AGRICULTURAL INNOVATION PROGRAM FINAL EVALUATION MARCH 23, 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International, A Tetra Tech Company. AGRICULTURAL INNOVATION PROGRAM FINAL EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Agricultural Innovation Program – Final Evaluation iii ACKNOWLEDGMENTS This report was made possible with the efforts of a number of people. Jacques Berard, team leader, and Shahid Umar, assignment manager, are the primary authors of the report. Special thanks to the sector specialists, Muhammad Qasim, Rookh Khan, and Kazim Bukhari, for their contributions in the preparation of data collection instruments, data collection, and data analysis. Babur Wasim, statistics and evaluation specialist, and Muhammad Danish, data analysis/survey specialist, significantly contributed to planning and managing the evaluation, developing data collection instruments, data collection, data analysis, and report writing. The evaluation team is also grateful to the data analysts, Umar Yar Khan and Saad Rajput, for their contribution in both qualitative and quantitative data analysis. Last, but not least, the evaluation team is thankful to the respondents (farmers, members of research organizations, sector experts, implementing partners, and government officials) for their time and cooperation. Agricultural Innovation Program – Final Evaluation iv CONTENTS Acronyms...........................................................................................................................vi Program Summary ..........................................................................................................vii Executive Summary.......................................................................................................... 1 Evaluation Purpose and Questions.................................................................................................. 1 Summary of Findings and Conclusions........................................................................................... 1 Evaluation Purpose and Questions.................................................................................. 3 Evaluation Questions.......................................................................................................................... 3 Program Background........................................................................................................ 3 Theory of Change................................................................................................................................ 5 Evaluation Methods and Limitations............................................................................... 5 Evaluation Methods............................................................................................................................. 5 Evaluation Limitations......................................................................................................................... 7 Findings and Conclusions.................................................................................................. 7 Question 1: Revitalizing Research and Innovation....................................................................... 7 Question 2: Building Research and Innovation Capacity..........................................................15 Question 3: Effectiveness of Program Activities........................................................................20 Question 4: Changes in Knowledge, Skills, and Attitudes.......................................................29 Concluding Remarks ....................................................................................................... 31 Annex 1: Statement of Work......................................................................................... 33 Annex 2: Assignment Work Plan .................................................................................. 45 Annex 3: Results Framework......................................................................................... 69 Annex 4: Data Collection Instruments ......................................................................... 70 Annex 5: Beneficiary Data and Online Surveys.......................................................... 101 Annex 6: Bibliography................................................................................................... 103 Annex 7: Sampling ........................................................................................................ 105 Agricultural Innovation Program – Final Evaluation v List of Tables Table 1: Program Summary.............................................................................................................................vii Table 2: Distribution of Beneficiaries by Type ........................................................................................101 Table 3: Number of Beneficiaries with CNIC and Contact Information ..........................................101 Table 4: Group Interviews – Research Organizations by Sector........................................................105 Table 5: Group Interviews – By Gender...................................................................................................105 Table 6: Group Interviews – Respondent Type by Gender.................................................................105 Table 7: Group Interviews – Research Organizations by Province....................................................105 Table 8: Group Interviews – Beneficiary Farmers..................................................................................105 Table 9: Group Interviews – Other Trainees..........................................................................................105 Table 10: Key Informant Interviews...........................................................................................................106 Table 11: Individual Survey Respondents by Sector...............................................................................107 List of Figures Figure 1: Map of Program Activities.............................................................................................................viii Figure 2: Program Architecture ...................................................................................................................... 4 Figure 3: Did AIP Contribute to Revitalizing Research and Innovation?............................................... 8 Figure 4: Germplasm Research and Innovation........................................................................................... 9 Figure 5: New Technologies and Practices Tested and Introduced....................................................... 9 Figure 6: Number of Research Proposals Submitted...............................................................................10 Figure 7: Factors Contributing to Revitalization of Research and Innovation ...................................11 Figure 8: Proportion of Respondents Reporting that AIP Built Organizational Research Capacity ........................................................................................................................................15 Figure 9: Indicators of Research Activity ....................................................................................................16 Figure 10: Factors Contributing to Capacity Building..............................................................................17 Figure 11: Effectiveness of AIP Activity Implementation.........................................................................20 Figure 12: Satisfaction with Training ............................................................................................................21 Figure 13: Strengths of AIP’s Training Approach......................................................................................22 Figure 14: Most Successful Training Delivery Mechanisms.....................................................................23 Figure 15: Factors Contributing to Effective Implementation................................................................24 Figure 16: Needs Assessed and Met by Sector.........................................................................................25 Figure 17: Effectiveness of AIP Training by Sector...................................................................................29 Figure 18: Effectiveness of AIP Training by Type of Trainee..................................................................30 Figure 19: AIP Results Framework ...............................................................................................................69 Figure 20: Individual Survey Sampling.........................................................................................................107 Agricultural Innovation Program – Final Evaluation vi ACRONYMS AIP Agricultural Innovation Program AVRDC Asian Vegetable Research and Development Center (World Vegetable Center) CGIAR Consultative Group for International Agricultural Research CIMMYT Centro Internacional de Mejoramiento de Maíz y Trigo (International Maize and Wheat Improvement Center) EGA Economic Growth and Agriculture FSC Farm Services Center GoP Government of Pakistan ILRI International Livestock Research Institute IP Implementing Partner IR Intermediate Result IRRI International Rice Research Institute KP Khyber Pakhtunkhwa MNFSR Ministry of National Food Security and Research MSI Management Systems International NARC National Agriculture Research Center OAPA Office of Afghanistan and Pakistan Affairs (USAID) PARC Pakistan Agricultural Research Council PIO Public International Organization UC University of California USAID United States Agency for International Development Agricultural Innovation Program – Final Evaluation vii PROGRAM SUMMARY Table 1 summarizes basic information about the Agricultural Innovation Program (AIP). TABLE 1: PROGRAM SUMMARY Title/Field Program Information Contract/agreement numbers PIO grant # AID-BFS-G-11-00002 Contracting/agreement officer’s representative (COR/AOR) Start date March 8, 2012 Completion date March 7, 2017 (extended to March 2018) Location Whole of Pakistan Implementing partner(s) International Maize and Wheat Improvement Center (CIMMYT) USAID/Pakistan Mission Strategic Framework objectives addressed • DO 2: Improved economic status of focus populations and sectors • IR 2.1: Improved economic performance of focus enterprises • IR 2.1.2: Improved capacity of workforce • IR 2.1.3: Increased use of modern technology and management practices • IR 2.2: Improved business enabling environment Budget $ Agricultural Innovation Program – Final Evaluation viii The map in Figure 1 illustrates the areas in which AIP operates and the agricultural subsectors it supports in each area. FIGURE 1: MAP OF PROGRAM ACTIVITIES Agricultural Innovation Program – Final Evaluation 1 EXECUTIVE SUMMARY The Agricultural Innovation Program (AIP), funded by the United States Agency for International Development (USAID) in Pakistan, aims to increase agricultural productivity and the income of farmers in four sectors (cereals, livestock, vegetables, and horticulture) by increasing the use of modern technology and management practices, improving the performance of value chains, and increasing the capacity of the public and private sectors to support the agricultural production system. Evaluation Purpose and Questions The four evaluation questions focus on various aspects of the program’s effectiveness. Answers to these questions will help program stakeholders such as the Government of Pakistan (GoP) and the AIP implementing partners (IPs) refine implementation. USAID/Pakistan will incorporate lessons learned into ongoing and future Economic Growth and Agriculture (EGA) activities. 1. To what extent has AIP contributed to revitalizing research and innovation in its focus subsectors? 2. To what extent has AIP’s collaboration with the public and private sectors built the capacity of partner institutions in research and development? 3. According to public and private sector partners of AIP, how effective was the implementation of project activities? What can be improved? 4. To what extent did AIP produce any change in knowledge, skills, and attitudes in its trainees? Summary of Findings and Conclusions Question 1: A large majority of interview respondents believed that AIP has contributed to revitalizing agriculture research and innovation, attributable mainly to introducing new practices and technologies. The provision of inputs such as genetic material and equipment (technology) was seminal in enabling cash-starved private and public research institutions to conduct applied research. Consequently, records of recipient organizations show sharp increases in the numbers of seed lines, practices, and proposals on which they are working. Thus, from the point of view of respondents, and according to hard indicators, AIP revitalized agricultural research and innovation. AIP’s success can be attributed to exposing Pakistani researchers to technologies and practices that are new to Pakistan. Providing Pakistani researchers with imported germplasm and equipment has shortened the time it takes to get products to market. However, as these basic inputs and technical assistance begin to bear fruit, funding constraints may jeopardize current and future activities. AIP’s termination raises concerns about the sustainability of what has been achieved, and even more about the wider dissemination after replication and multiplication. As the input provision period of AIP is nearing its end and the last trials are underway, the challenge now lies in disseminating the work to the majority of farmers, beyond the “innovators” and “early adopters.” Thus, future AIP activities should focus on completing all unfinished trials and adaptations and multiplying and more widely disseminating inputs (germplasm and technologies). Question 2: AIP has been successful at building the research capacity of partner institutions. Overall, 83 percent of interview respondents reported that AIP built the partners’ capacity in research and development. AIP’s technical training, supplying inputs, and developing linkages between individuals and organizations are the main contributors to capacity building. The program’s strategy was to rely on long- Agricultural Innovation Program – Final Evaluation 2 term “multiplier effects” of one-off technical training courses, which build the technical capacities of individuals, expecting that these individuals’ capacities will eventually spill over into the organization. AIP is a magnet using its funding power as a source of attraction, but USAID should work on reinforcing linkages and coordination between AIP and other projects that are working on improving the legislative framework and streamlining the procedural steps to bring innovations (notably germplasm) to market. Question 3: Roughly two-thirds of interview respondents thought that AIP’s implementation was effective and were particularly appreciative of the training. To the extent that AIP assessed beneficiaries’ needs, it was successful because it responded to the technical needs of organizations. While AIP’s implementation was effective overall, a relatively small number of respondents mentioned barriers to implementation including (1) unclear and ineffective marketing and dissemination strategies and (2) poor coordination with stakeholders, largely related to timely funding of activities. The evaluation team concludes that AIP’s implementation has been effective and that the effect of the above-mentioned shortcomings can be mitigated by having a clearer dissemination strategy, more visible marketing actions, and improved coordination between stakeholders. To enhance prospects for sustainability after AIP, USAID should communicate to IPs and the GoP its transition strategy, which should include a formal scaling-up plan. To increase uptake and bolster the dissemination of its results, AIP could consider locating demonstration plots more strategically to enhance visibility. Finally, IPs should explore ways to expand the role of extension workers in extending the use of AIP-supported germplasm and technologies to more remote areas of the provinces. Question 4: Eighty-one percent of survey respondents said that AIP assessed their individual and organizations’ needs. Although AIP did not conduct formal needs assessments, the program consulted with sector experts from all provinces to determine training priorities. An overwhelming majority of the survey respondents who participated in AIP training courses said that the training significantly or moderately increased their knowledge relevant to their research work; that they used the knowledge and skills often in their work; that the training had changed their research practices; and that the practices had become a routine part of their work. Factors that contributed to changing knowledge, skills, and attitudes included exposure to new technologies and practices, the usefulness of the information, technical assistance, and practical demonstrations. The short duration of some training courses, mostly in horticulture, limited their influence on changing knowledge, skills, and attitudes. In conclusion, respondents generally judged AIP a success in revitalizing agricultural research and innovation; building research capacity; and changing knowledge, skills, and attitudes. The bases for further dissemination are laid, but AIP is not ready to see its products and services reach the majority of farmers until and unless it gives further thought to formally scaling up results within the framework of a clear transition strategy toward the next phase of AIP-related programming. The evaluation team, based on feedback from stakeholders, recommends that local-language speakers be used more often to deliver training courses, and that training of trainers be conducted with extension workers to deploy need￾based training sessions to farmers. Finally, the team recommends increasing the frequency of training courses and ensuring that, after the courses, trainers do more frequent on-site follow-up visits with farmers. Overall, bearing in mind the methodological limitations of this evaluation, notably the possibly non￾generalizable character of the findings, stakeholders see AIP as being largely successful in revitalizing research and innovation and in changing the behaviors of trainees. Agricultural Innovation Program – Final Evaluation 3 EVALUATION PURPOSE AND QUESTIONS The evaluation assessed the extent to which, and how, AIP has built the agricultural research capacity of partner public and private research institutions; contributed to the diffusion of agricultural practices and technologies; and enhanced the knowledge, skills, attitudes, and behaviors of researchers, farmers, and others with whom it has engaged. USAID/Pakistan will incorporate lessons learned into ongoing and future EGA activities. The primary audiences for the evaluation include: • The USAID/Pakistan Mission, particularly the EGA team; • The USAID Office of Afghanistan and Pakistan Affairs (OAPA); • The IP, the International Maize and Wheat Improvement Center (CIMMYT); and • The GoP. Evaluation Questions The evaluation statement of work (Annex 1: Statement of Work) specified the following four evaluation questions, which are further elaborated in the assignment work plan (Annex 2: Assignment Work Plan). 1. To what extent has AIP contributed to revitalizing research and innovation in its focus subsectors? 2. To what extent has AIP’s collaboration with the public and private sectors built the capacity of partner institutions in research and development? 3. According to public and private sector partners of AIP, how effective were the implementation of project activities? What can be improved? 4. To what extent did AIP produce any change in knowledge, skills, and attitudes in its trainees? PROGRAM BACKGROUND AIP seeks to increase the agricultural productivity and incomes of farm households by improving the capacity of agricultural researchers to develop and disseminate productivity-enhancing technologies and practices and by strengthening value chains, among other interventions. The program focuses on four sectors: cereals and cereal systems (wheat, maize, and rice), livestock, perennial horticulture, and vegetables. Across all four sectors, the program also focuses on human resource development. Figure 2 illustrates the program architecture: the six research and development “projects” and the activities under each. During the program’s first five years, its primary activities have included: • Introducing improved seed/planting/genetic materials (e.g., wheat, maize, maize hybrids, vegetables, fruits, and rice) to public and private sector research organizations. The program anticipated that in the short term, the research organizations would test the material for applicability in different regions of Pakistan and then multiply and disseminate promising varieties. In the longer term, the genetic material should contribute to developing new seed varieties suitable to environments within Pakistan. Agricultural Innovation Program – Final Evaluation 4 • Introducing, testing, adapting, and disseminating innovative machinery and practices (e.g., Happy Seeder, two-wheel tractor, rice harvester, mung bean harvester, and protected cultivation) to improve agricultural productivity. • Strengthening the country’s human resource capacity for applied and adaptive agricultural research through vocational training (of members of research organizations, among others), field days, workshops, and support to students earning graduate degrees. • Establishing provincial agricultural research boards to manage a small competitive grants activity to support research. Project management is vested in a consortium of centers of the Consultative Group for International Agricultural Research (CGIAR); 1 the University of California, Davis (UC Davis); and the World Vegetable Center (AVRDC). CIMMYT is the overall lead and the technical lead on wheat, maize, and agronomy, while the International Livestock Research Institute (ILRI) leads livestock and dairy activities, the International Rice Research Institute (IRRI) leads rice activities, AVRDC manages vegetables, and UC Davis leads perennial horticulture. The Pakistan Agricultural Research Council (PARC) serves as both the hosting partner and the lead on the competitive grant system. CGIAR and USAID signed a contribution agreement in 2011 (amended in 2012) under which AIP was implemented in 2013. After a startup and priority-setting phase, AIP began implementation in earnest in 2015. FIGURE 2: PROGRAM ARCHITECTURE In 2011-12, USAID/Pakistan began discussions with CIMMYT and PARC to design a new agriculture project. The discussions culminated in a consultation with national stakeholders that identified the priority areas on which the project would work. CIMMYT subsequently identified partners to work in 1 CGIAR centers in the consortium are the International Maize and Wheat Improvement Center (CIMMYT), the International Livestock Research Institute (ILRI), and the International Rice Research Institute (IRRI). Agricultural Innovation Program – Final Evaluation 5 each area and engaged CGIAR consortium partners ILRI and IRRI to implement activities in livestock and rice, respectively, and non-CGIAR partners AVRDC and UC Davis to work on vegetables and horticulture, respectively. USAID/Pakistan engaged CIMMYT through the window three CGIAR fund, with the funds flowing through the World Bank to CIMMYT which then transferred funds to the other AIP partners, including PARC, on an annual basis. This mechanism allowed USAID to contribute to project design and to approve work plans. Although the mechanism does not require implementing partners to submit financial or progress reports, CIMMYT agreed to provide semi-annual progress reports at the request of USAID/Pakistan. Theory of Change The program’s theory of change posits that: • IF the capacity of public and private sector agricultural research institutions to develop and disseminate productivity-enhancing agricultural technologies and practices is improved; • AND these technologies and practices are disseminated to farmers, along with the information and skills to put them into practice; • AND producers are better linked to markets; • THEN agricultural productivity and household income will increase. The results framework reproduced in Annex 3: Results Framework further elaborates the theory of change. EVALUATION METHODS AND LIMITATIONS Evaluation Methods The evaluation employed a mixed-methods approach that included collecting quantitative and qualitative data from multiple sources (i.e., project documents, beneficiaries, stakeholders, and experts). 2 The evaluation team surveyed public and private research organizations; conducted group interviews with farmers, members of research organizations, extension workers, and other trainees such as teachers and students; and interviewed key informants such as heads of research organizations and various stakeholders. Annex 4: Data Collection Instruments contains the data collection instruments. Survey of Research Organizations The evaluation team conducted a structured online survey of all 92 AIP-supported research organizations and their members (Annex 4: Data Collection Instruments contains the survey instruments). The survey of heads of organizations collected organizational-level data on the organizations’ research capacity and interaction with AIP. The survey of organizations’ members also 2 The original statement of work suggested using outcome mapping to answer Question 4. This technique is best suited as a developmental evaluation tool to evaluate complex projects, generally bearing unclear objectives and characterized by an evolving package of activities applied in attempts to best impact behavior. It calls for a high intensity of interaction with stakeholders and is time-consuming. It also presumes that stakeholders have systematically documented (typically with ongoing project journals) changes related to the desired outcomes (behavior changes) of interest. For these reasons, the evaluation team discarded outcome mapping as a pertinent technique. Agricultural Innovation Program – Final Evaluation 6 asked broadly about organizational research capacity but focused mostly on individuals’ perceptions of the effectiveness of training with respect to building knowledge and skills and changing attitudes. From data provided by CIMMYT, the evaluation team identified the heads of the 92 research organizations that AIP reports supporting the most intensively3 and 532 members of those organizations who participated in AIP activities. The evaluation team invited the heads and members of these organizations to respond to the online survey. After repeated telephone reminders, 273 organization members (168 from the cereal subsector, 33 from livestock, 37 from vegetables, and 35 from perennial horticulture) responded to the survey for a response rate of 51 percent. Of these, 19 were women. Thirty-six organization heads responded to the survey, a response rate of 39 percent, which is relatively high for an online survey. Annex 5: Beneficiary Data and Online Surveys provides more detail on the surveys. Key Informant Interviews The team conducted 50 semi-structured interviews with key informants associated with the program and external sector experts who could provide feedback on the program’s relevance to agricultural development and its design and implementation. These interviews focused on developing a comprehensive understanding of the relevance and effectiveness of AIP in terms of revitalizing research and innovation; implementing program activities; and influencing the knowledge, skills, and attitudes of trainees. The evaluation team purposively selected stakeholders and experts to capture specific expertise, experience, or perspectives. Stakeholders included representatives of USAID, IPs, donor agencies, the National Agriculture Research Center (NARC), PARC, and the Ministry of National Food Security and Research (MNFSR), as well as officials from four provincial governments (see Annex 7: Sampling). Group Interviews The evaluation team conducted 65 group interviews with randomly selected4 beneficiary groups including farmers (37 groups), members of research organizations (13 groups), extension workers (7 groups), and other beneficiaries such as students and teachers (8 groups). The team conducted the group interviews in a sample of 15 districts in all 4 provinces and the Islamabad Capital Territory. It selected the districts based on the criteria of including different agro-climatic zones, various levels of maturity and concentration of AIP activities, and all AIP sectors and subsectors. The team also based the sample selection on accessibility (security). For details on group interviews, see Annex 7: Sampling. Secondary Data The evaluation team collected secondary data from information maintained by AIP (see Annex 6: Bibliography for a bibliography). This included program documents and program-supported research reports. The team also used beneficiary data provided by CIMMYT to assess the program’s reach and select samples for interviews (Annex 7: Sampling). 3 AIP reported working with 165 organizations including 7 non-governmental organizations, 17 universities, 28 private sector organizations, and 113 public sector research organizations. The 92 organizations that the evaluation surveyed were those that received grants to build their research capacity and could thus answer questions about this type of AIP support. 4 The team used a combination of random and convenience sampling to select group interview participants. In the first stage of sampling, the team randomly selected 10 beneficiaries for each group interview. In the second stage, it selected four respondents for convenience based on their availability and the team’s ability to contact the selected participants. Agricultural Innovation Program – Final Evaluation 7 Data Analysis The surveys yielded quantitative information regarding revitalization of research and innovation; capacity of research organizations; relevance and implementation of program activities; and changes in knowledge, skills, and attitudes of trainees. The analysis used descriptive statistics (frequencies, averages, and cross-tabulations) of the quantitative data to describe beneficiary characteristics and perceptions by subsector. When applicable, it used relevant statistical tests, such as the chi-squared and t-tests, to determine the statistical significance of observed differences between various levels of disaggregation, e.g., sector or type of respondent. The qualitative data from interviews helped explain how and why program activities worked or failed to work. The team used MAXQDA, a specialized software package, to organize, code, and identify patterns in the qualitative data. The analysis used an explanatory approach to integrate quantitative and qualitative data; qualitative findings helped explain trends and findings in the quantitative data. The integration was designed to triangulate the quantitative with the qualitative data and help explain how and why program activities worked—or failed to work—as expected. Evaluation Limitations Notwithstanding the efforts of the evaluation team to obtain valid and reliable data, the evaluation has the following limitations. • Selection bias: The evaluation team surveyed all heads and some members of 92 research organizations and randomly selected farmers, members of research organizations, extension workers, and other beneficiaries for group interviews. However, those who chose to respond to online surveys or agreed to participate in interviews may have been systematically different from those who did not respond or participate. This creates the potential for self-selection bias, although it is not practical to determine the magnitude or direction of any resulting bias. The team’s inability to contact many of the individuals in the beneficiary sampling frame because of incomplete contact information may also have introduced some bias, e.g., toward (larger) farmers with more reliable contact information. The evaluation design mitigated these potential sources of bias by triangulating findings from multiple sources. • Attribution: Rigorous attribution of outcomes to AIP activities was not possible because the program was not designed to accommodate impact evaluation. Instead, the evaluation team collected data to develop plausible explanations of AIP’s contribution to outcomes. FINDINGS AND CONCLUSIONS Question 1: Revitalizing Research and Innovation To what extent has AIP contributed to revitalizing research and innovation in its focus subsectors? Question 1 examines the extent to which, and the mechanisms by which, AIP partners adapted or adopted productivity-enhancing technologies and practices. It also explores the role of AIP in enhancing the capacities of public sector institutions to conduct research and disseminate results, and their intention and ability to continue doing so after AIP ends. Agricultural Innovation Program – Final Evaluation 8 Findings Overall, 90 percent of 123 respondents in key informant and group interviews believed that AIP introduced new technologies and practices to the agricultural research community (Figure 3). From that perspective, AIP has been successful in revitalizing agricultural research and innovation in Pakistan. Experts, all of whom were familiar with AIP, were somewhat less convinced of the program’s contribution than were other types of respondents, citing issues related to the researchers’ skills and overall coordination. For a discussion of the reasons for this discrepancy, see the “Factors Inhibiting Revitalization” section below. FIGURE 3: DID AIP CONTRIBUTE TO REVITALIZING RESEARCH AND INNOVATION? 5 Source: PERFORM key informant and group interviews. Quantitative data from the small survey of heads of AIP-supported research organizations corroborated the qualitative results reported in Figure 3, i.e., that AIP contributed to revitalizing research. The data show a 46 percent increase between 2013 and 2016 in the volume of new seed lines on which organizations reported they were conducting tests, and a 121 percent increase in the number of seed lines they had in the commercialization pipeline (Figure 4). The growth in both indicators coincided with AIP’s implementation trajectory, peaking around 2015 when AIP was fully implemented, suggesting that distributing genetic material contributed to increased research activity. The flat or downward trends beginning in 2016 may, at least in part, reflect the fact that the 2017 numbers cover only the first eight months of the calendar year. Extrapolating from the first 8 months suggests that the number of seed lines in the pipeline may have been as high as 44 and the number of lines tested as high as 874.6 CIMMYT further explained that as lines provided by AIP were tested, and less promising lines discarded, the number of lines on which an organization was working would naturally decline over time. 5 The interview guides did not ask directly about “revitalization.” The data in this figure represent the percentage of respondents who mentioned, during the interviews, that AIP contributed to enhancing some aspect of research and innovation capacity or practice. 6 The accuracy of extrapolated numbers depends on strong assumptions about the linearity of results over the course of the calendar year. Agricultural Innovation Program – Final Evaluation 9 FIGURE 4: GERMPLASM RESEARCH AND INNOVATION Source: PERFORM survey of beneficiary research organizations. The survey data also show a marked increase in the number of new practices and technologies (the latter being mostly in the form of machinery and, in the livestock sector, vaccines and artificial insemination products) on which AIP-supported organizations worked (Figure 5). The data show little growth after 2016 in the number of new technologies tested, perhaps because there are only so many types of technological innovations (e.g., zero till seeders, drip irrigation systems, and animal shelters) available and, more importantly, relevant to Pakistan. Therefore, once the technologies were tested and adapted, they required no additional work from the research organizations. Figure 5, like Figure 4, extrapolates earlier trends to the entire 2017 calendar year, illustrating that the number of new practices worked upon in 2017 might reach 84 and the number of technologies could be 17. The (extrapolated) 2017 upward trend in the introduction of new practices is consistent with the idea that practices follow the introduction of other inputs with which they are associated. FIGURE 5: NEW TECHNOLOGIES AND PRACTICES TESTED AND INTRODUCED Source: PERFORM survey of beneficiary research organizations. Agricultural Innovation Program – Final Evaluation 10 Following the program’s startup and priority-setting period (2013–2014), supported organizations submitted a much greater number of proposals, a clear sign of increased interest in research (Figure 6). Extrapolating to the entire 2017 calendar year suggests that the number of proposals submitted in 2017 could be as high as 17. This excludes the proposals that PARC, based on recent submissions from the provinces, was in the process of reviewing for award at the time this report was written. CIMMYT explained that the downturn in 2017 may reflect that the eight months of data available in 2017 excludes the end of the year when research centers and universities normally submit proposals. FIGURE 6: NUMBER OF RESEARCH PROPOSALS SUBMITTED Source: PERFORM survey of beneficiary research organizations. Factors Contributing to the Revitalization of Research and Innovation When asked how AIP contributed to revitalizing research and innovation, key informant and group interview respondents identified the introduction of new practices, training and exposure visits, provision of seeds and semen, and the technical assistance that came with the inputs. Figure 7 illustrates the percentage of responses associated with each of these themes, presenting only the most frequently mentioned factors. Respondents explained that the introduction of new practices was important because AIP, through its international network, introduced Pakistani scientists to hitherto unknown or unavailable international practices—for example, to diagnose wheat diseases: “CIMMYT-supported collaborations with international experts enhanced the capacity of our scientists. We developed linkages with international research centers through participation in international trainings. … For example, as a result of CIMMYT-supported training, I learned how to correctly identify the rust on wheat. In the past, sometimes we were not able to correctly diagnose the wheat disease [rust].” – Member of research organization The seeds and technologies that AIP introduced, as well as training in research methods, allowed scientists to apply their skills to testing and screening new seed lines and to adapting imported technologies (e.g., land leveling machinery, drill sowing, livestock shading, tunnel cultivation, and drip irrigation) to work in Pakistan. “AIP-supported trainings increased our research capacity. … I attended a training on research methods in Kenya … that was the most effective activity for my personal capacity building. In Agricultural Innovation Program – Final Evaluation 11 this training, I learned how to categorize the varieties and collect genetic data for evaluation purposes and conduct research on seed varieties.” – Member of research organization By importing into Pakistan inputs that had already been tried and marketed elsewhere, AIP contributed to revitalizing research and innovation by expanding the pool of genetic material on which researchers could work and by shortening, from 10 to 3, the number of years typically required to certify and register a crop variety. Similarly, according to a member of a public sector research organization, beginning with a proven technology rather than starting from scratch has also considerably reduced the amount of time it takes to adapt and bring to market a piece of equipment or other technology. FIGURE 7: FACTORS CONTRIBUTING TO REVITALIZATION OF RESEARCH AND INNOVATION Source: PERFORM key informant and group interviews. The interviews allowed multiple responses to this question, so totals do not necessarily sum to 100 percent. When disaggregated by sector, the responses show that the livestock, vegetable, and horticulture sectors benefited most from the introduction of new practices through training. With respect to beneficial practices, respondents most frequently mentioned drip irrigation and tunnel farming, insect nets, bags for seed preservation, laser land leveling, sowing wheat on ridges, effective use of fertilizers, artificial insemination in goats, and proper feeding and watering of livestock. The Happy Seeder, laser land leveler, push row planter, and GreenSeeker emerged as technologies that respondents found useful. The practical demonstrations they found most effective included ridge sowing, laser leveling, onion seed production, guava food processing, pruning in guava, preparation of animal feed, construction of sheds for small ruminants, and artificial insemination. Agricultural Innovation Program – Final Evaluation 12 But the provision of more than 1,000 seed lines in the cereal sector is the most important reason for the program’s success. This represents the highest volume of inputs in any sector and covers the crops—wheat and maize—on which CIMMYT concentrated its work. The evaluation team concludes that the provision of seeds in the cereal sector was seminal, essential, and necessary in revitalizing research and innovation and was AIP’s primary contribution to that end. The initial provision of inputs such as seeds—and the practices that accompany their use—remains the linchpin for revitalization of research and innovation, essential for technological advances, new practices, technical assistance, 7 and training. The sudden departure of UC Davis and, to a lesser extent, AVRDC, cut short efforts to finalize the testing of horticultural practices that would not bear fruit for 10 years or, in vegetables, that would require complementary measures (e.g., fertilization, pesticides, and soil testing). Factors Inhibiting Revitalization While few interview participants (17 percent) mentioned factors that may have limited AIP’s contribution to revitalizing agricultural research and innovation, their perceptions point to potential improvements in the design of similar USAID activities. Some of the factors do not lie within AIP’s manageable interest but can nevertheless inform future similar programming. Key informant and group interview respondents representing heads and members of research organizations, government officials, trainers, and trainees most frequently mentioned current and prospective funding constraints. AIP was fully operational for about 30 months after its initial two￾year priority-setting exercise. Yet, some respondents noted that the funding required to carry out trials and to finish those currently underway is already limited. Furthermore, those who noted that AIP is ending were concerned that it could not consolidate and deepen its gains without additional funding. Twenty-six percent of 35 responses (4 responses from members of research organizations, 2 each from IPs and experts, and 1 from a trainer) noted that government research organizations’ budgets are not sufficient to sustain the gains achieved through AIP (Question 3 addresses this issue in greater detail). “Our short-term needs in terms of operational expenses and limited staff training were fulfilled, but long-term institutional needs [funding] were not.” – Senior government official “We already had the genetic material but we did not have enough access to farmers to disseminate the genetic material and knowledge due to the lack of money.” – Head of research organization Funding constraints may also pose a greater threat than research skills to prospects for sustainability. In individual and group interviews, heads and members of Pakistani public and private sector institutions with which AIP worked reported that institution staff were qualified researchers. Researchers in these institutions explained that they possessed the requisite skills but, until AIP, lacked the raw materials (e.g., seeds and equipment) and especially funding that AIP provided to apply their skills. “Under the government structure, we always faced a shortage of funds for conducting research￾related activities. … We have no technical issue; the major problem is scarcity of funds. … Technically, I didn’t need any learning from AIP, but the major issue was funding.” – Member of research organization, Quetta 7 Technical assistance is defined as distinct from training. It consists of identifying agricultural needs in Pakistan and matching those with inputs, technologies, and practices that are available to AIP IPs but not previously available in Pakistan. It involves the preparation and assistance to import these various inputs, as well as knowledge and technological transfer that may occur between international and national experts when, for example, internationally sourced seeds are imported and introduced to the laboratories of senior researchers. When national researchers work with these inputs, technical assistance may also involve collaborative work with international researchers to adapt the inputs to Pakistan. Agricultural Innovation Program – Final Evaluation 13 Experts and IP respondents, however, believed that the research institutions do not have enough staff with a high level of technical expertise—for example, with skills acquired through American-style higher education. To help address this knowledge and skills gap, AIP trained researchers and also supported graduate degree programs for 14 students, 9 for master’s degrees and 5 for doctorates. The key informant who spoke to this issue reported that the selection process was transparent. At the time of the evaluation at the end of August 2017, all nine master’s students had completed their degrees and returned to Pakistan, but none had found employment. The doctoral students were working on their research. Only 4 percent of key informants raised a final concern, but these were mostly (42 percent) experts, respondents who have a unique, perhaps more holistic, perspective on the value that AIP adds to revitalizing research and innovation and its repercussions on the rest of the system. When asked to reflect on the role and contribution of AIP in the landscape of Pakistan’s agricultural sector, these respondents explained that regulatory and other policy-related barriers limit AIP’s contribution to increasing agricultural productivity, notably those concerning certification and registration. A sector expert, for example, explained the issues in the following terms: “The seed registration process is cumbersome and complicated in Pakistan. To be eligible for award of certification by the Federal Seed Certification and Registration Department, the seed companies interested in the KP [Khyber Pakhtunkhwa] seed business are legally bound to test their seed in KP for two years in at least two of the four agro-ecological zones of the province. … This is something difficult and [expensive] for companies to do. [Under AIP] we also tried to identify some farmers who could produce the seeds and register them with the seed certification department but … farmers faced some issues … and our effort was not successful. So, I think we should develop some mechanism to facilitate the farmers in this regard and should make things easy for them.” – Sector expert Thus, even though AIP may have provided the inputs and financial impetus to spur growth in the volume of research and innovation activity, and its products are about to be more widely distributed to the market, the administrative machinery to ensure buyers that the products meet quality standards is not in place. These missing reforms in the regulatory environment may limit AIP’s contribution to Pakistan’s agricultural sector. Conclusions The data strongly suggest that AIP contributed to improving research capacity and practices in the research organizations with which it worked by distributing imported inputs and training researchers on the new practices with which those inputs are associated. However, the two to three years required to complete tests have not passed, and funding has nearly lapsed, or has already done so in horticulture and vegetables. In these circumstances, AIP has only begun disseminating the results of revitalized research and innovation to farmers and others along the value chain. The evaluation suggests that “innovators” and “early adopters” are the primary beneficiaries of AIP’s work to date; getting the majority of (mostly small) farmers to buy and properly use the inputs AIP provided remains a challenge. 8 At the time of the evaluation, most AIP-supported tests and trials have reached maturity, but some remain to be concluded, notably in vegetables and horticulture. AIP now has an opportunity to 8 The database from which the evaluation team selected farmers for interviews was missing a large amount of contact information. If larger farmers were more likely to be innovators and early adopters, and were also more likely to have accurate contact information, the perspectives and experiences of the sample of farmers from which the team collected data may not reflect the experience of all farmers who engaged with AIP. Agricultural Innovation Program – Final Evaluation 14 consolidate the early results achieved and focus on dissemination. The strategy of engaging mostly with more progressive farmers to demonstrate new technologies and practices is sound. The challenge is to scale up AIP’s results to the majority of farmers. Given that not enough time has elapsed since 2015 to achieve economies of scale sufficient to make these inputs affordable to small farmers, AIP faces a challenge in crossing the chasm between richer, more progressive farmers and the rest of farmers. In short, and bearing in mind the limits to the data, the evaluation team concludes that although successful in what could be considered a first phase of revitalizing research and innovation, AIP was not in operation long enough to capitalize upon or consolidate the gains to which it contributed: changing technology and management practices, providing farmers with increased access to information, and improving the availability of technology. Identifying sources of funding to more widely disseminate new technologies and practices and continue on a trajectory of increased research activity will be crucial to consolidating AIP’s gains in revitalization of research and innovation and increasing agricultural productivity. Recommendations For USAID: To consolidate the results achieved so far, USAID should consider extending AIP funding with the explicit aims of: • Diverting funding to promising seed lines and technologies for which testing and adaptation are not finished. • Ensuring that next steps for completing work in horticulture are finished and the work more widely disseminated. • In preparation for the next phase of programming, developing a strategy to move innovations along the adoption trajectory from “innovators” and “early adopters” to the majority of farmers. Consider developing a separate project to scale up marketing of seed varieties, practices, and technologies at prices affordable to farmers. For IPs: • Complete an analysis of all seed lines and technologies upon which work remains to be done and focus funding on completing the work. Do not engage in new activities until the unfinished work is near completion and only fund new activities that can be completed in the remaining life of the project. • Identify and pursue a corrective course of action to provide further funding and technical assistance to the horticulture, vegetable, and—to a lesser extent—livestock sectors to reinforce the work done and establish the basis for further collaboration. • Develop and implement a marketing strategy, perhaps in collaboration with private (and public) sector partners, to disseminate the germplasm, practices, and technologies to the majority of farmers—i.e., move beyond demonstration plots to other means of placing the products in more remote areas of the provinces. • Identify and implement mechanisms to enhance subsistence farmers’ access to the inputs and practices hitherto available mostly to progressive—and often larger and more economically advantaged—farmers. Agricultural Innovation Program – Final Evaluation 15 Question 2: Building Research and Innovation Capacity To what extent has AIP’s collaboration with the public and private sectors built the capacity of partner institutions in research and development? This question examines the organizational capacity-building dimension of the program. AIP’s notion of building an organization’s research capacity is predicated on the hypothesis that the greater the number of individuals supported to work in their domain, the greater the capacity of the organization in which they serve. Findings Key informant and group interviews with sector experts, members of research organizations, extension workers, and IP representatives suggest that AIP has been successful at building individual researchers’ technical capacity to conduct research, and thus building the capacity of partner institutions (Figure 8). At the organizational level, increases in staff numbers and research activity and output reflect enhanced research capacity. Experts were more guarded than other types of respondents in their perspective on AIP’s capacity building. One reason for their divergent perspective is that experts did not see AIP contributing to establishing linkages within the broader interorganizational environment of relationships. FIGURE 8: PROPORTION OF RESPONDENTS REPORTING THAT AIP BUILT ORGANIZATIONAL RESEARCH CAPACITY Source: PERFORM key informant and group interviews. Figure 9 shows that as AIP finished its priority identification phase and began implementing activities in earnest in 2015, the number of researchers employed grew fivefold, while the number of research projects completed (a measure of organizational throughput capacity) grew by a factor of 10. If the 2017 trends were to continue over the course of the calendar year, up to 50 projects could be completed, and 69 researchers could be working on those. This does not include the researchers who might benefit from competitive grants after September–October 2017. These data, based on respondents’ records, corroborate the general perception highlighted in the survey that AIP succeeded in building the capacity of organizations in the sense of building the capacity of organizations’ researchers. Agricultural Innovation Program – Final Evaluation 16 FIGURE 9: INDICATORS OF RESEARCH ACTIVITY Source: PERFORM survey of beneficiary research organizations The extent to which AIP assessed and ultimately met beneficiaries’ needs is also an indicator of success. Overall, heads of organizations (86 percent) and the members of those organizations (88 percent) who responded to the survey thought that AIP properly assessed their organizations’ needs before starting to implement activities. Moreover, 84 percent of organization heads and 84 percent of members believed AIP’s assistance met their needs. Factors Contributing to Capacity Building Figure 10 summarizes the analysis of 594 coded excerpts from interviews that help explain how AIP support contributed to capacity building. Technical training, new seed varieties, and developing linkages between individual researchers and between institutions emerged as the most important contributors. Agricultural Innovation Program – Final Evaluation 17 FIGURE 10: FACTORS CONTRIBUTING TO CAPACITY BUILDING Source: PERFORM key informant and group interviews. The interviews allowed multiple responses to this question, so totals do not necessarily sum to 100 percent. Respondents spoke mostly of horizontal linkages (between individual researchers or between institutions working at the same level in the agricultural sector) but a few mentioned vertical linkages (from research to consumer). These linkages are important because they establish functional relationships between people and organizations that work along the entire value chain connecting agricultural research to production to markets, and they constitute a key element of institutionalization. “ILRI may serve as a bridge to connect livestock professionals with the farmers. Through my association with ILRI, I may be able to improve my skills and knowledge and transfer the same to livestock farmers for resolving their issues.” – Senior government official Members of research organizations in KP particularly found forums such as Farm Services Centers useful to establish vertical linkages and reach out to consumers. “In KP we have Farm Services Centers (FSCs) …. we link up the farmers and agriculture research institutes through FSCs to disseminate the inputs we received from AIP…. FSCs also played a role to develop the seed value chain…..Drip irrigation system is also another AIP￾supported intervention that was disseminated at the farm level through FSCs.” – Member of research organization Agricultural Innovation Program – Final Evaluation 18 One of the unintended benefits of AIP is that it created a pole of attraction for researchers and their institutions. Through its funding power and mere physical presence in Pakistan, AIP has grown a network of connections with itself at the center, and has also permitted the establishment of interconnections between the hitherto noncollaborating organizations that surround it as satellites. AIP also has offered a platform for CGIAR and non-CGIAR partners to work together under one umbrella and, in doing so, has established the foundations of a possible institutionalization of those relationships. “AIP brings all related agricultural technologies and expertise to one platform which brought positive change and enhanced the productivity of agriculture. AIP has basically provided a platform for researchers, scientists, the public sector, and private companies to collectively enhance the agriculture sector. … In other words, AIP worked as a hub for bringing people [scientists and researchers] to work together.” – Head of research organization and expert It is interesting to note that experts and IP respondents believed that developing linkages is the most important factor contributing to capacity building, whereas members of research organizations, students, and trainers identified technical training as the primary factor. This is understandable in light of the respective positions these respondents occupy within the constellation of organizations. Meanwhile, heads of research organizations emphasized the importance of money for facilities and supplies (e.g., laboratory equipment or work implements) or research grants, new genetic material, and the introduction of new research practices as key contributors to capacity building. The conclusion that emerges from these findings is that AIP’s key contributions to building the research and development capacity of partner institutions are money, genetic material, training for researchers, and establishing linkages. While AIP has contributed to establishing relationships within the agricultural research community, and begun to form horizontal linkages within the agricultural sector, some respondents expressed concern that once AIP shuts down, the tendency of agricultural research institutions to work in silos will once again prevail, and that the connections created between researchers of various organizations could be lost. A majority of heads of organizations (83 percent of 23 heads who responded to these survey questions) believed that AIP’s assistance—as delivered so far—produced lasting change in their organizations’ research practices. Even though this constitutes a subsample of a small sample, the responses showed no significant differences across sectors. Respondents from the horticulture sector provided no data for this question; this supports the finding, discussed under Question 4, that less than sustainable results may have been achieved in that sector. Factors Inhibiting Capacity Building PARC was charged with establishing provincial agricultural research boards, in part to better tailor technical assistance to the specific conditions prevailing at the provincial level. However, because of bureaucratic barriers that prevented PARC from establishing financial accounts to receive and disburse funds, this component did not fully materialize. Moreover, except in Punjab, which already had a board, the provincial boards were not established because the provincial assemblies did not pass the required legislation. Without the guidance of the provincial boards, even scientists’ efforts to build their individual capacities based on their research interests may have missed addressing localized needs. “We have the linkages, but they need to be strengthened because the international exposure and international research-related activities should mainly be undertaken at PARC, within the research system of the provincial government. If the linkages are strengthened, then the provincial government will also have access to new, innovative technologies.” – Senior government official Agricultural Innovation Program – Final Evaluation 19 Also, some program activities in vegetables and horticulture suffered from the untimely closure of the UC Davis and AVRDC components. While most respondents believed that establishing linkages was one of AIP’s most significant and important contributions to enhancing research capacity and outcomes, a few mentioned limitations with respect to linkages and coordination. The nine respondents who spoke to these issues believed that AIP needed to focus on building linkages along the entire value chain from research to markets in order to better connect beneficiary farmers with research and research outputs and, eventually, with markets. Two specifically noted that establishing public-private partnerships was crucial to scaling up and more widely disseminating research results to farmers. Conclusions Research organizations’ records of staff capacity, research activity, and throughput indicate that AIP successfully built the research capacity of organizations. It did so by reinforcing individuals’ research capacity by providing training, enhancing facilities, and supplying genetic material. By its mere presence in country, AIP contributed to establishing linkages between organizations. It acted as a hub, attracting researchers in quest of money or inputs, and brought together, for the first time in Pakistan, CGIAR and non-CGIAR partners to work together under one umbrella to address constraints in key subsectors of agriculture. However, perhaps because it had not developed or communicated an exit strategy, AIP did not manage to consolidate these one-off contacts. It did not leverage them to join or create self-sustaining networks of knowledge transfer that would support sustained change in organizational capacities in the absence of external funding. Because of the relative absence of government funding and coordination, there is little likelihood that such institutional contacts can be maintained without the presence of AIP’s IPs, the only viable source of research funding at the time of the evaluation. Recommendations For USAID and IPs: • Advocate with the government to better fund agricultural research and extension. Greater funding for research could extend and sustain the research-related gains of AIP. An adequately funded extension program will enable extension workers to be more effective partners to “socialize” the use of germplasm, technologies, and practices in more remote areas of the provinces. For IPs: • Use a more participatory approach to determining the needs of farmers in specific agro-climatic environments and deliver training and technical support accordingly. Support the GoP in extending the Model Farm Services Centers to regions other than KP to provide an easily accessible set of structures offering a platform to gather farmers’ inputs and ideas for improved training or technical assistance with germplasm and technologies in their areas. Agricultural Innovation Program – Final Evaluation 20 Question 3: Effectiveness of Program Activities According to public and private sector partners of AIP, how effective was the implementation of project activities? What can be improved? Findings The evaluation team assessed the effectiveness of AIP implementation approaches by gauging the extent to which they contributed to the intended outcomes on which the evaluation focused, i.e., revitalizing research and innovation, building the capacity of partner institutions, and enhancing the knowledge, skills, and attitudes of trainees. The interviews did not ask directly about “effectiveness” but, instead, asked whether an objective had been met (e.g., capacities enhanced) and then followed up with questions about “how” to explore linkages to AIP activities and approaches. This line of questioning also revealed implementation challenges that informed recommendations on how to improve performance. Fifty-eight percent of key informant and group interview respondents believed that AIP’s implementation was effective (Figure 11). This contrasts with findings for the other evaluation questions, where over 80 percent of respondents judged that revitalization occurred; that organizational capacity was built; or that AIP changed knowledge, attitudes, and skills. This is not necessarily inconsistent with other findings, however: respondents may be satisfied with the outcomes but still see room for improvement in the ways in which AIP achieved those outcomes. FIGURE 11: EFFECTIVENESS OF AIP ACTIVITY IMPLEMENTATION Source: PERFORM key informant and group interviews. Note: The evaluation team did not ask directly about problems with implementation. Instead, when asking about implementation, interviewers asked why implementation approaches were, or were not, effective. These numbers can be explained in terms of the factors that contributed to or inhibited effective implementation. Effectiveness of Training Implementation Along with introducing genetic material and technologies, training underpins AIP’s approach to revitalizing research and innovation. Because training differs from other activities in important ways, and Agricultural Innovation Program – Final Evaluation 21 because the evaluation team collected detailed information on training, this section focuses exclusively on implementation of training, while the next section addresses implementation of other program activities. A large majority of the 115 survey respondents9 who participated in training were satisfied with all aspects of the training (Figure 12). There was no statistically significant difference between men and women with respect to perceptions of training or other aspects of implementation. FIGURE 12: SATISFACTION WITH TRAINING Source: PERFORM survey of beneficiaries of AIP training. To elicit nuanced perceptions of how training contributed to changing knowledge, attitudes, and practices, the evaluation team asked trainees and students about their experiences with and perceptions of AIP’s training approach. Respondents most frequently mentioned exposure to new practices, the high quality and usefulness of the information and training, technical assistance,10 practical demonstrations, and exposure to new technologies as strengths of the training approach (Figure 13). 9 The number of respondents in Figure 12 is greater than 115 because some respondents represented multiple sectors. 10 Technical assistance is introducing and enabling the farmers to learn to use new and/or improved crop and livestock management equipment. This includes provision of new/improved machinery, equipment, utensils, etc., and related operational and maintenance skills. It excludes other training. Agricultural Innovation Program – Final Evaluation 22 FIGURE 13: STRENGTHS OF AIP’S TRAINING APPROACH Source: PERFORM key informant and group interviews. When asked why training was effective, individual and group interview respondents spoke of the efficacy of practical demonstrations, use of multimedia/pictorial materials, interactive sessions, and training in local languages (Figure 14). “In Thatta, nobody [has ever] succeeded in establishing a nursery due to the hot temperature, but we gave them the idea of a greenhouse shade net and they [farmers] saw the seedling production with their own eyes when we established a vegetable nursery there in the field. Now at least they have seen that they can establish a vegetable nursey by following the techniques we practiced in front of them. Previously they were bringing seedlings from Thailand, but now they are confident that they can establish a nursery for seedlings as well.” – Research organization member Agricultural Innovation Program – Final Evaluation 23 FIGURE 14: MOST SUCCESSFUL TRAINING DELIVERY MECHANISMS Source: PERFORM key informant and group interviews. The interviews allowed multiple responses to this question, so totals do not necessarily sum to 100 percent. Survey respondents noted, regardless of the frequency of training sessions, that their duration was insufficient. In key informant and group interviews, trainers, extension workers, and members of research organizations also noted that the length of individual training events was not sufficient (17 of 35 responses), which corroborated the survey responses. Farmers (11 responses) also mentioned that they needed more frequent follow-up sessions and practical demonstrations. “We usually design five-day agriculture-related training courses. But the people from CIMMYT visited us for half a day. They were coming for field observations but not frequently. Although we offered to let them stay in a safe and secure environment, they were coming only occasionally.” – Trainer and research organization member Factors Contributing to Effective Implementation of Other AIP Activities Figure 15 depicts the percentage of 390 responses from key informant and group interviews that mentioned specific factors that contributed to effective implementation of AIP activities. The qualitative data support the key finding of the survey: AIP was successful because it responded to the technical needs of organizations. Agricultural Innovation Program – Final Evaluation 24 FIGURE 15: FACTORS CONTRIBUTING TO EFFECTIVE IMPLEMENTATION Source: PERFORM key informant and group interviews. The interviews allowed multiple responses to this question, so totals do not necessarily sum to 100 percent. The factors listed in Figure 15 emphasize one strength of AIP’s implementation approach: that it identified and responded to recipients’ needs. Indeed, the new varieties of seeds that AIP provided were in response to expressed needs, and AIP also satisfied requests for more need-based activities that tracked with the crops’ schedules. The extent to which AIP assessed and met perceived organizational performance needs provides solid proxy indicators for respondents’ judgment about the program as a responsive, need-driven activity. While these perceptions varied across sectors, a majority of survey respondents in all sectors reported that AIP had assessed and met their needs (Figure 16). The somewhat lower percentages in the horticulture sector may reflect the early departure of UC Davis, well before the 5–10 years required for interventions to come to fruition. Agricultural Innovation Program – Final Evaluation 25 FIGURE 16: NEEDS ASSESSED AND MET BY SECTOR Source: PERFORM survey of beneficiaries of AIP training. Factors Inhibiting Effective Implementation of Other AIP Activities Relatively few key informant and group interview respondents mentioned suggestions or recommendations for improving program implementation.11 Among the 190 excerpts that noted issues with implementation, 35 percent mentioned issues with training (addressed above). Other issues respondents mentioned included: • Unclear and ineffective marketing and dissemination strategies (91 responses that included 29 from research organizations, 18 from experts, 12 from farmers, 9 from trainers, 8 from IPs, and 15 from other respondents). • Insufficient coordination (42 responses that included 13 from research organizations, 19 from experts, 5 from IPs, and 5 from other respondents), including: o Poor institutional linkages associated with inadequate coordination with stakeholders (13 of the responses that mentioned insufficient coordination), notably with the GoP (9 responses), and o IPs’ being insufficiently leveraged with the private sector (7 responses) and with extension workers’ (15 responses) and farmers’ interest groups (8 responses). Thus resulting in: • Insufficient targeting of interventions (29 responses that included 13 from research organizations, 11 from experts, 3 from IPs, and 2 from other respondents), in particular improper needs assessments and issues with funding. 11 The interviews did not ask directly about implementation. Instead, interviewers asked about the effectiveness of AIP activities, and of AIP overall, and then about reasons for activities’ being effective or ineffective. In the follow-on questions, interviewers probed specifically for implementation-related reasons for the effectiveness or ineffectiveness of activities. Agricultural Innovation Program – Final Evaluation 26 The rest of this section addresses each of these in detail. Dissemination and marketing: Respondents most frequently mentioned the underutilization of extension workers as a factor that limited AIP’s effectiveness in its role of disseminating technologies and practices to farmers. They believed that AIP should have provided extension workers with resources (e.g., funds and equipment) to work with farmers in their fields throughout the cropping season and to advise them as problems occurred. While respondents confirmed the efficacy of demonstration plots with progressive farmers as a dissemination strategy, some believed that there were not enough strategically located plots to exhibit results and thus encourage wide dissemination of promising seeds, practices, and technologies. Some also expressed concern that the relatively high cost of inputs and machinery constituted a barrier to widespread adoption of program-promoted technologies and practices, especially among small farmers. The private sector also wanted to be more involved in dissemination and marketing. Insufficient coordination: Although 73 percent of heads of organizations the team surveyed were satisfied with coordination, interviews with these heads, and with experts and senior government officials, revealed a more nuanced perspective. Among the 10 respondents, the most common coordination issue mentioned (by 4 respondents) related to timely release of funds to pursue research and dissemination, lapses that damaged relationships between implementers and farmers. One respondent from Balochistan noted the difficulty partners in locations far from Islamabad had in attending coordination meetings called on short notice, effectively excluding them from the decision￾making process. The absence of provincial agricultural research boards—except in Punjab—poses a problem in coordinating with extension departments at the district level. However, all respondents recognized that PARC potentially has a key role in coordinating work across and among provinces. Finding a way to enable PARC and its provincial delegations to manage funds may lay the foundation for a more coordinated, effective, and sustainable implementation of AIP activities. Ten percent of interview respondents (a majority of whom were heads of organizations) mentioned that expending more effort on developing linkages horizontally (between organizations at the same level) and vertically (between stakeholders at different levels of the research-inputs-production value chain) is likely to contribute to sustainable and visible change. Targeting of interventions: Analysis of the excerpts relevant to the targeting of AIP activities suggests that in some ways, AIP’s approach to identifying and addressing sector needs was not sufficient. “Projects should be designed to address the unmet needs of the sector; they should be need￾based. We can benefit from these projects only if the projects are designed and delivered according to the need of a specific region. Otherwise, we will not benefit from these projects [interventions].” – Head of organization, Faisalabad While this excerpt, and others like it, are not consistent with AIP’s mandate (i.e., AIP was not designed to meet every unmet need), it does illustrate a common theme that emerged from the interviews. The bureaucratic issues that prevented PARC from fully playing its anticipated role in provincial coordination, and the failure to establish the provincial boards, compromised AIP’s coordination at the provincial and district levels. During interviews, farmers and others often asked for additional interventions because the services and products AIP delivered did not always meet their localized needs. The evaluation team concluded that, while AIP largely met organizations’ needs, the needs identification process did not always reflect the range of perspectives of farmers. Since AIP did not incorporate “ordinary farmers” into the intervention delivery mode, it did not address some of their area-specific needs. Although working through progressive farmers is a tested method of dissemination, it was not Agricultural Innovation Program – Final Evaluation 27 enough in this instance, at least within the program time frame, to effectively “market” AIP interventions and promote uptake across the chasm from “early adopters” to “majority” users.12 Issues with funding: Funding was also an operational irritant that limited the effectiveness of AIP’s implementation. First, for some, there was not enough money to finish projects, let alone undertake new ones that beneficiaries saw as necessary. Members of research organizations and extension workers noted that after AIP support ends, they will not have the financial resources to continue expanding reach and supporting farmers (70 percent of 27 responses). Among those who addressed funding issues, 20 percent spoke of the need to streamline funding procedures so users, the majority of whom they expected to be subsistence farmers, can better access the next wave of AIP technology, and 17 percent called for some form of subsidization of small farmers. However, AIP was not designed to alleviate financial constraints through subsidization. “To make the interventions acceptable and get their buy-in, farmers should be given incentives such as free inputs and financial assistance. Later, when they will see improvement in production due to the intervention, they will adopt the new technologies and practices easily and their fellow farmers will follow them as well.” – Member of research organization Concerns about funding also arose in two other contexts: promoting dissemination and facilitating timely implementation of annual work plans. In the context of promoting dissemination and increasing uptake of AIP-supported products and services, not only did a few respondents call for subsidizing inputs to farmers, some also mentioned providing incentives for extension workers and scientists: “Unless we give a royalty to the private sector and allocate funds for research, we cannot develop our agriculture. Moreover, there is no incentive for a researcher who is doing research and innovation. There should be some reward system to encourage those who are working on research and innovation.” – Head of research organization Conclusions Of 164 interview respondents, 58 percent thought AIP’s implementation was effective. Analyzing the “effectiveness” of implementation from the point of view of the annual work plans and the semiannual monitoring reports AIP submits to USAID, the evaluation team concluded that the program effectively implemented its planned activities. The plans and reports emphasized building the technical capacity of individuals through training, supplying inputs, and supporting research projects. Although there are instances of “broken commitments” related to budget cutbacks for UC Davis and AVRDC, AIP largely delivered on its planned activities. Training was also effective, with the quality of the information, skills of the trainers, practical demonstrations, and appropriate and high-quality materials contributing to its effectiveness. The effectiveness of training increases when extension workers conduct the training in local languages, and when training accompanies the provision of inputs and reinforces their proper use, i.e., practical demonstrations. Although few respondents spoke of problems, their experiences and perceptions may point to ways to improve implementation. The evidence suggests that the project’s inability, due to limited government resources, to fully utilize extension workers may have compromised the dissemination of information and training on the use of 12 The conclusion that AIP did not effectively engage small farmers may be biased because the evaluation team found it difficult to identify and collect data from small farmer beneficiaries. Agricultural Innovation Program – Final Evaluation 28 new technologies to small farmers. Limited evidence also suggests that, to disseminate information more effectively, the program could modify its approach of working intensively with progressive farmers and demonstration plots to locate plots in more widely visible locations. The findings also lead the team to conclude that if provincial authorities, private sector actors, and farmers’ interest groups were more involved in AIP decision-making on who gets what, when, and where to produce what result, AIP could better tailor its interventions to match the needs of a wider range of beneficiaries. AIP has just finished providing inputs and is now ready to consolidate its technical contribution (i.e., finish trials and adaptions of machinery/technology and practices) and seek opportunities to scale up dissemination of improved varieties, technologies, and practices. Assuming a two-to-three-year time frame to achieve wide adoption of, for example, a seed variety, some of AIP’s products might be expected to be coming to maturity about now. AIP has reached a turning point in its life cycle where it now needs to focus on multiplication and accessing the market. Yet, some of the activities are facing constraints due to insufficient training (discussed further under Question 4) and unfinished business (due to limited funding or the end of funding), with the most acute problems in horticulture and vegetables. Better leveraging public-private partnerships may help support the tasks of completing ongoing research and expanding dissemination. Operational constraints impeded the effective implementation of some AIP activities. Chief among these were burdensome GoP procedures for decision-making and release of funds. Finally, the program did not share an exit strategy with the evaluation team or indicate that it has developed one. This is important because an exit strategy would dictate what next steps should be taken to build upon results. Recommendations For USAID: • Consider supporting innovative financing mechanisms, e.g., Development Credit Authority guarantees, to help poorer farmers gain easier access to productivity-enhancing technologies and practices. For IPs: • Continue using demonstration plots as a means of disseminating technologies and practices but consider locating them strategically to enhance visibility and effectiveness. • Explore ways to engage extension workers more fully to extend the use of AIP-supported technologies and practices to more remote areas of the provinces. • Ensure sufficient funding to finish adapting technologies and support partners in seed multiplication and disseminating technologies to ensure their widespread availability. • Increase the duration of training courses and, when offering them, ensure that trainers conduct more frequent on-site follow-up visits with farmers to help them promptly address their immediately pressing needs. • Make sure that training courses incorporate more practical sessions and continue to make use of visual materials, reducing the recourse to lectures. Agricultural Innovation Program – Final Evaluation 29 Question 4: Changes in Knowledge, Skills, and Attitudes To what extent did AIP produce any change in knowledge, skills, and attitudes in its trainees? To assess whether AIP training changed the knowledge, skills, and attitudes of members of research organizations, the survey asked training participants about the extent to which the training increased their knowledge and skills and about the extent to which these have become a routine part of their work. Findings Eighty-one percent of 115 survey respondents said that before providing training, AIP assessed their individual and organizational needs. Although AIP did not conduct formal needs assessments, the program consulted sector experts from all provinces to determine training priorities. An overwhelming majority of the survey respondents who participated in AIP training courses said that the training increased their knowledge relevant to their research work; that they used the knowledge and skills often; that the training had changed their research practices; and that the new practices had become a routine part of their work (Figure 17). FIGURE 17: EFFECTIVENESS OF AIP TRAINING BY SECTOR Source: PERFORM survey of beneficiaries of AIP training. The only significant difference between sectors was that respondents from the vegetables and cereal sectors were significantly more likely than respondents from the livestock and horticulture sectors to say they used new knowledge often in their work.13 The horticulture sector most probably lags behind the other sectors for two reasons: The IP, UC Davis, departed one year earlier than planned, and the horticulture crop cycle—5 to 10 years before trees bear commercial quantities of fruit—is much longer than the cycle for cereals. There was no statistically significant difference between the responses of men and women. 13 The difference between the cereal and horticulture sectors is significant at 𝛼 < 0.01, while the difference between the cereal and livestock sectors is significant at 𝛼 < 0.10. Agricultural Innovation Program – Final Evaluation 30 “Learning from training remains with us, and where needed, we put that into practice. For instance, whenever I write a research paper, learning from ‘Scientific Writing’ is put into practical use.” – Research organization member Findings from group and individual interviews were consistent with the survey findings; a majority of respondents reported positive changes in their knowledge, skills, and attitudes (Figure 18). Notably, 83 percent of all farmers—and 85 percent of female farmers—reported acquiring new knowledge and skills and changing their attitudes. By sector, 83 percent of farmers from the cereal sector, 72 percent from the vegetables sector, 73 percent from the horticulture sector, and 100 percent from the livestock sector reported that AIP training produced positive changes in their knowledge, skills, and attitudes. Survey evidence also suggests that trainees in the cereal sector realized somewhat more consistent benefits (across knowledge, skills, and practical application of knowledge) than those in other sectors. CIMMYT’s experience in Pakistan, the early closing of horticulture and vegetables activities, and the length of time required to realize the production benefits of some horticulture interventions may have contributed to these differences. FIGURE 18: EFFECTIVENESS OF AIP TRAINING BY TYPE OF TRAINEE Source: PERFORM key informant and group interviews. Factors Contributing to Changes in Knowledge, Skills, and Attitudes In key informant and group interviews, members of research organizations reported that they expected to continue practicing newly acquired skills because the skills are relevant to their work and have increased their research capacity and improved organizational productivity (23 of 50 responses). However, they also noted that a lack of funding for research or dissemination may limit their ability to continue new research practices or provide support to farmers. Similarly, of the 83 percent of farmers who reported learning new knowledge, skills, and attitudes, 100 percent said that they intended to continue the newly learned techniques and practices, their rationale being increased crop yield (38 percent of 107 responses), general usefulness of new practices to perform routine tasks (20 percent of responses), and increased income (15 percent of responses). The 21 farmers (17 percent of 122) who said they did not intend to continue the practices explained that the cost of the practices (67 percent of responses) and not being able to obtain required inputs in the Agricultural Innovation Program – Final Evaluation 31 market (33 percent of responses) may prevent them from doing so. The farmers who expressed these views were most likely small farmers with limited financial resources. “I will continue with the improved crop practices learned through my interaction with the project because I have personally observed the benefits. Before my association with the project, I was sowing local seed that was enough only to fulfill my domestic needs. Then I used the new recommended seed and fertilizers, which increased my production more than five times. I am now selling the extra production to generate income as well.” – Farmer, Quetta Conclusions Based on the perceptions of respondents, the evaluation can conclude that the new practices and technologies AIP-supported training introduced enhanced beneficiaries’ knowledge, skills, and attitudes. Farmers report intending to continue to use these new skills and practices because they have improved livestock and crop productivity, increased income, reduced manual labor, and improved time efficiency. However, affordability of and access to inputs and machinery, as well as the need for follow-up training to extend knowledge and cement skills, may limit small farmers’ uptake of new practices. The somewhat depressed results in the horticulture sector (relative to other sectors) illustrate the importance of a sustained interaction with farmers, especially in a sector where some interventions require years to affect production outcomes. Research organizations and extension workers report their intention to continue to use the skills they gained because they improved their organizations’ work productivity and benefited farmers. “We have worked with the farmers and they are so happy because through the direct seeding of rice technology their income has been increased to Rs. 10,000 per acre, and [they have gained] other savings like labor and time because they are now producing a resource-saving rice variety.” – Research organization member Recommendations For USAID and IPs: • Train extension workers to become trainers and use them more extensively to deploy need￾based training to farmers in the areas the extension workers serve. • Ensure that IPs better identify farmers’ needs and that training caters better to those needs. CONCLUDING REMARKS When gauged against the views of stakeholders, AIP was successful. As with any project, AIP experienced operational issues, and although respondents may be, on the whole, satisfied with the outcomes of AIP, some see the potential to improve the ways in which the program achieved those outcomes. The operational issues can be solved—e.g., funding processes can be streamlined and made even more transparent. Overall, however, the partners are effectively implementing the program, which emphasizes technical capacity building, distribution of genetic material and equipment, adaptive research, and exposure to new practices. AIP has enabled Pakistan to leverage an import substitution program that has shortened the time span that it takes to get inputs from the research stage to market. But now that the efforts to exploit and ready (certify and register) those inputs are nearing their end, AIP is confronted with the challenge of Agricultural Innovation Program – Final Evaluation 32 disseminating the fruits of its work beyond progressive farmers (the “innovators” and “early adopters”) to the broader majority of largely subsistence farmers. Given what some have called the “sprinkling” of AIP’s activities, the dissemination and marketing strategies need to be thought through—perhaps following a formal approach to scaling up, such as MSI’s own well-grounded one14—to move on to the next stage of this program. It is necessary to think through how to scale up current gains because the program has not reached a “tipping point” of a sufficiently large number of people to bring about the objectives stated in the AIP results framework. Although AIP has acted as a pole of attraction for stakeholders, it has not yet strengthened the network it has contributed to sufficiently to build functional and self-sustaining horizontal and vertical linkages along the research-production-marketing value chain. The decision regarding how and to what extent to go about building and strengthening these linkages depends more on the GoP (and PARC) than on USAID and AIP IPs. Involving the GoP in designing and managing what could be a package of follow￾through actions to AIP will not only build the GoP’s ownership but may also enable future activities to respond more sensitively to specific geo-climatic and crop needs articulated by those who, in the end, are responsible for increasing agricultural productivity in Pakistan: the provincial authorities. Consulting the GoP will also provide AIP with an opportunity to co-develop, from a program instead of a project perspective, an exit strategy that transitions the program into one that better satisfies grounded, endogenously determined needs and does so in a systematic approach to scaling up. These general conclusions lead the evaluation to recommend that AIP be extended with a clear plan to scale up its results and with the longer-term objective of institutionalizing its gains to ensure their sustainability. 14 Management Systems International, Scaling up—From Vision to Large-Scale Change: A Management Framework for Practitioners, 2nd ed. (MSI, 2012), http://www.msiworldwide.com/wp-content/uploads/Scaling-Up-Framework.pdf. Agricultural Innovation Program – Final Evaluation 33 ANNEX 1: STATEMENT OF WORK Agricultural Innovation Program – Final Evaluation 34 Agricultural Innovation Program – Final Evaluation 35 Agricultural Innovation Program – Final Evaluation 36 Agricultural Innovation Program – Final Evaluation 37 Agricultural Innovation Program – Final Evaluation 38 Agricultural Innovation Program – Final Evaluation 39 Agricultural Innovation Program – Final Evaluation 40 Agricultural Innovation Program – Final Evaluation 41 Agricultural Innovation Program – Final Evaluation 42 Agricultural Innovation Program – Final Evaluation 43 Agricultural Innovation Program – Final Evaluation 44 Agricultural Innovation Program – Final Evaluation 45 ANNEX 2: ASSIGNMENT WORK PLAN Agricultural Innovation Program – Final Evaluation 46 Agricultural Innovation Program – Final Evaluation 47 Agricultural Innovation Program – Final Evaluation 48 Agricultural Innovation Program – Final Evaluation 49 Agricultural Innovation Program – Final Evaluation 50 Agricultural Innovation Program – Final Evaluation 51 Agricultural Innovation Program – Final Evaluation 52 Agricultural Innovation Program – Final Evaluation 53 Agricultural Innovation Program – Final Evaluation 54 Agricultural Innovation Program – Final Evaluation 55 Agricultural Innovation Program – Final Evaluation 56 Agricultural Innovation Program – Final Evaluation 57 Agricultural Innovation Program – Final Evaluation 58 Agricultural Innovation Program – Final Evaluation 59 Agricultural Innovation Program – Final Evaluation 60 Agricultural Innovation Program – Final Evaluation 61 Agricultural Innovation Program – Final Evaluation 62 Agricultural Innovation Program – Final Evaluation 63 Agricultural Innovation Program – Final Evaluation 64 Agricultural Innovation Program – Final Evaluation 65 Agricultural Innovation Program – Final Evaluation 66 Agricultural Innovation Program – Final Evaluation 67 Agricultural Innovation Program – Final Evaluation 68 Agricultural Innovation Program – Final Evaluation 69 ANNEX 3: RESULTS FRAMEWORK FIGURE 19: AIP RESULTS FRAMEWORK Agricultural Innovation Program – Final Evaluation 70 ANNEX 4: DATA COLLECTION INSTRUMENTS AGRICULTURE INNOVATION PROGRAM (Survey Questionnaire for Members of Research and Partner Organizations) Serial number (FOR OFFICE USE ONLY) Start Time: hh:mm (AM/PM): Interviewer ID: End Time: hh:mm (AM/PM): Interview Date: DD/MM/YY Introduction: My name is______________ . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by CIMMYT and its partners. AIP aims to build the research and development capacities of partner institutions to revitalize agricultural research and innovation and affect the knowledge, skills, attitudes, and behavior of its trainees. The interview will take approximately 40-50 minutes. We will treat everything you say here confidentially. We will not use your names in reports or attribute anything you say to you. Module A: Background A1. Respondent name A2. Name of organization/Research institution A3. Location of organization/institution (City in which organization is located) A4. Is this a government (public sector) organization? Yes 1 No 0 (SKIP QUESTION A9) A5. Gender Male 0 Female 1 A6. Age of respondent in years Years Refused to answer -77 A7. Education of respondent Under graduate Graduat e/B. A/B.Sc. Masters Ph.D. Refused to answer 1 2 3 4 -77 A8. What best describes the nature of your current position Research 1 Managerial 2 Both reserch & managerial 3 Refused to answer -77 A9. What best describe the level of your position (Basic pay scale or equivalent) (NOTE: SKIP THIS QUESTION IF RESPONDENT ANSWERS “NO” TO QUESTION A4) Basic pay scale Refused to answer -77 A10. How many years have you worked in this organization (Years) Years Refused to answer -77 Years Agricultural Innovation Program – Final Evaluation 71 A11. How many years have you worked in your current position (Years) Refused to answer -77 A12. In which AIP activities were you engaged? (CIRCLE ALL THAT APPLY) Cereals: Wheat 1 Cereals: Maize 2 Cereals: Rice 3 Cereals: Agronomy 4 Livestock: Small ruminants 5 Livestock: Dairy animals 6 Livestock: Fodder and range management 7 Vegetable: Protected cultivation 8 Vegetable: New verities 9 Vegetable: Mung bean 10 Horticulture: Post-harvest losses 11 Horticulture: Value addition 12 Agricultural Innovation Program – Final Evaluation 72 NOTE: ➢ The following questions are meant to elicit responses of members of organizations that worked with AIP. ➢ Supervisors and enumerators should read every question along with responses aloud, unless instructed otherwise. Options for DK/NA are not to be read aloud. Module B: Institutional and Human capacity assessment DEFINITION: AIP: A USAID-funded project to enhance the research capacities of public and private sector agricultural and livestock research organizations with the aim of increasing agricultural and livestock productivity. It is implemented by the International Maize and Wheat Improvement Center (CIMMYT), International Livestock Research Institute (ILRI), International Rice Research Institute (IRRI), University of California-Davis (UC Davis), the World Vegetable Centre (AVRDC), and the Pakistan Agricultural Research Council (PARC). To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statement Strongly Agree Agree Disagree Strongly Disagree Don’t Know N/A Q1. Your organization is conducting research to improve agricultural or livestock productivity. 4 3 2 1 -88 -99 Q2. AIP support has improved your organization's capacity to conduct research to improve agricultural or livestock productivity. 4 3 2 1 -88 -99 Q3. AIP support has improved the quality of the research your organization conducts. 4 3 2 1 -88 -99 Q4. AIP support to your organization has created a lasting change in the organization's capacity to conduct research? 4 3 2 1 -88 -99 Q5. As a result of AIP support, the working environment at your organization is more supportive of research. 4 3 2 1 -88 -99 Q6. New research and development practices introduced by AIP have become part of the routine practice of this organization? 4 3 2 1 -88 -99 Q7. The leadership of your organization is more supportive of adopting new research techniques or practices since receiving assistance from AIP. 4 3 2 1 -88 -99 Q8. AIP accurately assessed your organization’s needs 4 3 2 1 -88 -99 Q9. AIP designed its assistance to meet those needs Agricultural Innovation Program – Final Evaluation 73 Question # Statement Strongly Agree Agree Disagree Strongly Disagree Don’t Know N/A Q10. AIP’s support improved your organization’s capacity for research. 4 3 2 1 -88 -99 Q11. AIP has effectively coordinated with your organization throughout the project period. 4 3 2 1 -88 -99 Question # Dimensions Significantly Improved Somewhat Improved Minimally Improved No Improvemen t Don’t Know N/A Q12. To what extent, if any, has AIP support to your organization improved the organization's capacity to conduct research? (CIRCLE ONE NUMBER) 4 3 2 1 -88 -99 Q13. To what extent, if any, has AIP support to your organization improved the organization's capacity to disseminate the results of your research? (CIRCLE ONE NUMBER) 4 3 2 1 -88 -99 Question # Training provided by AIP Q14. Did you receive any trainings through AIP? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q23 Q15. Before providing trainings, did AIP assess your own learning needs. (CIRCLE ONE NUMBER) Yes No Q16. Before providing trainings, did AIP assess your organization’s learning needs. (CIRCLE ONE NUMBER) Yes No 1 0 Q17. To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER FOR EACH QUESTION) Statement Strongly Agree Agree Disagree Strongly Disagree a. The training material was useful. 4 3 2 1 b. The trainer was well-prepared. 4 3 2 1 Agricultural Innovation Program – Final Evaluation 74 c. The training environment was conducive to learning. 4 3 2 1 d. The trainer used a participatory approach. 4 3 2 1 e. The training duration was enough to learn new knowledge or skills. 4 3 2 1 Q18. To what extent, if at all, did this training increase your knowledge relevant to your research work? (CIRCLE ONE NUMBER) Significantly increased Moderately increased Slightly increased Did not increase 4 3 2 1 GO TO Q22 Q19. How often do you use this knowledge in your work? (CIRCLE ONE NUMBER) Very often Often Sometimes Never 4 3 2 1 Q20. To what extent do you agree or disagree with the following statement: The new knowledge and skills you gained through AIP have become a routine part of your work. (CIRCLE ONE NUMBER) Strongly Agree Agree Disagree Strongly Disagree 4 3 2 1 Q21. To what extent, if at all, have the new knowledge and skills you gained through AIP changed the way you conduct research? (CIRCLE ONE NUMBER) Significantly Changed Moderately Changed Slightly Changed Did not Change 4 3 2 1 Q22. To what extent, if at all, did this training improve your organization’s overall capacity to conduct research? (CIRCLE ONE NUMBER) Significantly Improved Moderately Improved Slightly Improved Did not Improve 4 3 2 1 Supporting research through grants Q23. Did your organization receive any grants from AIP to support research? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q26 Q24. To what extent, if at all, was this funding relevant to address the needs of your organization so as to become better enabled to conduct research? (CIRCLE ONE NUMBER) Very Relevant Moderately Relevant Slightly Relevant Not Relevant 4 3 2 1 Q25. To what extent, if at all, did this funding actually improve your organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Significantly Improved Moderately Improved Slightly Improved Did not Improve 4 3 2 1 Upgrading research facilities or equipment Agricultural Innovation Program – Final Evaluation 75 Q26. Did your organization receive any assistance from AIP to upgrade the organization’s research facilities or equipment? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q29 Q27. To what extent, if at all, was this assistance relevant to address the needs of your organization so as to become better enabled to conduct research? (CIRCLE ONE NUMBER) Very Relevant Moderately Relevant Slightly Relevant Not Relevant 4 3 2 1 Q28. To what extent, if at all, did this assistance actually improve your organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Significantly Improved Moderately Improved Slightly Improved Did not Improve 4 3 2 1 Genetic material (Seed varieties/Sperm samples) Q29. Did your organization receive new genetic material (e.g., seed varieties or new or revived sperm samples) for research form AIP? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q32 Q30. To what extent, if at all, was this genetic material relevant to address the needs of your organization so as to become better enabled to conduct research? (CIRCLE ONE NUMBER) Very Relevant Moderately Relevant Slightly Relevant Not Relevant 4 3 2 1 Q31. To what extent, if at all, did this genetic material actually improve your organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Significantly Improved Moderately Improved Slightly Improved Did not Improve 4 3 2 1 New technologies Q32. Did AIP provide your organization with new agricultural or livestock technologies to test for applicability in Pakistan? (CIRCLE ONE NUMBER) Note: “Agricultural or livestock technologies” refers to physical items (e.g., machinery, greenhouses, irrigation equipment, vaccines, etc.) to improve agricultural or livestock production. Yes No 1 0 GO TO Q35 Q33. To what extent, if at all, were those technologies relevant to address the needs of your organization so as to become better enabled to introduce new productivity-enhancing technologies to farmers? (CIRCLE ONE NUMBER) Very Appropriate Moderately Appropriate Slightly Appropriate Not Appropriate 4 3 2 1 Q34. To what extent, if at all, did these technologies actually improve your organization’s capacity to introduce new productivity-enhancing technologies to farmers? (CIRCLE ONE NUMBER) Significantly Improved Moderately Improved Slightly Improved Did not Improve 4 3 2 1 New practices Agricultural Innovation Program – Final Evaluation 76 Q35. Did AIP provide your organization with new agricultural or livestock practices to test for applicability in Pakistan? (CIRCLE ONE NUMBER) Note: New agricultural or livestock practices refer to activities or ways of doing things that improve agricultural or livestock production. Examples may include cultivation practices (zero tillage planting), irrigation practices, harvesting or post-harvest practices, covered/vertical cultivation, feeding practices (fodder), animal husbandry practices, etc. Yes No 1 0 GO TO MODULE ‘C’ Q36. To what extent, if at all, were these practices relevant to address the needs of your organization so as to become better enabled to introduce new productivity-enhancing practices to farmers? (CIRCLE ONE NUMBER) Very Appropriate Moderately Appropriate Slightly Appropriate Not Appropriate 4 3 2 1 Q37. To what extent, if at all, did these technologies improve your organization’s capacity to introduce new productivity-enhancing practices to farmers? (CIRCLE ONE NUMBER) Significantly Improved Moderately Improved Slightly Improved Did not Improve 4 3 2 1 Agricultural Innovation Program – Final Evaluation 77 Module C: Information / Consent Form C1. Can you share your mobile number? (CIRCLE ONE NUMBER) Yes GO TO C1A No GO TO C2 Do not Have Mobile GO TO C2 (DO NOT READ) Don’t Know GO TO C2 Refused to Answer GO TO C2 1 2 3 -88 -77 C1A. Record mobile number Number1: Number 2: C2. Can you share your landline number? (CIRCLE ONE NUMBER) Yes GO TO C2A No GO TO C3 Do not Have Landline GO TO C3 (DO NOT READ) Don’t Know GO TO C3 Refused to Answer GO TO C3 1 2 3 -88 -77 C2A. Record landline number City/Area Code Phone number City/Area Code Phone number Note: if response is “yes” in either C1 or C2, then ask C3. C3. Can we contact you via mobile or landline in the future if we need additional information? (CIRCLE ALL THAT APPLY) Cell Landline No 1 2 3 Thank respondent Agricultural Innovation Program – Final Evaluation 78 AGRICULTURE INNOVATION PROGRAM (Organizational Survey) Serial number (FOR OFFICE USE ONLY) Start Time: hh:mm (AM/PM): Interviewer ID: End Time: hh:mm (AM/PM): Interview Date: DD/MM/YY Introduction: My name is . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT), in partnership with the International Livestock Research Institute (ILRI), International Rice Research Institute (IRRI), University of California- Davis (UC Davis), the World Vegetable Centre (AVRDC), and the Pakistan Agricultural Research Council (PARC). The interview will take approximately 45-60 minutes. We will treat everything you say here confidentially. We will not use your names in reports or attribute anything you say to you. Module A: Background A1. Respondent name A2. Name of organization/Research institution A3. Location of organization/institution (City in which organization is located) A4. Is this a government (public sector) organization? Yes 1 No 0 A5. In which AIP activities were you engaged? (CIRCLE ALL THAT APPLY) Cereals: Wheat 1 Cereals: Maize 2 Cereals: Rice 3 Cereals: Agronomy 4 Livestock: Small ruminants 5 Livestock: Dairy animals 6 Livestock: Fodder and range management 7 Vegetable: Protected cultivation 8 Vegetable: New varieties 9 Vegetable: Mung bean 10 Horticulture: Post-harvest losses 11 Horticulture: Value addition 12 A6. Interviewer name: A7. Entered by: A8. In which year did your organization start working with AIP? Agricultural Innovation Program – Final Evaluation 79 Module B: Institutional and Human Capacity Assessment Based on your organization’s records, please provide information on the following aspects of your organization’s activities from 2013 through 2017. Note: 1. Please provide information about each calender year e.g., 2013= January 01,2013 – December 31, 2013, and so on 2. Please provide annual numbers, NOT cumulative numbers on these indicators (don’t add numbers of previous years), unless otherwise stated. 3. Write -99 if not applicable. This section applies to all sectors. (This note will not be in the online instrument.) Question # Year 2013 2014 2015 2016 2017 Q1. How many proposals did this organization submit to obtain funding for agricultural or livestock research in each of the following years? Q2. How many of these proposals were successful in obtaining funding in each of the following years? Q3. How many researchers did this organization engage in each of the following years? Q4. How many research projects did this organization complete in each of the following years? To what extent do you agree or disagree with the following statements? (CIRCLE ONE NUMBER) Question # Statements about AIP implementation Strongly Agree Agree Disagree Strongly Disagree N/A Q5. AIP implementing partners accurately assessed my organization’s needs 4 3 2 1 -99 Q6. AIP implementing partners designed their assistance to meet my organization’s needs. 4 3 2 1 -99 Q7. AIP support has improved my organization's capacity to conduct research. 4 3 2 1 -99 Q8. AIP support has improved the quality of the research my organization conducts. 4 3 2 1 -99 Q9. AIP support to my organization has created a lasting change change in its capacity to to conduct research. 4 3 2 1 -99 Agricultural Innovation Program – Final Evaluation 80 Question # Statements about AIP implementation Strongly Agree Agree Disagree Strongly Disagree N/A Q10. AIP support to my organization has created a lasting change in its research practices. 4 3 2 1 -99 Q11. AIP effectively coordinated with my organization throughout the project period. 4 3 2 1 -99 Question # Training provided by AIP Q12. Did this organization receive any trainings through AIP implementing partners such as CIMMYT, ILRI, IRRI, AVRDC/WorldVeg or UCDavis? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q15 Q13. To what extent, if at all, was the training designed to address this organization’s capacity gaps to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Q14. To what extent, if at all, did the training improve this organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Supporting research through grants Q15. Did your organization receive any grants from AIP to support research? (CIRCLE ONE NUMBER) Yes No 0 GO TO Q18 Q16. To what extent, if at all, were these grants relevant to address the organization’s capacity gaps to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Q17. To what extent, if at all, did these grants actually improve the organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Upgrading research facilities or equipment Q18. Did your organization receive any assistance from an AIP implementation partner such as CIMMYT, ILRI, IRRI, AVRDC/WorldVeg or UCDavis to upgrade the organization’s research facilities, including equipment? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q 21 Q19. To what extent, if at all, was this assistance relevant to address the organization’s capacity gaps to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Q20. To what extent, if at all, did this assistance actually improve the organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Agricultural Innovation Program – Final Evaluation 81 Genetic material (Seed varieties/Sperm samples) Q21. Did your organization receive new genetic material (e.g., seed varieties or sperm samples) for research from an AIP implementation partner such as CIMMYT, ILRI, IRRI, AVRDC/WorldVeg and UCDavis? (CIRCLE ONE NUMBER) Yes No 1 0 GO TO Q26 Based on your organization’s records, please provide information on the following aspects of your organization’s activities from 2013 through 2017. Note: 1. Please provide information about each calender year e.g., 2013= January 01,2013 – December 31, 2013, and so on 2. Please provide annual numbers, NOT cumulative numbers on these indicators (don’t add numbers of previous years), unless otherwise stated. 3. Write -99 if not applicable. Year 2013 2014 2015 2016 2017 Q22. How many seed varieties not widely commercially available before 2013 was this organization testing for suitability to Pakistan in each of the following years? Note: This question refers to adaptive research to determine the suitability of an existing variety to Pakistan. It does not refer to selective breeding to develop new lines. Since testing may take several years, the values reported here may be cumulative. Report the number of varieties on which the organization was actively conducting research in each year. Q23. How many seed lines not widely commercially available before 2013 did this organization have its commercialization pipeline in each of the following years? Note: Report the number of lines on which the organization was actively conducting research, or had in the pipeline to be approved for commercialization, in each year. Numbers may be cumulated each year. Q24. To what extent, if at all, was this genetic material relevant to address the organization’s capacity gaps to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Q25. To what extent, if at all, did this genetic material improve the organization’s capacity to conduct research? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Agricultural Innovation Program – Final Evaluation 82 New technologies Q26. Did a AIP partner such as CIMMYT, ILRI, IRRI, AVRDC/WorldVeg or UCDavis provide your organization with new agricultural or livestock technologies to test for applicability in Pakistan? (CIRCLE ONE NUMBER) Note: “Agricultural or livestock technologies” refers to physical items (e.g., machinery, greenhouses, irrigation equipment, vaccines etc.) that have the potential to improve agricultural or livestock production. Yes No 1 0 GO TO Q30 Based on your organization’s records, please provide information on the following aspects of your organization’s activities from 2013 through 2017. Note: 1. Please provide information about each calender year e.g., 2013= January 01,2013 – December 31, 2013, and so on 2. Please provide annual numbers, NOT cumulative numbers on these indicators (don’t add numbers of previous years), unless otherwise stated. 3. Write -99 if not applicable. Year 2013 2014 2015 2016 2017 Q27. Apart from seed varieties, how many agricultural / livestock AIP promoted technologies not widely available before 2013 was this organization testing for their applicability to the Pakistan agricultural sector in each of the following years? Note: “Agricultural technologies” refers to physical items (e.g., machinery, greenhouses, irrigation equipment, etc.) that have the potential to improve agricultural production. Since testing may take several years, the values reported here may be cumulative. Report the number of technologies on which the organization was actively conducting research in each year. Q28. To what extent, if at all, were those technologies relevant to address the organization’s capacity gaps to introduce new productivity-enhancing technologies to farmers? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Q29. To what extent, if at all, did these technologies actually improve your organization’s capacity to introduce new productivity-enhancing technologies to farmers? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Agricultural Innovation Program – Final Evaluation 83 New practices Q30. Did AIP provide your organization with new agricultural or livestock practices on which to test for applicability in Pakistan? (CIRCLE ONE NUMBER) Note: New agricultural or livestock practices refer to activities or ways of doing things that have the potential to improve agricultural or livestock production. Examples may include cultivation practices (zero tillage planting), irrigation practices, harvesting or post-harvest practices, covered/vertical cultivation, feeding practices (fodder), animal husbandry practices, etc. Yes No 1 0 GO TO MODULE ‘C’ Based on your organization’s records, please provide information on the following aspects of your organization’s activities from 2013 through 2017. Note: 1. Please provide information about each calender year e.g., 2013= January 01,2013 – December 31, 2013, and so on 2. Please provide annual numbers, NOT cumulative numbers on these indicators (don’t add numbers of previous years), unless otherwise stated. 3. Write -99 if not applicable. Year 2013 2014 2015 2016 2017 Q31. How many AIP promoted livestock practices not widely used before 2013 was this organization testing for applicability to farmers in each of the following years? Note: “Livestock practices” refers to activities or ways of doing things that have the potential to improve livestock health or production. Examples may include feeding practices (fodder), animal husbandry practices, etc. Since testing may take several years, the values reported here may be cumulative. Report all livestock practicies on which the organization was actively conducting tests in each year. Q32. To what extent, if at all, were these practices relevant to improving your organization’s capacity to introduce new productivity-enhancing practices to farmers? (CIRCLE ONE NUMBER) Very Relevant Relevant Somewhat Relevant Not relevant 4 3 2 1 Q33. To what extent, if at all, did these practices improve your organization’s capacity to introduce new productivity￾enhancing practices to farmers? (CIRCLE ONE NUMBER) Great Extent Moderate Extent Some Extent No Extent 4 3 2 1 Agricultural Innovation Program – Final Evaluation 84 Module C: Information / Consent Form C1. Can you share your mobile number? (CIRCLE ONE NUMBER) Yes GO TO C1A No GO TO C2 Don not Have Mobile GO TO C2 (DO NOT READ) Don’t Know GO TO C2 Refused to Answer GO TO C2 1 2 3 -88 -77 C1A. Record mobile number Number1: Number 2: C2. Can you share your landline number? (CIRCLE ONE NUMBER) Yes GO TO C2a No GO TO C3 Don not Have Landline GO TO C3 (DO NOT READ) Don’t Know GO TO C3 Refused to Answer GO TO C3 1 2 3 -88 -77 C2A. Record landline number City/Area Code Phone number City/Area Code Phone number Note: if response is “yes” either C1 or C2, then ask C3. C3. Can we contact you via cell/landline in the future if we need additional information? (CIRCLE ALL THAT APPLY) Cell Landline No 1 2 3 Thank respondent Agricultural Innovation Program – Final Evaluation 85 Group Interview Guide Farmers Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A7. Start time: End time: A10. Moderator name: A11. Note taker name: Instructions to Note taker: 1. Assign each respondent a unique code, i.e., R1, R2, etc., and use these codes to identify individuals’ responses to each question. Instructions to interviewer: 1. Read the introduction below 2. Have members introduce themselves [name, village etc.] INTRODUCTION My name is _________________ . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). The interview will take approximately 90 -120 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If all respondents say yes, continue the interview. If no, try to motivate respondents by answering their questions and explaining the importance of recording the interview. If all respondents do not agree to the recording, do not record the interview. Agricultural Innovation Program – Final Evaluation 86 GROUP PROCESS: We value the contribution of each group member and therefore I expect that each person will respond in turn to the questions. First, I’d like to ask you all to introduce yourselves. Respondent Code Name Sex (M/F) Village District Sector R1 R2 R3 R4 Q1. In which type of trainings / learning activities did you participate? (EQ4) [Probe for demonstration plots, exposure visits, site advice from extension workers etc.] Q2. What did you expect to learn from trainings / learning activities provided by [name of AIP supported institution]? (EQ4) Q3. What did you learn from [name of AIP supported institution] trainings / learning activities? (EQ4) Q4. Which trainings / learning activities were most and least useful for you, and why? (EQ4) [Probe for frequency / intensity / quality of learning activities / ease of transferring knowledge to others etc.] Q5. How could learning activities be improved? (EQ4) [Probe for frequency / intensity / quality of learning activities / ease of transferring knowledge to others, content covered, etc.] Q6. What learning aids were most and least useful for you, and why? (EQ4) [Probe for factsheets, manuals, samples of equipment / materials, videos, radio programs, etc.] Q7. How could the learning aids be improved? (EQ4) [Probe for visual impact, ease of access, robustness of learning aids etc.] Q8. What, if anything, will you continue to apply from what you have learned from AIP trainings and other activities? Why or why not? (EQ4) Q9. How could the overall support from AIP be improved? (EQ4,3) [Probe for any other support such as structures, technology/tools, seed varieties etc.] Thank you for participating in this group interview Agricultural Innovation Program – Final Evaluation 87 Group Interview Guide Extension Workers Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Start time: End time: A7. Moderator name: A8. Note taker name: Instructions to Note taker: 2. Assign each respondent a unique code, i.e., R1, R2, etc., and use these codes to identify individuals’ responses to each question. Instructions to interviewer: 3. Read the introduction below 4. Have members introduce themselves [name, specialty, years in their current positions] INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). The interview will take approximately 2 hours (120 minutes). We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If all respondents say yes, continue the interview. If no, try to motivate respondents by answering their questions and explaining the importance of recording the interview. If all respondents do not agree to the recording, do not record the interview. Note to interviewers: If the respondents ask for the meaning of word ‘research’, read the following definition. Research is the process of developing or adopting livestock and agricultural productivity-enhancing technologies and practices. Agricultural Innovation Program – Final Evaluation 88 GROUP PROCESS: We value the contribution of each group member and therefore I expect that each person will respond in turn to the questions. First, I’d like to ask you all to introduce yourselves. Can you please give your name, your position or title, and the number of years of experience you have as an extension worker. Respondent Code Name Sex (M/F) Position/title Experience in years Sector Institution R1 R2 R3 R4 INTRODUCTORY QUESTION Q1. In which AIP supported activities have you participated? [Probe for learning, teaching, and adaptive research activities on technologies or practices] QUESTIONS ROLE AS PROVIDERS OF TRAINING / ORIENTATION Q2. What [AIP supported] orientation / training activities have you conducted? Describe how those activities were conducted? (EQ4) [Probe for the types of people for whom the activities were conducted; [Probe also for demonstrations, exposure visits, adaptive research on technologies or practices, field days, meetings with individual farmers, etc.] Q3. Which of these activities were most and/or least effective in teaching farmers about new technologies and practices, and why? (EQ4) [Probe for teaching techniques / frequency / intensity / quality of learning activities and materials] Q4. What types of orientation / training activities will you continue to conduct, and why? (EQ4) Q5. How could the learning / orientation activities for farmers be improved? (EQ4) [Probe for teaching techniques / frequency / intensity / quality of learning activities and materials]. Q6. What learning aids have you used? (EQ4) [Probe for factsheets, manuals, samples of equipment / materials, videos, radio programs, etc.] Q7. Which learning aids have been most and least effective in teaching farmers about new technologies and practices? Why? (EQ4) Q8. What types of learning aids will you continue to use, and why? (EQ4) [Probe for factsheets, manuals, samples of equipment / materials, videos, radio programs, etc.] Q9. How could the learning aids for farmers be improved? (EQ4) [Probe for visual impact, ease of access, robustness of learning aids etc.] Agricultural Innovation Program – Final Evaluation 89 QUESTIONS ON ROLE AS LEARNERS Q10. In which type of trainings / learning activities did you participate? (EQ4) [Probe for adaptive research methodology, lab techniques, management practices, teaching, demonstrations etc.] Q11. Which trainings / learning activities were most and least useful for you, and why? (EQ4) [Probe for frequency / intensity / quality of learning activities / ease of transferring knowledge to others etc.] Q12. What did you expect to learn from trainings / learning activities provided by [name of AIP supported institution]? (EQ4) Q13. What did you learn from [name of AIP supported institution] trainings / learning activities? (EQ4) Q14. What learning aids were most and least useful for you, and why? (EQ4) [Probe for facts sheets, training material / equipment, manuals, case studies, videos/audiovisual aids, etc.] Q15. How could the learning aids be improved, it all? (EQ4) [Probe for visual impact, ease of access, robustness of learning aids etc.] Q16. What, if anything, will you continue to apply from what you have learned from AIP trainings and other activities? Why or why not? (EQ4) Q17. How could the training you received from AIP be improved? (EQ4) [Probe for frequency / timeliness / intensity of trainings / quality of teachers / appropriateness of materials received] CLOSING QUESTION Q18. How could the overall support from AIP be improved? (EQ4, all) Thank you for participating in this group interview Agricultural Innovation Program – Final Evaluation 90 Group Interview Guide for Members of Research / Partner Organizations Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Start time: End time: A6. Moderator name: A7. Note taker name: Instructions to Note taker: 3. Assign each respondent a unique code, i.e., R1, R2, etc., and use these codes to identify individuals’ responses to each question. Instructions to interviewer: 5. Read the introduction below 6. Have members introduce themselves [name, specialty, years in their current positions] INTRODUCTION My name is ________________ . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). The interview will take approximately 90–120 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If all respondents say yes, continue the interview. If no, try to motivate respondents by answering their questions and explaining the importance of recording the interview. If all respondents do not agree to the recording, do not record the interview. Note to interviewers: If the respondents ask for the meaning of word ‘research’, read the following definition. Research is the process of developing or adotping livestock and agricultural productivity-enhancing technologies and practices. Agricultural Innovation Program – Final Evaluation 91 GROUP PROCESS: We value the contribution of each group member and therefore I expect that each person will respond in turn to the questions. Introductions: Respondent Code Name Sex (M/F) Sector Institution R1 R2 R3 R4 Q19. Why did your organization decide to collaborate with AIP; what needs did you expect the collaboration to address? (EQ2,1) Q20. What organizational needs did your collaboration with AIP meet? How did it meet these needs, and if did not, why not? (EQ2,1) Q21. Did AIP enhance your organization’s capacity to conduct research and disseminate results? If yes, please tell me about some of the most important capacities built and explain how AIP contributed to building these capacities? If AIP did not build your organization’s capacity, please explain why not? (EQ2) Q22. If AIP contributed to building your organization’s capacity, do you think this is a change you can sustain after AIP ends (in other words, a lasting change)? If yes, why / how is it sustainable? If no, why is it not sustainable? (EQ2) Q23. In which type of trainings / learning activities did you participate? (EQ4) [Probe for research methodology, lab techniques, management practices -- including seed selection, land preparation, irrigation, etc.--, specific technologies or practices, etc.] Q24. What did you expect to learn from trainings / learning activities provided by [name of AIP supported institution]? (EQ4) Q25. What did you learn from [name of AIP supported institution] trainings / learning activities? (EQ4) Q26. Which trainings / learning activities were most and least useful for you, and why? (EQ4) Q27. What learning aids were most and least useful for you, and why? (EQ4) [Probe for training material, manuals, videos etc.] Q28. What, if anything, will you continue to apply from what you have learned from AIP trainings and other activities? Why and how? If none, why not? (EQ4) Q29. How could the trainings and the learning aids from AIP be improved? (EQ4,3) Agricultural Innovation Program – Final Evaluation 92 [Learning activities: Probe for frequency / intensity / quality of learning activities / ease of transferring knowledge to others etc.] [Learning aids: probe for visual impact, ease of access etc.] Q30. How, if at all, could the program be improved? What practices should be continued? (EQ3) [Probe for support such as structures, technology/tools, seed varieties, etc.] Thank participant for the time afforded to the interview Agricultural Innovation Program – Final Evaluation 93 Group Interview Guide Trainers / Teachers Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Start time: End time: A6. Moderator name: A7. Note taker name: Instructions to Note taker: 4. Assign each respondent a unique code, i.e., R1, R2, etc., and use these codes to identify individuals’ responses to each question. Instructions to interviewer: 7. Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). The interview will take approximately 90-120 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If all respondents say yes, continue the interview. If no, try to motivate respondents by answering their questions and explaining the importance of recording the interview. If all respondents do not agree to the recording, do not record the interview. Agricultural Innovation Program – Final Evaluation 94 GROUP PROCESS: We value the contribution of each group member and therefore I expect that each person will respond in turn to the questions. First, I’d like to ask you all to introduce yourselves. Can you please give your name, your position or title, and tell us how many years you have been working in the sector. Respondent Code Name Sex (M/F) Position/title Experience in years Sector Institution R1 R2 R3 R4 Q31. What [AIP supported] teaching / training activities have you conducted? Briefly describe how were those activities conducted? (EQ4) [Probe for the types of people for whom the activities were conducted and how their needs were assessed] [Probe also for demonstrations, exposure visits, adaptive research on technologies or practices, contacts in the field and other experiential l /adult learning techniques] Q32. Which of these activities were most and/or least effective in teaching trainees about new technologies and practices, and why? (EQ4) [Probe for teaching techniques / frequency / intensity / quality of learning activities and materials] Q33. What types of trainings / learning activities should be continued, and why? (EQ4) Q34. How could the training / learning activities be improved? (EQ4) [Probe for teaching techniques / frequency / intensity / quality of learning activities and materials]. Q35. What types of teaching / training aids have you used? (EQ4) [Probe for factsheets, manuals, samples of equipment / materials, videos, etc.] Q36. Which teaching / training aids have been most and least useful? Why? (EQ4) Q37. How could the teaching / training aids be improved? (EQ4) [Probe for visual impact, ease of access, robustness of learning aids etc.] Q38. How could the overall support from AIP be improved? (EQ4, all) Thank you for participating in this group interview Agricultural Innovation Program – Final Evaluation 95 Key Informant Interview Guide IPs, USAID, PARC/MNFSR/Provincial Governments Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: 8. Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). The interview will take approximately 40–50 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Agricultural Innovation Program – Final Evaluation 96 Q1. Please explain your involvement in AIP Q2. How were the beneficiary research institutions selected? (EQ3) [Probe for selection criteria and ask interviewee to rank them] Q3. How, if at all, did your organization decide which AIP program activities to implement? (EQ3) Q4. Which needs did AIP intend to address? (EQ3) Q5. Which needs did AIP address? (EQ3) Q6. How well, or not, is the agricultural research and innovation system performing in Pakistan? Q7. How, if at all, did AIP contribute to building research capacity of organizations in each of the following sectors; cereals, vegetables, livestock, horticulture? (EQ2) Q8. Which interventions were most effective? Which interventions were least effective? Why or why not? (EQ2) [Probe for professional development, equipment, fund raising, staffing, promotion, technology, policy, work processes and procedures, culture or leadership] Q9. What are main bottlenecks in adoption of technologies introducted by AIP? Q10. How, if at all, could the program be improved? What practices should be continued? (EQ3) Thank respondent for his / her time Agricultural Innovation Program – Final Evaluation 97 Key Informant Interview Guide Heads of Research/Partner Organizations Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name and Gender: A7. Start time: End time: A8. Sector represented: (CIRCLE ALL THAT APPLY) 1. Cereals 2. Livestock 3. Horticulture 4. Vegetables A9. Moderator name: A10. Note taker name: Instructions to interviewer: 1. Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). The interview will take approximately 40–50 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If all respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Agricultural Innovation Program – Final Evaluation 98 Q39. Why did your organization decide to collaborate with AIP? (EQ3) Q40. What needs did you expect the collaboration to address? (EQ3) Q41. What organizational needs did your collaboration with AIP meet? How did it meet these needs and, if it did not, why not? (EQ3) Q42. How did AIP enhance, if at all, your research activities? Why or why not? (EQ1) Q43. How did AIP enhance, if at all, your organization’s capacity to conduct research and disseminate results? Why or why not? (EQ2) [Probe for most important capacities built and explain how AIP contributed to building these capacities] Q44. If AIP contributed to building the organization’s capacity, do you think this is a change you can sustain after AIP ends (in other words, a lasting change)? If yes, why / how is it sustainable? If no, why is it not sustainable? (EQ2) Thank participant for the time afforded to the interview Agricultural Innovation Program – Final Evaluation 99 Key Informant Interview Guide Experts Agriculture Innovation Program (AIP) Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name and Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: 2. Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to assess the impact of the Agricultural Innovation Program (AIP) implemented by the International Maize and Wheat Improvement Center (CIMMYT). AIP aims to (1) build the capacities of public and private sector agricultural and livestock research organizations to develop productivity-enhancing agricultural and livestock technologies and practices, (2) enhance the skills of researchers, and (3) promote dissemination of technologies and practices to farmers. The interview will take approximately 40–50 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No ) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Agricultural Innovation Program – Final Evaluation 100 INTRODUCTION: Have respondent describe what he / she does in the sector? Q1. What are the most pressing needs for improving the production and the quality of cereals, vegetables, livestock, and horticulture in Pakistan? (EQ3) Q2. What do you know about AIP? a. IF RESPONDENT KNOWS ABOUT AIP: How, if at all, did AIP contribute to the revival of research and innovation in Pakistan system? If no why not? Q3. In your opinion, how does a project that focuses on building local capacity to develop and disseminate productivity-enhancing agricultural and livestock technologies and practices contribute to meeting pressing needs of the Pakistani agricultural sector? (EQ3)? Q4. What are some of the impediments and / or facilitating factors for the functioning of agricultural research and development in Pakistan? Q5. How, if at all, could USAID improve the design and implementation of its interventions in the Pakistani agricultural sector? (EQ3) Q6. How, if at all, could USAID better support the agriculture and livestock sector in Pakistan? (EQ3) Thank respondent for the time invested in the interview Agricultural Innovation Program – Final Evaluation 101 ANNEX 5: BENEFICIARY DATA AND ONLINE SURVEYS CIMMYT Pakistan shared a list of 54,653 beneficiaries who had received assistance from any of the 5 IPs, i.e., CIMMYT, IRRI, AVRDC, ILRI, and UC Davis. The list contained information on beneficiary’s name, sector, CNIC number, sex, contact number, district, organizational association, and type. The list included 373 different categories of beneficiaries which were reclassified into six broad categories: i) farmers, ii) extension workers, iii) members of public sector organizations, iv) members of private organizations, v) staff members of IPs, and vi) other. The distribution of beneficiaries by IP and type is given in Table 2. TABLE 2: DISTRIBUTION OF BENEFICIARIES BY TYPE Beneficiary Type Number of Beneficiaries by IP CIMMYT IRRI AVRDC ILRI UC Davis Farmers 33,960 1,214 2,435 3,921 2,596 Extension workers 79 82 - - 295 Members of public sector organizations 5,365 13 - 352 2,880 Members of private organizations 40 1 - - 31 Project staff members 3 - - - 25 Other 1,005 - - 171 185 Total 40,452 1,310 2,435 4,444 6,012 Though the list says there were 54,653 beneficiaries, the number of unique beneficiaries was far lower. The list included an entry for a beneficiary each time she or he received assistance under AIP. The automated removal of repeated beneficiaries from the list was next to impossible, as CNIC numbers, contact numbers, and organizational association were not available in most of the cases. Moreover, for cases with contact numbers, more than 1,000 contact numbers were repeated 4 or more times. Of these, 8 contact numbers were repeated more than 100 times, and one number was repeated 389 times. The manual deletion of beneficiaries based on their names (which could be misspelled in many cases) and locations (which are missing in most cases) would be an unreliable way of preparing a list of unique beneficiaries. Therefore, in principle it was agreed that the evaluation team would use the original list in selecting the sample respondents, and in case the same person was selected more than once in a sample, she or he would be interviewed only once. Table 3 provides the number of beneficiaries of each type who were listed with CNIC and contact information. TABLE 3: NUMBER OF BENEFICIARIES WITH CNIC AND CONTACT INFORMATION Beneficiary Type No. of Beneficiaries With Contact Details With CNIC Information Farmers 44,126 22,085 21,667 Extension workers 456 336 70 Members of public sector organizations 8,610 6,179 911 Members of private organizations 72 55 20 Project staff members 28 22 4 Other 1,361 1,258 1,073 Total 54,653 29,935 23,745 Agricultural Innovation Program – Final Evaluation 102 Online Survey of Private and Public Organizations The evaluation used two types of surveys: i) a survey of beneficiary public and private organizations informing on organizations’ research and institutional capacity, and ii) a survey of members of public and private organizations informing on individuals’ perceptions of the effectiveness of training and outcomes with respect to changes in knowledge, skills, attitudes, and adoption. The evaluation team managed and monitored both surveys online without involving a subcontractor. CIMMYT Pakistan shared a list of 142 heads/principal investigators from 92 organizations that have worked with AIP, 130 of whom had both contact numbers and email addresses on the list. Six had phone numbers but no email addresses; the team contacted them by phone to obtain their email addresses. The link to the online organization-level survey and a questionnaire in Microsoft Word format were emailed to all 136 heads/principal investigators requesting them to provide information on the organization’s research and institutional capacity using the organization’s records and performance data. Seven days after sending the survey link, a gentle reminder email was sent to those who did not respond to the survey. If the reminder email did not produce a response within five days, the evaluation team contacted respondents by telephone twice with a gap of a week and asked them either to complete the survey using the online link or to fill in the Word document and return it. Despite repeated attempts, only 30 heads/principal investigators filled in the online survey or sent back the completed questionnaire in Word format. The link to the online survey was emailed to 467 members requesting them to provide information on individuals’ perceptions of the effectiveness of training and outcomes with respect to changes in knowledge, skills, attitudes, and adoption. Seven days after sending the survey link, a gentle reminder email was sent to those who did not respond to the survey. If the reminder email did not produce a response within five days, the evaluation team contacted respondents by telephone three times with a gap of three days and asked them to complete the survey either using the online link or on the phone. Despite repeated attempts, the team managed to complete the interview with only 262 respondents (both online and on the phone). In the data cleaning stage, the team found that there were 32 invalid interviews, and these were excluded from the analysis. Agricultural Innovation Program – Final Evaluation 103 ANNEX 6: BIBLIOGRAPHY Abedullah, S. Kouser, M.S. Mushtaq, and M.N.M. Ibrahim. Willingness to Pay (WTP) for Aflatoxin-Free Milk in Pakistan: Islamabad, Faisalabad & Lahore. Pakistan: ILRI, 2016. CIMMYT. Agricultural Innovation Program for Pakistan Work Plan: February 07, 2014. USAID, 2014. Project Internal Document. ———. Agricultural Innovation Program for Pakistan Work Plan: October 2014. USAID, 2014. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Work Plan: October 2015. USAID, 2015. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Work Plan: October 2016. USAID, 2016. Project Internal Document. ———. Agriculture Innovation Program Monitoring, Evaluation and Learning (ME&L) Plan. USAID, 2014. Project Internal Document. ———. Agricultural Innovation Program for Pakistan Report: January–June 2013. USAID, 2013. Project Internal Document. ———. Agricultural Innovation Program for Pakistan Semi-Annual Report: April 01–September 30, 2013. USAID, 2013. Project Internal Document. ———. Agricultural Innovation Program for Pakistan Semi-Annual Report: October 01–March 31, 2014. USAID, 2014. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Semi-Annual Report: April 01–September 30, 2014. USAID, 2014. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Semi-Annual Report: October 01, 2014 to March 31, 2015. USAID, 2015. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Semi-Annual Report: April 01, 2015 to September 30, 2015. USAID, 2015. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Semi-Annual Report: October 01, 2015 to March 31, 2016. USAID, 2016. Project Internal Document. ———. Agricultural Innovation Program for Pakistan (AIP) Semi-Annual Report: April 01, 2016 to September 30, 2016. USAID, 2016. Project Internal Document. ———. Maize Baseline Survey Punjab Report. USAID, 2015. Project Internal Document. ———. Maize Baseline Report – Khyber Pakhtunkhaw [sic] Province. USAID, 2015. Project Internal Document. ———. Wheat Baseline Report – Khyber Pakhtunkhaw [sic] Province. USAID, 2014. Project Internal Document. Agricultural Innovation Program – Final Evaluation 104 ———. Wheat Baseline Survey Report – Punjab Province. USAID, 2014. Project Internal Document. ———. Wheat Baseline Survey Report – Sindh Province. USAID, 2014. Project Internal Document. Duncan, Alan, Luke York, Ben Lukuyu, Arindam Samaddar, and Werner Stür. Feed Assessment Tool (FEAST) Focus Group Discussion Guide. Pakistan: ILRI, 2014. Project Internal Document. Management Systems International. Scaling up—From Vision to Large-Scale Change: A Management Framework for Practitioners. 2nd ed. MSI, 2012. http://www.msiworldwide.com/wp￾content/uploads/Scaling-Up-Framework.pdf. Mariner, Jeffrey. Thermostable Peste des Petits Ruminants Vaccine: Training Manual. ILRI and Tufts University School of Veterinary Medicine, 2014. Project Internal Document. Mustafa, Zeeshan, Abedullah, M. Shakeel Ashraf, Zahid Manzoor, Shahab Ali, and M.N.M. Ibrahim. Assessment and Evaluation of Constraints Faced by Dairy Farmers in Punjab and Khyber Pakhtunkhwa. USAID, 2015. Mustafa, Zeeshan, Abedullah, Nadhem Mtimet, Aftab Ahmad, Isabelle Baltenweck, Jane Poole, Nils Teufel, and M.N.M. Ibrahim. Dairy Value Chain Rapid Assessment in Districts of Punjab: Challenges and Way Forward. USAID, 2015. Nasir, M., Anwar M. Zubair, M. H. Shah, A. Ali, and K.M. Zahid Ullah. Baseline Report: Improved Mungbean Cultivation in World Vegetable Center Project Areas of Pakistan. Taiwan: World Vegetable Center, 2016. Nasir, Mohammad. Baseline Report: Vegetable Market Intermediaries in Pakistan. Taiwan: World Vegetable Center, 2015. NIGAB and NARC. Training on Field Application of Ultrasonograp[h]y in Small Ruminants. Islamabad: National Institute for Genomics and Advanced Biotechnology (NIGAB) and NARC, 2016. Project Internal Document. Saleem, Tariq and Akhter Ali. Maize Baseline Report – Azad Jamu & Kashmir and Gilgit Baltistan. USAID, 2014. Project Internal Document. ———. Maize Baseline Report – Sindh Province. USAID, 2015. Project Internal Document. Shah, Hassnain, Waqar Akhtar, Nadeem Akmal, Muhammad Azeem Khan, Waqas Farooq, Muhammad Islam, Girma T. Kassie, Barbara Rischkowsky, and Abdul Razzaq. “Small Ruminant Value Chain – Rapid Assessment: Village Level Case Studies from District Bahawalpur, Pakistan.” AIP unpublished draft, last modified April 2015. Shah, Hassnain, Waqar Akhtar, Nadeem Akmal, Tariq Hassan, Waqas Farooq, Muhammad Islam, Girma T. Kassie, Barbara Rischkowsky, and Abdul Razzaq. “Rapid Assessment of Small Ruminant Value Chain in Chakwal District, Pakistan.” AIP unpublished draft, last modified August 2015. UC Davis. Baseline Survey Report of Farmer Field Training Centres (FFTC) for Guava Growers. USAID, 2016. Project Internal Document. UC Davis. Baseline Survey Report of Pistachio Projects. USAID, 2016. Project Internal Document. Agricultural Innovation Program – Final Evaluation 105 ANNEX 7: SAMPLING Key Informant and Group Interviews TABLE 4: GROUP INTERVIEWS – RESEARCH ORGANIZATIONS BY SECTOR Group Interviews Cereals Vegetables Horticulture Multisector Total Research institutions 6 1 2 4 13 Total 6 1 2 4 13 TABLE 5: GROUP INTERVIEWS – BY GENDER Interviews No. of Participants Total Male Female Group Interviews 203 28 231 TABLE 6: GROUP INTERVIEWS – RESPONDENT TYPE BY GENDER Interviews Research Institutions Farmers Trainees Total Male Female Male Female Male Female Group interviews 42 0 124 19 37 9 231 TABLE 7: GROUP INTERVIEWS – RESEARCH ORGANIZATIONS BY PROVINCE Group Interviews Punjab KP Sindh Balochistan ICT Total Research institutions 5 2 3 2 1 13 Total 5 2 3 2 1 13 TABLE 8: GROUP INTERVIEWS – BENEFICIARY FARMERS Province Cereal Livestock Vegetable Horticulture Multisector Total Punjab 7 6 1 3 - 17 Sindh 3 1 2 - - 6 KP 6 1 - - 1 8 Balochistan 2 - 1 1 1 5 ICT - - - - 1 1 Totals 18 8 4 4 3 37 TABLE 9: GROUP INTERVIEWS – OTHER TRAINEES Provinces Extension Workers and Technicians Student Teacher Cereal Livestock Horticulture Multisector Total Punjab 2 - 1 - 3 3 5 Sindh - - 2 - 2 Agricultural Innovation Program – Final Evaluation 106 Provinces Extension Workers and Technicians Student Teacher Cereal Livestock Horticulture Multisector Total KP 1 - - - 1 Balochistan - - - 1 1 Totals 3 - 3 1 7 3 5 TABLE 10: KEY INFORMANT INTERVIEWS Respondent Type Number of Interviews USAID/Pakistan EGA office 1 Experts on cereals, livestock, vegetables, and horticulture 14 Pakistan Agricultural Research Council (PARC) 1 Ministry of National Food Security and Research (MNFSR) 1 Provincial governments (policymakers) 1 Provincial line departments 5 Trainers (who provide training in four focus sectors) 3 CIMMYT and other IPs (IRRI, ILRI, AVRDC, and UC Davis) 7 Research institutions and partner organizations 17 Total 50 Surveys CIMMYT shared a list of 8,682 beneficiaries from public and private research organizations. Out of these 8,682, the list provided phone numbers for only 6,234, and email addresses for none. From the 6,234 members with phone numbers, 1,238 unique members were identified manually using computerized national identity card (CNIC) number, phone number, organizational association, geographic information, name, and father’s name. All of the 1,238 uniquely identified members were contacted (maximum up to 4 times) on the given phone numbers to obtain their email addresses. The team managed to obtain email addresses from 467 members, and another 34 confirmed their availability for interviews over the phone as they did not have email addresses. Thirty-one members refused to take part in the survey because they either were not interested or could not recall the AIP activities so were unable to answer any questions regarding AIP. The remining 467 members could not be contacted because their phone numbers as listed were not correct, their phones were switched off, or calls were not answered despite repeated attempts. It is pertinent to mention here that 239 members contacted by the team reported they were not beneficiaries of AIP. The team managed to complete the interview with 262 respondents (49 percent response rate). Figure 20 depicts the sampling process. Agricultural Innovation Program – Final Evaluation 107 FIGURE 20: INDIVIDUAL SURVEY SAMPLING TABLE 11: INDIVIDUAL SURVEY RESPONDENTS BY SECTOR Subsector Number of Respondents Cereals 168 Livestock 33 Vegetable 37 Horticulture 35 Total 26215 From data provided by CIMMYT, the evaluation team identified the heads of all 92 AIP-supported research organizations and 532 organization members who had participated in AIP activities. The evaluation team invited these organization heads and members to respond to the online survey. After repeated telephone reminders, 273 organization members (168 from the cereal subsector, 33 from livestock, 37 from vegetables, and 35 from horticulture) responded to the survey for a response rate of 51 percent. Of these, 19 were women. Only 36 organization heads responded to the survey, a response rate of 39 percent, which is relatively high for an online survey. 15 The total number here is different (less) than the sum of the respondents from all sectors because some of the respondents work in multiple sectors. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523