EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS JULY/2023 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Ghazanfar Hoti, Tariq Husain, Aftab Ismail Khan, Fatima Abbas, Ilhamuddin, Dr. Farmanullah and Associates in Development Private Limited. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 1 Photo Credit: The photo on the cover page is of the field trials of wheat seed varieties at National Agriculture Research Centre (NARC). The photo was taken by the Evaluation Team Leader, Mr. Ghazanfar Hoti, in November 2022 during his data collection visit to the NARC. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 2 ABSTRACT This Performance Evaluation of Pakistan Agriculture Programs (PAP) focuses on four key sub-activities i.e., Wheat Productivity Enhancement Project (WPEP), Cotton Productivity Enhancement Project (CPEP), Phytosanitary Risk Management Program (PRMP) and Agriculture Service Providers Training Program (ASPTP), as the PAP invested most of its resources in these interventions. The evaluation included the following questions. ● Evaluation Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? ● Evaluation Question 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? ● Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Data collection methods included Key Informant Interviews (KIIs) and Group Interviews (GIs). Overall, the team conducted 88 KIIs and 17 Group Interviews (GIs) for the evaluation. In addition, the evaluation team also relied on data available from secondary sources including the projects’ records and reports available from implementing partners (IPs), sub-partners, and government agencies. Findings and Conclusions Through WPEP’s direct support, 21 new varieties of wheat were approved and released by the Provincial Seed Councils (PSC), accounting for about a third of all wheat varieties released during the last decade in Pakistan. On the other hand, three cotton varieties developed through CPEP support were approved and released by the PSC. Most of the interventions on development of seed varieties initiated through WPEP and CPEP were continued by the respective national partner research organizations. At the same time, the partner organizations benefited from the equipment and materials provided through these projects. However, many of the partner organizations suffered from lack of financial resources, which could potentially cause reduction in scale of activities in the varietal development process and affect the operation and maintenance of equipment and machinery provided to the research organizations through the projects’ support. In terms of successful adoption of technologies introduced through ASPTP, Gypsum bed planting in Punjab, Ridge planting in Sindh and Zero-till drill in Balochistan were successful because of effective field demonstration by Agriculture Service Providers (ASPs), and increased income for farmers. Banana chopper in Sindh, and Fertilizer Prediction Model (FPM) and happy seeder in Punjab, were not successfully adopted. For other technologies, the evaluation remains inconclusive. Most ASPs and farmers reported an increase in their annual income. Also, most ASPs reportedly continued provision of services to the farmers and the farmers continued using the services provided by ASPs. As for mitigating the impact of pests, PRMP was effective in reducing infestation and the use of chemical sprays and increasing farmers’ income during the life of the project. However, after the project’s closure, these gains were lost as most farmers stopped using biocontrol agents on their farms and reverted to the old practice of using chemical spraysfor multiple reasons elaborated in the report. Overall, both men and women benefited from the interventions under PAP’s four sub-activities. However, women’s participation remained low and was mainly limited to project supported capacity building activities. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 3 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS Evaluation Report July/2023 Contract Number: 391-USDA-13-0001 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 4 TABLE OF CONTENTS ABSTRACT 2 ACRONYMS 7 EXECUTIVE SUMMARY 9 EVALUATION PURPOSE AND EVALUATION QUESTIONS 9 BACKGROUND 9 EVALUATION DESIGN, METHODS, AND LIMITATIONS 11 FINDINGS AND CONCLUSIONS 11 RECOMMENDATIONS 15 EVALUATION PURPOSE AND EVALUATION QUESTIONS 16 EVALUATION PURPOSE 16 EVALUATION QUESTIONS 16 GENDER INTEGRATION 17 BACKGROUND 18 WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) 18 COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) 19 AGRICULTURE SERVICE PROVIDERS TRAINING PROGRAM (ASPTP) 20 PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) 21 EVALUATION METHODS AND LIMITATIONS 23 GENDER INTEGRATION 23 DATA COLLECTION METHODS 23 SECONDARY DATA 23 PRIMARY DATA 24 DATA ANALYSIS 25 METHODOLOGICAL STRENGTHS AND LIMITATIONS 25 EVALUATION TEAM COMPOSITION 26 FINDINGS AND CONCLUSIONS 26 WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) 27 FINDING 1 – INTENDED AND UNINTENDED OUTCOMES 27 FINDING 2 – SUSTAINABILITY 29 FINDING 3 - LESSONS LEARNED 30 FINDING 4 - GENDER INTEGRATION 31 CONCLUSIONS 31 COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) 32 FINDING 1 - INTENDED AND UNINTENDED OUTCOMES 32 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 5 FINDING 2 - SUSTAINABILITY 34 FINDING 3 – LESSONS LEARNED 35 FINDINGS 4 - GENDER INTEGRATION 35 CONCLUSIONS 36 AGRICULTURE SERVICES PROVIDERS TRAINING PROGRAM 37 FINDING 1 – INTENDED AND UNINTENDED OUTCOMES 37 FINDING 2 – SUSTAINABILITY 47 FINDING 3 - KEY LESSONS LEARNED 48 FINDING 4 - GENDER INTEGRATION 48 CONCLUSIONS 49 PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) 50 FINDING 1 – INTENDED AND UNINTENDED OUTCOMES 50 FINDING 2 – SUSTAINABILITY 53 FINDING 3 - KEY LESSONS LEARNED 55 FINDING 4 – GENDER INTEGRATION 55 CONCLUSIONS 55 RECOMMENDATIONS 57 ANNEXES 58 ANNEX I: EVALUATION STATEMENT OF WORK 58 ANNEX II: TIMELINE 67 ANNEX III: EVALUATION DESIGN DOCUMENT 69 ANNEX IV: EVALUATION DATA ANALYSIS PLAN 112 ANNEX V: DATA COLLECTION AND ANALYSIS TOOLS 125 ANNEX VI: SOURCES OF INFORMATION 165 ANNEX VII: LIST OF KEY INFORMANTS 166 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST 172 ANNEX IX: EVALUATION TEAM MEMBERS 173 ANNEX X: DETAILS OF APPROVED AND CANDIDATE WHEAT VARIETIES BY PARTNER RESEARCH INSTITUTES 175 ANNEX XI: LIST OF APPROVED VARIETIES IN PAKISTAN 185 ANNEX XII: ASP SELECTION CRITERIA FOR ASPTP 190 LIST OF TABLES Table 1 Background Information for Pakistan Agriculture Programs (PAP) .........................................................18 Table 2 Summary of Data Collection Methods by Project and Province ..............................................................23 Table 3 International Germplasm Evaluated at NARC between 2010 and 2017..................................................27 Table 4 List of Wheat Varieties Developed with WPEP’s Support........................................................................28 Table 5 Germplasm Acquisition and Screening ....................................................................................................32 Table 6 Cotton Varieties at Various Stages of Approval.......................................................................................33 Table 7 Changes in Income of ASPs......................................................................................................................45 Table 8 Changes in Income of Farmers.................................................................................................................45 Table 9 Changes in Yield .......................................................................................................................................46 Table 10 Summary of KII Responses from Interviews ..........................................................................................50 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 6 LIST OF IMAGES Image 1 Tissue Culture Storage Facility and Glasshouse......................................................................................30 Image 2 Laser Land Leveler in Operation .............................................................................................................37 Image 3 Unloading of Gypsum at ASP Shop (left); Soil Application of Gypsum (right).........................................38 Image 4 Zero-Till Drill in Operation in District Jaffarabad ....................................................................................39 Image 5 Drip Irrigation in Balochistan ..................................................................................................................40 Image 6 Wheat planted on Ridges in District Nankana Sahab .............................................................................41 Image 7 Rice Residue burning by Farmers (left); Happy Seeder in operation (right)...........................................42 Image 8 Banana Residue Chopper in Operation...................................................................................................43 LIST OF FIGURES Figure 1 Factors Enabling Adoption of Technologies............................................................................................44 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 7 ACRONYMS ARI Agriculture Research Institute ASP Agriculture Service Provider ASPTP Agriculture Service Providers Training Program ATI Agriculture Training Institute BARDC Balochistan Agricultural Research and Development Center BARI Barani Agricultural Research Institute BNS Breeder Nuclear Seed CLCuV Cotton Leaf Curl Virus CABI Center for Agriculture and Biosciences International CCRI Cereal Crops Research Institute CDRI Crop Disease Research Institute CIMMYT International Maize and Wheat Improvement Center CPEP Cotton Productivity Enhancement Project DPL Daily Paid Labor DUS Distinctness, Uniformity and Stability CSG1 Climate and Sustainable Growth FFS Farmer Field Schools FIDA Farmers’ Integrated Development Association FISH Fluorescence In Situ Hybridization FPM Fertilizer Prediction Model FSC&RD Federal Seed Certification and Registration Department GI Group Interview GB Gilgit Baltistan GOP Government of Pakistan HEIS High Efficiency Irrigation System ICARDA International Center for Agricultural Research for Dry Areas KIIs Key Informant Interviews NARC National Agriculture Research Centre NIBGE National Institute of Biotechnology and Genetic Engineering NIFA Nuclear Institute for Food and Agriculture NUWYT National Uniformity Wheat Yield Trials NWDSN National Wheat Diseases Screening Nurseries 1 Climate and Sustainable Growth (CSG) Office is the new name of Economic Growth and Agriculture (EGA) office. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 8 OECD-DAC Organization for Economic Cooperation and Development-Development Assistance Committee PAP Pakistan Agriculture Programs PARC Pakistan Agriculture Research Center POs Partner Organizations PCCC Pakistan Central Cotton Committee PCRWR Pakistan Council of Research in Water Resources PFVA All Pakistan Fruit and Vegetable Exporters, Importers and Merchants Association PRMP Phytosanitary Risk Management Program PSC Provincial Seed Councils RARI Regional Agricultural Research Institute REAP Rice Exporters Association of Pakistan SACAN South Asian Conservation Agriculture Network SARC Southern-zone Agricultural Research Center SAWCRI Soil and Water Conservation Research Institute SOW Statement of Work UAP University of Agriculture Peshawar Ug99 Uganda 1999 (Wheat Stem Rust Disease) USAID U.S. Agency for International Development USDA United States Department of Agriculture USG United States Government VEC Variety Evaluation Committee WPEP Wheat Productivity Enhancement Project WADO Women Agricultural Development Organization WOS Women Open School WRI Wheat Research Institute FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 9 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The primary purpose of this evaluation is to identify the achievements of selected components of the Pakistan Agriculture Programs (PAP) and inform management decisions on ongoing and future Climate and Sustainable Growth (CSG) activities. The findings will be highly useful to guide planning, design, implementation, and other related decisions that are likely to improve performance of similar investments in the future. The evaluation focused on four key sub-activities i.e., Wheat Productivity Enhancement Project (WPEP), Cotton Productivity Enhancement Project (CPEP), Phytosanitary Risk Management Program (PRMP) and Agriculture Service Providers Training Program (ASPTP), because the PAP invested most of its resources in these interventions. The primary audience for the evaluation includes • the USAID/Pakistan mission (particularly the CSG team) • the USAID Bureau for Asia (which includes Pakistan) • the implementing partner, US Department of Agriculture (USDA) including sub-awardees • the Government of Pakistan (GOP) The evaluation answers the following questions. 1) To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? a) How successful was the development of disease resistant wheat and cotton varieties? b) For ASP sub-activity, this question provides information on what technologies were most successful in terms of adoption by farmers and why? The question indicates if the adoption of technologies increased incomes of ASPs and farmers. c) What was the effectiveness of the PRMP and to what extent did it help to resolve plant health issues in the areas of project intervention which included Punjab, Sindh, Balochistan and Gilgit-Baltistan. 2) Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led to or deterred sustainability? 3) What are the lessons learned for future design and mechanism of similar projects? BACKGROUND The objective of the Pakistan Agriculture Programs (PAP) was to increase Pakistan's agricultural productivity by improving skills in research and development, introducing new varieties, and by increasing adoption of new agricultural practices and technologies. Through PAP, the USDA directly supported the Government of Pakistan’s efforts to increase agricultural productivity; build institutional capacities to better regulate animal and plant health, and food safety issues; and strengthen institutional research capabilities and scientific collaboration between the US and Pakistan. PAP had the following 14 sub activities out of which only four were included in the evaluation. 1. Watershed Rehabilitation and Irrigation improvement Project 2. Improving soil health and fertility through Extension 3. Foot and Mouth Disease Control and Surveillance Program 4. Peste des Petits (Pest) of the Small Ruminants-PPR Control and Surveillance Programs FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 10 5. Dairy Productivity Improvement Project 6. Animal Emergency Response and Mitigation program 7. Wheat productivity Enhancement (WPEP) 8. Cotton Productivity Enhancement (CPEP) 9. SPS Distance Learning Project 10. Phytosanitary Risk Management Project (PRM) 11. Water Dialogue 12. Aquaculture 13. Agriculture Service Provider Training Program (ASP) 14. Safer Food Through Aflatoxin Control in Pakistan A brief description of the four sub-activities included in the evaluation is given below. WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) Pakistan is a front-line state in the arms race between wheat farmers and wheat rusts. Large scale epidemics of leaf or yellow rust have caused huge losses in crop yield during the last 50 years, and stem rust, including the potential threat of Ug99 lineage, is a looming threat. To prevent future threats and safeguard Pakistan wheat production, USAID and USDA launched WPEP in 2010 as an outcome￾driven scientific collaboration with the International Maize and Wheat Improvement Center (CIMMYT), Pakistan Agricultural Research Council (PARC), and 11 regional Pakistani scientific organizations. The project was implemented from 2010 to 2021 by CIMMYT with the oval goal of development of high-yield and disease-resistant wheat varieties across all provinces of Pakistan. COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) Cotton is one of Pakistan’s most important crops, yet by the mid-1990s, the prevalence of the cotton leaf curl virus (CLCuV) had seriously limited production. USDA lists CLCuV as a top threat to U.S. cotton production and designed CPEP so that both the U.S. and Pakistan productivity would benefit and strengthen U.S. trade. U.S. cotton growers export their higher-grade cotton to Pakistan and other countries, where it is blended with local varieties for textile production. CPEP aimed to mitigate the effects of the chronic and lethal CLCuV disease especially for smallholder farmers, and then finding a source of resistance, and working with Pakistani breeders on developing resistant varieties. The project was implemented from 2010 to 2021 by the International Center for Agricultural Research in the Dry Areas (ICARDA) in collaboration with the Agricultural Research Service at USDA and national research organizations in Pakistan. AGRICULTURE SERVICE PROVIDERS TRAINING PROGRAM (ASPTP) Agriculture Service Providers Training Program (ASPTP) was implemented by the International Center for Agricultural Research in Dry Areas (ICARDA) from 2018 to 2021, in collaboration with USDA and five national organizations. A key objective of ASPTP was to increase adoption of selected technologies and practices by the farmers through service provision by the trained Agriculture Services Providers (ASP). These ASPs were trained on the technical knowledge, benefits and use of selected technologies and business entrepreneurial skills. PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) Farmers use pesticides that contain toxic chemicals, which have both an environmental impact and deleterious consequences for human health. Therefore, the Centre for Agriculture and Biosciences International (CABI) implemented Phytosanitary Risk Management Program (PRMP) from 2014 to FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 11 2019, to strengthen national agricultural system for adoption of eco-friendly technologies and to mitigate the impact of pre- and post-harvest pests of selected crops. EVALUATION DESIGN, METHODS, AND LIMITATIONS Data collection methods included Key Informant Interviews (KIIs) and Group Interviews (GIs) using purposive sampling for selection of respondents. Overall, the team conducted 88 KIIs and 17 GIs for the evaluation. In addition to the primary data, the evaluation team also reviewed data from secondary sources including the projects’ records and reports available from government agencies. A key limitation of the evaluation was reliance on qualitative data, since the results of qualitative data are not as generalizable as a statistically representative sample. Another limitation of the evaluation was reliance on retrospective perceptions of respondents, considering that some projects had ended years ago. To the extent possible, the team addressed these limitations by triangulation of data across various sources to ensure validity and reliability of the findings. FINDINGS AND CONCLUSIONS The aforementioned three evaluation questions are about: 1) achievement of project’s outcomes as stipulated in respective sub-questions; 2) likelihood of sustainability of the projects’ interventions (or outcomes); and 3) any lessons learned. As the evaluation findings and corresponding conclusions are distinct for each of the selected sub-activity or project, the findings and conclusions are therefore presented separately for each project. WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) FINDINGS Outcome – Development of Disease Resistant Wheat Varieties A total of 21 new varieties of wheat were developed by six partner research institutes from germplasm provided through WPEP support and were approved and released by the respective Provincial Seed Councils (PSC)2 . Furthermore, three varieties were pending approval of the respective PSCs, having completed the two years of National Uniformity Wheat Yield Trials (NUWYT)3 ; 18 varieties were being evaluated in the second year of NUWYT and 14 candidate varieties were being evaluated in the first year of NUWYT. According to Federal Seed Certification and Registration Department (FSC&RD)4 data, overall 71 new varieties have been approved and released in Pakistan since 2011, and 42 have been approved and released since 2017. Data from partner organizations and FSC&RD revealed that WPEP contributed to development and approval of 21 new disease resistant varieties, accounting for about a third of the total wheat varieties approved in Pakistan since 2011 and half since 2017. Sustainability The key findings for sustainability of WPEP’s interventions (and results) are as follows. ● Key informants from all partner organizations reported the continuation of WPEP’s intervention for development of new seed varieties. However, many partner organizations 2 PSC is the final approving authority for approval and release of seed variety in a province. 3 NUWYT are conducted by the National Wheat Coordinator at Pakistan Agriculture Research Council (PARC) to test promising breeding material/strains developed by breeders in Pakistan for adaptability under various agroclimatic conditions in the country, prior to their approval for general cultivation. These trials are conducted annually, and every new variety is tested twice in two consecutive years. 4 FSC&RD is responsible for testing the candidate seed varieties for Distinctness, Uniformity and Stability (DUS) and provides its recommendation to PSC. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 12 faced financial constraints in keeping up with the scale of activities implemented under WPEP and reported a reduction in activities. ● According to key informants, the partner organizations had been benefiting from the use of equipment and materials provided through WPEP. However, the continued use of the equipment depended on the availability of financial resources for its maintenance and operation. Lessons Learned Only national research organizations and seed corporations are entitled to produce pre-basic seed. The pre-basic seed produced by the research organizations is insufficient to meet the demand of the seed companies to produce basic, certified, and approved wheat seed for commercial use. One of the main reasons for the insufficient supply by research organizations is the limited availability of land for sowing seed to produce pre-basic seed. CONCLUSIONS WPEP supported the development of more than a third of wheat varieties approved and released during the last decade. The interventions initiated through WPEP support, including seed variety development and conducting wheat traveling seminars continued after the project ended. However, some of the activities suffered a reduction in scale due to financial constraints, and the likelihood of their continuation depends on the availability of financial resources to the research organizations. As far as commercialization of the new varieties is concerned, the partner research organizations are constrained by the capacity for facilitating commercial production of the new varieties. COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) FINDINGS Outcome – Development of Disease Resistant Cotton Varieties Of the 5,081 sets of germplasm accessions received by partner organizations over a period of eight years, 63 lines were found to be resistant to CLCuV. Most of the accessions identified as resistant during the screening process were infertile or produced small bolls. However, a germplasm (namely Mac-7) was identified as CLCuV resistant, while also having other promising characteristics suitable for breeding (crossing) with local germplasm. Through crossing (hybridization) of highly resistant accession acquired through CPEP with high yielding but susceptible local germplasm, new disease resistant cotton varieties were developed by CPEP’s partner research organizations. Three varieties developed through CPEP support were approved by Punjab Provincial Seed Council (PSC) and another eight were at various stages of the approval process. Sustainability Although the variety development interventions initiated under CPEP continued beyond the life of the project, the scale of implementation of activities reportedly varied among partner organizations. Whereas the respondent from NIBGE reported the organization was operating at full scale due to availability of funding from multiple sources, respondents from the other two research organizations reported operations at reduced scale due to diminished financial resources after conclusion of CPEP. In terms of sustainability of benefits of CPEP, the partner organizations reported that they had been benefiting from the continued use of equipment and materials provided through CPEP. However, since the storage facility provided through CPEP's support was not operational, the long-term storage of the germplasm had been compromised to the point of potential wastage of germplasm. None of the three approved three cotton varieties developed with CPEP support were commercially produced. The main reason cited for this was that the partner research organizations were constrained by their capacity for facilitating commercial production of the new varieties. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 13 Lessons Learned The aim of CPEP was to mitigate the effects of the chronic and lethal Cotton Leaf Curl Virus (CLCuV) disease especially for smallholder farmers. However, none of the three approved varieties developed through CPEP’s support had been commercially produced at the time of this evaluation. The research institutes did not have the capacity to provide adequate pre-basic seed to seed producing companies, and therefore, they were unable to produce commercially available seed for the farmers. CONCLUSIONS CPEP was successful in the development of disease resistant cotton varieties, to the extent of approval and release of three new cotton varieties by the PSC. The interventions initiated through CPEP’s support, including seed variety development, continued after the project ended. However, some of the activities suffered a reduction in scale due to financial constraints, and the likelihood of their continuation will depend on the availability of financial resources to the research organizations. As for commercialization of new varieties, the research organizations are constrained by the capacity for facilitating commercial production of new varieties. AGRICULTURE SERVICE PROVIDERS TRAINING PROGRAM (ASPTP) FINDINGS Outcome – Adoption of Technologies by Farmers and Changes in Income According to ASPs and farmers interviewed, gypsum and bed planting were successful in Punjab, ridge planting in Sindh, and zero-till drill was successful in Balochistan. The key factors that enabled the success of these technologies included demonstration of technologies in the field by ASPs, and increased income of farmers due to higher yield and reduced cost of inputs after adoption of the respective technologies. While banana chopper in Sindh, and Fertilizer Prediction Model (FPM) and happy seeder in Punjab were not successfully adopted, other technologies including laser land leveling, soil testing kit, High Efficiency Irrigation System (HEIS) and biozote had mixed results. Based on the feedback from farmers and ASPs, these technologies increased farmers’ income and yield, however their adoption was affected by factors in the local context in which they were introduced, e.g., absence of local maintenance workshop in KP affected the adoption of laser land leveling which was quite successful in Sindh where required market linkages were present. According to interviews with ASPs and farmers, most of the ASPs (16 of 18) and farmers (48 out of 51) reported an increase in their annual income through adoption of new technologies. The main factor cited for the increase in income for ASPs was demand for services from farmers. Most farmers attributed the increase in their income to an increase in crop yield and reduction in input costs. However, adoption of new technology did not necessarily reduce farmers’ input costs as inputs in some cases were expensive, though in the case of leveling most of the farmers using the technology reported that the increased input cost was a one-time cost and offset by increase in income over a longer period of time. Sustainability Most of the ASPs (16 out of 18) interviewed continued to provide services to farmers after ASPTP ended, while most of the farmers (46 out of 51) interviewed continued to avail themselves of services offered by ASPs. The main reason for ASPs’ continued provision of services was the demand from farmers. Similarly, farmers reportedly continued to avail themselves of their services because of yield increase, reduced input costs, water conservation, and improved soil and product quality. Main factor that constrained sustainability among ASPs and farmers was the lack of local availability of technology. The 2022 floods also disrupted the process. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 14 Lessons Learned The ASP approach has potential for replication, but requires careful selection of technologies, in view of their marketability, given local constraints such as government subsidies on technology adoption and availability of testing labs locally. CONCLUSIONS Gypsum and bed planting technologies were successful in Punjab, while ridge planting in Sindh and zero-till drill were successful in Balochistan. On the other hand, banana chopper in Sindh and FPM and happy seeder in Punjab were not successfully adopted. The evaluation remains inconclusive as to whether other technologies such as laser land leveling, soil testing kit, HEIS and biozote were completely successful or unsuccessful. While these technologies did increase farmers’ income and yield, their adoption was affected by other factors in the local context, such as absence of local maintenance workshops. The majority of ASPs and farmers experienced an increase in annual income through adoption of new technologies. Also, most of the ASPs continued to provide services after ASPTP ended, while most of their client farmers continued to avail them. PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) FINDINGS Outcome – Mitigating the Impact of Pests Most of the key informants from partner organizations (9 out of 10) and farmers interviewed reported a reduction in infestation (11 out of 14), and use of pesticides (12 out of 14) during implementation of the project. More than half (8 out of 14) of the farmers interviewed also reported an increase in income. However, all farmers in Balochistan and Sindh (7 out of 7) reportedly resumed the use of pesticides after PRMP ended. Key reasons for resuming the use of chemical sprays were the ineffectiveness of the biocontrol agents against pest attacks (i.e., mealybug in Sindh), damage caused by floods and hailstorms, and plant diseases5 such as root rot and leaf curl diseases. Sustainability Half (6 out of 12) of the respondents from partner organizations, mostly in Gilgit (3) and Balochistan (3), believed that the respective departments had adopted the capacity building support and the technology to rear biocontrol agents shared by CABI through PRMP. However, most respondents (5 out of 6) from partner organizations in Sindh and Baltistan reported that the interventions initiated through PRMP were not continued by the respective departments. It was observed that the labs in Gilgit and Balochistan were rearing parasitoids, but the labs in Baltistan (Skardu) and Sindh (Karachi) were not. Most farmers (12 out of 14) had not been using biocontrol agents at the time of the evaluation. The key reasons cited by the farmers were the ineffectiveness of biocontrol agents against pests, damage caused by plant diseases, adverse weather, unavailability of biocontrol agents and the lack of farmers’ interest in biocontrol technology due to no follow-up by the agriculture department after the project ended. Lessons Learned PRMP was implemented over a period of five years, with approximately two years of implementation in Balochistan and Sindh, and about two years in Gilgit and Baltistan. This suggests that most beneficiary farmers were associated with the project for one to two years or one to two crop seasons. Three (out of 12) key informants from partner organizations and three farmers (out of 14) suggested that the duration of the PRMP was too short to achieve results. Similarly, more than a third of the 5 Controlling these diseases was not a part of PRMP. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 15 farmers lost interest due to lack of agriculture departments’ follow-up visits to their fields after the project ended. CONCLUSIONS PRMP was effective in mitigating the impact of pests during the life of the project. However, once the project ended, its effectiveness in the majority of the focused areas (Balochistan and Sindh) was lost, as most farmers reverted to their old practice of using pesticides to control pests. In Gilgit and Baltistan, however, some efforts continued even after the project ended. Very little of the PRMP interventions has been sustained considering half of the PRMP supported labs were not operational and the other half were producing far less parasitoids than they used to when the project was active. GENDER INTEGRATION In both WPEP and CPEP, women from partner research organizations benefited from the project’s capacity building interventions. As projects focusing on research activities, WPEP and CPEP engaged with women from partner research organizations to improve their skills in wheat research. Women participated in local and foreign training such as Borlaug Fellowship program and basic wheat improvement course. CPEP also used Farmer Field School (FFS) and Women Open School (WOS) approaches to include women in the project’s agronomy related activities. Women participation in ASPTP interventions was minimal as only a few female ASPs benefited from the trainings. The project did not specifically target women as evidenced by the fact that women participation was not included in the ASPs’ selection criteria. Key reasons cited for lack of women’s participation included challenges faced by the project in accessing women and cultural barriers to women’s participation in agricultural activities in the intervention areas. In PRMP, more men benefited from the project interventions than women. This was demonstrated by the fact that about a third of training participants from government partner organizations and a fourth of beneficiary farmers were women. The reason cited for overall low female participation was the lack of female involvement in agricultural activities in the targeted areas. Nevertheless, in Gilgit where women were actively engaged in agricultural activities, PRMP actively supported women by providing training on control of pests through biocontrol agents. This was demonstrated by the fact that more than 60 percent of the beneficiaries from the activity in Gilgit were women. RECOMMENDATIONS The recommendations for the evaluation are as follows. 1. Given the constraints of the research institutions to facilitate the commercial production of wheat and cotton varieties, USAID should consider a component of commercialization to support commercial production of cotton and wheat varieties by involving the private sector, including seed producing companies. 2. Technologies, for which machinery and spare parts are not available locally,should not be rolled out for demonstration and adoption. 3. The interventions focusing on women on small farms should be well-defined at the design stage and duly reflected in beneficiary selection and training requirements. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 16 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The primary purpose of this evaluation is to identify the achievements of selected components of the Pakistan Agriculture Program (PAP) in line with their intended objectives and inform management decisions on on-going and future Climate and Sustainable Growth (CSG) activities. It also measuresthe contribution of PAP components to the focus areas in agriculture sector of Pakistan. The findings will guide the future planning, design, implementation and other related decisions that are likely to improve performance of similar investment efforts in the future. The learnings from this evaluation will also be shared with relevant government departments to inform future interventions in agriculture research and extension. The primary audience for the evaluation includes • the USAID/Pakistan mission (particularly the CSG team) • the USAID Bureau for Asia (which includes Pakistan) • the implementing partner, US Department of Agriculture (USDA) including sub-awardees • the Government of Pakistan (GOP) The evaluation focused on four key sub-activities i.e., Wheat Productivity Enhancement Project (WPEP), Cotton Productivity Enhancement Project (CPEP), Phytosanitary Risk Management Program (PRMP) and Agriculture Service Providers Training Program (ASPTP), because the PAP invested most of its resources in these interventions. Furthermore, these interventions were expected to produce more tangible results and create long term impacts in the focus areas. EVALUATION QUESTIONS The evaluation will address the following three key questions (as identified in the Scope of Work). Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? These results include the following. a) How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. b) For ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why? The question indicates if the adoption of technologies increased incomes of ASPs and farmers. c) What was the effectiveness of the PRMP and to what extent did it help to resolve plant health issues in the areas of project intervention which included Punjab, Sindh, Balochistan and Gilgit-Baltistan. Effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Key assumptions and definitions for evaluation design ● For the purposes of the evaluation, all relevant outcomes and results have been included explicitly in evaluation sub-questions (a), (b) and (c)6 . 6 According to Organization for Economic Cooperation and Development, Development Assistance Committee (OECD-DAC) 2010, Glossary of Key Terms in Evaluation and Results Based Management, 2010, (https://www.oecd.org/dac/evaluation/2754804.pdf), outcomes are “The likely or achieved short-term and medium-term effects of an intervention’s outputs” where an intervention is “An instrument for partner (donor and non-donor) support FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 17 ● Adoption7 is defined as “a process involving an individual that includes the series of stages one undergoes from first hearing about a product to finally accepting or using it.” ● “Farmers” means beneficiary farmers who adopted at least one new technology introduced through ASPs. ● “Focused horticulture crops” refersto papaya (in Sindh), apple (in Balochistan) and fruit plants (in GB). Question 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? This question will evaluate the likelihood that the interventions under the four identified sub-activities are sustainable i.e., the varieties, practices and technologies introduced will persist beyond the project support. Specific aspects will include institutionalizing these interventions in Government (both federal and provincial) and private organizations. Question 3: What are the lessons learned for future design and mechanism of similar projects? This question investigates the major challenges and successes especially about the USDA and their sub-partners. Recommendations focus on changes in the implementation approach that could improve performance in the new Participating Agency Program Agreement and future activities. GENDER INTEGRATION For all the three key evaluation questions, the evaluation will explore whether these interventions provided equal opportunities for both men and women. If not, what could have been done to encourage women participation in the activity? If yes, what were the approaches that encouraged women to benefit from the activities? aimed to promote development. Examples are policy advice, projects, and programs.” Results are “The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention.” 7 Literature Review, Scaling Agricultural Technologies and Innovation Diffusion, May 7, 2015, https://pdf.usaid.gov/pdf_docs/pa00kfqg.pdf FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 18 BACKGROUND The objective of the Pakistan Agriculture Programs (PAP) was to increase Pakistan's agricultural productivity by improving skills in research and development, introduction of new varieties, and by increasing adoption of new agricultural practices and technologies. PAP conducted 14 sub-activities, out of which four sub-activities were included in this performance evaluation because the PAP invested most of its resources on these interventions.8 The terms “project” and “sub-activity” have been used interchangeably throughout the report. Table 1 below provides a summary of background information for PAP. Table 1 Background Information for Pakistan Agriculture Programs (PAP) Agreement Number Contract No. 391-USDA-13-0001 Activity Start Date November 14, 2012 Activity End Date September 30, 2021 Total Estimated Cost $26,100,000 Location of Activities (Provinces/Districts) Nationwide Implementing Partner United States Department of Agriculture (USDA) USAID/Pakistan Results Framework Linkages DO 3: Increased Private Sector-led Inclusive Economic Growth. IR 3.2: Employment opportunities increased Sub IR 3.2.2: Agriculture value chains enhanced and Sub IR 3.2.2a: Number of individuals in the agriculture system who have applied improved management practices and technologies with USG assistance The following is a brief description of each of the four projects. WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) Wheat Productivity Enhancement Project (WPEP) was implemented from 2010 to 2021 by International Maize and Wheat Improvement Center (CIMMYT) to protect and enhance wheat productivity in Pakistan, given the history of crop losses the country has been facing due to large scale epidemics of leaf or yellow rust and stem rust, including the potential threat of Ug99 lineage.9 Aiming for the overall goal of identification, adoption, and optimal agronomic management of new, high-yielding and disease-resistant wheat varieties, the project has been implemented through a collaboration involving USDA, CIMMYT, Pakistan Agricultural Research Council (PARC), ICARDA and 11 agriculture research organizations working in all provinces of Pakistan.10 WPEP had the following three main objectives. 11 i. To establish a country-wide surveillance program in order to determine if new strains of stem rust disease (specifically the Ug99 strain) were present. ii. Identify and develop new high yielding, adapted varieties of wheat having genetic resistance to Ug99 and other major diseases. iii. Organize cooperative country-wide testing of germplasm and varieties, as well as greater 8USAID. PAP Evaluation Statement of Work, Section C, page 7 9 Ibid. 10 Ibid. 11 Ibid., page 8 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 19 collaboration and sharing of germplasm among Pakistani breeders, and release high-yielding, adapted, disease resistant wheat varieties to all wheat growing areas in Pakistan. The project aimed to achieve these objectives through the following activities. 12 i. Rust pathogen surveillance, which involved collecting plant samples from various wheat￾growing areas in Pakistan with information on varietal distribution, growth stage and rust incidence and severity. Rust samples collected were analyzed at Crop Disease Research Institute, Murree. ii. Pre-breeding/screening was conducted for varieties to determine the nature of genetic resistances, and those susceptible to infection are removed from the breeding process. iii. WPEP aimed for accelerated breeding for developing and testing rust-resistant, high￾performance candidate wheat varieties, a process that took many years. iv. Fast replacement of susceptible commercial varieties (vulnerable to new races of wheat rusts) with new rust-resistant and high-yielding varieties, through demonstration and seed distribution among farmers. v. Agronomic management practices among farmers were introduced to enhance the application of best practices. vi. Coordination and capacity building was ensured by initiating an exchange program for Pakistani scientists (drawn from national research institutions and agricultural universities) for long-term joint collaborative training. The national scientific organizations included the following.13 i. National Agriculture Research Center (NARC), Islamabad ii. Wheat Research Center (WRC), Sakrand, Sindh iii. Nuclear Institute of Agriculture (NIA), Tando Jam, Sindh iv. University of Agriculture Peshawar (UAP), Khyber Pakhtunkhwa v. Nuclear Institute for Food and Agriculture (NIFA), Peshawar, Khyber Pakhtunkhwa vi. Cereal Crops Research Institute (CCRI), Pirsabak, Khyber Pakhtunkhwa vii. Regional Agricultural Research Institute (RARI), Bahawalpur, Punjab viii. Wheat Research Institute (WRI), Faisalabad, Punjab ix. Barani Agricultural Research Institute (BARI), Chakwal, Punjab x. Agriculture Research Institute (ARI), Quetta xi. Balochistan Agricultural Research and Development Centre (BARDC), Quetta, Baluchistan xii. Crop Disease Research Institute (CDRI) in Murree, Punjab COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) Cotton Productivity Enhancement Project (CPEP) was implemented with the goal of mitigating the effects of the chronic and lethal Cotton Leaf Curl Virus (CLCuV) disease, especially for smallholder farmers, and then finding a source of resistance and working with the Pakistani breeders on developing resistant varieties. It also aimed to find the source of resistance and put it into U.S. cotton lines to prepare in case the cotton leaf curl virus became a problem in the U.S.14 The project was implemented from 2010 to 2021 by the International Center for Agricultural Research in the Dry Areas (ICARDA) collaborating internationally with the Agricultural Research Service at the US Department of Agriculture (USDA) and nationally with the following organizations. i. Central Cotton Research Institute (CCRI), Multan 12 CIMMYT. 2015. WPEP Phase 1, Final Technical Report, September 29, 2010 – September 30, 2015, pp.5-14. 13 CIMMYT. WPEP Brochure 14USAID. PAP Evaluation Statement of Work, Section C, page 8 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 20 ii. Pakistan Central Cotton Committee (PCCC), Multan iii. National Institute of Biotechnology & Genetic Engineering (NIBGE), Faisalabad iv. Pakistan Atomic Energy Commission (PAEC), Islamabad v. Cotton Research Institute (CRI), Multan and its Research Stations at Faisalabad and Vehari, Punjab vi. Agronomy Research Stations at Bahawalpur and Khanewal, Punjab vii. Farmer’s Integrated Development Association (FIDA) at Vehari, Punjab viii. Woman Agricultural Development Organization (WADO) at Khairpur, Sindh 15 Key project objectives included assisting the cotton research and development system of Pakistan to minimize the adverse effects of CLCuV disease, conducting the inheritance study of resistance to the virus, and the development and commercialization of CLCuV-resistant or highly tolerant cotton varieties utilizing virus-resistant USDA germplasm.16 The following approaches were adopted to achieve these objectives. 17 i. Supporting cotton best management practices for small farmers ii. Enhancing Pakistan’s cotton germplasm to broaden its genetic base iii. Development and utilization of sources of resistance to CLCuV iv. Capacity building of Pakistani researchers, institutes, and farmers for advanced research and development on cotton. Seven components or sub-activities were assigned to collaborating institutes to achieve project objectives. For effective coordination across institutes, these components were grouped into the following categories based on the nature of research work. i. Germplasm Group Screening of Germplasm imported from USA against CLCuV, and to utilize virus-resistant or highly tolerant USDA accessions in breeding program, to develop virus-resistant or highly tolerant cotton varieties. ii. Gene Mapping Group Germplasm screening, population development and identification of DNA markers associated with resistance to CLCuV. iii. Virology Group Evaluation of the mechanism underlying resistance against CLCuV observed in Mac-7 18 (a variety identified as CLCuV-resistant during previous phase of the project). iv. Agronomy and Integrated Pest Management (IPM) Program Identification of best management practices to minimize the effect of CLCuV, and training of male and female small farmers for these practices through Farmers’ Field School (FFS) and Women Open School (WOS).19 AGRICULTURE SERVICE PROVIDERS TRAINING PROGRAM (ASPTP) The overarching goal of ASPTP was to build upon the success of USDA’s previous soil fertility and health, and watershed rehabilitation work in Pakistan, and to develop a roadmap for the future, involving trained Agricultural Service Providers (ASPs).20 The project objectives were twofold: to 15 ICARDA. CPEP Final Technical Report, October 2017 – December 2018, page 5 16 ICARDA. CPEP Summary of Annual Technical Progress Report, October 2019 – September 2020, page 2 17 ICARDA. CPEP Annual Progress Report, October 2013 – September 2014 18During implementation, Mac-7 variety was discovered to be a source of high resistance. 19 ICARDA. CPEP Final Technical Report, October 2017 – December 2018, page 5 20 ICARDA. Increased Capacity of ASPs to Increase Adoption of Soil Fertility and Health and Improve Water Quality and Conservation Techniques, Final Technical Report, August 2018 – July 2021, page 8 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 21 increase capacity of ASPs for technologies and entrepreneurship; to increase adoption of soil fertility and health; and improve water quality and conservation techniques and technologies.21 ASPTP was implemented by ICARDA from 2018 to 2021, in collaboration with USDA and the following five agriculture research or extension institutions and non-government organization (NGO) as focal institutions. 22 i. Agriculture Research Institute (ARI), Quetta, Balochistan ii. Agriculture Training Institute (ATI) of Agriculture Extension Department, Hyderabad, Sindh iii. Soil and Water Conservation Research Institute (SAWCRI), Chakwal, Punjab (rain-fed system) iv. South Asian Conservation Agriculture Network (SACAN), Lahore, Punjab (irrigated system) v. Pakistan Council of Research in Water Resources (PCRWR), Peshawar, Khyber Pakhtunkhwa These institutes were selected by ICARDA, along with 11 technologies23 to promote through the project. This was followed by the selection of 70 ASPs based on the selection criteria listed in Annex XII. These ASPs were then trained in the technical knowledge, benefits and use of selected technologies and business entrepreneurial skills. They were also trained to maintain record-keeping registers of the number of farmers who availed their services, along with the income earned from provided service, to be shared with partner organizations during quarterly review meetings. ICARDA and collaborating institutes also developed and strengthened the linkages of these trained ASPs with government agriculture departments, relevant industry (manufactures, producers, and traders) and NGOs to facilitate service provision of technologies to the farmers. PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) Insect pests cause crop losses in Pakistan, adversely affecting domestic food security and the country’s export potential. Farmers use pesticides that contain toxic chemicals, leading to both environmental degradation and deleterious effects on human health. Biological control reduces pest infestation in the fields without using toxic chemicals.24 Therefore, to strengthen national agricultural system for adoption of eco-friendly technologies, Centre for Agriculture and Bioscience International (CABI) initially signed a 3-year agreement with the USDA from September 1, 2014, to September 30, 2017, to implement PRMP with the following objectives. i. Improve technical capacity of national partners to survey pests and to develop and deploy biocontrol agents to reduce the impact of plant pests in Pakistan. ii. Mitigate the impact of post-harvest pests of rice and horticultural crops and improve the capacity of plant health regulators to certify exports of agricultural commodities. The above objectives were achieved through the following three components. i. Pest monitoring to support biocontrol activities. ii. Developing new and utilizing existing biocontrol technologies. iii. Development of and training in proper post-harvest pest management practices.25 21 ICARDA. Final Technical Report, October 2017 – December 2018, page 8 22 ICARDA. ASPTP Final Technical Report, August 2018 – July 2021, page 8 23 11 technologies were drip/bubbler/sprinkler irrigation system, laser land levelling, ridge planting of crops, bed planting of crops, gypsum application for soil moisture conservation, biozote application to increase crop yield, fertilizer prediction model for accurate and balanced fertilizer use, rapid soil testing kit for fertilizer recommendation for crops, zero-till drill to plant wheat after rice harvest without land preparation, happy seeder for planting wheat after mechanical harvest of rice without land preparation, and banana chopper. 24 CABI. December 2019. PRMP Project Completion Report (August 2014 – September 2019), page 4 25 CABI. August 2018. PRMP Midterm Evaluation Report (September 2014 – April 2018), page 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 22 Considering the project components and findings of a baseline survey conducted in 2014, papaya mealybug in Sindh, and apple codling moth and apple spider mites in Balochistan were identified as pests of concern for the first two components, whereas for the third component, SPS compliance in horticulture and rice were identified as focus areas for project interventions. PRMP was subjected to its first amendment in 2016 when its scope was extended to Gilgit Baltistan (GB) to replicate all the three components in this region. Fruit flies and giant mealybugs were identified as pests of concern in Gilgit and Baltistan region, respectively. The project activities were initiated to implement a biological control program with the following components. iv. Pest monitoring to support biocontrol activities in GB. v. Developing new and utilizing existing biocontrol technologies in GB. vi. Development of and training in proper post-harvest pest management practices in GB. In 2017, the project received its second amendment based on which the issue of prevalence and identification of aflatoxins in horticultural crops was included in the project, thus adding the third objective of “analyzing and generating actionable information to regulate aflatoxins in fresh produce supply chains in Pakistan”. The following was added as the seventh component of the project. vii. Aflatoxins baseline study and biological control development in Pakistan. Based on these two amendments, PRMP got an extension of two years until September 2019, with the aim to implement biological control programs for papaya mealybug, apple codling moth, apple spider mites, fruit flies and giant mealybug; build capacity of national stakeholders and farmers; ensure SPS compliance in horticultural commodities and rice value chain; and to conduct a baseline study to identify aflatoxins in horticultural commodities in Punjab and Sindh.26 To implement these activities for beneficiary farmers, CABI collaborated with Pakistan Agricultural Research Council (PARC); four Provincial Departments of Agriculture in Sindh, Punjab, Balochistan and GB; Southern-zone Agricultural Research Center (SARC) in Karachi; All Pakistan Fruit and Vegetable Exporters, Importers and Merchants Association (PFVA); and Rice Exporters Association of Pakistan (REAP).27 26 CABI. August 2018. PRMP Midterm Evaluation Report (September 2014 – April 2018), pp. 1-3 27 CABI. August 25, 2022. PRMP PowerPoint Presentation FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 23 EVALUATION METHODS AND LIMITATIONS Data for this performance evaluation was drawn from both primary and secondary sources. Primary data was collected through Key Informant Interviews (KIIs) and Group Interviews (GIs), over six weeks of field work. A purposive sampling approach was used for sample selection. The evaluation planned to conduct a total of 88 KIIs with key stakeholders including implementing partners’ staff, participating organizations’ staff, government officials and other key individuals associated with PAP, as well as 19 GIs (3 farmers in each group) with beneficiary farmers of ASPTP. Overall, the team conducted 88 KIIs and 17 GIs. To the extent possible, the required information was also collected from other available sources. Details of completed interviews are presented in Table 2 below. Table 2 Summary of Data Collection Methods by Project and Province Project Name Punjab Sindh KP Balochistan Gilgit Baltistan Islama bad Total KIIs GIs KIIs GIs KIIs GIs KIIs GIs KIIs KIIs WPEP 8 3 3 1 - 4 19 CPEP 13 1 - - - 2 16 ASPTP* 12 9 3 2 3 2 5 4 - 1 24 KIIs, 17GIs PRMP 2 8 - 6 12 1 29 Total 35 9 15 2 6 2 12 4 12 8 88 KIIs, 17 GIs * ASPTP KIIs include those conducted with 18 Agricultural Service Providers (ASPs) In addition to the primary data, the evaluation team also collected data available from secondary sources, including the projects’ monitoring data, annual and quarterly reports, previous assessments and evaluation reports, and reports available from government agencies (such as approvals and registrations). GENDER INTEGRATION The evaluation team ensured that evaluation design, methodology, data collection, analysis and report adequately captured the situation and experiences of both males and females. The evaluation employed appropriate gender-sensitive data collection methodology which informed about the constraints in women participation in agricultural activities. For all the three key evaluation questions, the evaluation explored whether these interventions provided equal opportunities for both men and women. If not, what could have been done to encourage women participation in the activity? If yes, what were the approaches that encouraged women to benefit from the activities? DATA COLLECTION METHODS The evaluation employed qualitative research techniques to collect data from multiple sources to ensure multiple levels of triangulation. The data was obtained both from primary and secondary sources. The primary data was collected through KIIs and GIs, while the secondary sources are mentioned in the following section. SECONDARY DATA During the evaluation design phase, USAID and PAP implementing partners shared the following documents with the evaluation team. i. Annual and Quarterly reports (WPEP, CPEP, ASPTP, PRMP) FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 24 ii. Project completion report (PRMP) iii. Previous evaluation and assessment reports (ASPTP, PRMP) iv. Pakistan Agriculture Programs - Final Report, 2021 v. Pakistan Agriculture Programs Quarterly Reports (Quarter 4 - 2019, 2020, 2021) vi. Pakistan Agriculture Programs work-plans (2015, 2016, 2017, 2018, 2019, 2020) vii. Participating Agency Program Agreement, Modification Number 03 viii. Pakistan Agriculture Programs Monitoring, Evaluation and Learning (ME&L) Plan PRIMARY DATA KEY INFORMANT INTERVIEWS KIIs with key stakeholders provided overall background information and helped contextualize the overall evaluation framework. KIIs provided perspectives of stakeholders on accomplishments, best practices and lessons learned, as well as other factors that inhibited or promoted success of PAP. KIIs were conducted with stakeholders from each of the projects covered in this evaluation. In addition to including questions in all KIIs to understand the extent to which the sub-activities offered equal opportunities of participation and benefit to men and women, the evaluation team made a conscious effort to select and interview female key informants, wherever possible. The details regarding the list of organizations including the number of KIIs (ASPs and farmers for ASPTP) has been given in Annex VI. WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) The evaluation team conducted KIIs with key stakeholders of WPEP including the implementing partner CIMMYT, partner research organizations, government approving bodies and seed producing companies. A total of 19 KIIs were conducted with relevant stakeholders for WPEP interventions. COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) The evaluation team conducted KIIs with key stakeholders of CPEP including the implementing partner ICARDA, partner research organizations, government approving bodies, partner Non-Governmental Organizations (NGOs) and seed producing companies. A total of 16 KIIs were conducted with relevant stakeholders of CPEP interventions. AGRICULTURE SERVICE PROVIDERS TRAINING PROGRAM (ASPTP) The evaluation team conducted KIIs with key stakeholders of ASPTP, including the implementing partner ICARDA, program partner organizations and Agriculture Service Providers (ASPs) who received training through ASPTP. ASPs were purposively selected with representation from all provinces with at least two participants for each type of service provided. A total of 24 KIIs were conducted with ASPTP relevant stakeholders, including 18 ASPs (including one female respondent). Additionally, 17 group interviews (3 participants in each group) were also conducted with beneficiary farmers. To ensure diversity in geographical locations and beneficiaries’ experiences, farmers and ASPs were selected from Districts Nawabshah (in Sindh); Mardan and Peshawar (in KP); Ziarat, Quetta and Usta Muhammad (in Balochistan); and Sheikhupura, Sialkot, Faisalabad, Khanewal, Chakwal and Vehari (in Punjab). PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) The evaluation team conducted KIIs with key stakeholders of PRMP including the implementing partner CABI, program’s partner organizations, government officials and farmers (including female farmers, where possible) who participated in PRMP. Three farmers were purposively selected for each of the four crops i.e., papaya, apple, fruit trees and willow in Sindh, Balochistan, Gilgit and Baltistan regions, respectively. This led to a total of 29 KIIs (including one female farmer) with PRMP FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 25 stakeholders. To ensure diversity in geographical locations and beneficiaries’ experiences, beneficiaries were interviewed in Sultanabad, Danyor, Rahimabad and Oshkhandas in Gilgit; Sermik, Sumbal and Sundas in Skardu; Spin Takai (Pishin District), Tor Oskai (Killa Saifullah District) and Ghazaband (Quetta District) in Balochistan; Landhi (District Karachi), Memon Goth (District Karachi), Shahdadpur (District Sanghar), and Thatta in Sindh; and District Kamoki in Punjab. GROUP INTERVIEWS In addition to KIIs with key stakeholders, the evaluation team conducted group interviews with beneficiaries/farmers of ASPTP, who received services from ASPs on ten technologies introduced under the program. During the introductory meeting with implementing partner ICARDA, it was revealed that the contact information for these beneficiaries was not available, and therefore any quantitative sample survey was ruled out. The only way to reach out to these beneficiaries was through the ASPs. Therefore, a snowball sampling and convenience sampling approach were used to collect information from these beneficiaries. During KIIs with ASPs, they were requested to provide contact information of or to gather at least three farmers who were provided services by them at a location near ASPs’ home location or meeting spot. Where ASPs had directly assisted women farmers, they were requested to ensure nomination of women beneficiary farmers in the group interview as well. During the group interviews, all respondents were asked the same set of questions. A total of 17 group interviews with 51 farmers were conducted at 12 locations across four provinces. Annex VI illustrates the details of beneficiary farmers for group interviews. DATA ANALYSIS The evaluation largely relied on qualitative data i.e., KIIs and GIs with key program stakeholders and beneficiaries. The evaluation team employed a structured approach to analysis by designing a pre￾fieldwork data analysis plan, identifying potential variables, themes, and data sources against each evaluation question. Data analysis initiated with a thorough examination of field notes for key themes (including those mentioned in the data analysis plan and emerging from the fieldwork), coding responses according to these themes in a detailed tally-sheet or frequency sheet), and reporting frequencies and other quantitative summaries of responses, when possible and appropriate. When possible, the team also conducted comparative analysis across various groups of stakeholders to ensure triangulation of data sources. Where applicable, the evaluation disaggregated findings by gender, and drew specific conclusions and recommendations. The detailed data analysis plan is attached as Annex IV. METHODOLOGICAL STRENGTHS AND LIMITATIONS The evaluation methodology relied on triangulation of sources and methods to ensure the validity and reliability of results. The scope of this evaluation was limited to answering the evaluation questions to the extent that data was available. The evaluation team conducted data collection and analysis in a highly systematic manner by triangulating across multiple sources to ensure the reliability and validity of findings. The methodology allowed for data triangulation i.e., data was drawn from across stakeholder categories included in the evaluation. As the project had ended several years ago, it was challenging to locate the beneficiaries. Only a few beneficiaries were accessible, and they either did not recall project interventions or were unwilling to be interviewed. As a result, the team could not reach out to training participants in Punjab (for rice) and was able to interview only one rice exporter for training in post-harvest pest management. In the case of ASPTP, it had already been communicated to the team that beneficiary contact information is not available, so the team had to rely on convenience/snowball sample for selection of beneficiary farmers through the ASPs. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 26 A few key informants had also retired including Director, Barani Agricultural Research Institute (BARI) in Chakwal and National Wheat Coordinator in Islamabad. To overcome this limitation, the team interviewed the incumbent officers to receive required information. Qualitative data collection for this evaluation relied on purposive sampling, meaning that respondents were selected for their roles, responsibilities, and knowledge. The sample it has produced is not statistically representative. As a result, these respondents represented a variety of views on a topic, they may not have represented all relevant views. While the approach has produced rich data, the results are not as generalizable as results from a representative sample. Due to the evaluation’s reliance on retrospective perceptions of respondents, especially in cases where the project ended over three years ago, the evaluation results may potentially suffer from recall bias. Recall bias occurs when respondents do not accurately remember previous events or experiences, e.g., PRMP key informants and beneficiary farmers could not recall exact changes in crop yield and income due to project interventions. This was more prominent among respondents in Sindh and Balochistan where the project ended more than five years ago. Some project related documents were not shared with the evaluation team. In particular, the team did not receive Participating Agency Program Agreements (PAPA) from USAID and had to utilize information available from other sources, including projects’ annual and quarterly reports. This was communicated to USAID, and they agreed with this approach. CIMMYT was also requested for all annual and quarterly reports over the period of WPEP implementation from 2010 to 2021. Only some reports were shared, because the Country Manager had changed, and the organization did not have an updated database of documents. Finally, during KIIs with ASPTP partner organizations, the team requested income data of ASPs which had been recorded in their record registers during implementation. This information was not shared by any partner organization,so the team was unable to perform a trend analysis on the effect of project interventions on ASPs’ incomes. The specific limitations pertaining to each evaluation question and data collection method are presented in “Annex III – Getting to Answers Matrix,” along with measures to address these. EVALUATION TEAM COMPOSITION The evaluation team included six core team members including a Team Leader/Evaluation Specialist, two Qualitative Research Specialists, and three sector experts on Agriculture Economics, Seed Certification and Plant Pest Control. Additionally, the team had one Field Coordinator who supervised a field research team of four members to interview ASPs and beneficiary farmers for ASPTP across all provinces. Data collection, spanning over a duration of six weeks, lasted from November 14 to December 23, 2022. FINDINGS AND CONCLUSIONS The three evaluation questions mentioned above are about: 1) achievement of the projects’ intended (and unintended) outcomes; 2) likelihood of the sustainability of projects’ interventions (or outcomes); and 3) any lessons learned. As for the intended outcomes, the evaluation focused on specific outcomes or objectives, which were stipulated in the sub-questions under Evaluation Question No. 1. Specifically, the three sub-questions assessed the development of new disease￾resistant wheat and cotton varieties initiated under WPEP and CPEP respectively; adoption of technologies by farmers introduced under ASPTP, and changes in income of ASPs and farmers; and effectiveness of PRMP in terms of mitigating the impact of pests in targeted areas. As the findings and corresponding conclusions for the three evaluation questions are distinct for each of the selected FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 27 projects, the sections on findings and conclusions are therefore presented separately for each project. Additionally, the report also includes a section on ‘gender integration’ for each project. WHEAT PRODUCTIVITY ENHANCEMENT PROJECT (WPEP) FINDING 1 – INTENDED AND UNINTENDED OUTCOMES In this section, the evaluation assessed WPEP’s success in terms of development of new disease resistant wheat varieties approved by respective approving agencies in Pakistan. According to information received from partner research organizations, the varieties were mostly developed by using ‘selection’28 method. This method included the following steps. ● Micro Yield Trials: During this stage, the germplasms are evaluated at the research stations and promising lines are selected. ● Zonal Yield Trials: At this stage, the promising lines are evaluated at multiple locations for adaptability performance. ● National Uniform Wheat Yield Trial (NUWYT): Candidate varieties contributed from various institutions across Pakistan are tested nationally across 40 locations. Simultaneously, the candidate lines are tested in National Wheat Diseases Screening Nurseries (NWDSN) for diseases, various agronomic traits and for Distinctness, Stability and Uniformity (DUS) by Federal Seed Certification and Registration Department (FSC&RD), over a period of two years. The varieties are also tested for Value for Cultivation and Use (VCU) by the Wheat Varietal Evaluation Committee. Based on the results for VCU and DUS, the successful varieties are submitted to the respective Provincial Seed Councils (PSC)29 of each province for approval. The approved varieties are registered with FSC&RD for national listing and commercial cultivation. According to the evaluation Scope of Work (SoW), WPEP’s objective related to development of new varieties was to “identify and develop new high-yielding, climate resilient and adapted varieties of wheat, having genetic resistance to Ug99 and other major diseases”. To achieve this objective, WPEP provided technical and financial support to the local partner research organizations, including providing support in acquisition of wheat germplasm. According to WPEP reports, over 2,000 international germplasms received from CIMMYT were shared with the local partner organizations each year. These germplasms were evaluated for disease and other characteristics by the partner research institutes and over 1,000 lines were identified. For example, between 2010 and 2017, NARC received 75 sets of international wheat nurseries, consisting of 8,079 germplasms accessions of which 1,015 were identified for further evaluation. Table 3 below provides a summary of international germplasm evaluated at NARC between 2010 and 2017. Table 3 International Germplasm Evaluated at NARC between 2010 and 2017 Year No of sets received Number of lines tested No of lines selected 2010-11 9 925 45 2011-12 11 1151 95 2012-13 11 1350 180 2013-14 13 1382 261 2014-15 10 1377 289 2015-16 15 1027 112 2016-17 6 867 33 28All approved varieties at the time of evaluation were developed through ‘selection’ method. 29 PSC is the final approving authority for approval and release of seed variety in a province. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 28 Year No of sets received Number of lines tested No of lines selected Total 75 8079 1015 Source: Information received from NARC during field visit. After a series of micro, zonal and provincial trials, the most promising lines received from CIMMYT through WPEP’s support were entered into the approval process by the respective research institutes. At the time of evaluation, a total of 21 new wheat varieties, developed from these promising lines by six partner research institutes, were approved, and released by the respective PSCs. According to information received from the partner organizations during field visits, three varieties were pending approval after completion of two years of NUWYT, 18 varieties were being evaluated in the second year of NUWYT and 14 candidate varieties were being evaluated in the first year of NUWYT. The list of new wheat varieties at various stages of approval is summarized in Table 4 and details are provided in Annex X. The list of all approved varieties registered with FSC&RD is provided in Annex XI. Table 4 List of Wheat Varieties Developed with WPEP’s Support Institute NUWYT Y-1 (2022) NUWYT Y-2 (2021) DUS evaluation/ Pending Approval of PSC Approved Varieties Year of Approval WRI-Faisalabad V-19532 V-19347 - Anaj-2017 2017 V-20330 HYT-100-47 - Akbar-2019 2019 V-20337 HYT-100-74 - Subhani-2021 2021 - HYT-100-76 - MH-2021 2021 - - - Arooj-2022 2022 NARC￾Islamabad NR-546 NR-549 - Markaz-19 2019 NR-552 - - NARC Super 2021 NIA-Tandojam - NBW-232 - - - - SD-1060 - - - SD-1040 - - - - DF-1708 - - - - EST-28/11 - - - - EST-29/9 - - - RARI￾Bahawalpur HYT-100-95 180059 BF-1807 Ghazi-19 2019 BF-20105 - - Nawab-21 2021 - - - Sadiq-21 2021 CCRI- Pirsabak PR-146 PR-142 - Khaista-17 2017 PR-147 - - Wadaan-17 2017 PR-148 - - Pirsabak-2019 2019 PR-149 - - Gulzari-2019 2019 PR-150 - - Abaseen-2021 2021 - - - Zarghoon- 2021 2021 - Pirsabak-2021 2021 BARI-Chakwal 19C166 - 18C117 Barani 17 2017 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 29 Institute NUWYT Y-1 (2022) NUWYT Y-2 (2021) DUS evaluation/ Pending Approval of PSC Approved Varieties Year of Approval 20C207 - - MA-2021 2021 BARDC- Quetta - BARD-55 - - - WRI- Sakrand - E-107 SR- 6026 Sindu-16 2021 - HT-50 - IV-II 2021 - HT-25 - - - - HP-01 - - - Source: Information provided by partner research institutes during field visit. In addition to varieties mentioned here, the research institutes continued the development of varieties on which work had been initiated before the WPEP. These varieties included Ujala-16 (WRI￾FSD), Dilkash-2020 (WRI-FSD), Ihsan-16 (BARI), Pukhtoonkhwa-15 (CCRI), Borlaug-16 (NARC), NIA Zerkhaiz (NIA-TJ), NIA Shaheen (NIA-TJ), Gold-16 (RARI) and Jauhar-16 (RARI). According to FSC&RD data, 71 new varieties have been approved and released in Pakistan since 2011 and 42 have been approved and released since 2017. Data from partner organizations and FSC&RD revealed that WPEP contributed to development and approval of 21 new disease resistant varieties, accounting for about a third of the total wheat varieties approved in Pakistan since 2011 and half since 2017. FINDING 2 – SUSTAINABILITY FINANCIAL SUSTAINABILITY Key informants from partner organizations (8) reported that variety development interventions initiated with WPEP’s support had continued after the conclusion of the project. In addition to their own sources, the organizations received financial support from other sources including Pakistan Agriculture Research Board, National Agriculture Emergency Program, Public Sector Development Program (PSDP), Agriculture Innovation Program (AIP) managed by CIMMYT, provincial governments and CIMMYT. However, according to key informants (6 out of 8) some of the activities, such as surveillance and shuttle breeding were likely to suffer without additional funding due to financial constraints. USE OF EQUIPMENT According to key informants from partner organizations (8) equipment and machinery provided through WPEP support was operational, and research institutes were benefiting from them. For example, WPEP supported NARC in replacing a 30-year-old plot planter and plot combine harvester. This resulted in precision in harvesting and planting, because of which experimental errors and seed mixing were minimized, and quality of yield data improved. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 30 Image 1 Tissue Culture Storage Facility and Glasshouse Source: Picture taken during field visit Similarly, WPEP supported CDRI Murree by providing an independent power supply line, generator, and solar energy system for uninterrupted power supply, independent glasshouse cells with controlled environment, fresh air injection system and a culture storage facility (-80°C deep freezer). All the equipment was operational at the time of the evaluation team’s field visit to CDRI. COMMERCIALIZATION OF SEED According to the Seed Act, it is the mandate of research institutes to produce Breeder Nuclear Seed (BNS), which the research institutes use to produce pre-basic seed. The pre-basic seed is then shared with seed producing companies for commercial production of basic and certified seed. Primary data revealed that the quantity of pre-basic seed received by seed producing companies from the research institutes was insufficient to meet their demand to produce basic and certified seed for commercial use. The main reason cited for the low production of pre-basic seed was scarce land available to the research organization to produce adequate quantities of pre-basic seed to meet the seed producing companies’ demand. TRAVELING WHEAT SEMINARS According to the project’s reports, WPEP revived the annual Travelling Wheat Seminar (TWS) which aimed to identify wheat production issues and priority research areas; monitor NUWYT; and jointly assess the status of wheat crop. Each year the TWS team traveled through major wheat growing parts of the country, visited the federal and provincial wheat research institutes and discussed the status of research activities at these institutes. Key informants from all eight partner research institutes reported that they had participated in the TWS and expressed willingness to participate in future events. According to the National Wheat Coordinator, wheat planning meetings and traveling seminars are integral to the National Wheat Coordination Office and were conducted irrespective of availability of project funds. The two events were financed through funding from PSDP for the current year. The most recent Travelling Wheat Seminar was conducted in March 2022, and the Wheat Planning Meeting was conducted in September 2022. RUST SURVEILLANCE According to WPEP Annual Report 2020-2021, wheat rust surveillance was an important part of the WPEP, where partner organizations were provided funds and technical expertise to conduct surveillance for wheat rust throughout the country. This included visits to wheat growing regions in Pakistan to collect samples and providing these samples to CDRI for analysis. According to key informants (2) from CDRI Murree, rust surveillance continued after WPEP's closure, and the institute received wheat samples for rust analysis from all over the country. FINDING 3 - LESSONS LEARNED FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 31 COMMERCIAL SEED PRODUCTION The aim of WPEP was to enhance wheat productivity by mitigating among others, the effects of rusts (especially leaf rust disease) for smallholder farmers. However, the varieties developed through WPEP’s support were not produced in adequate quantities by the seed producing companies. As mentioned under the section on sustainability, the research institutes did not have the capacity to provide enough pre-basic seed to the companies and therefore, the seed producing companies were unable to produce certified and/or improved seed for the farmers. FINDING 4 - GENDER INTEGRATION Both primary and secondary data indicate that women staff from partner research organizations participated in local and foreign training programs supported by WPEP. For example, women from partner research organizations participated in Borlaug Fellowship program and received training on wheat stem rust at University of Minnesota and basic wheat improvement course at CIMMYT, Mexico. One key informant said that “we need to motivate women to enter farming; luckily, women are joining the scientific research domain and it’s a very welcoming sign”. Two key informants from the research organizations mentioned that they had hired female researchers and students to work on wheat breeding. Key informants (2 out of 8) from partner organizations believed that cultural barriers hindered women participation in projects like WPEP. A key informant from a partner organization in Balochistan said that “women participation in an agriculture project was not possible in Quetta due to cultural barriers”. CONCLUSIONS Based on these findings, the following are the key conclusions for WPEP. ● Development of disease resistant wheat varieties was successful to the extent that 21 new wheat varieties developed with WPEP’s support were approved by the respective provincial seed councils, whereas 35 promising new varieties were at various stages of approval process. These newly approved wheat varieties accounted for about a third of the total wheat varieties approved in Pakistan since 2011, and half since 2017. ● Evidence suggests that the interventions initiated through WPEP’s support, including variety development of seed and wheat traveling seminars, continued after the project ended. However, some of the activities suffered from reduction in scale due to financial constraints, and the likelihood of their continuation would depend on the availability of financial resources to the research organizations. ● As far as commercialization of the new varieties is concerned, the partner research organizations were constrained by their capacity for facilitating commercial production of the new varieties and ensuring its availability to farmers. ● As far as gender is concerned, both men and women from partner research organizations benefited from the WPEP sponsored training initiatives. As a project focusing on research activities, WPEP engaged with women from partner research organizations to improve their skills in wheat research. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 32 COTTON PRODUCTIVITY ENHANCEMENT PROJECT (CPEP) FINDING 1 - INTENDED AND UNINTENDED OUTCOMES In this section, the evaluation assessed CPEP’s success in terms of development of new disease resistant cotton varieties approved by respective approving agencies in Pakistan. Documents received from ICARDA indicate that 5,081 sets of germplasm accessions were distributed among the local partner research organizations i.e., Central Cotton Research Institute (CCRI), Multan; Cotton Research Institute (CRI), Multan, and its research stations in Faisalabad and Vehari; and National Institute of Biotechnology & Genetic Engineering (NIBGE) between 2011 and 2018 through the support of CPEP. These germplasms were evaluated by the local research organizations for disease resistance each year when the germplasm was received, and 63 germplasm accessions were reportedly found resistant to Cotton Leaf Curl Virus (CLCuV) over the life of CPEP. Table 5 provides a summary of germplasm accessions acquired and screened through CPEP’s support between 2011 and 2018. Table 5 Germplasm Acquisition and Screening Year No of Accessions30 G. hirsutum31 Other (Species) No. of Accessions Identified32 2011 370 4 366 - 2012 1486 803 683 - 2013 1240 1141 99 - 2014 1250 1160 90 - 2015 355 355 - - 2016 200 200 - - 2018 180 180 - - Total 5081 3843 1238 63 Source: PowerPoint Presentation shared during the introductory meeting with ICARDA. Both primary and secondary data indicate that the accessions identified as resistant during the screening process were mostly infertile and did not produce flowers. The varieties that were flowering produced large plants, small bolls, and low yields. These varieties were maintained by the research organizations as ratoon33 crops. Reportedly, these plants were regularly screened for CLCuV and were found to be disease free. As the germplasm received under this program did not generate the required results through ‘selection,’ the local research partners adopted crossing (hybridization) of highly resistant accessions, acquired through CPEP support, with high yielding local cultivars. A key informant from CCRI explained the process thusly. In crossing, we pick the variety that is the most resistant to the virus, but has low or zero yield, and cross it with some of our (local) varieties, which are not virus resistant but high yielding. Thus, we get a hybrid variety that is potentially disease resistant and high yielding. Then, we 30A distinct, uniquely identifiable sample of seeds representing a cultivar, breeding line or a population, which is maintained in storage for conservation and use. This definition can be found at the URL https://www.fao.org/wiews/glossary/en/ 31 G.hirsutum are species of cotton that were imported from the US through CPEP support and were screen for disease resistant by the local partner research organizations. 32 Information on the number of accessions identified as resistant for each year was not available. 33 Ratooning is the process of maintaining plants by pruning most of the above-ground portion of the plant but leaving the roots and the growing shoots intact to allow the plants to recover and produce a fresh crop in the next season. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 33 keep a check on it from F234 to F6 generation and select the varieties from these annually. These varieties are tested in fields where they are exposed to the natural environment. We have conducted more than a thousand crossings and have so far proceeded with six of the varieties. Two key informants from NIBGE revealed that after screening of almost 5,000 germplasm accessions, Mac-7 was identified as a source of resistance to CLCuV disease. Mac-7 was not only resistant to the virus but was also highly effective against ‘whitefly’ that transmits the virus. A key informant from NIBGE said that “usually most breeders send the resistant germplasm for breeding after its identification; unlike them, we did not proceed with the breeding, but we studied its genomics and identified the genes responsible for resistance”. Reportedly, Mac-7 was used as a source of resistant parent and was crossed with susceptible but high yielding local varieties. For example, NIBGE 11 and NIBGE 15 were developed as a result of hybridization of Mac-7 with other local varieties. Table 6 below presents the cotton varieties at various stages of the approval process. Table 6 Cotton Varieties at Various Stages of Approval Institute NCVT Y-1 (2022) NCVT Y-2 (2021) Pending Approval of PSC Approved Varieties Year of Approval NIBGE IR NIBGE-19 IR-NIBGE-17 IR-NIBGE-13 IR-NIBGE-11 2019 IR-NIBGE-20 - IR-NIBGE-16 IR-NIBGE-15 2021 NIBGE-PF-1 - - - - CCRI CIM-990 - Bt.CIM-775 - - CRI - - - MNH-1050 2021 Source: KIIs with local partner research organizations. According to key informants from partner research organizations, three varieties developed through support from CPEP were approved by Punjab Provincial Seed Council (PSC)35 . The same was confirmed by lists obtained from the Federal Seed Certification and Registration Department (FSC&RD)36 and Pakistan Cotton Central Committee (PCCC)37. Furthermore, three varieties were approved by the technical sub-committee and were pending approval from PSC. In addition to these, four varieties were undergoing varietal testing for the first year under National Coordinated Variety Trial (NCVT)38 and one variety was being tested for the second year. UNINTENDED BENEFITS Key informants from CRI reported that CPEP’s capacity building support helped in development of new cotton varieties not directly supported by CPEP, namely MNH-1020 developed by CRI Multan and FH￾490 developed by Cotton Research Station, Faisalabad. In another instance, respondents from NIBGE revealed that through CPEP’s capacity building support, the trained staff improved the capacity of other NIBGE staff by providing the same training to them. 34 Filial second generation or F2 is the progeny of the variety developed as a result of hybridization of USDA accessions and local varieties. F3 to F6 are the progeny of the varieties from respective preceding filial generation. 35 PSC is the final approving authority for approval and release of seed variety in a province. 36 FSC&RD is responsible for testing the candidate seed varieties for Distinctness, Uniformity and Stability (DUS) and provides its recommendation to PSC. 37 PCCC is responsible for testing the candidate seed varieties for Value for Cultivation and Use (VCU) and provides its recommendation to PSC. 38 NCVT are conducted by PCCC to test promising breeding material/strains developed by breeders in Pakistan for adaptability under various agroclimatic conditions in the country, prior to their approval for general cultivation. These trials are conducted annually, and every new variety is tested twice in two consecutive years. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 34 Additionally, doctoral students used the equipment and germplasm, acquired through CPEP’s support, for cotton research. FINDING 2 - SUSTAINABILITY FINANCIAL SUSTAINABILITY Key informants from NIBGE reported that interventions initiated under CPEP continued after the project concluded and work on new variety development was in progress. NIBGE continued receiving funding from other sources after the project ended, including funding from Project Development Fund, Technology Development Fund and Higher Education Commission (HEC) of Pakistan. It was reported that NIBGE had sophisticated infrastructure and trained work force because of which they were aiming to partner with private sector organizations to achieve long term sustainability. On the other hand, respondents at CRI and CCRI revealed that varietal development was the mandate of the research institutes and had continued after CPEP ended. However, the scale of activities had reduced after the project closure. The institute’s funding decreased after CPEP’s closure, which resulted in a decrease in plantation of nurseries, crossing, surveillance visits and trials. Moreover, maintenance of equipment, including lab equipment and incubators for seed germination, was also negatively affected due to lack of financial resources. EQUIPMENT Key informants from NIBGE reported that CPEP had supported the institute in building a state-of-the￾art tissue culture lab and cotton transmission lab. New glass houses and net houses were also constructed with support from CPEP. It was reported that three of NIBGE’s staff members who received training for Bioinformatics from Mississippi State University established a Bioinformatics lab at the institute with support of CPEP. Further, NIBGE did not have the infrastructure for sequencing and next generation sequencing before support from CPEP. Similarly, a key informant from CRI reported that CPEP supported CRI in upgrading the containment facilities and greenhouses and provided lab equipment. In addition to these, Vehari Research Station received solar panels and a light system to control temperature. At CCRI, the greenhouse, seed-testing lab and gene-back were upgraded. Respondents from the partner organizations revealed that the lab equipment and other material received through CPEP’s support were used for cotton research and variety development after CPEP’s closure. However, the Fluorescence in Situ Hybridization (FISH) microscope received through CPEP’s support was not being used by CCRI staff due to a lack of capacity on their part. GERMPLASM STORAGE According to a key informant from CRI, CPEP had supported the institute in establishment of gene￾bank for short (i.e. 25 years), medium (i.e. 50 years) and long term (i.e. 100 years) storage of the cotton germplasm that was acquired through CPEP. However, when the evaluation team visited the said facility, it was observed that the chiller in one of the storage rooms for long term storage was not operational. A key informant from NIBGE said the following. Pakistan received a huge number of germplasms from USDA (CPEP), but my biggest fear is that we will not be able to conserve it in the long run. Therefore, efforts should be made at the national level to ensure that the germplasms are stored under optimum conditions for future use. The germplasms must be multiplied and stored at multiple locations. Additionally, a key informant from NIBGE reported that Mac-7 variety was a “valuable source against CLCuV not only in Pakistan but anywhere in the world” and therefore “practical steps should be taken to conserve its purity”. It was reported that the purest form of Mac-7 was available at NIBGE, and its purity was being maintained by retaining the original source and multiplying the seed every year. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 35 CAPACITY OF HUMAN RESOURCE Primary data revealed that research organizations’ staff, after receiving capacity-building support through CPEP, continued to contribute to the respective partner research institutes after the project’s closure. For example, after receiving training in Bioinformatics the staff members became master trainers and trained the institute's other staff members and students in the respective field. In another instance, the trained staff implemented the learnings at the institute. However, in one institute the number of staff working in labs (and field) had reduced to less than half after CPEP concluded. FARMER FIELD SCHOOLS (FFS) AND WOMEN OPEN SCHOOLS (WOS) Key informant interviews with representatives from Women Agricultural Development Organization (WADO) and Farmers’ Integrated Development Association (FIDA) revealed that since the closure of CPEP, both organizations have not been able to provide support to farmers through FFS and WOS approach due to financial constraints. However, since the use of the knowledge and technology imparted to farmers through CPEP’s support benefited the farmers, they continued using it. FINDING 3 – LESSONS LEARNED COMMERCIALIZATION The aim of CPEP was to mitigate the effects of the chronic and lethal Cotton Leaf Curl Virus (CLCuV) disease, especially for smallholder farmers. Data from FSC&RD indicate that none of the varieties developed through CPEP’s support were commercially produced by the seed producing companies. As mentioned earlier, it is the mandate of research institutes to produce Breeder Nucleolus Seed (BNS), which is used to produce pre-basic seed. The pre-basic seed is shared with seed producing companies to commercially produce the next generation seed i.e., basic, certified, and approved seed. Primary data revealed that the research institutes had limited capacity to produce BNS and pre-basic seed due to scarce land and workforce. Interviews with representatives of two seed companies revealed that pre-basic seed received from the institutes was not enough for commercial production. The quantity of seed provided by the institute was enough for planting on only one acre of land whereas at least 20 to 25 acres of seed plantation is required for commercial production of seed. CLIMATE CHANGE Key informants from three partner research institutes and Pakistan Central Research Committee reported that currently “flood, drought, high temperature and unsuitable climate are the major issues” effecting cotton production in Pakistan. A key informant from a partner research institute said that “due to climate change we need to prepare varieties that are heat resistant; for the first time in the history of Pakistan we have received (cotton) germplasm on a huge scale” which can be used for research on traits such as heat tolerance. A key informant from ICARDA said that “climate change is having a negative effect on cotton production and quality. The main factors in climate change that need to be addressed are heat, flooding and drought tolerance through breeding and management”. Rashid et. al (2020)39 found that “based on the empirical results, it is recommended that those cotton varieties which are more temperature variations tolerant should be introduced. For this purpose, the government should focus on research and development”. FINDINGS 4 - GENDER INTEGRATION According to the key informants from partner research organizations, female staff of the respective organizations participated in the CPEP supported capacity building activities, including participation in 39 http://researcherslinks.com/current-issues/Impact-of-Climate-Change-on-Cotton-Production-in-Pakistan-An-ARDL￾Bound-Testing￾Approach/14/1/2771/html#:~:text=ARDL%20bounds%20test%20confirmed%20variables,negative%20effect%20on%20cott on%20production. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 36 Borlaug Fellowship and other national and international trainings and conferences. Two key informants mentioned that female staff participated in training on the use of specialized microscopes. Additionally, CPEP also supported men and women through activities conducted by FIDA and WADO. According to CPEP reports, women play an important role in cotton sowing, hoeing and picking. However, women often face challenges and have limited access to opportunities, such as under￾representation in learning groups. CPEP used Woman Open Schools (WOS) and Farmer Field Schools (FFS) approach to train women on improving cotton crop yield and raise awareness on better management of CLCuV, use of fertilizers and water management. According to information shared by ICARDA40 , 608 women41 (compared to 25,535 men) received capacity building support through the FFS and WOS initiatives, and 528 women participated in 23 farmer field days (as compared to 2,126 men). Although CPEP started in 2011, women participation in WOS and FFS occurred between 2018 and 2020. During this period, 2,298 participants were trained, which included 608 women. CONCLUSIONS Based on the findings above, following are the key conclusions for CPEP. ● CPEP was successful in the development of disease resistant cotton varieties, to the extent of approval and release of three new cotton varieties by the PSC. Although other varieties developed with support from CPEP were at various stages of approval, their fate remained uncertain until approved by the final approving authority, the PSC. ● The interventions initiated through CPEP’s support, including variety development of seed, continued after the project ended. However, in case of two partner organizations i.e., CRI and CCRI, some of the activities suffered from reduction in scale due to financial constraints, and the likelihood of their continuation will depend on the availability of financial resources to the research organizations. ● In terms of sustainability of benefits, the partner organization had been benefiting from the continued use of equipment and materials provided through CPEP. However, the long-term storage of the germplasm provided through CPEP’s support has been compromised to the point of potential wastage of germplasm. ● As far as commercialization of the new varieties is concerned, the partner research organizations were constrained by their capacity for facilitating commercial production of the new varieties, and finally, its availability to farmers. ● As far as gender is concerned, both men and women from partner research organizations benefited from the project’s capacity building interventions, as demonstrated by their participation in various training activities. As far as the agronomy related activities implemented by WADO and FIDA are concerned, CPEP used FFS and WOS approaches to include women in project activities. However, women were only included in the last four years of the project’s ten years implementation period. 40 ICARDA’s introductory presentation to the evaluation team. 41 Although CPEP started in 2011, women participation in WOS and FFS occurred between 2018 and 2020. During this period, 2,298 participants were trained, including 608 women. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 37 AGRICULTURE SERVICES PROVIDERS TRAINING PROGRAM FINDING 1 – INTENDED AND UNINTENDED OUTCOMES According to Feder et al. (1985), “final adoption at the individual farmer's level is defined as the degree of use of a new technology in long-run equilibrium when the farmer has full information about the new technology and its potential”42 . In a literature review, “Scaling Agriculture Technologies and Innovation Diffusion'' conducted by Management Systems International for USAID, “diffusion” was defined as “a process involving an individual that includes the series of stages one undergoes from first hearing about a product to finally accepting or using it”43 . During introductory meeting with the implementing partner, the team was informed that from ICARDA’s point of view, the term “adoption” meant that the farmer had used the technology at least once without considering any long term or continuous use of the technology. However, for the sake of this evaluation, adoption has been referred to as continuous or long-term use of the technologies by the farmers. TECHNOLOGIES ADOPTED BY FARMERS Laser Land Levelling and Ridge Planting of Crops According to ASPTP’s annual and quarterly reports, Laser land leveling technology and ridge planting were both introduced in Sindh, while in KP only Laser land leveling was introduced44 to farmers. Laser land leveler does precision levelling of the land, which saves water and allows its even distribution, leading to higher crop yields. Ridge planters help to plant crops on top of ridges, which saves about 30 percent water compared with traditional flatbed planting. Image 2 Laser Land Leveler in Operation Source: ASPTP Final Report One partner organization, ASPs (4 out of 4) and farmer groups (4 out of 4) interviewed reported that these technologies helped farmers save water, increase crop yield and incomes due to precision land leveling. The same finding was confirmed by a key informant from ICARDA. The final report of ASPTP notes the following. 42 Feder, G., Just, R., Zilberman, D. 1985. Adoption of Agricultural Innovation in Developing Countries – A Survey. World Bank Staff Working Papers. https://documents1.worldbank.org/curated/en/156141468767357523/pdf/multi0page.pdf. P 3 43 Management Systems International. May 7, 2015. Scaling Agricultural Technologies and Innovation Diffusion – Literature Review. https://pdf.usaid.gov/pdf_docs/pa00kfqg.pdf. P. 3 44 ICARDA and the collaborating institutes selected 70 ASPs through interviews of potential candidates following certain selection criteria. These ASPs were trained on the benefits & use of selected technologies. After training, the ASPs started service provision of selected technologies to the farmers in their areas. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 38 In Pakistan, over 53 percent cultivated land is irrigated with surface water. Traditionally, the farmers practice flood irrigation that wastes a lot of water, especially when land is not properly leveled. Laser land leveler does precision leveling of land that not only saves water but increases crop yield as well because of even distribution of irrigation water. It is pertinent to mention that ASPs (3 out of 4) and farmer groups (4 out of 4) both reported that the farmer’s input cost increased due to cost of diesel and high hourly rates of service provision. However, it was also reported that since the increased input cost due to laser leveling was a one-time activity that lasted 5-7 years, the higher costs were offset by higher crop yields over 5 to 7 years. In KP, ASPs (2) and farmer groups (2) both revealed that maintenance shops and spare parts for laser equipment were not available locally. Because of this reason, they either had to travel to far-flung areas for maintenance and repair of equipment or quit the technology. In Sindh on the other hand, ASPs did not face these challenges as the machinery and spare parts were locally available. Primary data revealed that laser land levelling and ridge planting were both successfully adopted in Sindh, while in KP, the limited availability of laser land leveler and its spare parts impeded its successful adoption. Gypsum According to project reports, gypsum was introduced to ASPs and farmers in rain-fed areas of Punjab. Based on information available from secondary data, the use of gypsum in rain-fed areas improved soil structure and allowed absorption of rainwater. Image 3 Unloading of Gypsum at ASP Shop (left); Soil Application of Gypsum (right) Source: ASPTP Final Report ASPs (2) and farmer groups (2) both shared positive feedback about the use of gypsum for improved soil and product quality, as well as lower input cost and higher yield. Moreover, a key informant from a partner organization said the following. Gypsum was provided at very nominal prices, and once applied to the soil it works for three years. The main function of gypsum is to retain moisture in the soil for longer periods. Usually, Potohar region gets rainfall in monsoon season, and we have no proper irrigation system here, so gypsum is applied before the monsoon season to conserve moisture in the soil. As far as adoption is concerned, we had a very good response for gypsum, because it can be easily applied to the soil and gives very good results. Another farmer in Punjab using gypsum shared the following. Farmers from nearby villages have also been noticing my yield and showing interest in using gypsum. Four or five farmers have personally met me to ask about this technology. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 39 Rapid Soil Testing Kit Soil testing kit was introduced to farmers and ASPs in Punjab for providing qualitative analysis of soil samples. ASPs analyze soil samples from farmers’ land and recommend fertilizers based on that analysis. ASPs (2) and farmers groups (2) reported that the soil testing kit benefited the farmers in terms of lower input cost, improved soil quality and a 20% to 40% increase in yield. However, in contrast to the views of farmers and ASPs, a key respondent from the partner organization, as well as the project’s final report, mentioned that the kit used for soil testing had become unaffordable after the increase in US Dollar exchange rate, despite its initial success. Zero-Till Drill Zero-till drill technology was introduced to ASPs and farmers in Balochistan. According to the project’s final report, the use of zero-till drill technology enabled farmers to save water as well as time and cost of land preparation, by allowing for early plantation of wheat in Balochistan. Similarly, according to ASPs (2) and farmers groups (2) interviewed, and a key informant from a partner organization, zero￾till drill increased farmers’ incomes and crop yield along with helping farmers reduce input cost by saving water and time for land preparation. The drill has also been reported to improve soil fertility. According to a key informant from the partner organization, the farmers noticed low input cost, decreased cost of cultivation, increased yield, increased income, and timely sowing of the wheat crop through the new technology, which led to a quick adoption of the technology by the farmers. Image 4 Zero-Till Drill in Operation in District Jaffarabad Source: ASPTP Final Report High Efficiency Irrigation System (HEIS) Drip/bubbler/sprinkler irrigation system (also called High Efficiency Irrigation System) was introduced to ASPs and farmers in Balochistan and rain-fed areas of Punjab. According to the project’s final report, HEIS was installed to supply water in a controlled fashion at the place it is needed the most. By doing so, it enables the farmers to save up to 80% of water, along with time for land preparation for cultivation of new crops. According to a partner organization, ASPs (2) and farmers groups (2) interviewed for this technology, the adoption of HEIS technology increased farmers’ incomes and crop yield, along with helping farmers reduce input cost by saving water and time for land preparation. Furthermore, according to a key respondent from the partner organization, In the upland areas of Balochistan, there is cultivation of fruits such as apples, cherries, olives, and grapes. In these areas, water is an expensive commodity, but more than 80 percent of FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 40 irrigation is done through ground water, which leads to water wastage. Additionally, load shedding of electricity adds to farmers’ suffering. Drip irrigation saves 80% to 90% of that water, thereby resolving one of the biggest problems faced by the farmers in Balochistan. Image 5 Drip Irrigation in Balochistan Source: Picture taken during field visit. Key informants from ICARDA and partner organization in rain-fed Punjab considered the high startup cost of PKR 120,000 to 140,000 per acre, and the absence of government subsidy for installation of drip irrigation system, as constraining factors for adoption of this technology by ASPs and farmers. The key informant from partner organization shared the following. Farmers are always ready to adopt new technologies, provided they get some sort of subsidy from the government. They are looking for incentives. Farmers cannot easily adopt the High Efficiency Irrigation System because it is expensive and out of budget for small farmers. Two out of four respondents (1 ASP and 1 farmer group) interviewed for HEIS in rain-fed Punjab mentioned high startup cost as a significant deterrent to its adoption by small farmers. Primary data revealed that the presence of government subsidy allowed successful adoption of HEIS in Balochistan, whereas farmers in Punjab were discouraged by the high startup cost of drip irrigation and lack of government support, thus constraining its adoption. Biozote Biozote was introduced to ASPs and farmers in Punjab. According to the project’s final report, biozote is a biofertilizer applied as seed coating just before sowing. It enabled the farmers to increase crop yield by 10 to 30 percent. According to ASPs (2) and farmer groups (2), biozote helped reduce input cost, increase both yield and incomes, and improve crop quality and soil fertility. According to a key informant from a partner organization, biozote was initially sold in packets by the National Agriculture Research Centre (NARC). During project implementation, the supply was commercialized by franchising to a private company, which charged a higher price. This made biozote expensive for farmers, hence hindering adoption despite its initial success. This was confirmed by three of the four respondents (1 ASP and 2 farmer groups) interviewed. They claimed that the private company created artificial shortage in the market to sell the product at a higher price. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 41 Bed Planting Bed planting was introduced to farmers in Punjab. Planting crops on raised beds saves water compared with traditional flatbed planting. According to the project’s reports, the results of previous phases of watershed project showed 30% saving of water in bed planting, compared with traditional flatbed planting. The technology was particularly useful for low-lying heavy soils of rice and wheat growing areas in Punjab, where occasional heavy winter rains create flood-like conditions and badly affects wheat crop. Similarly, a key informant from a partner organization added that bed planting was very useful for climate change; due to untimely monsoon or winter rains, crops get destroyed when they are grown traditionally due to excessive water, and therefore, there has been a rapid transition to growing these using bed planting. Image 6 Wheat planted on Ridges in District Nankana Sahab Source: ASPTP Final Report The contact information shared with the evaluation team by ICARDA included details of only one ASP and the same was interviewed by the evaluation field team. According to the ASP and farmer group interviewed, using bed-sowing they never went back to the traditional method of sowing after they reaped the benefits of lower input cost, higher income and yield, better product quality and water conservation. One of the farmers mentioned that input cost had reduced up to 35% due to reduction in use of weedicides and herbicides, because growth of weeds is slower on sowing beds. Happy Seeder Happy seeder technology was introduced to farmers in irrigated Punjab. According to the project’s final report, happy seeder enables farmers in rice-wheat system to plant wheat 7 to 10 days earlier than traditional practice, without removing or burning rice residue. The ASP who was trained in happy seeder was so successful that he purchased a second happy seeder and placed an order for a third one. Similarly, a key informant from a partner organization shared the following. ASPs’ clients were dealing with a persistent issue. Almost immediately after they harvested rice fields, they had to sow wheat within a short period of time. They coped by burning rice which has led to smog, resulting in serious health issues. When asked to protect the environment, the farmers asked for an alternative solution. The solution to this dilemma was provided by the Second-Generation Happy Seeder. While zero-till only treats anchored residual, second generation treats both loose and anchored by chopping and drilling loose FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 42 residual. This sold among farmers like hot cakes because it reduces the cost of land preparation. Image 7 Rice Residue burning by Farmers (left); Happy Seeder in operation (right) Source: ASPTP Final Report The ASP interviewed by the evaluation team observed that the new technology was a great introduction and had the potential to revolutionize agriculture in his region (Sheikhupura District). He added that unavailability of happy seeder in his district, along with lack of skills and awareness of its benefits among farmers, had discouraged him from providing services. Primary data showed that despite its potential to mitigate climate change and benefits to farmers, Happy Seeder was not successfully adopted because of its limited availability in Sheikhupura District. Fertilizer Prediction Model (FPM) FPM was introduced to farmers in irrigated areas of Punjab. According to the project report, the model calculates accurate and balanced quantity of Nitrogen and Phosphorus fertilizers for target yields of major crops. The ASPs collect soil samples from farmers’ lands and take them to district soil testing laboratories for analysis of soil and organic matter. Putting the test result values in the model, the ASPs calculate the amounts of fertilizers needed to get target yield of the desired crop and provides recommendation on use of fertilizer to the farmers. Both ASPs (2) and farmer groups (2) interviewed on the subject of FPM mentioned delaysin lab results because the labs were located at least 30 km away from the farmers’ villages. This delay made the test reportedly useless for the farmers. One of the ASPs interviewed added the following. We only charge PKR 200 per sample, but the cost of getting the report is more than PKR 200. Sometimes we wait to collect multiple samples from the farmers because the labs are quite distant (and going there for one sample is not feasible). This causes delays, so now the farmers do not prefer this technology. This technology can be successful if the reports are more readily available, otherwise the future of FPM looks bleak in our district (Sialkot District). Furthermore, farmers reportedly could not trust the results of lab reports because the same soil sample tested at two different labs had different results. Therefore, the farmers preferred making their own calculation of fertilizers to maximize crop yield. One of the farmers put it as follows. In my opinion, the results [of FPM] that were discussed and assumed in the training were practically not possible. Unfortunately, the accuracy of the lab report has been in doubt and needs to be reconsidered. Banana Chopper Banana chopper was introduced in Sindh where 90 percent of banana crop is grown. ICARDA imported one banana chopper from India to help farmers cut the banana residue and use it as mulch on soil or as organic manure after making its compost, instead of their traditional practice of burning the residue. However, as shared by a key informant from ICARDA, the imported machine had a mechanical problem that could not be fixed despite multiple attempts. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 43 Image 8 Banana Residue Chopper in Operation Source: ASPTP Final Report FACTORS ENABLING ADOPTION OF TECHNOLOGIES Overall, across 5 partner organizations, 18 ASPs and 51 farmers, factors that enabled the adoption of technologies by farmers include the following. ● Increase in income of farmers using the technologies (according to 48 farmers and 16 ASPs in all provinces, for all technologies except banana chopper and happy seeder). ● Increase in yield per acre (according to 39 farmers, 12 ASPs and 1 partner organization, in all provinces for laser land leveling, ridge planting, zero-till drill, soil testing kit, gypsum, biozote, bed planting and FPM). ● Reduction in input cost (according to 39 farmers, 7 ASPs and 3 partner organizations in Sindh, Punjab and Balochistan for laser land leveling, ridge planting, HEIS, zero-till drill, bed planting, soil testing kit, biozote, gypsum and FPM). ● Effective demonstration of the technology in field by ASPs (according to 24 farmers, 11 ASPs and 3 partner organizations in four provinces, for laser land leveling, ridge planting, HEIS, zero￾till drill, bed planting, FPM, soil testing kit and gypsum). ● Water conservation (according to 21 farmers, 8 ASPs and 2 partner organizations in four provinces, for laser land levelling, ridge planting, HEIS, zero-till drill, bed planting and gypsum). ● Improvement in soil quality and fertility (according to 24 farmers, 7 ASPs and 1 partner organization, all in Punjab for FPM, soil testing kit, gypsum and biozote). ● Improvement in product quality (according to 18 farmers and 6 ASPs all in Punjab, for bed planting, soil testing kit, FPM, gypsum and biozote). ● Reduction in time required for land preparation (according to 15 farmers, 5 ASPs and 2 partner organizations in KP, Balochistan and Punjab, for laser land leveling, HEIS, zero-till drill and FPM). These are summarized in the bar chart (Figure 1) below. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 44 Figure 1 Factors Enabling Adoption of Technologies Source: Field data collected by the evaluation team FACTORS CONSTRAINING ADOPTION OF TECHNOLOGIES According to project’s quarterly and final reports, factors that constrained the adoption of technologies included unavailability of maintenance workshop and/or spare parts locally (in the case of banana chopper) and high cost of adoption (for HEIS and soil testing kit). Moreover, based on the following excerpt from ICARDA’s final report of ASPTP, farmers’ traditional mindset also hindered their adoption of new technologies. Forty-three percent of ASPs faced issues of traditional beliefs and attitudes of farmers towards the use of new technologies in the farming system. Twenty-eight ASPs faced constraints associated with farmers’ lack of awareness about the new technologies and their potential effect on crop yields. 13% of farmers were unable to use new agricultural technologies on their farms due to lack of financial resources. Across all partner organizations, ASPs and farmers, factors that constrained the adoption of technologies include the following. ● Lack of local availability of machinery, its spare parts and/or maintenance workshops (according to 12 farmers, 5 ASPs and 1 partner organization in Punjab and KP, for soil testing kit, happy seeder, FPM, and laser land levelling). ● Unaffordability of the technology (according to 6 farmers, 1 ASP and 1 partner organization in Punjab, for soil testing kit and biozote). ● High startup cost of adoption (according to 3 farmers, 1 ASP and 1 partner organization in Punjab for HEIS). ● Farmers’ lack of trust in the technology (according to 6 farmers and 2 ASPs in Punjab for FPM). Changes in Incomes of ASPs and Farmers During data collection, ASPs were requested to report on changes in income after they started providing services to farmers on the respective technologies, and farmers were asked to report on changes in income after using the respective technologies introduced to them. In addition, the farmers were asked to report changes in crop yield and input cost after using the technologies introduced through ASPTP. For income and yield, the respondents reported data both in actual numerical (Pak Rupees) and percentage changes, and therefore both sets of data are reported in the next sections. 64 52 49 38 31 32 24 22 0 10 20 30 40 50 60 70 Higher Income Higher Yield Lower Input Cost Demonstration Saving Water Better Soil Quality Better Crop Quality Faster Land Preparation Factors Enabling Adoption of Technologies by the Farmers FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 45 Changes in Income of ASPs During primary data collection, the field research team specifically asked ASPs about changes in income as a result of services introduced by ASPTP. They reported this in Pak Rupees or as a percentage increase in income. Therefore, the primary data for change in income is being reported in the same way as it was received from ASPs. Out of 18 ASPs interviewed, 16 reported an increase in their incomes because of the provision of services to farmers. The remaining 2 no longer provided services for reasons mentioned in the previous section. Most of the ASP (8 out of 12) who reported changes in Pak Rupees, reported more than PKR 200,000 increase in their annual income. Of the three respondents reporting percentage changes in income, 2 reported an increase of more than 40% in their annual income. Table 7 below shows data reported by the ASPs. Table 7 Changes in Income of ASPs Annual Income Increase (Reported in PKR or percent) No. of ASPs* Province Technology ≤ PKR 50,000 2 Balochistan, Punjab Zero-till drill, FPM PKR 100,000 – 200,000 2 Punjab Gypsum, soil testing kit PKR 200,000 – 300,000 5 All 4 provinces Laser, ridge planting, bed sowing, HEIS > PKR 300,000 3 Balochistan, Punjab Zero-till drill, gypsum, biozote 20-30 percent 1 Punjab Soil testing kit 40-50 percent 2 Punjab FPM, Soil testing kit Did not share 1 Sindh Laser, ridge planting Total 16 * Percentages are drawn considering 15 ASPs because one ASP said that his income increased but did not share exact figures. Also, percentages have been drawn only for ASPs’ incomes, for an effective comparison of income data from both primary and secondary sources. Furthermore, a survey was conducted by ICARDA with 48 ASPs in February 2021 at 5 provincial stations. Based on that survey, 9 ASPs (19%) earned less than PKR 50,000 from service provision; 8 (17%) earned between PKR 50,000 and 100,000; 18 (37%) earned between PKR 100,000 and 200,000; 5 (10%) earned between PKR 200,000 and 300,000; and 8 (17%) are earned more than PKR 300,000. Changes in Income of Farmers Similarly, the farmers interviewed were asked to report on changes in income, yield and input costs. As mentioned earlier, for income and yield, the respondents reported either the numerical value or percentage changes, and therefore both sets of data are reported. Out of 51 farmers (17 GIs), 48 farmers reported an increase in their incomes. The remaining 3 farmers did not report any change in income because they were no longer using the technology, for reasons mentioned in the previous section. Most of the farmers (12 out 18), who reported in Pak Rupees, reported an increase of more than PKR 200,000 in their annual income. Of the 21 respondents reporting in percentages, 12 reported an increase of 10% to 30%, whereas 9 reported more than 30% increase in their annual income. Table 8 below shows the change in farmers’ income (Pak Rupees or Percentage) by province and by technologies adopted by them. Table 8 Changes in Income of Farmers Annual Income Increase (Reported in PKR or percent) No. of Farmers* Province Technology PKR 50,000 – 100,000 1 Punjab Gypsum FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 46 PKR 100,000 – 200,000 5 Sindh, Punjab Laser, ridge planting, gypsum PKR 200,000 – 300,000 6 Khyber Pakhtunkhwa Laser levelling > PKR 300,000 6 Punjab, Balochistan HEIS, gypsum 10-20 percent 8 Sindh, Punjab Laser, ridge planting, gypsum 21-30 percent 4 Punjab Bed sowing, soil testing kit 31-40 percent 6 Punjab Biozote, FPM, soil testing kit 41-50 percent 2 Punjab, Balochistan Biozote, zero-till drill >50 percent 1 Balochistan Zero-till drill Did not share 9 Balochistan, Punjab HEIS, zero-till drill, FPM, gypsum, bed planting Total 48 *Data is based on 39 farmers because 9 farmers said that their incomes increased but did not share exact figures. During primary data collection, the field team specifically asked farmers about changes in yield as a result of availing services from ASPs. They reported either in maunds per acre or as a percentage increase in yield. Therefore, the primary data for yield is being reported in the same way as was received from the farmers. Out of 51 farmers, 48 reported an increase in their yield per acre, the remaining 3 farmers did not report any change in yield. Of the 34 respondents reporting changes in yield in maunds per acre, half of the respondents reported an increase of up to 10 maunds per acre while the other half reported an increase of more than 10 maunds per acre. Table 9 shows data reported by farmers reflecting the increase in their yields (reported in maunds per acre and percentage) through technology adoption. Table 9 Changes in Yield Yield Increase (Reported in maunds per acre or percent) No. of Farmers* Province Technology < 5 maunds 3 Punjab FPM, gypsum 5-10 maunds 14 All 4 provinces Laser, ridge planting, drip irrigation, zero-till drill, gypsum 11-20 maunds 15 Sindh, Balochistan, Punjab Laser, ridge planting, drip irrigation, bed sowing, FPM, biozote 21-30 maunds 1 Punjab Bed sowing > 30 maunds 1 Balochistan Zero-till drill ≤ 20 percent 2 Punjab Biozote 21-30 percent 4 Punjab Soil testing kit 31-40 percent 4 Punjab FPM, soil testing kit, biozote > 40 percent 1 Punjab Biozote Did not share 3 Punjab Biozote, gypsum Total 48 * Reported data is based on 45 farmers because 3 farmers said that their yields increased but did not share exact numbers. Out of 51 farmers, 44 reported a change in their input cost. Of these 44 farmers, 10 farmers, majority (9) of whom were using laser technology, shared that their input cost increased by adopting new FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 47 technology. However, it was reported that the cost was a one-time activity and the land leveled by laser lasted for 5 to 7 years. The other 34 farmers reported a reduction in input cost due to saving water and/or fertilizers. Twelve out of 34 farmers attributed this reduction in input cost to saved water through drip irrigation, zero-till drill and bed sowing, while 15 farmers adopting bed sowing, gypsum, biozote and soil testing kit mentioned that they were able to reduce input cost because of a lower requirement for fertilizers due to these technologies. UNINTENDED BENEFITS OF PROGRAM ACTIVITIES A key informant from a partner organization in Punjab stated that the Punjab Government allowed ASPs to compete for the subsidy on Happy Seeder initiated in 2019-20. Moreover, bed planting was adopted by the Punjab Government under the National Agriculture Emergency Program (2019-2023). FINDING 2 – SUSTAINABILITY In answering this evaluation question, the focus was on assessing whether the trained ASPs continued to provide services to farmers using the introduced technologies, and whether the beneficiary farmers have continued to avail services from these ASPs and used the technologies, after ASPTP ended. CONTINUED PROVISION OF SERVICES BY ASPS ON SELECTED TECHNOLOGIES A key informant from ICARDA mentioned that out of the 70 ASPs trained during the project, 60 provided services until 2021. Based on data collected from the field, out of 18 ASPs interviewed, 14 reported continuation of service provision of introduced technologies after ASPTP ended. Two ASPs discontinued providing laser land levelling services, one each in Sindh and KP, while the other 2 discontinued FPM and happy seeder service provision in Punjab. The ASP in Sindh shared that he faced many difficulties to convince the farmers, because the farmers suspected that this technology would cost them more as this was an absolutely new technology and was unavailable in his area (Sakrand, Nawab Shah District). Regarding continuation of services, he said that he was not providing the facility to the farmers because they were not following his instructions. The ASP in KP mentioned that unavailability of machinery and its spare parts in his district made it difficult to provide the technology and its required equipment to farmers. Similarly, the ASP for happy seeder had discontinued services because of unavailability of happy seeder in his district. The ASP trained in FPM services shared that out of 350 farmers who participated in the orientation session, only 4-5 farmers were following instructions of the lab results, either due to delay in delivery of lab reports (as the labs were far away from the villages) or because of the unreliability of results. Details have been elaborated in the previous section related to FPM outcomes. Among 4 ASPs who discontinued service provision, the common factor was lack of availability of technology in their areas. Spare parts and maintenance workshops for equipment and machinery used were also distant from these districts, making mechanical malfunctions too costly for ASPs. Eight respondents (2 partner organizations in Punjab and Balochistan, 2 ASPs in Sindh, and 4 ASPs in Balochistan) discussed the damage caused by floods which destroyed machineries, crops or land, disrupting service provision and making fields uncultivable. Among factors that motivated ASPs to continue service provision, the most cited reason was an increasing demand for the technology, which led to an increase in income for the ASP (according to 16 out of 40 respondents). These include 2 partner organizations in Balochistan and Punjab, 8 ASPs and 6 farmer groups in KP, Balochistan and Punjab. According to a female ASP in Punjab, due to high demand for gypsum in her area, she and her father started buying it in bulk to sell it to individual farmers. With the help of demonstration and increasing awareness about the benefits of new technologies, the provision of services has extended beyond the initial group of farmers that ASPs engaged during FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 48 project implementation. This benefit of technology transfer was shared by 3 ASPs and 2 partner organizations in Punjab and Balochistan, and 10 farmer groups in all four provinces. As an example, according to a partner organization in Balochistan, more than 500 zero-till drills had been purchased by early 2022, since the farmers had learned of its benefits. According to 20 respondents (11 ASPs and 9 farmer groups), another factor that motivated ASPs to continue providing services was their desire to benefit farmers in their respective areas. CONTINUED USE OF TECHNOLOGIES BY FARMERS Out of 51 farmers interviewed, 46 continue to use services of ASPs on the introduced technologies after ASPTP ended. Among 74 respondents (5 partner organizations, 18 ASPs and 51 farmers), factors that enabled the continued use of new technologies among farmers include the following. ● Higher yield per acre (according to 36 farmers, 11 ASPs and 1 partner organization in all 4 provinces, for laser levelling, zero-till drill, soil testing kit, gypsum, HEIS, bed planting and biozote). ● Lower input cost (according to 30 farmers and 9 ASPs in Sindh, Punjab and Balochistan for zero-till drill, HEIS, bed planting, soil testing kit, FPM, gypsum, laser leveling and biozote). ● Improved soil quality and fertility (according to 21 farmers, 4 ASPs and 1 partner organization all in Punjab, for gypsum, FPM, soil testing kit and biozote). ● Improved product quality and water conservation (according to 18 farmers and 7 ASPs in 4 provinces, for laser leveling, HEIS, FPM, gypsum, soil testing kit and biozote). Farmers show a similar trend as ASPs vis-à-vis factors that constrained their continued use of technologies. In KP, Sindh and Punjab, 14 respondents (6 ASPs and 8 farmer groups) shared that unavailability of technology in their area is the most significant obstacle to sustained adoption. As one of the farmers in KP said, due to non-availability of the technology (laser) in our area, the program has had no positive impact on the farmers. In addition to laser land levelling in KP, respondents also expressed these concerns for FPM and biozote in Punjab. Farmers’ experience with these technologies has already been described above in findings under the section on Outcomes. Finally, 8 farmer groups (or 24 farmers) in Sindh, Balochistan and Punjab stated that floods damaged their land and crops and disrupted technology provision and adoption. FINDING 3 - KEY LESSONS LEARNED The ASP approach has a lot of potential for replication, provided there is continued effective demonstration of technology in the field and greater attention to sustainability at the design stage. This requires careful selection of technologies, keeping in view their marketability in the context of local constraints such as government subsidies on technology adoption and availability of testing labs. FINDING 4 - GENDER INTEGRATION Based on interview with a key informant from ICARDA, as well as the project’s quarterly and final reports, only 2 out of 70 ASPs trained were female. They were trained in service provision of gypsum and biozote technologies in the rain-fed areas Punjab. Out of 18 ASPs interviewed during primary data collection, only one was female. She was providing service of gypsum in rain-fed areas of Punjab. Both primary and secondary data revealed that women’s participation was not included as a selection criterion for ASPs as is reflected in Annex XII. ICARDA reports indicate that the project faced challenges in accessing females in the farming communities. In addition, due to cultural issues women were not allowed by family members to accept the technologies introduced through ASPTP. Primary data indicate women are traditionally not involved with handling machinery (tractor, etc.) in fields, though they help male family members or male farmers with sowing and harvesting. According to respondents, the key factors that hindered FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 49 women participation in agricultural activities are barriers to mobility (1 PO and 1 ASP in Punjab), domestic responsibilities (1 PO in Punjab), and cultural norms (1 PO in KP, 2 ASPs and 3 farmers in Sindh). Nonetheless, gypsum introduction in rain-fed areas of Punjab by a female ASP helped female farmers resolve the problem of delayed crop production due to wet soil. Through ASPTP support, the ASP taught these women to run successful businesses by teaching them to keep a close eye on their cash flows to ensure profitability and taught them how to deal with all the stakeholders involved, from the factories to the farmers and everyone in between. ICARDA taught them to be self-reliant businesswomen running profitable businesses. CONCLUSIONS Based on the findings above, following are the key conclusions for ASPTP. ● Gypsum and bed planting in Punjab, ridge planting in Sindh and zero-till drill in Balochistan were successful because of effective field demonstration by ASPs, higher yield, and reduced input costs, which led to increased income for farmers. While banana chopper in Sindh and FPM and happy seeder in Punjab were not successfully adopted. It is difficult to conclude confidently whether other technologies like laser land leveling, soil testing kit, HEIS and biozote were completely successful or unsuccessful. While these technologies increased farmers’ income and yield, their adoption was affected by the local context. For instance, the absence of a local maintenance workshop in KP affected the adoption of laser land leveling, which was quite successful in Sindh where required market linkages were present. ● Evidence suggests that the majority of ASPs and farmers experienced an increase in annual income through adoption of new technologies. The main factor for the increase in income for ASPs was demand for services by farmers. On the other hand, the farmers attributed the increase in their income to the increase in crop yield and reduction of input cost. However, adoption of new technology did not necessarily reduce farmers’ input cost as in some cases it was expensive, though in a majority of cases, the increased input cost was offset by increase in income over a longer period of time. ● Evidence suggests that most of the ASPs continued to provide services after ASPTP ended because of the demand for the services from farmers. Similarly, farmers continued to avail themselves of their services because of yield increase, reduced input cost, water conservation, and improved soil and product quality. Main factor that constrained sustainability was the lack of local availability of technology. The 2022 floods also disrupted the process. ● As far as gender is concerned, women participation in ASPTP interventions was minimal as only a few female ASPs benefited from the trainings. The project did not specifically target women as evidenced by the fact that women participation was not included in the ASP selection criteria. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 50 PHYTOSANITARY RISK MANAGEMENT PROGRAM (PRMP) FINDING 1 – INTENDED AND UNINTENDED OUTCOMES As mentioned earlier in the report, PRMP was implemented between 2014 and 2019. For the first two years of its implementation, the project focused on Balochistan and Sindh, and for the last two years, project activities were implemented in Gilgit and Baltistan. To assess the extent to which PRMP was able to mitigate the impact of pests on crops, data was collected on four indicators, namely changes in infestation, changes in the use of pesticides and other chemicals, changes in farmers’ farm income (including changes in yield) and changes in area under cultivation for the targeted crop. Table 10 below summarizes responses of 14 farmers interviewed on these indicators. Table 10 Summary of KII Responses from Interviews Indicator of mitigation of impact of pests Type of Answer Number of Farmers Reporting in Region (by crop) Gilgit (Fruit trees) N=4 Baltistan (willow) N=3 Balochistan (apple) N=3 Sindh (papaya) N=4 Total (14) Changes in Infestation Increased - - - - - Decreased 4 3 2 3 12 No-change - - 1 1 2 Use of Pesticides/ Chemical Sprays Increased - - - - - Decreased 4 3 2 3 12 No-change - - 1 1 2 Changes in Farm Income/Yield Increased 2 3 2 1 8 Decreased 2 - - 2 4 No-change - - 1 1 2 Changes in Area Under Cultivation Increased - 1 - 1 2 Decreased 1 - 1 3 5 No-change 3 2 2 - 7 Source: KIIs conducted with PRMP beneficiary farmers. As the evaluation required, four crops namely papaya, apple, fruit trees and willow were selected, one each for Sindh, Balochistan, Gilgit and Baltistan regions, respectively. In Punjab, rice was selected, but only one interview45 was conducted, and for the most part the respondent could not remember the effect of project interventions. Therefore, no findings related to rice are included in this section. The following are the key findings in this regard. CHANGE IN INFESTATION Key informants from partner organizations (9 out of 10) and most of the farmers (12 out of 14) reported a decrease in infestation due to the release of biocontrol agents in the targeted areas during PRMP implementation, while two farmers reported no change. In Gilgit, a key informant from partner organizations and farmers (3 out of 4) reported a decrease of 20% to 30% in infestation caused by fruit-fly on fruit trees. A key informant from a partner organization said that the intervention worked 45 The evaluation team requested CABI to provide the participant’s list of the training workshops on rice value chain; however, the evaluation team received contact information of only three participants (out of 187) and only one responded to the evaluation team’s meeting request. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 51 to some extent, however the infestation cannot be fully controlled by providing a few thousand traps or release of biocontrol agents. In Baltistan, both the key informants from partner organizations (2 out of 2) and the farmers (2 out of 3) reported a decrease in infestation by 80% to 90%. One of the farmers said that their trees were affected so badly that they could not touch or clean them, but after PRMP they are free from infestation, and they can easily cut them (for wood). In Balochistan, according to the PRMP Project Completion Report (December 2019)46, the project interventions resulted in 60% to 70% reduction in infestation on apple crop. Both key informants from partner organizations (2 out of 2) and the farmers (2 out of 3) reported a decrease in infestation during the project implementation period. According to one key informant from a partner organization, the reduction in infestation was 10% to 20% during that period. However, two (out of 3) farmers reported that their orchards were destroyed by floods and hailstorms, and no apple trees were left. In Sindh, key informants from partner organizations (3 out of 3) and farmers (2 out of 3) reported a decrease in infestation during the project implementation period. According to farmers (2 out of 3) and a key informant from a partner organization, there was a 50% to 80% reduction in infestation caused by mealybug on papaya crops during that period. USE OF PESTICIDE AND CHEMICAL SPRAYS According to the Post Project Assessment Report, a 75% reduction was observed in the application of pesticides and chemical sprays on different crops. Data from field visits indicate that most farmers (12 out of 14) reported a decrease in the application of pesticides and chemical sprays on their farms during the two years of project implementation. In Gilgit, all farmers (4 out of 4) reported that the application of pesticide sprays was reduced to a maximum of one spray per year. A key informant from a partner organization in Gilgit said that after awareness campaigns, training and release of biocontrol agents, the farmers stopped using sprays and they convinced other farmers to follow suit. Similarly in Baltistan, farmers (3 out of 3) reportedly stopped applying pesticides on their crops. One farmer in Baltistan reported that even though his income decreased after the use of biocontrol agents, he was still trying to promote the natural enemies (biocontrol agents). He did not use chemicals and pesticides because that will wipe out the Natural Enemies Field Reservoir (NEFR) population. The main reason for the reduction in pesticides application cited by the farmers (4 out of 7) in Gilgit and Baltistan was increased awareness about the harmful effects of pesticides. Farmers interviewed in Balochistan (2 out of 3), and Sindh (3 out of 4) reported a reduction in application of pesticide during the two years of PRMP implementation. However, all farmers (7 out of 7) resumed the use of pesticides after PRMP ended. In Balochistan the farmers resumed using pesticides because of the ineffectiveness of the biocontrol agents against pest attacks (reported by 1 out 3), damage caused by floods and hailstorms and plant diseases (1 out of 3), unavailability of biocontrol agents (1 out of 3), and hot weather (1 out of 3). Key informants (2 out of 2) from a partner organization in Balochistan reported that although the use of biocontrol agents was effective in controlling infestation, the farmers wanted quick results and therefore they used pesticides that killed the biocontrol agents, while the fruits were still infested. In Sindh, farmers resumed the application of pesticides due to the outbreak of diseases such as Sindh papaya China cutting disease (papaya leaf curl China virus) and root rot disease, which were not controlled by biocontrol agents (reported by 1 out of 4), mealy bug attack during summer (1 out of 4), 46 CABI. PRMP Project Completion Report, December 2019. Page 5. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 52 and use of pesticides by neighboring farmers. One farmer in Sindh stopped using pesticide sprays because mealybug disease ended on papaya in his farm. CHANGE IN INCOME/YIELD According to the Post-Project Assessment Report, a 19% increase was observed in the farm income of beneficiary farmers47. Data from field visits indicate that more than half (8 out of 14) of farmers reported an increase in their income during the implementation period of PRMP. However, four (out of 14) farmers reported a decrease in their income, while two (out of 14) farmers reported no change in income. On the other hand, a large number of key informants from partner organizations (11 out of 12) believed that farmers’ income had increased because of PRMP interventions. In Gilgit, about half (2 out of 4) respondents reported an increase in income, while the other half reported a decrease in their crop income. Farmers reporting increase in income attributed that to increased yields and improved quality of produce. On the other hand, farmers reporting a decrease in income attributed that to low yields due to increased infestation and lack of pesticide use. In Baltistan, all farmers (3 out of 3) interviewed reported an increase in their farm income. These farmers attributed the increase in income to the control of mealybugs that infested the trees. A key informant from a partner organization in Baltistan said that around 150 thousand trees were saved from mealybug infestation (through PRMP interventions), and also the quality of fruits and vegetables improved which helped the farmers increase their income. In Balochistan, a majority (2 out of 3) of farmers reported an increase in income during the two years of implementation of PRMP. The farmers attributed their increase in income to about 50% increase in yield of apples. However, after the project’s closure, two of the three farmers reportedly removed the apple orchards due to water scarcity and destruction caused by hailstorms and floods. The remaining one farmer had stopped using the biocontrol agents and attributed his current increase in income to use of pesticides. In Sindh, half (2 out of 4) of the farmers interviewed reported a decrease in their crop income, while one farmer reported an increase in income. The farmers attributed the decrease in income to the outbreak of diseases (2 out of 4) and use of pesticides by neighboring farmers. AREA UNDER CULTIVATION About half (7 out of 14) of the farmers interviewed reported no change in the area under cultivation for the targeted crops. About a third (5 out of 14) of the farmers reported a decrease in the area under cultivation and the remaining two reported an increase. In Gilgit and Baltistan, most farmers (5 out of 7) reported that their area under cultivation remained the same due to their limited land holdings. However, they reportedly planted more trees on the same field and benefited the Ten Billion Tree Tsunami Program initiated by the government. The farmers revealed that due to reduced infestation they were encouraged to plant more trees. One key informant from a partner organization in Gilgit said that “there is an increasing trend to cultivate fruit plants, and the credit for that goes to this project (PRMP). The Ten Billion Tree Tsunami Program has been on-going for the last three years, and the demand for plants is more than we can supply”. In Balochistan and Sindh, farmers (4 out of 7) reported a decrease in the area under cultivation for the targeted crops due to destruction of apple farms in Balochistan by floods and hailstorms, and outbreak of diseases that affected the papaya crop in Sindh. A key informant from a partner organization in Balochistan said that “the apple crop is water intensive and requires a lot more water than other plants. Balochistan has been experiencing drought and water shortage, and therefore the cultivated land under apple crop is decreasing”. UNINTENDED BENEFITS 47 CABI. PRMP Post-project Assessment Report, September 2014-September 2019, Page 4 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 53 Secondary data shared by CABI with the evaluation team indicate that 25 research documents including 12 research papers and 13 abstracts were published with PRMP support during the life of the project.48 A key informant from CABI reported that all research publications published with PRMP support were not included in original objectives of PRMP and therefore were the unintended benefits of the project. FINDING 2 – SUSTAINABILITY Sustainability was viewed from the perspective of continued rearing of parasitoids by labs upgraded with PRMP support; availability of required human resource; institutionalization of PRMP interventions; and whether biocontrol agents were used by farmers after PRMP concluded. The key findings for the sustainability of PRMP are as follows. USE OF LABS AND REARING OF PARASITOIDS During the evaluation field visits, the evaluation team visited labs upgraded through PRMP support in Gilgit, Baltistan, Balochistan and Sindh. It was observed that labs in Gilgit and Balochistan were rearing parasitoids49, whereas the labs in Baltistan (Skardu) and Sindh (Karachi) were not operational in terms of rearing parasitoids. The same fact was confirmed by the key informants from partner organizations in the respective locations. Key informants (1 out of 3) from partner organizations in Gilgit reported that about 30,000 to 50,000 parasitoids are annually reared in the lab located in Gilgit. It was reported that about 30,000 parasitoids were produced in the current year as compared to 638,936 parasitoids that were produced during the implementation period of PRMP. Key informants (2 out of 3) from partner organization in Balochistan reported that the lab has been operational and rearing parasitoids after PRMP ended, however, one key informant said that the limited resources hindered them from helping the farmers any further. Another key informant reported that lack of electricity was the reason for not producing parasitoids up to its capacity. In Sindh, the lab upgraded through PRMP is located in Karachi University. A key informant from a partner organization in Sindh reported that the lab was functional till 2019-2020, however, because of an incident of power outage in the university, all the parasitoids in the lab died and consequently the rearing of parasitoids was discontinued. In Baltistan, key informants (2 out of 3) from partner organizations reported that the lab was handed over to the Integrated Pest Management (IPM) department after PRMP concluded, however due to lack of financial resources the rearing of parasitoids was discontinued. AVAILABILITY OF STAFF AND TRAINING More than half (7 out of 12) of the key informants from the partner organization reported that there was no major change in the staffing structure of the departments in Gilgit, Balochistan and Sindh. However, the daily paid labor (DPL) supported by CABI through PRMP left after PRMP’s closure. In Baltistan, the respondent revealed that there was no lab staff in Baltistan; the PRMP supported staff left as the project ended and the regular staff members were transferred to other locations. About a quarter (4 out of 12) of respondents from the partner organizations believed that requisite training was provided, and knowledge was transferred to incumbent employees in respective departments, however the same number of respondents believed otherwise and reported that no training opportunities were available to incumbent employees. 48 CABI. August 25, 2022. PRMP PowerPoint Presentation. Slide 45 49 A parasitoid is an organism that lives in close association with its host at the host's expense, eventually resulting in the death of the host (harmful pests). Rearing parasitoids means production or multiplication of parasitoids in a laboratory or controlled environment. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 54 INSTITUTIONALIZATION OF INTERVENTIONS Half (6 out of 12) of the respondents from partner organizations, mostly in Gilgit (3) and Balochistan (3) believed that the respective departments have adopted the capacity building support and the technology to rear biocontrol agents shared by CABI through PRMP. Respondents (3 out of 4) from Gilgit reported that with support from the PRMP the department’s capacity has improved and the department is still rearing parasitoids and releasing them. Key informants (3 out of 3) in Balochistan reported that the department has continued PRMP activities, and the lab is still rearing biocontrol agents despite resource constraints. One key informant said that “we are trying to operate within the allocated budget for the directorate and still manage to provide brochures for awareness and arrange training sessions for the farmers”. Respondents (5 out of 6) from partner organizations in Sindh and Baltistan reported that the interventions initiated through PRMP were not continued by the respective departments. Respondents (2 out of 3) in Balochistan reported that even the staff trained through PRMP left the department and no parasitoids were being reared in the labs. Respondents (2 out of 4) in Sindh reported that the department did not have financial resources to continue the activities initiated through PRMP given that the biocontrol agents provided by PRMP were already dead because air￾conditioners were not working due to a 3-day long power outage in Karachi. USE OF PARASITOIDS BY FARMERS Most farmers (12 out of 14) interviewed reported that they had not been using biocontrol agents in their farms after the project ended. All farmers in Sindh (4 out of 4) and Balochistan (3 out of 3), and a majority of farmers in Gilgit (3 out of 4) and Baltistan (2 out of 3) reportedly stopped using the biocontrol agents after PRMP ended. The key reasons for not using biocontrol agents included non￾availability of biocontrol agents to farmers; elimination of the pest (in Baltistan); biocontrol agents not effective against diseases (Sindh); destruction of orchards due to floods and hailstorms (Balochistan); and loss of interest of farmers due to no follow-up from agriculture department staff after closure of PRMP. Physical observations by the evaluation team in the targeted villages (4 in district Gilgit, 2 each in district Skardu and Shigar, and 1 each in districts Quetta, Pishin and Kila Saifullah) of both GB and Balochistan revealed all the NEFRs and small farmer managed NEFRs established by PRMP in GB and Balochistan were abandoned. Both types of NEFRs were utilized as trashcans and storage facilities rather than rearing parasitoids. FACTORS THAT DETERRED SUSTAINABILITY Based on interviews with key informants and beneficiary farmers of PRMP, the key factors that deterred sustainability of PRMP interventions and its results are as follows. ● Unavailability of biocontrol agents to farmers (7 out of 14 farmers) ● Lack of financial resources for agriculture departments (4 out of 12 POs) ● Effect of floods and hailstorms that resulted in removal of trees from the orchards (2 out of 14 farmers) ● Limited access of agriculture department staff to reach out to farmers due to the vastness of distances involved in making physical visits (1 out of 12 POs and 2 out of 14 farmers) ● Loss of farmers’ interest in use of biocontrol technology due to lack of follow-up from agriculture department staff (1 out of 12 POs and 7 out of 14 farmers) ● Short duration of the project (3 out of 12 POs and 3 out of 14 farmers) ● Negative effects of the use of pesticide and chemicals by neighboring farmers (1 out of 12 PO and 3 out of 14 farmers) ● Lack of acceptance of biocontrol technology by the farmers (7 out of 12 POs) FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 55 FINDING 3 - KEY LESSONS LEARNED According to PRMP project reports and information from key informant interviews, the project was implemented over a period of five years with approximately two years of implementation in Balochistan and Sindh, and about two years in Gilgit and Baltistan. This suggests that most beneficiary farmers were associated with the project for one to two years or one to two crop seasons. According to key informants (3 out 12) from partner organizations and farmers (3 out of 14), the duration of the PRMP was short to achieve any significant and lasting results. One of the farmers interviewed said that “the project must not end until the results are achieved. If the project stops, then we stop too. The project should continue for a long term”. Similarly, 6 out of 14 farmers reported that they lost interest in using biocontrol agents because the concerned partner agriculture departments staff stopped follow-up visits to their fields. FINDING 4 – GENDER INTEGRATION According to the PRMP Project Completion Report (Dec 2019), the project’s overall outreach to men and women included the following. ● Training 1,221 (293 female and 928 male) stakeholders and government officials on biological control and Sanitary and Phytosanitary (SPS) ● Training 7,080 (1,389 female and 5,691 male) farmers on biological control ● Training 868 (232 female and 636 male) fresh produce industry personnel on SPS compliance ● Training 187 male rice industry personnel on rice value chain improvement Interviews with the agriculture officers, in Gilgit, Baltistan, Balochistan and Sindh indicate that female farmers were mostly engaged in Gilgit. According to information shared by CABI50 and key informants from partner organization in Gilgit, women comprised of about 49 percent of the agricultural labor force in Gilgit Baltistan region and were deeply involved in agricultural operations. Many women were considered as entrepreneurs, and they actively participated in PRMP supported activities including training courses on management of fruit fly through use of biocontrol technology. More than 60% of these training participants in Gilgit were female. However, in Baltistan, Balochistan and Sindh women participation in PRMP interventions was minimal due to lack of female participation in agricultural activities overall, however the project engaged women in its activities wherever possible (12 out of 12 POs). CONCLUSIONS Based on these findings, following are the key conclusions for PRMP. ● Evidence suggests that PRMP was effective in mitigating the impact of pests during the life of the project. This was demonstrated by reduction in infestation and use of chemicals and pesticides coupled with increase in farmers’ income in Gilgit, Baltistan, Balochistan and Sindh. However, once the project ended, its effectiveness in terms of mitigating the impact of pests could be seen in Gilgit and Baltistan, where the infestation caused by the target pests remained under control. In Balochistan and Sindh however, after PRMP ended the project effectiveness in terms of mitigating the impact of pests was lost, as evidence suggests that area under cultivation for farmers decreased (particularly in Balochistan) and most farmers reverted to their old practice of using pesticide and chemical sprays on their farms to control pests and diseases, and any increase in income and crop yield was not attributable to project. ● Very little of the PRMP interventions has been sustained considering half of the PRMP supported labs were not operational and the other half were producing far less parasitoids than they used to when the project was active. The main reasons for lack of sustainability are that partner organizations were suffering from lack of financial resources to continue activities 50 https://blog.cabi.org/2018/11/26/empowering-more-women-in-the-fight-against-fruit-flies-in-pakistan/ FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 56 related to biocontrol technology, and farmers had stopped using biocontrol agents in their fields and reverted to the use of pesticides and chemical spray. ● As far as gender is concerned, more men benefited from the project interventions than women. This was demonstrated by the fact that about a third of training participants from government partner organizations and a fourth of beneficiary farmers were women. The reason cited for overall low female participation was the lack of female involvement in agricultural activities in the targeted areas. Nevertheless, in Gilgit where women were actively engaged in agricultural activities, PRMP actively supported women by providing training on control of pests through biocontrol agents. This was demonstrated by the fact that more than 60 percent of the beneficiaries from the activity in Gilgit were women. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 57 RECOMMENDATIONS The recommendations for the evaluation are presented as follows. 1. Given the constraints of the research institutions to facilitate the commercial production of wheat and cotton varieties, USAID should consider a component of commercialization to support commercial production of cotton and wheat varieties by involving the private sector, including seed producing companies. 2. Technology for which machinery or spare parts are not available locally should not be rolled out for demonstration and adoption. Soil testing is typically required once in 2 to 3 years and is available from government and private sources. Therefore, the testing kit should not be included in future interventions for small farmers. Moreover, laser land leveling in KP, happy seeder, biozote and FPM labs in Punjab, and banana chopper in Sindh are among technologies that require careful location-specific assessment of sustainability at the design stage. 3. The interventions focusing on women on small farms should be well-defined at the design stage and duly reflected in beneficiary selection and training requirements. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 58 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK SECTION C – STATEMENT OF WORK C.1 OBJECTIVE The objective of this BPA Call is to provide evaluation services to the activities identified in Section C.2 below. C.2 ACTIVITIES TO BE EVALUATED UNDER THIS ORDER 1. This BPA Call will provide evaluation services for the following activities: 1. Wheat Productivity Enhancement Project (2010-2021) 2. Cotton Productivity Enhancement Project (2010-2021) 3. Phytosanitary Risk Management (PRM) Program (2014-2019) 4. Agriculture Service Providers Training Program (ASP) (2018-2021) C.3 ACTIVITY BACKGROUND 1. Activity Summary Pakistan Agreement Number: Contract # 391-USDA-13-0001 Activity Start Date: November 14, 2012 Activity End Date: September 30, 2021 Total Estimated Cost (TEC): $26,100,000 Location of Activities (Provinces/Districts): Nationwide Implementing Partner: United State Department of Agriculture (USDA) USAID/Pakistan Results Framework Linkages: DO 3: Increased Private Sector-led Inclusive Economic Growth IR 3.2: Employment opportunities increased Sub IR 3.2.2: Agriculture value chains enhanced Sub IR 3.2.2a: Number of individuals in the agriculture system who have applied improved management practices and technologies with USG assistance 2. Background Information 2.1 Country and Sector Context A major constraint to Pakistan’s agricultural sector, and consequently Pakistan’s economy and food security, is a lack of institutional capacity to address key issues affecting the efficient production and trade of agricultural products. Pakistan possesses a variety of relevant institutions at the national and provincial levels, both public and private, but often there is a lack of technical expertise or experience with innovative approaches that can enable effective use of resources on hand as well as FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 59 catalyze new ways of solving problems that affect food security, and economic growth, especially in rural areas. 2.2 Problem or Opportunity the Activity was Designed to Address The objective of the Pakistan Agriculture Programs (PAP) was to increase Pakistan's agricultural productivity by improving skills in research and development, introduction of new varieties, and by increasing adoption of new agricultural practices and technologies. Through PAP, the United State Department of Agriculture (USDA) directly supported the Government of Pakistan’s (GOP) efforts to increase agricultural productivity; build institutional capacities to better regulate animal and plant health and food safety issues; and strengthen institutional research capabilities and U.S.-Pakistani scientific collaboration. Under this activity, USDA (through its sub-awardees) implemented several interventions focusing on reducing plant and animal diseases, strengthening regulatory oversight for agricultural trade, and improving agricultural development and food security while promoting strong U.S. Government (USG) and university linkages with Pakistani agricultural scientists. PAP included the following 14 sub-activities: 1. Wheat Productivity Enhancement Project (2010-2021) 2. Cotton Productivity Enhancement Project (2010-2021) 3. Dairy Production Improvement Project (2014-2018) 4. Foot and Mouth Disease Control Program (2011-2015) 5. Control of Peste de petits Ruminants (PPR)-Demonstration Project (2014-2017) 6. Animal Emergency Response and Mitigation Team Development (2015-2017) 7. Sanitary and Phytosanitary (SPS) Distance Learning Project (2011-2017) 8. Phytosanitary Risk Management (PRM) Program (2014-2019) 9. Watershed Rehabilitation and Irrigation Improvement Project (2011-2016) 10. Aquaculture (2011-2017) 11. Water Dialogue (2013-2015) 12. Safer Food through Aflatoxin Control in Pakistan (2018-2021) 13. Improving Soil Fertility through Extension (2013-2018) 14. Agriculture Service Providers Training Program (ASP) (2018-2021) This evaluation will focus on four key sub-activities i.e., wheat, cotton, PRM and ASP because the PAP invested most of its resources on these interventions and these interventions were expected to produce more tangible results and create long term impacts on focus areas. The details of each of the four sub-activities/interventions, including how they addressed the problem are as follows: 2.1.1 Wheat Productivity Enhancement Project (WPEP) (2010-2021) The WPEP worked to protect and enhance the productivity of wheat in Pakistan, with particular attention to wheat rusts. Pakistan, with 9 million hectares of wheat, is a front-line state in the arms race between wheat farmers and wheat rusts. Large scale epidemics of leaf or yellow rust caused hundreds of millions of dollars in crop losses during the last 50 years, and stem rust, including the Ug99 lineage, is a looming threat. To prevent future threats and safeguard Pakistan wheat production, USAID launched WPEP in 2010 as an outcome-driven science collaboration involving USDA, CIMMYT, PARC, ICARDA, and 11 Pakistani scientific organizations working in all provinces of Pakistan. The program pursued six research objectives leading to the identification, adoption, and FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 60 optimal agronomic management of new, high-yielding, and disease- resistant wheat varieties, as well as building the capacity of national wheat scientists. WPEP significantly contributed to increased wheat production and productivity in the country with the production increased from 24 million to over 26 million tons during the performance period. The activity had following objectives: • To establish a country-wide surveillance program in order to determine if new strains of stem rust disease (specifically the Ug99 strain) were present; • Identify and develop new high-yielding, adapted varieties of wheat having genetic resistance to Ug99 and other major diseases; • Organize cooperative, country-wide testing of germplasm and varieties, as well as greater collaboration and sharing of germplasm among the Pakistani breeders; release high-yielding, adapted, disease resistant wheat varieties to all wheat growing areas in Pakistan. The activity interventions were designed to address the issues around low wheat productivity due to wheat stem rust (Ug99). The activity has achieved intended results by introducing 47 high yielding disease resistant varieties which contributed to 20% more yield gain in farmer fields, increasing the resistance of Pakistani wheat germplasm against wheat stem rust (Ug99) from 15% to 49%. The activity also scaled out approximately 800 tons of quality wheat seeds to small holder farmers giving them 20-50% extra wheat production while diversifying Pakistan wheat genetic base with introduction of more than 3,000 new lines/segregating populations. The activity also revitalized National Wheat Traveling Seminar and Annual Wheat Planning meeting and provided technical backstopping to the national wheat researchers throughout the country including AJK, and Gilgit￾Baltistan. 2.1.2 Cotton Productivity Enhancement Project (CPEP) (2010-2021) The objective of CPEP was to mitigate the effects of the Cotton Leaf Curl Virus (CLCuV) disease especially for smallholder farmers and then find a source of resistance and work with the Pakistani breeders on developing resistant varieties. Another objective was to find the source of resistance and put it into U.S. cotton lines so we would be prepared in case the cotton leaf curl virus became a problem in the U.S. All these goals were accomplished; 50 s 50 lines of cotton which are resistant to CLCuV have been developed. The new varieties are being introduced to the farmers by the research centers, extension agents, and farmer cooperative organizations that the activity worked with. 2.1.3 Phytosanitary Risk Management (PRM) Program (2014-2019) The PRM activity had three main objectives, namely to iImprove technical capacity of national partners to survey pests and to develop and deploy biocontrol agents to reduce the impact of plant pests in Pakistan, mitigate the impact of post-harvest pests of rice and horticultural crops and improve the capacity of plant health regulators to certify exports of agricultural commodities, and analyze and generate actionable information to regulate aflatoxins in fresh produce supply chains in Pakistan. The activity achieved these objectives by establishing six biocontrol laboratories across Pakistan, imparting 332 targeted training programs which trained over 1,000 technical experts and over 7,000 farmers in biocontrol rearing and release, and best management practices. The activity also resulted in approximately 22% increase in papaya production in Sindh,increasing the incomes of papaya farmers by 15%. 2.1.4 Agriculture Service Providers (ASP) Training Program (2018-2021) The ASP activity aimed to disseminate best soil and water management technologies and practices for the farmers and other stakeholders, and increase adoption of selected technologies and practices by the farmers through service provision by the trained ASPs. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 61 The activity achieved these objectives by training 70 ASPs in technical and business skills. These ASPs introduced agricultural technologies within their communities through demonstration, farmer training. As a result of activity interventions, more than 12,000 farmers adopted modern technologies and best practices. 2.3 Activity Documents or Performance Information Sources Available to the Evaluation Team Following documents/information sources will be available to the evaluation team: 1. Baseline information 2. Annual performance reports 3. Monitoring, Evaluation and Learning plan 4. Quarterly performance reports 5. Other Activity or sub-activity documents, if available. C.4 GEOGRAPHIC FOCUS PAP is implemented nationwide including Gilgit-Baltistan and AJK, with the focus limited to one or two provinces based on the nature of interventions under individual sub-activities. The geographical focus of the selected sub-activities is provided below: • WPEP- Nationwide • CPEP – Nationwide (with major focus on cotton producing districts in Punjab and Sindh) • PRM – Nationwide • ASP Training Program – Nationwide C.5 STATEMENT OF WORK The primary purpose of the evaluation is to identify the achievements of PAP in line with its intended objectives under Section 2.2 above and inform management decisions on on-going and future EGA activities. It will also measure the contribution of the PAP on the focus areas of the agriculture sector in Pakistan. The findings will be highly useful to guide the future planning, design, implementation and other related decisions that are likely to improve performance of future similar investment efforts. The learnings from this evaluation will also be shared with relevant government departments to help inform future interventions in agriculture research and extension. The primary audiences for the evaluation includes: 1) the USAID/Pakistan mission, particularly the EGA Team, 2) the USAID Bureau for Asia which covers Pakistan, (3) the implementing partner, USDA (including the sub-awardees), (4) and the Government of Pakistan (GOP). This is the final performance evaluation and will start on or about July 25, 2022 and will be completed on or about December 09, 2022. The PAP ended in September 2021, however, the sub￾awardees will have on ground presence to facilitate evaluation. A. Evaluation Questions As per ADS 201.3.6.2 - Evaluation Principles and Standards, planning for evaluation and identifying key evaluation questions at the outset should improve the quality of strategic planning and project and activity design, and should guide the collection, management, use, and delivery of data during implementation. Evaluations should address the most important and relevant questions about strategies, projects, or activities that cannot typically be answered alone through routine monitoring data, or other existing evaluations and studies, and that serve the informational needs of stakeholders. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 62 The three key questions that will guide this evaluation are: Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Explanation: These results include but not limited to: a. How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. b. As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why? The question should indicate if the adoption of technologies increased incomes of ASPs and farmers? c. What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project interventions which included Punjab, Sindh, Balochistan and Gilgit Baltistan. The effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Question 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Explanation: This question will evaluate the likelihood that the interventions under the four identified sub￾activities are sustainable i.e., that the varieties, practices, technologies introduced will persist beyond the project support. Specific aspects will include institutionalizing these interventions in the Government (both federal and provincial) and private organizations. Question 3: What are the lessons learned for future design and mechanism of similar projects? Explanation: This question should investigate the major challenges and successes especially about the USDA and their sub-partners. Recommendations should focus on changes in the implementation approach that could improve performance in new Participating Agency Program Agreement and future activities. Gender Integration: For all the three key evaluation questions, the evaluator must check whether these interventions provided equal opportunities for both men and women? If not, what could have been done to encourage women participation in the activity? If yes, what were the approaches that encouraged women to benefit from the activities? B. Data Collection and Analysis Methods As per 201.3.6.4, the selection of method or methods for a particular evaluation should consider the appropriateness of the evaluation design for answering the evaluation questions and the availability and accessibility of primary and/or secondary data, as well as balance cost, feasibility, and the level of rigor needed to inform specific decisions. The prospective evaluator will propose the most appropriate evaluation design incorporating innovative and appropriate data collection and data analysis methods to answer each evaluation question that generate the highest-quality, most credible data and evidence that correspond to the questions being asked, while taking into consideration time, budget, scale, feasibility, and other practical considerations. The evaluator will suggest both qualitative and quantitative methods, including appropriate sample size, specific to each evaluation question. The BPA holder must use gender-sensitive data collection methods and tools. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 63 The data analysis should progress from findings to conclusions and recommendations for future interventions. Generally, an evaluation will employ rigorous analytical methods to review qualitative and quantitative data. Qualitative data will provide rich evidence of the outcomes associated with activity interventions and, more importantly, how activity interventions contributed to anticipated results, reasons interventions may have failed to produce anticipated results, and unanticipated results. The evaluation team will use rigorous content analysis and coding techniques to identify key themes in the qualitative data and use quantitative analysis (e.g., descriptive statistics, cross￾tabulation, or regression) methods to report results and identify patterns and correlations in the data. If the evaluation team is able to obtain relevant and quality (quantitative) data from the sub￾activities’ M&E system and the sub-awardees’ record, it will use methods appropriate to the data to identify patterns and draw out findings. The analysis will disaggregate results by gender whenever applicable, and draw out specific conclusions and recommendations. The evaluator will also provide a clear explanation of limitations and risks associated with the proposed evaluation design. The design will include “Getting to Answers Matrix” based on an illustrative matrix given below: Evaluation Question Data Source Data Collection Method Method of Data Analysis Limitations/ risks C. Gender Considerations The evaluator will ensure that evaluation designs, methodologies, data collection, analyses and reports adequately capture the situations and experiences of both males and females. PAP includes female beneficiaries. It partnered with women Agriculture development organization (WADO) in Sindh and Farmer Development Organization in south Punjab and trained women in early detection of cotton leaf curl virus disease and introduction of improved agricultural practices to increase productivity. Appropriate gender-sensitive data collection and analysis methodologies will inform us about the potential constraints in women participation in agricultural activities. C.6 DELIVERABLES Deliverables under this evaluation include: 1. Evaluation Design: Evaluation design will include detailed methodology, key questions, data collection plan, plan, data collection instruments, data analysis plan, detailed timeline and staff composition for completion of the work requested in the SOW. Upon award of the call, the prospective BPA holder and USAID/Pakistan, through the call COR, will review the design again and will provide final approval for implementation. The final design will be approved by USAID/Pakistan within two weeks of the award of the contract. No data collection will be started without the approval of USAID/Pakistan. 2. Debriefing with USAID/Pakistan on Findings, Conclusions, and Recommendations: After completion of the data collection and analysis and before finalizing the first draft of the report, the evaluation team will present the major findings, conclusions, and recommendations to USAID/Pakistan. The evaluation team will consider USAID/Pakistan comments for use in the draft report, as appropriate. 3. Draft Evaluation Report: The draft evaluation report must meet the requirements described in ADS 201mah, USAID Evaluation Reporting Requirements: • The Evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 64 • The Evaluation report should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • The Evaluation report should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • The Evaluation methodology should be explained in detail and sources of information properly identified. • The Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • The Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • The Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. The draft evaluation report will follow the evaluation report template attached to the solicitation. At minimum, the evaluation report must: 1. Identify the evaluation as either an impact or performance evaluation per the definitions in ADS 201. 2. Include an abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the evaluation report cover. 3. Include an Executive Summary 2–5 pages in length that summarizes key points (purpose and background, evaluation questions, methods, findings, and conclusions). 4. State the purpose of, audience for, and anticipated use(s) of the evaluation. 5. Describe the specific strategy, project, activity, or intervention to be evaluated including (if available) award numbers, award dates, funding levels, and implementing partners. 6. Provide brief background information. This should include country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis, theory of change, or simply how the intervention addresses the problem. 7. State the evaluation questions. 8. Describe the evaluation method(s) for data collection and analysis. 9. Describe limitations of the evaluation methodology. 10. Include evaluation findings and conclusions. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 65 11. If recommendations are included, separate them from findings and conclusions. 12. Address all evaluation questions in the Statement of Work (SOW) or document approval by USAID for not addressing an evaluation question. 13. Include the following annexes: i. Evaluation SOW. If the SOW was revised over the course of the evaluation, the evaluation report should include the updated SOW as an Annex rather than the original SOW. The Contracting Officer’s Representative of the evaluation must agree upon, in writing, all modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline. ii. A description of evaluation methods (if not described in full in the main body of the evaluation report). iii. All data collection and analysis tools used, such as questionnaires, checklists, survey instruments, and discussion guides. iv. All sources of information—properly identified and listed. v. Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. vi. Signed disclosures of conflicts of interest from evaluation team members. vii. Summary information about evaluation team members, including qualifications, experience, and role on the team. 14. Include enough information on the cover of the evaluation report so that a reader can immediately understand that it is an evaluation and what was evaluated. The evaluation cover must: i. Include a title block in USAID light blue background color. ii. Include the word “Evaluation” at the top of the title block and center the report title underneath that. The title should also include the word “evaluation.” iii. Include the following statement across the bottom of the cover page: “This publication was produced at the request of the United States Agency for International Development. It was prepared independently by [list authors and organizations involved in the preparation of the report].” iv. Feature one high-quality photograph representative of the project being evaluated and include a brief caption on the inside front cover describing the image with photographer credit. v. State the month and year of report publication (e.g., when final and approved by USAID Pakistan). vi. State the individual authors of the report and identify the evaluation team leader. vii. As noted in ADS 201.3.5.17, evaluation reports must also conform to USAID branding requirements (see ADS 320, Branding and Marking) and comply with section 508 of the Rehabilitation Act (see ADS 302mak, USAID Implementation of Section 508 of the Rehabilitation Act of 1973). FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 66 The draft report will not exceed 40 pages, excluding annexes. The draft report will be submitted to the BPA Call COR for USAID/Pakistan review and comments. USAID will share feedback on the report within three weeks of its submission to the BPA Call COR. 4. Final Evaluation Report: The final report will address all USAID/Pakistan comments. The BPA holder will finalize the report and submit it to the COR for approval within two to three weeks of receiving comments from USAID/Pakistan’s BPA Call COR. 5. Raw Data: Per ADS 579 - USAID Development Data – all quantitative data collected for this evaluation will be submitted to USAID/Pakistan in electronic format within 30 days of Data collection completion. The BPA holder will submit the data in a format requested by USAID/Pakistan for internal archiving and also in formats specified in ADS 579 for uploading to the Data Development Library (DDL). Qualitative data will be delivered as 1) the coded segments used in analysis extracted from MAXQDA in an excel format or 2) tally sheets, as applicable to the analysis. The BPA holder will be responsible for submitting the data to DDL. 6. Development Experience Clearinghouse (DEC) Review: Prior to public dissemination, evaluation reports must be adjusted to remove information that falls under one of the “principled exceptions to the presumption in favor of openness” as described in ADS 201mae, Limitations to Disclosure and Exemptions to Public Dissemination of USAID Evaluation Reports. Accordingly, once the report is finalized, USAID/Pakistan may conduct a DEC review of the report. The COR will share the DEC version of the report with the BPA holder for final editing, formatting, and uploading to the DEC. C.7 TEAM COMPOSITION As noted in ADS 201.3.5.8, “evaluations must be conducted by individuals with appropriate training and experience, including but not limited to, evaluation specialists.” All team members should be familiar with the USAID ADS 201 operational policy on evaluation and the USAID Evaluation Toolkit. USAID encourages local evaluation specialists to lead or participate in evaluation teams. All team members will be required to provide a signed statement attesting that they have no conflict of interest or describing an existing Conflict of Interest (COI). The COI disclosure will be submitted with the quotations. C.8 SCHEDULE AND LOGISTICS The period of performance of this BPA call is from July 25, 2022 - December 09, 2022. The BPA holder will submit a detailed timeline including type of task, location, deliverable and tentative schedule. The BPA holder will be responsible for all the scheduling, logistics and security requirements related to the evaluation. C.9 EVALUATION MANAGEMENT The BPA holder will work closely with the COR on each task and will also frequently interact with USAID/Pakistan staff (through the COR) and relevant USAID/Pakistan’s IPs throughout the implementation of this call. Consequently, the BPA holder must work with the COR and the IPs to establish a coordinated system of communication and information sharing. The system should include several consultative sessions, both in person and/or virtually, with USAID staff throughout the period of performance of the call, as and when required by the COR, including upon award of the call to discuss the overall evaluation approach and other issues. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 67 ANNEX II: TIMELINE The evaluation, including conducting team planning meetings, data collection, analysis, and report writing and finalization will require approximately 19 weeks after the approval of the evaluation design. Key Activities Due Date Weeks November December January February March W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 Team Planning and Preparation 1 Team planning meeting and training of qualitative research team Nov￾1 1 Development of logistics management plan 0 Coordination with respondents and scheduling of data collection across all focused areas 2 Data Collection 6 Qualitative data collection in focused areas as per the approved data collection plan Nov￾14 Completion of interview notes/cleaning/tally sheet preparation Dec￾23 6 Submission of Data Completion Report to USAID Dec￾23 Data Analysis and Reporting 12 Data Analysis Debriefing with USAID on findings, 4 conclusions, and recommendations Jan￾19 Preparation of evaluation report Submission of draft evaluation report to USAID 3 for review Feb￾7 USAID's review and feedback Feb￾21 2 Incorporation of USAID's feedback and submission of final report to USAID Feb￾27 2 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 68 Key Activities Due Date Weeks November December January February March W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 USAID's approval on the final report Mar￾6 Submission of raw qualitative and quantitative data to USAID Mar￾9 1 Uploading raw data to Data Development Library Mar￾9 DEC review of evaluation report Editing, formatting, and uploading of report to DEC FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 69 ANNEX III: EVALUATION DESIGN DOCUMENT EVALUATION DESIGN FOR THE PERFORMANCE EVALUATION OF PAKISTAN AGRICULTURE PROGRAMS (PAP) SUBMITTED BY: ASSOCIATES IN DEVELOPMENT (AiD) PRIVATE LIMITED CALL ORDER NUMBER: 72039122F50005 SUBMISSION DATE: OCTOBER 27, 2022 This data collection plan is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Associates in Development (AiD) Pvt. Ltd. and do not necessarily reflect the views of USAID or the United States Government. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 70 ABBREVIATIONS AND ACRONYMS ARI Agriculture Research Institute ASP Agriculture Service Providers ASPTP Agriculture Service Providers Training Program ATI Agriculture Training Institute BARDC Balochistan Agricultural Research and Development Centre BARI Barani Agricultural Research Institute CABI Center for Agriculture and Biosciences International CDRI Crop Disease Research Institute CIMMYT International Maize and Wheat Improvement Center CPEP Cotton Productivity Enhancement Project EGA Economic Growth and Agriculture FSC&RD Federal Seed Certification and Registration Department FIDA Farmers' Integrated Development Association GOP Government of Pakistan ICARDA International Center for Agriculture Research for Dry Areas KIIs Key Informant Interviews NIBGE National Institute of Biotechnology and Genetic Engineering OECD-DAC Organization for Economic Cooperation and Development, Development Assistance Committee PAP Pakistan Agriculture Programs PARC PCCC Pakistan Agriculture Research Center Pakistan Cotton Central Committee PCRWR Pakistan Council of Research in Water Resources PPR Peste de Petits Ruminants PRMP Phytosanitary Risk Management Program RARI Regional Agricultural Research Institute SACAN South Asian Conservation Agriculture Network SARC Southern zone Agricultural Center SAWCRI Soil and Water Conservation Research Institute TEC Total Estimated Cost Ug99 Wheat Stem Rust Disease USAID United States Agency for International Development USDA United States Department of Agriculture USG United States Government VEC WPEP Variety Evaluation Committee Wheat Productivity Enhancement Project FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 71 WADO Women Agricultural Development Organization WRI Wheat Research Institute FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 72 BACKGROUND INFORMATION Table 1 Activity Summary Agreement Number: Contract # 391-USDA-13-0001 Activity Start Date: November 14, 2012 Activity End Date: September 30, 2021 Total Estimated Cost (TEC): $26,100,000 Location of Activities (Provinces/Districts): Nationwide Implementing Partner: United State Department of Agriculture (USDA) USAID/Pakistan Results Framework Linkages: DO 3: Increased Private Sector-led Inclusive Economic Growth IR 3.2: Employment opportunities increased. Sub IR 3.2.2: Agriculture value chains enhanced. Sub IR 3.2.2a: Number of individuals in the agriculture system who have applied improved management practices and technologies with USG assistance Purpose of Evaluation The primary purpose of the evaluation is to identify the achievements of selected components of the Pakistan Agriculture Program (PAP) in line with its intended objectives and inform management decisions on on-going and future EGA activities. It will also measure the contribution of the PAP components on the focus areas of the agriculture sector in Pakistan. The findings will be highly useful to guide the future planning, design, implementation, and other related decisions that are likely to improve performance of future similar investment efforts. The learnings from this evaluation will also be shared with relevant government departments to help inform future interventions in agriculture research and extension. The primary audience for the evaluation includes: 1) the USAID/Pakistan mission, particularly the EGA Team, 2) the USAID Bureau for Asia which covers Pakistan, (3) the implementing partner, USDA (including the sub-awardees), (4) and the Government of Pakistan (GOP). This evaluation will focus on four key sub-activities i.e., wheat, cotton, PRM and ASP because the PAP invested most of its resources on these interventions and these interventions were expected to produce more tangible results and create long term impacts on focus areas. Evaluation Questions The evaluation will address three key questions as identified in the SOW (section C of the contract). Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Explanation. These results include but are not limited to: FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 73 d) How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. e) As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why? The question should indicate if the adoption of technologies increased incomes of ASPs and farmers. f) What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project intervention which included Punjab, Sindh, Balochistan and Gilgit-Baltistan. Effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Key assumptions and definitions for evaluation design ● For the purposes of the evaluation, all relevant (included expected) outcomes and results have been included explicitly in evaluation sub-questions (a), (b) and (c).51 ● Adoption52 is defined as “a process involving an individual that includes the series of stages one undergoes from first hearing about a product to finally accepting or using it.” ● “Farmers” means beneficiary farmers who adopted at least one new technology introduced through ASPs. ● “Focused horticulture crops” refersto papaya (in Sindh), apple (in Balochistan) and fruit plants (in GB). Question 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Explanation This question will evaluate the likelihood that the interventions under the four identified sub-activities are sustainable i.e., the varieties, practices, technologies introduced will persist beyond the project support. Specific aspects will include institutionalizing these interventions in the Government (both federal and provincial) and private organizations. Key assumptions and definition for evaluation design ● The evaluation will focus on all interventions and outcomes/results of the four selected sub￾activities to evaluate the extent of sustainability and identify the contributing and hindering factors. 51 According to Organization for Economic Cooperation and Development, Development Assistance Committee (OECD-DAC) 2010, Glossary of Key Terms in Evaluation and Results Based Management, 2010, (https://www.oecd.org/ dac/evaluation/2754804.pdf), outcomes are “The likely or achieved short-term and medium-term effects of an intervention’s outputs” where an intervention is “An instrument for partner (donor and non-donor) support aimed to promote development. Examples are policy advice, projects, and programs.” Results are “The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention.” 52 Literature Review, Scaling Agricultural Technologies and Innovation Diffusion, May 7, 2015, https://pdf.usaid.gov/pdf_docs/pa00kfqg.pdf FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 74 Question 3: What are the lessons learned for future design and mechanism of similar projects? Explanation This question should investigate the major challenges and successes especially about the USDA and their sub-partners. Recommendations should focus on changes in the implementation approach that could improve performance in the new Participating Agency Program Agreement and future activities. Key assumptions and definition for evaluation design ● For the purposes of the evaluation, “mechanism” means Participating Agency Program Agreements53 (PAPA) and any aspect of implementation that emerges in a conclusion in answer to questions 1 and 2. EVALUATION METHODOLOGY This will be a performance evaluation and will rely on information drawn from both primary and secondary data. Primary data will be collected through non-probability sampling that will include Key Informant Interview (KII) and Group Interviews (GI). The evaluation will include a total of 88 KIIs which will be conducted with PAP key stakeholders including implementing partners’ staff, participating organizations’ staff, government officials and other key individuals associated with PAP. A total of nineteen (19) Group interviews (GI) will be conducted with beneficiary farmers of Agriculture Services Provider Training Program (ASPTP). In addition to the primary data, the evaluation will also collect data available from secondary sources including the sub-activity monitoring data, annual and quarterly reports, previous assessment and/or evaluation reports and reports available from government agencies (such as approvals and registrations). The details about KIIs and GIs are presented in the Data Collection Methods. Table 2 Summary of Data Collection Method by Sub-Activity Sub-Activity Data Collection Methods Key Informant Interview Group Interview WPEP 20 CPEP 16 PRMP 27 ASPTP 25 19 (57 individuals/ farmer) Total 88 57 Gender Integration The evaluation will ensure that evaluation design, methodology, data collection, analysis and report adequately capture the situations and experiences of both males and females. The evaluation will employ appropriate gender sensitive data collection methodology (detail provided in the following 53 PAPA is a participating Agency program Agreement under which USAID provides funds to another USG agency to execute its program which will also further USAID development objectives. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 75 paragraph and sections) which will inform the evaluation about the potential constraints in women participation in agricultural activities. For all the three key evaluation questions, the evaluation will explore whether these interventions provided equal opportunities for both men and women. If not, what could have been done to encourage women participation in the activity? If yes, what were the approaches that encouraged women to benefit from the activities? Data Collection Methods The evaluation will employ qualitative techniques to collect data from multiple sources to ensure multiple levels of triangulation. The data will be collected both from primary and secondary sources. The primary data will be collected through KIIs and Group Interviews, while the existing secondary sources are given below: Existing Data During the evaluation design phase, USAID and PAP implementing partners shared the following documents with the evaluation team. ● Annual and Quarterly reports (WPEP, CPEP, ASPTP, PRMP) ● Project completion report (PRMP) ● Previous evaluation and assessment reports (ASPTP, PRMP) ● Pakistan Agriculture Programs - Final Report, 2021 ● Pakistan Agriculture Programs Quarterly Reports (Quarter 4 - 2019, 2020, 2021) ● Pakistan Agriculture Programs work-plans (2015, 2016, 2017, 2018, 2019, 2020) ● Participating Agency Program Agreement, Modification Number: 03 ● Pakistan Agriculture Programs Monitoring, Evaluation and Learning, (ME&L) Plan Primary Data Key Informant Interviews KIIs with the key stakeholders are expected to provide overall background information and help contextualize the overall evaluation framework. In particular, KIIs will provide perspectives of key stakeholders on accomplishments, best practices and lessons learned, and other factors that may have inhibited or promoted the success of PAP. KIIs will be conducted with key stakeholders for each of the sub-activity included in this evaluation. In addition to including questions in all KIIs to understand the extent to which the sub-activities offered equal opportunities of participation and benefit to men and women, the evaluation team will make a conscious effort to select and interview female key informants, wherever applicable. A brief description of the sampling for each of the sub-activities is presented as follows: Wheat Productivity and Enhancement Project (WPEP) FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 76 The evaluation team will conduct KIIs with the key stakeholders of WPEP including the implementing partner, project partner research organizations, government approving bodies and seed producing companies. A total of 20 KIIs will be conducted from relevant stakeholders for WPEP interventions. The details regarding the proposed list of organizations including the number of KIIs are attached as Appendix II. Cotton Productivity and Enhancement Project (CPEP) The evaluation team will conduct KIIs with the key stakeholders of CPEP including the implementing partner, project partner research organizations, government approving bodies and seed producing companies. The Project Managers of the two partner NGOs engaged under CPEP to support farmers, including Farmers' Integrated Development Association (FIDA) and Women Agricultural Development Organization (WADO), through Farmer Field School (FFS) and Women Open Schools (WOS) are also included in the list of informants for KIIs. A total of 16 KIIs will be conducted from relevant stakeholders of CPEP interventions with the details attached as Appendix III. Phytosanitary Risk Management Program (PRMP) The evaluation team will conduct KIIs with the key stakeholders of PRMP including the implementing partner, program’s partner organizations, government officials and farmers (including female farmers, where applicable) who benefited from PRMP. Three farmers will be purposively selected for each of the four crops i.e., papaya, apple, fruit trees and willow in Sindh, Balochistan, Gilgit and Baltistan region, respectively. The evaluation team will conduct 27 Key Informant Interviews with PRMP stakeholders. The detailed table is attached as Appendix IV. Agriculture Service Providers Training Program (ASPTP) The evaluation team will conduct KIIs with the key stakeholders of ASPTP including the implementing partner, program partner organization(s) and Agriculture Service Providers (ASP) who received training through ASPTP on provision of ten types of services. ASPs will be purposively selected with representation from all the provinces and at least two participants for each type of service provided. A total of 25 KIIs are envisaged from ASPTP relevant stakeholders, including 19 ASPs. The list of proposed informants for ASPTP interventions is provided in Appendix V. A detailed table on the proposed sample by province and technology is attached as Appendix VI. Group Interviews54 In addition to KIIs with key stakeholders, the evaluation team will conduct group interviews with beneficiaries/farmers of ASPTP, who received services from ASPs on ten technologies introduced under the program. During the meeting with implementing partner ICARDA, it was revealed the 54 A group interview will include 2-4 individuals, on average. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 77 contact information for these beneficiaries was not available, and therefore any quantitative sample survey would not be possible. The only way to reach out to these beneficiaries would be through the ASPs. Therefore, a snowball/ convenience sampling technique is proposed to collect information from these beneficiaries. Since KIIs will be conducted with ASPs, the ASPs will be requested at the time of scheduling meeting to provide contact information of or gather at least three farmers who were provided services by them in the nearby location close to the ASPs home location or meeting spot. Where ASPs have directly assisted women farmers, they will be requested to ensure nomination of women beneficiary farmers in the group interview as well. During the group interview, all respondents will be asked the same set of questions. It is expected that a total of 19 group interviews with 57 individuals will be conducted in ten locations across four provinces. Appendix VII illustrates the proposed sample for beneficiary farmers for group interviews. Data Analysis The evaluation largely relies on qualitative data i.e., KIIs and group interviews with key program stakeholders and beneficiaries. The evaluation team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. When possible, the team will conduct comparative analysis across various groups of stakeholders to ensure triangulation of data sources. To the extent possible and where applicable, the evaluation will disaggregate findings by gender, and draw specific conclusions and recommendations. A detailed data analysis plan will be developed and shared with USAID for approval. Methodological Strength and Limitations The evaluation methodology relies on triangulation of sources and methods to ensure the validity and reliability of results. The scope of this evaluation will be limited to answering the evaluation questions to the extent that data is available. In cases where the program beneficiaries have received support several years ago or the project (PRMP) has ended at least three years ago, it may be challenging to locate the respondents. In case of ASPTP, it has already been communicated to the evaluation team that beneficiary contact information is not available, and the evaluation team will have to rely on convenience/ snowball sample for selection of beneficiary farmers through the ASPs. In case of selection of beneficiary farmers of PRMP, the evaluation team will have to rely on availability of beneficiary contact information to be shared by CABI. Since the country has suffered from heavy rains this year where millions of people have been displaced throughout the country due to floods, it may not be possible for the evaluation team to travel to flood affected locations. Another limitation for data collection could potentially be the non-availability of selected respondents due to displacement from home location due to floods. Because qualitative data collection for this evaluation relies on purposive sampling, meaning that respondents are selected for their roles, responsibilities, and knowledge, it will not produce statistically representative samples. As a result, although these respondents typically represent a FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 78 variety of views on a topic, they may not represent all relevant views. While the approach produces rich data, the results are not as generalizable as results from a representative sample would be. Due to the evaluation’s reliance on retrospective perceptions of respondents, especially in cases where the project has ended at least three years ago, the evaluation can potentially suffer from recall bias. Recall bias occurs when respondents do not accurately remember previous events or experiences. The specific limitations pertaining to each evaluation question and data collection method are presented in “Appendix 1 –Getting to Answers Matrix” along with measures to address these. The evaluation team will conduct data collection and analysis in a highly systematic manner by triangulating across multiple sources to ensure the reliability and validity of findings and conclusions. More precisely, the methodology allows for data triangulation i.e., data will be drawn from across stakeholder categories included in the assessment. EVALUATION DELIVERABLES The evaluation will produce the following deliverables. ▪ Evaluation Design: This document is the evaluation design. It includes detailed methodology, key questions, sampling techniques, detailed timeline, and staff composition for completion of the work requested in the SOW. USAID COR will review the design and will provide final approval for implementation. ▪ Data Collection Plan, Data Collection Instruments and Data Analysis Plan: The data collection plan, data collection instruments and data analysis plan will be submitted for COR’s approval before the start of field work. No data collection will be started without the approval of USAID/Pakistan. ▪ Data Collection Completion Report: At the conclusion of data collection, AiD will submit to the USAID COR a final data collection schedule indicating dates and location of data collection activities and persons or groups interviewed if relevant. ▪ Debriefing with USAID/Pakistan of Findings, Conclusions, and Recommendations: At or near the completion of data analysis and before finalizing the first draft of the report, the evaluation team will present the major findings, conclusions, and recommendations to USAID/Pakistan. The evaluation team will consider USAID/Pakistan comments for use in the draft report, as appropriate. ▪ Draft Evaluation Report: The draft evaluation report will answer the evaluation questions and will include findings, conclusions, and recommendations across the components/sub￾components. The draft evaluation report must meet the requirements described in ADS 201 mah, USAID Evaluation Reporting Requirements: o The Evaluation report should represent a thoughtful, well-researched, and well￾organized effort to objectively evaluate the strategy, project, or activity. o The Evaluation report should be readily understood and should identify key points clearly, distinctly, and succinctly. o The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 79 o The Evaluation report should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. o The Evaluation methodology should be explained in detail and sources of information properly identified. o The Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias and unobservable differences between comparator groups). o The Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. o The Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. o If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. o If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. The draft evaluation report will follow the evaluation report template attached to the contract (section J.5). At minimum, the evaluation report must: 1. Identify the evaluation as either an impact or performance evaluation per the definitions in ADS 201. 2. Include an abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the evaluation report cover. 3. Include an Executive Summary 2–5 pages in length that summarizes key points (purpose and background, evaluation questions, methods, findings, and conclusions). 4. State the purpose of, audience for, and anticipated use(s) of the evaluation. 5. Describe the specific strategy, project, activity, or intervention to be evaluated including (if available) award numbers, award dates, funding levels, and implementing partners. 6. Provide brief background information. This should include country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis, theory of change, or simply how the intervention addresses the problem. 7. State the evaluation questions. 8. Describe the evaluation method(s) for data collection and analysis. 9. Describe limitations of the evaluation methodology. 10. Include evaluation findings and conclusions. 11. If recommendations are included, separate them from findings and conclusions. 12. Address all evaluation questions in the Statement of Work (SOW) or document approval by USAID for not addressing an evaluation question. 13. Include the following annexes: i. Evaluation SOW. If the SOW was revised over the course of the evaluation, the evaluation report should include the updated SOW as an Annex rather than the original SOW. The Contracting Officer’s Representative of the evaluation must agree upon, in FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 80 writing, all modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline. ii. A description of evaluation methods (if not described in full in the main body of the evaluation report). iii. All data collection and analysis tools used, such as questionnaires, checklists, survey instruments, and discussion guides. iv. All sources of information—properly identified and listed. v. Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. vi. Signed disclosures of conflicts of interest from evaluation team members. vii. Summary information about evaluation team members, including qualifications, experience, and role on the team. 14. Include enough information on the cover of the evaluation report so that a reader can immediately understand that it is an evaluation and what was evaluated. The evaluation cover must: i. Include a title block in USAID light blue background color. ii. Include the word “Evaluation” at the top of the title block and center the report title underneath that. The title should also include the word “evaluation.” iii. Include the following statement across the bottom of the cover page: “This publication was produced at the request of the United States Agency for International Development. It was prepared independently by [list authors and organizations involved in the preparation of the report].” iv. Feature one high-quality photograph representative of the project being evaluated and include a brief caption on the inside front cover describing the image with photographer credit. v. State the month and year of report publication (e.g., when final and approved by USAID Pakistan). vi. State the individual authors of the report and identify the evaluation team leader. vii. As noted in ADS 201.3.5.17, evaluation reports must also conform to USAID branding requirements (see ADS 320, Branding and Marking) and comply with section 508 of the Rehabilitation Act (see ADS 302mak, USAID Implementation of Section 508 of the Rehabilitation Act of 1973). The draft report will not exceed 30 pages, excluding annexes. The draft report will be submitted to the COR for USAID/Pakistan review and comments. USAID will share feedback on the report within 3- 4 weeks. ▪ Final Evaluation Report: The final report will address all USAID/Pakistan comments. AiD will finalize the report and submit it to the COR for approval within two to three weeks of receiving comments from USAID/Pakistan’s COR. ▪ Raw Data: Per ADS 579 - USAID Development Data – all quantitative data collected for this evaluation will be submitted to USAID/Pakistan in electronic format within 30 days of Data collection completion. AiD will submit the data in a format requested by USAID/Pakistan for internal archiving and in formats specified in ADS 579 for uploading to the Data Development Library (DDL). Qualitative data will be delivered as 1) the coded segments used in analysis FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 81 extracted from MAXQDA in an excel format or 2) tally sheets, as applicable to the analysis. The Evaluation Team will be responsible for submitting the data to DDL. ▪ Development Experience Clearinghouse (DEC) Review: Prior to public dissemination, evaluation reports will be adjusted to remove information that falls under one of the “principled exceptions to the presumption in favor of openness” as described in ADS 201mae, Limitations to Disclosure and Exemptions to Public Dissemination of USAID Evaluation Reports. Accordingly, once the report is finalized, USAID/Pakistan may conduct a DEC review of the report. The COR will share the DEC version of the report with the evaluation team for final editing, formatting, and uploading to the DEC. TEAM COMPOSITION AND TASK ASSIGNMENT Evaluation Team Composition The key qualifications of the key evaluation team members are provided in Appendix VIII. Task Assignment of the Evaluation Team Position Name of Evaluation Team Member Task Assignment Evaluation Specialist/ Team Leader (01) Mr. Ghazanfar Hoti The Evaluation Specialist/ Team Leader is responsible for the overall technical management and delivery of the evaluation assignment. Specifically, he will perform the following tasks: ▪ Lead the preliminary meetings with USAID team, IPs and other relevant stakeholders and literature review to prepare an informed evaluation design. ▪ Work closely with the USAID team to understand and incorporate the feedback to finalize the design. ▪ Lead the preparation of data collection instruments with the support of sector FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 82 Position Name of Evaluation Team Member Task Assignment experts ▪ Lead selection of sample of informants and prepare the fieldwork plan. ▪ Prepare agenda and lead organization of the team planning meeting. ▪ Prepare and lead implementation of data transmission and storage plan which ensures data privacy and security ▪ Lead preparation of data analysis plan ▪ Lead qualitative interviews with key informants as per the fieldwork plan ▪ Participate in qualitative interviews conducted by the Qualitative Research Specialists for quality assurance and provide timely feedback for needed changes/ improvements ▪ Perform review and quality assurance of interview notes/ summary sheets prepared by the other qualitative research team members ▪ Lead analysis of evaluation data (both primary and secondary) with support of sector experts and Qualitative Research Specialists and ensure findings against all evaluation questions and sub-questions are backed by evidence ▪ Prepare and give debriefing presentation to USAID team. ▪ Lead the drafting of evaluation report with support from evaluation team members. Ensure that the draft evaluation report complies with USAID’s Evaluation Reporting Requirements provided in ADS 201mah; Branding and Marking requirements provided in ADS 201.3.5.17 and ADS 320; and Section 508 of the Rehabilitation Act as per the ADS 302mak ▪ Work closely with the USAID team to understand and incorporate feedback to finalize the report ▪ Provide guidance and technical support to all evaluation team members at all stages of evaluation ▪ Any other task relevant to the role of Team Leader/ Evaluation Specialist Agriculture Economist & Evaluation Specialist (01) Dr. Tariq Husain ▪ Participate in the preliminary meetings with USAID, PAP IPs, and other relevant stakeholders. ▪ Provide technical guidance and advice for preparation of informed and responsive evaluation design and data collection instruments. ▪ Participate in the team planning meeting to orient/ train the data collection teams on technical aspects as well as evaluation data collection ethics. ▪ Perform quality assurance of data collection process by reviewing the notes and providing timely feedback to the relevant evaluation team members. ▪ Participate in KIIs with PAP IPs and other relevant stakeholders, as assigned by the Team Leader ▪ Assist the Team Leader in analysis of primary and secondary data to prepare major findings, conclusions, and recommendations ▪ Participate in the debriefing meeting to USAID and assist the Team Leader in responding to questions, queries and/ or clarification requests by the USAID team ▪ Assist the Team Leader in preparing the draft evaluation report by drafting sections relevant to Agriculture Economy FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 83 Position Name of Evaluation Team Member Task Assignment ▪ Perform quality assurance of the draft evaluation report with the objective of ensuring technical efficacy of contents of the report, and its compliance with USAID’s evaluation reporting requirements. ▪ Assist the Team Leader in incorporating USAID team’s feedback on the draft evaluation report. ▪ Any other relevant task assigned by the Team Leader Qualitative Research Specialists (02) Ms. Fatima Abbas Mr. Aftab Ahmad ▪ Perform a detailed review of project documents and secondary literature relevant to the sub-activity assigned by the Team Leader ▪ Participate in the team planning meeting to develop an in-depth understanding of evaluation data collection and analysis requirements, processes, and expectations. ▪ Lead collection of primary data by conducting KIIs with stakeholders across all provinces/ territories ▪ Document the interviews by preparing interview summary sheets/ notes. Perform quality assurance of the notes prepared by the Note Takers, where applicable. The responsibility of preparing comprehensive and high-quality notes lies with the Qualitative Researcher ▪ Ensure compliance to evaluation data privacy and security plan. ▪ Assist the Team Leader and sector experts in analysis of primary and secondary data to prepare for the debriefing presentation. The Qualitative Research Specialists will also participate in the debriefing meeting. ▪ Assist the Team Leader in presenting the evidence for the evaluation findings in a concise manner (tabular and graphic presentation in particular) ▪ Assist the Team Leader in responding to USAID team’s feedback and comments on the draft evaluation report. ▪ Prepare the raw data files as per the formats specified in ADS 579 and upload to the Data Development Library (DDL) ▪ Any other relevant task assigned by the Team Leader Seed Certification Expert (01) Mr. Ilham￾ud-Din ▪ Perform review of secondary literature and project documents to develop a sound understanding of WPEP and CPEP ▪ Assist the Team Leader in identification of relevant stakeholders for primary data collection to help inform the evaluation design. ▪ Assist the Team Leader in preparing and finalizing the evaluation data collection instruments. ▪ Participate in the team planning meeting and orient the qualitative research team on technical terminologies and concepts related to CPEP and WPEP evaluation questions. ▪ Accompany the evaluation team during primary data collection and assist them in conducting KIIs with key stakeholders ▪ Perform a review of the data collection notes prepared by the qualitative research team to help ensure that the discussion is properly documented. ▪ Assist the Team Leader in data analysis and identification of major findings and FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 84 Position Name of Evaluation Team Member Task Assignment lessons learned ▪ Assist the Team Leader in devising recommendations. ▪ Participate in the debriefing meeting and assist the Team Leader in presentation of key findings, lessons learned and recommendations ▪ Perform a review of the draft evaluation report to ensure that the findings are presented with adequate evidence and in a technically sound manner ▪ Assist the Team Leader in responding to and/ or incorporating USAID’s feedback on the draft evaluation report ▪ Participate in meetings with USAID, IPs, or other relevant stakeholders ▪ Any other relevant task assigned by the Team Leader Entomologist/ Plant Pests Control Expert (01) Dr. Farman Ullah ▪ Perform review of secondary literature and project documents to develop a sound understanding of PRMP. ▪ Assist the Team Leader in identification of relevant stakeholders for primary data collection to help inform the evaluation design. ▪ Assist the Team Leader in preparing and finalizing the evaluation data collection instruments. ▪ Participate in the team planning meeting and orient the qualitative research team on technical terminologies and concepts related to PRMP evaluation questions. ▪ Accompany the evaluation team during primary data collection and assist them in conducting KIIs with key stakeholders. ▪ Perform a review of the data collection notes prepared by the qualitative research team to help ensure that the discussion is properly documented. ▪ Assist the Team Leader in data analysis and identification of major findings and lessons learned. ▪ Assist the Team Leader in devising recommendations. ▪ Participate in the debriefing meeting and assist the Team Leader in presentation of key findings, lessons learned and recommendations ▪ Perform a review of the draft evaluation report to ensure that the findings are presented with adequate evidence and in a technically sound manner ▪ Assist the Team Leader in responding to and/ or incorporating USAID’s feedback on the draft evaluation report ▪ Participate in meetings with USAID, IPs, or other relevant stakeholders ▪ Any other relevant task assigned by the Team Leader Field Researchers (04) To be selected ▪ Participate in the training to understand the evaluation purpose, terminologies, concepts, and interview techniques. ▪ Conduct primary interviews i.e., KIIs with ASPs and Group Interviews with beneficiary farmers. ▪ Document the discussion in the form of summary notes. The notes must provide a summary of responses to each evaluation question. ▪ Any other relevant task assigned by the Team Leader FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 85 Position Name of Evaluation Team Member Task Assignment Note Taker (03) To be selected ▪ Accompany the evaluation team members during interviews/ primary data collection and take notes of the discussion. ▪ Prepare and submit the interview notes to the leading team member in soft copy form FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 86 EVALUATION MANAGEMENT Anticipated Schedule The complete process, including finalization of the evaluation design, review of documents, developing data collection instruments, data collection, analysis, and report writing, and finalization will require approximately 23 weeks, including approximately 10 weeks after the approval of the evaluation design. Key Phases and Tasks Date / Dead line Months July & August September October November December January Weeks 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 Evaluation Design 12 Preliminary meeting with USAID COR & EGA team 2 Meetings with implementing partners of four selected sub-activities 4 Review of secondary data/ literature on PAP and the four selected sub-activities 5 Preparation of evaluation design including data collection plan, Getting to Answers matrix and data analysis plan 2 Submission of evaluation design including data collection plan to USAID for review 07- Sep USAID's review and feedback 09- Sep 0 Preparation of data collection instruments 2 Submission of revised evaluation design, including Data Collection Plan, Data Analysis Plan and Data Collection Instruments to USAID 22- Sep USAID's review and feedback 03- Oct 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 87 Key Phases and Tasks Date / Dead line Months July & August September October November December January Weeks 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 Submission of Final Evaluation Design including Data Collection Plan, Data Analysis Plan and Data Collection Instruments to USAID 14- Oct 2 USAID's approval on the evaluation design 18- Oct 1 Team Planning & Preparation 1 Team planning meetings and training of qualitative research team 1 Development of logistics management plan 0 Coordination with respondents and scheduling of data collection across all focused areas 23- Oct 2 Data Collection 4 Qualitative data collection in focused areas as per the approved data collection plan 20- Nov 4 Completion of interview notes/ cleaning/ summary sheets preparation 24- Nov Data Analysis & Reporting 8 Data Analysis Debriefing with USAID on findings, conclusions, 2 and recommendations 02- Dec Preparation of evaluation report Submission of draft evaluation report to USAID 3 for review 23- Dec USAID's review and feedback 30- Dec 1 Incorporation of USAID's feedback and submission of final report to USAID 06- Jan 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 88 Key Phases and Tasks Date / Dead line Months July & August September October November December January Weeks 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 USAID's approval on the final report 13- Jan Submission of raw qualitative and quantitative data to USAID 20- Jan 1 Uploading raw data to Data Development Library 20- Jan DEC review of evaluation report* Editing, formatting, and uploading of report to DEC FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 89 APPENDICES Appendix I: Initial Getting to Answers (G2A) Matrix Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? ● Implementing partners’ staff ● Partner organizations staff ● Other stakeholders e.g., government officials, private sector business community etc. ● Program beneficiaries ● Secondary data e.g., program reports, registration/certificat ion documents etc. ● Key informant interviews ● Document/ Literature Review Qualitative Data For qualitative data (key informant interviews and group interviews) the evaluation team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. ● Unavailability of key stakeholders for interviews ● Unavailability of documents or data ● To address the risk of non￾availability of key stakeholders, the evaluation team will schedule prior appointments with individuals to be included through phone and/ or email, to the extent possible with assistance from the Implementing Partners. ● In addition to meeting the key informants face to face, the evaluation team will conduct KIIs via telephone (WhatsApp, Google Meet, MS Teams and/ or a) How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. (Wheat) Implementing Partner ● CIMMYT Partner Organization ● Pakistan Agriculture Research Council (PARC) ● Wheat Research Institute (WRI), Faisalabad ● Regional Agricultural Research Institute (RARI), Bahawalpur, Punjab Key Informant Interviews ● Implementing partner staff ● Partner organization staff ● Key officials/mem ber of the government approving authorities. ● Unavailability of key stakeholders for interviews ● Unavailability of documents or data FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 90 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● Barani Agricultural Research Institute (BARI), Chakwal, Punjab ● Wheat Research Centre, Sakrand, Sindh ● Nuclear Institute for Agriculture, Tandojam, Sindh ● Cereal Crops Research Institute, Pirsabak, KP ● Arid Zone Research Center - Balochistan Agricultural Research and Development Centre (BARDC), Quetta ● Crop Disease Research Institute (CDRI) Murree ● National Agricultural Research Center, Islamabad Government Approving Agencies ● Federal Seed Certification and Registration Department (FSC&RD) ● PARC-VEC Document/ Literature Review Zoom) where appropriate and feasible to ensure maximum access to respondents. ● To address the risk of identifying the most appropriate candidate, the evaluation team has requested the implementing partners to provide a list of key stakeholders who have been involved in the implementation of PAP sub￾activities. The stakeholders/ individuals will be chosen from those lists to ensure the individuals most familiar with sub￾activities are interviewed. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 91 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● Provincial Seed Council Secondary Data ● WPEP Annual and Quarterly Reports ● PAP Annual and Quarterly Reports ● Meeting minutes of Provincial Seed Councils (if available) ● Certification/registra tion/approval documents. ● In case certain individuals are not available for interviews, alternate respondents will be selected. ● To ensure triangulation, the evaluation team will collect data in a highly systematic manner from multiple sources to ensure the reliability and validity of findings and conclusions. ● To mitigate the risk of accessing ASPs, the ASPs will be contacted via phone prior to interviews to ensure their availability. a) How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. (Cotton) Implementing Partner ● ICARDA Partner Organizations ● Central Cotton Research Institute, Multan. ● Cotton Research Institute, (CRI) Multan ● CRI Research Stations at Faisalabad and Vehari ● National Institute of Biotechnology & Genetic Engineering (NIBGE), Faisalabad ● Agronomy Research Station Bahawalpur Key informant interviews ● Implementi ng Partner Staff ● Partner Organizatio ns Staff ● Key officials/me mber of the government approving authorities. Document/Literatur e Review ● Unavailability of key stakeholders for interviews ● Unavailability of documents or data FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 92 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed Government Approving Authorities ● Federal Seed Certification and Registration Department (FSC&RD) ● Pakistan Cotton Central Committee ● Provincial Seed Council Secondary Data ● CPEP Annual and Quarterly Reports ● PAP Annual and Quarterly Reports ● Meeting minutes (if available) of approving authority ● Certification/registra tion/approval documents. b) As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why? The question should indicate if the adoption of technologies increased Implementing Partner ● ICARDA Partner Organizations ● Agriculture Research Institute (ARI), Quetta, Balochistan province ● Agriculture Extension Department, Key informant interviews ● Implementing Partner Staff ● Partner Organizations Staff ● Agriculture sector providers ● Unavailability of key stakeholders for interviews ● Travel restriction due to floods ● Unavailability of farmer contact information required for survey. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 93 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed incomes of ASPs and farmers? Hyderabad, Sindh province ● Soil and Water Conservation Research Institute (SAWCRI), Chakwal, Punjab province, rain-fed system ● South Asian Conservation Agriculture Network (SACAN), Lahore, Punjab province, irrigated system ● Pakistan Council of Research in Water Resources (PCRWR), Peshawar office Program Beneficiaries ● Agriculture service providers trained through ASP. ● Beneficiary farmers Secondary Data ● ASPTP Annual and Quarterly Reports ● ASPTP monitoring data. ● PAP Annual and Quarterly Reports trained by ASPTP. Group Interviews ● Beneficiary farmers Document/Literatur e Review ● Unavailability of farmers for group interview ● Unavailability of ASPs for interview FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 94 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed c) What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project interventions which included Punjab, Sindh, Balochistan and Gilgit Baltistan. The effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Implementing Partner ● CABI Key Stakeholders by Crop Papaya (Sindh) ● Southern Zone Agricultural Research Centre (SARC) ● Department of Agriculture Extension Sindh. Apple (Balochistan) ● Department of Agriculture, Balochistan Willow and Fruit Trees (GB) ● Department of Agriculture Research￾Integrated Pest Management (IPM) ● Department of Agriculture Extension Rice (Punjab) ● Rice Exporters Association of Pakistan Beneficiaries ● Training participants (Rice, horticulture, papaya, apple, timber, and fruit trees) ● Growers to whom Parasitoid were Key informant interviews ● Implementing Partner Staff ● Key stakeholder for selected crops ● Growers to whom Parasitoid were released. ● Training participants (Rice, horticulture, papaya, apple, timber, and fruit trees) Document/Literatur e Review ● Unavailability of key stakeholders for interviews ● Travel restriction due to floods ● Unavailability of farmers for interviews FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 95 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed released (Papaya, apple, timber, and fruit trees) Secondary Data ● PRMP Annual and Quarterly Reports ● PRMP monitoring data. ● PRMP previous assessments and evaluation reports ● PAP Annual and Quarterly Reports Question 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? WPEP Implementing Partner Staff ● ICARDA Partner Organizations Staff ● Central Cotton Research Institute, Multan. ● Cotton Research Institute, (CRI) Multan ● CRI Research Stations at Faisalabad and Vehari ● NIBGE, Faisalabad ● Agronomy Research Station Bahawalpur Private Sector/Other Stakeholders Key Informant Interviews ● Implementing partner staff ● Partner organization staff ● Key officials/mem ber of the government approving authorities. ● Other Stakeholders￾including executives of seed companies in Pakistan. Qualitative Data For qualitative data (key informant interviews and group interviews) the evaluation team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. ● Unavailability of key stakeholders for interviews FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 96 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● Punjab Seed Corporation ● Seed Companies CPEP Implementing Partner ● CABI Partner Organizations ● Central Cotton Research Institute, Multan. ● Cotton Research Institute, (CRI) Multan ● CRI Research Stations at Faisalabad and Vehari ● NIBGE, Faisalabad ● Agronomy Research Station Bahawalpur Private Sector/Other Stakeholders ● Punjab Seed Corporation ● Seed Companies Partner NGOs ● Women Agricultural Development Organization (WADO) ● Farmers' Integrated Development Association (FIDA) Key Informant Interviews ● Implementing partner staff ● Partner organization staff ● Key officials/mem ber of the government approving authorities. ● Other Stakeholders￾including executives of seed companies in Pakistan. ● Unavailability of key stakeholders for interviews FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 97 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ASPTP Program Beneficiaries ● Agriculture service providers trained through ASPTP. ● Beneficiary farmers Key informant interviews/ ● Agriculture sector providers trained by ASPTP. Group Interviews ● Beneficiary farmers who received services from ASPs ● Unavailability of key stakeholders for interviews ● Travel restriction due to floods ● Unavailability of farmer contact information required for survey. ● Unavailability of farmers for FGDs ● Unavailability of ASPs for group interview PRMP Implementing Partner ● CABI Key Stakeholders by Crop Papaya (Sindh) ● Southern Zone Agricultural Research Centre (SARC) ● Department of Agriculture Extension Sindh. Apple (Balochistan) ● Department of Agriculture, Balochistan Willow and Fruit Trees (GB) ● Department of Agriculture Research￾Key informant interviews ● Implementing Partner Staff ● Key stakeholder for selected crops ● Growers to whom Parasitoid were released. ● Training participants (Rice, horticulture, papaya, apple, FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 98 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed Integrated Pest Management (IPM) ● Department of Agriculture Extension Rice (Punjab) ● Rice Exporters Association of Pakistan Beneficiaries ● Training participants (Rice, horticulture, papaya, apple, timber, and fruit trees) ● Growers to whom Parasitoid were released (Papaya, apple, timber, and fruit trees) Secondary Data ● PRMP Annual and Quarterly Reports ● PRMP monitoring data. ● PRMP previous assessments and evaluation reports ● PAP Annual and Quarterly Reports timber, and fruit trees) Document/Literatur e Review Question 3: What are the lessons learned for future design and mechanism of similar projects? WPEP Implementing Partner Staff ● ICARDA Partner Organizations Staff Key Informant Interviews/Group Interviews Qualitative Data For qualitative data (key informant interviews and ● Unavailability of key stakeholders for interviews FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 99 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● Central Cotton Research Institute, Multan. ● Cotton Research Institute, (CRI) Multan ● CRI Research Stations at Faisalabad and Vehari ● NIBGE, Faisalabad ● Agronomy Research Station Bahawalpur Government Approving Authorities ● Federal Seed Certification and Registration Department (FSC&RD) ● Provincial Seed Council (Punjab) ● Implementing partner staff ● Partner organization staff ● Key officials/mem ber of the government approving authorities. ● Other Stakeholders￾including executives of seed companies in Pakistan. group interviews) the evaluation team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. CPEP Implementing Partner ● ICARDA Partner Organizations ● Central Cotton Research Institute, Multan. ● Cotton Research Institute, (CRI) Multan Key Informant Interviews ● Implementing partner staff ● Partner organization staff ● Key officials/mem ber of the government ● Unavailability of key stakeholders for interviews FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 100 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● CRI Research Stations at Faisalabad and Vehari ● NIBGE, Faisalabad ● Agronomy Research Station Bahawalpur Government Approving Agencies ● Federal Seed Certification and Registration Department (FSC&RD) ● Provincial Seed Council (Punjab) approving authorities. ● Other Stakeholders￾including executives of seed companies in Pakistan. ASPTP Partner Organization ● Agriculture Research Institute (ARI), Quetta, Balochistan province ● Agriculture Extension Department, Hyderabad, Sindh province ● SAWCRI, Chakwal, Punjab province, rain-fed system ● SACAN, Lahore, Punjab province, irrigated system Key informant interviews ● Implementing Partner Staff ● Partner Organizations Staff ● Agriculture sector providers trained by ASPTP. Group Interviews ● Beneficiary farmers ● Unavailability of key stakeholders for interviews ● Travel restriction due to floods ● Unavailability of farmer contact information required for survey. ● Unavailability of farmers for FGDs ● Unavailability of ASPs for group interview FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 101 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● Pakistan Council of Research in Water Resources (PCRWR), Peshawar office Program Beneficiaries ● Agriculture service providers trained through ASPTP. ● Beneficiary farmers Implementing Partner ● CABI Key Stakeholders by Crop Papaya (Sindh) ● Southern Zone Agricultural Research Centre (SARC) ● Department of Agriculture Extension Sindh. Apple (Balochistan) ● Department of Agriculture, Balochistan Willow and Fruit Trees (GB) ● Department of Agriculture Research￾Integrated Pest Management (IPM) ● Department of Agriculture Extension Rice (Punjab) Key Informant Interviews ● Implementing partner staff ● Stakeholders from research institutes ● Provincial agriculture department officials ● Exporters/bus inessmen ● Growers to whom Parasitoid were released (Papaya, apple, timber, and fruit trees) ● Training participants (Rice, ● Unavailability of key stakeholders for interviews ● Travel restriction due to floods ● Unavailability of farmers for FGDs ● Unavailability of ASPs for group interview FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 102 Evaluation Question Data Source Data Collection Method Method of Analysis Limitations/risks How Limitations/Risks will be Addressed ● Rice Exporters Association of Pakistan Beneficiaries ● Training participants (Rice, horticulture, papaya, apple, timber, and fruit trees) ● Growers to whom Parasitoid were released (Papaya, apple, timber, and fruit trees) horticulture, papaya, apple, timber, and fruit trees) FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 103 Appendix II: Proposed KII Sample for WPEP relevant Stakeholders S. No. Name of Organization Position of Interviewee Role Number 1 International Maize and Wheat Improvement Center (CIMMYT) Country representative The implementing partner and partner organizations are expected to answer questions related to the effectiveness of WPEP interventions as mentioned in Evaluation Question No. 1. They are also expected to answer questions related to overall sustainability of the project’s interventions (or outcomes) and any lesson learned. 1 2 Pakistan Agricultural Research Council (PARC) National Wheat Coordinator 1 3 Wheat Research Institute (WRI), Faisalabad Director 1 4 Regional Agricultural Research Institute (RARI), Bahawalpur, Punjab Director 1 5 Barani Agricultural Research Institute (BARI), Chakwal, Punjab Director 1 6 Wheat Research Centre, Sakrand, Sindh Executive Director 1 7 Nuclear Institute for Agriculture, Tandojam, Sindh Wheat Leader 1 8 Cereal Crops Research Institute, Pirsabak, KP Director 1 9 Arid Zone Research Center - Balochistan Agricultural Research and Development Centre (BARDC), Quetta Director 1 10 Crop Disease Research Institute (CDRI) Murree Principal Scientific Officer 1 11 Wheat Program, National Agricultural Research Center, Islamabad Program leader 1 12 Provincial Seed Council of KP, Punjab and Sindh, Director General Research The PSC members are expected to answer questions related to approval of seed varieties, submitted by WPEP partners. 3 13 Federal Seed Certification and Registration Department (FSC&RD) Director General DG FSC&RD is expected to answer questions related to registration of seed varieties submitted by WPEP partners at FSC&RD. 1 13 PARC – Variety Evaluation Committee (VEC) Member Crop Science The member crop science is expected to answer questions related to valuation of seed varieties 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 104 submitted by WPEP partners at VEC. 14 Punjab Seed Corporation Production Manager These stakeholders are expected to answer questions related to sustainability of seed varieties in terms of commercial production. 1 15 Seed producing companies Managing Director/CEO 3 Total 20 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 105 Appendix III: Proposed KII Sample for CPEP relevant Stakeholders S. No. Name of Organization Position of Interviewee Role Number 1 Implementing Partner International Center for Agricultural Research in the Dry Areas (ICARDA) Country representativ e The implementing partner and partner organizations are expected to answer questions related to the effectiveness of CPEP interventions as mentioned in Evaluation Question No. 1. They are also expected to answer questions related to overall sustainability of the project’s interventions (or outcomes) and any lesson learned. 1 2 Central Cotton Research Institute, Multan Director 1 3 Cotton Research Institute, (CRI) Multan Director 1 4 CRI Research Stations at Faisalabad and Vehari Cotton Botanist 1 5 NIBGE, Faisalabad Deputy Chief Scientist 1 6 Agronomy Research Station Bahawalpur Agronomist 1 7 Federal Seed Certification and Registration Department (FSC&RD) Director General DG FSC&RD is expected to answer question related to registration of seed varieties submitted by CPEP partners at FSC&RD 1 8 Pakistan Cotton Central Committee (PCCC) Director The Director PCCC is expected to answer questions related to valuation of seed varieties submitted by CPEP partners at PCCC. 1 9 Provincial Seed Council Director General Research The PSC members are expected to answer questions related to approval of seed varieties, submitted by CPEP partners. 2 9 Punjab Seed Corporation Manager Production These stakeholders are expected to answer questions related to sustainability of seed varieties in terms of commercial production. 1 10 Seed producing companies Managing Director/CEO 3 11 Partner Non-governmental organizations Project Managers The partner NGOs are expected to answer the questions pertaining to sustainability of Farmer Field School, Women Open Schools, and Children Ecology Club. 2 Total 16 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 106 Appendix IV: Proposed KII Sample for PRMP relevant Stakeholders S. No. Name of Organization Position of Interviewee Role Numbe r 1 Implementing Partner - CABI Program Manager The implementing partner and partner organizations are expected to answer questions related to the effectiveness of PRMP interventions as mentioned in Evaluation Question No. 1. They are also expected to answer questions related to overall sustainability of the project’s interventions (or outcomes) and any lesson learned. 1 2 Southern Zone Agricultural Research Centre (SARC), Karachi Director 1 3 Department of Agriculture Extension Sindh Deputy Director 1 4 Directorate of Agriculture Research, Quetta, Balochistan Director Plant Protection 1 5 Department of Agriculture Research, Gilgit, GB Deputy Director, Integrated Pest Management (IPM) 1 6 Department of Agriculture Extension, Skardu, GB Agriculture Officer, IPM 1 7 Department of Agriculture, Punjab Deputy Director 1 8 Rice Exporters Association of Pakistan Rice Exporters These are project beneficiaries and are expected to answer questions related to PRMP’s effectiveness in terms of mitigating impact of pests. They are also expected to answer questions related to sustainability of the results that may have accrued because of PRMP interventions. 3 9 Key farmers who received support from PRMP (papaya, apple, willow, fruit trees) Farmers 12 10 GoP officers trained by PRPM Various GoP officers who received training through PRMP are expected to answer questions related to PRMP’s effectiveness in terms of mitigating the impact of pests in their area. They are also expected to answer questions related to institutionalization /sustainability of PRPM’s interventions (or outcomes) in their respective departments. 5 Total 27 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 107 Appendix V: Proposed KII Sample for ASPTP relevant Stakeholders S. No. Name of Organization Position of Interviewee Role Number 1 Implementing Partner - ICARDA Country representative The implementing partner and partner organizations are expected to answer questions related to the effectiveness of PRMP interventions as mentioned in Evaluation Question No. 1. They are also expected to answer questions related to overall sustainability of the project’s interventions (or outcomes) and any lesson learned. 1 2 Agriculture Research Institute (ARI), Quetta, Balochistan province Deputy Director 1 3 Agriculture Extension, Sakrand, Sindh Principal, Agriculture Training Institute (ATI) 1 4 Soil and Water Conservation Research Institute (SAWCRI), Chakwal, Punjab province, rain-fed system Deputy Director 1 5 South Asian Conservation Agriculture Network (SACAN), Lahore, Punjab province, Irrigated System Chief Executive 1 6 Pakistan Council of Research in Water Resources (PCRWR), Islamabad Chairman 1 7 Agriculture service providers trained by ASPTP ASPs The ASPs are expected to answer questions related to changes in income of ASPs and that of the farmers to whom they provided services. They are also expected to answer questions related to their occupation as ASPs after ASPTP ended. 19 Total 25 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 108 Appendix VI: Breakdown of ASPs by Technology and Province S. No. Technology/ Training Number of ASPs Trained Balochista Role n KP Punjab Sindh 1 Drip/ Bubbler/ Sprinkler Irrigation System 2 - - - The beneficiary farmers are expected to answer questions related to changes in their income and adoption of new technologies introduced through ASPs. They are also expected to answer questions related to their continued use of new technology, after ASPTP ended. 2 Laser Land Leveling - 2 - - 3 Ridge Planting of Crops - - - 2 4 Bed Planting of Crops - - 2 - 5 Gypsum to Conserve Soil Moisture - - 2 - 6 Biozote to Increase Yield of Crops - - 2 - 7 Fertilizer Prediction Model - - 2 - 8 Rapid Soil Testing Kit - - 2 - 9 Zero-Till (ZT) Drill 2 - - - 10 Happy Seeder ZT Drill - - 1 Total 4 2 11 2 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 109 Appendix VII: Proposed Group interviews (GI) Sample for Farmer Beneficiaries S. No. Technology/ Service Number of Farmers who received services from trained ASPs Total Balochistan KP Punjab Sindh 1 Drip/ Bubbler/ Sprinkler Irrigation System 6 - - - 6 2 Laser Land Leveling - 6 - - 6 3 Ridge Planting of Crops - - - 6 6 4 Bed Planting of Crops - - 6 - 6 5 Gypsum to Conserve Soil Moisture - - 6 - 6 6 Biozote to Increase Yield of Crops - - 6 - 6 7 Fertilizer Prediction Model - - 6 - 6 8 Rapid Soil Testing Kit - - 6 - 6 9 Zero-Till (ZT) Drill 6 - - - 6 10 Happy Seeder ZT Drill - - 3 - 3 Total 12 6 33 6 57 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 110 Appendix VIII: Skills of Key Evaluation Team Members Team Leader/ Evaluation Specialist Ghazanfar Hoti - M.S. (Operations Research), 2011, and MPA (Economic Policy Management), 2008, Columbia University New York, USA: Mr. Ghazanfar Hoti is an experienced monitoring and evaluation (M&E) specialist with more than 20 years of professional experience. Most of his work experience revolves around designing, managing and leading evaluations and assessments in Pakistan. He has authored/ co-authored several international-quality evaluation reports, including those of projects funded by the bilateral aid agencies of the United States, Switzerland, and the United Kingdom. His sectoral and thematic experience includes agriculture development, economic growth, community mobilization, countering violent extremism (CVE), education, energy, gender and women’s empowerment, governance, stabilization, and health. Agriculture Economist & Evaluation Specialist Dr. Tariq Husain - Ph.D. (Economics), 1987 (dissertation in Agricultural Economics), University of Chicago: Dr. Tariq Husain has worked in development for 40 years, including 30 years as a consultant, in 21 countries. Most of his experience has revolved around programme design and strategy, evaluating operations and results, and helping institutions adapt to change. Dr. Husain is the author of 25 international publications and more than 30 articles on development and governance in the print media; and former member of the United Nations Development Programme (UNDP)’s Gender Advisory Panel for Asia-Pacific. High-level contributions include: the Pakistan country report presented by the President at the World Summit on Social Development (1995); the country progression report presented by civil society at the World Summit on Sustainable Development (2002); Pakistan’s first progress report on the Millennium Development Goals (2003); corporate evaluation policy of the International Fund for Agricultural Development (IFAD); IFAD country evaluations of Sri Lanka and Turkey, its work across 17 countries in Asia-Pacific, and its contributions to decentralization in Eastern Africa; and assistance to 18-agency working group on agriculture, rural development and poverty to operationalize the One UN approach in Pakistan. Sector and thematic experience include agriculture research and extension, water management, forestry, governance, political economy, community￾driven development, rural development, social sectors (including social protection), environment, gender, trade, and the informal economy. Qualitative Data Analyst Dr. Babur Wasim - PhD Development Economics (Policy Analysis Programme), National Graduate Institute for Policy Studies (GRIPS), Tokyo, Japan (Oct 2007 - Sep 2010): Dr. Babur Wasim Arif is a Research and Evaluation professional with cross-functional expertise in agriculture, poverty reduction, education/ human capital, health, and SMEs development. He has over 15 years of experience of research design, implementation, and data analysis on mixed-methods research and evaluation assignments. He brings experience of identifying and deploying research- FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 111 specific data analysis methods and tools and is well-versed in the use of data analysis software including STATA, SPSS, MAXDQA and NVivo. He has contributed to multiple activity, project and program research and evaluation studies commissioned by USAID, FCDO, World Bank, and Asian Development Bank. His thematic areas of expertise include agriculture research and development, financial inclusion, economic growth, and social protection. Seed Certification Expert Ilhammudin - M.S. in Seed Technology/Agronomy, July 1993, Mississippi State University, USA. Mr. Ilhammudin brings more than 30 years of hands-on experience in seed certification and registration. He has worked in the Federal Seed Certification and Registration Department as Deputy Director and oversaw Regional Directorate with responsibilities including coordination and supervision of technical and administrative matters of the department in different regions. Additionally, Mr. Ilham has also been providing consultancy services to the public and private sector in the fields of seed technology (variety registration & release system in Pakistan, seed production, seed quality control, seed conditioning, and seed storage). He has co-authored and contributed to multiple research publications on seed technology. Entomologist/ Plant Pests Control Expert Dr. Farman Ullah - Ph.D. Oklahoma State University, 1993, USA. Dr. Farmaullah is currently serving as Professor at the University of Agriculture Peshawar. He has more than 30 years of teaching experience, where he supervised doctoral and postgraduate students on the subjects of Sanitary and Phytosanitary Measures (SPS), Pesticides Resistance Management and Pest of Crops. Moreover, he also brings extensive experience of managing research projects on sectors including Pest Risk Analysis and IPM of Maize, Pest Management of Wheat Aphids, Mass Production of Trichogramma-IPM of Maize Stem Borer, and Insect Pests Management of Peach. Disclosure of conflict of interest: All evaluation team members have provided a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 112 ANNEX IV: EVALUATION DATA ANALYSIS PLAN PAKISTAN AGRICULTURE PROGRAMS PERFORMANCE EVALUATION DATA ANALYSIS PLAN DECEMBER 5, 2022 This data analysis plan is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Associates in Development (AiD) Pvt. Ltd. and do not necessarily reflect the views of USAID or the United States Government. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 113 ACRONYMS ASP Agriculture Service Providers ASPTP Agriculture Service Providers Training Program CPEP Cotton Productivity Enhancement Project EGA Economic Growth and Agriculture PAP Pakistan Agriculture Programs PRMP Phytosanitary Risk Management Program WPEP Wheat Productivity Enhancement Project FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 114 OVERVIEW This document presents an illustrative plan to guide the data analysis process for the final evaluation of the Pakistan Agriculture Programs (PAP). The primary purpose of the evaluation is to identify the achievements of selected components of the PAP in line with its intended objectives and inform management decisions on on-going and future Economic Growth and Agriculture (EGA) activities. It will also measure the contribution of the PAP components on the focus areas of the agriculture sector in Pakistan. Of the total 14 sub-activities implemented under PAP, the evaluation focuses on four key sub￾activities i.e., Wheat Productivity Enhancement Project (WPEP), Cotton Productivity Enhancement Project (CPEP), Phytosanitary Risk Management Program (PRMP) and Agriculture Service Providers Training Program (ASPTP) because the PAP invested most of its resources on these interventions and these interventions were expected to produce more tangible results and create long term impacts on focus areas. Specifically, the final evaluation answers the following three questions. Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? These results include but are not limited to: a) How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. b) As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why? The question should indicate if the adoption of technologies increased incomes of Agriculture Service Providers (ASPs) and farmers. c) What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project intervention which included Punjab, Sindh, Balochistan and Gilgit-Baltistan. Effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Question 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Question 3: What are the lessons learned for future design and mechanism of similar projects? The detailed data analysis plan for each evaluation question is given in the Table below. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 115 TABLE 1: DATA ANALYSIS PLAN Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods Question 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? a) How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at various stages of approval by the Government of Pakistan. (Wheat) Developme nt of disease resistant wheat and cotton varieties. Cotton and wheat varieties at various stages of approval. ● Number of cotton and wheat varieties in national trials. ● Number of cotton and wheat varieties registered at Federal Seed Certification and Registration Department. ● Number of cotton and wheat varieties pending approval of the technical committee of the provincial seed council. ● Number of cotton and wheat varieties approved by provincial seed councils. ● Key informant interviews with Implementing partner staff, partner organization staff and key officials/members of the government. ● Review of program records. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 116 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods b) As for ASP sub￾activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why? The question should indicate if the adoption of technologies increased incomes of ASPs and farmers? Adoption of selected technologie s. Technologies adopted by farmers. ● Perceptions of respondents regarding technologies that were successfully adopted by the farmers. ● List of factors that enabled the adoption of technologies by farmers. ● List of factors that constrained the adoption of technologies by farmers. ● Key informant interviews with Implementing partner staff, partner organization staff and agriculture service providers trained by ASPTP. ● Group Interviews with beneficiary farmers. ● Review of program records. Changes in incomes of ASPs and farmers. ● Number of ASPs reporting changes in income after providing services on new technology to farmers (to the extent possible, changes in annual income will be reported in rupees). ● Number of farmers reporting changes in income after adoption of new technologies (to the extent possible, changes in annual income will be reported in rupees). ● Number of farmers reporting changes in yield after adoption of new technologies. ● Number of farmers reporting changes in input cost after adoption of new technologies. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 117 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods c) What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project interventions which included Punjab, Sindh, Balochistan and Gilgit Baltistan. The effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Mitigation of the impacts of pests on horticultura l crops and rice. Effects of pest mitigation on farms and farmers. ● Number of respondents reporting changes in infestation caused by the targeted insect/pest. ● Number of respondents reporting changes in population of insects/pests. ● Number of respondents reporting changes in the use of chemicals/pesticides. ● Number of respondents reporting changes in income (to the extent possible changes in annual income will be reported in rupees). ● Number of respondents reporting changes in input cost (plant protection and fertilizers application). ● Number of respondents reporting changes in the crop yields. ● Number of respondents reporting changes in the cultivated area. ● List of factors that contributed to mitigating the impact of pests. ● List of factors that constrained mitigating the impact of pests. ● Key informant interviews with implementing partner (CABI) staff, key stakeholder for selected crops (Partner provincial government Agriculture Department officials), growers (beneficiary farmers) who received parasitoids & trainings and training participants (stakeholders and govt. officials). ● Review of program records. WPEP FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 118 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods Question 2: Have the interventions (or outcomes) introduced under the selected sub￾activities been, or are likely to be, sustained? What factors led or deterred sustainability? Sustainabilit y of intervention s. Institutional sustainability. ● Number of respondents reporting continued use of laboratory/equipment provided through WPEP support. ● Number of respondents reporting on capacity to store germplasm provided to institutes through WPEP support. ● Number of respondents reporting continued work on seed variety development after WPEP. ● Number of respondents reporting participating in surveillance seminar and/or planning meetings after WPEP. ● Whether the staff who received capacity building support were performing their duties in their respective organizations. ● List of factors that contributed to sustainability. ● List of factors that constrained sustainability. ● Key informant interviews with Implementing partner staff, partner organization staff, representatives of seed producing companies and key officials/members of the government. ● Review of program records. Financial sustainability. ● Perceptions of respondents on availability of financial resources for continuing the interventions initiated under WPEP. ● Number of respondents reporting having sustainability plans for continuation of WPEP interventions. ● List of factors that contributed to sustainability. ● List of factors that constrained sustainability. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 119 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods Sustainabilit y of benefits. Sustainable commercial production of disease resistant seed varieties. ● The number of respondents reporting seed varieties developed through WPEP support being commercially produced. ● List of factors that enable the commercial production of seed varieties developed with WPEP support. ● List of factors that constrained the commercial production of seed varieties developed with WPEP support. CPEP Sustainabilit y of intervention s Institutional sustainability. ● Number of respondents reporting continued use of laboratory/equipment provided through CPEP support. ● Number of respondents reporting on capacity to store germplasm provided to institutes through CPEP support. ● Number of respondents reporting continued work on seed variety development after CPEP. ● Number of respondents reporting participating in surveillance seminar and/or planning meetings. ● Whether the staff who received capacity building support were performing their duties in their respective organizations. ● List of factors that contributed to sustainability. ● List of factors that constrained sustainability. ● Key informant interviews with Implementing partner staff, partner organization staff, representatives of seed producing companies and key officials/members of the government. ● Review of program records FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 120 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods Financial sustainability. ● Perceptions of respondents on availability of financial resources for continuing the interventions initiated under CPEP. ● Number of respondents reporting having sustainability plans for continuation of CPEP interventions. ● List of factors that contributed to sustainability. ● List of factors that constrained sustainability. Sustainabilit y of benefits. Sustainable commercial production of disease resistant seed varieties. ● Number of respondents reporting seed varieties developed through CPEP support being commercially produced. ● List factors that enable the commercial production of seed varieties developed with CPEP support. ● List factors that constrained the commercial production of seed varieties developed with CPEP support. ASPTP Sustainabilit y of benefits. Continued provision of services by ASPs on selected technologies. ● Number of ASPs providing services on new technologies after ASPTP ended. ● List of factors that enable the continued provision of services by ASPs. ● List of factors that constrained the continued provision of services by ASPs. ● Key informant interviews with Implementing partner staff, partner organization staff and agriculture FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 121 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods Continued use of technologies by farmers. ● Number of farmers using services from ASPs on new technologies after ASPTP ended. ● List of factors that enabled the continued use of new technologies by farmers. ● List of factors that constrained the continued use of new technologies by farmers. service providers trained by ASPTP. ● Group Interviews with beneficiary farmers. ● Review of program records. PRMP Sustainabilit y of intervention s Institutional Sustainability ● Number of respondents reporting continued use of laboratory/equipment provided through PRMP support. ● Whether the staff who received capacity building support are performing their duties in their respective organizations. ● Perceptions of respondents on transfer of information and technology to the incumbent staff/employee. ● Perceptions of respondents on on-the-job-training provided to employees/staff working in labs upgraded through PRMP support. ● Key informant interviews with implementing partner staff, key stakeholder for selected crops, growers to whom parasitoid were released and training participants. ● Review of program records. Financial sustainability ● Perceptions of respondents on availability of financial resources for continuing the interventions initiated under PRMP. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 122 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods Sustainabilit y of benefits. Continued use of biocontrol agents by farmers beyond PRMP Institutional Sustainability ● Number of farmers reporting continued use of parasitoids at farms. ● List of factors that enabled the continued use of biocontrol technology at farms. ● List of factors that constrained the continued use of biocontrol technology at farms. Continued development and deployment of biocontrol agents ● Number of parasitoids reared by the PRMP￾supported facilities in a month after PRMP ended? ● Number of parasitoids sold to farmers in a month after PRMP ended? Question 3: What are the lessons learned for future design and mechanism of similar projects? Lessons for future similar projects. Lessons learned in implementatio n approaches of the four sub-activities. ● Perceptions of respondents on lessons learned from implementation of PAP’s four sub-activities i.e., WPEP, CPEP, ASPTP and PRMP. Lessons learned vis-à￾vis challenges faced in implementatio n of the four sub-activities. ● Perceptions of respondents on lessons learned from implementation of PAP’s four sub-activities i.e., WPEP, CPEP, ASPTP and PRMP. WPEP/CPEP ● Key informant interviews with Implementing partner staff, partner organization staff FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 123 and key officials/members of the government. ● Review of program records. PRMP ● Key informant interviews with implementing partner staff, key stakeholder for selected crops, growers to whom parasitoid were released and training participants. ● Review of program records. ASPTP ● Key informant interviews with Implementing partner staff, partner organization staff and agriculture service providers trained by ASPTP. ● Group Interviews with beneficiary farmers. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 124 Evaluation Question Key Constructs Dimensions Potential Variables Sources of Data/data Collection Methods ● Review of program records. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 125 ANNEX V: DATA COLLECTION AND ANALYSIS TOOLS Key Informant Interview Guide Wheat Productivity Enhancement Project (WPEP) – Federal Seed Certification and Registration Department (FSC&RD) Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue55 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's Name A9. Name A10. Designation A13. Note Taker’s Name56 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Wheat Productivity Enhancement Project (WPEP), implemented by the International Maize and Wheat Improvement Center (CIMMYT). Our team would like to ask you a few questions regarding the performance of WPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 40 – 50 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate. ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? 55 In the case of virtual interaction, this refers to the location of participants, not evaluators. 56 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 126 ◻ Yes ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, proceed with the interview without recording it. Preliminary Information 1. Please help us understand the role of FSC&RD in the registration of new wheat seed varieties. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(a): How successful was the development of disease-resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan (GoP). Q1. Are you aware of the Wheat Enhancement Productivity Project (WPEP) implemented by CIMMYT? If so, can you please tell us if any new seed varieties have been introduced by WPEP/CIMMYT? Q2. If yes, can you please tell us if any new varieties have been entered into the approval process? Please provide details. Prompts: a. Varieties in the process of Distinctness, Uniformity & Stability (DUS) testing. (Request the respondent to share the list of varieties). b. Varieties that have completed the process of DUS testing. (Request the respondent to share the list of varieties). c. Varieties submitted/ applied to FSC&RD for registration. (Request the respondent to share the list of varieties). d. Varieties registered by FSC&RD. (Request the respondent to share the list of varieties). Q3. In your opinion, what factors (if any) contributed to or hindered the registration of new seed varieties developed with support from WPEP/CIMMYT at FSC&RD? Please explain. Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q4. In your opinion, are the seed varieties introduced through WPEP/CIMMYT likely to be commercially produced in future? Please elaborate the factors which may contribute to or hinder the continuation. Please explain. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 127 Key Informant Interview Guide Wheat Productivity Enhancement Project (WPEP) – Partner Organizations Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue57 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name58 A11. Gender A10. Respondent’s Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Wheat Productivity Enhancement Project (WPEP), implemented by the International Maize and Wheat Improvement Center (CIMMYT). Our team would like to ask you a few questions regarding the performance of WPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 50 – 60 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes ◻ No 57 In the case of virtual interaction, this refers to the location of participants, not evaluators. 58 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 128 If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us if you have been involved in the implementation of WPEP. If so, please tell us about your role. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(a): How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. Q1. Did the organization receive any support on the development of new wheat seed varieties under WPEP? If yes, to what extent, and in what ways, did the support affect (if at all) the development of new wheat varieties? Prompts: a. Identification of disease-resistant varieties (screening) b. Availability of germplasm c. Breeding d. Testing Q2. Please tell us if any new seed varieties were developed by this organization through WPEP’s support. If so, have these new seed varieties been entered into the approval process? If yes, please tell us what stages of approval these are at. Please provide details. Prompts: a. Varieties included in National Uniform Wheat Yield Trials (NUWYT). [Request respondent to share the list of varieties]. b. Varieties submitted to the Variety Evaluation Committee (VEC). [Request respondent to share the list of varieties]. c. Varieties submitted for registration to the Federal Seed Certification & Registration Department (FSC&RD). [Request respondent to share the list of varieties]. d. Varieties registered with FSC&RD. [Request respondent to share the list of varieties]. e. Varieties approved by Provincial Seed Council (PSC). [Request respondent to share the list of varieties]. Q3. In your opinion, has the development of new varieties resulted in any change in the prevalence of leaf rust disease in the wheat crop? If yes, please explain how and to what extent this change has occurred. Q4. What other noteworthy changes have you observed, if any, because of support provided to your organization through WPEP? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 129 Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q5. In your opinion, are the seed varietal development interventions initiated with support from WPEP likely to continue beyond the life of the project? Please elaborate the factors which may contribute to or hinder the continuation. Q6. How likely are you to participate in the annual wheat surveillance seminar or planning meetings? Please elaborate on the factors which may contribute to or hinder your participation. Q7. Does your organization have adequate financial resources in its annual budget to enable continuation of wheat seed varietal development work? Please elaborate which factors may contribute to or hinder the availability of financial resources. Q8. Does your organization have a sustainability plan for continuation of WPEP interventions (or results thereof)? Please elaborate the factors which may contribute to or hinder the continuation. Q9. In your opinion, are the released seed varieties developed with WPEP support (if any) likely to continue in commercial production beyond the life of WPEP? Please explain. Prompt: a. How long (in years) before they start losing their disease or rust resistance? Q10. How are the germplasm received through WPEP stored? Are they likely to be used for development of new seed varieties beyond the life of WPEP? Please explain. Q11. Has your organization received any facility/ laboratory upgradation support through WPEP? If yes, please tell us whether the facilities/ laboratories upgraded with support from WPEP are likely to be utilized for research on wheat variety development beyond the life of WPEP. Please elaborate the factors which may contribute to or hinder the future continuity or utilization of these. Q12. In your opinion, will the on-going flood situation affect the sustainability of WPEP interventions (or results thereof)? If yes, please explain how and to what extent. Q13. In your opinion, what factors (if any) are likely to adversely affect the continuation of WPEP results in the future? Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q14. Please tell us if the organization faced any challenges during the implementation of WPEP. If so, how did you respond to those challenges? Q15. What support provided through WPEP to your organization was most useful? What was least useful? Please explain why. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 130 Q16. Did WPEP engage women in project interventions? If not, what could have been done to encourage women’s participation in the activity? If yes, please share how women were focused directly and indirectly under WPEP? Q17. What were the approaches, if any, that benefited women? What approaches can be adopted in similar future activities to ensure equal/ higher benefits for women? Q18. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 131 Key Informant Interview Guide Wheat Productivity Enhancement Project (WPEP) – Provincial Seed Council Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue59 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name60 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Wheat Productivity Enhancement Project (WPEP), implemented by the International Maize and Wheat Improvement Center (CIMMYT). Our team would like to ask you a few questions regarding the performance of WPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 40 – 50 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate. If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes 59 In the case of virtual interaction, this refers to the location of participants, not evaluators. 60 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 132 ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Identifying Question 1. Please help us understand the role of Provincial Seed Council (PSC) in the approval and release of new wheat varieties. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(a): How successful was the development of disease-resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. Q1. Are you aware of the Wheat Enhancement Productivity Project (WPEP) implemented by CIMMYT? If so, can you please tell us if any new seed varieties have been introduced by WPEP/ CIMMYT. Q2. If yes, can you please tell us if any new varieties have been entered into the approval process at the provincial level? If yes, please provide details. [Request the respondent for sharing the minutes of PSC meetings in which decisions regarding approval of new varieties were made]. Prompts a. Varieties in the process of evaluation by Expert/ Technical Sub-Committee. [Request the respondent to share the list of varieties]. b. Varieties cleared by Expert/ Technical Sub-Committee. [Request the respondent to share the list of varieties]. c. Varieties in the process of evaluation by the PSC. d. Varieties approved and released by the PSC. Q3. In your opinion, what factors (if any) contributed to or hindered the approval/ release of new seed varieties (developed by WPEP/CIMMYT)? Please explain. Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q4. In your opinion, are the seed varieties introduced through WPEP/CIMMYT likely to be commercially produced in future? Which factors may contribute to or hinder the continuation? Please explain. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 133 Key Informant Interview Guide Wheat Productivity Enhancement Project (WPEP) – Seed Companies Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue61 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name62 A11. Gender A1. Date A2. Language Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Wheat Productivity Enhancement Project (WPEP), implemented by the International Maize and Wheat Improvement Center (CIMMYT). Our team would like to ask you a few questions regarding the performance of WPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 50 – 60 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate. If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? 61 In the case of virtual interaction, this refers to the location of participants, not evaluators. 62 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 134 ◻ Yes ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us if this organization has received any support from WPEP/CIMMYT? If so, please explain. Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q1. Are you aware of the released wheat seed varieties developed with support from WPEP/CIMMYT? If so, has your company received the newly released varieties from WPEP/CIMMYT? If so, please tell us which wheat seed varieties (if any) introduced through WPEP/CIMMYT, have been produced by your company? Q2. If yes, in your opinion, are these seed varieties likely to be commercially produced by your company beyond the life of WPEP? Please elaborate the factors which may contribute to or hinder the continuation. Please explain. Prompts: a. Availability of early generation seed b. Demand from farmers for the seed varieties c. Yield d. Disease resistance Q3. In your opinion, will the ongoing flood situation affect commercial production of released varieties (mentioned in preceding questions)? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q4. What support provided through WPEP/CIMMYT to your company was most useful for you? What was least useful? Please explain why. Q5. How, if at all, could similar future activities be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 135 Key Informant Interview Guide Cotton Productivity Enhancement Project (CPEP) – Federal Seed Certification and Registration Department (FSC&RD) Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue63 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name64 A11. Gender A1. Date Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Cotton Productivity Enhancement Project (CPEP), implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of CPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 40 – 50 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. 63 In the case of virtual interaction, this refers to the location of participants, not evaluators. 64 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 136 Do I have your permission to record the interview? ◻ Yes ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, proceed with the interview without recording it. Preliminary Information 1. Please help us understand the role of FSC&RD in the registration of new Cotton seed varieties. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(a): How successful was the development of disease-resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan (GoP). Q1. Are you aware of the Cotton Enhancement Productivity Project (CPEP) implemented by ICARDA? If so, can you please tell us if any new seed varieties have been introduced by CPEP/ICARDA? Q2. If yes, can you please tell us if any new varieties have been entered into the approval process? Please provide details. Prompts: a. Varieties in the process of Distinctness, Uniformity & Stability (DUS) testing. (Request the respondent to share the list of varieties). b. Varieties that have completed the process of DUS testing. (Request the respondent to share the list of varieties). c. Varieties submitted/ applied to FSC&RD for registration. (Request the respondent to share the list of varieties). d. Varieties registered by FSC&RD. (Request the respondent to share the list of varieties). Q3. In your opinion, what factors (if any) contributed to or hindered the registration of new seed varieties developed with support from CPEP/ICARDA at FSC&RD? Please explain. Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q4. In your opinion, are the seed varieties introduced through CPEP/ICARDA likely to be commercially produced in future? Please elaborate the factors which may contribute to or hinder the continuation. Please explain. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 137 Key Informant Interview Guide Cotton Productivity Enhancement Project (CPEP) – Partner Organizations Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue65 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name66 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Cotton Productivity Enhancement Project (CPEP), implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of CPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 50 – 60 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes ◻ No 65 In the case of virtual interaction, this refers to the location of participants, not evaluators. 66 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 138 If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us if you have been involved in implementation of CPEP. If so, please tell us about your role. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(a): How successful was the development of disease resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. Q1. Did the organization receive any support on the development of new Cotton seed varieties under CPEP? If yes, to what extent, and in what ways, did the support affect (if at all) the development of new Cotton varieties? Prompts: a. Identification of disease-resistant varieties (screening) b. Availability of germplasm c. Breeding d. Testing Q2. Please tell us if any new seed varieties were developed by this organization through CPEP’s support. If so, have these new seed varieties been entered into the approval process? If yes, please tell us what stages of approval these are at. Please provide details. Prompts: a. Varieties included in National Coordinated Variety Test (NCVT). [Request respondent to share the list of varieties]. b. Varieties submitted to Pakistan Central Cotton Committee (PCCC). [Request respondent to share the list of varieties]. c. Varieties submitted for registration to the Federal Seed Certification & Registration Department (FSC&RD). [Request respondent to share the list of varieties]. d. Varieties registered with FSC&RD. [Request respondent to share the list of varieties]. e. Varieties approved by Provincial Seed Council (PSC). [Request respondent to share the list of varieties]. Q3. In your opinion, has the development of new varieties resulted in any change in the prevalence of Cotton Leaf Curl Virus (CLCuV) in the Cotton crop? If yes, please explain how and to what extent this change has occurred. Q4. What other noteworthy changes have you observed, if any, because of support provided to your organization through CPEP? Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 139 Q5. In your opinion, are the seed varietal development interventions initiated with support from CPEP likely to continue beyond the life of the project? Please elaborate the factors which may contribute to or hinder the continuation. Q6. How likely are you to participate in annual cotton surveillance seminar or planning meetings? Please elaborate on the factors which may contribute to or hinder your participation. Q7. Does your organization have adequate financial resources in its annual budget to enable continuation of cotton seed varietal development work? Please elaborate which factors may contribute to or hinder the availability of financial resources. Q8. Does your organization have a sustainability plan for continuation of CPEP interventions (or results thereof)? Please elaborate the factors which may contribute to or hinder the continuation. Q9. In your opinion, are the released seed varieties developed with CPEP support (if any) likely to continue in commercial production beyond the life of CPEP? Please explain. Prompt: a. How long (in years) before they start losing their CLCuV disease resistance? Q10. How are the germplasm received through CPEP stored? Are they likely to be used for development of new seed varieties beyond the life of CPEP? Please explain. Q11. Has your organization received any facility/ laboratory upgradation support through CPEP? If yes, please tell us whether the facilities/ laboratories upgraded with support from CPEP are likely to be utilized for research on Cotton variety development beyond the life of CPEP. Please elaborate the factors which may contribute to or hinder the future continuity or utilization of these. Q12. In your opinion, will the on-going flood situation affect the sustainability of CPEP interventions (or results thereof)? If yes, please explain how and to what extent. Q13. In your opinion, what factors (if any) are likely to adversely affect the continuation of CPEP results in the future? Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q14. Please tell us if the organization faced any challenges during the implementation of CPEP. If so, how did you respond to those challenges? Q15. What support provided through CPEP to your organization was most useful? What was least useful? Please explain why. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 140 Q16. Did CPEP engage women in project interventions? If not, what could have been done to encourage women’s participation in the activity? If yes, please share how women were focused directly and indirectly under CPEP? Q17. What were the approaches, if any, that benefited women? What approaches can be adopted in similar future activities to ensure equal/ higher benefits for women? Q18. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 141 Key Informant Interview Guide Cotton Productivity Enhancement Project (CPEP) – Provincial Seed Council Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue67 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name68 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Cotton Productivity Enhancement Project (CPEP), implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of CPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 40 – 50 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate. If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes 67 In the case of virtual interaction, this refers to the location of participants, not evaluators. 68 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 142 ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Identifying Question 1. Please help us understand the role of Provincial Seed Council (PSC) in the approval and release of new cotton varieties. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(a): How successful was the development of disease-resistant wheat and cotton varieties? Development will be measured by the number of varieties at different stages of approval by the Government of Pakistan. Q1. Are you aware of the Cotton Enhancement Productivity Project (CPEP) implemented by ICARDA? If so, can you please tell us if any new seed varieties have been introduced by CPEP/ ICARDA. Q2. If yes, can you please tell us if any new varieties have been entered into the approval process at the provincial level? If yes, please provide details. [Request the respondent for sharing the minutes of PSC meetings in which decisions regarding approval of new varieties were made]. a. Varieties in the process of evaluation by Expert/ Technical Sub-Committee. [Request the respondent to share the list of varieties]. b. Varieties cleared by Expert/ Technical Sub-Committee. [Request the respondent to share the list of varieties]. c. Varieties in the process of evaluation by the PSC. d. Varieties approved and released by the PSC. Q3. In your opinion, what factors (if any) contributed to or hindered the approval/ release of new seed varieties (developed by CPEP/ICARDA)? Please explain. Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q4. In your opinion, are the seed varieties introduced through CPEP/ICARDA likely to be commercially produced in future? Which factors may contribute to or hinder the continuation? Please explain. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 143 Key Informant Interview Guide Cotton Productivity Enhancement Project (CPEP) – Seed Companies Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue69 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name70 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Cotton Productivity Enhancement Project (CPEP), implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of CPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 50 – 60 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate. If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes 69 In the case of virtual interaction, this refers to the location of participants, not evaluators. 70 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 144 ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us if this organization has received any support from CPEP/ICARDA? If so, please explain. Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q1. Are you aware of the released cotton seed varieties developed with support from CPEP/ICARDA? If so, has your company received the newly released varieties from CPEP/ICARDA? If so, please tell us which cotton seed varieties, introduced through CPEP/ICARDA, have been produced by your company? Q2. If yes, in your opinion, are these seed varieties likely to be commercially produced by your company beyond the life of CPEP? Please elaborate the factors which may contribute to or hinder the continuation. Please explain. Prompts: a. Availability of early generation seed b. Demand from farmers for the seed varieties c. Yield d. Disease resistance Q3. In your opinion, will the on-going flood situation affect the commercial production of released varieties (mentioned in the preceding questions)? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q4. What support provided through CPEP/ICARDA to your company was most useful for you? What was least useful? Please explain why. Q5. How, if at all, could similar future activities be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 145 Key Informant Interview Guide Cotton Productivity Enhancement Project (CPEP) – Non-Governmental Organizations (NGOs) Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue71 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name72 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Cotton Productivity Enhancement Project (CPEP), implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of CPEP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 30 – 40 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate. If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes 71 In the case of virtual interaction, this refers to the location of participants, not evaluators. 72 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 146 ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. 1. Please tell us how your organization was involved under CPEP by ICARDA. Evaluation Question 2: Have the interventions (or outcomes) introduced under the selected sub￾activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q1. Did you provide capacity building support to farmers in your area? If yes, please elaborate on the type of support provided, and whether it was the standard approach recommended by CPEP/ICARDA or differed from the approach followed by other NGO partners. Q2. Did your organization provide support to farmers through Farmer Field School (FFS) in the targeted area? If yes, did FFS continue to function after the CPEP ended? Please tell us which factors (if any) contributed to or constrained the continuation of FFS. a) Did your organization continue to support farmers through FFS after CPEP ended? b) Have the farmers maintained linkages with government departments after CPEP ended? Q3. Did your organization provide support to farmers through Women Open Schools (WOS) in the targeted area? If yes, did WOS continue to function after the CPEP ended? Please tell us which factors (if any) contributed to or constrained the continuation of WOS. a) Did your organization continue to support farmers through WOS after CPEP ended? b) Have the farmers maintained linkages with government departments after CPEP ended? Q4. Did your organization provide awareness raising support to children through Children Ecology Club (CEC) in the targeted area? If yes, did CEC continue to function after the CPEP ended? Which factors contributed to or constrained the continuation of CEC. a. Did your organization continue to engage with children on awareness raising through CEC after CPEP ended? Q5. In your opinion, will the on-going flood situation affect benefits (if any) that accrued from capacity building of farmers through FFS/WOS/CEC? Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q6. What support provided through CPEP/ICARDA has been most useful? What was least useful? Please explain why. Q7. How, if at all, could similar future activities be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 147 Key Informant Interview Guide Agriculture Service Provider Training Program (ASPTP) – Agriculture Service Providers (ASP) Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue73 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name74 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Agriculture Service Provider Training Program (ASPTP) implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of ASPTP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 25 – 30 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes ◻ No 73 In the case of virtual interaction, this refers to the location of participants, not evaluators. 74 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 148 If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us something about your background. [Prompts: age, occupation, farm size (if relevant), education, and years of experience in agriculture and business]. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(b): As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why. The question should indicate if the adoption of technologies increased incomes of ASPs and farmers? Q1. Have you received any training through Agriculture Service Providers Training Program (ASPTP)/Partner Organization (spell out name of institute)? If yes, please tell us which technology(ies) you received training on. Q2. How much do you earn each year, on average, from providing agriculture services? Did your income change after your engagement under ASPTP? If yes, how much? [Please state your annual income before and after your engagement under ASPTP]. Q3. If new technology was introduced to the farmers, did the farmers adopt the new technology? If yes, in your opinion: a) Did the adoption of new technology by the farmers affect their annual income? If yes, please explain how and to what extent. b) Did the adoption of new technology by the farmers affect their crop yields? If yes, please explain how and to what extent. c) Did the adoption of new technology by the farmers affect their input costs? If yes, please explain how and to what extent. d) How many farmers were introduced to the new technology? Q4. Please tell us about the factors that contributed to or constrained the adoption of new technologies by the farmers. Prompts a) Availability of services b) Capacity of farmers c) Demonstrations of new technology Q5. In what ways, if any, did you reach out to women farmers and workers? How did they respond? Q6. What other noteworthy changes have you observed (if any) because of support provided to you through ASPTP? Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors promoted or deterred sustainability? Q7. Did you continue providing services to farmers on the new technology after the ASPTP ended? If yes, please let us know which factors (if any) enabled you to continue providing services to FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 149 farmers on new technology in your area. If not, please let us know about the factors (if any) that prevented you from providing services to farmers on new technology in your area. Q8. In your opinion, did the farmers continue using the new technology that you introduced to them after the ASPTP ended? If yes, please tell us which factors (if any) enabled the continued use of new technologies by farmers in your area. If not, please tell us about the factors (if any) that prevented the adoption of new technology by the farmers in your area. Q9. In your opinion, will the on-going flood situation affect your income and/or provision of services to farmers? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q10. What challenges were faced by you (if any) during the introduction of new technology to the farmers? How did you respond to those challenges? Q11. What support that you received from ASPTP was most useful? What was least useful? Please explain why. Q12. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 150 Group Interview Guide Agriculture Service Provider Training Program (ASPTP) – Beneficiary Farmers Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue75 Informant/ Respondent's Details Participant 1 Participant 2 Participant 3 A9. Name A10. Profession A11. Gender Group Interview Team A12. Moderator's Name A13. Note Taker’s Name76 Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Agriculture Service Provider Training Program (ASPTP) implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of ASPTP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 25 – 30 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. 75 In the case of virtual interaction, this refers to the location of participants, not evaluators. 76 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 151 Do I have your permission to record the interview? ◻ Yes ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us something about your background. [Prompts: age, occupation, farm size (if relevant), education, and years of experience in agriculture and business] Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(b): As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why. The question should indicate if the adoption of technologies increased incomes of ASPs and farmers? Q1. Did the Agriculture Service Providers (ASP) introduce any new technology to you? If yes, did you receive services on the new technology from the ASP? Please elaborate. Prompts a) Type of technology. b) Period and duration of service. c) Frequency of service provision. Q2. Did you use the new technology introduced to you by the ASPs? If yes: a) Did the usage of new technology affect your annual income? If yes, please explain how and to what extent. (Please state your annual income before and after introduction of the new technology). b) Did the usage of new technology affect your crop yields? If yes, please explain how and to what extent. (Please state crop yield in maunds per acre before and after adoption of new technology). c) Did the usage of new technology affect your input costs? If yes, please explain how and to what extent. (Please state the annual cost before and after adoption of new technology). d) How much did you pay the ASP per year, on average? Q3. Please tell us about the factors (if any) that contributed to your usage of new technology that was introduced to you by ASPs. Prompts a) Availability of services b) Capacity of farmers c) Demonstrations of new technology d) Any other factor. Q4. Please tell us about the factors (if any) that constrained the use of new technology that was introduced to you by ASPs. Q5. To what extent were the women who work on your farm involved with the new technology? How, if at all, did the new technology affect women and girls in your household? Q6. What other noteworthy changes have you observed (if any) because of support provided to you through ASPTP? To what extent did neighboring farmers show an interest in the new technology after seeing it on your farm? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 152 Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q7. Did you continue receiving services from the ASPs on the new technology after the ASPTP ended? If yes: a) Please tell us which factors (if any) enabled you to continue using the new technologies introduced by the ASPs. b) Please tell us which factors (if any), in your opinion, enabled the ASPs to continue providing services to farmers on new technology in your area. Q8. If no: a) Please tell us which factors (if any) constrained you from continuing to use the new technologies introduced by the ASPs. b) Please tell us which factors (if any) prevented the ASPs from providing services to farmers on new technology in your area. Q9. In your opinion, will the on-going flood situation affect your income and/or continued use of the new technology introduced by ASPs? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q10. Did you face any challenge(s) while using the new technology introduced to you by the ASPs? If yes, how did you respond to those challenge(s)? Q11. What support that you received from ASPTP was most useful? What was least useful? Please explain why. Q12. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 153 Key Informant Interview Guide Agriculture Service Provider Training Program (ASPTP) – Partner Organizations Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue77 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name78 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Agriculture Service Provider Training Program (ASPTP) implemented by the International Center for Agricultural Research in the Dry Areas (ICARDA). Our team would like to ask you a few questions regarding the performance of ASPTP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 50 – 60 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes ◻ No 77 In the case of virtual interaction, this refers to the location of participants, not evaluators. 78 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 154 If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us if you have been involved in the implementation of the ASP Training Program. If so, please tell us about your role. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(b): As for ASP sub-activity, this question will provide information on what technologies were most successful in terms of adoption by farmers and why. The question should indicate if the adoption of technologies increased incomes of ASPs and farmers? Q1. Did the institute provide training on new technologies to the ASPs through ASPTP? If yes, which new technologies did the institute provide training on to the ASPs? How many ASPs were trained? Q2. Did the ASPs, trained through ASPTP, provide services to farmers on the new technology in targeted areas? If yes, what type of farmers were the intended target groups for these technologies? [Prompt: characteristics of farmers in terms of farm size, crop, issues faced and other relevant characteristics]. Q3. In your opinion, did the farmers adopt the new technologies introduced to them by the ASPs? If yes, please identify the most successful and least successful technologies in terms of adoption by specific target groups i.e., the farmers. Q4. In your opinion, what factors enabled or constrained the adoption of new technologies by specific target groups i.e., the farmers? Q5. In your opinion, were there any changes in income of farmers who received services on new technologies from the ASPs? If yes, please state the factors that contributed to this change. Q6. In your opinion, were there any changes in income of ASPs after receiving support through ASPTP? If yes, please state the factors that contributed to this change. Q7. What other noteworthy changes have you observed (if any) because of the support provided by ASPTP? Q8. In what ways did ASPs reach out to men and women to promote the adoption of technologies? To what extent were these efforts successful for women? Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q9. In your opinion, did the ASPs continue to perform their intended role, i.e., provision of services on new technologies to farmers, after ASPTP ended? Please elaborate the factors which may have contributed to or hindered the continuation. Q10. In your opinion, did the farmers continue to use the new technology that was introduced to them by the ASPs, after ASPTP ended? Please elaborate the factors which may have contributed to or hindered the continuation. Q11. Does your institution plan for continuation of ASPTP interventions i.e., providing capacity building opportunities to ASPs after the end of the project? Please elaborate the factors which may contribute to or hinder the continuation. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 155 Q12. In your opinion, will the on-going flood situation affect the sustainability of ASTPS results? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q13. What challenges were faced by you (if any) during the introduction of new technology to the farmers? How did you respond to those challenges? Q14. What support that you received from ASPTP was most useful? What was least useful? Please explain why. Q15. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 156 Key Informant Interview Guide Phytosanitary Risk Management Program (PRMP) – Beneficiary Farmers Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue79 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name80 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Phytosanitary Risk Management Program (PRMP), implemented by the Centre for Agriculture and Biosciences International (CABI). Our team would like to ask you a few questions regarding the performance of PRMP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 25 – 30 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes ◻ No 79 In the case of virtual interaction, this refers to the location of participants, not evaluators. 80 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 157 If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us something about your background. [Prompts: age, occupation, farm size (if relevant), education, and years of experience in agriculture and business]. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(c): What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project interventions which included Punjab, Sindh, Balochistan and Gilgit Baltistan. The effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Mitigating Impact of Pest Q1. Did you receive any support from the Phytosanitary Risk Management Program (PRMP)? If yes, please tell us about the support you received from PRMP. Q2. Did the support you receive from PRMP affect (if at all) infestation caused by the targeted insect/pest in your cultivated crop? If yes, to what extent and how? Q3. Did the support you receive from PRMP affect (if at all) the population of insects/pests in your cultivated crop? If yes, to what extent and how? Q4. Did the support you receive from PRMP affect (if at all) the use of chemicals/pesticides on your farm? If yes, to what extent and how? [Please state the frequency of pesticides application to control the pest for the year before PRMP support and for the years after receiving PRMP support]. Q5. Did the support you receive from PRMP affect (if at all) your annual income from the cultivated crop? If yes, to what extent and how? How much of this, if any, was due to the reduction in losses? [Please state your annual income for the year before receiving PRMP support and for years after receiving PRMP support]. Q6. Did the support you receive from PRMP affect (if at all) the yield from the cultivated crop? If yes, to what extent and how? [Please state crop yield in maund per acre for the year before PRMP and for years after receiving PRMP support]. Q7. Did the support you receive from PRMP affect (if at all) the area under cultivation for the cultivated crop? If yes, to what extent and how? [Please state cultivated area in acre before and after PRMP support]. Q8. In your opinion, what factors (if any) contributed to mitigating the impact of pests in the targeted area? Please explain. Prompts: a. Pest management practices b. Availability of pest management information (source). c. Release of biocontrol agents [Natural Enemies Field Reservoirs (NEFRs)] d. Capacity of farmers/researchers/agricultural extension officers e. Availability of services from extension agents f. Changes in crop plantation g. Any other factor. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 158 Q9. In your opinion, what factors (if any) constrained mitigating the impact of pests in the targeted area? Please explain. Q10. What other noteworthy changes have you observed (if any) because of support provided to you through PRMP? Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q11. Did you continue the use of biocontrol technology after PRPM ended? Please tell us which factors (if any) contributed to or constrained the continued use of biocontrol technology on your farm. Q12. In your opinion, will the on-going flood situation affect the use of biocontrol technology on your farm? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q13. Please tell us if you faced any challenges in adopting and using the biocontrol technology on your farm? If so, how did you respond to those challenges? Q14. What support provided through PRMP to you was most useful? What was least useful? Please explain why. Q15. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 159 Key Informant Interview Guide Phytosanitary Risk Management Program (PRMP) – Partner Organizations Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue81 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name82 A11. Gender A1. Date A2. Language Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Phytosanitary Risk Management Program (PRMP), implemented by Centre for Agriculture and Biosciences International (CABI). Our team would like to ask you a few questions regarding the performance of PRMP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. Anonymity and Consent Statement The interview will take approximately 25 – 30 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. 81 In the case of virtual interaction, this refers to the location of participants, not evaluators. 82 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 160 Do I have your permission to record the interview? ◻ Yes ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us if you have been involved in implementation of the Phytosanitary Risk Management Program (PRMP). If so, please tell us about your role. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(c): What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project interventions which included Punjab, Sindh, Balochistan and Gilgit Baltistan. The effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Q1. Did your organization/department receive any support from PRMP? If yes, what type of support was provided? Please explain. Q2. Was PRMP able to mitigate the impact of pests in the targeted area? If yes, please explain to what extent and in what ways. [Request the respondent to share any documented evidence, if available]. Prompts: a. Changes in infestation caused by the targeted insect/pest. b. Changes in population of insects/pests. c. Changes in the use of chemicals/pesticides. d. Changes in the farmers’ income. e. Changes in farmer’s input cost (plant protection and fertilizers application). f. Changes in the crop yields g. Changes in the cultivated area. h. Any other changes? Q3. In your opinion, what factors (if any) are responsible for mitigating the impact of pests in the targeted area, as you have identified above? Please explain. [Request the respondent to share any documented evidence, if available]. Prompts: a. Pest management practices. b. Availability of pest management information [check the source] c. Release of biocontrol agents [Natural Enemies Field Reservoirs (NEFRs)] d. Capacity of farmers/researchers/agricultural extension officers e. Availability of services from extension agents f. Changes in crop plantation g. Any other factor. Q4. What other noteworthy changes have you observed, if any, because of support provided to your organization/department through PRMP? Prompts a. At the organizational level FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 161 b. At the farm/ field level Q5. In what ways did PRMP reach out to men and women for promoting the control of pests through biocontrol agents? To what extent were these efforts successful for women? Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors promoted or deterred sustainability? Q6. Has your organization/department received any facility/ laboratory upgradation support through PRMP? If yes, please tell us whether the facility/laboratory is being utilized for research on biocontrol of pests/insects and/or rearing of parasitoids after PRMP ended. Please elaborate the factors which have contributed to or hindered the continuity or utilization of these. Prompts: a) Are parasitoids available to farmers to use in fields? b) Are the farmers able to buy parasitoids? c) How much parasitoids are produced in a month? d) How much parasitoids are sold to farmers in a month? Q7. Has your organization/department received capacity building support for its staff through PRMP? If yes, what type of capacity development support was received? Are the staff, who received capacity building support, performing their duties in the organization after PRMP has ended? Please elaborate. Prompts: a) Are the staff/employees still employed by the organization? b) If employees have been posted out, whether information and technology has been transferred to the incumbent staff/employee. c) Is there any on-the-job-training for employees/staff working in labs upgraded through PRMP support? Q8. In your opinion, are the interventions (or the results thereof) institutionalized (or likely to be institutionalized) by your organization/department? Please explain to what extent and in what ways, including financial allocations. Q9. In your opinion, will farmers in your area continue to benefit from the results of PRMP interventions in terms of mitigating the impacts of targeted pests? Please elaborate the factors which may contribute to or hinder the continuation. Q10. In your opinion, will the on-going flood situation affect the sustainability of PRMP interventions (or results thereof)? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q11. Please tell us if the organization faced any challenges during the implementation of PRMP. If so, how did you respond to those challenges? Q12. What support provided through PRMP to your organization/department was most useful? What was least useful? Please explain why. Q13. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 162 Key Informant Interview Guide Phytosanitary Risk Management Program (PRMP) – Rice Exporters Performance Evaluation of USAID’s Pakistan Agriculture Program (PAP) A1. Date A2. Start Time A3. End Time A4. Province A5. District A6. City A7. Venue83 Informant/ Respondent's Details KII Team A8. Institution/ Organization A12. Moderator's A9. Name Name A10. Designation A13. Note Taker’s Name84 A11. Gender Instructions to the Interviewer 1. Read the introduction. 2. Read the anonymity and consent statement. Introduction Thank you for taking the time to meet us today. My name is _______________________________. Our organization, Associates in Development Pvt. Ltd., has been tasked with conducting the performance evaluation of USAID’s Pakistan Agriculture Program (PAP). One of the sub-activities under focus in this evaluation is the Phytosanitary Risk Management Program (PRMP), implemented by the Centre for Agriculture and Biosciences International (CABI). Our team would like to ask you a few questions regarding the performance of PRMP/ PAP. Your responses will contribute to the evaluation findings, which will help inform the design of similar future United States Government (USG) assistance/ activities. 83 In the case of virtual interaction, this refers to the location of participants, not evaluators. 84 The note-taker will take notes during the meeting, prompt the interviewer if a question has been missed, and prepare a summary in bullet form in this document for feedback from the interviewer. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 163 Anonymity and Consent Statement The interview will take approximately 25 – 30 minutes. We will treat everything you say here fully confidential. We will not use your name in our reports or give your name to anyone outside the research team. Do we have your consent to participate in this interview? ◻ Yes, I will participate ◻ No, I will not participate If no, please ask the reason and record that below. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? ◻ Yes ◻ No If the respondent says yes, continue the interview. If no, try to motivate the respondent by answering their questions and explaining the importance of recording the interview. If the respondent does not agree to the recording, do not record the interview. Preliminary Information 1. Please tell us something about your background. [Prompts: age, occupation, farm size (if relevant), education, and years of experience in agriculture and business]. Evaluation Question No. 1: To what extent have the activities achieved their intended (or unintended) outcomes in the selected sub-activities? Sub-question 1(c): What was the effectiveness of the PRM Program and to what extent did it help to resolve plant health issues in the areas of project interventions which included Punjab, Sindh, Balochistan and Gilgit Baltistan. The effectiveness will be measured by the extent to which the activity was able to mitigate the impact of pests in focused horticulture crops and rice. Mitigating Impact of Pest Q1. Did you receive any support from Phytosanitary Risk Management Program (PRMP)? If yes, please tell us about the support you received from PRMP. Q2. Did the support you receive from PRMP affect (if at all) infestation caused by the targeted insects/pests in rice storage facility/warehouse? If yes, to what extent and how? Q3. Did the support you receive from PRMP affect (if at all) the population of insect/pests in your rice storage facility/warehouse? If yes, to what extent and how? Q4. Did the support you receive from PRMP affect (if at all) the use of chemicals/pesticides in your rice storage facility/warehouse? If yes, to what extent and how? [Please state the frequency of chemical application to control the pest for the year before PRMP support and for the years after receiving PRMP support]. Q5. Did the support you receive from PRMP affect (if at all) your annual income from the rice sales/export? If yes, to what extent and how? How much of this, if any, was due to the reduction in losses? [Please state your annual income for the year before receiving PRMP support and for years after receiving PRMP support]. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 164 Q6. In your opinion, what factors (if any) contributed to mitigating the impact of pests in your rice storage facility /warehouse? Please explain. Prompts: a. Pest management practices b. Availability of pest management information (source). c. Release of biocontrol agents [Natural Enemies Field Reservoirs (NEFRs)] d. Availability of services from extension agents e. Any other factor. Q7. In your opinion, what factors (if any) constrained mitigating the impact of pests in your rice storage facility/warehouse? Please explain. Q8. What other noteworthy changes have you observed (if any) because of support provided to you through PRMP? Evaluation Question No. 2: Have the interventions (or outcomes) introduced under the selected sub-activities been, or are likely to be, sustained? What factors led or deterred sustainability? Q9. Did you continue the use of biocontrol technology after PRPM ended? Please tell us which factors (if any) contributed to or constrained the continued use of biocontrol technology in your rice storage facility/warehouse. Q10. In your opinion, did the on-going flood situation affect the use of biocontrol technology in your rice storage facility/warehouse? If yes, please explain how and to what extent. Evaluation Question 3: What are the lessons learned for future design and mechanism of similar projects? Q11. Please tell us if you faced any challenges in adopting and using the biocontrol technology in your rice storage facility/warehouse? If so, how did you respond to those challenges? Q12. What support provided through PRMP to you was most useful? What was least useful? Please explain why. Q13. How, if at all, could similar future interventions be improved? What practices should be continued and what should be avoided? FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 165 ANNEX VI: SOURCES OF INFORMATION BIBLIOGRAPHY ● CABI (2019). PRMP Project Completion Report August 2014-September 2019. Islamabad. ● CABI (2020). PRMP Post-project Assessment Report, September 2014-September 2019. Islamabad ● CABI (2022, August 25). PRMP Project Presentation June 2014-September 2019. Islamabad, Pakistan ● CIMMYT (n.d.). Wheat Productivity Enhancement Program (WPEP) for Pakistan. Islamabad, Pakistan ● CIMMYT (2015, November 24). WPEP Final Technical Report, 29 September 2010 - 30 September 2015. Islamabad ● CIMMYT. (2017). WPEP Final Technical Report, 1 October, 2015-30 September 2017. Islamabad ● Feder, G., Just, R. E., & Zilberman, D. (1985). Adoption of Agricultural Innovation in Developing Countries Retrieved from World Bank https://documents1.worldbank.org/curated/en/156141468767357523/pdf/multi0page.pdf ● ICARDA (2015). CPEP Annual Progress Report October 2013-September 2014. Islamabad ● ICARDA (2019). ASPTP Quarterly Technical Progress Report, July-September 2019. Islamabad ● ICARDA (2019). CPEP Final Technical Report October 1, 2017 - December 31, 2018. Islamabad ● ICARDA (2021). CPEP Summary of Annual Technical Progress Report, October 2019- September 2020. Islamabad ● ICARDA (2021). Increased Capacity of ASPs to Increase Adoption of Soil Fertility and Health and Improve Water Quality and Conservation Techniques, Final Technical Report, August 2018-July 2019. Islamabad ● ICARDA. (2021). CPEP Annual Technical Progress Report, October 2020-July 2021. Islamabad ● Jan, A. (2018). PRMP Midterm Evaluation Report September 2014-April 2018. Islamabad ● Johnson, M. (2015, May 7). Scaling Agricultural Technologies and Innovation Diffusion. Literature Review Retrieved from USAID Development Experience Clearinghouse: https://pdf.usaid.gov/pdf_docs/pa00kfqg.pdf ● OECD-DAC. (2010). Glossary of Key Terms in Evaluation and Results Based Management. Retrieved from https://www.oecd.org/dac/evaluation/2754804.pdf ● Rashid, M., Husnain, Z., Shakoor, U., & Iftikhar ul Husnain, M. (2020, March 3). Impact of Climate Change on Cotton Production in Pakistan: An ARDL Bound Testing Approach. Retrieved from Researchers Link: http://researcherslinks.com/current-issues/Impact-of-Climate-Change-on￾Cotton-Production-in-Pakistan-An-ARDL-Bound-Testing￾Approach/14/1/2771/html#:~:text=ARDL%20bounds%20test%20confirmed%20variables,negativ e%20effect%20on%20cotton%20production ● USAID (2022). Pakistan Agricultural Programs Evaluation. Statement of Work. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 166 ANNEX VII: LIST OF KEY INFORMANTS Province City Organization/Key Informant Number Role WPEP Key Informants Punjab Bahawalpu r Regional Agricultural Research Institute (RARI) 3 Partner Organization (PO) Faisalabad Wheat Research Institute (WRI) 1 PO Chakwal Barani Agricultural Research Institute (BARI) 1 PO Rahim Yar Khan Seed Producing Companies 2 They were asked questions related to sustainability of seed varieties in terms of commercial production. Murree Crop Disease Research Institute (CDRI) 1 Sindh Sakrand Wheat Research Center (WRC) 1 PO Tando Jam Nuclear Institute for Agriculture (NIA) 1 PO Provincial Seed Council (PSC) 1 The respondent was asked questions related to approval of seed varieties, submitted by WPEP partners. KP Nowshera Cereal Crops Research Institute, (CCRI) Pirsabak 1 PO Dera Ismail Khan Seed Producing Company 1 As above Peshawar Provincial Seed Council (PSC) 1 As above Balochista n Quetta Arid Zone Research Center - Balochistan Agricultural Research and Development Centre (BARDC) 1 PO Islamabad Capital Territory Islamabad International Maize and Wheat Improvement Center (CIMMYT) 1 Implementing Partner Wheat Program, National Agricultural Research Center, Islamabad 1 PO Federal Seed Certification and Registration Department (FSC&RD) 1 They were asked questions related to registration of seed varieties submitted by WPEP partners at FSC&RD. Crop Disease Research Institute (CDRI) 1 PO CPEP Key Informants Punjab Bahawalpu r Agronomy Research Station 1 PO FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 167 Province City Organization/Key Informant Number Role Multan Central Cotton Research Institute 2 PO Cotton Research Institute, (CRI) 1 PO Pakistan Cotton Central Committee (PCCC) 1 The respondent was asked questions related to valuation of seed varieties submitted by CPEP partners at PCCC. Vehari Farmers' Integrated Development Association (FIDA) 1 Partner NGO that was asked questions pertaining to sustainability of Farmer Field School, Women Open Schools, and Children Ecology Club. Faisalabad CRI Research Station 1 PO National Institute of Biotechnology and Genetic Engineering (NIBGE) 3 PO Rahim Yar Khan Seed Producing Companies 3 They were asked questions related to sustainability of seed varieties in terms of commercial production. Sindh Khairpur Women Agricultural Development Organization (WADO) 1 Partner NGO that was asked questions pertaining to sustainability of Farmer Field School, Women Open Schools, and Children Ecology Club. Islamabad Capital Territory Islamabad International Center for Agriculture Research for Dry Areas (ICARDA) 1 Implementing Partner Federal Seed Certification and Registration Department (FSC&RD) 1 They were asked questions related to registration of seed varieties submitted by CPEP partners at FSC&RD. ASPTP Key Informants Irrigated Punjab Lahore South Asian Conservation Agriculture Network (SACAN) 1 PO Rain-fed Punjab Chakwal Soil and Water Conservation Research Institute (SAWCRI), 1 PO Sindh Sakrand Department of Agricultural Extension 1 PO Balochista n Quetta Agriculture Research Institute (ARI) 1 PO Islamabad Capital Territory Islamabad ICARDA 1 Implementing partner Pakistan Council of Research in Water Resources (PCRWR) for Peshawar 1 PO FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 168 Province City Organization/Key Informant Number Role PRMP Key Informants Punjab Lahore Department of Agriculture 1 PO Rice Exporters Association of Pakistan (REAP) 1 Beneficiaries were asked questions related to PRMP’s effectiveness in terms of mitigating impact of pests, and sustainability of the results that may have accrued because of PRMP interventions. Sindh Karachi Key papaya farmers who received support from PRMP 3 Beneficiaries were asked questions related to PRMP’s effectiveness and sustainability of outcomes Southern Zone Agricultural Research Centre (SARC), Karachi 1 PO Department of Agriculture Extension 1 PO GoP officers trained by PRMP 2 Beneficiaries were asked questions related to PRMP’s effectiveness in mitigating the impact of pests in their area, and institutionalization/sustainability of PRMP’s interventions (or outcomes) in their respective departments. Thatta Key papaya farmers who received support from PRMP 1 As above Balochista n Quetta Directorate of Agriculture Research 3 PO GoP officers trained by PRMP 2 As above Khanozai Key apple farmers who received support from PRM 1 As above Gilgit Baltistan Gilgit Department of Agricultural Research 3 PO GoP officers trained by PRMP 1 As above Key fruit tree farmers who received support from PRMP 1 As above Skardu Department of Agriculture Extension 2 PO GoP officers trained by PRMP 1 As above Key fruit tree farmers who received support from PRMP 1 As above FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 169 Province City Organization/Key Informant Number Role Key willow farmers who received support from PRMP 3 As above Islamabad Capital Territory Islamabad Center for Agriculture and Biosciences International (CABI) 1 Implementing partner TOTAL 70 KIIs These exclude 18 ASPs interviewed for ASPTP and elaborated below LIST OF ASPS AND FARMERS (ASPTP) Province City Agricultural Technology Introduced/ Provided Beneficiaries Type Number of KIIs/GIs Balochistan Killa Saifullah/ Ziarat Drip/ Bubbler/ Sprinkler Irrigation System ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 Usta Mohammad Zero-Till (ZT) Drill ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 KP Charsadda/ Mardan Laser Land Leveling ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 Punjab Faisalabad/ Sheikhupura Bed Planting of Crops ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 170 Province City Agricultural Technology Introduced/ Provided Beneficiaries Type Number of KIIs/GIs Faisalabad/ Sialkot Fertilizer Prediction Model ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 Sialkot/ Vehari/ Gujranwala Rapid Soil Testing Kit ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 Sheikhupura Happy Seeder ZT Drill ASP trained through ASPTP KII 1 Chakwal Gypsum to Conserve Soil Moisture ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 Khanewal/ Chakwal Biozote to Increase Yield of Crops ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 Sindh Nawabshah ASP trained through ASPTP KII 1 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 171 Province City Agricultural Technology Introduced/ Provided Beneficiaries Type Number of KIIs/GIs Ridge Planting of Crops/Laser Land Leveling 3 farmers supported by ASP Group Interview ASP trained through ASPTP KII 1 3 farmers supported by ASP Group Interview 1 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 172 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST The disclosure forms have been removed from this version of the report uploaded at DEC to ensure privacy of personal identifiable information. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 173 ANNEX IX: EVALUATION TEAM MEMBERS TEAM LEADER/ EVALUATION SPECIALIST Ghazanfar Hoti - M.S. (Operations Research), 2011, and MPA (Economic Policy Management), 2008, Columbia University New York, USA: Mr. Ghazanfar Hoti is an experienced monitoring and evaluation (M&E) specialist with more than 20 years of professional experience. Most of his work experience revolves around designing, managing and leading evaluations and assessments in Pakistan. He has authored/ co-authored several international-quality evaluation reports, including those of projects funded by the bilateral aid agencies of the United States, Switzerland, and the United Kingdom. His sectoral and thematic experience includes agriculture development, economic growth, community mobilization, countering violent extremism (CVE), education, energy, gender and women’s empowerment, governance, stabilization, and health. QUALITATIVE RESEARCH SPECIALIST/QUALITATIVE DATA ANALYST Mohammad Aftab Ismail Khan – MBA, 2002, Pepperdine University, Malibu, California, USA: Mr. Aftab Khan has worked in development for over 20 years and his experience has revolved around designing, managing, and conducting evaluations, reviews, and studies; program development and coordination; project analysis, implementation, and management; communications; needs assessments; and report writing. His thematic experience includes governance; local state-building; conflict resolution; agriculture; infrastructure; environment; education; gender; health; trade; social protection; child protection; disaster relief, management, reconstruction, and rehabilitation; and poverty alleviation. His client experience includes USAID, DEC, FCO, SDC, UN agencies, GEF, ITC, IRW, CARE, Mercy Corps, IUCN, AKRSP, Oxfam-GB; ERRA, NDMA, and the Government of Gilgit-Baltistan. QUALITATIVE RESEARCH SPECIALIST/ QUALITATIVE DATA ANALYST Fatima Abbas – Master in Public Policy (International Development), 2011, National University of Singapore, Singapore: Ms. Fatima Abbas has 12 years of experience in development. Her skills include international-standard evaluation and qualitative research, team management, stakeholder coordination and high-quality report writing. Her sector and thematic experience spans poverty graduation, education, health, agriculture, transport and urban planning, aid governance, microfinance, digital finance and financial inclusion, water and sanitation, youth development, industrial growth, economic growth, and energy, with gender and women empowerment as an area of specialization in all sectors. She has worked on projects funded by the USAID and the U.S. State Department, the World Bank, UNICEF, UNESCAP, IFAD, Bill and Melinda Gates Foundation, EU, GIZ, State Bank of Pakistan, Telenor, Akhuwat and academic institutions including Harvard University, Oxford University, National University of Singapore, and Lahore University of Management Sciences (LUMS). AGRICULTURE ECONOMIST & EVALUATION SPECIALIST Dr. Tariq Husain - Ph.D. (Economics), 1987 (dissertation in Agricultural Economics), University of Chicago: Dr. Tariq Husain has worked in development for 40 years, including 30 years as a consultant, in 21 countries. Most of his experience has revolved around programme design and strategy, evaluating operations and results, and helping institutions adapt to change. Dr. Husain is the author of 25 international publications and more than 30 articles on development and governance in the print media; and former member of the United Nations Development Programme (UNDP)’s Gender Advisory Panel for Asia-Pacific. High-level contributions include: the Pakistan country report presented FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 174 by the President at the World Summit on Social Development (1995); the country progression report presented by civil society at the World Summit on Sustainable Development (2002); Pakistan’s first progress report on the Millennium Development Goals (2003); corporate evaluation policy of the International Fund for Agricultural Development (IFAD); IFAD country evaluations of Sri Lanka and Turkey, its work across 17 countries in Asia-Pacific, and its contributions to decentralization in Eastern Africa; and assistance to 18-agency working group on agriculture, rural development and poverty to operationalize the One UN approach in Pakistan. Sector and thematic experience include agriculture research and extension, water management, forestry, governance, political economy, community￾driven development, rural development, social sectors (including social protection), environment, gender, trade, and the informal economy. SEED CERTIFICATION EXPERT Ilhammudin - M.S. Seed Technology/Agronomy, July 1993, Mississippi State University, USA: Mr. Ilhammudin brings more than 30 years of hands-on experience in seed certification and registration. He has worked in the Federal Seed Certification and Registration Department as Deputy Director and oversaw Regional Directorate with responsibilities including coordination and supervision of technical and administrative matters of the department in different regions. Additionally, Mr. Ilham has also been providing consultancy services to the public and private sector in the fields of seed technology (variety registration & release system in Pakistan, seed production, seed quality control, seed conditioning, and seed storage). He has co-authored and contributed to multiple research publications on seed technology. ENTOMOLOGIST/PLANT PESTS CONTROL EXPERT Dr. Farman Ullah - Ph.D. Oklahoma State University, 1993, USA: Dr. Farmaullah is currently serving as Professor at the University of Agriculture Peshawar. He has more than 30 years of teaching experience, where he supervised doctoral and postgraduate students on the subjects of Sanitary and Phytosanitary Measures (SPS), Pesticides Resistance Management and Pest of Crops. Moreover, he also brings extensive experience of managing research projects on sectors including Pest Risk Analysis and IPM of Maize, Pest Management of Wheat Aphids, Mass Production of Trichogramma-IPM of Maize Stem Borer, and Insect Pests Management of Peach. FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 175 ANNEX X: DETAILS OF APPROVED AND CANDIDATE WHEAT VARIETIES BY PARTNER RESEARCH INSTITUTES WHEAT RESEARCH INSTITUTE - FAISALABAD: APPROVED/RELEASED VARIETIES V. Code Variety Name Parentage/Pedigree When was NUWYT Year 1 and Year 2 conducted Origin/ selection material/year Year of Release V-04178 AARI￾2011 SHALIMAR88/90A204//MH-97 PBP28645-21A-6A-3A-1A-0A-8A-0A 2007-08 & 2008-09 1996-97 Three ways cross 2011 V-05066 Punjab￾2011 AMSEL/ATTILA//INQ.91/PEW 'S' Pb.30196-1A-0A-2A-0A 2008-09 & 2009-10 1998-99 Hybridization 2011 V-08203 Ujala-16 KIRITATI/4/2*WEAVER/TSC//WEAV ER/3/WEAVER CGSS02B00125T-099B-099Y-099M￾099Y-4WGY-0B 2011-12 & 2012-13 2007-08 2016 V-11098 Anaj￾2017 BABAX/LR43//BABAX/6/MOR/VEE# 5//DUCULA/3/DUCULA/4/MILAN/5 /BAU/MILAN7/7SKAUZ/BAV92 CMSS05Y00558T-099TOPM-099Y￾099M-099Y-099ZTM-7WGY-0B 2014-15 & 2015-16 5 th SRRSN/2010-11 2017 N/A Akbar-19 BECARD/QUAIU#1 CMSS07B00230S-099M-099NJ￾099NJ-23WGY-0B N/A 34th ESWYT, Entry #139 2019 V-16005 Dilkash￾2020 WBLL*2/4/SNI/TRAP#1/3/KAUZ*2/ TRAP//KAUZ/5/PB.96//LU26/HD21 79 PB.35237-0A-0A-5A-0A-0A 2018-19 & 2019-20 2008-09/ Hybridization 2021 HYT-55-33 Subhani￾2021 KACHU/SAUAL*2/5/SERI.1B//KAUZ /HEVO/3/AMAD*2/4/KIRITATI CMSS10B010307-099TOPY-099M￾099NJ-WGY-0B 2018-19 & 2019-20 SABWGPYT - 07/2015-16 2021 V-12304 MH￾2021 WAXWING/4/SNI/TRAP#1/3/KAUZ */TRAP//KAUZ/4/TECUE#1 CMSS06B00468S-0Y-099ZTM-099Y￾099M-1WGY-0B 2014-15 & 2015-16 CSISA-EM/ 2011-12 2021 V-17179 Arooj￾2022 SOKOLL/3/PASTOR//HXL7573/2*B AU*2/6/OASIS/5*BORL95/5/CNDO /R143//ENTE/MEX175/3/AW.SQ/4 /2*OCI CMSA10M00159T-050Y-099ZTM￾099NJ-099NJ-5WGY-0B 2019-20 & 2020-21 24th SAWYT/ 2016-17 2021 WHEAT RESEARCH INSTITUTE - FAISALABAD: CANDIDATE LINES V. Code Parentage/Pedigree Origin/selection material/year Status FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 176 V-19347 BORL14*2//BECARD/QUAIU #1 CMSS12B00634T-099TOPY-099M-0SY-26M-0WGY 17th HTWYT/ 2018- 19 2 nd year of NUWYT HYT-100-47 SNTL/3/KACHU//WBLL1*2/BRAMBLING CMSS12B00561S-099M-099NJ-099NJ-3Y-0WGY SABWGPYT-04 2018-19 2 nd year of NUWYT HYT-100-74 MUCUY/BORL14//MUCUY CMSS12B00870T-099TOPY-099M-099NJ-099NJ-16Y￾0WGY SABWGPYT-05 2018-19 2 nd year of NUWYT HYT-100-76 BECARD #1/5/KIRITATI/4/ 2*SERI.1B*2/3/KAUZ*2/BOW//KAUZ*2/6/KFA/2*KACHU CMSS12B00911T-099TOPY-099M-099NJ-099NJ-31Y￾0WGY SABWGPYT-05 2018-19 2 nd year of NUWYT V-19532 KUTZ*2/5/UP2338*2/VIVITSI/3/FRET2/ TUKURU//FRET2/4/MISR 1 CMSS12Y00813T-099TOPM-099Y-099M-0SY-1M-0WGY 36th SAWSN/ 2018- 19 1 st year of NUWYT V-20330 MANKU/BOKOTA CMSS14Y00846S-099Y-099M-099Y-7M-0WGY 10th HPYT/ 2019-20 1 st year of NUWYT V-20337 VILLA JUAREZ F2009/3/T.DICOCCON P194625/AE.SQUARROSA(372)//3*PASTOR/4/WBLL1*2/B RAMBLING/8/PSN/BOW//SERI/3/MILA/4/ATTILA/5/KAUZ *2/CHEN//BCN/3/MILAN/6/WBLL1*2/SHAMA/7/IWA860 0211//2*PBW343*2/KUKUNA/9/SUP152/BLOUK#1 CMSS14Y01209T-099TOPM-099Y-099M-099Y-6M-0WGY 10th HPYT/ 2019-20 1 st year of NUWYT Barani Agriculture Research Institute - Chakwal: Approved/Released Varieties V. Code Variety Name Parentage/Pedigree When was NUWYT Year 1 and Year 2 conducted? Origin/Selec tion Material/tri al /year * Year of Release 6C016 Dharbi 11 HXL7573/2*BAU// PASTOR CMSS97Y03676S-040Y-050M￾040SY- 030M-21SY-010M-0Y￾0SY 2009-10 & 2010-11 CIMMYT 23rd SAWSN entry no. 127 year 2005-6 2011 6C002 Ihsan-16 Pastor/3/Altar84/Ae.Sq.//Opata CMSS97M00316S-040M-040SY￾030M-040SY-27M-0Y￾0SY 2013-14 2014-15 CIMMYT (SAWSN-W 2005, Ent#62) 2016 11C023 Barani 17 SOKOLL / EXCALIBUR CMSA04Y00612S-28ZTPOY￾010M-010SY-4M-02Y-0B 2014-15 2015-16 CIMMYT (SAWYT 2010-11, Ent#49) 2017 16C038 MA-2021 TRCH/SRTU//KACHU*2/3/KING BIRD #1 CMSS10Y00880T-099TOPM￾099Y-099M-7WGY-0B 2018-19 2019-20 CIMMYT (SAWYT 2015-16, Ent#44) 2021 Barani Agriculture Research Institute - Chakwal: Candidate Varieties FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 177 V. Code Parentage/Pedigree Origin/Selec tion Material/ /year * Status** 18C117 BECARD/QUAIU #1//BORL14 CMSS11B00516S-099M-0SY-18M 0WGY CIMMYT ESWYT (2017-18) Entry 40 year 2018 Two years NUWYT completed, DUS COMPLETED, Spot examination will be in year 2023 19C166 SAUAL/4/CROC_1/AE.SQUARROSA(205)//K AUZ/3/ATTILA/5/SAUAL/8/TACUPETOF200 1/6/CNDO/R143//ENTE/MEXI_2/3/AEGILO PSUARROSA(TAUS)/4/WEAVER/5/PASTOR/ 7/ROLF07/9/SAUAL/YANAC//SAUAL CMSS11Y01008T-099TOPM-099Y-099M￾0SY-16M-0WGY CIMMYT SAWSN-78 (2017-18) Entry 78 year 2019 Currently in NUWYT 1st year 2022-23 20C207 MUTUS*2/MUU//2*MUCUY CMSS13Y01148T-099TOPM-099Y-099M￾0SY-21M-0WGY CIMMYT ESWYT-33 (2019-20) Entry 33 year 2020 Currently in NUWYT 1st year 2022-23 21C244 QUAIU/MUNAL//QUAIU #2/3/SUP152/BAJ #1 CMSS14Y00266S-099Y-099M-0SY-10M￾0WGY CIMMYT ESWYT 2020-21 ENTRY #125 YEAR 2021 IN PROVINCIAL YIELD TRIAL 2022-23 21C252 MUCUY*2//SUP152/BAJ #1 CMSS13B01217T-099TOPY-099M-099NJ￾099NJ-27Y-0WGY CIMMYT SAWYT 2020-21- ENTRY 313 YEAR 2021 IN PROVINCIAL YIELD TRIAL 2022-23 21C253 MUTUS*2//TAM200/TURACO*2/3/KFA/2 *KACHU CMSS14B01483T-099TOPY-099M-0SY-41M￾0WGY CIMMYT SAWYT￾2020-21 ENTRY 342 YEAR 2021 IN PROVINCIAL YIELD TRIAL 2022-23 Cereal crops Research Institute - Pirsabak: Approved/Released Varieties V. Code Variety Name Parentage/Pedigree When was NUWYT Year 1 and Year 2 conducted? Origin/Sele ction Material/tr ial /year * Year of Release PR-125 Pirsabak-2021 KACHU/3/PBW343*2/KUKUNA //PBW343*2/KUKUNA CMSS09B00277S-099ZTM￾099NJ-099NJ-19WGY-0B 2018-19 2019-20 ESWYT V-28 (2015-16) 2021 PR-127 Zarghoon- 2021 SOKOLL/3/PASTOR//HXL7573/ 2*BAU*2/4/PASTOR//MILAN/ KAUZ/3/BAV92 CMSA09M00229T-050Y￾050ZTM-050Y-2WGY-0B 2018-19 2019-20 HTWYT V-32 (2015-16) 2021 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 178 PR-128 Abaseen-2021 SALEEM-200 X PS-05 CCSS10-11S-OCC-OK-5006CC1- OK-7016CC1-OK—9005CC1 2018-19 2019-20 Fix Line 9005 (2014-15) 2021 PR-119 Pirsabak-2019 NAC/TH.AC//3*PVN/3/MIRLO/ BUC/4/2*PASTOR/5/KACHU/6 /KACHU CMSS06B00734T-099TOPY￾099ZTM-099Y-099M-13WGY￾0B 2016-17 2017-18 SRN V-113 (2011-12) 2019 PR-112 Wadaan-17 YAV79//DACK/RABI/3/SNIPE/4 / AE. SQUARROSA CMSA04Y00649S-028CRE1Y￾010M-3SY-03CRE1Y-010M￾03Y-0B 2014-15 2015-16 SAWSN V￾166 (2010- 11) 2017 PR-110 Khaista-17 KAUZ//ALTAR 84/AOS/3/MILAN/KAUZ/4/HUI TES/7/ CAL/NH//H567.71/3/SERI/4/C AL/ NH//H567.71 /5/2*KAUZ/6/PASTOR CMSS05B00581S-099Y-099M￾099Y-099ZTM-2WGY-0B 2014-15 2015-16 ESWYT V￾17 (2011- 12) 2017 PR-103 Pukhtoonkhwa￾15 WBLL1*2/4/YACO/PBW65/3/K AUZ*2/TRAP//KAUZ CGSS01Y00054T-099M-099Y￾099M-099M-20Y-0B 2012-13 2013-14 ESWYT V￾48 (2009- 10) 2015 PR-105 Pirsabak-15 MILAN/S87230//BABAX CMSS97M03689T-040Y-030M￾020Y-030M-015Y-30M-3Y-1M￾0Y 2012-13 2013-14 ESWYT V￾11 (2007-08) 2015 PR-102 Pirsbak-13 CS/TH.SC//3*PVN/3/MIRLO/B UC/4/MILAN/5/TILHI CMSS97M04005T-040Y-020Y￾030M-020Y-040M-28Y-3M-0Y 2009-10 2010-11 ESWYT V￾120 (2006-07) 2013 PR-98 Shahkar-13 CMH84.3379/CMH78.578//MI LAN CMSS93Y006285-7Y-010Y￾010M-010Y-10M0Y-3KBY￾0KBY 2008-09 2009-10 IBWSN V￾90 (2004-05) 2013 Cereal Crops Research Institute - Pirsabak: Candidate Varieties V. Code Parentage/Pedigree Origin/Selecti on Material/ /year* Status** PR-142 KAUZ/ALTAR84/AOS/3/MILAN/KAUZ/4/…. / PR-105 CCSS-13-14S-0CC-0K-5077CC2-0K-7064CC4- OCC Fix line V-8327 (2017- 18) NUWYT- 2nd Year PR-146 SOKOLL/3/PASTOR//HXL7573/2*BAU*2/6/ OASIS/5*BORL95/5/CNDO/R143//ENTE/M EXI75/3/AE.SQ/4/2*OCI SAWYT V-4 (2018-19) NUWYT- 1st Year FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 179 CMSA10M00159T-050Y-099ZTM-099NJ￾099NJ-5WGY-0B PR-147 MUNAL*2/WESTONIA/4/KACHU/3/WHEAR //2*PRL/2*PASTOR CMSS14B00334S-099M-0SY-2M-0WGY ESWYT V-30 (2020-21) NUWYT- 1st Year PR-148 Pirsabak-13/ Khaista-17 CCSS-14-15S-0CC-5271CC14-0CC-7066CC2- 0CC FIX Line 8065 (2018-19) NUWYT- 1st Year PR-149 BORL14*2/8/REH/HARE//2*BCN/3/CROC_ 1/AE.SQUARROSA (213)//PGO/4/HUITES/5/ T.DICOCCON PI94624/AE.SQUARROSA (409)//BCN/6/REH/HARE// 2*BCN /3/ CROC_1/AE.SQUARROSA (213)//PGO/4/ HUITES/7/ MUTUS SABWGPYT-5 V-35 (2019-20) NUWYT- 1st Year PR-150 WHEAR/VIVITSI//WHEAR/3/FRNCLN/5/SH ORTENED SR26 TRANSLOCATION/4/3*CHIBIA//PRLII/CM65 531/3/MISR 2/6/BORL14 SBWGPYT-9 V-19 (2019-20) NUWYT-1st Year PS-88 SOKOLL/3/PASTOR//HXL7573/2*BAU*2/6/ OASIS/5*BORL95/5/CNDO/R143//ENTE/M EXI75/3/AE.SQ/4/2*OCI CMSA10M00159T-050Y-099ZTM-099NJ￾099NJ-5WGY-0B SAWYT V-4 (2018-19) KPWYT- 1 st Year PS-89 MUNAL*2/WESTONIA/4/KACHU/3/WHEAR //2*PRL/2*PASTOR CMSS14B00334S-099M-0SY-2M-0WGY ESWYT V-30 (2020-21) KPWYT- 1 st Year PS-90 Pirsabak-13/ Khaista-17 CCSS-14-15S-0CC-5271CC14-0CC-7066CC2- 0CC FIX Line 8065 (2018-19) KPWYT- 1 st Year PS-91 BORL14*2/8/REH/HARE//2*BCN/3/CROC_ 1/AE.SQUARROSA (213)//PGO/4/HUITES/5/ T.DICOCCON PI94624/AE.SQUARROSA (409)//BCN/6/REH/HARE// 2*BCN /3/ CROC_1/AE.SQUARROSA (213)//PGO/4/ HUITES/7/ MUTUS SABWGPYT-5 V-35 (2019-20) KPWYT- 1 st Year PS-92 WHEAR/VIVITSI//WHEAR/3/FRNCLN/5/SH ORTENED SR26 TRANSLOCATION/4/3*CHIBIA//PRLII/CM65 531/3/MISR 2/6/BORL14 SBWGPYT-9 V-19 (2019-20) KPWYT- 1 st Year PS-93 KACHU//KIRITATI/2*TRCH/3/KFA/2*KACH U CMSS13B00118S-099M-0SY-1M-0WGY HTWT V-41 (2019-20) KPWYT- 1 st Year PS-94 MERCATO/BECARD//BOKOTA CMSS12Y01173T-099TOPM-099Y-099M￾0SY-24M-0WGY IBWSN V-127 (2018-19) KPWYT- 1 st Year PS-95 PR-106/Trap Nursery Ent- 29 (2013-14)// PAKHTUNKHWA-15 /NARC-11 FIX Line 8186 (2018-19) KPWYT- 1 st Year FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 180 PS-96 MUTUS*2/MUU//2*MUCUY CMSS13Y01148T-099TOPM-099Y-099M￾0SY-16M-0WGY HTWYT V-31 (2019-20) KPWYT- 1 st Year PS-97 MUU/FRNCLN//FRANCOLIN #1/3/BORL14 CMSS13B00273S-099M-0SY-36M-0WGY HTWYT V-43 (2019-20) KPWYT- 1 st Year National Agriculture Research Center- Islamabad: Approved/Released Varieties V. Code Variety Name Parentage Pedigree When was NUWYT Year 1 and Year 2 conducted? Origin/Selecti on Material/ /year * Year of Release NR-356 NARC￾11 OASIS/SKAUZ//4*BCN /3/2*PASTOR CMSS00Y01881T￾050M-030Y￾030M-030WGY￾33M-0Y-01D 2008-09 and 2009-10 EBWYT 2006 Entry No. 514 2011 NR-397 Pakista n-13 MAX94.27.1.20/3/SO KOLL//ATTILA/3*BCN PTSS02B00132T￾0TOPY-0B-0Y-0B￾38Y-0M-0SY-0ID 2010-11 and 2011-12 RFUG 99RE 09 CIMMYT Selection # 11 (2009) 2013 NR-399 Borlaug -16 SOKOLL/3/PASTOR// HXL7573/2*BAU PTSS02B00015S￾0Y-0B-0Y-0B-1Y￾0M-0SY-0ID 2011-12 and 2012-13 RFUG99RE 09 CIMMYT Selection#08 (2009-10) 2016 NR-421 Zincol - 16 OASIS/SKAUZ//4*BCN /3/2*PASTOR/4/T.SP ELTAPI348449/5/BA V92/3/OASIS/SKAUZ //4*BCN/4/PASTOR/ 6/ WBLL1*2/CHAPIO CMSS07Y01302T￾099Y-19M-0Y￾2B-0Y-0ID 2012-13 and 2013 -14 6HPWYT (2010) Entry # 412 (BV2010\HP South Asia\34) 2016 NR-449 Markaz￾19 SOKOLL//FRTL/2*PIFE D CMSA04M00552S -040ZTP0Y￾040ZTM-040SY￾19ZTM-03Y-0B￾0ID 2014-15 and 2015-16 19th -SAWYT (2011) # 332 CIMMYT 2019 NR--499 NARC Super MUTUS*2/HARIL #1 CMSS08Y00871T￾099TOPM-099Y￾099M-099NJ￾7WGY-0B-0ID 2017-18-2018- 19 8 th EBWYT (2013) #18 2021 National Agriculture Research Center - Islamabad: Candidate Varieties FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 181 Entry Name Parentage Pedigree Origin Year in NUYT NR-546 HUIRIVIS #1*2/MURGA/3/TACUPETO F2001/BRAMBLING*2//KAC HU CMSS11Y00343S-099Y-099M￾099NJ-099NJ-11WGY-0B ESWYT 2017 E#7 1 st NR-549 UP2338*2/SHAMA/3/MILAN /KAUZ//CHIL/CHUM18/4/UP 2338*2/SHAMA*2/5/PBW34 3*2/KUKUNA*2//FRTL/PIFED CMSS11Y00993T-099TOPM￾099Y-099M-099NJ-099NJ￾2WGY-0B ESWYT 2017 E#14 2nd NR-552 NADI/COPIO//NADI CMSS11B00910T-099TOPY￾099M-099NJ-099NJ-37WGY￾0B SAWYT 2017- 18 E# 43 1 st Nuclear Institute of Agriculture - Tandojam: Approved/Released Varieties V. Code Variety Name Parentage/Pedigree When was NUWYT Year 1 and Year 2 conducted? Origin/Selectio n Material/ /year * Year of Release NIA Sundar Sarsabz/Sunco *2 2008-09-2009-2010 NUWYT Farmers demo trials in 2011-13 and subsequent release Varietal popularization trials in 2013-15 through WPEP program Local cross of srasbaz with CIMMYT germplasm Sunco 2011 NIA Saarang SHA 4/WEAVER//SKAUZ *2/SRMA 2007-08, 2008-09 NUWYT Farmers demo trials 2011-13 Varietal popularization trials in 2013-15 through WPEP program F3 BME4TE CIMMYT nursery 2013 NIA- Sunehri Cham4//Ures/Bow ‘S’ 2003-2005 NUWYT trials were conducted. Selection from ICW91 Nursery 2010 NIA Zerkhaiz KAUZ/GEN 2013-14, 2014-15 NUWYT Farmers demo trials 2017-2018 Varietal popularization trials in 2019-20 through WPEP program ESWYT, 94/95 2020 NIA Shaheen PBW 343*2/KONK 2014-15-2015-16 NUWYT Farmers demo trials 2017-2018 Varietal popularization trials in 2019-20 through WPEP program F2 selection 0f BME4TE nursery 2004 2020 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 182 Nuclear Institute of Agriculture - Tandojam: Candidate Varieties V. Code Parentage/Pedigree Germplasm received from CIMMYT Current stage of candidate lines ** NBW-232 TUKURU//BAV92/RAYON/6/NG8201/KAUZ /4/SHA7//PRL/VEE#6/3/FASAN/5/MILAN/K AUZ/7/TRCH/SRTU//KACHU 7TH SATYN Cycle 2017- 18 Being evaluated in 2nd Year NUWYT SD-1060 KAUZ//ALTAR84/AOS3/KAUZ/3/ATTILA/SO Y1/ATTILA/BCN/4/PASTOR-6 16TH ESBWYT; 2015-16 cycle DUS evaluation Completed NUWYT 2 years. Farmers demo trials are being conducted at large scale SD-1040 FRNCLN/ROLF07 33ESWYT CIMMYT TRIAL 2012- 13 cycle Completed NUWYT 2 years. Farmers demo trials are being conducted at large scale DF-1708 SAUAL/YANAC//SAUAL*2/3/TACUPETO F2001 11TH STEMRRSN: 2016-17 cycle Completed 2 Years of NUWYT evaluation EST-28/11 Selection from CIMMYT trial ESWYT 28TH ENTRY no. 11 ESWYT-28 no.11 Cycle 2007- 08 Completed 2 Years of NUWYT evaluation EST-29/9 Selection from CIMMYT trial ESWYT 29TH ENTRY no. 9 ESWYT-29 no 9 Cycle 2008- 09 Completed 2 Years of NUWYT evaluation SDW-4 BAKOTA/3UP2338*2KKTS*2/YANAC ESWYT 37 2016-17 Cycle completed evaluation in Zonal trials & ranked 1st Regional Agricultural Research Institute - Bahawalpur: Approved/Released Varieties V. Code Variety Name Parentage/Pedigree When was NUWYT Year 1 and Year 2 conducted? Origin/Selection Material/ /year * Year of Release 032862 AAS-11 PRL/PASTOR//V-6550/Sutlej-86 2006-07 & 2007-08 Local cross involving 1 exotic and 1 local parent 2011 076346 Gold-16 PR-32 (BAU)/Inqalab-91 (Exotic Local) 2011-12 & 2012-13 Local cross involving 1 exotic and 1 local parent 2016 099172 Jauhar-16 KAUZ/PASTOR//V.3009 (Exotic Local) 2013-14 & 2014-15 Local cross involving 1 exotic and 1 local parent 2016 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 183 122559 Ghazi-19 CNO79//PF70354/MUS/3/PASTOR/4 /BAV92/5/FRET2/… 2015-16 & 2016-17 Selection from 6HTWSN 2011- 12 (CIMMYT) 2019 BF-1705 Nawab￾21 HG094.7.1.12/2*QUAIU#1/3/VILLA JUAREZ F2009/SOLALA//… 2019-20 & 2020-21 Selection from 7-HPYT, 2016-17 2021 172190 Sadiq-21 QAIU/5/UP2338*2/SHAMA/3/MILAN /KAUZ/CHIL/… 2019-20 & 2020-21 Selection from 37ESWYT 2016- 17 (CIMMYT) 2021 Regional Agricultural Research Institute - Bahawalpur: Candidate Varieties V. Code Parentage/Pedigree Origin/Selection Material/ /year * Status** BF-1807 HG094.7.1.12/2*QUAIU#1/3/VILLA JUAREZ F2009/SOLALA//… Selection from 8-HPYT, 2017-18 (CIMMYT) Completed NUWYT, DUS trial under process 181601 V-3010/V-6377 Local cross involving both local parents Completed NUWYT, DUS trial under process 180059 BAV92//IRENA/KAUZ/3/HUITES*2/4/C ROC… Selection from CIMMYT germplasm *** NUWYT 2nd year HYT-100-95 OASIS/5*BORL95/5/CNDO/R143//ENT E/MEXI75/3/AE.SQ/4/2*OCI/6/KACHU /BECARD//WBLL1*2/BRAMBLING/7/K ACHU/BECARD//WBLL1*2/BRAMBLIN G Selection from CIMMYT germplasm (HYT-100, 2020- 21) NUWYT 1st year BF-20105 C80.1/3*BATAVIA//2*WBLL1/3/ATTIL A/3*BCN*2//BAV92/4/WBLL1*2/KUR UKU/5/IWA8600211//2*PBW343*2/K UKUNA/6/BORL14/7/BOKOTA Selection from CIMMYT germplasm (10-HPYT, 2019- 20) NUWYT 1st year ***information pending till Monday please, for nursery name and year Wheat Research Institute - Sakrand: Approved/Released Varieties V. Code Variety Name Parentage/Pedigree When was NUWYT Year 1 and Year 2 conducted? Origin/Sele ction Material/tr ial /year * Year of Release N/A Sindu-16 N/A N/A N/A 2021 N/A IV-II N/A N/A N/A 2021 Wheat Research Institute - Sakrand: Candidate Varieties FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 184 V. Code Parentage/Pedigree Origin/Sele ction Material/ /year * Status** SR- 6026 N/A N/A To be proposed to Technical Committee of PSC E-107 N/A N/A NEWYT Year 2 HT-50 NEWYT Year 2 HT-25 NEWYT Year 2 HP-01 NEWYT Year 2 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 185 ANNEX XI: LIST OF APPROVED VARIETIES IN PAKISTAN S. No Variety Name Breeding Centre Province Year of Release 1 Blue Silver WRI, AARI, Faisalabad Punjab 1971 2 Lyallpur 73 WRI, AARI, Faisalabad Punjab 1973 3 Pari 73 WRI, AARI, Faisalabad Punjab 1973 4 Sandal 73 WRI, AARI, Faisalabad Punjab 1973 5 Nuri 70 WRI, AARI, Faisalabad Punjab 1975 6 Yacora 70 WRI, AARI, Faisalabad Punjab 1975 7 ARZ PARC, Islamabad Islamabad 1976 8 LU 26 Univ. Agri. Faisalabad Punjab 1977 9 Pavon WRI, AARI, Faisalabad Punjab 1978 10 Indus 79 WRI, AARI, Faisalabad Punjab 1979 11 Jauhar 78 AEARC, Tandojam Sindh 1979 12 ZA 77 ARI, Tandojam Sindh 1979 13 Zarghoon 79 ARI, Sariab, Quetta Balochistan 1979 14 Bahawalpur 79 RARI, AARI, Bahawalpur Punjab 1980 15 Khyber 79 CCRI, Pirsabak NWFP 1980 16 Zamindar 80 ARI, Sariab, Quetta Balochistan 1980 17 Pak 81 CCRI, Pirsabak/AARI, Faisalabad NWFP/PB 1981 18 Punjab 81 WRI, AARI, Faisalabad Punjab 1981 19 Sindh 81 NIA, Tandojam Sindh 1982 20 Faisalabad 83 WRI, AARI, Faisalabad Punjab 1984 21 Kohinoor 83 WRI, AARI, Faisalabad Punjab 1984 22 Sarhad 82 CCRI, Pirsabak NWFP 1984 23 TJ 83 ARI, Tandojam Sindh 1984 24 Barani 83 WRI, AARI, Faisalabad Punjab 1985 25 Faisalabad 85 WRI, AARI, Faisalabad Punjab 1985 26 Punjab 85 WRI, AARI, Faisalabad Punjab 1985 27 Wadanak 85 WRI, AARI, Faisalabad Punjab 1985 28 Pirsabak 85 CCRI, Pirsabak NWFP 1986 29 Sarsabz NIA, Tandojam Sindh 1986 30 Sutlej 86 RARI, AARI, Bahawalpur Punjab 1986 31 Chakwal 86 WRI, AARI, Faisalabad Punjab 1988 32 Khyber 87 CCRI, Pirsabak NWFP 1988 33 Rawal 87 WRS, BARI, Rawalpindi Punjab 1988 34 Pasina 90 CCRI, Pirsabak NWFP 1990 35 Inqlab 91 WRI, AARI, Faisalabad Punjab 1991 36 Mehran 89 ARI, Tandojam Sindh 1991 37 Pasban 90 WRI, AARI, Faisalabad Punjab 1991 38 Rohtas 90 WRI, AARI, Faisalabad Punjab 1991 39 Soghat 90 NIA, Tandojam Sindh 1991 40 Pirsabak 91 CCRI, Pirsabak NWFP 1992 41 Anmol WRI, Sakrand Sindh 1993 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 186 S. No Variety Name Breeding Centre Province Year of Release 42 Bakhatawar 93 NIFA, Peshawar NWFP 1993 43 Sariab 92 ARI, Sariab, Quetta Balochistan 1993 44 Zardana ARI, Sariab, Quetta Balochistan 1993 45 Perwaz 94 WRI, AARI, Faisalabad Punjab 1994 46 Pothowar 93 WRS, (BARI), Rawalpindi Punjab 1994 47 Abadgar 93 WRI, Sakrand Sindh 1996 48 AZRI 96 AZRC, Quetta Balochistan 1996 49 Kiran 95 AEARC, Tandojam Sindh 1996 50 Kohsar 95 WRS, (BARI), Rawalpindi Punjab 1996 51 Nowshera 96 CCRI, Pirsabak NWFP 1996 52 Punjab 96 WRI, AARI, Faisalabad Punjab 1996 53 Shahkar 95 WRI, AARI, Faisalabad Punjab 1996 54 Sulaiman 96 CCRI, Pirsabak NWFP 1996 55 Bahawalpur 95 RARI, AARI, Bahawalpur Punjab 1996 56 Tatara NIFA, Peshawar NWFP 1996 57 Chakwal 97 BARI, AARI, Chakwal Punjab 1997 58 Drawar 97 RARI, AARI, Bahawalpur Punjab 1997 59 Fakhre Sarhad NIFA, Peshawar NWFP 1997 60 Kohistan 97 WRI, AARI, Faisalabad Punjab 1997 61 MH 97 WRI, AARI, Faisalabad Punjab 1997 62 Bahawalpur 97 RARI, AARI, Bahawalpur Punjab 1998 63 Daman 98 ARI, D.I. Khan NWFP 1998 64 Dera 98 ARI, D.I. Khan NWFP 1998 65 Durum 97 WRI, AARI, Faisalabad Punjab 1998 66 Ghaznavi 98 Agri. Univ. Peshawar NWFP 1998 67 Nishtar NIFA, Peshawar NWFP 1999 68 Zarlashta 99 ARI, Sariab, Quetta Balochistan 1999 69 Auqab 2000 WRI, AARI, Faisalabad Punjab 2000 70 BWP 2000 RARI, AARI, Bahawalpur Punjab 2000 71 Chanab 2000 WRI, AARI, Faisalabad Punjab 2000 72 Iqbal 2000 WRI, AARI, Faisalabad Punjab 2000 73 Nasir 2K ARI, D. I. Khan NWFP 2000 74 Punjnad 1 RARI, AARI, Bahawalpur Punjab 2000 75 Raj ARI, D.I. Khan NWFP 2000 76 Saleem 2000 CCRI, Pirsabak NWFP 2000 77 Takbeer NIFA, Peshawar NWFP 2000 78 AS 2002 WRI, AARI, Faisalabad Punjab 2002 79 Bhakkar 2002 AZARI, Bhakkar Punjab 2002 80 Chakwal 2002 BARI, AARI, Chakwal Punjab 2002 81 SH 2002 WRI, AARI, Faisalabad Punjab 2002 82 Ufaq 2002 ABRI, AARI, Faisalabad. Punjab 2002 83 KT 2000 BARS, Jarma, Kohat NWFP 2003 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 187 S. No Variety Name Breeding Centre Province Year of Release 84 KT 2003 BARS, Jarma, Kohat NWFP 2003 85 Lakki J03 ARS, Serai Naurang, Bannu NWFP 2003 86 Manthar 3 RARI, AARI, Bahawalpur Punjab 2003 87 Marwat J01 ARS, Serai Naurang, Bannu NWFP 2003 88 Bhittai NIA, Tandojam Sindh 2004 89 Haider 2000 CCRI, Pirsabak NWFP 2004 90 Marvi 2000 NIA, Tandojam Sindh 2004 91 Pirsabak 2004 CCRI, Pirsabak NWFP 2004 92 Zam ARI, D.I. Khan NWFP 2004 93 Raskoh 2005 ARI, Sariab, Quetta Balochistan 2005 94 Fareed 2006 RARI, AARI, Bahawalpur Punjab 2006 95 Imdad 05 WRI, Sakrand Sindh 2006 96 Khirman NIA, Tandojam Sindh 2006 97 Moomal 2002 WRI, Sakrand Sindh 2006 98 Sassui NIA, Tandojam Sindh 2006 99 Seher 2006 WRI, AARI, Faisalabad Punjab 2006 100 Shafaq 2006 WRI, AARI, Faisalabad Punjab 2006 101 SKD 1 WRI, Sakrand Sindh 2006 102 Chakwal 50 BARI, AARI, Chakwal Punjab 2008 103 Faisalabad-08 WRI, AARI, Faisalabad Punjab 2008 104 Gomal-8 ARI, D. I. Khan NWFP 2008 105 Hashim-8 ARI, D. I. Khan NWFP 2008 106 Lasani-08 WRI, AARI, Faisalabad Punjab 2008 107 Mairaj-08 RARI, AARI, Bahawalpur Punjab 2008 108 NIFA-Bathoor-8 NIFA, Peshawar NWFP 2008 109 Pirsabak-8 CCRI, Pirsabak NWFP 2008 110 BARS-09 BARS, Fateh Jhang/Chakwal Punjab 2009 111 NARC-09 CSI, NARC, PARC, IBD Islamabad 2009 112 Amin-2010 ARS, S. Naurang, Bannu KP 2010 113 Atta-Habib IB&GE, Agri. University, Peshawar KP 2010 114 KT-2010 BARS, Jarma, Kohat KP 2010 115 NIA-Amber NIA, Tandojam Sindh 2010 116 NIA-Sunhari NIA, Tandojam Sindh 2010 117 Siran-2010 IB&GE, Agri. University, Peshawar KP 2010 118 Tijaban-10 AZRC, Quetta Balochistan 2010 119 AARI-2011 WRI, AARI, Faisalabad Punjab 2011 120 AAS-2011 RARI, AARI, Bahawalpur Punjab 2011 121 Barsat-10 NIFA, Peshawar KP 2011 122 Dharabi-2011 BARI, Chakwal Punjab 2011 123 Janbaz Agri. University, Peshawar KP 2011 124 Millat-2011 WRI, AARI, Faisalabad Punjab 2011 125 NARC-2011 CSI, NARC, PARC, IBD Islamabad 2011 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 188 S. No Variety Name Breeding Centre Province Year of Release 126 Punjab-2011 WRI, AARI, Faisalabad Punjab 2011 127 BENAZIR-2012 WRI, Sakrand Sindh 2013 128 Galaxy-2013 WRI, AARI, Faisalabad Punjab 2013 129 Hamal-2012 WRI, Sakrand Sindh 2013 130 NIA-SARANG NIA, Tandojam Sindh 2013 131 NIA-SUNDAR 8/7 NIA, Tandojam Sindh 2013 132 NIFA Lalma 2013 NIFA, Peshawar KP 2013 133 Pakistan-2013 NARC, PARC, Islamabad Islamabad 2013 134 Pirsabak-2013 CCRI, Pirsabak KP 2013 135 Shahkar CCRI-2013 CCRI, Pirsabak, Noshera KP 2013 136 Ghanimat-e-IBGE IBGE, UAP KP 2015 137 Insaf NIFA, Peshawar KP 2015 138 Pakhtunkhwa-2015 CCRI, Pirsabak KP 2015 139 Pirsabak-2015 CCRI, Pirsabak KP 2015 140 Borlaug 2016 NARC, PARC, Islamabad Islamabad 2016 141 Fatehjang 2016 BARI, Chakwal Punjab 2016 142 Gold-2016 RARI, Bahawalpur Punjab 2016 143 Ihsan 2016 BARI, Chakwal Punjab 2016 144 Jauhar 2016 RARI, Bahawalpur Punjab 2016 145 NN Gandum 1 NIBGE, Faisalabad Punjab 2016 146 Ujala-2016 WRI, AARI, Faisalabad Punjab 2016 147 Zincol-2016 NARC, PARC, Islamabad Islamabad 2016 148 ANAJ-2017 WRI, AARI, Faisalabad Punjab 2017 149 Barani-17 BARI, Chakwal Punjab 2017 150 Fakhar-e-Bhakkar AZRI, Bhakkar Punjab 2017 151 Sindhu WRI, Sakrand Sindh 2017 152 Israr shaheed ARI, Dera Ismael Khan KPK 2018 153 Kohat-2017 BARS, Jarma, Kohat KPK 2018 154 Khaista-2017 CCRI, Pirsabak, Nowshera KPK 2018 155 NIFA Aman-2017 NIFA, Tarnab Peshawar KPK 2018 156 Paseena-2017 CCRI, Pirsabak, Nowshera KPK 2018 157 Shahid-2017 ARI, Dera Ismael Khan KPK 2018 158 Wadaan-2017 CCRI, Pirsabak, Nowshera KPK 2018 159 Pasta 2019 CCRI, Pirsabak, Nowshera KP 2018 160 Qaswa 2019 CCRI, Pirsabak, Nowshera KP 2018 161 Awaz 2019 NIFA, Tarnab, Peshawar KP 2018 162 Ghazi-19(122559) Regional Agri. Research Institute, Bahawalpur. Punjab 2019 163 Markz-2019(NR-449) PARC, Islamabad Islamabad 2019 164 Bhakkar Star Arid Zone Research Institute, Bhakkar Punjab/KP 2019/20 22 165 Akbar-19 Wheat Research Institute, AARI, Faisalabad Punjab 2019 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 189 S. No Variety Name Breeding Centre Province Year of Release 166 AZRC-DERA (Rainfed) Arid Zone Research Center, D.I. Khan KP 2020 167 Fahim-2019 BARS, Jarma, Kohat KP 2020 168 Pirsabak-2019 CCRI, Pirsabak, Nowshera KP 2020 169 Gulzar-2019 CCRI, Pirsabak, Nowshera KP 2020 170 I.V-2 Wheat Breeding Res. Inst. Sakrand Sindh 2020 171 NIA-Shaheen NIA, Tando jam Sindh 2020 172 NIA-Zarkhez NIA, Tandojam Sindh 2020 173 *Nishan (TWS1335-1/Bhakkar￾2020) AZRI, Bhakkar Punjab 2021 174 16C038 (MA-2021 BARI, Chakwal Punjab 2021 175 Dilkash-20 (V-16005) Wheat Research Institute, AARI, Faisalabad Punjab 2021 176 Subhani-21 (HY55-33) Wheat Research Institute, AARI, Faisalabad Punjab 2021 177 Swabi-1 ARS, Sawabi KP 2021 178 Abaseen-2021 CCRI, Pirsabak Nowshera KP 2021 179 Pirsabak-2021 CCRI, Pirsabak Nowshera KP 2021 180 Zarghoon-2021 CCRI, Pirsabak Nowshera KP 2021 181 TD 1 ARI, Tandojam Sindh/Punjab 2004/20 16 182 NARC SUPER (NR-499) Wheat Program, NARC, Islamabad Islamabad 2021 183 Arooj-22 (V-17179) Wheat Research Institute, Faisalabad Punjab 2021 184 Durum-2021 (D-15729) Wheat Research Institute, Faisalabad Punjab 2021 185 Rahbar-21 (NS-85-1) Neelum Seeds Punjab (PVT) 2021 186 SADIQ-21 Regional Agricultural Institute, Bahawalpur Punjab 2021 187 NAWAB-21 (FB-1705) Regional Agricultural Institute, Bahawalpur Punjab 2021 188 MH-21 Wheat Research Institute, Faisalabad Punjab 2021 189 Taskeen-2022 CCRI, Pirsabak, Nowshera KP 2022 FINAL PERFORMANCE EVALUATION OF THE PAKISTAN AGRICULTURE PROGRAMS | 190 ANNEX XII: ASP SELECTION CRITERIA FOR ASPTP ● Providing at least one of the identified/site-specific technology service to the farmers ● Preferably, literate and have some formal/informal skills ● Preferably, have some agriculture background ● Should be inclined and committed to provide the services ● At least 1 to 2 year experience of providing services to the farmers ● An experienced tractor driver for tractor driven machines ● Either having machine or committed to purchase for which he is interested among selected technologies ● Committed to established service-hub/center as entrepreneur.85 85 ICARDA. Final Report of ASPTP, August 2018-July 2012. Page 63 U.S. Agency for International Development Pakistan Mission Islamabad, 44010