BUILDING SUSTAINABLE AGRIBUSINESSES IN PAKISTAN FINAL EVALUATION OF THE AGRIBUSINESS PROJECT NOVEMBER 4, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Uzair Ahmed Khan, Zohra Khanum, Douglas Krieger, Zia Ur Rehman, Syed Noman Ali Shah, Jill Tirnauer, and Himat Ullah, of Management Systems International. BUILDING SUSTAINABLE AGRIBUSINESSES IN PAKISTAN FINAL EVALUATION OF THE AGRIBUSINESS PROJECT Management Systems International Corporate Offices 200 12th Street, South Arlington, VA 22202 USA Tel: + 1 703 979 7100 Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This evaluation is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International (MSI) and do not necessarily reflect the views of USAID or the United States Government. THE AGRIBUSINESS PROJECT EVALUATION i CONTENTS Project Summary ................................................................................................................................................................ v Executive Summary ........................................................................................................................................................... 1 Implementation ......................................................................................................................................................... 7 Development Hypothesis and Intended Results ............................................................................................... 9 Evaluation Methods and Limitations ............................................................................................................................ 10 Data Collection ....................................................................................................................................................... 10 Data Analysis............................................................................................................................................................ 12 Data Limitations and Interpretation ................................................................................................................... 13 Findings and Conclusions ............................................................................................................................................... 14 Apricot Value Chain ............................................................................................................................................... 14 Grape Value Chain ................................................................................................................................................. 17 HV/OSV Value Chain ............................................................................................................................................. 21 Potato Value Chain ................................................................................................................................................. 28 Meat Value Chain .................................................................................................................................................... 32 IMAP .......................................................................................................................................................................... 35 Miscellaneous Findings and Conclusions ........................................................................................................... 37 Conclusions by Evaluation Question .......................................................................................................................... 45 Design ........................................................................................................................................................................ 45 Implementation ....................................................................................................................................................... 46 Sustainability ............................................................................................................................................................. 47 Annexes ............................................................................................................................................................................. 50 Annex I: Evaluation Statement of Work ........................................................................................................... 51 Annex 2: Assignment Work Plan ........................................................................................................................ 75 Annex 3: Locations of Value Chain Clusters ................................................................................................... 81 Annex 4: Local Partners ........................................................................................................................................ 82 Annex 5: Documents Reviewed .......................................................................................................................... 83 Annex 6: Data Collection Instruments .............................................................................................................. 86 Annex 7: Interviews ............................................................................................................................................ 108 Annex 8: Sampling ............................................................................................................................................... 113 Annex 9: Qualitative Analysis Results ............................................................................................................. 117 Annex 10: Conflict of Interest Disclosures ................................................................................................... 134 THE AGRIBUSINESS PROJECT EVALUATION ii Lists of Tables and Figures Table 1: Project Summary................................................................................................................................................ v Table 2: TAP Performance Indicators......................................................................................................................... 10 Table 3: Data Collection Methods and Sources ....................................................................................................... 11 Table 4: Implementation Issues – Grape FEGs ......................................................................................................... 20 Table 5: Reasons for Increased Production .............................................................................................................. 23 Table 6: Implementation Issues – HV/OSV Individual Grant Recipients ............................................................ 28 Table 7: IMAP Events ...................................................................................................................................................... 35 Table 8: TAP Local Partners and Regional Value Chain Clusters ........................................................................ 37 Table 9: Challenge Grants ............................................................................................................................................. 41 Table 10: Women’s Participation in FEGs................................................................................................................. 44 Table 11: TAP Value Chain Clusters ........................................................................................................................... 81 Table 12: TAP Local Partners ....................................................................................................................................... 82 Table 13: List of Interviews and Site Visits ............................................................................................................. 108 Table 14: List of Beneficiaries by Location, Type, and Value Chain ................................................................. 114 Table 15: Number of interviews by Location, Type, and Value Chain ............................................................ 116 Table 16: Support Received from TAP (Apricot and Potato) ............................................................................ 117 Table 17: Support Received from TAP (HV/OSV and Grape)........................................................................... 117 Table 18: Support Received from TAP (IMAP, Meat, and total) ....................................................................... 118 Table 19: TAP’s Contribution to Value Chain Development............................................................................ 118 Table 20: Change in Technologies and Practices (FEGs) .................................................................................... 119 Table 21: Change in Technologies and Practices (Individual Grant Recipients and Processors/exporters) ........................................................................................................................................................................................... 120 Table 22: Reasons for Increasing Quantity Produced (Apricot, Potato, and HV/OSV) .............................. 121 Table 23: Reasons for Increasing Quantity Produced (Grape, Total, and Meat) .......................................... 122 Table 24: Change in Value of Sales (Apricot, Potato, and HV/OSV) ................................................................ 123 Table 25: Change in Value of Sales (Grape, Meat, Total, IMAP) ....................................................................... 123 Table 26: Change in Value of Exports (Apricot, Potato, and HV/OSV) .......................................................... 124 Table 27: Change in Value of Exports (Grape, Meat, Total, IMAP) .................................................................. 124 Table 28: Change in Employment (Apricot, Potato, and HV/OSV) .................................................................. 125 Table 29: Change in Employment (Grape, Meat, Total, IMAP) ......................................................................... 125 Table 30: Prospects for Sustainable Results (Apricot, Potato, and HV/OSV) ................................................ 126 Table 31: Prospects for Sustainable Results (Grape, Meat, Total, IMAP) ....................................................... 126 Table 32: Satisfaction with Implementation - Apricot ......................................................................................... 127 Table 33: Satisfaction with Implementation – Seed Potato ................................................................................ 128 Table 34: Satisfaction with Implementation – HV/OSV ....................................................................................... 129 Table 35: Satisfaction with Implementation – Grape ........................................................................................... 130 Table 36: Satisfaction with Implementation – Meat.............................................................................................. 131 Table 37: Satisfaction with Implementation – All Value Chains ........................................................................ 132 Table 38: Satisfaction with Implementation – IMAP ............................................................................................. 133 Figure 1: Location of Project Value Chain Activities ................................................................................................ vi THE AGRIBUSINESS PROJECT EVALUATION iii ACRONYMS ADB Asian Development Bank AI Artificial Insemination AITs Artificial Insemination Technicians AKRSP Aga Khan Rural Support Programme AOR Agreement Officer’s Representative ASEAN Association of Southeast Asian Nations ASF Agribusiness Support Fund ASP Assessment and Strengthening Program BDS Business Development Services BDSP Business Development Service Provider CIS Commonwealth of Independent States CNFA Citizens Network for Foreign Affairs EDT Enterprise Development Training EGA Economic Growth and Agriculture FATA Federally Administered Tribal Areas FEG Farmers ’ Enterprise Group FSC Farm Services Centers FTE Full-Time Equivalent GB Gilgit-Baltistan GCC Gulf Cooperation Council GoP Government of Pakistan HV/OSV High Value Off-Season Vegetables IMAP International Market Access Program kg Kilograms KP Khyber Pakhtunkhwa M&E Monitoring and Evaluation MSMEs Micro, Small and Medium Enterprises NGO Non-Governmental Organization NOC No Objection Certificate NRSP National Rural Support Programme OAPA Office of Afghanistan and Pakistan Affairs OIG Office of the Inspector General PAMCO Punjab Agriculture and Meat Company PMP Performance Management Plan RCDS Rural Community Development Society RMA Rapid Market Appraisal Rs. Pakistan Rupees SOW Statement of Work SRSP Sarhad Rural Support Programme TA Technical Assistance TAP The Agribusiness Project TDAP Trade and Development Authority UK United Kingdom USAID United States Agency for International Development USD United States Dollar USG United States Government THE AGRIBUSINESS PROJECT EVALUATION iv UVAS University of Veterinary and Animal Sciences VCP Value Chain Platforms THE AGRIBUSINESS PROJECT EVALUATION v PROJECT SUMMARY Table 1 summarizes basic information about The Agribusiness Project (TAP). TABLE 1: PROJECT SUMMARY The map in Figure 1 illustrates the locations of TAP’s value chain activities in the apricot, seed potato, high value off season vegetable (HV/OSV), grape, and meat value chains. The shaded districts represent the districts from which the evaluation team selected samples of respondents for site visits and interviews. 1 USAID/Pakistan. (2013). Modification of Assistance No. 6. Islamabad: USAID/Pakistan. 2 USAID/Pakistan. (2013). Modification of Assistance No. 6. Islamabad: USAID/Pakistan. Title / Field Project/Activity Information Contract/agreement numbers AID-391-A-12-00001 Agreement Officer’s Representative (AOR) Mohammad Ghani Khan, Project Management Specialist for Agriculture Start date November 10, 2011 Completion date November 9, 2016, revised to November 9, 20151 Location Nationwide, except for Balochistan Implementing partner(s) Agribusiness Support Fund (ASF) USAID/Pakistan Mission Strategic Framework objectives addressed DO 2: Improved economic status of focus populations. IR 2.1: Improved economic performance of focus enterprises IR 2.1.2: Improved skill development and job placement IR 2.1.3: Increased use of modern technology and management practices IR 2.2.2: Strengthened private sector and civil society engagement in policy-making. Budget USAID contribution: U.S. Dollar (USD) 89,412,942, reduced to USD 39,947,3812 ASF and grantee enterprises cost-share: USD 47,717,980 (the revised project document does not include a revised cost-share figure) THE AGRIBUSINESS PROJECT EVALUATION vi FIGURE 1: LOCATION OF PROJECT VALUE CHAIN ACTIVITIES THE AGRIBUSINESS PROJECT EVALUATION 1 EXECUTIVE SUMMARY USAID/Pakistan’s Economic Growth and Agriculture (EGA) office requested the end of project evaluation of The Agribusiness Project (TAP) to guide implementation of a newly designed project that focuses on developing agri-businesses and agricultural value chains. The EGA office also asked for recommendations on how best to engage local organizations as implementing partners. The evaluation addresses three questions: 1. To what extent was the revised project design relevant to achieving anticipated results? 2. To what extent were the project’s activities and implementation approaches effective in contributing to achieving expected results? 3. To what extent, and how, are project interventions likely to produce sustainable results that will endure beyond project assistance? To what extent are non-beneficiary farmers or enterprises replicating project interventions? Project Background USAID/Pakistan awarded the five-year USD 89.4 million cooperative agreement for TAP to the Agribusiness Support Fund (ASF), a local organization with no previous USAID experience, in November 2011. A 2013 Office of the Inspector General (OIG) audit concluded that ASF did not have the capacity to implement the project. In consultation with USAID, ASF scaled back the project, focusing on fewer value chains and a more limited geographic area, cutting the budget to USD 39.9 million, and reducing the timeframe to four years. TAP provides grants and technical assistance to small farmers organized into farmers’ enterprise groups (FEGs) and individual grants to agri-businesses, e.g., larger farmers, processors, and exporters. Grants help recipients adopt new technologies and practices to enhance competitiveness and ultimately “support improved conditions for broad-based economic growth, enhanced profitability, and employment opportunities and contribute to poverty alleviation through product and process transformation”.3 Data Collection Methods and Limitations Of the eight value chains which TAP supports,4 the evaluation addresses only five, apricot, high-value off￾season vegetable (HV/OSV), seed potato, grape, and meat. It relies almost entirely on qualitative data collected from randomly selected beneficiaries and stakeholders. The evaluation team conducted 67 individual interviews and site visits with project beneficiaries and stakeholders, 39 group interviews with 3 Agribusiness Support Fund. (no date). The Agribusiness Project Revised Project Document. Lahore: Agribusiness Support Fund. 4 Banana, chilies, citrus, apricot, grape, seed potato, high-value off-season vegetables, and meat. THE AGRIBUSINESS PROJECT EVALUATION 2 FEGs, and interviews with 22 individuals involved in design (USAID, ASF). When possible, the team used quantitative performance data and information from project documents to triangulate the qualitative findings. The data provide a great deal of information to answer how and why questions but suffer from the limitations common to qualitative data, i.e., differences in interview techniques, language barriers, general communication issues, the education level of respondents, and strategic behavior may all affect consistency, validity, and reliability. The evaluation team adequately addressed most of the limitations and is confident in the overall validity of the qualitative data. Findings and Conclusions ASF’s design of TAP was generally sound, i.e., it planned relevant and effective interventions. It also implemented most activities reasonably well after the mid-term course correction. Its primary implementation failings stemmed from a very long start-up period which delayed substantive engagement with many beneficiaries by two or three years. The late start deprived beneficiaries of the time necessary to gain experience with new technologies and practices before project support ends and may ultimately compromise the sustainability of results. The remainder of this section summarizes findings and conclusions for each value chain and value chain actor. Apricot FEGs: By relieving binding financial constraints, TAP grants were instrumental in transferring solar drying technologies to FEGs. The technologies and associated training (the design) effectively changed processing (drying) practices, reduced waste, and increased the quantity of dried apricots FEG members produced for sale. The farmers had not yet sold apricots, but when the drying season ends, they should earn additional income from increased quantities and better quality. Farmers in the region do not have a culture of hiring labor, so the intervention has not created jobs. ASF implemented the intervention well. It quickly and effectively corrected a design flaw in the drying tunnels. It is too early to assess the sustainability of the results, which will depend on the durability of the tunnels, and members of two of seven FEGs interviewed questioned their ability to repair the tunnels if they were damaged. Apricot Grant Recipients: TAP assistance addressed the financial constraints farmers faced establishing commercial apricot orchards. The orchards are not yet producing, so it is too early to assess production outcomes. Orchard owners are experienced apricot producers and, with the training in new cultivation practices provided with the grants, they should be able to increase their production of commercial apricot varieties and therefore their incomes. Once they are mature, the orchards will likely generate some jobs in orchard management and many seasonal jobs in harvesting and processing. However, the scale of the intervention (18 orchards) limits its immediate contribution to job creation or economic growth beyond the orchard owners. It is too early to assess the intervention’s success as a replicable model to promote investment in commercially oriented practices (e.g., varieties and cultivation practices). Grape FEGs: TAP support was instrumental in encouraging FEGs to invest in grape production. Few knew about grapes and none could afford the investment without external support. The FEGs are just beginning to produce and, based on what they had heard from acquaintances that produced grapes and observed market prices, farmers were optimistic about their own prospects. Because of their optimism and sunk investment, FEGs expected to continue producing grapes and had also observed other farmers replicating the vineyards. Grapes seem to offer prospects for full-time and casual labor (for men) but it is difficult to determine actual requirements for additional labor. From an implementation perspective, ASF struggled to provide trellis material on time and in the quantities expected by the FEGs. FEGs also reported suffering large losses due to birds and were disappointed that ASF did not provide nets. The sustainability of results is questionable for these FEGs if they cannot limit bird predation. THE AGRIBUSINESS PROJECT EVALUATION 3 Grape Grant Recipients: Individual grant recipients were relatively well-off landowners with other sources of income and were looking for investment opportunities. ASF-sponsored visits to farms with active vineyards convinced them that grapes represented a promising opportunity, a finding that strongly suggests that the demonstration effect of established vineyards contributes to replication, at least for those who can afford the investment. All four farmers the evaluation team interviewed had established their vineyards in early 2014 and had not yet produced or sold significant quantities of grapes. Grant recipients reported hiring labor for the vineyards, but the extent to which they hired new workers as opposed to merely re-assigning permanent labor is not clear. Women’s roles in grapes appear confined to packaging. Grant recipients reported that they expected to continue producing grapes if it was profitable and, based on their discussions with owners of established vineyards, they were optimistic about profitability. HV/OSV FEGs: ASF’s interventions with HV/OSV FEGs were largely well-designed and effective. A large majority of FEGs adopted the new technologies and practices, especially the hybrid seeds and vertical farming structures, and increased their production and sales (due to quantity and quality/price) as a result. However, most FEGs were still selling through the local mandis which do not generally reward quality. FEG-level vegetable production also generates a substantial number of jobs, many of which are open to women, due to more labor intensive production practices, higher yields, and a longer production season. FEG members reported that they intend to continue using the technologies and practices promoted by ASF because they are low cost and quite profitable relative to the alternative crops. However, they reported that difficulty finding good quality hybrid seeds in the market may disrupt their plans. ASF struggled to provide some inputs (seeds) on time and of good quality although this problem seems largely confined to activities in Lahore. HV/OSV Grant Recipients: ASF targeted relatively well-off landowners with capital to invest or specialized knowledge. The data suggest that ASF’s strategy of demonstrating new technologies and practices to raise awareness among financially capable farmers may have been effective in stimulating replication. At the individual grant level, the project design (i.e., tunnel farming) appears to have been effective in substantially increasing production, sales, and employment. The only issue with implementation emerged in Lahore and involved the design and/or inspection of tunnels. Tunnel farming appears to offer substantial employment potential for men and for women. Seed Potato FEGs: ASF’s support to seed potato FEGs appears to have successfully transferred new technologies and practices to FEG members, but outcomes (e.g., quantities produced or sold) were not yet directly observable, because most farmers had not harvested their first crop at the time of the evaluation. Based on the evidence from the few who had harvested and farmers’ assessment of the quality of the standing crop, the intervention should be effective in increasing the quantity and quality of potatoes and should increase the value of sales. At the FEG level, the potato value chain offers little potential for creating jobs. The intervention is low-cost and probably profitable in the short run, but difficulty obtaining good quality seeds may threaten the sustainability of results. ASF itself seems to have had difficulty providing good quality certified seeds on time. The project’s individual grants to establish storage cellars should address this constraint, but it will take a few seasons of experience to assess whether the cellars are effective in increasing the local supply of seed. Seed Potato Grant Recipients: The grants to support storage cellars addressed financial constraints that prevented farmers from constructing cellars and thus promoted better storage practices in the region. The cellars were appropriate technology that effectively eliminated spoilage during the winter. In addition to having a larger quantity to sell because of reduced spoilage, cellar owners were able to maintain the quality of their stored potatoes and thus receive higher prices in the market. If the practices are sustainable, the cellars should contribute to increasing access to local seed potatoes and THE AGRIBUSINESS PROJECT EVALUATION 4 thus address a production constraint FEGs mentioned. However, the high cost of the cellars may prevent the intervention from reaching scale through organic replication. Meat Value Chain: Training in meat cuts was effective in teaching new skills, but limited penetration into markets that use those skills to advantage has thwarted achieving broader outcomes on sales or employment. Retail butchers were the only trainees who reported results. Most processors and exporters have not yet developed the export market connections to sell high-value boneless cuts. ASF also trained artificial insemination technicians (AITs) to enhance farmers’ access to quality private sector artificial insemination (AI) services. The training was effective in strengthening AITs’ skills, and one trainee reported that farmers are increasingly aware of the benefits of AI and that his business is growing. However, two of the four trainees interviewed were not in a position to put their training into practice because they were unemployed, and two did not have access to transportation to reach farmers. Feedlot fattening training taught new skills, but not to individuals who were in a position to use them. From an implementation perspective, the experience with AI and feedlot fattening training points to issues with beneficiary selection criteria. International Market Access Program (IMAP): IMAP events effectively exposed participants to export markets and market requirements. To the large majority of participants (13 of 14 the IMAP participants the evaluation team interviewed) who had participated in similar events without TAP support, the events’ added value was the pre-event briefing and access to a branded booth or stall. ASF documented sales of over USD 33 million associated with IMAP events. Since most participants were established exporters, these sales may not (if they merely supplanted sales the participant would have made to other buyers anyway) represent expanded business or access to new markets, buyers, or products. ASF appears to have implemented the logistically difficult activity well with few substantive complaints and many kudos from participants. Gender Integration: The project explicitly considered women in design only by setting targets for the percentage of female members of project-supported FEGs. It did not have a gender strategy nor did it design interventions specifically to engage women. Development projects cannot create a role for women and, to have a meaningful impact, must deliberately design interventions in value chains in which women play an active role. The established role of women in apricot processing contributed to the project’s potential success in benefiting women in this value chain. Partnerships: ASF’s partnerships with local organizations were effective in quickly organizing FEGs for TAP implementation. Interviews with the local implementing partners and their closeout reports revealed two common implementation issues. First, ASF put little or no effort into connecting FEGs to markets. This reflects a limited appreciation for the value chain development approach and potentially limits results. Second, ASF often provided support too late for farmers to benefit fully, if at all. Frequent shifts in strategy, caused in part by USAID, coupled with ASF’s problems delivering on grants and technical assistance, complicated implementation for the partners and damaged their reputation with their constituent communities. Recommendations  If and when USAID/Pakistan engages local partners, it should do so slowly with relatively modest projects where the negative consequences associated with up-front learning are small. Developing the necessary capacities after the award may delay implementation and limit results. The agency might also consider placing an experienced international firm in a mentoring or capacity-building role and giving it the authority, at least initially, to manage parts of the project. THE AGRIBUSINESS PROJECT EVALUATION 5  USAID/Pakistan should adapt the value of exports indicator to reflect a more meaningful measure of enhanced access to export markets. A more meaningful measure of export capacity might include trends in overall export sales, sales to new buyers or markets, or offering new products to export markets. Any of these would be more direct and feasible measures of increased access to export markets that are not as misleading as the current value of exports indicator.  If USAID/Pakistan intends to continue supporting agricultural value chain development, it should consider continuing support to at least some of the more promising TAP-supported value chains and beneficiaries to give beneficiaries the additional technical assistance necessary to cement the lessons they have learned and address new challenges that arise as they gain experience with new technologies and practices. The most promising interventions seem to be HV/OSV and grape (FEGs and larger farmers), apricot orchards, and potato cellars.  If USAID/Pakistan wishes to continue supporting the meat sector with a focus on developing export markets, it needs to take a more holistic approach. A development process that starts by building the capacities of processors to meet a small demand for quality, value-added meat products and linking these processors to export markets may create a demand for better quality meat animals that donors can then support by investing in building feedlot capacity and improving herd quality.  To improve prospects for sustainability, USAID/Pakistan should require implementing partners to develop sustainability plans during the design phase, review the plans periodically to determine whether sustainability hypotheses are valid, and adapt implementation as necessary if the hypotheses do not hold true. Sustainability plans should be specific to each activity, i.e., in the TAP example, value chains.  When building and/or strengthening institutions is an important component of a sustainability strategy, it needs to start early in the project so the institutions are engaged, strong, and sustainable. In similar future work, USAID should ensure that building and strengthening institutions occurs early in the project work plan and follow up to make sure it happens. EVALUATION PURPOSE AND QUESTIONS The evaluation is an end of project evaluation that takes place six months prior to the official close of the project on December 9, 2015. The Mission has already designed future projects that will focus on developing agribusinesses and agricultural value chains. The evaluation will help USAID/Pakistan’s office of Economic Growth and Agriculture (EGA) make decisions about how the project(s) will be implemented. Furthermore, as USAID/Pakistan continues to emphasize implementing through local organizations, the Mission is particularly interested in recommendations and lessons learned regarding if, when, and how to use local organizations as implementing partners, and the evaluation will contribute to this learning. The primary audiences for the evaluation include the USAID/Pakistan Mission, particularly EGA; the USAID Office of Afghanistan and Pakistan Affairs (OAPA); and the implementing partner, the Agribusiness Support Fund (ASF). THE AGRIBUSINESS PROJECT EVALUATION 6 Evaluation Questions The evaluation Statement of Work (SOW) posed two evaluation questions, each with a detailed explanation that elaborated on the question. The questions focused broadly on project design (i.e., did the project do the right things) and implementation (i.e., did it do things right). The final version of the SOW included a sub-question about the sustainability of results. With the AOR’s concurrence, the evaluators created a third question by separating the issue of sustainability from implementation. Annex 1 contains the evaluation SOW with the completely elaborated evaluation questions. The Assignment Work Plan (Annex 2) includes the final evaluation questions. The top-line questions are: 1. To what extent was the revised project design relevant to achieving anticipated results? 2. To what extent were the project’s activities and implementation approaches effective in contributing to achieving expected results? 3. To what extent, and how, are project interventions likely to produce sustainable results that will endure beyond project assistance? To what extent are non-beneficiary farmers or enterprises replicating project interventions? At USAID/Pakistan’s request, the evaluation focused only on the apricot, high-value off-season vegetable (HV/OSV), seed potato, grape, and meat value chains. PROJECT BACKGROUND The Agribusiness Project (TAP) originated from a concept note ASF presented to USAID/Pakistan after it completed the five-and-a-half year USD 21.6 million Agribusiness Development Project funded by the Government of Pakistan (GoP) and the Asian Development Bank (ADB).5,6 The proposed technical approach framed the project as a continuation, with some improvements, of the ADB-funded project.7 ASF stated that the ADB project was the largest award it had ever received, and it was the only award from an international donor that ASF mentioned during interviews. USAID/Pakistan personnel familiar with the award process reported that the embassy was very keen to work with a local organization, and the Mission’s assessment of the sector found no other local organizations with the capacity to implement the project. The pre-award assessment conducted in January 2011 rated ASF as high risk, but the Mission ultimately awarded a non-compete sole source agreement to ASF in November 2011, before a follow-up pre-award assessment was completed.8 ASF management reported that it originally proposed a USD 50 million project, expecting USAID/Pakistan to reduce the award to around USD 30 million. Instead, USAID/Pakistan ultimately 5 Asian Development Bank. (2014). Validation Report: Agribusiness Development Project. Islamabad: Asian Development Bank. Accessed at: http://www.adb.org/sites/default/files/PVR-354.pdf 6 The project was originally approved at USD 49 million, but ASF spent only USD 21.6 million by the time the project closed. An independent evaluation concluded that actual expenditures fell short of anticipated expenditures because ASF did not complete many of the anticipated activities. 7 USAID/Pakistan. (2011). Cooperative Agreement No. AID-391-A-12-00001. Islamabad: USAID/Pakistan. 8 When it was completed, the follow-up pre-award assessment concluded that ASF had improved some of its processes and lowered the risk from high to medium. THE AGRIBUSINESS PROJECT EVALUATION 7 awarded an USD 89.4 million agreement. Out of concern for ASF’s capacity, USAID/Pakistan required ASF to partner with an international sub-contractor to provide technical assistance and services. The Citizens Network for Foreign Affairs (CNFA), a U.S.-based non-profit with experience in economic development and livelihoods, received the sub-award. A 2013 Office of Inspector General (OIG) audit of TAP concluded that ASF did not have the capacity to manage the project. It also concluded that, 15 months into implementation, the project was behind on its targets for awarding grants, had made no progress in achieving its goal of creating jobs, and ran a risk of not achieving its first goal of contributing to broad-based economic growth. The audit attributed the lack of results to ASF’s inexperience with USAID and its consequent difficulty developing compliant grants manuals and work plans. The audit also questioned the sustainability of grant-funded results for Farmers’ Enterprise Groups (FEGs) and individual farmers. In response to the OIG audit, USAID/Pakistan asked ASF to redesign the project with a reduced scope. ASF managers and USAID/Pakistan personnel describe the redesign as a collaborative exercise. An October 2014 modification (number 6) to the agreement formalized the parameters of the redesigned (“de-scoped” as ASF managers refer to it) project. The modification reduced the size of the award from USD 89.4 million to USD 39.9 million, cut the period of performance by one year, and incorporated a revised program description with a smaller scope. The revised project document limited the number of value chains to less than 10 and narrowed the geographic scope to clusters where the project could focus on transforming the value chain and providing effective demonstrations.9 Implementation The overall goal of the revised TAP design is to “support improved conditions for broad-based economic growth, enhanced profitability, and employment opportunities and contribute to poverty alleviation through product and process transformation of selected value chains in the horticulture and livestock sub-sectors”.10 The project has three objectives: 1. Strengthen capacities in horticulture and livestock value chains to increase sales to domestic and foreign markets; 2. Strengthen the capacities of smallholders (through FEGs, individual farmers, and agribusinesses) to operate effectively and efficiently; and 3. Increase productivity and profitability through the adoption of new techniques and technological innovations (among farmers, agribusinesses, and business development service providers).11 During the revision, ASF reduced the number of value chains from 16 identified in the original work plan to eight12 selected for their potential for “achieving quantitative impact on income and employment, as 9 Agribusiness Support Fund. (no date). The Agribusiness Project Revised Project Document. Lahore: Agribusiness Support Fund. Undated, untitled, electronic file provided by ASF and presumed to be the modified program description attached to agreement modification 6. 10 Agribusiness Support Fund. (no date). The Agribusiness Project Revised Project Document. Lahore: Agribusiness Support Fund. 11 Agribusiness Support Fund. (2014). Agribusiness Project Performance Management Plan Update. Lahore: Agribusiness Support Fund. 12 Bananas, chilies, citrus, apricots, grapes, seed potatoes, high-value off-season vegetables, and meat. THE AGRIBUSINESS PROJECT EVALUATION 8 well as with respect to the distribution of benefit with respect to impacting small producers and enterprises.” 13 The revised implementation approach includes two components, technical assistance and value chain transformation support. Annex 3 documents the locations of TAP’s work in the selected value chains. Technical Assistance: The technical assistance component forms and strengthens FEGs so their small farmer members can adopt new technologies, learn new practices, and build their capacities to engage productively and profitably in agricultural value chains. The revised project document describes the strategy of engaging with FEGs as a means to reach a significant number of small farmers and reap the benefits of scale. Support may consist of grants to introduce new crops (e.g., grapes) or new production technologies and practices (e.g., HV/OSV, potatoes, and apricots). ASF always couples grants with technical assistance and training which, at a minimum, involves organizational capacity building but may also include training in production and marketing. ASF sub-contracted the tasks of creating FEGs and building their organizational capacities to established local non-governmental organizations (NGOs) in each region to leverage their connections with local communities and their experience creating and supporting community organizations. Annex 4 summarizes the regions and value chains each NGO supported. The revised project document describes ASF’s strategy for engaging with FEGs as:  Determining realistic expectations for their participation in the value chain and preparing them to perform the corresponding functions successfully;  Partnering FEGs with players in subsequent legs of the journey to market; and  Fostering relationships in which a fair share of production and post-harvest value added accrues to farmers. The technical assistance component also includes institutional strengthening. This activity develops the capacities of private sector business development service providers (BDSPs) to provide the services required to contribute to growth in the agriculture and livestock sectors and links the BDSPs to project￾supported agribusinesses. Institutional strengthening also incorporates building and strengthening umbrella organizations to represent the value chains and facilitate transformation. These organizations include Value Chain Platforms (VCPs), Farm Services Centers (FSCs) in Khyber Pakhtunkhwa (KP) province and the Federally Administered Tribal Areas (FATA), associations, and cooperatives. Technical assistance also includes building producers’ and processors’ capacities to comply with international standards and the demands of international markets, e.g., certifications. This activity dovetails with the final activity of the technical assistance component, market access and linkages. This activity assists medium and large agribusinesses and FEGs in a number of ways, including conducting market assessments, developing marketing and branding strategies, organizing business-to-business meetings to establish commercial relationships, and facilitating participation in marketing and export promotion events to introduce agribusinesses to knowledge and potential buyers. 13 Agribusiness Support Fund. (no date). The Agribusiness Project Revised Project Document. Lahore: Agribusiness Support Fund. Undated, untitled, electronic file provided by ASF and presumed to be the modified program description attached to agreement modification 6. THE AGRIBUSINESS PROJECT EVALUATION 9 Value Chain Transformation Assistance: This component provides grants to agribusinesses to support investments that strengthen and expand value chains. Examples include “farm diversification, greater market penetration and expansion, supply chain participation of small farmers, and product and market diversification by agribusiness enterprises”. 14 Examples of these grants include establishing orchards of new (commercial) varieties of fruit, solar dryers for apricots, storage cellars for seed potatoes, and artificial insemination (AI) to improve the quality of meat animals. Development Hypothesis and Intended Results The results framework contained in the revised TAP Performance Management Plan (PMP) describes the project’s development hypothesis in broad terms. It implies that if the project strengthens market linkages and strengthens the capacities of smallholders and agribusinesses and introduces new technologies and practices, then the competitiveness of horticulture and livestock value chains will improve, which will increase income earning opportunities and contribute to improved conditions for broad-based economic growth. Table 2 summarizes the 13 performance indicators on which TAP reports which point to some of the key intended results, at least from a Mission management perspective. 14 Agribusiness Support Fund. (no date). The Agribusiness Project Revised Project Document. Lahore: Agribusiness Support Fund. Undated, untitled, electronic file provided by ASF and presumed to be the modified program description attached to agreement modification 6. THE AGRIBUSINESS PROJECT EVALUATION 10 TABLE 2: TAP PERFORMANCE INDICATORS Indicator Definition and Standard Indicator Reference Number of jobs attributed to program implementation (4.5-2) Project-related household incomes of U.S. Government (USG) targeted beneficiaries Value of incremental sales attributed to program implementation (4.5.2-23 modified) Value of exports of targeted commodities as a result of USG assistance (4.5.2-36 modified) Number of micro and small enterprises linked to larger-scale firms as a result of USG assistance to the value chain (Custom) Number of micro, small and medium enterprises (MSMEs), including farmers, receiving business development services (BDS) from USG-assisted sources (4.5.2-37) Value of new private sector investment leveraged with USG resources (4.5.2-28 modified) Number of persons receiving training on skill development (Custom) Number of entities (e.g., farmers or private enterprises) that have applied new technologies or management practices as a result of USG assistance (4.5.2-5 & 4.5.2-42 ) Number of hectares under improved technologies or management practices as a result of USG assistance (4.5.2-2) Number of USG-assisted organizations that participate in legislative proceedings and/or engage in advocacy at all levels (Custom) Number of rural households benefiting directly from USG interventions (4.5.2-13) Proportion of female participants in USG-assisted programs designed to increase access to productive economic resources (GNDR-2) Source: Agribusiness Support Fund. (2013). The Agribusiness Project Annual Progress Report - II, October 1, 2012 - September 30, 2013. Lahore: Agribusiness Support Fund. EVALUATION METHODS AND LIMITATIONS The evaluation relied almost entirely on qualitative data with the exception of triangulating qualitative results against the quantitative data ASF reported to USAID/Pakistan. The evaluation team also drew from project reports, studies, assessments, and other documents (see complete list in Annex 5) to assemble a complete picture of the rationale for and elements of the design, implementation strategies, accomplishments, and lessons learned. Data Collection The evaluation team reviewed a large number of documents to collect information about project design, implementation strategies, and reported results. It collected primary data through individual and group interviews and direct observation of project interventions. The team’s field data collection activities included 67 individual interviews with project beneficiaries (individual grant recipients) and stakeholders/participants (local partners/NGOs, BDSPs, technical assistance (TA) providers); 39 group interviews with FEGs; and interviews with 22 individuals involved in design (USAID/Pakistan, ASF) and/or implementation (ASF). Annex 6 contains the interview instruments. Table 3 summarizes the data collection methods the team used to answer each question and the sources of the data. Annex 7 THE AGRIBUSINESS PROJECT EVALUATION 11 summarizes the fieldwork schedule, interviews, and site visits. Table 15 in Annex 8 documents the distribution of interviews by value chain and beneficiary type. TABLE 3: DATA COLLECTION METHODS AND SOURCES Evaluation Question Data Collection Methods and Type Data Sources Question 1: Design Document review USAID/Pakistan’s Mission Strategic Framework TAP planning, agreement, performance, and implementation documents Semi-structured interviews TAP staff members, USAID personnel Question 2: Implementation Harvesting secondary performance data TAP monitoring and evaluation (M&E) data from databases and annual and quarterly progress reports Semi-structured interviews (qualitative) TAP beneficiaries (individual grant recipients, IMAP participants, trainees), TAP staff members, local partners/NGOs, participating TA providers and BDSPs Group interviews TAP beneficiaries (FEGs) Question 3: Sustainability Document review (qualitative) TAP planning and other documents Semi-structured interviews (qualitative) TAP staff members, USAID personnel, TAP beneficiaries (individual grant recipients, IMAP participants, trainees), TAP staff members, local partners/NGOs, participating TA providers and BDSPs Group interviews (qualitative) TAP beneficiaries (FEGs) Sampling The team selected samples of key informants purposively to capture unique or particularly informed perspectives. These included ASF staff members, USAID/Pakistan personnel, ASF’s local NGO partners, and TA providers. For all other interviews, the team selected samples randomly from among the beneficiaries and participants located in areas the team selected for site visits. Sites were selected based on the number of beneficiaries and participants within reach of the central locations in which the team stayed, coverage of major activities, and security. Based on these criteria, the team organized interviews in the areas in and around Islamabad, Lahore, Peshawar, and Gilgit with side trips to the districts of Chakwal, Attock, Charsadda, Manshera, Ghizer, and Hunza. One team member also traveled to Karachi to interview International Market Access Program (IMAP) participants. The districts the team could access from these central sites (the sampling frame) contained between 39 percent and 100 percent of the project’s beneficiaries (and 63 percent overall), depending on the value chain and beneficiary type. Annex 8 describes the details of the sampling approach and characteristics of the sample. ASF provided lists of all project beneficiaries in the selected value chains. The team filtered the lists to retain the selected districts, ordered the lists randomly, and contacted potential interview respondents starting at the top of the randomized lists until it had scheduled the required number of interviews. There is no formula for determining the number of interviews required for qualitative work, but the team selected the number of interviews per group or activity based loosely on the size of the group or activity (e.g., IMAP, meat sector training) and, when possible, tried for no fewer than four interviews per stratum. For example, ASF formed 335 FEGs focused on HV/OSV (55 percent of all FEGs). Consequently, 49 percent of the FEG sample consisted of HV/OSV FEGs. See Annex 8 for more detail. THE AGRIBUSINESS PROJECT EVALUATION 12 Data Analysis The data for the evaluation was almost entirely qualitative. The team used MAXQDA to prepare and analyze the qualitative interview data. The process of analysis included:  Developed a detailed data analysis plan with table shells that summarized the data in a manner relevant to answering the evaluation questions. For example, one table cross-tabulates themes related to “satisfaction with implementation” in rows with respondent type and value chain in the columns. The frequency of responses in each cell indicated the level of satisfaction or dissatisfaction with a particular facet of implementation by value chain and value chain actor (e.g., FEGs, individual grant recipients, processors/exporters).  Reviewed a subset of interview transcripts to identify common themes and develop a coding structure. In the context of the example above this meant identifying the facets of implementation respondents mentioned and their reasons for being satisfied or dissatisfied with TAP’s performance relative to that facet of implementation.  Reviewed all interview transcripts to identify text segments that corresponded to the identified themes and used the software to “code” the segment, i.e., associate the theme with the segment.  Populated the table shells developed in the data analysis plan with the frequencies of documents that contained one or more instances of a particular theme. Data from semi-structured interviews may contain multiple segments that illustrate a particular theme. Because the frequency of a theme within an interview does not reflect the intensity of the theme, especially in a group interview, using the document rather than coded segments as the unit of analysis is most appropriate.15 Annex 9 contains the tables the evaluation team developed to analyze the data.  Looked for patterns of responses in the tables. For example, the tables might reveal a pattern of large frequencies associated with a particular code across respondent types or value chains. They might also reveal a pattern of large frequency values across many codes for a particular respondent type or value chain.  Retrieved coded segments to develop a nuanced understanding of the story behind the observed patterns. For instance, if a large number of apricot FEG members expressed dissatisfaction with the design of the solar drying tunnels, the team reviewed the individual text segments to understand the aspects of tunnel design that were problematic. When possible, the team triangulated the results of the qualitative analysis with quantitative data obtained from the project’s reports and M&E system. 15 This means that the tables reflect the frequency of documents that contain one or more instances of the theme and not the frequency of coded segments. THE AGRIBUSINESS PROJECT EVALUATION 13 Data Limitations and Interpretation Qualitative data are well suited to answering how and why questions but not well suited to collecting quantitative data. Differences in interview techniques, language barriers, general communication issues, the education level of respondents, and the potential for strategic behavior by respondents may all affect the consistency, validity, and reliability of the data. Specific limitations include:  Qualitative data reflect respondents’ perceptions of design, implementation, and results. Perceptions may differ from reality, vary from person to person, and be difficult to interpret without understanding the broader context.  Qualitative data are well suited to explaining how and why interventions produce results, or fail to produce results; exploring how elements of design and implementation affect outcomes; and learning about contextual factors that may affect results. They are not appropriate for validating quantitative results. In the context of this evaluation, the qualitative data directly addressed the main evaluation questions but were not appropriate for directly validating the quantitative results ASF reported having achieved.  Translation can lead to a loss of fidelity in qualitative data as can poor communication between interviewers and respondents. The team encountered several occasions where respondents had difficulty consistently discriminating between past and present experiences, comparing across time periods, or differentiating the effects of TAP from other development projects.  Respondents may also respond strategically in hopes of receiving additional project support. The team addressed these inherent limitations by developing instruments collaboratively to ensure a common understanding of the instruments; identifying inconsistencies in interviews and probing to resolve them; whenever possible, conducting interviews in the languages with which respondents were most comfortable; and probing to reveal contextual factors that could have affected results. The evaluation team adequately addressed most of the limitations and is confident in the overall validity of the qualitative data. Nevertheless, two interviews had to be discarded because interviewers could not resolve inconsistencies. When discussing how a project was designed and implemented in a semi-structured interview, respondents are more likely to recall negative than positive aspects. The ratio of positive to negative responses to a particular aspect of the project is therefore meaningless. What is meaningful is the content and frequency of responses which reveal the extent of a perception and the nature of the positive or negative reaction. Similarly, in a semi-structured interview setting it is not possible to interpret “no response.” For example, if an individual or group fails to mention the effects of project interventions on potato production, it could mean that they did not experience a change or that they failed to mention a change. Keeping these limitations in mind, the analysis focuses on frequencies of responses rather than ratios of positive to negative responses and does not try to infer the meaning of “no response.” Finally, ASF’s progress reports did not provide a consistent time series dataset of indicators or other performance data. Reports often reported achieving a result that was merely in progress rather than completed. Most reports did not disaggregate key results by value chain or value chain actor. These deficiencies made it very difficult for the evaluation team to compile a clear picture of the evolution of project results or to triangulate quantitative with qualitative results. THE AGRIBUSINESS PROJECT EVALUATION 14 FINDINGS AND CONCLUSIONS In a departure from the usual organization of evaluation reports, this section presents findings and conclusions by value chain or component (i.e., apricot, grape, HV/OSV, potato, meat, IMAP, challenge grant) rather than by evaluation question. This organization allows the authors to tell an unbroken story for each value chain that tracks the value-chain-specific intervention from design through to results. The organization frames findings on design, implementation, and sustainability in the broader context of the specific value chain. The conclusions section summarizes conclusions by evaluation question. Each section begins by describing the development problem in the particular value chain and TAP’s strategy for addressing the problem, i.e., the project design. The narratives then draw on the qualitative and quantitative data to describe, in turn, the intermediate outcomes associated with the interventions (i.e., changes in technologies and practices, changes in quantities produced), the long-term outcomes that follow from achieving intermediate outcomes (i.e., value of sales, number of jobs created), and the prospects for sustainable results. In the apricot, grape, HV/OSV, and potato value chains, TAP supported FEGs and individual grant recipients with different interventions. These sections follow the outline described in the previous paragraph to present findings first for FEGs and then for individual grant recipients. Apricot Value Chain ASF’s assessment of the apricot sector in Gilgit-Baltistan (GB) reported that the region has great potential for apricot production and is, in fact, one of the main apricot-producing regions in the country.16 Most rural households cultivate apricot as a cash crop and for home consumption. The report concluded that developing the apricot sector has the potential to accelerate growth of rural economies in the region and offers good opportunities for income and employment in apricot processing. Furthermore, women are largely responsible for cultivating and processing (drying) apricots. However, there are many barriers to fully realizing the potential of apricot in the region. Many of the apricot varieties are not commercially valuable; fresh apricots are very perishable; poor transportation infrastructure and distance from main markets inhibit access to markets for fresh fruit; commercial scale processing (i.e., drying, pulping) capacity in the region is very limited; and traditional processing (drying) methods by which most apricots are preserved for sale (largely by individual households) have limited capacity and do not meet standards for high-end or export markets. All of these factors contribute to a high level of waste driven largely by non-commercial varieties, limited capacity to preserve large quantities in the short period of time available after harvest, and traditional drying methods that expose fruit to damage from rain and contamination. To address the limitations of the apricot value chain in GB, TAP: 16 Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Gilgit-Baltistan Apricot - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. THE AGRIBUSINESS PROJECT EVALUATION 15  Formed 80 apricot FEGs. The FEGs were comprised largely of women with a total of 599 female members and 201 male members. The numbers are suspect, however, since ASF’s data lists only men as members of several “women’s” groups. Furthermore, the seven randomly selected apricot FEGs the evaluation team interviewed were comprised entirely of women. TAP provided grants to individual FEG members to construct a total of 800 solar drying tunnels and supported the grants with training in drying. TAP anticipated that the dryers and technical assistance would increase drying capacity, improve quality, and increase income from sale of dried apricots (largely for women).17  Provided individual grants to establish 18 apricot orchards with commercial apricot varieties. ASF expected the orchards to serve as demonstrations of commercially oriented production and thus contribute to broader adoption of commercial varieties and improved practices.  Provided individual grants for 33 commercial scale solar dryers, perhaps 15 of which were used for apricots.18 ASF expected the solar dryers to increase processing capacity and provide a market outlet for a greater quantity of fruit. Apricot FEGs All seven apricot FEGs the evaluation team interviewed reported receiving solar drying tunnels and training (Table 16 in Annex 9). The grants to FEG members introduced technologies and practices they would probably not have adopted otherwise. Two of the seven FEGs reported that they were not aware of the technology prior to TAP. Three of the seven, however, said they were aware of the technology (two because of prior projects and one because a neighboring farmer had a drying tunnel) but could not afford to implement the technology without financial assistance. Six of the seven FEGs reported attending a four-day training that covered harvesting, washing, grading, drying, packing, storing, and marketing. One of these groups said that the trainer demonstrated processing techniques with a machine and the FEG does not have access to a machine. One group mentioned training in pruning and one in jam-making. All seven FEGs reported changing their practices as a result of the drying tunnels and training (Table 20 in Annex 9). In particular, all seven mentioned changing from traditional drying methods outdoors on trays, stones, or rooftops to the tunnels. The benefits of improved drying techniques included reduced waste (6 of 7 groups), faster drying times (3 of 7 groups), and less work (1 of 7 groups). “Before using these solar dryers we used to dry apricots through traditional techniques, i.e., putting all the produce into stick trays and placing them on the rocks in sunlight for days. That technique took apricots six to seven days to dry if there was no rain and/or winds. Additionally, it took a lot of hard work, for instance taking all the baskets inside the house (indoors) at night to prevent the produce from getting damaged from humidity and possible rainfall at nighttime.” (Apricot FEG) 17 TAP M&E data and Agribusiness Support Fund. (2013). The Agribusiness Project Annual Progress Report - II, October 1, 2012 - September 30, 2013. Lahore: Agribusiness Support Fund. 18 The data provided by ASF does not identify the use of the dryers. However, 15 grants were awarded to recipients in GB where TAP supported only apricot and seed potato. THE AGRIBUSINESS PROJECT EVALUATION 16 Six of the seven FEGs reported increasing the quantity of (dried) apricots they produced. They attributed the result to improved post-harvest practices that replaced traditional drying practices and reduced waste (6 of 7 groups - see previous paragraph). Even though the tunnels and training increased the quantities of dried apricots the FEGs produced, none reported increasing the value of sales (Table 24 in Annex 9). Two of the seven said it was too early to tell. They were using the tunnels for the first time and had not yet sold what they were currently drying. They both said that the quantity and quality were better than what they were able to achieve using their traditional methods, and they were hopeful that the price would be good, citing market prices of Rs. 15- 20 per kg for poor quality (faded color) apricots and up to Rs. 100 per kg for colorful and clean apricots. Regarding employment, none of the seven FEGs reported increasing employment. The three that spoke directly to the issue said that apricot production was a family business and, while they helped each other when necessary, they had no culture of hiring labor. The four that spoke about the role of men and women in apricot production and processing said that women performed the bulk of the labor. Men helped with harvesting and marketing, but women did all of the processing work. The FEGs were dissatisfied with three aspects of implementation, all related to the tunnels (Table 32 in Annex 9). All seven reported that heat, high winds, and heavy rains had destroyed the plastic covering on the tunnels. All of the members of one FEG and at least some in another FEG (both of whom were using their tunnels for a second season) reported losing much of their crop in the previous season because the tunnels were not adequately ventilated, which increased humidity to a level that spoiled the apricots (i.e., turned them black). They reported that ASF replaced the tunnel ends with breathable mesh in the second year and they were experiencing no problems. The mesh also reduced stress on the plastic during harsh weather and solved the problem of damage due to wind. ASF personnel confirmed the design flaw. None of the four FEGs that had received their tunnels the following year mentioned this issue. Six of the seven groups said that the tunnels were too small to dry large quantities of apricots and wanted larger tunnels so they could use them for other purposes as well, e.g., vegetable production and drying. Six of the seven groups reported that the Aga Khan Rural Support Programme (AKRSP, ASF’s local partner) had installed the tunnels near or at the end of the drying season in 2014, so they did not have a full season of experience with the tunnels. When asked whether they would continue to use the practices they had learned and what might cause them to stop, only two FEGs voiced concerns about continuing the practices, and their concerns were related to the anticipated expense of replacing the plastic covering if it was damaged. It was not possible with the data provided by ASF for the evaluation team to determine the value of sales or employment ASF attributed to its activities with apricot FEGs. Conclusions: ASF support was instrumental in relieving financial and awareness constraints to adopting solar drying technologies. The tunnels and training (the project design) were relevant and effective interventions for apricot FEGs. They changed production practices and increased the quantities of dried apricots the FEGs produced, largely by reducing waste, women’s work load, and drying time. Because the interventions increased the quantity, quality, and price of dried apricots produced, there is good reason to expect that the tunnels will contribute to increased value of sales as the women gain experience. ASF’s only apparent misstep in implementation was the initial design of the tunnels (insufficient ventilation), and it corrected the design quickly and effectively. At the FEG level, the apricot value chain holds little potential to create jobs but does effectively engage, and materially benefit, women. It is too early to assess the sustainability of results. However, the sustainability of any long-term outcomes that do eventually materialize depends on the sustainability of the tunnels, and two of seven FEGs questioned their ability to repair the tunnels if they were damaged. THE AGRIBUSINESS PROJECT EVALUATION 17 Individual Grant Recipients The six individual grant recipients the evaluation team interviewed in the apricot value chain received grants to establish orchards of improved (commercial) varieties. A majority (i.e., at least four) grant recipients reported receiving plants, training (orchard management), tools (for pruning), and infrastructure (poles and wire for a perimeter fence). Two of the six grant recipients said they would not have established the orchards without ASF because they had not considered organized orchards. Four of six said that they could not have afforded the cost of establishing the orchard without assistance. Only two of the six grant recipients reported changing their practices as a result of TAP, and both mentioned pruning (Table 21 in Annex 9). Since the orchards had not reached bearing age, none of the grant recipients reported changes in production or sales (Table 24 in Annex 9). Five of the six orchard owners reported hiring labor specifically for the orchard – two had permanent workers managing the orchards and four reported hiring casual labor to establish the orchards and build fence (Table 28 in Annex 9). One anticipated that the orchards would require more labor when they started producing fruit, especially for women in picking and processing. ASF did not report consistent data on results for apricot orchards. The disaggregated second quarter 2015 data provided by ASF reports seven full-time equivalent (FTE) jobs created to establish apricot orchards. The value does not seem unreasonable based on the evaluation findings. Conclusions: TAP assistance establishing commercial apricot orchards addressed the financial constraints farmers faced adopting the practice. It is too early to determine whether the orchards (the project design) are an effective intervention in terms of the long-term outcomes of leading a transformation to commercially oriented production practices. In the short run, however, there is every reason to believe that the trees will produce fruit and contribute to owners’ sales and incomes. Sales will depend on being able to get fresh fruit to markets or processing it, but ASF’s assessment of the value chain suggests that commercial varieties are rarely wasted, so there is good reason to believe that the orchard owners will find markets once the trees begin to bear fruit.19 Once they are mature, the orchards are likely to generate some jobs managing orchards and seasonal jobs in harvesting and processing. Grape Value Chain In interviews, ASF staff members reported that the Potohar region (the districts of Jhelum, Rawalpindi, Chakwal, and Attock) in northern Punjab provide an ideal opportunity for producing table grapes. The region can produce early-ripening varieties (Kings Ruby and Seedless Flame) that reach the market before it is glutted with other varieties and before the monsoon rains damage the crops. Prior to the de-scoping, ASF established 20 grape-oriented FEGs and provided 17 grants to individual farmers to establish vineyards. ASF discontinued its work on the grape value chain during the de-scoping but reinstated it in October 2014 due to demand from producers and USAID/Pakistan’s interest. 20 19 Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Gilgit-Baltistan Apricot - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. 20 USAID/Pakistan. (2014). Modification of Assistance No. 7. Islamabad: USAID/Pakistan. THE AGRIBUSINESS PROJECT EVALUATION 18 When ASF resumed its work in the grape value chain after the de-scoping, it shifted focus away from FEGs to building the capacity of larger vineyard owners to support commercial grape production.21 An interview with the National Rural Support Programme (NRSP), ASF’s local implementing partner in the region, confirmed this shift in focus. In the third annual report, ASF reported that it had:  Formed 17 grape-oriented FEGs and provided the FEGs with grants for trellises sufficient to establish three-acre vineyards and with tools and training to produce grapes.  Provided grants to 80 individual farmers to establish thee-acre vineyards. ASF staff members reported that the rationale for establishing these orchards was to demonstrate the opportunities in the grape sub-sector to other farmers and thus encourage replication. Grape FEGs The evaluation team interviewed six of the 17 ASF-supported grape FEGs.22 The six FEGs reported receiving trellises (4 of 6 FEGs), tools (3 of 6 FEGs), and training (4 of 6 FEGs) (Table 17 in Annex 9). Four of the FEGs reported receiving the trellises in 2013 and two received them in 2014. In addition to the trellises, one FEG member received netting to protect the fruit from birds (ASF provided a limited number of nets as a demonstration) and reported installing drip irrigation on his own. Tools included a pruning saw, a cutting tool for harvesting, and a first aid kit. Two of the FEGs said that ASF provided just one set of tools for all FEG members to share. The FEGs reported receiving training on erecting the trellis system, planting, and disease and pest management. The story that emerges from the FEG interviews suggests that ASF conducted most of these trainings in venues other than the farm and trained just a few members from each FEG. One FEG member specifically said that training did not include business training. None of the FEGs would have cultivated grapes without TAP support. All five of the FEGs that addressed the question said they had no knowledge of grapes and had not considered growing them (Table 19 in Annex 9). One FEG said they could not have afforded the trellises on their own even if they had known of the opportunity. Because they had no prior experience, producing grapes represented a new practice for all of the FEGs. Because it was their first experience with grapes, the FEGs were not able to say whether quantities produced had increased. The three who spoke about production said they were not sure about the quantity they could ultimately produce because they had only recently established the vineyards (one in 2013 and two in 2014) and they were not fully mature (Table 23 in Annex 9). One reported producing 300 crates of grapes (2,100 kg) from the three-acre vineyard. Because of immature vineyards and limited production, the three FEG that reported selling grapes sold limited quantities. The one that gave detailed information about sales reported selling 2,100 kg for Rs. 40,000 (Rs. 57/kg). One reported earning between Rs. 70,000 and 80,000 selling grapes and the other reported receiving between Rs. 500 and 700 per crate for Kings Ruby. In spite of limited production and experience so far, however, the FEGs were quite optimistic about their prospects with grapes. The project’s performance data for the 21 USAID/Pakistan. (2014). Modification of Assistance No. 7. Islamabad: USAID/Pakistan. 22 The sample included seven FEGs but one was not active and the team could not conduct an interview. THE AGRIBUSINESS PROJECT EVALUATION 19 second quarter of 2015 reports no sales for grape FEGs because ASF collected the data before the harvesting season. “Now due to this assistance things have changed, for example previously by cultivating a land of three acres we could not earn more than 100,000 per year. However now, thanks to Allah, with this assistance things will improve. Like for example this year (year two of plantation) even though we had some diseases and early rains and humidity affected our crop we still sold about Rs. 70-80,000 worth of grapes.” (Grape FEG) In grapes in particular, it was very difficult to determine whether farmers were employing additional labor. Larger farmers, and even some FEG members, employed permanent workers who lived on the farm, and they shifted these workers between tasks as necessary. It was sometimes difficult to determine whether a worker in a vineyard represented a new job or merely a new task for an existing employee. It does seem clear, however, that at least three of the FEGs had hired additional labor to manage the vineyards (Table 29 in Annex 9). Three mentioned hiring full-time employees to keep an eye on the vineyard, although these workers do not necessarily work only in the vineyard or represent new employees. Three FEGs also specifically said they employed casual labor for weeding and harvesting. The same caveats apply, i.e., that it is difficult to determine whether these workers represent new employment. The five FEGs that provided an answer all said women have no role in producing grapes. If they have any role at all, it is in packing grapes after harvest, a job they can perform inside their homes. ASF’s second quarter 2015 M&E data reports that grape FEGs employed eight FTE workers during the quarter, a value that seems reasonable in light of the interview data. Five of the FEGs said they expected to continue producing grapes (Table 31 in Annex 9). Four cited the substantial investment they had made in the infrastructure. Two acknowledged that, while they were optimistic, sustaining the intervention would depend on profitability. One alluded to the difficulty of repairing trellises but implied that they would manage to do so. Of five FEGs that responded to questions about replication, three reported that neighboring farmers had observed their vineyards and replicated the model. A majority of grape FEGs voiced issues with tools (5 of 6 FEGs), infrastructure quality (3 of 6 FEGs), infrastructure design (5 of 6 FEGs), training (5 of 6 FEGs), and timing (5 of 6 FEGs). Table 4 provides additional detail about the nature of the issues. FEGs viewed nets as critical to the success of their vineyards and were disappointed that ASF did not provide nets. ASF also appears to have had difficulty providing the material for trellises on time or in the quantities FEGs expected. Conclusions: TAP support was instrumental in encouraging FEGs to invest in grapes. While a few FEGs knew about grapes, none had considered producing grapes themselves. In addition to lack of awareness, without external support FEGs could not afford the investment in trellises necessary to produce grapes. Whether the design (i.e., trellises) is effective will depend on future production and sales. The farmers were optimistic that grapes will be more profitable than the alternatives but, because vines are not yet fully mature, they have no direct experience. The value chain seems to offer prospects for full-time and casual labor (for men) but it is difficult to determine actual requirements for new labor. From an implementation perspective, most FEGs complained that ASF provided the trellises too long after they planted the vines. FEGs also reported suffering large losses due to birds and were disappointed that ASF had not provided nets, material many could not afford on their own. The sustainability of results is questionable for these FEGs if they cannot limit bird predation. There is reasonably strong evidence that other farmers are attempting to replicate ASF-promoted grape production techniques on their own based on the demonstration effect of the FEG intervention. THE AGRIBUSINESS PROJECT EVALUATION 20 TABLE 4: IMPLEMENTATION ISSUES – GRAPE FEGS Issue Detailed Issues Number of FEGs Tools Poor quality tools relative to expectations 1 of 5 FEGs Received traditional tools they already have, not advanced designs they expected 2 of 5 FEGs Infrastructure design Need nets to keep birds from damaging crop 5 of 5 FEGs Training Need additional training (business, pruning, disease identification and treatment, marketing, storage) 3 of 5 FEGs Timing Pillars and wire arrived too long after plants (ASF recommended setting the trellis 3 months after planting and it took between 7 months and a year to receive the trellises.) 4 of 4 FEGs Infrastructure quality Quantity of pillars and wire not sufficient for area and less than promised 2 of 3 FEGs Poor quality pillars and wire 2 of 3 FEGs Individual Grant Recipients The evaluation team selected five grant recipients for interviews. However, nobody was available for an interview at one site, and the farm, including the vineyard, was untended and appeared not to be actively farmed. The findings are thus based on four interviews. Consistent with the design of the intervention, a majority of the five individual grant recipients reported receiving plants (3 of 4 recipients), training (4 of 4 recipients), and trellises (3 of 4 recipients) (Table 17 in Annex 9). Training topics included vineyard management, water management, integrated pest management, pruning, disease identification and management, basic accounting and record keeping, marketing, and trellis construction. Four of the five also said ASF had taken them to visit established vineyards, and two of the four said the visits were instrumental in their decisions to invest. All grant recipients were relatively well-off landowners with adequate financial resources. Only one, however, had any experience growing grapes, and he reported having only a couple of rows of vines. Of the three recipients who responded to the question, two specifically said they had the financial resources to invest in grape production, but all three said they lacked awareness of the opportunity (Table 19 in Annex 9). Since three of the four grant recipients the evaluation team interviewed had no previous experience producing grapes, the technologies and practices TAP introduced represented new practices. One specifically said that the most important change in his practices was keeping accurate records of inventory, expenses, and income. Similarly, since it was a new activity for most grant recipients, they could not respond to whether TAP had increased the quantities of grapes they produced or the value of sales. Only one grant recipient, the one with existing vines, reported selling grapes. Three others said their vines were immature and they had nothing to sell. The project’s M&E data reported no sales in the second quarter of 2015. Three grant recipients reported employing additional labor for the vineyard (Table 29 in Annex 9). One reported hiring a permanent vineyard management expert; one reported shifting two existing permanent employees to the vineyard and also hiring 8 to10 casual laborers during the working season (i.e., weeding, pruning, harvesting); and one said he had hired some casual workers to help erect the THE AGRIBUSINESS PROJECT EVALUATION 21 trellises. One also said he had allocated two permanent employees to the vineyard and employed three to four casual laborers for two weeks per month to help. None reported employing women, and all three recipients who responded to the question said that women played no role in grape production. ASF reported that individual grant recipients in the grape value chain created about 28 FTE jobs in the second quarter of 2015. Spread across 80 vineyards, this value seems not inconsistent with the interview data. When asked whether they intended to continue producing grapes, two of the grant recipients said they expected to sustain the practice. One cited the fact that he is replacing damaged plants as an indicator of sustainability. The other implied that he would decide on the basis of future (i.e., when the vineyard matured) profitability but, citing the experience of a friend in the business, he was confident it would be profitable. None of the four grant recipients said that other farmers (i.e., not involved in TAP) had adopted the TAP-promoted technologies and practices for grape production. However, three of the four grant recipients the evaluation team interviewed said visits to established vineyards helped convince them to invest. Two said that limited access to irrigation, the cost of waiting three years for a result, and unwillingness to invest in proof of concept for high risk activities may curb other farmers’ interest. One also said that limited access to BDSPs and technical assistance were important barriers to replication. One recipient said that the trellises were now readily available in the market, which would facilitate replication. “As mentioned earlier that the vineyard is only one year old. At present, profit earning is not the purpose. I just want to make it successful in terms of crop establishment. However, my friends who are engaged in the vineyard business are earning good money. A friend from Attock has earned about Rs. 2,800,000 from sale of grapes over an area of 50-60 kanals in a year. That shows that it is going to be a profitable business.” (Grape grant recipient) Grape grant recipients had no major complaints about implementation. Conclusions: Individual grant recipients were relatively well-off landowners with other sources of income and were looking for investment opportunities. Visits to farms with active vineyards and ASF’s outreach convinced them that grapes represented a promising opportunity. Whether the project is pursuing an effective strategy remains to be seen since most of the vineyards were not yet mature (all four established their vineyards in early 2014) so none of the farmers had produced or sold significant quantities of grapes. The experience of established orchards, however, implies that the design is appropriate. Some recipients appear to have hired some labor and many had assigned permanent workers to the vineyards, but it is difficult to determine the nature of employment effects. What is clear is that women play no role in the grape value chain, at least at the production level. The intervention and results appear sustainable, but ultimately sustainability will depend on profitability. ASF appears to have effectively implemented activities with individual grant recipients in the grape value chain. HV/OSV Value Chain ASF’s assessment of the HV/OSV value chain concluded that the demand for vegetables in the winter months is strong and that Pakistan has a comparative advantage (based on growing conditions, THE AGRIBUSINESS PROJECT EVALUATION 22 inexpensive labor, and geographic proximity) to serve these markets.23 However, it also identified a number of constraints to increasing the competitiveness of the industry. These include the lack of institutional support for developing markets; limited access to trade finance; low air transport capacity; lack of cold chain infrastructure; production and post-harvest practices that result in high losses and do not meet international demands and standards; and limited availability of production credit. ASF’s strategy in the HV/OSV value chain was to build the capacities of small and larger farmers to produce high-value vegetables and extend the season to provide some portion of production at times when the quantity supplied to the market was low and prices high. Interventions with FEGs and individual farmers focused on domestic markets, while those with traders/exporters (addressed through challenge grants and IMAP) addressed some of the constraints to international market access. TAP’s farm-level interventions in the value chain included:  Forming 335 FEGs in GB (53), KP (222), and Punjab (60). The evaluation team interviewed HV/OSV FEGs in KP (12), Punjab (6), and GB (1).24 The implementation approach varied by region. Interviews with FEGs and local partners revealed that:  The Rural Community Development Society (RCDS), the local partner in Punjab, provided tomato and cucumber seed; comprehensive training in production, harvesting, grading, packaging, marketing techniques, and business development; and tools and equipment (cultivation tools, harvesting baskets, scales, packaging material, wheel barrows, and watering equipment).  The Sarhad Rural Support Programme (SRSP) (the local partner in Peshawar) and the Hashoo Foundation (the local partner in Mansehra) provided poles and wire to construct structures for vertical farming; cultivation and harvesting tools and equipment (wheelbarrows, cutting tools, cultivating tools, and baskets); hybrid seeds (bitter gourd, tomato, cucumber, green chilies, round gourd, and straight gourd);25 and extensive training on vertical farming, cultivation practices, harvesting, business development and recordkeeping, and marketing.  Provided 315 grants to individual farmers to construct high tunnels for producing vegetables. Table 14 in Annex 8 documents the distribution of these grants across regions. HV/OSV FEGs TAP’s support to HV/OSV FEGs varied by region. FEGs in KP and Punjab reported receiving hybrid seeds, tools and equipment (cultivation tools, harvesting baskets, scales, packaging material, wheel barrows, and watering equipment), and comprehensive training in production, harvesting, grading, packaging, marketing practices and techniques, business development, and recordkeeping (Table 17 in Annex 9). The primary difference between implementation approaches in the two regions was that SRSP 23 Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) HVOSV - Value Chain Competitiveness Assessment for Bitter Gourd, Capsicum, Cucumber, Tomatoes, Squash. Lahore: Agribusiness Support Fund. 24 Table 14 in Annex 8 documents the distribution of FEGS and the sample across regions. 25 SRSP provided seed for bitter gourd, tomato, cucumber, and straight gourd. Hashoo Foundation provided seed for tomato, cucumber, round gourd, and green chilies. THE AGRIBUSINESS PROJECT EVALUATION 23 and the Hashoo Foundation (the local partners in Peshawar and Mansehra, respectively) promoted structure/vertical farming,26 while RCDS in Punjab did not. FEGs reported changing a number of practices (Table 21 in Annex 9). The most common were changes in general production practices (14 of 18 FEGs), new seed varieties (5 of 18 FEGs), and cropping patterns and marketing (4 of 18 FEGs each). Changes in production practices referred to structure farming (5 of the 14 FEGs from KP), changing from broadcasting seed to planting in rows with specified spacing (4 of 14 FEGs), and using small tunnels to establish seedlings early (3 of 14 FEGs). Fourteen of the 18 FEGs reported that the new production practices increased the quantity of vegetables they produced. They attributed the increase to reduced waste, higher yields, and new technologies and practices (Table 22 in Annex 9). They attributed reduced waste to structure farming, using proper baskets and bags for harvesting and transportation, and to training that helped them identify and treat diseases. They attributed higher yields to improved seeds, longer producing seasons, and structure farming. New technologies and practices referred to better quality seeds and structure farming (Table 5). TABLE 5: REASONS FOR INCREASED PRODUCTION Reasons Number of FEGs Detailed Reasons Number of FEGs Reduced waste 12 of 18 Structures that kept vegetables off the ground 6 of 12 Proper baskets and bags that reduced damage during harvest and transportation 4 of 12 Training to identify and treat diseases 3 of 12 Increased yield 11 of 18 Hybrid seeds that were more productive 4 of 11 Longer producing season 4 of 11 Structure farming 3 of 11 New technologies and practices 7 of 18 Better quality seed 5 of 7 Structure farming 2 of 7 “On average we have earned Rs. 500,000 to 600,000 per acre [with bitter gourd] as compare to Rs. 200,000 per acre in traditional way of cultivation.” (HV/OSV FEG) “Using structure farming, we produce 12 to 15 bags yield per canal [compared to 5 to 7 bags using traditional methods], and we collect the produce every four days. Also the structure farming and use of new seeds gives us yield till even four months.” (HV/OSV FEG) “Prior to support from TAP we used to grow vegetables during specific months (five to six months a year – because we also used to grow sugar cane) and now the cultivation is carried out almost all the year.” (HV/OSV FEG) 26 Farming on trellises that keep vegetables off the ground. THE AGRIBUSINESS PROJECT EVALUATION 24 Ten of the 18 HV/OSV FEGs reported increasing the value of their vegetable sales. Two who reported the value of sales noted values between Rs. 130,000 and Rs. 150,000 per year. None said the value of sales had stayed the same or declined (Table 24 in Annex 9). The 10 FEGs that repored increased value of sales attributed the increase to producing (and selling) larger quantities (7 of 10 FEGs), changing cropping patterns (3 of 10 FEGs), and obtaining higher prices due to grading and learning how to market (2 of 10 FEGs). The project reported incremental sales of USD 363,445 among HV/OSV FEGs in the second quarter of 2015, an average of USD 1,084 (Rs. 108,400) per FEG, values that seem roughly consistent with the thin interview data. “While growing sugarcane which was the most expensive (selling) of our traditional crop (wheat and corn) from one kanal of land we would make production of about five mann (one mann = 40 Kg) making a profit of about Rs. 20,000 per season. While bitter gourd at time of harvesting would give us 40 bags per week and we would sell one bag for Rs. 120, so in [just] a month during harvesting season we would make 25,000-30,000 so the profit difference is quite huge.” (HV/OSV FEG) Of the 18 HV/OSV FEGs the evaluation team interviewed, 8 reported hiring additional labor and 10 reported no change in employment (Table 28 in Annex 9). The eight FEGs that reported hiring more labor attributed the requirement to increased quantities produced, a longer producing season, and more labor-intensive structure farming methods (e.g., time tending plants). Specifically, the eight FEGs said they needed extra help with harvesting (5 of 10 FEGs), tending the growing crop (5 of 10 FEGs), and loading trucks for transport to market (1 of 10 FEGs). Although the group interviews with FEGs did not yield quantitative measures of new employment, the evidence suggests that additional employment could be substantial and engage a large number of women. Eight FEGs said they employed only men, three said they employed mostly women, and five said they employed men and women. The project’s performance data for the second quarter of 2015 reported 692 new FTE employees in project-supported FEGs, 67 percent of which were women. Spread across 335 FEGs, this implies just under one-half FTE (i.e., one person employed for six months) per FEG on average, a value that does not seem inconsistent with the interview data. “Only additional labor was required at the time of harvest. So structure farming has increased the number of days almost four times as compared to before. Before if they were required for 10 to 12 days, now we require additional labor for three months on average if we take six months season.” (HV/OSV FEG) A number of HV/OSV FEGs were very satisfied with the hybrid seeds (8 of 18 FEGs) and training (5 of 18 FEGs) they received through TAP (Table 34 in Annex 9). They found these interventions to be relevant, beneficial, and effective. However, FEGs also voiced a large number of complaints about project implementation. The most frequent complaints involved the timing of the intervention (10 of 18 FEGs), infrastructure design (8 of 18 FEGs), and training (6 of 18 FEGs). It is important to note that 4 of 18 FEGs (all among the 6 from Lahore) were dissatisfied with the quality of cucumber seed they received. Issues with timing included seeds arriving after the best season for planting (6 of 10 FEGs) and seeds and structures provided later than promised/expected (3 of 10 FEGs). Issues with infrastructure design focused on dissatisfaction with bamboo instead of steel structures (7 of 8 FEGs). It is important to note that FEGs were not promised steel but merely expressed a preference for it over bamboo. FEGs were not dissatisfied with the training per se, they just wanted more of it (4 of 6 FEGs) and training in marketing (2 of 6 FEGs). “Due to late receiving of seeds we didn’t cultivate any other vegetables on the 25 kanal for six months. ASF has wasted our time and money and now we are doing work as a laborer in THE AGRIBUSINESS PROJECT EVALUATION 25 others’ lands due to huge loss. If they had given money for the seeds instead we would have purchased good quality seeds and on proper time.” (HV/OSV FEG) “Another thing is that if were allowed to buy the same structure for ourselves and were only provided with cash grant support for this, the quality of material used for these structures in structure farming would have been better. Because bamboos didn’t work for us, they were damaged after some time. Now, almost 60 to 65 per cent of the bamboos have been replaced by us on our own.” (HV/OSV FEG) The four FEGs that were dissatisfied with the quality of the seed they received were all from Lahore and all received cucumber seed. The same four FEGs complained that ASF delivered the seed too late to plant at the correct time. When asked whether they would continue using the technologies and practices they learned through TAP, 11 FEGs said yes and 2 said no (Table 30 in Annex 9). The two who said they would not continue the practices cited not being able to find certified vegetable seeds in the market. Those who said they would continue cited the profitability (6 of 11 FEGs) and the investment in the structure (5 of 11 FEGs). One of the FEGs demonstrated sustainability by replacing the structure’s bamboo poles when they rotted. “We have stopped the cultivation of vegetables because we didn’t find that seed in the market and it’s also expensive too. And vegetable is more profitable than the maize, wheat crop. So we want it to continue if we are provided with the seeds.” (HV/OSV FEG) “But most importantly we need more seed especially hybrid seeds. We can buy them from market but in market the seeds are very low quality and it is difficult to find the trustworthy dealer or seeds locally.” (HV/OSV FEG) “We will use this practice for sure. For example some of the bamboos provided by TAP are not useable now, but we have replaced the damaged ones from our own pocket. As soon as one bamboo is damaged or is not good enough to use (to stand firm or to hold intact) we immediately replace it.” (HV/OSV FEG) “We will not stop using this practice. It is much better than before. We will even go for loans but we will not stop using structure farming.” (HV/OSV FEG) Conclusions: Interventions with HV/OSV FEGs were largely well designed and effective. A large majority of FEGs adopted the new technologies and practices, especially the hybrid seeds and vertical farming structures, and substantially increased their production and sales as a result. Increased sales were driven by both quantity and quality/price even though most FEGs reported selling through their normal channels in local mandis which do not generally reward quality. FEG-level vegetable production also generates a substantial number of jobs, many open to women, due to more labor-intensive production practices and harvesting larger quantities over a longer period of time. Sustainability of results seems likely, especially since the structures are relatively low-cost and quite profitable relative to the alternatives. In fact, several FEGs reported maintaining the infrastructure. The apparent difficulty finding good quality hybrid seeds in the market may threaten sustainability. ASF struggled to provide inputs (seeds) on time and of good quality, although this problem seems largely confined to Lahore. Individual Grant Recipients All individual HV/OSV grant recipients received four high tunnels in which to cultivate vegetables. The tunnels and associated training facilitated off-season production, incorporated vertical structures on THE AGRIBUSINESS PROJECT EVALUATION 26 which to cultivate high-yielding varieties, and improved quality and yield by providing a controlled environment. ASF staff stated that by providing cost-sharing grants to individuals with land and resources, ASF hoped to demonstrate new technologies and practices to stimulate further private sector investment. The seven grant recipients the evaluation team interviewed were relatively well-off landowners with other business interests. At least two were lawyers, two were horticulturalists who taught high tunnel farming and had consulted with ASF, and one mentioned being involved in multiple businesses. Six said they had no previous experience with tunnels or commercial vegetable production. When asked whether they could have made the investments without TAP support, five of the seven said they were not aware of the technology and its economic potential (of the two who were aware, one taught high tunnel farming and the other had been involved in multiple agricultural businesses). Only two (the horticulturalist/consultant and a farmer without other business interests) said they lacked the capital to make the investments. “It is very difficult to establish the high tunnel farm without TAP assistance because tunnel farming is a new method of farming but no one wants to take a risk to invest his money. It’s a quite expensive method.” (HV/OSV Individual Grant Recipient) All seven grant recipients the evaluation team interviewed reported receiving training. Topics included tunnel farming, disease identification and treatment, enterprise development, and marketing. One grant recipient mentioned practical training in preparing nurseries and transplanting seedlings into the tunnels. Three of the seven mentioned exposure visits to other tunnel farming operations. Five of the seven grant recipients specifically mentioned increasing the quantity of vegetables they produced due to the tunnels (Table 22 in Annex 9). Since at least six of the seven HV/OSV grant recipients had no previous experience with tunnels, merely adopting tunnel farming represents a change in practices. Six of the seven HV/OSV grant recipients reported that the value of vegetables they sold had increased. Because most had no previous experience producing vegetables commercially, any sale was an increase over previous years. The two respondents in Peshawar reported that a strong windstorm that damaged their tunnels while the crops were standing, but they were able to harvest some of the tomato and cucumber and seemed happy with the result, even though it was less than they expected. Two grant recipients attributed the increase in value of sales to being able to provide tomato to the market early in the season. One cited that he graded his produce and got good prices as a result. Three respondents reported lower than expected sales (one due to adverse weather and one to choosing poor quality seeds) but still seemed satisfied with the result. Although the quantiative data are inconsistent, the four respondents who provided sales data reported sales of Rs. 100,000/day, Rs. 225,000 - 1.1 million per year, Rs. 2,800 - 11,000/week, and Rs 40,000 - 50,000/year. ASF reported no sales for HV/OSV tunnel farms in the second quarter of 2015. “Despite all the damage due to natural calamity, we liked the new vegetable farming technique as it was far better than the traditional farming in terms of crop management and yield. The yield of hybrid tomatoes produced under tunnel is higher than the conventional vegetable growing method. The production of one acre in open filed in equal to 3 kanals (3/8 acre) of production under tunnel farming. So, we are very happy with this new technology.” (HV/OSV Individual Grant Recipient) Six of the seven HV/OSV grant recipients reported hiring additional labor. The six that reported hiring additional labor attributed the need for additional labor to “production” generally (4 of 6 grant recipients); weeding (2 of 6 grant recipients); harvesting (2 of 6 grant recipients); and marketing (1 o 6 grant recipients). Two of the six specifically mentioned hiring women; two said the culture in their area did not permit women to work in production; and four said that women do work in vegetable THE AGRIBUSINESS PROJECT EVALUATION 27 production for hoeing, transplating, pruning, and picking while men handle land preparation, irrigation, spraying, and marketing. ASF reported that HV/OSV grant recipients created 159 FTE jobs in the second quarter of 2015, 21 percent of which went to women. This represents an average of one-half FTE per grant recipinet, which does not seem inconsistent with the interview data. “Yes, we employed labor. Most of our labor was female. During harvesting and weeding, eight to nine women were involved, some on daily wages and some on monthly basis and they worked for about five to six months. The daily wagers charge Rs. 300/day. Women prefer working within protected areas. I also hired two permanent male laborers for spray and to go to mandi.” (HV/OSV Individual Grant Recipient) “They have hired three men as permanent labor and eight women as causal labor. During the picking season (almost 7 months in a year) they are hiring additional 20 women and eight men. (HV/OSV Individual Grant Recipient) HV/OSV individual grant recipients had few issues with implementation. Issues raised by three or more of the seven grant recipients the evaluation team interviewed included tunnel design, training, and timing (Table 6). Evidence from one HV/OSV grant recipient suggests that ASF’s attempt to link producers to large supermarkets was not particularly successful. The producers reported problems complying with buyers’ quality requirements, not being properly registered, the timing of purchases, and slow payment mechnisms. “They [ASF] created linkages with high end markets such as Hyper-star, Metro, Fresh Foods and a few other exporters. However we could not continue with Hyper-star and Metro due to the strict compliances. These stores do not allow farmers to sell their produce as a whole in the market. They take vegetables after grading and select only the best quality and most of our produce goes waste. Therefore we could not strengthen our linkages with such high end markets. Right now we have 35 bags (20 kg each) of bitter guard and if we take these bags to super markets, they will take half of our produce and will reject half. We had an agreement with Metro but could not sell. These super markets take the product in the morning and picking of vegetables is done in the evening. So, he had to hire a warehouse and two persons to keep the product till morning and marketing. But, the rate was not competitive so we discontinued. The intermediaries in the business take most of the benefit. The payment mechanism in both Hyper Star and Metro are 15 days. While, in the open market you get instant payments.” (HV/OSV Individual Grant Recipient) Two of the seven HV/OSV grant recipients whose tunnels were destroyed by wind were already demonstrating a level of sustainability by repairing the tunnels at their own expense. Two others said they expected to continue the practices if they remained profitable. Six of the seven HV/OSV individual grant recipients said that other farmers had visited their farms to see and talk about the tunnels. Two reported that other farmers had replicated the practice while five said the technology was too expensive for other farmers to replicate without outside assistance. The interviews collected no information about the characteristics of the farmers who replicated the technologies or those who chose not to replicate because of cost. THE AGRIBUSINESS PROJECT EVALUATION 28 TABLE 6: IMPLEMENTATION ISSUES – HV/OSV INDIVIDUAL GRANT RECIPIENTS Issue with Implementation Detailed Issues Frequency of Response Tunnel design Faulty foundation 2 of 7 grant recipients, all interviewed recipients from Peshawar Tunnels need adequate ventalation and different covers to control temperature and control environment 2 of 7 grant recipients Training Additional training required 3 of 7 grant recipients (all in Lahore) Timing Tunnel installed too late for planting 3 of 7 grant recipients Four of the seven grant recipients reported that they would not have engaged in tunnel farming without TAP because they were not aware of the technology or the benefits. Only two (teachers who promoted tunnel farming) said they lacked the financial resources but had the knowledge. One recipient confirmed the importance of the demonstration effect by mentioning his reluctance to take the risk on an unfamiliar technology without external support. Conclusions: ASF targeted relatively well-off landowners with capital to invest or specialized knowledge. The data are difficult to interpret but suggest that ASF’s strategy of demonstrating the technology and practices to raise awareness of the opportunity afforded by the tunnels among financially capable farmers may be effective in stimulating adoption. At the individual grant level, the project design (i.e., tunnel farming) appears to have been effective in increasing production, sales, and employment. The only issues with implementation emerged in Lahore and involved the design and/or inspection of tunnels and the quality and timeliness of provision of cucumber seed. Tunnel farming appears to offer substantial potential for employing men and women. Potato Value Chain ASF’s assessment of the potato value chain concluded that Pakistan’s ability to produce potatoes earlier than many other countries and its proximity to Association of Southeast Asian Nations (ASEAN), Gulf Cooperation Council (GCC), and Commonwealth of Independent States (CIS) countries gives it a comparative advantage over European exporters in potato production.27 To capitalize on these opportunities, the report recommended that Pakistan work on improving seed quality and the capacity of producers to meet quality standards and reduce post-harvest losses (in part by adopting better storage practices), promoting value addition, and developing higher value export markets. TAP addressed issues in the potato value chain at two levels.  It formed 174 seed potato FEGs in GB (Chitral, Daimer, Ghizer, Gilgit, and Hunza districts) and KP (Mansehra district). The FEGs build the capacities of small potato farmers to increase the quantities and quality of seed potatoes they produce. 27 Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Potato - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. THE AGRIBUSINESS PROJECT EVALUATION 29  It provided grants to 11 individual potato producers in the Ghizer and Hunza districts of GB to construct storage cellars. The cellars should reduce post-harvest losses and improve the quality of seed potatoes stored over the winter. Potato FEGs All six potato FEGs the evaluation team interviewed reported receiving training and tools (Table 16 in Annex 9). Training included technical training in potato planting methods (land preparation, spacing, and seed quality), crop management (fertilizers, irrigation, and identifying and treating diseases), harvesting, grading (distinguishing good seed potatoes from table potatoes), storage, and marketing. ASF or its partners delivered some training in the classroom and some in the field. Tools included hoes, spades, hand cultivators, knives, measuring tapes, gloves, magnifying glasses, and hand sprayers. Four of the six FEGs reported receiving seeds and two specifically said they did not receive seeds. All six FEGs said they grew potatoes prior to TAP. However, four of the FEGs reported that they were not aware of the new technologies (certified seed) and practices (cultivation, grading, harvesting, and marketing) prior to TAP. As a result of TAP training, five of the six FEGs reported changing their production practices (Table 20 in Annex 9). Changes included spacing plants properly (3 of 5 FEGs) and using (small) seed potatoes instead of (large) table potatoes for seed, using certified seed, knowing how to identify and treat diseases and pests, preparing and applying fertilizers, planting seed potatoes and table potatoes separately, and rotating potatoes with wheat annually (1 of 5 FEGs each). The one FEG that did not report changing practices said they are using the same cultivation techniques and that the seed ASF provided (late) was inferior to what they could buy in the market. “In the past most of our potatoes were damaged and they were of poor quality. Insects also damaged the potatoes. But this year we used certified seeds and tried the new methods (which we learnt from TAP providing trainings) and we are expecting that our produce will be good.” (Seed potato FEG) “Before this, we used Punjab seed for household consumption which were brought only from Hunza. In Punjab seed, if they give us one bag of seed, they would take two bags in return after harvesting. So, we were in total loss. Therefore, we did not grow on commercial scale. Now, we are still repeating the same seed provided by ASF and quality is good with medium size potatoes. We have not bought any seed from outside. But, we know that we can only use this seed for four to five years and then we need to arrange new seed. After the fourth year the production starts declining so it’s important to replace seed after every four to five years.” (Seed potato FEG) The FEGs did not receive support from ASF until early 2013 (3 of 6 FEGs) or early 2014 (3 of 6 FEGs). Four of the six FEGs reported that support from TAP increased the quantity of potatoes they produced (Table 22 in Annex 9). They attributed the increase to fewer rotten potatoes (2 of 4 FEGs), planting larger areas (2 of 4 FEGs), and higher yields (1 of 4 FEGs). Only one FEG reported increasing the value of sales, and this was relative to the value of growing wheat on the same land (Table 24 in Annex 9). Two others expected greater profits from selling potatoes based on the appearance of the standing crop but had not yet sold. Another was producing potatoes only for household consumption. The two remaining FEGs did not respond. “Before these potatoes we used to grow wheat in our four kanal land and we only had Rs. 8,000 to 10,000 profit from it, When we exclude Rs. 4,000 amount for the field expense we only left with Rs. 6,000 profit which was very minimal. In the same field of four kanal when we cultivated the potatoes we get Rs. 40,000 seed from the company and after excluding this seed THE AGRIBUSINESS PROJECT EVALUATION 30 amount and spraying cost we are left with Rs. 80,000 round about, so it was huge difference in the profit, (Seed potato FEG) Four of the six FEGs specifically said that they had not hired additional labor as a result of TAP (Table 28 in Annex 9). Only one implied that some smaller families hired labor to prepare the land (i.e., build ridges in which to plant) but it was not clear if this was a new practice. Those who did not hire labor claimed that farming was a family business and it was not their tradition to hire labor. The groups said that men were responsible for preparing the land, irrigating, harvesting, and marketing while women were responsible for preparing the seed, planting, weeding, and harvesting. ASF reported no employment or sales among seed potato FEGs in the second quarter of 2015. When asked whether they expected to continue using the practices and technologies promoted by TAP, two of the FEGs said they would and two said they would not. The two FEGs that said they would sustain the practices said they would do so because they found the practices profitable. The two that said they did not expect to continue the practices cited concerns about access to good quality seed potatoes but said they would sustain the practices if they could obtain improved seed varieties. In fact, three of the six FEGs claimed they had limited access to good quality certified seed. The project’s individual grants to establish storage cellars were designed to address this constraint but, probably because the intervention is very new, no FEGs reported purchasing seed from cellar owners. Three of the six FEGs reported that other farmers in their area were starting to adopt the practices promoted by TAP, and two of the FEGs said they were teaching others. Seed potato FEGs voiced several complaints about implementation. These included concerns about seeds (4 of 6 FEGs), seeds arriving too late for planting (4 of 6 FEGs), and poor quality tools (3 of 6 FEGs) (Table 33 in Annex 9). Issues with seeds included poor quality/rotten seeds (3 of 4 FEGs) and uncertified or ungraded seeds (2 of 4 FEGs). Members of one FEG, however, made a point of saying that the tools dramatically increased their productivity. “The contractor did not provide the seed that was promised, it was not good (mostly rotten). It had a mix of different varieties too and included Raja, Rado, and Cardinal. Most of it went to waste, for some members 80 percent of it went to waste. We had so many sacks lying by the road that year it was difficult to walk on our road, in the end we just buried them.” (Seed potato FEG) Conclusions: ASF’s support to seed potato FEGs appears to have been influential in promoting new technologies and practices, but the late start makes it difficult to observe results. The FEGs found the training useful and have changed practices as a result, but few have direct experience of quantities produced or the profitability of the intervention. Based on the evidence from those who have harvested and the perceptions of the other groups of the quality of the standing crop, there is good reason to believe that the intervention was effective in increasing the quantity and quality of potatoes the FEGs produced and should increase the value of sales. At the FEG level, the potato value chain offers little potential for creating jobs. The intervention is low-cost and probably profitable, but difficulty obtaining good quality seeds may threaten the sustainability of results in the short term. ASF itself seems to have had difficulty providing good quality certified seed and doing so on time. The project’s investments in storage cellars may ultimately address this constraint, but it will take additional experience to determine whether the cellars are effective in improving access to locally produced seed. Individual Grant Recipients The evaluation team interviewed five individual grant recipients who received grants to construct potato storage cellars. Three of the five grant recipients said the cellars cost too much for them to have taken THE AGRIBUSINESS PROJECT EVALUATION 31 the risk on their own. One said he was not aware of the technology prior to TAP. One grant recipient said he got his grant in 2013, two said they received the grants in 2014, and one received his grant in 2015. Consistent with the intervention design, all five reported receiving potato storage cellars (Table 16 in Annex 9). Three also reported receiving training (in business management and enterprise development) and tools (helmets, gloves, boots, masks, 45 bags in which to store potatoes, and a first aid kit). Only two of the five recipients specifically reported using the cellars instead of the traditional method of storing potatoes in a hole dug into the ground. One also reported storing potatoes in a rented shop prior to building the cellar. Four of the five grant recipients reported producing larger quantities of potatoes as a result of the cellars (Table 22 in Annex 9). In this context, increasing the quantity produced meant reducing wastage/spoilage. One reported spoilage losses of 30 percent in traditional storage, and one reported losses of 50 percent. All four respondents said the cellars had completely eliminated losses due to spoilage. “Before this cellar technology I as facing huge losses (30 percent) of my potato inventory during winter season. Even after completion of cellar, I was not mentally prepared to put all of my produce in it due to an unknown fear of loss. In the last season, I only stored 24 bags (50 kg) in my cellar as a trial. The result of the cellar was excellent as I did not observe any product loss/damage in my stored product during the winter season.” (Potato cellar grant recipient) Three grant recipients reported increasing the value of potato sales. One reported receiving a high price because he was able to sell potatoes from his cellar in February before other farmers could get potatoes from their traditional stores. The other three who reported increasing the value of sales attributed the increase to receiving a good price for higher quality seed potatoes. Respondents did not provide consistent data on the value of sales. “The price of potato seed available in the market was Rs. 2,000 per bag. According to him, the demand of his seed potato was high as compared to the seed available as people were well aware of the quality of his seed. According to him, he had received Rs. 500 higher price per 50 kg of seed potato as compared to available seed.” (Potato cellar grant recipient) Estimates of the value of sales obtained from the interviews do not seem consistent with the values ASF reported in the second quarter of 2015. One cellar owner attributed an increase in sale value to having a larger quantity to sell because of less spoilage. The implied value of what he did not lose was USD 260. Two others who reported selling potatoes said they received a better price because of better quality. Multiplying the difference in price by the quantity sold suggests that they earned USD 224 and USD 448 in additional income. ASF, on the other hand, reported an incremental value of sales of USD 9,093 in the second quarter of 2015, or USD 827 per cellar. The difference between the evidence from interviews and ASF’s performance data may reflect errors in ASF reporting, poor recall in the qualitative data, or inconsistencies due to data from different time periods. Only three of the grant recipients spoke about employment, and they all reported no change in employment due to the cellars. In fact, one said that it requires more labor to store by traditional methods than in the cellars. All five cellar recipients said they expected to continue the practice of using the cellars because they had experienced the benefit, and the infrastructure is durable. However, all five said that, in spite of their neighbors’ interest, financial constraints prevented others from replicating the intervention. THE AGRIBUSINESS PROJECT EVALUATION 32 The seed potato cellar recipients had no major complaints about implementation. In fact, three specifically said that the grant process was easy and quick. Conclusions: The grants to support cellars addressed financial constraints that prevented farmers from constructing cellars and thus promoted adoption of better storage technology. The cellars were appropriate technology and effective in reducing (or eliminating) spoilage during the winter. In addition to having a larger quantity to sell because of reduced spoilage, cellar owners were able to maintain the quality of their stored potatoes and thus receive higher prices in the market. While the intervention promises good returns to the cellar recipient, it holds little potential for broader effects such as creating jobs. Furthermore, the high cost of the cellars may prevent the intervention from reaching scale through organic replication. Meat Value Chain ASF’s competitiveness report for the meat industry concluded that Pakistan has a comparative advantage (large production, little domestic demand for quality meat, and location) in reaching large export markets for high-quality meat.28 The report put global beef exports in 2012 at USD 18.4 trillion annually for frozen meat and USD 20.0 trillion annually for fresh meat. Pakistan’s meat industry, however, serves almost entirely a relatively low-value market in fresh carcasses (which account for 98 percent of Pakistan exports) shipped to GCC countries. Several factors limit Pakistan’s ability to access higher value markets, particularly for boneless cuts. Pakistan’s livestock industry is dominated by small farmers who produce meat as a spin-off product of dairy farming. They do not have good meat breeds, so meat yield per carcass is low, and the meat does not meet the quality standards and expectations of high-end markets. Farmers have limited knowledge of modern farming practices and don’t feed or medically treat cattle in a manner conducive to producing quality meat. Furthermore, Pakistan’s meat processing industry is not familiar with meat cuts and does not possess the skilled labor to produce high-value boneless cuts. To capitalize on the market opportunities, the report concludes that Pakistan must:  Develop commercial feedlots that can produce high-quality and traceable meat animals;  Develop the processing skills to produce high-value boneless meat cuts;  Expand participation (i.e., quantity) and products (i.e., value added) in GCC markets and identify new markets in the region; and  Acquire Halal accreditations to gain access to a large and growing Halal market, much of it within reach of Pakistani exporters.29 After the de-scoping, ASF’s strategy in the meat sector focused only on the supply side, with the possible exception of the few meat processors/exporters who participated in IMAP events (covered 28 Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Bovine Meat - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. 29 The team’s interview with a technical service provider at the University of Veterinary and Animal Sciences in Lahore confirmed these conclusions. THE AGRIBUSINESS PROJECT EVALUATION 33 elsewhere in the report). ASF provided meat cut training to butchers in retail stores and in export￾oriented slaughterhouses. It also trained potential feedlot operators, or those who might influence animal feeding, and artificial insemination technicians (to build indigenous capacity for herd improvement). Meat Cuts and Standardization Training ASF trained 48 butchers in meat cuts and standardization. The evaluation team interviewed six recipients of meat cuts training: managers of four large processing/exporting companies and two small retail butchers (ASF conducted separate trainings for managers and butchers). The University of Veterinary and Animal Sciences (UVAS) in Lahore conducted the trainings, and the large processors/exporters all mentioned receiving on-site training from an Australian expert whom they praised as being especially good. Four of the trainees specifically praised the training for its relevance, utility, and quality of delivery. One manager credited the training with changing workers’ attitudes toward work and the work environment in his plant, reducing losses in chilling/production, and improving hygiene practices in the workplace. The only substantive negative comment on the training (4 of 6 trainees) was that a single training was not sufficient. One company the evaluation team interviewed was selling boned and vacuum packed meat cuts to a firm that held a contract to supply United Nations peacekeeping forces worldwide and also sold cuts to some other buyers. The owner of the firm said that the ASF training, particularly the Australian butcher, contributed to further building the skills his firm needed to access this market. The two retail butchers reported increasing the value of their sales as a result of the training. They reported that they were now able to offer meat cuts to customers and anticipated breaking into new markets such as hotels and elite class buyers. One estimated that his sales had increased by 5 percent. Both also said that what they had learned allowed them to operate a much more hygienic shop and their customers noticed this. Three of the processors/exporters reported that they had not yet put the training into practice because they had not received any orders for meat cuts. One mentioned the risk of entering the market due to export procedures, lack of a cool chain, and freight forwarding problems. None of the trainees reported increasing employment as a result of the training. Conclusions: The training was effective in teaching new skills, but limited penetration into markets that use those skills to advantage has thwarted achieving broader outcomes on sales or employment. Retail butchers are the only trainees reporting results so far. Processors and exporters have not yet developed the export market connections to sell high-value boneless cuts. The training will not produce results until processors/exporters develop the markets that demand such cuts but the training and skills are a necessary precursor to gaining access to those markets. The training is, therefore, relevant but not effective without additional effort to support producers/exporters to penetrate markets. One respondent suggested that that may take work on the enabling environment and other aspects of the meat value chain. Artificial Insemination Training ASF trained 42 individuals in artificial insemination (AI) to improve access to quality AI services among farmers and improve herd quality. The evaluation team interviewed four recipients of AI training. The trainees received an intensive (40-day) practical training at the Agriculture University of Peshawar. A representative of the university explained that they were able to bring cows in daily and provide the trainees with practical experience that they could get nowhere else. One veterinarian trainee THE AGRIBUSINESS PROJECT EVALUATION 34 corroborated this statement from his own experience in school. In addition to the training, all four trainees reported receiving a nitrogen tank for storing semen, an AI gun, gloves, a straw cutter, boots, and an apron. All four trainees said that the training was extensive and practical. They all specifically mentioned the value of the practical part of the training. Two of the four trainees, both practicing Doctors of Veterinary Medicine (DVMs), reported increasing the value of their sales of AI services as a result of the training. The two who did not report increasing the value of sales were both DVMs but unemployed. “Most of the farmers have become aware using AI for increased livestock productivity therefore, they are very conscious about this technique. Generally, they do not consider a DVM graduate as doctor if he was not able to perform AI on animals. This training helped us to gain all the practical knowledge of AI so, farmers always prefer us (AI experts). As the confidence of farmer’s increases, they contact us on preference basis to deal their cases and they also refer us to their relative’s. In this way our sales increase. I am in contact with my fallow trainees and they are also performing AI in their villages. This training provided practical knowledge which is a permanent change. By performing AI on animals the breed of my area will be improved in the long run and this equally important for all other villages in which trained and skilled ASF trainees perform AI.” (AI trainee) Only one of the trainees, a practicing DVM, performing one to two procedures per day, believed the results were sustainable. Conclusions: Based on four interviews, ASF’s design was effective in increasing skills for individuals who would not have developed the skills otherwise. It did not, however, live up to its potential to increase farmers’ access to quality AI services. Two of the four trainees were not in a position to put their training into practice because they were unemployed when ASF selected them for the training and they did not have access to transportation to reach farmers. From an implementation perspective, this points to issues with beneficiary selection criteria. Feedlot Fattening Training ASF trained 50 individuals in feedlot fattening practices. The evaluation team interviewed four of the trainees. Two were DVMs working for government and one was the owner of Stock Feed Company which conducts research on animal fattening and operates commercial fattening operations. Two of the trainees mentioned that the training was useful. While the trainees found the information useful, and one of the government employees said he would communicate what he had learned to farmers, none reported changing their practices or earning additional income because of the training. “It was a very new topic for me as in KP no one has done anything on the subject matter. I believe, if it is taken in its true spirit, it will become a very good business. The feedlot fattening is a unique opportunity to raise animals quickly for better meat production by feeding them both in green and supplementary feed under controlled environment” (Feedlot fattening trainee) Conclusions: Respondents provided little evidence that the feedlot fattening training was effective. While they said the training was useful, only one was in a position to put learning into practice. The government DVMs could pass information about animal nutrition on to their farmer clients but this is a far remove from promoting commercial feedlot fattening and, at this scale, will make no meaningful difference in the quality of animals available in the market. ASF may not have taken appropriate care in selecting trainees who could be influential but it may also be that the sector is not yet ready for this type of intervention. THE AGRIBUSINESS PROJECT EVALUATION 35 IMAP With IMAP, ASF aimed to introduce Pakistani processors and exporters to international markets with the ultimate goal of improving access to markets and increasing export sales. To accomplish this goal, ASF provided technical, logistical, and financial support to enable 76 processors/exporters to attend 10 different international exhibitions and fairs (Table 7). In addition to increasing immediate export sales, ASF anticipated that participants would establish networks; gain exposure to new innovations, ideas, and technologies for processing, marketing, value addition, and promotion; and strengthen trade relations. As of June 2015, ASF reported that IMAP events had stimulated USD 33.31 million in export sales for 23 businesses (Table 7). However, ASF counted any sale attributed to a connection established at an IMAP event as an increase in the value of exports as a result of U.S. Government assistance. Since many IMAP participants were established exporters, however, a sale attributable to an IMAP event may not represent an increase in the value of exports if it supplants a sale the firm would have made anyway. This in no way questions the well-documented sales figures reported by ASF, it merely questions whether exports attributable to an IMAP event represent new business, markets, or buyers attributable to USG interventions. TABLE 7: IMAP EVENTS Event Location Year Number of Participantsa Value of Sales/Exports (million USD) b Euro Tier 2012 Hanover, Germany 2012 4 4.95 World Food Moscow 2012 Moscow, Russia 2012 10 10.63 AGRAME 2013 Dubai, UAE 2013 10 5.60 Fruit Logistica 2013, 2014, 2015 Berlin, Germany 2013 21 6.81 FoodEx Saudi 2014 Jeddah, KSA 2014 8 2.42 Gulf Food Dubai 2014, 2015 Dubai, UAE 2014 19 2.90 World Food Kazakhstan 2014 Almaty, Kazakhstan 2014 1 - Grand Total 76 33.31 Source: The Agribusiness Project: Quarterly Progress Report – XV (April 1, 2015 to June 30, 2015). April 2015. a. Because individuals often attended multiple events, this column does not reflect the number of individuals who participated in IMAP events. The report states that 23 businesses reported receiving contracts through the events but not the number who participated. b. The project’s PMP defines the indicator as the incremental value of exports relative to the previous year. Project reports, however, appear to count all exports that results directly from an order received at a TAP-supported international exhibition or event. TAP selected participants through solicitations and its networks and provided grants that covered registration, transportation, lodging, meals, stalls, and design support. The project briefed participants prior to and after the trips. Many of the 14 participants the evaluation team interviewed were well￾established firms. Thirteen reported that they had attended international exhibitions before TAP, some with support from the Trade and Development Authority (TDAP) or donors and some on their own. The support from TAP made a difference, however. One participant each mentioned the importance of having a stand/stall, the value of the USAID logo, and the value of being well prepared. Participants spoke positively of the value of making connections and networking (5 of 14 participants) and learning (4 THE AGRIBUSINESS PROJECT EVALUATION 36 of 14 participants). Eight of the participants praised ASF for very well organized and successful experiences. “We were introduced to new products, their quality, color and packaging presented by international participants. I also came to know about the international standards and their requirements.” (IMAP participant) “I've been going to Gulf Food before for two years before ASP, but with the agribusiness people it was more informative. Beforehand they had seminars and meetings and sitting with them, one was from the U.S. and one from Australia, was very informative. Before that in this meet industry I was not aware of the circumstances for making good meetings.” (IMAP participant) Nine of the 14 participants the team interviewed mentioned negative experiences with IMAP events. Two said the stall was too small. They referred to the same exhibition where they said ASF had booked too late to get a good space. Six said the event in which they participated was not relevant to their business (e.g., a fruit exporter who visited a livestock-oriented event, a meat exporter who visited a dairy event, and an exporter who could not provide the approvals and no-objection certificates (NOCs) required to sell to European buyers at the event). Seven of the 14 participants said they gained new orders at the exhibitions. The orders ranged in value from USD 150,000 to USD 1.5 million. Gulf Food Dubai proved successful for meat producers. According to the Punjab Agriculture and Meat Company (PAMCO), local exporters received orders of approximately 450 tons of chilled meat as a result of the contacts made at the event. However, four participants specifically stated that they had not made any new sales as a result of IMAP events. “In AGRAME (2013), I met with a Turkish buyer- Haqan Foods (big company) and had a detailed meeting with them for future prospects. Just after this event, I participated in SIAL China (2013) and got chance to meet with Haqan foods again. The continuous participation in international exhibitions helps a lot to create a long lasting and professional image of any company in front of large international buyers. I treat AGRAME as the base of my relation with Haqan Foods. Furthermore, Haqan Foods introduced two buyers from Azerbaijan and Turkmenistan to our company and we were able to materialize both orders of worth USD 861,480 and USD 452,900 From January 2014 to May 2015.” (IMAP participant) Only two participants reported hiring new employees as a result of participating in an IMAP event. One routinely hired labor to can products when she had an order but her business was very small and sporadic. The other reported increasing sales of fruit by 60 percent to 70 percent and hiring extra labor to process and pack the fruit. A number of respondents mentioned external factors that limited results such as the enabling environment (6 of 13 participants), transportation (5 of 13 participants) and market linkages (4 of 13 participants). One mentioned not being able to capitalize on connections he established in Saudi FoodEx because he was not eligible to sell in Saudi markets. Another meat exporter complained that the GoP did not help exporters obtain approvals and NOCs for direct exports. A third participant noted that the lack of rules and regulations regarding exports constrained the firm’s ability to export. Limited air transport capacity also limits participants’ access to some export markets, particularly for perishable products such as fresh fruit, vegetables, and chilled meat. Another mentioned a requirement that he market his meat through a firm with a slaughterhouse, a requirement that wasted the effort he’d put into developing his brand. Conclusions: IMAP events were effective interventions to introduce processors/exporters to export markets and market requirements; gain knowledge about how to access these markets; and, for some, THE AGRIBUSINESS PROJECT EVALUATION 37 gain new buyers. Their added value to the individuals who had the capacity to attend the events without TAP support was the preparation and access to a branded booth or stall. It is difficult to determine whether the events increased sales however. Since most participants were established exporters, a new buyer does not necessarily mean an increase in the value of exports or even enhanced access to export markets. ASF appears to have implemented the logistically difficult activity well with few substantive complaints and many kudos from participants. Miscellaneous Findings and Conclusions This section covers findings that do not fall neatly into value chains or other project components but do contribute to answering the evaluation questions. These include findings on partnerships, challenge grants, sustainability, and gender integration. Partnerships ASF engaged local partners to form and strengthen FEGs. The strategy leveraged the partners’ established connections with communities and their experience forming and supporting community organizations. Local partners focused on particular regions in which they had a presence and, by default, the region specific value chains (Table 8). This section presents profiles of the partners gleaned from interviews, promotional material, and websites. TABLE 8: TAP LOCAL PARTNERS AND REGIONAL VALUE CHAIN CLUSTERS Value Chain Supported Value Chain(s) Region Rural Community Development Society (RCDS) Citrus, meat, HV/OSV Punjab Sindh Agricultural and Forestry Workers Coordinating Organization (SAFWCO) Chili, banana, HV/OSV Sindh Lasoona, Sarhad Rural Support Programme, Hashoo Foundation HV/OSV KP National Rural Support Programme (NRSP) HV/OSV, grape ICT, Potohar Aga Khan Rural Support Programme Apricot GB Aga Khan Rural Support Programme, Lasoona Seed potato GB, KP The evaluation team interviewed five of the partners, AKRSP, NRSP, Hashoo Foundation, and SRSP and reviewed the closeout report from another partner, RCDS. AKRSP: AKRSP has a long history of supporting agricultural development in GB. It signed a three-year agreement with ASF to form and support FEGs which the de-scoping reduced to two years. The final period of performance was from September 2012 through September 2014 with a no-cost extension through November 2014. AKRSP was responsible for forming FEGs; delivering enterprise development training (EDT) to FEGs; conducting rapid market appraisals (RMAs) for seed potato, apricot, and HV/OSV; and developing concept notes on apricot and HV/OSV. THE AGRIBUSINESS PROJECT EVALUATION 38 AKRSP’s project completion document reports that the organization was responsible for forming 250 FEGs in the apricot and seed potato value chains, a target ASF subsequently reduced to 166 due to the de-scoping.30 AKRSP reported forming 80 apricot and 92 seed potato FEGs with 2,336 members (1,536 in potato and 800 in apricot) by November 2014. Managers reported that ASF gave them a target of 30 percent women’s participation in FEGs and that AKRSP met this target by focusing on the processing side of the potato value chain, an activity that engaged women, and by leveraging women-oriented apricot processing FEGs with which they were already working. AKRSP managers believed the potato cellars and apricot drying tunnels were relevant interventions that addressed the needs of the communities and AKRSP observed real benefits due to these interventions, especially the solar drying tunnels for apricot. However, AKRSP managers believed that ASF should have helped FEGs develop linkages to markets, an activity in which AKRSP said ASF did nothing. The only challenge AKRSP managers mentioned about working with ASF was a lengthy procurement process which delayed delivery of infrastructure and tools. The de-scoping compressed the timeframe for achieving results and delayed procurement and delivery of tunnels (apricot) and tools (potato) by a year. It also created some challenges for the organization by limiting the period of time available in which to meet targets. Nevertheless, AKRSP managers believed it was a good move since it focused both partners on achieving targets in a timely manner. NRSP: NRSP claims to be the country’s largest program. It works with 135,000 community organizations representing more than 2 million households in 54 districts across Pakistan. The organization currently manages two USAID grants, the Assessment and Strengthening Program and the Small Grants and Ambassador’s Fund. It signed a three-year agreement with ASF in September 2012 and formed 145 FEGs in grape (25 in Attock and Chakwal), peach, HV/OSV (60), pomegranate, meat, and dairy by March 2014 when ASF terminated the contract early. ASF said that the early termination was a result of the de-scoping that shifted priorities to other value chains and regions. NRSP formed the FEGs and provided management and financial skill training. Others provided technical training in agricultural production and marketing. NRSP managers believed the technical experts ASF engaged to provide this trainer were “appropriate”. NRSP is experienced in agricultural development and has a section, natural resource development, which works in all sectors of agriculture. Organization managers reported that forming FEGs was part of NRSP’s normal business but value chains were new to them. Their concept of a value chain approach was one that not only helped farmers produce more but also linked them to markets. NRSP managers also defined a successful FEG as: “If their livelihoods are improving because of the intervention than we can say that the FEG is successful”, a statement that implies a livelihoods rather than a value chain development perspective. NRSP managers thought the project design (trellises) was appropriate but reported three major issues with ASF’s implementation. They felt that ASF should have given the communities the grant money and “trusted” them to find the best and most efficient way to obtain trellises rather than providing the trellises outright. With ASF’s process, trellises arrived too long after FEGs planted the vines, cost more than the communities believed was necessary, and were of poor quality. When pressed, managers could 30 Aga Khan Rural Support Programme. (2014). Agribusiness Support Fund Completion Report, Year 2013-2014. Aga Khan Rural Support Programme. THE AGRIBUSINESS PROJECT EVALUATION 39 not mention a specific quality issue but ASF itself determined that the I-shaped trellis supports provided to FEGs were not appropriate and thus provided Y-shaped supports to individual grant recipients. Finally, managers believed that ASF focused only on production and did not make the market linkages that could have enhanced benefits to the farmers. “The grapes FEGs had to purchase the seedlings on their own. They planted the seedlings and ASF has to provide them poles and wires two months after plantation so the plant can climb up on those things. And they provided these things too long after the plantation and the plants could not manage it. March plantation and delivery of the trellises was July or August. So there was a five or six month delay and this caused a lot of problems at the community level. We took up this issue with ASF and ask them to let the communities do the procurement by themselves. They can procure more quickly, much cheaper, and better quality.” (NRSP) NRSP managers also said that ASF’s criterion that FEG members own land was inconsistent with its target of 30 percent female members because land a household owns is almost never in the woman’s name. NRSP adapted the criterion to refer to household land ownership and met its targets for female FEG members. NRSP reported that women participated in all stages of grape production except marketing, a statement which directly contradicts information provided by the grape FEGs the evaluation team interviewed. Hashoo Foundation: The Hashoo Foundation focuses on three areas: economic development, human capital development, and social protection. Economic development focuses entirely on agriculture, i.e., honey, horticulture, agribusiness development, and dairy. Foundation managers reported they were already working in seed potato and HV/OSV. The foundation signed a two-year agreement with ASF in December 2012 and submitted concept notes to work in seed potato, HV/OSV, and dairy. ASF approved the foundation to work in the seed potato and HV/OSV value chains in March 2013. The Hashoo Foundation formed the FEGs and provided enterprise development training. ASF arranged for technical experts (extension agents) to provide technical training. The foundation’s close out document reported that it had formed 80 FEGs, 40 each in HV/OSV and seed potato. It formed 15 FEGs in each value chain in its first year and the balance in the second year.31 The foundation’s close out report concluded that the FEGs were effective organizations for collective action and had gained important capacities for production. FEGs are connected with markets and a contract farming arrangement was very successful. However, the report and an interview with foundation managers highlighted some implementation problems as well. In particular:  ASF never progressed, as planned, to form FEGs into clusters and associations which could have enhanced their collective strength in purchasing inputs and marketing.  ASF did not provide certified seed so FEGs produced table potatoes which sell for less than seed potato.  ASF implemented a livelihood approach by supplying only inputs and technical assistance to FEGs. It did not attempt to link FEGs to processors or markets. 31 Hashoo Foundation. (2014). Project Completion Report: The Agribusiness Project (TAP) Formation of Enterprise Groups. Hashoo Foundation. THE AGRIBUSINESS PROJECT EVALUATION 40  Delays in grants caused inputs to arrive too late for the agricultural season and compromised results. The closeout report also identified the lack of good quality seed as a potential barrier to sustainable results. RCDS: RCDS operates in Punjab and, under the de-scoped TAP, supported 60 HV/OSV FEGs. The project’s closeout document reports that it began working with ASF in October 2012 and ended in December 2014 but the report was otherwise poorly written and uninformative.32 RCDS formed the FEGs and trained them in management (legal and financial), leadership, accounting, and marketing. SRSP: SRSP signed a three-year agreement with ASF in October 2013 to form 400 FEGs in six districts of KP in the HV/OSV and strawberry value chains. The organization’s close out report and an interview with senior management found that ASF encouraged the organization to form the FEGs within the first two years and then spend a year consolidating the groups and forming associations. The de-scoping, however, reduced the term of the agreement to two years, limited the value chains to HV/OSV, and reduced targets to 134 FEGs in three districts. Furthermore, SRSP formed FEGs around structure farming, tunnel farming, mushroom farming, and food and fruit processing. ASF ultimately approved only the first two which limited the supported FEGs to 82. However, ASF did not provide the grants for tunnels on time so SRSP had to reorganize the 24 FEGs organized for tunnel farming to structure farming instead. The frequent strategy shifts frustrated and harmed farmers, delayed implementation, and damaged SRSP’s reputation. After forming the FEGs, SRSP trained them in business, FEG management, enterprise development, and the importance of value chain analysis. SRSP then helped the FEGs apply for grants from ASF, many of which had to be resubmitted as requests for structure farming after ASF did not provide tunnel farming grants in time for the season. Even these grants were late and the FEGs did not receive the structures until September 2014. During implementation, ASF took too long to approve grants and provide other support (tools and training) which resulted in much of the support arriving too late in the agricultural season to be useful. The delays created hardships for farmers when they’d set aside land for an intervention that did not materialize. SRSP’s closeout report contained some particularly thoughtful lessons and recommendations:  Timing is important in agriculture. Agricultural development projects need to develop, and strictly adhere to, pre- and post-production activity plans.  Small FEGs (4-7 like-minded and progressive members) work more effectively and seamlessly as an enterprise. These smaller groups should be clustered into associations to leverage collective bargaining power.  To create the capacity to identify issues early and seek timely remedies, at least one member of each FEG should participate in an advanced level/extensive technical training. 32 Rural Community Development Society. (2014). The Agribusiness Project's Completion Report. Rural Community Development Society. THE AGRIBUSINESS PROJECT EVALUATION 41  The office-bearers in each FEG should receive advanced book keeping training in order to improve cost accounting, profit and loss calculations, and record keeping for the entire group.  Facilitate networking with line agencies to help build farmers’ relationships with the available technical resource pool. Conclusions: The partnerships were largely effective in quickly organizing FEGs for TAP implementation. Interviews with the local implementing partners and their closeout reports revealed three common implementation issues. First, ASF put little or no effort into connecting FEGs to markets. This reflects a limited appreciation for the value chain development approach and potentially limits results at the FEG level. Second, ASF was habitually late delivering on its promised support. As a result it often provided support too late for farmers to benefit fully, if at all. Third, poor communication between ASF and partners contributed to misunderstandings and, perhaps, promises to FEGs that did not materialize. Poor communication and frequent shifts in strategy, caused in part by USAID, coupled with ASF’s problems delivering on grants and technical assistance, complicated implementation for the partners and damaged their reputation with their constituent communities. Challenge Grants The original project document recognized that: “Many Apex FEGS, medium and large businesses in the agriculture value chain, and agribusiness‐oriented NGOs have the capacity to productively absorb … multi‐million investments, contribute significantly to the objective of having a transformational impact on job creation, incomes and economic growth.” ASF anticipated using challenge grants to finance activities that would strengthen value chain linkages. The project document provided illustrative examples of using the grants to support 1) medium and large-sized producers, input providers, cold chain, packaging, and other enterprises that provide services on contract to other value chain actors and 2) investors taking advantage of liberalized marketing laws to establish more efficient wholesale marketing facilities. ASF awarded seven relatively large challenge grants to selected businesses with a total value of USD 143,569 (Table 9). TABLE 9: CHALLENGE GRANTS Grant Product District Purpose of Grant TAP Share USD Reefer Khairpur Provide cooling for removing field heat and temporary on-farm storage for fruits and vegetables. $9,072 Swabi $9,072 Screen House Hyderabad Promote availability of disease free fruit plants. $23,216 Lahore $23,216 Malakand $23,216 Fresh Pack Lahore Modified atmosphere packing facility to increase shelf life of fruits and vegetables for domestic and export markets and reduce waste. $45,928 Fruit Processing Solar Unit Malakand No information $9,849 Totals $143,569 Source: M&E data provided by ASF. THE AGRIBUSINESS PROJECT EVALUATION 42 The evaluation team interviewed two challenge grant recipients, Green Revolution and Fresh Pack. Green Revolution: Green Revolution supplies agricultural inputs such as pesticides, fertilizers, seeds, and plant stock (olives, grape, and citrus) to farmers. The company received a cost-share grant for two screenhouses. The screenhouses will help the business produce disease-free certified citrus plant stock. The firm was already providing certified citrus plants on a small scale from its existing (small) screenhouse and saw the challenge grant as a way to expand this aspect of the business. Green Revolution met with ASF, agreed on the structures and location, and signed a grant agreement in 2013. After Green Revolution deposited money with ASF, ASF selected a vendor and paid the vendor to build the screenhouse. However, ASF engineers rejected the screenhouse design after the vendor completed the work. Green Revolution reported that the vendor had built screenhouses for ASF before and these had also been rejected. Green Revolution incurred significant additional costs to bring the screenhouse into compliance with ASF standards. The screenhouses are not yet operational so the company has not yet benefited from their use. The company expects to increase sales of citrus plants and hire additional workers to manage the screenhouses when they are completed. Green Revolution managers believe that if ASF had checked on the vendor periodically, it could have avoided the problem with the unsatisfactory screenhouse. Green Revolution is optimistic about the future of its business but would welcome assistance to share technical knowledge with other (U.S.) companies. The company also saw value in a grant for equipment to treat citrus for export markets. Fresh Pack: Fresh Pack is a fruit and vegetable processor and exporter. The company received a cost￾share grant for modified atmosphere (vacuum) packing (MAP) and related equipment (generator, cold store, reverse osmosis water treatment equipment, and other smaller pieces). Fresh Pack also participated in an IMAP market exposure event and ASF helped link the firm to date growers in Dera Ismail Khan to procure a particular variety of date. The grant recipient was a former marketing director at ASF but left in 2012 to restart his export business. Fresh Pack applied for a challenge grant in 2013 and reported that the grant process was easy. However, it took two years to receive the (imported) MAP equipment, a delay Fresh Pack attributed to an “unorganized” effort by ASF and government red tape required to import machinery. The respondent reported that ASF was supportive and responsive in all other respects. The company has installed the equipment and is testing it before using it commercially. The respondent said he was not in a position to take the financial risk of purchasing the equipment without ASF support. Because the equipment is not yet in commercial operation, it has had no effect on the company’s production or sales. The respondent reported, however, that he is optimistic about export sales because export markets in the UK, U.S., Germany, and GCC countries demand vacuum packed products and prices are high. The respondent expects to increase his labor force proportionally to sales and said he prefers to hire women for processing, grading, cleaning, and packing because they are more productive than men. Fresh Pack expects to continue using the equipment because of the substantial investment and anticipated demand for the product. However, he said that there are no technical experts in MAP technology in Pakistan to help him implement the new practices. He believes that other investors are waiting to see how he fares before making similar investments. THE AGRIBUSINESS PROJECT EVALUATION 43 Conclusions: The relatively small investments in the two companies the evaluation team interviewed hold some promise for strengthening their respective value chains if they are connected to other value chain actors. Green Revolution supplies citrus trees and will be motivated to find customers. Fresh Pack may or may not increase sales sufficiently to expand the market for its inputs and thus producers. ASF proved incapable of effectively managing procurement processes which delayed results for both grantees. Sustainability Previous sections presented data on the likely sustainability of results in the four value chains. This section presents other evidence related to sustainability that is not value chain specific. The original project document mentioned sustainability in the context of ASF’s organizational sustainability and seemed to take the sustainability of beneficiaries’ results for granted: “The measure of sustainability achieved under the project will be the emergence of viable value chains in the horticulture and livestock markets. These value chains will be self‐sustaining because they are profit‐driven and not donor driven.” The revised project document (after the de-scoping) gave greater lip service to the sustainability of results but did not describe a strategy to achieve sustainability. It still seemed to take the sustainability of results for granted as a natural outcome of improved competitiveness: “Competitiveness of the chain as a whole will ensure sustainability of the market share and growth thus creating more jobs and income generation opportunities.” If ASF produced a separate sustainability plan or strategy, it did not share it with the evaluation team in spite of several requests. The revised project document seemed to adopt more realistic and scaled back expectations for the results that small farmers could achieve: “In this revised proposal we have combined a strengthened FEG mechanism with expanded options for sustainable and profitable small farmer value chain and market participation. Specifically, we propose to enlist private sector stakeholders within each value chain as primary entry and contact points for small producer beneficiaries. This approach will replace, in large part, the project’s prior dependence on an illusory scheme of converting small producers to impresarios and their associative organizations to vertically integrated enterprises.” To the extent that these revised expectations are more achievable, they may also be more sustainable. This does not, however, constitute a focus on sustainability or a sustainability strategy. ASF staff members described replication as an important part of the project’s strategy to achieve the scale necessary to transform value chains and achieve truly sustainable results. However, neither the original or revised project documents, which are attached to the agreement and specify the technical approach, nor any of the project’s annual reports, mention replication in this context. Gender Integration Much of Pakistan’s agricultural sector is male-dominated. ASF, in its original project description, attributes this to “social barriers” and a “result of low awareness of potential opportunities for women”. ASF sought to address the paucity of women’s involvement by actively seeking their participation in value chain activities and in their own hiring practices. ASF’s SOW proposed that at least 30 percent of the total project benefits accrue to women and also specified that it would hire a gender manager. ASF met its targets for involving women largely through FEGs. In fact, women’s involvement was one of the seven criteria ASF used to select the original 17 value chains. The de-scoping dropped most activities which specifically engaged women such as goats and kitchen gardening. According to one USAID/Pakistan official, the Mission encouraged ASF to drop these value chains because they were not successful from either a livelihood or a value chain perspective; the fact that they also eliminated many THE AGRIBUSINESS PROJECT EVALUATION 44 women-focused activities was an unanticipated consequence. Of the eight value chains selected as part of the redesign, women played a dominant role only in the apricot value chain (Table 10). TABLE 10: WOMEN’S PARTICIPATION IN FEGS Value Chain Number of FEGS Number of Male Members Number of Female Members Percentage Female Apricot 80 201 599 75% HV/OSV 335 3,693 1,227 25% Potato 173 2,388 373 14% Grapes 17 233 10 4% Total 605 6,530 2,209 25% The project document stated that ASF would develop a gender integration plan in the second year of the project. However, ASF did not provide the document to the evaluation team in spite of several requests. The plan, according to ASF’s third annual report, “included steps that will be required to incorporate gender during design, planning, implementation and monitoring and evaluation stages of the project interventions.” The only gender integration activities mentioned in progress reports involved data collection tools and training of ASF staff in December 2013. Very few women are engaged in exporting. According to the staff person in charge of IMAP, women were encouraged to participate, however, “only a handful of participants (they) worked with were women.” Similarly, ASF advertised grant opportunities and included sex as a selection criterion but few women applied. USAID/Pakistan personnel said they were aware of the difficulties ASF faced engaging women and said that ASF “kept their eyes open” for opportunities since they knew they would be under pressure from USAID/Pakistan. Conclusions: The project explicitly considered women in design only by setting targets for the percentage of female members of project-supported FEGs. There is no evidence that it deliberately selected value chains for their potential to benefit women or made any other design decisions to engage women. Not surprisingly, TAP did not directly benefit many women in terms of increasing their meaningful participation in supported value chains. The culture provides few, if any, employment opportunities for women in the potato, meat, and grape value chains. Women may work in these value chains but not as paid labor. Development projects cannot create a role for women and, to have a meaningful impact, must deliberately design interventions in value chains in which women play an active role. The established role of women in apricot processing contributed to the project’s potential success in benefiting women in this value chain. In the HV/OSV value chain, women stand to benefit from increased employment in harvesting. THE AGRIBUSINESS PROJECT EVALUATION 45 CONCLUSIONS BY EVALUATION QUESTION The conclusions in the findings and conclusions sections for each value chain and activity document value chain or activity-specific conclusions. The overarching conclusions in this section address issues of design, implementation, and sustainability more broadly. Design  The project’s planned interventions addressed real barriers to more profitable participation in agricultural value chains for many beneficiaries. Awareness of new practices and a comfort level sufficient to make the investment was a greater concern than financial constraints for both FEGs and larger farmers who received individual grants. The project’s grant support to influential larger farmers who could afford the investments without assistance can be justified only in the context of reducing the risk of an unfamiliar investment and thus developing demonstrations of project-supported interventions and the interviews provided persuasive evidence that the grant￾funded investments were effective demonstrations. Several individual grant recipients, particularly in the grape value chain, made their investment decisions based in part on the experiences of other farmers. At the FEG level, neighboring farmers were replicating low cost technologies and trying to learn the practices promoted by TAP.  ASF introduced appropriate technologies and practices in each of the four agricultural value chains (apricot, seed potato, grape, and HV/OSV). The interventions changed production practices and, when implemented in time, increased quantities produced and incomes. Interventions that had not yet matured showed good prospects for increased production and sales. Prospects for job creation varied by value chain and value chain actor. FEGs and individual grant recipients in the potato value chain produced few, if any, additional jobs while interventions in HV/OSV created a substantial number of jobs and grape a more modest number. HV/OSV was particularly effective at increasing employment because it is labor intensive but profitable enough to justify the additional labor. It is also an activity that often falls into the realm of women’s work. On the other hand, interventions in the potato value chain did not produce many jobs because quantities produced did not increase dramatically and producers have no culture of hiring labor.  Relatively low-cost investments at the FEG level in HV/OSV (structure farming) have the potential to yield substantial returns in terms of increased production, sales, and employment. Furthermore, the investment is relatively small, within the reach of many FEGs (as demonstrated by the fact that some are replacing rotted bamboos and other FEGs are replicating the structures), and very profitable. Additional technical assistance will probably be necessary to expand the intervention as will ready access to high yielding seed varieties.  TAP was not particularly successful at increasing access to export markets. Its flagship export￾oriented initiative, IMAP, was valuable in exposing participants to global standards, market demands, and new ideas. A relatively large number of participants reported making connections at the exhibitions that led to a sale. However, most IMAP participants were established exporters and many had been to exhibitions on their own prior to TAP. While they made a connection at an exhibition that resulted in a sale, current reporting does not reveal whether that sale represented increased business or whether it merely supplanted another sale. ASF also did not put much effort into facilitating international certifications necessary to access many export markets. Few IMAP participants, other than a few meat exporters, however, mentioned THE AGRIBUSINESS PROJECT EVALUATION 46 this as a constraint – but they were also selling largely into markets (e.g., GCC countries) with fewer requirements.  TAP worked across several levels of the selected value chains. It supported small producers (FEGs), larger producers (individual grants), and processors/exporters. However, ASF did not implement a true value chain development approach because it failed to effectively connect the actors at different levels of the value chain or to address value chain deficiencies its assessment reports identified. For example, FEGs in all supported value chains, and the local partners that formed the FEGs, reported that ASF made no attempt to link them to markets. Consequently, most were selling their produce in the mandis which effectively separated them from high-end buyers. The few attempts to connect FEGs directly to supermarkets failed because they overreached the capacities of the FEGs to supply these markets. ASF’s revised project document may actually have represented a realistic step back from a value chain development approach with FEGs when it proposed to “enlist private sector stakeholders within each value chain as primary entry and contact points for small producer beneficiaries” to replace “the project’s prior dependence on an illusory scheme of converting small producers to impresarios and their associative organizations to vertically integrated enterprises”. The evaluation team saw little evidence of private sector engagement with FEGs in this role. The meat value chain provides another example. Interventions in the value chain were largely disconnected and premature in the sense that they did not address some of the broader structural issues in the industry that influenced results (e.g., barriers erected by an unfavorable enabling environment, Halal accreditation, effectively connecting processors to buyers for value￾added meat products). ASF also seemed to have difficulty identifying appropriate beneficiaries for feedlot fattening and AIT training.  ASF paid little more than lip service to productively engaging women and empowering them in selected value chains. It is admittedly difficult to engage women in agriculture in Pakistan and ASF attempted this before the de-scoping with women-focused interventions such as goats and kitchen gardens. USAID/Pakistan was instrumental in removing these value chains during the de￾scoping because officials saw no benefit from a livelihoods or an economic growth perspective. While it was not the intent, the de-scoping removed many of the women-focused activities leaving only the apricot value chain that was dominated by women and directly benefited women. While USAID/Pakistan is partly responsible for weakening the project’s engagement with women, ASF did not appear to take engaging women seriously. It had no gender strategy document and the project document had no meaningful targets or specific strategies for benefiting women.  ASF targeted domestic and international markets appropriately. It did not try to connect FEGs, or many individual grant recipients, directly to international markets because few had the capacity to navigate complex export requirements and logistics. It focused its export-oriented interventions largely on established exporters and tried to connect them to new markets. To the extent that IMAP participants were also grant recipients or beneficiaries of other interventions, ASF started to develop the value chain to the extent that it simultaneously linked beneficiaries to markets and built their capacities to meet market demands. These examples seem to be the exception rather than the rule however. Implementation  The greatest challenge to achieving results was the late start of farm-level implementation. In all supported value chains, beneficiary farmers received the necessary support to begin production THE AGRIBUSINESS PROJECT EVALUATION 47 only since 2013 and many are producing and marketing for the first time in the 2015 season. Their limited experience with the interventions made it difficult to assess effectiveness in terms of long-term outcomes and calls into question the sustainability of the results since ASF won’t be around to provide additional technical assistance should it be necessary as farmers learn how to apply new practices. While ASF management of the project was largely responsible for the slow start, USAID is also culpable for awarding a very large agreement to a high-risk partner that had never implemented such a large project and had no experience with USAID processes and then dramatically changing strategy midcourse.  ASF struggled to provide some interventions on time, a serious misstep in an agricultural development project where seasons dictate strict adherence to timing. Local partners attributed the delays to slow grant and procurement processes. Whatever the cause, however, it points to poor management of a critical aspect of implementation that had serious implications for beneficiary farmers. While beneficiaries in all value chains complained of poor timing, the issue of sequencing came up much less often, (i.e., with two potato FEGs, five HV/OSV FEGs, and one grape FEG). Sequencing issues involved training coming after seeds (5 of 6 FEGs) and the structure coming after the seeds (1 of 6 FEGs).  ASFs management deficiencies harmed beneficiaries and the reputations of local partners. Farmers made decisions with financial and livelihood consequences based on expectations of timely support. When that support did not materialize on time or was of poor quality, it had financial repercussions for households with limited financial resilience to absorb the loss. Poor communication with local partners or a failure to honor commitments also damaged the reputation of some partners when they could not deliver on communities’ expectations.  ASF’s strategy of engaging local partners to form FEGs and build their capacities effectively leveraged partners’ expertise and presence in project-supported communities and probably contributed to forming FEGs much more quickly than ASF could have on its own. However, poor communication between ASF and partners led to misunderstandings that slowed implementation, formed incorrect expectations among FEGs, and damaged partners’ credibility with their constituent communities. Cumbersome procurement and payment procedures coupled with partners’ lack of familiarity with USAID requirements also slowed implementation and frustrated partners. Sustainability  Because interventions began very late and most are not fully mature, it is difficult to assess prospects for sustainability. Low-cost, relatively simple, and profitable interventions (e.g., structure farming for HV/OSV) are more likely to be sustainable than high-cost, technical, or low margin interventions (e.g., solar drying tunnels for apricot). In all cases, however, sustainability will depend on continuing technical assistance and the availability of key inputs (e.g., hybrid seeds). The technical assistance is not readily available in either the private or public sectors in Pakistan and vegetable and potato producers, in particular, reported that it was difficult to obtain high quality certified seeds. Continued donor support for technical assistance will likely be necessary to improve chances for sustainability, especially with FEGs. Donor support could also play an important role in improving access to productivity-enhancing inputs. THE AGRIBUSINESS PROJECT EVALUATION 48 RECOMMENDATIONS  ASF struggled initially to implement TAP and got off to a very slow start. It was not familiar with USAID’s requirements and did not fully grasp the value chain development approach. Nevertheless, ASF built its own capacity during the process and is now a more capable partner for future donor-funded projects. If and when USAID/Pakistan engages local partners, it should do so slowly with relatively modest projects where the negative consequences associated with up-front learning are small. The agency might also consider partnering the organization with an experienced international partner, at least initially, as a mentor as the organization builds its capacity. The partnership agreement will have to give the international partner the role or authority necessary to guide initial implementation appropriately.  When implementing through local organizations, USAID should ensure that the organization has the capacity to meet USAID’s reporting requirements and processes. Developing these capacities after the award may delay implementation and limit results. If a pre-award assessment concludes that an organization does not have these capacities, USAID/Pakistan should stipulate that the organization engage the expertise necessary to develop the processes and materials (e.g., accounting systems, grants manuals) quickly before or immediately at the start of implementation.  USAID/Pakistan should adapt the value of exports indicator to reflect a more meaningful measure of enhanced access to export markets. As ASF, and other implementing partners, now report on the indicator, it includes any sale participants make as a result of participating in a project￾supported marketing event. This is not a meaningful measure of competitiveness for established exporters. More meaningful measures might include trends in overall export sales, sales to new buyers or markets, or offering new products to export markets. Any of these would be more direct and feasible measures of increased access to export markets that are not as misleading as the current value of exports indicator.  If USAID/Pakistan intends to continue supporting agricultural value chain development, it should consider continuing support to at least some of the more promising TAP-supported value chains and beneficiaries. Because of the late start under TAP, many beneficiaries, particularly FEGs, will require additional technical assistance to cement the lessons they’ve learned and address new challenges that arise as they gain experience with new technologies and practices. For value chains and interventions deemed particularly promising, additional donor support may be required to accelerate replication and achieve the scale necessary to transform the value chain and produce truly sustainable results. The most promising interventions at this point seem to be HV/OSV and grape (FEGs and larger farmers), apricot orchards, and potato cellars.  If USAID/Pakistan wishes to continue supporting the meat sector with a focus on developing export markets, it needs to take a more holistic approach. ASF’s approach of working on several discrete and disconnected segments of the value chain achieved little because they were not mutually reinforcing and were not demand driven. A development process that starts by building the capacities of processors to meet a small demand for quality value-added meat products and linking these processors to export markets will create a demand for better quality meat animals that donors can then support by investing in building feedlot capacity and improving herd quality.  To improve prospects for sustainability, USAID/Pakistan should require implementing partners to develop sustainability plans during the design phase, review the plans periodically to determine whether sustainability hypotheses are valid, and adapt implementation as necessary if the hypotheses do not hold true. Sustainability plans should be specific to each activity, i.e., in the TAP example, value chains. THE AGRIBUSINESS PROJECT EVALUATION 49  When building and/or strengthening institutions is an important component of a sustainability strategy, it needs to start early in the project so the institutions are engaged. ASF did not establish the institutional platforms to support the value chains early or help them become effective advocates for the value chains. In similar future situations, USAID should ensure that building and strengthening such institutions occurs early in the project work plan and follow up to make sure it happens. THE AGRIBUSINESS PROJECT EVALUATION 50 ANNEXES THE AGRIBUSINESS PROJECT EVALUATION 51 Annex I: Evaluation Statement of Work THE AGRIBUSINESS PROJECT EVALUATION 52 The Agribusiness Project EVALUATION STATEMENT OF WORK The Agribusiness Project (TAP) - Performance Evaluation Statement of Work THE AGRIBUSINESS PROJECT EVALUATION 53 THE AGRIBUSINESS PROJECT EVALUATION 54 THE AGRIBUSINESS PROJECT EVALUATION 55 THE AGRIBUSINESS PROJECT EVALUATION 56 THE AGRIBUSINESS PROJECT EVALUATION 57 THE AGRIBUSINESS PROJECT EVALUATION 58 THE AGRIBUSINESS PROJECT EVALUATION 59 THE AGRIBUSINESS PROJECT EVALUATION 60 THE AGRIBUSINESS PROJECT EVALUATION 61 THE AGRIBUSINESS PROJECT EVALUATION 62 THE AGRIBUSINESS PROJECT EVALUATION 63 THE AGRIBUSINESS PROJECT EVALUATION 64 THE AGRIBUSINESS PROJECT EVALUATION 65 THE AGRIBUSINESS PROJECT EVALUATION 66 THE AGRIBUSINESS PROJECT EVALUATION 67 THE AGRIBUSINESS PROJECT EVALUATION 68 THE AGRIBUSINESS PROJECT EVALUATION 69 THE AGRIBUSINESS PROJECT EVALUATION 70 THE AGRIBUSINESS PROJECT EVALUATION 71 THE AGRIBUSINESS PROJECT EVALUATION 72 THE AGRIBUSINESS PROJECT EVALUATION 73 THE AGRIBUSINESS PROJECT EVALUATION 74 THE AGRIBUSINESS PROJECT EVALUATION 75 Annex 2: Assignment Work Plan THE AGRIBUSINESS PROJECT EVALUATION 76 THE AGRIBUSINESS PROJECT EVALUATION 77 THE AGRIBUSINESS PROJECT EVALUATION 78 THE AGRIBUSINESS PROJECT EVALUATION 79 THE AGRIBUSINESS PROJECT EVALUATION 80 6. COST ESTIMATE The estimated cost is broken down by four line items: Direct Labor Other Direct Costs Subcontractors MSI Core Support* *MSI Core Support includes pro1ect core costs that cannot be allocated directly to the assignment. This percentage is calculated by dividing the total approved core costs in the contract by the total contract value. 7. PERFORM COR APPROVAL [COR will indicate approval by signing below or indicating "approval" by return email]. Contracting Officer's Representative (COR) Date Sadia Naseer Khan, or designate PERFORM ASSIGNMENT WORK PLAN 4 THE AGRIBUSINESS PROJECT EVALUATION 81 Annex 3: Locations of Value Chain Clusters TABLE 11: TAP VALUE CHAIN CLUSTERS Value Chain Geographic Clusters Northern Region Central Region Southern Region Apricot Gilgit/Baltistan Banana Matiari, Tando Allahyar, Hyderabad Chilies Umerkot (Kunri Tehsil) Citrus Sargodha and Mandibahaudin High value/off season vegetables KP/FATA ICT/Potohar/Punjab (High value green house) Sindh Livestock and meat KP/FATA Lahore, Sahiwal, Okara, Multan, Muzaffargarh Karachi Seed potato Gilgit/Baltistan, Chitral, Mansehra Grapes ICT, Rawalpindi, Chakwal, Attock Source: TAP Revised Project Document THE AGRIBUSINESS PROJECT EVALUATION 82 Annex 4: Local Partners TABLE 12: TAP LOCAL PARTNERS Value Chain Supported Value Chain(s) Region Rural Community Development (RCDS) Citrus , Meat & HV/OSV Punjab SAFWCO Chili, banana, HV/OSV Sindh Lasoona, SRSP, Hashoo Foundation HV/OSV KP NRSP HV/OSV, grape ICT, Potohar AKRSP Apricot GB AKRSP, Lasoona Seed potato GB, KP THE AGRIBUSINESS PROJECT EVALUATION 83 Annex 5: Documents Reviewed Aga Khan Rural Support Programme. (2014). Agribusiness Support Fund Completion Report, Year 2013- 2014. Aga Khan Rural Support Programme. Agribusiness Support Fund. (2012). The Agribusiness Project Baseline Study Report - 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2012). The Agribusiness Project First Annual Progress Report, November 10, 2011 - November 9, 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2012). USAID's Agribusiness Proejct Quarterly Report, January 1, 2012 - March 31, 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2012). USAID's Agribusiness Project Quarterly Progress Report - IV, July 2012 - September 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2012). USAID's Agribusiness Project Quarterly Progress Report - IV, July 2012 - September 2012 . Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2012). USAID's Agribusiness Project Quarterly Report, April 1, 2012 - June 30, 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2013). The Agribusiness Project Annual Progress Report - II, October 1, 2012 - September 30, 2013. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2013). The Agribusiness Project Quarterly Progress Report - V, October 2012 - December 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2013). The Agribusiness Project Quarterly Progress Report - V, October 2012 - December 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2013). The Agribusiness Project Quarterly Progress Report - VI, January 2013 - March 2013. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2013). The Agribusiness Project Quarterly Progress Report - VII, April 2013 - June 2103. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2013). USAID's Agribusiness Project Quarterly Report, November 10, 2012 - December 31, 2012. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). Agribusiness Project Performance Management Plan Update. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Bovine Meat - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Gilgit-Baltistan Apricot - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) HVOSV - Value Chain Competitivess Assessment for Bitter Gourd, Capsicum, Cucumber, Tomatoes, Squash. Lahore: Agribusiness Support Fund. THE AGRIBUSINESS PROJECT EVALUATION 84 Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Potato - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project (TAP) Tomato - Value Chain Competitiveness Assessment. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project Annual Progress Report - III, October 1, 2013 - September 30, 2014. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project Quarterly Progress Report - X, January 1, 2014 - March 31, 2014. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project Quarterly Progress Report - XI, April 1, 2014 - June 30, 2014. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2014). The Agribusiness Project Quarterly Progress Report, October 1, 2013 - December 31, 2013. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2015). The Agribusiness Project Quarterly Progress Report - 13, October 1, 2014 - December 31, 2014. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2015). The Agribusiness Project Quarterly Progress Report - XIV, January 1, 2015 - March 31, 2015. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (2015). The Agribusiness Project Quarterly Progress Report - XV, April 1, 2015 - June 30, 2015. Lahore: Agribusiness Support Fund. Agribusiness Support Fund. (no date). The Agribusiness Project Revised Project Document. Lahore: Agribusiness Support Fund. Asian Development Bank. (2012). Completion Report: Agribusiness Deveelopment Project. Asian Development Bank. Asian Development Bank. (2014). Validation Report: Agribusiness Development Project. Islamabad: Asian Development Bank. Burki, A. A. (2010). Program Evaluation of Agribusiness Support Fund: Estimating the Effects of Treatment on Farmer Groups, Agribusiness and BDS Market in Pakistan. Lahore: Lahore University of Management Sciences. Hashoo Foundation. (2014). Project Completion Report: The Agribusinss Project (TAP) Formation of Enterprise Groups. Hashoo Foundation. National Rural Support Programme. (n.d.). Completion Report: USAID Agribusiness Support Project, January 2013-March 2014. National Rural Support Programme. Office of the Inspector General. (2013). Audit of USAID/Pakistan's Agribusiness Project, Audit Report no. G￾391-13-004-P. Islamabad: U.S. Agency for International Development, Office of the Inspector General. Rural Community Development Society. (2014). The Agribusiness Project's Completion Report. Rural Community Development Society. THE AGRIBUSINESS PROJECT EVALUATION 85 Sarhad Rural Support Programme. (2014). Farmer Enterprise Groups Formation, Agribusiness Support Fund, Project Completion Report. Sarhad Rural Support Programme. USAID/Pakistan. (2011). Cooperative Agreement No. AID-391-A-12-00001. Islamabad: USAID/Pakistan. USAID/Pakistan. (2013). Modification of Assistance No. 6. Islamabad: USAID/Pakistan. USAID/Pakistan. (2014). Modificaiton of Assistance No. 7. Islamabad: USAID/Pakistan. THE AGRIBUSINESS PROJECT EVALUATION 86 Annex 6: Data Collection Instruments The Agriculture Project Evaluation Interview Guide - Agribusinesses/Grant Recipients Learn how the respondent knows TAP (by what name) and use that name throughout the interview. 1. Can you briefly describe your business? a. What do you do? b. How long have you been in this business? 2. Do you own this business? 3. When did you begin receiving support from TAP? 4. What kinds of support have you or your company received from TAP. (Probe for training, certifications, tools/equipment, exposure visits, marketing exhibitions, grants, other) a. Grants: If the respondent received a grant: ask about the grant, e.g., purpose, application process, ease of application, how it was administered, timeliness of receipt, other b. Could you have made these investments in your business without TAP assistance? If yes, how? If not, why not? 5. Have you changed the way you do your business as a result of the assistance? How? If not, why not? (Probe for relevance of each type of assistance) 6. Did these changes affect the value of your sales? If yes how? (Probe for access to different markets, higher prices, larger quantities sold, standardization, certifications) 7. If relevant: Did these changes affect the value of your exports? If yes how? (Probe for access to different markets, higher prices, larger quantities sold, standardization, certifications) a. If relevant: How important were certifications to accessing international markets? 8. Do you employ only men, only women, or both men and women? a. Did these changes affect the number of workers you employ? If yes how? (Probe for producing/processing/marketing large quantities, more/less efficient practices, more/less labor intensive practices) i. If employment increased: What kinds of jobs were created (skilled/unskilled, seasonal/full-time, men/women) ii. Phrase as relevant: What roles do men and women have in your business? iii. Ask only if employment increased: Do you know what the new workers you hired were doing before you hired them? Probe to understand whether the jobs were new jobs (hire people who were not working) or better jobs (hired people who were working for less) 9. As relevant: You’ve told us that sales/exports/jobs have changed as a result of assistance from TAP. Do you think these are permanent changes or are they only temporary? Explain. (Probe for reasons results might be temporary) THE AGRIBUSINESS PROJECT EVALUATION 87 10. Since you started receiving support from TAP, are there other things internal or external to your organization that affected the value of your sales or exports or the number of people you employ? Please explain? 11. Are there other changes that could take place in your industry that would improve your business? Explain. (Probe for constraints in other parts of the value chain. Probe specifically for availability of BDS, financing, TA/human resource capacity) 12. Other than things we’ve already discussed, do you have suggestions for how the assistance could have been improved? (Probe as appropriate for quality of assistance/TA, timing, sequencing, relevance, completeness, grants if relevant) a. If not already mentioned and as appropriate, probe for plant varieties; crop production and management practices and technologies; value addition/processing; certifications. 13. Do you know of examples of other businesses that have started copying the practices or technologies you are using because of TAP. Explain. Probe to understand whether the replication was really a result of a demonstration effect of TAP. THE AGRIBUSINESS PROJECT EVALUATION 88 The Agriculture Project Evaluation Interview Guide – ASF Management 1. How did the project change from the original design to the de-scoping in 2013, in addition to the reduction in value chains? (Probe for approach, partnerships, grants, management) 2. In your opinion, what were the main reasons for the de-scoping? 3. To what extent were others outside ASF involved in the project re-design? (Probe for USAID, international partners, stakeholders) 4. How were the final eight value chains selected? (Probe for potential for results, geographic location, USAID priorities) 5. What were the strengths of the re-design/original design? 6. What were the weaknesses of the re-design/original design? 7. What was the role of CNFA and JE Austin? How well did the partnership work? a. What worked well b. What didn’t work 8. How did you select local partners (IPs)? a. Did they use different approaches in the field? Were some approaches more successful than others? b. Were some IPs stronger than others? If so, who? What made them stronger? 9. How did you go about selecting experts/institutions to provide TA and training? a. International and local expertise b. BDS providers 10. To what extent were the planned interventions sufficient for achieving desired results? 11. What other interventions should the project have implemented that wasn’t part of the original design to address weaknesses or gaps in the various value chains? 12. To what extent did the project incorporate sustainability into the re-design/original design of TAP? (Probe for whether it was an explicit objective or stated in a concept note or strategy) 13. Was there a specific objective of including women in the design of TAP? If so, what was the approach? Did it work? 14. What do you think is the project’s biggest success? 15. What do you think could have been done better? 16. Did ASF and its partners implement the project as designed? If not, how and why did you deviate from the design or planned implementation approaches? 17. Were there any challenges to managing a project of this size? How did ASF address these challenges? 18. Were there any challenges to implementation? Probe for: a. Changes in forming and working with FEGs THE AGRIBUSINESS PROJECT EVALUATION 89 b. Managing grants c. Managing partnerships d. Value chain platforms e. Linking value chain actors f. Certifications g. Managing vendors for infrastructure h. Technical assistance providers i. IMAP events) 19. Please describe how the M&E plan was developed following the redesign. How were targets set? Did they change? If so, how and why did they change? 20. Can you describe the role of BDSPs. How did you work with them and what do they do? 21. What constitutes a successful FEG? 22. What will make the value chain platforms/business associations created by TAP successful? What was there role in the project and how do you envision their role after TAP ends? 23. What are the prospects for sustainable results? Is there anything else the project or future projects should do to ensure sustainability? 24. Do you have an exit strategy? What is it? THE AGRIBUSINESS PROJECT EVALUATION 90 The Agriculture Project Evaluation Interview Guide – ASF Grapes Value Chain Lead Background 1. ASF added the grape value chain back in fairly late (October 2014). Can you explain why it was added back in and why ASF believed it was important? 2. What is the market for grapes? Demand? 3. Did ASF work in the grape sector under the ADB project? FEGs 4. What did ASF hope to accomplish with grape FEGs? 5. Did you accomplish these objectives? If not, why not? 6. How important was a demonstration effect to accomplishing results and did it materialize? 7. Why did ASF terminate support to grape FEGs in 2014 (NRSP)? 8. All of the grape FEGs we spoke with were selling in local mandis. Are there any examples where ASF has been able to link grape FEGs to other markets? If not, why not? a. Do FEGs have trouble reaching other markets because of inherent capacity limitations, insufficient time with ASF, implementation issues? 9. Did ASF experience any challenges working with FEGs in the grape sector? Explain? 10. What local partners did you work with to support grape FEGs? 11. What was your experience with local partners for grape FEGs? a. What were they responsible for (forming FEGs, training (what kind), marketing, …) b. Were they qualified to do what you expected of them? Explain. Grant Recipients 12. How were grant recipients in grapes selected? 13. What did ASF hope to accomplish with grape grant recipients? What were your objectives? 14. How important was the demonstration effect and has it materialized? 15. Most of the grant recipients we spoke with were selling in the local mandis. Are there any examples where ASF has been able to link grape FEGs to other markets? If not, why not? General 16. What were TAP’s most significant accomplishments in the grape sector? 17. What were the greatest challenges you faced and how did you overcome them? THE AGRIBUSINESS PROJECT EVALUATION 91 The Agriculture Project Evaluation Interview Guide - BDSPs Describe the project and learn how respondents refer to it (e.g., TAP, ASF) and use the name they’re familiar with throughout the interview. 1. Can you describe your business? What services do you provide, what kind of businesses do you work with? 2. What kinds of support have you received from TAP. (Probe for training, tools/equipment, other) 3. Has your business changed as a result of the assistance? How? If not, why not? a. Probe for ability to offer different services, quality of services, new clients, other b. Probe for relevance of each type of assistance 4. Did these changes affect the amount of money you earn from this business? If yes how? (Probe for access to different clients or services, higher fees, other) 5. Did these changes affect the number of workers you employ? If yes how? a. If employment increased: What kinds of jobs were created (skilled/unskilled, seasonal/full-time, men/women) b. Phrase as relevant: What roles do men and women have in your business? c. Ask only if employment increased: Do you know what the new workers you hired were doing before you hired them? Probe to understand whether the jobs were new jobs (hire people who were not working) or better jobs (hired people who were working for less) 6. As relevant: You’ve told us that sales/jobs have changed as a result of assistance from TAP. Do you think these are permanent changes or are they only temporary? Explain. (Probe for reasons results might be temporary) 7. Since you received training from TAP, are there other things internal or external to your organization that affected the value of your sales or exports or the number of people you employ? Please explain? 8. Are there other changes that could take place in your industry that would improve your business? Explain. 9. Other than things we’ve already discussed, do you have suggestions for how the assistance could have been improved? (Probe as appropriate for quality of assistance/TA, timing, sequencing, relevance, completeness, grants if relevant) THE AGRIBUSINESS PROJECT EVALUATION 92 TAP Evaluation Focus Group Discussion Guide –Farmers’ Enterprise Groups Introduction and Purpose of Discussion Assalam-o-alaikum, thank you for coming here today. We are grateful that you are giving us your time. Before we begin, we will introduce ourselves. [The moderator and the note-taker should introduce themselves] We are here to discuss with you about The Agriculture Project which is being implemented by the Agriculture Support Fund (ASF) and [name of local implementing partner] and how this project may have helped you with your [name of supported crop] growing, harvesting and selling. Through this discussion, we will ensure that your ideas and opinions are heard. We would like to record these discussions so that we remember and do not miss any of the ideas that you give us. The details of these discussions will not be shared with anyone and your names will be kept confidential - so please do not worry and feel free to express your ideas. Your participation in this discussion is voluntary and we really hope you are willing to participate actively and with enthusiasm. We would like to ask you about your experience with date growing and harvesting and selling and how the different training and equipment you have received has helped you improve your production, harvesting and farm sales. Will you allow us to record this discussion? Thank you THE AGRIBUSINESS PROJECT EVALUATION 93 Introductory Questions 1. Can you tell me something about your farming activities. How large is your farm and what do you grow? 2. When did this FEG begin receiving support from TAP? We want to talk first about your production of [name of supported crop] including the practices and technologies you use to grow and harvest [name of supported crop]. We’ll talk about processing and selling later. 3. Since you began working with TAP, what kinds of support have you received from [name of project/partner] for growing, harvesting, processing, or selling [name of supported crop]? Probe only if necessary  Did you receive any training? Explain.  Did you receive any tools? Explain.  Did you receive any equipment? Explain.  Did you receive any plants or seeds? Explain.  Did you receive help selling? Explain 4. How do you work together as a group to benefit from the support you received? Probes to understand how the group functions. Suggested probes:  How do you share infrastructure such as tunnels or dryers?  Do you all own land? If not, are some of you tenants on land owned by other member of the group?  Do you plant, cultivate, harvest, process, or sell individually or as a group? 5. Since you began working with TAP, have you received support from any other organization or project for growing, harvesting, processing, or selling [name of supported crop]? Describe. Probe (anything else?) to discover all types of support. 6. Are you using different equipment or practices to grow, harvest, or process [name of supported crop] because of the support you received from [name of project/partner]? Explain. Note that some respondents may not have grown [name of supported crop] prior to the project. Probe only if necessary to understand the extent to which TAP support influenced production, harvesting, or processing practices.  Are you using different varieties? Explain.  Are you using different techniques, tools, or equipment? Explain.  Are you using different inputs or using inputs differently? Explain.  Anything else? Explain. THE AGRIBUSINESS PROJECT EVALUATION 94 7. Have the different equipment and practices affected the quantity of [name of supported crop] you produce? Explain. Probe only if necessary. Probes should focus on learning how the practices affected the quantity they produced.  Has it changed the amount of land you planted to [name of supported crop]? How?  Has it helped increase yields? How? If not, why not? 8. How, if at all, have the different equipment and practices affected the quantity of [name of supported crop] you lost/wasted? Explain. Probe only if necessary  Better harvesting or handling practices.  Better production practices  If not, why not? 9. How, if at all, have these practices affected the cost of producing and selling [name of supported crop]? If it costs more, is the added cost worth it? Probe only if necessary. Probes should focus on learning how the practices affected production and marketing costs.  Did adopting the practices cost more than your usual practices?  Did the practices save you money, perhaps by saving time? 10. Which of the new equipment and practices you use to grow, harvest, process, or sell [name of supported crop], did you find the most useful in increasing your profits from [name of supported crop]? Why? Probe only if necessary and applicable about,  New varieties  New production practices  Tools and equipment  Value addition  Marketing 11. Are there other kinds of support you still need that could increase your profits from growing [name of supported crop]? Explain.  12. Did you know about these technologies and practices before receiving support from TAP? If yes, why weren’t you using them, why not? Probe only if necessary.  Did you not know about the practices?  Were you not able to afford to implement the practices?  Were you not able to obtain necessary tools or other inputs? 13. You’ve told us about new practices you are using now to produce, harvest, process, or sell [name of supported crop]. Can you think of any reasons you might stop using these practices in the future?  If a tool or piece of equipment breaks, could you afford to repair or replace it?  Is there anything else? Keep probing until participants can’t think of other reasons. 14. How likely is it that you will continue to use these practices in the future? Explain.  Employment 15. Do you employ any labor that you pay, either in cash or in-kind? Use this as a filter to know how to ask the following questions. THE AGRIBUSINESS PROJECT EVALUATION 95 16. Has the number of people you employ to produce, harvest, process, or sell [name of supported crop] changed when you started using the practices and technologies you received from TAP? Explain. Probe for the reasons they need more or less labor requirements for growing, harvesting, and processing? Probe for changes in number of men and women employed. 17. Are the laborers you employ men, women, or both men and women? Use this as a filter to know how to ask the following questions. 18. If they employ more labor: What kind of jobs are these people doing? What kind of jobs are men doing? What kind of jobs are men doing? Are the jobs seasonal jobs or full-time jobs? Probe for details on the different types of jobs men and women do in producing, harvesting, processing, and selling and what kind of jobs have been created. 19. Do you know what these laborers were doing before you employed them? Probe to understand whether the project created new or better jobs. Closing Questions 20. Is there anything that TAP could have done better? Explain. Probe if necessary:  Were there things about the support you received from TAP that did not work or did not work as well as you expected? Explain. 21. Since you received the support from TAP, do you know of other farmers or FEGs that have seen what you doing and have started doing the same things? Explain. Probe to understand whether the replication was really a result of a demonstration effect of TAP. THE AGRIBUSINESS PROJECT EVALUATION 96 TAP Evaluation Focus Group Discussion Guide –Farmers’ Enterprise Groups Introduction and Purpose of Discussion Assalam-o-alaikum, thank you for coming here today. We are grateful that you are giving us your time. Before we begin, we will introduce ourselves. [The moderator and the note-taker should introduce themselves] We are here to discuss with you about The Agriculture Project which is being implemented by the Agriculture Support Fund (ASF) and [name of local implementing partner] and how this project may have helped you with your [name of supported crop] growing, harvesting and selling. Through this discussion, we will ensure that your ideas and opinions are heard. We would like to record these discussions so that we remember and do not miss any of the ideas that you give us. The details of these discussions will not be shared with anyone and your names will be kept confidential - so please do not worry and feel free to express your ideas. Your participation in this discussion is voluntary and we really hope you are willing to participate actively and with enthusiasm. We would like to ask you about your experience with date growing and harvesting and selling and how the different training and equipment you have received has helped you improve your production, harvesting and farm sales. Will you allow us to record this discussion? Thank you THE AGRIBUSINESS PROJECT EVALUATION 97 Introductory Questions 22.Can you tell me something about your farming activities. How large is your farm and what do you grow? 23.When did this FEG begin receiving support from TAP? We want to talk first about your production of [name of supported crop] including the practices and technologies you use to grow and harvest [name of supported crop]. We’ll talk about processing and selling later. 24. Since you began working with TAP, what kinds of support have you received from [name of project/partner] for growing, harvesting, processing, or selling [name of supported crop]? Probe only if necessary  Did you receive any training? Explain.  Did you receive any tools? Explain.  Did you receive any equipment? Explain.  Did you receive any plants or seeds? Explain.  Did you receive help selling? Explain 25. How do you work together as a group to benefit from the support you received? Probes to understand how the group functions. Suggested probes:  How do you share infrastructure such as tunnels or dryers?  Do you all own land? If not, are some of you tenants on land owned by other member of the group?  Do you plant, cultivate, harvest, process, or sell individually or as a group? 26. Since you began working with TAP, have you received support from any other organization or project for growing, harvesting, processing, or selling [name of supported crop]? Describe. Probe (anything else?) to discover all types of support. 27. Are you using different equipment or practices to grow, harvest, or process [name of supported crop] because of the support you received from [name of project/partner]? Explain. Note that some respondents may not have grown [name of supported crop] prior to the project. Probe only if necessary to understand the extent to which TAP support influenced production, harvesting, or processing practices.  Are you using different varieties? Explain.  Are you using different techniques, tools, or equipment? Explain.  Are you using different inputs or using inputs differently? Explain.  Anything else? Explain. 28. Have the different equipment and practices affected the quantity of [name of supported crop] you produce? Explain. Probe only if necessary. Probes should focus on learning how the practices affected the quantity they produced.  Has it changed the amount of land you planted to [name of supported crop]? How?  Has it helped increase yields? How? If not, why not? 29. How, if at all, have the different equipment and practices affected the quantity of [name of supported crop] you lost/wasted? Explain. Probe only if necessary  Better harvesting or handling practices.  Better production practices  If not, why not? THE AGRIBUSINESS PROJECT EVALUATION 98 30. How, if at all, have these practices affected the cost of producing and selling [name of supported crop]? If it costs more, is the added cost worth it? Probe only if necessary. Probes should focus on learning how the practices affected production and marketing costs.  Did adopting the practices cost more than your usual practices?  Did the practices save you money, perhaps by saving time? 31. Which of the new equipment and practices you use to grow, harvest, process, or sell [name of supported crop], did you find the most useful in increasing your profits from [name of supported crop]? Why? Probe only if necessary and applicable about,  New varieties  New production practices  Tools and equipment  Value addition  Marketing 32. Are there other kinds of support you still need that could increase your profits from growing [name of supported crop]? Explain.  33. Did you know about these technologies and practices before receiving support from TAP? If yes, why weren’t you using them, why not? Probe only if necessary.  Did you not know about the practices?  Were you not able to afford to implement the practices?  Were you not able to obtain necessary tools or other inputs? 34. You’ve told us about new practices you are using now to produce, harvest, process, or sell [name of supported crop]. Can you think of any reasons you might stop using these practices in the future?  If a tool or piece of equipment breaks, could you afford to repair or replace it?  Is there anything else? Keep probing until participants can’t think of other reasons. 35. How likely is it that you will continue to use these practices in the future? Explain.  Employment 36. Do you employ any labor that you pay, either in cash or in-kind? Use this as a filter to know how to ask the following questions. 37. Has the number of people you employ to produce, harvest, process, or sell [name of supported crop] changed when you started using the practices and technologies you received from TAP? Explain. Probe for the reasons they need more or less labor requirements for growing, harvesting, and processing? Probe for changes in number of men and women employed. 38. Are the laborers you employ men, women, or both men and women? Use this as a filter to know how to ask the following questions. 39. If they employ more labor: What kind of jobs are these people doing? What kind of jobs are men doing? What kind of jobs are men doing? Are the jobs seasonal jobs or full-time jobs? Probe for details on the different types of jobs men and women do in producing, harvesting, processing, and selling and what kind of jobs have been created. 40. Do you know what these laborers were doing before you employed them? Probe to understand whether the project created new or better jobs. Closing Questions THE AGRIBUSINESS PROJECT EVALUATION 99 41. Is there anything that TAP could have done better? Explain. Probe if necessary:  Were there things about the support you received from TAP that did not work or did not work as well as you expected? Explain. 42. Since you received the support from TAP, do you know of other farmers or FEGs that have seen what you doing and have started doing the same things? Explain. Probe to understand whether the replication was really a result of a demonstration effect of TAP. THE AGRIBUSINESS PROJECT EVALUATION 100 The Agriculture Project Evaluation Interview Guide – IMAP Participants 1. Can you please tell us a little about your company; what is your main activity, do you export, what other types of activities are you engaged in (e.g. trading, processing, farming, etc.) 2. What kinds of support have you received from ASF Project? Probe if necessary for: a. Participate in any national/international business promotion/marketing event b. Branding and marketing support c. Support for improving packaging, export operation, and export documentation d. Support to obtain certifications required by international markets 3. Are you selling to new international markets/buyers (i.e., who you did not sell to before ASF) as because of the support you received from ASF? Explain. a. Probe to understand attribution, e.g., previous export experience and specifically how ASF support expanded opportunities or market access (certifications, introductions, exposure). Did new markets displace existing markets with little or no increase in overall sales volume? If so, were new markets more lucrative or more desirable? Exposure Visits 4. Can you tell us how the exposure visit/exhibition you attended with ASF support? a. What was the purpose, a study tour, exposure visit, marketing exhibition, etc. b. What costs did ASF cover and which were your responsibility? Probe as appropriate for travel, lodging and meals, exhibit space/booths, other) 5. What was the most important benefit to you of participating in the event? Probe if necessary for knowledge, contacts, contracts, other. 6. Did participating in the event affect the value of your sales and/or exports? Please explain how? If not why not? a. If you found new buyers, did you sell only once or did you establish a longer-term relationship with the buyer? 7. You incurred a cost in time and money to participate in the event. Was it worth it for your business to participate in the event? Why? 8. Had you been to similar events before your participation in this event ( with or without assistance e.g. donor, government) ? 9. Have you been to similar events since you received the assistance from ASF. Which ones? Did you receive any support to attend/participate? Other type of assistance e.g. branding, certifications, packaging, B2B linkages etc 10. If applicable: Did the other types of assistance you received from ASF affect the value of your sales and/or exports? Please explain how? If not why not? (Probe for change in quantities sold, changes in price or cost, more/less efficient practices or technologies) Employment THE AGRIBUSINESS PROJECT EVALUATION 101 11. Do you employ only men, only women, or both men and women? a. Did these changes affect the number of workers you employ? If yes how? (Probe for producing/processing/marketing large quantities, more/less efficient practices, more/less labor intensive practices) i. If employment increased: What kinds of jobs were created (skilled/unskilled, seasonal/full-time, men/women) i. Phrase as relevant: What roles do men and women have in your business? ii. Ask only if employment increased: Do you know what the new workers you hired were doing before you hired them? Probe to understand whether the jobs were new jobs (hire people who were not working) or better jobs (hired people who were working for less) Sustainability 12. If applicable: Do you think you continue the new market connections, practices, and technologies you adopted with ASF support after ASF is finished? If yes, how? If no, why not? 13. If applicable: What could make it more likely that your business will continue to experience the benefits you achieved with ASF assistance? Strengths and Weaknesses of TAP 14. In your opinion, what do you think ASF could have done to achieve better results for your business? Explain. THE AGRIBUSINESS PROJECT EVALUATION 102 The Agriculture Project Evaluation Interview Guide – Implementing Partners 1. What is the expertise of your organization? Did you have experience in agricultural value chains or agricultural development before working with TAP? Please describe that experience. 2. What was your role in implementing TAP? (Probe for role in forming and supporting FEGs.) 3. How did you come to partner with ASF on TAP? How would you describe your partnership? 4. What were your main objectives working with FEGs? 5. What latitude were you provided in designing and delivering technical assistance and/or training? 6. Did you work with local and international experts? If yes, what did you think of the quality of consultants provided? 7. Did you know that the project was redesigned in 2013? a. In addition to reducing the number of value chains, how did the project change from the original design? (Probe for changes in the approach, partnerships, roles of IPs, nature and use of grants, management) b. In your opinion, what were the main reasons for the re-design? c. Were you involved in the re-design? d. If yes, to what extent were you involved and who else was involved? (Probe for international partners, USAID, other stakeholders) e. Did the re-design affect your work or your role in TAP? How? 8. In your opinion, what were the strengths of project’s design? 9. In your opinion, what were the weaknesses of the project’s design? 10. Was there a specific objective of including women in the design and implementation of your work for TAP? If so, what was it? Did it work? 11. What challenges did you face implementing the project? (Probe for any problems with FEGs, grants, linkages to processors or markets, outside technical assistance) 12. How did you overcome these challenges? 13. In your opinion, what constitutes a successful FEG? 14. What challenges do FEGs or larger farmers/corporate farmers face linking to markets in the _____ value chain? 15. In your opinion, how likely is it that FEGs will be able to sustain the results they’ve achieved under TAP after TAP assistance ends? Why are the results sustainable? 16. What will be the challenges to sustaining results? 17. Is there anything else the project or future projects could do to increase prospects for sustainable results? 18. What would you have done differently as an IP for this project? THE AGRIBUSINESS PROJECT EVALUATION 103 The Agriculture Project Evaluation Interview Guide – Meat Industry Managers Learn how the respondent knows TAP (by what name) and use that name throughout the interview. 1. What kinds of support have you or your company received from TAP. (Probe for training, certifications, tools/equipment, market linkages, grants, other) 2. Have you changed the way you do your business as a result of the assistance? How? If not, why not? (Probe for relevance of each type of assistance) 3. Did these changes affect the value of your sales? If yes how? (Probe for access to different markets, higher prices, larger quantities sold, standardization, certifications, exports) 4. If relevant: Did these changes affect the value of your exports? If yes how? (Probe for access to different markets, higher prices, larger quantities sold, standardization, certifications) 5. If relevant: How important were certifications to accessing international markets? 6. Do you employ only men, only women, or both men and women? a. Did these changes affect the number of workers you employ? If yes how? (Probe for producing/processing/marketing large quantities, more/less efficient practices, more/less labor intensive practices) i. If employment increased: What kinds of jobs were created (skilled/unskilled, seasonal/full-time, men/women) ii. Phrase as relevant: What roles do men and women have in your business? iii. Ask only if employment increased: Do you know what the new workers you hired were doing before you hired them? Probe to understand whether the jobs were new jobs (hire people who were not working) or better jobs (hired people who were working for less) 7. As relevant: You've told us that sales/exports/jobs have changed as a result of assistance from TAP. Do you think these are permanent changes or are they only temporary? Explain. (Probe for reasons results might be temporary) 8. During the past 4 years, are there other things internal or external to your organization that affected the value of your sales or exports or the number of people you employ? Please explain? 9. Are there other changes that could take place in your industry that would improve your business? Explain. (Probe for constraints in other parts of the value chain. Probe specifically for availability of BDS, financing, TA/human resource capacity) 10. Other than things we've already discussed, do you have suggestions for how the assistance could have been improved? (Probe as appropriate for quality of assistance/TA, timing, sequencing, relevance, completeness, other) THE AGRIBUSINESS PROJECT EVALUATION 104 The Agriculture Project Evaluation Interview Guide – TA Providers Learn how the respondent knows TAP (by what name) and use that name throughout the interview. 1. What is your field of expertise and what role did you play in implementing ASF's project? 2. What knowledge do you expect trainees to acquire after successful completion of their courses or the assistance you provide? a. What results did the selected approach attain? Did you ever follow-up on your graduates (success ratio, best practices being followed) 3. How was the course work/syllabus designed (was it your prerogative based on your technical expertise or did ASF develop the material)? 4. What results did the selected approach attain? Did you ever follow-up on your graduates (success ratio, best practices being followed) 5. Were you approached at the right time by ASF regarding the technical assistance provision? (Probe about sequencing of interventions) 6. In your opinion what interventions/ assistance would have help more to produce better results in terms of skills, knowledge, productivity, sales etc.? 7. To what extent you think the project design accounted for the operating environment (including business development services, availability of finances, and technical capacity building)? 8. Do you think the designed course of assistance was beneficial in developing a successful business or partnership strategy(ies) in current context? 9. Did the course and/or technical assistance extended by you had any emphasis on women learning? 10. How do you think the technical assistance has changed the productivity and the subsequent sales? 11. How useful do you think the assistance packages provided were? (Tools and/or equipment if provided) 12. Were new crop varieties (i.e., grapes, potatoes, apricots) introduced by the project appropriate and effective (in terms of productivity, sales, market demand etc.)? 13. Do you think the crop varieties introduced by the project were according to the current market requirements? In terms of: a. Production and management Techniques b. Business development c. Increase in sales or exports 14. Did the assistance provide focused on domestic and international markets appropriately? Was its approach (market events, certifications) effective? 15. Was the quality, content, and delivery of training(s) and technical assistance, appropriate? 16. Were the interventions appropriate to the particular/ given context? Why? THE AGRIBUSINESS PROJECT EVALUATION 105 17. What kind of assistance/services did you provide to the ASF/TAP? (Adaptation of modern production techniques, marketing, business management technologies) 18. In your opinion the current services provision practice would increase in the production of target value chain and/or profitability of the value chain? 19. To what extent the capacity building services will change the efficiency? THE AGRIBUSINESS PROJECT EVALUATION 106 The Agriculture Project Evaluation Interview Guide – USAID 1. Were you involved in the original design of TAP? 2. In addition to the reduction in value chains, how did the project design or implementation change during the de-scoping in 2013? (Probes: approach; partners; grants; concerns about management) 3. In your opinion, what were the main reasons for the de-scoping? 4. How were the final eight value chains selected? (Probes: potential for results, geographic location, USAID priorities) a. Did USAID play any role in the decision? 5. Do you know how ASF selected local partners like NGOs, TA providers, international partners? 6. To what extent did the project design specifically address sustainability and how? 7. Did the project design include a specific strategy for including women? Explain. 8. Did ASF and its partners implement the project as designed? If not, how and why did it deviate from the design or planned implementation approaches? 9. To what extent were the planned interventions sufficient for achieving desired results? a. What other interventions should the project have implemented that wasn’t part of the original design to address weaknesses or gaps in the various value chains? 10. Were initial targets realistic and achievable? 11. How were new targets established after the redesign? What role did USAID play in setting new targets? 12. Did ASF encounter any particular challenges implementing the project? Explain. (Probes: managing grants, selecting appropriate partners, identifying appropriate technical expertise, facilitating links between value chain actors, …) 13. What role are the value chain platforms expected to play after TAP ends? a. What will make them successful? 14. Does the project have a specific strategy for sustainability or an exit strategy? 15. In your opinion, are project results sustainable? Why? a. Are results in some value chains or for some value chain actors (e.g., FEGs versus individuals) more sustainable than others? How and why? b. Is there anything else the project or future projects should do to ensure sustainability? 16. Can you think of examples where the project’s intervention has leveraged private sector resources to enhance results throughout the value chain? 17. What do you think is the project’s biggest success? 18. What do you think could have been done better? THE AGRIBUSINESS PROJECT EVALUATION 107 The Agriculture Project Evaluation Interview Guide – ASF Value Chain Leads 1. Can you describe your professional experience in the _____ value chain? 2. Why did ASF decide to work in the _________ value chain? (Probe for rationale for selecting the value chain, e.g., contribution to project objectives, USAID priority, ASF experience, other donor support) 3. What constraints and opportunities did ASF’s assessment of the _______ value chain identify? (Probe to identify all constraints and opportunities along the entire value chain, not just the things ASF chose to work on) a. Who conducted the assessments? (Probe for local or international experts, ASF or contracted) b. How satisfied are you with the quality of the assessment? Why? (Probes: was it complete, was it accurate, was it informative, did it cover the entire value chain – demand and supply sides,…) c. Were other stakeholders or local experts involved in the assessments? 4. What interventions did ASF design and implement to address identified constraints and take advantage of opportunities? (Probe for interventions along the entire value chain, which VC actors did ASF work with and how, were interventions sufficient, were any unnecessary) 5. In your experience, what worked particularly well in ASF’s support to the _______ value chain and why? (Probes: were interventions appropriate, were they effective in increasing sales, exports, jobs, or income) 6. In your experience, what did not work well and why? (Probes on whether interventions were appropriate, did ASF and USAID internal processes/procedures cause any problems, was the grant process easy and quick, a. Did you experience any internal or external challenges working with this value chain? Explain. (Probe for issues related to design or implementation, e.g., grant processes, procurement, experience with local partners/TA providers/vendors/service providers, other) 7. In your experience, was the project successful in increasing the value of sales and exports, creating jobs, and improving incomes for beneficiaries in the _______ value chain? How? a. In your opinion, which of the interventions was most effective or most important to achieving results? 8. In your opinion, what could ASF have done better to achieve better results? 9. In your opinion, how likely is it that the benefits beneficiaries have achieved will continue after ASF ends? How and why? THE AGRIBUSINESS PROJECT EVALUATION 108 Annex 7: Interviews TABLE 13: LIST OF INTERVIEWS AND SITE VISITS Individual Organization Position Location Role in TAP/evaluation 6/13/15 Shad M ASF CEO Islamabad Implementer Inamullah Khan Senior Advisor M&E Implementer 6/19/15 Saeed ur Rehman ASF Value Chain Lead, Grapes Islamabad Implementer Rhustam Khan Value Chain Lead, HV/OSV (north) Mohammad Akbar Regional Coordinator, FATA and KP Shafqat Ali Syed Director, Grants and Contracts Muhammad Ashan Riaz Value Chain Lead, Meat Sector Aeyesha Gulzar National Coordinator 6/22/15 Saeed Iqbal ASF Value Chain Lead, Regional Coordinator Islamabad Implementer 6/25/15 Muhammad Ashan Riaz, Ph.D. ASF Value Chain Lead, Meat Sector Islamabad Implementer 6/26/15 Ashlaq Husssain NRSP Project Coordinator Islamabad Implementing partner Mohammad Khalil Farooq NARC Director, Agriculture Poli￾Technique Institute Islamabad Pre-qualified service provider Raja Zaheer Ahmed Individual Farmer Vineyard Owner Islamabad Grape individual grant recipient Husnain Chaudry Individual Farmer Son of vineyard owner Shahid Mahmood Caudry Islamabad Grape individual grant recipients (2) 6/27/15 Mansoor Ayaz, M. Shafir Khan and M. Siddique Individual Farmer Son of vineyard owner MNA Ghulam Sawar Khan and nephew of vineyard owners M. Shafiq Khan and M. Siddique Islamabad Grape individual grant recipients (3) Syed Tanvir Individual Farmer Brother of Syed Asad Ali Shah, owner of the farm Islamabad Grape individual grant recipient Jibran Naseer Individual Farmer Son of vineyard owner33 Islamabad Grape individual grant recipient Noshaba Shehzad Masud Shehzad Asia International Owner Islamabad IMAP participant Mohammad Ghani Khan USAID/Pakistan AOR USAID/Pakistan AOR 6/28/15 Kot Key FEG Attock Grape FEG Choudhry Abad FEG Attock Grape FEG Zarkhaiz Pakistan FEG Attock Grape FEG Modrotha 1 FEG Attock Grape FEG 6/29/15 Mogla FEG group FEG Chakwal Grape FEG 33 His father, who was the grantee, recently passed away and his son inherited the farm. THE AGRIBUSINESS PROJECT EVALUATION 109 Individual Organization Position Location Role in TAP/evaluation interview Akwal FEG group interview FEG Chakwal Grape FEG Zaffar Iqbad Individual farmer Chakwal Grape individual grant recipient 6/30/15 Rustam Khan ASF Value Chain Lead, HV/OSV (south) Lahore Implementer Muhammad Hyat Jaspel, Ph.D. UVAS Professor, Meat Science Department, Business Incubation Center Lahore BDSP Farzand Ali Shafi Food Pvt. Ltd. Supervisor/Lead Butcher Lahore Meat Cuts and Standardization Training participant Adeel Supervisor/Lead Butcher Muhammad Junaid Kold Kraft Manager, Quality Assurance Lahore Meat Cuts and Standardization Training Ali Khan Production In participant -charge Amir Sajjad Individual farmer Lahore HV/OSV individual grant recipient 7/1/15 Mr. Murtaza RCDS Chief Executive Officer Lahore IP Farrah Hassan Individual Farmer Lahore HV/OSV individual grant recipient Shamoon Sadiq Fresh Pak Executive Manager Operations Lahore IMAP participant and Challenge Grant recipient Azam Sahib Abedin International Administrastor Lahore Meat Cuts and Standardization Training Mian Abdul Hannan Tazij Meat and Food Managing Director Lahore IMAP participant Mr. Konrad External Consultant Dr. Mohammad Ibrahim Feed Stock (Pvt) Ltd. Managing Director Lahore Feedlot Fattening Dr. Maqsood Shah Mohammad Dairy consultant Lahore Feedlot Fattening participant Multani Vegetable Group FEG Lahore HV/OSV FEG Pakaman Vegetable Group FEG Lahore HV/OSV FEG FEG-Sahara Vegetable Group FEG Lahore HV/OSV FEG FEG-Khudpur Vegetable Group FEG Lahore HV/OSV FEG 7/2/15 Haneef Farm Vegetable Group FEG Lahore HV/OSV FEG Khokar Vegetable Group FEG Lahore HV/OSV FEG Naveed Sadiq PAMCO Ex. Manager Operations Lahore IMAP Participant Muhammad Zahidd Durrani Hunza Organics CEO Lahore IMAP participant; resource person Tahir Yousef Green Revolution Assistant Technical Manager Lahore Challenge Grant recipient Mian Asif Maqbool Taha Trading Proprietor Lahore IMAP participant Ch. Muhammad Imtiaz Elahi Individual Farmer Lahore HV/OSV individual grant recipient Muhamad Tahir, son of Individual Farmer Nankana HV/OSV individual grant THE AGRIBUSINESS PROJECT EVALUATION 110 Individual Organization Position Location Role in TAP/evaluation Muhammad Ashraf recipient Zohaib Rameez Individual Farmers Nankana HV/OSV individual grant recipient (3 grants) Sajjid Mehmood Naveed Hussain 7/4/15 Phool FEG Peshawar HV/OSV FEG Maimar FEG Peshawar HV/OSV FEG Green House FEG Peshawar HV/OSV FEG Chand FEG Peshawar HV/OSV FEG Mohammad Naveed Peshawar AI Training participant Mohammad Nisar Private Veterinarian Peshawar AI Training participant Muhammad Nauman Peshawar AI Training participant Rafi Ullah Peshawar AI Training participant Shahzad Gul Tahir Shah Beef and Mutton Shop Partner Peshawar Meat Cuts and Standardization Training participants Lal Zada Partner Haji Habib ur Rehman Jan’s Meat Shop Peshawar Meat Cuts and Standardization Training participants Masood Ahmed Malik Sunny International Managing Director Peshawar IMAP participant Naseem Akthar New Shan Enterprises Peshawar IMAP participant; BDSP training provider Mr. Muhammad Ishaq MFEDO Peshawar HV/OSV FEG member; BDSP training provider 7/5/15 Dr. Saima Bibi Peshawar Feedlot Farming participant 7/6/15 Muhammad Akbar ASF Regional Coordinator, HV/OSF (north) Peshawar Implementer Zahid Kurshid SRSP Project Manager Peshawar Implementing partner Ahmed Rizwan NWFP Agriculture University Director of Planning and Development Peshawar Business Service Provider Abdus Salam Additional Director Planning and Development Muhammad Fayaz Ahmad Hamza Halal Foods Managing Director Peshawar IMAP participant; Meat Cuts and Standardization Training Siaf ur Rehman Individual farmer Peshawar HV/OSV individual grant 7/7/15 Mr. Muhammad Ishaq MFEDO Peshawar BDSP training provider Naseem Akthar Azam Farooq HRDC Coordinator Peshawar BDSP training provider Suleman Amin Individual farmer Peshawar HV/OSV individual grant Kastkar FEG Charsadda HV/OSV FEG Rohana Kissan FEG Charsadda HV/OSV FEG Sher Bahdar Kalay FEG Charsadda HV/OSV FEG Gulistan FEG Charsadda HV/OSV FEG 7/8/15 Muhammad Fayyaz Khan Hashoo Foundation Project Manager Islamabad Implementing partner Fouzia Sakhi MEAL Manager Bushra Usman Assistant Manager – Donor Liaison Col. Muhammad Sidiq Project Director Anum Mughal ECI Pvt Director of Client Relations Islamabad Business Service Provider THE AGRIBUSINESS PROJECT EVALUATION 111 Individual Organization Position Location Role in TAP/evaluation Asfandyar Khan Islamabad Manager Projects Falah FEG Manshera HV/OSV FEG Alfalah Group FEG Manshera HV/OSV FEG Itehad Welfare Group FEG Manshera HV/OSV FEG Jagger Sabza Group FEG Ghizer HV/OSV FEG 7/915 Saeed Iqbal ASF Regional Coordinator, VCL/VC Team Gilgit Implementer 7/10/15 Randy Chester USAID/Tanzania Office Director, Economic Growth Office Phone Office Director, Agriculture Office in USAID/Pakistan Bilal Ahmed Individual farmer Gilgit Apricot individual grant recipient Malik Miskeen Individual farmer Gilgit Apricot individual grant recipient Mehdiabad Group FEG Gilgit-Niltar Seed potato FEG Faizabad Nomal FEG Gilgit Apricot FEG Bulbul Shah Individual farmer Gilgit Seed potato individual grant recipient 7/11/15 Akthar Hussain Organo Botanica Manager Operations Gilgit Grant Recipient 7/12/15 White Apricot Group FEG Ghizer Apricot FEG Friends Business Group FEG Ghizer Apricot FEG Local Apricot Growers Golodas FEG Ghizer Apricot FEG Dry Apricot Producers Golodas FEG Ghizer Apricot FEG Naeem Khan Individual Grant recipient Ghizer Apricot individual grant recipient Taighoon Wali Khan Individual Grant recipient Ghizer Apricot individual grant recipient 7/13/15 Itifaq Group FEG Ghizer Seed potato FEG Sahibabad Sandi Bala FEG Ghizer Seed potato FEG Burndass Shah Group FEG Ghizer Seed potato FEG Fertile Group FEG Ghizer Seed potato FEG Saeed-ur-Rehman ASF Value Chain Lead, Grapes Islamabad Implementer Kifayat Zaman ASF Engineering Specialist Islamabad Implementer Mehboob Wali Khan Individual farmer Owner Phander, Gizar Seed potato individual grant recipient Faryad Hussain Individual farmer Owner Phander, Gizar Seed potato individual grant recipient 7/14/15 Brosha Group B FEG Hunza-Nager Seed Potato FEG Bercha Group Holshal FEG Hunza-Nager Seed Potato FEG Muhammad Ali Individual farmer Hunza-Nager Apricot individual grant recipient Musa Ali Individual farmer Hunza-Nager Seed potato individual grant recipient Abdullah Individual farmer Hunza-Nager Seed potato individual grant recipient Muhammad Iqbal Shah Associates Hunza-Nager Apricot individual grant recipient 7/15/15 THE AGRIBUSINESS PROJECT EVALUATION 112 Individual Organization Position Location Role in TAP/evaluation Amjad Wali AKRSP Gilgit Implementing partner Inam Ullah Mountain Fruits Pvt. Ltd. Managing Director Gilgit IMAP participant Gulmit Goze Women Group FEG Hunza-Nagar Apricot FEG Fotan Dar Women Group FEG Hunza-Nagar Apricot FEG 7/24/15 Farhan Rabbani Al Mahmood Establishment Owner Karachi IMAP participant 7/25/15 Abdul Karim Organic Meat Company (Pvt) Ltd. Exports Manager Karachi IMAP participant Ali Hussain Director Saeed Khan Saeed Khan Enterprises Owner Karachi IMAP participant 7/27/15 Wajahat Hussain Khan Seven Star International GM Operations Karachi IMAP participant Bilal Shahid Tata Tata Best Foods COO Karachi IMAP participant Muhammad Zaman Anas Anus Tropical Owner Karachi IMAP participant THE AGRIBUSINESS PROJECT EVALUATION 113 Annex 8: Sampling The team selected samples of FEGs, individual grant recipients, IMAP participants, and mean training participants randomly from among those within reach of central locations the team selected for site visits. Sites were selected based on the number of beneficiaries and participants within reach of the central location, coverage of major activities, and security. Based on these criteria, the team organized interviews in the areas in and around Islamabad, Lahore, Peshawar, and Gilgit with side trips to Chakwal, Attock, Charsadda, Manshera, Ghizer, and Hunza. One team member also traveled to Karachi to interview IMAP participants. Table 14 shows the location of project-supported FEGs, individual grant recipients, IMAP participants, and meat training participants by location, beneficiary type, and value chain. The shaded rows indicate the districts selected for site visits. The “# in sample districts” row at the bottom of Table 14 indicates the number of beneficiaries in the districts selected for site visits. The “% coverage” row shows the percentage of all beneficiaries that fell within the sampling frame. The sampling frame based on the selected districts encompassed between 39 percent and 100 percent of beneficiaries, depending on value chain and beneficiary type. The “% by beneficiary type” row indicates the percentage of a particular type of beneficiary (e.g., FEG, individual grant recipient) within each value chain and provides a point of reference for distributing the sample across value chains. Table 15 shows the distribution of the sample across location, type of beneficiary, and value chain. The “% by beneficiary type” row at the bottom of the table indicates the percentage of the sample of a particular beneficiary type allocated to each value chain and indicates the proportional representation of each beneficiary type/value chain set relative to the shares of all beneficiaries shown in Table 14. In addition to the interviews documented in Table 15, the team interviewed:  13 TAP staff members;  5 (of 7) local implementing partners (AKRSP, Hashoo Foundation SRDP, RCDS, and NRSP);  7 (of 35) technical assistance partners and BDSPs (NARC, UVAS, New Shan Enterprises, Modern Family & Enterprise Development Organization, NWFP Agriculture University, HRDC, ECI Pvt Islamabad);  2 USAID (Muhammad Ghani Khan, Randy Chester);  2 (of 7) challenge grant recipients (Fresh Pak, Green Revolution); and  4 (of 44) AI trainees. Except for TAP staff members USAID personnel which were selected purposively, the rest of these beneficiaries and participants were selected because they were located in or near the central locations the team visited, i.e., they represent convenience samples. THE AGRIBUSINESS PROJECT EVALUATION 114 TABLE 14: LIST OF BENEFICIARIES BY LOCATION, TYPE, AND VALUE CHAIN Province District FEGs Individual Grant Recipients IMAP Meat Training Apricot Grape HV/OSV Potato Apricot Grape HV/OSV Potato AJK Kotli 1 FATA Bajaur Agency 8 Khyber Agency 2 GB Chitral 10 Diamer 53 42 1 Ghanche 17 2 13 Ghizer 14 50 6 4 Gilgit 14 11 11 1 1 Hunza 35 21 1 1 7 KP Abbottabad 11 Charsadda 23 29 D.I. Khan 8 Haripur 26 Lower Dir 25 Malakand 1 10 Mansehra 40 40 14 Mardan 7 Nowshera 26 6 Peshawar 33 22 7 29 Swabi 30 24 Swat 44 6 Punjab Attock 12 45 29 THE AGRIBUSINESS PROJECT EVALUATION 115 Province District FEGs Individual Grant Recipients IMAP Meat Training Apricot Grape HV/OSV Potato Apricot Grape HV/OSV Potato Chakwal 3 17 11 Chiniot 1 Faisalabad 3 ICT/Rawalpindi 2 6 3 4 Kasur 10 Khanewal 1 Khushab 1 Lahore 16 14 14 86 Multan 6 Muridke 1 Nankana Sahib 19 12 Rawalpindi 11 11 Sargodha 3 Sharaqpur 1 Sheikhupura 25 26 Sindh Malir 1 Karachi 19 116 Kharipur 1 Number of beneficiaries 80 17 335 174 20 79 315 11 55 231 # in sample districts 63 15 131 122 18 62 133 11 41 231 % coverage 79% 88% 39% 70% 90% 78% 43% 100% 75% 100% % by beneficiary type 13% 3% 55% 29% 5% 19% 74% 3% 100% 100% Source: TAP M&E system data provided by ASF. THE AGRIBUSINESS PROJECT EVALUATION 116 TABLE 15: NUMBER OF INTERVIEWS BY LOCATION, TYPE, AND VALUE CHAIN Province District FEGs Individual Grant Recipients IMAP Meat Training Apricot Grape HV/OSV Potato Apricot Grape HV/OSV Potato GB Ghizer 4 1 4 2 2 Gilgit 1 1 2 1 1 Hunza 2 2 2 2 KP Charsadda 4 Mansehra 3 Peshawar 5 2 3 4 Punjab Attock 4 Chakwal 2 1 ICT/Rawalpindi 8 1 Lahore 6 3 5 5 Nankana Sahib 4 Sindh Karachi 6 Total interviews 7 7 19 6 6 9 9 5 16 9 % by beneficiary type 18% 15% 49% 15% 21% 31% 31% 17% 100% 100% Note: the number of interviews does not match the number of documents used in the quantitative analysis in some cases because some individual interviews with grant recipients turned into small group interviews. This table includes them as individual interviews because they covered separate businesses. However, the notes are in one document which is the unit of analysis for the qualitative analysis. THE AGRIBUSINESS PROJECT EVALUATION 117 Annex 9: Qualitative Analysis Results TABLE 16: SUPPORT RECEIVED FROM TAP (APRICOT AND POTATO) Support Value Chain and Value Chain Actor Apricot Potato FEG Individual grant recipients Processors/ exporters FEG Individual grant recipients Processors/ exporters Seeds/plants - 4 - 3 - - Certifications - 1 - - - - New varieties - 3 - - - - Training 7 5 - 6 3 - Tools/equipment - 4 - 6 3 - Infrastructure 7 5 - - 5 - Exposure visits 1 1 - - - - Marketing events 1 - - 1 1 - # of documents 7 6 0 6 5 0 TABLE 17: SUPPORT RECEIVED FROM TAP (HV/OSV AND GRAPE) Support Value Chain and Value Chain Actor HV/OSV Grape FEG Individual grant recipients Processors/ exporters FEG Individual grant recipients Processors/ exporters Seeds/plants 16 1 - 1 3 - Certifications - - - - - - New varieties 1 - - - - - Training 18 7 - 4 4 - Tools/equipment 13 1 - 3 - - Infrastructure 10 3 - 4 3 - Exposure visits 3 3 - 2 2 - Marketing events 2 2 - - - - # of documents 18 7 1 7 5 - THE AGRIBUSINESS PROJECT EVALUATION 118 TABLE 18: SUPPORT RECEIVED FROM TAP (IMAP, MEAT, AND TOTAL) Support Value Chain and Value Chain Actor All Value Chains IMAP Meat FEG Individual grant recipients Processors/ exporters Seeds/plants 21 8 - - - Certifications - 1 - 3 - New varieties 1 3 - - - Training 35 19 - 4 5 Tools/equipment 22 8 - - 1 Infrastructure 21 16 - - - Exposure visits 7 7 1 - 1 Marketing events 2 3 2 2 - # of documents 38 22 1 13 6 TABLE 19: TAP’S CONTRIBUTION TO VALUE CHAIN DEVELOPMENT Value Chain Value Chain Actor FEGs Grant recipients Processors/ exporters/IMAP Challenge grant recipients Apricot N=7 N=6 N= N= Lacked financial capacity 3 4 - - Lacked knowledge/awareness/comfort 2 2 - - Potato N=6 N=5 N= N= Lacked financial capacity - 3 - - Lacked knowledge/awareness/comfort 4 1 - - HV/OSV N=18 N=7 N=1 N= Lacked financial capacity 1 2 - - Lacked knowledge/awareness/comfort 9 5 - - Grape N=7 N=4 N= N= Lacked financial capacity 1 - - - Lacked knowledge/awareness/comfort 5 3 - - Meat N=0 N=0 N= N= Lacked financial capacity n.a. n.a. - - Lacked knowledge/awareness/comfort n.a. n.a. - - THE AGRIBUSINESS PROJECT EVALUATION 119 Value Chain Value Chain Actor FEGs Grant recipients Processors/ exporters/IMAP Challenge grant recipients All value chains N=38 N=22 N=1 N= Lacked financial capacity 5 9 - - Lacked knowledge/awareness/comfort 20 11 - - IMAP N= N= N=13 N= Lacked financial capacity - - 1 - Lacked knowledge/awareness/comfort - - 3 - TABLE 20: CHANGE IN TECHNOLOGIES AND PRACTICES (FEGS) Technologies and Practices Apricot Potato HV/OSV Grape All Value Chains New practice - - - - - No change - - 2 - 2 Post-harvest practices - 1 3 - 4 Management practices - - 1 - 1 New crops - - 3 5 8 New varieties - 1 5 - 6 Cropping patterns - 1 4 - 5 Storage 1 1 - - 2 Harvesting 3 1 1 - 5 Processing 7 - 1 - 8 Production - 6 14 1 21 Marketing 3 - 4 - 7 # of documents 7 6 18 7 38 THE AGRIBUSINESS PROJECT EVALUATION 120 TABLE 21: CHANGE IN TECHNOLOGIES AND PRACTICES (INDIVIDUAL GRANT RECIPIENTS AND PROCESSORS/EXPORTERS) Technologies and Practices Individual Grant Recipients Processors/ Exporters Apricot Potato HV/OSV Grape All Valve Chains IMAP Meat New practice - - - - - - - No change - - - - - - 1 Post-harvest practices - - - - - - Management practices - - - 1 1 - - New crops - - - - - - - New varieties - - - - - - - Cropping patterns 1 - 1 - 2 - - Storage - 2 - - 2 - - Harvesting - - - - - - - Processing - - - - - 1 1 Production 2 - 1 - 3 - - Marketing - - 1 1 2 - - # of documents 6 5 7 5 22 13 6 THE AGRIBUSINESS PROJECT EVALUATION 121 TABLE 22: REASONS FOR INCREASING QUANTITY PRODUCED (APRICOT, POTATO, AND HV/OSV) Quantity Produced Value Chain and Value Chain Actor Apricot Potato HV/OSV FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters N.A. - - - - - - 2 - - Increased 6 - - 4 4 - 14 5 - Reduced wastage 6 - - 2 4 - 12 1 - Increased yield - - - 1 - - 11 2 - Increased area - - - 2 - - 1 - - Commercial mindset - - - - - - 1 - - Technology and practices (post-harvest) 6 - - - 1 - - - - Technology and practices (harvest) - - - - - - - - - Technology and practices (pre-harvest) - - - - - - 7 1 - Improved varieties - - - - - - 2 1 - # of documents 7 6 - 6 5 - 18 7 1 THE AGRIBUSINESS PROJECT EVALUATION 122 TABLE 23: REASONS FOR INCREASING QUANTITY PRODUCED (GRAPE, TOTAL, AND MEAT) Quantity Produced Value Chain and Value Chain Actor Grape All Value Chains Meat FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters Processors/ exporters N.A. 1 - - 3 - - - Increased 2 1 - 26 10 - - Reduced wastage - - - 20 5 - - Increased yield 2 - - 14 2 - - Increased area - - - 3 - - - Commercial mindset - 1 - 1 1 - - Technology and practices (post-harvest) - - - 6 1 - - Technology and practices (harvest) - - - - - - - Technology and practices (pre-harvest) - - - 7 1 - - Improved varieties - - - 2 1 - - # of documents 7 4 - 38 22 1 6 THE AGRIBUSINESS PROJECT EVALUATION 123 TABLE 24: CHANGE IN VALUE OF SALES (APRICOT, POTATO, AND HV/OSV) Quantity Produced Value Chain and Value Chain Actor Apricot Potato HV/OSV FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters Yes - - - 1 3 - 10 6 No 1 5 - 1 1 - - 2 - N.A. – no production yet 1 5 - - - - - 1 - # of documents 7 6 0 6 5 0 18 7 1 TABLE 25: CHANGE IN VALUE OF SALES (GRAPE, MEAT, TOTAL, IMAP) Quantity Produced Value Chain and Value Chain Actor Grape Meat All Value Chains IMAP FEG Grant recipients Processors/ exporters Processors/ exporters FEG Grant recipients Processors/ exporters Yes - 1 - 2 11 10 - 2 No - 3 - - 2 11 - 2 N.A. – no production yet - 3 - - 1 9 - - # of documents 7 4 0 6 38 22 1 13 THE AGRIBUSINESS PROJECT EVALUATION 124 TABLE 26: CHANGE IN VALUE OF EXPORTS (APRICOT, POTATO, AND HV/OSV) Quantity Produced Value Chain and Value Chain Actor Apricot Potato HV/OSV FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters Yes - - - - - - - - - No - - - - - - - - - N.A. – no production yet - - - - - - - - 1 # of documents 7 6 0 6 5 0 18 7 1 TABLE 27: CHANGE IN VALUE OF EXPORTS (GRAPE, MEAT, TOTAL, IMAP) Quantity Produced Value Chain and Value Chain Actor Grape Meat All value chains IMAP FEG Grant recipients Processors/ exporters Processors/ exporters FEG Grant recipients Processors/ exporters Yes - - - - - - - 7 No - - - - - - - 2 N.A. – no production yet - - - 2 - - 3 3 # of documents 7 5 0 6 38 22 1 13 THE AGRIBUSINESS PROJECT EVALUATION 125 TABLE 28: CHANGE IN EMPLOYMENT (APRICOT, POTATO, AND HV/OSV) Quantity produced Value chain and value chain actor Apricot Potato HV/OSV FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters Increase - 5 - 1 - - 8 6 - No change 3 - - 3 3 - 10 - 1 # of documents 7 6 0 6 5 0 18 7 1 TABLE 29: CHANGE IN EMPLOYMENT (GRAPE, MEAT, TOTAL, IMAP) Quantity Produced Value Chain and Value Chain Actor Grape All Value Chains IMAP Meat FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters Processors/ exporters Processors/ exporters Increase 3 1 - 11 12 - 2 - No change 3 2 - 20 5 5 5 4 # of documents 7 4 - 38 22 7 13 6 THE AGRIBUSINESS PROJECT EVALUATION 126 TABLE 30: PROSPECTS FOR SUSTAINABLE RESULTS (APRICOT, POTATO, AND HV/OSV) Quantity Produced Value Chain and Value Chain Actor Apricot Potato HV/OSV FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters FEG Grant recipients Processors/ exporters Yes 6 4 - 2 5 - 11 2 - No 3 1 - 2 - - 2 - - Financial constraints 2 1 - 1 - - 2 - - # of documents 7 6 0 6 5 0 18 7 1 TABLE 31: PROSPECTS FOR SUSTAINABLE RESULTS (GRAPE, MEAT, TOTAL, IMAP) Quantity Produced Value Chain and Value Chain Actor Grape Meat All Value Chains IMAP FEG Grant recipients Processors /exporters Service providers Processors /exporters FEG Grant recipients Processors /exporters Processors /exporters Yes 5 3 - 2 1 24 15 1 3 No - - - 2 2 7 1 2 1 Financial constraints - - - 1 - 5 1 - 1 # of documents 7 4 0 4 6 38 22 1 13 THE AGRIBUSINESS PROJECT EVALUATION 127 TABLE 32: SATISFACTION WITH IMPLEMENTATION - APRICOT Dimensions of Implementation Value Chain Actor FEGs Grant Recipients Total Positive Negative Positive Negative Positive Negative Activities Exposure visits - - - - - - Marketing - - - - - - Inputs (seeds/plants) - - - - - - Tools/equipment - 2 - - - 2 Infrastructure quality - 7 - - - 7 Infrastructure design 1 6 - - 1 6 Training - 3 - 1 - 4 Management Timing - 6 - 1 - 7 Sequencing - - - - - - Infrastructure process - 2 - - - 2 Grant process - 1 - 1 - 2 ASF/partner expertise - - - - - - ASF responsiveness - - - - - - Other 1 - 2 - 3 - # of documents 7 6 13 THE AGRIBUSINESS PROJECT EVALUATION 128 TABLE 33: SATISFACTION WITH IMPLEMENTATION – SEED POTATO Dimensions of Implementation Value Chain Actor FEGs Grant Recipients Total Positive Negative Positive Negative Positive Negative Activities Exposure visits - - - - - - Marketing - 2 - - - 2 Inputs (seeds/plants) - 4 - 1 - 5 Tools/equipment 1 3 1 - 2 3 Infrastructure quality - - - 1 - 1 Infrastructure design - - - 1 - 1 Training 3 2 - 4 1 6 Management Timing - 4 - - 1 4 Sequencing - 2 - - - 2 Infrastructure process - - 1 - 1 - Grant process - - 3 1 1 1 ASF/partner expertise - - - - - - ASF responsiveness 1 - - - 1 1 Other 1 3 - 1 1 4 # of documents 6 5 22 THE AGRIBUSINESS PROJECT EVALUATION 129 TABLE 34: SATISFACTION WITH IMPLEMENTATION – HV/OSV Dimensions of Implementation Value Chain Actor FEGs Grant Recipients Total Positive Negative Positive Negative Positive Negative Activities Exposure visits - - - - - - Marketing - 4 - 1 - 5 Inputs (seeds/plants) 8 4 1 1 9 5 Tools/equipment 1 3 - - 1 3 Infrastructure quality - - - - - - Infrastructure design - 8 - 4 - 12 Training 5 6 2 3 6 9 Management Timing - 10 2 3 2 13 Sequencing 1 3 - - 1 3 Infrastructure process - 1 4 2 4 3 Grant process - - 5 1 5 1 ASF/partner expertise - 1 1 - 1 1 ASF responsiveness - 3 1 - 1 3 Other 1 5 - - 1 5 # of documents 18 7 25 THE AGRIBUSINESS PROJECT EVALUATION 130 TABLE 35: SATISFACTION WITH IMPLEMENTATION – GRAPE Dimensions of Implementation Value Chain Actor FEGs Grant Recipients Total Positive Negative Positive Negative Positive Negative Activities Exposure visits - - - - - - Marketing - 3 - 1 - 4 Inputs (seeds/plants) 1 1 - - 1 1 Tools/equipment 1 5 - - 1 5 Infrastructure quality - 3 - - - 3 Infrastructure design 1 5 - - 1 5 Training - 5 - 2 - 7 Management Timing - 4 - 1 - 5 Sequencing - 1 - - - 1 Infrastructure process - - - 1 - 1 Grant process - - 3 1 3 1 ASF/partner expertise 1 - 1 1 2 1 ASF responsiveness - 1 - 1 - 2 Other - 3 - - - 3 # of documents 7 4 11 THE AGRIBUSINESS PROJECT EVALUATION 131 TABLE 36: SATISFACTION WITH IMPLEMENTATION – MEAT Dimensions of Implementation Value Chain Actor Meat Cut Training Feedlot Training AITs Positive Negative Positive Negative Positive Negative Activities Exposure visits - - - - - - Marketing - 1 - - - - Inputs (seeds/plants) - - - - - - Tools/equipment - - - - - 1 Infrastructure quality - - - - - - Infrastructure design - - - - - - Training 4 6 2 - 4 1 Management Timing - - - - - - Sequencing - - - - - - Infrastructure process - - - - - - Grant process - - - - - - ASF/partner expertise - - - - - - ASF responsiveness - - - - - - Other - - - - - - # of documents 6 3 4 THE AGRIBUSINESS PROJECT EVALUATION 132 TABLE 37: SATISFACTION WITH IMPLEMENTATION – ALL VALUE CHAINS Dimensions of Implementation Value Chain Actor FEGs Grant Recipients Total Positive Negative Positive Negative Positive Negative Activities Exposure visits - - - - - - Marketing - 9 - 2 - 11 Inputs (seeds/plants) 9 9 1 2 10 11 Tools/equipment 3 13 - - 3 13 Infrastructure quality - 10 - 1 - 11 Infrastructure design 2 19 - 5 2 24 Training 7 17 2 10 9 27 Management Timing - 24 2 5 2 29 Sequencing 1 6 - - 1 6 Infrastructure process - 3 5 3 5 6 Grant process - 1 11 4 11 5 ASF/partner expertise 1 1 2 1 3 2 ASF responsiveness 1 4 1 1 2 5 Other 2 8 2 4 4 12 # of documents 38 22 60 THE AGRIBUSINESS PROJECT EVALUATION 133 TABLE 38: SATISFACTION WITH IMPLEMENTATION – IMAP Dimensions of Implementation Positive Negative Activities Exposure visits 5 3 Marketing 6 8 Inputs (seeds/plants) 1 - Tools/equipment - - Infrastructure quality - - Infrastructure design - 1 Training 1 2 Management Timing - - Sequencing - - Infrastructure process - - Grant process - 1 ASF/partner expertise - 1 ASF responsiveness - - Other - 1 # of documents 13 THE AGRIBUSINESS PROJECT EVALUATION 134 Annex 10: Conflict of Interest Disclosures THE AGRIBUSINESS PROJECT EVALUATION 135 THE AGRIBUSINESS PROJECT EVALUATION 136 THE AGRIBUSINESS PROJECT EVALUATION 137 THE AGRIBUSINESS PROJECT EVALUATION 138 THE AGRIBUSINESS PROJECT EVALUATION 139 THE AGRIBUSINESS PROJECT EVALUATION 140 THE AGRIBUSINESS PROJECT EVALUATION 141 THE AGRIBUSINESS PROJECT EVALUATION 142 THE AGRIBUSINESS PROJECT EVALUATION 143 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523