1 November 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Salmaan Farooq, Shabbir Hussain, Douglas Krieger, and Ann Schwartz under Management Systems International’s (MSI) Monitoring and Evaluation Project (MEP). Improving the Competitiveness of Small and Medium Enterprises in Pakistan: Final Evaluation of the Firms Project ACKNOWLEDGMENTS Evaluation is a collaborative effort that requires coordination among a number of people with different roles in the work. The Firms Project evaluation went smoothly because everyone involved cooperated exceptionally well. We would like to explicitly recognize the contribution of a number of groups and individuals to the evaluation. First and foremost, we wish to thank the many beneficiaries, Pakistani government officials, and stakeholders who generously gave us their time and patiently answered our endless questions. Without their participation the evaluation would have been impossible. Firms Project staff also deserve special recognition. Even as the project was closing, project personnel supported the evaluation exceptionally well; arranging meetings, joining the team in the field to make introductions and provide context, and responding quickly and competently to many last-minute requests for information and clarification. The logistics and other support provided by MEP staff were also invaluable. We have the MEP team to thank for effectively managing a demanding travel schedule, providing a sounding board for our thoughts, and coordinating the many moving pieces to make the experience successful and enjoyable. Finally, we wish to thank USAID, and particularly the Economic Growth and Agriculture Office and the Performance Management Unit, for their invaluable guidance and assistance. Thank you. Douglas Krieger Ann Schwartz Shabbir Hussain Salmaan Farooq IMPROVING THE COMPETITIVENESS OF SMALL AND MEDIUM ENTERPRISES IN PAKISTAN FINAL EVALUATION OF THE FIRMS PROJECT November 28, 2014 Contracted under Order No. AID-391-C-13-00005 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Project Summary ................................................................................................................................................................ v Executive Summary ...........................................................................................................................................................1 Evaluation Purpose and Questions ................................................................................................................................5 Project Background ...........................................................................................................................................................7 Evaluation Methods and Limitations ............................................................................................................................13 Findings ...............................................................................................................................................................................18 Conclusions .......................................................................................................................................................................51 Recommendations ...........................................................................................................................................................55 Lessons Learned ...............................................................................................................................................................58 Annexes .............................................................................................................................................................................60 Annex I: Evaluation Statement of Work ...........................................................................................................60 Annex 2: Data Collection Instruments ........................................................................................................... 101 Annex 3: List of Documents Reviewed .......................................................................................................... 134 Annex 4: Getting to Answers Table ............................................................................................................... 141 Annex 5: Sampling Protocols ............................................................................................................................ 143 Annex 6: Qualitative Data Collection by Value Chain ............................................................................... 144 Annex 7: Comparison of USAID and Firms Project Results Frameworks ............................................ 145 Annex 8: Project Expenditures by Component ........................................................................................... 147 Lists of Tables and Figures Table 1: Project Summary................................................................................................................................................ v Table 2: Data Collection Methods and Sources ....................................................................................................... 14 Table 3: Interview and Focus Group Details for VCD Activities ........................................................................ 15 Table 4: Evidence of Relevance of Project Components and Strategies to Host Country Government Priorities............................................................................................................................................................................. 19 Table 5: Prospects for Sustainable Results in Project-Supported Enterprises by Value Chain ..................... 53 Table 6: Sampling Protocols for Interviews and Focus Groups ........................................................................ 143 Table 7: Qualitative Data Collection Methods and Sources by Value Chain ................................................. 144 Table 8: USAID and Firms Project Results Frameworks, 2009 ......................................................................... 145 Table 9: USAID and Firms Project Results Frameworks, 2013 ......................................................................... 146 Table 10: Project Expenditure and Results by Value Chain ............................................................................... 147 Table 11: Expenditure per Unit of Result by Value Chain .................................................................................. 147 Figure 1: Location of Project VCD Activities ............................................................................................................ vi Figure 2: Firms Project Results Framework .............................................................................................................. 18 ii Figure 3: Evolution of Firms Project Objectives ....................................................................................................... 23 Figure 4: Project-Reported VCD Results .................................................................................................................. 25 Figure 5: Quantitative Results for Value Chain Components ............................................................................... 27 Figure 6: Trends in Value of Sales – Project-Supported Mango Processors ..................................................... 30 Figure 7: External Factors Affecting Results in the Date Value Chain ................................................................ 33 Figure 8: Quantitative Results for BEE ....................................................................................................................... 37 Figure 9: Factors and Conditions that Impede or Contribute to Achieving BEE Outcomes ........................ 45 Figure 10: Training by Value Chain, Topic, and Sex of Trainee ............................................................................ 50 iii ACRONYMS ADB Asian Development Bank AO Assistance Objective AOR Agreement Officer’s Representative AusAID Australian Agency for International Development BEE Business Enabling Environment CAD Computer Aided Design CMU Change Management Unit CNC Computer Numerical Control COR Contracting Officer’s Representative C-TPAT Customs-Trade Partnership Against Terrorism DO Development Objective EGA Economic Growth and Agriculture EPF Empower Pakistan: Firms (the original name of the project) ERP Enterprise Resource Planning EU European Union FTE Full-Time Equivalent GDP Gross Domestic Product GlobalG.A.P. Global Good Agricultural Practices GoB Government of Balochistan GoP Government of Pakistan GoKP Government of Khyber Pakhtunkhwa GSP Generalized System of Preferences HACCP Hazard and Critical Control Points KP Khyber Pakhtunkhwa KP-BOIT Khyber Pakhtunkhwa Board of Investment and Trade KP-CCI Khyber Pakhtunkhwa Chamber of Commerce and Industry KP-PPP Khyber Pakhtunkhwa Public Private Partnership KRA Key Results Area IDS Integrated Development Strategy IP Implementing Partner IR Intermediate Result ISO International Standards Organization IUA Infrastructure Upgradation Agreement LOP Life of Project M&E Monitoring and Evaluation MGA Mango GlobalG.A.P. Agreement MOU Memorandum of Understanding MSF Mission Strategic Framework OAPA Office of Afghan and Pakistan Affairs OIG Office of the Inspector General PaRRSA Provincial Reconstruction, Rehabilitation & Settlement Authority PIFMS Pilot Information and Financial Mobile Solutions PPP Public Private Partnership PRSP Poverty Reduction Strategy Paper Rs. Pakistani Rupees SME Small and Medium Enterprise iv SMEDA Small and Medium Enterprise Development Authority SOW Statement of Work UAE United Arab Emirates UNIDO United Nations Industrial Development Organization U.S. United States USAID United States Agency for International Development USD United States Dollar USG United States Government VCD Value Chain Development WRAP Worldwide Responsible Accredited Production v PROJECT SUMMARY Table 1 summarizes basic information about the Firms Project. TABLE 1: PROJECT SUMMARY Title / Field Project/Activity Information Contract/agreement numbers USAID GBTI II IQC TO # EEM-I-00-07-00008-00 Task Order No. 7 Contracting/Agreement Officer’s Representative (COR/AOR) Kanwal Bokharey, Economic Growth Advisor, Economic Growth and Agriculture (EGA) Office Start date May 07, 2009 Completion date December 31, 2014 Location Nationwide (geographical location for each activity is provided in Section F of this document) Implementing partner(s) Chemonics International, Inc. USAID/Pakistan Mission Strategic Framework objectives addressed DO 2: Improved Economic Status of Focus Populations IR 2.1: Improved Economic Performance of Focus Enterprises IR 2.2: Improved Business Enabling Environment Budget USD 92,255,031 vi The map in Figure 1 illustrates the locations of the Firms Project’s value chain development (VCD) activities the evaluation team investigated. Business enabling environment (BEE) activities did not have a specific location. The agricultural market reform and institutional strengthening for the Small and Medium Enterprise Development Authority (SMEDA) were national-level interventions. SMEDA is a federal-level authority. Agricultural market reform is province-specific, but the Firms Project worked on reform in all provinces. All other BEE activities involved support to various arms of the Government of the Khyber Pakhtunkhwa (KP) Province (GoKP). FIGURE 1: LOCATION OF PROJECT VCD ACTIVITIES 1 EXECUTIVE SUMMARY The Firms Project aimed to improve the competitiveness of Pakistan Small and Medium Enterprises (SMEs) in value chains relevant to contributing to expanding economic opportunities and jobs in districts vulnerable to extremist influences. It pursued these objectives through two components. The value chain development (VCD) component focused on building the capacities of SMEs, while the business enabling environment (BEE) component worked to establish a legal and regulatory environment conducive to SME growth. EVALUATION PURPOSE AND QUESTIONS This report is primarily a final performance evaluation with the objective of gauging whether the Firms Project achieved the results and targets outlined in its results framework. However, USAID/Pakistan’s Economic Growth and Agriculture (EGA) Office has designed a new activity to support Pakistani SMEs and the evaluation will also identify lessons to guide implementation of the follow-on activity. The primary audiences for the evaluation are USAID/Pakistan’s EGA Office, the Office of Afghanistan and Pakistan Affairs (OAPA), other key entities in Washington, relevant Government of Pakistan (GoP) departments, the private sector, and key institutions responsible for private sector led growth. The evaluation statement of work (SOW)1 poses four evaluation questions: 1. Relevance/appropriateness: To what extent was the project relevant as designed, and how did the implementing partner (IP) adjust to keep it relevant in the face of changing priorities? 2. Effectiveness: To what extent did the project achieve its stated objectives, and what lessons can be learned about the effectiveness of the business models / development approaches / implementation strategies adapted/adopted/developed by the project? 3. Sustainability: To what extent are the project’s outcomes as defined in the project results framework likely to continue once project assistance is formally concluded? 4. Gender: In what way were gender issues incorporated and/or addressed by the project interventions and what lessons can be incorporated in future programs? PROJECT BACKGROUND Economic growth is essential if Pakistan is to continue to develop and productively employ a growing population that will add 49 million new people to the labor force by 2025, with the largest growth (55 percent) in the 25-49 year-old age category – a key demographic in USAID/Pakistan’s strategy to combat extremist influences.2 The Firms Project aims to: “assist the Government of Pakistan to improve its 1 Annex I contains the evaluation SOW. 2 Planning Commission, Government of Pakistan, Pakistan: Framework for Economic Growth, May 2011. Accessed at: http://www.theigc.org/wp-content/uploads/2014/11/Pakistan-Framework-for-Economic-Growth-2011.pdf 2 service delivery and develop dynamic, internationally competitive private sector SMEs to accelerate sales, investment, and job growth to undercut the foundations of extremism”. DATA COLLECTION METHODS AND LIMITATIONS The mixed-methods evaluation relied on quantitative data taken from the project’s monitoring and evaluation system, primary qualitative data collected through semi-structured interviews with project beneficiaries and stakeholders, site visits to observe project interventions, and focus group discussions with project beneficiaries. The evaluation also used secondary qualitative data collected from project reports and other documents. The evaluators randomly selected VCD beneficiaries for semi-structured interviews and focus group discussions. The quantitative data provided numeric measures of effectiveness relative to the indicators in the project’s Activity Monitoring and Evaluation (M&E) Plan3 and required no analysis. The primary qualitative data provide rich evidence of how and why project interventions contributed, or failed to contribute, to results. Evaluators used content analysis to reduce the qualitative data; defining themes relevant to answering the evaluation questions and coding qualitative data under the themes. This approach allowed the evaluators to determine the number and percentage of respondents of different types who mentioned a particular theme and thus provided a measure of the validity or strength of the qualitative evidence. The number of discrete activities the evaluation examined and their geographic distribution limited the quantity of data the evaluation team could collect on each project component. The evidence is thus very thin, particularly for the BEE component. The evaluation addresses this limitation by triangulating findings across sources and methods whenever possible. The evaluators’ preliminary assessment of the project’s M&E data found it to be relatively comprehensive.4 Therefore, the evaluation relied on project￾collected data, validated when possible during interviews with project beneficiaries and stakeholders. However, the data tend to overstate some results, especially the value of sales and exports for export￾oriented value chains. FINDINGS AND CONCLUSIONS With respect to relevance, the evaluation concluded that the Firms Project’s results framework was well aligned with USAID’s Development Objective (DO) 2 results framework and thus relevant to USAID priorities. The broad areas on which the project focused (i.e., enhancing SME competitiveness and improving the business enabling environment) as well as specific implementation approaches and strategies also aligned well with host country government priorities documented in planning and strategy 3 As USAID’s guidance changed during the course of the project, the Firms Project referred to its M&E planning documents as Award Monitoring Plans, Performance Management Plans, and Activity M&E Plans. The remainder of this report refers to these documents generically by the correct name of Activity M&E Plans. 4 The OIG Audit (Office of the Inspector General, Audit of USAID Pakistan’s Firms Project. Accessed at: https://oig.usaid.gov/sites/default/files/audit-reports/g-391-12-001-p.pdf ) raised several concerns about how the project reported results. However, the project retroactively revised reported results in accordance with the audit findings and the evaluation used the revised figures. 3 documents. The highly consultative implementation strategy, particularly for BEE activities, ensured relevance in design and implementation for the federal and provincial government “clients” of BEE activities. In fact, seven of the nine individuals who spoke about the BEE process made a point of mentioning their appreciation of the highly consultative process. Furthermore, the project responded quickly and appropriately to major changes in design and implementation imposed by changes in USAID strategies and host country governments’ priorities. With respect to effectiveness, the evaluation concluded that the project probably met or exceeded all of its life of project (LOP) targets for key outcomes. Questions about the data on value of sales and exports, primarily in the fresh mango value chain, are responsible for the “probably” qualifier. Interventions to promote productivity-enhancing technologies and practices among farm enterprises were effective across the board in increasing quantities produced and improving quality. All six focus groups with project-supported farm enterprises (two with mango farmers, two with potato farmers, and two with peach farmers) reported that project-promoted technologies and practices had increased production, improved quality, and reduced waste. The project’s focus on upgrading infrastructure and technology, improving human capacities, and establishing market linkages was relevant in all project￾supported value chains the evaluation examined and, with the exception of date processing, effective in increasing competitiveness as measured by the value of sales. In export-oriented value chains (i.e., garments, fresh mango, and dates), certifications were particularly effective in opening doors to a wider range of buyers. The analysis-driven and consultative approach helped enhance effectiveness by addressing relevant constraints with appropriate interventions. BEE interventions were largely effective in building a firm foundation for reform and strengthening institutions. Beneficiaries praised the quality and timeliness of the project’s work. In fact, the eight BEE clients and stakeholders the evaluation team interviewed all claimed that the quality of the project’s work was “excellent.” It is difficult to assess the potential sustainability of project results. Sustainability of BEE results depends on federal and provincial governments building on the foundation provided by the Firms Project. This, in turn, depends on continuing government commitment. The project’s consultative approach to BEE activities will enhance prospects for sustainable results by ensuring relevance, fostering ownership, and creating champions to carry reform forward after project support ends. Some BEE interventions have progressed more than others towards implementation, largely because of provincial government commitment. The best prospects for sustainable results include public private partnership (PPP) in KP where the institutional framework was already in place for the PPP; the KP tourism policy which is expected to be launched in November or December 2014 (after the evaluation team wrote the report); and the Change Management Unit (CMU) and policy reforms in KP which will be implemented under the Integrated Development Strategy. Prospects for sustainability of the rest of the BEE activities are less certain and will depend on emerging host country government priorities and commitment. Sustainability of VCD activities depends on the willingness and ability of the project-supported SMEs to maintain or replace infrastructure, refresh training, renew certifications, and maintain market linkages. The evidence is mixed but prospects for sustainability of the results the project achieved are good for all project-supported value chains. Prospects are weakest in value chains in which the project got a late start (i.e., date processing) and where market actors still rely on local markets (i.e., dates and peaches). The project design included ambitious gender targets and plans for women-focused activities. It made good progress on women-focused activities in its year one work plan before USAID eliminated women￾focused objectives and activities from the project’s SOW in order to focus activities more directly on countering extremist influences and rebuilding after the 2010 floods. These changes in strategy focused project activities in regions and in value chains where it was more difficult to directly engage women. 4 RECOMMENDATIONS The evaluation SOW focused more on lessons learned than on recommendations. Nevertheless, a few recommendations emerged from the evaluation. Two relate to issues of attributing measured results, in terms of indicator values, to project interventions. To do so rigorously requires that USAID commit to designing and implementing an impact evaluation. Outside of the context of an impact evaluation, however, USAID should ensure, when applicable, that projects are reporting against indicators in a manner consistent with indicator definitions when the indicators allude to attribution, e.g., “as a result of USG assistance”. The remainder of the recommendations relate to areas where additional assistance may be required to enhance prospects for sustaining the results the Firms Project has achieved. Across all value chains, USAID should widely disseminate project research, analyses, and reports and promote project successes among donors and indigenous institutions with the aim of attracting the additional assistance required to enhance prospects for sustainability. The project’s exit strategy for VCD activities provides a relatively well-reasoned roadmap to the prospects for sustainable results and the actions necessary to enhance those prospects. Sustainability of BEE activities requires host country governments’ commitment to move draft reforms and regulatory frameworks to enacted legislation and regulations. If USAID can identify and nurture champions to shepherd reform movements forward, it will improve the chances that reforms will be enacted. LESSONS LEARNED Lessons relative to the effectiveness of business models, implementation approaches, and strategies include:  The consultative approach contributed to the effectiveness of both BEE and VCD activities by ensuring relevance and promoting ownership. Applied to designing the specifics of implementation, the consultative process ensured that activities addressed relevant needs in an appropriate fashion.  A focus on accessing foreign markets was an effective VCD strategy when supported SMEs were already exporting. In these cases, certifications such as the Worldwide Responsible Accredited Production (WRAP), Customs-Trade Partnership Against Terrorism (C-TPAT), Hazard and Critical Control Point (HACCP), and GlobalG.A.P. were particularly effective. Supported SMEs with little export experience may need more assistance learning how to navigate the intricacies of exporting.  Take care when engaging the private sector as market intermediaries between producers and markets to ensure that they do not perpetuate existing channels that often lock farmers into disadvantageous arrangements.  Anticipate and plan for procurement bottlenecks that can delay implementation and respond quickly to make alternative arrangements when delays threaten results. 5 EVALUATION PURPOSE AND QUESTIONS The evaluation SOW describes the evaluation as primarily a final performance evaluation with the objective of gauging whether the Firms Project achieved the results and targets outlined in its results framework. However, USAID/Pakistan’s Economic Growth and Agriculture (EGA) Office has designed a new activity to support Pakistani SMEs. If implemented under the current Mission Results Framework (MSF), the new activity will likely focus on the same key intermediate results the Firms Project addressed, i.e., improving the economic performance of enterprises on which the project focused and improving the business enabling environment. Therefore, the evaluation will also assess the effectiveness of the implementation approaches and strategies and business models adopted by the Firms Project with the aim of identifying lessons to guide implementation of the follow-on activity. The primary audiences for the evaluation are USAID/Pakistan’s EGA Office, OAPA, other key entities in Washington, relevant host country government departments, the private sector, and key institutions responsible for private sector-led growth. EVALUATION QUESTIONS The SOW poses four evaluation questions, each with an explanation. The questions focus broadly on the relevance and appropriateness of the project design to USAID and Pakistan government (federal, or provincial as applicable) objectives and priorities, the project’s effectiveness relative to stated objectives, the likely sustainability of project results, and the extent to which the project’s design and implementation addressed gender issues. The specific questions are: 1. Relevance/Appropriateness: To what extent was the project relevant as designed, and how did the implementing partner (IP) adjust to keep it relevant in the face of changing priorities? Explanation: The evaluation will assess the extent to which the project design responded to needs identified by the Pakistani government and contributed to USAID's MSF Development Objective (DO) 2 results framework. Relevant stakeholders include BEE, the federal and provincial government departments/ministries (representatives of the Ministries of Agriculture, Livestock, Mining, Government of Khyber Pakhtunkhwa (GoKP), Government of Balochistan, and the Planning & Development Department of GoKP). In addition, the evaluation should consider how the IP made decisions about what interventions to pursue and whether there are lessons learned which may be applied to the implementation of future projects. 2. Effectiveness: To what extent did the project achieve its stated objectives, and what lessons can be learned about the effectiveness of the business models / development approaches / implementation strategies adapted/adopted/developed by the project? Explanation: The evaluation should assess whether the project contributed to strengthening the level of competitiveness of the groups on which the project focused; identify key unintended results the project may have produced; identify key parameters/determining factors that may have influenced project outcomes; and identify lessons learned regarding the ability of a project of this nature to ensure policy reform recommendations are approved. Key stakeholders include 6 agriculture markets, livestock, mining, support for the Planning & Development Department of GoKP for BEE interventions; and a diverse sample of partner SMEs/relevant sector associations/chambers of commerce in the mango, agriculture implements, peach, knitted garments, hotels, and mobile information sectors under the VCD component. 3. Sustainability: To what extent are the project’s outcomes as defined in the project results framework likely to continue once project assistance is formally concluded? Explanation: The evaluation should assess whether a realistic and practical exit strategy is in place, and to what extent such a strategy has already been initiated, for the handover of continuing activities to relevant stakeholders. This includes federal and provincial government policymakers and a diverse sample of SME beneficiaries in the mango, agriculture implements, peach, knitted garments, hotels, and mobile information sectors, and reform work in agriculture markets and the livestock and mining sectors. 4. Gender: In what way were gender issues incorporated and/or addressed by the project interventions and what lessons can be incorporated in future programs? Explanation: The evaluation should assess the extent to which gender was addressed in the design and implementation of the project, particularly with regard to support for women’s chambers of commerce, and what lessons can be drawn for future projects. 7 PROJECT BACKGROUND Economic growth is essential if Pakistan is to continue to develop and productively employ a growing population that will add 49 million new people to the labor force by 2025, with the largest growth (55 percent) in the 25-49 year-old age category.5 Its inability to compete in global markets is a major barrier to Pakistan’s economic growth and development. In 2009, the World Economic Forum’s Global Competitiveness Index ranked Pakistan 101st out of 133 economies in terms of competitiveness.6 Promoting resilience in vulnerable areas is a key priority for the U.S. Government and the GoP. The MSF for USAID/Pakistan links resilience to economic opportunity, which depends on business growth and job creation.7 Pakistan’s SMEs account for about 30 percent of national gross domestic product (GDP), 25 percent of manufacturing exports, and 35 percent of manufacturing value-added.8 Furthermore, about 90 percent of Pakistan’s private sector businesses are SMEs and they employ an estimated 78 percent of the non-agricultural labor force.9 The evaluation SOW cites these figures to support the Firms Project's rational for focusing on SMEs: “The potential for this sector to contribute to the economic development objectives of Pakistan, including generating employment, increasing incomes, and reducing poverty is the justification for a strategy of support to unleash the sector’s potential.” THEORY OF CHANGE The Firms Project contributes to DO 2: “Improved economic status of focus populations” in the MSF. The development hypothesis embedded in the MSF suggests that increasing access to finance, improving skill development and job placement, and increasing the use of modern technologies and management practices will improve the economic performance of enterprises on which the project focuses. Furthermore, improving host governments’ abilities to develop and implement reform of policies, laws, and regulations and strengthening private sector and civil society engagement in policy making will improve the business enabling environment. Stronger enterprises, supported by a business enabling environment conducive to investment and growth, will contribute to the “improved economic status” of Pakistanis in the selected sectors.10 The Firms Project contributes to USAID/Pakistan’s development hypothesis with activities that focus on enhancing the skills of managers and the labor force of SMEs and increasing their use of productivity- 5 Planning Commission, Government of Pakistan, Pakistan: Framework for Economic Growth, May 2011. Accessed at: http://www.theigc.org/wp-content/uploads/2014/11/Pakistan-Framework-for-Economic-Growth-2011.pdf 6 World Economic Forum, The Global Competitiveness Report 2009-2010. Accessed at: http://www3.weforum.org/docs/WEF_GlobalCompetitivenessReport_2009-10.pdf 7 USAID/Pakistan, Mission Strategic Framework, Pakistan Fiscal Year 2013 – 2017 8 State Bank of Pakistan. Prudential Regulations for Small & Medium Enterprise Financing. Infrastructure, Housing & SME Finance Department. No date. 9 Shahid Saleem. SMEDA SME Policy Paper 2007 – A Critical Review, February 2008. Accessed at: http://mpra.ub.uni￾muenchen.de/7329/1/mpra_paper_7329.pdf 10 USAID/Pakistan, Mission Strategic Framework, Pakistan Fiscal Year 2013 – 2017 8 enhancing technologies and practices. With respect to the business enabling environment, the project works with governments at the federal and provincial levels to support reform. INTENDED RESULTS The Firms Project objective is to: “…assist the Government of Pakistan to improve its service delivery and develop dynamic, internationally competitive private sector SMEs to accelerate sales, investment, and job growth to undercut the foundations of extremism. Socioeconomic stabilization of vulnerable areas is in the government’s strategic interest; it is also an urgent priority for the U.S. Government. The primary prerequisite for this stabilization is a robust and competitive private sector resulting from a market-driven economic environment and enabling policies.” It pursues these objectives through activities that contribute to two Key Results Areas (KRAs): enhanced competitiveness in project-assisted SMEs in selected value chains, and improved business enabling environment at the federal, provincial, and district levels.11 THE FIRMS PROJECT COMPONENTS The Firms Project involved two main components: value chain development and business enabling environment. The VCD Component The VCD component focused on enhancing the competitiveness of SMEs in selected sectors and has supported 13 value chains in the agricultural, textiles, tourism, manufacturing, and mining sectors. These included agricultural implements, dates, fruit and vegetable pulping, Islampur weaving,12 knitted garments, hotels and fish farms, dried mango, fresh mango, marble and granite, peach, and potato. As applicable in each value chain, project activities focused on upgrading infrastructure, building human capacity, and establishing market linkages. Prior to designing a specific implementation approach in any value chain (with the exception of hotels and fish farms in Swat), the project conducted a thorough analysis to map the value chain and identify constraints and opportunities. The following very brief profiles of each value chain summarize salient features of the project’s approach to working with each of the nine value chains covered by the evaluation.13 11 USAID Firms Project Activity Monitoring & Evaluation Plan FY2009–FY2014. 12 Islampur is a village in the Swat District of the KP Province. Weavers in Swat are well-known for their handmade woolen shawls. 13 The Evaluation Methods and Limitations section provides the rationale for and details of selecting specific value chains for the evaluation. 9 Fresh Mango Value Chain: 14 The mango growing regions of Pakistan are far from the primary urban markets and export centers. Mangos are quite perishable and the lack of capacity in the growing regions for proper post-harvest handling, cold chain storage, and transportation significantly reduces the quality and shelf-life of mangos before they reach markets or export centers. Because it takes time to ship mangos to foreign markets, preserving shelf life is particularly important for export market access. Partly as a result of these constraints, Pakistan’s mango exports have typically focused on relatively low-quality regional or diaspora markets in the European Union (EU). Project interventions aim to increase mango yields, improve quality, and implement better post-harvest handling practices to build the sector’s capacity to reach new lucrative high-end export markets. The Firms Project worked at three different levels in the fresh mango value chain: Infrastructure Upgradation Agreement (IUA) farm enterprises, Mango Global Good Agricultural Practices (GlobalG.A.P.) Agreement (MGA) enterprises and small mango farm enterprises. Processing capacity within the mango producing regions aimed at increasing shelf-life is critical to reaching high-end export markets. Through cost sharing agreements (IUAs) the project established 14 local processing SMEs (IUA farm enterprises) in the mango producing districts of Multan in Punjab Province and Hyderabad in Sindh Province with processing, grading, packing, and cold storage infrastructure. The project supported these investments with relevant training, assistance obtaining HACCP certification required in many export markets, and exhibitions and other activities to establish market linkages. The project also signed agreements (MGAs) with 26 mango farmers to share the costs of making the investments necessary to obtain GlobalG.A.P. certifications required to access high-end export markets directly or indirectly. The project supported the training necessary to implement GlobalG.A.P. requirements. With IUA, MGA, and small mango farm enterprises, the project also provided tools (on a cost-share basis) and supported training in production, harvesting, and post-harvest handling to improve yields and product quality. Date Value Chain: 15 Pakistan is the fifth largest producer of dates in the world with a share of 6-8 percent of the world market. At the farm level, however, poor farming and post-harvest technologies and practices limit yields and compromise quality. At the processing level, limited awareness of quality control and food safety management and poor market linkages limit access to increasingly demanding export markets. Pakistan’s date farmers typically dry their dates on mats on the ground where they are subject to contamination with dirt and damage from rain. At the farm level, the Firms Project supported 45 date farmers by sharing the cost of production and post-harvest handling tools and equipment (e.g., cutting tools, sprayers, drying tables, crates, solar dryers) designed to reduce loss during harvest, reduce drying time (and the associated risk), and improve quality. The project also trained farmers in production and post-harvest technologies and practices. At the processor level, the project provided, on a cost-share 14 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Fresh Mango Program, June 2, 2014; USAID Firms Project, Value Chain Impact Assessment for Mango Sector, March 2014) and semi-structured interviews with Firms Project staff, mango sector stakeholders, and mango processors. 15 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Dates Program, June 2, 2014; USAID Firms Project Sales and Labor Multiplier Study for Dates Value Chain, November 2013) and semi-structured interviews with Firms Project staff, date sector stakeholders, and date processors. 10 basis, cold storage facilities to three processors, upgraded their processing equipment to meet HACCP certification requirements, trained managers and workers in food safety management and other topics, and introduced processors to new buyers at international exhibitions. Knitted Garments Value Chain: 16 The textile and garment sector is very important to Pakistan’s economy. It accounts for about 8.5 percent of GDP, 50 percent of the total value of exports, and employs an estimated 38 percent of the manufacturing workforce. The sector is comprised of about 74 percent SMEs that largely do not employ efficient production technologies, are unable to compete with countries that can produce at lower cost, and are unable to comply with the requirements of increasingly demanding export markets. To address these constraints and improve garment manufacturers' competitiveness and access to export markets, the Firms Project assisted 19 knitted garment manufacturing SMEs in Karachi by upgrading production equipment (machines and design equipment) on a cost-share basis, training managers and workers, helping firms meet requirements for compliance certification (e.g., WRAP and C-TPAT), and linking manufacturers to international buyers through exhibitions. Agricultural Implements Value Chain: 17 Pakistan’s more than 500 agricultural implements manufacturers serve a large local market of about 200 million U.S. Dollars (USD) annually and a smaller export market (USD 7 million annually) for low-end implements. Limited technical and management capacity, obsolete technology, poor quality control, low skill level of labor, and inefficient production processes limit the sector’s growth. To improve the competitiveness of the sector in local and export markets, the Firms Project helped 18 manufacturers upgrade equipment and modernize production processes. The project helped the manufacturers adopt assembly line production methods and trained managers and workers in planning, lean production, quality control, material selection, and other topics to improve production efficiency. On a cost-share basis, the project also helped manufacturers upgrade obsolete technology and install inventory and production management software. The project also helped manufacturers obtain International Standards Organization (ISO) 9000 certification and linked them (although market linkages was not a major focus) to potential foreign buyers. 16 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Knitted Garments Program, June 2, 2014; USAID Firms Project Lean Manufacturing and Quality System Training Final Report, June 2014) and semi-structured interviews with Firms Project staff and garment manufacturers. 17 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Agricultural Implements Manufacturing Sector Assistance Program, June 2014; USAID Firms Project, Agricultural Implements Sector Development Program Potential Economic Benefits of Distribution of Welding Plants to Agricultural Implements Sector, December 2013; USAID Firms Project, Agricultural Implements Cost of Production Study, January 2013) and semi-structured interviews with Firms Project staff and implement manufacturers. 11 Islampur Weavers Value Chain: 18 The shawls of the Islampur weavers are well known in KP Province. The weavers cluster in Islampur has approximately 2,000 to 2,500 weaving units producing shawls of different weights and quality. The weavers use largely outdated technologies that contribute to waste, have limited access to markets outside the local area where prices are low compared to urban markets, sell their products largely through agents, and are not market oriented since they have limited access to retail markets. The Firms Project sought to connect 44 weavers to retail markets by connecting them to buyers outside the Swat District. Direct connections with high-end retail buyers will also sensitize weavers to the colors and designs demanded by consumers. The Firms Project facilitated market linkages by helping to create a product catalogue and facilitating weavers’ participation in trade shows. The project also trained weavers in production and marketing techniques. Hotels and Fish Farms in Swat: 19 The Swat District is a well-known destination for domestic tourists in Pakistan. It is also well known for producing trout. The conflict that ended in 2009 and the floods in 2010 destroyed many hotels and fish farms in the Swat District. Rebuilding these businesses will contribute to local economic recovery. The Firms Project managed a grant program that provided cash and in-kind assistance to rebuild 239 destroyed hotels and 22 fish farms in the Swat District. It also helped develop a media campaign and a website to promote tourism in the valley. Peach Value Chain: 20 The Swat District produces 44 percent of Pakistan’s peach crop and contributes substantially to the local economy. However, a lack of trained labor, limited access to proper tools, and poor farm management and post-harvest practices inhibit productivity growth in the sector and contribute to high levels of waste. The Firms Project organized 1,350 peach farmers into clusters with the cluster leader acting as the point of contact between members and buyers, input suppliers, and financial service providers. The cluster model allows farmers to enjoy economies of scale in input and output markets. Project interventions also trained farmers in farm management, production, and post-harvest practices and provided tools required to implement the practices. 18 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Islampur Weaving Sector, June 2012) and semi-structured interviews with Firms Project staff and weavers. 19 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Malakand Small and Medium Enterprise (SME) Recovery Assistance Program, June 2012; USAID Firms Project Impact Assessment of the Malakand SME Recovery Assistance Program, June 2013) and semi-structured interviews with Firms Project staff and hotel and fish farm owners. 20 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Peach Program, June 2012; USAID Firms Project Monitoring Peach & Potato Trainings on Pre-Harvest Best Management Practices, December 2012) and semi-structured interviews with Firms Project staff. 12 Potato Value Chain: 21 Potato farmers in the Swat District can produce potatoes when other potato growing regions of the country cannot. This creates an opportunity for Swat farmers to supply local potato chip manufacturers when they cannot source potatoes elsewhere in Pakistan. To capitalize on this opportunity, however, farmers must adopt new production practices and technologies, plant appropriate varieties, improve quality, and free themselves from financial obligations (i.e., in-kind repayment of production loans) to input suppliers and agents. As with the peach sector, the project formed clusters of potato farmers to achieve economies of scale in input and output markets. The project also trained farmers in productivity-enhancing production and post-harvest practices and provided the tools necessary to apply the practices. After a disappointing attempt to link farmers directly to a large chip manufacturer, PepsiCo, the project connected potato clusters to a market intermediary, SE Trading, in the second year. SE Trading provides inputs and technical assistance in advance of planting and takes in-kind payment after harvest. The BEE Component The project’s BEE component focused on policy, regulatory, and institutional reform and institutional strengthening to support improvements in the business enabling environment. The BEE component included two national-level interventions: agricultural market reform and institutional strengthening of the Small and Medium Enterprise Development Authority (SMEDA). All other BEE activities the evaluation team examined occurred at the specific request of the GoKP and applied only to the KP Province. The findings for question 1 describe the BEE component with the exception of the Pilot Information and Financial Mobil Solutions (PIFMS). Pilot Information and Financial Mobile Solutions: 22 Farmers in remote regions have limited access to financial services and timely and relevant information about weather, production practices, and market prices that could help improve production and productivity. The recent conflict and floods in the Swat District have left farmers there particularly far removed from these services and information. Telenor, a local mobile phone service provider, has been piloting systems to provide financial services via mobile phones in Pakistan. It signed a tripartite partnership with USAID (through the Firms Project) and GoKP to bundle financial services with information relevant to farmers in the Swat District. The service is available to project-assisted peach and potato farmers and fish farm operators. 21 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Potato Program, June 2012; USAID Firms Project Monitoring Peach & Potato Trainings on Pre-Harvest Best Management Practices, December 2012) and semi-structured interviews with Firms Project staff. 22 Compiled from review of project documents (i.e., Fact Sheet: USAID Firms Project Pilot for Information and Financial Mobile Solutions, June 2012) and semi-structured interviews with Firms Project staff, GoKP officials, and Telenor personnel. 13 EVALUATION METHODS AND LIMITATIONS The evaluation used a mixed methods approach relying on quantitative and qualitative data collected through semi-structured interviews, focus group discussions, site visits, document review, and review and analysis of project-reported monitoring and evaluation data. DATA COLLECTION AND ANALYSIS The evaluation relied largely on primary qualitative data drawn from semi-structured interviews with project beneficiaries (for VCD activities), federal and provincial government “clients” (for BEE activities), and other stakeholders and participants in both components.23 It triangulated the primary qualitative data with secondary data (quantitative and qualitative) obtained from project and other documents, and quantitative data collected from the project’s M&E system. Annex 2 contains the data collection instruments, and Annex 3 lists the documents the team reviewed. The Getting to Answers table (Annex 4) from the evaluation Statement of Work summarizes the evaluation team’s plan for using these various data sources to answer the evaluation questions. Table 2 summarizes the data collection methods and sources the evaluators used to answer each evaluation question. 23 In this context, “participants” refers to individuals or organizations that supported the Firms Project. For example, a university that worked with the project to develop and deliver training or consultants (individuals and firms) the project hired to implement activities. 14 TABLE 2: DATA COLLECTION METHODS AND SOURCES Evaluation Question Data Collection Methods and Type Data Sources Relevance Document review (qualitative) USAID’s Mission Strategic Framework Firms Project planning and implementation documents Federal and provincial government planning and strategy documents Semi-structured interviews (qualitative) Firms Project staff, provincial government clients (BEE), and other stakeholders (BEE and VCD) Effectiveness Harvesting secondary performance data (quantitative) Project M&E data Semi-structured interviews (qualitative) Project beneficiaries (VCD) provincial government “clients” (BEE), participants, and stakeholders Focus group discussions (qualitative) Focus group discussions with project beneficiaries in the mango, date, peach, potato, and weaving value chains. Sustainability Document review (qualitative) Firms Project planning documents and exit strategy Semi-structured interviews (qualitative) Project beneficiaries (VCD) provincial government “clients” (BEE), participants, and stakeholders Focus group discussions (qualitative) Focus group discussions with project beneficiaries in the mango, date, peach, potato, and weaving value chains. Gender Document review (qualitative) Firms Project planning documents and gender strategy Semi-structured interviews (qualitative) Project beneficiaries (VCD) provincial government “clients” (BEE), participants, and stakeholders In view of the limited time and the extensive travel requirements for data collection, the evaluation examined a subset of nine of the 13 project-supported value chains and nine of the 15 discrete BEE activities. The evaluation team selected the activities in consultation with USAID to focus, for VCD activities, on relatively large interventions in terms of expenditure, scale, and magnitude of reported results and, for BEE activities, those specified by USAID. The value chains in which the project worked included agricultural implements, dates, fresh mango, Islampur weaving,24 knitted garments, hotels, fish farms, peach, potato, fruit and vegetable pulping, dried mango, and marble and granite. The evaluation examined the first nine in this list. The project’s BEE activities included SMEDA, agricultural market reform, KP public private partnership framework, KP tourism policy, KP Chamber of Commerce and Industry (KP-CCI), Provincial Reconstruction, Rehabilitation & Settlement Authority (PaRRSA) (personnel), PaRRSA (progress review system), KP Board of Investment and Trade (KP-BOIT), KP work plan, urban policy reforms, KP/Federally Administered Tribal Area Investment Promotion Council, livestock sector, mines and 24 Islampur is a village in the Swat District of the KP Province. Weavers in Swat are well-known for their handmade woolen shawls. 15 minerals, federal Board of Investment (BOI) program, and South Waziristan report. The evaluation focused on the first eight BEE activities. The project classified SMEDA as a VCD activity, but it has more in common with BEE. The evaluation therefore presents findings for SMEDA with findings for BEE activities. The evaluators selected VCD beneficiaries for semi-structured interviews and focus group discussions by randomly ordering lists of participants and contacting potential respondents working downward from the top of the list until reaching the desired number of individuals, i.e., eight to 10 for focus groups. Annex 5 describes sampling protocols for each set of interviews or focus groups. Stratifying sampling by sex was not relevant since all participants in the selected value chains were men. Table 3 summarizes the method, number of participants, and location of qualitative data collection activities for the project’s VCD components. TABLE 3: INTERVIEW AND FOCUS GROUP DETAILS FOR VCD ACTIVITIES Interview Subject Data Collection Method Total Number of Beneficiaries Number Interviewed Location Fresh mango processors Semi-structured interviews with site visits 15 3 Multan Mango farmers (MGA) One focus group discussion 26 5 Multan Mango farmers One focus group discussion 223 6 Multan Date processors Semi-structured interviews with site visits 3 3 Sukkur/Khairpur Date farmers Two focus group discussions 45 18 Sukkur/Khairpur Agricultural implement manufacturers Semi-structured interviews 18 2 Islamabad Knitted garment manufacturers Semi-structured interviews with site visits 19 3 Karachi Hotels Semi-structured interviews with site visits 239 3 Swat District Fish farms Semi-structured interviews with site visits 22 3 Swat District Islampur weavers Semi-structured interviews with site visits 44 1 Swat District One focus group discussion 44 10 Swat District Peach Two focus group discussions 1,350 22 Swat District Potato Two focus group discussions 547 21 Swat District Five of the individuals the team identified for interviews were not available in person or by telephone during the period of the evaluation. Data collection for SMEDA suffered the most since no SMEDA personnel were available for interviews during three attempts by the evaluation team. 16 DATA ANALYSIS The evaluation drew extensively on quantitative data from the project’s M&E system. The only “analysis” of these data, however, was assessing the quality and validity of the data and checking for consistency. The evaluators used content analysis to analyze the qualitative data. At the end of each day in the field, team members consolidated field notes and discussed the interviews internally to resolve differences in understanding and interpretation. The team continued this process immediately after the field work to produce a consistent set of field notes. MEP uses trained local moderators and note takers for focus groups. The evaluation team briefed the moderators and note takers on the instruments prior to conducting the focus groups. The moderators and note takers provided the team with summaries of the focus groups. The quality of the summary notes varied considerably. Notes from the four focus groups with mango and date farmers were very detailed. Notes from the five focus groups with peach and potato farmers and the Islampur weavers were less so. The qualitative evidence for these five value chains is therefore not as rich as that for the other value chains. Nevertheless, they contained sufficient information to support the basic findings. The qualitative data provide rich evidence of how and why project interventions contributed, or fail to contribute, to results. To analyze the qualitative data from the interviews and focus groups, the team first identified themes relevant to answering the evaluation questions and then counted the frequency with which the themes emerged in relevant interviews or focus groups. For example, project interventions may influence the value of sales by increasing the quantity sold or prices. Higher prices, in turn, may be the result of better quality or access to new markets. When assessing how project interventions affected the value of sales, the evaluation team looked for evidence of these broad themes in the quantitative data. This approach allowed the team to indicate the strength of qualitative evidence as the percentage of relevant sources that mentioned a particular theme. DATA LIMITATIONS The Firms Project worked directly with very small producers (e.g., potato and peach farmers, weavers), medium-sized processors/manufacturers (e.g., dates, mango, knitted garments, agricultural implement manufacturers), and federal and provincial governments. The number of discrete components the evaluation addressed (i.e., nine value chains and eight business enabling environment activities) and their geographic distribution (Figure 1) limited the quantity of primary data the evaluation team could collect on each component in the three weeks allocated to data collection.25 Furthermore, since many of the BEE activities provided assistance to a single host country government department or agency, the number of knowledgeable individuals for BEE activities was often very small and many times a single person. The breadth of qualitative data is therefore narrow, especially for many of the BEE activities. To address this limitation, the evaluation team triangulated qualitative findings with 25 The map in Figure 1 illustrates the locations of VCD activities the team visited. The team investigated BEE activities in all locations and also made a dedicated trip to Peshawar to examine BEE activities with the GoKP. 17 other sources such as document reviews and interviews with stakeholders whenever possible. Nevertheless, the evidentiary basis is particularly thin for BEE activities. Annex 6 summarizes the number and type of interviews and focus groups that provided the evidence for each VCD and BEE activity. Collecting primary quantitative data on such a broad range of value chain and BEE activities would have been prohibitively time consuming and expensive. The evaluators’ preliminary assessment of the project’s M&E data found it to be relatively comprehensive and accurate.26 Therefore, the evaluation relied on project-collected data, validated when possible during interviews with project beneficiaries and stakeholders. However, the qualitative data from interviews served primarily to explain the how and why behind the quantitative results and not to validate the results the project reported. The project’s reporting procedures for outcome indicators largely reflected practical approaches to difficult measurement issues, e.g., attribution to project activities. Furthermore, in most cases reporting was consistent with the indicator definitions in the project’s Activity M&E Plan, a document approved by USAID. Nevertheless, it is important to note two discrepancies between reporting and common understandings of terminology that particularly affected the interpretation of report results.  The project’s Activity M&E Plan defined the value of exports as the quantity exported multiplied by the price per unit. The reported values, however, include shipping costs. This practice can substantially inflate the value of exports relative to domestic sales and produce misleading conclusions about the relative value of the two markets (export and domestic). This is particularly evident in the fresh mango value chain where shipping costs by air to the EU may exceed the value of the mangos themselves.  In general, the project’s Activity M&E Plan defined the incremental value of sales as the difference in the gross value of sales between the reporting period and the previous year (the baseline). In two instances, however, the Activity M&E Plan specified a different measurement. When an SME sold to a buyer that a) required a certification the SME obtained with project assistance or b) sold to a buyer it met through a project-supported exhibition or other market linkage activity, the entire value of the sale was attributed to the project. In these two instances, the project could have reported a positive incremental value of sales even if the SME did not increase sales at all. The project’s approach to reporting is relevant only if the beneficiary firm did not reduce sales to other buyers to accommodate the project-facilitated sale. These methods could have over- or under-stated actual results and it is not possible to determine the direction of any resulting bias. The project’s M&E team also estimated indirect jobs associated with interventions in the peach, mango, and date value chains. Because the project’s M&E team considered the estimates unreliable, however, the project chose not to report indirect jobs even though they certainly exist, a decision that probably leads to understating job creation. 26 The OIG Audit (Office of the Inspector General, Audit of USAID Pakistan’s Firms Project. Accessed at: https://oig.usaid.gov/sites/default/files/audit-reports/g-391-12-001-p.pdf ) raised several concerns about how the project reported results. However, the project retroactively revised reported results in accordance with the audit findings and the evaluation used the revised figures. 18 FINDINGS FINDINGS FOR EVALUATION QUESTION 1: RELEVANCE Evaluation question #1 (Relevance/Appropriateness): To what extent was the project relevant as designed, and how did the implementing partner (IP) adjust to keep it relevant in the face of changing priorities? This question examines relevance of design from the perspectives of USAID’s MSF and host country government priorities. To answer this question, the evaluation team reviewed USAID’s MSF; federal and provincial governments’ planning and strategy documents; and Firms Project technical proposals, reports, and Activity M&E Plans. Interviews with Firms Project staff and host country government stakeholders also contributed to the findings. Relevance to USAID’s MSF To assess the relevance of the Firms Project to USAID’s objectives, this section compares project objectives to USAID objectives as articulated in the MSF. The mission adopted the current MSF in 2013, midway through the Firms Project 2009-2014 period of performance. To fully assess the relevance of the project design it is therefore also necessary to examine its alignment with the USAID results framework in place in 2009. Because results frameworks are relatively high-level strategy documents, assessments of relevance to USAID objectives are necessarily fairly general. Figure 2 illustrates the high level objectives of the Firms Project’s results framework as described in the project’s Activity M&E Plan.27 FIGURE 2: FIRMS PROJECT RESULTS FRAMEWORK In 2009, when Chemonics designed the Firms Project, the relevant Assistance Objective (AO) was AO 2: “improved conditions for broad-based economic growth”. Intermediate Results (IRs) under this AO 27 USAID Firms Project Activity Monitoring & Evaluation Plan FY2009–FY2014. June 30, 2014. Objective: Dynamic, internationally competitive firms developed to accelerate sales, investment, and employment growth KRA 2: Improved business enabling environment at the federal, provincial, and district levels KRA 1: Enhanced competitiveness in project-assisted SMEs in targeted value chains 19 included IR 2.1.1: “improved competitiveness of focus value chains” and IR 2.2.2: “improved business enabling environment”. The Firms Project’s KRAs map almost directly to USAID’s IRs. In USAID’s 2013 MSF, the Firms Project fell under Development Objective (DO) 2: “improved economic status of focus populations” which included IR 2.1: “improved economic performance of focus enterprises” and IR 2.2: “improved business enabling environment”. The Firms Project’s KRAs aligned well with the 2013 results framework at the DO and IR level. Furthermore, three sub-IRs of the 2013 MSF, (i.e., IR 2.1.2: “improved skill development and job placement”, IR 2.1.3: “increased use of modern technology and management practices”, and IR 2.2.1: “improved ability to develop and implement reform of policies, laws, and regulations”) mapped very closely to the outcomes of project activities (i.e., skill development, technology and management practices, developing and implementing reform of policies, laws, and regulations). Table 8 and Table 9 in Annex 7 compare the USAID results frameworks for 2009 and 2013, respectively, to the corresponding elements of the project’s results framework. Relevance to Host Country Government Priorities The essential elements of the Firms Project strategy with respect to value chain development are a focus on SMEs and addressing constraints in infrastructure, technology, tools, and equipment; human capacity (knowledge); and market access – export or local as applicable to each value chain. The choice of value chains on which to focus was also an important element of the project’s overall strategy. To assess the relevance of the project’s VCD and BEE activities to host country government priorities, the evaluation determined the extent to which federal and provincial planning and strategy documents identified these elements of the project strategy as pillars, priority measures to promote development objectives, or important constraints to development. Table 4 summarizes findings from the document review. TABLE 4: EVIDENCE OF RELEVANCE OF PROJECT COMPONENTS AND STRATEGIES TO HOST COUNTRY GOVERNMENT PRIORITIES Source Project Components SMEs Competitiveness Agriculture/ textiles Overall VCD strategy Ag markets BEE SME Policy    Framework for Economic Growth   KP Integrated Development Strategy     Poverty Reduction Strategy (PRSP II)     Strategic Trade Policy Framework 2009-2010      Medium Term Development Framework: 2005-2010     A Strategy for Accelerating Economic Growth and Improving Service Delivery (2009) – Punjab      Textile policy (2009-2014)   Note: A “” means that a document identified the project component as a pillar, a priority measure to promote development objectives, or an important constraint to development. 20 The document review found broad support for the relevance of the project’s focus on SMEs; the choice of agricultural and textile value chains; and the overall VCD strategy of addressing infrastructure and technology, human capacity, and market linkage constraints. A majority of the documents also mentioned agricultural market reforms. Since the “clients” of the project’s BEE activities were host country government bodies, the interviews with project beneficiaries provided a great deal of evidence on the relevance of specific BEE activities to host country governments. Consequently, the remainder of this section focuses on the relevance of BEE activities. Relevance of Business Enabling Environment Components Firms Project staff reported, and the host country government “clients” the team interviewed confirmed, that BEE activities were largely demand driven, i.e., that in most instances government bodies asked for Firms Project support for specific tasks.28 A request for assistance usually implies relevance. To further ensure relevance, the project designed specific approaches to requested BEE work in a consultative fashion with stakeholders.29 Interviews with BEE beneficiaries confirmed the importance of the consultative approach; seven of nine individuals who spoke about the process mentioned that the project employed a consultative approach to designing specific BEE interventions. Two of the BEE components had a national scope: its work with SMEDA and work on agricultural market reform in each province. The remainder of the BEE components focused on the KP Province. The remainder of this section reviews evidence of the relevance of each BEE component to the clients or beneficiaries. The evidence consists largely of descriptions of how project interventions addressed client's needs. Agricultural Markets: The Firms Project’s background research30 and group interviews with project staff, as well as each of the three agricultural market reform stakeholders the evaluation team interviewed about agricultural market reform contributed to a consistent picture of agricultural markets in Pakistan. Wholesale fruit and vegetable markets in Pakistan are largely controlled by commission agents who transact most sales. The monopolistic position of these agents leads to high commissions and their rent-seeking behavior drives markets to focus on volumes rather than quality. Furthermore, the auction system separates producers from buyers and their demands. Buyers who want a high quality product have difficulty finding it and producers have little incentive to produce quality because they receive little reward for doing so and have limited knowledge of buyers’ requirements. A senior provincial government official explained that if reform “gets the markets right”, markets will connect sellers to quality-conscious buyers which will help farmers capture a larger share of the high-value 28 Senior Firms Project staff said that BEE interventions were not necessarily entirely demand-driven in KP or Balochistan. Firms Project staff informed the GoKP that USAID assistance was available and the types of support Firms could provide and the GoKP requested support. In the case of agricultural market reform in Balochistan, Firms Project staff reported that they had to “seduce” the Government of Balochistan. The evaluation team heard of no other instances of BEE support not being demand-driven. 29 USAID Firms Project: Business Enabling Environment Final Report. October 2014. 30 Fact Sheet: USAID Firms Project Agriculture Markets Reform Program. June 2, 2014. Preston Pattie. USAID Firms Project. Agricultural Marketing Policy Reforms – Balochistan. October 2011. 21 market and provide the signals and incentives for farmers to invest in production and quality. At the request of the provincial governments of Punjab, Balochistan, and KP, the Firms Project helped draft agricultural marketing policy, laws, and regulatory structures for market reforms.31 In Sindh where the provincial assembly had already passed the Wholesale Agriculture Produce Markets Bill of 2010,32 the project helped revise the reform act. SMEDA: The Small and Medium Enterprise Development Agency (SMEDA) was organized in1998 under the Ministry of Industry. It serves as an SME policy-advisory body with a focus on “providing an enabling environment and business development services to SMEs.”33 Over time, however, SMEDA began to focus more on providing services than on facilitating service provision. It had “lost its way” and “was going everywhere and nowhere” according to a key stakeholder the evaluation team interviewed. Of the nine SMEs the evaluation team interviewed that mentioned SMEDA, eight believed the organization was not meaningfully supporting SMEs. In 2011, the Ministry of Industry asked USAID to help build SMEDA’s institutional capacity. The Firms Project reviewed the SME Policy and the regulatory framework for SMEDA; drafted a SMEDA Amendment Ordinance and an SME Development Act on the basis of the reviews; and, in consultation with SMEDA management, developed a roadmap to improve SMEDA’s service delivery to SMEs.34 Assistance to GoKP: Group and individual interviews with project staff and GoKP officials and KP strategy documents reported that the new government elected in KP in 2013 campaigned on a platform of change. The province’s Integrated Development Strategy (IDS) for 2014-201835 emphasized the importance of broad reforms: “The pivotal section in the IDS is based on priorities identified in the Strategic Development Partnership Framework that aims to infuse effective capabilities in delivering good governance, regional economic growth, and social development. Governance reforms are therefore embedded in all government.” Senior GoKP officials the team interviewed about overall support to the government reported that support from the Firms Project addressed the government’s need for developing policies, drafting laws, and developing implementation strategies to support comprehensive reform. “This government was voted in for change in May 2013…there are so many laws and so many drafts in the pipeline…Firms has provided us with a lot of support.” KP-CCI: Recent conditions in KP (i.e., energy crises, poor law and order situation, political and economic uncertainty) were not conducive to economic growth. The GoKP identified industry and tourism as promising sectors and asked the Firms Project, through a Memorandum of Understanding 31 Fact Sheet: USAID Firms Project Agriculture Markets Reform Program. June 2, 2014. 32 Accessed at: http://eshipibian.weebly.com/uploads/2/2/9/6/22969152/the_sindh_wholesale_agricultural_produce_markets.docx 33 SMEDA website. Accessed at: http://www.smeda.org/index.php?option=com_content&view=article&id=2&Itemid=689 34 Fact Sheet: USAID Firms Project’s Assistance to SMEDA. June 2, 2014. 35 Government of Khyber Pakhtunkhwa. Integrated Development Strategy 2014-2018. Accessed at: http://www.khyberpakhtunkhwa.gov.pk/cms/downloads/kp.gov.pk-downlaods-%204d03341e2270bac774adeed45144043c.pdf 22 (MOU) with KP-CCI, to develop vision documents, and companion implementation plans, for the two sectors that outlined short and long-term economic development strategies.36 KP-Tourism and KP-PPP: The GoKP’s Integrated Development Strategy 2014-201837 and the evaluation team’s interviews with the three government officials who spoke to the issue confirmed that the GoKP looks to leverage its substantial recreational assets to develop tourism as a growth sector. However, one of the government representatives the evaluation team interviewed explained that existing tourism laws were outdated and had not been adapted to the KP context after the 18th Amendment made tourism a provincial responsibility. Furthermore, the GoKP’s tourism sector vision document called for developing a PPP framework to facilitate private investment in the sector.38 A GoKP official told the evaluation team that the KP Tourism Department asked USAID for assistance to update tourism laws and policies, draft a PPP framework to promote investment in tourism, and develop a web portal to promote tourism. However, the Firms Project’s review of existing laws determined that it was more appropriate to develop an overarching (i.e., not sector specific) PPP framework, which it ultimately did under the auspice of the Planning and Development Department. KP-Board of Investment and Trade: The GoKP established KP-BOIT in 2012 to “provide a proactive one-window operation to promote investment and trade in the KP Province.”39 However, GoKP officials reported that as of June 2014 KP-BOIT lacked a legal structure (and status), rules and by-laws, a human resources manual and structure, an investment guide, promotional materials, a website, and mechanisms for coordinating with other agencies involved in promoting investment. At the request of the GoKP, the Firms Project identified and hired KPMG, an international professional services company, to provide technical experts to develop these materials. In addition, the KPMG team conducted feasibility studies for 12 potential investment projects identified by government departments and KP￾BOIT.40 A representative of KP-BOIT spoke to the relevance of the support as follows: “These were the right things, the basic things that have to be done for a foundation [for promoting investment].” Provincial Reconstruction, Rehabilitation & Settlement Authority: PaRRSA was created in July 2009 to manage reconstruction and rehabilitation of the Malakand Division following the conflicts in 2009. Its mandate was expanded to cover the entire KP province in 2012 and it now plans and coordinates overall reconstruction, rehabilitation, and resettlement projects throughout the province.41 A KP government official explained that rehabilitation includes supporting economic recovery but that PaRRSA had little capacity in this area. To support the institution, the Firms Project identified, hired, and placed six specialists in PARRSA’s Economic Growth Department and one in the Communications 36 Because KP-CCI was not available for interviews, the evaluation team drew the information in this paragraph from interviews with project staff and project documents, i.e., Fact Sheet: USAID Firms Project’s Implementation of MOU between USAID and the Khyber Pakhtunkhwa Chamber of Commerce and Industry (KPCCI). June 2, 2014. 37 Government of Khyber Pakhtunkhwa. Integrated Development Strategy 2014-2018. Accessed at: http://www.khyberpakhtunkhwa.gov.pk/cms/downloads/kp.gov.pk-downlaods-%204d03341e2270bac774adeed45144043c.pdf 38 Vision Document for the Tourism Sector of Khyber Pakhtunkhwa. February 2011. Accessed at: http://pdf.usaid.gov/pdf_docs/PA00K63H.pdf 39 KP-BOIT website. Accessed at: http://www.kpboit.com.pk/index.php?option=sl&menu=sub&refs=11 40 Interviews with KP-BOIT and Firms staff. 41 PaRRSA website. Accessed at: http://www.pdma.gov.pk/parrsaobjective.php 23 Department. The project also helped PaRRSA develop a web-based application so it could better monitor, manage, and report on its activities.42 Pilot Information and Financial Mobile Solutions (PIFMS): The one KP government official the evaluation team interviewed who could speak to this issue said that the GoKP was particularly interested in supporting Telenor in its pilot to use mobile technologies to disseminate information to farmers. He believed the concept had potential because farmers were comfortable with the technology and the information is relevant to their needs. Design Changes to Maintain Relevance Fourteen months after the inception of the Firms Project, USAID asked it to substantially revise its implementation approach in response to a shift in strategy designed to respond quickly to the devastating floods of 2010 and the threats posed by extremist influences; a change that would have far￾reaching effects on implementation. Figure 3 summarizes the original objectives contained in the May 2009 contract with USAID and the revised objectives specified in Modification 4 signed in June 2010. FIGURE 3: EVOLUTION OF FIRMS PROJECT OBJECTIVES Original Objectives (May 2009) Revised Objectives (June 2010)  Improve the productivity and competitiveness of sectors with strong potential for exports, job creation, value addition, and ability to absorb women into their workforce at all levels.  Improve the implementation capacity of existing and forthcoming Sector Management Companies and increase the capacity of existing public sector economic development and service agencies.  Improve the business environment by reducing the cost and time of doing business and affecting policy and regulatory reforms through consultative mechanisms.  Implement demand-driven business reforms at the district, provincial, and federal level.  Improve the quality, availability, and diversity of business services at affordable cost to SMEs so that they can grow and ultimately capture export markets.43  Promote, accelerate, and expand economic opportunities by creating domestic and export demand-driven sales opportunities for private businesses that will expand the number of available jobs in vulnerable districts of Pakistan and in other districts where project action support such opportunities in vulnerable districts.  Improve the business enabling environment in vulnerable districts, at the provincial level, and at the federal level (where relevant) by assisting the private sector to develop and articulate policy and regulatory reforms that can improve competiveness and by assisting various levels of government in Pakistan to design and implement such reforms on an international best practice basis. 42 Fact Sheet: USAID Firms Project’s Institutional Support to the Provincial Relief, Rehabilitation and Settlement Authority (PaRRSA). June 2, 2014. 43 USAID/Pakistan. Contract No. EEM-I-00-07-0008-00. May 14, 2009. 24 The three revisions that particularly affected implementation were focusing project activities on vulnerable districts, eliminating references to engaging women, and removing access to business development services as an explicit objective. In a group interview, project staff reported that the emphasis on vulnerable areas steered the project towards agricultural value chains and into sectors and regions where it was more difficult to engage women (see the findings on question 4 for more detail). In 2010, USAID asked the Firms Project to manage a grant program designed to rebuild hotel and fish farm SMEs in the Malakand Division of KP that had been destroyed by conflict and floods. The project also provided grants for training and equipment required to operate the SMEs and build their capacities. The changes represented a significant new intervention and mechanism for the Firms Project. In its annual reports, the project reported having developed the grant mechanism by the end of the first fiscal year (June 2010). By the end of the second fiscal year (June 2011) the project reported having disbursed grants totaling USD 1.2 million to 239 hotels and 22 fish farms, provided in-kind support (e.g., inputs, supplies, operating equipment), and trained fish farmers and hotel operators and staff. Project annual reports indicate that the Firms Project started working with the GoKP in its second fiscal year (May 2010 – June 2011) to help revive and develop the tourism industry. When the new government was elected in 2013 on a platform of change, it requested USAID assistance to support its ambitious and far-reaching policy reforms. The project’s annual reports state that this work began in the project’s fourth fiscal year (October 2012 – September 2013). Interviews with GoKP officials and Firms Project staff suggest that the requests came in very close to the end of the project and that the personal relationship between senior Firms Project personnel and GoKP officials was instrumental in responding to the requests quickly. FINDINGS FOR EVALUATION QUESTION 2: EFFECTIVENESS Evaluation question #2 (Effectiveness): To what extent did the project achieve its stated objectives, and what lessons can be learned about the effectiveness of the business models/ development approaches/implementation strategies adapted/adopted/developed by the project? The chapter on effectiveness presents findings for VCD and BEE activities separately. Both subsections begin by presenting the quantitative data collected and reported by the Firms Project. The presentation of findings then augments the quantitative data with qualitative data collected during interviews and focus groups with project stakeholders and beneficiaries. The qualitative data provide evidence of how and why project activities were, or were not, effective. The Lessons Learned chapter of this report uses the findings to develop lessons about the effectiveness of the business models, development approaches, and implementation strategies the project adapted, adopted, or developed. Effectiveness of Project Support to Value Chains The Firms Project designed and implemented a comprehensive M&E system. The system collected data on the four key VCD indicators included in the project’s Activity M&E Plan from SME participants in supported value chains. Furthermore, the project flagged suspiciously large or small reported values of 25 sales and exports and validated them by examining the paperwork generated by a sale.44 The project reported job creation in terms of full-time equivalents (FTEs). The FTE of a part-time job is the job’s proportion of a full-time job (2,080 hours per year). For example, a part-time job for 60 days at eight hours per day is 0.23 FTE.45 Figure 4 illustrates results reported by the project and their relationship to LOP targets. Note the caveats in the Data Limitations section regarding the validity of the value of exports figures. The primary concerns with value of exports figures are in the mango value chain and the project would likely have exceeded its LOP targets even without counting the value of exports in this value chain. FIGURE 4: PROJECT-REPORTED VCD RESULTS46 Panel 1: Cumulative Incremental Value of Sales Panel 2: Cumulative Incremental Value of Exports Source: Project annual reports. Source: Project annual reports. Panel 3: Cumulative Jobs Created Panel 4: Cumulative Private Sector Investment Source: Project annual reports. Source: Project annual reports. 44 Interview with M&E staff and observation of scanned copies of invoices and shipping receipts. 45 USAID Firms Project Activity Monitoring & Evaluation Plan FY2009–FY2014. June 30, 2014. 46 Represents all project-assisted value chains, not merely the nine the evaluation examined in depth. 10 20 30 40 50 60 Year 1 Year 2 Year 3 Year 4 Year 5 USD millions Project reporting years Cumluative value of sales LOP target 10 20 30 40 50 60 Year 1 Year 2 Year 3 Year 4 Year 5 (USD millions) Project reporting years Reported value LOP target 0 1,000 2,000 3,000 4,000 5,000 6,000 Year 1 Year 2 Year 3 Year 4 Year 5 Number of jobs created (FTE) Project reporting years Reported value LOP target 1 2 3 4 5 6 7 8 9 Year 1 Year 2 Year 3 Year 4 Year 5 USD (millions) Project reporting years Reported value LOP target 26 Figure 5 illustrates the relative contribution of the project-assisted value chains to the four key indicators included in the project’s Activity M&E Plan: “increase in sales revenue of project assisted firms in the selected sectors”; “value of exports of targeted commodities as a result of USG assistance”; “increase in employment for project-assisted firms”; “and value of private sector investment mobilized through formally engaged SMEs”.47 The fresh mango, knitted garments, and peach value chains made the greatest contributions to the value of sales, together accounting for 56 percent of the total value of sales and 88 percent of the value of exports. However, the project reported the gross instead of net value of sales. For export-oriented value chains where project-supported enterprises paid shipping costs, these costs are included in the value of sales and exports, a practice that can substantially overstate the value of the indicators. The overstatement is particularly large in the mango results and may also be a factor in results reported for garments. The upshot is that reported gross figures for the value of sales and exports may substantially overstate the net values that would be a more appropriate measure of these indicators.48 The project also attributed all sales that stemmed from a project-assisted market linkage to the project. As noted in the Data Limitations section, this is appropriate only if the enterprise did not sacrifice other sales to fill the new order.49 The Swat peach and hotel components together accounted for two-thirds of the jobs the Firms Project reported having created. The hotel industry is labor intensive and all jobs recovered due to rebuilding destroyed hotels are attributable to the Firms Project. Focus groups with peach farmers revealed that harvesting peaches requires a substantial amount of labor. Results for the value of exports reflect the market focus of individual value chains and the way in which the project attributed export sales to project activities. Only the fresh mango, date, and knitted garment value chains were heavily export-oriented. Project-supported mango processors were not able to export directly prior to the Firms Project intervention. Therefore the project claims all export sales (i.e., 6,448 tons in the 2012-13 season) as attributable to the project. Project-supported date processors were already heavily export-oriented prior to the Firms Project and reported no increase in exports after one year of project support. Project-supported knitted garment manufacturers sold exclusively to export markets, but the Firms Project claimed attribution for any sale that emerged from a market linkage facilitated by the project (see the Data Limitation section for a detailed explanation). 47 USAID Firms Project Activity Monitoring & Evaluation Plan FY2009–FY2014. June 30, 2014. 48 It is important to note that the project’s Activity M&E Plan is unclear about how to treat production costs when calculating the values of sales and exports (i.e., “Production cost is not counted under this indicator”). The project’s M&E officer interprets this to mean that production costs are not subtracted from sales revenue. The Activity M&E Plan is also silent on the issue of whether shipping costs should be included in the value of exports. 49 The project’s Performance Indicator Reference Sheets contained in the Activity M&E Plan and approved by USAID clearly describe this approach to measuring sales attributed to market connections. 27 FIGURE 5: QUANTITATIVE RESULTS FOR VALUE CHAIN COMPONENTS Panel 1: Cumulative Value of Sales (USD millions) Panel 2: Cumulative Value of Exports (USD millions) Source: Firms M&E data. Source: Firms M&E data. Panel 3: Cumulative Jobs Created Panel 4: Cumulative Value of Private Sector Investment Source: Firms M&E data. Source: Firms M&E data. Note: These results may appear inconsistent with those reported in Figure 4, but they report results for the subset of activities the evaluation examined while the results in Figure 4 reflect all project activities. $.0 $5.0 $10.0 $15.0 $20.0 Mango Fresh (IUA & MGA) Garments Peach Swat Tourism Swat Fisheries Potato Islampur Weaving Dates Pakistan Handmade Agri Implements Value of sales (USD millions) $.0 $2.0 $4.0 $6.0 $8.0 $10.0 Mango Fresh (IUA & MGA) Garments Peach Swat Tourism Swat Fisheries Potato Islampur Weaving Dates Pakistan Handmade Agri Implements Value of exports (USD millions) 0 500 1,000 1,500 2,000 2,500 Mango Fresh (IUA & MGA) Garments Peach Swat Tourism Swat Fisheries Potato Islampur Weaving Dates Pakistan Handmade Agri Implements Number of jobs created (FTE) $.0 $.5 $1.0 $1.5 $2.0 $2.5 Mango Fresh (IUA & MGA) Garments Peach Swat Tourism Swat Fisheries Potato Islampur Weaving Dates Pakistan Handmade Agri Implements Value of private sector investment (USD millions) 28 The following subsections on project-supported value chains draw on qualitative evidence from focus group discussions with beneficiaries and semi-structured interviews with beneficiaries and stakeholders to explore how and why project interventions were or were not effective. Production and Post-Harvest Practices, Value of Sales, and Employment Evidence from focus group discussions suggests that the agricultural value chains that worked with farmers to increase production (i.e., mango, peach, and potato farming) all increased the quantity produced and the value of sales. Furthermore, they did so in very similar ways. In each of the two focus groups with mango, peach, and potato farmers and interviews with two IUA mango processors who were also MGA farmers, participants said that productivity-enhancing production practices (e.g., appropriate use of inputs) had increased the quantity produced and sold. One IUA mango farmer gave the example of local experts recommending applying fertilizer just before trees set fruit, but the Firms Project experts said to apply fertilizer immediately after harvest. Potato farmers in both of the two focus groups spoke of the contribution of new varieties to increased yields. Participants in one of the two focus group discussions with mango farmers said the new production practices cost more than their traditional methods but that the increased value of sales more than offset the additional cost. Participants in all four focus group discussions with peach and mango farmers emphasized the importance of harvesting and post-harvest handling practices (i.e., grading and packaging) to reducing waste and increasing quality, both of which contributed to increasing the value of sales. They all linked better quality to higher prices. Participants in all focus groups with mango, peach, and potato farmers said that applying the new production practices required more labor. All four groups of mango and peach farmers mentioned the additional labor requirements of pruning and spraying. Mango farmers also spoke of requiring more labor for harvesting. Both focus groups with peach farmers also brought out the additional labor required for land preparation and both recounted that the demand for seasonal labor during harvest outstripped supply and had driven wages up. One MGA farmer said that following the practices required for GlobalG.A.P. certification had increased labor costs but that the value of the greater quantity produced more than offset additional labor costs. All focus group participants reported that women had no role except to collect and process fallen mangos. Effectiveness by Value Chain Fresh Mango Processing Value Chain Intervention: By sharing the cost of installing packing houses, the Firms Project established new mango processors in producing areas with the capacity to export directly, i.e., not indirectly through exporters. Because these businesses were new to processing and direct exports, baseline values for value of exports, jobs, and private sector investment (to support processing) were all zero. Since they were all established mango farmers, and selling mangos prior to the Firms Project, the baseline figure for value of sales is not zero. In fact, the project reported a baseline value of USD 8,563,650 29 in 2009. Semi-structured interviews with three project-supported mango growers/processors provided evidence of how project interventions influenced quantitative measures of effectiveness. Value of Sales/Exports: All three processors reported that they had been selling to wholesalers and exporters prior to participating in the Firms Project. They said that the ability to export directly had increased the value of their sales. Although all three had obtained HACCP certification through the Firms Project, only one mentioned it as an important contributor to (export) market access.50 All had attended the Fruit Logisitca exhibition in Berlin in 2010 and one specifically mentioned that he sold mangos to a buyer he met there. Figure 6 shows the project-reported total value of sales for the three mango processors the evaluation team interviewed.51 Processor 1, who had not yet used his equipment, showed no increase in value of sales. Processors 2 and 3 reported increases in 2010-11, the first season they used their processing lines, relative to the previous season. The three processors reported different experiences over the five-year period depicted in Figure 6. Processor 1, who did not use his equipment, experienced a 56 percent decline in the value of sales.52 Processor 2 reported a 73 percent increase but with a great deal of fluctuation. It is important to note that the project measured value of sales and exports as gross values, inclusive of shipping costs paid by the processor. Shipping costs by air are very high and can lead to gross values being an order of magnitude higher than net values. The large value of export sales reported by processor 2 in 2013 largely reflects the cost of air shipments and not the net value of export sales.53 Processor 3 reported a small (6 percent) decrease in the value of sales. 50 Note that the interview guide did not ask specifically about HACCP certification so it would be incorrect to conclude that processors who did not mention it did not find it important. 51 Project M&E data: LFA Master Tracker. 52 While processor 1 did not use his processing equipment, he contracted with another project-supported processor to process and export mangos. 53 Investigation of M&E data and discussions with project M&E staff. 30 FIGURE 6: TRENDS IN VALUE OF SALES – PROJECT-SUPPORTED MANGO PROCESSORS Panel 1: Processor 1 (equipment not operating) Panel 2: Processor 2 Source: Firms M&E data. Source: Firms M&E data. Panel 3: Processor 3 $0 $200,000 $400,000 $600,000 $800,000 $1,000,000 2009 2010 2011 2012 2013 Value of sales (USD) Value of local sales Value of exports Source: Firms M&E data. Job creation: The two processors who had operated their processing plants reported employing 30-40 people in processing throughout the 4-5 month season.54 Since these are new businesses that would not have existed without the Firms Project, the additional jobs are attributable to the project. Factors that Influenced Effectiveness: Two of the three mango processors and three of the four mango sector stakeholders the team interviewed spoke of the effects of the fruit fly on exports of mangos from Pakistan in the 2013-14 season. In an effort to avert a ban on exports to European markets, the Pakistan government instituted a requirement for a one-hour hot water bath on all mango exports to Europe. Since the processing lines of project-supported mango processors were designed for a five-minute bath, the new requirement significantly reduced throughput in the plants and substantially reduced exports to high-end European markets during the 2013-14 season. 54 Four months is one-third of a year so the 40 jobs per firm translates to about 13 full-time equivalents (FTEs) per firm per year. $0 $200,000 $400,000 $600,000 $800,000 $1,000,000 2009 2010 2011 2012 2013 Value of sales (USD) Value of local sales Value of exports $0 $200,000 $400,000 $600,000 $800,000 $1,000,000 2009 2010 2011 2012 2013 Value of sales (USD) Value of local sales Value of exports 31 All three project-supported mango processors the evaluation team interviewed said that they expected, and would have preferred, imported processing equipment, but received locally made equipment. They were all waiting for the grader that had not yet been delivered.55 One claimed that the equipment was not well made and that he had to continually pay for repairs. One mango processor believed the Firms Project had not focused enough on export marketing: “introducing the processor to the buyer was not enough”. He stated that the United Nations Industrial Development Organization (UNIDO) did a better job, working side-by-side with the processors throughout the entire process from production to shipping. He thought the Firms Project should have focused more on air shipments and on closer markets (e.g., United Arab Emirates [UAE], Malaysia) that are lucrative, less demanding, and easier to access. Mango Farmers (MGA and non-MGA) Intervention: The project supported both MGA and non-MGA mango farmers by promoting productivity-enhancing production practices. In addition, it helped MGA farmers obtain GlobalG.A.P. certification so they could supply exporters who required the certification; they were not direct exporters. The relevant measures of effectiveness are therefore value of sales (covered previously) and job creation. Job Creation: Focus group participants reported that productivity-enhancing production practices (e.g., systematic and frequent application of pesticides, micro nutrients, fertilizer, irrigation, pruning, harvesting with clippers, and cleaning the orchard after harvest) had increased employment and overall labor cost. One MGA farmer said that following the practices required for GlobalG.A.P. certification had increased labor costs, but the value of the greater quantity produced more than offset the additional costs. None of the respondents gave a quantitative estimate of employment effects. They all reported that women had no role except to collect and process fallen mangos. Factors that Influenced Effectiveness: Focus group participants brought up several issues they believed limited the effectiveness of project interventions. Participants in both focus groups cited a need for more technical support for mango growers and implied that government extension support was weak. One group spoke at length about the market linkages support provided by the project. The participants believed that the project should have focused more on improving access to regional (e.g., UAE, Korea, Japan), rather than EU, markets. They believed these markets were lucrative, less demanding (e.g., did not require irradiation which was not available in the area), and physically easier to reach. Participants in one group also said a publicly accessible value-added facility to support drying and provide cold storage would improve their access to markets and their profits. The project benefitted from previous and ongoing donor activity in the mango sector. The AusAID￾funded Australia Pakistan Agriculture Sector Linkages Program Phase II (December 1, 2010 – November 30, 2014) focused on improving mango production practices. Both focus groups with 55 Firms Project staff recounted that they ordered the grader from a local vendor who had not produced graders before. The vendor could not get the grader to work correctly. The project then solicited bids from other (foreign) vendors. The first vendor selected backed out and the project is still waiting for the second (selected) vendor to deliver the graders. 32 mango farmers as well as two of the three IUA mango farmers the evaluation team interviewed and two of three mango research organizations interviewed mentioned the AusAID project as providing a foundation for the work of the Firms Project, and priming some farmers’ interests in marketing to take advantage of increased production and quality. Several interview subjects and focus groups mentioned the EU-funded and UNIDO-implemented Trade: A Route to Prosperity Program’s (August 2009 – May 2016) work assisting with mango marketing, i.e., intensive hand-holding through the exporting process. Firms Project staff told evaluators that the project somewhat deliberately ceded the intensive marketing support work to UNIDO. Date Processing Intervention: The Firms Project provided cold storage on a cost-share basis to extend the shelf life of dates and the length of the processing season. It also helped the supported processors obtain HACCP certification to enhance their access to U.S. markets. Value of sales: None of the three project-supported date processors the evaluation team interviewed reported substantial increases in the quantity of dates sold since the start of the Firms Project. In fact, the project’s M&E data show a decline in sales between the 2011-12 and 2012-13 seasons, the only years for which data exist.56 However, the primary intervention that should have increased the quantity sold and thus the value of sales was cold storage, and three of five processors the team interviewed (three beneficiary and two non-beneficiary) reported having rented local storage prior to obtaining their own storage facilities. In fact, some reported still renting storage because the storage facilities they installed with Firms Project support are too small or not yet operational. However, all five reported that rented storage was inconvenient and expensive. Value of exports: In interviews, the three project-supported date processors reported being highly export-oriented. However, two of the three said that the HACCP certification they gained through the Firms Project gave them access to the U.S. market. Only one of the beneficiary processors reported selling to U.S. markets. Other markets included Hong Kong (one processor), Canada (two processors), Moscow (one processor), Germany (two processors), United Kingdom (one processor), and South Africa (one processor). Job creation: The three date processors said that employment depended almost entirely on the quantity they processed. While Firms Project interventions should, in theory, increase the quantity processed by increasing quantity sold, the processors had not had their equipment long enough (one season at most) to have experienced a significant change in sales or employment. Factors that Influenced Effectiveness: Three beneficiary processors, two non-beneficiary processors, and one industry stakeholder the team interviewed mentioned several factors that influenced the effectiveness of the project and may affect future development of the sector. Figure 7 summarizes the key factors interview subjects brought out. 56 The evaluators could not find timely data on date production, prices, or exports to determine whether the 2011-12 or 2012- 13 seasons were anomalies. 33 FIGURE 7: EXTERNAL FACTORS AFFECTING RESULTS IN THE DATE VALUE CHAIN Contributors/Barriers to Achieving/Sustaining Results Number of Responses (out of 6) Limited cold storage capacity (private and publicly accessible) 4 Expense of cold storage (costs to operate own storage or rent private storage) 2 More market linkage support/promotion 2 Tissue culture (additional varieties) 3 Subsidies by other producing countries/competition from other markets 3 Contamination (oil and pesticides)/lack of standards 2 Continued education of farmers 2 Date Farmers In two focus group discussions, project-supported date farmers said they sold only through agents to local markets; they did not export. Intervention: The Firms Project shared the cost of drying tables, solar drying tunnels, and trolleys (for moving and drying dates in the tunnels). The support was aimed at increasing the quality and value of dried dates. Relevant measures of effectiveness are therefore the value of sales and employment. Value of sales: In both of the two focus groups with date farmers, participants reported that the tables, trolleys, and solar tunnel dryers had increased the cleanliness and quality of their dates and therefore the price and the value of sales. Job creation: Participants in both focus groups said that the equipment and tools provided through the Firms Project helped them use labor more efficiently. However, speed was of the essence in order to minimize the risk of loss due to rain. Farmers in both focus groups reported that they had not reduced the number of workers they employed; they used the same number and accomplished the tasks more quickly. Factors that limited effectiveness: Participants in both focus groups spoke at length about the insufficient quantity of equipment. Participants in both groups said that the quantity of tables provided was sufficient for only a small portion (e.g., 20 percent) of the crop. They dried the remainder in the traditional way, on mats on the ground. Participants in one group also told of how the lack of access to cold storage limited their direct access to markets and made them dependent on commission agents to store their product. Peach Farmers Intervention: The Firms Project promoted new practices to increase production and new harvest and post-harvest practices to enhance quality and reduce waste. The relevant measures of effectiveness are therefore value of sales and employment, both covered previously. Factors that limited effectiveness: Focus group participants mentioned several factors that may limit project effectiveness. Both groups spoke of the need for access to production credit to free 34 them from depending on agents for sales and the need for better research or information regarding diseases and appropriate pesticides and their use. One group each mentioned the following: scarcity of water, the need for cold storage and transportation to maintain quality and direct access to (local) markets, and research on value-added products and the local capacity (factories) to produce value-added products. Potato Farmers Intervention: The Firms Project promoted new practices to increase production, introduced different varieties required by a potato chip processor (PepsiCo), and linked the farmers to PepsiCo through an intermediary (SE Trading). Relevant measures of effectiveness are therefore the value of sales and employment, both covered previously. Factors that limited effectiveness: Firms Project staff reported that the project first linked potato farmers (organized in clusters) to PepsiCo directly. However, because of adverse weather conditions, the 80 tons farmers delivered did not meet PepsiCo’s expectations of 200 tons. The project then connected the potato clusters to an intermediary trader, SE Trading, which was already working with other potato clusters and selling to PepsiCo. Two focus group discussions with project-supported potato farmers, however, suggest that the experience has not been particularly good for farmers. Both groups reported that SE Trading had not provided the required inputs to farmers on time and had still not paid for potatoes the farmers delivered two years ago. They reported that most farmers were therefore hesitant to experiment with different marketing channels and continued to sell through agents. Knitted Garments Intervention: The Firms Project supported 19 knitted garment manufacturers by providing, on a cost-share basis, labor-saving sewing machines and Computer Aided Design (CAD) equipment to improve the efficiency of fabric layout and reduce waste of raw material. In addition, the project introduced software to improve planning and management. It supported the interventions by providing training in management, production methods, and use of the new machines. The project also helped manufacturers obtain certifications (e.g., WRAP and C-TPAT) required by some buyers. The project also hired consultants to help link manufacturers to new buyers and took manufacturers to the Market Magic trade show in the U.S. to introduce them to potential buyers. Value of sales: The three project-supported manufacturers reported that certifications, most importantly WRAP, had helped them meet the requirements of a greater number of buyers and increased their orders and production. They also said that being able to demonstrate improved production and quality control processes, due in large part to project-supported training and equipment, helped them attract orders and buyers. All three beneficiary manufacturers commented positively on the intensive and hands-on training. Job creation: All three manufacturers interviewed reported that project-supported interventions (most importantly machines that automatically trimmed threads) increased labor efficiency. However, they also reported that the increased labor requirements associated with a greater quantity produced more than offset gains in labor efficiency and that they had hired additional labor (men and women) to meet the increased quantities they were producing. One said specifically that women were easier to train and more suited to operating the newer machines. 35 Other factors: All three interview respondents said that the Generalized System of Preferences (GSP) Plus agreement between Pakistan and the EU, which exempts buyers from import duties, has increased the share of their sales going to the EU relative to that going to the U.S. Two of the manufacturers reported that they preferred selling to the EU because order sizes were smaller and more manageable and because buyers were not as aggressive about negotiating lower prices. One specifically mentioned that buyers like Wal-Mart ordered very large quantities and very aggressively demanded a low price. Two of the three beneficiary manufacturers the team interviewed reported that recent changes in exchange rates and the economic conditions in the U.S. had depressed sales. Two of the three manufacturers reported that they had not received the complete Enterprise Resource Planning (ERP) accounting and management system, in particular, the module on merchandizing.57 Agricultural Implements Intervention: The Firms Project provided equipment and production and management training with the aim of increasing the efficiency of production. Although the intervention did not specifically target export markets (and in fact, focused on import substitution), the project facilitated a visit to an exhibition in Dubai to introduce beneficiaries to potential buyers. Value of sales, value of exports, and employment are all relevant measures of effectiveness. Value of sales: Interviews with two owners of project-supported implement manufacturers told evaluators that the intervention increased their efficiency of production by establishing assembly line methods, installing new equipment (welders), and teaching lean manufacturing techniques. These interventions led to much lower electricity costs (e.g., from Rs. 100,000/month to Rs. 60,000/month), increases in quantity produced (71 and 50 percent increases), and better quality. Interviewed manufacturers emphasized assembly line production, savings in electricity, and planning as being particularly important factors in improving their competitiveness and ability to reliably meet orders. Higher quality attracted buyers and both manufacturers reported increasing the value of their sales – one by about 25 percent. Value of exports: Although the intervention did not focus on export markets, one manufacturer reported increasing his export sales from four units in 2012 to 92 units in 2013, and 83 units through October 2014. The other did not mention export markets. Job creation: Both manufacturers said that greater production efficiency increased production capacity, but so far they had met increased demand for equipment with the same number of skilled workers producing more efficiently. Both said they had hired additional unskilled labor to handle parts and finished machines and manage office operations. One specifically mentioned the lack of skilled labor available if he wanted to further increase his capacity. External factors affecting results: The two manufacturers evaluators interviewed mentioned several factors that could have enhanced results or improved prospects for sustainability. One spoke 57 Project-supported manufacturers received different components of the ERP software depending on their needs and willingness to share the cost. 36 mostly about the support provided by the project. He said that the business management software system provided by the project was installed too late and for too few beneficiaries. Manufacturers require additional training to realize the full benefits of the system. The manufacturer also said that the exposure visit to Turkey would have been more useful if it had occurred earlier.58 The exposure visits also raised beneficiaries’ expectations about the type of equipment they would receive and they were then disappointed to receive different equipment (e.g., arc rather than gas welders, manual rather than power presses, no Computer Numerical Control (CNC) or multiple drilling machines). The second manufacturer spoke more about industry-level issues including working on rules, regulations, and standards to guide the industry; the need for a common testing/certification facility; and increasing awareness among manufacturers of the value of working in clusters – they should not work as “jack of all trades, master of none”. Both manufacturers said that forming market linkages requires more attention than the project provided. They also said that the lack of skilled labor constrained their growth prospects and one suggested that a technical training institute would help ease this constraint. Islampur Weavers Intervention: The Firms Project supported the Islampur weavers primarily by connecting them to domestic buyers outside of Swat. Measures of effectiveness relevant to the intervention are therefore value of sales and employment. Value of sales: Participants in the one focus group discussion with the Islampur weavers said that the value of their sales had increased, and attributed the increase to market connections established through the Firms Project. They also mentioned being better able to meet market demands after learning about appropriate quality and designs from the buyers. “Due to the support extended by the project and participation in different exhibitions, weavers got an opportunity to meet different new buyers and understood market requirements at different places. This resulted into increased number of buyers and increased sales of the products. Opportunity for work has increased for all weavers as when they receive huge orders; all of them put their resources together to fulfill those demands in time.” Job creation: Focus group participants said that, together, they had to employ about 25 additional workers to produce the shawls required to satisfy orders obtained through project-supported market linkages. 58 The exposure visit was designed to introduce manufacturers to more efficient production processes that similar-sized manufacturers used in Turkey. 37 Effectiveness of BEE Activities The definition of what constitutes effectiveness is somewhat different for BEE than for VCD activities. For the most part, the project’s BEE activities responded to specific host country government requests such as drafting laws, regulatory frameworks, and implementation plans. The project was responsible only for producing results related to the number of policy reforms identified and the number of related documents submitted, accepted, or in some stage of development. Figure 8 shows the project’s progress on each of these four key results as reported in its periodic performance reports.59 FIGURE 8: QUANTITATIVE RESULTS FOR BEE Panel 1: Policy Reforms Identified Panel 2: Recommendations Submitted Source: Firms M&E data. Source: Firms M&E data. Panel 3: Reforms Accepted Panel 4: Reforms in Some Stage of Development Source: Firms M&E data. Source: Firms M&E data. Note: No LOP target reported. The year 5 annual report gives a cumulative total of 105, but individual annual reports document only 44. 59 USAID Firms Project Annual Progress Report – V October 2013-September 2014. 0 20 40 60 80 100 Year 1 Year 2 Year 3 Year 4 Year 5 Cumulative mnumber of reforms Policy reforms identified LOP target 0 20 40 60 80 100 Year 1 Year 2 Year 3 Year 4 Year 5 Cumulative number of reforms Recommendations submitted LOP target 0 20 40 60 80 100 Year 1 Year 2 Year 3 Year 4 Year 5 Cumulative number of reforms Recommendations accepted LOP target 0 20 40 60 Year 1 Year 2 Year 3 Year 4 Year 5 Cumulative number of policies Number of policiesat a given stage of development 38 Interviews with project staff, stakeholders, and beneficiaries suggest that progress on legal, policy, and regularly reform and institutional change results occurs along a continuum and takes time. The Firms Project works at one end of the continuum, implementing activities (e.g., hiring consultants) that produce outputs such as draft laws and regulations, implementation plans, etc. The ultimate outcomes related to these activities emerge only after laws are passed and regulations implemented. These results depend on factors outside the control of the project such as the willingness of federal and provincial governments to take up the issues, their institutional capacity to implement, and their continued commitment – particularly challenging given the frequent turnover of senior government officials in Pakistan. The Firms Project was effective in the sense that it met or exceeded its targets for the required high￾quality BEE outputs. To assess effectiveness in a broader context, admittedly beyond the project’s control, the evaluation examined three criteria that may influence the likelihood that the BEE work will ultimately contribute to the desired long-term results; whether the project provided the required support, quality, or relevance of the work to the client; and whether the project had submitted products to the immediate client.  The six “clients” of the project’s BEE work that the evaluation team interviewed all reported that the project produced and delivered all the required outputs.  The eight clients and stakeholders the evaluation team interviewed all claimed that the quality of the project’s work was “excellent.” For example, one said that the project provided “senior people and showed real dedication to the task.”  Two provincial government clients and one stakeholder specifically mentioned that even though governments had not yet passed laws based on the project’s work, the project added significantly to the base of knowledge necessary to make future progress. FINDINGS FOR EVALUATION QUESTION 3: SUSTAINABILITY Evaluation question #3 (Sustainability): To what extent are the project’s outcomes as defined in the project results framework likely to continue once project assistance is formally concluded? It is generally premature to expect sustainable outcomes from a development project within its planned period of performance. In fact, the Firms Project’s results framework identifies only intermediate results for BEE activities, recognizing that responsibility for implementing reforms (which should ultimately yield outcomes) rests with host country governments and is beyond the project’s control. The findings in this section draw primarily from interviews with project beneficiaries, stakeholders, and federal and provincial government “clients” to assess the likelihood of sustainable results in each value chain and BEE activity. It addresses sustainability of VCD results from the perspective of the four primary outcomes: value of sales, jobs, value of exports, and private sector investment. 39 Strategies for Sustainability The issues of sustaining results beyond the life of the project and achieving scale are different concepts. However, the Firms Project proposal describes a sustainability strategy that relies on emulation, or scale: “Our plan for sustainability is simple: firmly embed a sales-driven development model, proven over time in a variety of countries, into the SMEs and key private and public institutions that are EPF’s counterparts.” “Technical assistance and other support to a group of progressive manufacturers will develop self￾sustaining momentum; emulation will transform the sector.”60 Nevertheless, the evaluation focuses only on assessing the prospects for sustaining the key outcomes identified in the project’s Activity M&E Plan (value of sales, jobs, value of exports, and private sector investment among project beneficiaries) with direct project beneficiaries and does not introduce the issue of scale. The project’s exit strategy for VCD activities identified four things that the project must do to improve prospects for sustainable VCD results, the most important of which involves disseminating project learning to key sector stakeholders to secure commitments from qualified institutions to implement project-designed plans moving the sectors towards sustainability. The exit strategy concluded broadly that outdated agricultural marketing laws threaten the sustainability of the project’s work in agricultural value chains. The value chains that built capacities to bypass wholesale markets (i.e., fresh mango and potato) were most successful and further along the sustainability continuum, while those that still relied on wholesale markets (i.e., peach and date) had not progressed as far. The exit strategy report assessed the manufacturing activities (i.e., knitted garments, agricultural implements, weaving) to be in the early part of the third of fourth phases of sustainability: adoption. Beyond developing a detailed exit strategy, there is no evidence that the project implemented any of the exit strategy plans prior to closeout.61 Findings on the Sustainability of VCD Results The project’s value chain interventions focused on three main activities: upgrading infrastructure and providing tools and equipment, developing human capacity through training, and connecting beneficiaries to markets. The relative importance of the three components varied across value chains depending on the specific needs of actors. In the context of the project’s value chain work, sustainability will depend on four factors: 60 Empowering Pakistan: Firms: A Task Order Proposal under the GBTI II IQC. Chemonics International Inc., October 14, 2008. 61 Personal communication with three project staff members. 40  Beneficiaries’ willingness or ability to maintain or replace infrastructure, equipment, and tools;  Refreshing and reinforcing knowledge gained during training;  The sustainability of market linkages and marketing capacities; and  The absence of prolonged adverse conditions that could affect production or marketing opportunities. Training to enhance human capacity cuts across all of the project-supported value chains. While all VCD beneficiaries spoke of the value of training, only three, two processors who buy from farmers and a research institute, spoke of the importance of reinforcing lessons with repeated training: “Farmers are also uneducated and need repeated training to reinforce lessons. Once is not enough.” The following subsections present findings gleaned from interviews and focus groups related to the likelihood of maintaining and replacing equipment, sustaining market linkages, and continuing to apply productivity-enhancing practices promoted by the Firms Project. Fresh Mango Processing Equipment maintenance/replacement: The fact that all mango processors had made significant cost-share investments (an average of USD 54,364 per firm)62 in their processing plants suggests that they have the financial capacity to maintain or replace equipment if they deem it important. The interviews with three of the 14 project-supported mango processors provided additional evidence of their willingness and ability to purchase and maintain equipment.  One processor had installed, at his own expense, a screen separating the receiving and packing areas of his packing house to accommodate Chinese requirements for fruit fly control. However, he was looking for investors to help defray the estimated USD 40,000 cost of increasing the capacity of the hot water bath in his processing line. He did not specifically mention whether he would be able to maintain or replace equipment.  One processor reported that he would not have made the initial investment in processing equipment without the Firms Project, but it was because of a lack of knowledge and confidence rather than financial ability. In fact, he said that he probably would have made the investment on his own eventually. He also reported having to repair his equipment frequently at his own expense.  The third processor had not yet used his packing house and had no response to the question of maintaining or replacing equipment. He did mention, however, that it took him some time to arrange financing for his share of the cost of establishing the processing and packing house. 62 Firms Project M&E data. This figure represents the average over 15 MGA farms, 11 IUA/MGA farms, and four IUA only farms. However, the reported data do not distinguish between IUA and MGA investments. 41 Market linkages: According to the project’s M&E data, all 14 project-supported mango processors were selling mangos to local markets and to exporters prior to engaging with the project. The project focused on linking processors directly to export markets, and this section thus examines evidence of linkages to export markets. The M&E data indicate that two of the 14 project-supported processors exported mangos in 2009, prior to engaging with the Firms Project. By 2013 (the latest data available), all reported having exported in at least one year, and 12 had exported consistently (i.e., every year) once they started exporting. These data suggest that engagement with the Firms Project built the export capacity of the processors and that they maintained the market linkages they formed. However, the track record of exporting is relatively short, i.e., one processor (7 percent) had exported for five years, one (7 percent) for four years, seven (50 percent) for three years, four (29 percent) for two years, and one (7 percent) for one year. The data are questionable since one of the three processors the evaluation team interviewed reported having exported directly only once, whereas the M&E data reported exports for three consecutive years – in one of which the farmer reported being out of the country and selling to contractors. Subsequent discussions with USAID and project staff suggest that the discrepancy may result from farmers reporting only direct export sales during interviews, while the M&E data include both direct and indirect exports. External factors also threaten the sustainability of market linkages. During the 2013-14 season, the GoP dictated that all exports of mangos to the EU undergo a one-hour hot water bath to control the fruit fly. All three processors the evaluation team interviewed, a mango trader, and all three of the research organizations involved in the mango sector mentioned this as a significant threat to exports of mangos from Pakistan. Date Processing Maintenance/replacement: The evaluation team interviewed the owners of all three project￾supported date processing plants. One said that his family was involved in other businesses such as textiles and cement. Coupled with the fact that all had made significant cost-share investments in their plants (an average of USD 122,071 per firm) as part of their infrastructure upgradation agreements with the Firms Project, this suggests that they had the financial capacity to invest.63 Two of the three processors provided specific evidence of their willingness and ability to invest in equipment. One recently purchased an automated washing line to clean dates to support his strategy to expand sales of value-added products (i.e., paste) to export markets.64 He reported spending Rs. 11 million (USD 110,000) on the machine. Another mentioned plans to double the size of his cold storage at his own expense. Market linkages: All three project-supported date processors were processing and selling dates well before receiving support from the Firms Project. They reported having good connections with buyers – almost entirely in foreign markets. All three processors reported selling consistently every 63 Firms Project M&E data. 64 The evaluation team observed the equipment. It had been installed but had some problems in the first season and was not in use at the time of the team’s visit. 42 year. One said that the HACCP certification he obtained through the Firms Project helped him access new buyers in the U.S. The experience of one plant owner illustrates how financial constraints limit his ability to promote his product and invest in equipment. He reported that he is willing to pay his own way to future market exhibitions but can’t afford the USD10,000 cost of a stall. He explained that he must buy all the dates he needs to keep his plant operating for the year within 20 days of harvest before prices start to increase. He therefore has all his capital tied up in dates and can’t afford to purchase equipment. He is not willing to take bank loans because of high (24 percent) interest rates which would eliminate his profit margin. He reported that collateral requirements and documentation are also barriers to obtaining bank credit. Borrowing is also risky because he could lose everything if he is unable to repay the loan. Knitted Garments Manufacturing Maintenance/replacement: All three garment manufacturers the evaluation team visited and interviewed were well established firms with considerable investments in plant and equipment. They had made substantial cost-share investments under the Firms Project – USD 34,511 per plant on average.65 One reported buying additional machines after receiving some under the agreement with the Firms Project. All three implied that they were willing to make additional investments, but they mentioned constraints to doing so. For example, one reported that market opportunities and skilled labor, not access to capital, limited investment. Another said he was planning to invest in the capacity for screen printing, washing, developing his own brand, and beginning to market online. The third said his money was “stuck” with the government taxation and rebate department and this was delaying his plans for investing to expand production. Market linkages: All three garment manufacturers were marketing substantial quantities exclusively to export markets prior to engaging with the Firms Project. Reported monthly production ranged from 40,000 to 60,000 units. All three reported selling larger quantities after the Firms Project intervention than before and that certifications (e.g., WRAP, C-TPAT) opened doors to new markets and increased their competitiveness. Agricultural Implements Maintenance/replacement: The two implement manufacturers the evaluation team interviewed were established firms; one had been in business for 52 years and the other for 39 years. During the interviews, they did not indicate whether they were willing or able to repair or replace equipment. However, on average, project beneficiaries in the agricultural implements sector had invested USD 9,272 as their share of project-supported equipment.66 Furthermore, both firms reported that they had incurred the cost of obtaining ISO 9000 certification and planned to continue renewing the certification in the future. 65 Firms Project M&E data. 66 Firms Project M&E data. 43 Market linkages: The Firms Project did not focus on linking agricultural implement manufacturers to markets. It organized a visit to the 2014 AGRA Middle East exhibition in Dubai for eight implement manufacturers but did nothing further to promote market linkages. Nevertheless, the two implement manufacturers both reported increased sales since engaging with the Firms Project; one reported a 25–30 percent increase and the other an increase of more than 70 percent with an increasing quantity to export markets. Date Farming Continuing productivity-enhancing practices: The two focus group discussions with date farmers provided evidence of the likelihood that farmers will continue to use the productivity-enhancing production practices they adopted as a result of participating in the Firms Project and barriers to their continued use. A small number of respondents in both groups said that the equipment required to apply practices was too expensive to replace if it was lost or destroyed (e.g., tunnels). However, most respondents in both groups indicated that they could afford to replace or repair relatively inexpensive items but would look for less expensive options (e.g., wood instead of plastic tables). In fact, a number said they were already doing so – gradually increasing their stock of crates and tables. “We have profit, and we have started buying our own things related to new technology; because we know these things give more profit; but we will not buy expensive items, yes only that we can afford such as tables, crates; sprayer is good but we will see if we can buy in case broken.” Peach Farming Because the Firms Project support to small farmers focused more on training than on providing equipment and tools, willingness and ability to invest in productivity-enhancing production practices is a more relevant indicator of sustainability than their ability to maintain or replace equipment. Continuing productivity-enhancing practices: Participants in both of the two focus groups with peach farmers said they would likely continue to use the productivity-enhancing practices they had learned under the Firms Project because they had seen the benefit in terms of higher quality and profits. One group said they would “definitely” continue to use practices such as plowing and pruning but both groups said they would have to stop applying practices that required inputs if the cost increased beyond what they could afford. One group reported that, in the future, they would take a more proactive approach to obtaining inputs and knowledge by visiting the research center for seeds, pesticides, and learning about updated practices. Potato Farming Continuing productivity-enhancing practices: Participants in both of the two focus groups with project-supported potato farmers said they would continue to use the production practices promoted by the Firms Project. However, both also said that if input costs increased too much they may have to stop using some of them (i.e., better seeds, pesticides, and fertilizers). Market linkages: The Firms Project linked the potato clusters to SE Trading as an intermediary between the farmers and PepsiCo, the buyer of the chip stock potatoes the farmers produced. However, participants in one of the focus groups said the company had not provided required inputs in time and still had not paid for the product the farmers delivered two years ago. This incident has made the farmers wary of experimenting with new marketing connections. 44 Mango Farming Continuing productivity-enhancing practices: Participants in both focus groups with mango farmers (MGA farmers and other farmers) reported that the production and post-harvest practices they learned through the Firms Project were useful. However, they both cited the cost of applying the practices (e.g., fertilizers and sprays) as potential barriers to their continued use. One of the two groups mentioned that the cost of labor has increased because workers using improved equipment (e.g., clippers) demand higher wages. The other group mentioned that it is sometimes difficult to find the tools and equipment required to apply the practices in the local market. Both of these statements arose in the context of a discussion of factors that might prevent the farmers from continuing to apply the practices. Weavers The Firms Project supported the Islampur weavers only in facilitating market linkages. Market linkages: The one focus group with weavers provided very little information about the sustainability of the market linkages created through the Firms Project. The one summary quote that spoke to the issue cited the cost of attending exhibitions (and meeting new buyers) as the primary barrier to sustainability: “Since majority of the weavers are very poor, they may not be able to keep exploring markets for new requirements and participate in exhibitions on their own. Resultantly, this capacity of penetrating into new markets with quality production may reduce with the passage of time.” Sustainability of BEE Results The Firms Project only produced draft laws, analyses, frameworks, etc., and strengthened institutions through its BEE work and this is all it was responsible for in its Activity M&E Plan. All of the 11 individuals (excluding Firms Project staff) the evaluation team interviewed about BEE activities said that the project had been effective in producing its required outputs; adding to the body of knowledge, and advancing the reform process. However, only governments can implement laws and regulations, and it is these outcomes that are the meaningful measures of the sustainability of the BEE project’s results. While achieving sustainable BEE results is beyond the control or responsibility of the Firms Project, content analysis of two interviews with project staff, six interviews with provincial government “clients”, two interviews with consultants the project placed in government departments, and three interviews with BEE stakeholders identified nine conditions or factors that either impede or contribute to sustainable results. Figure 9 lists these conditions and factors and the number and percentage of the 13 interview respondents who mentioned them. 45 FIGURE 9: FACTORS AND CONDITIONS THAT IMPEDE OR CONTRIBUTE TO ACHIEVING BEE OUTCOMES Contributors/Barriers to Achieving Sustainable Outcomes Number And Percentage of Responses Political will is important 11 (85%) Establishing appropriate institutional frameworks (e.g. marketing boards) is essential 10 (77%) Negotiating with stakeholders and keeping them engaged and informed of progress during the planning and implementation process is important 9 (69%) The process of reform takes time so outcomes are not evident in the short-term, nor is it realistic to expect outcomes in the short-term 7 (54%) Clarity on organizational roles and strategies is important 6 (46%) The complexity of the steps required and resistance from existing structures and vested interests are barriers 5 (38%) Having public sector “champions” who were committed to moving things forward is essential. Prospects for achieving outcomes can flounder with the loss of individuals who were highly committed to change 5 (38%) Additional capacity building support and “handholding” may be required to implement change 5 (38%) Frequent turnover of high-level Pakistani government officials (e.g. at minister or secretary levels) may lead to a loss of institutional memory, change in priorities, and loss of momentum and thus delay or derail progress 4 (31%) Interviews with stakeholders provided examples of some factors that could influence the likelihood of achieving outcomes in specific reforms. For example, one stakeholder the team interviewed reported that agricultural market reform is complex in the sense that there are powerful vested interests in the current marketing system. The Firms Project discussion with the Karachi market committee indicated that reform would have to address any jobs that would be lost as a result of reform. Another stakeholder intimately familiar with the situation believed that reforms at SMEDA would be difficult because the organization was not convinced of the necessity and members of the SMEDA board didn’t agree on the appropriate role for the organization. Key members of the Firms Project’s staff reflected on other factors that could have improved the prospects for the sustainability of BEE work. These included:  More intensive and sustained (“incessant and regular follow-up”) engagement with governments and emphasizing the importance of reforms, when applicable, to achieving results in other work.  More outreach with communicators and dissemination of messages on reform to the broader community of stakeholders and donors to engage them in advocacy.  Ensure handover and complementarity with other programming. For instance, they cited the Punjab Enabling Environment Project (PEEP), which began in early 2014, as an example of an opportunity for USAID to deliberately build on the results achieved under the Firms Project.  Donors could better promote the importance of agricultural market reform and coordinate their work on this topic. 46 FINDINGS FOR EVALUATION QUESTION 4: GENDER Evaluation question #4 (Gender): In what way were gender issues incorporated and/or addressed by the project interventions and what lessons can be incorporated in future programs? The Firms Project’s original technical proposal defined ambitious goals for engaging women and specific women-focused activities.67 Fourteen months after the project’s inception, however, USAID revised its strategy to focus attention on countering extremist influences and recovering from the floods in 2010. To implement this strategy it asked, among other things, that the Firms Project remove gender indicators and women-oriented activities and the project complied.68 Therefore, this section examines the evaluation question in two distinct periods, the pre-modification period from April 2009 to June 2010 and the post-modification period from June 2010 to November 2014. Pre-Modification Findings The project’s technical proposal, embedded in the contract with USAID, emphasized the importance of addressing gender issues. “[The] contractor must demonstrate a keen awareness to gender issues in Pakistan with a particular emphasis on the barriers and constraints in pursuing income generating opportunities in formal/informal settings in targeted sectors and 20 districts.”69 The original project design, as outlined in the technical proposal, included a detailed strategy for identifying women’s issues and incorporating women into sector development activities, business development services, and business enabling environment work.70 The proposal’s results framework included three women-focused KRAs: 1.6: “women entrepreneurship and skills development supported”; 2.3: “service provided to women entrepreneurs increased”; and 3.4: “business environment for women improved” and three indicators with specific targets for women. The proposal also included a set of activities in each of the women-oriented KRAs and a detailed year one work plan for implementing women-focused activities. The implementation strategy called for a highly consultative approach with women’s stakeholder groups to identify women’s needs and constraints to aid in designing specific activities. The staffing plan also specified hiring a gender specialist. The first project Activity M&E Plan (April 2010) included three women-specific “scorecard” indicators with targets:71  Number of work plans identified specifically targeting women's economic participation in selected sectors (target=1) 67 Empowering Pakistan: Firms: A Task Order Proposal under the GBTI II IQC. Chemonics International Inc., October 14, 2008. 68 Personal communication with USAID staff and USAID/Pakistan. Modification 4. Contract No. EEM-I-00-07-0008-00 Task Order No. 7. July 13, 2010. 69 USAID/Pakistan. Contract No. EEM-I-00-07-0008-00. May 14, 2009. 70 Empowering Pakistan: Firms: A Task Order Proposal under the GBTI II IQC. Chemonics International Inc., October 14, 2008. 71 The Firms Project: Project Award Monitoring Plan FY2009–FY2013. April 2010. 47  Number of women participating in capacity building programs (target=450)  Value of program budget allocated to meaningful change for women in business (target=25 percent) Furthermore, the indicator for change in employment in project-assisted sectors included an overall target of a 10 percent increase in employment for men and 25 percent for women. In accordance with its work plan, in year one the project consulted extensively with the private sector, women business owners, chambers of commerce, and others to identify the factors contributing to and inhibiting women’s economic participation. The draft gender strategy the project submitted to USAID in August 2010 noted that: “During the first nine months of the Firms Project hundreds of discussions were conducted with private sector companies, women business owners and female university students in Sindh, Punjab and Balochistan to understand the factors contributing to women’s economic participation.”72 These activities were intended to inform the final gender strategy and activity planning. The project’s first annual report in 2010 documented progress on a number of women-focused activities including (amongst others) gender sensitization training for Pakistani government officials and project staff; training in gender mainstreaming for district economic development strategies; conducting an awareness seminar for female mango farm owners; and supporting women-owned businesses from Multan, Karachi, Islamabad, and Lahore and connecting them to export markets with Pakistan Handmade under the Market Readiness Program.73 Over a period of nine months in year one (prior to the modification) the project also consulted with women’s stakeholder institutions (universities, associations and chambers of commerce)74 to better understand the issues of women and women-owned businesses and provide services that responded to identified needs.75 Post-Modification Findings The project’s draft gender strategy and project staff during a briefing with the evaluation team reported that USAID instructed the project to cease all national-level activities except Pakistan Handmade in August 2010. Under this instruction, the project was allowed to continue women-focused activities only within the mango sector and other sectors that USAID may approve in the future.76 Other than the modification itself, which did not contain such prescriptive language, the evaluation team was not able to find written evidence of an actual “directive” or instruction. However, after the modification only one general reference to women remained in the project’s SOW: “Recognizing the important role that women can play in stabilization, increasing competitiveness and investing a higher percentage of 72 Firms Project. Gender Strategy: Increasing Women’s Economic Participation (draft). August 2010. 73 USAID/Pakistan: Firms Project. Annual Progress Report – 1: July 30, 2010. 74 Including, but not necessarily restricted to, Women’s Chambers of Commerce and Industry (Sindh, Karachi Islamabad, Lahore), IBA Sukkur, IMSciences, Sardar Bahadur Khan Women’s University, and the National Association of Business, Professional, and Agricultural Women (ABP&AW). 75 USAID/Pakistan: Firms Project. Annual Progress Report – 1: July 30, 2010. 76 Firms Project. Gender Strategy: Increasing Women’s Economic Participation (draft). August 2010. 48 resources in health and education, the [Private Sector Development] PSD component will emphasize expanded incorporation of women into economic activities”.77 The 2012 Activity M&E Plan contained no targets for women and the two modified technical proposals produced after the modification did not reference specific activities engaging with or benefitting women.78 The draft gender strategy identified opportunities for engaging women under the restricted statement of work. These included, for example, building on activities under the Market Readiness Program; using social networks to identify prospective participants in sector-based programs; focusing on additional female mango farmers in technical assistance activities; and others. However, subsequent annual reports indicate that the project did not implement most of the activities outlined in the gender strategy and identified in the pre-modification work plan. Since the modification, the project has implemented a few discrete women-focused activities (e.g. gender sensitivity training) that addressed women’s participation or specifically focused on women. Interviews and focus group discussions with processors, farmers, and business owners revealed that women played no substantive role in the mango processing, weaving, agricultural implements, peach, potato, or fisheries sectors. However, the project’s M&E data on job creation reported an increase in women’s employment in the fresh mango value chain – from 47 FTE jobs in 2009 to 124 FTE jobs in 2014 (a 164 percent increase) relative to a 28 percent increase among men (from 1,162 to 1,484 FTE jobs). The project’s M&E data also reported 20 additional jobs for women among five project-assisted knitted garments manufacturers for which data existed. Overall the project reported training 9,342 individuals, 292 (3 percent) of whom were women. Figure 10 illustrates the share of men and women trained in various topics in the mango, garment, and date value chains, the only value chains with relevant data. The project trained the greatest number of women in mango value addition (pickles, chutney, preserves, etc.), an exclusively women-oriented activity. In the garment and date value chains, the greatest number of women received training in firefighting and first aid in the garment sector (associated with WRAP certification)79 and food safety management in the date sector (associated with HACCP certification). The project’s fourth annual report re-introduced a gender indicator: “proportion of women participants in USG-assisted programs designed to increase access to productive economic resources”. The project reported on this indicator in its fourth and fifth annual reports. The fifth report presented a cumulative figure of 18 percent and an LOP target of 16 percent. This value seems quite high in light of other evidence presented in the evaluation. Project staff explained that the numerator includes 3,360 women in the peach sector who worked during the harvest season sorting fruit and preparing food for hired labor. Most, if not all, of these women were unpaid family members, as were the male members of the households. The project counted these women as “participants” in the context of the indicator. 77 USAID/Pakistan. Modification 4. Contract No. EEM-I-00-07-0008-00 Task Order No. 7. July 13, 2010. 78 The Firms Project: Project Award Monitoring Plan FY2009–FY2013. April 2010. 79 WRAP website. Accessed at: http://www.wrapcompliance.org/news 49 Women’s Roles in Project-Supported Value Chains The focus group with project beneficiaries and interviews with supported SMEs provided some insights into the roles women do and can play in the value chains. Findings include: Mangoes: Of the three mango processors the evaluation team interviewed, one reported engaging women in grading/sorting mangos, one said that women could play a role in sorting/grading once he installed an automatic grader, one reported that women packed mangos in boxes, and two reported that women collected fallen fruit for drying and pickling. The two focus groups with mango farmers confirmed the findings from the interviews. Both groups reported the same roles for women, illustrated here with a quote from one of the groups: “Women do only one single job at mango farms - to collect all those mangos dropped down due to heavy storm or rain. These women collect these mangos cut and dry them into powder form and sell it in local market for household usage in the kitchen.” Dates: All five date processors the evaluation team interviewed said that they contracted pitting work to women who worked in their homes as contract labor. Practices varied inside the processing facilities however. In three of the four plants the evaluation team visited, it observed many women sorting, cutting, and wrapping dates. The other two processors (including one the team did not actually visit) did not hire women. One reported that it was the culture in the factory, and the other said male supervisors did not like managing women, and that it was too much trouble to arrange for their safety (i.e., transporting them to and from work). Both of the two focus groups with date farmers revealed that women are involved in date harvesting, drying, and sorting at the farm level. Garments: The project initially engaged women entrepreneurs directly by introducing them to potential buyers in fashion shows. Men dominate the labor pool in the knitted garments industry, although all three manufacturers the evaluation team interviewed employed some women. The three interviewed firms reported that female employees made up about 5 percent (of 82 total employees), 10 to 12 percent (of 100 employees), and 25 percent (no total reported) of the total workforce. The firms reported that women worked as machine operators and croppers. One firm reported employing women to operate the new machines obtained through the Firms Project because “women are easier to train and manage.” Potato, Agricultural Implements, Fish Farms, and Weaving: Focus groups participants in these sectors and subjects of interviews with SMEs reported no women working in these sectors. Peaches: Participants in both of the two focus groups with peach farmers reported that women prepared food. The participants explained that in the context of peach harvesting this is not a trivial job, since farms hire a lot of itinerant labor and, as part of the employment contract, are required to provide food. 50 FIGURE 10: TRAINING BY VALUE CHAIN, TOPIC, AND SEX OF TRAINEE 0 400 800 1,200 1,600 Awareness Programme Mango MUAVAN Balanced Nutrients Application Extension Workers Training HACCP Training Integrated Crop Protection Management Mango Dormancy & Irrigation Management Mango Sudden death Syndrome Mango Value Addition for Women Pack House Operations Management Post Flood Management Pre & Post Harvest Management Tree Canopy & Pruning Fire Fighting and First Aid training Productivity enhancement Efficient marker making Production planning and management Material resource planning Export marketing QA system Modular Layout & QA Systems Time & Motion Study Date Palm Farming Practices Date Palm Disease Cure Management Date Palm Nursery Development Food Safety Management Export Marketing for Dates Products Fresh mango Garments Dates Number of people tained Men trained Women trained 51 CONCLUSIONS CONCLUSIONS ON RELEVANCE The Firms Project design was aligned well with USAID strategies and host country government priorities. The project’s highly consultative implementation strategy ensured relevance, particularly for BEE activities. The project’s BEE activities were largely demand-driven, i.e., requested by host country government bodies, which implies that the activities were relevant to clients’ needs. The consultative process during implementation ensured that the specific products and the design and content of the products were relevant. In its VCD activities, the project ensured relevance of its strategy in each value chain by consulting (during project design and in the early stages of implementation) with potential beneficiaries and sector stakeholders to identify the packages of support it would offer in each value chain. However, it was necessarily (because there were not individual “clients”) less consultative, and specific, about implementation strategies. For example, many recipients of equipment seemed to expect imported equipment instead of the locally manufactured equipment the project provided. Other examples include how the project implemented market linkage activities and the types of markets on which it focused. Although it did not come out specifically in the findings, the project implemented activities largely through short-term consultants. For instance, it hired consultants to implement most BEE services, conduct value chain analyses, and train and provide technical assistance to VCD beneficiaries. This may have been one factor that explains the project’s ability to quickly adapt its implementation approach to maintain relevance through several changes in strategy and activities imposed by USAID and Pakistani governments. It adapted quickly to changes in the geographic areas on which it focused and the consequences for the value chains the project would support, dictated by USAID in 2010. It also quickly took up management of the grants program to support reconstruction in the Malakand Division in 2010 and many requests from the GoKP for BEE support in 2013. CONCLUSIONS ON EFFECTIVENESS The project met or exceeded its targets for creating jobs and attracting private sector investment. It also most likely met its targets for the value of sales and exports although ambiguity about net versus gross values and the treatment of shipping costs call into question the validity of reported values. Furthermore, the qualitative evidence links changes in quantitative indicator values to project activities. The project overstated the value of exports compared to a conventional understanding of the concept (i.e., included the cost of shipping) and this probably distorted reported values in the fresh mango value chain. The project’s cumulative total value of exports exceeded the LOP target by USD 6.8 million. The fresh mango value chain produced USD 8.5 million in cumulative exports. Therefore, if shipping costs accounted for less than 80 percent of the reported cumulative value of fresh mango exports (which seems very likely), the project would have met its LOP targets even if the value of exports did not include shipping costs. Deliberately or otherwise, the fresh mango value chain built on and benefited from the work of other donors in the sector. Several beneficiaries and stakeholders commented on the effectiveness of mutually supportive interventions. AusAID’s previous work in the mango sector focused largely on production, and primed some farmers’ interests in marketing to take advantage of increased production and quality. The Firms Project’s primary emphasis in the value chain, processing facilities, built marketing capacities. 52 Several interview subjects and focus groups mentioned UNIDO’s current work assisting with marketing, i.e., intensive hand-holding through the exporting process. The project’s analysis-driven value chain approach identified appropriate packages of interventions in each value chain to address relevant constraints and capitalize on opportunities. For example, the overall focus on infrastructure, human capacity, and market linkages appears to have been an effective approach for strengthening beneficiary businesses in the processing and manufacturing value chains. The consultative process the Firms Project used to implement BEE and the quality of consultants it provided contributed to effectiveness by promoting ownership, averting potential resistance, producing quality products that contributed to the knowledge base, and developing implementation strategies to facilitate progress beyond the project’s engagement. CONCLUSIONS ON SUSTAINABILITY BEE: Sustainability in the context of BEE activities implies that the project’s work contributes to enacted reforms after the project ends. The demand-driven nature of the BEE work and the consultative process the project employed in developing BEE products enhances prospects for sustainable results by ensuring that the support addressed host country government priorities and needs and by building ownership and commitment. Nevertheless, sustainability in the sense of lasting reform is out of the project’s control. Some BEE interventions have progressed more than others towards implementation, largely because of provincial government commitment. The best prospects for sustainable results include PPP in KP, where the institutional framework was already in place for the PPP; the KP tourism policy, which is expected to be launched in November or December 2014; and the Change Management Unit (CMU) and policy reforms in KP, which will be implemented under the Integrated Development Strategy. Prospects for sustainability of the rest of the BEE activities are less certain and will depend on emerging host country government priorities and commitment. VCD: In the context of the project’s work with value chains, sustainability of results will depend on the willingness and/or ability of beneficiaries to continue investing in processing and productivity enhancing technologies and practices; their ability to sustain, and establish new, market outlets; and access to resources to reinforce training. Additional investments will include repairing or replacing tools and equipment, purchasing inputs, and renewing certifications. Table 5 summarizes conclusions about the prospects for sustainable results among project-supported enterprises in each project-assisted value chain. 53 TABLE 5: PROSPECTS FOR SUSTAINABLE RESULTS IN PROJECT-SUPPORTED ENTERPRISES BY VALUE CHAIN Value Chain Conclusions About Prospects for Sustainability Fresh mango processing Mango processors are large farmers with substantial financial resources. They have demonstrated the willingness and ability to make substantial investments in developing their capacity to export directly and some have made additional investments to upgrade and repair the facilities obtained through the Firms Project. They voiced their intentions to continue renewing their HACCP certifications, which will require training. The new processing enterprises appear to have developed relatively sustainable market linkages in the short time they have been involved in direct exporting and also have substitute market outlets that will reward them for quality. There is good reason to expect that most of these enterprises will be sustainable businesses. Threats to sustainable results include sustaining market connections through disruptive events such as the fruit fly, the (not fully tested) viability of sea shipments to high-end markets in Europe, and the ability to identify and reach alternative markets that are less demanding. Unreliable electricity supply also substantially increases costs and threatens the financial viability of the project-supported enterprises. Mango farm enterprises The project has successfully demonstrated the value of productivity-enhancing interventions to increasing mango quality and production in a cost-effective manner. The cost of applying the practices, including labor costs, may inhibit sustainability but most farmers seemed more concerned with rising costs than with current costs. Access to markets that reward quality could enhance the prospects for sustainability. Date processing Date processors are also enterprises with substantial financial resources. They have also demonstrated their willingness and ability to invest in technologies and practices to enhance their access to markets. The three project-supported processors have established and durable links to export markets and HACCP certifications obtained with project assistance have opened new markets. There is every reason to expect that they will continue as viable businesses beyond the life of the Firms Project. The equipment necessary to help the processors implement the largest part of the project’s date sector strategy – the coolers – have not been in place long enough to know whether they will effectively improve processors’ competitiveness. The fact that all had been renting cold storage to access these markets suggests that the overall strategy is sustainable. However, unreliable electricity supply substantially increases costs and threatens the financial viability of the project-supported interventions. Date farm enterprises The project has successfully demonstrated the benefits of productivity-enhancing production practices and improved drying technologies and practices. Beneficiaries reported being willing and able to invest in continuing the practices but with lower cost equipment when possible. Replacing expensive items will be challenging. The evidence suggests that farmers will continue using productivity-enhancing production practices and technologies and improved drying methods. In fact, farmers are already investing in increasing their capacity to dry dates. Garments Project-assisted garment manufacturers were well established businesses with the financial capacity to adopt project-promoted interventions. They all recognized the benefits and have made additional investments to enhance these benefits. There is good reason to expect that project-supported manufacturers will sustain, and expend on, results. 54 Value Chain Conclusions About Prospects for Sustainability Peach Project interventions have successfully demonstrated the benefits of productivity-enhancing technologies and practices. Farmers are most likely to continue the relatively low cost practices of plowing and pruning. Their willingness or ability to continue using improved inputs will depend on their cost. There is good reason to expect that farmers will continue to apply at least some of the project￾promoted practices and thus sustain project results. Potato Project interventions have successfully demonstrated the benefits of productivity-enhancing technologies and practices. Farmers are most likely to continue employing relatively low cost practices. Their willingness or ability to continue using improved inputs will depend on their cost. There is good reason to expect that farmers will continue to apply at least some of the project￾promoted practices and thus sustain project results. Much will depend on the success of the private sector intermediary model to facilitate cost effective and reliable access to inputs, services, and markets cost and this is still somewhat in question. Weaving Prospects for sustainable results appear bright. The only potential threat appears to be the cost of sustaining connections with distant retail market outlets. Ag implements Project interventions have demonstrated the benefits of improved production and management practices to increasing production efficiency, improving quality, and expanding sales. Beneficiaries demonstrated their willingness and ability to make the necessary investments and did not mention the associated costs. There is good reason to expect sustainable results for project￾supported manufacturers. Hotels All of the 239 project-supported hotels were operating prior to the conflict and floods. With project support to rebuild and refurnish the hotels and improve management capacity there is every reason to expect they will return to being sustainable commercial operations. Risks to sustainability include future conflicts or other factors that disrupt tourist activity in the Swat District or unduly delay recovery. Fish farms All of the 22 project-supported fish farms were operating prior to the conflict and floods. With project support to rebuild and re-equip the fish farms and improve management there is every reason to expect they will return to being sustainable commercial operations. Threats to sustainability include a recurrence of conflict or future damaging floods. CONCLUSIONS ON GENDER The original proposal, the contract, the budget, the initial indicators, and the initial work plan showed a project with ambitious targets for engaging women and specific implementation strategies for doing so. At the outset, not only was gender well integrated into plans for sector development, but there were interventions addressing the particular barriers women faced in accessing services and in the challenges women faced in employment and engaging in business. The project appears to have implemented its women-focused activities as planned during the first 14 months, including an extensive consultative process to help guide the design of specific women-oriented activities. However, the change in USAID strategy that removed most women-focused activities from the statement of work put an abrupt end to activities specifically focused on women. From a promising start, it is not possible to determine what the project could have accomplished. 55 RECOMMENDATIONS Additional support to Firms Project activities: Some of the Firms Project’s activities are probably not sustainable without further support. When contemplating future support to project-supported value chains, USAID should consider the suggestions of the well-reasoned VCD exit strategy developed by the Firms Project.80 Many of the following sector-specific recommendations draw on the material in the exit strategy validated by the evaluator’s findings. Since sustainability and scale depend critically on demonstration and emulation, USAID should disseminate project research and activity documents and actively seek institutions to carry the work forward. These could be other USAID-funded projects (e.g., projects working on agricultural and livestock market reforms), or indigenous institutions. As part of this effort, USAID should seek opportunities to promote project successes in forums with industry stakeholders and donors. Recommendations specific to project-supported activities include: Date processing: The Firms Project’s exit strategy concludes, and the evaluators concur, that it would be wise to provide sufficient technical support to project-supported date processors to confirm that their cold storage units are operating correctly. The fact that all three project￾supported processors reported renting cold storage to extend their seasons prior to the Firms Project suggests that they are already familiar with the (low-end domestic) markets the strategy opens for them and need little additional support on that front. An additional season of technical assistance to ensure that project-installed equipment is operating correctly seems a small price to pay to solidify USAID’s substantial investment in date processors and to test the demonstration effect necessary to achieve scale. Date farming: This evaluation concluded that the Firms Project’s investments in date farming reduced losses and improved quality, thereby increasing income to project-supported date farmers. Furthermore, date farming enterprises seem willing and able to continue, and expand their use of, project-promoted practices. Beyond linking project-supported enterprises to institutions for additional support and promoting the success to encourage emulation, little additional support is probably necessary. Fresh mango processing: The 15 project-supported fresh mango processors are new businesses established by the Firms Project. Most seem to have taken advantage of project support to gain direct access to export markets. The fruit fly incident during the 2013-14 season derailed the project export strategy to some extent, and an additional season of support may be warranted to solidify results and enhance prospects for successful demonstration and emulation to take the results to scale. To this end, if the activities are still operating, USAID should look to the AusAID￾funded work in the mango sector as well as the Agribusiness Support Fund for additional assistance to project-supported processors. 80 USAID Firms Project Exit Strategy for Value Chain Development (VCD) Programs. July 2014. 56 Mango farming: Lack of capital and limited access to markets that reward quality appear to be the primary constraints to sustainability and scale of project results among mango farmers. Solving these problems is not a small task and, unless USAID wishes to commit to a sustained engagement, is probably best addressed by seeking other partners to carry the work forward, e.g., if they continue past current funding, AusAID or the Agribusiness Support Fund, or indigenous institutions. Passage of agricultural market reforms may help project-supported farmers gain access to markets that reward them for quality, but this is not likely to occur in the short term. Knitted garments: Beyond disseminating project research and reports and promoting project success to encourage emulation, there is little USAID needs to do to sustain results in this sector. Islampur weavers: Short of disseminating project reports and work and promoting the work to local institutions and donor-supported projects to maintain and enhance market linkages, there is little USAID needs to do in this sector to ensure sustainable results. Agricultural implements: Project interventions in this sector seem to have produced meaningful results. Although the evaluators concluded that results were likely sustainable in project-supported manufacturers, the exit strategy report suggests that some manufacturers have yet to receive the full package of equipment and training - especially in lean manufacturing techniques, of which the two interviewed manufacturers spoke highly. USAID should consider assessing the current status of support to these manufacturers and completing the project’s commitments in these areas to further extend results. Peach and potato farming: Short of seeking to link project-supported peach farmers to other institutions or donor-supported projects, there is little USAID needs to do to sustain results for this value chain. PIFMS: Although the PIFMS pilot is far from mature, it is unclear what USAID could do to further support the effort. Private-sector financial interests will ultimately determine sustainability and perhaps USAID could continue to support the effort by facilitating interaction among the players. Business enabling environment: The Firms Project made significant contributions to the body of knowledge necessary to support reforms aimed at improving the business enabling environment for SMEs and other enterprises. Host country government commitment is necessary to move the reforms forward to enacted legislation and actual reform. Other than a few results that are most likely sustainable (i.e., KP-PPP, KP tourism policy, the KP Change Management Unit, policy reforms in KP which will be implemented under the Integrated Development Strategy, and agricultural policy reform in Sindh), USAID could enhance prospects for sustainable results by nurturing champions and continuing to work with federal and provincial governments to push results forward. The GoP’s 2025 vision document also emphasizes the importance of policy reforms promoted by the Firms 57 Project. These include policies to support private sector growth and investment, PPP, increases in manufacturing and agricultural competitiveness, and agricultural market reforms.81 Defining outcome indicators: In the future, USAID should review reporting procedures more carefully to ensure that they conform to detailed indicator definitions and the spirit of the indicator. The Firms Project did not claim that changes in its reported indicators could be rigorously attributed to project activities. In fact, in many cases, its M&E procedures reflected a practical approach to plausible attribution, i.e., establishing a baseline against which to compare indicator values throughout implementation. However, in some instances, the project reported on indicators in a manner that substantially overstated results, the most egregious example being counting shipping costs as part of the value of a sale. Not only did the practice overstate results, it also made it difficult to determine the effectiveness of the project’s strategy in activities that targeted export markets and where project￾supported enterprises paid for shipping, e.g., fresh mango and date processors. 81 Government of Pakistan. Planning Commission. Ministry of Planning, Development & Reform. Pakistan2025: One Nation – One Vision. No publication date. Accessed at: http://pakistan2025.org/wp-content/uploads/2014/08/Pakistan-Vision-2025.pdf 58 LESSONS LEARNED Lessons on effectiveness of approaches: The evaluation findings support conclusions about the relative effectiveness of the business models, development approaches, and implementation strategies the project employed. Distinct approaches or strategies include the overall VCD approach of addressing infrastructure, human resource, and market linkage constraints; the choice of working at the upper end (processors) or the lower end (small farmers) of selected value chains; a focus on export markets in some value chains; engaging the private sector as a market intermediary; sea shipment trials; and a highly consultative process.  Almost all project beneficiaries appreciated the project’s consultative approach to designing and implementing activities. The approach was effective in ensuring relevance and promoting ownership. It contributed to effectiveness by ensuring that interventions were relevant to beneficiaries’ needs. Where the consultative process did not cover as broad a range of decisions about implementation (e.g., the choice of specific equipment, the choice of markets for some value chains) beneficiaries voiced more dissatisfaction with implementation and often described their dissatisfaction in the context of effectiveness, e.g., equipment that did not perform well, a focus on difficult-to-reach markets when more accessible and lucrative markets existed.  Building SMEs’ capacities to access foreign markets was effective when project-supported SMEs were already selling to those markets (i.e., knitted garments and dates). In these cases, the qualitative data suggest that simple interventions like certifications (WRAP and C-TPAT for garments and HACCP for date processors) contributed most to increasing access to foreign markets. The project adopted a riskier strategy in the fresh mango value chain by trying to build the capacities of new processing businesses to access high-end markets that were physically difficult to reach cost-effectively with existing export practices. The project effectively extended the shelf-life of mangos by processing them closer to production areas. However, the fruit fly problem in the 2013-14 season disrupted the export market and prevented a good test of the marketing strategy. Consequently, the effectiveness of the strategy to access high-end European markets in a cost-effective manner by sea is still in question.  Engaging private sector market intermediaries may seem like a good approach to improve the chance of sustainable results. However, projects must be careful when promoting such relationships to ensure they do not merely replicate existing relationships between farmers and traders, which lock farmers into what they often view as exploitative relationships. The thin evidence on the relationship between project-supported potato farmers and SE Trading is mixed. The Firms Project reports the relationship between potato clusters and SE Trading as successful. However, potato farmers who participated in focus groups said they were disappointed in the result and would continue to sell in their traditional manner through agents. SE Trading also reported some problems working with project-supported potato clusters. Some of the reported problems may reflect a phase in the relationship where each of the partners is learning about the expectations of the other and how to work together. Consistency of Strategy/implementation: It is obvious but deserves emphasis that a consistent strategy and approach is more likely to yield positive results than an inconsistent approach. This is most evident in the case of gender results for the Firms Project. The project seemed on track to achieve its ambitious results for women with activities that specifically identified relevant opportunities and focus activities for women. The change in strategy in 2010 sidetracked many of these activities and, consequently, the project generated few tangible results for women. 59 Support for Marketing: Especially for beneficiaries with limited marketing experience (e.g., fresh mango, agricultural implements), building the capacity to navigate complicated exporting processes is important. Introducing farmers to potential buyers is only the first step; it may be necessary to provide intensive support throughout the process of developing contracts with buyers, arranging local transportation and storage, managing the preparation of shipping containers, contracting with shippers, and managing any issues when the product arrives at its destination. There is evidence that the Firms Project provided at least some of this type of support, at least in the fresh mango value chain. Many producers, however, cited the need for more assistance in marketing. Anticipate predictable bottlenecks: The infrastructure intensive interventions, especially date processing, were plagued by delays in receiving and installing equipment. The consequence was a very late start in implementation that limited the time available for beneficiaries, and the project, to iron out problems and learn how to use the equipment effectively. This was particularly evident in the date processing value chain where beneficiaries received their coolers only in the final season of the intervention. To some extent these problems were predictable and avoidable. Contractor, USAID, and customs requirements often extend the time it takes to procure equipment and, when the risk of a lengthy procurement process is high, choosing another approach may be better than leaving beneficiaries with “half a machine” and little time to integrate new processes into their operations before project support ends. The choice to purchase equipment from a local vendor, while laudable from the perspective of sustainability and building local capacity, poses risks of delays when the equipment is complex and local manufacturers have no experience producing it. Timing/sequencing: Careful planning to ensure timeliness and a good first experience are particularly important in agricultural value chains that work with risk averse small farmers with little resilience. Delays in delivering inputs or payments can set back trust and progress significantly, and also substantially harm intended beneficiaries. It is important to recognize the very real potential for procurement delays and the time required to develop new technologies and commercial relationships and plan accordingly. Gender: In the Pakistan context, it is particularly difficult to engage women in many development activities. The experience of the Firms Project emphasizes a lesson that has emerged from other evaluations conducted in Pakistan:82 projects that try to fit women into activities that are not specifically women-focused are not often effective. Meaningfully engaging and empowering women requires a deliberate approach that identifies specific opportunities to engage and empower women and focuses on those activities. 82 EGA Meta-analysis, WEEB evaluation, Entrepreneurs evaluation. 60 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 ANNEX 2: DATA COLLECTION INSTRUMENTS FIRMS PROJECT EVALUATION INTERVIEW GUIDE KP PUBLIC PRIVATE PARTNERSHIP Background on the PPP and Firms’ Support 1. Please explain when the work began on setting-up the PPP framework? 2. Please explain what the PPP framework is designed to support and it main areas of focus? 3. In addition to an emphasis on tourism investment, what are the other key sectors and infrastructure investments the PPP will work on. How are the principles of risk sharing and full cost recovery for the private sector selected in the PPP? 4. Would you please explain exactly how the Firms project has been helping you plan and set up this PPP framework? Usefulness of the visions and implementation plans 5. Would you explain how the Vision Documents and Implementation Plans that the Firms’ project helped develop have been used in planning the PPP framework? Did these help? 6. How well did you find that these plans address what is needed for a successful public-private partnership that facilitate investments, especially in tourism? Institutional and regulatory reforms required and considered 7. What will be the institutional and regulatory forms that will be required in order to move forward the PPP? 8. Which of these are currently under consideration? Current status of the PPP framework 9. Please explain the current status of this framework? Is it being implemented as yet? 102 FIRMS PROJECT EVALUATION INTERVIEW GUIDE AGRICULTURAL IMPLEMENT MANUFACTURERS Business profile 1. Can you briefly describe your equipment manufacturing business: type of products do you produce, where and to whom do you sell, what is the size of your operation? 2. Since you were once the president of the Agricultural Implements Manufacturers Association, could you briefly describe Pakistan’s agricultural equipment industry—(typical size of operations, markets served, level of technology, production costs and productivity). 3. Would you please explain any major changes, if any, you have seen in this industry over the past 2 or 3 years? 4. In your experience, what are the major constraints or challenges that the agricultural implements industry faces in Pakistan if it wants to grow and be more competitive? Interaction with Firms Project 5. How did you learn about and become involved with the Firms Project? 6. Can you describe the types of support you received from the Firms Project (Probe for specific types of training, equipment, market connections, certifications)? 7. How did you decide what type of support you would accept from the Firms Project (try to understand if it involved a consultative process)? As appropriate 8. What kind of training did you receive, who was trained, and what impact has the training had on your business (probe for specifics)? 9. What kind of equipment did you receive and what impact has the equipment had on your business (probe for specifics, e.g., less waste, lower production costs, better quality)? 10. What types of certifications did you obtain and what impact have they had on your business (probe for specifics, e.g., new markets)? 11. How did the Firms Project help connect you to markets? Did these connections lead to sales? Sales 12. How have your sales changed in the past 5 years (probe for volume, types of equipment, domestic/export markets, etc.)? 103 Relevance of Firms support 13. Overall, was the support you received from the Firms Project the kind of support you needed to grow your business? Explain. 14. Are there other kinds of support you would have liked to have received from the Firms Project? How would these types of support have helped your business? 15. Could the Firms Project have provided the support in a better way? Explain. SMEDA 16. Are you aware of the Small and Medium Enterprise Development Association (SMEDA)? 17. What is your impression of SMEDA? 18. Have you ever received any assistance from SMEDA? Explain. The Future 19. What is your vision for your company in the next 5-10 years? 20. What are the main risks or barriers to achieving this vision? 21. Do you think the support you received from the Firms Project will help you achieve this vision? Explain. 104 FIRMS PROJECT EVALUATION INTERVIEW GUIDE AGRICULTURAL MARKETS REFORM Agriculture Coalition-role 1. Would you please explain the background and role of the Agriculture Coalition? 2. Who is involved in the Agriculture Coalition? Which agencies do you work the most closely with? Private sector marketing constraints 3. Would you please explain what you understand as the main regulatory constraints that have limited private sector investment and better access to markets? (i.e., restraints the system of marketing through government notified areas and market committees 4. Would you please explain what are the priority agricultural marketing reforms that, in your view are needed to support private sector horticulture marketing? Knowledge of the Firms Project 5. Would you please explain your understanding of what the Firms’ project has been trying to do reduce marketing distortions and strengthen private sector agricultural marketing? 6. Has the FIRMS team worked with you in the past or currently to seek your advice and set these priorities? 7. We understand that the FIRMS project has worked on revised Agricultural Marketing Legal Frameworks (for Sindh, Punjab, Balochistan and KP). Would you please explain the current status of these Agricultural Marketing Legal Frameworks? Have these yet been adopted with regulations drafted? What is actually in effect right now? 8. What are the next steps for these new frameworks to actually be put into effect? About how long is this likely to take? Private sector marketing progress and results to date 9. In Punjab, would you explain the progress-to-date in allowing private sector markets to be set￾up? (i.e. wholesale markets, farmers markets, collection points and consumer/retail markets). Where does this stand now? Is this moving ahead? 10. As far as you know, do there remain any “notified areas” or market committees? 11. To your knowledge, what progress has there been in setting-up an agriculture marketing board? 12. Are the wholesale markets run by government market committees now being turned into companies that could be financially sound and keep operating in future? 13. To date, has permission been given to the private sector to set up horticulture markets? 105 Difference made by Firms Project 14. Would you please explain what difference you expect the Agriculture Markets Reform efforts being made by the Firms project have made so far? Could potentially make once they are in full effect? 15. In your view were there things the Firms project might have done differently to ensure better and faster market reform? Please explain. 106 FIRMS PROJECT EVALUATION INTERVIEW GUIDE DATE PALM RESEARCH INSTITUTE Brief profile 1. Can you briefly describe the role and work of the Date Palm Research Institute 2. Does our institute work on research topics such as: improving nursery management, improving date yields, disease control, pre and post-harvest handling methods Engagement with the Firms Project 1. Would you please explain what you know about the Firms project and its work with date farmers and date processors? 2. How has the Firms Project engaged with the Research Institute? (For example, has the project sought your advice or other assistance in the work it has done with farmers and date processors?) Please explain 3. If, under the Firms Project, the Institute has provided training or technical advice to date farmers and date processors, would you explain the specific areas of support you provided for date farmers? For date processors? 4. Have you provided any advice to the Firms project regarding equipment requirements for date production and handling? For date processing? Major challenges for dates sector and Firms support 5. In your view, what are the main challenges that the dates sector faces for producing quality products for better serving domestic markets? 6. In your view, what are the main challenges that the dates sector faces for producing quality fresh dates for regional and international markets? 7. What is your view on whether Firms support is in fact providing the kinds of support needed to address these major challenges? Please explain. Firms support for date farmers 10. What are the major challenges that date farmers/producers face in order to improve quality and production and obtain better prices to their products? 11. The Firms Project has provided trainings to about 800 farmers on farm management, disease cure. From what you have seen, what changes, if any, have date farmers made to their practices after this training? 12. Would you please explain what you believe are the conditions needed for improved farm production of more and better quality dates with fewer losses? 107 13. How do most date farmers sell their dates? Do they sell to wholesalers/buyers or directly to processors? Do any farmers sell directly to the market? 14. About what percentage of the dates that farmers harvest are typically wasted or spoiled before they are able to sell them? 15. Are there standards and grades for date quality? Would you please explain how these work and the challenges farmers face in meeting standards? 16. In your opinion, what difference, if any have solar tunnel dryers and plastic tables made to the quality and acceptability of dates produced? (Have you had the chance to visit farms where these are being used? (For example, has this equipment allowed farmers to sell more and better quality dates and obtain better prices?) 17. How affordable are these solar tunnel dryers and plastic tables for farmers who are not assisted by the project? Have you seen other farmers purchase these after seeing their use by project￾assisted farmers? Firms support for date processors 18. The Firms Project has provided support to three date processors for cold storage equipment and for back-up generators. These cold storage units have been installed very recently (August 2014). Would you please explain what difference, in your opinion, consistent cool temperatures will make to date quality and reduced spoilage for shipping to export markets. 19. What else is needed, in your opinion, for the production and processing sides of the date business to ensure that the industry grows and is even more competitive in international markets. Assessment – what might have worked better? 20. In your opinion, how well has the support of the Firms project served the needs of the farmers and processors it has worked with? 21. If you had been in charge of planning and carrying out the Firms project with dates farmers and the dates industry, what would you have changed? Would you explain two or three things that you would have done differently? The Future 22. Would you please explain what, in your view, are the most crucial changes required in dates production and processing for the future if the industry is to become competitive in high value markets? 108 FIRMS PROJECT EVALUATION INTERVIEW GUIDE DATE PROCESSORS Brief business profile 1. Can you briefly describe the nature of your business: your activities, size of your processing operation, main markets, employees, products you sell, length of time in operation. Engagement with the Firms Project 2. When did you first start working with the Firms project? 3. When did you sign an Infrastructure Upgrading Agreement with the Firms project? 4. What kind of equipment did you invest in with the support of the Firms project? 5. What was the cost-sharing arrangement with the Firms project for this equipment? How much did you have to pay? 6. Was all of the equipment delivered and installed on time. Explain. 7. Was the equipment what you expected and needed? Explain. 8. Is the equipment you received now in operation? If so, for how long have you been using it? Were you able to use it for the latest season? (ending late August?) 9. Has the Firms project help you with identifying new markets? What kinds of market linkages or development support did you receive under the Firms project? 10. Did you participate in any training provided by the Firms project? Please explain. 11. Was the training useful to your business? Explain. How, if at all, did the training affect your business? 12. In your experience, how well would you say that the Firms project delivered on your agreement with them– was it what you had expected and needed to build your business? Please explain. Processing and marketing 13. For how long has your business been processing fresh dates for domestic markets? 14. For how long has your business been processing fresh dates for export markets? 15. We understand that your involvement with the Firms project has been for a relatively short time. However, I’d like to ask you some questions about how your business may have changed over the past three years since the 2011 season so we can compare how it was before you started working with the Firms project and now. I’ll ask a number of questions about how you processed and marketed dates in 2011 and then we’ll compare that to how you do things now. 2010/11 (before Firms) Now (2013/14) Reasons for difference (probe for connection to Firms) 109 Domestic marketing 16. What sort of standards (processing, grades, packaging, etc.) did direct buyers require? 17. What sort of standards (processing, grades, packaging, etc.) did wholesale buyers require? 18. About what percentage of the dates you processed did you have to sell at a lower grade than necessary? Why? 19. How did you process/pack dates for domestic markets (describe the process from the point where dates arrived at your facility to when the dates left.) 20. About what percentage of the fresh dates you processed were wasted or spoiled before you were able to sell them? 21. What were the main challenges you faced selling dates for domestic markets? Export marketing 22. What countries did you sell dates to? 23. What sort of standards (processing, grades, packaging, etc.) did direct buyers require? 24. What sort of standards (processing, grades, packaging, etc.) did wholesale buyers require? 25. (If graded) About what percentage of the dates you handled did you have to sell at a lower grade than necessary because of poor post-harvest handling? 26. How did you process/pack dates for export markets (describe the process from the point where dates arrive at your facility until they leave) 27. About what percentage of the dates you handled were wasted or spoiled before you were able to sell them? 28. What were the main challenges you faced selling dates to export markets? Employment 29. How many full-time employees (disaggregated by men and women) did you have? 30. How many part-time employees did you hire during the date season? About how many days of part-time labor did you employ? (disaggregated by men and women) Market linkages 31. Has the Firms project been able to help you build links with buyers in international markets? If so, how? Have you been able to keep up these links? Have these connections actually helped you sell to new markets? Expand sales to markets you already were selling to? 32. What are the main challenges you face selling to export markets? 33. What are the main benefits of selling to export markets? 110 Volume of Dates Processed 34. From how many farms in your area do you purchase dates? 35. Do you know if the Firms Project provided any training or tools to these date farms to help them increase yields or quality of dates? 36. If yes, have you noticed any changes in the quantity or quality of dates you buy from these farmers and has that affected your business? Explain. Policies and regulations affecting marketing of dates 37. What laws or regulations in Pakistan affect your date processing and selling business? Please explain. 38. Have you been affected by any recent changes in laws or regulations governing date marketing? Explain. 39. Would you say that, for your business, the “environment” for conducting business in Pakistan has improved, stayed the same, or become worse over the past 5 years? Please explain. Services offered by SMEDA 40. Do you know about Pakistan’s Small and Medium Enterprise Support Agency (SMEDA)? Have you ever asked for or received any help from them to further grow your business? If yes, have there been any changes in the types of quality of services SMEDA offers to SMEs in the past 5 years? Explain. Assessment – what might have worked better? 41. In your opinion, how important has your engagement with the Firms Project been in improving the competitiveness of your business: very important, somewhat important, not important? Explain. 42. If you had been in charge of planning and carrying out the Firms project with the Dates industry, what would you have changed? Would you explain two or three things that you would have done differently? The Future 43. Have you made any other investments in your processing business that were not supported by the Firms Project? 44. Do you plan to keep the equipment you installed with the support of the Firms Project? What might cause you to stop using the equipment? 45. How do you expect your business to grow or change in the next five years? What factors could make that change take place faster and which factors could slow it down? 46. What are the three main barriers you now face being able to compete successfully in dates export markets? 111 FIRMS PROJECT EVALUATION INTERVIEW GUIDE SUPPORT TO PARRSA 1. First we’d like to understand the history of the Firm’s Project support to PaRRSA. a. How did the assistance begin - did PaRRSA request assistance from the Firms Project or did the project offer assistance? b. How did the Firms Project decide what sort of assistance to provide – did PaRRSA request specific types of assistance or did Firms and PaRRSA decide together? 2. What types of assistance did the Firms Project provide to PaRRSA? 3. Was the assistance relevant to PaRRSA’s needs at the time? Explain. 4. How, if at all, did the assistance from the Firms Project help PaRRSA? 5. Are the effects of the Firms Project assistance temporary or did the project have lasting effects on PaRRSA’s capacity? Explain. a. Are any of the specialists the Firms Project placed in PaRRSA still there? If so, are they being paid by PaRRSA? 6. How would you describe the quality of the assistance the Firms Project provided? 7. In your opinion, are there things the Firms Project could have done better? Explain 112 FIRMS PROJECT EVALUATION INTERVIEW GUIDE KNITTED GARMENT MANUFACTURERS Brief business profile 1. Could you start by telling us very briefly a little about your business; how long have you been in business, what do you produce and how much, where do you sell, how many people do you employ? Types of support received from the Firms Project 2. Can you tell us when you started working with the Firms Project and what types of support have you received from the project? Markets and marketing challenges 3. What are the major challenges you face selling to export markets? 4. Which markets require compliance certifications? Do you have any certifications? Have you had difficulties obtaining these certifications? 5. If you’ve received compliance certifications, how have they helped your business? 6. What are the primary challenges you face in your production processes and how do these affect your business, particularly your competitiveness in domestic and export markets. Probe for impacts on production costs, timeliness of deliveries, getting orders, and meeting buyers’ requirements if necessary Training and productivity 7. What specific kinds of training and technical assistance have you received? Prompt for improving machine layouts and converting to a modular system if necessary? 8. Who provided this training? 9. How has this affected your business? Prompt for improved efficiency or reduced waste if necessary? Technology upgrading 10. What kinds of new or improved technology or technological processes have you adopted because of the Firms Project assistance? Prompt for better stitching machines and CAD if necessary. Did you receive financial assistance to introduce these new or improved technologies to your manufacturing processes. Linkages to markets 11. Has your engagement with the Firms Project helped you set up any direct linkages with markets in the US or EU? If so, please explain how this has happened. 113 12. Have any of the market contacts you made when you participated in the Magic Market Weeks trade shows in the US resulted in serious inquiries or orders as yet? Impact on sales and employment 13. Since you started working with the Firms Project, have you had to hire any more employees in order to expand production? Please explain and quantity the change in employment. Probe to determine attribution to Firms. 14. Would you please explain whether your sales to export markets have changed since participating in the Firms program? To which markets have these sales been made? How did Firms support help you expand sales? 15. What would you say your % increase in sales has been? What were your annual sales before you started participating in the Firms project? Assessment – what might have been improved upon? 16. Has the Firms Project support met your expectations? Explain. 17. What suggestions do you have about what could have been improved upon in the type or level of support you have received from this project. The Future 18. How do you see the future for your manufacturing businesses production and sales? Please explain, for example, what you expect to accomplish in the next three years-by October 2017? 114 FIRMS PROJECT EVALUATION INTERVIEW GUIDE KP PUBLIC PRIVATE PARTNERSHIP Background on the PPP and Firms’ support 1. Please explain when the work began on setting-up the PPP framework? 2. Please explain what the PPP framework is designed to support and it main areas of focus? 3. In addition to an emphasis on tourism investment, what are the other key sectors and infrastructure investments the PPP will work on. 4. How are the principles of risk sharing and full cost recovery for the private sector relected in the PPP? 5. Would you please explain exactly how the Firms project has been helping you plan and set up this PPP framework? Usefulness of the visions and implementation plans 6. Would you explain how the Vision Documents and Implementation Plans that the Firms’ project helped develop have been used in planning the PPP framework? Did these help? 7. How well did you find that these plans address what is needed for a successful public-private partnership that facilitate investments, especially in tourism? Institutional and regulatory reforms required and considered 8. What will be the institutional and regulatory forms that will be required in order to move forward the PPP? 9. Which of these are currently under consideration? Current status of the PPP framework 10. Please explain the current status of this framework? Is it being implemented as yet? 115 FIRMS PROJECT EVALUATION INTERVIEW GUIDE KP TOURISM Background on KP tourism and Firms support 1. Would you please explain the mandate and role of the KP Tourism Department? 2. Which are the priority topics or areas of focus for the Department? 3. Would you please explain what kind of technical assistance the Firms’ project has provided to your Department? Challenges for KP tourism development 4. In your view, which are the development challenges that your department faces in trying to revitalize and develop tourism. Please explain those that you believe are most critical to address? Policy development 5. How has the Firm project helped you with Policy Development? Please explain. 6. Would you please explain the composition of the Tourism Working Group and its role in providing input on policy development, priority-setting and plans? 7. How did the input of the four regional focus groups help inform the Draft Policy? 8. What is the current status of the Draft Tourism Policy? 9. If this policy has been adopted, how is it helping your department in its planning and priority setting? Draft tourism laws 10. Has the Firms project provided technical support to your department in revising and updating Tourism Laws? Please explain. PPP framework 11. We understand that the Firms’ project helped you develop a possible PPP framework? What is the current status of that PPP framework? 12. What would you describe as the main features of this PPP framework? 13. What do you expect to achieve if this framework is implemented? 116 SWAT tourism web portal 14. We understand that the Firms Project helped set up a Tourism Web Portal that was handed over to the management of the Tourism Corporation. As far as you know, how has this portal helped the industry to date? Next steps 15. Would you mind explaining how you are planning to move forward on policy implementation and making sure the PPP framework is set up and working well? The Future 16. We understand that tourism growth opportunities are very significant. What is your picture of what tourism will look like three years from now-by October 2017? Assessment of Firms’ support 17. How well would you say that the assistance provided by Firms met your expectations? On a scale of 1 to 5 where 1 is not very useful and 5 is highly useful, where would you rate this assistance. 18. In your view, were there ways that this support provided by the Firms project could have been improved upon? 117 FIRMS PROJECT EVALUATION INTERVIEW GUIDE KP BOIT Mandate and role of KP-BOIT and Firms support 1. Can you explain the mandate and role of the Board? When was KPBOIT established? 2. In your experience what are the main challenges and constraints in trying to promote investment and trade in KP? For example, how do you go about identifying possible investment areas for potential private investors? Engagement with the Firms Project 3. We understand that the Firms Project commissioned KPMG to provide technical support to KPBOIT. In what areas did the Firms Project provide support? 4. How did you determine the type of support you needed and what the Firms Project would provide? KP-BOIT’s organizational development 5. Would you please explain in which areas the Firms Project is helping you strengthen your organization and management structures? For example, how did the Firms Project help KP￾BOIT’s Human Resources Dept.? 6. Do you expect that this strengthened organization and management will help you to carry out your role as KP’s apex promotion body? How? 7. How do you expect that this strengthened organization and management will help you strengthen public-private sector linkages? Identifying possibilities for private sector investment 8. Can you explain how KPBOIT goes about identifying potential areas for private sector investment and seeking out potential investors? 9. Has the support of the Firms’ project helped you to identify the main sectors and opportunities for private sector investment? How? 10. In your opinion, which of the twelve sector feasibility studies showed the greatest possibilities for investment? Investment promotion 11. Can you explain how you promote investment opportunities to potential investors? For example, do you carry out general promotional and communications activities? What are these? 12. How do you actually target, attract and actively pursue potential investors? 118 13. Has the support of the Firms Project helped you with your promotion strategy and activities? How? Providing investment services 14. We understand that it a fairly new setup and still under development, but what are the investment services that KP-BOIT is now, as of Oct. 2014, able to provide to the private sector? 15. Which services do you intend to offer in the future? 16. Has the support of the Firms Project helped you in your anticipated service delivery to date? How? Website development 17. We understand that the Firms Project has helped KP-BOIT develop a website. Has this website helped you achieve your objectives? How? What has the response been to date by stakeholders and potential investors? Investment-friendly reforms 18. Which, in your view, are the most important topics that should be addressed for investment￾friendly policy and regulatory reform? Relevance and effectiveness of Firms support 19. To what extent was the support you received from the Firms Project aligned with your needs? Explain. 20. Are there areas of support you would have liked to receive that were not provided by the Firms Project? Explain. 21. How well did the Firms Project deliver the support, what was the quality of the support you received? Explain. 22. Are there things the Firms Project did particularly well? Are there things the Firms Project could have done better? Explain. 23. Is there anything at all you would like to add either about Firms’ support to KP-BOIT or about KB-BOIT’s plans? 119 FIRMS PROJECT EVALUATION INTERVIEW GUIDE KPMG Background on role of KP-CCI 1. Would you please explain the role of the KPCCI, particularly with reference to its role in working towards greater competitiveness of industry and tourism sectors in regional markets? 2. In your view, what are the most important constraints or challenges that now affect the competitive position of KP’s industry and tourism sectors. How would you compare these challenges with those in 2009-2010? Firms technical support activities and where these stand 3. Would you please explain exactly what support the FIRMS project provided to the KPCCI? 4. Is our information correct that this work was short-term Technical Assistance over a two￾month period to develop both a Vision Document for Tourism and one for Industry? If so, would you explain to us what the industrial sector covers? Outputs of firms’ technical support to KPCCI 5. Would you please explain how the work on the Vision Documents took place and what these documents include? Do these vision documents in fact include implementation plans? 6. We understand that the Tourism Vision includes providing for policy development and a PPP until 2025? How these visions and plans have been used 7. Did these documents meet your expectations as a guide for future development and support for future development? Please explain? 8. Since these planning documents have been handed over to the government of KP in January 2011, in what ways have you seen these plans being moved forward over the past more than three and one-half years? What actions have been taken as far as you know? 9. Is there anything else you would like to add about the technical assistance provided by the Firms project or what is needed for the Tourism Vision to be realized? 120 FIRMS PROJECT EVALUATION INTERVIEW GUIDE MANGO PROCESSORS Brief business profile 1. Can you briefly describe the nature of your business: how long you have been in business, the size of your farm and processing facility, the products you sell? 2. When did you sign an Infrastructure Upgrading Agreement with the Firms project? When does this IUA end? Engagement with the Firms Project 3. What kind of equipment did you purchase under the IUA with the Firms Project and how was it different than what you had prior to the Firms Project? Probe for details. 4. In your experience, how well would you say that the Firms project delivered on this agreement – was it what you had expected and needed to build your business? Explain. Probe for timing of support, appropriateness of equipment, etc. 5. Did you receive any training or other support for mango production through the Firms Project? Explain. Mango production 6. Has the quantity of mangos you produce changed since you started working with the Firms Project? If yes, why? Probe for connections to Firms Project support? 7. Has the quality of mangos you produce changed since you started working with the Firms Project? If yes, why? Probe for connections to Firms Project support? 8. Have post-harvest losses changed since you started working with the Firms Project? If yes, why? Probe for connections to Firms Project support? 9. Do you purchase mangos from other farms or markets in your area for your processing business? 10. Have you faced any particular challenges in terms of quantity or quality when buying mangos from others? 11. Have you noticed any recent changes in the quantity or quality of mangos you are able to purchase for processing? Explain. Probe for connection to Firms training of mango growers. Mango marketing 12. What types of mangos or mango products do you sell? Probe for fresh, processed, or both. 121 13. How do you sell mangos? Probe for directly to buyers, through agents/mandis, other. 14. What are the characteristics of the different markets for mangos? Probe for prices, standards, grades, payment terms, etc. 15. Has the Firms Project linked you to new buyers? Explain. 16. Has your work with the Firms Project changed how you sell mangos, i.e., the types of markets you can access, the products you sell, quality, prices, etc.? Explain. Employment 17. How many full-time employees (disaggregated by men and women) do you have? 18. How many part-time employees do you hire during the mango season? (disaggregated by sex and production/processing) 19. Has the number of men and women you employ changed since you started working with the Firms Project? Explain. Probe for attribution to the Firms Project. Policies and regulations affecting marketing of mangoes 20. Have laws or regulations in place in Pakistan negatively affected your business? Explain. 21. Would you say that, for your business, the “environment” for conducting business in Pakistan has improved, stayed the same, or become worse? Please explain. Services offered by SMEDA 22. Do you know about Pakistan’s Small and Medium Enterprise Support Agency (SMEDA)? Have you ever asked for or received any help from them to further grow your business? If yes, have there been any changes in the types of quality of services SMEDA offers to SMEs in the past 5 years? Explain. Assessment – what might have worked better? 23. In your opinion, how important has your engagement with the Firms Project been in improving the competitiveness of your business? 24. In your opinion did the Firms Project do the right things to increase the competitiveness of the mango sector? Explain. 25. In your opinion, was the way the Firms Project worked with mango producers and processors appropriate and effective? Explain. 122 The Future 26. Do you plan to keep the packing house (processing lines, cold storage units and all the equipment in operation)? 27. How do you expect your business to grow or change in the next five years? What factors could accelerate that change and which factors could slow it down? 28. What are the three main barriers you face being able to compete successfully in mango export markets? 123 FIRMS PROJECT EVALUATION INTERVIEW GUIDE MANGO RESEARCH STATION Brief profile 1. Can you briefly describe the role and work of the Mango Research Station? 2. What type of research does the station do to support the mango sector? Major challenges facing the mango sector 3. In your view, what are the main production challenges the mango sector faces? Probe for varieties, production practices, inputs, … 4. In your view, what are the main marketing challenges the mango sector faces? Probe for differences in domestic and export markets, infrastructure, quality, prices, and impacts of marketing act. Engagement with the Firms Project 5. What do you know about the Firms Project and its work with mango farmers and processors? 6. How has the Firms Project engaged with the Research Station? Probe for consultation, equipment, training, etc. 7. What is your view on whether the Firms Project provided the kinds of support needed to address challenges in the mango sector from both production and marketing perspectives? Explain. Effects of Firms Project 8. Do you have any evidence that mango farmers that have worked with the Firms Project are producing more or better quality mangos? If so, how? Probe for changes in production, post￾harvest handling/losses, quality, etc. 9. Do you have any evidence that mango processors that have engaged with the Firms Project are better able to access markets? Explain. What has been the impact on prices, quantities, types of markets reached, etc.? 10. In your opinion, has the support from the Firms Project made beneficiaries more competitive in export markets? 11. The Firms Project is relatively small. In your opinion, can it have an impact on the competitiveness of the entire mango sector? If so, how? If not, what would this take? 12. In your opinion, did the Firms Project do the right things to support the mango sector? Explain? 13. In your opinion, was the way in which the Firms Project supported mango producers and processors appropriate? Explain? 124 FIRMS PROJECT EVALUATION INTERVIEW GUIDE PARRSA SPECIALISTS (Background note: KP Government established PaRRSA to drive the recovery effort and to provide a single window facilitation point for Fed/Prov governments, donors and partner organizations). Background and role of PaRRSA and Firms’ support 1. Please explain the background of PaRRSA, when it was established and why. Please describe its main role. 2. What are PaRRSA’s main areas of focus in its efforts to “drive” recovery efforts and serve as a single coordinating and information window? 3. What are the main challenges that PaRRSA has faced in undertaking this important recovery and economic development role? 4. Please explain the kinds of support that PaRRSA requested and also has received from the Firms project. Firms’ technical assistance activities and where these stand 5. Are Firms’ two specialists and four associates in the Economic Growth Department and one specialist in the Communications Department are still in place? 6. Policies Developed and their Implementation: Would you please explain which policies and have been developed and are being carried out with the help of the Firms’ technical support? a. Economic Growth Projects: would you please explain which specific rehabilitation and Economic Development projects PaRRSA has planned and implemented to date with the support of Firms’ specialists and associates in the Economic Growth Dept. ? What are the key components of PaRRSA’s agenda? 7. Communications Strategy: how has the Firms project helped with communications? What are the main topics and who are the target audiences? 8. In your opinion, how well has the Communication strategy and methods succeeded in engaging people in what PaRRSA is trying to do? How would you say that Communications activities have actually reached PaRRSA’s potential beneficiaries for recovery and development activities? 9. Progress Review System/ database: what is the status of the support that the Firms project has provided to help PaRRSA develop its PRS database? Has this database been set up and handed over to PaRRSA? This was scheduled to be handed over in June 2014. 10. Has this happened? Who has been using it so far and for what purposes? Even though this is quite recent could you please explain how PaRRSA and its Steering Committee intend to use it? 125 11. How engaged was PaRRSA’s management in developing this database? Is it fully operational by now? How has PaRRSA used this so far? Specifically, how has this been used for monitoring the progress of PaRRSA led recovery, rehabilitation and development schemes? PaRRSA’s own assessment: capacity and limitations 12. Please compare PaRRSA’s current organization in supporting recovery efforts and in communicating about these with the situation when you began and then received assistance from the Firms,’ project beginning April 2010 13. In which specific areas do you feel that your organization’s capacity has significantly improved because of Firms’ technical support? 14. Would you please which areas you have identified where PaRRSA continues to have capacity gaps that you would like to address for greater organizational effectiveness? Usefulness of Firms’ technical assistance 15. We understand that PaRRSA had to plan and carry out projects that would lead to quick recovery results but also support economic development. In your view, which were the projects that Firms’ specialists helped you design that are now showing or at least starting to show promising results? Please explain 16. Which of these projects you have just noted would you not have been able to do at all? As well? Had Firms’ specialists not been in place to provide technical assistance? 17. Thinking about Firms’ assistance, in what ways could their support have been improved upon? Please explain. 126 FIRMS PROJECT EVALUATION INTERVIEW GUIDE SE TRADING Background 1. History of interaction with Firms Project on potato? 2. Role in Firms Project potato VC? SE Trading model and interaction with Firms 3. How do you work with farmers and with PepsiCo a. Financial arrangements with farmers – when contracted, price determination, payment terms b. Role of cluster leader? 4. Impact on farmer income/production/prices (net) 5. Jobs? How could Firms have done better? 6. In what ways do you think the Firms Project could have been more effective in its interaction with potato farmers? a. Did the Firms Project provide the right kind of support? b. Are there other things the Firms Project have done to get even better results c. Did it provide the support in the right way? Explain. 7. Have you faced any challenges connecting Swat potato farmers to markets? Explain. 8. What is your vision for the future of your work with potato farmers? a. What are the main threats to achieving this vision? SE Trading – Pilot Information 9. Can you describe how you became involved in the Pilot for Information and Financial Mobile Solutions. What is the role of SE Trading in the pilot? 10. What services does the system provide now? What services in the future? 11. How did you decide which services to provide? 127 12. How does the system work? a. Is registration required? b. Do users have to pay for the service? c. What is the source of the information you provide (e.g., market prices)? d. Who is the financial partner for financial mobile solutions? 13. Can you describe the role of the service centers? Are these in operation now? What is your role in setting up the service centers? 14. How, if at all, does working with the pilot support your other activities? 15. Do you believe the business model is sustainable? If so, how will it be financially sustainable? 16. Do you have plans to extend the service to different regions or sectors? Explain 128 FIRMS PROJECT EVALUATION INTERVIEW GUIDE SE TRADING Background on role of SMEDA and FIRMS’ support 1. Please explain the mission and role of SMEDA in supporting SMEs? 2. What are SMEDA’s main areas of activity in helping SMEs to develop? 3. In your experience what are the main constraints to SME development in Pakistan? 4. What were the specific topics or areas where SMEDA requested technical assistance form the Firms’ project? Please explain these Firms’ TA activities and where these now stand 5. We are interested in finding out what difference Firms’ TA has meant to SMEDA, it work and its effectiveness in supporting SMEs. We would like to discuss with you each of the technical assistance areas which the Firms’ project has supported and the results-to-date from this support. We would like to first look at this by topic-i.e. policy review, SME regulatory framework review, drafting of laws, organizational development and services, strategy development. 6. Policy Review: When was the SME policy most recently revised? How does this policy define SMEs? Who (which agencies) have the main responsibility to carry out which elements of this policy? 7. How well, in your view is this policy being carried forward or implemented? What are the constraints or major challenges in implementing this policy? 8. Exactly how did the TA FIRMs provided help you strengthen your policy? 9. Regulatory Framework: Would you please explain in which ways (if any) the support of the Firms project helped with identifying needed revisions to SME laws and the regulatory framework? How did you use the draft for the SME Amendment Ordinance and the SME Development Act? 10. In your view, how useful and realistic were the proposed revisions to the SME laws and regulatory framework? 11. At what stage are these now? Have the laws and regulations been approved? Are these being implemented? What steps are remaining 12. SMEDA’s Organization and Services: We understand that the Firms project worked with you to look at how your organization could best respond to the service demands of SMEs. In which ways, if any, was this helpful to you in planning your services and how to structure the organization to meet these demands? 129 13. How have you used the Organizational Development and Services Strategy, Implementation Plan and performance measures up until now? 14. What changes have you been able to make already in your services or how these are delivered? 15. What changes do you intend to make and approximately when will these be put into effect? 16. SMEDA’s “Cohesive Strategy”: We understand that the Firms’ project has helped you assemble a cohesive strategy that includes suggested policy and regulatory revisions as well as strategic directions and solutions to improve service delivery. 17. Would you please explain how this strategy will, once it is being implemented, help support SME development and the services you will be able to provide to SMEs? SMEDA’s five-year business plan - carrying it out 18. What period of time does this Business Plan cover? How are you using it at the moment to guide your activities and services? 19. What are the major constraints you face in moving forward (carrying out) this Strategy and Business Plan? 20. What are the next steps you have planned? SMEDA’s assessment: capacity and limitations 21. In your own opinion, how would you assess SMEDA’s existing capacity to help SMEs? 22. In your view, how effective is SMEDA now in influencing SME policy and regulations and their implementation? 23. In your view, how effective is SMEDA now in reaching SMEs and meeting their support needs? Usefulness of Firms’ technical assistance 24. How would you rate the usefulness of Firms’ assistance on a scale of 1-5 where 1 is “not very useful” and all and 5 is “highly useful”? 25. Thinking about Firms’ assistance, in what ways could their support have been improved upon? Please explain. 130 FIRMS PROJECT EVALUATION INTERVIEW GUIDE FORMER COP 1. What were the most important changes in the SOW associated with Modification 4 and what effect did they have on implementation? a. Geographic areas b. Choice of value chains (did it contribute to shift away from marble for instance) c. BDS – it seems that this took on less importance post-modification. Is this true and did the modification contribute to that? What role did Firms play in providing BDS? d. Gender 2. Did you receive any other communication from USAID regarding women-focused activities? 3. The project strategy for scaling up in the mango processing, date processing, and knitted garment activities in particular, depend on a demonstration effect or “emulation.” Do you have a sense from your time here about the effectiveness of that strategy and the factors that might threaten the strategy? 4. From your time as the COP, did you form any ideas of what might have worked better that could inform future similar projects? 131 FIRMS PROJECT EVALUATION INTERVIEW GUIDE SUPPORT TO GOKP VIA PLANNING & DEVELOPMENT (P&D) DEPARTMENT Background on the assistance to GoP via P&D by Firms Project 1. It is our understanding that the Firms project has provided support to the GoP via P&D in a number of areas: Change Management Unit; P&D strengthening, Bureau of Statistics Assistance, Earned Value Based M&E System; Health Sector Reforms and Education Sector Reforms; Urban Sector Legal and Regulatory Reforms. 2. Would you please explain when this support began and when it is scheduled for completion? 3. What is the KP government’s goal in strengthening these six areas. Why were these the areas selected? 4. Given that this support from the Firm’s project began only about 4 months ago, what are the most important outputs of this support that P&D expects to see completed? 5. We would like to ask you about progress and plans in each of the six areas where the Firms project is providing support. Firms TA activities and where these now stand Change Management Unit 6. What have the three experts assigned to the Change Management Unit accomplished for the P&D? 7. How is the Change Management Unit now working as a unit within P&D to engage various departments in vision and the reform process? 8. Since the CMU is a very new unit, now that the basic institutional documents and workplan have been drafted, how will P&D use these to set priorities and move forward? 9. Would you provide some examples of the types of reforms that have been suggested to improve service delivery? Decisions on “sectors” to support 10. How were decisions made about which “sectors” were the priorities for certain reforms? P&D strengthening 11. The Firms project provided support for P&D’s institutional development. Would you please explain what the Firms project did for P&D. 12. In particular how did the Capacity Building Action Plan help you identify areas where capacity could be improved? Please explain 132 13. What has been P&D’s response to this capacity building document and to the draft Final Report provided by the Firms project? Bureau of Statistics 14. Would you please explain what the Firms project did for the KP BOS? 15. As far as you know, how does the BOS intend to use the institutional analysis and technical needs analysis that the Firms project supported? 16. How will P&D be able to monitor and influence the follow-up of this work? Earned value based M&E system 17. Would you please explain how you understand that earned-value based M&E frameworks can be used by departments involved in planning and implementing development activities? 18. How do you think these tools will be useful for managing your whole portfolio of development activities as well as activities and individual projects. 19. How have the reports on this topic been used by P&D? 20. We understand that training tools have been developed for Senior Managers so they can use this earned value Monitoring and Evaluation approach. Has this training been conducted? Health sector 21. How has the Firms project helped the Department of Health towards the development of a Provincial Health Policy? 22. What has the Firms project contributed towards the establishment of a Management Information System? 23. As far as you know, how has the Firms project helped in building a framework for training of nurses and paramedics? 24. How would you assess the capacity of the Health Department to move forward on the basis of these documents and frameworks? Urban sector legal and regulatory framework 25. We understand that the Firms project is providing technical assistance in drafting laws that reflect best practices in (i) Rental Law, (ii) Condominium Law, (iii) Land use and building rules and regulations and (iv) Mass Transit Systems 26. To your knowledge, has this work been completed? 27. As far as you know, how has this work been adapted to the context of KP? 28. P&D’s Assessment of KP Departments’ Capacity to Implement the Reform Package 29. What are the capacity challenges that P&D faces in trying to oversee and ensure that the desired changes (reforms) move forward in a timely manner? 133 30. How would you assess the capacity of the departments selected (that are important for service delivery and development) to actually implement the plans, frameworks, draft laws, etc.? 31. What kinds of support will these departments likely require, especially from the Change Management Unit? Next steps 32. Would you please explain the next steps that the P&D Department is intending to take to ensure that reforms move forward? 33. Which are the most important reforms that you will prioritize for completing by this time next year? Usefulness of Firms’ technical assistance 34. How would you rate the usefulness of Firms’ assistance on a scale of 1-5 where 1 is “not very useful” and all and 5 is “highly useful”? 35. Thinking about Firms’ assistance, in what ways could their support have been improved upon? Please explain. 134 ANNEX 3: LIST OF DOCUMENTS REVIEWED I. General Documents 1) Program Descriptions and Modifications:  Technical Proposals (3 documents)  Contract (1 document)  Contract Modifications (22 documents) 2) Work Plan:  Firms Annual Implementation Plan (AIP) (6 documents) 3) Quarterly & Annual Reports:  Quarterly reports (16 documents)  Annual reports (4 documents) 4) MSF, PMP, M&E Plans and other related documents:  EGA results framework (1 document)  Firms Activity M&E Plan s (9 documents) 5) Sectoral Evaluations:  Sector assessments and strategies (8 documents) 6) Project-generated studies and assessments:  Value chain impact assessment - Mango  Sales and labor multiplier study - Peach  Sales and labor multiplier study - Dates  Cost of production study – Agricultural Implements  Follow up cost of production study – Agricultural Implements  Impact assessment of the Malakand SME Recovery Assistance Program  Workforce development study – Karachi  Workforce development study – South Punjab 7) Fact Sheets:  VCD and BEE fact sheets (24 documents) II. List of Documents for VCD Component: 1) Project Related Reading Documents of Ag Implement Sector List of Documents:  Agricultural Implement sectors’ Need Assessment Report”  Technical sheet “One pager”  Sector Work Plan / Budgets  Quarterly and annual Progress Reports  Presentation of exposure visit and exhibition. 135  Cost of Production Study (report)  Report of consultants  Energy Audit Reports Administrative Documents:  Contracts with Beneficiaries  Contracts with Consultants  SOWs & TPARs (if needed)  Acknowledgement Forms Reference Material/Reports:  Findings of Baseline Survey  Capacity Need Assessment of Agricultural Implements Manufacturing Sector (USAID Firms Project)  Agricultural Machinery Industry in Pakistan: Present status and future prospects by University of Agriculture Faisalabad  Diagnostic Study Report on Agricultural Implements Industry by Raj Kumar Arora http://www.ediindia.org/DSR/KARNAL%20DS.pdf 2) Project Related Reading documents of Marble sector List of Documents:  Event Report - Sector Level Consultative Session  Marble sector’s strategy presentation  Sector Work Plan with Budget  Demand and Supply Gap Analysis Report - Marble and Granite Sector  Marketing Assessment Report of 12 Beneficiaries  Internal Audit Report of 06 Units by ISO Implementation Consultant  Lean Manufacturing Implementation Reports of 06 Units by Lean Manufacturing Consultant  Environmental Compliance units Assessment and Monitoring Reports of 08 units  Final Report - Assistance in Quarry Management Administrative Documents:  Assistance Agreements with 12 Beneficiaries  Contracts with local and international Consultants  Contracts with Sub Cons  SOWs & TPARs (if needed) Reference Material/Reports:  Marble Sector Strategy - PISDAC Project 136 3) Projects Related Reading Documents of Mango sector List of Documents:  Year four and Five Mango sector work plan (Presentation)  Technical sheet “One pagers”  Quarterly and annual Progress Reports  Reports and Presentation of exposure visit and exhibition  ST consultants (Reports, Presentations)  Events (Training/Non-Training) Booklets and presentations Administrative Documents:  MOUs and Agreements with beneficiaries  Contracts with ST Consultants  SOWs & TPARs (if required)  Sector Work Plan / Budgets and Gantt Chart Reference Material/Reports:  Mango Sector Capacity Assessment Report  MAUVAN Program Packinghouse Infrastructure Assessment Report 4) Project Related Reading Documents of Fruit and Vegetable Pulping Sector List of Documents:  Consultation with all value chain players of the fruit and vegetable pulping sector  sector’s strategy presentation  Sector Work Plan with Budget  Identification and selection of progressive pulp units in Punjab and Sindh  Sector Gap Analysis Report – Fruit and vegetable Sector  Profiling and Capacity Need Assessment of Pulping Units  Marketing Assessment Report of 5 Beneficiaries  Internal Audit Report of 05 Units by ISO/HACCP Implementation Consultant  Environmental Compliance units Assessment and Monitoring Reports of 05 units Administrative Documents:  Assistance Agreements with 06 Beneficiaries  Contracts with local and international Consultants  Contracts with Sub Cons  SOWs & TPARs (if needed) 137 Reference Material/Reports:  Presentation for Marketing of fruit pulp  Training on Product and process development of fruit and vegetable pulping  Training on Good manufacturing practices for fruit and vegetable pulping in line with international food safety standards 5) Project Related Reading documents of Garments sector List of Documents:  Training Report – Lean Manufacturing and Quality Systems  Training Report – Time and Motion Study  Training Report – Technical trainings  Garments sector’s strategy presentation  Sector Work Plan with Budget  WRAP – Report by Rajab Ali  CTPAT – Report by Adnan ul Hasan Administrative Documents:  Assistance Agreements with 18 Beneficiaries  Contracts with local and international Consultants  Contracts with Sub Cons  SOWs & TPARs (if needed) 6) Project Related Reading documents of Dates Sector List of Documents:  Dates sector’s strategy document  Export Marketing Strategy for Dates sector  Dates sector Program Presentation  Dates Sector Work Plan with Budget  Dates sector Factsheet  Reports and Presentations of STTA Consultants  Training Programs Presentations Administrative Documents:  Infrastructure Upgrading Agreements with 48 Dates sector Beneficiaries  Contracts with STTA Consultants  Contracts with Vendors and Subcontractors  Scope of Work for STTA consultants  SOWs & TPARs (if needed) 138 Reference Material/Reports:  Booklet for Flood Affected Date Palm Management  Business directory of Dates sector Beneficiaries  Business Plan for Commercial Cold Store for Dates 7) Project Related Reading Documents of Peach Sector Peach:  Sector Presentation  Pre and Post-Harvest Training Material  Peach Pulping Line Pre-Feasibility Report  Peach Pulping Line machinery installation and commissioning Report 8) Project Related Reading documents of Potato Sector Potato:  Sector Presentation  Pre and Post-Harvest Training Material 9) Project Related Reading documents of Islampur Weaving Sector Islampur Weaving:  Mobilization Work plan  B2B marketing training material 10)Project Related Reading documents of PIFMS Sector PIFMS:  Agri Trading report  Gap Analysis Mobile Money in Pakistan report  Market analysis and Price collection mechanism for GoKP Report 11)Project Related Reading Documents of KP-BOIT Sector KP-BOIT:  Legal Structure and Frame work  HR policies 139  Pre-Feasibility studies III. List of Documents for BEE Component: Sector Report Title Agriculture Agriculture Marketing Policy Framework, Punjab and Sindh Agriculture Marketing Policy Framework, Balochistan Agriculture Marketing Policy Framework, KP Agriculture Marketing Legal Framework, Punjab Agriculture Marketing Legal Framework, Sindh Agriculture Marketing Legal Framework, Balochistan Agriculture Marketing Legal Framework, KP Livestock Livestock Policy Review, Punjab and Sindh Livestock Legislative Review, Punjab Animal Welfare Act Quality Meat and Slaughter Act Livestock Policy, KP Livestock Policy, Balochistan Fisheries Fisheries Policy Framework Mines & Minerals Mines and Minerals Policy Recommendations Report, KP and Balochistan Board of Investment Analysis of Existing State of BOI South Waziristan Agency South Waziristan Agency Report Investment Promotion Council The Structural and Organizational Options Paper KPCCI Industry Vision Document Tourism Vision Document KP Tourism KP Tourism Policy Report Draft Tourism Laws Document Public Private Partnership Framework Document GB Tourism GB Tourism Policy Report, Draft Tourism laws Document PPP Framework Report PPP Law Document PaRRSA Progress review meetings between USAID and PaRRSA SMEDA Technical Assistance to SMEDA: Consolidated Summary Textiles Reform of Regulatory Framework: Cotton Ginning Sector 140 Sector Report Title KP Workplan M&E Portfolio Management Framework Project Management Framework PM Information Systems Framework Health Legislative Framework Literature Review Urban Sector Draft Physical Planning Act Draft Mass Transit System Act Draft Tenancy Act Establishing CMU CMU-Agenda and Structure Document CMU-Work Plan and Budget Document CMU-HR Requirements Document KP Reform Action Plan Document Strengthening of P&D Work Plan for New Inductions and Processes Document Technical Assistance matrix Document Capacity Building Action Plan Document Final Report Institutional Analysis of BOI Institutional and Technical Assessment Reports Final Report Urban Sector-Forman Christian College Grant Position Paper on "Water and Sanitation" Position Paper on "Real Estate Markets" Position Paper on "Governance" Position Paper on "Trade Friendliness" Position Paper on "Transport" Core Working Group Meetings Minutes-1st Meeting Core Working Group Meetings Minutes-2nd Meeting Core Working Group Meetings Minutes-3rd Meeting Core Working Group Meetings Minutes-4th Meeting 141 ANNEX 4: GETTING TO ANSWERS TABLE Evaluation Question Type of Answer/ Evidence Data Collection Data Analysis Methods Method Sources Sampling RELEVANCE / APPROPRIATENESS: To what extent was the project relevant as designed and did IP adjust design or implementation to maintain relevance in the face of changing priorities? Descriptive and Analytical  Document review  Key expert interviews  Document review – project design and activity documents, EGA results framework, USAID strategy  Key expert interviews with key stakeholders (Federal and Provincial government departments/ministries) and USAID and IP staff  Purposive sampling for key expert interviews Document evidence of relevance in design from activity design documents. Review IP reports for evidence of changes in implementation specifically to maintain relevance. Augment with data from individual interviews. EFFECTIVENESS: To what extent has the project achieved its stated objectives and what lessons can be learned about the effectiveness of the business models/development approaches/ implementation strategies adapted/adopted/developed by the project? Descriptive and Analytical  Document review  Data analysis  Key expert interviews  Focus group discussions  Document review – project design and activity reports, M&E reports  Analysis of project-collected M&E data if applicable  Key expert interviews with project staff (M&E) and selected beneficiaries/participants (e.g., trade/industry bodies, government agencies, chambers of commerce, SMEs  Focus group discussions with groups of beneficiaries (e.g., SMEs, farmers, artisans) as feasible or appropriate in each activity  Field visits to supported SMEs (particularly the date and mango VCC activities)  Purposive sampling for key expert interviews  Random sampling (when feasible) for focus group discussions Document review and interviews with project staff to document project-reported objectives, reported results, measurement/ reporting procedures and issues, and specific business models/approaches/strategies. Validate reported results and unanticipated results/consequences from interviews with participants/ beneficiaries, observation, and analysis of project data if applicable. Validate results and link to specific business models/approaches/strategies through individual interviews and focus group discussions. 142 Evaluation Question Type of Answer/ Evidence Data Collection Data Analysis Methods Method Sources Sampling SUSTAINABILITY: To what extent are project results likely to be sustained once the project assistance is formally concluded? The aspect will look into the likelihood of the replication and scalability of the project’s benefits after its conclusion. Descriptive and Analytical  Document review  Key expert interviews  Focus group discussions  Document review – project planning/strategy documents and progress reports  Key expert interviews with project staff, sector experts, government stakeholders, trade/industry bodies, chambers of commerce, beneficiary SMEs, institutional beneficiaries  Focus group discussions with groups of beneficiaries (e.g., SMEs, farmers, artisans) as feasible or appropriate in each activity  Purposive sampling for key expert interviews  Random sampling (when feasible) for focus group discussions Review documents to understand project’s strategy for sustainability, replication, and scaling up. Use individual interviews and focus group discussions to explore likelihood of sustainable results. Document examples of replication or scaling up. GENDER: In what way were gender issues incorporated and/or addressed by the project interventions and what lessons can be incorporated in future programs? Descriptive and Analytical  Document/data review  Key expert interviews  Focus group discussions  Document review – project planning documents and reports, gender strategy  Field visits to supported SMEs (particularly the date and mango VCC activities)  Key expert interviews with artisans, trade/industry bodies, chambers of commerce, SMEs.  Focus group discussions with cluster leaders (and others as appropriate)  Purposive sampling for key expert interviews  Random sampling (when feasible) for focus group discussions Document project planning with regard to gender issues in design and implementation. Use individual interviews and focus groups to gain an understanding of how the project addressed gender issues during implementation, whether the approaches were relevant/effective, and how they could have been more so. Field observations of respective roles of men and women in production and processing activities. 143 ANNEX 5: SAMPLING PROTOCOLS The evaluation team developed sampling protocols to select respondents for semi-structured interviews and focus group discussions with project beneficiaries. All samples reflect a two-stage design where the team selected a subset of beneficiaries based on convenience in the first stage and then, in the second stage, randomly selected subjects from the first-stage convenience sample. Table 6 describes the sampling protocols for each set of beneficiaries. TABLE 6: SAMPLING PROTOCOLS FOR INTERVIEWS AND FOCUS GROUPS Respondents Sampling Mango processors The Firms Project supported 15 mango processors in Sindh and in Punjab with infrastructure upgrades (Infrastructure Upgradation Agreement or IUA farms). To save time, the evaluation team visited processors and farmers in and around Multan in Punjab province only. Firms Project staff assured the team that processors and farmers in Sindh and Punjab were similar. The sample is entirely a convenience sample consisting of all three processors within a practical driving distance (1.5 hours) of Multan. Firms Project staff assured the evaluation team that the processors are typical of the eight assisted processors in Punjab. Mango farmers (GlobalG.A.P.) The Firms Project trained 26 mango farmers in Punjab in GlobalG.A.P. production practices (Mango GlobalG.A.P. Agreement or MGA farms). Eleven of the 26 were also IUA signatories. Firms Project staff assured the evaluation team that all would be willing to come to Multan for a focus group discussion. The team therefore randomized the list of 15 farmers who received only GlobalG.A.P. training, and starting at the top of the list, called farmers until it filled its quota for the focus group. Five farmers participated in the focus group discussion. Mango farmers Firms Project staff believed that all 223 farmers in Punjab trained in mango production would be willing to come to Multan to participate in a focus group discussion. The evaluation team therefore randomized the entire list and, starting at the top of the list, called farmers until it filled its quota for the focus group. Eight farmers participated in the focus group discussion. Peach farmers The Firms Project started working in the peach sector in Swat in 2012 with 440 orchards. The project included additional orchards in 2013 and 2014. However, in 2014, restricted access prevented the project from completing the training. The evaluation team therefore selected a sample from among the 888 farmers who had participated since 2012. Participating farmers are located across Swat and recruiters let each farmer decide whether he was willing to travel to the focus group site in Mingora. Eleven farmers participated in the first and focus group discussion and 11 in the second. The sample is likely skewed towards farmers who live closer to Mingora. The likelihood, direction, and magnitude of any self-selection bias are unknown. Potato farmers The Firms Project has been working with potato farmers in Swat since 2012 and included additional farmers in 2013 and 2014. The evaluation team randomly ordered the list of 52 farmers in the Miandam cluster that had participated since 2012 and, starting at the top of the list, called farmers until it filled its quota for two focus groups. Eleven farmers participated in the first and focus group discussion and 10 in the second. The sample is likely skewed towards farmers who live closer to Mingora. The likelihood, direction, and magnitude of any self￾selection bias are unknown. Islampur weavers The evaluation team randomly ordered the list of 47 project-supported weavers and, starting at the top of the list, contacted weavers until it filled its quota for the focus group. Ten weavers ultimately attended the discussion. Knitted garment manufacturers The Firms Project supported 19 knitted garment manufacturers in Karachi. The evaluation team randomly ordered the list of manufacturers and visited the three at the top of the list. Date farmers The Firms Project trained 48 date farmers in Sukkur and Khairpur in the Sindh Province. The evaluation team randomly ordered the entire list and, starting at the top of the list, contacted farmers until the desired number agreed to participate in two focus groups. Moderators conducted the discussions in the Chancery Guest House in Sukkur. Seven participated in first focus group and 11 participated in the second. 144 ANNEX 6: QUALITATIVE DATA COLLECTION BY VALUE CHAIN TABLE 7: QUALITATIVE DATA COLLECTION METHODS AND SOURCES BY VALUE CHAIN Value Chain Qualitative Data Collection Fresh mango value chain Interviews with 3 of 14 project-supported processors Interviews with 3 sector stakeholders Focus group discussion with two groups of mango farmers Date value chain Interviews with 3 of 3 project-supported date processors Interviews with 3 sector stakeholders Focus group discussions with two groups of date farmers Knitted garment value chain Interviews with 3 of 18 project-supported manufacturers Interview with one sector stakeholder Agricultural implements value chain Interviews with 2 of 18 project-supported manufacturers Peach value chain Focus group discussions with two groups of peach farmers Potato value chain Focus group discussions with two groups of potato farmers Interview with marketing intermediary Islampur weavers value chain Focus group discussion with one group of weavers Hotels Interviews with 3 of 239 hotel owners Fish farms Interviews with 3 of 22 fish farmers 145 ANNEX 7: COMPARISON OF USAID AND FIRMS PROJECT RESULTS FRAMEWORKS TABLE 8: USAID AND FIRMS PROJECT RESULTS FRAMEWORKS, 2009 USAID Results Framework Firms Project Results Framework AO 2: Improved conditions for broad-based economic growth Objective: Dynamic, internationally competitive SMEs with accelerated sales, investment, and employment IR 2.1: Increased income generating opportunities IR 2.1.1: Improved competitiveness of focus value chains KRA1: Enhanced competitiveness in project-assisted SMEs in targeted value chains IR 2.1.1.1: Strengthened market linkages in focus value chains IR 2.1.1.2: Improved technological innovation and management practices IR 2.1.1.3: Improved water management in focus chains IR 2.1.2: Improved workforce development systems IR 2.1.3: Improved access to financial services for SMEs IR 2.2: Improved economic governance IR 2.2.1: Improved fiscal discipline at federal and provincial level IR 2.2.2: Improved business enabling environment KRA2: Improved business enabling environment at the federal, provincial, and district levels IR 2.2.3: Improved analysis, policies, laws and regulations governing natural resources use and management IR 2.2.4: Effective civil society for oversight, engagement and advocacy 146 TABLE 9: USAID AND FIRMS PROJECT RESULTS FRAMEWORKS, 2013 USAID Results Framework Firms Project Results Framework DO 2: Improved Economic Status of Focus Populations Objective: Dynamic, internationally competitive firms developed to accelerate sales, investment, and employment growth IR 2.1: Improved Economic Performance of Focus Enterprises KRA1: Enhanced competitiveness in project-assisted SMEs in targeted value chains IR 2.1.1: Increased Access to Finance IR 2.1.2: Improved Skill Development and Job Placement IR 2.1.3: Increased Use of Modern Technology and Management Practices IR 2.2: Improved Business Enabling Environment KRA2: Improved business enabling environment at the federal, provincial, and district levels IR 2.2.1: Improved Ability to Develop and Implement Reform of Policies, Laws, and Regulations IR 2.2.2: Strengthened Private Sector and Civil Society Engagement in Policy-Making 147 ANNEX 8: PROJECT EXPENDITURES BY COMPONENT TABLE 10: PROJECT EXPENDITURE AND RESULTS BY VALUE CHAIN Value Chain Total Expenditure (USD) Value of Sales (USD) Value of Exports (USD) Jobs Created (FTE) PSI (USD) Dates 8,683,037 61,698 61,698 0 150,997 Agricultural implements 2,646,606 n.a. 31 166899 Knitted garments 3,302,649 6,159,369 6,159,369 287 430,100 Mango 23,223,965 8,752,097 8,519,553 922 2,207,804 Peach and potatoa 4,258,226 5,637,026 n.a. 1,167 1,855,527 Islampur weaving 727,457 67,883 n.a. 0 27,867 Hotels and fish farmsb 11,854,625 3,742,009 n.a. 2,053 2,270,628 Total 54,696,566 24,420,082 14,740,620 4,460 7,109,822 a. Peach accounted for 1,094 (94 percent) of the jobs created in the peach and potato value chains. b. Hotels accounted for 1,911 (93 percent) of the jobs created in the hotel and fish farm value chains. TABLE 11: EXPENDITURE PER UNIT OF RESULT BY VALUE CHAIN Value Chain Total Expenditure (USD) Cost Per Unit Of Result Value of Sales (USD) Value of Exports (USD) Jobs Created (FTE) Private Sector Investment (USD) Dates 8,683,037 140.73 140.73 0.02 Agricultural implements 2,646,606 n.a. 85,374 0.06 Knitted garments 3,302,649 0.54 0.54 11,507 0.13 Mango 23,223,965 2.65 2.73 25,179 0.10 Peach and potato 4,258,226 0.76 n.a. 3,649 0.44 Islampur weaving 727,457 10.72 n.a. 0.04 Hotels and fish farms 11,854,625 3.17 n.a. 5,774 0.19 148 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523