SATPARA DEVELOPMENT PROJECT EVALUATION IRRIGATED AGRICULTURE IN GILGIT-BALTISTAN OCTOBER 24, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared (in alphabetical order) by Izhar Hunzai, Uzair Ahmed Khan, Zohra Khanum, Douglas Krieger, Zia Ur Rehman, Syed Noman Ali Shah, and Himat Ullah, of Management Systems International. SATPARA DEVELOPMENT PROJECT EVALUATION IRRIGATED AGRICULTURE IN GILGIT-BALTISTAN Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. CONTENTS Acronyms........................................................................................................................................................................... iii Project Summary.............................................................................................................................................................. iv Executive Summary ..........................................................................................................................................................1 Evaluation Purpose and Questions...............................................................................................................................5 Project Background ..........................................................................................................................................................6 Evaluation Methods and Limitations...........................................................................................................................12 Findings and Conclusions ..............................................................................................................................................16 Component 1: Efficient Use of On-Farm Water............................................................................................16 Component 2: Enhancing Productivity of High -Value Horticulture and Dairy Products ....................21 Component 3: Enhancing Processing and Marketing Capacity of the Area .............................................27 Component 4: Create an Enabling Environment for the Agriculture Sector to Grow.........................31 Conclusions......................................................................................................................................................................35 Design .......................................................................................................................................................................35 Implementation.......................................................................................................................................................35 Sustainability ............................................................................................................................................................36 Recommendations ..........................................................................................................................................................36 Annexes.............................................................................................................................................................................38 Annex 1: Evaluation Statement of Work..........................................................................................................38 Annex 2: List of Documents Reviewed ............................................................................................................66 Annex 3: Data Collection Instruments .............................................................................................................68 Annex 4: Fieldwork Schedule and List of Interviews.....................................................................................83 Annex 5: Qualitative Data Analysis Tables ......................................................................................................86 Annex 6: Performance Data ................................................................................................................................93 List of Tables Table 1: Project Summary.............................................................................................................................................. iv Table 2: Interviews by Role in SDP and Interview type.........................................................................................14 Table 3: Water Availability ...........................................................................................................................................20 Table 4: Causes of Increased Quantities Produced ................................................................................................25 SATPARA DEVELOPMENT PROJECT EVALUATION i Table 5: Change in Marketing Practices by Group..................................................................................................29 Table 6: List of Interviews and Site visits...................................................................................................................83 Table 7: SDP Contribution to Adopting Technologies and Practices.................................................................86 Table 8: Change in Practices ........................................................................................................................................87 Table 9: Reasons for Increasing Quantity Produced...............................................................................................88 Table 10: Change in Value of Sales .............................................................................................................................89 Table 11: Employment ...................................................................................................................................................89 Table 12: Prospects for Sustainable Results .............................................................................................................90 Table 13: Satisfaction with Implementation ..............................................................................................................91 Table 14: Targets and Achieved Results – Component 1 .....................................................................................93 Table 15: Percentage of Target Achieved by Project Year – Component 1.....................................................96 Table 16: Explanation of Reported Results – Component 1 ................................................................................98 Table 17: Progress Against Indicators – Component 1 ...................................................................................... 103 Table 18: Targets and Achieved Results – Component 2 .................................................................................. 105 Table 19: Percentage of Target Achieved by Project Year – Component 2.................................................. 109 Table 20: Explanation of Reported Results – Component 2 ............................................................................. 112 Table 21: Targets and Achieved Results – Component 3 .................................................................................. 116 Table 22: Percentage of Target Achieved by Project Year – Component 3.................................................. 118 Table 23: Explanation of Reported Results – Component3 .............................................................................. 120 Table 24: Explanation of Reported Results – Component 4 ............................................................................. 124 List of Figures Figure 1: Location and Context of Project SDP Activities...................................................................................... v Figure 2: Satpara Dam Command Area.......................................................................................................................8 Figure 3: SDP Results Framework ..............................................................................................................................13 SATPARA DEVELOPMENT PROJECT EVALUATION ii ACRONYMS AIT Artificial Insemination Technician AKF Aga Khan Foundation AKRSP Aga Khan Rural Support Programme ARC Agriculture Resource Center DoA Department of Agriculture DoL Department of Livestock EGA Economic Growth and Agriculture FEG Farmers’ Enterprise Group FFS Farmer Field School FGD Focus Group Discussion FSCRD Federal Seed Certification and Registration Department FTE Full-Time Equivalent GB Gilgit-Baltistan GBLA Gilgit-Baltistan Legislative Assembly GoP Government of Pakistan HV/OSV High-Value Off-Season Vegetables ICT Information and Communication Technology IR Intermediate Result LBC Left Bank Canal LOP Life of Project LSO Local Support Organization MoU Memorandum of Understanding OFWMD On-Farm Water Management Department PCC Plain Cement Concrete PCPS Pre-cast Parabolic Section RCC Reinforced Cement Concrete RBC Right Bank Canal SDP Satpara Development Project SOW Statement of Work USAID United States Agency for International Development USD United States Dollar WAPDA Water and Power Development Authority WPG Women’s Processing Group WUA Water Users’ Association SATPARA DEVELOPMENT PROJECT EVALUATION iii PROJECT SUMMARY Table 1 summarizes basic information about the Satpara Development Project (SDP). TABLE 1: PROJECT SUMMARY Title / Field Project/Activity Information Contract/agreement numbers AID-391-A-12-0002 Contracting/Agreement Officer’s Representative (COR/AOR) Ayaz Muhammad, Project Management Specialist Start date March 12, 2012 Completion date March 11, 2017 Location Command area of Satpara Dam in Gilgit-Baltistan Implementing partner(s) Aga Khan Rural Support Programme (AKRSP), Aga Khan Foundation USAID/Pakistan Mission Strategic Framework objectives addressed DO 2: Improved Economic Status of Focus Populations IR 2.1: Improved Economic Performance of Focus Enterprises IR 2.2: Improved Business Enabling Environment Budget USD 19.75 million The map in Figure 1 shows the area affected by the project and the location of the major infrastructure elements (i.e., dam, left bank canal, and right bank canal) of the Satpara Dam Project which formed the basis for SDP. SATPARA DEVELOPMENT PROJECT EVALUATION iv FIGURE 1: LOCATION AND CONTEXT OF PROJECT SDP ACTIVITIES Source: Aga Khan Foundation. (n.d.). Satpara Development Project. 1st Annual Progress Report: March 12, 2012 - March 31, 2013. Islamabad, Pakistan: Aga Khan Foundation. SATPARA DEVELOPMENT PROJECT EVALUATION v EXECUTIVE SUMMARY The Satpara Development Project (SDP) is far from meeting targets for extending irrigation to additional land and, at the time of the evaluation, was well behind targets for constructing irrigation infrastructure. The former is a direct result of that fact that key assumptions underpinning the development hypothesis have failed to materialize, through no fault of the implementing partners, the Aga Khan Foundation (AKF) and the Aga Khan Rural Support Programme (AKRSP). The Economic Growth and Agriculture (EGA) office of USAID/Pakistan has commissioned a mid-term evaluation of SDP to help decide whether, and under what conditions, to extend the current agreement and to identify any corrective actions (i.e., changes in design, management, or implementation approaches) that may improve performance. The evaluation addresses three questions: 1. How appropriate was the project design to achieving anticipated results? 2. To what extent were the project’s activities and approaches appropriate to achieving anticipated results and how/why? 3. To what extent are results likely to be sustainable beyond the end of the project and why? Project Background With USAID/Pakistan’s assistance, Pakistan’s Water and Power Development Authority (WAPDA) completed the Satpara Dam near Skardu in Gilgit-Baltistan (GB) in 2012 with the multiple objectives of generating electricity to relieve chronic load shedding in Skardu, enhancing access to domestic water, alleviating downstream flooding, and increasing the supply of water for irrigation. The irrigation infrastructure associated with Satpara Dam, the left and right bank canals (LBC and RBC, respectively) were to carry the additional water to areas outside the traditional Satpara Lake command area and extend irrigation to about 15,500 acres. USAID/Pakistan designed SDP to enhance agricultural outcomes associated with the additional irrigation water Satpara Dam was expected to deliver. It implements the U.S. Dollar (USD) 19.75 million SDP through a cooperative agreement with the Aga Khan Foundation (AKF) with a sub-award to the Aga Khan Rural Support Programme (AKRSP). The project focuses on improving irrigation infrastructure to make more efficient use of water, establishing community and government institutions to maintain irrigation infrastructure and manage water, providing grants to agribusinesses to spur adoption of modern production and processing technologies and practices, and contributing to an enabling environment that will facilitate agricultural sector growth. Project objectives include improving water use efficiency, increasing the irrigated area from about 2,689 acres to about 15,500 acres, enhancing agricultural production, increasing the incomes of about 8,547 households by 15 percent (Rs. 61,959), and creating about 4,850 full-time equivalent (FTE) jobs. The key assumption underpinning SDP, i.e., that additional water from Satpara Dam could be used to extend the irrigated area outside the traditional command area of Satpara Lake, failed to materialize. Environmental issues prevented adding an additional source of water to Satpara Dam, and traditional water rights holders were unwilling to relinquish their rights to existing water to serve new areas. SATPARA DEVELOPMENT PROJECT EVALUATION 1 Data Collection Methods and Limitations The evaluation relies largely on qualitative data from individual and group interviews. Over a two-week period, the evaluation team conducted 45 individual and group interviews with implementers, beneficiaries, and stakeholders and conducted site visits in the Satpara Dam command area. During analysis, the team triangulated qualitative data from the interviews with quantitative data collected from project reports and AKRPS’s results tracking system. The team produced detailed transcripts of each interview; reviewed the transcripts to identify themes relevant to answering the evaluation questions; used MAXQDA to code text segments illustrating the themes and identify patterns in the themes; and analyzed the coded segments to develop detailed evidence of if and how AKF/AKRSP’s design and implementation of SDP had contributed to achieving intended results. Qualitative data are well suited to answering how and why questions but not well suited to validating quantitative data. Furthermore, differences in interview techniques, language barriers, general communication issues, the education level of respondents, and strategic behavior by respondents may all affect the consistency, validity, and reliability of the qualitative data. Particular to SDP, many beneficiaries seemed to have participated in multiple project-supported activities which made it difficult for the team to determine the effects of individual interventions. The team addressed these inherent limitations by identifying inconsistencies in interviews and probing to resolve them; whenever possible, conducting interviews in the languages with which respondents were most comfortable; and probing to reveal contextual factors that could have affected results. Findings and Conclusions The report presents findings and conclusions by project component. Component 1 is “efficient use of on-farm water”; component 2 is “enhancing productivity of high-value horticulture and dairy products”; component 3 is “enhancing processing and marketing capacity”; and component 4 is “creating an enabling environment for the agricultural sector to grow.” Component 1: Efficient Use of On-Farm Water By reducing loss due to seepage and increasing water flow rates, improved channels and tertiary watercourses have unquestionably increased the availability of irrigation water. Residents in the Satpara Dam command area also benefited from the regulated flow from the dam which affected the seasonality of water availability. Those outside the command area, who still depend largely on seasonal streams, benefitted from more water when the streams were flowing but did not benefit as much from more seasonal reliability. Improvements in water availability saved time and labor, improved irrigation scheduling, and extended irrigated land. Factors largely outside of AKF/AKRSP’s control affected their ability to meet project objectives for irrigating 15,500 acres of land. These include primarily the many factors that prevented utilization of the LBC and RBC to transfer water to the unserved areas. This situation occurred because new sources of water were not diverted to Satpara Dam and traditional water rights holders were unwilling to relinquish their rights and transfer surplus water to new areas. To meet targets, water rights holders must use water more efficiently and be willing to relinquish some of their rights (perhaps through a SATPARA DEVELOPMENT PROJECT EVALUATION 2 formal trade or market mechanism) and allow surplus water to flow through the LBC and RBC to areas outside the traditional command area. Establishing an institutional structure to maintain the LBC and RBC is also necessary. Alternatively, if Shatong Nullah1 water is eventually diverted to the dam, residents outside the Satpara Lake command area will have rights to that water. Work on tertiary watercourses progressed more slowly than anticipated largely due to the need for a much more intensive master planning exercise than originally anticipated and delays developing a grants manual and environmental documentation while USAID and AKF ironed out details of the cooperative agreement. At the level of main and secondary channels, disagreements with communities about the size and shape of channels (all four construction packages), short working seasons (one construction package), and a contractor’s inability to mobilize quickly (one construction package) delayed progress.2 Most of the disagreements stemmed from communities’ efforts to protect their access to water when they perceived that the new and unfamiliar construction techniques and materials employed by AKF would reduce water availability. Water Users’ Associations (WUAs) did not speak directly about whether increased water availability was sustainable. Most (7 of 8), however, believed that pre-cast parabolic section (PCPS) technology was not durable and cited their experience with broken sections and water loss. Communities’ perceptions may not reflect reality, and AKF/AKRSP claims that PCPS is a proven technology and farmers’ objections are a result of their lack of familiarity with the technology. Better communication to establish buy-in may have moderated community resistance and ameliorated delays. Component 2: Enhancing Productivity of High-Value Horticulture and Dairy Products AKRSP promoted effective productivity-enhancing interventions and, for the most part, implemented them well. A large majority of beneficiaries attributed project interventions with increasing the quantities of agricultural and livestock products they produced and sold. The greenhouses, in particular, seem to have spurred a shift from household to commercial production, and participants are now seeing producing vegetables as a viable business opportunity. In addition to the greenhouses, hybrid seeds and training seemed to be particularly effective interventions. Because component 2 is largely independent of component 1, i.e., the project’s productivity-enhancing interventions did not depend on improved availability of irrigation water, the assumptions underpinning component 1 results (i.e., that water would be available to utilize the LBC and RBC to deliver water to areas outside the traditional Satpara Lake command area) that failed to materialize did not affect results of component 2. However, the delay in developing grants manuals and environmental compliance documents, which was related to negotiating the respective roles of AKF and AKRSP in the cooperative agreement, delayed grant-funded activities and activities contingent on grant-funded infrastructure or equipment by at least a year. 1 Shatong Nullah is the alternative water source that was not diverted into Satpara Dam as originally planned. 2 Aga Khan Foundation. (2015). 3rd Annual Progress Report: April 1, 2014 - March 31, 2015. Peshawar: Aga Khan Foundation. SATPARA DEVELOPMENT PROJECT EVALUATION 3 Component 3: Enhancing Processing and Marketing Capacity AKRSP’s training in post-harvest practices seems to have been particularly effective in raising awareness of the value of proper grading, storage, and packaging practices. Its marketing training and exposure visits have sensitized producers to market demands, and many now report grading, storing, and packaging to meet these demands. Although AKRSP did not focus on linking producers directly to buyers, its training seems to have changed the way beneficiaries interact with existing markets in beneficial ways. Other than the Friday bazaar, which has been a particularly effective innovation for linking producers directly to consumers, AKRSP did not seem to focus much on linking producers to alternative markets. Few producers mentioned new marketing channels but spoke instead of engaging with existing markets more knowledgably, i.e., grading to meet market demands, presenting produce attractively, collecting information on prices, and selling collectively in larger quantities and with greater bargaining power. Investments in dairy processing appear effective on a very small scale in terms of both the size of the supported enterprises and the number of grants. Dehydration units for apricots seem to have been instrumental in orienting farmers toward commercial production and processing. Component 4: Creating an Enabling Environment for the Agricultural Sector to Grow The enabling environment component has progressed slowly, largely because it has been difficult to engage policy makers and stakeholders. However, SDP has made some important headway. Formalizing WUAs and the Jheel Committee and expanding the Jheel Committee’s membership are important steps in developing representative water institutions with the authority and historical legitimacy to address emerging challenges and opportunities, such as redefining water rights, developing water markets, and attracting investments. SDP has also facilitated establishing local plant inspection and certification services and thus the supply of locally adapted certified rootstock, a major development for the fruit sub-sector. Recommendations • USAID/Pakistan should continue investing in improving main and secondary channels and tertiary watercourses in the existing Satpara Dam command area and in the areas anticipated to be served by the LBC and RBC. Project-supported irrigation infrastructure has been exceptionally effective in increasing water availability, even in the areas not served by Satpara Dam, and will continue to do so for decades. • If AKF/AKRSP continues the irrigation infrastructure work, it needs to communicate better with affected communities and engage them more fully in planning. In particular, it needs to convince communities that PCPS technology is suitable to the area and durable. If failures noted by communities and the evaluation team are the result of poor construction or planning, AKF/AKRSP may need to better oversee contractors. AKRSP-supported exposure visits to established PCPS irrigation systems do not appear to have been widely effective, and broader exposure may be necessary. • While improved irrigation infrastructure in the areas not served by Satpara Dam has been effective, its impact on water availability will be limited (particularly in extending seasonal availability) until water from the source becomes more reliable. AKF/AKRSP should continue SATPARA DEVELOPMENT PROJECT EVALUATION 4 the work of identifying and developing alternative sources of water. If the assessment indicates that improving the alternative water sources (above the LBC and RBC) has the potential substantially to improve water reliability, and if directing water into the LBC and RBC does not seem likely, then USAID/Pakistan may want to consider developing the alternative water sources. • If USAID/Pakistan decides to improve water sources in the areas served by the LBC and RBC, it should design the work so that water can relatively easily be redirected into the LBC and RBC if and when they become operational. • In future support to irrigation schemes, USAID must thoroughly understand the structure of traditional water rights and incorporate these rights in design and implementation from the start. • To get the most out of the investment it has already made in the LBC and RBC, USAID/Pakistan must facilitate resolving issues of infrastructure ownership and water rights. Unless USAID/Pakistan deems these issues intractable within a reasonable timeframe or determines that it does not have sufficient leverage to influence decisions, it should continue to support developing local institutional structures to manage water allocation and distribution and maintain irrigation infrastructure. Investments in creating an enabling environment are relatively low-cost and, in this case, could have a large payoff. • AKF/AKRSP’s work on components 2 and 3 has been effective. However, it will require more time and technical assistance to cement the skills and enhance prospects for sustainable results. For the remainder of the project, AKF/AKRSP should focus on supporting capacity building by providing technical assistance, and not on providing new grants that may not have time to mature to sustainable status. • To enhance prospects for sustainability, AKF/AKRSP should focus on building sustainable public or private sector technical service providers who can support greenhouses, orchards, nurseries, processing, and marketing. EVALUATION PURPOSE AND QUESTIONS With SDP behind on many of its results targets (at the time USAID commissioned the evaluation), USAID wishes to understand fully the causes of delays to inform decisions about whether, and under what conditions, to extend the current agreement and to identify any corrective actions (i.e., changes in design, management, or implementation approaches) that may improve performance. The evaluation should also identify lessons about project design and implementation that could improve the design and/or performance of future USAID programming in infrastructure or agriculture. The primary audiences for the evaluation include: 1) the USAID/Pakistan Mission, particularly the Economic Growth and Agriculture (EGA) team; 2) the USAID Office of Afghanistan and Pakistan Affairs; and 3) the implementing partners, the Aga Khan Foundation (AKF) and the Agha Khan Rural Support Programme (AKRSP). Recommendations from the evaluation will help USAID/Pakistan, AKF, and AKRSP improve project performance. Lessons learned will help all audiences design and implement more effective projects in the future. SATPARA DEVELOPMENT PROJECT EVALUATION 5 Evaluation Questions The evaluation statement of work (SOW) poses three evaluation questions, each with an explanation. The questions focus broadly on understanding whether the project design is appropriate to achieving intended results (i.e., did the project do the right things), the effectiveness of project implementation (i.e., did the project do things right), and the likelihood that results will be sustainable after USAID support ends. The top-line evaluation questions are: 1. How appropriate was the project design to achieving anticipated results? 2. To what extent were the project’s activities and approaches appropriate to achieving anticipated results and how/why? 3. To what extent are results likely to be sustainable beyond the end of the project and why? The evaluation statement of work (SOW) in Annex 1 contains the original questions and detailed explanations for each question. PROJECT BACKGROUND Pakistan’s Water and Power Development Authority (WAPDA) began building the Satpara Dam near Skardu in Gilgit-Baltistan (GB) in 2003 with the multiple objectives of generating electricity, providing domestic water, alleviating downstream flooding, and increasing the availability of irrigation water. WAPDA had largely completed the dam and related infrastructure by 2010 but ran out of funds to finish the project. In 2011, the Government of Pakistan (GoP) and USAID/Pakistan signed a 19 million U.S. Dollars (USD) agreement to complete the dam project. In addition, USAID provided USD 7 million to help construct two canals, the left bank canal (LBC) and right bank canal (RBC), to extend the dam’s command area to irrigate barren land and areas irrigated by seasonal streams (nullahs) upstream and downstream of the existing command area (Figure 2). WAPDA officials the evaluation team interviewed reported that they expected Satpara Dam to obtain half its water from the Satpara Lake catchment and the other half from water diverted from Shatong Nullah, a nearby watershed. 3 However, they reported that it had not conducted a proper environmental impact assessment and that environmental organizations4 successfully argued that diverting water from Shatong Nullah violated the Wildlife Act. WAPDA deferred the diversion and engaged a consultant to study its likely environmental impact. Development Problem and USAID’s Response Gilgit-Baltistan (GB), the province in which Satpara Dam is located, lies in the mountainous north of Pakistan, an area that encompasses the Karakoram and western Himalayan mountain ranges and contains 5 of the world’s 14 peaks over 8,000 meters high. The region’s remoteness (from input and 3 Shatong Nullah is a river in the nearby Deosai Plains, a designated national park. 4 The World Wildlife Fund, the International Union for Conservation of Nature, and the Himalayan Wildlife Fund. SATPARA DEVELOPMENT PROJECT EVALUATION 6 output markets), poor transportation infrastructure, and harsh winters make it a challenging environment for agricultural production. A recent study concluded that GB has experienced respectable economic growth in recent years.5 A gradual shift from subsistence agriculture to fruits and other cash crops has contributed to this growth, especially in areas better connected to markets through the Karakoram Highway. In spite of relatively good growth, however, average per capita income levels are still only about 90 percent of the national average, and the 29 percent incidence of poverty in 2004-2005 was higher than the national average of 24 percent with substantial disparity across districts – rural districts faring worse than urban. A majority of the population (86 percent) is rural, and major development challenges remain as the rural poor move out of subsistence agriculture. AKRSP’s agreement document states that GB, and the Indus River valley around Skardu in particular, has substantial agricultural potential. The soils are fertile and suitable to many food and cash crops. Farmers in the area produce a variety of fruits (apricots, cherries, apples, plums, almonds, walnuts, mulberries, grapes, and peaches), vegetables (onions, tomatoes, cauliflower, cabbage, turnips, carrots, peas, beans, capsicum, chilies, and radishes), field crops (potatoes and wheat), and fodder crops (barley, millet, sorghum, and maize). With appropriate technologies and practices, the area has the capacity to produce high-value off-season vegetables to replace products imported during the winter months. The area also has substantial capacity for fruit production with potentially large national and international export markets. In spite of its potential, the agriculture sector in Skardu faces many challenges including:6 • Scarce and inefficient use of irrigation water – Irrigation water is scarce in the Satpara watershed, due in large part to high levels of loss in unlined irrigation channels and poor water management practices. Furthermore, few farmers have experience with efficient irrigation techniques. • Low agricultural productivity – Farmers tend to focus on traditional crops and have limited knowledge of, or access to, productivity-enhancing production practices and technologies, labor, and land. In this context, even if farmers minimize water loss, they still may not use water as productively as possible. Extreme winter weather conditions also limit production and constrain agricultural productivity. • Limited processing or marketing capacity – Limited knowledge of good harvest and post￾harvest practices, few storage or processing options, and poor access to markets all contribute to high levels of waste in the fruit and horticulture sectors in the Skardu area. Public and private sector agricultural support services (inputs, extension, and marketing) are also very weak. • Unfavorable enabling environment – Examples of enabling environment issues that affect agricultural sector growth include government interference in markets and the absence of seed laws and nursery registration to ensure the quality of seeds and rootstock. 5 Government of Gilgit-Baltistan; Asian Development Bank; World Bank. (2011). Gilgit-Baltistan Economic Report: Broadening the Transformation. 6 USAID/Pakistan. (2012). Cooperative Agreement No. AID-391-A-12-0002. Islamabad: USAID. SATPARA DEVELOPMENT PROJECT EVALUATION 7 FIGURE 2: SATPARA DAM COMMAND AREA Source: Presentation from Aga Khan Rural Support Programme. SATPARA DEVELOPMENT PROJECT EVALUATION 8 To enhance Satpara Dam’s contribution to the agricultural sector, USAID entered into a USD 19.75 million cooperative agreement with the Aga Khan Foundation (AKF) to implement the Satpara Development Project (SDP). At the time of the award, USAID thought that AKRSP was a subsidiary of AKF and did not require a sub-grantee status. This turned out not to be true, and USAID had to substantially modify the agreement with AKRSP assuming responsibility for implementing SDP through a sub-award with AKF, which was responsible for inter-agency coordination, monitoring, and grants management. The change in the agreement took quite a long time and delayed development of key operating documents (e.g., grants manuals and environmental compliance procedures) and implementation. The SDP agreement document sets a target of using water from Satpara Dam to irrigate 15,500 acres of agricultural land, 2,689 acres within the traditional command area of Satpara Lake and the remainder in new areas to be served by the LBC and RBC (Figure 2), areas that have traditionally relied on seasonal streams (nullahs). However, the agreement document recognized that the water available from Satpara Dam was not sufficient to meet the needs of simultaneously generating power, providing drinking water, and irrigating 15,500 acres of land at current rates of water use.7 It concluded that, to meet targets, existing on-farm irrigation infrastructure would have to be constructed or rehabilitated to reduce seepage/loss and improve water use efficiency. “Out of 93,000 acres-feet storage capacities, more than 62,500 acres-feet will be required for irrigation [for 15,500 acres]. The remaining 30,500 acres-feet of water is hardly enough to generate one-third of the installed [electricity generation] capacity. In order to use the dam water efficiently for irrigation purpose the existing on-farm irrigation infrastructure needs to improve [to reduce seepage and loss] and new irrigation infrastructure is needed to be constructed for further land development in the command area.” Though the deferred diversion of Shatong Nullah and low water levels in Satpara Dam seemed initially to constrain AKRSP’s ability to extend irrigation water to new areas, traditional water rights have also emerged as a barrier. Prior to the dam’s construction, 12 villages held the rights to the water from Satpara Lake. These villages have argued that, since Satpara Dam does not collect water from other (new) sources (i.e., Shatong Nullah), they still have the rights to all the water released from the dam. They have been unwilling to relinquish these rights and divert water into the LBC and RBC to serve new areas.8 Furthermore, because no institution has ownership of the LBC and RBC, they are quickly falling into disrepair and are now damaged and blocked by landslides. Implementation SDP aims broadly to increase the irrigated area, agricultural productivity, income, and employment in the (expanded) command area of the Satpara Dam by directly addressing problems with irrigation infrastructure and constraints to agricultural sector growth in the region. Project activities focus on 7 The water rights study commissioned by AKF concluded that “Overall supply of water after construction of Satpara Dam has increased substantially and made more reliable but not enough to irrigate designed area of 15,500 but sufficient to irrigate 7,310 acres against the 2,689 acres irrigated before the construction of Dam.” 8 Khan, M. J., Zulfiqar, M., & Ali, J. (June 2015). Traditional Irrigation Practices and Proposed Efficient Irrigation Water Management system for SDP Command Area (Benefitting from Satpara Dam) Skardu - Satpara Development Project. Peshawar: University of Agriculture, Peshawar. SATPARA DEVELOPMENT PROJECT EVALUATION 9 farmers and other agricultural enterprises in the command area of the Satpara Dam irrigation infrastructure, including the areas expected to be served by the LBC and RBC. Geographically, this encompasses the area along the south bank of the Indus River from the village of Hoto in the west to Thorgo Bala in the east (Figure 1). AKRSP personnel reported that AKF designed SDP on the assumption that Satpara Dam would include the Shatong Nullah diversion and provide an increased volume of water, and that the LBC and RBC would be operational. Without the Shatong Nullah diversion, and with the traditional water rights holders unwilling to relinquish their rights, water to operate the LBC and RBC did not materialize, and the command area did not expand. The failure of a key design assumption to materialize had a cascade effect on all SDP components. In an attempt to meet targets in an environment without additional water to charge the LBC and RBC, AKRSP commissioned studies of water rights and alternative water sources in the areas outside the traditional Satpara Lake command area. Even without water in the LBC and RBC, AKRSP has been upgrading primary and secondary canals and tertiary watercourses in these areas. The project focuses primarily on small farmers (organized into producers’ groups and associations), processing enterprises, agribusinesses (e.g., milk collection centers), and support enterprises (e.g., artificial insemination service providers), with an emphasis on the high-value vegetable and dairy sub￾sectors. The project engages men and women as appropriate. Project activities are organized around four primary components corresponding to the four intermediate results in the EGA results framework. • Component 1: Efficient use of irrigation water – To implement this component, AKF developed an irrigation master plan and designed specific irrigation projects (secondary canals and tertiary watercourses) based on the plans. AKRSP /AKF9 then engaged contractors to construct the secondary canals; established and strengthened Water Users’ Associations (WUAs) to build, maintain, and manage tertiary watercourses; and provided grants to WUAs to construct the tertiary watercourses. The project also planned to introduce more efficient on￾farm irrigation technologies and practices such as land leveling and drip irrigation systems. • Component 2: Enhancing productivity of high-value horticulture and dairy products – Activities under this component included providing grants to individual farmers to spur adoption of productivity-enhancing technologies and practices. The project established farmers’ enterprise groups (FEGs) in the horticulture and dairy value chains; trained them (sometimes through farmer field schools) in productivity-enhancing technologies and practices; and exposed them to more efficient irrigation systems, markets, and new technologies and practices through 9 AKF played a particularly important role in grants and sub-contracts. AKRSP had the authority to authorize grants up to USD 10,000 (eventually increased to USD 20,000), while AKF had to authorize larger grants and contracts (e.g., for constructing canals). AKRSP personnel reported that AKF prepared the base documents (e.g., grant manual, contracting procedures, and HR manual). Even after addressing these inception issues, however, the project still faces administrative delays caused by multiple layers of oversight and approvals from AKRSP, AKF, and USAID. Sometimes, signatories are not available at their stations, and signing of checks takes weeks and months, thus further slowing down operations. SATPARA DEVELOPMENT PROJECT EVALUATION 10 exposure visits. The project also provided grants to develop agribusiness enterprises to support growth in the sector. Uses of grants include: § Establishing registered nurseries and commercial orchards to support development of the fruit sub-sector; § Constructing greenhouses to demonstrate off-season vegetable production and processing; and § Establishing enterprises (e.g., farms, artificial insemination (AI) centers, milk collection centers, and feed suppliers) to support development of the dairy sub-sector. Under component 2, SDP also established women’s producer groups to engage women where appropriate in production, add value to products, and enhance market opportunities for processed products. • Component 3: Enhancing processing and marketing capacity of the area – Activities under this component included forming women’s processing groups (WPGs), introducing the groups to improved processing practices and technologies; training group members in processing practices, business, and post-harvest handling; and providing grants to support investments in processing enterprises. • Component 4: Create an enabling environment for the agricultural sector to grow – SDP focuses on empowering local communities to work with the public and private sectors to develop new and improve existing rules, regulations, and acts related to water user rights; certify horticulture products; introduce good agricultural practices; and improve agriculture extension services of the relevant government departments, e.g., Agriculture and Livestock Husbandry. Development Hypothesis and Intended Results The project’s results framework, contained in the project management plan document,10 outlines the development hypothesis. The strategic objective is: “To increase on-farm water use efficiency, agro￾productivity and agro-marketing facilities within the command area of the Satpara Dam Irrigation Scheme in order to provide income generating and employment opportunities.” Four intermediate results (IRs) contribute to the objectives: 1) increased efficiency of use of on-farm irrigation water; 2) enhanced productivity of higher-value horticulture and dairy products; 3) enhanced agro-processing and marketing capacity; and 4) create an enabling environment for the agriculture sector to grow. Sub-IRs under each IR further elaborate the development hypotheses. Figure 3 illustrates the linkages between the project’s IRs and sub-IRs. In summary terms, the development hypothesis is: If (assumption) Satpara Dam and LBC/RBC are constructed and function as planned, then if secondary and tertiary watercourses are improved in the traditional Satpara Dam command area and in adjacent areas served by LBC/RBC, and on-farm water use efficiency is improved, then more water will be 10 Aga Khan Foundation. (2012). Project Monitoring Plan (PMP). Peshawar: Aga Khan Foundation. SATPARA DEVELOPMENT PROJECT EVALUATION 11 available for irrigation, and additional land will be brought under irrigation, and if farmers are introduced to, and adopt, productivity-enhancing technologies and practices and value addition and processing capabilities are enhanced and enabling environment improved, then 15,500 acres will be irrigated, the agricultural sector in the command area will grow, incomes of about 8,547 households will increase by 15 percent (Rs. 61,959), and about 4,850 full-time equivalent (FTE) jobs will be created. Available project documents do not cite additional anticipated outcomes. EVALUATION METHODS AND LIMITATIONS The evaluation relied largely on qualitative data collected through semi-structured individual and group interviews with implementing partner staff, stakeholders, and beneficiaries. The evaluation team also reviewed project documents to develop an understanding of project design and implementation approaches. It also gathered quantitative performance data from project reports, AKRSP’s results tracking system, and PakInfo in order to triangulate the qualitative findings. Data Collection The evaluation team reviewed a large number of project documents (Annex 2) before assembling in Islamabad for a team planning workshop. Through the document review, the team developed an understanding of project objectives, design, implementation strategies, and results. During the workshop, AKF made a presentation to the team to explain the project further. The team spent the remainder of the workshop developing a data collection plan and instruments (Annex 3). In its two weeks of field work, the evaluation team conducted 13 semi-structured individual interviews with AKRSP staff members, stakeholders, and individual grant recipients. It also conducted 34 group interviews with AKRSP staff members, local support organization (LSO) staff members, stakeholders, and project beneficiaries (i.e., artificial insemination technician (AITs) trainees, farmer field school (FFS) participants, grant recipients, WUA members and presidents, FEG members, and WPG members). Table 22 summarizes interviews by role in SDP and type (i.e., individual or group). Annex 4 contains a detailed schedule of the field work. Sampling The team selected key informants purposively to represent specific perspectives on project design or implementation. Key informants included AKF and AKRSP staff members and stakeholders (i.e., WAPDA, Department of Agriculture (DoA), and Government of GB officials). For all other interviews, the team selected participants randomly from lists of beneficiaries and participants provided by AKRSP. The team identified the various types of beneficiaries and participants; created separate lists for each type; randomly ordered the lists; and, starting from the top of each list, selected the required number of interview subjects of each type. The team determined the number of interviews by type of beneficiary/participant roughly on the basis of their proportion among all beneficiaries/participants. SATPARA DEVELOPMENT PROJECT EVALUATION 12 FIGURE 3: SDP RESULTS FRAMEWORK Strategic ObjecHve: To increase on-farm water use efficiency, agro-producHvity and agro-markeHng faciliHes within the command area of the Satpara Dam IrrigaHon Scheme in order to provide income generaHng and employment opportuniHes IR-1 Increased efficiency of use of on-farm irrigaHon water Sub-IR 1.1 Improved exisHng structures and constructed new irrigaHon infrastructure Sub-IR 1.2 Improved management of the irrigaHon system IR-2 Enhanced ProducHvity of Higher Value HorHculture & Dairy Products Sub-IR 2.1 Improved supply of high quality fruit for export out of the region Sub-IR 2.2 Improved Supply of in-season and off-season vegetables Sub-IR 2.3 Improved producHvity of the local dairy industry IR-3 Enhanced Agro￾Processing and MarkeHng Capacity Sub-IR 3.1 Minimize fruit & vegetable producHon wastage Sub-IR 3.2 Improved markeHng of higher value fruit, vegetable and dairy products Sub-IR 3.3 Improved farm to market access infrastructure (roads, bridges, culverts) IR 4: Created an Enabling Environment for the Agriculture Sector to Grow Sub-IR 4.1 EffecHve and improved water uHlizaHon operaHng and maintenance system and rules developed and established Sub-IR 4.2 Enhanced effecHveness of agriculture extension services Sub-IR 4.3 Standards and cerHficaHon for natural/ organic mountain products developed and implemented Sub-IR 4.4 Farmer Enterprise Groups are federated at the Skardu level in order to influence policy and pracHces Sub-IR 4.5 Range of relevant polices and legal instruments in support of farming communiHes and entrepreneurs approved by GBLA SATPARA DEVELOPMENT PROJECT EVALUATION 13 TABLE 2: INTERVIEWS BY ROLE IN SDP AND INTERVIEW TYPE Role In SDP Individual Interviews Group Interviews/ Discussions AI trainees 1 Dehydration unit grant recipients 1 FFS participants - unidentified 4 FFS participants – animal husbandry 1 FFS participants – vegetable production 1 Greenhouse grant recipients 2 4 FEGs - HV/OSV 2 Implementing partner 2 2 Individual grant recipients - orchards 2 1 Individual grant recipients - nurseries 1 Individual grant recipients - vegetables 1 Stakeholders 2 3 Nursery and orchard management trainees 1 Soil testing trainees 1 WUA members 5 WUA presidents 2 WPG – mulberry, apple, apricot 3 Milk collection and processing grant recipient 3 Total 12 33 Data Analysis The data for the evaluation is almost entirely qualitative. The team used MAXQDA to prepare and analyze the qualitative interview data. To prepare and analyze the data, the team: • Developed a detailed data analysis plan with table shells that summarized the data in a manner relevant to answering the evaluation questions. For example, one table specified themes related to satisfaction with implementation in rows and respondent type in columns. The frequency of responses in each cell indicated the level of satisfaction or dissatisfaction with a particular facet of implementation by respondent type (e.g., WUAs, greenhouse FEGs, or individual grant recipients). • Reviewed a subset of interview transcripts to identify common themes. In the context of the example above, this meant identifying the facets of implementation respondents mentioned and their reasons for being satisfied or dissatisfied with AKRSP’s performance relative to that facet of implementation. SATPARA DEVELOPMENT PROJECT EVALUATION 14 • Reviewed all interview transcripts to identify text segments that corresponded to the identified themes and used the software to “code” the segments, i.e., associate the theme with the segment. • Populated the tables developed in the data analysis plan with the frequencies of documents that contained one or more instances of a particular theme. Data from semi-structured interviews may contain multiple segments that illustrate a particular theme. Because the frequency of a theme within an interview does not reflect the intensity of the theme, especially in a group interview, using the document rather than coded segments as the unit of analysis is most appropriate. This means that the tables report the number of documents that contained one or more instances of a coded theme, not the frequency of coded segments. Annex 5 contains the tables the evaluation team developed to analyze the data. • Looked for patterns of responses in the tables. For example, the tables might reveal a pattern of large frequencies associated with a particular theme across respondent types. They might also reveal a pattern of large frequency values across many themes within a particular respondent type. • Retrieved coded segments to develop a nuanced understanding of the story behind the observed patterns. For instance, if a large number of WUA members expressed dissatisfaction with the design of tertiary watercourses, the team reviewed the individual text segments to understand the aspects of watercourse design that were problematic. When possible, the team triangulated the results of the qualitative analysis with quantitative data obtained from the project’s reports and results tracking system. Data Limitations Qualitative data are well suited to answering questions of how and why something happened but not well suited to collecting or verifying quantitative data. Differences in interview techniques, language barriers, general communication issues, the education level of respondents, and the potential for strategic behavior by respondents may all affect the consistency, validity, and reliability of the data. Limitations specific to the SDP evaluation include: • Qualitative data reflect respondents’ perceptions of design, implementation, and results. Perceptions may differ from reality, vary from person to person, and be difficult to interpret without understanding the broader context. • Qualitative data are well suited to explaining how and why interventions produce results, or fail to produce results; exploring how elements of design and implementation affected outcomes; and learning about contextual factors that may affect results. They are not appropriate for validating quantitative results. In the context of this evaluation, the qualitative data directly addressed the main evaluation questions but were not appropriate for directly validating the quantitative results AKRSP reported having achieved. • Interview results, even when collected from randomly selected respondents, are not generalizable to the population, largely because the interviews were not structured and the sample sizes were too small. • Translation can lead to a loss of fidelity in qualitative data, as can poor communication between interviewers and respondents for any other reason. SATPARA DEVELOPMENT PROJECT EVALUATION 15 • Project beneficiaries often engaged in more than one project-supported activity. For example, an individual farmer might be a member of a WUA, receive a grant for a greenhouse, and participate in a vegetable production FFS. The team therefore had difficulty separating the effects of one intervention from those of another. • Respondents may respond strategically in hopes of receiving additional project support. • AKRSP adapted implementation to the evolving operating environment. This involved adding, dropping, and modifying activities as the development context changed. Consequently, activity reports did not provide a clear linear picture of activities or progress against targets. This made it difficult for the evaluation team to understand activities and progress solely from documents. Meetings with AKF/AKRSP resolved these issues during the field work. The team addressed these inherent limitations by hiring local translators for the few instances when an individual or group preferred to speak in local languages (i.e., not Urdu), identifying inconsistencies in interviews and probing to resolve them; and probing to reveal contextual factors that could have affected results. The evaluation team adequately addressed most of these limitations and is confident in the overall validity of the qualitative data. To resolve inconsistencies in the quantitative data, the team pieced together an implementation timeline from reports, AKRSP’s results tracker, and PakInfo11 and then contacted AKRSP to resolve inconsistencies. When discussing their experiences with a project in a semi-structured interview, respondents are more likely to recall negative than positive aspects. The ratio of positive to negative responses to a particular aspect of the project is therefore meaningless. What is meaningful is the content and frequency of responses that reveal the nature of the positive or negative reaction. Similarly, in a semi-structured interview setting it is not possible to interpret “no response.” For example, if an individual or group fails to mention whether they produced a larger quantity of potatoes as a result of the project’s interventions, it could mean that they did not experience a change or that they failed to mention a change. Keeping these limitations in mind, the analysis focuses on frequencies of responses rather than ratios of positive to negative responses and does not try to infer the meaning of “no response.” FINDINGS AND CONCLUSIONS Component 1: Efficient Use of On-Farm Water Component 1 activities focus on constructing or improving irrigation infrastructure; forming and strengthening WUAs to construct and maintain tertiary water courses and manage irrigation water; improving farmers’ capacities to use irrigation water more efficiently (e.g., pressurized and drip irrigation, land leveling); and installing other infrastructure and equipment associated with the watercourses and water management (e.g., piped irrigation, water reservoirs, washing areas, information and communication technology (ICT) water monitoring). 11 PakInfo is USAID/Pakistan’s internal results reporting and management information system. SATPARA DEVELOPMENT PROJECT EVALUATION 16 Quantitative findings for component 1 come from the project’s results tracker which documents specific activities and annual progress against activity targets (summarized in Table 14 in Annex 6 for component 1); annual reports which provide narrative explanations for the quantitative results (summarized in Table 16 in Annex 6 for component 1); and PakInfo, which contains the quantitative data the project reports against its USAID performance indicators (summarized in Table 15 in Annex 6 for component 1). The results tracker reflects revised targets and activities, while the annual reports reflect unrevised targets and work plans. Comparing the two reveals what AKRSP set out to accomplish, what it reports having accomplished, and reasons for discrepancies. Highlights by project year include: • Year 1: AKRSP formed 91 WUAs and registered 84. It developed a plan for the irrigation infrastructure which it ultimately realized was not nearly detailed enough for its intended purpose. The time required to develop a more detailed plan, along with delays in developing grants manuals and environmental documentation (caused by a prolonged negotiation with USAID over the respective roles of AKF and AKRSP in the cooperative agreement) prevented AKF/AKRSP from achieving targets for establishing demonstration plots, conducting exposure visits to efficient irrigation systems, training farmers in efficient irrigation, and designing and constructing tertiary watercourses. • Year 2: AKRSP formed an additional 127 WUAs, registered 73, and began to train them in watercourse construction and recordkeeping. It also hired a consultant for the (expanded) irrigation master plan and improved 94 tertiary watercourses with Plain Cement Concrete (PCC) technology. It failed to meet targets for forming 10 WUA networks, procuring ICT water monitoring equipment (deferred until larger channels were completed), installing irrigation pipe, constructing water reservoirs (later dropped due to cost), constructing washing stations, leveling land (later dropped due to cost), and conducting training visits for WUAs. • Year 3: AKRSP completed the irrigation master plan at the beginning of year three (April 2015). With the master plan in place, it hired contractors and began constructing the main and secondary channels. It did not meet its targets, however, because of community disagreement about the shape and size of the channels, internal community issues and interference with construction, delays in contractors’ start dates, and adverse weather (see detailed findings for more information).12 The master plan also recommended a layout of tertiary watercourses that did not correspond to the WUAs AKRSP had formed to date. Starting essentially from scratch, AKRSP realigned 111 WUAs to the new tertiary watercourse layout and registered 79. It also began work on the tertiary watercourses and completed 16 using Pre-cast Parabolic Section (PCPS) technology. • Year 4, quarter 1:13 AKRSP continued the activities of year three. It realigned an additional 16 WUAs, trained two in recordkeeping and maintenance, conducted exposure visits for WUAs, and held a WUA managers’ conference. It continued work on main and secondary channels and tertiary watercourses but failed to meet targets for either. • Current status: AKRSP has realigned 127 WUAs representing 93 percent of the life of project (LOP) target of 137. However, it is well behind cumulative targets for constructing main and 12 Aga Khan Foundation. (2015). 3rd Annual Progress Report: April 1, 2014 - March 31, 2015. Peshawar: Aga Khan Foundation. 13 The last quarterly report provided to the evaluation team. SATPARA DEVELOPMENT PROJECT EVALUATION 17 secondary channels and tertiary watercourses. At the end of the first quarter of year four, it has completed 12,553 meters of main and secondary channels, a figure that represents 43 percent of what AKRSP expected to have completed by that time and 29 percent of the 43,590 meters LOP target.14 It is difficult to determine the precise progress on tertiary watercourses, but AKRSP reports either working on or having completed 87 percent of the watercourses associated with LBC and 65 percent associated with RBC. The indicator “Number of acres with irrigation and drainage services” is most directly associated with component 1 activities. The project’s reported values against this indicator also illustrate that it is well behind targets for extending the irrigated area. By the end of Fiscal Year 2015, it had achieved only 38 percent of its cumulative target to that point in time and 12 percent of the LOP target. • Note: After the team submitted the draft report, AKF provided updated information on irrigation infrastructure through the end of the third quarter of year four – six months beyond the information the evaluation team had at its disposal. As of December 2015, AKF/AKRSP reports that it has completed 35,469 meters of a planned 34,684 meters of main and secondary channels (98 percent of the LOP target).15 Disagreements with communities still contribute to shortfalls in meeting targets. Since the interview data reflect the situation on the ground in the first quarter of 2015 (and not the updated information AKF subsequently reported), the evaluation team did not modify qualitative findings to reflect the new quantitative information on construction progress. The team did, however, incorporate the new information in the conclusions as appropriate. Even with construction of channels and tertiary watercourses behind schedule (at the time of the evaluation fieldwork), farmers have noticed a substantial improvement in water availability. Twenty-four of 31 groups (77 percent) reported that they had more water available for irrigation (Table 3). Seventeen of the 24 groups who reported increased water availability (71 percent) described improved availability in terms of greater quantities of water available, and all attributed the greater quantity to reduced seepage/waste as a result of lined irrigation channels. Twenty-one of the 24 groups (88 percent) described increased availability as more reliable or frequent access to water, and 16 of these groups (76 percent) attributed increased reliability/frequency to a faster flow of water in lined channels. “We don’t have to wait as long as previously the water flow was very slow and it took around 3-4 hours to receive water at any designated area. Now, the speed of water is increased tremendously. Now we receive water immediately at the time during continuous flow of water. Now, water is available for 24 hours in my area and I can use it any time as per my requirements. Watercourses also control the wastage of water as previously, most of the water was wasted due to seepage and weak channels developed by mud.” (Apricot orchard owner) “Another thing is that the quantity of water that we get now through these water channels is higher than what we received through the traditional water channels. This allows us to cultivate more land in the last two years than we used to cultivate before.” (Greenhouse owner) 14 AKRSP made significant additional progress constructing main and secondary channels after the end of the evaluation period. As of December 2015, it reported having completed 61 percent of the LOP target for main and secondary channels. 15 The differences between the LOP targets in this paragraph and the previous paragraph reflect revisions to the targets as planning and implementation progressed. SATPARA DEVELOPMENT PROJECT EVALUATION 18 Groups from outside the traditional Satpara Lake command area (those who do not currently get water from the dam) were just as likely as those who are getting water from the dam to report an increase in water availability. Seventy-eight percent of 17 groups outside the command area and 80 percent of the 5 groups within the command area reported increased water availability (Table 3). The tangible effects of increased water availability included being able to get irrigation water 24 hours per day (4 of 24 groups who said availability increased), faster flow and greater quantities that allowed farmers to irrigate their land more quickly (9 of 24 groups) and share excess water with others (2 of 24 groups), and being able to irrigate more frequently (4 of 24 groups). WUAs spoke more directly than other groups about the process of constructing channels and watercourses. Of the eight WUAs the team interviewed, half or more voiced negative perceptions of the design of the watercourses (7 of 8 WUAs), issues with costs (4 of 8 WUAs), and the process of constructing the watercourses (4 of 8 WUAs) (Table 13 in Annex 5). Concerns about the design of watercourses centered on a dislike or distrust of pre-cast parabolic section (PCPS) technology (7 of 8 WUAs) due to its perceived fragility (relative to the more familiar plain cement concrete (PCC) and reinforced cement concrete (RCC) technologies) and anticipated maintenance cost (3 of 4 WUAs). Three WUAs also believed the watercourses AKRSP proposed (and in some cases built) were too small to carry their allocation of water during the high flow summer months. One WUA explained that they wanted the capacity to receive all the water they were entitled too, even when the flow was high. Two specifically spoke of overflowing water and the associated waste and damage caused by the water eroding the banks of the watercourse and undercutting the structure. The evaluation team documented examples of broken, displaced, and eroded PCPS watercourses and some that were not properly backfilled, but it is not possible to determine the cause or whether the incidence of damage was higher or lower than with PCC and RCC. “Parabolic technology is vulnerable to climate factors, especially to severe freezing conditions in Skardu, which moves and dislodges the panels when the earth/foundation is frozen and thawed. They are also vulnerable to landslides, movement of tractors during the ploughing time, felling of trees, and breakage during transportation. The life of parabolic technology is five years at a maximum, while PCC can last for at least 20 years.” (WUA) All four WUAs (including the two interviews with groups of WUA presidents who collectively represented 12 WUAs) who voiced complaints about the process of developing the watercourses said that AKRSP should have consulted the community to learn about the local situation and community needs and preferences when designing the watercourses. Two of the eight WUAs (and 2 of 4 FEGs) said that AKRSP did not explain costs and responsibilities well, a problem they attributed in part to English language documents (2 of 2 WUAs and 1 of 4 FEGs). “However our only concern is that at time of devising the master plan we should have been involved or at least our requirements should have been considered, e.g., they wanted to construct a channel in our locality which was not sufficient for us and would eventually be a cause of concern for everybody so we did not agree with its construction.” (WUA) Representatives of LSOs raised a similar concern about the lack of communication with the communities. SATPARA DEVELOPMENT PROJECT EVALUATION 19 TABLE 3: WATER AVAILABILITY Characteristics of Increased Water Availability Analysis by Group Analysis by Locationa FFS FEGs Individual grant recipients WUAs WPGs Sum of all groups Satpara Dam command area Outside Satpara Dam command area Water availability increased 3 4 7 8 1 24 4 13 Quantity increased 2 3 5 6 1 17 2 11 Water loss reduced 2 3 5 6 1 17 2 11 Reliability/frequency increased 3 4 7 6 0 21 3 12 Faster flow 3 2 7 3 0 16 1 10 Number of documents 7 4 9 8 3 31 5 17 Excludes four individual grant recipients who were not farmers. a. The number of groups by location does not equal the total number of groups because it was not always possible to identify the location of a group, e.g., it may have contained individuals from both locations. “SDP prepared [the irrigation master plan] without consultation with the user communities. So, when it came to executing that plan, there were many technical and social problems. The master plan was based on many unrealistic assumptions based on stellate imagery, which were not validated on the ground. For instance, the design and size of major and minor channels was based on topography and not living settlements and the assumption that there will be water in RBC/LBC. It also ignored the water rights issues and the fact that there are many sources of water, other than Satpara, which the communities use during peak summer flows, for which larger water courses are needed.” (LSO representatives) In spite of complaints about design and process, however, many groups were pleased with the results. One WUA made a point of saying: “The development work undertaken by SDP and AKRSP during last three year is incomparable with the government activities during past fifty years and so we are completely satisfied with this whole program.” (WUA) For residents of the Satpara Dam command area, the positive results associated with improved irrigation infrastructure were largely independent of the incomplete status of LBC and RBC. In the areas outside the command area, however, improved infrastructure did not address the limitations imposed by obtaining water from seasonal streams. More importantly, inability to utilize the LBC and RBC set AKRSP back on meeting targets for extending irrigation to 15,500 acres. “They were assuming water availability all the time after the construction of RBC, but the Olding people did not give us dam water, and we have to rely on our own seasonal water. We told them that we need to divert all the water available in the nullah during the short summer season, but they did not listen to us. When it was built, water was overflowing, and now they SATPARA DEVELOPMENT PROJECT EVALUATION 20 are thinking of raising the height. They should have listened to us in the first place. Otherwise, our new watercourse is very good, the flow of water is fast and it is easy to operate.” (WUA) To improve water availability in areas outside the dam command area, AKRSP has commissioned a study of alternative water sources. Conclusions By reducing loss due to seepage and increasing the speed at which water moves, improved channels and tertiary watercourses have unquestionably increased the availability of irrigation water. Residents in the Satpara Dam command area also benefited from the regulated flow from the dam, which affected the seasonality of water availability. Those outside the command area, who still depend largely on seasonal streams, benefitted from more water when the streams were flowing but did not benefit as much from more seasonal reliability. Improvements in water availability saved time and labor, improved irrigation scheduling, and extended irrigated land. Meeting project objectives for irrigating 15,500 acres of land depends on being able to utilize the LBC and RBC to transfer water to the unserved areas. To meet targets, water rights holders must use water more efficiently and be willing to relinquish some of their rights (perhaps through a formal trade or market mechanism) and allow surplus water to flow in the LBC and RBC. Establishing an institutional structure to maintain the LBC and RBC is also necessary. Alternatively, if Shatong Nullah water is eventually diverted to the dam, residents outside the Satpara Lake command area will have rights to that water. Work on irrigation infrastructure initially progressed more slowly than anticipated largely due to AKF/AKRSP’s realization that a much more involved planning process was required. The time required to negotiate the respective roles of AKF and AKRSP in the cooperative agreement and consequent delays developing a grants manual and environmental documentation further delayed construction. Even without these delays, however, poor communication with communities to help them understand and accept the PCPS technology would have delayed implementation. The delays in improving irrigation infrastructure set back other activities that were contingent on the infrastructure. WUAs did not speak directly about whether increased water availability was sustainable. Indirectly, they questioned the durability of PCPS technology and mentioned that they had already observed broken sections and water loss. Apart from the issue of its perceived fragility, a perception based on very little direct experience over a short period of time, community resistance to the PCPS technology seems rooted in two issues. First, it is unfamiliar technology and many communities are not convinced that the smaller PCPS channels can deliver as much water (because of a faster flow rate) as the larger and more familiar PCC and RCC channels they have used in the past. Second, while communities can construct PCC and RCC channels with their own materials and labor, PCPS technology requires special materials and skills. Building and repairing PCPS channels thus requires a greater cash outlay and less in-kind (labor) contribution from the communities. Component 2: Enhancing Productivity of High -Value Horticulture and Dairy Products Activities under this component include forming and training farmer enterprise groups (FEGs) in the horticulture and dairy value chains. Under component 2, AKRSP also provides grants to establish registered nurseries and commercial orchards to support development of the fruit sub-sector; construct SATPARA DEVELOPMENT PROJECT EVALUATION 21 greenhouses to demonstrate off-season vegetable production and processing; establish agribusiness enterprises (e.g., farms, artificial insemination (AI) centers, milk collection centers, and feed suppliers) to support dairy sub-sector development; develop packaging design and production facilities; and construct a soil testing laboratory housed in the DoA. AKRSP also provides training to support these activities. The project’s results tracker (summarized in Table 18 in Annex 6 for component 2), narratives and results gleaned from annual reports (summarized in Table 20 in Annex 6 for component 2), and indicator values obtained from PakInfo (summarized in Table 19 in Annex 6 for component 2) provide quantitative evidence of project activities and results. Key results by year include: • Year 1: In year one, AKRSP began forming FEGs (6 of a planned 10) but stopped because the absence of a grants manual prevented it from providing grants. Consequently, except for facilitating a market exposure visit for six FEG members, it did not train FEG members as planned. With orchard and nursery activities, AKRSP provided four grants to establish fruit orchards and two grants to establish fruit nurseries. In vegetables, AKRSP began training in greenhouse management but had no other activities in its work plan. In the livestock sub-sector, AKRSP began preparations to train Department of Livestock (DoL) staff in AI. • Year 2: AKRSP reported no work with FEGs in year two. Its activities associated with orchards and nurseries included establishing 3 additional nurseries from an annual target of 4, distributing 20,000 plants, training nursery owners in nursery management, establishing 14 of a planned 16 orchards, and developing 2 certified nurseries. It did not improve 10 existing orchards as planned. Activities to support vegetable production included constructing 25 of a planned 20 plastic greenhouses, training owners in greenhouse management and post-harvest handling, and establishing 5 of a planned 10 vegetable and seed production farms. AKRSP did not conduct planned FFSs in seed production or off-season vegetable production as planned. • Year 3: In year 3, AKRSP formed 14 additional FEGs and trained them in packaging. It reported establishing an additional 14 orchards and 4 certified nurseries and training owners in orchard and nursery management, respectively. To support vegetable production, AKRSP reported providing grants for 150 glass greenhouses and training owners in greenhouse management, initiating FFSs in seed and off-season vegetable production, and engaging a consultant to redesign the greenhouses. In livestock, it trained farmers in fodder production and storage and trained livestock extension workers. It also reported establishing a youth-led enterprise for packaging design, establishing two enterprises for developing packaging, and completing the first phase of establishing a soil testing laboratory within the DoA. • Year 4, quarter 1: In the first quarter of year four, AKRSP reported forming and training four additional FEGs, providing packaging material to FEGs, conducting a feasibility study for a storage facility, establishing a model animal shed to protect cattle in the winter, and improving five existing sheds. • Current status: By the end of the first quarter of year four, AKRSP reported having formed 44 FEGs (but reports document only 13); establishing 32 model fruit orchards, 6 nurseries, and 4 certified nurseries; distributing 70,000 certified plants to nurseries; constructing 175 greenhouses; facilitating 28 FFSs in animal husbandry, fodder production and storage, seed production, and off-season vegetable production; and establishing a soil testing laboratory within the DoA. The analysis for component two draws on all interviews with groups and individuals directly involved in agricultural or livestock production. This includes seven FFS groups, four FEGs, nine individual grant SATPARA DEVELOPMENT PROJECT EVALUATION 22 recipients,16 and three WPGs. Some, but not all, received grants for infrastructure and equipment, and all received some form of training. SDP interventions were instrumental in encouraging beneficiaries to adopt new technologies and practices, especially for individual grant recipients and FEGs (Table 8 in Annex 5). Of nine production￾oriented grant recipients, seven said they lacked the knowledge or awareness to have adopted new technologies and practices without SDP support, and five said they lacked the financial capacity. Respondents mentioned financial constraints only in the context of relatively large investments such as orchards and nurseries (5 individual grant recipients) and greenhouses (2 of 4 FEGs, 1 of 3 WPGs, and 1 of 7 FFS groups). Similarly, knowledge limitations applied largely to technical knowledge about how to operate orchards, nurseries, and greenhouses (4 orchard and nursery groups representing 9 owners and 4 greenhouse groups representing 11 owners). “I would not have done it because we had less knowledge and resources. Root stock was also not available.” (Nursery grant recipient) “It won’t be possible for me to make such type of investments on my own, besides financial problems, it would be difficult for me to research and explore such type of trainings services, content, resource, greenhouse concept, technical and quality parameters, etc.” (Greenhouse grant recipient) The training and, to a lesser extent, infrastructure and equipment, encouraged SDP beneficiaries to adopt new management, production, marketing, and post-harvest practices (Table 8 in Annex 5). No strong patterns emerged across beneficiary types with the possible exception that FEGs and individual grant recipients were more likely than others to mention changing production and cultivation practices, particularly with respect to greenhouse production. The findings for component 3 present evidence of changes in marketing and post-harvest practices. This section presents findings for changes in other practices. Six of nine individual grant recipients mentioned changing their use of inputs. Four of eight respondents (2 FFS groups and 6 individual grant recipient groups) mentioned changing their use of fertilizers (natural and chemical) and using quality hybrid seeds. Three of the eight mentioned learning how to use pesticides properly. “We changed our practices pertaining to use of seeds, fertilizers, and agriculture chemicals after 2013. Before that we did not know about proper usage of pesticides and fertilizers. We didn’t know about the correct procedures to use organic or artificial fertilizers. Prior to assistance, we used to buy seeds from traders without any technical knowledge about the quality and varieties.” (Group interview with eight vegetable farmers) Seven individual grant recipient groups and FEGs spoke of growing different crops because of better profit potential. Six of the seven respondents mentioned shifting their emphasis to vegetables, and one shifted from vegetables to apricots. 16 It does not include the four individual grant recipients who received grants for processing or training. SATPARA DEVELOPMENT PROJECT EVALUATION 23 “I have a total of 19 kanal of land and I only used to grow cattle fodder before the program, the reason for that was low prices in the market. Now however, after the construction of my greenhouse I for the first time grew vegetables, mainly tomatoes. My tomatoes were available early in the market so they fetched me good prices and I sold them for Rs. 70 per kg, now more will be ready soon and they will be fetch me around Rs. 50 per kg.” (Group interview with female greenhouse owners) Individual grant recipients spoke of adopting new harvesting practices. They mentioned learning about the importance of handling fruit properly during harvest to maintain quality and improve shelf-life, using ladders or climbing trees to pick fruit individually instead of shaking the tree, and carefully harvesting nursery stock instead of allowing customers to “pluck” whatever they wanted. Respondents from all beneficiary types mentioned adopting new cultivation practices. The most common new practice they mentioned (8 of 10 respondents) was learning to sow seeds in beds with proper spacing instead of broadcasting the seed onto level ground. Both fruit growers spoke of learning the value of proper planting and pruning. “In case of tomatoes we would just sow it anywhere, there was no concept of plant bed, which would result in a lot of wastage. Once I received the greenhouse grant I started planting using bed methods and tried to change the old ways of plantation. It took time, like more than a year, however now the practices are changing and people are thinking in terms of doing business and making money.” (Greenhouse FEG) Ten respondents spoke about greenhouses, which they said had changed their practices by facilitating off-season production (4 of 10 respondents) and allowing them to produce seedlings for sale or to transplant into their own fields (8 of 10 respondents). “Since I received modern green house facility, now I learned about how to use water in greenhouses; how to control temperature; how to plant seed properly; what are the characteristics of high quality seeds, hybrid seeds; and what type of varieties suit the environment and soil condition in our area. Now I don’t use any type of pesticide in my greenhouses as it is hazardous for human health. I use farming tools in my greenhouses to properly cut, prune, and clean my plants.” (Greenhouse owner) Twenty of 23 production-oriented groups and individuals (a majority of all beneficiary types) reported that the practices they had learned and technologies they had adopted as a result of SDP support had increased the quantity of agricultural products they produced (Table 9 in Annex 5). FFS participants, FEGs, and individual grant recipients attributed increased quantities produced to increased yield and a transition to commercial farming. FFS participants and WPGs mentioned reduced waste. Table 4 summarizes the reasons respondents gave for increasing the quantities they produced. “A few years back selling milk was not considered respectable. After the program created some collection points, we go there to sell milk. This is all because of SDP, people are now even doing ‘Kist” system (kind of a loan in this case); so a person would take money from a shop keeper and purchase a cow and would sell milk from that cow to the shop keeper who in turn would pay him and also take some money off for the loan he has given out to the first person.”(FFS group) “Animals are now fed with more fodder and in fact good quality fodder. We have now also started feeding them with mulberry blocks, and also we give them water to drink now three to SATPARA DEVELOPMENT PROJECT EVALUATION 24 four times a day. These changes have increased the quantity of milk from our animals.” (Fodder production and dairy FFS participant) “Now however with the new and improved growing techniques and hybrid seed we are producing three times more from the same field as we used to before the training. At least 20 percent of produce was wasted within the field while some were damaged during transportation. However now that is not the case.” (Greenhouse owner) TABLE 4: CAUSES OF INCREASED QUANTITIES PRODUCED Result Explanation Frequency of Explanation High-yield hybrid vegetable seeds were more productive. 4 vegetable groups Increased yield Improved feeding and watering regimens increased milk yield. 3 dairy groups New cultivation practices reduced damage to orchard plants. 1 orchard group Planting in rows and off the ground reduced damage to vegetables in the field. 1 fruit production FFS group, 1 greenhouse FEG, 1 vegetable production FFS group Reduced waste Picking fruit instead of shaking the tree reduced fruit damage during harvest. 1 WPG group Grading out poor quality fruit for home consumption or processing reduced waste. 1 WPG group A ready market for mulberry blocks increased demand for surplus fruit that would otherwise have gone to waste. 1 dairy FFS group Chillers reduced milk spoilage. 1 dairy individual grant recipient Transition to commercial farming Farmers are starting to produce for more than just home consumption and are now producing to sell. 13 greenhouse groups and 1 dairy group Farmers are starting to approach farming as a primary business rather than a sideline. 3 groups Nineteen of 23 production-oriented groups (83 percent) specifically mentioned that the value of their sales had increased (Table 10 in Annex 5). However, no clear pattern emerged to explain the causes. All four FEGs attributed increased sales to increasing the quantity they produced, two of three WPGs attributed it to higher prices, and two of four FEGs reported receiving higher prices because they participated in the SDP-sponsored Friday bazaar. Three of the five groups and individuals who mentioned receiving a higher price said that prices have increased generally since the start of SDP but did not attribute the overall increase to SDP. One orchard owner reported that they now get higher prices because they sell collectively and have greater bargaining power with traders. The women in a group of WPG members mentioned receiving training in apricot drying techniques and getting higher prices for dried fruit due to better quality. A substantial number (half or more) of FEGs and individual grant recipients reported increasing employment as a result of SDP interventions (Table 11 in Annex 5). The two (of four) FEGs were greenhouse recipients. One reported hiring a person to sell seedlings. The other reported that they SATPARA DEVELOPMENT PROJECT EVALUATION 25 need to hire additional labor for weeding, pruning, and other general tasks because of increased production. Individual grant recipients were more likely to report increased employment. Eight of the 11 who reported increasing employment said they needed additional labor to handle increased production (1 orchard, 2 nurseries, 2 milk collection centers, and 3 greenhouses). Five said the new practices they had adopted required more labor. One greenhouse owner and one orchard owner said they hired temporary labor to help establish the orchard and construct the greenhouse. Only the milk collection centers said they hired permanent workers. Except for the milk collection centers, which hired only men, grant recipients said they hire both men and women. Fifteen of 23 production-oriented individuals and groups (6 of 7 FFS participants, 4 of 4 FEGs, 4 of 9 individual grant recipients, and 1 of 3 WPGs) believed the results they achieved through SDP were sustainable (Table 12 in Annex 5). They cited the profitability and effectiveness of the new technologies and practices and the value of the training and other support they had received through SDP. Two mentioned already investing in maintaining infrastructure, i.e., contributing to a community fund to maintain watercourses and replacing broken glass in greenhouses. “If our watercourse is damaged we will repair it. It is our common village asset and everyone understands this fact. It gives us benefits; we depend on this for our survival, so there is no question that we will maintain it. If we don’t maintain, it will be our own loss!” (WPG group) “It is our permanent dependency now. We will not stop practicing farming through greenhouses. If I had to sell my gold ornaments I would repair and/or construct greenhouse for me.” (Greenhouse FEG) Three respondents cited threats to sustainable results, including doubts about the availability of water in the future if the watercourses are not sustainable, and the need for further technical support. “Supply of water in future will be the only reason to discontinue such progressive practices. We are quite unsure about the channels sustainability in future.” (Group of vegetable farmers) “Our production from new plants is expected by 2018-19 whereas SDP is going to close in 2017. This situation is not good for us at all as we will be on our own without any proper guidance and further support in terms of processing, storing, and marketing of product.” (Orchard owner) Except for watercourses (covered in the findings of component 1) and training (covered in the findings of component 3), the only implementation issues mentioned by at least half of a particular type of respondent related to the grant process (2 of 4 FEGs) and the timing of activities (2 of 4 FEGs). The two comments on the grant process were that AKRSP did not pay on time (1 comment on watercourses) and that agreements and documentation were in English and not Urdu (2 respondent groups). The two comments on timing related to providing grants for greenhouses during the winter – before they were useful and when material costs were higher. Conclusions AKRSP promoted effective productivity-enhancing interventions and, for the most part, implemented them well. A large majority of beneficiaries credited the interventions with increasing the quantities of agricultural and livestock products they produced and sold. The greenhouses, in particular, seem to have spurred a shift from subsistence to commercial production, and participants are now seeing vegetable SATPARA DEVELOPMENT PROJECT EVALUATION 26 production as a viable business opportunity. In addition to the greenhouses, hybrid seeds and training seemed to be particularly effective interventions. Because component 2 is largely independent of component 1, i.e., the project’s productivity-enhancing interventions did not depend on improved availability of irrigation water, the inability to utilize the LBC and RBC to deliver irrigation water did not affect the results of component 2. However, the time it took USAID, AKF, and AKRSP to finalize roles in the cooperative agreement and the consequent delays preparing grants manuals and environmental compliance documents set back grant-funded activities and activities contingent on grant-funded infrastructure or equipment by at least a year. Component 3: Enhancing Processing and Marketing Capacity of the Area Component 3 focused on building local capacity to process and market agricultural and dairy products. Key activities included forming FEGs and WPGs, introducing the groups to improved processing practices and technologies, training group members (e.g., in processing practices, business, post-harvest handling, and marketing), and providing grants to support investments in processing enterprises. The project’s results tracker (summarized in Table 21 in Annex 6 for component 3), narratives and results gleaned from annual reports (summarized in Table 23 in Annex 6 for component 3), and indicator values obtained from PakInfo (summarized in Table 22 in Annex 6 for component 3) provide quantitative evidence of project activities and results. Key results by year include: • Year 1: In the first year, AKRSP focused on forming WPGs and training them in business development; building the capacity of its own staff in agribusiness, marketing, processing, and labor market analysis; and conducting market surveys and value chain assessments. It planned to issue grants to processing enterprises but could not due to absence of a grants manual and environmental documentation. It met most of its targets otherwise. • Year 2: In year two, AKRSP continued to form WPGs (16 against a target of 10) and FEGs (3) and trained members in fruit and vegetable processing. It also initiated studies on a variety of processing activities (e.g., fruit and vegetable processing, buckwheat processing, fresh milk, and yogurt) as a basis for establishing related enterprises. It failed to meet any of its targets for establishing the enterprises, however, citing unfinished feasibility studies. • Year 3: In year three, AKRSP formed 10 additional WPGs and provided grants to establish a milk processing center and a yogurt processing center. It missed targets for actually establishing the centers, however, because grantees had not yet procured equipment or started operations. AKRSP also provided cost share grants and associated training to WPGs for 20 dehydration units to dry apricots, cherries, mulberries, and other fruits and vegetables; established an animal husbandry FFS; took some local products from progressive farmers to exhibitions and fairs; and exposed four young web developers to market information systems. • Year 4, quarter 1: In the first quarter of year four, AKRSP continued forming and training processing groups; initiated grants for 25 additional home-based dehydration units with processing groups; and completed the milk collection center and yogurt processing center initiated in year three. • Current status: By the end of the first quarter of year four, AKRPS reports having formed 33 processing groups (men and women) and provided most (reports are not clear about the actual number) with dehydration units. It has trained FEGs and processor groups in business SATPARA DEVELOPMENT PROJECT EVALUATION 27 development, processing, and marketing; established a milk collection center and a yogurt processing center; and helped 15 farmers present their products at markets in Islamabad. The evaluation team’s interviews that touched on processing consisted of a group interview with seven farmers who received dehydration units and individual interviews with owners of two milk collection/processing centers. The following qualitative analysis of processing support is based on these interviews. The analysis of marketing support is based on all interviews with all FEGs, WPGs, individual grant recipients, and FFS participants (most of whom were producers who participated in production￾oriented activities). Processing None of the three FEGs or two WPGs that reported receiving dehydration units spoke specifically about using the dehydration units or about any outcomes associated with the dehydration units. The following narratives document findings from the dehydration unit group interview and the two milk collection centers. Dehydration units: The evaluation team conducted one focus group discussion (FGD) with seven dehydration unit owners. The group reported that AKRSP provided them with small plastic drying units and training in 2010. They did not say whether these units were effective or even whether they used them. The units AKRSP provided in 2013 were much more “effective.” The group reported that the dryers have opened their eyes to the market potential of apricots and that apricots are now one of the most important crops in their portfolio. When asked about the most important changes in their business in the past four years, they mentioned modern dehydrators which improve the quality and cleanliness of their dried produce and the fact that they can use the dehydration units year-round to dry other fruits and vegetables. They also reported that they would not have made the investment on their own because they were not aware of the technology and could not have afforded it. They expect to continue the drying activity because it is profitable. Milk collection center #1: The owner recounted that he started his business in 2012 to provide fresh milk in Skardu. The business grew gradually from selling a very small quantity of milk from the owner’s own 5 cows to working with 26, mainly female, suppliers delivering about 150 liters of milk per day. As the size of the business grew, the owner needed to establish a milk collection center to handle the increased volumes and preserve quality. AKRSP provided a cost-share grant to purchase a chiller, a boiler, a refrigerator, a motorbike (for delivering products), and milk testing equipment. The business now sells milk in plastic bags and also produces yogurt and butter. Since the owner started the business, the value of sales has increased from about Rs. 27,000 per month to about Rs. 240,000 per month – Rs. 130,000 from yogurt, Rs. 80,000 from milk, and Rs. 30,000 from butter. The grant recipient reported that he is selling about 240-320 kg of yogurt per day and employs three male workers in the collection center. The respondent attributed the growth in his business to SDP support, without which he said it would have taken several years to reach his current size. The respondent said that two problems he faced with his business were the limited availability of fresh fodder in winter and the low milk production of local breeds. He suggested that SDP should introduce improved breeds for milking, help create an association of dairy farmers, and introduce high-value fodder varieties. Milk collection center #2: The grant recipient reported that he started his milk and yogurt retail business three years ago after four years working in the dairy business. Before receiving support from SDP, he had a storage capacity of 50-60 liters in a small freezer. Because of load shedding, he would lose SATPARA DEVELOPMENT PROJECT EVALUATION 28 80-120 liters per month. He reported selling about Rs. 70,000 to Rs. 120,000 per month. He would collect the milk in reused plastic bottles, check the quality by taste or by dipping his hand in it, and package it in plastic bags for sale. He had no awareness of hygiene or food safety in his milk handling practices or in his shop. AKRSP provided him with a cost-sharing grant to purchase a 250 liter capacity chiller which has eliminated loss due to spoilage, a generator, milk testing equipment, and the finances to move to and equip a new shop. The grant recipient reported that he is now selling about 100-110 liters of milk and yogurt daily and earning Rs. 120,000 to Rs. 200,000 per month. Based on the training provided by AKRSP, he reported that he now collects milk in stainless steel containers (and encourages his suppliers to do the same). He and his employees are now aware of hygiene and food safety practices. He sells his milk in food-grade plastic bags and will soon use his own branded bags. He reported that he has hired one additional permanent employee because of increased volumes. The most important changes in his business are the chiller, generator, and testing equipment. The respondent mentioned that he has trouble sourcing sufficient quantities of high-quality milk. Local breeds are not productive, farmers do not have a tradition of selling milk, and they have no knowledge of proper animal husbandry practices required to improve milk quality and yield. He said he did not have the financial capacity or knowledge to have made the investments in his business without SDP support. Marketing The information in this section is based on all interviews with producers, i.e., FEGs, WPGs, individual grant recipients, and FFS participants. Of the 27 groups, 14 spoke of changing marketing practices, 4 of 4 FEGs, 5 of 13 individual grant recipients, 2 of 3 WPGs, and 3 of 7 FFS participants (Table 5). TABLE 5: CHANGE IN MARKETING PRACTICES BY GROUP Practice FEGs Individual Grant Recipients WPGs FFS Participants All Groups Marketing 4 5 2 3 14 Friday Market 4 - 1 3 8 Post-harvest practices 3 10 3 5 21 Grading 2 7 3 4 16 Storage 1 8 - 4 13 Packaging 1 9 2 2 14 Number of documents 4 13 3 7 27 Of the 14 groups who spoke about how they changed their marketing practices, 6 reported that they now sell directly to customers instead of through traders, and 4 of the 6 specifically said that the Friday bazaar contributes to this opportunity. Four other groups mentioned that they sold at the Friday bazaar but did not mention it in the context of selling directly (although that is implied). Two groups said they now grade their produce so they can charge a higher price for better quality. A fruit plant nursery owner reported that he used to price trees on the basis of size and age but now considers variety as SATPARA DEVELOPMENT PROJECT EVALUATION 29 well. Respondents attributed the change in their practices to SDP-supported training and to the opportunities to sell directly through the Friday bazaar. Table 5 also documents that 21of the 27 groups (78 percent) mentioned changing post-harvest practices related to marketing, i.e., grading (17 of 21 groups), storage (13 of 21 groups), and packaging (14 of 21 groups). Fourteen of the 17 groups who spoke of grading mentioned changing their grading criteria, e.g., from simply damaged/undamaged to the demand-driven criteria of size, color, and quality for fruits and vegetables and from age and size to variety for fruit trees. Of the 14 groups who mentioned adopting different grading criteria, 5 mentioned getting a better price for higher grades, 2 mentioned grading for personal consumption versus sales, and 2 mentioned selling higher grades and processing lower grades into jams or jellies. “We do three categories of fruit, one is the fruit that is fully ready to be ripened, is without spots, and is in good color. The second category is the produce that is ready to be ripened but with a little flaw on it like a spot or two, or if it does not look good in terms of its color. And the third category is the produce that is small in size and/or having spots on it or is a little damaged during the harvest.” (WPG) The 13 groups who spoke of storage mentioned “storing” fruits and vegetables for winter use by processing them into tomato paste, jams, jellies, etc. (4 of 13 groups); keeping them short-term, post￾harvest in plastic crates and in a cool place out of direct sunlight (4 of 14 groups); and using milk chillers to improve storage conditions and reduce spoilage (2 of 2 respondents). Both nursery owners spoke of changing their practices from essentially nothing to wrapping roots in jute bags, moistening the bags, and keeping them in a cool place until sold. “Since we got trainings on post-harvest and marketing practices, now I keep my produce in plastic crates with appropriate volume (20 – 22 kg per crate). I store my produce in a covered room to avoid any type of contamination and climate affects e.g. rains, heavy winds, high sunlight etc. I also clean my plastic crates with water prior to storing my produce in them.” (Greenhouse grant recipient) The 14 groups who spoke of packaging mentioned starting to use plastic crates or cardboard cartons for packaging fruits and vegetables (5 of 14 groups); packing fruits and vegetables in transparent plastic bags so buyers can see what they are getting (5 of 14 groups); and using branded cartons (2 of 14 groups). Four groups reported that improved packaging helped maintain the quality and extend the shelf life of fruits and vegetables. “Before [the training in] 2010 we used to take our produce in sacks to the market, with mud on them. The perception was that if the vegetables would have mud on them the buyer would buy it without asking whether it is fresh or not. However, now we properly wash them and remove all the mud and then take it to the market in plastic bags so that produce is clearly visible and the buyer gets attracted to it. It’s like how shopkeepers display their products in the showcases so that people get attracted to it and buy it.” (Vegetable FEG) AKRSP’s interventions to support processing and marketing relied to a large extent on training. A greater number of trainees expressed positive (5 of 27 groups) than negative (1 of 27 groups) views of the trainings and said the trainings were effective. The one negative opinion of the training process was that the training was conducted in English which was difficult to understand. Eight of 27 groups said they were happy with the training but wanted more. SATPARA DEVELOPMENT PROJECT EVALUATION 30 “We are very satisfied with these trainings and the content was easy to understand, they were very cooperative and would answer any questions we had very politely. They especially stressed us on asking as many questions as we want because they said you have an opportunity to learn so make the most of it.” (FFS participants – Dairy) Production, harvesting, processing, and marketing practices are complementary activities that contribute jointly to changes in higher-level outcomes such as the value of sales. Therefore, the findings and conclusions for component 2 cover the effects of processing and marketing on these outcomes. Similarly, it is difficult to separate the quality of implementation specific to marketing from that associated with processing. These too are covered in the findings for component 2. Conclusions AKRSP’s training in post-harvest practices seems to have been particularly effective in raising awareness of the value of proper grading, storage, and packaging practices. Its marketing training and exposure visits have sensitized producers to market demands and many now report grading, storing, and packaging to meet these demands. Although AKRSP did not focus on linking producers directly to buyers, its training seems to have changed the way beneficiaries interact with existing markets in beneficial ways. Other than the Friday bazaar, which has been a particularly effective innovation for linking producers directly to consumers, and introducing some farmers to markets in Islamabad, AKRSP did not seem to focus much on linking producers to alternative markets. Few producers mentioned new marketing channels but spoke instead of engaging with existing markets more knowledgably, i.e., grading to meet market demands, presenting produce attractively, collecting information on prices, selling collectively in larger quantities and with greater bargaining power. Investments in dairy processing appear effective on a very small scale in terms of both the size of the supported enterprises and the number of grants. Dehydration units appear to have been effective in increasing the profitability of apricot production and processing. Component 4: Create an Enabling Environment for the Agriculture Sector to Grow Component 4 was designed to facilitate and sustain project results. It addresses policy and strategic issues, and its main objective is to create a viable and favorable enabling environment to support agricultural sector growth. AKRSP planned four sets of interventions to achieve this objective. These included improving water governance systems and rules; enhancing the effectiveness of extension services; forming networks of WUAs, production, and marketing groups and enterprises to represent the groups’ interests; and establishing forums for policy dialogue among stakeholders. Since the outputs of the first three components directly contribute to component 4, the scope of this component is still evolving, and progress is still at an early stage. The project’s annual reports (summarized in SATPARA DEVELOPMENT PROJECT EVALUATION 31 Table 24 in Annex 6) provide the following picture of how component 4 activities have evolved. • Year 1: AKRSP did not plan any work on component 4 in year 1 beyond holding dialogues with the departments of agriculture, livestock, and water management to develop linkages and discuss collaboration. • Year 2: AKRSP planned to conduct two studies but deferred both. It facilitated a multi￾stakeholder meeting to review plans for the irrigation master plan; submitted a draft memorandum of understanding (MoU) to the Secretary of Agriculture GB to define roles and responsibilities of both parties in future collaboration; presented SDP’s activities to the Chief Secretary GB; and signed an MoU with the DoA to establish a soil testing laboratory in Skardu. It facilitated two exposure visits but deferred two activities planned for the year, forming WUA and FEG networks, and supporting extension services. • Year 3: AKRSP made little progress against its year 3 targets. It hired consultants for two studies slated for year 1, established a soil testing laboratory, and conducted 10 of a planned 12 thematic conferences. It made little or no progress on 12 additional planned activities, 5 of which were removed from the project scope of work. • First quarter of year 4: During the first quarter of year 4, AKRSP reported accomplishing each of the its three planned tasks: producing a water rights study, supporting LSOs/networks to form WUAs and handle social/water issues, and convening one thematic conference. To collect evidence for component 4, the evaluation team conducted in-depth interviews with seven key stakeholders or stakeholder groups, including representatives of WAPDA, DoA, Department of Education, Jheel Committee,17 GB administration, artificial insemination technicians (AITs), and LSOs. Interviews with AKRSP also contributed to the findings for component 4. Implementation of component 4 relies heavily on close interaction with policy makers. AKRSP reported that elections, frequent transfers of government staff, and the lack of local initiative or authority have made meaningful and consistent interaction difficult. Consequently, work on component 4 has proceeded in fits and starts. Interviews with public sector stakeholders revealed many challenges, including unresolved ownership and operations and maintenance responsibility for the dam and its canal and energy infrastructure; lack of capacity and motivation on the part of public sector partners; and lack of interest and policy initiatives by the GB government, which depends on the federal government for policy decisions. “The government line departments lack capacity to sustain extensive work done by SDP. The On-Farm Water Management Department (OFWMD) is very weak, is not fully established, and not oriented to manage water issues. SDP is providing training to OFWMD. The department has also been invited to oversee the construction work on watercourses on site and gain experience, but there is very little appetite for learning and taking on extra work.” (SDP personnel) 17 Lake Committee SATPARA DEVELOPMENT PROJECT EVALUATION 32 Despite a number of meetings with departmental heads and thematic conferences with relevant stakeholders, there is no evidence (e.g., draft papers or reports on policy issues and recommended reforms) of developing a firm agenda and consensus on the nature and scope of reforms to be undertaken. Improving Water Governance Systems In 2014, the Commissioner, Baltistan, issued a Notification, formalizing the Jheel Committee and expanding its membership to include public sector representatives, including the Deputy Commissioner, Skardu; Assistant Commissioner, Skardu; representatives from the Public Works Department and WAPDA; the Tehsildar and Naib Tehsildar;18 and permanent representatives of the 12 villages of Satpara. Members of the Jheel Committee believed that the committee’s notification and induction of public sector representatives into the committee may be the first step in developing water governing institutions. “The Jheel Committee should be upgraded to a formal water management institution with enhanced scope for maintenance, water efficiency, and water trading, but without altering the ownership and water rights of the original owners. The government departments and administration (AC/ DC) should only assist and facilitate the proper functioning of the Jheel Committee, and should not influence its decisions when it comes to the rights and distribution of water. The primary role of the government should be management and maintenance of the dam, power stations, and LBC/RBC.” (Jheel Committee members) Forming and registering WUAs and the notification of the Jheel Committee represent a transition from informal to formal water management institutions. Members of one WUA the evaluation team interviewed believed registration with OFWMD provided them legal cover, created the first tangible link with state institutions, and made them eligible for future support and services. WUAs also reported that training in record keeping has increased transparency and reduced the likelihood of conflict. “After formation and registration of the WUA, we have formalized the water management system. The WUA is our village institution that combines traditional and modern methods for water management. This is very important for unity and equal participation in decision-making in our community. The role of WUAs will improve with time, with the functioning of the entire system.” (WUA member) The transition to formal institutions has not been painless. WUAs are responsible for maintaining local irrigation infrastructure but have shifted largely from a labor-based to a cash-based approach to share costs among community members. Some communities are unhappy with this change, but others see the benefits (see findings for component 1). Enhancing Effectiveness of Extension Services AKRSP staff said that the high turnover of public sector employees has limited the effectiveness of its training and exposure visits. However, stakeholders did mention a number of examples of SDP’s effects on public sector extension services. These include: 18 Revenue officer and deputy revenue officer SATPARA DEVELOPMENT PROJECT EVALUATION 33 • AKRSP’s action to engage the Federal Seed Certification and Registration Department (FSCRD) in 2014 to screen and certify plants at SDP-supported nurseries started a local crop inspection and certification system. This is significant since fruit plants supplied from other sources are not certified and carry a risk of spreading disease. Availability of certified fruit plants adapted to the local environment is a major step forward. • AKRSP staff reported that the SDP-supported soil testing laboratory housed in the DoA is underutilized, and current staff does not know how to operate its hi-tech equipment. However, this may be a case of poor initial planning of a much needed investment, and is rectifiable. • The Director of DoA reported that the department and district administration are actively supporting SDP’s Friday bazaar initiative, which appears to have prompted some new thinking on the part of DoA. Findings of component 3 suggest that the Friday bazaar is a beneficial initiative for farmers. “The passive solar greenhouse technology introduced by SDP is very effective, which I have seen and like. In fact, I have formulated a PC-1 for public funding to replicate this design in other areas of Baltistan. Another idea of SDP, which I have supported and planning to take to scale, is the Friday Market for which I have prepared another PC-1. I have seen this (farmers’ market) in Japan and it works well as it generates direct and better cash incomes for farmers and serves as a powerful incentive for them to improve production and value addition. We can work with SDP to establish a more permanent platform for farmers to market their produce directly, as well as to serve as a point of aggregation.” (Director, DoA, Baltistan) • The shift to modern and beneficial production technologies and practices promoted by SDP has created a demand for public and private sector extension services. The DoA has been slow to respond to this demand, but senior managers are thinking about replicating some of these ideas by integrating them into their own annual development plans. • A key activity, though still in planning stages, is to establish agriculture resource centers (ARCs) under LSO/FEG Networks. The ARCs will bring private and community interests together, with public support, to work with and adopt a variety of new technologies promoted by SDP. The primary actors in ARCs are LSOs, FEGs, line departments, and small agribusinesses. AKRSP reports that it plans to increase the role of professional organizations, small businesses, and their networks in extension services to support the ARCs. AKRSP has accomplished little under the two final outputs of component 4: forming networks of WUAs, production, and marketing groups and enterprises to represent the groups’ interests and establishing forums for policy dialogue among stakeholders. AKRSP has accomplished more in component 4 than is apparent from project documents. The project philosophy has evolved from attempts to writing unwanted policy papers and guidelines and providing training to creating demand and open-sourcing services. The SDP Chief of Party said the project has changed activities, not outcomes, and that SDP has succeeded in devising a strategy to integrate component 4 objectives in the larger ecology for change and transformation, spearheaded by AKRSP in the project area. But to get there is a long process. Time is of essence and a limitation for SDP. Conclusions The enabling environment component has progressed slowly, largely because it has been difficult to engage policy makers and stakeholders. However, SDP has made some important headway. Formalizing WUAs and the Jheel Committee and expanding the Jheel Committee’s membership are important steps SATPARA DEVELOPMENT PROJECT EVALUATION 34 in developing representative water institutions with the authority and historical legitimacy to address emerging challenges and opportunities, such as redefining water rights, developing water markets, and attracting investments. SDP has also facilitated establishing local plant inspection and certification services and thus the supply of locally adapted certified rootstock, a major development for the fruit sub-sector. CONCLUSIONS Design The four project components were individually well designed to achieve objectives. Upgrading irrigation infrastructure under component 1 has been very effective in increasing the availability of water. Grant￾funded infrastructure and productivity-enhancing technologies and practices promoted under component 2 have effectively increased agricultural production, sales, and income. Investments in processing under component 3 have produced less dramatic results but have nevertheless increased production and processing capacity in locally important value chains, e.g., fruits, vegetables, and dairy. AKRSP has struggled to gain traction with enabling environment interventions under component 4 but has made some important strides in formalizing water management institutions. AKRSP staff believe they have adapted component 4 activities appropriately to the environment and still have a good chance to achieve objectives before the end of the project. Results are remarkably robust given the absence of a functioning LBC or RBC to deliver water to an expanded Satpara Dam command area. Improving irrigation infrastructure is still relevant, even in the areas intended to have been supplied by the LBC and RBC. Dropping activities designed to promote pressurized irrigation (deemed inappropriate given small landholdings) and land leveling (not cost￾effective) and the fact that drip irrigation systems have not yet been introduced mean there is no link between component 1 and components 2 and 3. This is because, with the possible exception of orchards and nurseries, results from productivity-enhancing practices and technologies are largely independent of improved access to irrigation water. The inability to extend irrigation water from the dam beyond the traditional command area has most affected components 1 (by limiting the additional land that can be irrigated) and 4 (by raising issues of water rights and putting pressure on institutions to resolve the problems). But these issues have made component 4 all the more important. The greatest impact the nonfunctional LBC and RBC have had on results is on the scale. SDP will almost certainly fail to meet its targets for irrigating 15,500 acres, and this will significantly reduce the magnitude of the benefits households in the extended areas will receive from improved irrigation infrastructure, i.e., the project will not be able to do much to regulate irrigation water supply throughout the agricultural season for households outside the traditional command area. Implementation Implementation progressed relatively smoothly. Few communities or individuals had substantive issues with the grant process, timing or sequencing of interventions, or interaction with AKRSP. The only major issue that emerged from interviews with communities involved irrigation infrastructure, which is addressed in more detail in subsequent conclusions. The time it took USAID, AKF, and AKRSP to resolve issues with the cooperative agreement deferred progress on producing a grants manual, developing environmental documentation, and hiring an engineer. These delays, in turn, substantially set SATPARA DEVELOPMENT PROJECT EVALUATION 35 back the start of grant-supported and major infrastructure interventions. These delays were particularly serious in the context of a seasonally dependent agricultural development project and may have affected project outcomes. Despite a late start on irrigation infrastructure, however, AKF and AKRSP moved quickly to complete the work in the second half of 2015 and are now largely finished with construction work on main and secondary channels. The ultimate resolution on the cooperative agreement created a cumbersome administrative relationship between AKF and AKRSP that further slowed some aspects of implementation, particularly grant-making. These issues had the combined effect of delaying implementation of key project components by over a year. AKF/AKRSP’s realization that a much more involved planning process than they originally anticipated was necessary to design the irrigation infrastructure also contributed to the delayed implementation of component 1 activities, greatly complicated the WUA formation process (i.e., causing AKRSP to realign most of the WUAs), and delayed important results. Furthermore, from the communities’ perspective, AKF did not consult them sufficiently in the planning process or in designing the channels and watercourses, and the plans demonstrated little sensitivity to the importance and nature of traditional water rights. The consequent community resistance to the size of proposed canals and watercourses further delayed component 1 activities and results. AKF/AKRSP state that they engaged in a very consultative process and that community concerns stem largely from their unfamiliarity with the PCPS technology and its performance. Nevertheless, community resistance delayed construction activities, and additional time sensitizing communities to PCPS may have attenuated these objections. Sustainability It is too soon to determine whether results are sustainable. The infrastructure which underpins many results (e.g., irrigation, greenhouses) appears to be relatively durable. In these cases, sustainability will depend more on whether communities and farmers have the capacities, and profit incentive, to use the infrastructure effectively without further support. Farmers will probably require additional technical assistance and experience to build and cement their skills before production and income results are fully sustainable. Building the capacity of public sector extension services will help, but SDP has not made much progress on this front yet. Enhancing and sustaining results from the Satpara Dam in general will require developing sustainable institutional arrangements to manage the dam, irrigation infrastructure, and water distribution. AKF/AKRSP have made some progress on this front by supporting formalization of WUAs and the Jheel Committee. Significant challenges remain, however, including resolving disputes about payment for services and responsibilities for managing the dam and maintaining the LBC and RBC. Farmers will likely need ongoing technical assistance to gain experience and cement their skills. Building the skills of public sector extension services to provide assistance relevant to the technologies and practices SDP has supported will enhance prospects for sustainable results at the producer and processor levels. AKF/AKRSP have not yet made substantial progress on this front, which threatens the sustainability and scalability of the SDP investment. SATPARA DEVELOPMENT PROJECT EVALUATION 36 RECOMMENDATIONS • USAID/Pakistan should continue investing in improving main and secondary channels and tertiary watercourses in the existing Satpara Dam command area and in the areas anticipated to be served by the LBC and RBC. Project-supported irrigation infrastructure has been extremely effective in increasing water availability, even in the areas not served by Satpara Dam, and will continue to do so for decades. • AKF/AKRSP, and future partners implementing similar projects, need to communicate better with affected communities and engage them more fully in planning. In particular, they will need to convince communities that PCPS technology is suitable to the area and durable. If failures noted by communities and the evaluation team are the result of poor construction or planning, AKF/AKRSP may need to better oversee contractors. AKRSP-supported exposure visits to established PCPS irrigation systems do not appear to have been widely effective, and broader exposure may be necessary. • While improved irrigation infrastructure in the areas not served by Satpara Dam has been effective, its impact on water availability will be limited (particularly in extending seasonal availability) until water from the source becomes more reliable. AKF/AKRSP should continue the work of identifying and developing alternative sources of water. If the assessment indicates that improving alternative water sources (above the LBC and RBC) has the potential substantially to improve water reliability, and if directing water into the LBC and RBC does not seem likely, then USAID/Pakistan may want to consider developing the alternative water sources. • If USAID/Pakistan decides to improve water sources in the areas served by the LBC and RBC, it should design the work so that water can relatively easily be redirected into the LBC and RBC if and when they become operational. In doing so, however, USAID/Pakistan will need to adequately consider existing water rights. • To get the most out of the investment it has already made in the LBC and RBC, USAID/Pakistan will have to facilitate resolving issues of infrastructure ownership and water rights. Unless USAID/Pakistan deems these issues intractable within a reasonable timeframe or determines that it does not have sufficient leverage to influence decisions, it should continue to invest in the enabling environment. Investments in creating an enabling environment are relatively low-cost and, in this case, could have a large payoff. • AKF/AKRSP’s work on components 2 and 3 has been effective. However, it will require more time and technical assistance to cement the skills and enhance prospects for sustainable results. For the remainder of the project, AKF/AKRSP should focus on supporting capacity building by providing technical assistance and not on providing new grants that may not have time to mature to sustainable status. • To enhance prospects for sustainability, AKF/AKRSP should focus on building sustainable public or private sector technical service providers who can support greenhouses, orchards, nurseries, processing, and marketing. SATPARA DEVELOPMENT PROJECT EVALUATION 37 ANNEXES Annex 1: Evaluation Statement of Work SATPARA DEVELOPMENT PROJECT EVALUATION 38 SATPARA DEVELOPMENT PROJECT EVALUATION 39 SATPARA DEVELOPMENT PROJECT EVALUATION 40 SATPARA DEVELOPMENT PROJECT EVALUATION 41 SATPARA DEVELOPMENT PROJECT EVALUATION 42 SATPARA DEVELOPMENT PROJECT EVALUATION 43 SATPARA DEVELOPMENT PROJECT EVALUATION 44 SATPARA DEVELOPMENT PROJECT EVALUATION 45 SATPARA DEVELOPMENT PROJECT EVALUATION 46 SATPARA DEVELOPMENT PROJECT EVALUATION 47 SATPARA DEVELOPMENT PROJECT EVALUATION 48 SATPARA DEVELOPMENT PROJECT EVALUATION 49 SATPARA DEVELOPMENT PROJECT EVALUATION 50 SATPARA DEVELOPMENT PROJECT EVALUATION 51 SATPARA DEVELOPMENT PROJECT EVALUATION 52 SATPARA DEVELOPMENT PROJECT EVALUATION 53 SATPARA DEVELOPMENT PROJECT EVALUATION 54 SATPARA DEVELOPMENT PROJECT EVALUATION 55 SATPARA DEVELOPMENT PROJECT EVALUATION 56 SATPARA DEVELOPMENT PROJECT EVALUATION 57 SATPARA DEVELOPMENT PROJECT EVALUATION 58 SATPARA DEVELOPMENT PROJECT EVALUATION 59 SATPARA DEVELOPMENT PROJECT EVALUATION 60 SATPARA DEVELOPMENT PROJECT EVALUATION 61 SATPARA DEVELOPMENT PROJECT EVALUATION 62 SATPARA DEVELOPMENT PROJECT EVALUATION 63 SATPARA DEVELOPMENT PROJECT EVALUATION 64 SATPARA DEVELOPMENT PROJECT EVALUATION 65 Annex 2: List of Documents Reviewed Aga Khan Foundation. (2012). 2nd Quarterly Progress Report: July 1, 2012 - September 30, 2012. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2012). Project Monitoring Plan (PMP). Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2012). Quarterly Progress Report No 1: March 12, 2012 - June 30, 2012. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2013). 1st Quarterly Progress Report - Year 2: April 1, 2013 - June 3-, 2013. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2013). 3rd Quarterly Progress Report: October 1, 2012 - December 31, 2012. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2013). 6th Quarterly Progress Report: July 1, 2013 - September 30, 2013. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2014). 1st Quarterly Progress Report - Year 3: April 1, 2014 - June 30, 2014. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2014). 2nd Annual Progress Report: April 1, 2013 - March 31, 2014. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2014). 2nd Quarterly Progress Report - Year 3: July 1, 2014 - September 30, 2014. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2014). 3rd Quarterly Progress Report - Year 2: October 1, 2013 - December 31, 2014. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2014). 4th Quarterly Progress Report: January 1, 2014 - March 31, 2014. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2014). Third Annual Work Plan: April 1, 2014 - March 31, 2015. Peshwarar: Aga Khan Foundation. Aga Khan Foundation. (2015). 1st Quarterly Progress Report - Year 4: April 1, 2015 - June 30, 2015. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2015). 3rd Annual Progress Report: April 1, 2014 - March 31, 2015. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2015). 3rd Quarterly Progress Report - Year 3: October 1, 2014 - December 31, 2014. Peshawar: Aga Khan Foundation. Aga Khan Foundation. (2015). 4th Quarterly Progress Report - Year 3: January 1, 2015 - March 31, 2015. Peshawar: Aga Khan Foundation. SATPARA DEVELOPMENT PROJECT EVALUATION 66 Aga Khan Foundation. (2015). Fourth Annual Work Plan: April 1, 2015 - March 31, 2016. Peshawar: Aga Khan Foundation. Government of Gilgit-Baltistan; Asian Development Bank; World Bank. (2011). Gilgit-Baltistan Economic Report: Broadening the Transformation. Khan, M. J., Zulfiqar, M., & Ali, J. (June 2015). Traditional Irrigatoin Practices and Proposed Efficient Irrigation Water Management system for SDP Command Area (Benefitting from Satpara Dam) Skardu - Satpara Development Project. Peshawar: University of Agriculture, Peshawar. International Food Policy Research Institute (IFPRI) and Aga Khan Foundation (AKF). (No date). Satpara Development Project Household Survey Baseline Report. Peshawar: Aga Khan Foundation. USAID/Pakistan. (2012). Cooperative Agreement No. AID-391-A-12-0002. Islamabad: USAID. SATPARA DEVELOPMENT PROJECT EVALUATION 67 Annex 3: Data Collection Instruments The Satpara Development Project Evaluation Interview Guide – USAID 1. Were you involved in the original design of SDP? 2. What is your role in managing the project? 3. Do you have specific concerns about how the project was designed or implemented? Explain. 4. To what extent did the project design specifically address sustainability and how? a. Does it have a documented strategy for sustainability or an exit strategy? b. In your opinion, to what extent is the approach to sustainability likely to be effective and why? c. How, if at all, could it have been more effective? d. Are some results, or results for some groups, more likely to be sustainable than others? e. What are the primary obstacles to sustainable results? 5. Did the project design include a specific strategy for including women? Explain. a. In your opinion, to what extent has the approach to engaging/benefiting women been effective and why? b. How, if at all, could it have been more effective? 6. Did AKRSP implement the project as designed? If not, how and why did it deviate from the design or planned implementation approaches? 7. To what extent were the planned interventions sufficient for achieving desired results? a. What other interventions should the project have implemented that wasn’t part of the original design to address weaknesses or gaps in the various value chains? 8. Were initial targets realistic and achievable? If not, why not? 9. Did AKRSP encounter any particular challenges implementing the project? Explain. (Probes: managing grants, selecting appropriate partners, identifying appropriate technical expertise, facilitating links between value chain actors, …) a. In your opinion, has the partner adapted in an appropriate manner to minimize the impacts of problems it has encountered? Why or why not? b. What could the partner have done better to respond to problems? 10. Does the project have a specific strategy for sustainability or an exit strategy? 11. Can you think of examples where the project’s intervention has leveraged private sector resources to enhance results throughout the value chain? 12. What do you think is the project’s biggest success? 13. What do you think could have been done better? SATPARA DEVELOPMENT PROJECT EVALUATION 68 The Satpara Development Project Evaluation Interview Guide – AKRSP Management 1. What challenges or problems have you encountered implementing this project and how have you dealt with those problems? a. In particular, the project is behind on many targets. What were the primary causes for delays and what has AKRSP done to adapt design or implementation to minimize the impacts of delays? b. In retrospect, are there things you would change if you could about the project design or implementation approach? Explain. 2. Other than issues we just discussed, did AKRSP implement the project as designed? If not, how and why did you deviate from the design or planned implementation approaches? 3. In your opinion, what are the strengths of the project design? 4. In your opinion, what are the weaknesses of the project design? 5. To what extent were the planned interventions sufficient for achieving desired results? 6. What other interventions should the project have implemented that weren’t part of the original design to address weaknesses or gaps in the various value chains? 7. To what extent did the project incorporate sustainability into the design of SDP? (Probe for whether it was an explicit objective or stated in a concept note or strategy) a. What are the prospects for sustainable results? Is there anything else the project or future projects should do to ensure sustainability? b. Do you have an exit strategy? What is it? 8. Was there a specific objective of including women in the design of SDP? If so, what was the approach? Did it work? (probe for the presence of a gender strategy, training,…) 9. What do you think is the project’s biggest success? 10. What do you think could have been done better? 11. Other than the issues we’ve already discussed, did you face any other challenges during implementation? Probe for: a. Changes in forming and working with groups b. Issues around water rights c. Managing grants d. Linking value chain actors e. Managing vendors for infrastructure 12. What constitutes a successful FEG, WPG, WUA, FFS? Add questions that arise during the field work. SATPARA DEVELOPMENT PROJECT EVALUATION 69 SDP Beneficiary Instrument Note to moderator: Do not mention SDP or AKRSP in your introduction. Assalam-o-alaikum. Thank you for meeting with us today. We are grateful that you are giving us your time. Before we begin, we will introduce ourselves. [The moderator and the note-taker should introduce themselves]. We are here to discuss with you about the support you may have received to improve your production of agricultural and livestock products and how that support has affected you. Through this discussion, we will ensure that your ideas and opinions are heard. We would like to record these discussions so that we remember and do not miss any of the ideas that you give us. The details of these discussions will not be shared with anyone and your names will be kept confidential - so please do not worry and feel free to express your ideas. Your participation in this discussion is voluntary and we really hope you are willing to participate actively and with enthusiasm. We would like to ask you about your production and sales of agriculture and livestock products. Will you allow us to record this discussion? Thank you. Note to moderators: AKRSP has been active in the area for a long time and respondents may have received other kinds of support from AKRSP. Always make sure they are telling you about assistance they received under the Satpara Development Project that started in 2012. Do not mention SDP or AKRSP until they are mentioned in a question. Introduction 1. What is your main business related to agriculture or livestock? Probe to understand whether the respondent’s primary agriculture or livestock business is focused on production, processing, marketing, etc. 2. Can you please tell me a little about your business? Ask as appropriate to the type of business: a. What do you produce/process/sell? b. What crops/livestock do you grow/raise? c. How much land do you farm? d. Do you employ labor other than family members? e. How long have you been in this business? Changes in Practices since 2010 (before SDP) Note to moderator: This section focuses on if and how respondents have changed the way they produce, process, and sell agriculture and livestock products since 2010, a date well before the start of SDP. Do not ask about specific SDP interventions and do not mention SDP or AKRSP. We are interested only in whether practices have changed, how, and why. Ask the “why not” question only when a respondent says they were exposed to a change in practice or technology that they chose not adopt. Probe very thoroughly on the how, why, and why not. When probing on how, ask specifically how they did it in the past (2010) and how they do it now. Ask why not only when the respondent said they received support or advice that they chose not to adopt? 3. Compared to 2010 have you changed the way you produce, process, or market agricultural or livestock products? Ask each of the following sub-questions that are applicable to the respondent. a. Have you changed the way you produce agricultural or livestock products? i. Have you changed your use of inputs such as seed, fertilizer, agricultural chemicals, or other technologies or practices (e.g., using artificial SATPARA DEVELOPMENT PROJECT EVALUATION 70 insemination)? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? ii. Have you changed your production practices (e.g., using different infrastructure or tools, changed the time of planting or harvesting, changed the crops you grow, changed livestock feeding practices, changed the way you manage livestock, etc.)? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? b. Have you changed the way you harvest agricultural products or collect milk? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? c. Have you changed the way you handle agricultural or livestock products after harvest? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? Focus on post-harvest practices such as cooling or other practices that affect quality immediately after harvest and before sale or storage. d. Have you changed your practices with respect to grading? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? In most cases, they probably did not grade so give them the option of saying they did not or do not grade. e. Have you changed the way you store agricultural or livestock products prior to sale or use? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? f. Have you changed the way you package your agricultural or livestock products for sale? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? g. Have you changed how or where you sell your agricultural or livestock products? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? h. Have you changed the way you transport your agricultural or livestock products to buyers? If yes, how did you do it in 2010 and how do you do it now? Why did you change your practices?/why not? Change in Employment 4. Do both men and women work in your business or is it only men or only women? i. If both men and women work, ask: What jobs do men do? What jobs do women do? j. Has the number of people working in your business changed since 2010? If yes how and why? (Probe for producing/processing/ marketing large quantities, more/less efficient practices, more/less labor intensive practices) i. Ask only if employment increased: 1. What kinds of jobs were created? (skilled/unskilled, seasonal/full-time, men/women) 2. Are new workers unpaid family members or did you hire new labor from outside? 1. If they hired labor from outside the family, ask: Do you know what the new workers you hired were doing before you hired them? Probe to understand whether the jobs were new jobs (hire people who were not working) or better jobs (hired people who were working for less) SATPARA DEVELOPMENT PROJECT EVALUATION 71 Outcomes Associated with Changes in Practices The purposes of this question are to understand a) what worked and what did not and b) establish a credible causal link between SDP interventions and outcomes (quantity produced value of sales, incomes). Probe very thoroughly to understand how each type of change the respondent reported in the previous questions contributed to the outcomes. It may be difficult to separate individual effects of each change in practices or technologies. In that case, probe thoroughly to understand if and how the entire group of changes (not individual changes) the respondent mentioned in the previous questions affected production, sales, and income associated with agricultural or livestock products. Probe thoroughly to understand how changes in technologies or practices affected outcomes. For example: If quantity produced increased was it because of a greenhouse or the training, was it because of higher yields (what caused higher yields?), or more area planted to the crop (why were they able to plant more area?). For value of sales and income, probe to understand if sales/income increased because of selling more (why are they selling more – more to sell or access to higher value markets), higher prices (because of access to higher value markets, better quality, value addition)? Ask specifically about the effects of type of support they r reported receiving. 5. Have the changes in your agricultural and livestock production, processing, and marketing practices affected the quantity of products you produced, the value of your sales, or your income from agricultural and livestock activities? How? 6. Based on what we’ve discussed, what do you think are the most important changes in your business in the past four years? Why were they the most important? 7. Since 2010, are there other things that we have not discussed that affected your production, sales, income, or the number of people you employ? Please explain? This refers to anything they have not yet mentioned and should include factors outside of the project’s influence such as the economy, weather, etc. External Support for Water, Production, Processing, and Marketing 8. Since 2010, did you receive any support to help construct or improve watercourses or canals that affected your agricultural or livestock production? If the answer is yes, ask the following: a. Has constructing/improving watercourses changed the way your community manages irrigation water? Explain. Ask how they managed water before (2010) and how they manage water now and why it changed. i. Who provided this support? Probe for all sources. b. Has the support for constructing/improving watercourses affected your situation with respect to water for irrigation? (Let respondents tell you about effects but probe if necessary for getting water more often, getting more water when it’s your turn, water lasts longer into the dry season) c. How does your community manage irrigation water? Explain. d. Do you believe your community will face any challenges maintaining the watercourses in the future? Explain. Probe to understand how the community will share costs and labor involved in maintaining watercourses. SATPARA DEVELOPMENT PROJECT EVALUATION 72 9. Since 2010, what other kinds of support have you received to support your production, processing, or marketing of agricultural and livestock products? We don’t need a lot of detail with this question. The purpose is to attribute the outcomes they’ve mentioned to SDP interventions. k. Did you receive any support to help you improve production of agricultural or livestock products? i. Infrastructure (e.g., greenhouses, watercourses/canals, irrigation equipment, etc.), ii. Technical assistance/training (e.g., exposure visits, farmer field schools, etc.), iii. Tools, or iv. Other support v. Who provided the support? Probe for all sources. l. Did you receive any support to help you process agricultural or livestock products? i. Infrastructure (e.g., processing equipment), ii. Technical assistance/training (e.g., exposure visits, farmer field schools, etc.), iii. Tools, or iv. Other support v. Who provided the support? Probe for all sources. m. Did you receive any support to help you sell agricultural or livestock products? i. Technical assistance/training (e.g., exposure visits, farmer field schools, etc.), ii. Tools/packaging, or iii. Other support iv. Who provided the support? Probe for all sources. SDP Support We want to talk now only about the support you received from the Satpara Development Project implemented by AKRSP. 10. Did you receive any grants from SDP/AKRSP? 11. If yes: a. What was the grant for? b. How did you learn about the grant opportunity? c. How was the process of applying for and receiving a grant? Probe for ease of the application process, how it was administered, timeliness of receipt, other. A recipient may have received several grants. Ask about each type of grant separately. 12. Could you have made these investments without the grant assistance from SDP? If yes, how? If not, why not? Probe for access to financial resources, knowledge of the technologies and practices, and comfort with the investment. SATPARA DEVELOPMENT PROJECT EVALUATION 73 Sustainability 13. You’ve mentioned that your production, sales, and income (ask only as supported by previous answers) have increased since 2010. Can you think of any reasons you may not be able to sustain these increases in the future? Probe for things like inability to afford improved technologies or practices, lack of external support, increasing input prices, limited access to inputs or technical assistance, lack of sustainable market connections, etc. Sufficiency and Implementation 14. Do you face other problems increasing production or sales? Explain. Probe for constraints in other parts of the value chain, e.g., availability of BDS, financing, TA/human resource capacity, etc. 15. Other than things we’ve already discussed, do you have suggestions for how the support you received from SDP could have been improved? Probe as appropriate for quality of assistance/TA, timeliness of support in the context of agricultural seasons, sequencing of interventions, relevance to recipients’ needs, grant process if relevant. 16. Do you know of examples of other businesses that have started copying the practices or technologies you are using because of SDP? Explain. Probe to understand whether others are replicating the technologies and practices, e.g., building a greenhouse, adopting practices SATPARA DEVELOPMENT PROJECT EVALUATION 74 The Satpara Development Project Evaluation Interview Guide – Enabling Environment Stakeholders Water management questions, Representatives WAPDA 1. What are the key characteristics of the Satpara Watershed? 2. What are the key purposes/ functions of Satpara Reservoir? 3. What is the future management plan? 4. Is design appropriate? 5. What are the key results/ benefits that have been realized? 6. What are major issues? 7. Are LBC and RBC a failure? 8. Is WAPDA going to transfer the management of Reservoir to GB? When? 9. What is the annual revenue of the dam? From water, from electricity? Other services? 10. What are the revenue grow forecasts for the next 5 years? 11. Is the project financially sustainable? Water management questions, Representatives of AKF/SDP Team 1. Water management before and after SDP? 2. What does SDP do that is different? 3. Is design appropriate? 4. Evolving roles in water management, your comments? 5. Alternatives to traditional practices, your comments? 6. What could have been done better? 7. Sustainability? Water management questions, Representatives DoA/ Water Management 1. What is the role of government agencies, communities, and private sector actors in water use in agriculture? 2. Specifically, who is responsible for developing, managing and regulating water resources, O&M of infrastructure, and ensuring equal access to water for irrigation in SDP Command area? 3. Who are the key stakeholders in water /agriculture management? 4. Please describe the salient features of water rights in SDP Command area? SATPARA DEVELOPMENT PROJECT EVALUATION 75 5. What are the strengths, weaknesses, opportunities and threats (SWOT) of traditional water management system in SDP command area? 6. What suggestions can you make to improve irrigation efficiency and increasing the value of agricultural production? 7. What can be done to formalize and update traditional water management systems in SDP command area? 8. What support/ incentives are needed to assist communities to democratically reform their traditional water management rules/ customs? Water management questions, Representatives of WUAs/ LSOs/ VOs/ WOs 1. What are the key objectives of WUAs? 2. What is the legal status of WUAs? 3. What is the role of WUAs in water management? 4. How different this role is from traditional system? 5. Do you follow a set of rules and regulations? 6. Who makes and amends such rules? 7. Are there any disputes in the way WUAs work? 8. How these disputes are resolved? 9. Does WUAs collects water charges? 10. 1How does WUAs pays for it services? 11. Does WUAs have a business plan? 12. Have you thought of revenue generation by selling surplus water to non-water right holders? 13. Are WUAs work autonomous and each works differently? 14. Is there a common Charter of WUAs? 15. What role LSOs play in forming WUAs? 16. What are some of your achievements? 17. Name some of the major issues that WUAs face 18. What is your future plans? Water markets questions, owners of greenhouses, orchards, nurseries, vegetable farms and vegetable shop owners at local vegetable market 1. DO you get water in adequate quantity and quality on time to meet your requirements? 2. Do you have water rights? 3. Do you pay any water charges? SATPARA DEVELOPMENT PROJECT EVALUATION 76 4. Are you willing to pay for extra water if you need it? 5. What are your water requirements and are you satisfied by the services of WUAs? Water rights questions, from representatives of farmers who are in the extended SDP command area 1. Do you have water rights for your land, if not why not? 2. Are you willing to pay for the water that you need for your land to irrigate? 3. Are you willing to accept some sort of water/ land swapping arrangements? Agricultural extension questions, Representatives of DoA 1. Please describe the state of agricultural extension services, their structure, effectiveness and gaps? 2. Who are the providers of agricultural extension services, and what are some of the areas of improvement? 3. Give us some examples of public-private partnerships (PPPs) in agricultural extension services? 4. What policy changes are required to promote PPPs in the SDP Command area? 5. What is the experience of soil testing lab (STL), which is a PPP project? 6. Is STL is being used by farmers? 7. Are SDP interventions are relevant and implemented in the right way? 8. What are some of the gaps for the provision of quality inputs for agricultural production? 9. What is the potential and what can be done to produce high quality seeds in SDP command area? 10. Do you know what SDP is planning to do in this area? Good Agricultural Practices (GAP) questions, Representatives of FSCRD 1. What are some of the barriers on local agricultural products to compete in national and international markets? 2. Who are the main actors for research, setting standards, certification regimes, and promotion and branding? 3. What do you think of developing standards and establishing a Mountain GAP? 4. What local or national agencies can help in establishing Mountain GAP, provision of training and monitoring? FEG Network Questions, Representatives of FEGs 1. What do FEGs do and how are they organized? SATPARA DEVELOPMENT PROJECT EVALUATION 77 2. What services do they provide to individual producers? 3. How FEG has worked for you personally? 4. What advantages do you see in creating a network of FEGs? Policy Dialog questions, Representatives of GBLA/ District Gov 1. Who makes laws to regulate water for agriculture? 2. Is there an irrigation/agricultural policy that is in force in GB? 3. What are some of the pressing needs/ priorities to reform agricultural policy? 4. What role do you see for private sector / WUAs in Water management? 5. Are you willing to consider water trading? 6. What legal instruments can be created to manage water efficiently, equitably and sustainably? SATPARA DEVELOPMENT PROJECT EVALUATION 78 SDP Evaluation Group Discussion Guide – Water Users’ Associations Assalam-o-alaikum. Thank you for meeting with us today. We are grateful that you are giving us your time. Before we begin, we will introduce ourselves. [The moderator and the note-taker should introduce themselves] We are here to discuss with you about the Satpara Development Project which is being implemented by the Aga Khan Rural Support Program (AKRSP) and how this project may have affected you. Through this discussion, we will ensure that your ideas and opinions are heard. We would like to record these discussions so that we remember and do not miss any of the ideas that you give us. The details of these discussions will not be shared with anyone and your names will be kept confidential - so please do not worry and feel free to express your ideas. Your participation in this discussion is voluntary and we really hope you are willing to participate actively and with enthusiasm. We would like to ask you about your experience with date growing and harvesting and selling and how the different training and equipment you have received has helped you improve your production, harvesting and farm sales. Will you allow us to record this discussion? Thank you Note to moderators: AKRSP has been active in the area for a long time and respondents may have received other kinds of support from AKRSP. Always make sure they are telling you about assistance they received under the Satpara Development Project that started in 2012. 1. Where does your village get water for irrigation? (Probe: Satpara Dam, other sources) 2. Please tell about your water user association? a. When was it formed? b. When was it registered and with whom? c. What is the role of the WUA in managing water in your village? (Probe: maintain water channels, distribution of water among members) d. How many households belong to your WUA? e. How many kilometers long is the irrigation system you command? f. How many acres of land are irrigated through the system your WUA manages? 3. How do you allocate water among the WUA members? Probe about: a. Process b. Recording system c. Payment / Fees /Penalties d. Dispute resolution 4. Is there a system of trading/selling water among your WUA members or other villages? Please explain. 5. Have members of this WUA ever traded or sold water between themselves or with other villages? Can you explain how that worked and the result? Probe to understand the details of the transaction, e.g., terms of payment or trade, duration of the agreement, who had to agree to the sale/trade, etc.) 6. What kind of support have you received from SDP to improve the existing water courses? SATPARA DEVELOPMENT PROJECT EVALUATION 79 7. How was your WUA involved in the improvement of the water course with SDP assistance? (Probe: labor, money, in-kind.) 8. Are you satisfied or dissatisfied with improved water courses? Please explain reason for satisfaction or dissatisfaction. (Probe for changes in the quantity of water, reduced loss, easier to manage, etc.. Also probe to understand how the improvements in the water course affected these outcomes.) a. Has the amount of water your members receive changed as a result of the improved water courses? 9. What other assistance did your WUA receive? Let respondents talk and probe for the following if they don’t mention them: a. Formation/registration of WUA b. Training in record keeping and management c. Exposure visits 10. Has this assistance changed the way you manage and distribute water to your member households? How? (Probe: water fees, wastage/seepage, accounting, fines for misuse of water, allocation of water to different members of WUA.) 11. How did you manage water distribution before SDP? Probe for details. 12. Do you find the current management structure better or worse than how you managed water before SDP? Why? 13. Do you work with the OFWM department? a. How do you work with OFWM? b. Has the nature of your work with them changed since you started working with SDP? How? 14. In your opinion, how likely is it that the WUA will be able to continue playing its role for the next 20 years? 15. Is there anything that might prevent the WUA from performing these roles in the future? (Probe: lack of financial resources, insufficient expertise, disagreement from members.) 16. Are there other things that need to be done to improve access to irrigation water for households in your area? Explain. 17. Is there anything that SDP could have done better? Please explain. SATPARA DEVELOPMENT PROJECT EVALUATION 80 The Satpara Development Project Evaluation Interview Guide – Artificial Insemination Trainees Learn how the respondent knows SDP (by what name) and use that name throughout the interview. 1. Can you briefly describe your business? a. What do you do? b. How long have you been in this business? c. Do you own this business? 2. When did you begin receiving support from SDP? 3. What kinds of support have you or your company received from SDP. (Probe for training, tools/equipment, exposure visits, grants, other) a. Grants: If the respondent received a grant, ask about the grant, e.g., purpose, application process, ease of application, how it was administered, timeliness of receipt, other. b. Could you have made these investments in your business without SDP assistance? If yes, how? If not, why not? 4. Have you changed the way you do your business as a result of the assistance? How? If not, why not? (Probe for relevance of each type of assistance) 5. Did these changes affect the value of your sales? If yes how? (Probe for access to different markets, higher prices, larger quantities sold, standardization, certifications, exports) 6. Do you employ only men, only women, or both men and women? a. Did these changes affect the number of workers you employ? If yes how? (Probe for producing/processing/marketing large quantities, more/less efficient practices, more/less labor intensive practices) i. If employment increased: What kinds of jobs were created (skilled/unskilled, seasonal/full-time, men/women) ii. Phrase as relevant: What roles do men and women have in your business? iii. Ask only if employment increased: Do you know what the new workers you hired were doing before you hired them? Probe to understand whether the jobs were new jobs (hire people who were not working) or better jobs (hired people who were working for less) 7. As relevant: You've told us that sales/exports/jobs have changed as a result of assistance from SDP. Do you think these are permanent changes or are they only temporary? Explain. (Probe for reasons results might be temporary) 8. Are there other changes that could take place in your industry that would improve your business? Explain. (Probe for constraints in other parts of the value chain. Probe specifically for availability of BDS, financing, TA/human resource capacity) 9. Other than things we've already discussed, do you have suggestions for how the assistance could have been improved? (Probe as appropriate for quality of assistance/TA, timing, sequencing, relevance, completeness, other) SATPARA DEVELOPMENT PROJECT EVALUATION 81 10. Do you know of examples of other businesses that have started copying the practices or technologies you are using because of SDP. Explain. Probe to understand whether the replication was really a result of a demonstration effect of SDP? SATPARA DEVELOPMENT PROJECT EVALUATION 82 Annex 4: Fieldwork Schedule and List of Interviews TABLE 6: LIST OF INTERVIEWS AND SITE VISITS Organization Position Type of Interview Location Role in TAP/Evaluation AKF ?? Individual Islamabad Management AKRSP SDP M&E Specialist Individual Islamabad Implementer AKRSP CEO Head of Engineer Unit Social Development M&E Governance Value Chain Group Skardu Implementer WUA Astana Pine President and focal Person Group Astana Pine WUA WUA Brazgang Arungpa President and focal person Group Brazgang Arungpa WUA Individual Farmer Owner Individual Kushmara Greenhouse recipient Individual Farmer Owner Individual Shigari Kalan Greenhouse recipient WUA Biafo President and members Group Biafo WUA Milk collection and processing center Owner Individual Yadgar Chowk Grant recipient Milk collection and processing center Owner Individual Biafo Grant recipient FEG Focal Person Group Greenhouse grant recipients FEG Focal Person Group Sundus Greenhouse grant recipients WPG Focal Person Group Fapa Apricot WPG WPG Focal Person Group Thorgu pine(Thombo Grong) Mulberry WPG WPG Focal Person Group Rzasna yaar khor Apple WPG Deosai Yogurt Center Owner Individual Patwal Olding Grant recipient FFS - Focal person Group Shairthang FFS participants FFS - Focal person Group Sundus FFS participants SATPARA DEVELOPMENT PROJECT EVALUATION 83 Organization Position Type of Interview Location Role in TAP/Evaluation Individual farmer Owner Individual Rizvia Individual nursery grant recipient Individual farmer Owner Individual Astana Individual nursery grant recipient Individual farmer Owner Individual Gamba Individual orchard grant recipient Individual farmer Owner Individual Malkhor Gamba Individual orchard grant recipient WAPDA Project Director, SATPARA Dam Executive Engineer, Satpara Dam Superintendent Engineer, Satpara Dam Group Skardu Management of Satpara Dam Department of Agriculture Director Individual Skardu Head of Agriculture Department Orchard Owner Group Focal person Group Ghayul Individual orchard grant recipients Vegetable group Focal Person Group Astana Pine Individual vegetable grant recipient FEG Focal Person Group Fapa HV/OSV FEG FEG Focal Person Group HV/OSV FEG WUA President and focal Person Group Khalang Ranga WUA WUA President and focal Person Group Shargran WUA WUAs Presidents 1 President Group Skardu WUA WUAs Presidents 2 Presidents Group Skardu WUA WUA Newranga Cluster President and Focal Person Group Newranga WUA FFS Focal Person Group Sondus Gond FFS participants FFS Focal Person Group Bain FFS participants Green House Focal person Group Shagri Kalan Greenhouse grant recipient Department of Agriculture Focal Person Group Skardu Nursery and orchard management trainees Greenhouse Owner Focal Person Group Skardu Greenhouse grant recipients SATPARA DEVELOPMENT PROJECT EVALUATION 84 Organization Position Type of Interview Location Role in TAP/Evaluation FFS Focal Person Group Shagri Kalan FFS participants – animal husbandry (dairy) FFs Focal Person Group Gangchan Colony FFS participants – vegetable production Department of Education Focal Person Group Skardu Soil testing trainees AI Trainees Focal Person Group Skardu AI trainees Jheel Committee Member and focal Person Group AKRSP Office Members of Jheel Committee AKRSP CEO Head of Engineer Unit Social Development M&E Governance Value Chain Group Skardu Implementer Government of Administration Commissioner, Baltistan Individual Commissioner Office Skardu Stakeholder FF Dehydration Unit Focal Person Group Dehydration unit grant recipients LSOs Focal Person Group LSO Kisan, Office Skardu Stakeholders SATPARA DEVELOPMENT PROJECT EVALUATION 85 Annex 5: Qualitative Data Analysis Tables TABLE 7: SDP CONTRIBUTION TO ADOPTING TECHNOLOGIES AND PRACTICES Barriers to Adoption FFS FEGs Individual Grant Recipients WPG All Production Groups AITs WUAs Lacked financial capacity 1 2 5 1 9 1 n.a. Lacked knowledge/awareness/comfort 0 1 7 1 9 1 n.a. Number of documents 7 4 9 3 23 2 8 Shaded cells represent a majority of respondents in a column. SATPARA DEVELOPMENT PROJECT EVALUATION 86 TABLE 8: CHANGE IN PRACTICES Practice FFS FEGs Individual Grant Recipients WPG All Production Groups AITs WUAs No change 1 - 2 - 3 - n.a. Inputs 2 - 6 - 8 - n.a. New management practices 1 - 2 - 3 1 n.a. New crops/experience 1 2 5 - 8 1 n.a. Improved/new varieties 4 4 6 - 14 1 n.a. Changed cropping patterns 2 1 4 - 7 - n.a. Transportation - - 4 - 4 - n.a. Animal husbandry 3 - - - 3 - n.a. Marketing 3 4 4 2 13 1 n.a. Friday Market 3 4 - 1 8 - n.a. Post-harvest practices 5 3 7 3 18 1 n.a. Grading 4 2 6 3 15 - n.a. Storage 4 1 5 - 10 1 n.a. Packaging 2 1 6 2 11 1 n.a. Harvesting 1 2 4 1 8 - n.a. Processing 3 1 2 2 8 1 n.a. Production 1 4 8 1 15 1 n.a. Cultivation Practices 2 4 3 1 10 - n.a. Greenhouse 1 4 4 1 10 - n.a. Fruit Drying - - - 1 1 - n.a. Nursery 1 4 3 - 8 - n.a. Off-season - 1 2 1 4 - n.a. Number of documents 7 4 9 3 23 2 8 Shaded cells represent a majority of respondents in a column. SATPARA DEVELOPMENT PROJECT EVALUATION 87 TABLE 9: REASONS FOR INCREASING QUANTITY PRODUCED Reasons for Changes in Quantity Produced FFS FEGs Individual Grant Recipients WPGs All Production Groups AITs WUAs N.A. - no production - - 1 - 1 - n.a. Increased 6 4 7 3 20 1 n.a. Wastage reduced 4 1 1 2 8 - n.a. Increased yield 4 4 4 1 13 - n.a. Increased area 2 0 1 1 4 - n.a. Speculative 0 0 1 - 1 - n.a. Transition to commercial farming 6 3 7 1 17 - n.a. Technology and practices (post-harvest) - - - - - - n.a. Technology and practices (harvest) - - 1 - 1 - n.a. Technology and practices (pre-harvest) - 1 1 - 2 - n.a. Improved varieties - 1 1 - 2 - n.a. Decreased - - 3 - 3 - n.a. External factors - - 3 - 3 - n.a. Number of documents 7 4 9 3 23 2 8 Shaded cells represent a majority of respondents in a column. SATPARA DEVELOPMENT PROJECT EVALUATION 88 TABLE 10: CHANGE IN VALUE OF SALES Change in Value of Sales/Cause FFS FEGs Individual Grant Recipients WPGs All Production Groups AITs WUAs Not applicable, no production yet 0 0 1 0 1 - n.a. Increased 6 4 6 3 19 2 n.a. Market intelligence 0 1 0 0 1 - n.a. Quantity 3 4 3 1 11 2 n.a. Price 0 0 3 2 5 - n.a. Quality 1 1 3 1 6 1 n.a. Friday Market 0 2 0 1 3 - n.a. Value Addition 2 0 0 0 2 1 n.a. Other 0 0 1 0 1 - n.a. Number of documents 7 4 9 3 23 2 8 Shaded cells represent a majority of respondents in a column. TABLE 11: EMPLOYMENT FFS FEGs Individual Grant Recipients WPGs All Production Groups AITs WUAs All Groups Increased 2 2 11 - 12 1 - 16 No change 4 1 3 2 9 1 - 11 Opportunities for men and women Men and women - - 7 - 6 - 1 8 Mostly men - - 1 - - 1 - 2 Mostly women - - 3 - 2 - - 3 Number of documents 7 4 13 3 27 2 8 37 Shaded cells represent a majority of respondents in a column. SATPARA DEVELOPMENT PROJECT EVALUATION 89 TABLE 12: PROSPECTS FOR SUSTAINABLE RESULTS Sustainability FFS FEGs Individual Grant Recipients WPGs All Production Groups AITs WUAs No 1 - 2 - 3 - 1 Yes 6 4 4 1 15 2 - Only with access to inputs 0 - 1 - 1 - - Number of documents 7 4 9 3 23 2 8 Shaded cells represent a majority of respondents in a column. SATPARA DEVELOPMENT PROJECT EVALUATION 90 TABLE 13: SATISFACTION WITH IMPLEMENTATION Dimension of Implementation FFS FEGs Individual Grant Recipients WUAs All Production Groups AITs WUAs Activities Watercourses Incomplete 2 3 - - 10 - 3 Complete - 1 - - 2 - 1 Negative - 3 2 - 10 - 4 Positive - 1 1 - 4 - - Seeds/plants - - Positive - - 1 - 2 - - Negative - - 1 - 2 - - Tools/equipment Positive - - - - - - 2 Negative - 1 - 1 4 - - Infrastructure/material quality - Positive - 1 - - 2 - 2 Negative - 1 - - 2 - - Infrastructure design - Positive - - - - - - 1 Negative - 1 2 1 8 - 7 Training Negative 2 2 0 2 12 - 1 Positive 4 - - - 8 - - Management Group formation process Positive - 1 - - 2 - - Grant process Negative 2 2 1 - 10 - - Positive - - 4 1 10 1 - Marketing Positive - 1 - - 2 - - Negative 1 - 1 1 6 - - Timing Positive 0 1 - - 2 - - Negative 1 2 - - 6 - - SATPARA DEVELOPMENT PROJECT EVALUATION 91 Dimension of Implementation FFS FEGs Individual Grant Recipients WUAs All Production Groups AITs WUAs Infrastructure process Positive - - - - - - 1 Negative 0 1 1 - 4 - 2 Other Positive 1 - - - 2 - - Negative - - 2 - 4 - - Infrastructure cost - - - 1 2 - 4 Number of documents 7 4 9 3 23 2 8 Shaded cells represent a majority of respondents in a column SATPARA DEVELOPMENT PROJECT EVALUATION 92 Annex 6: Performance Data TABLE 14: TARGETS AND ACHIEVED RESULTS – COMPONENT 1 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Results to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Planning Hire consultant for survey and irrigation master plan Days 180 - - 180 180 - - - - 180 Survey for irrigation master plan Survey 1 1 1 - - - - - - 1 Identify, conduct feasibility studies, and design (tertiary) irrigation projectsa Projects 200 50 24 - - - - - - - Assessment study of water resources outside existing Satpara Dam Study - - - - - - - 1 1 b Forming and strengthening WUAs Form WUAs (re-alignment) WUAs 138 - - - - 78 87 Form and realign WUA's WUAs - - - - - - - 14 16 16 Form WUA networks Network 10 - - 10 - - - - - - Train WUAs Trainings 16 - - 16 5 4 3 2 2 10 Exposure visits (WUAs) Visits 6 - - 2 - 2 2 1 1 3 WUA managers conference Conferences - - - - - - - 1 1 1 Building farmers’ capacities for using irrigation water more efficiently Establish demo plots Plots 4 4 - - - - - - - - Exposure visits (community members) Visits 6 2 1 2 - - - 2 1 2 Install pressurized irrigation system Acres 25 - - - - 25 - - - SATPARA DEVELOPMENT PROJECT EVALUATION 93 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Results to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Install drip irrigation system Acres - - - - - - - 2 9 c Infrastructure Procure ICT based water monitoring equipment Equipment 75 - - 75 - - - - - - Provide/install irrigation pipes Meters - - 500 - - - - - - Construct water reservoirs Projects 6 - - 6 - - - - - - Construct washing areas Projects 50 - - 50 - - - - - - Land leveling Kanals 400 - - 400 - - - - - - Construct main & secondary channels (LBC- Package 1) Meters 11,168 - - - - 4,629 1,316 3,991 2,820 4,136 Construct main & secondary channels (LBC Package 2) Meters 14,425 - - - - 5,700 1,799 5,405 - 1,799 Construct main & secondary channels (RBC- Package 1) Meters 8,402 - - - - 1,890 921 2,400 1,720 2,641 Construct main & secondary channels (RBC- Package 2) Meters 8,805 - - - - 2,816 1,727 2,350 2,250 3,977 Hold dialogues and start construction of watercoursesa Projects 200 50 20 - - - - - - - Construct watercourses (Chaks) at LBC Construct watercourses (Chaks) at RBC Chaks Chaks 138 - - - - 39 45 22 8 d - - - - 50 27 35 28 d Building staff capacity Engineering staff capacity building training Trainings - - - - - - - 2 2 2 Exposure visit (field engineers to efficient irrigation systems) Visits - - - - - - - 1 1 1 SATPARA DEVELOPMENT PROJECT EVALUATION 94 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Results to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Exposure visit (staff to efficient irrigation systems Visits - - - - - - - 1 - - Source: AKRSP results tracker a. Appears in the year 1 annual report but not in AKRSP’s results tracker. b. Consultant selected, final proposal received c. Potential sites selected d. Work in progress SATPARA DEVELOPMENT PROJECT EVALUATION 95 TABLE 15: PERCENTAGE OF TARGET ACHIEVED BY PROJECT YEAR – COMPONENT 1 Activities Unit LOP Target % of cumulative target achieved % of LOP target Notes Year 1 Year 2 Year 3 Year 4, First Quarter Preparation of irrigation master plan Survey 1 0% 0% 100% 100% 100% Proposals shortlisted in year 1, consultant hired in year 2, plan completed early year 3 Establish demo plots on farmers land Plots 4 0% 0% 0% 0% 0% Exposure visit to community members (national) Visit 6 50% 25% 25% 33% 33% Form WUA networks at primary/secondary canal level Network 10 0% 0% 0% 0% Procurement of ICT base water monitoring equipment Equipment 75 0% 0% 0% 0% Dropped Hire consultant for survey and preparation of master plan Days 180 100% 100% 100% 100% Provision/installation of irrigation water pipes Meters 0% 0% 0% 0% Dropped Construction of water reservoirs Projects 6 0% 0% 0% 0% Const. of washing areas Projects 50 0% 0% 0% 0% Dropped Land leveling Kanals 400 0% 0% 0% 0% Dropped Training to WUAs Training 16 31% 40% 45% 63% Exposure visit to WUAs (National) Visit 6 0% 50% 60% 50% Construction of Main & Secondary Channels LBC￾Package 1 Meters 11,168 28% 48% 37% Late start due to community disagreement with shape and size of channels Construction of Main & Secondary Channels LBC Package 2 Meters 14,425 32% 16% 12% Construction of Main & Secondary Channels RBC￾Package 1 Meters 8,402 49% 62% 31% SATPARA DEVELOPMENT PROJECT EVALUATION 96 Activities Unit LOP Target % of cumulative target achieved % of LOP target Notes Year 1 Year 2 Year 3 Year 4, First Quarter Construction of Main & Secondary Channels RBC￾Package 2 Meters 8,805 61% 77% 45% Construction of Watercourses (Chaks) at LBC Chaks 61 115% 87% 87% Work in progress Construction of Watercourses (Chaks) at RBC Chaks 85 54% 65% 65% Work in progress Installation of pressurized Irrigation System Acre 25 0% 0% 0% Dropped Installation of drip irrigation system Acres 450% n.a. Sites selected Assessment study of water resources outside existing Satpara Dam Study 100% n.a. Consultant identified, final proposal received WUA managers conference Conferences 100% n.a. No LOP target Engineering staff capacity building training Training 100% n.a. No LOP target Exposure visit of field staff to efficient irrigation systems of Pakistan (Field Engineers) Visit 100% n.a. No LOP target Exposure visit of staff to efficient irrigation systems (Regional) Visit 0% n.a. No LOP target Formation and realignment of WUA's WUAs 200 182% 151% 81% 93% 93% Results in year 3 reflect the realignment which required AKRSP to reformulate existing WUAs and reduced the target from 200 to 138 Source: AKRSP results tracker Note: Blank cells indicate there was no target for the activity in that year. 0% means there was a target. SATPARA DEVELOPMENT PROJECT EVALUATION 97 TABLE 16: EXPLANATION OF REPORTED RESULTS – COMPONENT 1 Activity Year 1 (March 2012-March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Planning Prepare irrigation master plan Expected to be completed in year 1 but only received proposals N.R. N.R. The AKRSP team leader for infrastructure reported that the master plan was completed in April 2014. Engage consultant to conduct survey and prepare irrigation master plan N.R. Consultant engaged, survey and irrigation master plan finalized Not in year 3 work plan Develop Programmatic Environmental Documentation Form (PEDF) Submitted revised form to USAID for approval Develop grants manual Submitted to AOR Revised thresholds to allow AKF to authorize grants below $20,000. Study to assess water resources outside dam command area Not in year 1 work plan Not in year 2 work plan Not in year 3 work plan Consultant hired Form, realign, register, strengthen WUAs Form WUAs 91 WUAs formed of a planned 50. 84 registered. Of a target of 100, formed 127 WUAs and registered 73. Abandoned all previously formed WUAs due to irrigation plan. Reformed/realigned 111 WUAs and registered 79. Reformed/realigned 16 WUAs and registered 19. Train WUAs in construction and record keeping N.R. From a target of 16, 5 WUAs trained. Not in year 3 work plan 2 trainings against target of 2 Form WUA networks at primary/secondary canal level N.R. Of a target of 10, created no WUA networks after discussions with USAID deferred the activity to year 3 pending institutional assessment of existing WUAs Not in year 3 work plan SATPARA DEVELOPMENT PROJECT EVALUATION 98 Activity Year 1 (March 2012-March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Exposure visits (WUAs) N.R. 2 planned, none conducted 2 planned, 2 conducted 1 planned, 1 conducted WUA managers conference N.R. Not in year 2 work plan Not in year 3 work plan 1 planned, 1 conducted Improve capacity to use irrigation water more efficiently Establish demonstration plots No plots established due to absence of project engineer, irrigation master plan, environmental documentation, and grants manual Not in year 2 work plan Not in year 3 work plan Exposure visits (communities) Fell short of target (1 of 2 planned visits) due to absence of project engineer, irrigation master plan, environmental documentation, and grants manual Target of 2 deferred to next quarter N.R. 1 visit conducted against target of 2 Train farmers in efficient irrigation No training delivered due to absence of project engineer, irrigation master plan, environmental documentation, and grants manual N.R. N.R. Install pressurized irrigation system N.R. N.R. 25 acres planned, nothing reported Install drip irrigation system N.R. N.R. N.R. 9 sites selected from a target of 2 Land leveling N.R. Dropped due to shortage of funds N.R. SATPARA DEVELOPMENT PROJECT EVALUATION 99 Activity Year 1 (March 2012-March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Construct main and secondary channels 1,316 meters constructed Construction of Main & Secondary Channels LBC￾Package 1 N.R. N.R. against target of 4,629 meters (28%). Late start due to community disagreement with shape and size of channel. Work ended in December due 2,820 meters constructed against target of 3,991 meters (71%). to adverse weather. 1,799 meters constructed Construction of Main & Secondary Channels LBC￾Package 2 N.R. N.R. against target of 5,700 meters (32%). Late start due to community disagreement with shape and size of channel and contractor’s inability to mobilize 0 meters constructed against target of 5,405 meters (0%). equipment and labor on time. Construction of Main & Secondary Channels RBC Package 1 N.R. N.R. 921 meters constructed against target of 1,890 meters (49%). Late start due to community disagreement with shape and size of channel 1,720 meters constructed against target of 2,400 meters (72%). 1,727 meters constructed Construction of Main & Secondary Channels RBC￾Package 2 N.R. N.R. against target of 2,816 meters (61%). Late start due to community disagreement with shape and size of channel and interference from the 2,250 meters constructed against target of 2,350 meters (96%). community. Construct tertiary watercourses Identify, conduct feasibility survey, and design, and construct irrigation projects (tertiary watercourses) Target of 50. Completed 24 plans. Fell short of targets due to absence of project engineer, irrigation master plan, environmental documentation, and grants manual Of a target of 100, constructed 94 watercourses serving 26 acres using PCC, work in progress on remaining 6. SATPARA DEVELOPMENT PROJECT EVALUATION 100 Activity Year 1 (March 2012-March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Hold third dialogues and start constructing watercourses 16 projects submitted to AKF, 4 projects approved, no projects started Construction of Watercourses (Chaks) at LBC Completed 11 watercourses of a target of 39. Remainder in progress. AKRSP shifted to using PCPS in year 3. 8 completed against target of 22. Remainder in progress Construction of Watercourses (Chaks) at RBC Completed 5 watercourses of a target of 50. Remainder in progress. AKRSP shifted to using PCPS in year 3. 28 completed against target of 35. Remainder in progress Other infrastructure/equipment Install piped irrigation N.R. Target of 500 meters, none installed. Will include in new packages. N.R. Construct water reservoirs N.R. Dropped due to shortage of funds N.R. Construct washing areas N.R. None constructed from an annual target of 50. Will include in new packages. N.R. Procure ICT water monitoring equipment N.R. Deferred until larger irrigation channels are completed and WUA networks formed. N.R. Human resource capacity building Train staff and government on environmental compliance Target achieved N.R. N.R. Engineering staff capacity building training N.R. N.R. N.R. 2 trainings conducted against target of 2 Exposure visit to efficient irrigation systems (field engineers) N.R. N.R. N.R. 1 visit conducted against target of 1 SATPARA DEVELOPMENT PROJECT EVALUATION 101 Activity Year 1 (March 2012-March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Exposure visit to efficient irrigation systems (staff) N.R. N.R. N.R. 0 visits conducted against target of 1 Sources: Year1, Year2, and Year 3 Annual Progress Reports. AKRSP results tracker. N.R. means “not reported”. SATPARA DEVELOPMENT PROJECT EVALUATION 102 TABLE 17: PROGRESS AGAINST INDICATORS – COMPONENT 1 Indicator Type FY 2013 FY 2014 FY 2015 LOP Target 2.1.1b. Number of micro, small and medium enterprises (MSMEs), including farmers, receiving business development services from USG assisted sources (also MSF 3.4.2-a, F 4.5.2-37) Target 165 205 13 625 Actual 270 330 139 2.1.2b. Number of persons receiving training on skill development (also MSF 3.4.1b) Target 1,224 1,472 1,221 4,124 Actual (male) 339 377 572 Actual (female) 218 749 1,183 Actual (total) 557 1,126 1,755 2.1.3a. Number of acres with irrigation and drainage services Target 2,372 4,316 4,789 15,500 Actual 272 1,222 1,801a 2.1.3b. Number of farmers and others who have applied new technologies or management practices as a result of USG assistance (F 4.5.2-5) Target 21,703 3,000 1,406 5,000 Actual 102 2,085 1,990 2.1.3d. Number of hectares under improved technologies and management practices as a result of USG assistance (F 4.5.2-2) Target 1,920 1,440 1,842 4,481 Actual 119 677 750 2.1a. Value of incremental sales attributed to program implementation (F 4.5.2-23- mod) Target 18,718 20,590 61,959 Actual 5,141 24,888 0 2.1b. Value of exports of targeted commodities as a result of USG assistance (F 4.5.2- 36-mod) Target 3,631 131,779 243,904 610,380 Actual 3,631 118,941 248,998 2.1c. Number of micro and small enterprises linked to a larger-scale firm as result of USG assistance to the value chain Target 150 250 226 750 Actual 200 98 148 2.1d. Number of rural households benefiting directly from USG interventions (F 4.5.2- 13) Target 14,043 2,585 1,050 8,547 Actual 4,051 2,396 3,548 2.2.2a. Number of USG-assisted organizations that participate in legislative proceedings and/or engage in advocacy at all levels Target 4 3 4 10 Actual 25 4 1 2.2a. Number of policies/regulations/administrative procedures in development stages of analysis, drafting and consultation, legislative review, approval or implementation as a result of USG assistance (F 4.5.1-24) Target 2 1 1 3 Actual 0 0 0 0 SATPARA DEVELOPMENT PROJECT EVALUATION 103 Indicator Type FY 2013 FY 2014 FY 2015 LOP Target 2a. Number of jobs attributed to program implementation (also MSF 3.4a, F 4.5-2 - mod) Target 944 1,566 1,486 4,850 Actual (male) 353 827 265 Actual (female) 17 681 27 Actual (total) 370 1,508 292 2c. Project-related household incomes of USG targeted beneficiaries Target 15% 15% 15% 15% Baseline 29,050 8,664 897 Income 930,680 237,828 7,738 Percent change 3% 4% 12% 2d. Proportion of female participants in USG-assisted programs designed to increase access to productive economic resources (assets, credit, income or employment)(F GNDR-2) Target 30% 30% 30% 35% # of women 1,934 98 35 Total 6,550 161 67 % women 30% 61% 52% 3.3a. Number of beneficiaries receiving improved infrastructure services due to USG assistance (also MSF 2.1.3e, F 4.4-8) Target Actual (male) 2,080 6,093 3,654 Actual (female) 2,080 6,092 3,654 Actual (total) 4,160 12,185 7,308 a. As of December 2015 (after the end of the evaluation period) AKF reported that SDP had extended irrigation and drainage services to 9,500 acres. SATPARA DEVELOPMENT PROJECT EVALUATION 104 TABLE 18: TARGETS AND ACHIEVED RESULTS – COMPONENT 2 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Result to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Baseline survey and mapping Survey 1 1 1 0 0 0 0 0 0 1 FEGs FEGs formed FEGs 50 10 6 20 20 7 14 4 4 44 FEGs trained in use of improved packaging Training - - - - - 1 1 0 0 1 Training of FEG members Training - - - - - 7 4 4 4 8 Grants to FEGs for production level activities FEGs - - - - - - - 1 1 1 Orchards and nurseries Nursery management training Training 2 1 - - 1 1 1 - - 2 Fruit orchards management training Training 3 1 - - - 1 1 - - 1 Mother fruit nurseries established Nursery 6 - - 4 3 - - - - 3 Model fruit orchards established Orchards - - 4 16 14 16 14 - - 32 Existing fruit Orchards improved Orchards - - - - - - - - - 0 Rootstock for commercial certified nurseries Stock - - - 20,000 20,000 50,000 50,000 - - 70,000 Support to input store Enterprise - - - - - 1 1 - - 1 Certified fruit nurseries strengthened and developed Nursery - - - - - 4 4 6 - 4 Rootstock for commercial certified fruit nurseries and orchard imported Sapling - - - - - - - 1,500 - 0 SATPARA DEVELOPMENT PROJECT EVALUATION 105 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Result to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Certified fruit nurseries developed and strengthened Nursery - - - 2 2 4 4 - - 6 Vegetable production Plastic tunnels/ glasshouses constructed Glass house 300 - - 20 25 150 150 - - 175 Greenhouse management training Training 5 1 1 1 6 5 7 - - 14 Training in post-harvest crop handling Training - - - 5 8 - - - - 8 Off-season vegetable and seed production farms established (3 kanal/farm) Farms - - - 10 5 12 - - - 5 Facilitators in off-season vegetable production trained Training - - - 1 2 - - - - 2 FFS in seed production FFS - - - 5 2 5 initiated 5 - initiated FFS in off season vegetable production FFS 50 5 - 4 2 10 initiated - - 2 FFS in vegetable seed production FFs - - - - - - - 5 - 0 Consultants' visit for improvement in greenhouses' design Visit - - - - - 1 1 - - 1 Exposure Visit (National) for off season vegetable and seed production grant recipients Visit - - - - - - - - - 1 Livestock FFS in animal husbandry FFS 45 5 1 10 10 10 10 5 started 21 Grants for establishing dairy farms Farms 5 5 5 - - - - - - 5 SATPARA DEVELOPMENT PROJECT EVALUATION 106 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Result to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Public sector AI services strengthened Activity 2 1 - - - - - - - 0 Consultant to carryout value chain study hired Study report 1 - - 1 1 - - - - 1 Al kits procured Kit 10 - - 8 9 - - - - 9 Establishment of commercial dairy farms Initiated feasibility - - - 1 1 - - - - 1 Feasibility study for storage facility conducted Feasibility - - - - - 1 Not initiated - - 0 Al input supply chain established Enterprise - - - - - 1 1 - - 1 Training in AI services for 10 participants Training - - - 1 1 - - - - 1 FFS in fodder production & storage FFS - - - 5 - 10 10 - - 10 FFS in fruit production FFS - - - - - 10 initiated - - 0 Agri/Livestock extension workers training Training - - - - - 2 2 - - 2 Model animal shed established Grants - - - - - - - 1 1 1 Existing animal sheds improved Grants - - - - - - - 5 5 5 Packaging Enterprises for improved packaging established Enterprise - - - - - 2 2 - - 2 Youth led enterprises for packaging design established Enterprise - - - - - 1 1 - - 1 Provision of packaging material for FEGs lump sum - - - - - - - 1 3 3 Miscellaneous SATPARA DEVELOPMENT PROJECT EVALUATION 107 Activities Unit Project Target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Result to Date Target Actual Target Actual Target Actual Q1 Target Q1 Actual Capacity building of farmers and entrepreneurs through FFS Training - - - - - 1 initiated - - 0 Value chain study by international expert Study report - - - - - 1 1 - - 1 Soil testing lab established in Skardu Lab - - - - - 1 1 - - 1 Seed testing lab established in Skardu Lab - - - - - 1 1 - - 1 Feasibility study of storage facility Study - - - - - - - 1 1 1 Training in business development Training - - - - - 2 3 1 1 running Soil testing lab established in Skardu Labs - - - - - - - 1 - 0 Facilitate DOA, DOL, FSC&RD staff in providing extension services to grantees Lump sum - - - - - - - 0.25 - 0 SATPARA DEVELOPMENT PROJECT EVALUATION 108 TABLE 19: PERCENTAGE OF TARGET ACHIEVED BY PROJECT YEAR – COMPONENT 2 Activities Unit LOP Target % of cumulative target achieved % of LOP Target Year 1 Year 2 Year 3 Year 4, First Quarter Baseline survey and mapping Survey 1 100% 100% 100% 100% FEGs FEGs formed FEGs 50 60% 100% 108% 107% 88% FEGs trained in use of improved packaging Training 100% 100% Training of FEG members Training 57% 73% Grants to FEGs for production level activities FEGs 100% Orchards and nurseries Nursery management training Training 2 0% 100% 100% 100% Fruit orchards management training Training 3 0% 50% 50% 33% Mother fruit nurseries established Nursery 6 75% 75% 75% 50% Model fruit orchards established Orchards 88% 100% 100% Existing fruit Orchards improved Orchards 0% 0% 0% Rootstock for commercial certified nurseries stock 100% 100% 100% Support to input store Enterprise 100% 100% Certified fruit nurseries strengthened and developed Nursery 100% 40% Rootstock for commercial certified fruit nurseries and orchard imported sapling 0% Certified fruit nurseries developed and strengthened Nursery 100% 100% 100% Vegetable production Plastic tunnels/ glasshouses constructed Glass house 300 125% 103% 103% 58% Greenhouse management training Training 5 100% 600% 200% 200% 280% Training in post-harvest crop handling Training 160% 160% 160% Off-season vegetable and seed production farms established (3 kanal/farm) Farms 50% 23% 23% SATPARA DEVELOPMENT PROJECT EVALUATION 109 Activities Unit LOP Target % of cumulative target achieved % of LOP Target Year 1 Year 2 Year 3 Year 4, First Quarter Facilitators in off-season vegetable production trained Training 200% 200% 200% FFS in seed production FFS 40% 20% 13% FFS in off season vegetable production FFS 50 0% 50% 11% 11% 4% FFS in vegetable seed production FFs 0% Consultants' visit for improvement in greenhouses' design Visit 100% 100% Exposure Visit (National) for off season vegetable and seed production grant recipients Visit 50% 50% 50% Livestock FFS in animal husbandry FFS 45 20% 100% 84% 70% 47% Grants for establishing dairy farms Farms 5 100% 100% 100% 100% Public sector AI services strengthened Activity 2 0% 0% 0% 0% Consultant to carryout value chain study hired Study report 1 100% 100% 100% 100% Al kits procured Kit 10 113% 113% 113% 90% Establishment of commercial dairy farms Initiated feasibility 100% 100% 100% Feasibility study for storage facility conducted Feasibility 0% 0% Al input supply chain established Enterprise 100% 100% Training in AI services for 10 participants Training 100% 100% 100% FFS in fodder production & storage FFS 0% 67% 67% FFS in fruit production FFS 0% 0% Agri/Livestock extension workers training Training 100% 100% Model animal shed established Grants 100% Existing animal sheds improved Grants 100% Packaging Enterprises for improved packaging established Enterprise 100% 100% Youth led enterprises for packaging design established Enterprise 100% 100% SATPARA DEVELOPMENT PROJECT EVALUATION 110 Activities Unit LOP Target % of cumulative target achieved % of LOP Target Year 1 Year 2 Year 3 Year 4, First Quarter Provision of packaging material for FEGs Lump sum 300% Miscellaneous Capacity building of farmers and entrepreneurs through FFS Training 0% 0% Value chain study by international expert Study report 100% 100% Soil testing lab established in Skardu Lab 100% 100% Seed testing lab established in Skardu Lab 100% 100% Feasibility study of storage facility feasibility 100% Training in business development Training 150% 133% Soil testing lab established in Skardu Labs 0% Facilitate DOA, DOL, FSC&RD staff in providing extension services to grantees Lump sum 0% Source: AKRSP results tracker Note: Blank cells indicate there was no target for the activity in that year. 0% means there was a target. SATPARA DEVELOPMENT PROJECT EVALUATION 111 TABLE 20: EXPLANATION OF REPORTED RESULTS – COMPONENT 2 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) Baseline survey and mapping Field enumeration completed. No targets for year 2. No targets for year 3. FEGs FEGs formed 6 of 10 planned FEGs formed. Formation stopped waiting for sub￾grants to be awarded. No mention of this activity or result in report. 14 achieved against a target of 7. FEGs trained in use of improved packaging No targets for year 2. No targets for year 2. 1 training conducted of target of 1. Training of FEG members No targets for year 2. No targets for year 2. Report says 7 of 5 Grants to FEGs for production level activities No targets for year 2. No targets for year 2. No targets for year 3. Orchards and nurseries Nursery management training 0 of 1 planned training conduced. Training will be imparted next year Unplanned activity (in this year) Achieved target of 1. Fruit orchards management training 0 of 1 planned training conduced. Training will be imparted next year No targets for year 2. Achieved target of 1. Mother fruit nurseries established Report says plants purchased and planted in 2 nurseries 3 nurseries established of target of 4. Remaining nursery in progress, expected to be completed by mid￾April 2014. No targets for year 3. Model fruit orchards established 4 established from target of 0. 14 of a planned 16 achieved. Target of 16 but only 14 qualified awardees identified. Existing fruit orchards improved No targets for year 2. 0 of a planned 10 achieved. 6 orchards selected. Activity will start in first quarter of next year. No targets for year 3. Rootstock for commercial certified nurseries No targets for year 2. 20,000 of a planned 20,000 plants provided. Report says 4 of 0. Activity is carried forward to year IV Support to input store No targets for year 2. No targets for year 2. 1 of a target of 1 achieved. SATPARA DEVELOPMENT PROJECT EVALUATION 112 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) Certified fruit nurseries strengthened and developed No targets for year 2. No targets for year 2. In report only. The activity was linked with import of rootstock from international sources, now it is carried forward to year IV Rootstock for commercial certified fruit nurseries and orchard imported No targets for year 2. No targets for year 2. No targets for year 3. Certified fruit nurseries developed and strengthened No targets for year 1. 2 of a planned 2 achieved. 4 of a planned 4 achieved. Vegetable production Plastic tunnels/ glasshouses constructed No targets for year 2. Constructed 25 of planned 20. Exceeded target because of the great interest by community. These were plastic tunnels. Shifted to glass houses and initiated/constructed 150 against a target of 150. Greenhouse management training Report says 0 of 1 completed, tracker says 1 of 1. Preparations completed. Training will be imparted next year 6 of a target of 1 training conducted. Report says 5 of 5. Training in post-harvest crop handling No targets for year 2. 8 trainings conducted against a target of 5. No targets for year 3. Off-season vegetable and seed production farms established (3 kanal/farm) No targets for year 2. Reported establishing 5 farms against a target of 10. Linked to FFS which started late. No farms established against a target of 12. Facilitators in off-season vegetable production trained No targets for year 2. 2 Sessions Completed No targets for year 3. FFS in seed production No targets for year 2. Reported conducting 2 FFS of a target of 5. Linked to FFS which were completed late. FFS deferred to next year. 5 FFS initiated but not completed (due in June 2015) FFS in off season vegetable production Preparations in progress, target deferred to next year. Training of facilitators completed in March. Remaining FFS deferred to April 2014. 10 of planned 10 in progress, will be completed in June 2015 FFS in vegetable seed production No targets for year 2. No targets for year 2. No targets for year 3. SATPARA DEVELOPMENT PROJECT EVALUATION 113 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) FFS in fruit production No targets for year 2. No targets for year 2. 0 of 10 planned FFS conducted. Consultants' visit for improvement in greenhouses' design No targets for year 2. No targets for year 2. 1 consultant engaged of target of 1. Exposure visit (National) for off season vegetable and seed production grant recipients No targets for year 2. 1 visit conducted of target of 2. All grantees combined into one trip. No targets for year 3. Livestock FFS in animal husbandry Preparations in progress, target deferred to next year. 10 of a planned 10 achieved. 10 of a planned 10 achieved. Grants for establishing dairy farms Process completed for 5 of planned 5 grants. Implementation will start in April 2013. No targets for year 2. No targets for year 3. Public sector AI services strengthened Dialogues held with DoL Skardu. MOU prepared. Support will be provided next year No targets for year 2. No targets for year 3. Consultant to carryout value chain study hired No targets for year 2. 1 of a planned 1 consultant hired. No targets for year 3. Al kits procured No targets for year 2. 9 kits provided against a target of 8. No targets for year 3. Establishment of commercial dairy farms Initiated No targets for year 2. Feasibility study completed. No targets for year 3. Feasibility study for storage facility conducted No targets for year 2. No targets for year 2. Consultant hired, study will be completed by April 2015. Al input supply chain established No targets for year 2. No targets for year 2. 1 AI supply chain established of target of 1. Training in AI services for 10 participants No targets for year 2. 1 training conducted of target of 1. No targets for year 3. FFS in fodder production & storage No targets for year 2. 0 of a planned 5 FFS conducted. Trained facilitators late and activity deferred to next year. Conducted 10 of a planned 10 FFS. SATPARA DEVELOPMENT PROJECT EVALUATION 114 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) Agri/Livestock extension workers training No targets for year 2. No targets for year 2. 2 of a planned 2 trainings conducted. Model animal shed established No targets for year 2. No targets for year 2. No targets for year 3. Existing animal sheds improved No targets for year 2. No targets for year 2. No targets for year 3. Packaging Enterprises for improved packaging established No targets for year 2. No targets for year 2. Report says 2 of 1. Youth led enterprises for packaging design established No targets for year 2. No targets for year 2. 1 enterprise established of target of 1. Provision of packaging material for FEGs No targets for year 2. No targets for year 2. No targets for year 3. Miscellaneous Capacity building of farmers and entrepreneurs through FFS No targets for year 2. No targets for year 2. 0 of a planned 1 activity achieved. Requires services of Center for Agriculture and Biosciences International. Deferred until CABI completed (June 2015) Value chain study by international expert No targets for year 2. No targets for year 2. 1 of a planned 1 study completed. Soil testing lab established in Skardu No targets for year 2. No targets for year 2. First phase completed. Not yet operational. Seed testing lab established in Skardu No targets for year 2. No targets for year 2. Report says 1/0. Scope changed. Feasibility study of storage facility No targets for year 2. No targets for year 2. No targets for year 3. Training in business development No targets for year 2. No targets for year 2. Reported 3 trainings of a target of 2 (in annual report only). Soil testing lab established in Skardu No targets for year 2. No targets for year 2. No targets for year 3. Facilitate DOA, DOL, FSC&RD staff in providing extension services to grantees No targets for year 2. No targets for year 2. No targets for year 3. Source: Annual and quarterly reports. SATPARA DEVELOPMENT PROJECT EVALUATION 115 TABLE 21: TARGETS AND ACHIEVED RESULTS – COMPONENT 3 Activities Unit LOP target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Total to date Target Actual Target Actual Target Actual Target Actual Baseline survey Survey 1 1 1 - - - - - - 1 Training on use of processing technology and marketing Training 1 1 - - - - - - 1 Formation of Women Processor Groups Groups 2 2 10 16 10 10 - - 28 Issuing grants to processors Groups 2 - - - - - - - - Scaling of women producer groups (WPGs) Groups 5 4 - - - - - - 4 Training in business development Training 1 1 1 1 2 3 - - 5 Acquire service of fruit processing and technology consultant Days - - 30 22 - - - - 22 Formation of FEGs Groups - - 3 3 - - - - 3 Fruit and vegetable processing training Training - - 2 5 - - - - 5 Initiate establishment of buckwheat processing center Feasibility - - 1 1 - - - - 1 Initiate establishment of fruit processing center Feasibility - - 1 1 1 - 1 1 Establishing processing enterprises Enterprise - - 6 - - - - Exposure visits - - 1 3 2 2 1 1 5 Initiate establishment of fresh milk center Feasibility - - 1 1 1 - 1 Initiate establishment of yogurt processing units Feasibility - - 1 1 1 - 1 Provision of improved dehydration units Plant - - - - 20 initiated - SATPARA DEVELOPMENT PROJECT EVALUATION 116 Activities Unit LOP target Year 1 (2012) Year 2 (2013) Year 3 (2014) Year 4 (2015) Total to date Target Actual Target Actual Target Actual Target Actual Establishing of fruit processing centers Center - - - - 1 initiated - Training in Drying technology Training - - - - 2 1 1 Linkages with national/international markets Visits - - - - 1 1 1 Facilitation in the establishment of fresh milk collection centers Center - - - - 1 initiated 1 1 - FFS in animal husbandry - - - - 10 10 Facilitation in the establishment of yogurt processing collection units Center - - - - 1 initiated 1 1 - Facilitate processors/producers in fair-trade certification Event - - - - 1 1 1 Participation in national exhibition, fairs etc. Persons - - - - 15 15 5 5 15 Formation of youth group Groups - - - - 1 1 1 Exposure visit of youth group Visit - - - - 1 1 1 Formation of men and women processor groups (P/WPGs) Groups 5 5 5 Post-harvest handling training Training 2 2 Provision of improved dehydration units at HH level Plant - - - - - - 25 25 25 Establishment of fruit dehydration center Center - - - - - - 1 - - Grant to establish market information system Grants - - - - - - 1 - Facilitation in establishment of weekly bazaar Weekly Market - - - - - - 1 1 1 SATPARA DEVELOPMENT PROJECT EVALUATION 117 TABLE 22: PERCENTAGE OF TARGET ACHIEVED BY PROJECT YEAR – COMPONENT 3 Activities Unit LOP Target % of cumulative target achieved % of LOP Target Year 1 Year 2 Year 3 Year 4, First Quarter Baseline survey Survey 1 100% 100% 100% 100% 100% Training on use of Processing technology and marketing Training 100% 100% 100% 100% No LOP target Formation of Women Processor Groups Groups 100% 150% 127% 127% No LOP target Issuing grants to processors Groups 0% 0% 0% 0% No LOP target Scaling of Women Producer Groups (WPGs) Groups 80% 80% 80% 80% No LOP target Training in business development Training 100% 100% 125% 125% No LOP target Acquire service of fruit Processing and technology Consultant Days 73% 73% 73% No LOP target Formation of FEGs Groups 100% 100% 100% No LOP target Fruit and vegetable Processing Training Training 250% 250% 250% No LOP target Initiate establishment of buckwheat processing center Feasibility 100% 100% 100% No LOP target Initiate establishment of fruit processing center Feasibility 100% 50% 33% No LOP target Establishing Processing Enterprises Enterprise 0% 0% 0% No LOP target Exposure visits Visits 300% 167% 150% No LOP target Initiate establishment of fresh milk center Feasibility 100% 50% 50% No LOP target initiate establishment of yogurt processing units Feasibility 100% 50% 50% No LOP target Provision of improved dehydration units Plant 0% 0% No LOP target Establishing of fruit processing centers Center 0% 0% No LOP target Training in Drying technology Training 50% 50% No LOP target Linkages with national/international markets Visits 100% 100% No LOP target Facilitation in the establishment of fresh milk collection centers Center 0% 50% No LOP target FFS in animal husbandry FFS 100% 100% No LOP target Facilitation in the establishment of yogurt processing collection units Center 0% 50% No LOP target SATPARA DEVELOPMENT PROJECT EVALUATION 118 Activities Unit LOP Target % of cumulative target achieved % of LOP Target Year 1 Year 2 Year 3 Year 4, First Quarter facilitate processors/producers in fair-trade certification Event 100% 100% No LOP target Participation in National Exhibition, Fairs etc. Persons 100% 100% No LOP target Formation of youth group Groups 100% 100% No LOP target Exposure visit of Youth Group Visit 100% 100% No LOP target Formation of Men and Women Processor Groups (P/WPGs) Groups 100% No LOP target Post-Harvest Handling Training Training No LOP target Provision of improved dehydration units at HH level Plant 100% No LOP target Establishment of fruit dehydration center Center 0% No LOP target Grant to establish market information system grants 0% No LOP target Facilitation in establishment of weekly bazaar Weekly Market 100% No LOP target Source: AKRSP results tracker Note: Blank cells indicate there was no target for the activity in that year. 0% means there was a target. SATPARA DEVELOPMENT PROJECT EVALUATION 119 TABLE 23: EXPLANATION OF REPORTED RESULTS – COMPONENT 3 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Baseline survey Field enumeration completed. Report not finalized. N.R. N.R. Training on use of processing technology and marketing Staff trained on use of technology in processing and marketing during an exposure visit to down country N.R. N.R. Formation of Women Processor Groups 2 formed against target of 2 16 groups formed against target of 10 due to community interest Target of 10 achieved Issuing grants to processors No grants distributed against a target of 2 due to absence of grants manual, environmental documentation, etc. N.R. N.R. Scaling of Women Producer Groups (WPGs) 4 groups formed against target of 5. First dialogues have been held, formation of remaining groups is in process. Will be fully achieved by end of April. N.R. N.R. Training in business development Conducted 1 training against a target of 1 N.R. N.R. Acquire service of fruit processing and technology consultant N.R. 22 days expended against a total of 30. Activity is deferred to first two quarters of year III, considering the fruit picking season. N.R. Formation of FEGs N.R. Target of 3 achieved N.R. SATPARA DEVELOPMENT PROJECT EVALUATION 120 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Fruit and vegetable processing training N.R. 5 trainings against target of 2 due to community interest N.R. Initiate establishment of buckwheat processing center N.R. Target achieved. Study complete and final report due from the external consultant. N.R. Initiate establishment of fruit processing center N.R. Target achieved. Study complete and final report due from the external consultant. N.R. Agreement development with the applicants finalized Establishing processing enterprises N.R. No enterprises established against target of 6. To commence next year N.R. Exposure visits N.R. 3 visits conducted against target of 1. N.R. Initiate establishment of fresh milk center N.R. 1 initiated against target of 1 (but not really). To be completed after finalization of feasibility study from the consultant. N.R. Initiate establishment of yogurt processing units N.R. 1 initiated against target of 1 (but not really). To be completed after finalization of feasibility study from the consultant. N.R. Provision of improved dehydration units N.R. N.R. Target of 20 achieved Establishing of fruit processing centers N.R. N.R. None achieved against a target of 1. Carried forward to year 4. SATPARA DEVELOPMENT PROJECT EVALUATION 121 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Conducted 1training against Training in drying technology N.R. N.R. target of 2. 1 carried forward to year 4. Linkages with national/international markets N.R. N.R. Tracker claims to have met target of one. Report claims no achievement. None established against Facilitation in the establishment of fresh milk collection centers N.R. N.R. target of 1. Grant agreement signed procurement of equipment in progress by grantee, will be completed in June 2015 1 established from previous year target FFS in animal husbandry N.R. N.R. Will be concluded in June 2015 5 started against target of 5 (from previous year) Facilitation in the establishment of yogurt processing collection units N.R. N.R. None established against target of 1. Grant agreement signed procurement of equipment’s in progress by grantee, will be completed in June 2015 1 established and operational from previous year target Facilitate processors/producers in fair-trade certification N.R. N.R. No certifications against a target of one. Scope of activity changed. Participation in national exhibition, fairs etc. N.R. N.R. Participated in 10 against target of 15 5 of a target of 5 (held over from previous year target) Formation of youth group N.R. N.R. One established against target of 1 Exposure visit of youth group N.R. N.R. One visit against target of 2. Formation of men and women processor groups (P/WPGs) N.R. N.R. N.R. 5 WPGs formed against target of 5 SATPARA DEVELOPMENT PROJECT EVALUATION 122 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Post-harvest handling training N.R. N.R. N.R. 2 trainings completed against target of 2 Provision of improved dehydration units at HH level N.R. N.R. N.R. Established 25 against target of 25. Purchase order for remainder issued. Establishment of fruit dehydration center N.R. N.R. N.R. No action against target of 1. Change in scope anticipated. Grant to establish market information system N.R. N.R. N.R. Discussion of system design ongoing Facilitation in establishment of weekly bazaar N.R. N.R. N.R. 1 of target of 1 established. SATPARA DEVELOPMENT PROJECT EVALUATION 123 TABLE 24: EXPLANATION OF REPORTED RESULTS – COMPONENT 4 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Output 1: Improved water governance system and rules Study traditional and state rules on water rights and find gapes Not in year 1 work plan 1 study planned, not undertaken 1 study planned but not undertaken 1 planned and done Meetings with community/Govt. to discuss policy issues and formulate reforms Not in year 1 work plan 1 meeting planned and done 2 meetings planned and implemented 1 planned, not implemented Produce papers and reports on policy issues & recommended reforms Not in year 1 work plan 1 study planned, not undertaken 1 study planned, not implemented Disappears from work plan, without explanation Exposure visits for community and public sector reps to show them best practices Not in year 1 work plan 1 planned and conducted Not in year 3 work plan Not in year 4 work plan Output 2: Enhanced effectiveness of extension services Provide support to DoA/DoL to revitalize the extension infrastructure & service through PPPs Not in year 1 work plan 2 planned, not implemented Not in year 3 work plan Not in year 4 work plan Initiate crop inspection system for certification with FSCRD Not in year 1 work plan 1 planned, not implemented 1 planned and implemented Not in year 4 work plan SATPARA DEVELOPMENT PROJECT EVALUATION 124 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Facilitate in crop inspection system for certification with FSCRD Not in year 1 work plan Not in year 2 work plan Not in year 3 work plan 1 planned, not implemented Training & licensing of crop inspectors through FSC&RD on seed certification Not in year 1 work plan Not in year 2 work plan 1 planned and implemented 1 planned and implemented Institutional review of OFWM/ WUAs Not in year 1 work plan Not in year 1 work plan 1 planned and initiated Ongoing Output 3: Networks created of WUAs, production and marketing groups and enterprises Form WUA network for policy influencing Not in year 1 work plan 10 planned, not implemented After discussions with USAID, this activity was deferred pending institutional assessment of existing WUAs Status unchanged Form and mobilize FEG networks to influence agriculture policies Not in year 1 work plan 1 planned, partially implemented 2 planned, implemented Ongoing Support FEG networks to develop local product brands with private sector Not in year 1 work plan 4 planned, 1 implemented Ongoing Ongoing Support LSO/FEG and networks to influence agriculture policies Not in year 1 work plan Not in year 2 work plan 3 planned 5 implemented Ongoing SATPARA DEVELOPMENT PROJECT EVALUATION 125 Activity Year 1 (March 2012 – March 2013) Year 2 (April 2013 – March 2014) Year 3 (April 2014 – March 2015) First Quarter of Year 4 Hold policy dialogues with Govt./GBLA to remove legal issues in agricultural marketing Not in year 1 work plan 2 planned, 1 initiated 4 planned and initiated Ongoing Output 4: Forums established for policy dialogue among stakeholders Establish Agriculture Resource Centers (ARC) under LSO/ FEG Networks Not in year 1 work plan Not in year 2 work plan 3 planned (on-hold) Revised proposal on cost sharing under review Thematic Conferences Not in year 1 work plan Not in year 2 work plan 12 planned, not implemented 1 planned and implemented Media Outputs (Press release, feature report etc.) Not in year 1 work plan Not in year 2 work plan 1 planned and implemented Not in year 4 work plan Video Documentary Not in year 1 work plan Not in year 2 work plan 1 planned and implemented Not in year 4 work plan Support LSOs/Networks to form WUAs and handle social/water issues Not in year 1 work plan Not in year 2 work plan Not in year 3 work plan 1 planned, initiated SATPARA DEVELOPMENT PROJECT EVALUATION 126 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523