June 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under Pakistan Poverty Alleviation Fund (PPAF), Cooperative Agreement 391-A-00-03-01010-00 Pakistan Economic Growth Program Evaluation Evaluation Summary Report DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS INTRODUCTION ...................................................................................................................1 Key Findings and Recommendations..................................................................................2 SUMMARY OF PROJECTS..................................................................................................3 Business Development ........................................................................................................3 Financial Services................................................................................................................4 Agriculture ..........................................................................................................................4 Energy .................................................................................................................................4 PROGRAM RESULTS ...........................................................................................................4 Relevance ............................................................................................................................4 Effectiveness .......................................................................................................................6 Impact..................................................................................................................................8 Efficiency ............................................................................................................................9 Sustainability And Replication..........................................................................................11 Gender Considerations......................................................................................................13 USAID Reporting And Branding Requirements, Outreach And Communications.........................................................................................................16 ANNEX 1. COMPETITIVE AND NON-COMPETITIVE CONTRACT AWARDS..........................................................................................................17 ANNEX 2. WAS PROGRAM MANAGEMENT DATA COLLECTED?........................19 ANNEX 3. OVERALL EVALUATION QUESTIONS ......................................................20 TABLES Table 1. USAID Pakistan Economic Growth Project Evaluations: Summary of Efficiency Measures................................................................................................10 Table 2. USAID Pakistan Economic Growth Projects: Summary of Gender Impacts............14 PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT i PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 1 INTRODUCTION Pakistan’s economy featured both growth and setbacks and a significant downturn during the 1990s. In 2001, Pakistan’s growth rate dropped below 2 percent1 before recovering, caused by a lack of competitiveness in international markets, low technology, low labor productivity and protected and inefficient manufacturing. During the past seven years Pakistan aggressively pursued reforms that paved the way for private sector investment and growth, yet Pakistan still struggles to compete in the global market. In 2001, after a five year absence from the country, USAID returned to Pakistan and re-started its assistance program. During this process in 2002, USAID recognized the need to rapidly boost Pakistan’s economic growth as a way to reduce poverty. In 2003, USAID initiated a $72 million, five￾year Economic Growth Program to quickly implement programs to support the Government of Pakistan’s reform efforts and augment inclusive economic growth. The Pakistan Economic Growth Evaluation and Design (PEGED) project completed a comprehensive evaluation of USAID Pakistan’s Economic Growth (EG) portfolio to inform the development of a new five-year strategy. The project included this summary evaluation and mid or final evaluations of all eight projects and the design of four new economic growth projects. In most cases, the evaluations analyzed project results through December 2007. Only two of the eight projects were fully completed by December 2007. This report analyzes the eight evaluations to capture ten key cross-cutting findings and recommendations. This report does not intend to summarize each project evaluation, as the individual project evaluations contain those details. Using this report, USAID can develop its new strategy based on the lessons learned in these past projects. USAID can also use this report to share the evaluation findings with other donors and the Government of Pakistan (GoP). The evaluations used the same ten evaluation questions for each project (see Annex 3). However, this report covers results from only the six most critical questions regarding impact, effectiveness, efficiency, relevance, gender, and sustainability/replication. The partners who conducted the eight projects provided training, equipment, technical assistance, and other inputs. The evaluations examined how those inputs generated outputs, and then analyzed how those outputs improved Pakistan’s economic growth prospects. The evaluations centered on key questions, such as whether the projects were relevant to Pakistan’s economic problems and the needs of the Pakistani people; implemented in an efficient and effective manner; whether they affected men and women differently; whether the project benefits generated are sustainable and can the project approach be replicated. However, the implementing project designs, scopes of work and contracts did not explicitly address each of these questions. This, as well as no requirements for baseline data collection or adherence to a Performance Management Plan (PMP) resulted in significant constraints to calculating impact and efficiency. The team found it difficult to quantify and value many of the policy and institutional change objectives, particularly at the beneficiary level, because they were too recent to measure any attributable results. In addition, only two of the eight projects were completed. Therefore, the evaluation team feels that full impact is not possible to measure at this time, though the team did estimate possible future impacts. Only two of the projects2 were centered at the local level where, through better access to data, benefits and costs could be calculated with some confidence. For the other six projects, impact figures are rough estimates at best. 1 Jesus Felipe, “A Note on Competitiveness and Structural Transformation in Pakistan,” ERD Working Paper No. 10, December 2007. 2 The MEDA Access to Contemporary Markets for Homebound Women Embroiders Project and the FAO Balochistan Agricultural Project. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 2 The eight project evaluations are available from USAID Pakistan’s website and the USAID Washington Development Experience Clearing House. Key Findings and Recommendations • The GoP wants to encourage economic growth and reduce poverty but finding a model that works has proven difficult. Subsidies and special privileges run the danger of being politically motivated and often have negative results. The GoP sought the advice of USAID, and as a result, USAID created business competitiveness projects and developed industry sector working groups. • As a result of the USAID program, the private sector is more organized and actively promoting new approaches. The Ministry of Industry, Production and Special Initiatives (MoI&P)) reviewed USAID policy papers and implemented many USAID’s suggested strategies. The GoP invested more than $125 million of its own funds. The challenge now is for USAID to build on the government’s receptivity and the private sector’s responsiveness to expand the most appropriate programs and implement the most applicable policy reforms. • Unfortunately in Pakistan, the government believed in subsidizing poor borrowers with below market interest rates and used ineffective lending agencies that cannot cover their costs. With the intervention of USAID microfinance projects, however, the GoP now understands the need for a market-based interest rate and sustainable approaches for administering such programs. • At a relatively low cost, the MEDA project (Behind the Veil: Access to Markets for Homebound Women Embroiderers in Pakistan) reached rural women in some of the poorest regions. The program created capacity in local partner NGOs to train a cadre of women sales agents who provided the linkage to markets. Thus rural women gained opportunities to work at home on embroidery and garments as well as sell their products at a competitive price to higher income buyers in other regions of Pakistan. Though on a small scale, the project became very effective using its value chain approach, technical assistance and technology. The GOP and the Canadian International Development Agency recently signed a $6.7 million contract to greatly expand the project. USAID should identify more successful efforts like MEDA and help replicate and/or expand them. • Unless projects specifically targeted gender impacts, like MEDA, the impact on gender was marginal at best. For projects to measurably address gender issues, USAID must state those objectives in all project designs as requirements for implementation. • USAID focused mainly on quick implementation and results for the Economic Growth (EG) program. USAID rarely collected baseline data, and created very few monitoring and evaluation plans (see Annex 2). In this environment, project managers struggled to assess progress, alter efforts to correct problems or identify the most successful techniques. In addition, evaluators faced difficulties in qualifying as well as quantifying project performance. Future USAID projects need a robust Monitoring and Evaluation (M&E) component. USAID must require baselines and Performance Management Plans (PMP) to track performance for multi-year projects more rigorously. • USAID was certainly in a hurry to re-start the program and lacked management capacity at the time of awards. Instead of completing the long and slow competitive bid process, USAID awarded contracts seven of the ten projects non-competitively.3 (see Annex 1) The evaluation 3 The count of 10 projects includes PISDAC I, PISDAC II, and PISDAC III as separate projects. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 3 team believes this directly contributed to the lack of performance monitoring and ultimately benefit in some projects. • USAID lacked monitoring of project performance and requirements for projects to address gender issues. At the time, USAID was extremely short staffed and therefore could not critically review periodic project reports or make regular site visits. Thus, project managers and USAID staff did not receive the information necessary to properly manage these efforts. For future projects, USAID should consider the staff needed for effective USAID operational oversight. • USAID’s forthcoming economic growth portfolio would benefit from a blend of both higher level policy projects and grassroots projects that work directly with beneficiaries. Local projects produce direct benefits to the people, which are clear and easily measured. However, these benefits can be difficult to scale-up to provinces or the country at large. Large projects, on the other hand, focusing on macro, sector and policy reforms are more difficult to measure direct benefits but in time can generate widespread change throughout the economy. • While not an original part of the evaluation, USAID needs to consider employment as an issue that affects productivity and economic growth. Between 1970 and 2000, Pakistan’s population more than doubled from 60 million to 144 million. The population ages 15 – 49 more than tripled from 14 million to 68 million. When the number of people in the 15 – 49 age group rises, so does the demand for jobs. Projections indicate that Pakistan’s population will increase to 229 million by 2025, and the population aged 15 – 49 will increase to 121 million, nearly double the level in 20004 . If productive employment keeps pace with the growing work force, this will help Pakistan by adding to GDP growth. Conversely, if Pakistan does not invest in producing a more skilled, technically qualified and educated workforce, population growth will create serious socio-economic problems . Currently, Pakistan’s reported unemployment rate is 7 percent. There is also substantial underemployment5 . The GoP’s current supply-push approach is based on outdated needs because the young people need industry-oriented skills. Pakistan needs a demand-pull approach concentrating on the technical skills, knowledge and literacy required by specific industries and service sectors in both the formal and informal markets. Pakistan can accomplish this through skills mapping, and by working collaboratively with industry, NGOs, the government and the academic community. SUMMARY OF PROJECTS Business Development 1. The Pakistan Initiative for Strategic Development and Competitiveness (PISDAC) helped develop a public/private dialogue on economic policy measures and reforms. In addition, PISDAC helped form Strategy Working Groups (SWOGs) and Sector Management Companies (SMCs) to improve competitiveness in key industrial sectors. 2. The Competitiveness Support Fund (CSF) was established in collaboration with the Ministry of Finance. CSF developed proposed policy and regulatory reforms and public-private partnerships to support competitiveness. By working closely with the GoP and industry groups, CSF designed and implemented sector and industry studies. 3. The Access to Contemporary Markets for Homebound Women Embroiders Project provided technical assistance to a local NGO, the Entrepreneurship Career Development Institute. The 4 Source: UN, World Population Prospects: The 2006 Revision Population Database; and U.S. Census Bureau, International Data Base (online).   5 CIA Factbook, 2008. NGO trained sales agents in marketing, costing, pricing, and design, and these agents assisted rural women embroiderers in improving the marketability of their products. Financial Services 4. USAID worked in collaboration with Khushhali Bank (KB) in Sindh, Balochistan, and the Federally Administrated Tribal Areas (FATA) to provide micro-credit loans of $50 to $500 to the rural poor in non-banked areas. 5. The Enterprise Development Facility (EDF), implemented through a partnership with the Pakistan Poverty Alleviation Fund (PPAF), worked to support microfinance service providers. Those providers met the needs of entrepreneurs not served by either microfinance institutions or commercial banks. They provided loans ranging from Rs. 30,000 ($500) to Rs. 100,000 ($1,666) and training to borrowers in vocational and managerial skills. 6. The Widening Harmonized Access to Microfinance Project (WHAM) sought to stimulate industry at the national level and to expand access for the “missing middle” of micro and small enterprises. WHAM supported loans from $500 to $30,000 through commercial bank down-scaling and microfinance up-scaling. For up-scaling, WHAM provided training to six of the top microfinance providers, including credit-granting NGOs, rural support programs and microfinance banks. WHAM also worked with three commercial banks on down-scaling their efforts to reach the upper group of the “missing middle.” Agriculture 7. USAID worked with the United Nations Food and Agricultural Organization (FAO) to implement the Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project. This project helped agricultural producers in Balochistan improve water resource management and livestock and crop production. Energy 8. The South Asia Regional Initiative for Energy (SARI/Energy) is an eight country program promoting regional energy cooperation and security. SARI/Energy completed wind and solar maps of Pakistan, helped Pakistan develop a strategic and business operating plan for the South Asian Association for Regional Cooperation (SAARC) Energy Center, and assisted with the design of a National Policy Framework for liquefied natural gas. PROGRAM RESULTS Relevance: How well was the project focused on the needs of the beneficiaries? The Pakistan Initiative for Strategic Development and Competitiveness (May 2004 to May 2008) This policy and strategy project repositioned several important industrial sectors on a more competitive basis. USAID designed it well to enhance competitiveness for the sectors in which it works. PISDAC Strategy Working Groups typically include leaders in the sector that galvanize the industry and show other members the value of uniting, cooperating and collaborating. Though beneficial, these leaders, often from the larger firms in the sector, seem to accrue benefits first. Strategy Working Groups do not include two key stakeholders, which are workers and consumers. The Competitiveness Support Fund (March 2006 to December 2007) This project: i) Boosts enterprise competitiveness based on improved strategy and innovation; ii) Finances pilot projects and; iii) Supports small and medium enterprises (SME). When USAID set up CSF, the GoP approved PC-1, a legal document that defines CSF as a government-supported company. The GoP later changed, through focusing on policy related reforms, CSF’s primary PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 4 customers (the target beneficiaries) from small and medium enterprises to the Government of Pakistan (GoP) and business groups. CSF developed proposed policy and regulatory reforms and public-private partnerships to support competitiveness. The GoP and producers (mainly business councils and industry and trade groups) played a primary role designing and implementing sector and industry studies to improve competitiveness. With CSF’s help, the GoP can now present its case more effectively to the World Economic Forum. Also, the Ministry of Finance (MoF) better understands how to analyze and utilize competitiveness indicators and data because of CSF’s technical assistance. Access to Markets for Homebound Women Embroiderers in Pakistan (September 2004 to December 2007) The MEDA project focused on the needs of beneficiaries who are rural women embroiderers working out of their homes. The project provided training, input supply linkages and marketing by women sales agents, which increased income for the rural embroiders and provided appropriate financial incentives for the sales agents. Highly successful in being relevant to the target beneficiaries, the project might have disproportionately benefited women sales agents more than the women homebound producers. The Khushhali Bank (KB) Developing Non-bankable Territories for Financial Services Project (October 2003 to September 2010) This project targeted the poor in Balochistan, Sindh and FATA with KB serving as the local implementing partner. The project met an important need of the target beneficiaries: micro-credit loans. However, USAID quickly designed the project with one product -- micro-business loans. The focus resulted in the project overlooking financial services, particularly savings instruments and non￾business loans. Although the project did miss these opportunities, it did well in its relevance for target beneficiaries. The Enterprise Development Facility (September 2003 to September 2007) The beneficiaries were micro-enterprises with a loan demand profile of $500 to $1,500. Formal financial institutions do not lend in this range and neither do micro-finance institutions. EDF’s implementing partner, The Pakistan Poverty Alleviation Fund (PPAF) and USAID were the two key EDF stakeholders. Because the project lacked adequate targets, EDF had problematic loan terms and conditions for partner organizations (POs). As the same POs also received funds from PPAF for non￾USAID projects, the POs were thus reluctant to point out any EDF project problems. Even though the project reached target borrowers, it suffered from the lack of open communications and feedback. Widening Harmonized Access to Microfinance ((March 2005 to March 2008) This project targeted the missing middle, meaning those not serviced by microfinance institutions or commercial banks. The project barred use of donor funds for loan capital to avoid cheap-credit donor dependence and the corresponding negative consequences for long-term operational and financial sustainability. The project addressed the needs of beneficiaries, but had other problems. The project design failed to recognize high interest spreads that encouraged commercial banks to continue lending to large, traditional borrowers rather than the missing middle. Also, the project scope of work included a Cost Share requirement that measured only loan disbursements instead of a wider array of financial instrument objectives. As a result, WHAM developed and launched only loan products, even though anecdotal evidence shows that targeted groups, especially women, can save and thus demand savings type instruments. The focus on loans meant that the project missed opportunities to provide other financial services. The Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project (December 200 to December 2008) The project approach of concentrating on agriculture, research and extension activities was well suited to poverty alleviation in Balochistan. The project’s speed of design, scope and geographic focus reflected USAID’s interest in establishing a visible presence quickly among rural people in a PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 5 strategically important region. This, however, meant the design lacked a fully participatory approach, though most of the important stakeholders were involved. Ideally, the project would have included more women and stakeholders experienced with community development. During implementation, the project worked directly and successfully with beneficiaries and stakeholders. They helped the project better identify changing needs and adapt interventions to meet those needs. The South Asia Regional Initiative for Energy (September 2004 to September 2008) This project promoted technical and institutional frameworks for regional energy planning and infrastructure investments in the South Asia region. Though some of the training was not relevant to Pakistan’s needs, it did raise awareness among energy officials and the media. Wind and solar mapping provided some useful data, but contained technical problems. Consequently, the maps might not be used by Pakistani agencies. The overall program design only partially addressed Pakistan’s energy needs. Effectiveness: Has the project accomplished its objectives? The Pakistan Initiative for Strategic Development and Competitiveness (PISDAC) PISDAC achieved all but one of its objectives. PISDAC held several workshops with a broad range of industry groups. As a result, six industry groups agreed to work together and formed sector Strategy Working Groups (SWOGs). They operated much like business trade associations to improve competitiveness. The SWOGs set a good precedent to establish cross industry collaboration. Also, the SWOGs developed their own sector strategies that identified policy and workforce constraints as well as more than 50 strategic and policy reforms. As a result of PISDAC, the GoP established five private sector management companies to implement the strategies and fund sector initiatives. One sector management company is now fully operational and independent of PISDAC. The Competitiveness Support Fund (CSF) CSF developed proposed policy and regulatory reforms and public-private partnerships to support competitiveness. The GoP and producers (mainly business councils and industry and trade groups) played a primary role designing and implementing sector and industry studies. CSF raised awareness of competitiveness issues in the GoP through public-private sector dialogues on competitiveness and the role of the private sector. These discussions identified 30 policy and institutional reforms. At the time of the evaluation, the GoP had passed 11 of them. CSF completed feasibility studies for 12 matching sector grants. Ten of these supported joint-facilities training centers and two were for technical assistance. As a result of these CSF policy studies, the GoP embraced competitiveness as a way to reduce poverty and support economic growth. For example, in February 2008, the GoP provided $8 million and the Sindh Government provided $2 million to implement the CSF-defined Karachi Fisheries Harbor management upgrade. CSF promoted public-private partnerships in 3 action plans. Also, the CSF horticulture study helped secure a PC-1 grant for $50 million and a plot of land near the Lahore Airport. Access to Contemporary Markets for Homebound Women Embroiders (MEDA) Implemented by Mennonite Economic Development Associates (MEDA), this project focused successfully on increasing women’s income with a value-chain that provided appropriate information on consumer demand through a linkage of sales agents. Sales agents worked closely with embroiders to improve their skills, product quality and price. The project met all performance indicators and targets were met if not greatly exceeded. Product quality and design steadily improved. The embroiderers have higher income, greater status in their household, more children going to school. The project faced the challenge of political unrest and violence, which limits market access from supporting input sellers and product buyers. Even so, the project showed how to reach homebound women even in the most conservative areas of Pakistan. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 6 PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 7 The Kushhali Bank (KB) Development of Non-bankable Territories (Sind, Balochistan and FATA) KB provided rural micro-credit loans of $50 to $500 and financial services in non-banked areas of Pakistan. The project will likely meet its target number of loans (362,000), value of loans ($64.5 million) and number of clients (103,800) in Balochistan and Sindh. However, KB disbursed only 4 percent of the target number of loans in the Federally Administrated Tribal Areas (FATA). This was primarily due to the dangerous security situation. Repeat borrowers were significantly below target at approximately 30%, which is extremely low. The high dropout rate may indicate that KB did not tailor its loan products to meet customer needs. KB loan policy placed equal weight on men and women customers, which resulted in the project not reaching the target for women borrowers. The project target was 33 percent, and women received only 15.3 percent of loans and 12.4 percent of the total value of loans. In 2005, the Asian Development Bank Institute (ADBI) conducted a large-scale field survey. This survey determined how beneficiaries fared compared to those who did not receive loans. The survey found that households with access to KB loans spent more on health care, were more likely to seek medical care, spent more on education, had greater measures of women’s empowerment and were more likely to have children in school. These results strongly suggest that KB loans improved beneficiary household income and welfare. The Enterprise Development Facility (September 2003 to September 2007) Through the Pakistan Poverty Alleviation Fund (PPAF), this project provided loans in the range of $500 to $1,666 to micro-entrepreneurs. These micro-entrepreneurs needed loans larger than those available through micro-credit institutions yet smaller than those from formal financial institutions. The average loan size was $903. The project design had a Performance Monitoring Plan (PMP) and met nearly all end-of-project loan targets. However, the project was very late in training partner organization staff and failed to meet its objective of training borrowers. Unfortunately, PPAF had an inadequate risk assessment and auditing system. It did selective monitoring and evaluation at best, and over 60 percent of its loans went to one company. This company, Network Leasing Corporation Ltd. (NLCL), committed fraud and later went bankrupt. NLCL made loans to ineligible borrowers and failed to abide by the Cooperative Agreement rules. USAID first learned of this after commissioning an audit of EDF loans made through NLCL. USAID ended the project shortly thereafter6 . Widening Harmonized Access to Microfinance (WHAM) WHAM had an overarching goal of strengthening microfinance policies and coordination at the national level. WHAM also worked to expand financing for the missing middle, meaning those not served by microfinance institutions or commercial banks with loans ranging from $500 to $25,000. Regarding downscaling, WHAM applied a sound selection strategy for technical assistance and training for loan activity, yet only three commercial banks participated in the project. Furthermore, only the state-owned National Bank of Pakistan’s (NBP) participation directly led to increased loan activity. This may well have been due to attractive and attainable spreads in the organizations’ core business. With upscaling activities, WHAM experienced more success. WHAM concluded agreements with four partner organizations that have an important presence in the microfinance market, including those that exclusively focus on women borrowers. WHAM monitored progress more on the basis of the lending performance than on the intermediate goal of improved staff capacity. All of the interviewed senior managers from partner organizations stated that the technical assistance and training activities provided by the project were effective, useful and relevant. 6 Khan, Memoona, USAID Pakistan Email of April 20, 2008 In terms of industry strengthening, WHAM improved knowledge and enhanced visibility of the microfinance industry through the delivery of seminars and conferences. WHAM also disseminated updated industry information and worked closely with the Pakistan Microfinance network (PMN). In the second half of 2007, WHAM launched the Human Resource Development Initiative, a training center for microfinance practitioners. Given its very recent establishment, the evaluation team did not examine the effectiveness of this initiative. WHAM appears on-track to meet or exceed all of its activity targets except for the number of loan officers trained (77% of target achieved with one remaining quarter) and number of female borrowers (42% percent of target with one remaining quarter). However, WHAM exceeded targets for the value of loans disbursed, loans at risk, number of repeat borrowers, and annual loan revenue. The Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project) This project met or exceeded its technical targets of improving cereal crop and livestock production by 10 percent and on-farm water use efficiency by 40 percent. The project increased average cereal yield by 35 percent and average livestock yield by 21 percent. As a result, average household income increased by 23 percent. While project research demonstrated the ability to reach productivity targets, it has been slow in establishing community organizations and its capacity is spread too thin to disseminate results widely to all 223 community organizations. The project did not have a specific gender component of set of targets and therefore generated limited benefits for women. As household members, they may indirectly benefit through rising household income. The South Asia Regional Initiative for Energy (SARI/E) As a technical and policy project, SARI/E was designed to promote technical and institutional frameworks for regional energy planning and infrastructure investments. It only partially accomplished its objectives of increasing training and research capacities, increasing awareness, introducing new technology, cooperative research and developing strategic communications and regional energy planning between Pakistan and its neighbors, particularly India. Impact: To what extent has the project benefited the people of Pakistan? USAID did not design or implement any of the projects to facilitate quantitative impact assessment. Only two projects collected any baseline data and none had comprehensive monitoring and evaluation plans with indicators and targets that could generate needed data for impact assessment. (See Annex 2 for details on the types of data collected by each project.) The evaluation team attempted to recreate baselines and data. Thus, the resulting impact measures are much less complete and accurate than projects with impact assessment in their design. The Pakistan Initiative for Strategic Development and Competitiveness (PISDAC) PISDAC generated an estimated $6.4 million (present value in 2007) in monetary benefits in the dairy and gems and jewelry sectors. Direct beneficiaries included about 748 owners of cooling tanks for whom the project increased annual incomes by an average of $2,300, at least 748 newly employed tank operators earning $800 per year, perhaps as many as 25,000 individual farmers, about 167 owners of model farms earning an estimated $17,000 more each year, and about 200 jewelry artisans for whom the project increased average annual incomes by an estimated $3,300. Activities in the other sectors had not advanced enough to generate measurable impacts. The Competitiveness Support Fund (CSF) Begun in March 2006, the project was evaluated in September of 2007, which meant it had only a short history. Though its efforts influenced policymaking and set in motion several sector and industry reforms, the project had not generated measurable financial benefits. CSF did raise awareness in the government and the private sector through its “State of Pakistan’s Competitiveness Report” and a series of papers that the GoP included in its Poverty Reduction Strategy Plan II. In addition, CSF’s five sector studies and action plans (motorcycles, automotive, Karachi Fisheries, food processing and horticulture) became catalysts for both public and private initiatives. CSF’s small grant component PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 8 would likely have generated monetary impacts for the selected sectors and, potentially, for workers in those sectors. However, CSF has not disbursed any grants yet. Access to Contemporary Markets for Homebound Women Embroiders (MEDA) The MEDA project increased average annual incomes by an estimated $81, or 5%, for about 9,330 rural embroiderers. It increased average annual incomes for about 174 sales agents by an estimated $1,177 which was a 75 percent increase. MEDA also increased incomes for about 239 urban garment makers (up $300) and 40 designers (up $1,175). Anecdotal evidence also suggested that the project empowered some of the women participants by increasing their status in the household and enhancing their mobility. Kushhali Bank (KB) Development of Non-bankable Territories The Asian Development Bank Institute (ADBI) conducted a survey that suggests the project increased incomes for households that received KB loans and empowered women who received these loans. However, the evaluation team could not quantify the monetary impact of the loans on household incomes. The Enterprise Development Facility (EDF) The evaluation did not attempt to assess the impacts of EDF lending. Widening Harmonized Access to Microfinance (WHAM) The project contributed to substantial growth in the microfinance industry, particularly increasing loans in the “missing middle” range of $500 to $25,000. While it is difficult to positively attribute project activities to increased lending, it appears that the project’s scaling up component may have provided access to loans to at least 35,000 borrowers who would not have had access to loans at that time without the project. The evaluation team conducted a survey that estimated the project’s scaling up lending increased household incomes by an average of $83 per year for about 19,900 borrowers engaged in retail trade. Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project The project improved incomes by an average of $850 per year (a 23% increase) for an estimated 3,200 farmers in the rural Balochistan provinces of Mastung, Loralai, and Killa Saifullah. The South Asia Regional Initiative for Energy (SARI/E) The project produced no measurable impacts for Pakistani individuals or households. Efficiency: How efficient has the project been in utilizing its resources to achieve results? The eight projects experimented with innovative approaches to improve economic growth and reach the poor. For future projects, USAID needs to know the most efficient techniques to reach the target population. Most of the projects did not focus on a PMP or collect complete cost/benefit data. As best as possible, Table 1 provides measures of project benefits and costs. It shows that due to partial and sometimes no quantitative data, the evaluation team could not calculate even marginally reliable measures of benefits for all projects. The evaluation team did reconstruct some measures for four of the eight projects. Similar to the constraints mentioned in the impact and effectiveness sections, the evaluation team was unable to measure efficiency without data generated from a baseline and applicable set of indicators through a PMP. For example, some projects lacked benefit/cost data because they incurred costs, but had not yet generated any monetary benefits. For the projects with benefit/cost information, MEDA PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 9 PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 10 clearly stood out as a success. PISDAC, WHAM, and FAO seem likely to generate benefits well in excess of costs over the next five years. TABLE 1. USAID PAKISTAN ECONOMIC GROWTH PROJECT EVALUATIONS: SUMMARY OF EFFICIENCY MEASURES Economic Growth Projects Actual Internal Rate of Return Possible Future Internal Rate of Return Benefit-Cost Ratio Net Present Value Benefit per $1 of Costs PISDAC -15 % +41 % 0.60 -$ 3.7 mil. $ 0.34 CSF7 N.A. N.A. N.A. N.A. N.A. MEDA +59 % N.A. 4.80 N.A. $ 4.80 KB8 N.A. N.A. N.A. N.A. N.A. EDF9 N.A. N.A. N.A. N.A. $ 2.73 WHAM9 +2 % +57%10 1.00 $ 0.0 mil $ 1.00 FAO + 5 % +25 % 1.05 +$ 0.3 mil $ 1.12 SARI/E N.A. N.A. N.A. N.A. N.A. See footnotes at the bottom of this page. N. A. = Data not available. The Pakistan Initiative for Strategic Development and Competitiveness (PISDAC) This project successfully reached most of its objectives, although the evaluation team could not precisely quantify the magnitude of the benefits without additional data. The present value of measurable benefits ($6.4 million in 2007) does not cover the present value of costs ($9.1 million in 2007). The resulting internal rate of return is a negative 15 percent. However, estimated benefits over the next five years might reach $144 million, generating an internal rate of return of 41 percent. The Competitiveness Support Fund (CSF) Currently, CSF has not realized any measurable monetary impact because implementation of its studies and recommendations are just beginning. As such, the evaluation team did not calculate measures of efficiency. With future results, it will remain difficult to measure progress because the project M&E plan lacks targets, baseline data, clear indicator definitions and an approach to regular data collection and analysis. Access to Contemporary Markets for Homebound Women Embroiders This project completed its 3-year project at the end of 2007 and reached 9,783 direct beneficiaries. The present value of project benefits ($3.16 million in 2007) exceeds the present value of costs ($658,000) and produces a benefit cost ratio of 4.8 and an internal rate of return of 59%. Measures of monetary benefits are relatively complete and verifiable except that the project did not establish firm measures of baseline incomes. Kushhali Bank (KB) Development of Non-bankable Territories (Sindh, Balochistan and FATA) The Asian Development Bank Institute conducted surveys that indicated that households that received KB loans had improved welfare compared to households that did not. However, because the evaluation team could not identify quantitative measures of project impact, monetary measures of efficiency were not possible. The evaluation team did determine that KB has high operating costs and 7 CSF activities have just started to produce results. They have not achieved any monetary benefits. 8 KB and EDF evaluations were unable to identify monetary benefits. The EDF evaluation determined that it cost a $1 to make a $2.73 loan, though that represents only a small share of total EDF costs and benefits. 9 WHAM data refers only to approximately one-third of the project that focused on scaling-up microfinance institutions to provide larger loans to microenterprises. It includes benefits only to the estimated 56% of scaling-up loan recipients who ran retail businesses. 10 The estimated net present value in 2012 of the scaling up portion of the WHAM project is $61.8 million. is saddled with a large number of non-performing loans. The program also lacked a meaningful or useful performance monitoring and evaluation (M&E) plan. As a result, management did not receive feedback on operational achievements to improve operations. The evaluation team lacked the data to estimate an internal rate of return for the project. The Enterprise Development Facility (EDF) This project did not identify or collect benefit measures. It was not possible to calculate benefit cost ratios or internal rates of return. EDF records show an average loan value of $903 and a cost per loan of $331. In other words, it costs EDF nearly $1 to lend $3. An optimistic break-even point would require a lending interest rate of 20 to 30 percent. EDF lent money to partner organizations at 8 percent and the partner organizations on-lent to borrowers at 17 percent. With those costs and normal loan delinquencies and defaults, the business model does not appear to be commercially viable. Widening Harmonized Access to Microfinance (WHAM) For the loan scaling-up program with microfinance institutions, WHAM’s measured benefits equaled their costs. It cost $84 per beneficiary to generate $83 of benefits. It had a benefit/cost ratio of 1.0 and an internal rate of return of 2 percent. Similar data on the down-scaling program with commercial banks is not available. Increasing financial access for women was not an explicit part of the project. WHAM disbursed a total of 37,036 loans and women received 2,179, or 6 percent. WHAM was more successful at reaching women by working with women’s organizations, such as Asasah. The Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project This project improved incomes by an average of $850 per year for 3,200 households at an annual cost of $756 per household. That yields a benefit/cost ratio 1.05 or an internal rate of return of 5 percent. Rainwater harvesting projects in semi-arid regions of other countries also have similarly low rates. Though the project ends in 2008, its capital investment will outlive the project period and the actual rate of return for future years could be much higher. With reasonable “decay rates” for future benefits, and few new costs, the stream of benefits is a steady but declining. In that situation, the estimated internal rate of return for 2004-2013 might reach 25 percent. The South Asia Regional Initiative for Energy (SARI/E) This project provided training, technical assistance and small grants. SARI/E did not conduct a training needs assessment or did it establish indicators to measure training impact. SARI/E lacked a Pakistan specific Performance Monitoring Plan, including indicators to quantify impacts and efficiency and did not collect performance data. Therefore, the evaluation team could not assess project efficiency. Sustainability And Replication: Are the activities and results likely to be sustained after the project is completed? To what extent can the activities and results of the project be replicated? Original project designs did not include sustainability and replication, and therefore were not a focus during project implementation. The Pakistan Initiative for Strategic Development and Competitiveness Project (PISDAC) This project did not have sustainability as a project objective. Currently, only Pakistan Dairy has become fully independent from PISDAC assistance. The Sector Management Companies (SMCs) and their strategies are not self-sustaining because they depend heavily on government funding. However, replication of the project approach looks very promising. The Ministry of Industries contracted separately with the implementing partner to establish two new sector-based Strategy Working Groups. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 11 The Competitiveness Support Fund (CSF) This project raised awareness of competitiveness, especially within the GoP. CSF policy studies caused the GoP to embrace competitiveness as a way to reduce poverty and support economic growth. The GoP allocated substantial funding to implement several CSF recommendations. Without external funding, either from a donor or the GoP, CSF is not sustainable. Similar to PISDAC however, given the political will, it has great potential to be replicated in other ministries and countries. Access to Contemporary Markets for Homebound Women Embroiders MEDA successfully increased women’s income through a value-chain approach. Already, the project expanded to include urban garment makers and designers. The incentives and information flows on design and markets support sustainability. With a sound business model, MEDA appears likely to continue delivering benefits now that the project has ended. The project approach is sustainable and can be replicated. In fact, the Canadian International Development Agency (CIDA) plans to provide $7 million to replicate the project approach and expand the targeted sectors in other areas of Pakistan. The Kushhali Bank (KB) Development of Non-bankable Territories (Sindh, Balochistan and FATA) USAID designed this project to increase rural micro-credit and financial services. Several indicators raise serious questions about its sustainability and replication. The bank kept approximately half of its funds in ultra safe GoP Treasury securities rather than increasing its lending. Even with a high loan subsidy, there is an extremely high drop out rate with nearly 70 percent of borrowers not borrowing a second time. This indicates that the bank is not nurturing and serving its customers or the borrowers see little value in the credit offered. Many bank branches have high default levels and operate at a loss, even after the USAID cost-share. KB also does not always take action on borrowers who default on their loans. With the present approach, there is little possibility of sustainability or replication. The Enterprise Development Facility (EDF) This project provided loans in the range of $500 to $1,500 to micro-entrepreneurs who needed loans larger than those available through micro-credit institutions and smaller than those from formal financial institutions. EDF worked through partner organizations that make the loans. The partner organizations are charged only 8 percent interest and they lend to borrowers at 17 percent. These rates are well below market rates and not high enough to cover operating costs. The EDF works closely with the Pakistan Poverty Alleviation Fund, which suffers from an extremely hierarchical decision￾making structure and weak operating procedures. Furthermore, there are no incentives for the partner organizations to institute badly needed management and financial reforms. Given these problems, the EDF approach is not likely to be sustainable nor should it be replicated. Widening Harmonized Access to Microfinance (WHAM) This project targeted the missing middle, in other words, those not serviced by microfinance institutions or commercial banks. WHAM’s sustainability depends on how well it can build relationships with the financial organizations receiving assistance and with other stakeholders in the financial sector. Building mutual trust and confidence requires a substantial level of resources and energy, and even more importantly, a long period of time. WHAM had only four years to complete the project, which is not long enough. The evaluation team found a lack of incentives for commercial banks to move away from typical commercial loans because during the project period, banks benefited from high spreads in their primary target audience (large business). This undermined project efforts to encourage downscaling of loans to smaller borrowers. Scaling-up shows greater opportunity and potential for replication. Both sustainability and replication prospects are uncertain. The Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project This project gives training and hands-on experience, which enhanced agricultural research capacity, but retaining that capacity is beyond the project’s control and might not continue once the project ends. The community organizations are too small and immature to continue without outside support. The project included plans for replication into similar agro-climatic regions of the NWFP and FATA, but the evaluation team noted a lack of skilled water, livestock and crop experts available to work on PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 12 PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 13 such new projects. Even with those constraints, replication could make sense. The project demonstrated very strong results in a difficult regional and economic environment, and is a strong candidate for scaling-up in other semi-arid regions of Pakistan. The South Asia Regional Initiative for Energy (SARI/E) As designed this project promoted technical and institutional frameworks for regional energy planning and infrastructure investments. Though possible to support research, training, technical assistance, improved coordination and demonstration projects, this works only if the donor and government see a priority need. The sustainability of SARI/E as a regional program was beyond the evaluator’s scope of work. As for replicability, without stronger involvement and influence by the participant country in determining the program’s activities, replication is not recommended. USAID/ Pakistan did not fund the effort (regional USAID funds were used) and SARI/E did not fit neatly in the Pakistan Economic Growth program. Gender Considerations When designing a new program, USAID analyzes economic, institutional, and policy issues along with the social soundness of its approach. Often, gender does not become a key issue in this analysis. Without an understanding of how program design affects men and women differently, projects run the risk of not including, and benefiting from, women appropriately. USAID needs to measure benefits, set up meaningful targets and indicators, and establish a plan to monitor and achieve those targets. For example, in Pakistan the Kashf Foundation provided credit primarily to women, but the evaluators discovered that men actually used 90 percent of sector loans11. In addition, those who implement activities (project staff, partner staff and managers) with gender targets must be aware of any cultural constraints and social norms. For example, Pakistani custom does not allow male project staff to visit the homebound women embroiders. In fact, the simply could not meet with them, regardless of where. Thus, trained women staff and staff are essential. As previously mentioned, almost all of the projects failed to incorporate gender in their design and contracted scopes of work. Given this, as Table 2 indicates, except for MEDA, most projects had few women beneficiaries. In fact, MEDA was the only project that collected data on the monetary value of benefits received by women. Other projects rarely targeted women as primary beneficiaries, although several did try to incorporate women as both staff and beneficiaries where possible. This is especially unfortunate in Pakistan, which the World Economic Forum Gender Gap Study ranks as 126 out 128 countries in the world. 11 Daly, Jorge, “Impact of USAID Investments in the Pakistan Micro-Finance Sector.” Draft, April 2008 TABLE 2. USAID PAKISTAN ECONOMIC GROWTH PROJECTS: SUMMARY OF GENDER IMPACTS Project Number of Direct Female Beneficiaries and as a (% ) of Total Beneficiaries Annual Benefit Per Female Beneficiary Benefits as a % of Annual Income PISDAC 50 trainees* 3% 30% of PISDAC Staff of which 70% are mid or senior staff N.A. N.A. CSF 5 of 13 CSF Staff (40%) 3 of 15 CSF Board Members (20%) N.A. N.A. MEDA 9,295 Embroiders (100%) 488 Sales agents, designers, garment workers (100%) $81 Embroiders $2,745 Sales Agents 5 % 174 % KB 10,862 loans (11.3%) N.A. N.A. EDF 1,409 loans (28%) N.A. N.A. WHAM* 2,179 loans (6%) 114 trained professionals (16%) N.A. N.A. FAO Households are beneficiaries, not women 8 of 37 staff (20%) 77 of 223 Community Organizers (35%) N.A. N.A. SARI/E 10 women trainees (6%) N.A. N.A. * Note: Fifty PISDAC gem stone trainees were women. After completing their training, half were not allowed to work by their brothers or fathers. WHAM technical assistance to Asasah and The Kashf Foundation indirectly generated 2,179 loans to women and helped Asasah develop a savings component for borrowers. N.A. = Data not available The Pakistan Initiative for Strategic Development and Competitiveness (PISDAC) This project lacked gender-specific targets or objectives in its SOW, though a general objective mentions employment of women. However, the project made an expressed effort to include women in its priority sectors, many traditionally closed to women. PISDAC trained women in the jewelry business, stone cutting, and mosaic making. Though women had competitive market advantage in some areas (mosaic setting and gem setting), PISDAC still faced entrenched cultural biases. Although PASDEC trained 50 women in assembling mosaics, brothers and husbands did not permit half of those trained to work. Even so, PISDAC felt that the project made progress in changing cultural stereotypes. The PISDAC project had hired about 30% women as of this evaluation; 70% of these were middle or senior staff. The Competitiveness Support Fund (CSF ) This project has limited impact on women beneficiaries to date. More than 40 percent of CSF staff and 3 out of 15 board members are women. One of the eight CSF studies examines empowerment of women and linkages to competitiveness, but the study has not had any impact to date. Access to Markets for Homebound Women Embroiderers in Pakistan MEDA worked successfully on increasing women’s income by using a value-chain approach that provided appropriate information on consumer demand through a linkage with sales agents. Sales agents worked closely with embroiders to improve their skills, product quality and price. The project met all performance indicators and targets and exceeded several others. Product quality and design have improved. The embroiderers have higher income, greater status in their household, more PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 14 children going to school and the general quality of life has improved. Women were the primary target beneficiaries and in actual practice that has been the result. The Khushhali Bank (KB) Developing Non-bankable Territories for Financial Services Project This project mobilized 15,683 community groups of which 9 percent were women’s groups. The Asian Development Bank Institute (ADBI) completed a large field survey in 2005 to determine how beneficiaries fared compared to those that did not receive loans. This study found that households with access to KB loans spent more on health care, were more likely to seek medical care, spent more on education, had greater measures of women’s empowerment and were more likely to have children in school. These results strongly suggest that beneficiary household income and welfare improved as a result of KB loans. While the project target was 33 percent, only 15 percent of clients were women. The average value of a woman’s loan was 80 percent of the average man’s loan. Only 6 percent of loan officers were women, which likely had negative implications with respect to outreach to women The Enterprise Development Facility (EDF) The evaluation team found no evidence that the project targeted women in a systematic manner or urged its partner organizations to actively recruit women borrowers. Toward the end of the project, the participation of women sharply increased and women received 28 percent of loans. The average value of a loan received by a woman was 82 percent of that received by a man. Widening Harmonized Access to Microfinance (WHAM) The project design did not include increasing financial access for women. Despite this, WHAM did set a target for the number of women borrowers, but missed it by 50 percent. A total of 37,036 loans were disbursed of which women received only 6 percent. WHAM was somewhat more successful at hiring women (7 of its 15 full- time staff were women) and women were 16 percent of bank professionals who received training. WHAM did find that very few women occupy the market segment of owning and managing their own small enterprise in Pakistan. Recognizing this, WHAM calibrated the importance of this gap and enlisted microfinance organizations that cater their services exclusively to women. For example, WHAM helped build the institutional capacity of Asasah, a successful micro-savings institution, managed and staffed by women. As a result, Asasah provided 1,154 loans to women and created a new savings account feature. WHAM technical assistance to The Kashf Foundation indirectly generated 1,025 loans to women. The Food Security/Poverty Alleviation in Arid Agriculture Balochistan Project This project generated limited direct benefits for women in community organizations, poultry, drinking water, kitchen gardens and food processing. Out of 37 staff members, 8 (20 percent) were women. Even though women do have a limited role in livestock care, gender considerations were not part of the program. As household members, women may indirectly receive benefits with rising household income. The South Asia Regional Initiative for Energy (SARI/E) March 2006 USAID RFP The overall management and coordination of the SARI/E program in Pakistan included a full-page gender statement. However, the implementing partner scopes of work had no reference to gender. Women did not make up a significant proportion of SARI/E trainees (10% or 6 out of 60), probably because the training targeted a level of officials that includes very few women. Prime USAID contractors did not ensure that women were represented in partner organizations nor did they focus on women as anything more than energy consumers. Project managers said: “The SARI/E program already has too many foci. Project training is targeted at a level of officials that includes few (if any) women. Targeting women in a sector and culture in which women do not lead is difficult. A single project cannot change years of cultural history.” PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 15 USAID Reporting And Branding Requirements, Outreach And Communications In general, projects submitted financial and operations reports in a timely manner and met USAID branding requirements. On the other hand, public relations and media coverage had various results. The project designs and scopes of work often did not address or emphasize this subject. Both CSF and PISDAC developed new initiatives and received extensive press and TV coverage. Most of the other projects received project attention in local newspapers for conferences or special reports, but rarely at other times. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 16 PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 17 ANNEX 1. COMPETITIVE AND NON-COMPETITIVE CONTRACT AWARDS Award Type, Project Dates, USAID Funding Activity and Implementing Partner Competitive or Non-Competitive Contract Award Award Type Project Start and End Dates USAID Commitment in $ Million’s Total $53.653 PISDAC Pakistan Initiative for Strategic Development and Competitiveness. R. Nathan and J.E. Austin PISDAC I Competitive PISDAC II Non-Competitive PISDAC III Non-Competitive Contract Contract Amendment Contract 05/10/04 to 08/12/04 08/12/04 to 08/31/06 02/08/06 to 05/31/08 Total $ 0.350 $ 3.952 $ 6.996 $11.298 CSF Competitiveness Support Fund GoP Ministry of Finance Non-Competitive Limited Grant Agreement with GoP Ministry of Finance 03/03/06 to 12/31/08 $11.823 MEDA Access to Markets for Homebound Women Embroiderers Mennonite Economic Development Association (MEDA), Enterprise and Career Development Institute (ECDI) Competitive USAID Washington Cooperative Agreement Microenterprise Implementation Grant Program 09/30/04 to 12/31/07 $ 0.506 KB Developing Non￾Bankable Territories for Financial Services Khushhali Bank Non-Competitive Cooperative Agreement 10/01/03 to 09/30/10 $11.052 EDF Enterprise Development Facility Pakistan Poverty Alleviation Fund Non-Competitive Cooperative Agreement 09/30/03 to 09/30/07 $ 6.32012 WHAM Widening Harmonized Access to Microfinance Competitive Request for Cooperative Agreement 03/06/05 to 03/31/08 $ 5.000 12 EDF was terminated early by USAID because of poor project management. Only $2.529 million was disbursed. PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 18 Activity and Implementing Partner Competitive or Non-Competitive Contract Award Award Type Project Start and End Dates USAID Commitment in $ Million’s Total $53.653 Shore Bank Advisory Services Assistance FAO Poverty Alleviation in Arid Agriculture U.N. Food and Agricultural Organization Non-Competitive Grant Agreement 12/01/04 to 12/31/08 $ 5.970 SARI/E South-Asia Regional Initiative for Energy PA Government Services, Haggler Bailey, U.S. Energy Association, National Renewable Energy Laboratory, NEXANT, Winrock International Non-Competitive Regional USAID Contract 2000 to 2011 No USAID Pakistan Funds. An estimated $1.684 from regional funds. Included in USAID total commitment above. ANNEX 2. WAS PROGRAM MANAGEMENT DATA COLLECTED? USAID EG PROJECTS Monitoring & Evaluation Plan (M&E) Performance Monitoring Plan (PMP) Baseline Data Collected End of Project Targets Performance Indicators Gender Targets & Indicators PISDAC None, SOW had no reporting requirements None No indicators None None None. Contractor submitted quarterly reports to USAID No gender targets, or indicators CSF YES, but limited in scope and utility None None None None. Contractor submitted reports to USAID No gender targets, or indicators MEDA None None YES YES YES, 3 YES KB None None None None YES, 4 indicators YES 1 One-third of borrowers women EDF YES, but limited in scope and utility None None YES, 5 YES, 16 YES 1 Value of loans to women WHAM None None None YES YES, 3 No gender targets, or indicators FAO None None YES YES, 6 YES 5 performance 135 output YES, 3 30%-40% Participati on rate SARI/E None None None None None None PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 19 ANNEX 3. OVERALL EVALUATION QUESTIONS 1. Relevance: How well was the project focused on the needs of the beneficiaries? Was the project well designed to address the needs of the beneficiaries? How well was the project adjusted to address the needs of the beneficiaries? To what extent did the design of the activity utilize participatory techniques? Was the activity designed to meet a felt need of a specific community, target audience, or influential stakeholder? Were stakeholders involved in a substantive way throughout the project life cycle? Was the targeting appropriate in hindsight? 2. Effectiveness: Has the project accomplished its objectives? How were the initial targets established for each activity? Were the targets appropriate and realistic? To what extent were the targets achieved? What are the lessons learned for setting targets in future activities in accordance with the requirements of USAID’s Performance Monitoring Plan (PMP)? 3. Efficiency: How efficient has the project been in utilizing its resources to achieve results? To the extent possible, what is the internal rate of return for this project, as calculated in a cost benefit analysis? How cost-effective has the project been? How do overhead and administrative costs for this activity compare to others across differing types of implementation mechanisms (e.g. Contract, Limited Scope Grant Agreement, Grant, Cooperative Agreement) and for the different types of implementing entities (e.g. local vs. international firms, non-profits vs. for-profits, etc)? 4. Impact: To what extent has the project benefited the people of Pakistan? How has the program benefited the intended beneficiaries? What were the primary and secondary positive and negative impacts of the projects? How large has the impacts been or are likely to be? To what extent can the impacts be attributable to the project? How were the impacts distributed by region, sector and gender of the beneficiaries? Were any of these benefits or losses unexpected? 5. Sustainability: Are the activities and results likely to be sustained after the project is completed? Were the activities designed in a manner which focuses on their sustainability after project completion? Were the activities implemented in a manner which focuses on their sustainability after project completion? PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 20 Was the initial timeframe for the activity realistic to achieve sustainable results? Were any of the activities fundamentally designed and implemented in a way which creates donor dependence? Is it reasonable to expect the project to achieve sustainability during the funding life of the project given internal and external factors? 6. Replication: To what extent can the activities and results of the project be replicated? Were the activities designed in a manner which focuses on their replication? Were the activities implemented in a manner which focuses on their replication? Can the activities be replicated in other areas with similar socio-economic features? Can the activities be replicated in dissimilar areas? To what quantified extent can the project be replicated? 7. Reporting: Have the prime contractors and grantees fulfilled all of their reporting requirements in an accurate, comprehensive and timely manner? Have the partners fulfilled all of their reporting requirements? Were the reports useful to USAID staff? Were all branding guidelines followed? Were the reported results accurate and verifiable? How can the reporting requirements and formats be improved? 8. Public Relations/Media Coverage: How effective has the project been in telling its story? How effectively has the project highlighted success stories? How effective were the project’s public relations events/activities in terms of frequency, nature, profile, content and design, branding and participation? To what extent have they raised awareness of the activity and influenced the perception of beneficiaries? How can the impact of the public relations component of future programming be improved? 9. Coordination: How effectively has the project coordinated with other parties? How effectively has the project coordinated with the Government of Pakistan? How effectively has the project coordinated with other USG projects? How effectively has the project coordinated with other donors? How effectively has the project coordinated with other stakeholders? To what extent were synergies developed between the project and other individual USAID EG activities, other donor programs and/or GOP initiatives? What concrete steps should be taken to improve coordination and maximize synergies in future activities? 10. Gender: To what extent has the project benefited women? To what extent has the project included women in its staff, partners, agents, etc.? To what extent has the project targeted women in its activities? PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 21 PAKISTAN ECONOMIC GROWTH PROGRAM EVALUATION EVALUATION SUMMARY REPORT 22 How effective has the project been in reaching women? What are the direct quantified benefits that the project has had on women?