FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT June 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under Pakistan Poverty Alleviation Fund (PPAF), Cooperative Agreement 391-A-00-03-01010-00 Enterprise Development Facility (EDF) Project DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS PROJECT SUMMARY...............................................................................................................III ACRONYMS................................................................................................................................IV EXECUTIVE SUMMARY .......................................................................................................... V SUMMARY EVALUATION REPORT................................................................................... VII ANNEX 1: FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT............................................................................................. 1 I. INTRODUCTION ....................................................................................... 2 II. THE DEVELOPMENT PROBLEM ......................................................... 3 A. Problem Statement .............................................................................. 3 B. USAID Intervention............................................................................ 4 III. EVALUATION METHODOLOGY .......................................................... 5 A. Evaluation Purpose and Questions...................................................... 5 B. Evaluation Design............................................................................... 6 C. Data Collection Sources and Methods................................................ 7 D. Data Limitations.................................................................................. 7 IV. FINDINGS AND CONCLUSIONS ON THE OVER ARCHING QUESTIONS ........................................................................... 8 A. Relevance ............................................................................................ 8 B. Effectiveness....................................................................................... 9 C. Efficiency.......................................................................................... 15 D. Impact ............................................................................................... 16 E. Sustainability..................................................................................... 18 F. Replication ........................................................................................ 20 V. FINDINGS AND CONCLUSIONS ON THE CROSS CUTTING ISSUES ................................................................................... 21 A. Gender............................................................................................... 21 B. Reporting........................................................................................... 22 C. Public Relations/Media Coverage..................................................... 22 D. Coordination...................................................................................... 22 VI. RECOMMENDATIONS .......................................................................... 22 VII. LESSONS LEARNED............................................................................... 23 ANNEX 2: REFERENCES ......................................................................................................... 24 ANNEX 3: LIST OF PEOPLE INTERVIEWED ..................................................................... 28 ANNEX 4: GETTING TO ANSWERS MATRIX .................................................................... 29 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT i TABLES Table 1: Estimated Program Budget of the EDF .............................................................................5 Table 2: Getting to the Answers for Overall Evaluation Questions.................................................6 Table 3: Project Goals, Expected Results, Indicators, End-of-Project Actual Results and Percentage of Target Achieved .....................................................................................10 Table 4: Loans Disbursements to POS as of June 2007 ................................................................11 Table 5: EDF Loaning by Sector from January 2004 to June 2007...............................................12 Table 6: Primary Monetary and Non-Monetary Benefits of Microfinance Lending .....................17 Table 7: Loan Disbursements by Gender as of June 2007.............................................................21 FIGURES Figure 1: Results Framework for EDF.............................................................................................4 Figure 2: Disposition of Current Expenditures..............................................................................15 Figure 3: Selected Expenditure Ratios...........................................................................................16 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT ii PROJECT SUMMARY Project Name Enterprise Development Facility (EDF) Initiative Life of the Project (LOP) September 2003 to September 2007 Project Location Nationwide Implementing Partner Pakistan Poverty Alleviation Fund (PPAF) Activity Number 391-A-00-03-01010-00 Type of Activity Cooperative Agreement Budget USAID Share US$ 6.3m PPAF Share: US$ 0.8m Total: US$ 7.1m FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT iii ACRONYMS AR Annual Report BoD Board of Director CBO Community Based Organization CED Credit and Enterprise Development Unit of PPAF CFI Commercial Financing Institution CO Community Organization CSC Community Support Concern CSO Civil Society Organization CTO Cognizant Technical Officer of USAID CWCD Center For Women Cooperative Development DFID U.K Department for International Development DO Development Objective EDF Enterprise Development Fund EMF Environmental Management Framework FMFB First Micro Finance Bank ERD Evaluation, Research and Development Unit of PPAF FSP Financial Services to Poor FY Financial Year GOP Government of Pakistan GTZ German Technical Cooperation Agency HID Human and Institutional Development Unit of PPAF IFAD International Fund for Agricultural Development JICA Japan International Cooperation Agency KB Khushhali Bank KASB Bank Khadim Ali Shah Bukhari Bank MFB Micro Finance Bank MFI Micro Finance Institution M & E Monitoring and Evaluation MIS Management Information System NGO Non Government Organization NLCL Network Leasing Corporation Limited NMFB Network Micro Finance Bank Limited Non-FSP Non-Financial Services to Poor NWFP North-West Frontier Province OLP ORIX Leasing Pakistan Limited PMP Performance Monitoring Plan POMFB Pakistan-Oman Micro Finance Bank Limited PO Partner Organization PPAF Pakistan Poverty Alleviation Fund PRSP Punjab Rural Support Program QPR Quarterly Progress Report RCDS Rural Community Development Society RMFB Rozgar Micro Finance Bank Limited RSPN Rural Support Program Network SAFWCO Sindh Agricultural & Forestry Workers Cooperative Organization SBF Small Business Finance SBI Shore Bank International Limited SECP Securities and Exchange Commission SME Small & Medium Enterprises TMB Tameer Micro Finance Bank USAID United States Agency for International Development WHAM Widening Harmonized Access to Micro Finance FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT iv EXECUTIVE SUMMARY In August 2007, the United States Agency for International Development (USAID) commissioned Management Systems International (MSI) to conduct the evaluation of eight projects of its Pakistan Economic Growth Activities. Enterprise Development Facility (EDF) is one of three projects that seeks to increase access to financial services to the benefit of households and micro and small businesses in rural and urban areas across Pakistan. This report addresses the final evaluation of EDF. The structure and terms of EDF are specified in the Cooperative Agreement No 391-A-00-03-01010-00 of September 30, 2003 signed with the Pakistan Poverty Alleviation Fund (PPAF). Initially valid for a period of three years, the Agreement was amended on September 26, 2006 for a period of one additional year. The project budget is $7.1 million, and USAID contributes $6.3 million (89%). The project’s contractual arrangement dictates its direct execution by PPAF, the beneficiary agency. In 2000, PPAF launched the EDF facility to address the economic development constraints of the relatively less poor, otherwise known as “potentially poor” in Pakistan. Sponsored by the Government of Pakistan (GOP), PPAF is an apex organization with a seed capital of $13.3 million donated by two multilateral donors, the Asian Development Bank (ADB) and the World Bank. As an apex organization, PPAF acts as a wholesaler and intermediary of funds. By funding NGOs, PPAF plays a significant role in the expansion of the micro credit market of Pakistan. Operationally, the EDF project consisted of two main components: (a) provision of financial services to the potentially poor (FSP); and (b) extension of non-financial services to the poor (non-FSP). The first component consists of $4.493 million, which EDF disbursed to the actual beneficiaries through its partner organizations (POs). In turn, the POs provided individual loans between $500-1,666 to entrepreneurs who had already successfully serviced conventional micro credit loans (loans under $500 from previous PPAF micro credit programs). Intended to enable the entrepreneurs to expand their businesses, these larger loans generated larger streams of revenue and increased businesses’ capacity to buy more assets. LOANS DISBURSEMENTS TO POs AS OF JUNE 2007 Partner Organization No. of Loans (and Percentage of Total) Total Value of Loans (and Percentage of Total) Average Loan Size CWCD 904 (18.2%) $738,000 (16.9%) $816 RCDS 697 (14.1%) $499,000 (11.4%) $716 PRSP 196 (4.0%) $137,000 (3.1%) $699 NLCL 3,044 (61.4%) $2,894,000 (66.1%) $951 SAFWCO 62 (1.3%) $44,300 (1.0%) $714 CSC 54 (1.1%) $68,000 (1.6%) $1,259 Total 4,957 $4,379,000 $883 Non-FSP, on the other hand, made up 15% of the total USAID contribution. This component provided technical support to build the capacity of beneficiaries in order to generate more economic value for their micro-enterprises. Typically, this technical support took the form of training programs to improve entrepreneurial skills, improve marketing techniques, and give access to information networks. The non￾FSP component of ESF also included activities to increase the capacity of the POs to enable them to manage and deliver the EDF credit program. The evaluation of the EDF Project focused on 10 primary themes. The following table summarizes the main conclusions for each theme: FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT v Evaluation Theme Key Conclusions Relevance • The EDF project effectively focused on the real needs of a sizeable group of beneficiaries among the Pakistani population. • Loans were channeled through partner organizations that also benefit from non￾financial assistance by PPAF. This may have inhibited initiatives of partner organizations from suggesting program adjustments that could have better met the needs of the end-user beneficiaries. Effectiveness • On the FSP side, the EDF project met or significantly exceeded its targets for six of its objectives with respect to providing sub-loans to relatively poor entrepreneurs. • EDF failed to reach its target of seven new POs to undertake enterprise development activities (it had 6, of which only 3 were active by the end of the project). PPAF relied on “known quantities” rather than a careful examination of the potentially positive opportunities offered by other retailers not participating in the program. • On the non-FSP side, the project was hugely effective in training PO staff, but fell short in meeting its objective of training borrowers. Impact • EDF lending did impact the welfare of beneficiaries, but produced no firm evidence that it affected household incomes or other monetary measures. Efficiency • With no monetary measures of benefits, the evaluation team could not calculate the usual efficiency measures (i.e., net present value, benefit cost ratio, internal rate of return). Sustainability • The EDF project was neither designed nor implemented in ways that focused on the sustainability of partner organizations after project completion. Replication • Expanding EDF-type activities in all geographical areas (except FATA) is contingent on whether the PPAF or some other apex organization(s) follow improved management procedures and practices. Gender • Women received nearly double the pre-determined project targets for loans. Despite that fact, women received considerably fewer loans than men. Reporting • EDF submitted the Planning and Implementation Plans and quarterly reports with goals and activities of the project. EDF also tracked actual performance against actual targets. The evaluation team did not find evidence of branding guidelines in place. Communication and Outreach • EDF drafted and disseminated well-written case studies of loan impact to donors and GOP agencies. Coordination • PPAF implemented its poverty alleviation activities in accordance with the goals of the GOP, as specifically stated by the SBP. PPAF also provided technical inputs 0used by the World Bank for the preparation of the Poverty Reduction Strategy Paper. Recommendations To the extent that it remains engaged in finance projects, USAID should build on the credit assessment of PPAF recently conducted by Shorebank International, and consider the following: (a) Restructure the finance and development components of PPAF, and define clear objectives and roles for each. (b) Replicate EDF under the following conditions: (1) wholesaling of long-term loans at commercial rates; (2) credit-granting NGOs develop specialized credit-only arms as a sine-qua￾non condition for accessing assistance; and (3) continuing analysis of POs for financial self￾sufficiency progress. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT vi FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT vii SUMMARY EVALUATION REPORT1 INTRODUCTION In August 2007, the United States Agency for International Development (USAID) commissioned Management Systems International (MSI) to conduct the evaluation of eight projects within the purview of its Pakistan Economic Growth Activities. Three of these projects seek to increase access to financial services for households and micro and small businesses in rural and urban areas (Widening Harmonized Access to Microfinance (WHAM), Enterprise Development Facility (EDF) and Developing Non￾Bankable Territories for Financial Services). This report addresses the final evaluation of EDF. The structure and terms of EDF are specified in the Cooperative Agreement No 391-A-00-03-01010-00 of September 30, 2003 signed with the Pakistan Poverty Alleviation Fund (PPAF). Initially valid for a period of three years, the Agreement was amended on September 26, 2006 for a period of one additional year. Total project budget amounts to $7.1 million, of which $6.3 million (89%) is funded by USAID. The project’s contractual arrangement dictates its direct execution by PPAF, the beneficiary agency. y in wn l of zation, focus on the poor and the disadvantaged. s t businesses, thus generating larger streams of revenue and increasing their capacity to buy more assets. PPAF launched the EDF facilit 2000 to address the economic development constraints of the relatively less poor, otherwise kno as “potentially poor” in Pakistan. Sponsored by the Government of Pakistan (GOP), PPAF is an apex organization with a seed capita $13.3 million donated by two multilateral donors, the Asian Development Bank (ADB) and the World Bank. As an apex organi PPAF acts as a wholesaler and intermediary of funds. With its large endowment fund, PPAF places significant volumes of funds in interest-bearing accounts. Also, PPAF obtains financial resources at zero or 0.75% interest rate, which allows for short-term sustainability. By funding NGOs, PPAF plays a significant role in the expansion of the micro credit market of Pakistan. Following a risk assessment, PPAF applies a set of criteria to select NGO loan retailers. This includes the candidate NGO’s demonstrated track record on loan quality portfolio, outreach, social mobilization, and its level of Operationally, PPAF divided the EDF project into two main components: (a) provision of financial services to the potentially poor (FSP); and (b) extension of non-financial services to the poor (non-FSP). The first component consists of $4.493 million, which EDF disbursed to the actual beneficiaries through its partner organizations. Management of the program cost $886, 438. The POs provided individual loan between $500-1,666 to entrepreneurs who had already successfully serviced conventional micro credi loans (loans under $500 from previous PPAF micro credit programs). These larger loans enabled the entrepreneurs to expand their 1 The full evaluation report is included as Annex 1, included information on data sources and additional annexes. Non-FSP, on the other hand, was allocated $939,983 or 15% of the total USAID contribution. This included technical support to build the capacity of the beneficiaries, which generated more economic value for their micro-enterprises. Typically, this technical support took the form of training programs to improve entrepreneurial skills, improve marketing techniques, and give access to information networks. Non-FSP activities also included increasing the capacity of the POs for managing and delivering the EDF credit program. Evaluation Methodology The evaluation of the EDF Project relied on the following data collection methods. • Review of relevant documents including (a) the cooperative agreement and its amendment; (b) the project proposal presented to USAID, project implementation plans and ten quarterly reports; (c) important documentation pertaining to the PPAF and its partner organizations such as their Annual Reports, their presentations on their EDF lending component, and their system procedures; and (d) a major audit report done on the implementation of the Co-operative Agreement from October 1 to June 30, 2005. The review was conducted when the Team Leader arrived in Islamabad on October 12, 2007. • Interviews with key informants. These included (a) five representatives of the donor community; (b) one representative each from a microfinance bank and a Rural Support Program; (c) an official of the State Bank of Pakistan; and (d) a representative of a civil society organization – the Pakistan Microfinance Network. A list of the key informants is included in Annex 3. • PPAF/EDF: Interviews with (a) the Chief Strategic Planning Officer, the General Manager, the Management Executive and several high ranking officials; and (b) principals and staff of three major partner organizations that are recipient of credit from EDF, i.e., the National Leasing Corporation Ltd (NLCL) located in Karachi, the Lahore-based Center for Women and Community Development (CWCD), and the Rural Community Development Services (RCDS), which is located in Mandi Faizabad. The team conducted these interviews from October 17th to November 28th in 2007. A list of these individuals is also included in Annex 3. • Group interviews with 20 selected borrowers from CWCD and 11 borrowers from RCDS. The evaluation team prepared a comprehensive design to conduct the evaluation using a Getting to Answers (GTA) Matrix. This planning tool helps determine information to answer each evaluation question, from where and how the data can be obtained, and appropriate analysis techniques. The GTA for this evaluation is included in the main EDF report. FINDINGS AND CONCLUSIONS Relevance The beneficiary target group for the EDF project was micro-entrepreneurs who had a loan-demand profile in the range of $500 to $1666. These loans were larger than the kinds of loans available through conventional micro-credit programs. On the other hand, these loans were smaller than those available from formal financial institutions. Micro-credit institutions offered very small microloans of $160-250, but these do not help a micro-entrepreneur develop business potential, even if they received these loans on a repetitive basis. Despite the recent entry of specialized microfinance banks in Pakistan, loan supply from formal financial institutions in the $500-1,666 range is still very thin. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT viii Given its mandate and functional linkages to NGOs, PPAF was a feasible, effective and suitable organization to assume the main responsibility for implementing this initiative. In 2003, PPAF had a pool of 28 partner organizations (POs). Several of these were eligible to conduct the EDF program. PPAF involved them via a roundtable discussion for their feedback, comments and recommendations. Some partner organizations also obtained grants from PPAF for activities in the areas of infrastructure, health and education. For example, RCDS obtained health grants from PPAF to establish two clinics for its Mother and Child Care program. Also, RCDS and CWCD obtained micro-credit loans from PPAF through projects and initiatives not associated with the EDF facility. Conclusions First, EDF focused in a very relevant way for a large group of beneficiaries by providing loans in the range of $500-1666 to micro-entrepreneurs. These beneficiaries had loan needs unmet by either micro￾credit institutions or formal financial institutions. Second, EDF channeled loans through partner organizations that also benefit from non-financial assistance by PPAF. This may have inhibited partner organizations from suggesting program adjustments that could have better met the needs of the end-user beneficiaries. Effectiveness The Performance Monitoring Plan (PMP) identified five major results, as shown below. In order to track progress toward these results, the plan included 16 performance indicators, 7 for the FSP component and 9 for the Non-FSP component. The indicators had end-of-project targets, some of which were revised due to the one year extension. See following table for details. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT ix FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT x PROJECT GOALS, EXPECTED RESULTS, INDICATORS, END-OF-PROJECT ACTUAL RESULTS AND PERCENTAGE OF TARGET ACHIEVED EDF Goals as Stated in the Cooperative Agreement Original Expected Results as Stated in the Coop. Agreement End-of-Project Targeted Indicators in the PMP End-of-Project Actual Data (and Percentage of Target Achieved) 3,610 sub-loans to entrepreneurs 4,957 (137.3%) 200 sub-loans to repeat borrowers 334 (167.0%) 722 enterprises financed with women participation 1,409 (195.2%) Extend at least 2,295 enterprise sub-loans of greater than Rs 30,000 (US $517) each $898,489 disbursed to women entrepreneurs $1,302,205 (144.9%) Maintain the quality of the 10 % maximum of portfolio at risk <10 percent overall loan portfolio (Portfolio at Risk not exceeding 10 percent) 90 % maximum recovery rate of dues >90 percent Achieve an average ex￾post incremental increase of 10 % in the incomes/ assets of borrowing households 10 % increase in income and/or assets of borrowers No Data Available2 80 % of businesses expanded No Data Available (see f/n 5 below) Secure sustainable livelihoods for the poor through support to entrepreneurship and local micro￾businesses enabling them to generate income/assets 800 borrowers trained in Enterprise Skill Development & managerial skills 654 (81.8%) Institutionally 7 new POs inducted 6 (85.7%) capacitate selected org’ns to undertake second-generation innovations in micro-credit/ enterprise dev’t Enable at least 7 partner organizations (POs) to undertake enterprise development activities 18 PPAF Partners’ staff trained in program management & skill dev’t 125 (694.4%) $4,492,759 credit disbursement to Pos $4,379,031 (97.5%) $586,207 disbursements for Entrepreneurs Skill Dev’t & managerial skills $35,276 (6.0%) $172,414 disbursements for POs’ Staff Development Events $15,284 (8.9%) $12,931 disbursement for Market/Product/Other Research $32,986 (25.5%) $887,931 total disbursement for Non￾Financial Component $83,546 (9.4%) Both Goals Disburse Rs 295 million (US $5.0 million) for credit and capacity building (Total for credit & capacity building—i.e., FSP and Non-FSP Components, which was targeted in the original Cooperative Agreement at $5 million) $4,462,577 (89.3% of target in column 2) Source: USAID, PMP, 2006, and PPAF’s Quarterly Progress Reports 14 and 15 2 Data was not included in the PMP within the Quarterly Reports. A baseline conducted by Gallup Poll in 2007 did not report baseline income and changes in this income of individual respondents over time, or changes in income of who received loans, against those who did not (the control group). Financial Services to Poor EDF met or far exceeded its targets for six of the seven FSP-related performance indicators: number of sub-loans to entrepreneurs, number of sub-loans to repeat borrowers, number of enterprises financed with women participation, dollar amount disbursed to women entrepreneurs, percentage maximum of portfolio at risk, and percentage maximum recovery rate of dues. In fact, EDF exceeded its targets (by amounts ranging from 37 percent to 95 percent) for the first four of those indicators, namely indicators related to sub-loans to beneficiaries. (The other two FSP indicators related to the quality of the overall loan portfolio: portfolio-at-risk not exceeding 10 percent and 90 percent maximum recovery rate of dues.) Dollar amount of credit disbursement to POs was the only FSP indicator for which the end-of-project target was not fully achieved. Of the $4,492,759 targeted, EDF disbursed $4,379,031 (97.5 percent). All enlisted POs extended loans, on average, in the range of $500 - $1,666. According to external audit reports, however, POs disbursed 61 percent of loans to NLCL for amounts that exceeded the upper limit of $1,666. To achieve the average loan size of $951, NLCL most likely extended a significant number of loans close or perhaps even below the lower limit of $500. Significantly, the EDF loan portfolio was very heavily skewed toward the NLCL. This PO represented 66 percent of total loan value and 61 percent of the total number of loans disbursed. Regarding usage of loans, beneficiaries used 40 percent of the loans for commerce and trading purposes, 10 percent for the handicraft and cottage industries, and 7 percent for light manufacturing. EDF identifies a third of the total loans issued through June 2007 as “other,” and does not provide a definition of this. Therefore, the team is unable to ascertain whether these loans meet the project’s objective to “support entrepreneurship and local micro-businesses enabling them to generate income/assets.” Technical Assistance Partner Organizations As noted earlier, EDF aimed to “institutionally capacitate selected organizations to undertake second￾generation innovations in micro-credit/enterprise development.” Partner organizations play a critical and key role in the successful delivery of typical apex loans. The project design called for EDF to initiate loan activity with three POs in the first year, an additional three in the second year, and one more in the third year. This totaled seven during the project. However, EDF worked with only two POs in the first year and with four more in the second and third years. And, of these six, EDF did not continue with three POs. EDF failed to reach its PO target for various reasons. PPAF bureaucratic procedures slowed down a more dynamic, faster project kick off. Also, POs needed to meet strict criteria. The PPAF criteria required that eligible POs were previous clients of PPAF. Additional criteria included other risk-mitigating factors, such as length of time in business (minimum of two years), good governance, adequate internal control procedures, and commitment to establish a separate administrative unit to manage the EDF loan product. Unfortunately, these strict criteria did not prevent serious problems in the project. For example, the Punjab Rural Support Program, a well-known and respected credit-granting NGO, failed to deliver on performance targets. PPAF terminated the contract after only nine months of lending. PPAF suspended RCDS in June 2006 for not meeting contract conditions, but reinstated it a year later. NLCL was the most regrettable choice of PO and went bankrupt due to mismanagement and fraud. Unfortunately, NLCL distributed the most loans in the project, which violated principles of business risk analysis. PPAF channeled two thirds of all loans through this organization. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT xi FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT xii PO and EDF Capacity Building The project’s non-FSP component activities also aimed to increase the capacity of those POs to undertake innovative micro-credit/enterprise development. POs needed direct technical assistance because they had no previous experience in extending loans in the range of the pre-determined “missing middle”($500 to $1,666). With the exception of NLCL, the POs specialized in loan sizes under $250. Non-FSP activities included training of EDF staff. From the beginning, PPAF supported EDF with technical staff. In 2005, two top managers of EDF traveled to the United Kingdom to attend specialized training in micro lending and investment risk analysis. Unfortunately, they left the organization soon thereafter. At the time of this evaluation, the EDF appeared to be understaffed. Although PPAF gave support, EDF did not have the full-time staff of eight people outlined by the project design.3 EDF had a target of 18 for trained PO staff in program management and credit-skill development. By the end of the project, almost seven times the targeted number of staff (125) obtained training in program management and credit-skill development. In one sense, this remarkable achievement over target clearly indicated EDF effectiveness; however, based on the evaluation team’s experience, this may suggest a weak indicator of increased capacity. The high number of trained staff does not reflect the quality, relevance or usefulness of the training and its impact on organizational performance. In the CWCD site visit, only 50% (6 in number) of the trained management staff of CWCD praised the training modules on financial management and monitoring and evaluation as relevant and useful. Micro-Entrepreneur Skill Building The non-FSP component aimed to raise the skills of micro entrepreneurs. This training included several activities, such as training in enterprise skill development, marketing and information linkages, and systems development services. The project targeted 800 micro entrepreneurs. As of June 2007, 654 (82% of target) received skill training. The budget allocation for this sub-component was $586,207, equivalent to 66% of the total budget for non-FSP activities. However, EDF used only 6% ($35,276) of the targeted budget for entrepreneurial skill development training. This is a great discrepancy between the substantial number of trained micro entrepreneurs and the low amount of resources used by EDF for training. EDF and the POs had differences in requirements for the venue, training content and the trainers themselves. EDF envisioned consultants would conduct off-site training, but the POs did not consider such training to be meaningful for their borrowers. Their clientele were usually unable to leave their businesses for protracted periods of time, as they would forego incomes and earnings during training periods. This was particularly true for micro entrepreneurs in trade and manufacturing. Moreover, POs felt borrowers could benefit more from practical training, rather than conceptual or classroom learning. Conclusions First, for the FSP component, EDF met or significantly exceeded its targets for six of its seven objectives to provide sub-loans to relatively poor entrepreneurs. However, the evaluation team is unable to determine how beneficiaries used a third of the loans. Second, EDF failed to reach its target of seven new POs to undertake enterprise development activities (EDF worked with six, and only three of those were active by the end of the project). PPAF relied on “known quantities” instead of carefully examining the potentially positive opportunities offered by other retailers that did not participate in the program. 3 Depriving resources to a project is not unusual in organizations that draw donor support from multiple sources, for they usually contend with competing, if not conflicting objectives that lead to unproductive staff shuffling and, often, taxing human resources to the limit. Third, for the non-FSP component, the evaluation team concludes that the project was hugely effective in training PO staff, but fell short in meeting its objective of training borrowers. Efficiency Because the evaluation team did not estimate monetary benefits associated with the project, it is not possible to calculate the usual economic measures of efficiency, such as net present value, benefit cost ratio, or internal rate of return. Project expenditure data provide some measures of spending efficiency. See the following summary of spending in several categories as percentages of project costs, as of September 2007: • Labor accounts for 30%; • Loan capital accounts for 54.2%; • Other Direct Costs (ODCs) account for 12.3%; and • Outputs (training, capacity building, research, systems development) account for 3.5%. Excluding loan capital, labor accounted for 65% of expenditures and ODCs accounted for 27% and outputs for only 8%. Conclusions There are a number of possible measures of project efficiency that compare costs to some measure of output. In this case, however, no monetary measures of benefits exist so it was not possible to calculate the usual efficiency measures (i.e., net present value, benefit cost ratio, internal rate of return). Impact USAID did not ask the evaluation team to estimate the monetary impacts of the EDF project. However, the evaluation team conducted group interviews of 20 CWCD borrowers. Of these, 10 were women, and 11 were RCDS beneficiaries. These interviews provided anecdotal evidence of EDF loan impact on businesses and households. Interviews indicate that a large majority of the loans received by interviewees financed both working capital and capital equipment investment. Most interviewed borrowers also revealed that loans financed the hiring of non-family workers. All women borrowers interviewed indicated that loans increased their sense of self-empowerment. Conclusions Estimating a monetary measure of micro credit impacts is notoriously difficult. This case is no exception. Group interviews suggested that EDF lending did impact the welfare of beneficiaries, but produced no firm evidence that it affected household incomes or other monetary measures. Sustainability EDF extended loans to its POs at an interest rate of 8 percent, which was clearly below the prevailing market rate. The POs, in turn, provided credit to their customers at 17 percent, which was not high enough to cover the typical costs for Pakistan’s nascent microfinance industry. Consequently, the project granted interest rate subsidies both to loan retailers and borrowers. This scheme does not encourage micro loan providers with incentives to introduce institutional reforms. These reforms lead to the establishment of linkages with commercial banks. In addition, these reforms help transform micro loan providers into FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT xiii deposit-taking entities. Without institutional reforms, donor dependency occurs and providers cannot achieve long-term operational and financial independence and sustainability. In the short and medium term, PPAF is well supported by donors and the GOP, which allows EDF to continue. Conclusions USAID did not design or implement EDF in ways that focused on the sustainability of partner organizations after project completion. PPAF’s contractual terms between itself and its POs did little to enhance the latter’s long-term self-sufficiency. Still, short-term sustainability seems likely because EDF is sponsored by an apex organization with access to substantial financial resources. Replication PPAF is already replicating EDF activities on a small scale, such as the World Bank-funded PPAF II that features a small EDF window and IFAD’s MIOP with a private sector lending window. For greater effectiveness and sustainability, PPAF needs to wholesale long-term loans at commercial rates of interest. This sends unequivocal signals to potential POs that financial self-sufficiency is necessary. Along the same lines, credit-granting NGOs must develop specialized credit-only arms, which should be an essential condition for accessing assistance. Finally, successful replication must address staffing issues. Certain staff members need cross-cutting developmental and finance skills. Moreover, full-time staff must be technically competent from the beginning. At the beginning of the project, PPAF assigned a full-time chief executive and a full-time manager to run the EDF project. As noted earlier, they left shortly after attending training programs in the United Kingdom. Contrary to the implementation plan, eight new technical staff were never hired, and any project replication must hire additional staff. Conclusions EDF activities offer much for Pakistan, especially for the “missing middle” group of entrepreneurs. In fact, PPAF is currently implementing some on a small scale. To replicate in other areas of the country, however, improved management procedures and practices are necessary. Gender EDF nearly doubled the end-of-project targets for the number and value of loans extended to women borrowers. Despite this, loans to women still accounted for only 28 percent of all loans received. Given the need in Pakistan to increase outreach to women, these targets seem comparatively modest. During the period of 2005-06, the average loan size for women was $794, equivalent to 68 percent of that for men. NLCL loans caused some of this disparity. These loans were relatively large and 70 percent of them went to men. After the collapse of this entity, average loan size for men dropped to $952, and the average loan size of women was equivalent to 82 percent of that for men. LOAN DISBURSEMENTS BY GENDER AS OF JUNE 2007 Loan/Gender Male Female No. of Loans 3548 (71.6% of all loans) 1409 (28.4% of all loans) FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT xiv Value of Loans 3.3 million 1.08 million Average Loan 952 780 Among the POs, only CSC lent exclusively to women. Unfortunately, this totaled only 54 loans. In Mandi Faizabad, RCDS made only 48 loans to women compared to 649 loans to men during a two and a half year period. CWCD extended a significantly larger number of loans to women than men in its earlier years, but at the end of the project, the gender division changed to 53 percent for men and 47 percent for women. Conclusions Women received considerably fewer loans than men, even though the number of loans to women nearly doubled the projected targets. The evaluation team found no evidence that project activities targeted women in a systematic manner. The project did enlist a woman-oriented partner organization (CWCD), but channeled most resources to an organization (NLCL) that made no effort to increase lending to women. EDF did not urge its partner organizations to deliver more loans to women, and did not make it a condition of providing technical assistance in the non-FSP component. Reporting The Planning and Implementation Plans, as well as quarterly reports, described the goals and activities of the project. EDF did track actual performance against actual targets. However, the evaluation team found that EDF adjusted targets several times during the project with no explanation. EDF did not document loan repayments from partner organizations, so the team cannot discern the pattern of loan arrears of partner organizations. Furthermore, the team cannot determine the adequacy of risk assessment methodologies. Lastly, the team found no evidence that EDF applied branding guidelines. Communications and Outreach EDF prepared well-written case studies of impact on loan end-users and disseminated these case studies among donors and GOP agencies. USAID was certainly a visible presence in the areas served by the EDF project. For example, borrowers in Mandi Faizabad near Sheikhupura knew that USAID was the source of their funding. Overall, however, the evaluation team did not find evidence of EDF making substantive efforts to participate in high-profile events and seminars. Coordination EDF coordinated well with the GOP and the World Bank. PPAF implemented its poverty alleviation activities according to GOP goals as specifically stated by the SBP. PPAF also provided technical inputs that the World Bank used to prepare the Poverty Reduction Strategy Paper. RECOMMENDATIONS To the extent that it remains engaged in finance projects, USAID should build on the credit assessment of PPAF recently conducted by Shorebank International, and consider the following: FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT xv FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT xvi (a) Enhance PPAF’s capabilities in typical wholesaling operations. This program of technical assistance should include, at minimum: • Development of strategic, business and operational plans • Market research techniques • Risk-assessment tools • Client selection strategy • Monitoring systems that measure and forecast risks • Communication strategy with relevant stakeholders (b) Restructure the finance and development components of PPAF, so as to define clear objectives and roles for each. For example, under finance, more strict selection criteria in selecting the retailers, risk assessment, stronger monitoring of the retailers’ portfolio and operations, and more defined key financial performance indicators. (c) Restructure the development role of PPAF. This includes a more robust technical assistance program to the intermediaries, or POs. This kind of assistance includes improved management capacity, loan officer training, better selection of end users, and more focused and effective outreach of business development services. (d) Replicate the EDF model by: (1) Setting up wholesaling of long-term loans at commercial rates; (2) Developing credit-only arms at credit-granting NGOs, as a sine-qua-non condition for accessing assistance; and (3) Continuing analysis of POs for financial self-sufficiency progress. LESSONS LEARNED When beneficiaries themselves execute projects, USAID must conduct regular meetings to monitor progress, track results, and determine necessary program adjustments for maximum impact. USAID must set up ways to control finances of the project that ensures a close monitoring of decisions and expenditure patterns. ANNEX 1: FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 1 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 2 I. INTRODUCTION In August 2007, the United States Agency for International Development (USAID) commissioned Management Systems International (MSI) to conduct the evaluation of eight projects within the purview of its Pakistan Economic Growth activity. Three of these projects seek to increase access to financial services to the benefit of households and micro and small businesses in rural and urban areas across Pakistan (Widening Harmonized Access to Microfinance (WHAM), Enterprise Development Facility (EDF) and Developing Non-Bankable Territories for Financial Services). This report addresses the final evaluation of EDF under conditions specified in the Cooperative Agreement No 391-A-00-03-01010-00 of September 30, 2003. Initially valid for a period of three years, the Agreement was amended on September 26, 2006 for a period of one additional year. Its contractual arrangement is similar to the Developing Non-Bankable Territories for Financial Services project, which is implemented by Khushhali Bank (KB). Both were implemented directly by the beneficiary agencies, unlike WHAM, which an independent contractor implemented. EDF is an initiative launched by the Pakistan Poverty Alleviation Fund (PPAF) to address the economic development constraints of the relatively less poor, otherwise known as “potentially poor in Pakistan.”4 Therefore, the evaluation team viewed EDF within PPAF goals and activities. Sponsored by the Government of Pakistan (GOP), PPAF is an apex organization set up in 2000 with a seed capital of $13.3 million donated by two multilateral donors, the Asian Development Bank (ADB) and the World Bank. PPAF uses these donor funds to give partner organizations (POs) for implementing focused development interventions. Its stated mission is to “help the poor [and potentially poor] gain access to resources [for] productive self-employment, and [conduct] activities [leading to] income generation [and] poverty alleviation5 .” The POs, in essence, are non-government organizations (NGOs) and civil society organizations (CSOs). These organizations have a geographic and cultural proximity to the poor and also have experience at the grass root level. This gives them an opportunity to be valuable agents of community empowerment as well as socio, economic, and cultural change. The partnership between PPAF and the NGOs mirrors the multiplicity of roles that the latter have undertaken in Pakistan. PPAF’s organizational structure is not based on the undertaking of exclusive, specialized apex operations. Consequently, PPAF includes a unit devoted to reconstruction and rehabilitation (due to the devastating earthquake of 2005), another to health and education, and another to community infrastructure. In all, PPAF extends lines of credit for the expansion of poverty-targeted micro credit programs. PPAF provides grants on a cost-sharing basis for the development of community physical infrastructure and the quality improvement of health and education services in poor communities. Critically, PPAF also extends grants directly to its partner organizations in order to strengthen their human and institutional capacities. At the time of this evaluation, PPAF worked with 68 partner organizations, with 38 of them receiving loans. As an apex organization, PPAF acts as a wholesaler and intermediary of funds. With its large endowment fund, PPAF places significant volumes of funds in interest-bearing accounts. Also, PPAF obtains financial resources at a zero or 0.75% interest rate, which allows for short-term sustainability. By funding NGOs, PPAF plays a significant role in the expansion of the micro credit market of Pakistan. Following a risk assessment, PPAF applies a set of criteria to select NGO loan retailers. This includes the candidate Ngo’s demonstrated track record on loan quality portfolio, outreach, social mobilization and its focus on the poor and disadvantaged. 4 The poor, defined in terms of food intake, are those people who consume less than 2250 calories a day. In the case of Pakistan there is a high concentration of people clustered around the poverty line, so that if the line is moved marginally down, the population of the poor can increase by 10% to 20% of the total population. People clustered on the upper margins of the poverty line may be said to constitute the potentially poor. 5 Project Work Plan Year 1. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 3 Box 1: NGOs in Pakistan NGO activity in Pakistan dates back to 1947 with engagements in the fields of health and education. More recently, in the early 1980s, the Aga Khan Rural Support Program and the Orangi Pilot Project began developmental work in the Northern Areas of Pakistan and in Karachi, respectively. These NGOs implemented programs that led to increasing the capacity of their clienteles to construct and maintain infrastructure and sanitation networks through their labor contributions. The development interventions of these ‘early starters’ were characterized by their good performance and high management standards. Even more recent is the proliferation of NGOs focused in poverty alleviation, or engaged in diverse fields, such as human rights and women issues. One impetus for the extraordinarily rapid growth of NGOs was the decrease in Government spending on the social sectors in the 1980s, a result of the World Bank-sponsored structural adjustment reforms. Another catalyst has been the willingness of donors -- USAID, UK’s DFID, Japan’s JICA, Germany’s GTZ, -- to extend various types of financial support to NGOs. In fact, a diverse number of NGOs, both large and small, of varying caliber have benefited from donor assistance. Unfortunately, their symbiotic relationship may also have been instrumental in breeding a culture of dependency. The extension of micro credit services has become one of the distinguishing characteristics of NGOs. To illustrate, the main mandate of Kashf Foundation and the National Rural Support Program (NRSP) has been, from their very inception, the provision of micro-credit lending for poverty alleviation. As of this writing, the lending arms of Kashf Foundation and NRSP are in the process of preparing the groundwork for the transformation of their micro-finance operations into specialized microfinance banks. More recently, a number of newer credit-granting NGOs have emerged, some of which cater their services exclusively to low-income women, such as Asasah, or target mixed clienteles in smaller towns and their peripheries, such as the Narowal Rural Development Society, and the Rural Community Development Society (RCDS). Other credit-granting NGOs, such as the Center for Women Cooperative Development (CWCD), Community Support Concern, (CSC) and the Sindh Agricultural and Forestry Workers Cooperative Organization (SAFWCO) serve the poorer areas in major urban centers. Source: Interviews conducted with key informants. II. THE DEVELOPMENT PROBLEM A. Problem Statement According to the State Bank of Pakistan (SBP)6 , the total number of clients who obtain loans from formal sources of finance barely surpasses 5 million, and 22 percent of these clients are micro and small borrowers. Credit-granting NGOs, in combination with KB and a handful of specialized microfinance banks, extend credit to approximately one million micro borrowers. This is equivalent to approximately just 5% of potential micro loan demand.7 In Pakistan, the vast majority of micro borrowers are offered loans ranging between $160 and $250, which are repaid in frequent, small installments. While these loans may have a positive effect for regularizing and protecting incomes of borrowers, the impact of loans of this size to enhance productive capacity is indeed limited. Therefore, potentially dynamic micro entrepreneurs who wish to develop their business are hampered by lack of access to adequate finance for obtaining information about domestic or international market demand for their products. More importantly, these loans are too small for entrepreneurs to introduce improved techniques, materials and designs that would enable to supply their products at higher quality and better prices. With access to loans of larger size at reasonable cost, entrepreneurs can increase their productive capacities and, ultimately, their incomes and sustainability. 6 Source: Interview conducted with the Director of the Microfinance Department 7 Shorebank International has estimated that the market size of micro credit comprises 7 million households and no less than 10 million individuals. Within poverty bands, these are classified as transitory vulnerable and transitory poor. B. USAID Intervention According to the terms of the Cooperative Agreement, USAID agreed to provide $6.32 million to PPAF over a period of three years. The development objectives of the project are consistent with the overall objectives of USAID-Pakistan's Economic Growth Strategy, and consist of: • Focusing on and targeting the ‘missing middle' of the potentially poor. The missing middle is defined as those prospective micro borrowers who are neither covered through the conventional poverty programs of civil society organizations, nor by formal sector financial institutions. • Securing sustainable livelihoods for the potentially poor through financial support to entrepreneurship activities. Institutionally capacitating selected organizations to undertake innovations in micro credit/enterprise development, and also providing market linkages, information and improvement in skills to entrepreneurs straddling the aforementioned missing middle. Operationally, PPAF divided EDF into two main components: (a) provision of financial services to the potentially poor (FSP); and (b) extension of non-financial services to the poor (non-FSP). The first component consists of $4.493 million, which EDF disbursed to the actual beneficiaries through its partner organizations. Management of the program cost $886,438. The POs provided individual loans between $500-1,500 to entrepreneurs who had already successfully serviced conventional micro credit loans (loans under $500 from previous PPAF micro credit programs). These larger loans enabled the entrepreneurs to expand their business, thus generating larger streams of revenue and the capacity to buy more assets. FIGURE 1: RESULTS FRAMEWORK FOR EDF Non-FSP, on the other hand, was allocated $939,983 or 15% of the total USAID contribution. This included technical support to build up the capacity of the beneficiaries, which generated more economic value added into their micro-enterprises. Typically, this technical support took the form of training programs to improve entrepreneurial skills, improve marketing techniques, and give access to information networks. Non-FSP activities also included increasing the capacity of the POs for managing and delivering the EDF credit program. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 4 TABLE 1: ESTIMATED PROGRAM BUDGET OF THE EDF Budget Category USAID Contribution PPAF Share Total EDF Program A. Financing for Micro-Enterprise Development $ 4,493,579 B. Program Management $ 886,438 C. Capacity Building $ 939,983 Total $ 6,320,000 $ 778,621 $ 7,098,621 Source: USAID and PPAF, September 30, 2003: Award of Cooperative Agreement No. 391-A-00-01010-00; III. EVALUATION METHODOLOGY The evaluation team consisted of MSI staff member Jorge L. Daly, Team Leader, assisted by Ms. Memoona Khan, Micro and Small Enterprise Development Specialist who collaborated in conducting interviews with stakeholders and in conducting data collection and analysis. Douglas Krieger led the tasks centered in efficiency and impact analyses, and Maliha Hussein, a microfinance specialist, reviewed the draft for content, quality, and contextual accuracy. The evaluation tasks began when the Team Leader arrived in Islamabad on October 12th 2007. Because his arrival coincided with the Eid Holidays, effective kick-off of activities started on October 17th. Ms. Khan joined the team on November 1st 2007. The Team Leader returned to Washington DC on November 20th after conducting a partial data collection in Islamabad, Karachi and Lahore, and assisting in the survey design for impact analysis. Efficiency and impact analyses were conducted from February 18th, 2008 through March 16, 2008. On November 19th, right before the return of the Team Leader to Washington DC, a presentation on preliminary findings was made to USAID. A. Evaluation Purpose and Questions The purpose of this evaluation is to conduct the final assessment of the EDF Project. In doing so, the evaluation will determine and inform USAID the impact and sustainability of the project’s activities and results. The evaluation provides answers to both general and specific questions. See the following general questions: • Relevance: How well was the EDF Project focused on the needs of the beneficiaries? • Effectiveness: Has the EDF Project accomplished its objectives? • Efficiency: How efficient has the EDF Project been in utilizing resources to achieve intended results? • Impact: To what extent has the EDF Project benefited the people of Pakistan? • Sustainability: Are activities and results likely to be sustained after the EDF Project is completed? • Replication: To what extent can the activities and results of the EDF Project be replicated? FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 5 • Gender: To what extent has the EDF Project benefited women? • Reporting: Has the EDF Project reported in a timely and useful manner? • Public relations: How effective has the EDF Project been in getting its story out? • Coordination: How effectively has the EDF Project been in coordinating with other parties? B. Evaluation Design The EDF evaluation is designed to be independent, objective, and findings-based. The evaluation questions are answered based on a general approach that progresses from findings to conclusions to recommendations. The process starts with findings which are observed and collected facts. These are “produced” during the data collection phase of the evaluation. Using analysis, interpretation and judgment, conclusions are drawn from the findings. In turn recommendations are based on the conclusions. Recommendations aim to identify practical actions for the consideration of project managers in the event USAID decides to continue to focus on finance projects. This approach gives report reviewers a clear view of the evidence that supports each of the study’s conclusions and recommendations. The evaluation methodology can be best illustrated and summarized by the “Getting to Answers Matrix” (GTA). This key evaluation design tool provides a format that allows the key evaluation questions to drive the identification of relevant data sources and data collection methods. This process – the completion of the GTA matrix – creates the point of departure for the collection of data necessary to the key evaluation questions. Table 2 provides an illustration of how a general GTA works, applicable to the first two of the overall questions – relevance and effectiveness. A specific GTA matrix for the EDF Project is included in Annex 4. TABLE 2: GETTING TO THE ANSWERS FOR OVERALL EVALUATION QUESTIONS Evaluation Questions Relevance: How well was the project focused on the needs of the beneficiaries? Effectiveness: Has the project accomplished its objectives? Type of Answer or Evidence Needed Comparative of targets related to beneficiaries with results, Comparative of what was targeted versus felt needs, Strategic or cause and effect Quantitative comparison of targets to baseline data, if available Strategic or cause and effect Method of Data Collection Review of project documents, Interviews, Direct observations, Surveys, Effectiveness and impact findings Review of project documents, interviews, focus groups, direct observations, surveys Data Source Project data, USAID and project personnel, Key informants, Beneficiaries, Evaluation findings USAID and project personnel, partners, participants, beneficiaries, observers, outside groups Selection Criteria Knowledgeable Persons, Randomly selected beneficiaries, stratified as appropriate for the project in PEGED surveys Knowledgeable persons, random selection of beneficiaries, stratified as appropriate Data Analysis Methods Comparisons Strategic analysis Comparison, quantified number of beneficiaries (disaggregated) and quantified benefits as possible FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 6 C. Data Collection Sources and Methods The evaluation of the EDF Project relied on the following data collection methods. • Review of relevant documents including (a) the cooperative agreement and its amendment; (b) the project proposal presented to USAID, project implementation plans and ten quarterly reports; (c) important documentation pertaining to the PPAF and its partner organizations such as their Annual Reports, their presentations on their EDF lending component, and their system procedures; and (d) a major audit report done on the implementation of the Co-operative Agreement from October 1 to June 30, 2005,. The review was conducted when the Team Leader arrived in Islamabad on October 12, 2007. • Interviews with key informants. These included (a) (a) fiver representatives of the donor community; (b) one representative of a microfinance banks and one of a Rural Support Program; (c) an official of the State Bank of Pakistan; and (d) a representative of a civil society organization – the Pakistan Microfinance Network. A list of the key informants is included in Annex 3. • PPAF/EDF: Interviews with (a) Chief Strategic Planning Officer, General Manager, Management Executive and several high ranking officials; (b) principals and staff of three major partner organizations that are recipient of credit from EDF: (a) the National Leasing Corporation Ltd (NLCL), located in Karachi; (b) the Lahore-based Center for Women and Community Development (CWCD); and (c) the Rural Community Development Services (RCDS), which is located in Mandi Faizabad. The team conducted these interviews during the period October 17 – November 28 2007. A list of these individuals is also included in Annex 3. • Group interviews with 20 selected borrowers of CWCD and 11 of RCDS. D. Data Limitations In the process of collecting data, the evaluation encountered the following limitations: • The task coincided during a period of high political tension in Pakistan. The data collection stage was dominated by important events such as: a) the attempt on the life of opposition leader Benazir Bhutto on October 19th and her subsequent assassination on December 27th 2007; b) the imposition of a state of emergency by President Musharraf on November 1st 2007; and c) the postponement of national elections. These events created unintended obstacles, including sudden cancellation of flights and appointments with key informants. • Given the above, some interviews with key informants were rescheduled and eventually conducted in less allotted time. Others could not be conducted at all, notwithstanding the efforts of the team. The most important key appointments that were missed were with the Ministry of Finance and the World Bank. • The analysis of project impact was constrained by the lack of a baseline data constructed prior or shortly after the start of the project. Getting reliable measures of before-project incomes was therefore not possible. Nor was it possible to identify a suitable control group. As a consequence, it was not possible to estimate reliable measures of monetary impact. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 7 IV. FINDINGS AND CONCLUSIONS ON THE OVER ARCHING QUESTIONS A. Relevance The purpose of this section is to determine if the design and implementation of the EDF Project was focused on the needs of the beneficiaries. The beneficiary target group for the EDF project was micro-entrepreneurs who had a loan-demand profile in the range of $500 to $1666. These loans were larger than the kinds of loans available through conventional micro-credit programs. On the other hand, these loans were smaller than those available from formal financial institutions. Micro-credit institutions offered very small microloans of $160-250, but these do not help a micro-entrepreneur develop business potential, even if they received these loans on a repetitive basis. Despite the recent entry of specialized microfinance banks in Pakistan, loan supply from formal financial institutions in the $500-1,666 range is still very thin. It is indeed very rare for commercial banks to offer large volumes of loans below $10,000. Market conditions for successful downscaling do not exist because of high transactions costs, the perception that micro and small entrepreneurs are high risks and, critically, existing profitable opportunities elsewhere. Findings Through EDF, PPAF proposed a vehicle addressing the unmet needs of the micro-entrepreneurs described above. By providing loan products of a larger size (i.e., $500-1,500) to that target group, EDF enabled PPAF’s relatively less poor clients—i.e., “the missing middle”— to marginally improve upon their subsistence levels of income, and relieve them of the financial constraints. It was assumed by both EDF and PPAF that after two or three loan cycles, the more dynamic entrepreneurial borrowers would require additional funding and by this time, attain an increased capacity to bear debt. Given its mandate and functional linkages to NGOs, PPAF appeared to be a feasible, effective and suitable organization to implement this initiative. In 2003, the PPAF had a pool of 28 partner organizations (POs), including a few that were eligible to conduct the EDF program. PPAF commissioned a study to determine the institutional needs of its POs and the credit demand of their clienteles. USAID based the agreement on the findings of this study, which meant the program design appeared to include the needs of the POs and their borrowers. PPAF and USAID were the only stakeholders in the EDF project doing consultations during the design phase. At the same time, however, the team found no evidence that USAID was sufficiently engaged so as to anticipate problems that subsequently surfaced during project execution. For example, USAID was not aware of the relationship between PPAF and the POs. Managers of the partner organizations who were interviewed revealed that PPAF alone set the terms and conditions of the EDF loans. Therefore, USAID did not have a correct identification of the institutional capacities of the POs, nor did USAID have a way to receive partner’s feedback that would strengthen the project. Additionally, some enlisted POs obtained grants from PPAF for activities in the areas of infrastructure, health and education. For example, RCDS obtained health grants from PPAF for two clinics for its program Mother and Child Care. Also, RCDS as well as CWCD obtained micro-credit loans from PPAF through projects and initiatives not associated with EDF. Although PPAF sets up a roundtable discussion with partner organizations for feedback, comments and recommendations, POs most likely felt obligated to accept existing circumstances in order to continue funding for other programs. Consequently, the EDF project did not receive feedback for adjustments to more effectively address the needs of both POs and the end-user beneficiaries. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 8 Conclusions First, by providing loans in the range of $500-1500 to micro-entrepreneurs who needed loans larger than those available through micro-credit institutions and smaller than those available from formal financial institutions, the EDF effectively focused on the real needs of a sizeable group of beneficiaries among the Pakistani population. Second, EDF channeled loans through partner organizations that also benefit from non-financial assistance by PPAF. This may have inhibited initiatives of partner organizations to suggest program adjustments that could have better met the needs of the end-user beneficiaries. B. Effectiveness The purpose of this section is to determine if the EDF Project accomplished its objectives. The Performance Monitoring Plan (PMP) for the project identified five major results and 16 performance indicators. The indicators had end-of-project targets, some of which were revised upon extending the project by a fourth year. The original goals and expected results, PMP targeted indicators, and actual end￾of-project results are detailed in Table 3. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 9 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 10 TABLE 3: PROJECT GOALS, EXPECTED RESULTS, INDICATORS, END-OF-PROJECT ACTUAL RESULTS AND PERCENTAGE OF TARGET ACHIEVED EDF Goals as Stated in the Cooperative Agreement Original Expected Results as Stated in the Coop. Agreement End-of-Project Targeted Indicators in the PMP End-of-Project Actual Data (and Percentage of Target Achieved) 3,610 sub-loans to entrepreneurs 4,957 (137.3%) 200 sub-loans to repeat borrowers 334 (167.0%) 722 enterprises financed with women participation 1,409 (195.2%) Extend at least 2,295 enterprise sub-loans of greater than Rs 30,000 (US $517) each $898,489 disbursed to women entrepreneurs $1,302,205 (144.9%) Maintain the quality of the 10 % maximum of portfolio at risk <10 percent overall loan portfolio (Portfolio at Risk not exceeding 10 percent) 90 % maximum recovery rate of dues >90 percent Achieve an average ex￾post incremental increase of 10 % in the incomes/ assets of borrowing households 10 % increase in income and/or assets of borrowers No Data Available8 80 % of businesses expanded No Data Available (see f/n 5 below) Secure sustainable livelihoods for the poor through support to entrepreneurship and local micro￾businesses enabling them to generate income/assets 800 borrowers trained in Enterprise Skill Development & managerial skills 654 (81.8%) Institutionally 7 new POs inducted 6 (85.7%) capacitate selected org’ns to undertake second￾generation innovations in micro-credit/ enterprise dev’t Enable at least 7 partner organizations (POs) to undertake enterprise development activities 18 PPAF Partners’ staff trained in program management & skill dev’t 125 (694.4%) $4,492,759 credit disbursement to POs $4,379,031 (97.5%) $586,207 disbursements for Entrepreneurs Skill Dev’t & managerial skills $35,276 (6.0%) $172,414 disbursements for POs’ Staff Development Events $15,284 (8.9%) $12,931 disbursement for Market/Product/Other Research $32,986 (25.5%) $887,931 total disbursement for Non￾Financial Component $83,546 (9.4%) Both Goals Disburse Rs 295 million (US $5.0 million) for credit and capacity building (Total for credit & capacity building—i.e., FSP and Non-FSP Components, which was targeted in the original Cooperative Agreement at $5 million) $4,462,577 (89.3% of target in column 2) Source: USAID, PMP, 2006, and PPAF’s Quarterly Progress Reports 14 and 15 8 Data was not included in the PMP within the Quarterly Reports. A baseline conducted by Gallup Poll in 2007 did not report baseline income and changes in this income for individual respondents over time, or changes in income of who received loans, against those who did not (the control group). In the remainder of this section, the evaluation discusses findings and adds more detail to the results presented in the table above. The discussion is organized by the two project components, FSP and non￾FSP. B.1 FSP Component Loans The data presented in Table 2 above show that the EDF met or far exceeded its targets for six of the seven FSP-related performance indicators: number of sub-loans to entrepreneurs, number of sub-loans to repeat borrowers, number of enterprises financed with women participation, dollar amount disbursed to women entrepreneurs, percentage maximum of portfolio at risk, and percentage maximum recovery rate of dues. In fact, EDF exceeded its targets (by amounts ranging from 37 percent to 95 percent) for the first four of those indicators, namely indicators related to sub-loans to beneficiaries. (The other two FSP indicators for which targets were met were those dealing with the quality of the overall loan portfolio: portfolio-at-risk not exceeding 10 percent and 90 percent maximum recovery rate of dues.) Dollar amount of credit disbursement to POs was the only FSP indicator for which the end-of-project target was not fully achieved. Of the $4,492,759 targeted, EDF disbursed $4,379,031 (97.5 percent). Table 4 below indicates the value of accumulated loans disbursed to each of PPAF’s six POs. All six POs extended loans, on average, within the range of the missing middle. According to external audit reports, however, 61 percent of loans disbursed to NLCL retailed for amounts that exceeded the upper limit of $1,500. To achieve an average loan size of $951, NLCL most likely extended a significant number of loans close or perhaps even below the lower limit of $500. With data that allows only the computation of average loan sizes, the evaluation team was unable to assess the extent, if any, to which other POs may have provided loans below or above the targeted range of $500 to $1,500 for project beneficiaries. TABLE 4: LOANS DISBURSEMENTS TO POS AS OF JUNE 2007 Partner Organization No. of Loans (and Percentage of Total) Total Value of Loans (and Percentage of Total) Average Loan Size CWCD 904 (18.2%) $738,000 (16.9%) $816 RCDS 697 (14.1%) $499,000 (11.4%) $716 PRSP 196 (4.0%) $137,000 (3.1%) $699 NLCL 3,044 (61.4%) $2,894,000 (66.1%) $951 SAFWCO 62 (1.3%) $44,300 (1.0%) $714 CSC 54 (1.1%) $68,000 (1.6%) $1,259 Total 4,957 $4,379,000 $883 As seen from the table above, the EDF loan portfolio was very heavily skewed toward the NLCL. This PO represented 66 percent of total loan value and 61 percent of the total number of loans disbursed. The next two major partners in the program were CWCD and RCDS, which comprised 17 percent and 11 percent, respectively, of total dollar value of loans. PRSP was dropped in June 2005 and operations with NLCL were terminated early in 2007 with the collapse of this institution. The relatively small numbers for SAFWCO and CSC, which together make up only 2.6 percent of total dollars lent, were late additions to the project, and were extended lines of credit only as of October 2006. Use of Loans by Borrowers Table 5 shows that beneficiaries used approximately 40 percent of the loans for commerce and trading purposes, 10 percent for handicraft and cottage industries, and 7 percent for light manufacturing. Significantly, the “other” uses of loans comprise fully a third of the total loans issued through June 2007. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 11 The PPAF quarterly reports do not explain this large percentage, which is troubling given the rather comprehensive list of specific uses in the rest of the table. Without an explanation, the evaluation team cannot ascertain whether so many “other” loans were used to meet the project’s objective to “support entrepreneurship and local micro-businesses enabling them to generate income/assets.” TABLE 5: EDF LOANING BY SECTOR FROM JANUARY 2004 TO JUNE 2007 Sectors Total Loans Rs. In m US$ m Commerce & Trading 2,004 (40%) 122.451 (45%) 2.04 Handicraft/Cottage 492 (10%) 32.68 (12%) 0.54 Agriculture/Cropping 215 (4%) 13.45 (5%) 0.22 Livestock/Poultry/Fisheries 166 (3%) 8.80 (3%) 0.15 Manufacturing/Light engineering 309 (7%) 14.57 (5%) 0.24 Food Processing 74 (1%) 3.36 (1%) 0.06 Others 1,697 (34%) 73.39 (27%) 1.22 Total 4,957 268.69 4.48 Loans to Women vs. Men According to PPAF Quarterly Progress Reports, loan disbursements to women entrepreneurs amounted to $1.302 million, which surpassed the end-of-project targets by 45 percent. The number of financed women-managed enterprises was 1,409, which is the project target of 722 nearly doubled. Twenty-eight percent of the total 4,957 loans issued through the EDF went to women, while 72 percent went to men. In all but one PO, women received fewer loans than men. Community Support Concern was an exception, and women received 100 percent of the 54 loans issued. Even in the Centre for Women Cooperative Development, women received 45 percent of the loans and men 55 percent; but in that case the average size of loans was higher for women than for men (Pak Rs 51,096 vs. 47,261). On average, however, across all six POs, women received smaller loans than men—Pak Rs 46,810 vs. 57,094, respectively. B.2 Non-FSP Component Partner Organizations As noted earlier, EDF wanted to “institutionally capacitate selected organizations to undertake second￾generation innovations in micro-credit/enterprise development.” Partner organizations play a critical and key role in the successful delivery of typical apex loans. The project design called for EDF to initiate loan activity with three POs in the first year, an additional three in the second year, and one more in the third year. This totaled seven during the project. However, EDF worked with only two POs in the first year and with four more in the second and third years. And, of these six, EDF did not continue with three POs. EDF failed to reach its PO target for various reasons. PPAF bureaucratic procedures slowed down a more dynamic, faster project kick off. Also, POs needed to meet strict criteria. The PPAF criteria required that eligible POs were previous clients of PPAF. Additional criteria included other risk-mitigating factors, FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 12 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 13 such as length of time in business (minimum of two years), good governance, adequate internal control procedures, and commitment to establish a separate administrative unit to manage the EDF loan product. Unfortunately, these strict criteria did not prevent serious problems in the project. For example, the Punjab Rural Support Program, a well-known and respected credit-granting NGO, failed to deliver on performance targets. PPAF terminated the contract after only nine months of lending. PPAF suspended RCDS in June 2006 for not meeting contract conditions, but reinstated it a year later. NLCL was the most regrettable choice of PO and went bankrupt due to mismanagement and fraud. Unfortunately, NLCL distributed the most loans in the project, which violated principles of business risk analysis. PPAF channeled two thirds of all loans through this organization. Regrettably, NLCL eventually went bankrupt due to mismanagement and fraud, as pointed out by the new management of the NLCL to the Securities and Exchange Committee of Pakistan (SECP), and the National Accountability Bureau (NAB) in a letter to the Chief Executive of the PPAF, Mr. Ahmed Jamil. The activities leading to this unfortunate situation were also corroborated by the findings of a special audit report conducted by Anjum Asim Shahid Rahman, 2007. The question arises as to why PPAF chose NLCL as its major partner organization in the EDF project. For one, PPAF was impressed by the commercial orientation of NLCL and by the fact that it had a large geographic outreach. In addition, PPAF saw a good opportunity since NLCL had a track record of extending loans close to the upper end of the loan limit. In fact, prior to contract arrangement with this entity, the size of the average loan was $1,200. Finally, NLCL appeared to be sound and credible. It was regularly monitored by the Securities and Exchange Companies of Pakistan (SECP), which gave confidence not only to PPAF but also to eighteen commercial banks and even donors (the World Bank and GTZ besides USAID) to engage in financial contractual arrangements with NLCL. Unfortunately, both the SECP and the external auditors of NCLC did not do their jobs rigorously. So, while the choice of NLCL appears in retrospect to be injudicious, PPAF was not the only organization with misplaced faith in NLCL. PO and EDF Capacity Building The project’s non-FSP component activities aimed to increase the capacity of those POs to undertake innovative micro-credit/enterprise development. Direct technical assistance to POs was justified by the fact that they had no previous experience in extending loans within the range of the pre-determined “missing middle”--$500 to $1,500. With the exception of NLCL, the POs credit skills specialized in loan sizes under $250. Non-FSP activities included training of EDF staff. From the beginning, PPAF supported EDF with technical staff. In 2005, two top managers of EDF traveled to the United Kingdom to attend specialized training in micro lending and investment risk analysis. Unfortunately, they left the organization soon thereafter. Within the current management, the General Manager of Credit Operations and the EDF Manager have benefited from visits to Bangladesh to observe the operations of the Grameen Bank, and have participated in SME lending workshops in the United Kingdom. At the time of this evaluation, the EDF appeared to be understaffed. Although PPAF gave support for Monitoring and Evaluation, Research, Evaluation and Policy, and the Human and Institutional Department, EDF did not have the full-time staff of eight people outlined by the project design. 9 Of the three key partner organizations that the evaluation team visited, CWCD and RCDS placed strong emphasis on staff training. CWCD coordinated closely with the Human and Institutional Development unit in PPAF regarding their staff training needs assessment, while RCDS preferred to assess their own training needs. Both POs sent their financial executives to the course arranged by Shore Bank on “How 9 Depriving resources to a project is not unusual in organizations that draw donor support from multiple sources, for they usually contend with competing, if not conflicting objectives that lead to unproductive staff shuffling and, often, taxing human resources to the limit. MFIs should Access Commercial Bank Lending.” Interestingly enough, none of the three POs visited were aware that they were eligible for funding in training in information technology, even though they had needs to update databases and management information systems. NLCL staff received no training at all. EDF had a target of 18 for trained PO staff in program management and credit-skill development. By the end of the project, almost seven times the targeted number of staff (125) obtained training in program management and credit-skill development. In one sense, this remarkable achievement over target clearly indicated EDF effectiveness; however, based on the evaluation team’s experience, this may suggest a weak indicator of increased capacity. The high number of trained staff does not reflect the quality, relevance or usefulness of the training and its impact on organizational performance. In the CWCD site visit, only 50% (6 in number) of the trained management staff of CWCD praised the training modules on financial management and monitoring and evaluation as relevant and useful. Micro-Entrepreneur Skill Building The non-FSP component aimed to raise the skills of micro entrepreneurs. This training included several activities, such as training in enterprise skill development, marketing and information linkages, and systems development services. The project targeted 800 micro entrepreneurs. As of June 2007, 654 (82% of target) received skill training. The budget allocation for this sub-component was $586,207, equivalent to 66% of the total budget for non-FSP activities. However, EDF used only 6% ($35,276) of the targeted budget for entrepreneurial skill development training. This is a great discrepancy between the substantial number of trained micro entrepreneurs and the low amount of resources EDF used for training. EDF and the POs had differences in requirements for the venue, training content and the trainers themselves. EDF envisioned consultants would conduct off-site training, but the POs did not consider such training to be meaningful for their borrowers. Their clientele were usually unable to leave their businesses for protracted periods of time, as they would forego incomes and earnings during training periods. This was particularly true for micro entrepreneurs in trade and manufacturing. Moreover, POs felt borrowers could benefit more from practical training, rather than conceptual or classroom learning. In this regard, CWCD trained micro-entrepreneurs with a practical approach within the aforementioned constraints. Such practical training was delivered by a “master trainer,” generally the most skilled person in the relevant field—for example, someone who operated a lathe machine or stitched clothes. The master trainer worked in the same geographical area, which allowed micro-entrepreneurs to easily take out one or at most two hours two to three times a week for training. As explained by Farida Tariq Baloch, the CEO of CWCD, as well as Waqar Toor, their Training Manager, women benefited from this approach because they could work from home. Resources made available through partner organizations to improve marketing techniques of their micro enterprise clienteles was only 25 percent of the target. Of all the POs, CWCD used these funds the most. Feedback from focus group meetings indicated that both men and women micro-entrepreneurs enjoyed their training in this field, including techniques for naming and branding of their businesses, as well as how to improve the packaging of products. Conclusions On the basis of actual achievements against the targeted performance indicators, the evaluation team concludes the following: First, for the FSP component, the EDF project met or significantly exceeded its targets for all six of its objectives to provide sub-loans to relatively poor entrepreneurs. However, the evaluation team is unable to determine how beneficiaries used a third of the loans. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 14 Second, EDF failed to reach its target of seven new POs to undertake enterprise development activities (EDF worked with six, and only three of those were active by the end of the project). PPAF relied solely on “known quantities” instead of carefully examining the potentially positive opportunities offered by other retailers that did not participate in the program. Third, for the non-FSP component, the evaluation team concludes that the project was hugely effective in training PO staff but fell short in meeting its objective of training borrowers. Although the EDF fell far short of meeting its objectives with respect to disbursements for non-FSP activities, it is not possible to conclude that this was a project failure because it is not clear whether more could have (or should have) been done. C. Efficiency Because the evaluation team did not estimate monetary benefits associated with the project, it is not possible to calculate the usual monetary measures of efficiency, such as net present value, benefit cost ratio, or internal rate of return. It is possible to calculate a cost effectiveness measure, i.e., cost per loan provided. According to project records, EDF disbursed 4,662 loans through March, 2007 at a total cost of $1,545,045 (not counting loan capital) for a cost per loan of $331, roughly a third of the average loan size of $905. It is also possible to examine the pattern of expenditures. According to project records, expenditures through September, 2007 amounted to $3,368,082. Three-quarters of current expenditures ($2.5 million) were from USAID funds and PPAF provided the balance ($0.8 million). Loan capital accounted for 54% of total expenditure. Figure 1 illustrates the disposition of expenditures through September, 2007. FIGURE 2: DISPOSITION OF CURRENT EXPENDITURES Expenditure ratios also provide an indication of project efficiency. Figure 2 presents selected expenditure ratios. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 15 FIGURE 3: SELECTED EXPENDITURE RATIOS Labor accounted for 65% of expenditures excluding loan capital while ODCs accounted for 27%. Outputs (i.e., training, capacity building, research, systems development, establishing a baseline, and evaluation) accounted for only 8%. Conclusions There are a number of possible measures of project efficiency that compare costs to some measure of output. In this case, however, no monetary measures of benefits exist, so the evaluation team could not calculate the usual efficiency measures (i.e., net present value, benefit cost ratio, internal rate of return). D. Impact USAID did not ask the evaluation team to estimate the monetary impacts of the EDF project. Table 6 summarizes the types of monetary and non-monetary benefits typically associated in the literature with microfinance lending. Evidence of these benefits, particularly the monetary benefits, is largely anecdotal. Evidence from more rigorous approaches to identifying these impacts is mixed: changes are often small, marginally significant, and inconsistent across studies. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 16 TABLE 6: PRIMARY MONETARY AND NON-MONETARY BENEFITS OF MICROFINANCE LENDING Monetary benefits Non-monetary benefits • Increased business income/profit • Increased household income • Increased asset value (business and household) • Increased business activity (sales) • Increased household expenditure and consumption • Increased spending on education • Increased employment • Women’s economic empowerment (for loans to women) and status in household • Improved family health and hygiene • Increased school enrollment • Improved family nutrition • Improved food security Estimates of monetary impacts for microfinance lending in Pakistan do exist. They are not, however, particularly reliable as they are generally based on inadequate assessment methods, small samples, and largely anecdotal information. Although the studies are not particularly strong, they do show that microfinance lending in Pakistan does generate impacts even if estimating a monetary value is difficult. Findings of selected studies include: • In 2007, PPAF commissioned a large (3,000 households) study of the impacts of its lending. The study included both treatment and control groups but did not attempt to estimate monetary impacts. The study concluded that the loans had (1) increased personal and household incomes, (2) increased incomes from agriculture and livestock, (3) increased enterprise income, (4) increased consumption of food items, (5) increased operating surplus, (6) improved living conditions, (7) enhanced social status of men and women. • In 2007, WHAM commissioned a survey of Asasah and NBP borrowers that concluded (without much confidence) that the loans had increased average annual incomes for Asasah (downscaling) clients by an estimated Rs. 7,620 and by an estimated Rs. 48,137 for NBP (upscaling) clients. The survey also found increased household consumption among borrowers. • In 2005, the Asian Development Bank Institute (ADBI) commissioned a large-scale study of the impacts of Khushhali Bank lending. The study did not generate monetary measures of impacts but did find that KB clients (1) spent more on health care, (2) were more likely to seek medical treatment, and more likely to do so from trained professionals, (3) exhibited greater measures of women’s empowerment (i.e., ability to get small amounts of cash from their own assets when necessary), (4) spent more on education, and (5) were more likely to have children enrolled in school than were members of a control group. Findings The evaluation team conducted group interviews of 20 CWCD and 11 RCDS beneficiaries, which provided anecdotal evidence of the impact of EDF loans and training on businesses and households. Of the 20 beneficiaries of CWD, there were 10 women and 10 men. All 11 beneficiaries of RCDS were men The findings that are detailed below combine both monetary and non-monetary benefits. Regarding CWDC, the team found that: • 18 beneficiaries indicated that sales had increased as a result of loans. • Six indicated that loans helped finance purchase of capital equipment. • Five indicated that loans helped financed working capital • Four bought gold with loan proceeds • Five indicated that they used loans to pay house rent and/or pay education bills. • Eight indicated that loans had helped increase outside employment in their businesses. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 17 • All women confirmed that loans had increased their sense of empowerment. Regarding RCDS, the team found that: • 10 beneficiaries indicated their sales rose following receipt of loans • 6 indicated they had hired non-family workers • 8 indicated loans helped finance working capital and capital equipment. Conclusions Conclusion: A large body of evidence suggests that microcredit produces measurable impacts on household welfare. Estimating a monetary measure of those impacts, however, is notoriously difficult. This case is no exception. Available data collected from group interviews suggest that EDF lending did impact a number of welfare indicators, but the team found no firm evidence that it affected household incomes or other monetary measures. E. Sustainability The purpose of this section is to determine whether the activities and results of the EDF Project are likely to be sustained after the project is completed. EDF extended loans to its POs at an interest rate of 8 percent, which was clearly below the prevailing market rate. The POs, in turn, provided credit to their customers at 17 percent, which was not high enough to cover the typical costs for Pakistan’s nascent microfinance industry. Consequently, the project granted interest rate subsidies both to loan retailers and borrowers. This scheme does not encourage micro loan providers with incentives to introduce institutional reforms. These reforms lead to the establishment of linkages with commercial banks. In addition, these reforms help transform micro loan providers into deposit-taking entities. Without institutional reforms, donor dependency occurs and providers cannot achieve long-term operational and financial independence and sustainability. The EDF project can be sustainable as long as PPAF is. At the moment, there is no question as to the financial sustainability of the latter, as it enjoys strong donor and government support. PPAF may not be operating at a high efficiency rate. Even though PPAF identified the potential POs (except for NLCL) as a “known quantity,” EDF did not efficiently select final POs with delays in processing applications as well as setting and approving loan limits. PPAF’s organizational culture relies on a hierarchical structure with decisions made at the top. If high-ranking officials are absent, decisions are delayed. Certain PPAF procedures also take more time than necessary. For example, PPAF approves an initial loan limit for one year, and then the Board of Directors must renew it annually. This procedure contravenes the PPAF’s own initial policy decision to state credit limits only once and for three years. In any event, decisions on credit limits were not taken uniformly for all partner organizations. Ironically, PPAF gave only NLCL a credit limit authorization for three years, though may or may not have led EDF to set credit limits once a year. Currently, PPAF reviews and re-approves credit limits during the March meeting of the Board of Directors. These delays have caused quite severe credit crunches for the POs, as reported by the CWCD and the RCDS. Senior managers of CWDC and RCDS reported that PPAF strictly monitored oversight and enforced compliance with its rules regarding the EDF funding agreement. So how did the EDF not anticipate the failure of NLCL? Possibly, PPAF applied most of its monitoring to its basic loan product and not the additional risks involved in the loans extended by the EDF project. In addition, PPAF did not have the risk assessment systems in place for entities with which they did not have a prior relationship in the micro credit program. The lesson is that launching higher-size loan activities demands intensive staff training and modifications to monitoring systems. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 18 At the time of the evaluation, PPAF was already in the process of developing and implementing a more robust system that would enable the timely identification and mitigation of risks emanating from the activities of a potential PO. To make this initiative more effective, PPAF set up a separate risk assessment unit, and commissioned Shorebank International to critically examine PPAF lending policies and operational manuals. In addition, Shorebank will analyze the characteristics of market segments wherein EDF could operate. Significantly, Shorebank’s analysis also includes the assessment of eight partner organizations to determine their threshold exposure levels. Box 2: Monitoring PO Performance and Risk: A Tale of Two Cases The process whereby EDF management responded to key implementation problems facing two of their partner organizations is quite revealing. Punjab Rural Support Program (PRSP) forms part of the rural support programs of Pakistan. At first sight, this entity seemed to be a good choice to be enlisted as a PO. When in the fiscal year of 2004-2005 the entity was unable to meet the pre-agreed loan target (200 loans), two senior managers at once visited their head office in Lahore to resolve this issue. The issue turned out to be far more serious than just meeting loan targets. PRSP faced high default rates of 60% in Gujranwala and 29% in Sialkot, which threatened to unravel the program with this organization. The problem was that PRSP unwisely entered a district with a history of high default rates (Gujranwala) and was adept only at managing risks tied to very small loans, not to the higher credits pertaining to the range of the missing middle. EDF management decided to withdraw the credit line for PRSP operations and requested to collect the arrears on their loans. This PRSP did, and loans were eventually fully reimbursed to EDF. These were tough measures that EDF management adopted promptly in response to the problem. On the other hand, the management response to problems with NLCL was very different. The partnership with NLCL was initiated at the same time as with PRSP, in July 2004. By the middle of 2005, NLCL was already facing some problems in repayment and informed EDF of this issue. But apart from a yearly monitoring visit to the NLCL Head Office in Karachi, no action was taken on the matter. It was left up to external auditors (Ernst and Young, 2006) to find out the irregularities involved in the operations of the entity. Inexplicably, however, even after the audit findings, PPAF continued to disburse loans to NLCL, albeit at substantially lower amounts than before, from April 2006 until June 2007. In May 2007, NLCL finally declared bankruptcy and was taken over by the management of KASB Bank, a well-known and financially sound commercial bank. All this indicates that EDF did not monitor their program with NLCL carefully. In the partnership financing agreement EDF had committed to monitoring the NLCL operations every 3 months, but in actuality a monitoring team would only visit the entity once a year, clearly not enough to detect the irregularities unveiled by the external auditors. The monitoring system clearly failed and management mistakes were clearly incurred. In this regard, it is worth underlining that other partner organizations, with which EDF had significantly less exposure, were monitored no less than twice a year. Conclusions First, USAID did not design or implement EDF in ways that focused on the sustainability of partner organizations after completion date. PPAF’s contractual terms between itself and its POs, did little to enhance the latter’s long-term self-sufficiency. PPAF extended loans to partner organizations at below market rates, which does not provide incentives to these organizations to institute badly needed management and financial reforms for facilitating access to lines of credit from commercial banks. Second, and given the fact that the EDF project is sponsored by an apex organization that at present has access to substantial financial resources, sustainability of the EDF Project is not in question, at least in the short run. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 19 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 20 F. Replication10 This section determines to what extent the activities and results of the EDF Project can be replicated. Need for Replication Given the very low degree of micro loan outreach existing in Pakistan, there is no question that there is high demand for the activities of an EDF-type project, which would target similar customers. This excludes FATA where lack of security blocks the normal execution of finance operations. Because EDF project ended, partner organizations now must seek alternative sources of loans. At the time of this evaluation, PPAF was negotiating with donors to obtain funding for $22 million to launch a product as a spin-off of the EDF project. This is known as the IFAD prism product. It included an equity injection to increase the capital base of PPAF, a credit enhancement scheme, and rights to place interest￾bearing deposits in commercial banks. Replicability of the EDF Model by PPAF On a small scale, activities similar to EDF are already being replicated in other programs currently being implemented by PPAF. For example, the World Bank-funded PPAF II features a small EDF window, and IFAD’s MIOP has a private sector lending window. For greater effectiveness and sustainability, PPAF needs to aggressively market the product to potential partners, wholesale long-term loans at commercial rates of interest, and send unequivocal signals to potential POs that selection will be based on rigorous analysis of paths leading to financial self-sufficiency. The General Manager of EDF confirmed that these projects need a clear-cut separation of roles and functions between the finance and developmental branches. Appropriately for EDF, the finance branch controlled the project, but depending too much on an organizational structure and functions that allowed for easy intermingling of credit and non-credit goals and objectives. For example, the selected POs also received funds for poverty alleviation programs. This dependency caused beneficiaries to mirror the mission, goals and activities of donors. This is precisely the case of the multi-purpose NGOs that became clients of PPAF. Their goals tended to be diffuse in part because the assistance activity that PPAF offered was so varied. To avoid this, EDF could have required that the credit-granting NGOs develop specialized credit-only branches for accessing assistance. Although EDF did set up conditions for some NGOs, EDF did not enforce them. Finally, successful replication must address staffing issues. Certain staff members need cross-cutting developmental and finance skills. Moreover, full-time staff must be technically competent. At the beginning of the project, PPAF assigned a full-time chief executive and a full-time manager to run the EDF project. As noted earlier, they left shortly after attending training programs in the United Kingdom. Only one manager, assisted by a finance specialist, was hired to perform the tasks of these people, and he could devote only one-fifth of his time to running the program. Contrary to the implementation plan, eight new technical staff were never hired, and any project replication must hire additional skilled staff. Conclusions First, clearly Pakistan can benefit from more activities similar to EDF, especially for the “missing middle” group of entrepreneurs. Some of these activities are already being replicated on a small scale in other PPAF programs. 10 Much of what is discussed in this section is based both on the evaluation team’s understanding of what went right and wrong in the EDF and on lessons from experience with other micro-credit programs elsewhere, of what is needed for a successful program. Second, to replicate in other areas of the country (other than FATA), however, these activities must include improved management procedures and practices. V. FINDINGS AND CONCLUSIONS ON THE CROSS CUTTING ISSUES A. Gender The number and value of loans extended to women borrowers nearly doubled end-of-project targets. Given the need in Pakistan to increase outreach to women, these targets however, may have been too modest. So, notwithstanding the good performance against target, loans to women accounted for only 28 percent of all loans received. During the period 2005-06, the average loan size for women was $794, equivalent to 68 percent of that for men. The distorting effect of NLCL loans explains some of this disparity because those loans were relatively large, and 70 percent of those went to men. Subsequently, and following the collapse of this entity, average loan size for men dropped to $952, and the average loan size of women was equivalent to 82 percent of that for men. TABLE 7: LOAN DISBURSEMENTS BY GENDER AS OF JUNE 2007 Loan/Gender Male Female No. of Loans 3548 (71.6% of all loans) 1409 (28.4% of all loans) Value of Loans 3.3 million 1.08 million Average Loan 952 780 Among the POs, CSC lent exclusively to women, but it made only 54 loans. In Mandi Faizabad, RCDS made only 48 loans to women vs. 649 to men over the course of two and a half years. CWCD extended significantly larger number of loans to women than to men in its earlier years. However, at the end of the project, the gender division was in the ratio of 53 percent for men to 47 percent for women. This entity employed a gender-parity methodology to reach out to women: they lend to a pair of people, mother and son, husband and wife, or brother and sister. Women made up the senior staff of CWCD, including the chief executive officer and the manager of the monitoring and evaluation unit, but all the loan officers were men. Conclusions First, actual performance with respect to the number of loans to women was nearly double the pre￾determined project targets, which may suggest that the targets were too modest in the first place. Despite that fact, women received considerably fewer loans than men, which may be explained by the project’s apparent lack of emphasis on loans to women. Second, the evaluation team found no evidence that project activities targeted women in a systematic way. As explained in the section on project effectiveness, women did benefit as recipients of loans that served to finance their micro enterprises. Although, the project did enlist a woman-oriented partner organization (CWCD), but most of the resources went to an organization (NLCL) that made no effort whatsoever to increase lending to women. Furthermore, not just with this failed entity but with the other FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 21 partner organizations as well, the evaluation team found no evidence that EDF urged the delivery of more loans to women, or made technical assistance conditional based on loans to women. B. Reporting The Planning and Implementation Plans, as well as quarterly reports, submitted were descriptive of the goals and activities of the project. Program budgets are stated in US dollars and rupees, as well as projections for FSP and non-FSP activities. However, those pertaining to the period 2004-05 were missing, as well as the first six quarterly reports. The evaluation team did not receive a satisfactory explanation of this problem. EDF did monitor actual performance against actual targets. However, the evaluation team found that targets were adjusted several times during the project with no information explaining why. Information on loan repayments from partner organizations was also not documented, so it is impossible to ascertain, from documentation review, the pattern of loan arrears of partner organizations, let alone the adequacy of risk assessment methodologies. Lastly, the evaluation team found no evidence that branding guidelines were followed. C. Public Relations/Media Coverage EDF prepared well-written case studies of impact on loan end-users of loans and disseminated these case studies among donors and GOP agencies. USAID was certainly a visible presence in the areas being served by the EDF project. For example, borrowers in Mandi Faizabad near Sheikhupura were certainly aware that the source of their funding had been provided by USAID. Overall, however, the evaluation team found no evidence of substantive efforts on part of the project to participate in high-profile events and seminars. D. Coordination In interviews with the General Manager of EDF, the evaluation team found that PPAF implemented its poverty alleviation activities in accordance with the goals of the GOP, as specifically stated by the SBP. In this regard, EDF played an important role as well as in the technical inputs submitted to the World Bank for the preparation of the Poverty Reduction Strategy Paper. These activities show that EDF coordinated with these two important entities of the GOP and the World Bank. VI. RECOMMENDATIONS To the extent that it remains engaged in finance projects, USAID should build on the credit assessment of PPAF recently conducted by Shorebank International, and consider the following: (a) Further enhancing PPAF’s capabilities in typical wholesaling operations. This program of technical assistance should include, at minimum: • Development of strategic, business and operational plans • Market research techniques • Risk-assessment tools • Client selection strategy FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 22 • Monitoring systems that measure and forecast risks • Communication strategy with relevant stakeholders (b) Restructuring the finance and development components of PPAF, so as to define clear objectives and roles for each. For example, under finance, more strict selection criteria in selecting the retailers, risk assessment, stronger monitoring of the retailers’ portfolio and operations, and more defined key financial performance indicators. (c) Similar to the point above, restructuring the development role of PPAF. This includes a more robust technical assistance program to the intermediaries, or POs. Such assistance would comprise better systems, improved management capacity, loan officer training, better selection of end users, and more focused and effective outreach and business development services. In other words, a change towards more aggressively marketing the EDF loan product to potential partners, rather than simply reacting to forthcoming requests. (d) If replicating the EDF model, USAID should: • Set up wholesaling of long-term loans at commercial rates; • Require credit-granting NGOs to develop specialized credit-only arms, as a sine-qua-non condition for accessing assistance; and • Select all POs partners based on rigorous analysis of the candidate PO’s plan to achieve financial self-sufficiency. VII. LESSONS LEARNED When a beneficiary is responsible for executing the project, USAID should conduct regular meetings to appraise in progress-results and determine the need for program adjustments leading to maximize program impact. Furthermore, USAID should have financial control mechanisms to closely monitor decision making processes and expenditure patterns. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 23 ANNEX 2: REFERENCES Anjum Asim Shahid Rahman, 2007 (AASR 2007) Report of factual findings, Network Leasing Corporation Limited for the Year ended June 30, 2007. Anjum Asim Shahid Rahman Chartered Accountants. Karachi, Pakistan, September 17, 2007. Burki, Hussan Bano and Shah, Mehr, 2007 (Burki & Shah 2007) The Dynamics of Microfinance Expansion In Lahore. United State Agency for International Development and Shorebank International Limited. Lahore, Pakistan, 2007. Center for Women Co-operative Development (CWCD 2004) Hammara Credit Need Evaluation Form Enterprise Development. Developed by CWCD for EDF Loan. Lahore, Pakistan, 2004. CWCD, Business Development Manager (CWCD BDM 2006) Annual Report 2006 SME Department. Center for Women Co-operative Development. Lahore, Pakistan, 2006. CWCD, Business Development Manager and Training Manager (CWCD BDTM 2006) Annual Report 2006 SME Department. Center for Women Co-operative Development. Lahore, Pakistan, 2006. CWCD, Business Development Manager (CWCD 2006) Annual Report 2006 SME Department. Center for Women Co-operative Development. Lahore, Pakistan, 2006. CWCD, Training Department, Micro Enterprises Support Cell, 2006 (CWCD, TMSC 2006) CWCD. Lahore, Pakistan. Report On One Day Training Course on Traffic Laws and Auto Rikshaw Engine Maintenance, December 15, 2006. One Day Training Course on Animal Health Awareness, December 22, 2006. Enterprise Development Fund, Pakistan Poverty Alleviation Fund (EDF, QPR No) List of “Quarterly Progress reports”. Pakistan Poverty Alleviation Fund (PPAF). Islamabad • “Quarterly Progress report, January-March 2005 (QPR 6)” • “Quarterly Progress report, April-June 2005 (QPR 7)” • “Quarterly Progress report, July-September 2005 (QPR 8)” • “Quarterly Progress report, October-December 2005 (QPR 9)” • “Quarterly Progress report, January-March 2006 (QPR 10)” • “Quarterly Progress report April-June 2006 (QPR 11)” • “Quarterly Progress report, July-September 2006 (QPR 12)” • “Quarterly Progress report, October-December 2006 (QPR 13)” • “Quarterly Progress report, January-March 2007 (QPR 14)” • “Quarterly Progress report, Aril-May 2007 (QPR 15)” FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 24 Ernst and Young (EY, 2006) Results of first recipient contracted audit (RCA) of PPAF. October 1, 2003 to June 30, 2005. October 11, 2006. Gallup Pakistan, 2007 (GP 2007) Baseline Study of PPAF’s Enterprise Development Facility 2004-2005. Final Report Gallup. Islamabad, Pakistan, February 8, 2007. Kashf Foundation, 2006 (KF 2006) Annual Report 2006, Growing opportunities for the Unbanked. Lahore, Pakistan, 2006. Montgomery, Heather, 2005 (Montgomery 2005) Meeting The Double Bottom Line, The Impact of Khushhali Bank’s Microfinance Program in Pakistan. Asian Development Bank Institute, ADBI policy paper No 8. Tokyo, Japan, 2005. Network Leasing Corporation Limited (NLCL, 2007) Annual Report, 2007. Network Leasing Corporation Limited. Karachi, Pakistan, October 5, 2007. Pakistan Microfinance Network and Shorebank International Limited (PMN & SIL 2007) Microwatch Issue 3: Quarter 1 (January-March 2007). United State Agency for International Development . Lahore, Pakistan, June 2007. Pakistan Poverty Alleviation Fund (PPAF 2004) Financing Agreement, Pakistan Poverty Alleviation Fund, Enterprise Development Facility and Network Leasing Corporation Limited. Islamabad, Pakistan, 2004. Pakistan Poverty Alleviation Fund (PPAF May 2007) Financing Agreement Between Pakistan Poverty Alleviation Fund, Enterprise Development Facility and Center for Women Cooperative Development. Islamabad, Pakistan, May 4, 2007. Pakistan Poverty Alleviation Fund (PPAF Feb 2007) Financing Agreement, Pakistan Poverty Alleviation Fund, Enterprise Development Facility and Rural Community Development Society (Phase-III). Islamabad, Pakistan, February 1, 2007. Pakistan Poverty Alleviation Fund (PPAF Annual Reports Numbers) List of “Annual Reports”. Pakistan Poverty Alleviation Fund (PPAF). Islamabad • Annual Report, September 27, 2004. • Annual Report, August 18, 2005. • Annual Report, September 13, 2006. • Annual Report, September 17, 2007. Pakistan Poverty Alleviation Fund, 2006 Scan 0001 (EDF QPR 11). PPAF. Islamabad. 2006. PPAF, Credit Enterprise Development (CS-CED 2004) Profiles in Courage, Case Studies. Pakistan Poverty Alleviation Fund. Islamabad, Pakistan, 2004. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 25 PPAF, Credit Enterprise Development (CS-DA 2007) Driving Ambitions, Case Studies. Pakistan Poverty Alleviation Fund. Islamabad, Pakistan, March 9, 2007. Qureshi, Uzma and Zaka, Sarah, 2007 (Qureshi and Zaka 2007) Survival or Take-off: A Study of MFEs in Lahore Small Business Finance Market. Lahore, Pakistan, October 2007. Rural Community Development Society (RCDS 2007) Presentation Outlines. Mandi Faizabad, Pakistan, November 28, 2007. Rural Community Development Society (RCDS 2006) Annual Report 2006. Mandi Faizabad, Pakistan, March 2007. Rural Community Development Society, 2006 Microfinance Program CED and EDF. Mandi Faizabad, Pakistan. 2006 Rural Community Development Society (RCDS 2007) Institutional Development through Capacity Building Program. Mandi Faizabad, Pakistan, 2007. United States Agency for International Development, 2003 (USAID 2003) Implementation Plan 2003-4 (IP Year 1). Enterprise Development Fund, (USAID). Washington. United States Agency for International Development, 2004 (USAID 2004) Implementation Plan 2004-5 (PIP Year 2. Enterprise Development Fund, United States Agency for International Development. Washington, November 25, 2004. United States Agency for International Development, 2006 (USAID 2006) Implementation Plan 2005-6 (PIP Year 3). Enterprise Development Fund, United States Agency for International Development. Washington, September 4, 2006 United States Agency for International Development (USAID PIP 2007) Yearly Work Plan October 2005-6 (PIP Year 3). Enterprise Development Fund, United States Agency for International Development. Washington, November 17, 2006 – end September, 2007. United States Agency for International Development and PPAF, 2003 (USAID, PPAF 2003) Award of Cooperative Agreement No. 391-A-00-01010-00; Enterprise Development Facility Initiative. Pakistan Poverty Alleviation Fund. Islamabad, Pakistan, Oct 1, 2003. United States Agency for International Development (USAID, PMP 2006) USAID Pakistan/Performance Monitoring Plan. Washington D.C, January 2006. United States Agency for International Development, Islamabad (USAID Audit 2006) Audit of USAID Enterprise Development Facility (EDF) Program for the period from October 1, 2003 to June 30, 2005. Letter and Enclosure of 3 copies of Audit documents, Ford Rhodes, Sidat Hyder and Co. Islamabad, Pakistan, August 31, 2006. United States Agency for International Development, Islamabad (USAID, PMP 2006) Pakistan Poverty Alleviation Fund Response to IG Case Number L- 0705 G- 06-14-07 Memo 08- 06-07. Islamabad, Pakistan, August 6, 2007. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 26 United States Agency for International Development and PPAF, 2007 (USAID PPAF 2007) USAID-EDF Budget - Revised Final, 2000.xls. USAID. Islamabad, 2007. United States Agency for International Development and PPAF, 2004 (USAID PPAF 2003) EDG Timeline and Budget, December 9, 2003. USAID. Islamabad, 2004. USAID, Chief Technical Officer, 2004 (CTO, USAID 2004) PPAF Budget by CTO, 2000 xls. USAID. Islamabad, 2004. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 27 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 28 ANNEX 3: LIST OF PEOPLE INTERVIEWED PPAF/EDF and Partner Organizations Ahmad Jamal, Chief Strategic Planning Officer. Pakistan Poverty Alleviation Fund. Mr. Tariq Khan Baluch, General Manager, Credit and Enterprise Development, Pakistan Poverty Alleviation Fund. Mr. Zulqarnain Rafiq, Management Executive Credit and Enterprise Development, Pakistan Poverty Alleviation Fund. Mr. Asghar Memon, Manager Credit and Enterprise Development, Pakistan Poverty Alleviation Fund. Mrs. Farida Tariq, Chief Executive Officer, Center for Women Cooperative Development. Mr. Muhammad Abu Bakar, Manager Micro Credit Operations, Center for Women for Development. Mr. Yasir Tariq, Business Development Manager, Center for Women Cooperative Development Ms. Saira Hassan, Financial Manager, Center for Women Cooperative Development. Mr. Waqar Toor, SME Training Manager Center for Women Cooperative Development. Mr. Shahzad Khan, CWCD Shalimar Branch Manager, Lahore Mr. M. Sualeh Faruqui, Head of Human Resource Department and Administration, Net Work Leasing Corporation Limited, Mr. Mohammed Farooq, Senior Manager, Net Work Leasing Corporation Limited Mr. M. Nadeem Ahmed, the Chief Financial Officer, Net Work Leasing Corporation Limited, Mr. Murtaza, Chief Executive Officer, Rural Community Development Society, Microfinance Program. Mr. Rasheed Ahmad, Manager (HR), Rural Community Development Society, Microfinance Program Syed Mujahid Hussain, Senior Manager and Program Manager, Microfinance Program, Rural Community Development Society Key Informants Gregory C. Chen, Country Representative, Pakistan, Shore Bank International. Nadeem Hussain, President & CEO. Tameer Microfinance Bank Limited. Mr. Qasim Nawaz, Director State Bank of Pakistan, Mr. Haroon Sharif, Senior Advisor, Private Sector Development Department for International Development (DFID), British High Commission, Zach Orend, United States Agency for International Development Khalid Nawaz. Project Director. Financial Sector Strengthening Programme, Swiss Development and Cooperation Aban Haq, Research Analyst. Pakistan Microfinance Network Mohammed Azim Hashimi, Investment Officer. Pakistan Resident Mission. Asian Development Bank Rashid Bajwa, CEO. National Rural Support Programme ANNEX 4: GETTING TO ANSWERS MATRIX Evaluation Questions Type of Answer or Evidence Needed Method of Data Collection Data Source Selection Criteria Data Analysis Methods 1. Have project activities directed to partner organizations increased their capacity? Work with project experts to determine appropriate questions to assess capacity building. 2. Have technical support and non￾credit interventions to the poor improved their business skills? Work with project experts to determine appropriate questions to assess capacity building. Surveys Loan recipient surveys 3. How many loans have been disbursed in each quarter since EDF started operations? (by region, partner organization, and gender) Comparison with number of loans without EDF Document review Project documents review Describe trajectory of loan disbursements since EDF began operations. Break down by province/district, partner organization, gender. 4. What are projections for loan disbursements in the future? (by region, partner organization, and gender) Trend in loan disbursements Interviews Key informants interview Partner organization interview Project staff interview Project number of loans through ______ (date) by province/district, partner organization, and gender. 5. How many loans would have been disbursed (in target loan range?) without USAID intervention over the life of the project and into the future? (by region, partner organization, and gender) Comparison with number of loans disbursed with EDF Interviews Document review Surveys Key informants interview Review of historic records of partner organizations Potential survey of financial institutions not partnered with EDF Project trend in lending from pre-EDF period. FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 29 FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 30 Evaluation Questions Type of Answer or Evidence Needed Method of Data Collection Data Source Selection Criteria Data Analysis Methods 6. What would business income have been without the loans? (by region, business type, gender) Serves as control for estimating loan impacts Document review Surveys Review of business statistics if timely and applicable Control group businesses survey Loan recipient surveys If control group surveyed, then stratify by region and business type. If from loan recipient survey, then stratify survey of loan participants by loan size and by region (and by lender?) Given the number of loans and nationwide scope, a clustered sample may be necessary. Need detail on location of loans for sampling. Estimate business income without EDF support. Determine appropriate control based on available data and feasibility of control group survey. 7. What impact have loans had on the income of recipient businesses? (by region and gender) Comparison of business income with and without loans. Document review Surveys Project documents (including recent survey in Lahore) Loan recipient surveys Stratify survey of loan participants by loan size and by region (and by lender?) Given the number of loans and nationwide scope, a clustered sample may be necessary. Need detail on location of loans for sampling. Stream of with and without impacts over time. 8. How were the loans used and what impact did they have on the business? Descriptive Surveys Loan recipient survey Stratify survey of loan participants by loan size and by region (and by lender?) Given the number of loans and nationwide scope, a clustered sample may be necessary. Need detail on location of loans for sampling. Narrative descriptions of impacts of loans on the business 9. How, if at all, have the benefits of the loan to the business affected the income of the owner and the welfare of the owner’s household? (by region, Descriptive Quantitative Surveys Loan recipient survey Stratify survey of loan participants by loan size and by region (and by lender?) Given the Narrative descriptions of impacts of loans on the business Estimates of current and FINAL EVALUATION OF THE ENTERPRISE DEVELOPMENT FACILITY PROJECT 31 Evaluation Questions Type of Answer or Evidence Needed Method of Data Collection Data Source Selection Criteria Data Analysis Methods loan size, and gender) number of loans and nationwide scope, a clustered sample may be necessary. Need detail on location of loans for sampling. lasting impacts on household income. 10. Have the loans generated any secondary impacts as a result of increased business activity? (by region, loan amount, and gender) Surveys Existing model or multipliers Loan recipient survey Stratify survey of loan participants by loan size and by region (and by lender?) Given the number of loans and nationwide scope, a clustered sample may be necessary. Need detail on location of loans for sampling. Existing multipliers applied to estimates of additional employment or income for recipient businesses. 11. In what ways, if any, has the EDF training helped your institution or its employees? Descriptive Interview Interviews with partner organizations Narrative, content analysis of common benefits or merits of training 12. Has the project increased your organization’s capacity to provide credit to entrepreneurs that have outgrown traditional microfinance markets? Explain. Descriptive Interview Interviews with partner organizations Narrative, content analysis of common ways project has increased capacity 13. Are the borrowers from the EDF successfully pursuing other sources of commercial financing? Descriptive Surveys Loan recipient survey Proportion of loan recipients who have obtained other commercial loans 14. Are the terms and requirements for accessing EDF loans appropriate Descriptive Interview, Surveys Interviews with partner organizations, Loan recipient survey Narrative, content analysis of common complaints about requirements