DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT APRIL 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under Cooperative Agreement 391-A-00-03-01011-00 EVALUATION REPORT: DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT Implemented by Khushhali Bank (KB) in the provinces of Balochistan and Sindh, and in the Federal Administrative Tribal Areas under a Cooperative Agreement with USAID/Pakistan DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS PROJECT SUMMARY...............................................................................................................III ACRONYMS................................................................................................................................IV EXECUTIVE SUMMARY .......................................................................................................... V SUMMARY EVALUATION REPORT................................................................................... VII ANNEX 1: DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT............................................................................................. 1 I. INTRODUCTION............................................................................................... 2 II. THE DEVELOPMENT PROBLEM................................................................ 3 A. Problem Statement ................................................................................. 3 B. USAID Intervention ............................................................................... 4 III. EVALUATION METHODOLOGY............................................................... 6 A. Evaluation Purpose and Questions......................................................... 6 B. Evaluation Design .................................................................................. 8 C. Data Collection Sources and Methods................................................... 8 D. Data Limitations..................................................................................... 9 IV. FINDINGS AND CONCLUSIONS ON THE OVERARCHING QUESTIONS / ISSUES............................................................................. 9 A. Relevance............................................................................................... 9 B. Effectiveness........................................................................................ 12 C. Impact................................................................................................... 16 D. Efficiency............................................................................................. 19 E. Sustainability ........................................................................................ 20 F. Replication............................................................................................ 22 V. FINDINGS AND CONCLUSIONS ON THE CROSS CUTTING QUESTIONS ........................................................................ 23 A. Gender.................................................................................................. 23 B. Reporting.............................................................................................. 24 C. Communications and Outreach ............................................................ 24 D. Coordination ........................................................................................ 25 VI. RECOMMENDATIONS ............................................................................... 25 VIII. LESSONS LEARNED ............................................................................. 26 ANNEX 2: BIBLIOGRAPHY/REFERENCES........................................................................ 27 ANNEX 3: LIST OF INDIVIDUALS INTERVIEWED ......................................................... 29 ANNEX 4: GROUP INTERVIEWS.......................................................................................... 31 ANNEX 5: GETTING TO ANSWERS MATRIX KB PROJECT......................................... 32 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT i TABLES Table 1: Getting to the Answers for Overall Evaluation Questions....................................................8 Table 2: Loan Targets and Actual Performance – Balochistan and Sindh.......................................14 Table 3: Loan Targets and Actual Performance - FATA..................................................................14 Table 4: Loan Targets and Actual Consolidated Performance .........................................................14 Table 5: Primary Monetary and Non-Monetary Benefits of Microfinance Lending ........................17 Table 6: Sample Distribution............................................................................................................17 Table 7: Comparison of Households with KB Loans and Households Without KB Loans .............18 Table 8: Loan Performance – Gender...............................................................................................23 FIGURES Figure 1. USAID Branch Disbursement Trends: Total quarterly Disbursements in Pakistan Rupees ...............................................................................................................13 Figure 2. Active Clients at USAID-Supported Branches: Number of Active Clients.....................13 Figure 3. Number of Loans Disbursed..............................................................................................14 Figure 4. Selected Expenditure Ratios..............................................................................................20 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT ii PROJECT SUMMARY Project Name Developing Non-Bankable Territories for Financial Services Life of the Project (LOP) October 2003 to September 2008 (Sindh and Balochistan) and September 2010 (FATA) Project Location Selected Districts in Sindh, Balochistan and FATA Implementing Partner Khushhali Bank Activity Number 391-A-00-03-01011-00 Type of Activity Cooperative Agreement Budget USAID Share: US$ 11.05m KB Share: US$ 03.08m Total: US$ 14.14m DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT iii ACRONYMS ADBI Asian Development Bank Institute ADB Asian Development Bank EDF Enterprise Development Facility FATA Federally Administered Tribal Area GTA Getting to Answers Matrix GOP Government of Pakistan KB Khushhali Bank NBP National Bank of Pakistan MSI Management Systems International PMP Performing Monitoring Plan RSP Rural Support Program SBP State Bank of Pakistan USAID United States Agency for International Development WHAM Widening Harmonized Access to Microfinance ZTBL Agricultural Development Bank DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT iv EXECUTIVE SUMMARY As demonstrated by experience in developed and developing countries alike, a robust and efficient financial sector helps mobilize savings, foster productive investments, and support balanced economic growth. Since political independence sixty years ago, Pakistan did not create these conditions to make all this possible. Starting in the early to mid 1990s, however, the Government of Pakistan (GOP) introduced important financial reforms that are gradually contributing to higher rates of economic growth. This included enacting measures to strengthen the microfinance sector through the active involvement of the State Bank of Pakistan (SBP) in policy making, supported by the establishment of a consultative group drawn from industry stakeholders. For example, in 2001, Pakistan set up the Microfinance Institutions Ordinance, which regulates micro lending operations and establishes rules for the creation of specialized microfinance banks. Large parts of Pakistan’s territory do not present ideal conditions for the development of banking services. These are areas characterized by: (a) very low population density; (b) poorly maintained roads, outdated electricity grids, undeveloped communication networks and other problems present in low-quality economic infrastructure; (c) lack of security; (d) reliable information on potential customers that is not easily accessible; (e) low-quality of education and population incomes that are low and volatile; and (f) cultural traits that reinforce resistance to social change. These problems defy simple solutions. Since the early 1970s, however, successive Pakistani governments have tried to reinforce state presence and deliver credit to the marginalized populations through the operations of the Agricultural Development Bank (ZTBL) and the active support to rural support programs (RSPs). More recently, in August 2000, the SBP facilitated the establishment of Khushhali Bank (KB). The creation of this bank originated in a public￾private partnership initiative that sought to improve timely access to financial services for the benefit of low-income households and micro enterprises, especially those located in rural areas. This report addresses the final evaluation of the Developing Non-Bankable Territories for Financial Services project. KB implemented the project in the provinces of Balochistan and Sindh, and in the Federal Administrative Tribal Areas (FATA). By offering almost a third of total financial outreach in Pakistan, KB is a significant presence in Pakistan’s microfinance industry. KB significantly contributed to the impressive rate of clientele growth (50% and above) achieved by the microfinance industry in the last three years. By the end of 2006, KB’s indicators for operational and financial sustainability were 75% and 57% respectively.1 The KB Project did not target the destitute, that is, people living in either chronic or extreme poverty. Rather, the project focused on the poor people of Pakistan, which are characterized as those households living just above the poverty line (transitory vulnerable) and those living just below the poverty line (transitory poor). The transitory vulnerable and the transitory poor are approximately 57.5 million people and represent 36% of all households in Pakistan.2 Households and Poverty Bands in Pakistan Source: Centre for Research on Poverty Reduction and Income Distribution: Pakistan Human Condition Report 2003 1 Pakistan Microfinance Review [2006]. KB’s own calculations of operational and financial sustainability are 74.38% and 69.28% respectively. 2 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT v Source: Centre for Research on Poverty Reduction and Income Distribution: Pakistan Human Condition Report 2003. As a principal activity, the project accomplished loan disbursal as displayed by the following table. This table covers loan activity in Balochistan and Sindh, and in the FATA region. TABLE 1: LOAN TARGETS AND ACTUAL PERFORMANCE Balochistan and Sindh FATA Indicator/Performance Actual Result (as of 9/2007) Project Target (to 9/2008) Actual Result Project Target (as of 9/2007) (to 9/2008) # of Loans Disbursed 327,352 362,583 1,483 51,369 Value of Loans Disbursed (US $) 57.6 million 64.5 million 245,000 11.2 million # of Active Clients 94,806 103,858 973 18,043 The evaluation of the KB Project focused on 10 primary themes. The following table summarizes the main conclusions for each theme: Evaluation Theme Key Conclusions Relevance • Partnering with KB was relevant as a mechanism to meet USAID’s objective to expand the provision of financial services to underserved areas, with a focus on serving clients who have been largely marginalized by mainstream finance. • The project emphasized the delivery of one single loan product whose terms and conditions do not provide an optimal match with the full set of needs of the target clientele. In particular, it would have been useful for KB to have developed a savings product. Effectiveness • The project will meet its loan targets in Balochistan and Sindh, both in terms of the number of loans and the amount of funds disbursed. In FATA, to date KB has only been able to disburse 3% of the target number of loans it had hoped to disburse. • With respect to client retention, the project had a target to retain 50% of clients in the second loan cycle and 55% in the third loan cycle. As of November 2007, for the project￾supported branches, KB reported that the percentage of borrowers who repeat loans a first and second time is, respectively, 27% and 12%. Impact • Micro credit produces measurable impacts on household welfare. Estimating a monetary measure of those impacts, however, is notoriously difficult. Available data suggest that KB lending did impact a number of welfare indicators but produced no evidence that it affected household incomes or other monetary measures. Efficiency • With no monetary measures of benefits, it was not possible to calculate the usual efficiency measures (i.e., net present value, benefit cost ratio, internal rate of return). Sustainability • The project was not designed or implemented with a focus on the achievement of long￾term sustainability (the self-financing of operational costs after project completion). Replication • The project’s design should be re-visited before replication activities are considered. Additionally, plans to continue activity in FATA should be reexamined in light of project experience to date and in consideration of accomplishing project goals in an area where security is likely to hamper project performance. Gender • KB does try to reach out to women clients, but efforts under the USAID project have not been as successful as KB’s performance in non-project branches. Reporting • Reports were submitted in a timely manner. Branding requirements were followed. Communication and Outreach • KB has recently adopted more sensible loan policies, and since its incorporation as a member of the Pakistan Microfinance Network (PMN), its image is gradually improving. Coordination • More active interface of KB with industry stakeholders has yielded positive results. The most important of all was the decision not to subsidize loan rates. Recommendations 1. USAID should renegotiate the terms of the contract with KB regarding the Balochistan and Sindh operations. For these branches, the Mission should stop financing branch operations. The current activity leads to increased donor dependency. Instead, USAID should offer a package of technical assistance with microfinance best practices emphasizing the following: market assessments, loan product development, loan product pricing, development of savings and other financial instruments, corporate governance, human resources policy development and financial management. 2. The activities in the FATA region should be suspended until the overall security situation improves significantly. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT vi SUMMARY EVALUATION REPORTT he active 3 INTRODUCTION In August 2007, the United States Agency for International Development (USAID) commissioned Management Systems International (MSI) to conduct the evaluation of eight projects within its Pakistan Economic Growth portfolio of activities. Three projects corresponded to activities that seek to increase access to financial services for the benefit of households and micro and small businesses in the rural and urban areas of Pakistan (Widening Harmonized Access to Microfinance (WHAM), Enterprise Development Facility (EDF) and Developing Non-Bankable Territories for Financial Services). This report addresses the final evaluation of Developing Non-Bankable Territories for Financial Services, implemented by Khushhali Bank (KB) in the provinces of Balochistan and Sindh, and in the Federal Administrative Tribal Areas (FATA), under conditions stipulated in the Cooperative Agreement 391-A-00- 03-01011-00 of September 2003, and subsequently modified two years later, in September 2005. The KB project started operations in October 2003 and ends its Balochistan and Sindh operations in September 2008, and its FATA operations in September 2010. Total project budget is $14.1 million, 78% of which was funded by USAID. Similar to EDF, the beneficiary agency directly implements the project’s contractual arrangement. Donor interventions in finance-related projects are entirely justified when they propel the development of financial markets. As demonstrated by experience in developed and developing countries alike, a robust and efficient financial sector helps mobilize savings, foster productive investments, and support balanced economic growth. Since political independence sixty years ago, Pakistan did not create these conditions to make all this possible. Starting in the early to mid 1990s, however, the Government of Pakistan (GOP) introduced important financial reforms that are gradually contributing to higher rates of economic growth. This included enacting measures to strengthen the microfinance sector through t involvement of the State Bank of Pakistan (SBP) in policy making, supported by the establishment of a consultative group drawn from industry stakeholders. For example, in 2001 the Microfinance Institutions Ordinance in 2001began to regulate micro lending operations and establish rules for the creation of specialized microfinance banks. 4 5 Prior to this, the SBP used a special ordinance to establish KB in August 2000. The creation of the bank originated in a public-private partnership initiative that sought to improve timely access to financial services for the benefit of low-income households and micro enterprises, especially those located in rural areas. Prompted by the GOP, public and private commercial banks contributed equity. In addition, end-user credit received funding from $70 million of a $150 million loan from the Asian Development Bank (ADB). Currently the issued, subscribed and paid-up capital totals $28.4 million, and the state-owned National Bank of Pakistan (NBP) owns the largest stake in KB with 23% of the shares ($6.6 million). Fifteen private domestic and foreign banks hold the remaining shares, most importantly Habib Bank Limited and Muslim 3 The full evaluation report is included as Annex 1, included information on data sources and additional annexes. 4 Poverty Reduction Strategy Paper [2001]. 5 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT vii KB informed the Evaluation Team that, effective April 1 2008, the operations of KB are regulated by the Microfinance Ordinance of 2001. Commercial Bank Ltd. They each contributed $5 million (18%), and the United Bank Limited and the Allied Bank of Pakistan Ltd., which each contributed $3.33 million (12%). At the time of this evaluation, the government was conducting high-level discussions to sell off the NBP shares, which turns KB into an entirely private-owned commercial bank.6 Headquartered in Islamabad, KB operates with 1,870 employees in 113 branches in eighty eight districts of the country. When KB began, the government had a high priority in sending resources to poverty-stricken areas. Responsible for financing community infrastructure projects, KB also conducted other typically non￾core activities, such as granting scholarships to raise technical skills of prospective clients and providing business development services to customers. KB policy equally targets men and women. At the end of 2006, the number of active borrowers was 236,917, of which 120,715 were women.7 However, for the USAID-funded project, loans to women fell way below bank targets (33%) and actual performance (50.9%) in the country at large. By offering almost a third of total financial outreach in Pakistan, KB is a significant presence in Pakistan’s microfinance industry. KB significantly contributed to the impressive rate of clientele growth (50% and above) achieved by the microfinance industry in the last three years. By the end of 2006, KB’s indicators for operational and financial sustainability were 75% and 57% respectively.8 Evaluation Methodology The evaluation team used the following data collection methods. • Review of relevant documents including the project proposal presented to USAID, project implementation plans and quarterly reports. The review was conducted with the arrival of the Team Leader in Islamabad on October 12, 2007. • Interviews with key informants: These included (a) four representatives of the donor community; (b) one representative of a microfinance bank; (c) an official of the State Bank of Pakistan; (d) three officials of the Pakistan Poverty Alleviation Fund (PPAF); and (e) two representatives of a civil society organization – the Pakistan Microfinance Network. The interviews were conducted in Islamabad, Lahore and Karachi, during the period October 17 – November 19 2007. • Interviews with key personnel of KB: (a) the President, Operations Manager, the Manager of the USAID-funded Project at headquarters in Islamabad; and (b) regional business managers, branch managers, territory portfolio managers, relationship managers, credit specialists and business development officers of the following branches: Quetta and Pishin (Balochistan); Hyderabad, Nabawshah, Sukkur and Jaffarabad (Sindh); and Khyber and Khurram in FATA. The interviews were conducted during from November 1, 2007 to March 2, 2008. • Group interviews with selected borrowers that have been extended loans by the KB Project in the aforementioned branches: The interviewed borrowers numbered 69, of which 45 were men and 24 were women. The evaluation team developed a comprehensive methodology to conduct the evaluation, using a Getting to Answers (GTA) Matrix. This planning tool helps determine the necessary information to answer each evaluation question, from where and how the data can be obtained, and appropriate analysis techniques. The GTA for this evaluation is included in the main KB report. 6 KB informed the Evaluation Team that NBP intends to divest no later than June 30, 2008. If and when this happens, KB will be completely privatized. 7 Pakistan Microfinance Review [2006] 8 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT viii Ibid. On the other hand, KB’s own calculations of operational and financial sustainability are 74.38% and 69.28% respectively. FINDINGS AND CONCLUSIONS Relevance Evaluation question: How well was the project focused on the needs of the beneficiaries? As principal stakeholders of this project, USAID and KB recognized the necessity of launching a project to expand finance into underserved areas of the country, including rural localities. The project targeted households and micro enterprises largely marginalized from mainstream finance. Many exist in non￾bankable territories, which means areas difficult for the sustained extension of formal financial services. With a mandate to serve these areas, KB extends at least 75% of the bank loans in rural areas. Five key informants said share-holding commercial banks perceive KB as an instrument of the Ministry of Finance or the SBP. The evaluation team could not obtain details of the Board composition, nor ascertain what parties attend the board meetings and how frequently these are conducted. The ADB exerts an extremely limited influence in operational decisions. For example, KB used 70% of the ADB funds reserved for loans ($48 million) for treasury bills instead.9 Project design included: (a) Emphasizing loan delivery to beneficiaries, and disregarding other financial instruments, such as savings products, which could have an important difference in the improvement of the livelihoods of the beneficiaries; (b) Setting objectives and undertaking activities outside the core functions of typical commercial banking, such as the provision of business development services to prospective clients; and (c) a lack of focus on the achievement of long-term operational sustainability. Other select findings include the following: • The KB Project did not target the destitute, that is, people living either chronic or extreme poverty. Rather, the project focused on the poor people of Pakistan, characterized as households living just above the poverty line (transitory vulnerable) and those living just below the poverty line (transitory poor). The transitory vulnerable and the transitory poor are approximately 57.5 million people and represent 36% of all households in Pakistan.10 • According to KB top management, the project launched in areas with weak demand for loan products. At the same time, top management said that the clientele is already heavily indebted to both formal and informal sources. • KB has branch presence all over the country, operating in areas where there is a large presence of other commercial banks and financial entities. USAID funded the project for areas where KB is the only source of formal financial services—five districts in Balochistan, three districts in Sindh, and seven agencies in FATA. • From interviews conducted in field visits with branch managers and loan officers, the team found evidence that they do indeed have a good grasp of clientele possibilities, as well as understanding of their economic potential and the problems they confront. Conclusions: By partnering with KB, USAID met its objective to provide financial services to underserved areas, and therefore the project is highly relevant. USAID designed the project with an emphasis on the delivery of a single loan product that did not address the full set of needs of the target clientele. The project did not design variations of loan products for education, emergencies, or housing. In particular, KB did not develop a savings product or a way for branch managers to give flexibility on loan repayment schedules and on increasing the limits for second and third loans. 9 Information provided by Mr. Azim Hashimi, Project Implementation Officer of the ADB. 10 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT ix Source: Centre for Research on Poverty Reduction and Income Distribution: Pakistan Human Condition Report 2003. Effectiveness Evaluation question: Has the project accomplished its objectives? The evaluation team reviewed KB documents, and found evidence that KB’s top management was well aware of the monumental challenges. For example, KB knew of the institutional problems associated with rapid start up, including putting management systems in place. KB also needed to overcome skepticism from industry stakeholders, including the commercial banks that contributed capital. KB management also faced pressures from government to show quick results in non-bankable areas. Not surprisingly, KB experienced a difficult start. Operations in Balochistan provide a useful illustration of the difficulties the bank ran into, well before it entered into an agreement with USAID. After one year of operations in this province (two years before project kick off), KB was operating with high operational costs. It was also saddled with a large number of non-performing loans. In this context, KB approached USAID for support to operate more effectively in non-bankable areas of Balochistan, Sindh and FATA. USAID wanted to collaborate with KB to establish a firm foundation for further expansion of financial operations into other highly volatile, risky, yet underserved territories. The project does not provide funding for loan activity. Instead, the project finances the establishment of branches and their operational costs in selected districts of Balochistan, Sindh and FATA. The team found no evidence that USAID actively set targets for project activities even though KB proposed them in a document submitted to USAID. Furthermore, a useful performing monitoring plan (PMP) for the project was never developed. Project Targets: Project targets reflect a multi purpose approach to banking; unfortunately, these do not contribute to successful long-term sustainability of commercial bank operations. These targets included: (a) Disbursing 363,000 new loans in Balochistan and Sindh, and 51,000 in FATA; (b) Retaining 50% of clients in the second loan cycle and 55% in the third loan cycle; (c) Mobilizing 18,298 community groups; (d) Stabilizing the incomes of 10% of KB borrowers who have taken loans for three consecutive years; and (e) Building a loan portfolio comprising at least one third female borrowers. The project’s targets did not include performance indicators fundamental to a rural credit program, such as financial viability and profitability of branch operations, arrears and outright default on loans. Tables 1 and 2 respectively highlight the results of loan activity in Balochistan and Sindh, and FATA. TABLE 1: LOAN TARGETS AND ACTUAL PERFORMANCE – BALOCHISTAN AND SINDH Indicator/Performance Actual Result (as of 9/2007) Project Target (to 9/2008) # of Loans Disbursed 327,352 362,583 Value of Loans Disbursed (US $) 57.6 million 64.5 million # of Active Clients 94,806 103,858 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT x TABLE 2: LOAN TARGETS AND ACTUAL PERFORMANCE - FATA Indicator/Performance Actual Result (as of 9/2007) Project Target (to 9/2010) # of Loans Disbursed 1,483 51,369 Value of Loans Disbursed (US $) 245 thousand 11.2 million # of Active Clients 973 18,043 Conclusion: The project will meet its loan targets in Balochistan and Sindh, both in terms of the number of loans and the amount of funds disbursed. In FATA, KB disbursed only 3% of the target number of loans of the target. However, this low number is caused by a dangerous security situation that prevented staff from operating in the area. The project set a target retention rate of 50% of clients in the second loan cycle and 55% in the third loan cycle. In November 2007 for the project-supported branches, KB reported that the percentage of borrowers who repeat loans a first and second time is, respectively, 27% and 12%, significantly below the targets specified in the project proposal. In other words, the borrower drop out rate is over 70%, which is extremely high.11 Such high rates suggest that KB struggled to tailor loan products for customer needs in the USAID-funded branches. In fact, the demand for loans in this particular market segment is thin. As of September 2008, KB mobilized 15,683 community groups, close to the end-of-project target of 18,298. Women-only groups total 1,342. When the project began, KB relied on local NGOs to set up the groups, but later decided to make loan officers responsible for this task. Impact Evaluation question: To what extent has the project benefited the people of Pakistan? The monetary impacts of microfinance lending are notoriously difficult to quantify even with well designed impact assessments.12 This section reviews the available evidence and conclusions regarding monetary and non-monetary impacts associated with the project. The Asian Development Bank Institute (ADBI) conducted a large scale nationwide impact assessment of KB lending in 2005.13 The evaluation team did not have the resources to collect comparable data, and instead obtained the raw data from ADBI and re-estimated the models for the KB project areas (i.e., Sindh and Balochistan). The ADBI survey did not include households in the FATA. In summary, the survey of KB clients found that households with access to KB loans: • Spent more on health care, • Sought medical treatment more than others, and more likely to do so from trained professionals, • Exhibited greater measures of women’s empowerment (i.e., ability to get small amounts of cash from their own assets when necessary), • Spent more on education, and • Enrolled children in school more than other households. 11 In KB as a whole, that is, in all both donor-funded and non-donor funded branches, the borrower drop out rate is 41.2%. 12 Hussein, M., & Hussain, S. (2003). The Impact of Micro Finance on Poverty and Gender Equity: Approaches and Evidence from Pakistan. Pakistan Micro Finance Network. 13 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT xi Montgomery, H. (2005). Meeting the double bottom line: the impact of Khushhali Bank's microfinance program in Pakistan. Tokyo, Japan: Asian Development Bank Institute Publishing. The team was not able to obtain statistically significant monetary measures of lending impacts from the data. Conclusion: The team found strong evidence suggesting that micro credit produces measurable impacts on household welfare. Available data suggest that KB lending did impact a number of improved welfare indicators. However, the evidence collected was insufficient to determine if it affected household incomes or other monetary measures. Efficiency Evaluation question: How efficient has the project been in utilizing its resources to achieve results? Project expenditure data provide some measures of spending efficiency. The following is a summary of spending in several categories: • Labor accounts for 41% of project costs to date; • Administration accounts for 20% of project costs to date; • Other Direct Costs (ODCs) account for 22% of project costs to date; and • Outputs (implementation and branch refurbishment) account for 15% of project costs to date. Conclusion: There are a number of possible measures of project efficiency, but these measures require a comparison of costs to some measure of output. In this case, however, no monetary measures of benefits exist, so the team was unable to calculate the usual efficiency measures (i.e., net present value, benefit cost ratio, internal rate of return). Sustainability Evaluation question: Are the activities and results likely to be sustained after the project is completed? Successful international microfinance meets two basic criteria, which are interrelated: outreach and sustainability. Microfinance providers must meet the demand for financial services for as many micro borrowers as possible (outreach). At the same time, providers must provide services efficiently over time (sustainability). To accomplish both, providers must lower explicit transaction costs as well as lessen dependency on outside funds, whether from the government or donors. The team found evidence that suggests KB viewed outreach and sustainability as contradictory objectives before the project began. For example, the bank tried to target loans to as many new clients as possible. (This reveals a loan “supply-push” policy typical of financial entities influenced by pressures from government authorities). Yet bank authorities allocated a very high proportion of funds to safe, high-yield treasury bills, which contradicts the objective of increasing outreach. At the time of this evaluation, the team found an even division of loan fund allocation and treasury bill allocation (50% each). This particular policy underlines the practice of healthy, cautious banking.14 Furthermore, KB’s loan policy itself contains contradictions between outreach and sustainability. The high drop out ratio suggests that KB’s drive for a “supply-push” of loans caused a focus on gaining new customers instead of finding ways to retain existing ones. Therefore, this did not lead to institutional sustainability. Instead, KB made decisions likely to bear quick measurable results on important indicators (such as the number of new clients incorporated into formal finance through loans). Conclusion: The project was not designed or implemented with a focus on the achievement of long-term sustainability (the self-financing of operational costs after project completion). For example, the project did not develop a strategy for deposit mobilization. 14 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT xii Yet, a 50-50 liquid asset composition between loans and financial investments is probably overly cautious, definitively too conservative. In successful micro finance intermediaries, fund allocation to loans is no less than 70%. At the very least, microfinance best practices include a loan pricing policy based on cost-recovery basis, an incentive-pay system for loan officers, flexibility and responsiveness to client situations in regard to payment schedules, and incentives to borrowers for early repayment. Because the project has not implemented these principles, significant financial improvements in the projected-support branches are unlikely. In some branches, the project did not consistently enforce credit discipline. These flawed procedures caused a perception among prospective clients that KB is a typical state-owned entity that condones debts. Replication Evaluation question: To what extent can the activities and results of the project be replicated? USAID can replicate the project only in areas with direct evidence of private microfinance providers not operating there. With 770 branches, Punjab is by far the province with the largest concentration of bank offices in the country, with almost 62% of the total. Sindh has significantly less with 269 branches, which is equivalent to 22% of total bank offices. Branches exist in every district of these two provinces. In contrast, Balochistan, NFWP, and FATA have a much different banking situation. In Balochistan, seventeen districts have no formal banking presence at all. Four districts in the NWFP and five of the seven agencies of FATA also lack banking services.15 16 KB could presumably work in these districts. (Because of the high-risk security situation in FATA, the project is performing poorly. The current security and development conditions in FATA may mean the project cannot achieve any objectives in this area, or the project may need to adjust the design.) The team bases the assumption of successful replication on whether the project addresses current major design flaws and implementation problems. Conclusion: USAID needs to revisit the project’s design before considering replication activities. Additionally, USAID needs to examine plans to continue activity in FATA given current project experience and the difficulty of accomplishing project goals in an area where security is likely to hamper project performance. Gender Evaluation question: To what extent has the project benefited women? Table 3 summarizes project results in the delivery of loans to women. TABLE 3: LOAN PERFORMANCE – GENDER Value of Loans # of Loans # of Active Result/Indicator Disbursed Disbursed Clients (US$) Actual Result (as of 9/2007) 328,835 57.9 million 95,779 Women 50,415 7.2 million 10,862 Percentage Participation of Women (%) 15.3 12.4 11.3 The stated KB loan policy places equal weight to men and female customers. The total number of women borrowers is 50,415 as of September 2007, which is equivalent to 15.3% of all project clients. This result is significantly below the 33% target specified in the project proposal. Even more significantly, the actual 15 Source: Micro Watch, Issue 4 (April – June 2007) 16 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT xiii The assumption is that the security situation will improve significantly in these districts, most notably in those located in NWFP and FATA. result is considerably below the proportion of women borrowers in the overall KB loan portfolio (slightly above 50%). Clearly, the project has not been effective in reaching women. If the project employed more female loan officers, better results would be possible. Currently, the total number of female loan officers is only 9 out of 151 in the entire project; that is, barely 6%. Overall, in non-project supported branches, women comprise 30% of the loan officer workforce. KB attributes this lack of effectiveness to the difficulties inherent in the recruitment and placement of female loan officers in rural and marginalized areas. Deeply entrenched cultural biases against women create obstacles for the hiring of women staff. Conclusion: KB, as an institution, does try to reach women clients, but KB has not met its targets for lending to women (33%) in USAID supported branches. Reporting Evaluation question: Have the prime contractors and grantees reported on time and in a useful manner? Conclusion: KB submitted reports in a timely manner, and followed branding requirements. Communications and Outreach Evaluation question: How effective has the project been in getting its story out? Conclusion: After joining the Pakistan Microfinance Network (PMN), KB’s image is gradually improving. It also helps that the bank president is publicly recognized as a competent, well-intentioned professional. However, the bank’s reputation will ultimately be defined by its ability to achieve long-term sustainable growth. Coordination Evaluation question: How effectively has the project coordinated with other parties? Conclusion: When KB interacted more directly with industry stakeholders, the project achieved positive results. Most importantly, KB decided not to subsidize loan rates. As a result, microfinance banks now penetrate market segments in these areas. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT xiv RECOMMENDATIONS First, USAID should renegotiate the terms of the contract with KB for its Balochistan and Sindh operations, due to be terminated in September 2008, and for those in FATA, scheduled to end in September 2010. For the Balochistan and Sindh branches, the Mission should stop financing branch operations. The current activity leads to increased donor dependency. Instead, USAID should offer a package of technical assistance to bring KB up to date with the principles and procedures of microfinance best practices. Second, at a minimum, the USAID-funded support should include training and technical assistance in the following areas: • Market assessments • Loan product development • Loan product pricing • Development of savings and other financial instruments • Corporate governance • Human resources policy development • Financial management Third, the activities in the FATA region should be suspended until the overall security situation improves significantly. Fourth, USAID should engage in discussions, and support efforts that could lead to the complete privatization of KB. USAID can consider an offer to extend the aforementioned technical assistance to the eventual new owners and managers. Fifth, USAID should revisit the project’s performance monitoring system to develop improved performance measures on the quality of the loan portfolio, sustainability, cost-recovery measures and gender accomplishments. LESSONS LEARNED First, to improve prospects for success, microfinance projects implemented by public banks need to embrace and implement microfinance best practices. Otherwise, the prospects of failure are high. Second, at times, geopolitical and economic circumstances dictate the need to launch projects rapidly. However, projects need well thought-out design if they are to succeed in the long run. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT xv ANNEX 1: DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 1 I. INTRODUCTION In August 2007, the United States Agency for International Development (USAID) commissioned Management Systems International (MSI) to conduct the evaluation of eight projects within its Pakistan Economic Growth portfolio of activities. Three projects corresponded to activities that seek to increase access to financial services for the benefit of households and micro and small businesses in the rural and urban areas of Pakistan (Widening Harmonized Access to Microfinance (WHAM), Enterprise Development Facility (EDF) and Developing Non-Bankable Territories for Financial Services). The KB project started operations in October 2003 and ends its Balochistan and Sindh operations in September 2008, and its FATA operations in September 2010. Total project budget is $14.1 million, 78% of which was funded by USAID. Similar to EDF, the beneficiary agency directly implements the project’s contractual arrangement. Donor interventions in finance-related projects are entirely justified when they propel the development of financial markets. A robust and efficient financial sector helps mobilize savings, foster productive investments, and support balanced economic growth. Since political independence sixty years ago, Pakistan did not create these conditions to make all this possible. Starting in th early to mid 1990s, however, the Government of Pakistan (GOP) introduced important financial reforms that are gradually contributing to higher rates of economic growth. This included enacting measures to strengthen the microfinance sector through the active involvement of the State Bank of Pakistan (SBP) in policy making, supported by the establishment of a consultative group drawn from industry stakeholders. For example, in 2001 the Microfinance Institutions Ordinance in 2001began to regulate micro lending operations and establish rules for the creation of specialized microfinance banks. e 17 18 Prior to this, the SBP used a special ordinance to establish KB in August 2000. The creation of the bank originated in a public-private partnership initiative that sought to improve timely access to financial services for the benefit of low-income households and micro enterprises, especially those located in rural areas. Prompted by the GOP, public and private commercial banks contributed equity. In addition, end￾user credit received funding from $70 million of a $150 million loan from the Asian Development Bank (ADB). Currently the issued, subscribed and paid-up capital totals $28.4 million, and the state-owned National Bank of Pakistan (NBP) owns the largest stake in KB with 23% of the shares ($6.6 million). Fifteen private domestic and foreign banks hold the remaining shares, most importantly Habib Bank Limited and Muslim Commercial Bank Ltd. They each contributed $5 million (18%), and the United Bank Limited and the Allied Bank of Pakistan Ltd., which each contributed $3.33 million (12%). At the time of this evaluation, the government was conducting high-level discussions to sell off the NBP shares, which turns KB into an entirely private-owned commercial bank.19 17 Poverty Reduction Strategy Paper [2001]. 18 KB informed the Evaluation Team that, effective April 1 2008, the operations of KB are regulated by the Microfinance Ordinance of 2001. 19 KB informed the Evaluation Team that NBP intends to divest no later than June 30, 2008. If and when this happens, KB will be completely privatized. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 2 Headquartered in Islamabad, KB operates with 1,870 employees in 113 branches in eighty eight districts of the country. When KB began, the government had a high priority in sending resources to poverty￾stricken areas. Responsible for financing community infrastructure projects, KB also conducted other typically non-core activities, such as granting scholarships to raise technical skills of prospective clients and providing business development services to customers. KB policy equally targets men and women. At the end of 2006, the number of active borrowers was 236,917, of which 120,715 were women.20 However, for the USAID-funded project, loans to women fell way below bank targets (33%) and actual performance (50.9%) in the country at large. By offering almost a third of total financial outreach in Pakistan, KB is a significant presence in Pakistan’s microfinance industry. KB significantly contributed to the impressive rate of clientele growth (50% and above) achieved by the microfinance industry in the last three years. By the end of 2006, KB’s indicators for operational and financial sustainability were 75% and 57% respectively.21 II. THE DEVELOPMENT PROBLEM A. Problem Statement Pakistan has both highly sophisticated financial services and credit-granting non-government organizations (NGOs) and informal providers of credit that cater their services to customers perceived as high risks or too poor. In other words, commercial banks see these customers as “non-bankable.” According to the SBP, the total number of clients who obtain loans from formal sources of finance barely surpasses 5 million, and micro borrowers make up 20% of those. KB, in combination with NGOs and a handful of specialized banks, extends credit to approximately one million micro borrowers, which is equivalent to approximately just 5% of potential micro loan demand.22 This means financial outreach to the “non-bankable” customer is low. Vast numbers of households and individuals have the potential to demand micro loans, and currently they obtain credit from informal sources such as money lenders, suppliers, traders, shopkeepers, relatives and friends. Many areas of Pakistan have unfavorable conditions for developing banking services. These areas have: (a) very low population density; (b) poorly maintained roads, outdated electricity grids, undeveloped communication networks and other problems that underline the presence of low-quality of economic infrastructure; (c) lack of security; (d) reliable information on potential customers that is not easily accessible; (e) low-quality of education and population incomes that are low and volatile; and (f) cultural traits that reinforce resistance to social change. Since the early 1970s, successive governments have tried to reinforce state presence and deliver credit to the marginalized populations through the Agricultural Development Bank (ZTBL). In addition, the GoP supports rural support programs (RSPs) and multipurpose NGOs that develop the rural economy through the operation of social and credit programs. Despite these efforts, 80% of the rural population lives with an income of $1/day or less.23 20 Pakistan Microfinance Review [2006] 21 Ibid. On the other hand, KB’s own calculations of operational and financial sustainability are 74.38% and 69.28% respectively. 22 Shorebank International has estimated that the market size of micro credit comprises 7 million households and no less than 10 million individuals. Within poverty bands, these are classified as transitory vulnerable and transitory poor. 23 Consultative Group of Assistance to the Poor [2007]. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 3 B. USAID Intervention USAID began this project right after the Mission reopened activities in Pakistan. This meant the project needed high visibility as quickly as possible. To increase economic opportunities for the poor, in October 2003, USAID entered into partnership with KB to launch a project with the following expected results: (a) Increased access to micro credit and micro finance services in the rural economy; (b) Expanded access to quality education in business and agriculture for the poor; and (c) Increased market-based opportunities for the rural economy. Mainly, the project funded the operations of branches in districts of Balochistan, Sindh, and FATA in the following districts: • Ten in Balochistan: Quetta, Loralai, Pishin, Zhob, Sanjavi, Barkhan, Sibbi, Jafferabad, Naseerabad and Gawadar. • Ten in Sindh: In the first year of operations, Jacobabad, Dadu, NasheroFeroze and Larkana. During the second year of the project, six additional districts of Sindh were added: Sukkur, Khairpur, Hyderabad, Ghotki, Nawabshah and Sanghar. 24 • Two agencies in FATA – Khyber and Khurram. Project funds supported the following activities by bank staff in the target branches: • Improving credit delivery systems. • Using innovative marketing strategies to lure prospective customers into the use of formal banking services. • Launching communication strategies aimed at improving financial discipline of potential clients. • Providing business development services to clients so as to stabilize their incomes and, consequently, mitigate credit risks. • Building of the KB image as a trusted, financially strong and friendly bank that wants to help the rural poor to improve their incomes and reduce their debt. • Enhancing the institutional capacity of the bank to turn it into a prime practitioner of microfinance best practices. • Providing higher education opportunities to the qualified poor from the project area so as to build future KB local capacity to support banking operations. 24 Two more FATA agencies -- Bajaur and Mohmand – are to become operational in 2008 with USAID support. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 4 The FATA Region Project documents describe FATA as “a rugged mountainous terrain spread over 27,220 square kilometers. The area is a 1200-kilometer narrow track with its breadth varying between 60-80 kms, bordering with Afghanistan. According to the 1988 census, it has a total population of 3.2 million, of which 97.3% is rural, with a predominantly Pashtun ethnicity. Population density averages 116.7 persons per sq km. Administratively it is directly controlled by the federal government, through the governor of the North West Frontier Province (NWFP), and is constituted by seven regions called ‘agencies’, each governed by a political appointee. The agencies are: Bajaur, Khyber, Kurram, Mohmand, Orakzai, North and South Waziristan.” “During last three decades, the entire region has gone through great turmoil, in part due to the situation across the border in Afghanistan. The ongoing crisis in this neighboring country has seriously affected the lives of the people of FATA and has contributed adversely to the rampant levels of poverty, illiteracy and under development. The problem is greatly compounded by geo-economic factors – the harshness of the terrain provides limited economic opportunities for the majority of the people, who continue to live in abject poverty.” “Recent developments are bringing a rapid change in the geo-political environment in and around FATA. The institutionalization of the democratic process in neighboring Afghanistan is ushering in more indigenous development focus with international support. In addition, the GOP is substantially increasing funding for health, education and infrastructure development.” “It is envisaged that these programs, provided they are executed with transparency and efficiency, can unleash an era of new economic opportunity for the people of the region. The larger question, however, is if the geo political problems that give way to the generalized insecurity that permeates the region can subside significantly before the public investments materialize.” Source: KB Project Implementation Plan [2007] USAID funding totals $11.05 million. KB contributes $3.08 million, which gives a total project budget of $14.1 million. Of this sum, KB allocated $9.37 million to support branch operations in the provinces of Sindh and Balochistan, and $4.76 million in FATA. As noted, in Sindh and Balochistan, the project will run until September 2008, and for the activities in FATA until September 2010. The USAID intervention took significant risks. Most importantly, USAID risked working in an environment where key industry stakeholders assumed KB operated like a typical state development-bank with little managerial autonomy and insufficient emphasis on self-sufficiency. In actuality, state-owned financial entities offer successful financial intermediation. See Box 2 for examples of these entities and successful features. Indeed, the GoP modeled KB after the Bank Rakyat Indonesia (BRI), a state-owned bank. Since the mid to late 1980s, this bank successfully provided financial services to millions of Indonesia’s low-income rural inhabitants in a financially sustainable manner. State development banks find legitimacy in market segments that are shunned by private commercial banks. Explicitly or implicitly, they pursue a developmental agenda with a social mission defined by political objectives. As an added challenge, these banks must incorporate these political objectives in a corporate strategy that must strictly abide by market-oriented rules and the operational decisions that guarantee financial self-sufficiency. Many public finance entities fail to achieve this. In Pakistan, state activism dominated economic policy well into the early 1990s. The evaluation team identified factors to clarify KB’s own efforts in meeting this particular challenge. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 5 Successful State Development Banks Latin American and Asian have many examples of state development banks that efficiently extend financial services to micro and small borrowers. Chile’s Banco Estado is a stellar performer. Established in 1851, Banco Estado is a state-owned bank that provides financial services to under served clienteles, including small and medium sized enterprises (SME) firms. It operates as a typical commercial bank but with financial technologies tailored to the demand profile of micro, small and medium enterprises both in urban and rural areas. The bank provides services in a rapid, personalized, and flexible manner, generally with the best interest rates available. The bank operates with a profit, and does not depend on government funding. Significantly, the bank mobilizes large volumes of savings from large and small depositors alike. Thailand’s Bank for Agriculture and Agricultural Cooperatives (BAAC) is a government-owned bank established in 1966 whose policies and operations are controlled by the Ministry of Finance. Indonesia has the most impressive examples, such as the Bank Kredit Kecamatan (BKK) and Bank Kredit Desa (BKD), both owned and operated by provincial government, and, most notably, the Bank Rakyat Indonesia (BRI), the oldest and one of the largest government-owned banks in the country. Until the early 1980s, BRI operated mainly as a “faucet” and not as a bank. BRI delivers subsidized credit from the central government to its target groups. In 1984, BRI introduced small-scale banking activities through the unit desa network, consisting of more than four thousand branches and sub-branches, scattered throughout the country. Each unit desa is an independent profit center that is run with a very small cadre of administrative employees and loan officers. This model follows a commercial approach to banking, that is, its operations are client-tailored, credit is not targeted, and loan and savings rates are set above costs and inflation. By the mid 1990s, the BRI achieved impressive results: low arrears growing loan portfolio, composed mainly of credits to small rural customers and growing savings accounts, which surpassed the volume of loans outstanding by a factor of two. Ten years after the sweeping institutional reform, the bank posted profits. Remarkably, the unit desa network weathered the financial crisis that swept the Asian economies in the late 1990s very well. III. EVALUATION METHODOLOGY The evaluation team consisted of MSI staff member Jorge L. Daly, Team Leader, assisted by Aroona Kamal and Sarah Tirmazi, independent consultants. Ms. Kamal and Ms. Tirmazi collaborated in conducting interviews with stakeholders, branch managers and selected borrowers. They also assisted in data collection and analysis. Douglas Krieger led the tasks centered in efficiency and impact analyses, and Maliha Hussein, a microfinance specialist, reviewed the draft for content, quality, and contextual accuracy. The evaluation tasks began with the arrival of the Team Leader in Islamabad on October 12th 2007. Because his arrival coincided with the Eid Holidays, effective kick-off of activities started on October 17th. Ms. Kamal was immediately available and Ms. Tirmazi joined the team in late February 2008 to conduct the visits to the FATA branches. The Team Leader returned to Washington DC on November 20th after conducting a partial data collection in Islamabad, Karachi and Lahore, and Quetta, and assisting in the survey design for impact analysis. Efficiency and impact analyses were conducted from February 18th, 2008 through March 16, 2008. On November 19th, the team presented preliminary findings to USAID before the Team Leader returned to Washington DC. A. Evaluation Purpose and Questions This evaluation conducts the final assessment of the KB Project in its Sindh and Balochistan operations, and a formative/midterm evaluation of project operations in FATA. For the Sindh and Balochistan provinces, the evaluation attempts to determine the summative effect and impact of the project. In the case of FATA, the evaluation seeks to inform USAID decisions for further operations. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 6 The evaluation provides answers to both general and specific questions. The general questions are the following: • Relevance: How well was the KB Project focused on the needs of the beneficiaries? • Effectiveness: Has the KB Project accomplished its objectives? • Efficiency: How efficient has the KB Project been in utilizing resources to achieve intended results? • Impact: To what extent has the KB Project benefited the people of Pakistan? • Sustainability: Are activities and results likely to be sustained after the KB Project is completed? • Replication: To what extent can the activities and results of the KB Project be replicated? • Gender: To what extent has the KB Project benefited women? • Reporting: Has the KB Project reported in a timely and useful manner? • Communications and Outreach: How effective has the KB Project been in getting its story out? • Coordination: How effectively has the KB Project been in coordinating with other parties? Specific questions of this evaluation included the following: • How many loans has the project generated? • How many loans have been disbursed in each quarter since Khushhali Bank started operations in the region? (by region and gender) • What are projections for loan disbursements in the future? (by region and gender) • How many loans would have been generated without the project? • How many loans would have been disbursed without USAID intervention over the life of the project and into the future? (by region and gender) • What were the monetary and non monetary impacts of the loans on recipient households? • What impact have loans had on the income of recipient households? (by region and gender) • How were the loans used and what impact did it have on household wellbeing? • Have the loans generated any secondary impacts as a result of increased income? (by region) • How many total and new clients did Khushhali Bank have in each quarter? (by branch) • Are the new branches profitable? • Are they operating without subsidy? • How can the KB program be made more effective for women entrepreneurs in FATA? • How effective have Khushhali Bank’s training and media campaigns been in building capacity and raising awareness? DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 7 B. Evaluation Design The KB evaluation is designed to be independent, objective, and findings-based. The team answered the evaluation questions following a general approach that progresses from findings to conclusions to recommendations. The process starts with findings, which are observed and collected facts. These are “produced” during the data collection phase of the evaluation. Using analysis, interpretation and judgment, conclusions are drawn from the findings. In turn, recommendations are based on the conclusions. Recommendations identify practical actions for the consideration of project managers as the Mission reviews the potential for ongoing work in FATA. This approach gives a clear view of the evidence that supports each of the study’s conclusions and recommendations. For more details, see the “Getting to Answers Matrix” (GTA). This key evaluation design tool provides a format that allows the key evaluation questions to drive the identification of relevant data sources and data collection methods. The process of completing the GTA matrix creates the point of departure for the collection of data necessary to answer the key evaluation questions. As an example, Table 1 applies the GTA to the first two evaluation questions: relevance and effectiveness. Annex 5 contains the full GTA matrix. TABLE 1: GETTING TO THE ANSWERS FOR OVERALL EVALUATION QUESTIONS Relevance: How well was the project Evaluation Effectiveness: Has the project focused on the needs of the EVALUATION REPORT 8 Questions accomplished its objectives? beneficiaries? Type of Comparative of targets related to beneficiaries with results, Comparative of what was targeted versus felt needs, Strategic or cause and effect Quantitative comparison of targets to baseline data, if available Strategic or cause and effect Answer or Evidence Needed Method of Review of project documents, Interviews, Direct observations, Surveys, Effectiveness and impact findings Review of project documents, interviews, focus groups, direct observations, surveys Data Collection Project data, USAID and project personnel, Key informants, Beneficiaries, Evaluation findings USAID and project personnel, partners, participants, beneficiaries, observers, outside groups Data Source Knowledgeable Persons, Randomly selected beneficiaries, stratified as appropriate for the project in PEGED surveys Knowledgeable persons, random selection of beneficiaries, stratified as appropriate Selection Criteria Comparison, quantified number of beneficiaries (disaggregated) and quantified benefits as possible Data Analysis Comparisons Strategic analysis Methods C. Data Collection Sources and Methods The evaluation of the KB Project relied on the following data collection methods. • Review of relevant documents, including the project proposal presented to USAID, project implementation plans and quarterly reports. The review was conducted with the arrival of the Team Leader in Islamabad on October 12, 2007. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT • Interviews with key informants: These included (a) four representatives of the donor community; (b) one representative of a microfinance bank; (c) an official of the State Bank of Pakistan; (d) three officials of the Pakistan Poverty Alleviation Fund (PPAF); and (e) two representatives of a civil society organization – the Pakistan Microfinance Network. The interviews were conducted in Islamabad, Lahore and Karachi, during the period October 17 – November 19 2007. See Annex 3 for a list of the key informants. • Interviews with key personnel of KB: (a) the President, Operations Manager, the Manager of the USAID-funded Project at headquarters in Islamabad; and (b) regional business managers, branch managers, territory portfolio managers, relationship managers, credit specialists and business development officers of the following branches: Quetta and Pishin (Balochistan); Hyderabad, Nabawshah, Sukkur and Jaffarabad (Sindh); and Khyber and Khurram in FATA. The interviews were conducted during the period November 1, 2007 – March 2, 2008. A list of these individuals is also included in Annex 3. • Group interviews with selected borrowers that have been extended loans by the KB Project in the aforementioned branches: The interviewed borrowers numbered 69, of which 45 were men and 24 were women. See Annex 4 for a list of the composition of these interviews. D. Data Limitations In the process of collecting data, the evaluation encountered the following limitations: • The task coincided during a period of high political tension in Pakistan. Important events occurred during this time period: a) The attempt on the life of opposition leader Benazir Bhutto on October 19th and her subsequent assassination on December 27th 2007; b) the imposition of a state of emergency by President Musharraf on November 1st 2007; and c) the postponement of national elections. These events created unintended obstacles, including sudden cancellation of flights and appointments with key informants. • Given these circumstances, the team rescheduled some interviews with key informants and conducted these interviews in less time than planned. Others could not be conducted at all, notwithstanding the efforts of the team, most notably with the Ministry of Finance and the World Bank. • Prior or shortly after the start of the project, baseline data did not exist. Without information on before-project incomes, the team could not analyze the project impact with reliable measures of monetary impact. IV. FINDINGS AND CONCLUSIONS ON THE OVERARCHING QUESTIONS / ISSUES A. Relevance Evaluation question: How well was the project focused on the needs of the beneficiaries? USAID and KB, as the principal stakeholders, jointly recognized the necessity of launching a project to expand finance into underserved areas of the country, including rural localities. The intended beneficiaries were households and micro enterprises that have remained largely marginalized from mainstream finance. Many of them exist in non-bankable territories, meaning areas very difficult for the sustained extension of formal financial services. The GoP established KB with a mandate to serve these areas. KB extends at least 75% of its loans in rural areas. Findings DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 9 • The KB Project is focused on the poor, but not on clienteles that are destitute and living in abject poverty. In 2006, KB conducted a baseline survey in Sindh and Balochistan and found the average income of respondents was $800, which is about 10% higher than per capita income in the country. Only 10% of respondents reported income below $330 (less than half of per capita income). Slightly less than half (43%) owned agricultural land and two thirds owned shops. • In Balochistan, seventeen districts have no formal banking presence. Four districts in the NWFP and five of the seven agencies of FATA are also devoid of banking services. • According to the aforementioned baseline survey, 16% of respondents had a bank account and receive loans from both formal and informal sources. Microfinance institutions provide most of the formal loans, which accounts for 37% of total respondents. However, informal sources dominate lending: 30% of respondents indicated that they receive from relatives, 19% from friends, and 7% from money lenders and the rest from other sources, including suppliers. • The ADB, the principal donor of KB, exerts limited influence in operational decisions. KB used 70% of the ADB funds allocated to loans ($48 million) for the purchase of treasury bills instead.25 • Some relevant features of project design included: (a) emphasis on the delivery of loans to beneficiaries; other financial instruments, such as savings products, which could have an important difference in the improvement of the livelihoods of the beneficiaries, were not considered; (b) setting objectives and undertaking activities that are outside of the core functions of typical commercial banking, such as funding of community infrastructure projects and the extension of business development services; and (c) a lack of focus on the achievement of long￾term operational sustainability. • According to KB top management, the project launched in areas with weak demand for loan products. At the same time, top management said that the clientele is already heavily indebted to both formal and informal sources. • KB has branch presence all over the country, operating in areas where there is a large presence of other commercial banks and financial entities. The USAID-funded project targeted districts where KB constitutes the only source of formal financial services – five districts in Balochistan and three in Sindh. This feature also applies to the two branches in FATA. • Other microfinance entities operate in the same districts as KB, such as the National Rural Support Program in Hyderaband. In this district, three microfinance entities (Pak Oman, Tameer Microfinance Bank and Kashf Foundation) started operations after KB. In Nawabshah, SAFCO and NRSP had already opened before KB began, then Tameer and Pak Oman had announced plans to open operations. Moneylenders operated in the Sukkur district and they charge 10% per month. • The KB Project offers one loan product to individuals that is collateralized by exclusively male, female and mixed groups of sizes that vary (3-5 members in urban areas, and 10-25 in rural areas). Loan size varies between $50 and $500, but 81% of them lay in the range of $166 and $320. KB set loan rates as fixed on declining balances and increased them from 10% to 20% in 2005. • KB did not design any other loan products, such as ones for education, emergencies, housing and the like. Branch managers of Sukkur and Jaffarabad express that more loan products should be 25 Information provided by the ADB. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 10 offered, including consumer loans and collateralized loans. The limit for loans is $500. According to a branch manager in Sukkur, this limit is too low and should be doubled. • The evaluation team conducted interviews during field visits with branch managers and loan officers. These interviews revealed a good grasp of clientele possibilities, a valuable understanding of economic potential, and of the problems that exist. • From group interviews with the 69 borrowers, the evaluation team learned that nine obtained loans from credit-granting NGOs and five expressed a preference for individual non￾collateralized group loans. They felt that group loans jeopardized the eligibility for everyone if only one member defaulted on a loan. These five borrowers had loans for $500. • The ten borrowers that expressed preference for larger loan sizes were in the third or forth loan cycles. Consequently, they developed a credit history with KB, and had gained confidence in their ability to repay installments in time. • Seventeen of the 69 interviewed borrowers expressed a strong preference for access to key infrastructure, especially water, over access to financial services. • Twenty seven borrowers revealed capacity to generate savings and expressed demand for savings products; most of this group stated that they cannot find convenient outlets to safely place their deposits, so they keep them at home. Bank visits for deposits are also difficult because of the distance to the bank and lack of adequate transportation. • The State Bank of Pakistan conducted a KB survey, which shows that the clientele does have capacity to save. The average deposit size is $333, which approximately doubles the average loan size. SBP also revealed that some credit-granting NGOs have succeeded in capturing deposits from the clientele and placed them in commercial banks. • At the moment of conducting the evaluation, the KB Project had not designed nor launched savings products. KB top management informed the evaluation team that the bank was designing a savings product that would be available soon. Quetta, Sukkur and Jaffarabad branches strongly emphasized the need for savings instruments. Conclusions First, the KB project does address an important existing need of target beneficiaries. Seventeen districts in Balochistan and five agencies in FATA have no formal banking services at all. When residents of the target districts do obtain loans, they are likely to come from informal (and usually more expensive) sources, including relatives, money lenders and friends. Expanding the options available for households and microentrepreneurs in these areas is critical. Second, partnering with KB did meet USAID’s objective to expand the provision of financial services to underserved areas. However, the project was hastily designed with insufficient focus on the goals. For example, the design included activities directed at facilitating technical transfer and business information to raise productivity and sales revenues of micro enterprises, prior to the extension of financial services. These activities are most relevant for technical assistance providers, and not actual banks. Third, the project placed emphasis on the delivery of a single loan product. This assumed that the market (demand) would follow the product (supply), and not the other way around. In doing so, the project overlooked other financial services, particularly savings instruments, that better match the full set of needs of the target clientele. Fourth, regarding credit services, KB did not design variations of loan products, such as for education, emergencies, housing and the like. In particular, KB did not develop a savings product or a way for DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 11 branch managers to give flexibility on loan repayment schedules and on increasing the limits for second and third loans. B. Effectiveness Evaluation question: Has the project accomplished its objectives? Findings: The evaluation team reviewed KB documents that revealed top management was well aware of the monumental challenges. KB faced institutional problems associated with rapid start up, including set up management systems. In addition, KB faced skepticism from industry stakeholders, including the commercial banks that contributed capital. KB management also faced pressures from the government to show quick results in non-bankable areas. Not surprisingly, KB had a rocky start. Operations in Balochistan provide a useful illustration of the bank’s difficulties, well before it entered into an agreement with USAID. After a year of starting operations in this province (three years before partnering with USAID), KB had high operational costs. Also, KB had a large number of non-performing loans. In this context, KB approached USAID for support to operate more effectively in non-bankable areas of Balochistan, Sindh and FATA. USAID wanted to collaborate with KB to establish a firm foundation for further expansion of financial operations into other highly volatile, risky, yet underserved territories. It is important to note that the project does not provide funding for loan activity. Instead, the project finances the establishment of branches and their operational costs in selected districts of Balochistan, Sindh and FATA. The team found no evidence that USAID actively set targets for project activities even though KB proposed them in a document submitted to USAID. Furthermore, a useful performing monitoring plan (PMP) for the project was never developed. Project Targets: Project targets reflect a multi purpose approach to banking, one that is, unfortunately, not conducive to successful long-term sustainability of commercial bank operations. These targets included: (a) Disbursing 363,000 new loans in Balochistan and Sindh, and 51,000 in FATA; (b) Retaining 50% of clients in the second loan cycle and 55% in the third loan cycle; (c) Mobilizing 18,298 community groups; (d) Stabilizing the incomes of 10% of KB borrowers who have taken loans for three consecutive years; (e) Awarding 80 education scholarships – 50 in Balochistan and Sindh and 30 in FATA – with the intention of employing graduates in KB; and (f) Building a loan portfolio consisting at least of a third of female borrowers. The project’s targets did not include most performance indicators considered fundamental to a rural credit program, such as financial viability and profitability, and arrears and outright default on loans. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 12 Total Disbursements: Figure 1 shows the disbursements of USAID-supported KB branches, between September, 2003 and January, 2007. The chart shows that over the period of project operation average quarterly disbursements varied significantly, but have generally trended upward.26 FIGURE 1. USAID BRANCH DISBURSEMENT TRENDS: TOTAL QUARTERLY DISBURSEMENTS IN PAKISTAN RUPEES Active Client Trends: Figure 2 shows the number of active clients of USAID-supported KB branches, between September, 2003 and January, 2007. The chart indicates that over the period of project operation the number of active clients has held fairly steady, with the exception of a steep increase during the quarter ended in September of 2007, the last quarter for which data are available.27 FIGURE 2. ACTIVE CLIENTS AT USAID-SUPPORTED BRANCHES: NUMBER OF ACTIVE CLIENTS 26 Source: KB Project Implementation Reports. 27 Source: KB Project Implementation Reports. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 13 Table 2 summarizes actual performance against targets of loan activity in Balochistan and Sindh and Table 3 depicts the results in FATA. Table 4 is a consolidation of overall project performance. Figure 3 shows the cumulative number of loans disbursed over time. TABLE 2: LOAN TARGETS AND ACTUAL PERFORMANCE – BALOCHISTAN AND SINDH Indicator/Performance Actual Result (as of 9/2007) Project Target (to 9/2008) # of Loans Disbursed (cumulative) 327,352 362,583 Value of Loans Disbursed (US $) 57.6 million 64.5 million # of Active Clients 94,806 103,858 TABLE 3: LOAN TARGETS AND ACTUAL PERFORMANCE - FATA Indicator/Performance Actual Result (as of 9/2007) Project Target (to 9/2010) # of Loans Disbursed (cumulative) 1,483 51,369 Value of Loans Disbursed (US $) 245 thousand 11.2 million # of Active Clients 973 18,043 TABLE 4: LOAN TARGETS AND ACTUAL CONSOLIDATED PERFORMANCE Indicator/Performance Actual Result (as of 9/2007) Project Target (to 9/2008) # of Loans Disbursed (cumulative) 328,835 413,952 Value of Loans Disbursed (US$) 57.9 million 75.7 million # of Active Clients 95,779 121,901 FIGURE 3. NUMBER OF LOANS DISBURSED DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 14 In Balochistan and Sindh, with 12 months of project operations remaining, the actual number and value of loans disbursed, as well as actual number of clients, reached about 90% of project targets. At current growth rates, projected accumulated results for these three indicators will easily surpass project targets in these two provinces. Table 2 illustrates that FATA is the problem. Actual results of number and value of loans disbursed represent 3% and 2% respectively of project targets at termination date. Actual results-to￾target with respect to the indicator of number of clientele fares slightly better at 5%. Though unsurprising because of the worsening security situation in this region, the performance is disappointing. KB authorities could not give guarantees on safety to the evaluation team for field visits to this region. More significantly, KB management forbids bank personnel from headquarters and from branches in other provinces to visit FATA branches.28 FATA operations will not meet project targets at close out date. Table 4 shows how these results drag down the project’s consolidated results-to-target. KB can meet overall project targets only if loan activity in Balochistan and Sindh significantly increase, which appears unlikely given the current political instability and economic uncertainty in the business environment. In addition, a significant increase in loan activity could also have adverse effects on the quality of KB’s loan portfolio. As a more realistic goal, the project should revise these indicators downwards (e.g. 10% or less) to better reflect the reality of the project’s operating environment. Or, the project can establish individual targets from the distinct regions of operation. Regarding other performance indicators, the evaluation team reports the following findings: • Loans disbursed to women reached 50,415 clients, equivalent to 15.3% of total project borrowers. A more extensive discussion of this issue is detailed in Section V.A. • As of September 2008, KB set up 15,683 community groups, which is close to the end-of-project target of 18,298. There are 1,342 female-exclusive groups. Community groups are synonymous with borrowing groups. When the project began, KB relied on local NGOs to set up the groups, but then decided to use KB loan officers, which increased operational costs. • Because of budget constraints, KB lowered the number of scholarships for students in Balochistan and Sindh from 50 to 36. Subsequently, the targets were raised again. In all, KB granted 42 scholarships through an agreement with a university in Karachi. For FATA, KB granted 52 scholarships. • KB set up distance learning programs with a third-party, and those were scheduled to begin in early 2008. • As of November 2007, for the project-supported branches, KB reported that the percentage of borrowers who repeat loans a first and second time is, respectively, 27% and 12%, way below the specified targets. Put differently, the borrower drop out rate is over 70%, which is extremely high.29 At the very least, such high rates suggest the bank’s inability to tailor the loan product to the actual profile of the clientele, or a thin market demand for loans in this market segment. • KB cannot offer repeat loans above 20%. Put simply, the KB Project increases loan amounts for as much as 20% on each repeat loan, no matter customers’ capacity to repay larger loans. KB enforced this rule without exception. 28 While conducting the KB evaluation, the team leader was requested by USAID to review a proposal to launch a finance project focused exclusively in FATA. Findings stated in this evaluation, hopefully, should give pause to the Mission on embarking in such a risky initiative. Finance projects cannot possibly succeed in highly insecure environments. 29 In KB as a whole, that is, in all both donor-funded and non-donor funded branches, the borrower drop out rate is 41.2%. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 15 • Loan processing takes the same time (two weeks) for first and subsequent repeat loans. • Two borrowers in Quetta want KB to be flexible in rescheduling loans in time of need. For example, when riots that lasted a month in Quetta forced the borrowers to close shop, they could not sell their products to make money to pay back the loans. KB did not change the payment schedule, and consequently, borrowers had to take desperate measures – one selling assets and another receiving loans from suppliers. • Regarding individual group-collateralized loans, a majority of interviewed borrowers said they formed a group in two weeks (even though they had an advantage of knowing each other for 20 years or more). Loan processing takes another two weeks, and this includes program introduction in site visits, cash flow analysis and loan approval. In total, loan disbursement takes four to six weeks. • Loan officers collect data and do a simple verification: age, must be a business, two years of operation, they must not be in default, etc. They also determine sources of income and patterns of expenditures. They give this to a software analyst who computes a cash flow analysis, which the loan officers use to approve the loan or not. • KB funds community infrastructure projects on cost sharing basis: 80% KB and 20% community. Typical projects include water storage, community centers, irrigation pipelines, drinking water, and electricity. KB stopped this funding to concentrate on earthquake affected areas. Conclusions First, the project will meet its loan targets in Balochistan and Sindh, both in terms of the number of loans and the amount of funds disbursed. In FATA, KB disbursed only 3% of the target number of loans it had hoped. The failure to achieve loan targets in FATA can primarily be attributed to the dangerous security situation, which prevented staff from operating in the area. Second, KB takes too long to set up groups and process loans. Repeat loans are processed in the same time than first time loans. The consequence is that project effectiveness is undermined. Third, the KB Project did not meet the target of retaining 50% of clients in the second loan cycle and 55% in the third loan cycle. The percentage of borrowers who repeat loans a first and second time is, respectively, 27% and 12%. This suggests that the KB Project may have been focused on segments where loan demand is weak. It also suggests that the project may not have succeeded in tailoring products to customers’ needs and/or allowing for more flexibility in the offer of repeat loans. Fourth, lack of flexibility on loan terms sapped project relevance and effectiveness. Clients of good credit standing, tested and proven for long, should never be put in a position to sell assets to meet repayment schedules in situations that are temporary and generated by external factors. KB needed more flexibility to reschedule the loan and, in this way, protect the income-generating capacity of an excellent customer. C. Impact Evaluation question: To what extent has the project benefited the people of Pakistan? The monetary impacts of microfinance lending are notoriously difficult to quantify even with well designed impact assessments.30 Without baselines or comparison groups, the evaluation team could not 30 Hussein, M., & Hussain, S. (2003). The Impact of Micro Finance on Poverty and Gender Equity: Approaches and Evidence from Pakistan. Pakistan Micro Finance Network. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 16 estimate reliable measures of monetary impact. This section reviews the available evidence and conclusions regarding monetary and non-monetary impacts associated with the project. Table 5 summarizes the types of monetary and non-monetary benefits typically associated in the literature with microfinance lending. Evidence of these benefits, particularly the monetary benefits, is largely anecdotal. Evidence from more rigorous approaches to identifying these impacts is mixed: changes are often small, marginally significant, and inconsistent across studies. TABLE 5: PRIMARY MONETARY AND NON-MONETARY BENEFITS OF MICROFINANCE LENDING Monetary benefits Non-monetary benefits • Increased business income/profit • Increased household income • Women’s economic empowerment (for loans to women) and status in household • Increased asset value (business and household) • Improved family health and hygiene • Increased business activity (sales) • Increased school enrollment • Increased household expenditure and consumption • Improved family nutrition • Improved food security • Increased spending on education • Increased employment Findings – Current Monetary Impacts The evaluation team found no evidence that KB lending produced measurable monetary impacts for households. This does not mean that the loans had no impact, rather this means the data were not adequate to determine whether the loans had an impact on household income. The Asian Development Bank Institute (ADBI) conducted a large scale nationwide impact assessment of KB lending in 2005.31 The evaluation team obtained the raw data from ADBI and re-estimated the models for the KB project areas (i.e., Sindh and Balochistan). The ADBI survey did not include households in the FATA, but the evaluation team found the ADBI survey data useful. The survey included both treatment groups (current KB borrowers) and control groups (households that had been approved for a loan that had not yet been disbursed). Table 6 reports characteristics of the sub-sample that provided data for this evaluation. TABLE 6: SAMPLE DISTRIBUTION Size of treatment Size of comparison Province/district Total group (households) group (households) Sindh 453 521 974 Urban (Karachi) 174 219 393 Jacobabad 161 159 320 Nawabshah 118 143 261 Balochistan 141 203 344 Urban (Quetta) 47 87 134 Loralai 94 116 210 Totals 594 724 1,318 31 Montgomery, H. (2005). Meeting the double bottom line: the impact of Khushhali Bank's microfinance program in Pakistan. Tokyo, Japan: Asian Development Bank Institute Publishing. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 17 The data were collected at one point in time from both groups. They thus provide a snapshot of differences between the two groups at that point in time. This is not enough to establish impact, however, unless it is reasonable to assume that the two groups were the same prior to disbursement of loans to the treatment group. Table 7 compares the two groups on a number of characteristics. TABLE 7: COMPARISON OF HOUSEHOLDS WITH KB LOANS AND HOUSEHOLDS WITHOUT KB LOANS Treatment Control (N=568) (N=721) Household characteristics Average family size 7.66 7.33 less than 90% Percent rural 62.8% 57.7% 94% Income (Rs.) Percent with income from agriculture 34.7% 24.2% 100% Percent with micro-enterprise income 33.7% 21.1% 100% Percent with income from labor 75.2% 76.2% less than 90% Percent with income from women 42.1% 34.2% 100% Micro-enterprises 33.7% 21.1% 100% Male-run (%) 26.3% 17.5% 100% Female-run (%) 10.1% 4.8% 100% Non ag enterprise (female) (N=44) (N=29) Average asset value 4,780 4,074 90% Average annual sales 38,918 29,278 less than 90% Average annual profit 16,994 12,475 90% Agriculture (N=206) (N=175) Average annual sales 28,030 13,149 100% Livestock Average sale of animal products 17,666 13,209 90% Exenditure and consumption (annual) Average household food 47,701 47,634 less than 90% Average per capita food 7,676 7,922 less than 90% Average household non-durables 77,805 62,354 99% Average per capita non-durables 11,098 9,589 less than 90% Average household total 125,507 109,988 99% Average per capita total 18,775 17,510 less than 90% Borrowing activity Average number of KB loans/household 1.47 0 100% Average number non-KB loans/household 0.15 0.1 91% Average loan size 14169 33357 100% Average number of informal loans/household 0.11 0.13 91% Experimental group Probability that Characteristic groups are different The comparison suggests that the two groups are substantially different. Households that received KB loans (the treatment group) are significantly more likely to be rural than are households that did not receive loans (the control group). These households are also larger in size, although the difference is on the margin of statistical significance (i.e. 89%). They are more likely to have income from agriculture, micro-enterprises, and from women of the household. Treatment group households were much more likely to run micro-enterprises, both by females and males, than were control group households. Female-run non-ag enterprises had greater asset values and larger profits than those run by women in control group households. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 18 In agriculture and livestock, treatment group households had greater average sales of agricultural and livestock products. Data were not adequate, however, to reliably estimate net income from agriculture or livestock. Similarly, data were not adequate to estimate a reliable measure of net household income. Treatment group households had significantly larger gross household incomes than did control group households, but the cost structures of their income earning activities may have been very different. Treatment group households also had higher expenditure and consumption of non-durables than control group households although the difference evaporates when calculated on a per capita basis (although estimated family size differences fell just under the usual 90% threshold for statistical significance). As expected, the two groups have significantly different loan histories. Treatment group households were more likely to have obtained loans from sources other than KB but less likely to have obtained loans from informal sources (i.e. money lenders, family). The average loan size among treatment group households was also significantly smaller than for control group households. So, both groups had access to credit but KB clients had access to lower cost (i.e., KB) credit than did control group households. It could be that access to loans caused the observed differences between the two groups but without a baseline to verify they were identical before the KB loans, it is not possible to tell. Findings – Non-Monetary Impacts The ADBI report focused on measures of welfare and did not estimate impacts on income, perhaps because the data were not sufficient to calculate incomes. The present analysis generally confirms the non-monetary impacts presented in the ADBI report. Again, however, the absence of a baseline prevents clear attribution of the observed impacts to KB lending. In summary, the ADBI report found that households with access to KB loans: • Spent more on health care, • Were more likely to seek medical treatment, and more likely to do so from trained professionals, • Exhibited greater measures of women’s empowerment (i.e., ability to get small amounts of cash from their own assets when necessary) • Spent more on education, and • Were more likely to have children enrolled in school. Conclusion: The team found strong evidence suggesting that micro credit produces measurable impacts on household welfare. Available data suggest that KB lending did impact a number of improved welfare indicators. However, the evidence collected was insufficient to determine if it affected household incomes or other monetary measures. D. Efficiency Evaluation question: How efficient has the project been in utilizing its resources to achieve results? The evaluation team was unable to identify monetary measures of benefits, and therefore could not calculate the usual measures of efficiency (i.e., net present value, benefit cost ratio, internal rate of return). DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 19 How the project spent its money provides some measures of efficiency. Figure 4 presents selected expense ratios. Labor accounts for 41% of project costs to date, administration for 20%, ODCs for 22%, and outputs (implementation and branch refurbishment) for 15%. FIGURE 4. SELECTED EXPENDITURE RATIOS Conclusion: There are a number of possible measures of project efficiency, but these measures require a comparison of costs to some measure of output. In this case, however, no monetary measures of benefits exist, so the team was unable to calculate the usual efficiency measures (i.e., net present value, benefit cost ratio, internal rate of return). E. Sustainability Evaluation question: Are the activities and results likely to be sustained after the project is completed? Successful international microfinance meets two basic criteria, which are interrelated: outreach and sustainability. Microfinance providers must meet the demand for financial services for as many micro borrowers as possible (outreach). At the same time, providers must provide services efficiently over time (sustainability). To accomplish both, providers must lower explicit transactions costs as well as lessen dependency on outside funds, whether from the government or donors. The team found evidence that suggests KB viewed outreach and sustainability as contradictory objectives before the project began. For example, the bank tried to target loans to as many new clients as possible. (This reveals a loan “supply-push” policy typical of financial entities influenced by pressures from government authorities). Yet bank authorities allocated a very high proportion of funds to safe, high-yield treasury bills, which contradicts the objective of increasing outreach. At the time of this evaluation, the team found an even division of loan fund allocation and treasury bill allocation (50% each). This particular policy underlines the practice of healthy, cautious banking.32 32 Yet, a 50-50 liquid asset composition between loans and financial investments is probably overly cautious, definitively too conservative. In successful micro finance intermediaries, fund allocation to loans is no less than 70%. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 20 Findings: The evaluation team discovered these findings from project documents, interviews with key informants, and branch visits: • Five of the interviewed key informants stated that KB is not sustainable. For several years since its inception, the bank a) Neglected to pursue untapped, profitable niches; b) Did not embrace a commercial, profit oriented approach; c) Employed former NRSP employees who set up a typical NGO culture); and d) Charged subsidized loan rates lower than those charged by other operators.33 These key informants however recognized that KB is slowly working itself out of the traditional state bank profile – by improving loan repayment rates and being more careful in assessing risks. • According to the Director of Microfinance of the SBP, the SBP is trying to instill KB on the adoption of loan pricing on cost-recovery basis, and to stay away from subsidized funding. • As of October 2007, KB reported that the two agencies of FATA and 16 of the 20 branches in Balochistan and Sindh were operating at net loss after USAID cost-share. The results do not vary significantly after factoring in a $960,000 USAID grant, as only three more branches in these two regions reversed the operational losses.34 • Branches in Baluchistan have very high default rates: 1,180 out of the 1,250 active clients of the Pishin branch are in default. This branch stopped disbursements of new loans. • The Sukkur branch revealed that the KB Project is rapidly losing its market share to competitors. • In Balochistan, the KB Project does not take action against borrowers who default on their loans. The Hyderaband branch acknowledged that the project is very lenient with default borrowers. • Loan officers are not paid incentive bonuses on performance related to volume and quality of loans they generate. • Branches do not offer borrowers incentives for early repayment of loans, such as interest rate rebates or quicker approvals of repeat loans. • KB adequately applies systematized procedures to assess loan repayment capacities of borrowers, but those applied to willingness to pay are basic. Conclusions: First, this project was not designed or implemented with a focus on the achievement of long-term sustainability (the self-financing of operational costs after project completion). For example, the KB Project did not develop a strategy for deposit mobilization. Despite the clients’ poverty, the evaluation team found on field visits that customers did indeed save both in kind and cash. Without deposit facilities, they kept these savings at home. 33 One of these five key informants emphasized that KB, until 2005, was charging 10% on loan rates, or half the rate of what private operators were charging. According to this view, KB increased loan rates to 20% in 2006 after intensive lobbying from private stakeholders that participate in the Consultative Group of Microfinance. This view, however, is disputed by KB Management. 34 The four branches that report net profits without the $1 million USAID grant are Larkana, Khairpur, Nawabshah and Jaffarabad. Those that turned out operational profits after this grant are Naseerabad, Hyderabad, Ghotki. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 21 Although KB will introduce facilities to mobilize deposits, this is too late to affect the sustainability of the current project. USAID did not require deposit facilities as an object, and therefore lost an opportunity to create a local basis to support lending and to reduce outside dependency on funds. Second, KB reported an operational sustainability index at 75%, and financial sustainability was significantly lower at 57%. Regarding USAID-funded branches specifically, the evaluation team found no evidence that actual indices deviate significantly from those posted by the bank. In fact, the indices may be even lower, especially they do not include USAID subsidies and grants. Third, KB did not follow principles of microfinance best practices, including a loan pricing policy based on cost-recovery basis, an incentive-pay system for loan officers, flexibility and responsiveness to client situations in regard to payment schedules, and incentives to borrowers for early repayment. Until KB implements these measures, the projected-support branches are unlikely to experience significant financial improvements. Fourth, the KB Project did not consistently enforce credit discipline. These flawed procedures ultimately support the perception, among prospective clients, that KB is a typical state-owned entity that is prone to condone debts. F. Replication Evaluation question: To what extent can the activities and results of the project be replicated? Findings: In principle, USAID could replicate the project in any area without private microfinance providers. If the private sector is already present or about to enter an area, USAID should not provide donor support to KB operations. With 770 branches, Punjab is by far the province with the largest concentration of bank offices in the country, with almost 62% of the total. Sindh trails way behind, with 269 branches, equivalent to 22% of total bank offices. In fact, branches exist in every district of these two provinces. The situation is much different in Balochistan and, to a lesser extent, in the NFWP and FATA. In Balochistan, seventeen districts have no formal banking presence. Four districts in the NWFP and five of the seven agencies of FATA also lack banking services.35 KB could presumably enter districts to extend financial services to those customers deemed as non￾bankable by private providers.36 However, the project performed the most poorly in FATA because of the security concerns. September 2010. Without significant adjustments to the project’s design and expectations for FATA, the project cannot achieve any objectives in this area. Furthermore, USAID should not initiate replicating the project under the present regulatory regime. Private microfinance banks are regulated by the Microfinance Ordinance of 2001, whereas KB is regulated by the Ordinance of 2000.37 Although KB officials claim these two ordinances are the same, KB still has unfair regulatory advantages. Stakeholders in the donor community as well as the private sector cite the following differences: 35 Source: Micro Watch, Issue 4 (April – June 2007) 36 The assumption is that the security situation will improve significantly in these districts, most notably in those located in NWFP and FATA. 37 In late April, KB informed the Evaluation Team that, effective April 1 2008, KB operations were being regulated by the Microfinance Ordinance of 2001. The alluded unfair regulatory advantages, consequently, have been eliminated. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 22 • The KB ordinance of 2000 confers a very important advantage over private microfinance providers, insofar as it sanctions access to two development funds provided by the ADB. Lack of access to sources of commercial funds for on-lending constitutes one critical impediment that microfinance providers face. • KB is allowed to operate like a typical public bank, which translates into government support that is not available to private providers. • KB by law is not obligated to distribute dividends. • Authorization for the opening of new KB branches is granted more rapidly than that applied to private requests. Conclusion: First, USAID should study the project’s design before considering replication activities. Second, USAID should examine plans to continue activity in FATA considering current project experience and the unlikely ability to accomplish project goals where security is uncertain. V. FINDINGS AND CONCLUSIONS ON THE CROSS CUTTING QUESTIONS A. Gender Evaluation question: To what extent has the project benefited women? Findings: Table 8 provides data on loans disbursed to women in the project focus areas. The total is 50,415 clients, equivalent to 15.3% of total project clients. This gives solid evidence to how the project failed to build a significant larger portfolio composed of female borrowers, meaning at least the proposed target of 33%. Female participation on number and value of loans disbursed is also correspondingly low. Interestingly, as noted in Section I, the proportion of women borrowers in the bank’s total loan portfolio is slightly above 50%. The average loan size extended to women ($143) is equivalent to 80% of the average loan size of total activity ($176). This disparity is not uncommon in settings where females are perceived as borrowers who generate less income than males. TABLE 8: LOAN PERFORMANCE – GENDER Value of Loans # of Loans # of Active Result/Indicator Disbursed Disbursed Clients (US$) Actual Result (as of 9/2007) 328,835 57.9 million 95,779 Women 50,415 7.2 million 10,862 Percentage Participation of Women (%) 15.3 12.4 11.3 The evaluation team does not underestimate the difficulties in effectively targeting female customers in settings with a deeply entrenched cultural bias against women. Despite this, KB does seem committed to a more balanced gender distribution of its borrowers, which is evident in its overall loan portfolio. Furthermore, interviews with top managers in headquarters and branch personnel during field visits, in fact, confirmed that key personnel understand the substantial benefits from targeting women borrowers. In practice, however, KB’s commitment to gender equality needs improvement. For example, an increase DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 23 of female loan officers would very likely create significant increases in the number of female borrowers. And yet, KB employs only 9 women as loan officers out of the 151 loan officers (6%). And of these nine, seven work in branches of Sindh, a province allegedly less difficult for banking than Balochistan and FATA. According to KB management, women loan officers are difficult to recruit and place. KB believes this explains why the project failed to replicate its success in other areas. Still, quarterly reports do not highlight targets and achieved results on outreach to women borrowers. This oversight contrasts with the objectives in the Project Implementation Plan, where KB purports that “[g]ender analysis [would] remain pivotal to Khushhali Bank’s operating policies, programs and projects, the empowerment of women [would] be recognized as key to sustainable development, and adequate resources [would] be allocated to gender mainstreaming.”38 Conclusions: First, KB does try to reach out to women clients but efforts under the USAID project have not been as successful as KB’s performance in off-project branches. The project did not meet the objective of increasingly reaching women borrowers, both in absolute numbers and as proportion of the project loan portfolio. Accordingly, this project needs gender targets as project performance indicators. Second, KB could obtain better results by employing more female loan officers. Currently, they make up only 9 out of 151 in the entire project, that is, barely 6%. Overall, in branches not participating in the project, women comprise 30% of the loan officer workforce. B. Reporting Evaluation question: Have the prime contractors and grantees reported on time and in a useful manner? KB submitted the following reports, according to the reporting requirements given in its Memorandum of Understanding with USAID: • Quarterly Reports. • An Annual Report that includes an Independent Directors Report, and an Audit Committee Report, as part of a consolidated report for KB operations in Pakistan. • A Performance Monitoring Plan (PMP) was developed and submitted soon after start of activities, with key indicators, and targets by year, as well as methods for data collection. • A Project Implementation Plan (PIP) is submitted on a yearly basis. • A Benchmark Survey for Impact Analysis was completed in 2006 to establish benchmark indicators, and one for FATA is currently being undertaken. Conclusion: KB submitted reports in a timely manner, and followed branding requirements. C. Communications and Outreach Evaluation question: How effective has the project been in getting its story out? According to project documents, KB developed and implemented a set of communications and outreach activities. These consist of the following: 38 Project Implementation Reports DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 24 • Using outdoor billboards and print and electronic media spots • Creating signs for enhanced branch recognition and awareness among target audience • Using of a marketing plan, which targeted universities in Balochistan and Sindh, in order to raise awareness among educated youth in the area and promote KB’s image as a quality employer. • Planning to refurbish all 22 project branches in ways that make them more customer-friendly • Granting scholarships to build a good image among clienteles in marginalized territories In spite of these activities, loan officers and prospective borrowers determine the awareness among intended beneficiaries. The evaluation team found no evidence that the project is highlighting particular success stories. The evaluation team visited branches that displayed the USAID logo, but not prominently. Conclusion: After joining the Pakistan Microfinance Network (PMN), KB’s image is gradually improving. It also helps that the bank president is publicly recognized as a competent, well-intentioned professional. However, the bank’s reputation will ultimately be defined by its ability to achieve long-term sustainable growth. D. Coordination Evaluation question: How effectively has the project coordinated with other parties? KB coordinates with other parties in its overall activities. The bank keeps in close touch with important government agencies, such as the Ministry of Finance and the SBP. KB does have an institutional presence in the policy-making committees of SBP. Being the first established commercial microfinance bank, KB participated in discussions that led to the microfinance ordinance of 2001. KB also participated in initiatives launched by the USAID-funded WHAM project to strengthen the microfinance industry in Pakistan. For example, KB participated in the Round Table Discussion “Exploring the Middle Management Growth Needs” of February 2007 (sponsored by Shorebank, in collaboration with the Lahore University of Management Science). Also, KB assisted in the Shorebank￾led path breaking study on the dynamics of the microfinance expansion in Lahore. KB increased the prestige of PMN by becoming a member Therefore, PMN receives more operational and financial data for publication in the quarterly Micro Watch bulletins and in the Pakistan Microfinance Review. Significantly, the president of KB is the chairperson of the editorial board for this publication. Conclusion: As KB interacts with industry stakeholders, certain positive results occur. Most importantly, KB substantially increased loan rates from 10% to 20%. Microfinance banks are slowly entering the market segments where KB operates. VI. RECOMMENDATIONS First, USAID should renegotiate the terms of the contract with KB for its Balochistan and Sindh operations, which ends in September 2008, and in FATA, which ends in September 2010. For the Balochistan and Sindh branches, USAID should stop financing branch operations. The current activity leads to increased donor dependency. Instead, USAID should offer technical assistance for the principles and procedures of microfinance best practices. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 25 Second, at minimum, the USAID-funded support should include training and technical assistance in the following areas: • Market assessments • Loan product development • Loan product pricing • Development of savings and other financial instruments • Corporate governance • Human resources policy development • Financial management Third, USAID should suspend the activities in the FATA region until the overall security situation improves significantly. Fourth, USAID should be actively engaged in discussions leading to the complete privatization of KB. USAID should offer the aforementioned technical assistance to the eventual new owners and managers. Fifth, USAID should revisit the project’s performance monitoring system to develop improved performance measures on the quality of the loan portfolio, sustainability, cost-recovery measures and gender accomplishments. VIII. LESSONS LEARNED First, to improve prospects for success, microfinance projects implemented by public banks need to embrace and implement microfinance best practices. Otherwise, the prospects of failure are high. Second, at times, geopolitical and economic circumstances dictate the need to launch projects rapidly. However, projects need well thought-out design if they are to succeed in the long run. DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 26 ANNEX 2: BIBLIOGRAPHY/REFERENCES Annual Report and Accounts 2006, Khushhali Bank, Islamabad, Pakistan, 2007 Benchmark Survey for Impact Analysis in Sindh & Balochistan, A.C Nielsen, Khushhali Bank, Islamabad, Pakistan, 2006 Grassroots Jan - Mar 2007, a quarterly newsletter, Khushhali Bank, Islamabad, Pakistan, 2007 Grassroots July - Sept. 2007, a quarterly newsletter, Khushhali Bank, Islamabad, Pakistan, 2007 Human Resource Development Initiative, A Microfinance Training and Development System for Sector Scale-Up, USAID/Shorebank International, Islamabad, Pakistan, 2007 KB Ordinance 2000, Government of Pakistan, 2001 Knowledge and Attitudes of Stakeholders on Microfinance in Pakistan 2006, PMN/Shorebank, USAID￾WHAM Project, Islamabad, Pakistan, 2007 Meeting the Double Bottom Line, The Impact of Khushali Bank’s Microfinance Program in Pakistan, Heather Montgomery, Asian Development Bank Institute, ADBI Policy Paper No 8, Tokyo, Japan2005 Microcredit Financing: Assessment of Outcomes 2005, Gallup Pakistan, PPAF, Islamabad, Pakistan, 2007 Microcredit Financing: Assessment of Outcomes 2002, Gallup Pakistan, PPAF, Islamabad, Pakistan, 2004 Microfinance Performance in Pakistan 1999-2005: Growth but a Structural Flaw Persists, PMN/ShoreBank, USAID-WHAM Project, USAID, Islamabad, Pakistan, 2006 MicroWatch Jan - March 2007, a Quarterly Update on Microfinance Outreach in Pakistan, PMN/ShoreBank, USAID-DFID, Islamabad, Pakistan, 2007 A study of Informal Finance Markets in Pakistan, Adnan Qadir, Pakistan Microfinance Network, Islamabad, Pakistan, December 2005 Pakistan Country Level Effectiveness and Accountability Review with a Policy Diagnostic, CGAP, 2007 Pakistan Study: Microfinance and Branchless Banking, James Dailey and Janine Firpo, for WHAM and AMPER Projects, USAID, Islamabad, Pakistan, 2007 Pakistan Microfinance Review 2006, Shades of Growth, Pakistan Microfinance Network, Islamabad, Pakistan, 2007 Performance Indicators Report 2005, Pakistan Microfinance Network, Islamabad, Pakistan, 2005 Performance and Transparency, A Survey of Microfinance in South Asia, MIX/CGAP, World Bank Poverty and Social Impact Assessment: Pakistan Microfinance Policy, DFID/Oxford Policy Management, May 2006 Poverty Reduction Strategy Paper. 2001 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 27 Preparing Middle Management for Growth: Building Capacity & Leadership, Pete Sparreboom, Shorebank/LUMS, USAID-WHAM Project, Islamabad, Pakistan The Dynamics of Microfinance Expansion in Lahore, Hussan Bano Burki & Mehr Shah, PMN/ShoreBank, Islamabad, Pakistan, 2007 Project Documents Khushhali Bank Proposal for USAID, 2003 Cooperative Agreement, July 2007 Project Implementation Plan, YR1, YR2, YR3, Consolidated Draft, USAID, 2004, 2005, 2006 Project Monitoring Plan, PMP, USAID, 2004 Project Plan FATA Five-years, Draft, USAID, 2006 Quarterly Reports (Quarters 1st - 16th), Oct 2003 - Sept 2007 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 28 ANNEX 3: LIST OF INDIVIDUALS INTERVIEWED I. Key Informant Interviews USAID Zack Orend PPAF Ahmad Jamal, CEO Tariq Khan Baluch, General Manager, Credit and Enterprise Development Amir Naeem, Manager, Credit and Enterprise Development Shorebank Gregory Chen, Chief of Party, USAID-WHAM Project Asian Development Bank Azim Hashimi, Project Implementation Officer Pakistan Microfinance Network (PMN) Rashid Bajwa, Chair PMN/Chief Executive National Rural Support Program (NRSP) Aban Haq, Research Analyst Swiss Development and Cooperation Khalid Nawaz, Project Director, Financial Sector Strengthening Program DFID Haroon Sharif, Economic Adviser Tameer Microfinance Bank Nadeem Hussain, President & CEO. II. Staff Interviews - Khushali Bank: Corporate Office and Branch Offices Head Office Ghalib Nishtar, President, Yawer Afridi, Operations Manager Nusrat Sultana, Project Manager Regional / Branch Offices: Quetta Branch, Balochistan Shafiullah Marwat, Territory Portfolio Manager (TPM), Quetta Region Ms. Uzma Naqvi, Relationship Manager (RM), Quetta Branch Manager Ms. Rubina, Credit Specialist (CS) Business Development Officers (BDOs) - 2 Male & 1 Female and Sales Officers Pishin Branch, Balochistan Shafiullah Marwat, Territory Portfolio Manager (TPM), Quetta Region Acting Relationship Manager (RM) / Branch Manager Business Development Officer (BDO) DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 29 Jaffarabad Branch, Balochistan Mr. Niaz Ahmed, Regional Business Manager (RBM), Sukkur Region Ashfaq Hussain Luhrani, Relationship Manager (RM) / Branch Manager Business Development Officers (BDOs) - 3 Male Hyderabad Branch, Sindh Asim Anwar, Regional Business Manager (RBM), Karachi Region Subhas, Territory Portfolio Manager (TPM), Hyderabad Territory Taqi Khokar, Relationship Manager (RM) / Branch Manager Credit Specialist (CS) Business Development Officers (BDOs) - 5 Male and 4 Female Nawabshah Branch, Sindh Mohd Ishaq, TPM, Nawabshah Territory Yassir Ali, Relationship Manager (RM) / Branch Manager Business Development Officers (BDOs) - 9 Credit Specialist (CS) Sukkur Branch, Sindh Mr. Niaz Ahmed, Regional Business Manager (RBM), Sukkur Region Mr. Khurram Relationship Manager (RM), Branch Manager Business Development Officers (BDOs) – 4 Khyber and Khuramm, FATA Marwat, Mumtaz Khan Yameen, Waqar Ullah Afridi, Saqib, Bashir, Zafar Afzal, Javed DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 30 DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 31 ANNEX 4: GROUP INTERVIEWS I. Community / Group Interviews Quetta Branch i. Female Group: 3 members, Enterprise Development, at Mominabad ii Mixed Group: 2 Males and 3 Females, Enterprise Development, at Rehmat Colony iii. Male Group: 2 members, Enterprise and Livestock at Huda, Quetta Urban Service Center staff interview - Female Service Center Officer (SCO), and Business Development Officers (BDOs). Also present TPM, Relationship Manager (RM), and BDOs Pishin Branch i. Female Group: 4 members, Enterprise Development at Christian Colony Also present TPM, Relationship Manager (RM), and BDO Hyderabad Branch i. Female Group: 3 members, Livestock, Service Area Sumarchand, Matiari UC ii. Male Group: President and 9 members, Agriculture, Sachedino Barecho Village Also present RBM, TPM, RM and BDOs Nawabshah Branch i. Male Group: 3 members, Enterprise Development, Nawabshah Urban ii. Mixed group: 3 females and 1 male, Enterprise Development and Livestock, Nawabshah Urban Also present TPM, RM and BDOs Sukkur Region i. Male group: 13 Clients, Date Palm trees and Cotton, Buxan Bhambro Also present RBM, RM, and BDOs Jaffarabad Region i. Male Group: 14 members, Agriculture and Livestock, USAID funded School ii. Female Group: 3 members (and 4 potential members), Stitching and Embroidery, house of member Also present RBM, RM, BDOs and Accounts Officer FATA: 5 male borrowers ANNEX 5: GETTING TO ANSWERS MATRIX KB PROJECT Type of Answer Evaluation Questions or Evidence Method of Data Data Analysis Needed Collection Data Source Selection Criteria Methods 1. How many loans have been disbursed in each quarter since Khushhali Bank started operations in the region? (by region and gender) Counts of loans by quarter Review of bank records Interview with bank staff or request to bank staff Describe pattern of loan disbursements over life of project to date by region and gender 2. What are projections for loan disbursements in the future? (by quarter, region, and gender) Counts of loans by quarter Interview Interview with bank staff or request to bank staff Projections of future loan disbursements by region and gender 3. How many loans would have been disbursed without USAID intervention over the life of the project and into the future? (by quarter, region, and gender) Counts of loans by quarter Interview Interview with bank staff or request to bank staff Counterfactual projection of pattern of loan disbursement without USAID involvement by region and gender 4. What impact have loans had on the income of recipient households? (by quarter, region, and gender) Comparison of income with and without loans Interview Survey of loan recipients Interview with bank staff or request (cash flow) Stratify sample to obtain sufficient responses from different regions and from women. Estimate average household impact by region and gender 5. How were the loans used and what impact did they have on household wellbeing? Descriptive Surveys Survey of loan recipients Stratify sample to obtain sufficient responses from different regions and from women Narrative description of impacts on households or individuals business or family/personal wellbeing 6. Have the loans generated any secondary impacts as a result of increased income? (by region) Aggregate increases in incomes by quarter and region Drawn from existing studies Existing multipliers if available Apply multipliers to estimate additional income or jobs created and increased income DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 32 Type of Answer Evaluation Questions or Evidence Method of Data Data Analysis Needed Collection Data Source Selection Criteria Methods 7. What are the likely secondary impacts of the project? Probe beyond the immediate impacts. Descriptive Interview Project staff Narrative descriptions Key informants Partner organizations 8. Have there been unintended benefits or costs associated with the project? Descriptive Interview Project staff Narrative descriptions Key informants Partner organizations 9. How many total and new clients did Khushhali Bank have in each quarter? (by branch) Comparison Document review Project documents or request to bank staff Trend in new clients and total client base over time. 10. How can the KB program be made more effective for women entrepreneurs in FATA? Descriptive Interviews Key informant interviews Survey of women loan recipients in FATA Narrative 11. How effective have Khushhali Bank’s training and media campaigns been in building capacity and raising awareness Comparison Document review Project brand awareness survey Increases in brand awareness DEVELOPING NON-BANKABLE TERRITORIES FOR FINANCIAL SERVICES PROJECT EVALUATION REPORT 33