PROJECT HOPE ALLEVIATING THE EFFECTS OF POVERTY ON WOMEN AND CHILDREN: An Expansion of Project HOPE's Village Health Bank Approach Final Evaluation Report Cooperative Agreement FAO-A-00-98-00028-00 September 1, 1998 – August 31, 2003 Prepared by: Frank Sullivan Locus Grove, VA Submitted to: U.S. Agency for International Development BHR/PVC Matching Grant Program Washington, DC Submitted by: The People-to-People Health Foundation, Inc. (Project HOPE) Millwood, Virginia 22646 Tel: (540) 837-2100 Fax: (540) 837-1813 August 2003 TABLE OF CONTENTS 1.1 Evaluation Profile Sheet.....................................................................................1 1.2 Summary of Conclusions and Recommendations............................................2 1.3 Program Background ........................................................................................6 1.3.1 Brief history of the Matching Grant program.......................................................6 1.3.2 Status of interventions at the beginning of the program.......................................6 1.3.3 Rationale for the program.....................................................................................7 1.3.4 Midterm Evaluation..............................................................................................7 1.4 Program Effectiveness........................................................................................8 1.4.1 Program Model .....................................................................................................8 1.4.2a Achievement of Objectives...................................................................................9 1.4.2b Impact on Target Populations: Further Discussion............................................19 1.4.3 Cross-cutting Issues ............................................................................................21 1.4.3.1 Partnership ..........................................................................................................21 1.4.3.2 New Tools, Guidance or Standards....................................................................21 1.4.3.3 Sustainability and Scale-Up ................................................................................23 1.4.4 Lessons Learned and Program Recommendations .............................................24 1.4.4.1 Lessons Learned..................................................................................................24 1.4.4.2 Program Recommendations ................................................................................27 1.5 Program Management ......................................................................................28 1.5.1 Management Approach.......................................................................................28 1.5.2 Quality and Status of Detailed Implementation Plan (DIP)................................28 1.5.3 Financial Management ........................................................................................29 1.5.4 Monitoring and Evaluation (M&E) ....................................................................29 1.5.5 Information Systems...........................................................................................31 1.5.6 Staffing and Supervision.....................................................................................31 1.5.7 PVC Program Management ................................................................................32 1.5.8 Program Management Lessons Learned and Recommendations .......................32 1.6 Evaluation Methodology ..................................................................................36 1.7 Evaluation Attachments...................................................................................37 Appendix A: Approved SOW Appendix B: DIP Matrix and Results Status- Overall project and per country Appendix C: Program Strengths and Weaknesses Table Appendix D: Final Evaluation-Malawi Appendix E: Final Evaluation-Guatemala Appendix F: Final Evaluation-Peru Appendix G: List of persons/groups interviewed Appendix H: Bibliography Appendix I: VHB Photographs 4 FINAL EVALUATION ALLEVIATING THE EFFECTS OF POVERTY ON WOMEN AND CHILDREN PROJECT HOPE VILLAGE HEALTH BANKING PROJECT 1.1 Evaluation Profile Sheet ß PVO name: The People-to-People Health Foundation, Inc. (Project HOPE) ß Cooperative Agreement number: FAO-A-00-98-00028-00 ß Country program sites: Ecuador, Malawi, Guatemala, Peru ß Duration of Grant: Sept. 1, 1988 - Aug. 31, 2003 ß Beneficiary Populations: 13,000 impoverished women in three countries ß PVC-PVO match: USAID: $2.4 million; Project HOPE: $2.999 million ß PVC-PVO match funds disbursed to date: PVC: $ 2,400,000 PVO: $ 2,425,260 ß Date DIP was first approved by PVC: October 1999 ß Evaluation Start Date: June 1, 2003 ß Evaluation End Date: August 15, 2003. 5 1.2 Summary of Conclusions and Recommendations The Village Health Banking (VHB) project has had qualified success in achieving its goal to alleviate poverty and its effects on women and children. The five-year project has ‘graduated’ operations in one country, Ecuador, and effectively created a loan portfolio in three very different, start-from-scratch country settings, Malawi, Guatemala, and Peru, each of which has represented specific challenges. VHB has accomplished a number of targets at 100% or more of goal; however, under￾anticipated, heavy attrition has detracted from the project’s realizing its full potential. Operational sustainability has been achieved in one country but not yet in the other three. While achieving operational sustainability in these countries was likely an overly ambitious goal, it will probably be achieved in one, perhaps two, countries in the foreseeable future. Of a total target of 13,000 participants, the Final Evaluation demonstrates substantial improvement in health and incomes for 2,250 women, those who remain in the project four or more loan cycles, many of whom continue in the project even now. To a lesser degree, improved health and income can be inferred for other 4,000 or so women who participated a year or more during the project but who have since withdrawn. The following table summarizes overall project achievements of Strategic Objectives. Overall Goal: To alleviate poverty and its effects on women and children Proposed Results Current Status SO 1: Strengthened [HQ] capacity to implement Village Health Bank programs Formation of an Income Generation Department in HOPE HQ Accomplished Development of a common, flexible VHB methodology Accomplished Implementation of VHB in 3 new countries and 1 expansion Accomplished Greater sustainability through diversified funding sources Accomplished Increased sustainability of HQ technical services Accomplished Establishment of a program performance monitoring system Establishment of a program impact assessment system Two-thirds accomplished SO 2: Financial services and health education provided in a sustainable manner Develop local staff capacity to implement VHB methodology Accomplished Achieve sustainability of three country operations Average: 57% Serve a significant population 48%, 51%, 77% or 131% (depending upon method of calculation) Maintain high program quality [in financial terms] 49% under-achievement in arrears Loan losses controlled in 2 countries SO 3: Improved income and health of impoverished women and children Form financially viable, democratic community institutions 130% accomplished Improve economic opportunities for women (loans disbursed) Accomplished 20% increase in incomes of participants Accomplished for 2,251 women, less for another 4,046 women Increased exposure of women to health talks Accomplished Improved health status of participating women 32% accomplished 6 From the Headquarters perspective, the project has been effectively managed in most of its tangible elements. Regarding the provision of financial services and health education, local staff has been effectively trained in the VHB methodology, and a significant population has been served. Operational sustainability, a key project goal, has been achieved 58% in Malawi, 69% in Guatemala and 45% in Peru. Indicators of high quality performance in micro￾finance have been achieved at 43% of goal, in large part due to discouraging performance in Peru. Regarding improved income and health, more than 100% of targeted village banks have been opened while 65% of them remain open at time of the evaluation. More loans have been disbursed than planned also. Increased income due to the loans has been achieved for 6,297 women, more for 2,251 four-loans-or-more women still involved in the project than for another 4,046 who are no longer involved. In terms of health, the goal of 12,000 health talks has been exceeded. Improved health status for participating women has been achieved at an estimated 32% of goal. The following table contrasts project accomplishments country-by-country. VHB Accomplishments per Country Objectives Indicators Malawi Guatemala Peru • VHB policies and procedures defined for successful implementation in this area VHB model Accomplished throughout project adapted to characteristics of the country • Completed lesson plans for health and business education that is appropriate to the needs of the beneficiaries On the way in health and business On the way in health; not underway in business Well on the way in health; not underway in business • New VHBs created 164% accomplished 122% accomplished 88% accomplished Expansion of the VHB program • Women receiving loans 122% total, 53% active accomplishment 123% total, 53% active accomplishment 154% total, 32% active accomplishment • Arrears over 30 days less than 5% of outstanding loans Not accomplished Accomplished Not accomplished • Loan loss rates of less than 2% of average portfolio Accomplished Accomplished Not accomplished Maintenance of high program quality • Operational efficiency of less than 15% by grant end Accomplished Accomplished Not accomplished • New staff trained and oriented in VHB methodology Improved Accomplished throughout project institutional capabilities • Development of planning, administrative, and management systems and documentation to support a growing institution Accomplished Accomplished Accomplished but … 7 Objectives Indicators Malawi Guatemala Peru • Upgrade of the Financial Management Information System to assess economic impact on beneficiaries Accomplished throughout project • Development of an Impact Assessment System Accomplished in micro-finance but can be strengthened; not accomplished in health Accomplished in M-f; well advanced in health • Achieve a 100% level of operational sustainability; 58% accomplished 69% accomplished 45% accomplished Self￾Sufficiency • Accessing in-country resources to complement program operating and growth needs Not accomplished Partially accomplished Not accomplished • Income of participating women increased by 20% annually 20% for 755 women; less for 1,588. 50% for 829 women; less for 1,974 15% for 667 women; less for 484. Increased beneficiary impact • 70% of VHB members will be engaged in improved home-based health practices Yes for 750+ women Yes for 830+ women Yes for 650+ women In other areas: Partnership was not a specific objective of the grant but the VHB has nevertheless established a number of effective partner relationships at the country level. New tools: The project created an innovative “peer evaluation” process during the Midterm that showed promise. VHB-Guatemala created an innovative loan monitoring system that could be widely distributed; VHB-Peru created an excellent health talk monitoring tool. Sustainability seems assured through FY04 with non-grant funding provided by HOPE HQ. The Ecuador VHB has become completely independent (as projected) during the course of this grant. Scale-up is taking place in three additional countries with non-USAID funds. A number of lessons were learned. 1. The VHB goal may have been overly ambitious. 2. Substantial benefit in health and incomes has accrued to over 2,200 women, with another 4,000 who received some benefit. 3. It is important to focus on those who remain in the project four cycles or more. 4. The pace of VHB program change and organizational learning needs study. 5. High dropout rates have not yet become a major project concern. 6. The importance of savings may have been underestimated. 7. Significant improvements in Adult Education have been achieved in several countries and need to be consolidated. 8. Improved data collection in health needs to take place (and is about to). 9. Improved data collection in income will make it easier to show improved impact in subsequent years of project operation. 10. Improved loan making may become the next “systemic” improvement in VHB. 11. Operational sustainability needs to improve. 12. Several Malawi decisions need support and the program strategy in Peru needs to be thoroughly re-thought. 8 13. Leadership styles of the HOPE Country Directors (and perhaps the Program Director) may need to be re-thought. Recommendations evolve from lessons learned. 1. Project HOPE may need to scale down goal expectations in future proposals; also a need to dedicate a larger amount to grant financing in start-up operations. 2. Country staff needs to “think outside the box;” a performance indicator can be added to VHB manager annual evaluations regarding “proactivity.” 3. It is recommended that staff address disguised dropouts and establish performance targets related to reducing this. 4. Orient project staff to the economic impact of improved member savings; create a realistic program indicator; establish a portfolio target. 5. Use Peru staff as resource to consolidate gains in adult education in Guatemala and somehow even in Malawi. 6. Collect data for 1st cycle entrants in health and in income and measure improvements in both areas on all participants at the conclusion of the 4th cycle. 7. Embark on economic analysis of individual loan applications on a small number of applicants being careful not to over-tax field staff with this extra burden. 8. Mount an aggressive campaign of cost-savings to improve cost/sustainability ratio. Release supernumerary staff. HQ to provide more hands-on oversight of country budgets. 9. HQ support of the decision to continue VHB presence in Mulanje; also, HQ to push hiring an Adult Health Educator in Malawi. 10. In Peru, conceptualization and creation of a new, more viable program strategy 11. With Country Director input, the Program Director to articulate a “management philosophy” paper thinking through country leadership and portfolio quality; performance indicators for supervision of VHB to get added to C.D. annual performance appraisals. 12. Continued HQ efforts to scale the VHB up in new countries; preparation of project proposals for new donors. 9 1.3 Program Background 1.3.1 Brief history of the Matching Grant program The People-to-People Health Foundation, Project HOPE, is a non-profit, non-sectarian, a￾political organization with a mission “Helping People Help Themselves.” HOPE has been conducting assistance and development programs throughout the world since the first HOPE ship in 1960. Major recent programs have included Child Survival programs in half a dozen countries; emergency response; health education and training for physicians, nurses and other health professionals; preventative health care, policy work through the Center for Health Affairs; Village Health Bank Programs; and others. Origins of the current matching Village Health Bank (VHB) project lie in a 1992-1997 USAID Cooperative Agreement to allow HOPE to establish a village health bank methodology in Honduras and Ecuador. Significant program discussions took place in the long period between the preparation of a follow-on project proposal in December 1997 and submission of the Detailed Implementation Plan in October 1999. For FY 1998-2003, additional funding was requested from USAID and from the UK Department for International Development, DFID, and from UK National Lottery Charities Board to strengthen the project in one country (Ecuador) and expand it to three new countries where HOPE had other on-going programming: Malawi, Guatemala and Peru. A Midterm evaluation was conducted in this project in June-September 2000; this is the Final evaluation of that project. 1.3.2 Status of interventions at the beginning of the program A Final Evaluation was conducted by an external evaluator at the close of the matching grant in September 1997. The Executive Summary highlights key accomplishments: a tested, well-functioning approach was set up in two countries; both projects established viable, democratically managed village health banks and exceeded DIP targets; an effective financial management information system was established; and both programs had moved steadily toward sustainability, 109% achieved in Honduras, 77% in Ecuador. In addition, an institutionalization of the program was accomplished both at HOPE HQ and in country offices. Four Recommendations were offered. 1. Continued development of HOPE HQ’s ability to implement Income Generating projects · More definition of core principles · Develop clearer program parameters · Clarify lines of authority and supervision · Strengthen staff development. 2. Consolidation of VHB systems · Carry out country-specific financial reviews · Improve the monitoring of socio-economic and health impact · Re-assess the importance of business education modules 10 · Expand the financial management information system 3. Meaningfully integrate health education as a full project component · “Effectiveness of Project HOPE in improving health has been compromised by… limited attention to the …. health component...” · Prioritize activities to enhance health education · Clarify lines of authority for health education including adequate technical support from HOPE Center. · Develop strategies for a more comprehensive and systemic approach 4. Enhance the transition from a traditional HOPE “project” focus · Create full-blown country offices · Promote field offices into indigenous Non-Governmental Organizations · Scale up the Income Generation portfolio to more countries 1.3.3 Rationale for the program The Overall Objective of the project is to “alleviate poverty and its effects on women and children by increasing incomes and improving health conditions and personal well￾being.” This is to be achieved through the accomplishment of three Strategic Objectives: SO 1 Strengthened capacity of Project HOPE to implement successful Village Health Bank programs. SO 2 Provide valued financial services and health education in a sustainable manner. SO 3 Improve income and health levels of impoverished women and children. The Project HOPE model of Village Health Banks provides credit and health education to women, building on the traditional village banking method of providing credit to solidarity groups. The program is frequently described as “credit-plus”, meaning that health education is added to the project mix to enhance the impact of gains in income. The project is directed to the inter-relationship between low incomes and poor health influenced by poor housing and hygiene, inadequate access to health care, and limited educational opportunities. It also builds on the well-documented fact that increases in women’s incomes get spent more on health and nutrition items than increase in most men’s incomes. Project HOPE, as a health-focused institution, is particularly concerned with the project’s health impacts. The project hypothesizes a synergism between the two mutually reinforcing objectives. 1.3.4 Midterm Evaluation From July to September 2001, a Midterm evaluation was carried out on the grant with the following results. Findings regarding strengthened capacity of Project HOPE to implement successful VHB programs were that creation of an Income Generation Department, development of a common VHB methodology, and expansion of the VHB to three new countries had been 11 accomplished fully. Diversified funding and increased sustainability of HQ technical services were on target, as was creation of a performance monitoring system. Not yet accomplished was a fully functioning impact assessment system. Findings regarding providing financial services and health education were that in four countries developing local staff to implement the VHB model had been accomplished. Improved sustainability of operations was on target, as were serving a significant population and maintaining high financial quality. Findings regarding improved income and health of VHB members were that the project was providing increased economic opportunities for thousands of rural women and some improvements in income were taking place but required more study. The quality of many health talks suffered from weaknesses that were discussed in detail. Recommendations were formulated: for HOPE HQ, more involvement from the Board of Directors, creation of a Health Educator position at HQ, and strengthening the position of Vice President of International Programs. Portfolio management, though extremely strong in financial realms, needed strengthening in health. Recommendations were offered to strengthen adult education methodology in the VHB project. Country-specific recommendations were also offered for two countries the external evaluator visited during the Midterm. 1.4 Program Effectiveness 1.4.1 Program Model Two reflections are offered on the VHB program model. Succinctly, throughout the developing world, millions of women are unable to receive services from their over￾worked and under-funded Ministries of Health. When these women fall sick, there is nowhere to go that does not require sometimes hours (or half-days) of travel. Moreover, preventative public health activities are all but unheard of. The VHB model offers a way to fund well-trained primary health promoters outside of, and complementary to, the Ministries of Health to promote women’s ‘demand-driven’ search for better health services. If financial sustainability can be achieved, the model offers an effective way for women who would otherwise not receive health education to learn how to improve their lives. At the same time, it offers marginalized rural and peri-urban women an opportunity to improve their household incomes (and the health of their children), as well as to grow in self-esteem. The evaluator finds the model compelling. In places like Malawi, Guatemala and Peru where infant and maternal mortality is still quite high, the model should be made to work: it is a way tens of thousands of such women can be served. In addition, though there is debate on this topic, the evaluator views favorably the strategy of Project HOPE to create a country platform in three countries like these independent of grant funding. Growth in professionalism in career national staff and long-term organizational development are hard to achieve when a country operation is surviving from one short-term grant to another. The evaluator supports the implicit 12 ‘program philosophy’ that a successful VHB project could create a program base on which Project HOPE can build. 1.4.2a Achievement of Objectives This major section will be presented in four tables, three for HOPE HQ and one for the three countries; these latter data will be extracted from individual country Final evaluations that are attached as Appendices D, E and F. A narrative will accompany each table. SO 1 Accomplishment Table 1: SO 1 Accomplishments of the Current Project Overall Goal: To alleviate poverty and its effects on women and children Proposed Results Current Status SO 1: Strengthened capacity [at HQ]to implement Village Health Bank programs Formation of an Income Generation Department in HOPE HQ Accomplished Development of a common, flexible VHB methodology Accomplished Implementation of VHB in 3 new countries and 1 expansion Accomplished Greater sustainability through diversified funding sources Accomplished Increased sustainability of HQ technical services Accomplished Development of independent national organizations Indicator discontinued Establishment of a program performance monitoring system Establishment of a program impact assessment system Two-thirds accomplished Formation of an Income Generating Department in HOPE HQ has been fully accomplished, ratifying a full-time staff in the person of the Director-Income Generation; he is assisted by an administrative assistant split between VHB and other assignments. Among the observed characteristics of this unit are: quality of proposal and report writing, donor relations, monitoring, nurturing of staff, portfolio analysis, leadership, collegiality, and openness to criticism. Yearly VHB sharing events take place with project staff. On the financial side (though health will be discussed in different terms), the VHB is as competently supervised as one could hope for. This unit has also been strengthened by the hiring of three country Project Managers, and two half-charged Adult Educators, one in Latin America and one in Africa. Evaluator criteria suggest these are all talented, well-qualified staff. Development of a common, flexible methodology has been fully accomplished. Based on interviews with the Program Director and discussions with field staff, it is clear that guiding principles have been promulgated from HOPE HQ during the grant, with local adaptation being encouraged depending on country conditions. Overall principles of Micro-finance have been respected: market rate interest, portfolio quality guidelines, women’s participation, VHB Board of Directors formation, loan monitoring mechanisms, health education as an integral part of the project, etc. A number of country variations were also observed: change in the planned repayment schedule from fifteen days to thirty, change in how loan “points” are charged, change in the average size of the group, 13 change in the average size of the loan, and others. The program has adapted appropriately to local conditions without sacrificing any essential element. Manuals were seen and briefly reviewed in each country visited that attest to the ‘diversity within similarity’ of the VHB model. Replication of the VHB to three new countries, Malawi, Guatemala and Peru, and expansion in Ecuador have been verified during the Midterm evaluation (Guatemala and Ecuador) and the Final Evaluation, (Malawi, Guatemala and Peru.) As above, the design of the program, implementation, monitoring and evaluation are similar in each, but each country has been allowed to adjust various policies to fit market conditions. Greater sustainability of the VHB programs through diversification of funding— commercial financing—was discarded as an indicator based on Midterm recommendations and a HOPE Board of Directors decision not to pursue this type of financing. Increased sustainability of HQ technical services is evidenced by three facts. 1) Acquiring complementary funding for this project from two U.K. sources, the Department for International Development (DFID) for expanded program operations in Malawi, and the National Lottery Charities Board for expanded program operations in Guatemala. 2) Expansion into two countries, Thailand and the Dominican Republic without requesting USAID funds. 3) The Vice President for International Programming confirmed on July 29, 2003 that continued HOPE funding would be made available for FY04 for Malawi, Guatemala and Peru as necessary to keep the VHB operating; he also confirmed the decision to hire a Deputy Program Director-Income Generation. Development of independent national organizations that operate the VHB program was recommended to be discarded during the Midterm evaluation. However, at that point, plans were well advanced to “graduate” the Ecuador operation, the Fundación Para El Desarrollo Integral, ESPOIR, to full institutional independence; and HOPE decided to go ahead with that activity. As a result, one VHB, Ecuador, is now completely independent, and the other three remain in the HOPE portfolio. Establishment of a performance monitoring and program impact assessment systems has been accomplished perhaps “about three-quarters.” On the financial side, a format called the VHB-TRAK monitors bank formation, number of participants, target number of loan disbursements, disbursement amounts, total loan portfolio, savings, operational sustainability, arrears, as well as a number of other indicators. This part of the project is a source of pride to Project HOPE, and is completely under control. Impact monitoring on the micro-credit side will be shown to require some fine-tuning. On the health side, monitoring and impact assessment are less well developed and have taken some years to firm up. However, an acceptable methodology and format have been firmed up in the last several months and are ready to go into program-wide implementation. SO1 Summary. From the Headquarters perspective, the project has been effectively managed in almost all of its tangible elements. 14 SO2 Accomplishment Table 2: SO2 Accomplishments of Current Project Proposed Results Current Status SO 2: Financial services and health education provided in a sustainable manner Develop local staff capacity to implement VHB methodology Accomplished Create local NGOs to continue program beyond LOP Indicator discontinued Achieve sustainability of operations Malawi: 58%, Guatemala: 69%; Peru: 45%; Average: 57% Serve a significant population 48%, 51%, 77% or 131% depending upon method of calculation. Develop access to resources to support long-term growth Indicator discontinued Maintain high program quality [in financial terms] 43% accomplishment Develop local staff capacity to implement the VHB methodology has been fully accomplished. Final evaluation field travel was able to visit and converse with 13 staff in Malawi, 14 in Guatemala and 10 in Peru. This does not take into account administrative personnel who also work on the project with whom interviews were generally not held. The majority of staff in each country was evaluated as fully competent in their discipline; one or two personnel per country were judged excellent in community promotion. Create local NGO institutions to continue the program beyond the grant end was described above: accomplished in Ecuador by the formation of an Ecuadorian Board of Directors and legal incorporation (Personería Jurídica) of ESPOIR. Creating local NGOs in the other three countries was discarded at the recommendation of the Midterm evaluation. Achieving sustainability of operations has been one of the highest priorities of VHB. The following lists the status and some of the items of discussion from the respective country final reports. · Malawi: 58% achieved at time of the evaluation visit. The Malawi VHB is “likely on the way” to operational sustainability. Future viability depends on continued success in expanding the peri-urban portfolio in Blantyre. A major constraint has been high-cost operations associated with rural Mulanje, but economies of scale seem achievable without having to withdraw from this area. · Guatemala: 69% achieved. “On the way.” Gains in operational sustainability have varied over the months, but a steady pattern of improvements seems the most likely scenario over the next year. The Project Manager anticipates achieving operational sustainability within nine months. To do so does not require success in the untested urban market of Quetzaltenango. Operational sustainability will be somewhat slowed by the project’s having to absorb more of its “fair share” of Guatemala City head office costs. 15 · Peru: 45% achieved. Operational sustainability does not seem achievable in this portfolio for the foreseeable future. Operating expenses are high because of high levels of staffing, high dropout rates, and by the project’s “churning” of clients: working with large numbers of ever-new banks and clients while large numbers of older clients withdraw. Competition from other lenders and debt-forgiveness by the Government of Peru, coupled with a culture of non-payments has proven problematic. The Final Evaluation also speculates the project may not have been sufficiently proactive in responding to these external threats. · Average of the three countries: 57% Evaluation narratives discuss this issue in three different country contexts. With a (relatively) small amount of seed capital, +/- $250,000 per country, there is experience from comparable organizations to suggest aiming for operational sustainability in five years in three start-from-scratch operation may have been an overly ambitious goal. Serving a significant population was planned to be reported three ways: total participants, total beneficiaries and total banks formed. The original proposal did not discuss the issue of dropouts. Taking into account dropouts means the number of people who are currently receiving benefit from the project is a percentage of those who received services at one time or other. Calculating significant population served can be done in several ways. 1.) First one can look at the accumulated total, 16,136 of a goal of 12,300, 131% of goal. 2.) Next, one can track total clients vs. current active clients. Current active clients as a percentage of goal is 48%. Participant Totals and Currently Active Members Total Goal Active Dropout Malawi 6,720 5,500 2,935 56% Guatemala 4,332 3,500 1,861 57% Peru 5,084 3,300 1,071 79% Total 16,136 12,300 5,867 64% The data can also be used to show a dropout percentage, (total participants minus active divided by total). Dropouts will be seen as a serious issue for the project, 64%. 3.) Another way to calculate participants is to look at those who have remained in the project for a year. As reported by project management, the data are as follows. End of Year Clients Staying a Full Year 6/99 6/00 6/01 6/02 Total Malawi, 444 438 706 755 2,343 16 Guatemala 616 1,358 829 2,803 Peru 86 398 667 1,151 Total 444 1,140 2,462 2,251 6,297— 51% There is double counting in these figures: from one end-of-year to the next, women who stay more than four loan cycles are counted twice or three times. These figures will be explored more in depth in the section on improved income. 4.) Another way to calculate participants is to compare them to an “internal management target,” as reported by the Program Director, 3,200 active participants in Malawi by EOP, 2,125 in Guatemala and 2,160 in Peru. Accomplishment of this internal goal 92% in Malawi, 87.6% in Guatemala, and 50% in Peru, 77% total. Thus, deriving a ‘snapshot’ percentage to show the total of those who benefited meaningfully from the project is a challenge. If one measures total participants at one time or other, the figure is 131% of goal; if one reports vis-à-vis the internal management target, it is 77% of goal; if currently active participants, the figure is 48%; of those who have stayed a year or more, the figure is 51% of goal. From the evaluator’s perspective, somewhere between the two middle figures seems the most appropriate. Number of total beneficiaries (family members, etc.) has been achieved at 132% of goal. Based on currently active beneficiaries, this represents 55% achievement of goal though the same caveat as in the preceding paragraph also applies. Total Active Est. Malawi, 47,000 25,500 Guatemala, 23,800 10,200 Peru 28,000 5,900 Total 98,800 41,600 Goal 75,000 75,000 Percentage 132% 55% Total number of village health banks formed has been achieved at 215% of proposed goal; current active banks represent 130% achievement of original goal. Total Goal Active Act./Goal Malawi, 251 120 197 164% Guatemala, 177 85 104 122% Peru 185 80 70 88% Total 613 285 371 130% Attracting commercial financing was discarded as a programming strategy by the Project HOPE Board of Directors. Maintain program quality, 30-day arrears less than 10%: 49% under-achievement. 17 Arrears Net portfolio value Malawi 13% of $215,000 Guatemala 3% of $163,000 Peru 28% of $178,000 Goal <10% Weighted average 14.87%, $556,000 48.7% under-achievement Maintaining loan losses at less than 2% has been accomplished in Guatemala and Malawi, in part due to a deliberate management policy to delay booking the loss, in recognition that some old clients repay little by little over time. SO2 Summary: Regarding the provision of financial services and health education, local staff has been effectively trained in the VHB methodology. A significant population has been served: between 55% and 77% of targeted participants and total beneficiaries and between 65% (active) and 130% (total) banks. Operational sustainability has been achieved 58% in Malawi which is “likely on the way,” 69% in Guatemala which is on the way, and 45% in Peru which seems unachievable. Indicators of high quality performance in micro-finance, maintaining 10% or less arrears in loans has been 49% under-achieved, though loan losses are under control in two countries. These last two indicators have been highly affected by the discouraging results in Peru. SO 3 Accomplishment Table 3: SO 3 Accomplishments of Current Project Proposed Results Current Status SO 3: Improved income and health of impoverished women and children Form financially viable, democratic community institutions Of goal, 130% formed, 65% active Improve economic opportunities for women (loans disbursed) Accomplished 20% Increased incomes of participants Accomplished for 2,250 women, less for another 4,046 Increased exposure of women to health talks 108% accomplished Improved health status of participating women 32% accomplished A total of 613 democratically managed community institutions were formed; 371 are functional at time of the Final evaluation, 130% of the goal of 285 in these three countries. Improved economic opportunities for impoverished women have been accomplished. Loan placement has averaged 2.03 disbursements per 16,136 total clients, of the proposed goal of 53,000, 62% of accomplishment. Average loan amount of the portfolio is $169. Disbursements Avg. loan Malawi, 11,967 $139 Guatemala, 10,562 $200 Peru 10,221 $173 Total 32,750 $169 18 Estimating increased incomes for project participants has proven a complex task. Narratives in the individual country evaluations speak of the existence of a quality income monitoring and evaluation system. Further investigation during this write-up suggest that while the data collection system is strong, the amount of information collected has been less than ideal, thus requiring some extrapolation of the data to reach program-wide conclusions. Three concepts are involved: 1) sample size and gains in income reported by survey; 2) long-term participants who have benefited from those income gains; 3) gains to others who have left the project. 1.) The following chart taken from individual country narratives shows sample size and reported gains in income. Table 4 Reported Quantitative Improvements in Household Income (Same Clients) Criterion FY01 FY02 Malawi Year 3 (N=10) Year 4 Improvements in household expenditure 120% Not reported Increase in households reporting better incomes 93% Not reported Decrease in households reporting worse incomes 60% Not reported Peru Year 3 (N=87) Year 4 (N=179) Percentage increase in household expenditure 7% 11% Percentage increase in households incomes 18% 5% Percentage increase reporting better incomes 35% 17% Guatemala Year 3 (N=136) Year 4 (N=145) Year 5 (N=242) Percentage increase in household expenditure 18% 18% 12% Percentage increase in households incomes 48% 52% 37% Percentage increase reporting better incomes 59% 57% 24% Regarding the issue of how representative these results are: -- The Malawi data are unreliable: the chart shows one year’s figures on ten people of a four-cycle-or more total of 438 women. -- In Peru in FY01, it appears all 87 people of the four-cycle-or more universe were interviewed. In FY02, 179 people of a total of 398 were interviewed. -- In Guatemala in FY01, 136 of 616 were interviewed; in FY02, 145 of 1,358, and in FY03, 242 of 829. -- In Peru, quantifiable gains in income are not large but the sample size is good. In Guatemala, the reported gains are large but the sample size is not as big. 2.) Project management presented post-field trip data showing four-cycle-or-more participants at the end of each fiscal year. The evaluator has reworked the data in an effort to interpret the figures in light of other information collected during the field travel. Client Movement from FY to FY/ Net Gains to Long-term Clients 19 Malawi 6/99 6/00 6/01 6/02 Total No. B/F start of year 873 444 438 706 New clients start of year 620 844 790 Clients lost during year 429 626 576 741 Clients staying full year 444 438 706 755 Long-term client gains 444 -6 268 49 755 Guatemala 6/00 6/01 6/02 Total No. B/F start of year 969 616 1358 New clients start of year 1134 19 No. lost during year 353 392 548 No. staying full year 616 1358 829 Long-term client gains 616 742 -529 829 Peru 6/00 6/01 6/02 Total No. B/F start of year 618 86 398 New clients start of year 1225 787 No. lost during year 532 913 518 No. staying full year 86 398 667 Long-term client gains 86 312 269 667 Net gains to long term clients 2,251 Based on all available evidence, the evaluator estimates these 2,251 clients are those who have benefited by increased income to the full extent described in the quantitative surveys, perhaps about 20% in Malawi, 10%-15% in Peru and 40-50% in Guatemala. 3.) At the same time, there is likely some validity to extrapolating gains in income for those FY99-FY01 clients who remained in the project four cycles but who withdrew from the project for one reason or another. This conclusion is based on an inference that women who have been less successful have left the project more frequently than fully successful women who continue in the project from one year to the next. Thus, income gains for these women have been estimated less than for the 2,251 immediately above. Combining data from previous tables produces the following. Clients Staying a Full Year with Some Income Gains 6/99 6/00 6/01 Total Malawi 444 438 706 1,588 Guatemala 616 1,358 1,974 Peru 86 398 484 Total 4,046 20 Quantification of this income gain has not been attempted though one infers it will be less than those who still participate in the project. In summary, as closely as an outsider can determine, VHB has brought substantial economic benefit in the 15%-40% per year range for 2,251 women, and less but nonetheless important benefit to another 4,046 women. Proposed recommendations to strengthen measurement of impact in income may make this exercise easier in future years of the project. Increased exposure of impoverished women to health education lessons: Total health talks since the start of the program have been 13,003, 108% of goal. As described in each country’s evaluation, in FY02 and FY03 a change took place in the VHB health education methodology when project staff assumed direct responsibility for the health talks. Qualitative assessment, generally good, is reported in each country’s narrative. Regarding improved health status (by 25%) for participating women (75%): In the absence of reliable quantitative data, the observer speculates significant gains in health for 24% of VHB women (2,250/13,000X75%) for long-term participants, and less significant gains in health for those FY99-01, four-cycles-or-more women who left the project. See also Section 1.4.2b below for further discussion. Recent improvements in health data monitoring and reporting will permit this estimate to be more objective in subsequent years. Women experiencing Women experiencing long-term health gains FY99-01 health gains Malawi 755 1,588 Guatemala 829 1,974 Peru 667 484 Total 2,250 (25% gain) 4,046 (some gain) Goal 24% of goal 41% of goal Overall accomplishment 32% of goal SO3 Summary: More than 100% accomplishment has been achieved in bank formation; 65% of them remain open at time of the evaluation. More loans have been disbursed to women than planned. Increased income due to these loans has been achieved for 2,250 women, plus 4,046 women benefiting somewhat less. The goal of women receiving quality health talks has been accomplished 108% of goal. Improved health status for participating women has been achieved at an estimated 32% of goal. Accomplishments per Country Table 5: Comparison of Accomplishments per Country Objectives Indicators Malawi Guatemala Peru VHB model adapted to • VHB policies and procedures defined for successful implementation Accomplished throughout project 21 Objectives Indicators Malawi Guatemala Peru characteristics of the country • Completed lesson plans for health and business education that is appropriate to the needs of the beneficiaries On the way in health and starting in business On the way in health; not underway in business Well on the way in health; not underway in business • New VHBs created 164% accomplished 122% accomplished 88% accomplished Expansion of the VHB program • Women receiving loans 122% total, (2935/5500) 53% active accomplishment 123% total, (1861/3500) 53% active accomplishment. 154% total, (1071/3300) 32% active accomplishment • Arrears over 30 days less than 5% of outstanding loans Not accomplished Accomplished Not accomplished • Loan loss rates of less than 2% of average portfolio Accomplished Accomplished Not accomplished Maintenance of high program quality • Operational efficiency of less than 15% by grant end Accomplished Not accomplished • New staff trained and oriented in VHB methodology Accomplished throughout project • Development of planning, administrative, and management systems and documentation to support a growing institution Accomplished Accomplished Accomplished but … • Upgrade of the Financial Management Information System to assess economic impact on beneficiaries Accomplished throughout project Improved institutional capabilities • Development of an Impact Assessment System Accomplished in micro-finance; not accomplished in health Accomplished in M-f; well advanced in health • Achieve a 100% level of operational sustainability. 58% accomplished 69% accomplished 45% accomplished • Commercial funding utilized to finance expansion of the loan portfolio: Not applicable Self￾Sufficiency • Accessing in-country resources to complement program operating and growth needs Not accomplished Partially accomplished Not accomplished • Income of participating women increased by 20% annually 20% for 755 women, less for 1,588 50% for 829 women, less for 1,974 15% for 667 women, less for 484 Increased beneficiary impact • 70% of VHB members engaged in improved home-based health practices Yes for 750+ women Yes for 800+ women Yes for 650+ women VHB model adapted to characteristics of the country has been accomplished in all three countries. Improved health education modules are most advanced in Peru. Reported earlier, new banks still active in Malawi are 164% of goal, in Guatemala 122% of goal, and in Peru 88% of goal; Peru is likely overstated, as recently formed banks will 22 likely follow the trend of high drop out rates. Taking into account currently active clients, accomplishments are 53% in Malawi, 53% in Guatemala and 32% in Peru. Expansion of the VHB program in number of banks has taken place in all three countries, 164% of goal in Malawi, 176% of goal in Guatemala and 88% of goal in Peru. In Peru, this figure is also likely overstated. In terms of currently active participants, the percentages are 53%, 53% and 32%, respectively. Maintenance of high program quality has been achieved as follows: · 30 days arrears less than 5% only in Guatemala; · loan write-offs <2% in Malawi and Guatemala; · operational efficiency near 15% in Malawi and Guatemala. Improved institutional capabilities regarding trained staff, upgraded financial management and micro-finance impact measurement have been accomplished in all three countries. In proactive management of problems, Peru may be a little behind the others while Peru is advanced in improving its impact measurement in health. Self-sufficiency has not been fully achieved in any country and is most advanced in Guatemala and least advanced in Peru. The search for new donors is most advanced in Guatemala and seems rather underdeveloped in Malawi and Peru. Significantly improved health is likely for 2,250 project participants. 1.4.2b Impact on Target Populations: Further Discussion Individual country narratives discuss the impact of the project on target populations with care. In general terms, broad program impact in health could not be discerned, i.e., widespread uplift in health status, or changes in health behavior in a substantial number of women. Project staff in each country expressed concern that the evaluator’s time did not allow the capturing of these changes though they are taking place. In fact, large-scale changes cannot be documented because of the methodology chosen for health data collection. In addition, the country evaluations note large movement of clients over time—not only constant dropouts but also constant new entrants— such that the number of women exposed to the full cycle of health talks is quite reduced. It is only these four￾cycles or more women where improved health results can easily be discerned. The country evaluations do attest to important individual gains in health. In order to capture these individual gains, the narratives use a “case study” methodology, reporting on results of random interviews of VHB members at the end of health talks. It may be useful to cite some of those stories. In Malawi, mention was made of 33 “A-plus” groups that have shown attentiveness to project messages, steadfastness of membership and good loan repayments. It was postulated that women in these groups, 550 people, have likely changed their health behavior in significant ways. Certainly, the one “A+” group the evaluator visited showed notable improvement in knowledge of sanitation and hygiene. 23 In Guatemala, some of the life stories that came out of individual interviews were the following: · One woman says before VHB, her husband would not eat a meal without meat. After the nutrition talks, she changed her cooking and now the whole family eats balanced meals with lots of vegetables, and eats meat only every now and then. · Several women think the greatest value they found from the health talks was that they have learned to value themselves as women: self-esteem has improved. · One woman had never attended such meetings before; the talks woke her up and have had a liberating effect; she says, “Before we were alone; now we have colleagues.” · One woman learned in the talks of the importance of self breast-examination, and full vaccinations for children. To her, the talks reinforce health messages she has heard elsewhere in the community and encourage her to do something about accessing health services. · Many women remember the PAP talks and some now get their PAP smears done regularly. · New topics for many women were HIV/AIDS, cancer, and the importance of marital fidelity. · For other women, the most important thing they learned is that women have the right to go to the police and other social welfare offices if they are mistreated. · Several women reported group members who have escaped the clutches of moneylenders. In Peru, some of the life stories that came out of interviews were the following: · One woman now sees herself as a village leader and says the health talks are “the best thing that could possibly happen.” She remembered nearly every talk ever given and has applied what she has learned to her life. · Another woman is “enchanted” with the health talks. She commented that the sessions allow her to relax and have fun, and “step out of herself” for a little while. She loves the dynamics. · One woman remembered many of the topics and when asked if she applied them to her day-to-day life, responded: “Of course! They ask us to make promises, and I keep them!” · Another takes what she learns from the health talks and shares it with her neighbors and friends; any talk she is too old to apply to her own life, she teaches to her grown children so they can raise their children better. · Another says she is more conscientious about covering the food at home and hygiene matters to her, and she gets her PAP smear check ups now, where before she was afraid to. · Several women reported members who got out of the clutches of moneylenders. It bears mentioning these women were selected for interview at random and a deliberate effort was made not to interview any member of the VHB Board of Directors, (thinking 24 that Board women might not be representative of the wider group.) With these case stories out of a random sample of 54 interviewees, there is validity to the proposition that in spite of relatively recent changes in the health methodology; in spite of the lack of demonstrated program-wide quantitative impact; and in spite of high member turnover, the project has managed to have substantial impact on health for a number of women. 1.4.3 Cross-cutting Issues 1.4.3.1 Partnership Strengthening partnership was not a stated objective of this Matching Grant thus has not figured as a major project activity. However, several important partnerships have been formed. Among those observed during the evaluation process were: · In Malawi, formal membership in, and serving as Secretary for, the Malawi Micro-Finance network of NGOs. · In Guatemala, informal, cordial relationships with several micro-finance NGOs, including FUNDESPE and CARITAS. · In Peru, VHB took a formative role in creation of, and membership in the Central de Riesgos, a data bank for micro-credit agencies to crosscheck loan defaulters. VHB-Peru regularly participates in a Round table of micro-credit institutions to share experiences. In addition, cordial relationships exist between the project and several credit NGOs, and an intimate relationship exists with the Departmental Ministry of Health. In addition, a collaborative relationship exists with the Agronomy Department of the University of San Martín. · In Peru, discussions are on going with two other NGOs regarding the signing of a tri-partite contract to provide health education to 4,500 women. In all three countries, the evaluator observed that VHB project managers were perceived as well-qualified professionals by their peers. Observations detailed in each country narrative discuss the generally strong performance of most of VHB field staff. In one country (Guatemala,) HOPE was lauded by a colleague organization as “the” expert in rural health education. 1.4.3.2 New Tools, Guidance or Standards Midterm “peer” evaluation The VHB conducted an innovative “peer evaluation” during the Midterm evaluation, as follows. An external evaluator was hired as chief evaluator, team leader and author of the eventual document. To make the evaluation participatory and to foster a climate of organizational learning, three national teams were formed to accompany, assist (and in one instance replace) the external evaluator. These teams were composed to two or three VHB national staff that traveled to another country to carry out the midterm assessment: Peru staff visited Guatemala, Guatemala staff visited Ecuador, and Ecuador staff visited Peru. Each team was given a seven-module questionnaire to fill out during its travel. 25 The modules included a questionnaire to be applied to senior managers and field staff, a management systems questionnaire, a VHB banking procedure checklist, a health talk assessment format, a client satisfaction interview format, and a colleague organization interview guideline. Applying those instruments, the national teams prepared a summary report of findings that was used by the external evaluator in preparation of the document. This process was of sufficient interest to the Agency for International Development that a separate document was authored by the external evaluator describing it. Regarding outcomes, the peer evaluation produced a number of high quality products and organizational, country-level, and personal learning. A system for internal evaluation was created in HOPE, and a methodology to provide on-going annual follow-up. The peer evaluation process evolved largely as conceived. Highlights included extensive pre￾work on interview guidelines; good complementarity between external and internal evaluators; careful selection of peers evaluators; and good team planning, facilitation and field travel. A few outcomes did not fulfill all expectations: the biggest drawback may have been the learning curve of team members. However, none of the limitations interfered with the overall quite positive impact of the methodology. Conclusions were that the peer review produced a number of clear benefits to the project. Notwithstanding ‘growing pains,’ the peer review offered important advantages in the context of process evaluation. One recommendation was that the document be shared widely in BHR/PVC. As this writer was both judge-and-jury throughout the effort, it is impossible to comment with objectivity what others—the donor and the client—have finally thought with regard to its success. The Program Director-Income Generation mentioned in passing that he would do it differently the next time, though that comment was never developed. Either positively or less positively, the peer evaluation does seem to be a lesson learned. Loan portfolio monitoring VHB-Guatemala is cited in its country narrative for innovation in developing a staff performance monitoring system in microfinance each month. Data from the monthly financial report are inputted on 7 indicators: new banks formed during the month, total monthly loans distributed, active portfolio, and active portfolio less < 30 days arrears, and arrears of 31-60, 61-90 and >91 days. A percentage combining these indicators is then prepared and each staff is presented with graph of his/her performance compared to other field staff at the end of the month. This seems an excellent supervisory (and motivating) mechanism other VHB countries could adopt. Health talk monitoring VHB-Peru is cited in its country narrative for innovations in designing a health talk ‘check list’ that measures the technical skills of the facilitator as well as the technical content and quality of the talk. The format has four rating sections, each of them generating a numerical score: 1) quality of steps and procedures; 2) knowledge, ability and behavior of the facilitator; 3) responses of the women during the session (including behavioral observations as well as post-event interview); 4) quality of the visual aides 26 and overall quality of the event. This observer has been struggling to systematize health talk monitoring for years and concludes that the Peru VHB supervisory mechanism is as good as he has ever seen. The Peru Health Educator uses this format whenever she supervises field staff and was able to show a file full of these forms over time, and improvement in staff scores over the last six or nine months. This is truly a great stride forward and could be looked at for health talks supervision anywhere in the world. 1.4.3.3 Sustainability and Scale-Up Regarding sustainability of current efforts, in a June 2 conversation the Director-Income Generation indicated HOPE’s commitment in principle to a net transfer of resources of $50,000 each to Malawi and Peru for support of FY04 VHB activities. In the latter days of the evaluator’s stay in Peru, the FYO4 budget was approved by the Latin America Regional Director, effectively guaranteeing continued operation of VHB-Peru for another year. In a July 28 discussion with the Vice President for International Programs, he indicated a commitment that HOPE Center resources need to be found for FY04 to continue VHB program operations in these three countries as well as the decision to hire a deputy for the Program Director. Regarding scale-up, approximately two years ago Project HOPE began to look at the possibilities of replicating the VHB in other program settings. The result of those investigations, donor search, proposal preparation, etc. is that HOPE has now established a Village Health Banking project in the Dominican Republic, in Thailand, and in Mozambique. Each scale-up has adapted basic VHB methodology to country conditions. · In the Dominican Republic, the primary adaptation has been peri-urban credit in the barrio of Herrera, Santo Domingo, and the provision of health education through an established primary health care clinic staffed by part-time Ministry of Health counterparts. · In Thailand, the innovation has been the provision of credit for rural agricultural activities. In addition, health talks are more sophisticated in Thailand than elsewhere in the VHB portfolio because of the sophistication of the Thai rural primary health care network. · In Mozambique, in a newly starting program, the VHB methodology is being directed to clients affected by HIV/AIDS. Staff from current VHB programs provided mentoring and consulting services to the scale-up. Four Malawi staff visited Thailand for two weeks in FY03 to advise Thai staff on management issues, health education and micro-finance mechanisms. Two HOPE Mozambique personnel visited Malawi to look at the VHB operation. The Program Director traveled several times to the Dominican Republic to advise in that start-up. He reports the current scope of activities in the three scale-up countries as follows: · Dominican Republic: 50 VHBs and 1,155 clients; FY 04 budget $ 122,000 for direct costs. 27 · Thailand: 33 VHBs and 800 clients; FY04 budget $300,000 (including HQ and indirect costs.) · Mozambique: start-up; FY04 budget is $435,000. On July 28 2003, the Vice President for International Programs also judged that Village Health Banking could become a “core expertise” of Project HOPE, and he is looking for donor interest to expand the VHB model (further) into HIV/AIDS response populations and perhaps to targeted minorities also. Post-grant sustainability in Ecuador has been achieved with the “graduation” of the Ecuador portfolio to complete independence. The local entity—ESPOIR, now no longer Project HOPE—has an independent Board of Directors, an Executive Director, staff, etc. Continuing a pattern set in the late Nineties by HOPE Honduras, which developed a sustainable portfolio platform to support other projects, ESPOIR is self-sufficient of Project HOPE funding. ESPOIR is also actively pursuing local grants as part of its organizational evolution. 1.4.4 Lessons Learned and Program Recommendations 1.4.4.1 Lessons Learned The following evaluator comments do not necessarily reflect the opinions of VHB staff and are phrased to stimulate reflection. Over-ambitious goal Reported in the country evaluations, there are data to suggest the VHB was overly ambitious in proposing to achieve financially viable loan portfolios in three start-from￾scratch operations. When operational expenses were deducted from the grant, the money available for loan financing was no more than approximately $250,000 per country. Colleague organization experiences in Guatemala and Peru where operational sustainability was achieved in five years indicate these organizations started with more loan financing in the first place, or that a previous loan operation was rolled into the micro-credit operation, or both. If VHB had started in three countries instead of four (including Ecuador), operational expenses would have been less a drain on the loan fund, and divvying up available loan financing among one less country would have made more resources available for loans. It appears there may be a certain threshold loan fund necessary to achieve operational sustainability in a time-bound grant Midterm adjustments The VHB should be sincerely congratulated on taking Midterm recommendations to heart. Primary among them was the recommendation that the project move away from relying on village women to deliver high quality health talks: the proverbial Training-of- (village)-Trainers was not working. Instead, it was recommended VHB should convert 28 bank promoters into health extensionists. This recommendation was fully implemented. VHBs in each country changed the title and job description of the field staff from banking promoters to community development promoters or some similar title. Training modules were developed to guide non-health staff on how to cover key messages in a given subject. A training calendar is being carried out for staff in the new skills. Making a program change of this nature is a big undertaking; VHB took it on, and is carrying it through. This is a substantial step in the right direction. A question may be usefully raised, however. The Midterm recommendation was written in August 2001. The first response to the recommendation was taken by VHB-Peru, which contacted a health education consultant who began working part-time in January 2002. In March 2002, a multi-country VHB conference took the program decision to proceed. One wonders if six months to ratify the decision was an appropriate amount of time. One also wonders what the other two countries were doing with this idea while Peru was in the vanguard. At this writing 15 months after the decision, only one of four health modules is fully complete and in implementation; a second is in the field-testing stage. When will the other modules be finished? Is this pace of innovation appropriate and has the project come as far as it should since the Midterm? In terms of 180? change, the project is to be congratulated. Perhaps more rapid progress could have been achieved. Organizational learning curve There are several instances where appropriate action does not seem to have been taken until long after the problem had been identified. As detailed in the three country narratives, staff has been struggling with health data collection since FY00. The Malawi narrative is illustrative. For three years, the project has struggled to track and document improved health. Baseline heath data were collected a year after the project started in August 2000 (N=269). Another effort was undertaken in FY01 (N=256.) In FY02, a third effort was carried out (N=282). In all cases, a detailed survey report was prepared. Alas … FY01 and FY02 data were substantially ‘co-mingled’ with recently entering women. FY02 data, for instance, show only 17% of the sample were women who had been in the program four or more cycles. In order to show some movement, in the FY02 Annual Report, FY01 data on seven indicators (of 32 on which data were collected) were compared to the Baseline and then reported as “average improvement of 35% across a variety of indicators.” When an exercise was conducted during this SOW using all 32 variables, the average difference between Baseline and 2001 data was only 5%. When FY02 results on the seven indices were compared to Baseline, the average improvement was 15%. The problem with these figures, of course, is that the ‘co-mingling’ of new entrants with old makes all the figures suspect. Aware of these difficulties, the HQ Program Director tried to tighten the relevance of FY02 data by comparing some 1st cycle and 4th cycle clients (N=75 and N=89 respectively). At his suggestion, a second round of analysis was conducted on 16 FY02 indictors where 1st and 4th cycle data were available. After deleting two indicators that are individual responses, ten indicators show improvement, four show deterioration: overall change: 13%. At this writing a health survey is being conducted in Blantyre (N=200) comparing 1st and 3rd cycle VHB women. , the fundamental problem remains: different women being compared. 29 The same conundrum was being faced in Guatemala and in Peru and it is only late in FY03 that a solution is being found. As narrated in the Peru document “… [S]everal months ago, VHB-Peru came to a significant conclusion. After the struggle of the previous three years, the change is quite simple: collect health data at the start of a women’s career in VHB just as income and expenditure data are collected; and measure improvements in health at the close of the 4th loan cycle just as improvements in income and expenditure are measured. In fact, the HQ Program Director’s recent trip reports to other countries came to this conclusion at about the same time. VHB-Peru is the only operation to have reached the conclusion on its own.” In an organization like Project HOPE dedicated to health, why did it take VHB four years to recognize and deal with problems in health data and how did it take so long to discover a solution built into income- generation tracking from the very start of the project? On a related note, the VHB portfolio in Peru is in serious financial difficulty. A reading of the FY02 Annual Report shows VHB-Peru was in difficulty last September. Individual country final evaluations speak of the strong VHB financial monitoring system. Could/ should this financial monitoring system have addressed Peru’s problems earlier? Combining the two ideas: Have the learning curve, innovation and pro-activity of VHB staff been all that one could hope for? Has VHB staff been sufficiently thought provoking and challenging of one another in this grant? Dropouts A reading of the country narratives demonstrates that dropouts and desertion have been with the project from the beginning. During the Midterm, the Program Director was asked about dropout rates and replied that VHB was experiencing more or less industry trends. Currently in Malawi, the drop out rate is 56%; in Guatemala, it is 57%; the case of Peru is special with drop out rates perhaps as high as 75%. These are figures not contemplated in the original project proposal. If, as recommended in individual country narratives, project staff can find the time to provide oversight as to the quality of individual loan requests (admittedly a big “if”), it may be possible to shoot for meaningful reduction in these figures. Low savings Disguised dropouts may also be occurring. Women, whose loans are not successful, as they withdraw from the project, surrender their compulsory savings to repay the loan balance. Country narratives report that member savings compared to the original goal is low: in Malawi, 29% of goal, and in Guatemala, 18% of goal. It should be noted these goals were developed based on the Ecuador VHB experience, which has turned out to be overly ambitious in other contexts. Nevertheless, as narrated by several interviewees, if a woman’s ability to create and maintain a savings account is a life-changing improvement 30 (like increased income to which it is related), it would appear that VHB has been less successful in this area than one could have hoped. More vigorous tracking of a relationship between savings and loans may stimulate staff to encourage better (i.e., higher profit) loan endeavors, so that women generate more savings at the end of each loan cycle. The VHB Model Perhaps one of the biggest questions for those who sit in “macro” positions is how much the VHB model has demonstrated its viability. In ten-word summaries: · In Malawi, micro-finance is strong, but health talks are not yet producing widespread gains. · In Guatemala, financial progress is strong but health gains are mostly still incipient. · In Peru, health is a tremendous strength but the financial side is unsustainable. It may be worth reflecting whether VHB has demonstrated itself as a deliverer of the proposed synergism between health and income. Can field staff be expected to provide quality oversight to these different disciplines? In the person of stellar performers described in country narratives, the answer to this question is clearly “yes.” How to attract and retain stellar performers is an issue that may need thoughtful consideration. 1.4.4.2 Program Recommendations The country evaluation narratives spell out six recommendations held in common: a) Continued HOPE Center support for VHB operations: financial and budgetary support from HQ for FY04 to enable the project to continue. b) Improved Adult Education: consolidating the gains of the new health education philosophy on health talks (except Peru where the observation is “keep up the good work.”) c) Improved health data collection: collect baseline information on health just as economic information is collected at the start of a woman’s participation in VHB, to be measured four cycles later. d) Improved income data collection: collect baseline income information at the start of a woman’s participation, and measure all clients, not just a sample, four cycles later. e) Improved loan making: consider moving into occasional supervision of individual loan applications and perhaps begin talks on better business planning (as Malawi is doing.) f) Continued search for operational sustainability: cost savings and efficiencies. Two other recommendations are country specific: In Malawi: Continued presence in Mulanje, and hiring an Adult health Educator. In Peru: Creation of a new program strategy. 31 1.5 Program Management 1.5.1 Management Approach Overall project management is appropriate based on evaluator field trip observations. The three project managers were judged generally effective project leaders: on top of the details of their portfolio, in touch with field realities, aware of program and staff strengths and weaknesses. All three come from non-health backgrounds, but this does not appear to be an obstacle to their being committed to and involved in the health side of the project. This was not the case in Guatemala for a time where project leadership was not much interested in health, but that situation was remedied two years ago. Regarding country management, several HOPE Country Director seemed rather distant from the VHB project—delegating completely to the project manager. The comment is formulated cautiously, since with strong project leadership an appropriate management response is to delegate. Nevertheless, all HOPE Country Directors are health specialists: one a Ph.D., one a public health physician, one a public health nurse. If experienced country leaders in health had been more involved with VHB, weaknesses in health data collection might have been resolved earlier. Laissez faire is not the same thing as appropriate delegation. HOPE Country Directors do not appear to have exercised all due VHB oversight. Regarding portfolio management, the observation is also formulated to stimulate reflection. In discussions with the Program Director during the Midterm and Final, the topic of appropriate levels of Head Office involvement came up. The Director spoke during the Midterm of his high involvement during initial stages of the project; during the Final, he described a deliberate, capacity-building strategy to step back from day-to-day decisions, pushing countries to take decisions on their own. Intellectually, it is hard to dispute such a management stance. However, the project appears to have needed steady direction throughout its life: evolving country situations require tough decisions; problems that crop up (viz. Peru) require rapid response; measuring improved incomes needs more rigor in some areas. As HQ has the only program-wide perspective, it is the responsibility of Headquarters to provide this. The Vice President of International Programming discussed his vision of creating VHB as a pole of sectoral excellence— strengthening program quality and technical supervision and breaking away from the HOPE regional management structure. Combining this vision with the goal of growing the portfolio, there is likely a difficult balance between head-office-led and country-led portfolio quality that needs to evolve. 1.5.2 Quality and Status of Detailed Implementation Plan (DIP) The evaluator was completely satisfied with the quality of the DIP and the Annual Reports, in terms of clarity and adequacy of the narratives, description of objectives and indicators, and of activities. Country offices have their own individualized DIPs and prepare individual annual reports that are then ‘rolled up’ at the central level into the final document submitted to the donor. The evaluator finds this is project strength. 32 One area of minor improvement might be that project managers did not seem to know how charts reporting improved income (included in the Annual Report appendices) had been developed. This might be an indication that preparation of this report could be delegated. 1.5.3 Financial Management The project appears to be well managed financially. When the evaluator asked budget questions, the answers were provided by return-of-e-mail. It appears all budgets have been expensed to at or near 100%. When a brief analysis was conducted of each country’s expenditures, the ratio of staff expenses to total expenses and country office overhead to project expenses were both judged in line with industry standards. Project HOPE appears to have a somewhat old-fashioned financial system whereby expenses are not booked until inputted at Headquarters, and more evaluator time could have been expended in a review of financial operations than was available; but the overall judgment would likely not be any different. The Program Director is a strong performer in financial realms and has controlled the VHB budget well. Noted above in the paragraph on Scale-up, Project HOPE is leveraging additional resources beyond the match in annual funding for VHB operations in the Dominican Republic, Thailand and Mozambique. 1.5.4 Monitoring and Evaluation (M&E) M&E system As described elsewhere in this document, on the credit side the VHB’s monitoring system supplies accurate, reliable and timely performance data. The monthly VHB-TRAK monitors bank formation, number of participants, target number of loan disbursements, disbursement amounts, total loan portfolio, savings, operational sustainability, arrears, as well as a dozen other indicators. The micro-credit evaluation mechanism collects economic and household data on each entrant to the VHB during the 1st cycle and then again on a sample of those women who are still active at the close of the 4th loan cycle. One weakness that came to light during this Final evaluation is that of sample size. In one country, the sample was substantially inadequate; in another, it was adequate but incomplete. The Final evaluation process also demonstrated that potential income gains to over 3,400 clients who left the project had to be inferred rather than documented, because improved income data were not captured at the time those women were still involved in the program. On the health side, monitoring and evaluation are less well developed and have taken some years to firm up. See the full discussion on the “learning curve” issue above. With the HOPE HQ/Peru decision to collect health data at the start of a woman’s career in the project at the same time as economic data are being collected, an acceptable methodology 33 and format have been established and are ready to go into full implementation throughout VHB. Country impact evaluation Reported at length in the Midterm evaluation, in March 2001 a Ph.D. research anthropologist carried out an evaluation of the health impact of the village health bank program in Ecuador. The study sample involved current clients and a control group: 482 people were interviewed, and biological samples and anthropometric measurements were taken on 326 VHB and 156 control women and children. Among the most important findings were the following: · There were no statistically significant differences between VHB and control women on many socio-demographic variables; but there was in VHB members using rainwater run-off for drinking purposes more (p. 0.001), and more reported VHB household expenditures on health care (p. 0.02). · There were no statistically significant differences between VHB and control women on most health access indicators; but there was for members’ screening for cervical cancer (p. 0.019) and those screened in the last 12 months (p. 0.035). There was also a statistically significant difference in VHB members reporting being able to finance health care needs during the past month (p. 0.016) than control women. · There was no statistically significant difference between VHB and control women in morbidity indicators: food and water borne infections, upper and lower respiratory infections, etc; but there was an enormous statistical difference in VHB members reporting reduced incidence of mosquito-borne malaria and dengue (p. 0.0000003). · There were no statistically significant differences between VHB and control women in most anthropometric indicators: mean weight, height, median arm circumference, etc. There were two puzzling statistically significant differences between VHB and control women’s perceptions of family health: control women reporting their current health status as good or very good compared to VHB members (p. 0.011) and VHB members reporting themselves more “food insecure” and “food insecure with hunger” (p. 0.022). These differences perhaps reflect increased aspirations in VHB women. Impact Results Summary: VHB women were likely more forward thinking than their peers were and exploited their environment more. They took better care of their adult children and grandchildren. There was evidence their expectations of good health had gone up because of the project. They engaged in improved health care seeking for them and their children and grandchildren, and they used of some form of family planning more. They experienced an enormous reduction in malaria and dengue, two devastating diseases in the area. From the profits from their village banking activities, they spent nearly double on medicines compared to non-VHB members. 34 The study demonstrated convincingly that the VHB had a substantial impact on the health of project participants. It was also evaluated as high quality, impact directed research. 1.5.5 Information Systems Regarding VHB’s sharing of program information, all communication is conducted via e￾mail; all programs have access to the internet. VHB uses database systems to collect, analyze and report information. HOPE Guatemala has embarked on automated data entry. As a program, VHB is looking at using hand-held computers to enhance monitoring and evaluation. Regarding shared learning, annual meetings of the Latin American VHBs are held— sometimes including participation from “graduated” countries. These workshops have rotated among the different VHBs, each year focused on a specific theme. From the Midterm on, peer visits are also a part of VHB. Peru staff visited Ecuador and Honduras; Guatemala staff visited Honduras. Honduras staff visited Guatemala and (noted elsewhere) Malawi staff visited Thailand, and Thailand staff, Malawi. In addition, Project HOPE has an annual Fall Leadership event at HOPE Center that country representatives attend. 1.5.6 Staffing and Supervision Country narratives attest to the fact that in general the project is adequately staffed at the field level; there is only one exception to this pattern (Malacatán, Guatemala). In fact, the opposite might be more an issue: the project appears to be over-staffed in two countries. Project management suggests an appropriate ratio of clients to staff is 300:1, more or less an industry pattern. Counting field staff and supervisors, the ratio in Malawi is 417:1, quite efficient; however in Guatemala, it is 169:1, and in Peru it is 153:1. Comparable experience from colleague organizations in Guatemala and Peru suggest these ratios represent quite intensive supervision. One way to build a portfolio is to hire over current requirements; another is to grow “organically.” VHB has opted for the former strategy in two countries; in times of scarcity, the latter may become appropriate. Country narratives discuss staff qualifications and performance as follows: · In Malawi: All field staff are ex-teachers with considerable personal warmth and ‘presence’ to carry out vibrant, community promotion. Adult education skills of these staff are still developing, in part, because they are ex-teachers, in part because the previous health education called for “transmission” of a lot of content, and in part for other reasons. · In Guatemala: Most though not all field staff demonstrated good use of names, adequate preparation for the session, and carried out the health talk largely as planned. Overall effectiveness of the talks was as follows: two sessions with good facilitation but incomplete technical competence; two sessions with adequate technical content and facilitation; one session with good content and facilitation; one session with good technical content and excellent facilitation. 35 · In Peru, most field promoters are strong performers in health talks. Overall rating of the sessions observed produced three “A’s,” good technical content and wonderful facilitation; two “B+s,” good facilitation and good content; and one “C”, adequate facilitation, less adequate technical content. Supervision of staff varied throughout the program. In Malawi, field staff had not been supervised much for almost a year. In Guatemala, the supervision was good in one geographical area, and insufficient in the other. In Peru, the supervision in health was excellent, perhaps less so in micro-credit. 1.5.7 PVC Program Management It appears that BHR/PVC’s oversight of the grant has improved in the last several years. VHB reports that oversight was practically absent during the first several years; recently staff has initiated more dialogue and contact. VHB’s reporting and documentation are apparently well viewed in PVC. These comments having been said, the level of PVC involvement has not been a hindrance to project management. 1.5.8 Program Management Lessons Learned and Recommendations Summarizing this document, the following seem to be key lessons learned and recommendations that come out of the five-year VHB project. The table is followed by a brief paragraph on each topic. Table 6: Key Lessons Learned and Recommendations Lessons Learned Recommendations 1. Ambitious goal formulation HOPE to scale down some of its goal expectations in future proposals; also to dedicate a larger amount to grant financing in start-up operations. 2. Pace of program changes Establish calendars for program adaptations and accelerate accomplishment timetables. 3. Organizational learning Country staff to be tasked to “think outside the box;” performance indicator added to project manager annual evaluations regarding “proactivity.” 4. High dropout rates Re-emphasize to staff the problem of disguised desertion and high dropout rates; establish performance indicators aimed at reducing these rates. 5. Importance of member savings Orient program staff to the economic impact of improved member savings; create a realistic program indicator; establish portfolio and staff performance indicators. Consider studying the indicator as one related to individual sustainability. 6. Continued HOPE HQ financial support for VHB FY04 budget support for three country operations. 7. Improved Adult Education Use Peru staff as resource to consolidate the gains in adult education in Malawi and Guatemala. 8. Improved health data collection Collect health data for first cycle entrants; re-collect after 4th cycle; supervise compliance by requiring reporting to HQ on these data every six months. 9. Improved income data collection Collect income data for first cycle entrants; re-collect on all fourth cycle participants. 10. Higher profit loans Embark on economic analysis of individual loan applications on a limited number of applicants being careful not to over-tax field staff with this extra responsibility. 11. Operational Mount aggressive campaign of cost-reductions to improve cost/sustainability 36 Lessons Learned Recommendations sustainability ratio in current VHB countries. Release supernumerary staff in each country. HQ to provide more hands-on oversight of country budgets. 12. Malawi decision￾making HQ support of the decision to continue VHB presence in Mulanje; also, HQ to push hiring an Adult Health Educator in Malawi. 13. Peru portfolio survival Conceptualization and creation of a new, more viable program strategy. 14. Country Director involvement With Country Director input, Program Director-Income Generation to articulate in writing a “management philosophy” paper thinking through country leadership styles and quality portfolio issues. Performance indicators for supervision of VHB gets added to C.D. annual performance 15. Program Director management style With VP-International Programming support, HQ Program Director-Income Generation articulates a “management philosophy” paper thinking through leadership and quality portfolio issues. 16. Scaling up Continued HQ efforts to scale the VHB up in new countries; preparation of project proposals for new donors Ambitious goal formulation. The conceptual attractiveness of the HOPE VHB “credit plus” model may be responsible for underestimating operational difficulties. Country￾specific adaptations will always be required and take time to recognize and implement. The ratio of clients per bank was adapted from Ecuador, a sui generis context; all growth￾of-portfolio projections based on those figures have been shown to be overambitious— with significant impact on the accomplishment of program targets for the entire life of grant. Future VHB proposals can be more realistic in what is achievable in a given grant period. Pace of program change. Continuing the earlier discussion on this topic: was country staff sufficiently active in moving forward on Midterm recommendations? Peru was the first to move forward regarding a reformulation of the health education methodology. Guatemala moved forward on an organizational self-assessment that identified three strategic areas of improvement: administrative manuals, institutional strengthening, and portfolio growth. If the answer is “probably yes,” the project moved forward with as much speed as one could legitimately expect given the complexity of the change, the question then becomes “how can HOPE accelerate the process the next time?” Relatedly, what is the extent of local country commitment to VHB? Are the HOPE country operations “implementers” of VHB, or “owners?” Organizational learning. Repeating the question from earlier in the narrative: have the learning curve, innovation and pro-activity of VHB staff been all that one could hope for? Has VHB staff been sufficiently thought provoking and challenging of one another during this grant? One tends to conclude the answer to this question is “no.” To the outsider, it seems VHB has been a bit of a “train on the track,” steaming down the path that was laid out without adjusting as much as was needed to changed condition along the way. The strength of VHB’s financial monitoring may have lulled project staff into thinking that all program issues were being addressed. It seems HOPE HQ and individual country operations need to think “outside the box” more. High dropout rates. The dropout rate is particularly acute in Peru but also has a bearing on the other two operations. It appears VHB has accepted that high dropout rates, characteristic of the micro-finance industry, are a fact of life. However if some 37 reductions in these rates could be achieved, the project would become more efficient as well as more effective. If women’s businesses become more successful, more women should be interested in staying in the project longer. It may be appropriate for VHB leaders, the Program Director and the project managers, to think about striving for some “systemic” improvement in this area. At the same time, much of this analysis suggests the project can begin to look more carefully at those who complete four cycles, aggressively monitoring improvements in health and in income. Much of the speculation on improvement in four-or-more￾cycle participants can be documented if women who complete this milestone are carefully monitored, surveyed and their changed situation documented. Member savings. The project currently tracks member savings each month. However, there is no explicit linkage between accumulation of member savings and the overall effectiveness of the project. When a business fails and a woman withdraws from VHB, her accumulated (enforced) savings are captured as a last-resort mechanism to pay off the loan. In business terms, this is justified. However, VHB staff does not yet perceive it as a setback to project impact. One recommendation is for VHB to create an explicit ratio of loans-distributed compared to member savings, or portfolio value compared to savings to strengthen the focus on the importance of steady growth in member savings. HOPE HQ support for VHB. Given that none of the three countries has yet achieved operational sustainability, Project HOPE should continue to finance the VHB operation in two of the three countries at the conclusion of the current grant and study seriously the situation in Peru. The program model is an effective one that responds to key needs and should be continued. One hopes full operational sustainability can be achieved in two countries in the near future, and HOPE HQ financial support should be provided to allow that goal to be achieved. On another front, the V.P.-International Programming informed the evaluator that a new position has recently been created at HOPE Headquarters, that of Deputy Director for Income Generation. The commitment of additional HOPE resources to the hiring of a head office resource person to support VHB is another indication that VHB has become “mainstreamed” in the institution, one of the primary goals of the project. Improved Adult Education. The Midterm recommendation for hiring an adult educator at HOPE HQ was not acted on—for financial reasons that, if unfortunate, are understandable. However, a good alternative to this recommendation was the hiring of the Peru Adult Educator, whose performance has been evaluated highly in the Peru country narrative. A similar position has been created in the Malawi portfolio. HOPE HQ support for these two positions should continue. A less important recommendation regarding off-budget support for adult education materials may also be worth considering. Improved health data collection. As determined by the project a short time ago, health data should be collected at the start of a women’s career in VHB just as income and 38 expenditure data are collected; and improvements in health should be measured at the close of the 4th loan cycle just as improvements in income and expenditure are measured. Improved income data collection. Continue to collect income data at the start of a women’s career in VHB and measure improvements in income at the close of all clients’ 4 th loan cycle loans. Improved loan profitability. Until now, as long as the VHB group proposed the loan and it did not seem too risky, the project has not much focused on business analysis behind the loan applications. Within understood time constraints, field staff could begin to study the underlying soundness of individual loans. So as not to over-burden field staff with this role on top of their health education responsibilities, such analysis could be done on a limited basis: perhaps on one or two of the newer members joining a successful bank, or perhaps mostly on 1st or 2nd loan cycle clients. Several colleague NGOs are experimenting with this idea. Operational sustainability. In each VHB country, a cost-reduction exercise should begin (or continue if already started). Appropriate changes in each country could be to reduce over-staffing, improve control of operating expenses, use vehicles better, carefully control VHB country head-office support, and reduce office size and institutional “footprint.” Paradoxically, VHB salaries may need to be increased in one country or other to retain qualified staff. At the same time, if a campaign to reduce dropouts is successful, program revenues will rise thereby enhancing operational sustainability. Malawi decision making. It is encouraging that the recommendation in the Malawi country evaluation to keep the Mulanje office open has been accepted. At the same time, a campaign to hire a national-staff Adult Educator to complement the skills of the expatriate Adult Educator should take place. Peru portfolio survival. It is recommended VHB-Peru give serious study to determining a viable program strategy. The project has not been able to slow the exodus of its clients. As a result, the project is constantly opening new (and untried) banks and promoting new and untried clients, while older banks and older clients leave. A new program design needs to take place to adjust to these realities. HOPE HQ staff, the Peru Country Director and the Project Manager should give serious thought to this issue. Country Director and Program Director involvement. It appears that leadership of the Country Directors and the Program Director may have been overly “delegative.” The management philosophy of situational leadership suggests when things are going well, the leader steps back; when things are deteriorating, the leader needs to intervene, sometimes quite vertically. Management literature1 suggests that “democratic” leadership may not always be appropriate and that management for performance is not necessarily hands-off. If this perception is valid, one way to pursue the idea would be to debate vigorously within VHB and prepare a position paper on what effective, results￾oriented management would look like in operational terms in VHB. 1 Kouzes and Posner; Goleman, Bouatzis and McKee; Lencioni. 39 Scaling up. If, as desired by the Vice President for International Programming, the VHB is to become a core pillar of excellence in Project HOPE, new donors and new programs will need to be found. The findings and conclusions of this document may strengthen HOPE’s ability to improve the current portfolio and attract new donors for VHB. 1.6 Evaluation Methodology The evaluation took place from June 2 to Aug. 15 as follows. Three half-days of interviews were held at HOPE Headquarters. Field travel took place during the weeks of June 15-20 to Malawi, June 29-July 5 to Guatemala, and July 6-12 in Peru. Write-up took place from July 21 to August 14. Four stand-alone documents were produced: Final Evaluation reports for each of the three country operations, plus the USAID Matching Grant Final Evaluation, which is this document. The methodology was as follows: review of documents; in-depth discussions with HQ program management; field visits of one week each to Malawi, Guatemala and Peru; holding in-depth discussions with the project managers and program support technicians during that travel; field travel to a number of VHB meetings, including direct observation of health talks and loan operations; and semi-structured interviews with randomly selected VHB members. The following table summarizes the field travel. Table 7: Summary Information on Country Visits Program VHBs Visited Members Participating Individual Interviews Field staff Observed Discussions w/ Staff Malawi 7 83 4 6 (of 8) 8 Guatemala 9 111 22 5 (of 7) 14 Peru 8 99 30 4 (of 5) 6 Total 24 294 56 15 28 Of twenty HOPE field staff, fifteen were observed as they engaged in project promotion, 75%. Twenty-four banks were visited for approximately two hours each. Participation of members in these meetings was 294 women, 77% attendance not including excused absences. Post-event conversations took place with fifty-six women selected at random from the members who attended the bank meetings. During the travel, discussions were also held with field and administrative support staff, 8 in Malawi, 14 in Guatemala, and 6 in Peru. In addition, interviews were held with two colleague organizations in Guatemala and six in Peru. This large number of informants allowed the evaluator to form a comprehensive picture of most of the strengths and weaknesses of the project in the three countries visited. The evaluation was carried out by a thirty-year experienced development practitioner with specialization in rural agricultural development, rural sociology, primary health care, and organizational management. The evaluator was also the external evaluator of the VHB Mid-Term in September 2001, allowing some additional comparisons between 40 the time of Midterm and Final evaluation; he was also the author of the Midterm Peer Review study paper. 1.7 Evaluation Attachments Appendix A: Approved SOW Appendix B: DIP Matrix and Results Status- Overall project and per country Appendix C: Program Strengths and Weaknesses Table Appendix D: Final Evaluation-Malawi Appendix E: Final Evaluation-Guatemala Appendix F: Final Evaluation-Peru Appendix G: List of persons/groups interviewed Appendix H: Bibliography Appendix I: VHB Photographs 41 APPENDIX A Scope of Work Project HOPE Matching Grant Final Evaluation Program Context: Project HOPE received a USAID Matching Grant for the period 1998 – 2003 in order to replicate HOPE’s Village Health Bank methodology which integrates access to credit with health education for impoverished women to three new countries: Malawi, Guatemala, and Peru. The overall objective was to alleviate poverty and it’s effects on women and children by increasing incomes and improving health conditions and personal well-being. Specific program goals to achieve this objective include: · Strengthened capacity of Project HOPE to implement successful Village Health Bank Programs · To provide valued financial services and health education in a sustainable manner · To improve income and health levels of impoverished women and children Objectives: In accordance with the requirements of the USAID Matching Grant, Project HOPE is completing a final evaluation of achievement of the goals, indicators, and impact of the projects funded with this grant. Specific objectives of this evaluation include: 1. Conduct an evaluation of Project HOPE’s Village Health Bank activities in Malawi, Guatemala, and Peru. 2. Assess HOPE Headquarters progress with the grant. 3. Prepare a report consolidating all the findings generated by this evaluation. Role of Evaluator: Project HOPE will hire a mutually agreeable independent consultant to conduct and report upon the evaluation. Specific responsibilities include: · Review supporting documents pertaining to the USAID Matching Grant. This includes the grant application, detailed implementation plan, annual reports, policy and procedure documents, evaluations, and other internal information. · In partnership with Project HOPE and relevant USAID personnel, coordinate planning of activities for the execution of this final evaluation. · Conduct an assessment of Project HOPE’s Village Health Bank programs in Malawi, Guatemala, and Peru in achieving the grant objectives, indicators, and impact on target populations using the below evaluation guide and accompanying tables. · Conduct an assessment of Project HOPE/Headquarters progress in achieving the grant objectives and indicators using the below evaluation guide and accompanying tables. · Develop recommendations for HOPE to strengthen our continuing execution of the VHB programs. 42 · Prepare a final report consolidating all the findings generated by this evaluation. This report should follow the below guide, with a distinct report on each country, along with an overall report. We are planning to engage Frank Sullivan (who conducted our mid-term evaluation) to lead this final evaluation. Mr. Sullivan will conduct this evaluation independently of participation by Project HOPE Headquarters personnel. Methodology: The principal strategy for this final evaluation is to have the same person who conducted the mid-term evaluation also conduct the final evaluation and use the previous experiences as a basis for comparison with current results to document progress and determine success in achieving grant objectives. Specific elements include: · Review of supporting documents related to the USAID Matching Grant and it’s projects. Specifically, review of the mid-term evaluation reporting. · Field visits to 3 countries (Malawi, Guatemala, Peru) · Visit to HOPE Headquarters to interview key staff and review information systems involved in implementation of the USAID Matching Grant. · Interviews with key program staff involved in implementation of the projects. · Open discussion with program staff about program advances, challenges, and results. · Review of internally generated operational documents, performance reports, and impact information. · Verification of the systems that generate the above information. · Direct observation of field activities specifically related to health education. · Informant interviews and focus group discussions with direct project beneficiaries to verify their perceptions and specific benefits gained. Report Format: A report of acceptable quality in accordance with this Scope of Work will be prepared. This report will follow the format prescribed by USAID and address the specific evaluation questions. A distinct report will be prepared on each country, along with an overall report summarizing. These reports will be completed in Microsoft Word. Proposed Deadlines: · Completion of SOW June 1st · Project Assessments June 15th – July 12th · Completion of draft report July 25th · Stakeholders review August 8th · Revisions & final report completion August 15th · Evaluation debriefing August 22nd Evaluation Questions: Section numbers refer to the actual report document. 43 1.4 Program Effectiveness 1.4.1 Program Model or Approach ß Determine if the hypotheses and assumptions under-pinning the program model were sound. 1.4.2a Achievement of Objectives Identify the program objectives and assess progress made to date toward achieving each major objective defined in the DIP, for the PVO’s HQ program and for each country program. Under each main objective (as defined in the DIP): Assess progress toward each major objective. Provide evidence, criteria for judgment and cite data sources. ß Identify major successes, challenges and constraints in achieving each objective. Include a chart that summarizes the program’s successes and weaknesses (see suggested format in Annex A). Attach a summary of the program’s DIP results status using template given in Annex B; ß Assess the impact of the program on strengthening the capacity of the PVO’s operations at HQ and in the field to deliver sustainable services; and ß Discuss any unintended impacts to date - positive or negative. Assess if the main strategies and activities employed to achieve results were effective. Identify any changes made during implementation. ß Have the strategies and activities been effective in achieving mid-term/end of project (EOP) targets? ß Did the PVO conduct a mid-term evaluation or data review? Was program implementation on-track? Were mid-course corrections made and what are the effects of these changes? 1.4.2a Impact on Target Populations ß Discuss the impact of the program on target populations; ß Review program impact information and the design, systems, instruments, and procedures used to collect and document it. ß Verify the results through target population input. 1.4.3 Cross-cutting Issues Partnerships (Note – as this issue was not a stated objective it should be commented upon but not used as part of the evaluation criteria): Assess the status and outcomes of the PVO’s partnerships with local organizations, NGOs, networks, businesses or intermediate support organizations: 44 ß Assess the effectiveness of the PVO’s approach to building in-country partnerships. Identify the key elements that contributed to success. ß Identify local partners and cite changes in local partners’ institutional capacities that occurred as a result of the program partnerships. Identify what activities were most instrumental in strengthening local partners; ß Cite changes in the PVO’s institutional capacities, as a result of learning from in￾country partners; ß Discuss if these partnerships are mutually satisfactory and beneficial. Identify and discuss major constraints to developing durable and productive partnerships; ß Identify outcomes of program links with any coalitions, networks or associations; and ß Attach Partnerships Table (see template in Annex C) New Tools, Guidance Or Standards: Identify if new tools/guidance, approaches, or program standards were developed under this program: Assess if these tools, approaches etc. were effective and merit broader distribution or application. Sustainability and Scale-Up: Assess program sustainability and scale-up plans of operations: ß Magnitude of the program: size of beneficiary population by age and sex; ß Progress and potential for scale-up or replication; ß Progress made by the program toward achieving sustainability and if the PVO is planning to measure post-grant sustainability; ß Progress program is making toward diversifying resources, for example building alliances with in-country business organizations. 1.4.4 Lessons Learned and Program Recommendations ß Outline the main lessons learned from the MG program that would be applicable beyond the program sites. ß Provide recommendations for the PVO, partner organizations and USAID/PVC 1.5 Program Management 1.5.1 Management Approach ß In the context of the program model and the changing country situation, is the overall approach to program management flexible, appropriate and adequate? ß Is the program cost effective and timely in converting inputs into outputs and outcomes? 1.5.2 Quality and Status of Detailed Implementation Plan (DIP) ß Discuss the quality of the DIP, i.e. the clarity and adequacy of the objectives, indicators, baseline studies and activities. 45 ß Comment on the utility of the DIP as a management tool for the PVO, partners and PVC. 1.5.3 Financial Management ß Are adequate financial control systems in place (the intent of this question is to comment upon observed systems not audit)? ß Is the PVO leveraging additional resources (beyond the match)? 1.5.4 Monitoring and Evaluation (M&E) System ß Does the M&E system supply accurate, reliable and timely performance data? ß How effective are the steps (if any), taken to institutionalize M&E at PVO HQ? ß Assess if the partner organizations have increased their capacity to monitor and evaluate their work, document program achievements, and use data for decision￾making and program advocacy. ß Has the program undertaken to date, any special studies to assess program operations or impact? Comment on the quality and utility of these studies. ß Assess PVO and partner use of data to make management decisions. ß What more could be done to improve the M&E systems and use data for decision￾making and program advocacy? ß Verify data pertaining to a random sample of indicators. 1.5.5 Information Systems ß Has the program increased in-country partners’ access to information technologies? ß What steps have been taken by the PVO and its partners to share program information and learning? 1.5.6 Staffing and Supervision ß Do the PVO and partner organizations have an adequate number of staff with relevant expertise for supervising/backstopping the program? 1.5.7 PVC Program Management ß Assess PVC’s oversight and backstopping of the cooperative agreement (based upon the PVO’s feedback). 1.5.8 Program Management Lessons Learned and Recommendations ß Discuss the management lessons learned and recommendations to the PVO, in￾country partners and PVC. 1 ANNEX B1 – DIP RESULTS STATUS – VHB OVERALL Strategic Objective 1: Strengthened [HQ] Capacity to Implement Successful Village Health Bank Programs DIP OBJECTIVES DIP INDICATOR Baseline EOP TARGETS ACHIEV EMENTS VERIF IED? COMMENTS A) Formation of an Income Generation department at HOPE Center 1) Team of technical specialists working together to advance the VHB methodology 1 full time person 3+ persons working together for an equivalent of 1.5 full time positions Accmplshed Yes Hope Center team: 1+ ½ time Country Project Managers 3 LA Reg. Adult Educator 1 Africa Adult Educator ½ time 1) Development of technical & operational guidelines No guidelines by each program Technical manuals for the methodology, with modifications Accmplshed Yes Manuals seen and reviewed in three countries; adapted to country conditions 2) Definition of core health objectives for all VHB programs No core health objectives Defined objectives and common educational processes Accmplshed Yes In a change from Midterm, staff in all three countries is now clear this is primarily a health project. B) Development of a common, yet flexible methodology building on success experiences 3) Development of MIS for program monitoring and impact None Functioning systems Accmplshed Yes Microfinance monitoring/ impact measurement are generally strong. Health monitoring/ impact measurement are not yet up to speed, but agreed-upon methodology is ready for wide replication 2 DIP OBJECTIVES DIP INDICATOR Baseline EOP TARGETS ACHIEV EMENTS VERIF IED? COMMENTS 4) Creation of a network of VHB practitioners Minimal interactio n Exchange opportunities 2 times per year with mentoring responsibilities Accmplshed Yes Experienced staff available in each country. Cross￾fertilization taking place in joint annual workshops; cross-visits also taking place at least once a year. 1) VHB model adapted to unique characteristics of each new implementation site as evidenced by: - % that are operationally sustainable on monthly basis by grant end; 0% 100% Accmplshed Not Accmplshed Yes Methodology has been implemented in three very different countries. 100% operation sustainability has not been achieved in any of the 3 start-up countries. C) Implementation of the VHB methodology in varied settings through replication to 3 new countries, and expansion in 1, all with successfully operating programs - % with high program quality of 30 day arrears < 5% and annual loan losses < 2% 1 program 4 programs with 30 day arrears less than 5% of outstanding loans and annual loan losses < 2% Accmplshed in two countries Yes Achieved in Guatemala and Malawi; not achieved in Peru. 1) % of USAID support 50% of resources from USAID 20% of resources from USAID Accmplshed Two non-USAID donors for current project. Three non-USAID donors for scale-up of VHB in three other countries D) Greater sustainability of the VHB programs through diversification of their funding sources 2) % of VHB programs utilizing commercial or investment capital 0 of 2 programs using commerci al capital 80% of 5 programs N/A N/A No longer relevant per mid￾term evaluation. HOPE BOD has made a policy decision not to pursue such financing. 3 DIP OBJECTIVES DIP INDICATOR Baseline EOP TARGETS ACHIEV EMENTS VERIF IED? COMMENTS 1) % of USAID support in annual budget 50% of resources from USAID 25% of resources from USAID Accmplshed Yes Complementary U.K resources added to two country operations, Malawi and Guatemala Two country operations opened, Dom. Republic and Thailand, without USAID funding E) Increased sustainability of HQ technical services 2) % of funding from client payments in annual budget 0% 25% No No No third-country support planned for HOPE Center costs 1) Number of independent national organizations 0 4 organizations N/A N/A Indicator no longer relevant per mid-term evaluation. F) Development of independent national organizations that operate the VHB program 2) % of partner organizations that are operationally self￾sufficient on monthly basis n/a 100% N/A N/A Evaluator comment: Not applicable G) Establishment of a program performance monitoring system 1) Functioning performance monitoring system None Functioning system Accmplshed Yes Microfinance monitoring is strong; health monitoring is not yet fully up to speed, but methodology and format have been developed. H) Establishment of a program impact assessment system 1) Functioning impact assessment system None Functioning system Half accmplshed Yes Microfinance impact measurement needs to be strengthened. Health impact measurement not yet in place but methodology and format have been developed. 4 Strategic Objective 2: Provide Valued Financial Services and Health Education in a Sustainable Manner DIP OBJECTIVES DIP INDICATOR Baseline EOP TARGETS ACHIEV EMENTS VERIF IED? COMMENTS 1) # of new staff trained in VHB methodology 1) n/a 50 Accmplshed Yes FY03 field staff observed: Malawi: 13 observed Guatemala: 14 observed Peru: 10 observed A) Develop local staff capacity to implement the VHB methodology 2) % of new VHB programs with documented policies & procedures 2) 0 100% (3 programs) Accmplshed Yes Manuals seen and reviewed in three countries; adapted to country conditions B) Create local NGO institutions to continue the program 1) Number of independent national organizations 1) 0 organizatio ns 4 organizations N/A N/A Indicator discarded at recommendation of Midterm evaluation. C) Achieve sustainability of operations 1) % of partner organizations that are operationally self￾sufficient 1) n/a 75% [Ed. 100%?] Not accmplshed Yes EOP status: Malawi: 58% Guatemala: 69% Peru: 45% D) Serving a significant population 1) # of new VHB program participants reached with resources of this grant 1) 0 12,300 participants 131% of goal total, but note indicators for “1yr.+” women and currently active women Yes Total >1 Yr. Active Malawi: 6,720 2,343 2,935 Guate. 4,332 2,803 1,861 Peru 5,084 1,151 1,071 Total 16,136 6,297 5,867 131% 51% 36% 5 2) # of beneficiaries (participants + family members) of the resources of this grant 2) 0 75,000 beneficiaries 132% of goal total, but note other indicators also Yes Estimate Total >1 Yr. Active Malawi 47K 16.4K 26K Guate. 24K 15.4 10K Peru 28K 6.3 6K Total: 99K 38K 42K Of Goal 132% 51% 55% 3) # of VHBs within Project HOPE 3) 230 VHBs 285 130% of goal No Malawi: 251, active 197 Guatemala: 177; active 104 Peru: 185; active 70 Total: 613. Active 371, 130 % of goal, not counting Ecuador E) Develop access to resources to support long term growth 1) % of VHB programs utilizing commercial or investment capital 1) 0% (2 VHB programs) 75% (5 VHB programs) N/A N/A Indicator no longer relevant per mid-term evaluation. 1) Arrears more than 30 days old 1) n/a < 10% of portfolio 14.87% Yes Malawi: 13% of $215 K Guatemala: 3% of 163 K Peru: 28% of 178 K Weighted total: 82.7 K/556 K F) Maintain high program quality 2) Annual loan losses 2) n/a < 5% of portfolio Accmplshed Yes Accomplished in Guatemala and Malawi 6 Strategic Objective 3: Improve Incomes and Health Levels of Impoverished Women and Children DIP OBJECTIVES DIP INDICATOR Baseline EOP TARGETS ACHIEV EMENTS VERIF IED? COMMENTS A) Financially viable, democrat. managed community institutions 1) # of new Village Health Banks formed 1) n/a 285 new VHBs 130% of goal Yes Malawi: 251, active 197 Guatemala: 177; active 104 Peru: 185; active 70 Total: 613. Active 371, 130% 1) # of women receiving loans 1) n/a 12,300 Women 131% of goal; Note other figures as per SO 2.D.1 Yes Total loans: 16,136, 131% > 1 Yr. loans: 6,297, 51% Active clients 5,867, 48% 2) # of loans disbursed 2) n/a 55,000 loans disbursed 59.5% accmplished Yes Malawi: 11,967 (6/02/03) Guatemala: 10,562 (6/30/03) Peru: 10,221 (6/30/03) Total: 32,750 3) % of loan disbursements less than $300 3) n/a 75% Accmplshed Yes Average loan size FY03 Malawi: $139 Guatemala: $200 Peru $ 173 B) Improved economic opportunities for impoverished women 4) Participation by impoverished women 4) n/a 75% of participants report incomes below poverty line No Data requested, not yet received 7 C) Increased incomes of program participants 1) Average change in sample of participant incomes 1) n/a 20% improvement Accomplish ed as per the comments column Yes See narrative explanation. Big Gains Some imp. Malawi 20% 755 1,588 Guate 50% 829 1,974 Peru 15% 204 484 1) Number of educational sessions provided 1) n/a 12,000 education sessions D) Increased Accmplshed Yes Total LOP 13,003 exposure of impoverished women to health education lessons 2) Average # of health education sessions received 3) n/a 15 Not likely No High drop out rates make collection of this information extremely difficult E) Improved health status of participating women 1) Changes in consolidated health score 1) n/a 75% of participants report 25% or better improvement 32% accmplshed Yes See narrative. Substantial improvements in health for a est. 2,250; some improvements for another 4,046. Total 32% of goal for both groups combined. 8 ANNEX B2 DIP STATUS RESULTS - MALAWI DIP Objectives DIP Indicators EOP Status Year 5 Status • VHB policies and procedures defined for successful implementation in this area; Accomplished - Appropriate credit policies and procedures developed and in place. Documents discussed and reviewed between the evaluator and the Project Manager during evaluation travel. Mulanje and Blantyre conditions are very different, but procedures seem to be working in the latter area as they did in the former area. Since the evaluation VHB￾Malawi has taken the decision to continue working in Mulanje. VHB model adapted to socio￾economic characteristics of Mulanje District. • Completed lesson plans for health and business education that is appropriate to the needs of the beneficiaries; Not yet accomplished but action under way Health education plans are still in development, but are showing good movement forward. See narrative. Business plans have not yet been systematized but VHB has contracted an outside expert to give training to selected village women on better business planning. Evaluation recommends that this be an area of project interest in FY04. • 120 new VHBs will be created; 164% accomplished - 251 Village Health Banks have been opened with 197 still in active operation, 164% of goal. In addition, the Evaluation exercise developed a project consensus that 32 banks are “A+”. Expansion of the VHB program to Malawi • 5,500 women will receive loans Accomplished - Total 6,720 women have received loans, 161% of goal - Total 1 yr. or more, 2.343, 43%. - Total 2,935 are active, 53% of goal. • Arrears over 30 days less than 5% of outstanding loans; Not accomplished - Arrears over 30 days are 13%. • Loan loss rates of less than 2% of average portfolio; Accomplished - Loan defaults for FY03 are less than 1%, in part because the project is cautious in declaring the loan uncollectible. Maintenance of high program quality • Operational efficiency of less than 15% by grant end. Data not collected. • 25 staff trained in VHB methodology Accomplished - 28 staff trained in the methodology. Improved institutional capabilities • Development of planning, administrative, and management systems and documentation to support a growing institution; Accomplished - Loan management and Accounting Systems are in place. Performance monitoring system is being used. Overall management of the portfolio seems strong. - Decision to enter Blantyre was strategic and well-thought out. Decision to leave Mulanje was less well researched. 9 DIP Objectives DIP Indicators EOP Status Year 5 Status • Upgrade of the Financial Management Information System to assess economic impact on beneficiaries; Accomplished - Project has established an effective tracking system to measure improvements in income by collecting economic and household data on each entrant to the VHB during the 1st cycle and compared the results at the conclusion of the 3rd loan cycle. Improvements ranges from 21% to 37% (see narrative) and are consistent over time. • Development of an Impact Assessment System; Accomplished in micro-finance; not accomplished in health - Economic impact is well measured, as per the above. - Health studies show expenditure of great effort, but methodology has contributed to unreliable data. - Anecdotes of successes have been prepared but were not of much usefulness for evaluation purposes. See extensive discussion on this subject in the country text. • Achieve a 100% level of operational sustainability; Not accomplished - Operational sustainability is 58%. Plans to achieve 100% depend on the continued success of the Blantyre initiative. • Commercial funding is utilized to finance expansion of the loan portfolio; N/A - No longer applicable. Self-Sufficiency • Accessing in-country resources to complement program operating and growth needs; Not accomplished - Not accomplished. Malawi is a very difficult environment for anything except donor programming. HOPE Malawi has not successfully marketed any large new projects in several years • Income of participating women increases by 20% annually; Fully accomplished for perhaps 755 clients; less impact for 1,588 - Considerable confidence to the assertion that 755 clients have benefited by increases of 20% or a little more; less impact estimated for another 1,588. Increased beneficiary impact • 70% of VHB members will be engaged in improved home-based health practices; Accomplished for perhaps 750 women - Extrapolating from “A+” banks, it is possible the project has had substantial impact on the lives of approximately five hundred and fifty people; somewhat less impact may be surmised in the lives of another similar number. There has not been sufficient time to show improvement in Blantyre participants. It seems difficult to assume substantial improvements in health for the majority of participants in the Malawi VHB program. 10 ANNEX B3 DIP STATUS RESULTS - GUATEMALA DIP Objectives DIP Indicators EOP Status COMMENTS • VHB policies and procedures defined for successful implementation in this area; Accomplished - Appropriate manuals, credit policies and procedures developed and in place. Documents discussed and reviewed between the evaluator and the Project Manager during evaluation travel. Highland and Coastal conditions are different, but procedures work well in both areas. Project is well managed administratively VHB model adapted to socio￾economic characteristics of the San Marcos Department including the Altiplano and Boca Costa regions • Completed lesson plans for health and business education that is appropriate to the needs of the beneficiaries; Not yet accomplished - Health education plans are still in development, but are showing good movement forward. See narrative. Work has not yet begun on improving the “business plan” aspect of women’s loan requests, but this topic was included as a recommendation in the Final Evaluation. Expansion plan into new program areas developed; Promotional plan developed . • 85 new VHBs will be created; 122% accomplished - A 177 Village Health Banks have been opened with 104 still in active operation, 122% of goal Expansion of the VHB program to Guatemala • 3,500 women will receive loans; 53% accomplished - Total client receiving loans, 4332, 123 % of goal - Current active clients are 1,861, 53% of goal. • Arrears over 30 days less than 5% of outstanding loans; Accomplished - Arrears over 30 days are 3%. • Loan loss rates of less than 2% of average portfolio; Accomplished In FY03, loan loss is 0% Maintenance of high program quality • Operational efficiency of less than 15% by grant end; Accomplished Operational efficient achieved 16.78%, judged all-but 100% of goal. • 15 new staff trained and oriented in VHB methodology; Accomplished Current staff of 14 with certain longevity appear well trained and most are competent. Three new staff not evaluated. Project manager reports four staff training events in FY03: in credit analysis, billing, planning, and health education. • Development of planning, administrative, and management systems and documentation to support a growing institution; Accomplished Improved institutional capabilities • Upgrade of the Financial Management Information System to assess economic impact on beneficiaries; Accomplished - Overall management of the portfolio is strong. Systems reported and reviews are: annual work planning, expansion and self-sufficiency plan, promotion plan, monthly performance planning, computerized loan registers, creations of credit policies, procedure manual, paperwork standardization, and others. .Loan management and Accounting Systems are in place. Performance monitoring system is being used. 11 DIP Objectives DIP Indicators EOP Status COMMENTS • Development of an Impact Assessment System; Accomplished in micro￾finance; not accomplished in health - Economic impact is well measured, as per the above. - Health studies show expenditure of great effort, but methodology has contributed to unreliable data. See extensive discussion on this subject in the country text. • Achieve a 100% level of operational sustainability; 69% accomplishment - Operational sustainability is 69%. Project Manager is confident 100% will be achieved in the next nine months, and the evaluator is disposed to agree with the confidence. • Commercial funding is utilized to finance expansion of the loan portfolio; N/A - No longer applicable. Self-Sufficiency • Accessing in-country resources to complement program operating and growth needs; Partially accomplished Local resources have been researched, but have not yet produced results. Project has been partially supported by the National Lottery Charities Board of the United Kingdom. for several years, to good effect. • Income of participating women increases by 20% annually; Accomplished for 829 women; less for another 1,974 - Considerable confidence to the assertion that 829 clients—those who have completed 4 loan cycles or more and may still be active in the project—have benefited by increases in income of up to 50%. -The project has likely had some—but less—economic impact on the lives of 1,974 more women, those who have participated in the project four loan cycles or more but who left the project. Increased beneficiary impact • 70% of VHB members will be engaged in improved home-based health practices; [Same comment] Accomplished for 829; less for another 1,974 [Same comment] - Considerable confidence that 829 clients—those who have completed 4 loan cycles or more and may still be active in the project—have benefited by substantial improvements in health. -The project has likely less impact on the health of 1,974 more women, those who have participated in the project four loan cycles or more but who left the project. 12 ANNEX B4 DIP STATUS RESULTS - PERU DIP Objectives DIP Indicators EOP Status Comments • VHB policies and procedures defined for successful implementation in this area; Accomplished - Appropriate manuals, credit policies and procedures developed and in place. Documents discussed and reviewed between the evaluator and the Project Manager during evaluation travel. Project is well managed administratively VHB model adapted to socio￾economic characteristics of San Martin Region • Completed lesson plans for health and business education that is appropriate to the needs of the beneficiaries; “Well on the way” in health; not underway in micro-credit - One Health Education module is complete and others are in active development, but are showing excellent movement. See narrative. Work has not yet begun on improving the “business plan” aspect of women’s loan requests as in other countries, but may be needed as part of creating a needed revision to program strategy. • 80 new VHBs will be created; 88% accomplished but…. - A 185 Village Health Banks have been opened with 70 in active operation., 88% of goal. However, project is faced with “killing” rates of drop-out; based on this trend, some (new) active banks will likely drop out soon. Project has recently decided to open operations in Lima. Expansion of the VHB program to Peru • 3,300 women will receive loans; Accomplished but with high dropouts - Total clients were 5,034, 153% of goal. - Total 1 yr. or more participation: 1151 - Current active clients are 1,071, 32% of goal at time of the Final evaluation. Due to likely future drop out, this is probably over-stated. • Arrears over 30 days less than 5% of outstanding loans; Not accomplised - Arrears over 30 days past due are 28%. • Loan loss rates of less than 2% of average portfolio; Not accomplihsed - Loan defaults—arrears more than 1 yr. old—for FY03 are 3.48% Maintenance of high program quality • Operational efficiency of less than 15% by grant end; Not accomplished 24%. High operating costs coupled with high loan arrears. • 15 new staff trained and oriented in VHB methodology; Accomplished Current staff of 11 with certain longevity appear well trained and most are competent. Three Lima staff not evaluated. Improved institutional capabilities • Development of planning, administrative, and management systems and documentation to support a growing institution; Accomplished but …. - Loan management and Accounting Systems are in place. Performance monitoring system is being used. - Developing an effective response to difficult ‘macro-’ context elements was not observed. - Decision to enter Lima was researched but slowness in the project’s ‘gaining traction’ has been encountered. - Quality and quantity of health talks is very well supervised. 13 DIP Objectives DIP Indicators EOP Status Comments • Upgrade of the Financial Management Information System to assess economic impact on beneficiaries; Accomplished - Project has established an effective tracking system to measure improvements in income by collecting economic and household data on each entrant to the VHB during the 1st cycle and compared the results at the conclusion of the 4th loan cycle. For those who are still involved in the program for four cycles or more, 200 or so, improved income may be in the 10%-15% range. • Development of an Impact Assessment System; Accomplished in micro￾finance; well advanced in health - Economic impact is well measured, as per the above. - Health studies show expenditure of great effort, but methodology has contributed to unreliable data. See extensive discussion on this subject in the country text. Recent decisions regarding change of methodology in health data collection should show positive results soon. • Achieve a 100% level of operational sustainability; Not accomplished - Operational sustainability is 45%; it does not seem achievable in this portfolio for the foreseeable future. • Commercial funding is utilized to finance expansion of the loan portfolio; N/A Not applicable Self-Sufficiency • Accessing in-country resources to complement program operating and growth needs; Not accomplished Local resources have been researched, but have not yet produced results. Peru is a difficult country in which to find funding except through development grants. HOPE Peru has been unable to develop new grant funding in several years. One new initiative is under negotiations with the Caja Rural. • Income of participating women increases by 20% annually; Accomplished for 200+ women; ; less for another 484 - Some confidence to the assertion that 205 clients have substantially improved their income—those who have completed 4 loan cycles or more and who are still involved in the program. Less impact is estimated for 484, those 1 yr. or more who are no longer in the program. Increased beneficiary impact • 70% of VHB members will be engaged in improved home-based health practices; Accomplished for 200+ women, less for another 484. -The project has likely had substantial impact on the lives of 200 or so, those who have participated in the project four loan cycles or more with less impact on 4-cycle members who have left, 484. Because of high drop-outs and other factors, it seems difficult to assume substantial improvements in health for the rest of participants in the Peru VHB program. 1 APPENDIX C VHB PROGRAM STRENGTHS AND WEAKNESSES Objectives Strengths Weaknesses SO 1: Strengthened Headquarters capacity to implement VHB programs Formation of an Income Generation Department in HOPE Headquarters -Good leadership; good systems; good management. -Importance of VHB as strategic part of Project HOPE portfolio has improved with arrival of new V.P.- Programming. Development of a common, flexible VHB methodology Fully accomplished; good overall designs; good local adaptations. Implementation of VHB in 3 new countries and 1 expansion VHB Ecuador graduated to full institutional autonomy; VHB fully implemented in three new countries. Greater sustainability through diversified funding sources -Complementary funding received from two U.K. donors for VHB operations. -Expansion into two new countries without USAID assistance. Increased sustainability of HQ technical services HOPE resources committed for the hiring of Deputy Director-I.G. from FY04. Establishment of a program performance monitoring system Establishment of a program impact assessment system Quite strong in micro-finance in both monitoring and impact assessment. Has taken length of the grant to determine a viable methodology for monitoring and impact assessment in health. Impact evaluation in incomes can also be strengthened. SO 2: Financial services and health education provided in a sustainable manner Develop local staff capacity to implement VHB methodology Country project managers generally well qualified and exercising good project leadership; quality field staff placed in three countries; supervision and oversight good (though varied by component and by country.) -Skills of some field staff in the new education methodology are not yet consolidated. -Some suggestion of over-intensive staffing in some countries. Achieve sustainability of three country operations Sustainability likely to be achieved in Guatemala within the next 9-12 months and possibly to be achieved in Malawi in the next 12-18 months. Sustainability not achievable in Peru for the foreseeable future. Serve a significant population Accomplishment of original goal of 12,300 clients was achieved at 131%. Proposal did not discuss implications of drop-outs. Without dropouts, project well exceeded the goal. Taking into account dropouts means that effective achievement of project goal, currently active clients or those 1 yr. or more in the project, is quite a bit less than original goal. Maintain high program quality [in financial terms] Achieved in Guatemala. Not achieved in Malawi under difficult programming circumstances. Far from being achieved in Peru. SO 3: Improved income and health of impoverished women and children Form financially viable, democratic community institutions Currently active banks, 371, of an original target of 285 represent 130% achievement of original goal. Improve economic opportunities for women(loans disbursed) Total loans disbursed, 16,136 compared to a goal of 12,300, 130% of goal. 20% increased incomes of participants Approximately 24% increased incomes accomplished for 2,251 participants, less for -Proposal did not discuss drop-outs and desertion. 2 Objectives Strengths Weaknesses another 4,046. -Many clients have dropped out and likely did not substantially improve incomes compared to those who have. Some of this reflects micro￾finance industry trends; some of it seems particular to VHB. Increased exposure of women to health talks Accomplished quantitatively at 108% of goal. Qualitatively, health talks have taken a big turn for the better in the last 18 months. Improved health status of participating women Substantial improvements in health accomplished for 2,251 participants, less for another 4,046. As in improved income, drop-outs and desertion likely mean that many clients have not substantially improved their health compared to those who have. 3 APPENDIX D: MALAWI FINAL EVALUATION APPENDIX E: GUATEMALA FINAL EVALUATION APPENDIX F: PERU FINAL EVALUATION 1 APPENDIX D FINAL EVALUATION MALAWI VILLAGE HEALTH BANK PROJECT 1.0 INTRODUCTION and BACKGROUND 1.1 Origins Of the Current Project The People-to-People Health Foundation, Project HOPE, is a non-profit, non-sectarian, a-political organization with a mission “Helping People Help Themselves.” HOPE conducts assistance and development programs throughout the world since the first HOPE ship in 1960. Major recent programs have included Child Survival in half a dozen countries, emergency response, health education and training for physicians and other health professionals, preventative health care, policy work through the Center for Health Affairs, Village Health Banks, and others. Origins of the current Village Health Bank (VHB) project lie in a 1992-1997 USAID Cooperative Agreement to allow HOPE to establish the program in Honduras and Ecuador. For FY 1998-2003, additional funding was requested from DFID and from USAID. A Midterm evaluation was conducted in this project in June 2000 and a program visit in August 2001; this document is the Final evaluation of that project. 1.2 Scope Of Work The Scope of Work (Appendix A) tasked the consultant to conduct an evaluation of the Village Health Bank activities in Malawi (and also Guatemala and Peru.); assess HOPE Headquarters progress with the grant; and develop recommendations to strengthen continuing execution of the VHB programs. Main topic were: ß Discuss if the hypotheses under-pinning the program model were sound. ß Discuss progress toward achievement of major objective; major successes, challenges and constraints; and unintended impacts. ß Identify changes made during implementation. ß Discuss impact of the program on target populations. ß Discuss whether adequate financial control systems exist. ß Discuss whether the M&E system supplies accurate, timely performance data. ß Discuss program management including an adequate number of staff with relevant expertise and supervisory systems. ß Discuss Lessons Learnt and Recommendations. 1.3 Timing of the Evaluation The evaluation took place from June 10 to 30, 2003 beginning with two days of orientation with the Director-Income Generating in HOPE Center. Field travel to Malawi took place from June 14 to 21. During the visit, one day of program discussions took place with the Programme Manager, followed by three days of field travel to VHBs. Seven banks were visited. A discussion was held with project field staff on June 20, as well as an ongoing conversation with the Programme Manager. A presentation of preliminary findings 2 was made to the Malawi Country Director, the Programme Manager and the Technical Support Manager on June 21. Project write-up took place during the week of June 23-28. Submission of the final document to Project HOPE took place on June 28. 1.4 Methodology and Evaluator The methodology was typical of many evaluations: ß review of documents; ß in-depth discussions with program management and program support technicians; ß field travel, including direct observation of village health talks and loan operations; and ß semi-structured interviews with randomly selected VHB members. Six of eight HOPE field staff were observed as they engaged in project promotion. Seven banks were visited for two hours each. Post-event conversations took place with eleven women at four different banks. Discussions were held with eight Mulanje field and administrative support staff at the close of the field travel. Both program areas were visited, Mulanje and peri-urban Blantyre. A visit was requested to FINCA Mulanje to compare program experiences but could not take place because of a FINCA scheduling conflict. The evaluation was carried out by Frank Sullivan, a thirty-year experienced development practitioner with specialization in rural agricultural development, rural sociology, primary health care, and organizational management and development. Mr. Sullivan worked in overseas capacities for almost twenty-five years, in his later years as Country Director of operations to $6.5 million annually. He was also the external evaluator of the VHB Mid-Term in September 2001. 1.5 Structure of report The document is structured in chapters. Chapter 1, Introduction, has now concluded. Chapter 2 outlines the design of the project and its conceptual underpinning. Chapter 3 summarizes various Mid-term evaluations. Chapter 4 presents Findings in Health, Chapter 5, Findings in Microfinance. Chapter 6 discusses project finances briefly. Chapter 7 presents Conclusions and Chapter 8, Recommendations. A conscious effort was undertaken to make this a stand-alone a document (perhaps resulting in occasional repetition in Chapter 2 and 3), so that a reading of this paper will result in an overall understanding of the context in which the project was written as well as its current status and goal accomplishments. 3 2.0 PROJECT STRUCTURE, FRAMEWORK and STATUS 2.1 Project Structure The overall Objective of the project is to alleviate poverty and its effects on women and children by increasing incomes and improving health conditions and personal well-being. This was to be achieved through three Strategic Objectives: SO 1: Strengthened capacity of Project HOPE-HQ to implement successful Village Health Bank programs. SO 2: Valued financial services and health education provided in a sustainable manner. SO 3: Improved income and health levels for impoverished women and children. Drawing on Project HOPE micro-finance experience in Latin America, the design called for the establishment of a village health banking system in Malawi. The Malawi experience was to be part of a wider effort in three other (Latin) countries with complementary, non-DFID funding. 2.2 Conceptual Framework The HOPE model of Village Health Banks provides credit and health education to women, building on credit to solidarity groups. The program is frequently described as “credit-plus,” meaning that health education is added to project interventions to enhance the impact of gains in income. The project is directed to the inter-relationship between low incomes and poor health influenced by poor housing and hygiene, inadequate access to health care, and limited educational opportunities. It also builds on the well-documented fact that increases in women’s incomes get spent on more family health and nutrition than increase in most men’s incomes. As a health-driven institution, Project HOPE is particularly interested in the project’s health impacts. 2.3 Current Status Updating a table from the Midterm evaluation, a current snapshot of the project is as follows. Table 1: Overall status of Intermediate Results Accomplishments Proposed Results Current Status SO 1: Strengthened capacity [at HQ]to implement Village Health Bank programs Formation of an Income Generation Department in HOPE HQ Accomplished Development of a common, flexible VHB methodology Accomplished Implementation of VHB in 3 new countries and 1 expansion Accomplished Greater sustainability through diversified funding sources On target Increased sustainability of HQ technical services Accomplished Development of independent national organizations Objective discontinued based on Midterm recommendations Establishment of a program performance monitoring system Accomplished Establishment of a program impact assessment system Accomplished SO 2: Financial services and health education provided in a sustainable manner Develop local staff capacity to implement VHB methodology Accomplished Create local NGOs to continue program beyond LOP Objective discontinued based on Midterm recommendations Achieve sustainability of operations See discussion 4 Serve a significant population Accomplished Develop access to resources to support long-term growth See discussion Maintain high program quality [in financial terms] See discussion SO 3: Improved income and health of impoverished women and children Form financially viable, democratic community institutions Accomplished Improve economic opportunities for women (loans disbursed) Accomplished 20% Increased incomes of participants Accomplished Increased exposure of women to health talks Accomplished Improved health status of participating women See discussion The project has accomplished its objectives of strengthening HOPE HQ in its institutional capacity to manage micro-finance programming; in improving VHB management systems; and in improved supervision of VHB activities. As part of the USAID grant but not the DFID one, these topics will not be discussed further in this evaluation. Regarding the formation of local NGOs, the Midterm recommended abandonment of the target. Sustainability, serving a significant population, resources for long term growth and high program quality will be discussed below. Regarding program impact on women’s lives, an adequate number of loans have been disbursed. Illustrative data on net improvement in women’s income suggest this program goal is being accomplished. The issue of how much has been improved in terms of women’s health will be discussed at length in this document. 5 3.0 MIDTERM EVALUATIONS Three Midterm evaluations took place in this project. The first—early on—was carried out by an African microfinance consulting organization, Vulindlela, in June 2000. The second was a portfolio review led by the current evaluator in two Latin countries in Aug.-Sept. 2001. The third was a replica of the Latin American Mid-term review conducted by a HOPE Ecuador staff member in October 2001. 3.1 Midterm Evaluation- Vulindlela- June 2000 This assessment found that “most of the indicators are positive except portfolio quality.” For some indicators, it was too early to judge whether they would be achieved due to the fact that the program had been operating only a year. Salient recommendations from that document which have been judged accomplished by this evaluator include: ß development of an expansion plan to take the program out of Mulanje to other high economic growth districts of Malawi; ß link bank promoters’ performance to portfolio quality reports; ß review [and make appropriate adjustments to] group size; ß staff trained in ratio analysis; ß reduce the growth pace in VHB formation; ß development of comprehensive staff training programs; ß raise the initial savings deposit to 10% of the loan; ß develop a direct link of loss of savings as a consequence of non-payment of loan; ß match loan size to business type and cash flow cycles; ß tighten delinquency management; ß development of a comprehensive lending policies and procedures manual. Salient recommendations which have not been fully accomplished are: ß streamlining and simplifying project documentation; ß development of an incentive package including preferential pricing for goods; ß development of operational budgets managed at program site. 3.2 Midterm Evaluation- Village Health Banking Project- Sept. 2001 Refer to Table One, p. 3-4 for an overview of the current status of the project based on a format discussed in this midterm review. Additional themes discussed in that document that will find echo in this one are: ß the importance of health outcomes and health impact assessment as the raison d’être of the VHB project given Project HOPE’s mission; ß the relative weakness of the health component compared to micro-credit; ß thinking through what could be legitimately aspired to in half-hour slots once every two weeks; 6 ß suggested change in methodology of health education away from relying on village women as teachers; ß strengthening good Adult Education pedagogy at the field level; ß creation of the post of Adult Education advisor at HOPE HQ. 3.3 Malawi Field Report, USAID Matching Grant Mid-Term Evaluation [October] 2001 Summary observations from this assessment that will be touched on in are: ß targeting more commercial areas; ß correct some deficiencies found during the VHB meetings; ß implementing new procedures to minimize arrears; ß suggested change in methodology of health education; ß need for training of staff on adult education and monitoring; ß need for a new management information system to track the growing portfolio. 7 4.0 FINDINGS in HEALTH Improving women’s health is a principle goal of the project. Discussion has been grouped in ascending levels of analysis: client issues, program issues, and conceptual issues. A fourth section describes scale-up of the model to Thailand and Mozambique. A final section will discuss the evaluator’s carefully weighed judgment regarding the project’s overall impact in health. 4.1 Client Issues Four subjects will be discussed: community interest, constant movement of clients, negative macro￾societal trends, and the prevalence of a strong male bias in Malawi society. Community interest It is clear from field visits there is high community interest in the project. This can be inferred from several points of view. First, attendance at bi-weekly community meetings in the seven VHBs visited during the field travel was 76 of 107 not including excused absences, over three-quarters attendance. Additionally, the contrast between member self-presentations—where serious faces, subservience, and timidity reflect rural women’s traditional carriage—and when health talk began was dramatic—wide smiles, energetic participation, forthright opinions, and singing and dancing to the session’s slogan becoming the norm. This juxtaposition between the two behaviors was striking. Questions from the evaluator to selected post-event interviewees reinforced this perception: VHB members are enjoying the meetings a great deal. Another characteristic of village banks is that a forum has been created whereby women can discuss issues and exchanges experiences that has likely never existed before. This was seen to be true in a non-controversial module on water and sanitation, and also in the more challenging and difficult sessions on HIV/AIDS and sexual mores. In the project, women are talking to one another as they have never done before, and sharing experiences on topics that are seldom discussed in public in the Malawi society. Constant client movement One characteristic of the VHBs that almost every bank suffers from is a constant departure of old clients and influx of new ones. Data to be presented in the micro-finance chapter show that virtually half of the current clients in Mulanje are new to the project. Similar trends are beginning to appear in Blantyre at an earlier stage of the project. During the Mid-term evaluation, the HQ Program Director commented that this drop-out rate is not unusual in the micro-finance sector. Observed during the travel were the departure of the first few women at the end of the first or second loan cycle, the departure of one or two more in many subsequent loan cycles, the entering of new members from the second cycle on, and (even) the entry of new members up to the eighth cycle. Implications for Adult Education are serious since even if the project were delivering the most vibrant and impact-filled Adult Education in health (AE-h) topics in the world, new entrants wouldn’t have heard them. Also, with an attrition rate of up to 50% by the end of the fourth cycle, half the members who received one or another health module will no longer be part of the group. As will be seen, this has had important ramifications in accomplishment of health goals. 8 Negative macro-societal trends One ‘macro-’ trend the project has had to deal with is high illiteracy, approximately 42% in 1998, worse in the rural areas. This fact has required the project to adjust some of its credit tracking and loan monitoring paper flow, but it also has an impact on women’s ability to absorb health messages. For instance, all health messages must be short and terse: the project is unable to leave behind pamphlets or other descriptive information. Messages must also be easily remembered: the essence must be distilled and communicated to be remembered. Slogans become the ‘shorthand’ of each health module. While illiteracy needs to be factored into many health programs, Malawi’s high illiteracy rate has been an obstacle that may have slowed project achievement. Another macro-trend—familiar to experts on African development but a shock to a Latin American expert—is the decimation that HIV/AIDS is wreaking on the Malawi society. Current HIV/AIDS prevalence is one of the highest in the world: in women attending antenatal clinic, from 10% (rural) to 30% (urban); 14% in the 15-49 age cohort; four to six times the infection rate in younger females than males.2 These figures are astronomical. It seems there is not a family in the country which has not lost a relative to the pandemic; many of the victims come from the young, productive cohort. Even in the short space of this evaluation, the effects of AIDS was felt: ß The HOPE Mulanje demographer was attending a family funeral the day the evaluator arrived; ß one community health session scheduled for an evaluation visit was cancelled because the community was attending a funeral; ß one VHB that went ahead with its fortnightly meeting came to the session having buried one of its villagers earlier in the morning. This, in the space of four days. Trying to visualize the impact of the relentless personal and economic loss to the country is very difficult. Thankfully only a few VHB members have died during these years. But as described in Malawi Poverty Reduction Strategy Paper, “the spread of HIV/AIDS is threatening to undermine all attempts to reducing poverty in Malawi, both directly through the prevalence of HIV/AIDS and indirectly through the resulting shortages of skilled human resources.” The economic drag of HIV/AIDS on the VHB project cannot be gainsaid. A third macro- trend is the low and deteriorating level of socio-economic conditions. According to Malawi Poverty Reduction Strategy Paper: ß 65%, 6.3 million people, are poor, consuming on average only two-thirds of the recommended dietary allowances; 28% of the population is in dire poverty. ß 67% of the rural population live in poverty. ß Life expectancy has deteriorated from 43 years in 1996 to 37 years in 2000, an alarming deterioration from a not-high base. ß Infant and child mortality are 104 and 189 per 1000 live births, quite high. ß Maternal mortality is 1120/100,000, alarmingly high. ß Almost half (49%) of children under 3 were stunted (low height for age.) ß Inflation has averaged 30% per year from 1994 to 2000. 2 Malawi Poverty Reduction Strategy Paper, October 2001, p. 109. 9 ß Per capita GDP shrunk by 0.4% in 2000 and by 3.5% in 2001. ß GNP is $660 per capita (World Factbook 2001). The list could be extended. Summarizing: the 1998 UN Human Development Index rates Malawi 165th of 174 countries, one of the fifteen poorest countries in the world. Strong male bias in Malawi society. The last macro trend to be noted is the prevalence of cultural traditions that are substantially detrimental to women’s health and empowerment. The first is well-documented male sexual promiscuity. While the evaluator was in country, the UNFPA representative was quoted in the paper saying that hetero-sexual contact was the biggest single transmitter of AIDS in Malawi. Several health talks discussed factors that contribute to male sexual promiscuity—most of the women’s comments subtly blaming themselves: “the wife isn’t sufficiently attractive, she isn’t a good cook, she doesn’t keep a neat house, too many noisy kids….” Marital rape and seeking out prostitutes are expected and frequent outcomes of excessive alcohol consumption. Moreover, a number of strong tribal customs exist that contribute to the spread of AIDS: ß chokolo— whereby the brother-in-law inherits his brother’s widow as wife whether she is willing or not; ß kusansa fumbi— initiation of the female adolescent into the tribe is ratified by her having forced sex with one or more men; ß biriya— whereby the teenage girl is forced to have her first sexual relations with her brother-in-law or the old man in the village; ß fisi— whereby the neighbor is asked to impregnate the wife because the husband is sterile; ß chimwanamaye— wife swapping with a good friend; ß banyira— polygamous marriage to the younger sister of the current wife. Enumeration of these cultural mores does not represent a value judgment, but description by VHB members of societal factors which contribute to the spread of AIDS. Naturally, they have a substantial negative effect on women’s self-esteem also. The project has dealt creatively with the societal pattern of the husband as ‘head of the family’ who has veto power and control of the family budget by requiring the husband ‘endorse’ the VHB loan signifying his acceptance of the pledged collateral items and that the loan will be managed by his wife (though exceptions were observed during the evaluation week.) Nevertheless, the pervasiveness of a number of anti-women cultural patterns has had a significant slowing effect on the project—perhaps one not recognized when the project goals were being written based on Latin American experience. 4.2 Program Issues Program issues to be discussed are: direct delivery of health education messages, revised curriculum, staff qualifications, staff supervision, attempts to document impacts in health, and a close-out decision that may need to be re-visited. Direct health education delivery 10 One of the recommendations of the Mid-term evaluation was that the VHB project should move away from the idea that village women could be trained to the point of delivering high quality health talks. Barely literate, overworked village women were judged largely incapable of delivering high quality messages the project needed; the proverbial Training-of-(village)-Trainers was not working. Instead, it was recommended that HOPE convert bank promoters into health extensionists, ‘direct health education delivery’ by HOPE. To the evaluator’s pleasure, this recommendation has been fully implemented. HOPE Malawi changed the title and job description of the bank promoters from village banking promoters to Community Development Promoters (CDP) about eighteen months ago. Training modules were developed to guide these non-health staff how to cover key messages in any given subject. Also, a training calendar was carried out for the staff in the new skills. From April 2002, HOPE field staff, not village women, have been carrying out the health talks. This is a substantial step in the right direction. Revised modules As part of this switch to direct delivery, the evaluator was pleased to note a significant realignment in the health curriculum has recently taken place. Discussions were held with the expatriate (African) Malawi Technical Support Manager, a health professional and an adult educator, who indicated that on her arrival the then-curriculum was under serious scrutiny as to effectiveness and impact. Her and the HOPE Malawi team’s judgment was that the modules were overly didactic, overly technical, and many more than could be completed in half-hour slots over a two year period. Under her guidance over months of design, pre-testing, etc., the modules have been reduced to six (with multiple themes per module) sufficient to be covered in 28 sessions, four loan cycles; their content has been ‘scaled down’ technically; indicators of success have been couched in behavioral terms; and their methodology has been made far more Freirean. The evaluator enthusiastically endorses these changes. Unfortunately, called to fulfill numerous other functions including proposal development, close-out of other projects, etc., the Technical Support Manager has only just now completed the first draft of these modules; CDP training in the modules has yet to take place. Summarizing historically, then, adult health education in the project has roughly gone through the following stages: ß Year One and Year Two: start of financial operations; no health education in first loan cycle. ß Year Three: change of focus to direct delivery, development of a didactic curriculum; also substantial staff turnover and learning curve for new staff; determination the didactic curriculum was not adequate; beginning of training for staff in community adult education. ß Year Four: determination the didactic curriculum was not adequate. ß Year Five: development of a much improved curriculum; ß (As of June ‘03 as this evaluation was taking place:) experimentation with first few modules. Thus Adult Education (AE) in health has been problematic for much of the life of the grant. Staff qualifications Regarding staff attitudes, seven of the eight CDPs were observed in action during the evaluation week. All six demonstrated considerable personal warmth, enthusiasm for the jobs, energy in their 11 presentations, good rapport with the women, and the necessary ‘presence’ to carry out vibrant, fun￾filled community promotion. All are ex-teachers and obviously comfortable in front of a group. Releasing a number of staff with different skills two years ago, HOPE Malawi has found exactly the right kind of CDP. Unfortunately, adult education skills of these staff are still developing. AE and non formal learning (as discussed in the Mid-term) require a different mind set and different tools than formal, didactic ‘schoolroom’ education. In part because these field workers are ex-teachers, in part because the previous health education called for the “transmission” of a lot of content, in part for other reasons, —despite the best of intentions— CDPs are not yet fully versed in AE. Things like seating the women in rows of school benches, the facilitator standing and talking instead of sitting at the women’s level, reading from the module guide, and others classroom techniques show staff are still learning the new philosophy and methodology. More complex concepts like creating a ‘space’ for women to interject when they want to—and drawing on such interjections as knowledge—are yet unknown. Given CDPs’ positive attitude, there is little doubt they will improve, probably rather quickly, but there is a ways to go at this point. Supervision of staff CDP staff have not been well supervised in this transition. The previous VHP Health Coordinator—the individual primarily responsible for the didactic curriculum—resigned for personal reasons in November 2002 (perhaps not coincidentally, several months after the arrival of the expatriate Technical Support Manager.) A review of her travel from April to June 2002 indicates she supervised 25 CDP talks, a more-or-less acceptable level of supervision. In the period July-Sept., that figure dropped to twelve; though data are not available, one speculates her travel in the month prior to her resignation was likely zero. The in-coming Technical Support Manager has only been able to make four CDP visits in 11 months. Thus to all intents and purposes CDPs have been without technical support and supervision in health for over a year. Said one staff: “just as CDPs need supervision in finances, so too they need it in health.” HOPE Malawi made a presentation to HOPE HQ to replace the health educator immediately on her departure, but the idea was put on hold, thinking the Technical Support Manager could fill that responsibility. This has not happened. The number of health talks is reported each month, and a matrix showing talks per bank is prepared quarterly and annually. The numbers are acceptable, 1408 reported in FY02 and 1496 reported to date in FY03. This report is not designed to measure quality of the talks. Unfilled is an Adult Educator position at HOPE Center recommended by the Mid-term evaluation. The evaluator recognizes this recommendation was one of the more ambitious of the Mid-term. Hiring a Head-Office technical expert requires an executive decision, and cost implications are substantial; thus it is not completely surprising HOPE HQ didn’t accede to the recommendation. Nevertheless, as predicted, the effect has been that field efforts to improve Adult Education are not meaningfully backstopped or supported. Field staff comments also identified this weakness. Documenting improved health For three years, the project has struggled to track and document improved health. Baseline heath data were collected a year after the project started in August 2000 (N=269). Another effort was 12 undertaken in FY01 (N=256.) In FY02, a third effort was carried out (N=282). In all cases a detailed survey report was prepared. Alas, because of the high drop-out rates discussed earlier, FY01 and FY02 data were substantially ‘co-mingled’ with recently entering women. FY02 data, for instance, show only 17% of the sample were women who had been in the program four or more cycles. In order to show some movement, in the FY02 Annual Report, FY01 data on seven indicators (of 32 on which data were collected) were compared to the Baseline and then reported as “average improvement of 35% across a variety of indicators.” When an exercise was conducted during this SOW using all 32 variable, the average difference between Baseline and 2001 data was only 5%. When FY02 results on the seven indices were compared to Baseline, the average improvement was 15%. The problem with these figures, of course, is that the ‘co-mingling’ of new entrants with old makes all the figures suspect. Aware of these difficulties, the HQ Program Director tried to tighten the relevance of FY02 data by comparing some 1st cycle and 4th cycle clients (N=75 and N=89 respectively). At his suggestion, a second round of analysis was conducted on 16 FY02 indictors where 1st and 4th cycle data were available. After deleting two indicators which are individual responses, ten indicators show improvement, four show deterioration: overall change: 13%. At this writing a health survey is being conducted in Blantyre (N=200) comparing 1st and 3rd cycle VHB women. Still the fundamental problem remains: different women being compared. Taking such figures into account and analyzing the data this way and that, one can speculate that change in health behavior may be in the 5% to 15% range—though it must be recognized a serious social scientist would reject such a figure out of hand. What is clear from the documents is that project staff invested much time and effort in collecting the information. A recommendation will be offered in this regard. Qualitative gains: “A+” Banks Beyond quantitative gains in health surmised in the previous paragraph, qualitative gains merit discussion using the village of Lunguni as an example. Based on the fact that its fortnightly meeting was one of the days the evaluator was in the field, the community of Lunguni was visited. This is a group of 14 women (see photo, Appendix E) who live in an isolated part of Mulanje. They have been together for eight loan cycles with no drop-outs since the 1st cycle three years ago (!); repayments are steady; the group is cohesive; and the health education session on water and sanitation showed they have learned a great deal on the topic and, indeed, were able to offer additional comments beyond those in the health talk. One member has become sufficiently successful she has taken a loan for MK 25,000, one of the biggest in the Mulanje portfolio; another banana-and-maize seller has accumulated MK 7,500 in savings. This is a terrific group—in fact, in the evaluator’s judgment everything the project could possibly aspire to. In a staff meeting the following day, the evaluator asked CDPs to identify by name groups like Lunguni, dubbed “A+.” Thirty-two groups were named, with a membership of 553, representing 36% of the Mulanje portfolio. Staff clarified that some of these groups were excellent in loan repayment, some excellent in health, and some like Lunguni excellent in both. Assuming staff assessments can be relied on in this ‘snapshot’ judgment—and there was general consensus in the 13 room when each community was named—it is likely the project has had a profound effect on at least a third of its clients, those who stick with it through four or more loan cycles. Additionally, though not discussed with staff, the evaluator speculates that another third of the Mulanje clientele may be ‘pretty good’, a one-third/one-third/one-third division being a recurring pattern in this writer’s experience. This would make the portfolio of ‘very-good-to-pretty-good’ clients about a thousand people or so. Those this is an admittedly subjective judgment, it appears to this evaluator the VHB has had a big impact on the lives of some of its clients; though very real and palpable, such benefits are difficult to quantify. Closure of Mulanje office During the course of the field travel, the evaluator became aware that program operations in Mulanje are schedule to close from October 1, a development he views with some concern. While the decision was based on financial criteria and discussion on it could logically be postponed to the next chapter, such a decision has many health implications and will be discussed here. The issue of financial sustainability has been a major program goal since the write-up of the project. A review of FY00 and ’01 Annual Reports shows the project was still far from this goal by Sept. ’01. At that point the programme team began re-aligning project activities to begin working in an area with more economic potential—the periphery of Blantyre where HOPE has its country office. First promotional activities in Blantyre began in FY02 and the results were dramatic; by the end of the year, 41 new banks had been formed: one-third the number of Mulanje banks had been created in six months in Blantyre compared to four years in Mulanje. It is unclear whether this decision was formally shared with the donor before it was taken, but even if not, the evaluator supports the decision. Mulanje results over time demonstrate that a self-sustaining operation in that rural, underdeveloped area is a challenge. In FY03, another look was taken at the Mulanje portfolio, and evidently even the rapid movement toward financial sustainability in Blantyre appeared not enough to achieve full sustainability if Mulanje continued to require subsidy. A decision was taken in March, ’03 that all on-going loan activities in Mulanje would cease. It bears repeating that the project works on a ‘quick turnaround’: as soon as one loan cycle is fully paid, the next one begins. Thus the announcement of a cessation of follow-on loans was greeted with shock by Mulanje staff and by some VHB members. After consultation with HOPE HQ, the decision was re-thought, and the current deadline for closure was extended to October, 2003. The only non-health activity taking place in Mulanje at this writing is recuperation of existing loans. This decision contradicted the comments in the FY02 Annual Report that declared: “Nor do we want to withdraw from Mulanje because of the true value in operating there.” When the issue was brought up in the evaluator’s debriefing session, the Country Director expressed anguish at having been forced to take this decision. She and the Programme Manager attributed it exclusively to financial considerations: they believe all other alternatives have been explored and that the struggle to reach financial sustainability requires they take this drastic action. 14 The evaluator is not fully convinced. First, evidence was not presented to demonstrate that cost savings and a reduced ‘institutional footprint’ have been thoroughly explored in taking this decision. Second, if the assessment of the previous section is accurate to within 10%, it is hard to believe that ways cannot be found to make continued presence and program activity financially viable for over a thousand high quality clients like the women of Lunguni. Project management contends that with limited funds, there is an inevitable trade-off between Mulanje and Blantyre; the evaluator would like to see the issue explored further. A recommendation will be offered in this regard. 4.3 Conceptual Issues Before concluding this section on health, it seems appropriate to offer two brief reflections on the VHB program model. Succinctly, throughout the developing world, millions of women are unable to receive services from their over-worked and under-funded Ministries of Health. When these people fall sick, there is nowhere to go that does not require sometimes hours (or half-days) of travel. Moreover, preventative public health activities are all but unheard of. What the VHB model offers is a way to fund well-trained primary health promoters outside of, and complementary to, the Ministries of Health to promote women’s ‘demand-driven’ search for better health services. When financial sustainability is achieved, the model offers an effective way for women who would otherwise not receive health education to learn how to improve their lives. At the same time, it offers marginalized rural and peri-urban women an opportunity to improve their household incomes (and the health of their children), as well as to grow in self-esteem. This evaluator finds the model compelling. In a place like Malawi where infant mortality is still over 100 per thousand, and maternal mortality is one of the worst in the world, the model should be made to work: it is a way tens of thousands of such women can be served. And though there is debate on this topic, the evaluator views favorably the strategy of Project HOPE to create a country platform in places like Malawi independent of grant funding. Growth in professionalism in career national staff and long-term organizational development are very hard to achieve when a country operation is surviving from one short-term grant to another. The evaluator supports the implicit ‘program philosophy’ that a successful VHB project could create a program base on which to build. 4.4 Expansion/ Replicability Exemplifying the penultimate paragraph, approximately two years HOPE HQ began to look at the possibilities of replicating the VHB in other program settings. The result of those investigations, donor search, proposal preparation, securing funding, is that HOPE has now established a Village Health Banking project in Thailand, and an HIV/AIDS micro-lending program in Mozambique. In both cases HOPE Malawi has proved the training ground for the scale-up. Four Malawi staff visited Thailand for two weeks in FY03 to advise Thai staff on project implementation—both from the management perspective as well as providing on-the-ground advice on health education and micro￾finance mechanisms. Two HOPE Mozambique personnel crossed into Mulanje to look at its VHB operation as a model how to establish micro-finance ledgers, etc.; more visits are planned. Project HOPE has used experience in Malawi as a training ground and springboard to replicate the project—as one would hope with an innovative program. 4.5 Overall Quality and Effectiveness 15 We now come to an overall assessment-cum-reflection on all that has been discussed in adult health education. In general terms, the project is working in a quite difficult environment. Some of the difficulties are similar to those found in other developing countries: low economic activity, high unemployment, poor health status, high illiteracy. Other macro trends are specific to Malawi but are also substantial obstacles: extremely high levels of HIV infection that are drastically slowing the pace of development, sexual and cultural mores that marginalize Malawi women to a substantial degree, and others. One suspects the combination of these macro- and societal currents has impacted negatively on project accomplishments far more than conceived of when the proposal was formulated. Regarding program issues, HOPE accepted the recommendation of the Mid-term evaluation and has changed its education methodology from village trainers to ‘direct delivery’ by project staff, a quite positive step. Modules have been redesigned to emphasize adult education (Freire) principles to good effect. The VHB project has also found field staff with the right disposition to carry out direct delivery. Unfortunately, the new methodology has come late in the project so that long-term impact of this change will not be great; also, CDPs are not yet up to speed in the skills required by this new focus, and have not been much supervised over the last 12 months in absorbing these changes. The constant in-flow and outgo of VHB participants has proven a significant obstacle to project goal accomplishment—perhaps as detrimental as the hostile macro-societal trends. The project recognized this trend early but struggled without notable success to adjust to it. As a result, tracking and measuring improved health have proven difficult. The overall effectiveness of health education talks seems debatable. The project decision, a good one, to abandon the previous curriculum in Year Four suggests that design was proving unsatisfactory. Two senior project staff thought the (current) overall quality of the talks was “4-to-6 out of ten,” an assessment with which the evaluator concurs. Of six talks visited, one was of high quality (in fact it was a joy to observe); but the other five—while showing lesson preparation and innovative techniques—demonstrated room to improve: in adapting further to the new methodology and the newness of the modules, and incorporating other adult-education principles. While the number of talks is quite acceptable, how much they have changed people’s lives is an area of ambiguity. Extrapolating from the Lunguni experience, it is possible, indeed likely, the project has had substantial impact on the lives of approximately five hundred and fifty people— clients who have benefited from the project exactly as was conceptualized. Somewhat lesser impact may be surmised in the lives of another similar number. There has not been sufficient time to show improvement in Blantyre participants. Beyond these factors, because of the number of hostile external factors and because of a number of internal factors related to HOPE Malawi’s ‘learning curve,’ it seems difficult to assume substantial improvements in health for the majority of participants in the Malawi VHB program. 16 5. FINDINGS in MICROFINANCE The Mid-term evaluated the credit side of this project as one of the most well-supervised and well￾documented program components in this writer’s experience (Appendix E illustrates the idea.) That judgment has not changed. One role of the evaluation then is to update FY03 figures for selected indicators, sometimes visually. The discussion begins with a restatement of the reasoning behind the Blantyre initiative discussed earlier. Improved incomes will also be assessed, as well as comments offered on arrears, operational sustainability and summary concluding comments. 5.1 Expansion into Blantyre To make more rapid progress in achieving financial sustainability, in FY02 the project scaled back efforts in Mulanje and started a second program area in the peri-urban areas of Blantyre. At this writing, the number of Blantyre banks is equal to that of Mulanje, in one half the amount of time. Because of the economic potential of peri-urban Blantyre, these clients’ absorptive capacity is higher: average loans in Blantyre are 18% larger than those in Mulanje, MK 28,834 compared to 24,454 for the period Feb.-May 2003. The net result of these two factors means as of May 15, ’03 the Blantyre portfolio is MK 15.27 million compared to MK. 4.81 million in Mulanje, three times bigger. The evaluator judges this was an important and justified program decision. Building on the (comparative) economic dynamism of Blantyre, it should be possible to achieve project financial sustainability more quickly—and, one hopes, generate a margin that would allow HOPE Malawi to subsidize the less efficient Mulanje operation. Extrapolating from current trends, the Programme Manager speculates full financial sustainability could be accomplished by the end of FY04. It is reported this is the target of the Program Manager also. 5.2 Membership in Malawi Micro-finance Network HOPE Malawi has joined a network of international and national NGOs working in micro-finance, serving as Secretary of the network. This provides a good opportunity for VHB staff to stay aware of other micro-finance programs in the country, to share experiences, and to learn of macro￾initiatives that may impact on project operations. 5.3 Growth in Number of Banks The number of Mulanje banks grew steadily from Year One to Year Four, while in Year Four the number of Blantyre banks increased quickly. In late 2003, a ‘winnowing’ of Mulanje banks that were no longer functioning was carried out. The net total at time of the evaluation is 197 total banks. The project goal was 120, representing 195% of target. 5.4 Growth of Membership The number of VHB members grew slowly from Year One to Year Four in Mulanje and took a sudden jump with the Blantyre expansion. Total membership as of May 15, 2003 is 4,171 compared to a target of 3,240, 130% of target. 5.5 Cumulative Loan Distribution 17 Cumulative loan disbursements grew steadily through the life of the grant—if more slowly than project designers had planned. These figures have been reported on in the Annual Reports, but since the reporting has been in hard currency and the value of the Kwacha has fallen by over 100% during the period, the growth of the portfolio has been obscured. The chart shows the growth of portfolio disbursements in Kwacha at the time of the evaluation. The total is almost MK 150 million, at current exchange rates, £1,000,000. Growth in Portfolio (MK millions) 0 50 100 150 '99 '00 '01 '02 '03 The next chart reflects the current value of the portfolio in hard currency at MK87: $1.00 (22/6/03). Total value is $172,300 compared to a target of $1.6 million. Note the loss of portfolio value in FY02 due to devaluation. Current Portfolio Value in US$000 $0.00 $50.00 $100.00 $150.00 $200.00 '99 '00 '01 '02 '03 5.6 Total Member Savings VHB members are required to save 10% of their intended initial loan and to deposit 16% of the value of the loan as they make loan repayments. This is an appropriate strategy. Profit margins on most of these micro-businesses are razor-thin: imagine selling used clothes or bananas along a roadside where thirty or forty others are doing the same thing to get the picture. The policy of enforced savings promotes that at the end of the loan cycle, the woman will have something left for her effort. In fact, in some cases this savings can be substantial. The evaluator was surprised to interview one women selected at random who had saved over MK 7,500. This is an enormous accomplishment at her level of economic activity. Withdrawals from savings are permitted when a member is leaving the VHB, in case of family emergency, or to pay off a balance due on the loan. The chart shows the growth of savings up to the time of the evaluation in hard currency. 18 Growth in Savings in US$ 0 20 40 60 80 '99 '00 '01 '02 '03 Total dollar amount is $70,654 compared to a target of $240,793, 29% of target. 5.7 Mulanje Bank Closings As part of a cost reduction exercise that followed the opening of the Blantyre office, the project cleaned up its active portfolio in Mulanje. This resulted in the official closing of 41 banks of a total of 124, one-third. In most cases, this seems to be an appropriate decision; the revised portfolio is now ‘cleaner.’ For unexplained reasons, CDPs report two “A+” banks were also closed. 5.8 Dormant Banks When a bank does not pay 100% of its loan within 120 days, it enters a status called “dormant” by the project. This means the VHB no longer receives health education talks, and that further contacts between HOPE and VHB are suspended until the loan balance is paid. This is a form of pressure to encourage the banks to continue repaying. At the time of the evaluation, the number of dormant banks was 12 of 76 in Mulanje and 43 of 81 in Blantyre, 30% in total. 5.9 Member Drop-out The issue of member drop-out has been mentioned earlier. Among the factors discussed in various Annual Reports are: an over-ambitious initial loan amount, failure of the business resulting in non￾repayment, husband borrowing the money and losing it, husband’s jealousy of the wife’s success, transfer of the husband to a wage-earning job, divorce, poor health, and others. In total the number of dropouts to date is the difference less one between total active clients 2,935 (6/30/03) vs. total loan clients during the life of project, 6,720, 56%. A recommendation will be offered. 5.10 Improved Business Training One recent innovation is that the project has hired on an eight-month consulting contract a management expert to conduct business training for VHBs in the native language. Until now, as long as the VHB group proposed the loan and it did not seem overly ambitious, the project has not much focused on the business analysis behind the loan application. The thinking in hiring this specialist is that with a modicum of business training in the native language, women may be able to perceive more profitable opportunities in their environments. For instance, one interviewee narrated that during most of the year, she sells bananas; when the Malawi ‘hungry season’ comes, November to February, she moves into maize because people don’t have enough money for bananas, and maize becomes the staple in those months. This is a kind of rudimentary business forecasting that could eventually help women chose more profitable endeavors. 19 5.11 Quantifiable Improvements in Income At the project level, VHB has established an effective tracking system to measure improvements in income by collecting economic and household data on each entrant to the VHB during the 1st cycle. For those who are still active at the close of the 3rd loan cycle, the survey is repeated on a sample basis, and the changes are measured. The following table represents those results in Malawi as reported in the Year Three and Year Four Annual Reports, with improvement in household expenditures being taken as a proxy for increased income. Table 2: Quantitative Improvements in Household income Criterion Year 3 (N=10) Year 4 Improvements in household expenditure 120% Not reported Increase in households reporting better incomes 93% Not reported Decrease in households reporting worse incomes 60% Not reported The system collects data in a methodologically sound manner. However, the smallness of the Malawi sample in Year 3—10 people of a total of 438 who reportedly completed four cycles during FY01— make the data unreliable, as does the fact that comparable information was not collected in Year 4. Data in these two reports that compare women entering the project with those still in suggest gains of perhaps 20%, however one cannot infer with complete confidence such quantifiable improvements in income. 5.12 Non-quantifiable Impact Discussions with interviewees suggested numerous non-quantifiable impacts. Women’s ability to represent themselves has clearly improved. Also, project documents speak of women being able to receive attention in commercial banks for the first time ever. Staff report that women are beginning to perceive economic consequences to poor health, though this was not commented on during the field travel. Several interviewees indicated they had made enough money to have built a house; others indicated they had purchased a bicycle—in rural Malawi a major asset purchase; others report paying school fees with their profits. When the evaluator probed what effect the ‘hungry season’ had on one member’s family, the woman responded that while many around her went hungry, she and her family were relatively untouched. She used her accumulated resources to feed her family at that time of year. In all of the discussions during the entire week, the subject of flooding, hunger and famine never arose—perhaps the Sherlock Holmes ‘dog that never barked’? If during “the most difficult January/March [2002] hungry season in a decade”3 VHB members and project staff do not report themselves overly affected, surely this is an important non-quantifiable impact and a proxy of success. 5.13 Arrears 30 Days Past Due Project documentation and micro-finance literature suggest one important way to measure the soundness of a loan portfolio is to track arrears more than 30 days past due. The proposed goal was “>30 days in arrears under 5%”; status at time of evaluation is 13%. Normally such high arrears 3 USAID/Malawi Annual Report 2002, p. 1 20 would be cause for concern. Taking into account that accumulated savings can be used to offset un￾repaid loans; a Malawi pattern of slow repayment even after many months; the occasional seizure of collateral; and under one-half of one percent of written-off loans until now, the Programme Manager suggests the loans he considers seriously at risk are those 76-90 days and >90 days past due. If this is true, the (real) at-risk portfolio (May 30 data) would be 7.9%. Digging into banking intricacies, what also happens sometimes is that a deposit does not get credited to a VHB account properly, either because the treasure didn’t fill out the deposit slip accurately or due to bank error. Sometimes the money sits in a “unallocated” account for up to ninety days while the bank, the VHB, the CDP, and the project catch up with the paperwork. May 30 ‘03 data show MK 114,600 in this ‘unallocated’ category, representing another 1% reduction in arrears. In such a case, the seriously at-risk percentage would be 6.9% instead of 13%. Still higher than the goal but substantially less worrisome than 13%. The question is whether the perception of the Programme Manager in this regard will be confirmed over time. 5.14 Operational Sustainability Operational sustainability, all costs covered by project income, is one of the ‘gold standards’ of micro-financing projects, and as noted throughout the narrative it has been a major concern of the Malawi VHB project from the beginning. Data indicate that operational sustainability has hovered around 30% for much of the grant (30% FY99, 45% FY00, 29% FY01, 27% FY02.) Current operational sustainability at time of the evaluation is 58%. Reasons for the substantial jump in operational sustainability during the last eight months are the ‘coming-on-line’ of the Blantyre portfolio, operational efficiencies being achieved in the Mulanje portfolio, and cost-reductions. Of course, 58% is substantially less than the 100% goal proposed in the project design. 5.15 Overall Assessment The total number of people affected by the program is approximately 47,000, seven-member families for 6,720 cumulative membership. Improvements in income have been suggested, perhaps in the 20%-35% range for women who remain in the project to the 4th cycle and beyond. In Malawi this is apparently about 750 people; lesser impact may be assumed for those who completed four cycles but have left the project, 1,588. For VHB members in Blantyre who are recently completing the 3rd cycle, the impact will be less. Benefiting two thousand needy Malawi women is an important accomplishment. Sustainability is still somewhere in the future with current operations sustainability 58%. In the space of this grant, it has not yet been achieved or indeed approximated. One has to enquire, however, whether achieving operational sustainability of a start-from-scratch operation is feasible within five years. In a relatively stable and moderately underdeveloped country, perhaps the answer to this question is “yes.” The evaluator leaves that judgment to micro￾credit experts assessing an exclusively micro-credit program. Other factors may need to be weighed in answering such a question in the case of HOPE Malawi. ß Given a program goal that was established based on a development experience in a less prejudicial context; 21 ß given Malawi’s position on the development ladder as the fifteenth poorest country in the world; ß given HIV/AIDS decimation that is daily affecting the population and the project; ß given a program model that hopes to improve exceedingly poor health indices at the same time as it improves women’s historically low earning power and culturally low status; ß given all these factors, the answer seems much less clear. Beyond the question whether operational sustainability was an appropriate goal, the evaluation attests to a number of non-quantitative, anecdotal data to suggest significantly improved incomes. In addition to verifiable income gains and the existence of “A+” groups described earlier, in one VHB meeting the president gave testimony with evident feeling to her members’ having been able to build houses, purchase bicycles, pay school fees, and improve their children’s lives. Reduced food insecurity was commented on by several members. In addition, the project has placed a number of individual loans to highly credit-worthy clients—sometimes from $500 to $700. One member in an isolated rural village talked about her being able to repay a loan of nearly $300; another attested to the fact that she had accumulated almost $100 in savings. These are important gains. As reported by the Project Director at HOPE HQ, Project HOPE is committed to providing $50,000 to Malawi for the VHB in FY04 plus technical assistance and non-charged overhead support. This will give the project at least one more year to become self-sufficient. HOPE HQ believes this is a program that should be given another year to achieve its goal. In the end, that, too, is the assessment of the evaluator. 22 6. OVERVIEW OF FINANCIAL MANAGEMENT To foster this document’s stand-alone potential, the following summary financial information is provided. Verification of these data was beyond the Scope of Work, thus they are the same figures provided by HOPE in a separate end-of-grant financial report. Table 3: Grant Budget and Final Grant Expenditures in £ Programme Costs Original Grant Revised Grant Actual Expenditures Diff./ Grant Personnel Benefits International Travel Capital Equipment Supplies Contractual Services Training Credit Capital Other Direct Costs Total UK Project Development & Admin. (10%) Gross Programme Costs Less Programme Income Net Programme Costs DFID Share (50%) 278,950 80,687 29,962 51,403 55,946 31,379 10,287 200,909 164,161 903,684 90,368 994,052 254,489 739,563 369,782 282,007 86,670 31,587 83,310 38,396 18,837 4,758 128,439 117,740 791,744 79,174 870,918 131,354 739,564 369,782 254,898 68,408 27,423 67,370 77,024 19,582 8,250 153,979 191,209 868,145 86,815 954,960 134,862 820,097 369,782 -9.6% -21.1% -13.2% -19.1% +106% +4% +73.4% +19.9% +62.4% +9.7% +9.7% +2.7% +10.9% Total Malawi-specific project expenditures are £ 868,145. This represents 9.7% over- budget of the revised budget based on devaluation, lower average loan requests and other factors; but it represents 96% of the original budget. Financial support from USAID and portfolio income were used to cover the over-expenditure. DFID funds were expensed as budgeted. Project over-expenditures in supplies, capital and other costs relate to the scale-up of the Blantyre operation; over-expenditures in training were for staff capacity building. Under-expenditures in personnel and benefits relate to the Country Director’s salary being charged to other projects in the Malawi portfolio, and the delayed arrival of the expatriate technical advisor. The offset of over￾expenditure vs. under-expenditure was fortuitous given cost overruns in the larger HOPE Malawi mission, and the FY04 budget will need to be carefully monitored as the project moves into the post-grant era. A brief analysis was also conducted of Blantyre central office by reviewing non-project costs billed to the grant from the period March 2002 to May 15, 2003. The percentage of these costs absorbed by the project from March ‘02 to Feb. ‘03 was 11%, and from March ’03 to May 15 2003, 15%. In the evaluator’s experience, these are acceptable levels of country office support and in line with usual industry practices. 23 7. CONCLUSIONS The HOPE Village Health Bank project appears to have been a qualified success in Malawi. Starting from scratch, operating in a quite difficult development context, and working with a program model that addresses two intractable developmental problems, poor health and poor incomes, the project is on the way to achieving financial self sufficiency. Over the course of five years, health and income generating services have been offered to 6,720 poor rural and peri-urban women, affecting the lives of 47,000 people. There is strong evidence, anecdotal as well as hard data, to support the conclusion that up to two thousand women have received substantial and long￾lasting benefit from the project in income—less in health, significantly improving their quality of life. For the remaining 2,100 current clients, many of them new entrants into the project, the impact is surely less. Systems and program operations have shown steady improvement over the life of the project with some quality innovations coming ‘on line’ just as grant financing ended, such that continued improvement in the lives of up to 3,500 women may continue. The success of this program could create an institutional platform such that Project HOPE can continue to run the project successfully for some years, and build on this program base to attract other donors for much needed health projects in Malawi. Operating the program in Mulanje has proven problematic because of the low economic activity of the district (one of the more impoverished in Malawi) and because of geographical isolation. Nevertheless even in Mulanje, the project has achieved dramatic success with probably about a thousand clients. Ways must be found either under the HOPE umbrella or outside of it for these women to continue on their development path. The project has confronted a number of “moments of insight.” One of the insights was that VHB members do not have many young children thus Child Survival messages were not particularly germane; also that the ‘window’ of learning for VHB women was much smaller than for Child￾Survival participants—maximum 45 minutes per session instead of several hours per theme. Another set of insights was that direct delivery of health talks was necessary to achieve quality; that a different skill set was needed in field staff to do so; and that the health education curriculum needed considerable re-working to adjust to these facts. A third insight was the recognition that if the project was to succeed, it would have to open operations in a more economically dynamic area to achieve financial sustainability. Possibly some of these insights could have been arrived at earlier than they were. Nevertheless, throughout the five years the project has continued to learn, experiment and improve. Health monitoring and reporting have proven to be a huge headache. Staff should be encouraged to keep trying. Fortunately, the work already done can serve to guide the way. Also, ways should be found to track the quality of health education better. Belt-tightening has started recently on the eve of the cessation of grant funding. More can be done in this area. Finally, the high drop-out rate has been a drag on program efficiency for the life of the project. It may be true that micro-finance ‘standard practices’ suggest high participant turnover are part of the business, but high drop-outs could affect Blantyre sustainability as much as it did Mulanje. Ways 24 could be found to study more in depth the factors that force women to withdraw; while some are likely beyond the control of the project, others may be capable of being addressed. DFID grant reporting requests the evaluator to address specific issues. The following table has been prepared to summarize those themes. Table 4: DFID Indicators of Performance Issue Evaluator Comment Relevance Health and income generation are two key constraints women face in Malawi; project is well designed to address these issues. The health and microfinance components have become well integrated in the person of the CDP. Equity The VHB focuses entirely on women in Malawi who are the most discriminated against and highly marginalized of any group. All women are treated equally in the project, and the growth of skills in rural women is evident. Women occupy senior leadership positions in HOPE Malawi, and also are adequately represented in the CDP cadre. Efficiency and Effectiveness Program is a conceptually attractive mix of health activities and income generation. With the decision to move to ‘direct delivery,’ effectiveness in Adult Education health messages will increase. At one point program opted to convert a planned vehicle purchase into more loan funds. Overall expenditures were within 10% of budget and appear largely justifiable with expansion into a new project area. Overall cost to DFID was £370,000. Impact Increase in income for 4th cycle-and-above participants has been demonstrated in the 20%-35% range. Furthermore, 500 people seem clearly impacted in significant and sustainable ways; another 500 less so but probably still importantly; of the remaining 2,300, many are part of the Blantyre cohort which is only 12 months functioning. Sustainability Big jump in operational sustainability achieved between Oct. ‘02 and May ’03 with the Blantyre portfolio coming of age, but sustainablity has not yet been achieved. HOPE HQ has committed additional FY04 resources to achieve sustainability. Replicability Program has proven ‘expandable’ with all systems intact into another project area and into new countries. Operational sustainability may be achieved more quickly in areas of high economic activity (such as peri-urban ones) but improved incomes have also been demonstrated in isolated rural settings. Further study is required to see if the program model can be ‘scaled up’ anywhere where sufficient program capital is available. Lessons Learnt Smaller than anticipated loan sizes and smaller groups than planned in Mulanje had far reaching budget and sustainability implications that the project struggled with for much of the grant. High drop-out rate—a characteristic of many microfinance projects—was also a drag on program accomplishments. Program made a good adjustment in health education methodology. One suspects the program can become more cost efficient with minimal decrease in effectiveness. 25 8. RECOMMENDATIONS 8.1 Continued HOPE Center Support for VHB Operations in Malawi The first recommendation is obvious: Project HOPE should continue to finance the VHB operation in Malawi at the conclusion of the DFID and USAID grants. The program model is an effective one that responds to key needs and should be continued. It is to be hoped that full operational sustainability can be achieved in the course of the next year, and HOPE HQ financial support should be provided to allow that goal to be achieved. 8.2 HOPE Center Support for Program Quality in Adult Education It is likely a recommendation for the hiring of an adult educator at HOPE HQ would face the same fate as that recommendation in the Mid-term. However, this should not detract from the fact that HOPE HQ can benefit from the expertise of an Adult Education specialist. There are reports of some movement in this area in the Peru VHB program. At the same time, another of the constraints on each country’s dedicating resources to improving adult education materials—even in rural Malawi some would benefit from colorful posters, pamphlets, local drawings adapted to the culture, etc.—has been the overhanging cloud of sustainability: ‘non-essential’ expenses detract from the financial sustainability target. Perhaps a way to get country offices to invest more attention in quality adult education materials would be for HOPE Center to fund a budget line item and look for a country match—funded by HOPE Center out of the VHB program budget but not counted against the financial sustainability of one country’s operation. In truth this is a bit of a shell-game, robbing Peter to pay Paul, but it may be a way to get VHB country management to think how to improve their AE quality more. 8.3 Continued Support to Successful Mulanje VHBs HOPE Malawi needs to re-visit the decision to close operations in Mulanje. While the evaluator is sympathetic to the cost and sustainability issue, the case has not been convincingly made that Malawi staff have been adequately thorough or creative in thinking how a reduced operation could be financed. Program management should look at the following ideas and try to keep Mulanje in operation. ß Close out the Mulanje office as planned. Release all support and miscellaneous staff as planned. ß Transfer the fewest number of essential support staff to Blantyre (planned). ß Re-allocate the charging of the program manager’s salary and program support salaries proportional to the respective loan portfolios, not geography. ß Keep two CDPs based in Mulanje, and provide them with the newest of the motorcycles (planned). ß Co-finance with these staff an office-cum-residence where they could live together, and make one of the rooms the VHB office. ß Establish a level of acceptable subsidy the Blantyre operation can be expected to generate in favor of Mulanje. 26 In the worst of cases, if a rigorous cost analysis demonstrates that Mulanje must close, the “A+” groups, and after them the “A and B” groups, should be transferred to other micro-finance lenders in the area. Project management has held preliminary conversations with FINCA and one other micro-lender in this regard without tangible results to date. This overture should be pursued zealously to its conclusion. One wonders whether there isn’t an ethical responsibility to these groups to see this process through. In fact, other micro-lenders should be delighted to take over the ‘stars’ of HOPE Malawi’s operations. This activity should take place immediately, that is, by mid-July 2003. Already at this writing, a number of “A+” banks are reaching the end of their loan cycle and their next round of loan requests is not being processed. 8.4 Improved Health Data Collection By dint of much experimentation in health data collection, the project has come very close to creating a viable health measurement system; it only needs minor ‘tweaking’ to begin capturing improved health. The recommendation is simple after the difficulties of the last three years: collect health data at the start of a women’s career in VHB exactly as income and expenditure data are collected, and measure improvements in health at the close of the 4th loan cycle exactly as improvements in income and expenditure are being investigated. In fact, the HQ Program Director’s last field trip report came to this conclusion, though the Malawi staff have not yet begun acting on the recommendation. It seem eminently do-able. 8.5 Improved Income Data Collection Regarding income data collection, the recommendation is also simple. Collect data on income indicators at the start of a woman’s career in VHB and measure improvements at the close of the 4th loan cycle on all participants—not only on a sample. This will eliminate much of the guess work of estimating improved income for long-term participants. 8.6 Hire a Adult Educator in Health The health educator who left last November needs to be replaced. The job of monitoring the quality of health activities is too important for it to be left unattended any longer; and the multiple responsibilities of the expatriate Technical Support Manager means she will not be able to fulfill this function except in a back-stopping role. The quality of the health education talks needs to be looked at more as current monitoring only tracks quantity. This can most effectively be done only by direct supervision by an outside observer, and the Technical Support Manager and the new health educator have an important role to play in this area. 8.7 Improved Loan-making: Business Plans The project’s recent experimentation with hiring a local-language consultant to teach business management to VHB members seems worth continuing. The overall thrust should be to get clients to begin moving into more profitable activities. A suggestions that arose in one VHB meeting is that there are VHB members who are sufficiently entrepreneurial that they themselves might be able to become ‘visiting experts’ at nearby banks talking about this-or-that aspect of their business! 27 If this and similar activities go forward, one would like to think the project could begin to address the issue of high drop-out. If businesses become more successful, more women should be interested in staying in the project longer. Reducing the drop-out rate makes the project more efficient at achieving its goals. At the same time, a year ago project management began to conduct ‘exit interviews’ with departing bank members to understand better the various factors that influence a women’s decision to withdraw. Knowing more about these factors may give the project an opportunity to design other interventions. The target could be to reduce the drop-out rate in Blantyre to a more manageable figure, perhaps 40% by the 4th cycle? Tracking drop-out rates and trying to reduce it could become the next ‘systemic goal’ of a follow-on VHB project. 8.8 Continued Search for Self-sufficiency The cost-reduction exercise that began recently should be continued, looking wherever possible for improved efficiencies. A realistic target for cost savings should be established. Four changes worth considering seem to be the following: · Closing down the current Mulanje office should result in substantial savings in salaries as well as in general operations. · Second, in the judgment of the evaluator, the project is overstaffed at the support staff level. When the Mulanje office closes, at least three support staff will be transferred to Blantyre— duplicating some of the portfolio management functions already being performed by other Blantyre staff. Perhaps there is a viable job description for one, not for three. All three are talented staff thus the decision will be difficult, but it should take place. · Paradoxically, it appears that VHB salaries should be increased—in spite of the financial implications. HOPE Malawi needs to retain qualified staff if this project is to succeed, and there are inklings some staff would go elsewhere quickly if there were other job offers. This is not a way to build program quality. The issue requires HOPE Malawi study, but the closure of the USAID portion of this grant (Aug. ’03) will create an opportunity for annual contracts to be renegotiated. At that point, salaries could be adjusted upward to reflect the quality performance being delivered by the VHB personnel who will remain on. · From Sept. ‘03, VHB will become the only large program left in the HOPE Malawi portfolio. Central office staff—hired at the time of an expanding portfolio—will need to be downsized significantly to take into account the much-reduced portfolio. At the same time, great care must be exercised so that the VHB does not assume more than its fair share of head office costs. This can be a hard line to toe in lean times but is the only way VHB will be able to achieve its goal of self-sustainability. 28 APPENDIX E FINAL EVALUATION￾Village Health Banking Project-Guatemala 1.0 INTRODUCTION 1.1 Origins of the Current Project The People-to-People Health Foundation, Project HOPE, is a non-profit, a-political, non-sectarian organization with a mission “Helping People Help Themselves.” Origin of the current Village Health Bank (VHB) project lies in a 1992-97 USAID Cooperative Agreement to allow HOPE to establish the program in Honduras and Ecuador. For FY 1998-03, additional funding was requested from USAID and the U.K. National Lottery Charities Board to start a VHB program in Guatemala. A Midterm evaluation was conducted in this project in August 2001; this document is Part Two (Guatemala) of the Final Evaluation of that project. 1.2 Scope of Work The Scope of Work detailed an effort to conduct an evaluation of the Village Health Bank activities in Guatemala (and also Malawi and Peru.); assess HOPE Headquarters progress with the grants; and develop recommendations to strengthen continuing execution of the VHB programs. Main topics were to: ß Discuss if the hypotheses under-pinning the program model were sound. ß Discuss achievement of major objectives, successes, challenges, constraints, and unintended impacts. ß Identify changes made during implementation. ß Discuss impact of the program on target populations. ß Discuss whether adequate financial controls exist. ß Discuss whether the M&E system supplies accurate, timely performance data. ß Discuss program management including an adequate number of staff with relevant expertise, and supervisory systems. ß Discuss Lessons Learned and Recommendations. 1.3 Timing, Methodology, and Evaluator The evaluation took place beginning with two days of orientation with the Director-Income Generating in HOPE Center on June 2-3. Field travel to Guatemala took place from June 29 to July 4, 2003. During the field visit, program discussions took place with the project manager followed by three days of travel to VHBs. A presentation of preliminary findings was made to the Guatemala Country Director, the Deputy Country Director and the VHB Project Manager on July 4. The methodology involved the following: presentation by the project manager and staff of major program highlights; review of documents; in-depth discussions with program management and program support technicians; field travel including direct observation of village health talks and loan operations; and semi-structured interviews with randomly selected VHB members. 29 Five of seven HOPE field staff, técnicos, were observed as they engaged in project promotion. Nine banks were visited for approximately two hours each; each bank was chosen at random by the evaluator from those banks which were holding meetings during the three field visit days. Post￾VHB event interviews took place with 22 women at nine different banks—all of them selected at random, and as far as is known, none of them the (more active) VHB board members. Informational interviews were also held with the Manager of the Fundación de Desarrollo de la Micro-empresa, FUNDESPE, and with the microfinance Project Manager of CARITAS-San Marcos. The following table shows the distribution of the evaluation visits. Table 1: Village Health Banks Visited during the Evaluation Week VHB Name Mem bers Cycle Rural/Urban Highland/ Coast Técnico Coatepeque 15 5 th X X Heber Paz Femenil Bethania 17 2 nd X X Fco. Sandovál Ayuda Mutua 13 5 th X X Noemi García Las Samaritanas 16 5 th X X Oscar Barrios Nuevo Amanecer 14 6 th X X Fco. Sandovál El Triunfo/ Las Triunfadoras 18 12 7 th 2 nd X X X X Jelmo Figueroa La Ceiba 16 8 th X X Jelmo Figueroa El Jardín 12 6 th X X Noemi García TOTAL 133 5 4 5 4 The evaluation was carried out by a development practitioner who has substantial experience evaluating programs in Guatemala and who was also the external evaluator of the VHB Midterm evaluation in September 2001. 1.4 Structure of report The document is structured in sections. Section 1, Introduction, has now concluded. Section 2 presents Findings in Health; Section 3, Findings in Microfinance. Section 4 discusses several management issues. Section 5 presents Conclusions; and Section 6, Recommendations. 30 2.0 FINDINGS in HEALTH The Project HOPE model of Village Health Banks provides credit and health education to women, building on the village banking method of solidarity group credit. Health education is directed to the relationship between low incomes and poor health influenced by poor housing and hygiene, inadequate access to health care, and limited educational opportunities. It also builds on the well￾documented fact that increases in women’s income are spent more on health and nutrition than increases in most men’s income. As a health-focused institution, Project HOPE is particularly concerned with the project’s health impact, while funding comes from the PVO Office of Matching Grants that is primarily concerned with the project’s economic outcomes. This section discusses the project’s health component, the next chapter, the micro-finance component. This section discusses “first stage” changes, second stage needs, quantitative and qualitative gains in health, Midterm recommendations, and summary comments. 2.1 “First generation” changes To a gratifying extent, many first generation changes called for in the Midterm evaluation in Health and Training have taken place. Direct health education delivery Of fundamental importance was the Midterm recommendation that the VHB project should move away from the concept that village women could be trained to the point of delivering high quality health talks. Sometimes barely literate, overworked village women were judged at the Midterm to be largely incapable of delivering the high quality messages the project needed; the proverbial Training-of-(village)-Trainers was not working. Instead, the Midterm recommended that HOPE convert bank promoters into health extensionists, ‘direct health education delivery.’ VHB Guatemala field staff is now delivering health talks in every bank meeting. Training modules have been developed to guide these non-health personnel how to cover key messages in the subject, and an on-going training calendar is being carried out to build the new skills. A 180-degree change in Health Education methodology has taken place: project staff, not village women, are leading the discussions. This is a substantial step in the right direction. Revised modules As part of this switch, the evaluator was pleased to note a significant realignment in the health curriculum also. An Adult Education (AE) expert has been hired in Peru. With extensive consultation with Guatemala staff, she is in the process of developing new modules to guide the village health talks. Under her guidance, over months of design and pre-testing including a visit to Guatemala last year, a major shift away from ‘didactic’ to participatory talks has taken place. Substantial emphasis is given to games, learning-centered content, and other Adult Education techniques such that the sessions have become quite enjoyable; virtually all of the hundred or so women observed in the talks were seen to be enjoying themselves, laughing, joking, taking pleasure in the module and generally having fun. This is an important gain. Modules have been reduced to four ejes—loosely translated as axis of intervention: healthy women, women and children, healthy families, and healthy communities, each having multiple 31 themes per module. Currently one module has been completed and is in implementation, another in the testing stage, two others have yet to begin. The modules are Adult Education-based following a structure of: Introduction and review of past session; Development of the theme; Review; Conclusion and segue to the next session. Participants are encourage to take on “promises,” personally applying the health lesson before the next meeting—an excellent innovation. The methodology is highly participatory and learner-centered. Moreover, the length of the health talk has been extended from ½ hour during the Midterm to a full hour. Tellingly, while many women in the Midterm complained of having to spend an extra half-hour then, none of the interviewees complained of having to spend an hour now. Clearly, the quality of the talks has gone up so much that women are pleased to be attending. The evaluator enthusiastically endorses these changes. Field staff is showing great creativity in preparing the props needed for the sessions, and lots of innovation in creating low-cost charts, brightly hued visuals, and ‘homemade’ lesson aides. All of this has been carried out at almost no financial outlay—simply with ingenuity and creativity. Improving staff skills Staff skills in AE techniques have improved from the Midterm when a laborious reading of project materials was the norm. Time control was good in all sessions observed. Most field staff (though not all) demonstrated good use of women’s names. All staff demonstrated adequate preparation for their session, knew what was expected, and carried out the health talk largely as planned. The overall effectiveness of the talks will be assessed further below. Supervision Recently an educator and a nutritionist from another HOPE Guatemala project have begun providing part-time support to the VHB nurse educator. This is a positive development because it strengthens the VHB project while also enhancing cross-project learning. It was also reported (though not observed) that the two educators conduct on-the-job training for each field worker by carrying out each new modules in every field worker’s village. This is a good way for the técnico to see how a session should be carried out. It may be that additional ‘coaching’ sessions should be carried out after this first on-the-job training to see how well the techniques are being applied. A monitoring instrument to measure health talk quality was not observed during the field travel. 2.2 Second generation changes Now that the fundamental program shift has taken place, a number of second generation issues need to be addressed. These involve making maximum use of women’s time and enhancing their learning. Overly based on games/ loss of focus? One observation is that health sessions in VHB Guatemala have gone from the extreme of being a highly theoretical, boring recitation of facts, to the other extreme of being an unrelenting series of games, role-plays and AE dynamics. Over half the sessions observed used four different dynamics in the space of 60 minutes- one after the other. With this saturation of techniques, the balance 32 between the theoretical and the practical appears to have swung too far. Women are moved from one fun-filled simulation to the next, with seconds between one and the next and with no pause for reflection or opportunity to interiorize the experience. In half the sessions, the game-playing before and after the message took fifty minutes, leaving a mere 10 minutes of content; in several others, the message content was only 15 minutes. In two sessions, setting up the simulation took 35 minutes, a dynamic that had clearly gotten out of control. These errors do not maximize participants’ learning. In this somewhat frenetic rush from one dynamic to the next, the overall objective of the session appears to have gotten lost. Staff demonstrated some wonderful group techniques, but the evaluator left the session sometimes thinking: “what was it all about?” The module guide spells out clearly what the learning objective is, so the problem is not that the goal of the session is not articulated. The problem seems to lie more in losing sight of the forest amid all the trees of activity. Under-emphasized are the concepts that the purpose of a workshop should drive the selection of every module, and that every technique should be directed to the accomplishment of the learning objective. A related concept is that each session should be focused on actual learning, not theory. In a talk on balanced nutrition, the module called for classifying food groups; and various dynamics were used to get women familiar with the classification. Well and good, but in some communities the subtext of the message was that fish was part of a balanced diet, when women in this rural mountainous community had probably not eaten fish in five years. Thus, learning appropriate to the context is overshadowed by the theory behind the session. In another session, the técnico slipped into discussing the properties of vitamins, A, B, C, D, etc., clearly over the head of the women as judged by their responses. A wonderful question put by one woman in another community: “how do I achieve a balanced diet since I am poor?” was never answered; another, “what do you do if you can’t eat vegetables every day?” was similarly never answered. In carrying out these modules, staff needs to begin thinking of practical, actual learning in the contexts in which the women find themselves. A third concept recommended by the Midterm is that most sessions should become focused on behavioral change. What does the project want women to do differently as a result of this talk? How will the project measure the effectiveness of the session in terms of how people change the way they live? This is a complicated theme, and VHB Malawi is beginning to think about this issue. See Attachment C for work so far. Apparently there is an all-VHB seminar scheduled for August that will address this theme. Session facilitation A number of positive facilitation techniques were cited above, but others are not yet evident. One important one is taking a few seconds to reinforce the correct response. In the sessions observed, almost no responses were reinforced. Instead, what happens is that correct answers and incorrect answers are dealt with the same—that is, with little or no verbal reinforcement of the comment and moving on to the next answer. This misses the opportunity to clarify for the women what is correct content and what is not. Other reinforcement opportunities are also being lost. In a discussion on the nutritional value of vegetables, one woman said, “Before I do anything with vegetables, I need to wash my hands,”—a wonderful example of the key message of improved hygiene from a previous module!—but the comment received no verbal ‘reward’ from the facilitator. This is easy 33 to fix: a simple “wonderful comment, Mrs. X, now let’s move on” would suffice, but this is not yet taking place. Use of women’s names has been noted above. Técnicos who have not yet memorized the names of their clients should be encouraged to do so. This is good AE pedagogy and an important way to move the VHB from a project/client relationship to an AE facilitator/learner one. Many of the sessions observed were repetitive: for instance, dividing 12 people into three groups and then each group tediously repeating almost the same content as what the first group presented. A more efficient use of time could be for the three groups to address different topics; another would be to select only one group to make the presentation/game, and ask the others to add comments. Full participation seems to take precedent over best use of time. It was observed that several of the field staff were not fully in command of simple content: having to look on the nutrition chart where to put eggs, misplacing the ‘fats’ in the ‘meat’ category, etc. Small slip-ups to be sure, but one has to wonder whether if these errors are taking place in one of the simpler modules, what could be the misrepresentation of facts in more complicated modules such as women’s health, self-esteem, and HIV/AIDS. Finally, it appears from the sessions the observer participated in that there is a substantial difference in rural women’s ability to understand and process health messages compared to their urban counterparts. The quality of the facilitation being equal, urban groups were quicker to capture the point of the session than were rural women, and urban women were able to make connections between the talk and their daily lives more easily. Since supervisory staff generally agreed with this comment, it appears some field staff need to develop the ability to ‘scale down’ the complexity of their modules when presenting in rural communities. This probably means slower-paced sessions, more repetition of the technical content, and more repetition in the wrap-up. Naturally, these adaptations would mean less time for the dynamics, or using dynamics that are completed more quickly. It will be seen that this recommendation was also made during the Midterm: “take into account literacy and socio-cultural characteristics of the audience.” Observed quality of the health talks In a previous section it was noted that staff skill in AE have improved since the Midterm but the discussion was left incomplete. Having analyzed aspects of staff’s facilitation skills, it is now possible to assess the overall effectiveness of the eight sessions observed. Summarizing evaluator findings based on extensive note taking during each of the sessions: · two sessions suffered from either weak content or sterile facilitation; · two sessions were carried out with good facilitation and obvious concern for the women’s well-being but lacked full technical competence; · two sessions were adequate in both technical content and in facilitation; · one session was good in both content and facilitation; · one session was superb: good technical content and wonderful facilitation. · In sum: two “D’s”; two “Cs”; two “B”; one “A”; one “A+.” 2.3 Quantitative Documentation of Improved Health 34 For three years, the project has struggled to track and document improved health. Baseline heath data were collected in July 2000, 22 months after the project started, (N=200). A Midterm KAP study was undertaken in October 2001 (N=313.) In March 2003, a third effort was carried out (N=346), 146 women who were new to the program, 200 who were 4th cycle or beyond members. In all cases, a detailed survey report was prepared. Unfortunately, methodological weaknesses characterized most of these efforts: the “co-mingling” of newer VHB members with older ones. Discussions in the FY02 and ’03 Annual Reports regarding project impact in health should thus be interpreted carefully. In the Midterm KAP study, comparisons between 59 older members and 102 new members are sometimes contrasted, sometimes reported together; graphically they are intermingled. The FY03 survey contrasts 146 new members with 200 older members, but then sometimes sub-classifies the distinction between 52 4th cycle respondents from 2001 with 148 4th cycle respondents from FY02. Somewhat confusing. Aware of these difficulties, the ’03 study did track health changes over time (on some indices but not on all) in the same population. Unfortunately, due to dropouts and other selection criteria, the “N” of this study is only 52. Conclusions to be derived from a sample this size should be voiced tentatively. Again, graphical representation of differences among the three groups is rather muddled. Analyzing the FY03 data carefully, for women who remain in the project four loan cycles or more, one can speculate there may be some widespread change in health knowledge: in recognition of danger signals during and after pregnancy, and in pre-natal check-ups—though this may also be attributable to a more literate clientele. However, the co-mingling of the cohorts must be recognized as a methodological error, and a social scientist would reject such conclusions as unwarranted. Extrapolation from this sample to the larger VHB population is similarly questionable. What is clear from the many documents is that project staff invested much time and effort in collecting the information. A recommendation will be offered in this regard. Numerically, one can speculate on health gains for approximately 2,800 women: 300 women with multi-year four-or-more cycle participation who are still involved in the project, plus 2,500 four-or￾more cycle women from years past. 2.4 Qualitative Documentation of Improved Health Beyond quantitative gains in health surmised in the previous paragraph, qualitative gains merit discussion using two VHBs and several individual narratives as examples. Cruz de Piedra is a rural village bank of 14 women in their 8th cycle. By happenstance and great luck, this was one of the villages selected at random for a final evaluation visit, having also been visited during the Midterm. The group is led by Margarita López, an indigenous woman of perhaps 45 years old who has four children under twelve. In the interim since the Midterm, Doña Margarita has found employment in the city of San Marcos while continuing her representational functions in the VHB, and has led the group with great energy to negotiate with the mayor of San Marcos on a number of village concerns. (In fact, just the day previously she had been in the mayor’s office lobbying.) Because of these initiatives, a road has been opened to the village (!), and a budget has been prepared for the construction of a Q. 2.4 million water system (!). These are huge gains for an 35 indigenous community led by an indigenous woman. Though one must credit Doña Margarita with the personality and drive to have carried these initiatives through, Margarita tells everyone she got her start as a leader of the village health bank, and it was with the VHB that she learned the power of organization and representation. Margarita is not alone in this village: several other women were articulate in contributing to the health talk. El Jardín is an urban VHB composed of 12 women, recently completing their 5th cycle. Results in El Jardín are not so dramatic—group leadership does not have the dynamism of Doña Margarita— but the changes could be equally profound. In a module on women’s health, this group was strongly affected by the talk on self-esteem, “negotiating equality” with spouses, defending the rights of abused women in the neighborhood, and related issues. Said one woman: “if I think about it [women’s rights] much, I will start to cry.” Another said: “we need to stand up for one another when one of us is being abused by her husband.” In post-event interviews, four women said this was the first time such a subject had ever been talked about; also that sessions on vaginal cancer, detection of breast cancer, menopause and other women’s health issues were similarly new and changing their lives. Life stories that came out of individual interviews—selected at random by the evaluator—are the following ‘un-cued’ comments that reflect many interviewee responses: · Dina Rodriguez says the health talks have seriously changed her life. Before, her husband would not eat a meal if it did not have meat. After going to the nutrition talks, she came home and shared what she had learned with him, and now the whole family eats balanced meals with lots of vegetables, and eats meat only every now and then. · Norma Rodriguez, and others, thinks the greatest value she found from the health talks was that women have learned to value themselves as women: self-esteem has improved. · Amparo Orusco had never attended such meetings before; the talks “woke her up and have had a liberating effect; before we were alone-- now we have colleagues.” She now gets her PAP smear regularly where she never did before. · Anselma de Leon learned in the talks of the importance of self-examination of the breast, and children with full vaccinations. To her the talks reinforce health messages she has heard elsewhere in the community and encourage her to do something about accessing the services. · Gloria Alicia Mérida Ochoa remembers the PAP talks, but highlights the learning “how we value ourselves; we women have rights too.” · New topics for Araseli Orosco and others were AIDS, cancer, and the importance of marital fidelity. · For Elba Teresa Yoc, the health talks have been fun; the most important thing she learned is that women have the right to go to the police and other GOH social welfare offices if they are mistreated. Staff reports corroborated by the evaluator suggest that some women take out minimal loans—or only save rather than borrow—in order to be able to participate in the health talks. It appears the VHB has had a big impact on the lives of clients who remain in the program for a number of cycles, even if such benefits are difficult to quantify. 2.5 Midterm Evaluation Recommendations 36 During the week of July 2001, a Midterm evaluation was carried out on the Guatemala VHB project. Since the evaluator/writer of this Final Evaluation was the leader of that event, it is possible to compare Guatemala final status with midterm recommendations. The following table demonstrates the status of midterm recommendations as discussed above. Table 2: Accomplishment of Midterm Recommendations in Health Midterm Recommendations Status Village health talks demonstrate serious weaknesses but weaknesses were identified previously and a plan to improve is underway. First stage weaknesses have been identified and improved. Recommendation that commu nity talks be directed more to needs and interests of the women. Accomplished. Talks are now more responsive to women’s interests. Recommendation that VHB staff assume a larger role in conducting health sessions; also recommendation that staff job description evolve into “development promoter” instead of “credit promoter.” Accomplished. Staff job descriptions and job titles changed. Recommendation to use MSP staff and HOPE Child Survival staff as resources more. Not being accomplished as far as one could observe. Recommendation for better selection of village health volunteers (vocales). Accomplished. The methodology of most observed sessions is a laborious reading of project materials, rather than effective learning events. Methodology completely changed. Recommendation to re-think village training strategy in light of the question: what behavioral change can be legitimately expected from twelve half-hour sessions? Not yet accomplished but improvement has taken place. Recommendation to re-think training methodology. Every session should: 1) address behavioral change objectives; 2) appropriately select a topic based on villagers’ priorities and needs; 3) take into account the skills of the facilitator(s); 4) use accurate, quality IEC materials; 5) use appropriate adult education methodology; 6) take into account literacy and socio-cultural characteristics of the audience Accomplished. 1) Not yet accomplished in full. 2) Accomplished. 3) Facilitators are HOPE staff. 4) Improving w/ good creativity. 5) Accomplished. 6) Not yet fully accomplished but large steps have been taken. Health Coordinators’ Plan de Acción is adequate but needs more concrete outputs and more follow-up. (not observed) Recommendation that Health Coordinators travel to VHB sessions, personally co-facilitate talks, and begin to serve as in situ coaches for HOPE promoters and village vocals. Being accomplished At the least level of expectation, every session should start by drawing on village women’s experiences and conclude with a review of key points discussed. Being fully accomplished in the “A” and “B” sessions; not much in “C” and “D” sessions. Recommendation for significant technical assistance from an IEC expert. Accomplished. 2.6 Conceptual Issues Before concluding this section on health, it seems appropriate to offer two brief reflections on the VHB program model. Succinctly, throughout the developing world, millions of women are unable to receive services from their over-worked and under-funded Ministries of Health. When these people fall sick, there is nowhere to go that does not require sometimes hours (or half-days) of travel. Moreover, preventative public health activities are all but unheard of. What the VHB model 37 offers is a way to fund well-trained primary health promoters outside of, and complementary to, the Ministries of Health to promote women’s ‘demand-driven’ search for better health services. When financial sustainability is achieved, the model offers an effective way for women who would otherwise not receive health education to learn how to improve their lives. At the same time, it offers marginalized rural and peri-urban women an opportunity to improve their household incomes (and the health of their children), as well as to grow in self-esteem. This evaluator finds the model compelling. In a place like Guatemala where infant and maternal mortality are still, the model should be made to work: it is a way thousands of women and children can be reached who would otherwise not be. And though there is debate on this topic, the evaluator views favorably the strategy of Project HOPE to create a country platform in places like Guatemala independent of grant funding. Growth in professionalism in career national staff and long-term organizational development are very hard to achieve when a country operation is surviving from one short-term grant to another. The evaluator supports the implicit ‘program philosophy’ that a successful VHB project could create a program base on which to build. 2.7 Overall Quality and Impact Reflecting on the overall impact of the project in health: In the design of the health talks, VHB Guatemala accepted the recommendation of the Midterm evaluation and changed its education methodology from village trainers to direct delivery by project staff, a quite positive step. Redesigned modules emphasize Adult Education (Freire) principles to good effect. Several of the field staff have good-to-excellent abilities in this area. Others are not yet up to speed in the skills required by the new focus. The overall effectiveness of health education talks varies. The new methodology is a pleasure to watch, but the overall learning appears to run from a great deal with skilled field staff, to not so much with less skilled staff. For five of the eight sessions, an overall assessment might be “6-out￾of-ten” while two were 8/10 and one, 10/10. Thus, there is room to grow in consolidating the gains of the new methodology and the new modules; in incorporating other adult-education principles; and in homogenizing the quality of the talks throughout the VHB. In general terms, broad program impact could not be discerned, i.e., widespread uplift in health status, or changes in health behavior in a substantial number of women. Project staff in Guatemala expressed concern that the evaluator’s time did not allow the capturing of these changes though they are taking place. In fact, large-scale changes cannot be documented because of the methodology of health data collection. In addition, data in the next section will show large movement of clients over time—not only constant dropouts but also constant new entrants— such that the number of women exposed to the full cycle of health talks is quite reduced. For these factors and because of the newness of the methodological shift, six months or less in implementation, long-term program￾wide impact in health has not yet been observed. Meanwhile, individual gains in improved health are clearly taking place. 38 3.0 FINDINGS in MICROFINANCE The Midterm judged the credit side of this project as one of the most well-supervised and well￾documented program components in this writer’s experience. As such, one purpose of a final evaluation becomes to update indicators with FY03 data. Many indicators could be tracked based on the project’s extensive monitoring system. The most important are: number of banks and members, growth in total loans distributed, arrears, project impact, and operational sustainability. Accomplishment of Midterm recommendations will be documented, and concluding comments will be offered. 3.1 Expansion into Quetzaltenango Project management has recently taken the decision to expand into Quetzaltenango. At this writing, only five banks are operating there though there is hope the number will grow. Similar to VHB decisions elsewhere, management believes the growth of the portfolio and operational sustainability will occur more rapidly if the project works where there is higher demand for loans and higher absorptive capacity of VHB clients. The evaluator makes no comment on this strategic decision other than to hope that opening in Quetzaltenango does not take resources away from the rural portfolio excessively. 3.2 Growth in Number of Banks The number of project banks has grown steadily throughout the life of the project. In late 2002, a ‘winnowing’ of non-functional banks was carried out. The net total of active banks at time of the evaluation is 177 of a project goal of 85, 208% of target. 3.3 Growth of Membership VHB membership has grown steadily throughout the project. Total membership as of June 15, 2003 is 2,684 compared to a target of 2,125, 126% of target; 1,861 are active. 3.4 Cumulative Loan Distribution The chart reflects the cumulative growth of the portfolio in hard currency at Q. 7.75: $1.00 (22/6/03). Total value is $1.6 million compared to a target of $ 2.37 million, 67% of goal. Annual Reports narrate this shortfall is due to smaller loan sizes than planned. 0 500 1000 1500 2000 '99 '00 '01 '02 '03 Cummulative Disbursements in $000 3.5 Total Member Savings 39 VHB members are required to save 10% of their profits. This is an appropriate strategy that promotes that at the end of the loan cycle the woman will have something left for her effort. In fact, in some cases the savings can be substantial. The evaluator was surprised to interview one women selected at random who had saved over Q. 1500, $ 220. This is a substantial accomplishment at her level of economic activity. Withdrawals from savings are permitted when a member is leaving the VHB, in case of family emergency, or to pay off a balance due on the loan. In 2002, an effort was undertaken to clean the project portfolio of non- or under-performing banks. When this was done, a substantial amount of member savings was returned, noted in the jump in withdrawals in FY ’03, over half a million quetzals. The chart shows the savings accumulation net of these deductions. 0 100 200 300 400 500 600 700 6/'00 6/'01 6/'02 6/'03 Savings and Withdrawals in Q. '000 Withdrawals Net Balance Total dollar amount at Q7.75: $1.00 is $64,875 compared to a target of $ 358,111, 18% accomplishment. Savings targets, like several other numerical projections, were based on the Ecuador experience, which proved unrealistic in the Guatemala replica. 3.6 Inactive Banks When a bank does not pay 100% of its loan within 90 days, it enters a status called “inactive” by the project. This means the VHB no longer receives health education talks, and that further contacts between HOPE and VHB are suspended until the loan balance is paid. This is a form of pressure to encourage the banks to continue repaying. At the time of the evaluation, the number of inactive banks was 46 of 150, 31% in total. 3.7 Member Drop-out Member dropout in Guatemala is significant. The total number of dropouts to date is the difference between total current borrowers vs. cumulative borrowers, 1861 of 4,332, total 53%. A modest recommendation will be offered on this subject. 3.8 Improved Income At the project level, VHB has established an effective tracking system to measure improvements in income by collecting economic and household data on each entrant to the VHB during the 1st cycle. For those who are still active at the close of the 3rd loan cycle, the survey is repeated on a sample basis, and the changes are measured. The following table represents those results in Guatemala as reported in the Year Three and Year Four Annual Reports (and to be reported in Year Five), with improvement in household expenditures being taken as a proxy for increased income. 40 Table 3: Quantitative Improvements in Household income Criterion Year 3 (N=136) Year 4 (N=145) Year 5 (N=242) Percentage increase in household expenditure 18% 18% 12% Percentage increase in households incomes 48% 52% 37% Percentage increase households reporting “better” incomes (56%/88%) 59% (56%/88%) 57% (58%/76%) 24% These figures have been prepared in a methodologically sound manner. Demonstrated results are consistent over time. The project appears to have increased income for a number of project participants: for those who remain in the program over several years, 829 people as of June 30, 2003, substantially improved income has resulted, perhaps by as much as 40%-50%. In addition, in the absence of hard data on the subject, the evaluator speculates on some gains in income for those FY99-FY01 clients who remained in the project four cycles but who no longer participate in the project, 1,358 people. This assumption grows out of a belief that fully successful women will continue in the project; those who are somewhat successful will have less economic incentive to stay in. Thus, income gains for these women have been estimated less than for the 829 immediately above. Quantification of this gain has not been attempted. 3.9 Non-quantifiable Impact Discussions with interviewees selected at random suggested numerous non-quantifiable impacts also. Women’s ability to represent themselves has clearly improved, as in the Cruz de Piedra example. In addition, project documents speak of women being able to receive attention in commercial banks for the first time. Several interviewees indicated they made enough money to become financially self-sufficient (at least for a time); others reported paying school fees with their profits. Among the most salient comments: a) Amparo Baptista (6th cycle, from the beginning) sells vegetables and clothes in the market. She has Q. 1702 ($220!) in savings and is proud of her accomplishment down to the last quetzal. b) Norma Rodriguez (6th cycle, from the beginning) has developed a new habit of saving where she saves throughout the year to invest in her business at peak times. She had Q. 1400 ($200) in savings and then invested Q. 1000 of that into her business of quilt-making right before Christmas. c) For Ilaria Elvira Aguilar (2nd cycle), the VHB “opened new doors;” she makes and sells tortillas to local shops, and is paying regularly and comfortably on a loan of $260. d) For Magdali López Morales (5th cycle), the VHB loan has enabled her to expand the stock of her village shop with more milk, rice and sugar. She has gotten out of her previous chicken-raising business because it was not making any money. e) Heidi Areli (2nd cycle) sells Guatemalan shoes across the border in Mexico. The VHB loan allowed her to move into higher fashion shoes, with a higher profit margin for her efforts. She reports no difficulty making the fortnightly payments. f) For Elba Teresa Yoc (1st cycle), the $200 loan has allowed her to expand the foodstuffs at her store. Her husband died seven years ago, and the profits from the store have allowed her to put her child into technical school. 41 g) With her $260 loan, Elisira Méndez (3rd cycle) grows greens and vegetables on a larger plot than before, selling part of the harvest every week depending on her cash needs. She finds the bi-weekly loan repayments easy: “all I do is sell a little more of my vegetables.” Beyond these self-descriptions, though the numbers are small, several interviewees indicated some of their bank colleagues had used the VHB loan to get out of the clutches of moneylenders who charge 20% a month. This too must be counted as a significant project accomplishment for a limited number of women. 3.10 Arrears 30 Days Past Due Project documentation and micro-finance literature suggest one important way to measure the soundness of a loan portfolio is to track arrears more than 30 days past due. The proposed goal was “30 day arrears under 5%.” Status at time of evaluation is 3%. This is an excellent accomplishment. 3.11 Operational Sustainability Operational sustainability (all costs covered by project income) is one of the gold standards of micro-financing projects, and has been a major focus of the Guatemala VHB from the beginning. Current operational sustainability at time of the evaluation is 69%. The project manager estimates full operational sustainability will be achieved by about March 2004. This is certainly a goal to be striven for. 3.12 Comparative Experiences with Operational Sustainability To get an idea of the San Marcos micro-finance context, interviews were held with two colleague micro-finance NGOs, the Fundación de Desarrollo de la Pequeña Empresa, FUNDESPE, and CARITAS-San Marcos. The experiences of these two entities—sometimes colleagues, sometimes competitors—provide an interesting sidelight on difficulties of achieving operational sustainability in San Marcos. Selected indicators from those informal interviews are presented in the table below Table 4: Selected Comparative Indicators of Colleague NGOs in Micro-finance Index HOPE FUNDESPE CARITAS Clients 1860 480 3,800 Portfolio size in $US 168,000 387,000 774,000 Average loan size $US $200 $800 $225 Client/ técnico ratio* 206 120 633 (?) Approximate desertion rate 31% <10% +/-8% Arrears 3% 11-12% 5.6% Operational sustainability 69% 60% >100% * (active clients/técnico + supervisors. CARITAS data may be mis-recorded.) FUNDESPE began its micro-finance program in 1988, ten years earlier than VHB. There was a major housecleaning of the FUNDESPE portfolio in the early Nineties when millions of quetzals were written off. Notwithstanding a lower client/staff ratio (meaning an operation that is less efficient) and a more carefully chosen clientele, FUNDESPE has not been able to achieve operational sustainability since the portfolio restructuring. 42 The CARITAS loan program started in 1992 with an agricultural credit program that evolved into a pure micro-finance effort in 1996. CARITAS is a major actor in credit in San Marcos with double the number of clients of VHB and a portfolio five times bigger. Half of this bigger portfolio comes from the rollover of agricultural borrowers before the start of the (pure) micro-finance component. Its average loan size is similar to that of the VHB, but its group size can sometimes be as high as 50, much bigger. CARITAS has achieved full operational sustainability over the last five years, a laudable achievement. Building on the agricultural loan portfolio base, CARITAS’s loan funding over the five years of subsidized operations was a Q. 3 million, $387,000, while the VHB loan fund has been $ 300,000. The CARITAS example suggests that achieving operational sustainability in five years is do-able under the right conditions and with good management. However, doing so may be facilitated by building on an existing program base (not starting from scratch); it may also require a somewhat bigger financial ‘jump-start’ than the VHB Guatemala grant contemplated. 3.13 Midterm Recommendations The following chart represents accomplishment of Midterm recommendations in micro-finances. Table 5: Accomplishment of Midterm Recommendations in Microfinance Midterm Recommendations Current Status Generally well run and under control. Remains so. Lots of time and energy spent on simple bookkeeping; opportunity to streamline and dedicate time to other tasks; recommendation to use computers more. Accomplished. Procedures are much more computerized resulting in substantial saving of staff busy-work. Recommendation to create a “window of discussion” in each village session to analyze problems and discuss other (non￾bookkeeping) themes. Not observed. Recommendation to minimize staff transfers Accomplished in the Highlands. Not accomplished on the Coast (see Section 4.) Recommendation for more staff training in credit, community development and other themes. Accomplished. 3.14 Overall Assessment To a large extent, the project is reaching its goal in micro-finance and income generation. This is seen in the way the micro-finance activities are carefully implemented and monitored; the growth in the number of banks; the number of women participants; and growth of total loan disbursement, 67% of target. Member savings is 18% of goal, somewhat, but not much, attributable to fall in the value of the quetzal. The number of inactive banks compared to active banks and member dropout compared to overall membership do not appear to exceed industry trends though they are higher than for the two Guatemalan colleague organizations. Arrears thirty days past due represent accomplishment of the project goal in that area. Increases in women’s income have been demonstrated via a logically consistent methodology. For those who remained in the program for a substantial period of time, 829 people as of June 30, ’03, improvement in income is perhaps in the 40%-50% range, a significant accomplishment. Lesser impact is also inferred for 1,358 four-cycles-or-more participants who are no longer in the project. Interviewee comments confirm the impression that project has had an important economic impact on women’s lives. 43 Operational sustainability has not been achieved within the life of the five-year grant. There are encouraging signs—and project management enthusiasm— that operational sustainability will be achieved within the next 12 months. One hopes the enthusiasm is justified. Experience from colleague organizations in San Marcos leads one to hypothesize that achieving operational sustainability within five years in a start-from-scratch operation may not have been a realistic goal. 44 4.0 MANAGEMENT ISSUES Several management issues came up during the field week that have bearing on the accomplishment of project goals. To be discussed are expenditures, rotation of staff, consolidation, staff skills, staff￾to-client ratios, morale, and the program area of Malacatán. Accomplishment of Midterm recommendations will also be discussed. 4.1 Budget vs. Expenditures The following summary financial information is provided. Verification of these data was beyond the Scope of Work, thus these figures were provided by HOPE. Project expenditures for the life of the project will be approximately $1.132 million, 100% of original budget, broken down as follows: · Headquarters support is $145K, 13%. · Local personnel cost is approximately $318K, 28%. · Grant financing is $ 300K, 26.5%. · Other direct operating costs are $143.3K, 13%. · Indirect cost is $223.7K, 19.7%. In other VHB countries, the issue of support for the in-country head office was discussed. As reported by the Project Manager, in Guatemala the VHB contributes nothing to Guatemala City support except a portion of the salary of the Country Director. A review of these figures suggests the project is in line with NGO industry standards. 4.2 Rotation at the Project Manager Position The Guatemala VHB has had three project managers in five years. While a turnover rate like this is not unusual in development projects, the effect in the VHB project has been noticeable. The first project manager was relieved of his responsibilities just at time of the Midterm evaluation. Though the man was not interviewed during the Midterm, it was clear that his management style had not created a sense of team, and the morale of the field staff at the Midterm was low. Additionally this individual did not seem overly involved in or interested in the health side of the project. Thus, almost two full years of project momentum were slowed by weaknesses at the top. Shortly after the Midterm, a replacement came on board. This individual only lasted four months, causing another half year of leadership vacuum. The current project manager, judged a strong performer in micro-finance and interested in and committed to health, has been in the position for just over two years. Thus strong project management can only be posited for about 40% of the life of the grant. 4.3 Program Consolidation 45 During the last two years, it appears the program has consolidated many gains. Illustrative of this consolidation is a presentation made by the Project Manager on the first day of the evaluation visit. This included listing a series of weaknesses, many of them “internalized” by the project based on the Midterm event, followed by a presentation of strategies to address the problems: reorganizing the project structure, institutional strengthening, portfolio growth and improving the quality of health talks. Regarding the reorganization of the project structure, the presentation indicated that: · Staff functions have been revised to include health and micro-credit activities. · Work plans are now prepared by the staff (not the supervisor). · A monthly performance and ranking system of field staff has been implemented. · The formal system of performance evaluation has been strengthened. · Preparation of manuals has been accomplished on staff functions, credit methodology, credit rules, a procedure manual and an education manual. Regarding institutional strengthening, the presentation highlighted: · Staff training during FY03 in methodology, use of electronic spreadsheets, credit analysis, collection strategies, planning educational talks and others. · Computerization of a number of repetitive loan register forms. · Strengthening of internal project controls, including electronic worksheets that automatically calculate disbursements, balances, check writing and portfolio record keeping. Regarding institutional growth: · Development of a strategy to concentrate on the urban ‘core’ near the VHB San Marcos office; also moving the project into Coatepeque and Quetzaltenango. · A promotion plan that included strategies, activities and educational materials. In health, in addition to aspects already discussed, the presentation highlighted: · Self-identification by staff of the health component as a “competitive advantage.” · Carrying out PAP smear campaigns via community-wide efforts to 591 women. · Creation of a work group charged with continuous improvement in the health education component. This is an impressive list of gains achieved in two years since the Midterm. 4.4 Rotation of Field Staff Quantifiable data were not collected on full staff movement during the life of the grant and might be confusing to present in any case. Anecdotal data suggest that rotation of field staff may have been substantial. One VHB group observed it has had five VHB técnicos in four years. Only two field staff of seven interviewed during the Midterm were still employed at time of the Final evaluation. Of nine current técnicos and supervisors, four had less than eight months experience and only three a year to two years in the project. The Malacatán office has lost six staff since its inception, four of whom were released due to poor performance. During the Midterm evaluation, two of three field positions were vacant in Malacatán; by coincidence at the Final, two of three field positions were again vacant in this area. 46 Since much of the success of village banking rests on a relationship of trust and confidence between the project and its clients, each of these staff changes—and however many more have taken place— represents a setback. By contrast, CARITAS has lost or fired two field staff of six in a period of five years. Retention of skilled staff is an essential component of VHB, and the issue of remuneration vs. responsibilities may require study in VHB Guatemala. 4.5 Staff Skills As discussed earlier, the skills of the field staff vary a good deal. One was judged to be excellent, another quite good. One was judged with developing skills, but with the right mental attitude to become good field worker. Two were judged substantially weak. (Three new employees were not evaluated.) Passing comments suggested one of these two had been under observation for a year, yet no decision had been taken on his long-term future. Releasing under-performing staff is always a hard job, and VHB Guatemala needs to find— and retain— high quality performers. 4.6 Management Supervision Mention was made earlier that there is no systemic evaluation of health talks. One should note however, a well-done monitoring form the Project Manager has developed that tracks staff performance in microfinance each month. Data from the monthly financial report are automatically downloaded on 7 indicators: new banks formed during the month, total monthly loans distributed, active portfolio, and active portfolio less < 30 days arrears, and arrears of 31-60, 61-90 and >91 days. A percentage combining these indicators is then prepared and each staff is presented with graph of his/her performance compared to the other field staff at the end of the month. This seems an excellent supervisory (and motivating) mechanism other VHB countries could adopt. 4.7 Staff Morale Staff morale has improved since the Midterm. This is reflected in a higher esprit de corps; staff comments on improved training opportunities—six of them in FY03 alone; one project staffer about to depart on an international training assignment; and an overall sense of purpose that was not perceived at time of the Midterm. Project management may be more sensitive to staff concerns than two years ago. In addition, HOPE Guatemala has recently undertaken a strategic planning exercise (document not reviewed) perhaps giving the country office a clearer sense of direction. 4.8 Staff Ratio One Midterm recommendation was that management should look at the ratio of field staff to clients and the subject may need to be re-visited. Currently seven field staff (three new) are servicing 1861 clients, 265:1, not much different from the time of the Midterm. Two supervisors are not counted in that total. The Project Manager suggested it should be possible over time to move to a ratio of 300- 325:1. If so, management theory of span of control suggests one supervisor should be sufficient to provide adequate supervision to at least six field staff. 4.9 Malacatán 47 Malacatán is the Coastal area of the project and represents approximately half the project portfolio. It is different in a number of ways from the Highlands around San Marcos: in geography (humid, coastal); in culture (mestizo); in agriculture (cattle, coffee, and plantation); in economic activity (more commerce near the Mexican border.) As noted earlier, staffing in Malacatán has proven problematic for several years, both at the Midterm, now, and (perhaps) in between. Because of a recent staff departure, at the time of the evaluation, Malacatán’s approximately nine hundred clients were being serviced by two people—an impossible load. Though financial records and repayments currently seem in order, if more field staff cannot be hired and retained to work this half of the portfolio, the Malacatán situation could represent an undocumented risk to project sustainability. 4.10 Midterm Recommendations in Management The following table represents the status of Midterm management recommendations. Table 6: Accomplishment of Midterm Recommendations in Management Midterm Recommendations Current Status Past manager weaknesses have been identified and some corrective action taken. Accomplished in the hiring of the incumbent Project Manager and also the Deputy Director (see below.) Urgent need to bring the incoming project coordinator up to speed quickly. Accomplished with the incumbent. Urgent need to bring the proposed Deputy Director up to speed quickly HOPE Guatemala has hired an experienced physician to oversee the entire portfolio, and this individual is well versed in VHB strengths and weaknesses. Recommendation for better strategic and annual planning: with banks, with MSP, with others. Strategic planning taking place in HOPE/G. Unclear whether strategic planning with banks and Ministry of Health is underway. Ecuador experience suggests that project is currently overstaffed by 33% based on current client loads; recommendation that project standard should be 350 socias per promoter, not 250 as at present. Not accomplished. Project still appears overstaffed—though this is an area of concern to the Project Manager. Ecuador ratio would suggest a reduction in field staff of at least one field worker (1800/300= 6) and perhaps one field supervisor. Recommendation to improve sense of team, organizational climate and staff/ management relations. Accomplished. 48 5.0 CONCLUSIONS The HOPE Village Health Bank project appears to be a qualified success in Guatemala. At the Midterm, the project was struggling. A shake-up in senior project leadership had just taken place and the new manager was not yet on board. Health talks were of low quality, due in part to an ill-conceived design that called for village women to give the health talks, but also because of weakness in VHB’s implementation. Microfinance seemed to be under control, though the paperwork burden was quite time-consuming; but there were weaknesses being reported in the Highland loan portfolio. Problems of low staff morale and few training opportunities were voiced. Staff continuity in the Coastal region was a concern. At the Final, many of these areas have shown big changes. · Senior project leadership has been solid and uninterrupted during the intervening two years, and the project seems more holistically managed than heretofore. · Computerized financial record keeping has freed field staff time to allow them to assume responsibility for health talks that are now being carried out by HOPE técnicos instead of village women: substantial qualitative improvements are now possible. · The potential gains from this program shift have not yet been fully realized. It took the VHB project (and VHB Guatemala along with it) nine months to begin implementing Midterm health recommendations starting in March 2002. With additional time required to re-tool the curriculum, experiment with the new modules, train-up staff, etc., the new methodology is only a few months old. Though the sharpened focus is impressive, widespread changes in women’s behavior will be less than optimum as the project comes to the end of its grant financing. · Measuring impact in health has proven a frustrating process for VHB/G staff. In spite of the best of intentions, the VHB evaluation methodology has meant that quantitative gains that might have been occurring could not be captured in any meaningful way. In fact, because of weakness in health education, one suspects program-wide gains in health have not been achieved. · Personal interviews suggest that substantial individual gains have taken place—in some few women, world-shifting changes; in a number of others, significant improvement in family and personal health and self-esteem. At least half the interviewees suggested they get their PAP check-ups regularly—some attributing their willingness to do so directly to the project. There are several reports of VHB participants, diagnosed in the early stages of uterine cancer, who have had their uteruses removed and are now in good health because of a VHB talk. Though this is not widespread program impact, it certainly has had a life-and-death effect on these few.4 4 Graciously, the CARITAS manager said at the conclusion of his interview: “CARITAS wants to begin broadening our portfolio focus and get into Health; HOPE is the expert in Health in San Marcos.” 49 · In income generation, the project has worked with 2,684 women, seventy percent of whom (1,861) are still active, a solid achievement. Growth in the number of clients and the number of banks has been more than projected; growth in the portfolio disbursed and in the members’ accumulation of savings has been much less than planned. · Overall, improvements in income for over 830 long-term participants can be reasonably postulated in the 40-50% range, a substantial success for these women and for the project. Lesser gains improved income can be inferred for 1,358 four-cycle-or-more participants who have left the project. · In organizational terms, the project has accomplished its goal of keeping arrears under 5%; in the Highlands in fact, arrears are zero. Organizational sustainability has not yet been achieved—perhaps an overly ambitious goal in a start-up operation—but appears achievable in the next 12 months. · Many management issues identified in the Midterm have been addressed, demonstrating serious organizational commitment to learning and improving, and the project is clearly better run than two years ago. The issues of under-performing staff and staff rotation are themes likely requiring further HOPE attention, as is weighing the as-yet-unmeasured risk in Malacatán. 50 6.0 RECOMMENDATIONS As grant financing ends, the VHB Guatemala will come under more pressure to become financially independent. This pressure is probably an excellent thing at this stage of the evolution of the Guatemala portfolio. 6.1 Continued HOPE Center Support for VHB Operations in Guatemala The first recommendation is clear: Project HOPE should continue to finance VHB operation in Guatemala at the conclusion of the USAID and National Lottery Charities Board grants. The program model is an effective one that responds to key needs and should be continued. It is to be hoped that full operational sustainability can be achieved in the course of the next year, and HOPE HQ financial support should be provided to allow that goal to be achieved. 6.2 Improved Adult Education VHB Guatemala has benefited enormously from the hiring of an Adult Education specialist in Peru. This has not only taken the form of curriculum work described above; the AE person has also visited Guatemala twice during the year, once to deliver a workshop and once to conduct field visits. This expert assistance had been the driver of VHB Guatemala’s improvements in health education. These visits should be scheduled at least twice a year, and field travel and coaching should become an integral part of the visits. The seconding the Child Survival project educators part-time to strengthen VHB health education is another way to improve quality. These two people report that they “model” each curriculum module with every field staff during the validation stage. This practice should continue.5 Monitoring the quality of the health education needs to be strengthened, as current monitoring does not yet address quality adequately. It was reported that the VHB took a program decision in March 2003 to use a health evaluation format developed in Peru last year. There was no evidence of such a format being used in Guatemala. It is attached as Appendix B. It is strongly recommended VHB/G begin using this excellent format to enhance health talk supervision. One idea generated by the Project Manager during a de-briefing session was to use strong técnicos as coaches to show the rest of the field staff how first-class health education is carried out. This idea could help improve health talk quality with very little money. Another excellent technique to enhance staff skills is to create a feedback loop at the end of each session visited by a supervisor. This implies simply taking 10 or 15 minutes after a session has concluded for the supervisor and the técnico to think aloud what went well and what could be improved the next time. Such a feedback loop does not yet exist in VHB Guatemala but could be implemented easily and effectively. 5 Victoriana Salanic, an indigenous woman recently hired by VHB in Quetzaltenango, was herself the beneficiary of a village banking project some years ago and narrates freely that it changed her life: “I never knew how to save until then.” She wants to share her positive experience with VHB women like her, and is particularly appreciative of VHB’s focus on health. After Victoriana’s technical skills get up to speed, she could serve as a resource person for the rest of the VHB staff how to make the program more easily understandable to the indigenous population. 51 Under-performing staff who do not demonstrate necessary skills in health education and who are not able to improve their performance in a defined period, say six months, should be released. Last, a program of upgrading staff skills should begin.6 This should not consist of sporadic, one-off events, but become a training and skills-enhancement program. To do so may involve embarking on a new organizational philosophy: the project demanding the best performance from every staff, every session, every day; employees growing in career skills throughout long service with VHB. The evaluator can put VHB/G in contact with a Guatemalan consultant in this area if requested. 6.3 Improved Health Data Collection Regarding health data collection, the recommendation is quite simple after the difficulties of the last three years. Collect data on health indicators at the start of a woman’s career in VHB exactly as income and expenditure data are collected, and measure improvements in health at the close of the 4 th loan cycle exactly as improvements in income and expenditure are being investigated! Economic and household data show convincing improvements and the methodology has not proven overly burdensome; applying the same methodology to health data collection should produce the same results. The HQ Program Director’s last field trip report came to this conclusion, though Guatemala staff has not yet acted on the recommendation. It seems quite a feasible adjustment. 6.4 Improved Income Data Collection Regarding income data collection, the recommendation is also simple. Collect data on income indicators at the start of a woman’s career in VHB and measure improvements at the close of the 4th loan cycle on all participants—not only on a sample. This will eliminate much of the guess work of estimating improved income for long-term participants. 6.5 Improved Loan-making: Business Plans Given that gains in health education have yet to be consolidated, the following recommendation is articulated as something to be considered over the next several years, not necessarily an idea for immediate implementation. For VHB to have more impact on the lives of its clients, a new direction could be to get clients to begin moving into activities that are more profitable. Until now, as long as the VHB group proposed the loan and it did not seem too risky, the project has not much focused on the business analysis behind the loan application. The thinking in this recommendation is that within current time constraints, field staff could begin to study the underlying soundness of individual plans more. With the project experimenting with a modicum of business focus—and perhaps adding an occasional module on basic business training—women may be able to perceive more profitable opportunities in their environments. A way to do this is for VHB staff to give more scrutiny to the fundamental cost-effectiveness of individual bank loan requests. So as not to over-burden field staff with this role on top of their health education responsibilities, such analysis should be done on a limited basis: perhaps on one or two of the newer members joining a successful bank, or perhaps mostly on 1st and 2nd loan cycle clients. 6 recommended rather elliptically in the Midterm as “more management attention to Human Resources” 52 If this concept goes forward, the project may be in a position to begin addressing high dropouts indirectly. If businesses become more successful, more women should be interested in staying in the project longer. At the same time, project management could begin to conduct ‘exit interviews’ with departing bank members to understand better the various factors that influence a women’s decision to withdraw. Knowing more about these factors might enable the project to design other interventions. The target could be to reduce the dropout rate further, perhaps no more than 25% at the end of 4th cycle. Trying to reduce dropout rates could become a systemic goal of follow-on VHB programming. In addition, a suggestion that arose in one VHB meeting is that there may be VHB members who are sufficiently entrepreneurial that they themselves might be able to become ‘visiting experts’ at nearby banks talking about this-or-that aspect of their business. 6.6 Continued Search for Self-sufficiency Cost-reductions directed to rapid achievement of operational self-sufficiency should begin. Changes worth considering seem to be the following: · The project may be overstaffed at the field level. A ratio of 300:1 suggests at least one field person could be released. At the same time, the two empty Malacatán positions must be quickly filled. A suggestion was also discussed with the Project Manager of considering ‘span of control’ issues to reduce to one supervisor rather than the (current) two or (planned) three. Client loads should be gradually increased toward the proposed 300-plus target, up from the current 265:1. Thereafter, staff should be brought on only if the portfolio expands, at an “organic” pace. · Paradoxically, it appears that raising VHB salaries should be studied—in spite of the financial implications. The departure of one or two high performers after the Midterm was noted by the evaluator, and HOPE Guatemala needs to retain qualified staff if this project is to succeed; current staff rotation suggests this is an issue. The topic requires careful study, but the closure of grant funding (Aug. ’03) may create an opportunity for annual contracts to be renegotiated. At that point, salaries could be adjusted to reflect the quality of performance of VHB personnel. · If it is true that VHB Guatemala does not yet support a share of HOPE Guatemala Head Office costs (beyond some portion of the Country Director’s salary), it is appropriate that this begin. Though this will push the achievement of operational sustainability back, it is an important principle in NGO management that no project should be given a ‘free ride.’ While the HOPE Guatemala portfolio still has a variety of donors and the financial burden of each project is light, it is time for VHB to begin assuming this obligation. 53 APPENDIX F FINAL EVALUATION Village Health Banking Project-Peru 1.0 INTRODUCTION 1.1 Origins Of the Current Project The People-to-People Health Foundation, Project HOPE, is a non-profit, non-sectarian, a-political organization with a mission “Helping People Help Themselves.” Origins of the current Village Health Bank (VHB) project lie in a 1992-1997 USAID Cooperative Agreement to allow HOPE to establish the program in Honduras and Ecuador. For FY 1998-2003, additional funding was requested from USAID to start a VHB project in Peru. A Midterm evaluation was conducted in this project in August 2001; this document is Part Three (Peru) of the Final evaluation of that project. 1.2 Scope Of Work The Scope of Work tasked the consultant to conduct an evaluation of the Village Health Bank activities in Peru (and also Malawi and Guatemala); assess HOPE Headquarters progress with the grant; and develop recommendations to strengthen continuing execution of the VHB programs. Main topics were: ß Discuss if the hypotheses under-pinning the program model were sound. ß Discuss progress toward achievement of major objective; major successes, challenges and constraints; and unintended impacts. ß Identify changes made during implementation. ß Discuss impact of the program on target populations. ß Discuss whether adequate financial control systems exist. ß Discuss whether the M&E system supplies accurate, timely performance data. ß Discuss program management including an adequate number of staff with relevant expertise and supervisory systems. ß Discuss Lessons Learned and Recommendations. 1.3 Timing, Methodology, and Evaluator The evaluation took place in July 2003 beginning with two days of orientation with the Director￾Income Generating in HOPE Center. Field travel to Peru took place from July 6-11, 2003. During the visit, program discussions took place with the project manager followed by three days of field travel to VHBs. A presentation of preliminary findings was made to the Country Director, the Adult Education Consultant and the VHB Project Manager on July 11. The methodology involved the following: in-depth discussions with the project manager; field travel including direct observation of village health talks and loan operations; and semi-structured interviews with randomly selected VHB members. Three of four Tarapoto field staff plus the Health Coordinator were observed as they engaged in project promotion. Eight banks were visited for approximately two hours each; banks were selected by the Project Manager based on geographical proximity and those which had already scheduled meetings during the three field days available. Post-VHB event conversations took place with 28 women at the eight banks—all of them 54 selected at random as, as far as is known, only one of them was a (more active) VHB board member. Informational interviews were also held with the Managers of the PRISMA (USAID￾funded) micro enterprise project, CEPCO (the Centro para Estudios de Promoción Comunal del Oriente) and Fundacón Manuela Ramos, three NGOs working alongside HOPE in microfinance in the Department of San Martín. The evaluation was carried out by a development practitioner who has substantial experience evaluating programs who was also the external evaluator of the VHB Midterm evaluation in September 2001. 1.4 Structure of report The document is structured in sections. Section 1, Introduction, has now concluded. Section 2 presents Findings in Health, Section 3, Findings in Microfinance. Section 4 discusses several management issues. Section 5 presents Conclusions and Section 6, Recommendations. 55 2.0 FINDINGS in HEALTH The Project HOPE model of Village Health Banks provides credit and health education to women, building on the village banking method of solidarity group credit. Health education is meant to enhance the impact of gains in income and is directed to the relationship between low incomes and poor health influenced by poor housing and hygiene, inadequate access to health care, and limited educational opportunities. It also builds on the well-documented fact that increases in women’s income get spent more on health and nutrition items than increase in most men’s income. As a health-focused institution, Project HOPE is particularly concerned with the project’s health impacts; while funding comes from the PVO Office of Matching Grants that is primarily concerned with the project’s economic outcomes. This section discusses the project’s health component, the next chapter the project’s micro-finance component. To be discussed are: structure of the educational curriculum; staff skills; supervision in health; a question of ‘hyperactivity’; discussion of extremely high rates of client rotation and its effect on accomplishment of quantifiable gains in health; qualitative gains in health; and overall summary comments. 2.1 Structure, Methodology and Content of Health Education Modules The issue of the VHB health education component formed a large part of the discussion of the Midterm evaluation. To a quite gratifying extent, almost all of the Midterm recommendations in Health and Training have taken place. Direct health education delivery Of fundamental importance was the recommendation of the Midterm evaluation that the VHB project should move from the idea that village women could be trained to the point of delivering high quality health talks. The Midterm evaluation showed that overworked village women were largely incapable of delivering the high quality messages the project needed: the proverbial Training-of-(village)-Trainers was not working. Instead, it was recommended that HOPE convert the role of bank promoters into health promoter, ‘direct health education delivery’ by HOPE. To the evaluator’s pleasure, this recommendation has been fully implemented. Three months after the Midterm (December 2001), HOPE Peru identified an Adult Education (AE) consultant who began to work on re-conceptualizing the health education methodology. This individual facilitated an all￾country VHB seminar in March 2002 that ratified a major conceptual shift in the VHB project. As of June 2002, she became a full-time consultant to HOPE for the entire Latin American program with special emphasis on VHB. Fifteen months later, VHB field staff in Peru are now delivering health talks in every bank meeting. Training modules have been developed to guide non-health staff how to cover key messages in the subject. A 180? change in Health Education methodology has taken place and project field staff (asesor/ asesora), not village women, are leading the discussions. This is a substantial step in the right direction. Revised modules As part of this change, the evaluator was pleased to note that a significant realignment in the health curriculum has also taken place. The AE expert is in the process of developing new modules to guide the village health talks. Under her guidance over months of design and pre-testing, including 56 two visits to Peru field staff , a major shift away from ‘didactic’ talks to participatory talks has taken place. Substantial emphasis is given to games and other adult education techniques such that the sessions have become quite enjoyable; virtually all women observed in the talks were seen to be enjoying themselves, laughing, joking, taking pleasure in the module, and generally having fun. This is an important gain in Adult Education. The modules are broken into four major themes: healthy women, women and children, healthy families, and healthy communities, each having multiple topics per module. Currently one module has been completed and is in implementation, another in the testing stage, two others have yet to begin. The structure of the modules is Adult Education based: introduction and review of past session, development of the theme, review, conclusion and segue to the next session. “Promises” are encouraged from participant at the close of each session: tasks for them to accomplish before the next meeting related to themes discussed—an excellent innovation. The methodology is highly participatory and learner-centered. Moreover, the length of the health talk has been extended from ½ hour during the Midterm evaluation to a full hour now. Tellingly, while many women in the Midterm complained of having to spend an extra half-hour at that time, none of the interviewees complained of having to spend an hour now. Clearly the quality of the talks has gone up so much that women are pleased to be attending. The evaluator enthusiastically endorses these changes. Visual Aides Field staff is showing enormous creativity in preparing the props needed for these sessions. Flip charts hung from clotheslines or taped to the wall; cutouts and drawings of fruits and vegetables; homemade ‘fishponds’ and ‘fishing rods’; balloons with health messages; ‘treasure hunts’ for slogan cards hidden in the room; health sentences to be pasted on the chart; paper houses with swinging doors where messages are ‘stored’; chicken cut-outs and grains of corn used for the participants to evaluate quality of the lesson; story-telling case studies and more: the variety of the visual aides and the sheer creativity behind them are inexhaustible—and a veritable delight to observe. There is a clearly observable synergism between the AE curriculum developer in Lima and the Health Educator on the ground in Tarapoto that is creating ‘homemade’ AE materials as good as this observer has ever seen. Astonishingly, all of this has been carried out with almost no financial outlay—simply ingenuity and creativity. Bravo! 2.2 Staff Skills/ Quality of Talks However innovative and creative a new methodology, it is no good unless the staff using it are trained in implementing it: for good AE, both personal and human skills are necessary, along with control of technical content. Based on the evaluator’s experience and extensive note-taking during each of seven talks observed, the quality of the events is summarized in the following table. Table 1: Evaluator Assessment of Peru Health Talks VHB Location Human skills Technical content Overall Comment Santa Inés A A A Unión Solidaria A A A Las Orquidias A A B+ Facilitator physically sick Sagrado Corazon A A A 57 Forteleza de Diós B+ B B+ Difficult theme; first exposure Diós Amanece para Todas A A A Las Perseverantes B C C+ Lost control of time and content Overall: three “A’s”; two “B+s”; one “C”. This is substantially better than either of the other two countries visited during this final evaluation. 2.3 Supervision of Health Talks Not only are staff skills important, supervision and continuous coaching are required. VHB-Peru is well advanced in this area also. As an outgrowth of the Midterm evaluation, over the last year the AE Advisor and the Health Educator have been working on the design of a health talk ‘check list’ that measures the technical skills of the facilitator as well as the technical content. The format has four rating sections, each of them generating a numerical score: quality of steps and procedures; knowledge, ability and behavior of the facilitator; responses of the women during the session (including behavioral observations as well as post-event interview); quality of the visual aides and overall quality of the event. Evaluating the quality of health talks is a complex subject and this observer has been struggling with how to systematize the process for years. Experimenting with the Peru format in two health talks during the week, the evaluator comes to the conclusion that the Peru VHB supervisory mechanism is as good as he has ever seen. Indeed, a recommendation has been made to other VHB programs to adopt this format. The Health Educator uses this format whenever she supervises field staff. She was able to show the evaluator a file full of these forms, and talked about the numerical improvement in staff scores over the last six or nine months. This is truly great stuff. (One surprise is that this format was supposedly made VHB program policy in March but was not in use in the other two countries, in spite of those other countries’ struggling with this issue. But this is no reflection on the VHB-Peru project.) 2.4 A Question of ‘Hyperactivity?’ One note of caution in the wonderful changes in AE that have taken place is that the sessions have gone from the extreme of being theoretical and boring ‘expositions’ to a rapid, unremitting series of ice-breakers, games, and AE dynamics. Women are moved from one exciting and fun-filled simulation into the next, with hardly a minute between to reflect on or interiorize the experience. In truth, in VHB- Guatemala, this was a bigger problem than in VHB-Peru. In Peru, staff has been able to maintain the delicate balance between dynamics and content and should be congratulated for doing so. It is an balance that needs to be continually kept in mind however, even in Peru, for which it is noted here. In this rush from one dynamic to the next, the overall learning objective of the session needs to stay in the forefront. The module guideline spells out clearly what the learning objective is, so the problem is not that the goal of the session is not articulated. The problem seems to lie more in losing sight of the forest amid all the tree activities. 58 A related concept is that each session should be focused on actual learning, not theory. In one session on balanced nutrition, the module called for classifying food groups and various dynamics were used to get women familiar with the classification. The purpose of the knowledge—getting women to make changes in the family meals—was never mentioned or even implied. Thus, learning gets overtaken by theory. Recommended by the Midterm, most sessions should become focused on behavioral change. What is it that we want women to do differently as a result of this talk? How will we measure the effectiveness of the session in terms of how people change the way they live? Though this is a complicated theme, VHB-Peru is at the stage where it can and should begin paying more attention to the question. Apparently there is an all-VHB seminar scheduled for August that will address this theme, and VHB-Peru staff should take a high-profile in exploring the topic. 2.5 Quantifiable Gains in Health For three years, the project has struggled to track and document improved health. Baseline heath data were collected 7 months after the project started in April 2000 (N=97). Another effort was undertaken in February ‘01 (N=164.) In Sept. ‘02, a Lot Quality Assurance Sampling (LQAS) effort was carried out (N=124), complemented by a simultaneous impact study (N=67). In September and December 2002, two qualitative studies were carried out. In March ’03, an impact study was carried out (N= 56). This is a large list of activities. In all cases a detailed survey report was prepared. All of this activity took a great deal of time and staff commitment and energy. Unfortunately a methodological error (common in the three VHB countries) took place in most of these efforts, the ‘co-mingling’ of new VHB members with older ones. During the course of this study, a detailed comparison was made of the results of these various studies. Results of that comparison showed at least three things. First, one would have expected some commonality between baseline conditions with “before” project women; in most items there was little correlation. Next, the LQAS (Lot Quality Assurance Selection) data were strange, either because of statistical anomalies in the lot selection or for other reasons: in at least two indices, LQAS data appeared frankly beyond belief. Third, one would have hoped that 90 women who had participated in the program’s health talks for at least four cycles would show some improvements in health scores; few of the indices showed improvement compared to baseline; few showed improvement compared to data on newly entering women taken at the same time. In sum, because of the co-mingling of new and older women in most of these studies; because of a devastating rate of attrition to be discussed momentarily; and in spite of considerable staff effort in carrying out these studies over time, there are little quantitative data to suggest program-wide improvements in health. Recognizing these weaknesses several months ago, VHB-Peru came to a significant conclusion. After the struggle of the previous three years, the change is quite simple: collect health data at the start of a women’s career in VHB just as income and expenditure data are collected; and measure improvements in health at the close of the 4th loan cycle just as improvements in income and expenditure are measured. In fact, the HQ Program Director’s recent trip reports to other countries came to this conclusion at about the same time. VHB-Peru is the only operation to have reached the conclusion on its own. 2.6 Qualitative Gains in Health 59 Beyond quantitative gains, qualitative gains merit discussion. If the project has not had a wide￾spread impact, it has had an impact on individual women. Given that adjustments to the schedule permitted interviewing a fairly large number of women (N=28), it is possible to use the stories of these women to illustrate. a) Milagros Gutiérrez Seminario (in from the beginning of her bank, Vista Alegre, 2 years ago) works a juice kiosk in Nueva Cajamarca. She sees herself as a “leader-ess” and says the health talks are “the best thing that could possibly happen” for the women, because so much trouble is caused by not knowing. She remembered nearly every talk ever given in detail and has herself applied what she has learned to her life. b) Leonida García Pinchi (began in the first cycle of the bank, but took 2 cycles off , for a total of 6 cycles) is “enchanted” with the health talks. She commented that the sessions allow her to relax and have fun, and “step out of herself” and her worries for a little while. She loves the dynamics. c) Sarita Pasmina Ruiz (1st cycle) remembered many of the talk topics and when asked if she applied them to her day to day life she responded, “of course! they ask us to make promises, and I keep them!” d) Madrona Díaz Montilla (3 years in the program) finds the health talks to be an opportunity for relaxation and laughter; moreover, she takes everything she learns from the health talks and shares them with her neighbors and friends in the market, and any talks she is too old to apply to her own life she goes home and teaches her grown children about so they can raise their children better. e) Juana Villaro Deciurlizza says she is more conscientious about covering the food at home and hygiene matters to her, and she gets her PAP smear check ups now, where before she was afraid to. It bears repeating that these women were selected for interview by the evaluator at random and a deliberate effort was made not to interview any member of the VHB Board of Directors, (thinking that Board women might not be representative of the wider group.) With these case stories out of a random sample of 28, there is some validity to the proposition that in spite of relatively recent changes in the health methodology; in spite of the lack of demonstrated program-wide quantitative impact; and in spite of high member turnover (to be discussed next), the project has managed to have substantial impact on health for a limited number of women. Building on this idea, one can posit with confidence that significant gains in health have taken place for 205 women currently active in VHB who have participated in health talks during four or more recent loan cycles. Lesser gains in health are inferred for FY99-FY02 clients who remained in the project four cycles but who no longer participate in the project, 946 in total. 2.7 High Drop-out of Clients One overwhelming characteristic of the Peru VHB is that almost every bank suffers from a constant departure of old clients and the influx of new ones. Data also to be presented in the micro-finance section show that more than eighty percent of women are new to the project. During the Mid-term evaluation, the HQ Program Director commented that a high drop-out rate is not unusual in the micro-finance sector, but the Peru numbers are devastating: active clients are 1,199 of 5,084, 76% dropout. 60 Implications for Adult Education are serious since even if the project were delivering the most vibrant and impact-filled Adult Education topics in the world—which in large part comes close to being true— new entrants won’t have heard the talks. In addition, with attrition rates this high, by the end of the 4th cycle the number of clients who have been exposed to the full set of health talks is very small indeed. Clients exposed to only one or two health talks are quite numerous. Implications in health are serious indeed. 2.8 Conceptual Issues Before concluding this section on health, it seems appropriate to offer brief reflections on the VHB program model. Succinctly, throughout the developing world, millions of women are unable to receive services from their over-worked and under-funded Ministries of Health. When these people fall sick, there is nowhere to go that does not require sometimes hours (or half-days) of travel. Moreover, preventative public health activities are all but unheard of. What the VHB model offers is a way to fund well-trained primary health promoters outside of, and complementary to, the Ministries of Health to promote women’s ‘demand-driven’ search for better health services. When financial sustainability is achieved, the model offers an effective way for women who would otherwise not receive health education to learn how to improve their lives. At the same time, it offers marginalized rural and peri-urban women an opportunity to improve their household incomes (and the health of their children), as well as to grow in self-esteem. This evaluator finds the model compelling. In a place like Peru where infant and maternal mortality is still high, the model should be made to work: it is a way thousands of women can be served. 2.9 Summary Comments VHB-Peru accepted the recommendation of the Mid-term evaluation and changed its education methodology from village trainers to direct delivery by project staff, a quite positive step. Redesigned modules emphasize adult education (Freire) principles to extraordinarily good effect. Most field staff are delivering some of the highest quality health talks this evaluator has ever seen: VHB-Peru has found field staff with the right disposition to carry out direct delivery and the new curriculum adequately prepares them to deliver talks of high technical quality. Unfortunately, the constant in-flow and outgo of VHB participants has proven a significant obstacle to project goal accomplishment. Also the new methodology has come somewhat late in the project so that long￾term impact of this change will not be great. It is most likely that program-wide improvements in health have not taken place. However, those few women who have stayed with the project over the span of two or three years have likely changed some key behaviors—one speculates, as much due to the personal example and enthusiasm of VHB field workers as due to the dynamism of health talks themselves. The number of talks given in FY03 under this new methodology is 850 as of June 30, 1,750 in the last two years. If the high drop-out problem could be solved, the project’s impact in health could be dramatic. 61 3.0 FINDINGS in MICROFINANCE Of many indicators in micro-finance that could be tracked based on the project’s extensive monitoring system, those that will be discussed are growth in the number of banks and membership, growth in total loans distributed, arrears, project impact, and operational sustainability. The narrative begins however by picking up on the theme of high drop-out rates. 3.1 Member Drop-out The total number of dropouts is the difference between total current borrowers vs. cumulative borrowers, 2,911 of 5,084 respectively, total 42.7%. In fact, the figure may be much worse. Calculating the difference between 1,199 active client (6/30/03) and 1,712 current borrowers who are not active, the Project Manager estimates that no more than 30% of the latter group will return to the project. This would make the effective drop-out number at least 3,371/5,084, over 66%, a devastating figure. If those who return are less than 30%, the percentage will be even higher. In some months, individual drop-out rates at one bank or another can be as high as 30% a month. This drop-out rate has affected every one of the indicators reported below. Discussions with colleague organizations (to be reported below) suggest VHB-Peru is not the only project to be affected. Project data as of July 31 show 1,071 active members (128, or 10%, falling out in July alone.) Two hundred and five of them have remained in the project for four loan cycles or more, 19% of current active clients, 4% of total cumulative borrowers. 3.2 Expansion into Lima Management has recently taken the decision to expand into the peri-urban area of Lima. At this writing, twelve banks are in operation in Lima though there is hope the number will grow. Similar to a decision taken in Malawi and Guatemala, management believes portfolio growth may be achievable if the project works in areas where there is higher demand for loans and higher absorptive capacity of VHB clients. 3.3 Growth in Number of Banks The number of project banks has grown steadily throughout the life of the project. While the total at time of the evaluation was 185 total banks formed, only 70 are active. Taking the latter as a more accurate figure, it represents 87% (effective) achievement of the project goal of 80. 3.4 Growth of Membership VHB membership has also grown throughout the life of the project. Development of the numerical goal was based on the experience in Ecuador which proved an overly optimistic target. Cumulative membership is 5,084 (6/30/03) compared to a program goal of 2,160. As above, if adjustments are made for current active members (1,199) and possible returning members (30% X 1712=510) likely effective membership will not be more than 1,710, 79% of target. 3.5 Cumulative Loan Distribution The chart reflects the cumulative loan distributions in hard currency at S/ 3.35: $1.00 (7/15/03). 62 0 200 400 600 800 1000 1200 1400 Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Cummulative Disbursements in $000 Total value is $ 1.378 million compared to a target of $ 1.747 million, 79% of goal. 3.6 Current Value of Loan Portfolio In the chart below, the larger bars reflect the end-of-fiscal-year values of the loan portfolio in hard currency at S/ 3.35: $1.00 (7/15/03). Current value of the portfolio is $177,688, of a target of $405,400, 44% of goal. 0 50 100 150 200 Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Loan Portfolio & Member Savings in $000 Portfolio Savings 3.7 Total Member Savings VHB members are required to save 10% of their loan amounts. This is an appropriate strategy that promotes that at the end of the loan cycle a woman will have something left for her effort. Withdrawals from savings are permitted when a member is leaving the VHB, in case of family emergency, or to pay off a balance due on the loan. The smaller bars in the chart above shows the growth of savings up to the time of the evaluation in hard currency. Current savings is $17,290; the goal was $262,170, representing 6.6% achievement of goal. On this index, one can see clearly the effects of high drop-out rates. 3.8 Inactive Banks When a bank does not pay 100% of its loan within 90 days, it enters a “inactive” status. This means the VHB no longer receives health education talks, and that further contacts between HOPE and VHB are suspended until the loan balance is paid. At the time of the evaluation, the number of inactive banks in VHB-Peru was 115 of 185, 62% of the total. 3.9 Improved Income 63 At the project level, VHB has established an effective tracking system to measure improvements in income by collecting economic and household data on each entrant to the VHB during the 1st cycle. For those who are still active at the close of the 3rd loan cycle, the survey is repeated on a sample basis, and the changes are measured. The following table represents those results in Peru as reported in the Year Three and Year Four Annual Reports, with improvement in household expenditures being taken as a proxy for increased income. Table 3: Quantitative Improvements in Household Income Criterion Year 3 (N=87) Year 4 (N=179) Percentage increase in household expenditure 7% 11% Percentage increase in households incomes 18% 5% Percentage increase households reporting better incomes (37%/57%) 35% (38%/46%) 17% These data have been collected in a methodologically sound manner; however they do not show a clear a trend. For those who remain in the program over several years, 205 women at the close of FY03, improved income may be in the 10%-15% range. In addition, the evaluator infers some gains in income for those FY99-FY01 clients who remained in the project four cycles but who no longer participate in the project, another 946 over the life of the project. This inference grows out of a belief that women who have been only somewhat successful leave the project, while fully successful women continue in the project year after year. Thus, income gains for these women have been estimated less than for the 205. Quantification of this gain has not been attempted. 3.10 Non-quantifiable Impact Discussions with interviewees selected at random suggested non-quantifiable impacts. The following personal histories are the most salient: a) Elsida Rivera (3 rd cycle), a start-from-nothing entrepreneur, sells food to farmers in outlying homesteads. With the extra income she makes using the VHB loan, she is paying tuition for her daughter to attend university. b) Madrona Díaz Montilla (3 years in the program) was having serious financial difficulties raising her family after her husband’s death. Now, as she says, “thanks to the little bank, I’m fine.” She observed that before there was no possibility to save as all extra money went into the family. She now has S./ 350 ($100) in savings. c) Juana Villaro Deciurlizza (10th cycle, bank secretary) changes her sales business slightly to fit the conditions, and now has approximately. S./ 180 ($54) in savings. d) Sonya Savedra (5th cycle) makes women’s underclothes, using the VHB loan to purchase the raw materials. Her savings is S./ 500, nearly US$ 150, half again as much as her loan. e) Cristina Campana Avalo (6th cycle) uses the VHB loan to provide major maintenance to the motorized rickshaw she and her husband run. She has done sufficiently well at this business that she has hopes of buying a second one in the next year. f) Felicita Regaderos Frias (4th cycle) is a fish-seller in the municipal market of Nueva Cajamarca, expanding into fish marketing three years ago after having done it on a smaller scale from her house. She has almost $100 in savings and has used the loan to buy a freezer and other equipment for her fish business. 64 During these interviews, several interviewees also narrated that the VHB has allowed several of their friends and acquaintances to escape the clutches of money lenders who charge 20% interest a month. Though the numbers may not be great, the economic impact on the lives of these women is surely great. 3.11 Arrears 30 Days Past Due Project documentation and micro-finance literature suggest one important way to measure the soundness of a loan portfolio is to track arrears more than 30 days past due. The project goal was “30 days arrears under 5%”; status at time of evaluation is 29%. Two years ago, the project hired a part-time lawyer to begin taking to court clients in arrears to try to lower this high arrears rate, but the strategy has not proven overly successful, in part because of the part-time nature of the lawyer￾VHB relationship and in part because the individual has been sick some of the time. Twenty-nine percent is an alarming and unsupportable rate of arrears. 3.12 Operational Sustainability 100% operational sustainability (all costs covered by project income) was another important goal of the project. Current operational sustainability at time of the evaluation is 45%. After five years, this is quite low. 3.13 Comparative experiences with operational sustainability To get an idea of the Tarapoto micro-finance context, interviews were held with three colleague micro-finance NGOs: the Centro de Estudios de Promoción Comunal del Oriente, CEPCO; the (USAID) PRISMA project; and the Fundación Manuela Ramos. The experiences of these entities—mostly colleagues, sometimes competitors—may provide insight to the difficulties of achieving operational sustainability in rural Peru. CEPCO is a rural micro-financing entity that began in 1998 with a grant from DFID-UK (for a rural portfolio) and Miséreor-Germany (for an urban portfolio.) It started with a S./ 1 million grant (US$ 300 K) and a two-year subsidy of operational costs. Phase I of its operations resulted in a rate of arrears of 20%; in Phase II, it has been able to reduce it to 7%- to-12%. It serves 300 urban clients, 75% women, and 400 rural ones, 25% women; many of the latter loans are tied to agricultural production linked to marketing. Several years ago in order to reduce costs, CEPCO took the decision to reduce its operations from four zones to one in an attempt to become more competitive. The urban portion of the portfolio is covering its costs, the rural, not yet. PRISMA is a USAID-financed project that began as a national primary health care project that spun off the micro-credit sector in 1998. At that time, the portfolio was characterized by high desertion rates and arrears of 40% or more. A drastic restructuring of the portfolio was required, and millions of Soles obligations had to be written off.7 It currently serves 1,420 loan holders, of a total of 2,200 clients. The Tarapoto interviewee was not aware of the current operational sustainability figure but thought it might be about 60%; he hopes 100% sustainability might be achieved in the next 15 months. 7 Newspaper reports the next day suggested another restructuring may be in process. 65 The Manuela Ramos Foundation is a Peruvian NGO with national coverage. It began credit operations in Tarapoto in July 1998. It serves 2,600 women clients in 130 banks. The Manuela Ramos Regional Director finds loan operations in Tarapoto complicated by a number of factors: high desertion rates (“maybe eight of my 130 banks have their original membership intact”); one￾third new members in any given bank; the reality that if two or three good members leave, the bank crumbles; and any missteps at the beginning means the portfolio is in serious trouble. Manuela Ramos is the only one of the four NGOs that has achieved operational sustainability. Other comments from these knowledgeable informants were interesting: · Over-indebtedness: many clients owe to more than one lender. · The sector suffers from heavy competition from the (Government of Peru) Caja Municipal and the Banco de Fomento that have entered with easy credit. · Increased rates of desertion occur with the entrance of the Cajas Municipales-- credit is pretty well available in this part of rural Peru. · In fact, some feel NGO credit cannot compete (or co-exist) with government-sponsored credit; one NGO has deliberately stayed out of areas where government credit is available and carefully chosen areas where there is little or no competition. · Peru does not have a ‘culture of repayments;’ indeed, some politicians actively promote non-payment for political gain. · A widespread culture of corruption exists in Peru and exposure in the press of high political corruption creates a hostile climate for personal fiscal integrity. · In the Tarapoto area, an aggressive campaign of legal action is necessary to convince clients of the seriousness of repayments. Having the institutional image of an NGO makes this more difficult. · The biggest micro-financing weakness in Tarapoto is lack of trained personnel; the methodology (and culture) of communal banks is unknown. · Several NGOs report having to move into solidarity groups (4 or 5 members) and individual loans whether they wanted to or not because of the slow rhythm of loan activities in the region. · One NGO attempted to start a new office in 2003 and was forced by lack of demand to close it a month later (Yurimaguas where VHB-Peru was also unsuccessful). Selected indicators from those informal interviews are presented in the table below. Table 4: Selected Comparative Indicators of Colleague NGOs in Micro-finance Index VHB-Peru CEPCO PRISMA M. Ramos Active Clients 1,199 700 1,420 2,600 Portfolio size $178,000 $298,000 $448,000 $291,000 Average loan size $US $173 $319 $315 $197 Client/ staff ratio 1 1732 319 315 197 Reported desertion rate 66% 10% 35% 33% Reported Arrears 28% 7-12% 14% 0.15%3 Operational Sustainability 4 45% ~76% ~60% 126% 1 Active clients/field personnel + supervisors; lower figure = less efficient. 2 Excluding Lima start-up. 3 Previously 8% but in FY00 eight banks were written off. 4 Tarapoto operations only. 66 Key comparisons seem to the following: · VHB-Peru’s active clients are the second-smallest of the group, second only to CEPCO which has an average loan almost double that of VHB. · VHB’s average loan size is the smallest of the group. · VHB’s client to staff ratio is the least efficient of the group—but not all that distant from Manuela Ramos. · The last three VHB indicators are the “show-stoppers:” drop-out rates, arrears and operational sustainability far worse than colleague organizations. The table suggests that micro-credit in Tarapoto does seem a difficult challenge, though not an insurmountable one. 3.14 Viability of Agricultural Loans One field visit was made to a rural community where VHB is financing subsistence agricultural production, Chiricayu. Because this was an out-of-schedule visit and because clients interviewed were native Quechua speakers, the quality of the information was not as high as some other. Nevertheless, the following finding is worth mention. In this writer’s experience, rural, stand-alone financing of basic grain production is not usually an economically viable proposition. Experience with the Green Revolution demonstrates that without a series of modern inputs: improved seed, chemical fertilizers, modern pesticides, modern storage, and careful marketing, it is very unusual that basic grain production can generate a return sufficient to amortize a 36% loan. The evaluator was informed that VHB entered this village to complement the on-going HOPE Child Survival program—and in response to villager complaints that they should be able to receive credit if another HOPE project (VHB) was offering it elsewhere. He was also informed that VHB activities in Chiricayu are linked to a crop-diversification study being conducted by an agricultural research university, a valuable collaboration. In spite of these facts, the evaluator is concerned the economic profitability of basic grain production has not been adequately studied by VHB. If one imputes a cost to labor (which subsistence farmers usually do not) there is a real possibility that instead of benefiting farmers with this credit, a subtle de￾capitalization is taking place. Project staff were sensitive to the observation. A cost-benefit analysis by an agricultural economist seconded from a nearby university could shed further light on this concern. 3.15 Membership in Tarapoto Micro-finance Network Two years ago, VHB-Peru participated in the formation of a network of micro-finance NGOs working in creating a Central de Riesgos, a registry of loan defaulters in the Tarapoto area. The agreement among these members is that no loan applicant who owes money to another member will be accepted into a second micro-finance activity until the first loan is repaid. This is a good initiative, and represents a private sector response to the debt-forgiveness attitude prevalent among Government of Peru lenders. VHB staff also participate actively in a roundtable of micro-credit lenders in the Department. 3.16 Overall Assessment 67 It appears VHB-Peru is in serious financial jeopardy. Uncontrollably high drop-out rates as well as high arrears mean that the project is in dire financial shape. As in the other VHB countries, micro￾finance activities are carefully documented and monitored; but in spite of this the project has not found a way to stem the hemorrhaging of clients. While growth in the number of banks and total loan disbursement was accomplished to approximately 80% of target, impact-oriented goal accomplishment is very low. The current value of the portfolio at 44% of goal, member savings at 6.6% of goal. The number of inactive banks compared to active banks, and member drop-out compared to overall membership are both in the 65%-75% range, seriously unsustainable. Increases in some women’s income may be a ray of sunshine in this bleak picture. There is evidence to suggest that some gains in income have accrued to women who have participated long￾term in the project, perhaps in the 10%-to-15% range. Unfortunately this cadre of women as of July 31 is 205 people, with another 946 women who have let the project after a year showing some, but likely less, improvement in income. Taking both these groups together represents only 35% of the goal of 3,300. Current operational sustainability is under 50% and to this writer it does not appear achievable in the foreseeable future. Experience from colleague organizations working in Tarapoto leads one to hypothesize that achieving operational sustainability in five years in a start-from-scratch operation was not a realistic goal with start-up capital of only $250,000. More seriously, as long as VHB￾Peru continues to “churn” clients: creating new banks with a few left-over participants from banks that have gone into arrears or signing up new clients with little experience with or commitment to solidarity loans, operational sustainability will not be achieved for quite some time. To be discussed shortly, the move to Lima does not seem to offer the possibility of quick payoff as in other VHB countries. 68 4.0 MANAGEMENT ISSUES Management issues with a bearing on the accomplishment of project goals are expenditures, rotation of staff, staff skills, morale, the ratio of clients to staff, one new program initiative, and improved client focus. 4.1 Budget vs. Expenditures The following summary financial information is provided. Verification of these data was beyond the Scope of Work, thus these figures were provided by HOPE as of July 10, 2003. Project expenditures for the life of the project will be approximately $1.097 million, of an original budget of $1.149 million, 96% of budget. This is broken down as follows. [John: please check] · Headquarters support is $25 K, 2%. · Local personnel costs are approximately $349 K, 31%. · Grant financing is $ 250 K, 23%. · Other direct operating costs are $250 K, 23%. · Indirect cost is $224 K, 21%. A review of these figures suggests the project is in line with NGO industry standards. 4.2 Expansion: Yurimaguas and Lima In an effort to expand, in October 2000 the project took a decision to open a sub-office in Yurimaguas. Over the next year, a total of 40 VHBs were opened and 688 clients were provided loans. Unfortunately, the credit experience was never very satisfactory here, and in December 2002 the project closed the Yurimaguas operation. At this writing, only sporadic repayment of outstanding obligations is taking place. In April 2003, a decision was taken to enter the peri-uban area of Lima. Similar to decisions taken in other VHB countries, the expectation is that because of a higher economic potential of Lima clients and a greater number of clients in a smaller geographic concentration, the project could become more efficient. At the time of the evaluation, a total of 12 banks have been formed with total membership of 159. Though the decision to enter Lima was researched and a risk-analysis was prepared, the Project Manager indicated the Lima market has proven more complicated, and bank formation requires more time, than anticipated. Other lessons from the Yurimaguas experience should be thought through also as the Lima operations expands. 4.3 Stability of staff The Project Manager position has been staffed by the same person for the life of the grant. The Health Educator was transferred to the project from HOPE Peru’s Child Survival project about two￾and-a-half years ago. Of nine field staff, three are newly hired for Lima, three have less than a year in VHB, and three have a year to three years. Turnover of staff does not appear to be as much an issue in VHB-Peru as the high turnover of clients. For instance, several long-running banks have had the same promoter for multiple loan cycles, thereby strengthening the impact of the health 69 messages through the strong personal relationship built up between the members and the VHB staffer. 4.4 Staff skills and morale As discussed, skills of most field staff in Health are good, indeed, in several cases, excellent. Time constraints did not permit participation in the credit side of the banks visited but it is possible several field staff have had difficulties in being sufficiently “tough” in loan collection. Staff morale appears quite high throughout VHB-Peru. 4.5 Staff Ratio One Midterm recommendation was to look at the ratio of field staff to clients in light of the proposed VHB standard of 300:1. Currently in VHB-Peru six field staff (excluding three in Lima) are servicing 1040 non-Lima clients, a ratio of 173:1. This is substantially less than the ratio in other countries during the Midterm (265:1) and the proposed program goal. The health supervisor is not counted in that figure. Plans are underway to release one field person at the beginning of October but even then the ratio would be only 208:1. Getting to the program’s proposed ratio in the Tarapoto region would require reducing field staff by half. It will be noted the staff/client ratio in Lima is 80:1, not counting the supervisor. Though a more intensive level of activity is needed in any start-up operation, it may be that this ratio of staff to clients needs to be looked at also. 4.6 Caja Rural VHB-Peru is in the midst of preparing a project proposal for a tri-partite agreement with two other Tarapoto entities. The idea is that VHB would conduct health education, the Caja Rural would be in charge of rural lending, and a private clinic, MaxSalud, would provide health services. If the proposal is funded, it would move VHB into new and desirable relationships. 4.7 Client Focus In individual interviews, some women expressed dissatisfaction with several aspects of VHB operations. Two frequent comments were the project’s requirement for repayments every 15 days even when the woman’s business generates profits only once a month and that the project does not pay interest on savings. Regarding the former, the project has, indeed, modified its policies on entering the Lima market: in Lima, monthly payments are the norm. Regarding the latter comment, VHB-Guatemala was seriously intrigued by the possibility of paying interest on member savings, indicating at least in principle that VHB-Peru could consider the idea also. The point of the paragraph is not to offer this-or-that recommendation but to suggest that in a highly competitive market, VHB-Peru may have been somewhat slow in adapting its policies to customer needs. It may be appropriate to begin adjusting VHB policies and procedures to respond to the best of its clients. 70 5.0 CONCLUSIONS VHB-Peru is struggling for survival. Wonderful accomplishments are taking place in health education. With a redesigned curriculum, high-quality delivery by most field staff, and high quality supervision, the health talks are a joy to observe. The methodology is well conceived and well executed. Women are having great fun and, a year into the new curriculum, they are engaged in exciting new learning. The number of talks given in FY03 is impressive: 3179 as of June 30. The potential gains from this program shift have not yet been fully realized. VHB-Peru began implementing Midterm recommendations in this area in January 2002. With the time required to re￾tool the curriculum, experiment with the new modules, train-up staff, etc., the new methodology is only 12 months old. Coupled with the high drop out rate, though the change in focus is impressive, the impact of the change on women’s behavior is not yet widely evident as the project comes to the end of grant financing. Measuring impact in health has proven a frustrating process for VHB-Peru staff. In spite of the best of intentions, the evaluation methodology chosen has meant that widespread program gains that might be occurring could not be captured in any scientific way. Due to high client turnover, program-wide improvements in health do not appear to have taken place. Personal interviews suggest that substantial individual gains may have taken place—for a few women, world-changing improvements; for a number of others, substantial improvements in family and personal health, and self-esteem. Numerous interviewees suggested they get their PAP check￾ups regularly, attributing their willingness to do so directly to the project. There were other reports of VHB participants’ having been diagnosed in the early stages of uterine cancer who have had their uteruses removed and are now in good health because of a VHB talk. Though this is not widespread program impact, the project certainly has had a life-and-death effect on these few. For those women who stay with the project over the span of two or three years, slightly over 200, it seems likely important changes in health behavior have taken place. Less improvement in health is inferred for an additional 950 or so. The high turnover rate that is slowing health accomplishments is killing the microfinance side of the project. “Low-end” estimates put turnover at 66%. High-end estimates (if 30% of those in arrears do not return) suggest turnover could be 75% or even more. The effects have been felt in member participation, formation of stable banks, retaining a stable client base, loan distribution, loan repayments, arrears, and operational sustainability. Project management is fully aware of these unfortunate facts and has been struggling to deal with them for years. The problem is also felt by other micro-lenders in the project area. Competition from Government of Peru lenders with their emphasis on political influence rather than on micro-finance disciplines seems to be a key factor in the picture. Several colleague comments suggest that an NGO loan program cannot compete in such an environment. 71 Improved income in the 10-15% range has been demonstrated for 205 clients, somewhat less for another 946 working on the inference of four-loan-cycles or more participation. This is at most 35% of target. On the other hand, one has to ask whether VHB-Peru has yet identified a viable program strategy to respond to these conditions. One speculates the project may have been “churning out” new banks to meet planning targets without looking at whether there were better ways of responding to Tarapoto challenges. Ideas that come to mind are the following: · Geographical concentration: e.g., stay in Tarapoto but concentrate all bank operations in one province—as one NGO seems to have done. Or decide to work in provinces where there is no competition from the Government and other NGOs. · Scale down to a very small operation and rebuild the portfolio “organically.” · Concentrate on the quality of individual loan applications and on more intensive individual client response, as several NGOs have decided to do. Substantial and continued HOPE HQ financial support will be required to enable this project to survive. Serious cost-cutting will become necessary everywhere. Moreover, as VHB becomes the only large-scale project left in the HOPE Peru portfolio, contributing to Lima head office costs will add a tremendous additional challenge. 72 6.0 RECOMMENDATIONS As grant financing comes to a close, the VHB will come under pressure to reach self-sufficiency. This is probably a good thing but serious challenges need to be faced. 6.1 Continued HOPE Center Support for VHB Operations Project HOPE needs to decide whether to continue VHB operation in Peru at the conclusion of the USAID grant. This is not as clear-cut a decision as in the other two VHB countries because the program has not yet proven itself effective. Yet it has the best health component of any. It is to be hoped that operational sustainability will eventually be achieved, but it is hard to guess when that might happen. In the meantime, HOPE HQ support of some magnitude will be required in order to keep the VHB doors open. As the evaluator was leaving the country, HOPE HQ had, indeed, taken this decision for FY04. 6.2 Creating of a New Program Strategy It is recommended that VHB-Peru give serious study to determining its true program niche in light of Tarapoto conditions. The drop-out rates, the quite high arrears, and the resulting low operational sustainability have come about in part because the project has not been able to slow the exodus of its clients. As a result, the project is constantly opening new (and untried) banks and promoting new and untried clients, while older banks and older clients leave. Several ideas for further consideration are the following: · Devise a program response that invests less staff time in opening a large number of new banks and more time in working with old banks that may be struggling. Investigate why banks are closing in such numbers and try to address those issues. · Devise a program response to work with solidarity groups of four to six members. · Devise a program response to look at the quality of individual loan applications to see whether enough profit margins exists in the activities women are proposing · Devise a program response to combine some of the stronger members of a disintegrating group into a new, strong group rather than letting them disappear and/or having them start over with new untested members. · Devise a program response to scale-down a (somewhat) spread-out operation and concentrate on one program area, as one colleague NGO has done. Or devise a program response to withdraw from most areas where GOP competition (a.k.a debt forgiveness) is contaminating the micro-credit environment. · Devise a program response to reduce the number of field staff until the portfolio is able to support a larger number. · Devise a program response to manage the Lima start-up taking into account cost and quality implications of long-distance supervision. And devise a program response to reduce the three salaried positions in Lima when the portfolio is only 12 banks. With only twelve banks active in Lima, it is early to wonder whether the same “churning” that took place in Tarapoto will take place here. In addition, as Lima grows it will be important to learn from the past. In any case, quick portfolio growth in an urban context does not seem to hold an immediate key to success. 73 It is obvious that HOPE HQ staff, the Peru Country Director, and the VHB Project Manager will need to give serious thought to these possibilities. 6.3 Improved Adult Education It is difficult to recommend any improvements in Adult Education that VHB-Peru does not already have well in hand. · The project has benefited greatly from the hiring of an Adult Education specialist and the services of this individual should be continued. · The project has a quite creative, grass-roots Health Educator who is providing the necessary touch of ‘field reality’ to the AE expert’s conceptualization. · Quality of the health education is being well monitored through an instrument these two staff have created. Improvements in staff performance are being noted based on the use of this form and based a feedback loop at the end of each session visited by the Health Educator. · Health education materials are being created in an amazing variety at virtually no cost. Clients are delighted at the variety and innovation, and good learning is taking place. The only recommendation could be “keep up the good work.” 6.4 Improved Health Data Collection Regarding health data collection, the recommendation ratifies the recently taken VHB decision in this regard: Collect data on health indicators at the start of a woman’s career in VHB just as income and expenditure data are collected, and measure improvements in health at the close of the 4th loan cycle just as improvements in income and expenditure are being investigated. Applying the same methodology to health data collection as to economic and household data should produce good results. 6.5 Improved Income Data Collection Regarding income data collection, the recommendation is also simple. Collect data on income indicators at the start of a woman’s career in VHB and measure improvements at the close of the 4th loan cycle on all participants—not only on a sample. This will eliminate much of the guess work of estimating improved income for long-term participants. 6.6 Cost Cutting Serious cost cutting is necessary in all parts of the project via: · Reduced staffing · Reduced administrative, secretarial and MIS support · Reduced Tarapoto office size and smaller institutional “footprint” · Lima field operations made more cost efficient · Reduced Lima head office support · Reduced Indirect HOPE HQ. 74 It is to be hoped these measure will be enough to keep the project alive. There is much to be lost in high quality health if this project disappears. 75 APPENDIX G List of Interviewees HOPE Center John Bronson VHB Technical Program Advisor, HOPE HQ Bob Grabman Latin America Regional Director, HOPE HQ Petra Reyes Africa Regional Director, HOPE HQ Randy Wykoff Senior V.P. for International Programs, HOPE HQ VHB Malawi Staff Shalote Chipamaunga Technical Support Manager Howard Bowa VHB Project Manager Lansen Chikopa VHB Health Education specialist Christopher Gremu MIS Davie Malonie VHB Supervisor Bernard Bvunguti Community Development Promoter Kingsley Kumatso Community Development Promoter Edith Maxwell Community Development Promoter Collins Namathawa Community Development Promoter Geoffrey Ziyada Community Development Promoter Colles Nyirenda Community Development Promoter Lucy Banda Community Development Promoter Charles Gwembere Community Development Promoter VHB Malawi Clients Mercy Mpama VHB member- Blessings VHB Letisia Brash VHB member- Lunguni VHB Eleni Sonbole VHB member- Chitukuko Dhimwaza VHB Agnes Minjolo VHB member- Chitukuko Chimwaza VHB VHB Guatemala Staff Julieta Afre Nutricionista (PSI) Anabela Aragón Deputy Director Juan Carlos Lau Project Manager Juan Carlos de León Supervisor Area 1 Noemí Elizabeth García Técnico Area 1 Oscar Rene Barrios Técnico Area 1 Alma de Jesús Hernández Educadora de Salud Jelmo Guobany Figueroa Supervisor Area 2 Herber Hiraldo Paz Técnico Area 2 Francisco Leonel Sandoval Técnico Area 1 Nestor Giovanni Rodríguez Educador VHB Guatemala Clients Amparo Orusco Coatapeque VHB Concepción Chanvac Coatapeque VHB Anselma de León Esquipulas Palo Gordo Gloria Alisia Merido Ochoa Esquipulas Palo Gordo 76 Margarita Estela López Miranda Las Samaritanas VHB Araseli Orozco Ayuda Mutua VHB, San Marcos Elsira Méndez Ayuda Mutua VHB, San Marcos Venancia Navarro Nuevo Amanecer VHB Floridalma Leticia Orusco Pérez Nuevo Amanecer VHB Maria Carmelina Ardia Nuevo Amanecer VHB Paula Ramos Nuevo Amanecer VHB Elba Teresa Yoc Triunfo VHB Heidi Areli Triunfo VHB Elida Gloria Ramos Triunfo VHB Magdali López Morales La Ceiba VHB, San Pablo María Enriqueta Rodríguez La Ceiba VHB, San Pablo Dina Rodríguez La Ceiba VHB, San Pablo Augusema Morales La Ceiba VHB, San Pablo Alisia Domingo Chávez El Jardín VHB Ilaria Elvira Aguilar El Jardín VHB Norma Rodríguez El Jardín VHB Amparo Baptista El Jardín VHB Guatemala Colleague Organizations César López Gerente: FUNDESPE, Fundación de Desarrollo de la Pequeña Micro-Empresa, San Marcos Eric Godínez Gerente, Micro-Financiamiento, CARITAS, San Marcos VHB Peru Staff Sidlia Torres Arevalo Project Manager Nancy García Health Educator Carlos Ballena VHB MIS Karina Saavedra VHB Asesora Selena Sandoval VHB Asesora Noemí Coronel VHB Asesora Fernando López VHB Supervisor VHB Peru Clients Palmir Arué VHB Santa Inés Lily Paredes VHB Santa Inés Belen Sangama Chiricayu VHB Rosario Sangama Chiricayu VHB Berta Oyola Ordoñas Chiricayu VHB Darcila Cachique Sangán Chiricayu VHB Lorenza Salas Tapugima Chiricayu VHB Emarvil Luna Rengifo Unión Solidaria VHB Sonia Saavedra Unión Solidaria VHB Rosana Reateicaro Unión Solidaria VHB Marina Flores Unión Solidaria VHB Juana Villaro Deciurliza Unión Solidaria VHB Cristina Campana Avalo Las Orquídeas VHB Judit Tuanama Las Orquídeas VHB 77 Leonida García Pinchi Las Orquídeas VHB Dorita Cordoba Armas Las Orquídeas VHB Rosa Jansica Pezo Utia Sagrada Corazón VHB Rosa Maria Portugal Ríos Sagrada Corazón VHB Sarita Pasmina Ruiz Sagrada Corazón VHB Madrona Días Montolla Sagrada Corazón VHB Teresa Riatege Fortaleza de Diós VHB Ligner Velásquez Fortaleza de Diós VHB Felicita Regaderos Frias Christo Rey VHB Aurelia Rodes Ramírez Las Hermosas VHB Milagros Gutieírrez Semenario Vita Alegre VHB Licena Ayachi Chujulalla Dios Amanece Para Todas VHB Nilsa Balverde Remayna Dios Amanece Para Todas VHB Elcida Rivera Diós Amanece Para Todas VHB Peru Colleague Organizations Mercedes Ruiz Project Coordinator, CEPCO Salvador Bartola Project Coordinator, PRISMA Davis Linares Regional Director, Fundación Manuela Ramos Information also presented to Marta Arce HOPE Adult Education Consultant-L.A. Víctor Calderón HOPE Country Director- Guatemala Sandra Contreras Agulo HOPE Country Director-Peru Dorothy E. Namate HOPE Country Director-Malawi 1 APPENDIX H Bibliography Government of Malawi: Malawi Poverty Reduction Strategy Paper, Oct. 2001. Micro Banking Bulletin: MicroBanking Standards Project, Issue # 7, Nov. 2001. Project HOPE: Annual Report No. 1, Jan. 2000 _____ Annual Report No. 2, Jan. 2001 _____ Annual Report No. 3, Jan. 2002 _____ Annual Report No. 4, Jan. 2003. _____ Detailed Implementation Plan (DIP) October, 1999. _____ Midterm Evaluation, October, 2001 _____ Village Health Banking Program (Project Proposal), 1999. Project HOPE Guatemala: Encuesta KPC Medio Término, Oct. 2001. _____ Estudio Comparativo de Línea Base Sobre Conocimientos en Salud, Coberturas y Practicas de las Socias, Marzo, 2003. _____ Estudio Comparativo de Línea Base Sobre Conocimientos en Salud, Coberturas y Practicas de 52 Socias Entrevistadas en los Estudios Realizados Durante los Años 2,001 y 2,003, Abril 2,003. _____ Estudio Comparativo de Línea Base Sobre Conocimientos en Salud, Coberturas y Practicas de Las Socias Que Participan en Las Sesiones Educativas de los Bancos Comunales en Salud, Mayo 2,003. Project HOPE Malawi: Annual Health Survey, 2001, Oct. 2001. _____ Annual Health Survey, 2002, Jan. 2003 _____ Baseline Health Survey, Aug. 2000. _____ Credit Policies Revision, March, 2002 _____ Detailed Implementation Plan, Oct. 1999. _____ Health Education Training Modules (undated) _____ Internal Regulations, Jan. 1998. _____ Loan Approval Procedures, March 2001. _____ Quarterly Report, July-Sept. 2002. _____ Quarterly Report, Oct.-Dec. 2002. Project HOPE Perú: Análisis KPC 2000-2003 _____ Bancos Comunales de Salud, Estudio Cualitativo (Septiembre, 2002) _____ Componente de Salud de Bancos Comunales de Salud, 2003 _____ Guia De Supervision Y Monitoreo [en Salud] _____ Encuesta KPC, Resultados de Impacto 2000-2003 _____ Eje: Mujeres Saludables _____ Módulo 3: Seguridad Y Alegría, Papanicolaou al Día _____ Módulo 6: Mujer Saludable: Sexo Seguro Y Agradable USAID Malawi: Annual Report 2002. World Bank: World Development Indicators 2003.