USAID/PVC Matching Grant Evaluation Series: Capacity Building for Microenterprise Development In Mexico, Romania and U.S. Headquarters Matching Grant FAO-A-00-99-00055-00 between Enterprise Development International and USAID/PVC February, 2002 Conducted under USAID/CDIE Evaluation Indefinite Quantity Contract # AEP-I-02-00-00024-00, Task Order 2 Evaluation team members: Lawrence Locklin (Enterprise Development International) Harry Carr (MSI, Team Leader) H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc i Evaluation of “Capacity Building for Microenterprise Development in Mexico, Romania and U/S. Headquarters” Matching Grant FAO-A-00-99-00055-00 between Enterprise Development International and USAID/PVC TABLE OF CONTENTS List of Acronyms Used..........................................................................................................................iii Evaluation Identification Sheet............................................................................................................iv 1.0 Executive Summary* .......................................................................................................................1 1.1 Overview..............................................................................................................................1 1.2 Key conclusions and recommendations.................................................................................1 1.3 Acknowledgements...............................................................................................................3 2.0 Evaluation Methodology and Team Composition*.........................................................................4 3.0 Matching Grant Background* ........................................................................................................6 3.1 Historical & technical context and partners* .........................................................................6 Background to Grant.............................................................................................................6 Project Context - Mexico ......................................................................................................6 Project Context - Romania ....................................................................................................6 Methodology, Approach, Strategy and Target Groups...........................................................7 3.2 Project goal, objectives, and major hypotheses to be tested*..................................................7 4.0 Purpose of the Evaluation*..............................................................................................................9 5.0 Program Implementation Evaluation Questions...........................................................................10 5.1 The Detailed Implementation Plan ......................................................................................10 5.1.1 Meeting DIP targets and data accuracy....................................................................10 5.1.2 Quality of DIP and degree of success in implementation*........................................11 5.1.3 Familiarity with DIP and design* ............................................................................17 5.1.4 Major successes and shortfalls in implementation*..................................................17 5.1.4 Impact Results* ......................................................................................................19 5.2 Assessment of project model and hypotheses ......................................................................19 5.2.1 Project hypotheses articulated in CA* .....................................................................19 5.2.2 Replication and scale-up of approaches in project area or elsewhere* ......................20 5.3 Advocacy under the project.................................................................................................21 5.3.1 Advocacy activities and impact* .............................................................................21 5.3.2 Partner/PVO roles in advocacy* -- N/A...................................................................22 5.4 Implementation Lessons Learned ........................................................................................22 6.0 Partnership Questions ...................................................................................................................23 6.1 Analysis of Partnership Schemes*.......................................................................................23 6.2 Measuring Institutional Capacity*.......................................................................................23 6.3 Constraints to Partnership* .................................................................................................26 6.4 Information Technology* -- N/A.........................................................................................26 6.5 Use of local networks and service organizations* -- N/A.....................................................26 H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.docii 7.0 Program Management...................................................................................................................26 7.1 Strategic Approach and Program Planning* ........................................................................26 7.2 Country Initiatives* -- N/A .................................................................................................27 7.3 Conflict Management* -- N/A.............................................................................................27 7.4 Monitoring and Evaluation*................................................................................................27 7.5 Overall Management*.........................................................................................................28 7.6 Sustainability* ....................................................................................................................30 7.6.1 Overall sustainability survey* -- N/A ......................................................................30 7.7 Financial Management ........................................................................................................30 7.7.1 Effectiveness of financial management*..................................................................30 7.7.2 Leveraging other donor funds* -- N/A.....................................................................31 7.7.3 Cost effectiveness of technical approach*................................................................31 7.7.4 Repercussions of “matching” requirement on program* ..........................................32 7.8 PVO’s Information Management*.......................................................................................32 7.9 Logistics* -- N/A ................................................................................................................33 7.10 Project Supervision*.......................................................................................................33 7.11 USAID Management* ....................................................................................................33 8.0 Conclusions and Recommendations* ............................................................................................34 Annexes................................................................................................................................................36 Annex A: Key Events Timeline* .........................................................................................................37 Annex B: Detailed Implementation Plan Table* ................................................................................39 Annex C: Evaluation Scope of Work * ...............................................................................................48 Annex D: Solidarity Group Summary Information for Avance Chalco ............................................58 Annex E: Client Summary Information for HIRO.............................................................................59 Group Summary Information for HIRO............................................................................................60 Annex F: List of Persons Interviewed .................................................................................................63 Annex G: Simplified Activity and Financial Reports.........................................................................64 Annex H: EDI Criteria for International Programs and Partnerships..............................................70 H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc iii LIST OF ACRONYMS USED AMEXTRA Asociacion Mexicana de Transformacion Rural y Urbana (Mexican Association for Rural and Urban Transformation) BHR Bureau of Humanitarian Relief CA Cooperative Agreement DOSA Discussion-Oriented Organizational Self-Assessment EDI Enterprise Development International LNGO Local non-governmental organization HIRO Help Integra Romania MEDA Mennonite Economic Development Associates NGO Non-governmental organization NICRA Negotiated Indirect Cost Rate Agreement OI Opportunity International PVC Private Voluntary Cooperation PVO Private Voluntary Organization (usually U.S.) TBF Transylvania Business Forum USAID United States Agency for International Development WOF Women’s Opportunity Fund H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.dociv EVALUATION IDENTIFICATION SHEET PVO name Enterprise Development International Matching Grant Title Capacity Building for Microenterprise Development in Mexico, Romania and U/S. Headquarters Cooperative agreement number FAO-A-00-99-00055-00 Amount of Grant $ 649,990 Period of Grant 1999 and 2002 Any (cost/no cost) extensions? No Current status of MG In final stages USAID/PVC Grant Officer (s) Tom Kennedy Technical area of grant Microenterprise development Date of the evaluation February, 2002 Countries of program activity Mexico, Romania and U.S. Headquarters Country programs evaluated Mexico, Romania and U.S. Headquarters Evaluation Team Members (organization) Harry Carr, MSI Lawrence Locklin, EDI H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 1 Evaluation of “Capacity Building for Microenterprise Development in Mexico, Romania and U/S. Headquarters” Matching Grant FAO-A-00-99-00055-00 between The Nature Conservancy and USAID/PVC 1.0 EXECUTIVE SUMMARY* 1.1 Overview Enterprise Development International (EDI) has the goal of providing increased capacity to its partner NGOs in Mexico and Romania to support the delivery of sustainable credit with training to poor women. Working with a three-year Cooperative Agreement for $649,990 initiated in September, 1999, EDI is strengthening its headquarters capacity as well as the capacity of its partners in Romania – Integra and HIRO (Help Integra in Romania) – and Mexico – AMEXTRA (Mexican Association for Rural and Urban Transformation) and Avance Chalco. Solidarity groups of primarily women entrepreneurs are offered saving and borrowing services in Mexico, while in Romania primarily women clients are offered business training and loans. Management Systems International (MSI) was contracted by PVC to conduct this summative evaluation, which was carried out between late September and early February, 2002 by a team comprised of one MSI evaluation specialist and one EDI representative. Team members reviewed relevant documents and interviewed key informants at EDI headquarters in Fairfax, Virginia as well as the staff and board members of the partner organizations in Mexico and Romania. EDI was new to USAID as a grantee prior to this Cooperative Agreement (CA). The Evaluator concludes that PVC has had a successful Matching Grant experience with EDI. Its goals fit very well with the Matching Grant program’s micro-enterprise development goals, and its focus on women as a means of sustaining family income fits well with USAID’s gender goals. By the end of its second of three years, EDI’s two programs have reached their targets for operational sustainability and it is possible that the Mexico program will be 100% operationally sustainable before the end of the CA. It should be noted that this success in operational sustainability comes on top of early portfolio-at-risk problems for each partner, which were overcome quickly. The conclusion can be made that PVC has found a new PVO partner to work with in achieving its goals. 1.2 Key conclusions and recommendations Conclusions: (a) Almost half – 47% (24 out of 51 indicators) of Year Three, end-of-project, targets were already met, 16% (8 out of 51 indicators) of Year Two targets were not met and only 6% -- 3 indicators – represent a problem for EDI in Year Three; (b) Regarding program efficiency, Avance’s and HIRO’s low reported performance on these indicators is probably overstated, that is, in the cases of both Avance and HIRO the real “inefficiencies” were in Year One and they have, since then, been overcome. In both cases, however, the problem loans made in Year One have created a rolling effect through Year Two. This is reflected in under-achievement in that year for efficiency measures. (c) EDI partner managers do not appear to use the DIP and its components as a guide to management or source of modifications in the program. EDI Headquarters, however, does utilize reported results against indicators as a guide to oversight and supervision vis a vis its partners. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc2 (d) Regarding impact for the Matching Grant Program, out of twelve indicators which could be considered as measuring impact, nine have been achieved for Year Two, evenly distributed between HQ, Avance Chalco and HIRO objectives. (e) EDI is a faith-based private voluntary organization. There is a close, mutually supportive bond of Christian values between EDI and all of its partners. The resulting trust among partners increases program effectiveness through better coordination and collaboration. There is high motivation among EDI and partner staff for achieving real program impact. Staff dedication reinforces client outreach and client-loan officer relationships. The resulting amount of time and assistance given by loan officers to their clients could be a key determinant of the program’s success. (f) In the course of the Matching Grant implementation and, particularly, in response to problem loans in Year One, EDI has modified its practice in small degrees away from the very poor and women in order to accommodate the program’s needs for viability and sustainability. The evaluators observed that it is sometimes difficult for all partners to maintain a balance in loan recipient criteria such that the integrity of the original development hypothesis is guarded. In most cases EDI and its partners tend toward the developmentally needy, but there is an attraction towards the safer loans with more substantial businesses in order to protect portfolio viability. (g) Even though advocacy and networking with other NGOs for the sake of lobbying are not objectives of the CA, each local partner has found it necessary and effective to join other similar entities and associations either to increase their understanding of national and regional issues of importance and/or to advocate certain positions before government. This has been particularly salient for each partner regarding national registration laws and regulations. (h) In the partnership arrangement for the EDI Cooperative Agreement, EDI relates primarily with AMEXTRA and Integra, the sponsoring institutions. Supervision of the local partners, Avance and HIRO, is left to the sponsoring institutions. The oversight and supervision of local partners, however, varies in effectiveness. The Integra Program Advisor, who is partially paid for by EDI under the Matching Grant, makes many visits, has adequate communication and gives good direction to HIRO. AMEXTRA appears to leave Avance more on its own, Avance having more contact with EDI directly. EDI has tended to fill the gap in Mexico by relating more directly with Avance for supervision. (i) The increased ability of EDI to plan for the future, reinforced by significantly improved financial and administrative management systems allowing for better reporting and projecting, has been so remarkable that it is referred to among the significant successes of the Matching Grant. (j) One of EDI Headquarter’s objectives is to provide its implementing partners with effective management information systems (MIS). EDI has had particular difficulty in identifying an appropriate MIS for Avance Chalco; one that meets all its program requirements. This delay has stretched the institutional capacity of Avance, although to￾date the problem has been managed. However, the rapidly growing size of Avance’s portfolio makes solution of this problem even more urgent. EDI believes it has, as of the writing of this evaluation, found the appropriate system and is negotiating with the vendor for its use in Mexico. (k) The progress reporting consisting of monthly reports on program issues, loan status and financial management is appropriate in content and timeliness. EDI has been able to both note and follow problem issues, operations and financial performance. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 3 Recommendations: (a) EDI should make every effort to ensure an adequate MIS software for Avance Chalco as soon as possible. The goal of its negotiations is to obtain such a software package – COBIS – in early 2002. Contract negotiations are underway at the writing of this report and the contract should be signed in April. (b) EDI should try to instill more of a sense of ownership on the part of its partners in the DIP and their respective parts of it, particularly their performance indicators, by doing the following: fi Involve partners more actively in the program design, particularly goal and target setting, so that partners take more ownership in plans and fi Perhaps build into the budget an “all-partner manager” multi-day workshop for DIP formulation and team building, as well. (c) There is an important lesson for PVC in the effect on program impact provided by the sharing of basic values and operating principles as EDI and its partners do. In their case the bond is one of shared Christian values. It could be that similar common bonds exist or could be fostered among PVOs pursuing objectives in human rights, humanitarian assistance, child survival and disaster relief. (d) EDI should be attentive, in its program monitoring, of the extent to which its partners are accommodating viability and sustainability to the expense of its mission of assisting women and the very needy. This can be done in several ways: fi Definition of explicit performance indicators designed to track the socio-economic conditions of its borrowers and clients as well as their businesses; fi A careful re-consideration of performance monitoring results, even including the indicators EDI now has, with the purpose of maintaining its desired balance between a viable and sustainable program and one that serves its mission as closely as possible. (e) Because EDI is responsible under the CA for improved capacity in the local partners, EDI should be particularly attentive to the guidance, support, and assistance they may need and may not be getting from their sponsoring institutions. Particular areas include: fi Local staff training; fi Local board training; and fi Client training. 1.3 Acknowledgements The Evaluation Team Leader would like to acknowledge the Headquarters staff of EDI and the staffs of Avance Chalco in Mexico and HIRO in Romania whose names are given in Annex F. They provided great support, assistance and availability for discussions and interviews with the Team Leader. Moreover, their participation in the evaluation by providing the necessary data, interpretations and clarifications added value to its conclusions and recommendations. In all cases staff not only participated willingly and openly but also incorporated some recommendations into their operations even in the course of the evaluation. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc4 2.0 EVALUATION METHODOLOGY AND TEAM COMPOSITION* Approach: The overall approach to this evaluation was participatory. The evaluation was carried out by an MSI Team Leader/Evaluation Specialist in participation with (during the Mexico phase) the EDI Program Manager. The evaluation methodology involved review of program documents, information gathering interviews with program and partner staff both at headquarters and in the field sites (Romania and Mexico), field visits for data gathering, interviews with program clients and program activity observation. Data gathered at headquarters and in the field consisted of secondary information in program reports, correspondence and records as well as primary data gathered through protocol guided interviews and observation of representative and stratified clients and client groups. Evaluation tasks were undertaken during the following five major phases: Phase One: Preparatory work in Washington - Half-day Team Planning Meeting with EDI and USAID officials (September 27) and document review in Washington, DC (September 26 to October 4). Phase Two: Visit to EDI Headquarters – The Team Leader visited EDI Headquarters in Fairfax, Virginia on two occasions (November 8 and 20 – 21) to interview EDI staff and gather data and additional documents. Phase Three: Field visit to Mexico – The Team Leader, accompanied by the EDI Program Manager, visited the AMEXTRA/Avance Chalco site in Mexico to interview partner directors, board members and staff, visit and observe client solidarity groups, and review data and gather documents (November 10 to 17). Phase Four: Field visit to Romania – The Team Leader visited Oradea, Romania and the HIRO site to interview the Executive Director, Program Director and staff, to visit program client borrowers, to review data and gather documents (January 12 to 15). Phase Four: Preparation/submission of draft report (January 15 to 31). Incorporation of USAID and EDI comments, and production of final report. As indicated in the various sections of this report, data were verified through the review of key documents, observation and interviews with relevant individuals and groups (Annex F provides a list of persons contacted). Findings are based on the information collected, while conclusions and recommendations are the opinions and contributions offered by the evaluation team. Report Format: It should be noted that the team was asked to use a pre-determined format for the preparation of this report. That is because this is one of a set of some 12 final evaluations of Matching Grant for which MSI was contracted by USAID/PVC. Therefore, to facilitate possible future study at the level of the overall PVC grant program, it was determined that all reports would employ the same sections and sequence. Non-applicable sections are marked simply “N/A.” Team Composition: H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 5 Based on the Scope of Work (see Annex C), the evaluation team was comprised of the following members: v Harry Carr, an Evaluation Specialist with thirty years of years of economic development consulting experience most of which has been spent in evaluation and the related activities of strategic planning, performance monitoring and organizational development. v Lawrence Locklin, EDI’s Program Director, a position created by the matching grant, Mr. Locklin has worked in small and medium enterprise development in economic development circumstances since 1993. The primary limitation of the evaluation was the need to re-schedule the Romania field trip late in the process due to international security reasons prohibiting necessary country clearance. This postponement reduced the amount of time available to analyze, absorb and write-up the data and findings. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc6 3.0 MATCHING GRANT BACKGROUND* 3.1 Historical & technical context and partners* Background to Grant Enterprise Development International was awarded its current three-year Matching Grant, the organization’s first, in September 1999. The three components of the grant are Enterprise Development International’s headquarters office in Fairfax, Virginia, and partners in Mexico and Romania. Since the inception of the grant, Enterprise Development’s Matching Grant partners, AMEXTRA in Mexico and Integra in Eastern Europe, have each created subsidiary organizations specializing in microenterprise development. AMEXTRA created Avance Chalco to serve the residents of the Chalco Valley outside Mexico City. Integra created Help Integra Romania (HIRO) to serve women in Oradea, Romania. Through the grant, Enterprise works to strengthen its ability to support sound microfinance services and to build the institutional capacity of Avance Chalco and HIRO to provide microenterprise development services. Project Context - Mexico The project is located to the southeast of Mexico City in the State of Mexico, within the larger metropolitan area that Mexico City encompasses. The project serves the Valley of Chalco, specifically the municipalities of Ixtapaluca, Chimaluacan, La Paz, Chalco, Valle de Chalco, Chicoloapan and Nezahuayacoyotl, which contain more than 1 million inhabitants. The population is made up of first and second generation migrants from the countryside who have settled in the area because they are unable to afford living in Mexico City proper. Some communities now have regular services such as water, drainage and electricity, while others have simply improvised these. The adult population has, on average, a 6th grade education, and 15% are illiterate. Project Context - Romania Oradea is located in Bihor County and has a population of 222,000 out of 22 million people living in the country of Romania. Even though it is considered one of the most significant economic, social and cultural centers in northwestern Romania, it is worse (as determined by the baseline work) than the national average in several key economic indicators: · lower average monthly salary · higher unemployment · a high percentage of women in the “at risk” category · a low number of courses and participants in programs for retraining or providing additional training for unemployed workers In addition, baseline work determined that Bihor county is characterized by a high percentage of women that would like to own their own business. Although the vast majority of the companies in Bihor County are privately owned, they operate primarily in the trade and services sectors. Government owned companies are still primarily accountable for the industrial output (which accounts for 55% of the county GDP) in the county. The primary industry for Oradea and Bihor County is agriculture. Much of this sector is characterized by small farms that are unable to support many employees, thus there is need for new small businesses in the service or light industry sector that can develop to employ additional employees. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 7 The educational opportunities under the communist regime were abundant and have continued to the present even though the funds available for programs and educators have not kept pace with inflation. Therefore, the quality and cost (direct and indirect) of higher education discourages many from continuing their studies. “Retraining” programs for the unemployed are also not very prevalent (currently only four programs are in existence), thus do not provide the opportunities for enabling the unemployed to improve their chances for employment. Methodology, Approach, Strategy and Target Groups The methodology employed at the US PVO level is for Enterprise Development International to partner with independent local microenterprise-development organizations to build their institutional capacity by providing them with training, technical assistance and funding. In Mexico, AMEXTRA’s subsidiary, Avance Chalco uses a cooperative organizational structure to provide savings-led microfinance services to poor residents of the Chalco Valley outside Mexico City. Avance Chalco also partners with a local organization specializing in training to offer its clients regular business training opportunities. In Romania, Help Integra Romania (HIRO), a subsidiary of the Integra Foundation, offers intensive business training, followed by business plan assistance and loans to the poor women of the city of Oradea. Summary data on the clients of Avance Chalco and HIRO are shown in Annexes D and E. They provide a representative portrayal of EDI’s partner client base. 3.2 Project goal, objectives, and major hypotheses to be tested* EDI’s program goal is to enhance Enterprise Development International’s capacity to support its partners, and to improve the income, social status, and welfare of very poor residents of the Chalco Valley outside Mexico City, and in Oradea, Romania by enabling them to support themselves. EDI’s overall objectives in its USAID funded MG program are 1) to improve Enterprise Development’s technical assistance, training, and capacity building services to its network of partners, and 2) to develop the capacity of Enterprise Development’s partners in Mexico and Romania to achieve increased outreach and to work toward operational self-sufficiency. EDI works to achieve these objectives through three sets of subordinate objectives focused on its Headquarters and its programs in Romania and Mexico. The following table summarizes the hierarchy of objectives, as presented in the Detailed Implementation Plan (DIP) and amended, in the case of Headquarters, in July 2000: H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc8 Table 3.2: Project Hierarchy of objectives Goal: Headquarters: Objective 1: Board Development: Equip local boards to improve their capacities in organizational vision, leadership, oversight, and representation Objective 2: Partner Staff Training: Train local staff in all aspects of program implementa￾tion, management, evaluation, growth, and accountability (including impact assessment) Objective 3: Client Training: Equip partner staff to adapt and use appropriate training materials and methodologies for their clients Objective 4: Partner MIS: Equip partners with effective management information systems Objective 5: Organizational Assessment: Enable partners to accurately and constructively assess their own organizations and plot a course for improvement based on results of the assessment Objective 6: Program Officer: Add to staff a Program Officer who will provide high quality technical services to partners and backstopping in home office Objective 7: VP Technical Services: Expand purview and capacity of VP Technical Services to include broader array of partner services and development of home office tools Objective 8: Government Accounting: Develop staff expertise in accounting and reporting for US government grants Objective 9: Grants Management and Program Monitoring: Develop staff expertise and improve internal systems for major grant management and program monitoring Objective 10: Computer Network: Improve home office computer network to better serve staff need Objective 11: Strategic Planning: Develop five-year corporate strategic plan to set direction for the organization Objective 12: Sustainability: Improve home office financial sustainability Objective 13: US Public Awareness: Increase US public awareness of the work of EDI Romania: Objective 14: Establish Integra institutional capacity to deliver microenterprise credit and training services in Romania Objective 15: Achieve significant outreach among low-income women Objective 16: Implement successful business training Objective 17: Achieve program efficiencies Objective 18: Progress toward sustainability of the MED institution Objective 19: Achieve significant impact Mexico: Objective 20: Expand AMEXTRA’s institutional capacity to deliver MED services Objective 21: Establish significant program breadth Objective 22: Establish significant program depth Objective 23: High Program Efficiency Objective 24: Progress toward sustainability Objective 25: Achieve significant impact H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 9 4.0 PURPOSE OF THE EVALUATION* The specific purposes of the evaluation were the following: · To document the achievements of the EDI program against its objectives and · To report on progress made towards specific PVC Matching Grant program objectives related to partnership relations and aspects of program management, including: fi Sustainability; fi performance monitoring; fi financial management; fi information management; and fi relations with USAID management. The evaluation is envisioned to be used in the following ways: · To tell PVC, and EDI itself, how EDI and its partners have performed on these dimensions in order to provide guidance for each in the preparation and review of the second EDI grant application and also · To tell PVC lessons that have been learned by EDI in the course of the MG, again on the dimensions listed above, so that PVC might adjust its procedures and policies accordingly. (See Annex C for the evaluation scope of work.) H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc10 5.0 PROGRAM IMPLEMENTATION EVALUATION QUESTIONS 5.1 The Detailed Implementation Plan 5.1.1 Meeting DIP targets and data accuracy Annex B contains detailed information on EDI’s implementation of the CA and its objective. It shows, particularly, data on EDI’s accomplishments against its performance indicators. The findings, conclusions and recommendations presented below refer to those data. Findings: (a) As shown in Table 3.2 above, EDI has a total of 25 objectives divided over three categories: Headquarters Improvement; Avance Chalco Performance and HIRO Performance: fi For Headquarters Improvement, EDI has 14 objectives in its planning matrix and 19 performance indicators of which – · Two indicator targets were not reached for Year 2; · Six indicator targets for Year 2 were reached; · Ten indicator targets for Year 3 were reached; and · One indicator for which target data were not collected. fi For Avance Chalco/AMEXTRA EDI has 6 objectives and 17 performance indicators of which – · Six indicator targets were not reached for Year 2; · Two targets for Year 2 were reached; · Seven indicator targets for Year 3 were reached; and · One indicator was removed and data for one target were not collected. fi For HIRO/Intrega EDI has 6 objectives and 17 indicators of which – · Eight indicator targets for Year 2 were not met; · Two targets for Year 2 were reached; and · Seven targets for Year 3 were met. (b) In summary, regarding EDI’s reaching its performance indicator targets: fi 16 % were not reached fi 78 % were reached in Year 2 for either Years 2 or 3 fi 6 % could not be counted Conclusions (a) Targets which were not met and which present a problem for EDI – 3 fi Lack of an adequate MIS for Avance, reflected in two indicators fi HIRO’s underachievement in number of active borrowers (b) Targets not met but which should be or have been either eliminated or reduced -- 4 (c) Targets which were not met but which EDI is confident will be met – 6 (d) Targets which were not met and for which EDI has justification -- 4 H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 11 Recommendations (a) As is discussed below in Sections 5.1.4 and 6.4, EDI should make every effort to ensure adequate MIS software for Avance Chalco as soon as possible. The goal of its negotiations is to obtain such a software package – COBIS – in early 2002. Contract negotiations are underway with the writing of this report. (b) HIRO’s failure to reach its target in number or borrowers could, indeed, be very good management as a response to early procedures which allowed unfeasible loans to be made. Since this is potentially a serious problem and it is reflected in HIRO underachievement for other targets, EDI should pay attention to the indicator’s results and assure itself that HIRO is doing something more to bring its numbers up. 5.1.2 Quality of DIP and degree of success in implementation* Findings: Regarding Avance Chalco’s compliance -- (a) At the end of its second year, the EDI program in Mexico has achieved most of its Year Two targets and almost all of its final targets. (See Annex B.) All indicators for Objectives 1 and 5 have been achieved. All but one indicator for Objectives 2, 3, and 6 have been achieved. The first indicator for Objective 2 has all but been reached, with 1,261 clients saving against a targeted 1,350. (The DIP planning matrix calls this indicator “number of persons receiving training and saving.” This is because originally, the vision was to provide business training to every client. See (c) directly below for an explanation of this change.) The first indicator for Objective 3, regarding program depth, looking for 95% participation by women among the client population, has been dropped, as was reported in the Year One Progress Report. EDI reports that the second indicator of Objective 5 percent client retention – was, “included in the planning matrix in error.” This effectively leaves Objectives 2, 3 and 5 achieved. Neither indicator of Objective 4 for Year Two has been achieved. Regarding HIRO’s compliance -- (b) The same indicator has been eliminated from the HIRO planning matrix for the same reasons. In the case of HIRO, however, 97% of its clients are women. (c) By the end of Year Two in Romania, eight out of its 18 indicators, including number of women borrowers, have not been reached, two of its Year Two targets have been achieved and 10 indicator Year Three targets have been reached. Regarding EDI Headquarters compliance -- (d) EDI has failed to achieve two of its 19 HQ improvement indicator targets. The single indicator target has several components, among which providing partners with adequate MIS has not been achieved for Mexico. Targets for establishing an endowment were not achieved in Year Two because of the events of September 11 and the impact on the Mexican economy. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc12 Conclusions: (a) The elimination of the indicator for women participation in the program has a certain logic. What in fact was eliminated was the requirement that women be targeted exclusively as clients. Men can now be members. AMEXTRA felt that if there was bias for women, then male borrowers might simply force their wives into enrolling while they, the men, actually made the saving and loan decisions. There is heavy participation of women in the program and will be almost by its nature. More importantly almost all Cashiers, or group accounts keepers and money collectors, are women who play a crucial role in the groups and who are a significant element in the success or failure of a group. (b) The training of clients in Mexico has changed from the DIP. Originally, all clients were slated to receive business training. Providing business training for all clients, however, proved to be expensive, a poor response to clients needs, and a poor use of many clients’ time. Therefore it was decided that it would be better to provide business training on a demand basis, and periodic business seminars were planned. Three business seminars were conducted by Avance Chalco, reaching a total of 106 people. It was then decided to partner with MicroMEDA Mexico, an organization which specializes in business training for microenterprises in the Chalco Valley. These courses began in November 2001. (See the Year Two Report for a description of MEDA’s methods.) This accounts for Avance’s under-achievement on the first indicator for its Objective 3. (c) In Romania, the HIRO staff was trained by Intregra in its client training and subsequently has used it, accounting for the under-achievement on Objective 3. (d) The evaluator concludes, regarding program efficiency, that Avance’s and HIRO’s low reported performance on these indicators is probably overstated, that is in the cases of both Avance and HIRO the real “inefficiencies” were in Year One and they have, since then, been overcome. In both cases, however, the problems loans made in Year One have created a rolling effect through Year Two. This is reflected in under-achievement in that year for efficiency measures. Recommendations: (a) EDI has identified the elimination of the women’s training indicator for Mexico noted above in its Year Two Report. However, the same elimination in the case of HIRO’s planning matrix has not been reported. EDI should report it and any other changes. (b) Because of the possible overstatement of inefficiencies in Mexico and Romania, EDI should pay extra close attention to these indicators and Avance and HIRO performance on them in Year Three to make sure that partners’ performance does, indeed, “catch up”. Highlights of implementation experience, based on review of the DIP, are summarized Table 5.1.2 on the following pages. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc13 Table 5.1.2 Objectives and Outcomes Matrix Objective Organizational Level Key Outcomes Successes Constraints Headquarters Objective 1: Board Development: Equip local boards to improve their capacities in organizational vision, leadership, oversight and representation EDI Board: Yes Avance Board: Somewhat, but still growing, should/could give training EDI Board: Undergone revision and new members; members involved in program HIRO Board: Planning expansion Objective 2: Partner Staff Training: Train local staff in all aspects of program implementation, management, evaluation, growth and accountability (including impact assessment) AMEXTRA/Avance, EDI and HIRO: Staff trained in several ways and venues on all dimensions of MED AMEXTRA/Avance: Very positive efforts and impact on all dimensions HIRO: Particular success in training of Finance Manager Objective 3: Client Training: Equip partner staff to adapt and use appropriate training materials and methodologies for their clients AMEXTRA/Avance: MicroMeda carries out training for clients HIRO: Has adopted Integra’s model/materials AMEXTRA/Avance: AMEXTRA/Avance: Not many clients seem reached; Should have cashier training HIRO: Has not found best methods for follow-on training Objective 4: Partner MIS: Equip partners with effective management information systems AMEXTRA/Avance: In negotiation with COBIS system HIRO: Has adopted Integra systems AMEXTRA/Avance: COBIS seems to have potential for Avance and others HIRO: Making excellent use of Integra systems AMEXTRA/Avance: System is very late in coming and program suffers for its lack Objective 5: Organizational Assessment: Enable partners to accurately and constructively assess their own organizations and plot a course for improvement based on results of the assessment AMEXTRA/Avance, EDI and HIRO: DOSA is faithfully and effectively applied AMEXTRA/Avance, EDI HQ and HIRO; All have had the DOSA applied and have learned from it Objective 6: Program Officer: Add to staff a Program Officer who will provide high quality technical services to partners and backstopping in home office Program Officer is in place, working on all aspects of program and countries Program Officer is highly qualified in MED, educates himself further, particularly on micro￾finance and is dedicated Objective 7: VP Technical Services: Expand purview and capacity of VP Technical Services to include broader array of partner services and development of home office tools The Technical Officer is on-board and working on all aspects of the DIP requirements for her position Technical Officer has designed a board development course, a strategic planning course, and an introduction to microfinance course There has been a consider￾able constraint in finding an MIS system for Mexico Objective 8: Government Accounting: Develop staff expertise in accounting and reporting for US government grants Information and financial systems installed and in use; NICRA calculated; financial administrative systems employed; staff expanded Outcomes mentioned have truly transformed EDI, allowing it to increase its reach considerably H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc14 Objective Organizational Level Key Outcomes Successes Constraints Objective 9: Grants Management and Program Monitoring: Develop staff expertise and improve internal systems for major grant management and program monitoring · Upgraded EDI’s financial management system · All USAID reporting is accurate and on-time Objective 10: Computer Network: Improve home office computer network to better serve staff need Computers purchased and all staff networked Objective 11: Strategic Planning: Develop five-year corporate strategic plan to set direction for the organization Three Year Strategic Plan formulated and revised each year as “rolling” plan Objective 12: Sustainability: Improve home office financial sustainability · Direct mail campaign be-gun in November 2001 · Second mailing in January 2002 to an expanded list · NICRA calculated Establishment of NICRA opens EDI to new sources; MG with USAID opens new private sources Events of September 11 delayed the initiation of the direct mail campaign. The decline in technology sector wealth precludes raising an endowment in the near term. Objective 13: US Public Awareness: Increase US public awareness of the work of Enterprise · Annual reports, new brochures and video produced and distributed Very professional video produced at very low costs Romania Objective 14: Establish Integra (HIRO) institutional capacity to deliver microenterprise credit and training services in Romania Association HIRO established along with SC HIRO CONS SRL, an Ltd. Company established as a lending vehicle · Staff hired and trained · MIS systems in place and working · Clients experiencing growth Lack of specific legislation for MFIs in Romania and lack of support from the Minister of Finance Objective 15: Achieve significant outreach among low-income women 25.2% of HIRO clients are living under the poverty line, which is $50 per month. The personal income of 80% of HIRO clients is under the average income in Romania, which is $100 per month. HIRO provides an opportunity for low￾income women in Bihor County to have a business financed in a county where that otherwise would not be possible. If the clients are too poor, this could have a negative impact on the program because they tend to use the revenues from business to cover family expenses. Objective 16: Implement successful business training An initial training course of 12 hours implemented. On-going training and consulting delivered to the clients by the Loan Officers especially during site visits The course is very appreciated by the women for its rich content and practicality. 100 % of the trainees rated the training as very useful for the information received and relationships created. 91 % of them said that training empowered them. There are no other courses available for small entrepreneurs. Follow-up training is more difficult to deliver to groups because the clients can’t find the time for it, even if they consider it very beneficial. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 15 Objective Organizational Level Key Outcomes Successes Constraints Objective 17: Achieve program efficiencies Percent of portfolio in default is bigger than projected due to the first loans issued and to the lack of experience of staff. HIRO corrected the problem Many lessons learned from the first year of operation have been incorporated into HIRO operations and loan requirements and problems HIRO had have been overcome The number of clients projected as targets was very high, which created pressure on the inexperienced staff in the beginning, which, in turn, caused the high early loan default rate. Objective 18: Progress toward sustainability of the MED institution Cost coverage percent was higher than targeted (42% vs. 30%). FTE interest income was higher than projected ($3,381 vs.$1,100) HIRO has recuperated some money even from the written-off loans The delinquent clients contributed to reducing both of these indicators Objective 19: Achieve significant impact 64% of the clients experienced increases in business sales in the last 12 months. The target was 75%. 64% of the clients experienced increases in the household income in the last 12 months. The target was 50% $HIRO helped many women to improve their lives and self-esteem; $HIRO created a place where women feel safe and respected; and $The program has a positive impact on the clients’ families. The economic situation is very bad. Inflation was still high (29%) and people generally can’t afford to buy much. Due to the conditions in the country, women are afraid to start bigger businesses and production because the risk would be too high Mexico Objective 20: Expand AMEXTRA’s institutional capacity to deliver MED services AMEXTRA’s capacity is expanded by: $ Avance Chalco MFI model application; $ Increased network with training and MED associations; and an $ Innovative flexible, savings-led solidarity group model for delivering microfinance services. Avance revised its model, requirements and procedures for lending and, through close monitoring, found significant improvement. Avance finds that for efficient lending and low at risk portfolio it needs to lend to higher income level borrowers and businesses than fits EDI’s initial vision. Objective 21: Establish significant program breadth Avance Chalco has spread to employment of AMEXTRA services significantly in Chalco Valley; Avance models are being employed by AMEXTRA in other parts of Mexico, rural and urban Avance has expanded its membership beyond its target and highest expectations. Indeed, increased membership is pressing Avance to make changes in MIS. The MIS currently used is inadequate for the size of Avance’s membership and loan servicing either for loan tracking or for Avance financial management. Objective 22: Establish significant program depth Women are a significant (large) part of Avance and play a key role in the program through the cashiers being primarily women All Avance’s loan officers and staff are women, except the Director. Cashiers are mostly women NOTE: Avance no longer designates women as an exclusive target borrower population H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc16 Objective Organizational Level Key Outcomes Successes Constraints Objective 23: High Program Efficiency Avance has reached 9.9% of portfolio in default by the end of Year Two EDI made a mistake in assigning the 5% target desired as an end-of-project status in Years One and Two. The target should have been ramped up. Objective 24: Progress toward sustainability High cost (84%) recovery establishes platform from which AMEXTRA/Avance can raise private funding The rate reached by the end of Year two is well above the 58% end-of-project target Objective 25: Achieve significant impact Wide geographic reach Avance has expanded in new and poor neighborhoods and had significant impact as businesses surveys show H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc17 5.1.3 Familiarity with DIP and design* Findings: (a) The DIP is used by EDI Headquarters and its partners for the following purposes: · It provides a uniform structure for reporting; · It focuses staff attention on performance indicators; and · It is a guide for targets to be achieved. (b) Headquarters staff are extremely conscious of the DIP as a strategy for achieving EDI’s goals, as a statement of EDI’s benchmark requirements, and as a roadmap for operations. They refer to it among themselves at crucial times of decision-making. Conclusions: (a) Partner managers do not appear to use the DIP and its components, particularly the indicators and a periodic reporting against them, as a guide to management or source of modifications in the program. In one case the DIP was reported as serving as a roadmap (with an implication of rigidity) to program implementation, obviating the need for an active board, closely overseeing progress, and making changes when necessary. EDI Headquarters, however, does utilize reported results against indicators as a guide to oversight and supervision vis a vis its partners. Recommendations: (a) EDI should try and instill more of a sense of ownership on the part of its partners in the program and their respective parts of it, particularly their performance indicators, by doing the following: · Involve partners more actively in the program design, particularly goal and target setting, so that partners take more ownership in plans, and · Perhaps build into the budget an “all-partner manager” multi-day workshop for DIP formulation and team building, as well. 5.1.4 Major successes and shortfalls in implementation* Highlights of major successes and setbacks from EDI’s implementation experience are shown below in Table 5.1.4. They are based on review of relevant documents, analysis of data, visits to program sites and interviews with program and partner staff and clients. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc18 Table 5.1.4: Major Successes and Shortcomings in Implementation Implementation Experience at a Glance Major Successes Major Shortcomings 1. EDI Headquarters staff and operations have benefited from the Matching Grant experience by increasing their capacity to plan further into its future, generate more funding, and, particularly, to program and support local NGOs in MED. Most of these improvements have stemmed, either directly or indirectly, from the information technology acquired by HQ, the communications it has allowed, and the reporting requirements of USAID which are made possible by the IT improvements. 1. Not enough start-up and learning allowed for in the DIP, creating too high a volume of lending without adequate and tested procedures and requirements to ensure quality loans and reduce default. 2. All parties seem to have learned from early mistakes and recovered quickly. Examples of improvements include: EDI HQ – (1) Its understanding of how USAID works and (2) allowing partners to change models when needed Avance Chalco – (1) Loan terms and maximum lending, (2) Loan Officer incentives, (3) Group requirements HIRO – (1) Loan terms and maximum lending, (2) Greater assistance to clients in business plans, (3) Eliminating solidarity groups, (4) Training prior to loan approval 2. Not providing an adequate loan tracking and financial management software to Avance Chalco has not only restrained its growth in lending but, more importantly, leaves the program’s records vulnerable and causes significant program inefficiencies. Secondary negative effects of this are that the Program Coordinator has been required to spend inordinate amounts of time on loan tracking and financial management because of a break between financial managers. (See Section 6.4.) It should be noted that the development of a flexible, cost-effective MIS is a chronic difficulty in MED work around the world. Negotiations are currently in the final stages with Microbanx for use of a web-hosted version of its COBIS software. 3. EDI has remained faithful to its vision of helping poor women gain a living and support families through business initiatives in the face of realities which argue for a bias towards lending to more established and solvent entrepreneurs. 3. Both programs have suffered from regulatory uncertainty. Changing regulations ( in the case of Mexico) and unclear regulations (in the case of Romania) have necessitated a re-registration process for both organizations – a process which has had a high cost in terms of the Program Directors’ time and energy. 4. The EDI program seems to be extremely efficient. Essentially one Program Manager, supported on a part￾time basis by financial and administrative staff, supports two micro-finance development projects, each of which was new and each of which is successful after two years, based on visits to representative cross-sections of clients. (See Section 7.7.3) 4. EDI should make a greater effort to provide on￾site technical assistance to its partners in capacity building. There should be closer direct support to partners and assistance in solving problems. 5. USAID management, strategic planning and performance monitoring methods have become institutionalized in EDI. 5. EDI has not done as much as it can in assisting partners in board capacity building through such things as implementation of its board training seminars used at HQ for partners. Implementation of this seminar at 2 non-Matching Grant partners is scheduled for 2002. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 19 5.1.4 Impact Results* Findings: (a) The DIP contains impact indicators some of which have been categorized as “sustainability” measures and some as “impact” measures. These include: For EDI Headquarters – fi Percent of endowment goal raised fi Number of individual donors fi Revenue from institutional giving fi Marketing of client products For Avance Chalco – qua sustainability -- fi Percent of cost coverage fi Interest income per staff member qua impact – fi Percent business owners with increased sales in last 12 months fi Percent business owners with increased household incomes in last 12 months For HIRO – qua sustainability – fi Percent of cost coverage fi Interest income per staff member qua impact – fi Percent business owners with increased sales in last 12 months fi Percent business owners with increased household incomes in last 12 months (b) EDI, HIRO and Avance are efficient in the sense that they provide effective microfinance development services at very low costs. Staff rates are low. Staff travel only when necessary and inexpensively, both in travel and lodging. However, in the cases of both HIRO and Avance, their efficiency indicator targets are all underachieved. (See Section 5.1.2 above and, particularly Conclusion C.) Conclusions: (a) As is shown in Annex B, one of EDI’s HQ sustainability measures have not been achieved. (b) For Avance Chalco, three out of the four sustainability and impact measures have been achieved in Year Two. (c) For HIRO, also, three out of the four sustainability and impact measures have been achieved in Year Two. 5.2 Assessment of project model and hypotheses 5.2.1 Project hypotheses articulated in CA* Findings: The hypotheses and models of the EDI program, although not expressed in program documents as such, can be constructed from program documents in the following way. From EDI’s Strategic Plan for 2001 – 2003, the “Organization Description and Mission” are described as follows: H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc20 “Enterprise Development International is a Christian nonprofit 501 (c)(3) organization serving the poor in the United States and around the world. Enterprise raises funds and provides financial and technical support to programs implemented by locally registered organizations. Through these partnerships, poor families are equipped to support themselves and strengthen their communities. The Mission of EDI is to demonstrate God’s love for the poor by enabling them to free themselves from poverty. The mission is accomplished in the following way: · Providing business training to low-income persons who have the potential and desire to become self-supporting; · Distributing small loans to poor entrepreneurs who have viable business ideas but need capital; · Mentoring participants through on-going personal and professional encouragement.” EDI’s Matching Grant application amplifies the hypothesis and model as expressed, within its “Rationale for Undertaking the Program” section, in the following paragraph: “Enterprise strongly believes that the best way to address poverty in the developing world is through sustainable economic development programs...The proposed program is designed to improve the social and economic status where the poor, in particular very poor women, have a great need for job creation, income generation, and overall community development.” Most succinctly, although in less detail, the EDI hypothesis is expressed in its CA in the following way: “The goal of the program is to enhance the income, social status, health, and welfare of very poor women and their families in Mexico and Romania. This program attempts to address poverty alleviation through a sustainable microenterprise development program that will focus on job creation and income generation for impoverished women and men...The program’s purpose is to develop the sustainability of (EDI’s) affiliate organizations by enhancing their institutional capacity, training, and technical development.” 5.2.2 Replication and scale-up of approaches in project area or elsewhere* This sub-section covers various aspects of model replication. It discusses EDI models and approaches which have been replicated, modified or have not been replicated at all by its partners. It also discusses key principles of EDI which have been successfully or unsuccessfully applied by its partners. It also contains observations and conclusions about why replication has succeeded to the extent that it has. Findings: (a) Replications and scaling up of Enterprise Development programs which have taken place over the course of the Matching Grant include: fi Replication of the Avance Chalco model in Chiapas – a combined effort of AMEXTRA, the Kellogg Foundation, and Enterprise Development. fi Integra is using the lessons it has learned about microenterprise development in Romania through the Matching Grant and applying them to an expansion of its work, in the southeast of the country. fi Enterprise Development has signed a new partnership agreement with an organization in Guatemala and has used many MG lessons to design that program. Fundraising for that work is underway and lending operations are due to begin in mid-2002. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 21 fi Enterprise Development has significantly scaled-up its work in India and the Philippines during the period of the Matching Grant, again drawing on program lessons and home office technical expertise gained through the MG. (b) The community transformation bank model, utilizing two-tiered fellowship groups for support, learning and mutual responsibility, has not been followed strictly by either partner, with the agreement of EDI. Mexico does use solidarity groups but not in two￾tiered fashion. From the beginning HIRO, did not choose to adopt the savings aspect of the model. Moreover, Romania has abandoned solidarity groups, finding early on that clients do not trust each other sufficiently to work in groups and that they are more in competition with each other than in cooperation. (c) The loan tracking and financial management software and model used by MikroFond, Integra’s partner in Bratislava, is used by HIRO in Romania. MikroFond provided training and technical assistance to HIRO staff in its installation and application. Conclusions: (a) Although the community transformation bank model is one which EDI applies in its international practice and is one it believes in, it does not require its Matching Grant partners to apply it either strictly or at all. The HIRO program, since it does not offer savings, doesn’t apply. Mexico has applied the model to a certain extent as described above. (b) The MikroFond MIS/FIS systems used by HIRO are among the strongest points of HIRO’s operational success. 5.3 Advocacy under the project 5.3.1 Advocacy activities and impact* Advocacy is not included as a goal, purpose, or activity in the EDI Cooperative Agreement. Consequently, no systematic research was done on this issue. However, EDI partners did find it necessary to undertake advocacy activities. In Mexico, the problem being addressed through advocacy is that AMEXTRA, the sponsoring institution under which Avance Chalco operates, is a social NGO, while Avance is acting as an economic NGO. If an NGO raises and handles money, it is regulated differently than a social NGO. The problem in Romania with HIRO is that it was originally registered as a social NGO. Changes in and uncertainties about national legislation regarding NGO regulations have made it unclear whether HIRO, which partially acts as an economic NGO providing loans, fits properly. Each partner has determined different courses of action, through the options each learned in their networking. Avance has decided to either restructure itself within AMEXTRA or to split off from AMEXTRA, while HIRO has registered itself as an economic NGO for the purposes of lending. Realizing that the issue of advocacy was important to PVC, the evaluator did gather information on advocacy whenever such activities were identified. In the interests of informing PVC about these activities and outcomes, they will be discussed here in terms of lessons learned. Findings: (a) Avance Chalco has become involved with the Mexican association of microfinance organizations, PRODESARROLLO, and other NGO associations such as Red de H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc22 Finanzas Sociales (Network of Social Finance), to inform itself and to help inform the Government of Mexico about the issues, pros, and cons involved in the current debate over microfinance, NGO registration legislation, and regulations. (b) HIRO has participated with the Transylvania Business Forum (TBF) and the nascent association of Romanian micro-finance institutions in meetings and conferences to discuss current NGO registration and regulations and to enter into dialogue with the Government of Romania. (c) HIRO and Integra have taken a leading role in founding an association, as yet unformed and un-named, of Romanian micro-finance institutions. Integra, together with TBF, participated in an Ethics in Government seminar and, based on that experience, are preparing a proposal for funding from Canada and Great Britain to support its establishment. Conclusions: (a) This networking has been productive in terms of informing EDI’s partners about the status, issues and options in NGO registration and regulations as they affect Avance and HIRO. They do not, however, appear to have had much influence on how registration legislation and regulations might end up being reformulated. (b) The information about options obtained in discussions with other NGOs has been beneficial to EDI’s partners in enabling them to formulate strategies of their own and to take actions regarding their registration which will allow them to operate regardless of the outcome of national debate and consequent regulations. This information was gained through networking with the purpose of influencing the national dialogue. (c) HIRO’s involvement itself with the Transylvania Business Forum and its attempt to form a micro-finance institution association as it is now doing, are significant steps in a society that is used to “top down” directives rather than “bottom up” lobbying. Recommendations: (a) Avance and HIRO should continue in the relationships with PRODESARROLLO and the Transylvania Business Forum, as well as other associations and NGOs in order to further strengthen their registration and protect their forms of operation. 5.3.2 Partner/PVO roles in advocacy* -- N/A 5.4 Implementation Lessons Learned This section is designed to include tools, approaches and methodologies developed by the program that have benefited or could benefit other MED projects or institutions. Findings and Conclusions: (a) COBIS software – EDI is negotiating testing and using this new microfinance loan tracking and financial management package in Mexico with Avance Chalco. (b) HIRO’s re-structuring and “double registration” could be useful for other NGOs not only in Romania but in Eastern Europe through the Integra and OI network. (c) Integra is applying the DOSA training in its Bratislava project. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 23 6.0 PARTNERSHIP QUESTIONS 6.1 Analysis of Partnership Schemes* Findings: (a) Ostensibly the partnership arrangements and relationships in the EDI program are straightforward. EDI supports Avance Chalco and HIRO through AMEXTRA and Integra respectively. There are two examples of local subcontracting: fi In Mexico, where Avance subcontracts Micro MEDA Mexico, the local affiliate of the Mennonite Economic Development Associates (MEDA), to carry out client training in Microenterprise Capacity Building and fi In Romania, where HIRO was supposed to subcontract the Transylvania Business Forum to provide training to HIRO staff in the early stages of its development. The training was not carried out. (b) Micro MEDA functions completely independent of Avance. It has its own curriculum, which has not been reformulated for the EDI program. It provides its own training materials, although Avance takes responsibility for reproducing the copies. (c) There are relationships between the various international partners – EDI, Integra, MEDA, AMEXTRA and OI – that are outside and precede the EDI Matching Grant program. All institutions are oriented by Christian values. They cooperate, attend conferences, and share methodologies and experiences with each other. They know each other, as institutions, quite well. · There are relationships, outside of the Avance – EDI and HIRO – EDI relationships, between EDI and AMEXTRA and Integra. EDI has a second program with AMEXTRA in Chiapas. EDI has another program with Integra in Slovakia. · There is, also, a common relationship held by both Integra and EDI with Opportunity International (OI), headquartered in Chicago. That relationship is one of staff, and consequently, experience and ideas stemming from OI into EDI and Integra. Conclusions: (a) The common organizational cultures and shared Christian values is particularly strong “mortar” for cementing these institutions and enhancing the opportunity for program success even when conditions might get difficult. (See above under Section 5.2.2.) 6.2 Measuring Institutional Capacity* Findings: (a) EDI has set up a monthly reporting routine and format for its partners consisting of three parts: Program Summary with textual description of activities carried out, problems encountered and/or resolved, etc.; Program Data with training, loan tracking and portfolio performance data; and Financial Accounting with management data. (b) Through its monthly reports and sometimes direct sharing of data via internet, EDI follows all indicator results explicitly, i.e. each indicator is reported on. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc24 (c) EDI has done some specific assessments – early loan default problems and the adequacy of WOF for training in Mexico and Romania -- once problems have been identified. In most cases EDI leaves the analysis and conclusions to the local partner. (d) EDI has set the DOSA in place in Mexico and Romania as a way to enhance local measurement of capacity (self assessment) by facilitating one DOSA session for each partner. (e) EDI makes visits to its local partners, with the Program Officer making four visits to Romania and four to Mexico. (f) EDI’s Director of Communications has made visits to each partner to document “success stories.” (g) EDI relies on its sponsoring institutions -- AMEXTRA and Integra -- for monitoring and supervision. Integra staff visit partners more frequently and are in closer communication than EDI staff, while Avance appears to have about as much contact with EDI as it does with AMEXTRA . (h) EDI keeps informed about partner activities and performance through its network of related institutions (see Section 6 above). Conclusions: (a) EDI and its partners have reacted quickly to indicator results, making quick and effective changes when the need was identified, as with the loan default problems arising early in the experience of Avance and HIRO. (b) EDI appears to follow and be more concerned with loan tracking and financial management (efficiency and sustainability) indicators and less concerned with organizational management and capacity building indicators. Training of partner staff and clients, for example, and technical assistance and board development for partners could be provided to a greater extent although board development and staff training are priorities for EDI TA this year under the Matching Grant. It is as if EDI and its partners have been most focused on making sure the models works and the programs operate, remembering that both started from zero. (See Recommendations below.) (c) EDI has been slow to deliver on the need by Avance of an MIS system that will allow it to adequately track loan information. (See Section 5.1.4 above and Section 6.4 below.) (d) Leaving as much discretion and authority in the hands of its local partners in cooperation with their regional sponsors, as described in (g) above, EDI has not always provided adequate oversight of partners. For as capable and dedicated as partner managers are, sometimes they have not known what to do. While the major problems of early loan default were addressed by both EDI and its partners effectively, lower order needs that may not be showing up yet in the performance indicators may be being overlooked until they become big problems. Possible examples, worthy of attention and oversight, include: fi In Mexico, the need for MIS, financial management in general, and the building of staff capacity to match the rapidly growing portfolio; fi In both Romania and Mexico, the need for board development and partner reactions to the uncertainty of the current regulatory environments. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 25 It should be noted that the delegation of authority is part of EDI’s methodological approach, making decisions is an important aspect in developing ownership and effective responsibility for local programs, which in terms re-enforces partner sustainability. (e) The supervisory visits carried out by the Program Officer are proper in number and timing, but the Evaluator concludes that they are inadequate, possibly in their duration; the issues that are focused on; the technical assistance partner management might be given during the visits; or the planning and coordination that could be done with AMEXTRA and Integra.1 (f) The oversight and supervision of local partners by their sponsoring institutions varies in effectiveness. The Integra Program Advisor, who is partially paid for by EDI under the Matching Grant, makes many visits, has adequate communication, and gives good direction to HIRO. AMEXTRA appears to leave Avance more on its own, Avance having more contact with EDI directly regarding. (g) Regarding results of the DOSA -- fi The DOSA was facilitated for HIRO only in September, 2001. It generated twelve “action items” of which none have as yet been addressed. fi The DOSA was carried out in Mexico in June, 2001, and it generated four “action items” of which two have been implemented and one is being completed. Recommendations: (a) EDI should devote more attention to training and technical assistance to partner staffs. Since delegation of authority is important to the EDI-partner relationship (see sub-section 7.4 (c) below), giving partners the necessary tools is also important. Specifically: fi Board training seminars that EDI has developed and implemented for its own board should be given to its partners and partner boards, which are currently being expanded and re-structured for both Avance and HIRO; EDI notes that it currently has plans for carrying out board training for Integra/Slovakia, and its partner in Nicaragua. fi Partner staff training in Mexico should be given somewhat more attention. The courses at New Hampshire College in MED and at the Microfinance Center in Poland for loan tracking and financial management that have been provided through the grant have been effective. The Finance Manager in Avance, who, although an accountant, is new as of November, 2001, and could use training in loan tracking and 1 This conclusion was based on evaluator observations and analysis and was developed into a hypothesis – that EDI’s partners, being newly formed, sometimes felt “out on a limb” or in unfamiliar territory – around the general issue of how to deal with the local partners’ registration. For both Avance Chalco and HIRO this issue brought up several thorny decisions and actions the partners needed to take. In order to test the hypothesis, the evaluator asked several different kinds of questions to various partners, including the Program Officer and other EDI Headquarters staff, designed to identify the depth and breadth of “supervision”. The evaluator was particularly interested in the dimensions of “supervision” which might also be termed “support” or “direction” that EDI might have provided to partners at times when they, being newly organized, might have been in doubt or uncertain. The local partners always felt that they had handled the issues well, or as best they could, and that they did not expect any further support or direction from EDI. They generally kept EDI closely informed and acknowledged that EDI Headquarters did help them as much as possible. The local partners never expressed any lack of satisfaction or feeling that EDI should have done more for them. EDI Headquarters staff, as was noted in the report, felt that to some extent the process of local partners learning on their own was part of developmental value. So any lack of supervision, defined as “support” and “direction”, was a conclusion of the evaluator’s and should be viewed in juxtaposition with EDI’s management approach of delegated decision￾making and problem solving. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc26 financial management. Moreover, once the COBIS system is installed, the Project Coordinator as well as the Finance Manager will need training in its application. 6.3 Constraints to Partnership* Findings: (a) The constraints to EDI’s relationship with its local partners include: fi Distance – EDI is very careful about the money it spends on travel to the field. (b) The constraints in the relations between regional and local partners include: fi AMEXTRA has overlapping but broader goals than Avance Chalco, making oversight and assistance difficult, in that AMEXTRA does not do lending and saving. Since AMEXTRA’s methodological approach does not include economic mechanisms for community development like savings and lending, their objectives are not always consonant; fi In Mexico – Avance is outreaching and overtaking AMEXTRA in the Chalco Valley in terms of need for physical space and coordination with other community center activities; and fi Integra seems to have a very close fit, in terms of its methods, expertise and objectives, with HIRO, making for a close and effective partnership. Constraints were not found by the Evaluator. 6.4 Information Technology* -- N/A Since neither EDI partner existed before the CA it is not possible to evaluate its use of and capabilities in information technology in a before and after sense. EDI Headquarters, on the other hand, has experienced an information technology improvement described in terms of “light years” by EDI staff and verified by the Evaluator. Much of the discussion above, in several sections, related to financial management and capacity building at Headquarters. 6.5 Use of local networks and service organizations* -- N/A 7.0 PROGRAM MANAGEMENT 7.1 Strategic Approach and Program Planning* Findings: (a) EDI has adopted new methods and carried out several activities that have improved its ability to plan, not only in its international program but also in its domestic activities. fi It designed a seminar for its board to carry out strategic and program planning; fi It carried out an outside evaluation that was aimed at informing as well as improving its own planning; fi It formulated its first three-year plan as part of the Matching Grant; and H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 27 fi EDI carried out the DOSA, which is serving as a guide to and baseline for future actions EDI is planning. (b) The improvements in its information systems, described in Section 6.4 above, have improved EDI’s ability to plan and adjust its plans. Conclusions: (a) The increased ability of EDI to plan for the future, reinforced by significantly improved financial and administrative management systems allowing for better reporting and projecting, has been so remarkable that it is referred to among the significant successes of the Matching Grant. 7.2 Country Initiatives* -- N/A 7.3 Conflict Management* -- N/A 7.4 Monitoring and Evaluation* Much has already been said about EDI’s management of the program. Below are presented summary findings, conclusions, and recommendations about its overall management, with references to the appropriate sub-section for more details. Findings: (a) There is a bond between EDI and its partners of shared values which holds the partnership together, beyond the functional and financial relationship. This bond reinforces trust and fundamental understanding amongst all partners. It makes program management easier and more effective. (See Section 5.2.2 above.) (b) EDI has adopted new methods and carried out several activities which have improved its ability to plan (see Section 7.1 above): · It designed a seminar for its board to carry out strategic and program planning; · It carried out an outside evaluation that was aimed at informing as well as improving its own planning; · It formulated its first three year plan as part of the Matching Grant; and · EDI carried out the DOSA which is serving as a guide to and baseline for future actions EDI is planning. (c) The improvements in its information systems, described in Section 6.4 above, have also improved EDI’s ability to plan. (d) Financial management in general in EDI’s Headquarters includes all necessary elements for effective financial control. (See Section 7.7.1 below.) (e) EDI’s program is efficient (see Section 7.7.3 below): · EDI’s current indirect costs are extremely low at 18.51%; H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc28 · EDI’s direct costs for personnel are also extremely competitive. All costs, and particularly salaries, are intentionally kept low so as provide more services to program clients; · The Matching Grant program is carried out by one full-time Program Officer support by part efforts from EDI’s VP for Technical Support, the Director of Finances, the VP for Development and the VP for Programs; and · As of September 30, 2001, administrative and fundraising expenses represent only 9.2% of total expenditures. Conclusions and Recommendations: (a) Regarding results of the DOSA -- fi The DOSA was facilitated for HIRO only in September, 2001 by the EDI Program Officer. It generated twelve “action items” of which none have as yet been addressed. These action items included: · Need to improve staff skills in impact surveys and assessment; · Contingency measures needed for lack of operational funds; · Develop an operational cash reserves policy; · Longer terms for bigger loans need to be offered to clients; (this is being considered) · Board more involved in the strategy of the agency; · Better communication between board and agency (board empowering staff); · Better understanding of strategic management; · Better top-down communication from board to staff; · Connect mission and objectives; · Better communication with general public; · Linkages with private sector (develop a plan); and · More formal application of impact survey. fi The DOSA was carried out in Mexico in July, 2001 by the EDI Program Officer, and it generated four “action items” of which two have been implemented and one is being completed. These action items were: · Hiring of new staff, including an accountant and cashier; · Defining the specific mission and objectives of the organization; · Creating a strategy for developing relationships with the private business sector in the Chalco Valley, in coordination with AMEXTRA and Enterprise Development; and · Determining the prudent level of operating reserves for the program. 7.5 Overall Management* Much has already been said about EDI’s management of the program. Below are presented summary findings, conclusions, and recommendations about its overall management with references to the appropriate sub-section for more details. Findings: (e) There is a bond between EDI and its partners of shared values, which holds the partnership together, beyond the functional and financial relationship. This bond H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 29 reinforces trust and fundamental understanding amongst all partners. It makes program management easier and more effective. (See Section 5.2.2 above.) (f) EDI has adopted new methods and carried out several activities which have improved its ability to plan (see Section 7.1 above): · It designed a seminar for its board to carry out strategic and program planning; · It carried out an outside evaluation that was aimed at informing as well as improving its own planning; · It formulated its first three year plan as part of the Matching Grant; and · EDI carried out the DOSA which is serving as a guide to and baseline for future actions EDI is planning. (g) The improvements in its information systems, described in Section 6.4 above, have also improved EDI’s ability to plan. (h) Financial management in general in EDI’s Headquarters include all necessary elements for effective financial control. (See Section 7.7.1 below.) (e) EDI’s program is efficient (see Section 7.7.3 below): · EDI’s current indirect costs are extremely low at 18.51%; · EDI’s direct costs for personnel are also extremely competitive. All costs, and particularly salaries, are intentionally kept low so as provide more services to program clients; · The Matching Grant program is carried out by one full-time Program Officer supported by part-time efforts from EDI’s VP for Technical Support, the Director of Finances, the VP for Development and the VP for Programs; and · As of September 30, 2001, administrative and fundraising expenses represent only 17.7% of total expenditures. Conclusions and Recommendations: (a) EDI and its partners have reacted quickly to indicator results, making quick and effective changes when the need was identified, as with the loan default problems arising early in the experience of Avance and HIRO. (b) Since delegation of authority is part of EDI’s methodological approach (see Section 6.2 above): · EDI should devote more attention to training and technical assistance to partner staffs. Specifically board training seminars, client training, and partner staff training. · EDI should provide greater direct supervision of its partners and their sponsoring institutions, including coordination and planning on a periodic and institutionalized basis and longer oversight visits. (c) The increased ability of EDI to plan for the future, reinforced by significantly improved financial and administrative management systems allowing for better reporting and H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc30 projecting, have been so remarkable that it is referred to among the significant successes of the Matching Grant. 7.6 Sustainability* Finding: (a) As was described above under Impact Results in Section 5.1.5, EDI has scored well on its sustainability indicators. (b) Both HIRO and Avance are preparing themselves for the possibility that a second Matching Grant will not be awarded to EDI and that they will have to continue on their own. To that end HIRO and Avance have taken several steps: fi EDI is making a strong effort to resolve the MIS problem in Mexico; fi Avance Chalco and AMEXTRA are clarifying their relationship and are formulating Avance’s restructuring, given expected changes in national legislation regarding NGOs; fi Avance is expanding membership in its board; fi HIRO is expanding and widening the role of its board; and fi HIRO has recently registered its lending operation as a limited company. Conclusions: (a) Both HIRO and Avance have reformed loan requirements, loan officer incentive schemes, and taken other measures that have proven successful in reducing portfolio at￾risk such that they will meet their targets for operational sustainability by the end of Year Three. Avance will significantly pass its target, possibly to the extent of achieving 100% operational sustainability. 7.6.1 Overall sustainability survey* -- N/A 7.7 Financial Management Much has already been said about EDI’s financial management of the program. Below are presented conclusions about its overall financial management with references to the appropriate sub-section for more details. 7.7.1 Effectiveness of financial management* Findings: (a) Financial management in general in EDI’s Headquarters includes all necessary elements for effective financial control. (b) At HIRO, the only weaknesses to financial management are the following: H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 31 fi Cash is handled – received, noted and recorded and kept – by several officers including loan officers, the Cashier and the Executive Director and fi The program loan data system and the Avance financial management system are not linked and the data for each are kept on different computers. (c) At Avance, the following weaknesses have been identified, and partially discussed above: fi The program loan data system and the Avance financial management system are not linked and the data for each are kept on different computers; fi The expected COBIS MIS system will contain loan tracking as well as financial management components that will be integrated; fi Cash is received and stored in Avance’s office, which is housed in the AMEXTRA community center in circumstances extremely vulnerable to theft ; and fi The Avance Accountant is new and learning her responsibilities. Conclusions: fi Only the problems in Avance represent a concern for the overall financial management of the EDI program. Recommendations: (a) Ensure that the COBIS system is obtained and installed. (b) Train the new Avance Accountant and give her a clear set of responsibilities. (c) Provide a computer with enough memory to hold and manipulate all program data. 7.7.2 Leveraging other donor funds* -- N/A 7.7.3 Cost effectiveness of technical approach* Findings: (a) EDI’s current indirect costs are extremely low at 18.51% (b) EDI’s direct costs for personnel are also extremely competitive. All costs, and particularly salaries, are intentionally kept low so as provide more services to program clients. It was explained to the Evaluator and verified through the two field trips that this practice stems from the core Christian values followed by EDI and its partners. (d) Although a quantitative analysis was not undertaken by the evaluator, the Matching Grant program is carried out by one full-time Program Officer support in part by efforts from EDI’s VP for Technical Support, the Director of Finances, the VP for Development and the VP for Programs. (e) As of September 30, 2001, administrative and fundraising expenses represent only 17.7% of total expenditures. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc32 Conclusions: (a) It appears, on an intuitive level to an Evaluator with considerable experience, that the $649,990 awarded to EDI to be spent over a three-year period in Romania and Mexico is having a very large impact. 7.7.4 Repercussions of “matching” requirement on program* Findings: (a) The fact that a major international institution is investing in it gives EDI high credibility among its corporate and individual donors. (b) The Matching Grant program has allowed EDI to have broader coverage and has provided it with more “success stories,” making for good public relations and development. EDI has been able to meet its matching commitment through contributions from individuals, family foundations and, in the case of Integra/HIRO, a grant (directly to Integra) from the Shell Foundation. The “matching” aspect of the program has been attractive to several of these donors. Rapid growth in Avance Chalco put strains on Enterprise matching funds in December 2001, especially given decreased donations after the events of September 11. However, in the end, Enterprise was able to raise the necessary funds for Amextra during this time, due to a few timely end-of-year donations. Conclusions (a) The Matching Grant has made raising outside funds easier. (b) While the Matching Grant has helped Enterprise with individual donors and family foundations, the youth of the programs has prevented EDI from accessing big corporate and institutional donors (i.e. major foundations) as EDI had hoped, at least not during the first two formative years of HIRO and Avance. However, Integra was able to win a significant grant from the Shell Foundation for HIRO. (c) The grant has provided EDI with its two most marketable programs regarding general fund raising as well as fund raising for the USAID match. 7.8 PVO’s Information Management* Findings: (a) As is described above in Section 6.2, both HIRO and Avance submit monthly progress reports to EDI which contain three sections: fi Program Summary with textual description of activities carried out, problems encountered and/or resolved, etc.; fi Program Data with training, loan tracking and portfolio performance data; and fi Financial Accounting with management data. (b) EDI submits annual reports to PVC which follows a PVC-prescribed format; one that reflects the DIP. It contains the following sections, each divided into sub-sections covering Mexico, Romania and Headquarters: H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 33 fi Background and Project Context; fi Project Methodology; fi Monitoring and Evaluation; fi Review and Analysis of Headquarters/Support Functions; fi Financial Report; fi Lessons Learned and Long Term Project Implications; and fi Attachments. Conclusions: (a) The progress reporting is appropriate in content and timeliness. EDI has been able to both note and follow problem issues, operations, and financial performance. Follow-up communication is carried on most frequently via e-mail and telephone. (b) The annual reports to PVC report on performance indicators and target achievement, with explanations and clarifications where needed, should serve PVC as comprehensive evaluative progress reports identifying successes, problems, and actions to be taken. 7.9 Logistics* -- N/A 7.10 Project Supervision* Findings: (a) EDI’s formal agreements are with AMEXTRA and Integra and give oversight and supervisory responsibilities for the execution of the EDI program in Mexico and Romania to them. (EDI has a formal agreement with HIRO only because HIRO’s bank requires it.) EDI took the primary role in designing the program and works in close consultation with AMEXTRA and Integra in its execution. (b) On a more informal level, EDI has a close operational relationship with Avance Chalco mainly because its Program Coordinator is an EDI representative. (c) In each case, EDI provides support and assistance directly to HIRO and Avance when needed. However, as was described and discussed above under Section 6.2, most supervision of local partners falls to the sponsoring institutions. 7.11 USAID Management* Findings: (a) EDI was new to USAID as a grantee with this CA. Its knowledge of USAID procedures and familiarity with USAID’s organizational culture was negligible. (b) EDI staff took various seminars at USAID and in PVC to familiarize itself with USAID management. (c) When EDI asked for clarification from its initial Contracts Officers, it received quick and informative responses. Some months into the CA, however, the Contracts Officer H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc34 changed. The current Contracts Officer is found to be uncommunicative and unresponsive. As a consequence, EDI has asked fewer questions and increasingly tried to figure things out on their own. (d) EDI has found that communication with the Technical Officer, who has not changed over the course of the CA, has been helpful and supportive. (e) As EDI enters its third year under the Matching Grant, it feels more confident of its understanding of USAID and how it works. 8.0 CONCLUSIONS AND RECOMMENDATIONS* The overall conclusions of the evaluation are presented below in outline form since the details of each have already been presented above. 1. The direct capacity building services from EDI to local partners is limited. As was noted above in Section 6.2, EDI delegates much decision-making and problem resolution to its partners. The Evaluator has found situations, however, where somewhat more support and assistance to HIRO and Avance are warranted. These are described in summary below. 1.1. Regarding EDI training of partner staff: fi Strategic planning or team building seminars have not been provided to the boards of either Avance or HIRO and yet they could be, since EDI has developed them; fi Methods and materials for client training, which were supposed to have been provided through Women’s Opportunity Fund (WOF), but instead have been replaced in Mexico under a subcontract with Micro-MEDA and in Romania by materials developed by ISVOR/OI in Slovakia, do not appear to the Evaluator to be reaching many clients or increasing the skills of those who do take them; and fi The staff training and technical assistance described in the Application and for which the Transylvania Business Forum was subcontracted was not provided. 1.2. Regarding support in MIS to EDI partners: fi Support to Avance Chalco weak and late, although it appears to have been overcome in January, 2002 and fi Support to Romania good but comes through Integra. 1.3. Regarding capacity building in performance monitoring, the only model passed on to local partners is the Planning Matrix and its indicators and there is not much training on how to use or modify them. 1.4. Regarding the role of sponsoring partners in capacity building, much more capacity building and support comes from AMEXTRA and Integra: fi AMEXTRA – administrative support, strategic planning, models, networking fi Integra – models, MIS software and financial systems, networking 1.5. Regarding local partner problems with registration, EDI has not been able to be of much help regarding the fit and adjustment of local partners and their registration as institutions to local laws and regulations, their changes and uncertainties (see below). 2. Local partners are successful and sustainable because of: 2.1. The loan funds themselves which provide fi incentives to client participation and fi influence by local partners to strengthen client businesses; 2.2. The models that are applied; 2.3. The shared organizational values (see below); and 2.4. The excellent recruitment and quality of staff. 3. EDI Headquarters has increased its own capacity greatly from USAID link: H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 35 3.1. In its MIS and particularly financial systems; 3.2. Through its application of the DOSA; 3.3. Through strategic planning and other methods/models required by USAID; 3.4. Through its calculation of its NICRA as prescribed for government grants; and 3.5. Through fundraising capacity. 4. The common foundation of Christian motivation and values among all EDI’s partners is very beneficial: 4.1. As a “glue” that provides trust and solidarity between partners as well as a network for their support and coordination; 4.2. Through the lower costs offered by selfless motivation; and 4.3. Through steadfast attention to the quality of end products and services provided to clients. 5. All parties seem to have learned from early mistakes in the CA and to have recovered quickly: 5.1. EDI generally in its understanding of how USAID works; 5.2. Avance Chalco as it encountered higher than anticipated default rates, and overcame them through modifications in procedures and requirement, such as -- fi Loan terms modification and maximum lending reduction; fi Loan officer incentive remuneration; fi Group requirements; and fi Additional loan officers hired. 5.3. HIRO also had high portfolio servicing problems and overcame them successfully through -- fi The elimination of solidarity group lending; fi Training before loan acceptance through a three-day business seminar; fi Business plan development and acceptance before loan acceptance; fi Lowering of loan size; and fi Stricter requirements. 6. Both local partners have had problems with how they are registered and in their ability to understand and adapt to changes and uncertainties in national laws regarding NGO registration: 6.1. Both have had to, in HIRO’s case, or is seriously considering, on the case of Avance Chalco, moving to some form of limited incorporation as opposed to not-for-profit status to better enable their lending practice and 6.2. Both organizations have found it necessary to carry out advocacy activities and engage in networking to influence legislation and regulations, as well as to keep informed. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc36 ANNEXES H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc37 37 ANNEX A: KEY EVENTS TIMELINE* Tabular form FY99: 1-Oct-98 to Sep-99 FY 00: 1-Oct-99 to 30-Sep-00 FY01: 1-Oct-00 to 30-Sep-01 Date Event Date Event Date Event Avance Chalco legal registration process Nov 1998 Outside evaluation of EDI recommending USAID matching grant application Jan 2000 OI mentoring exercise Oct 2000 Enterprise Development Strategic Plan ratified by Board of Directors Jan 2000 Director of HIRO hired Nov 2000 Years 2 and 3 budget revised Dec 1998 Proposal submitted March 2000 DIP Submitted May 2000 DIP Review June 2000 Partner Program Directors trained in US June 2000 HIRO legal registration July 1999 Matching Grant award announced July 2000 Update of HQ DIP indicators June 2001 Program Officer training in New Hampshire July 2000 DOSA exercise at Enterprise Dev’t. HQ June 2001 OI Mentoring Excercise August 2000 Survey of partners July 2001 NICRA updated August 2000 WOF training in Mexico July 2001 DOSA applied in Mexico Sept 2000 HIRO begins training and loans Sept 1999 EDI Matching Grant CA awarded Sept 2000 OI mentoring exercise Sept 2001 DOSA applied in Romania FY02: 1-Oct-01 to 30-Sep-02 Date Event Oct 2001 Year 3 budget revised Jan 2002 Final EDI evaluation for PVC H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc38 38 Key Events Timeline in Graphic Form 1998 1999 2000 2001 2002 October November December January February March April May June July August September October November December January February March April May June July August September October November December January February March April May June July August September October November December January February March April Evaluation recommends USAID MG application Proposal submitted Enterprise Development notified of Matching Grant award Enterprise Development Matching Grant CA awarded OI mentoring exercise; HIRO Director hired DIP submitted DIP review Partner Directors trained in US; HIRO legally registered DOSA at EDI HQ; Update of HQ DIP Indicators Survey of partner TA priorities; WOF training in Mex. HIRO begins training and loans; OI mentoring EDI Strategic Plan ratified by Boar d MG budget revised for years 2 and 3 Program Officer training in NH; OI mentoring NICRA updated; DOSA applied in Mexico DOSA applied in Romania Year 3 budget revised Final EDI evaluation for PVC H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 39 ANNEX B: DETAILED IMPLEMENTATION PLAN TABLE* Headquarters: Goal/Objective/ Activity Indicator End of Project Target Accomplishments Data verified?2 Explanation for Change in Indicator or Target, if any3 Target Met? Goal: Improve Enterprise’s capacity to provide high quality, sustainable technical assistance to implementing partner organizations. Technical Capacity: Objective 1: Board Development: Equip local boards to improve their capacities in organizational vision, leadership, oversight and representation Partners improving their Boards in identified areas of weakness Conduct and evaluate 2 seminars with revised materials Year Two target was to complete 1 seminar. EDI’s board training seminar, “Developing a Great Board” has been completed. The implementation of the second board seminar, originally scheduled for Sept. 2001, was postponed until early 2002. Y Neither indicator nor target changed Yes for Year 2 Activities for Obj. 1 (a) Study other institution board training materials (b) Design a strategic planning course for the board Objective 2: Partner Staff Training: Train local staff in all aspects of program implementation, management, evaluation, growth and accountability (including impact assessment) Increased effective￾ness of partners in areas in which they received training Conduct and evaluate 2 training events w/ revised and expanded materials ·Completion of survey of partner training needs. ·Completion of training tools ·Delivery of a training on impact assessment to Avance Chalco. Y (records) Y Y Neither indicator nor target changed Yes for Year 2 Activities for Obj. 2 (a) Design and carry out a survey and EDI partner needs for staff training (b) Design training materials including: strategic planning; strategic planning; loan assessment; delinquency management; and impact assessment (c) Organize a training in Mexico Objective 3: Client Training: Equip partner staff to adapt and use appropriate training materials and methodologies for their clients Partners using appropriate training materials Revised and expanded mater-ials, according to partner input Enterprise client training materials were revised during FY01. Most prominently, the accounting portion of the business training curriculum was enhanced. Y Neither indicator nor target changed Yes for Year 2 Activities for Obj. 3 (a) Carry out training needs assessment of partners and their staffs 2 “Y”, indicates data were verified by evaluators and “N” that it was not possible for evaluators to substantiate PVO data. 3 Revisions to indicators during the life of the project as well as any changes in targets for indicators. Space is also used to provide explanations where targets for Year Two were not achieved by EDI H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc40 Goal/Objective/ Activity Indicator End of Project Target Accomplishments Data verified?2 Explanation for Change in Indicator or Target, if any3 Target Met? (b) Identify staff training opportunities in the regions and in the US (c) Objective 4: Partner MIS: Equip partners with effective management information systems. Increased timeli-ness, relevance and accuracy of partner management infor￾mation and reporting Implement for 2 other partners HIRO in Romania has successfully implemented a new Management Information System (MIS). Avance Chalco in Mexico has tested and plans to contract for an MIS in February, 2002 Y Neither indicator nor target changed Accomplishment of the MIS for Mexico is a need Yes for Year 2 Activities for Obj. 4 (a) Incorporate, test and refine Integra loan and client data systems, as well as QuickBooks for HIRO accounting (b) For Mexico, research “off-the-shelf” software that meets Avance Chalco’s needs (c) For Mexico and other EDI partners, negotiate an agreement with Microbanx Objective 5: Organizational Assessment: Enable partners to accurately and constructively assess their own organizations and plot a course for improvement based on results of the assessment ·Partners perform-ing self-assessment ·# actions taken per partner as a result of self-assessment Facilitated self￾assessments, and use of their results, for 2 partners. The Discussion-Oriented Organiz￾ational Self-Assessment (DOSA) tool was facilitated for both part-ners as well as HQ staff and both organizations have begun implementing DOSA action items Y (records) Y (records) Neither indicator nor target changed Neither indicator nor target changed Yes for Year 2 Yes Activities for Obj. 5 (a) Learn the DOSA methodology and how it is to be facilitated (b) Prepare local partner staffs for the training, its benefits Home Staff Development: Objective 6: Program Officer: Add to staff a Program Officer who will provide high quality technical services to partners and backstopping in home office Program Officer successfully providing quality technical services and backstopping 4 program visits; 4 conferences & seminars attended; Boulder or NHC training ·4 visits to Romania, 4 to Mexico and 2 to Nicaragua ·Program Officer has attended 2 SEEP Gen. Meetings and works on SEEP Impact Assessment Working Group, AIMS seminar, US Chamber of Commerce conference, a one-day PVC seminar and several courses including two weeks at New Hampshire College’s summer MED Institute Y Neither indicator nor target changed Yes for Year 3 Objective 7: VP Technical Services: Expand purview and capacity of VP Technical Services to include broader array of partner services and development of home office tools VP Technical Services success-fully providing quality technical services and having successfully deve-loped home office tools in Section I. Attain targets for Board Dev’t, Partner Staff Training, Client Training and Partner from Section I. ·Designed a board capacity building course and delivered it to EDI Board ·Designed board strategic planning training and used it to develop the EDI 2001-2003 Strategic Plan ·Negotiating with software manufacturer for EDI-wide use Y ·Regarding staff training targets have been met ·Regarding MIS, there is a serious need for loan tracking system in Mexico which has not yet been met and leaves Avance Chalco vulnerable No Activities for Obj. 7 (a) Study MIS needs of partners in Mexico and Romania and identify compatible software H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 41 Goal/Objective/ Activity Indicator End of Project Target Accomplishments Data verified?2 Explanation for Change in Indicator or Target, if any3 Target Met? Objective 8: Government Accounting: Develop staff expertise in accounting and reporting for US gov’t grants Timeliness and accuracy of financial reporting to US Gov’t Accurate & timely submission of quarterly Financial Status and Federal Cash Transaction reports, and end of grant reports ·Calculated EDI’s NICRA for first time allowing gov. grants ·Upgraded EDI’s financial management system ·All USAID reporting is accurate and on-time ·Accounting assistant hired Y (records) Neither indicator nor target changed Yes for Year 3 Activities for Obj. 8 (a) Carry out the OI mentoring program to learn government accounting requirements (b) Hire an accounting assistant (c) Take courses at USAID and other gov. Related institutions Objective 9: Grants Management and Program Monitoring: Develop staff expertise and improve internal systems for major grant management and program monitoring Accurate and timely internal program monitoring and internal reporting Expanded use of successful monitoring system ·Internal communications has been upgraded through internet and e-mail capability ·The DOSA has been employed and its results used in HQ and partners staffs Y Neither indicator nor target changed Yes for Year 2 Activities for Obj. 9 (a) Carry out the OI mentoring program Infrastructure, Planning and Sustainability: Objective 10: Computer Network: Improve home office computer network to better serve staff needs No. of computer upgrades; computer usage level; no. of system failures Operation and maintenance of office system working well ·IT services company was hired · Twelve new computers purchased ·Several software packages purchased Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 10 (a) Purchase computers, and software, upgrade computers (b) Contract with MIS specialist to make upgrades, install network in HQ and provide other IT services Objective 11: Strategic Planning: Develop five-year corporate strategic plan to set direction for the organization Existence and use of strategic plan Annual updates completed ·EDI Strategic Plan for 2001 to 2003 completed and ratified ·DOSA for HQ staff completed providing Strategic Plan updated Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 11 (a) Design a board training, with outside facilitator, for strategy development (b) Have Program Manager trained in the DOSA delivery H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc42 Goal/Objective/ Activity Indicator End of Project Target Accomplishments Data verified?2 Explanation for Change in Indicator or Target, if any3 Target Met? Objective 12: Sustainability: Improve home office financial sustainability ·% of endowment goal raised; ·no. of individual donors; ·revenue from institutional giving; ·product marketing outcome 40% of $10 million endowment goal raised; 25% increase in no. of individual donors; 15% increase in revenue from institutional giving. ·Concept papers for endowment and marketing products completed ·Direct mail campaign included in Strategic Plan ·A plan for marketing, shipping and selling program client products in the US completed ·48% increase in individual donors ·15.6% increase in revenues from institutional donors N Y (records) Y (records) N ·Plan for establishing endowment postponed due to 2001 economic situation ·EDI decided not to market￾ing client products in the US which complies with its target No Yes Yes Yes Activities for Obj. 12 (a) Establish an endowment (b) Conduct donor acquisition mailings and pursue institutional donors (c) Investigate marketing products produced in the program areas by program clients Objective 13: US Public Awareness: Increase US public awareness of the work of Enterprise ·No. publications; ·No. website hits; ·Number of mail recipients 25% increase in # of hits on website ·Annual Reports improved ·No. of EDI brochures increased by 150% (from 2 to5) ·EDI website hits/day increased from 1 to 13 ·EDI video was produced ·17,710 direct mail recipients in November 2001. 20,268 in January 2002. Y Y Y (records) Y N Neither indicator nor target changed Neither indicator nor target changed Yes for Year 3 Yes for Year 3 Yes Activities for Obj. 13 (a) Publish and distribute an Annual Report, calendar and other promotional materials (b) Expand and improve website (c) Conduct donor acquisition mailings H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 43 AMEXTRA/Avance Chalco Planning Matrix Goal/Objective/ Activity Indicator End of Project Target Accomplishment Data verified? Explanation for Change in Indicator or Target, if any Target Fully Met? Goal: The DIP shows no goal Objective 1: Expand AMEXTRA’s institutional capacity to deliver MED services Qualified Program Director in place In place Qualified Program Director hired Y Neither indicator nor target changed Yes for Year 3 Qualified loan officers in place 4 The program had hired 5 qualified loan officers Y Neither indicator nor target changed Yes for Year 3 Qualified MIS in place In place An Excel-based MIS was established and recently upgraded. Y Implementation of a sophisticated MIS is planned for February, 2002 No DOSA in use Demonstrated improvement DOSA was facilitated in July, 2001 Y Neither indicator nor target changed Yes for Year 2 Activities for Obj. 1 Select director Provide training Hire/train loan officers Select/implement MIS and train users Implement DOSA Objective 2: Establish significant program breadth Number of persons receiving training and saving 2200 The program had 1,261 saving clients as of September 30, 2001. On December 31, 2001 the program had 1,900 savers, well ahead of schedule to meet the end-of-project target. Therefore, as of 12/31/01, this target is essentially being met. Y The training strategy has changed since the elabor￾ation of the DIP and not every one of those saving has been trained. Training is now offered independently of saving, and is open both to those who are financial clients of the organization and to those who are not. Avance Chalco is now partnering with a local Mennonite Development Associates (MEDA) affiliate which specializes in training to deliver this service to re￾sidents of the Chalco Valley. No for year 2, see “accomplish￾ments” Number of active of active borrowers 800 433 active borrowers on 9/30/01. 803 active borrowers on 12/31/01. Y Neither indicator nor target changed Yes for Year 3 Valuation of loan portfolio $137,480 $ 168,077 on 9/30/01; $349,470 on 12/31/01. Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 2 Organize training for clients Recruit clients H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc44 Goal/Objective/ Activity Indicator End of Project Target Accomplishment Data verified? Explanation for Change in Indicator or Target, if any Target Fully Met? Capitalize loan fund and issue loans Objective 3: Establish significant program depth Percent of women participants 95% 69 percent of participants are women Y As reported in the Year One Progress Report, the program no longer has women as an exclusive target. Women are focused on but men are not excluded. No Number of women staff 5 Seven staff, all but the director, are women Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 3 Recruit women participants Recruit women staff Objective 4: High program efficiency Percent of portfolio in default Less than 5% 9.9% Y The majority were loans disbursed before two changes were instituted: One, a system for rewarding loan officers on the basis of portfolio quality was implemented, two, loan size maximums were lowered substantially. No (NOTE: EDI acknowledges that it made a mistake in not “ramping up” its target over years) Client/average FTE staff 133 borrowers 366 savers 94 borrowers per staff and 223 savers per staff as of 12/31/01 Y Y The no. of loan officers were increased in order to service the unexpected growth in borrowers No No Activities for Obj. 4 Educate clients on importance of repayment Follow-up delinquencies Train loan officers Keep overhead low Objective 5: Progress towards sustainability Percent cost coverage 58% 84 % Y Larger loan portfolio than expected and higher interest rates than anticipated has helped improve Avance Yes for Year 3 Percent client retention 75% N/A Y This indicator was included in the planning matrix in error. In the narrative portion of the DIP for Amextra, in section E “Sustainability Strategy” this is, correctly, not included among the indicators listed. Avance does not encourage constant N/A H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 45 Goal/Objective/ Activity Indicator End of Project Target Accomplishment Data verified? Explanation for Change in Indicator or Target, if any Target Fully Met? borrowing, so retention is not appropriate Interest income per staff member $6,935 $8,449 Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 5 Keep repayment rates high Provide excellent service to clients Objective 6: Achieve significant impact % business borrowers with increased sales in last 12 months 67% 29% Y Data are based on a less than 100% survey of borrowers. EDI believes, also, that it is difficult for borrower to attribute increased business sales to a loan No % business borrowers with increased household income in last 12 months 50% 56% Y Neither indicator nor target changed Yes for Year 3 % savers using savings to buy food, clothing, medical care, home improvements or children’s education in last 12 months 70% N/A Y Information on this indicator was inadvertently omitted from the September 2001 survey. It will be included in 2002 – final -- survey N/A Activities for Obj. 6 Annual surveys HIRO Planning Matrix Goal/Objective/ Activity Indicator End of Project Target Accomplishment Data verified? Explanation for Change in Indicator or Target, if any Target Fully Met? Goal: To enable poor women in Western Romania to develop and manage profitable microenterprises that improve their standard of living. Objective 1: Establish Integra institutional capacity to deliver micro-enterprise credit and training services in Romania Registered Romania organization In place HIRO is registered as an NGO for training and as an Ltd. for loans Y Neither indicator nor target changed Yes for Year 3 Board of Advisors In place Board is to be expanded in membership in near future to make it less dependent on EDI and the matching grant Y Neither indicator nor target changed Yes for Year 3 Qualified program director In place Qualified Program Director, Ms. Florica Cherches, in place. Y Neither indicator nor target changed Yes for Year 3 Number of qualified loan officers 4 Five loan officers are in place and working Y Neither indicator nor target changed Yes for year 3 H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc46 Goal/Objective/ Activity Indicator End of Project Target Accomplishment Data verified? Explanation for Change in Indicator or Target, if any Target Fully Met? Qualified MIS In place Integra MIS installed Y Neither indicator nor target changed Yes for Year 3 DOSA Improved DOSA score over two years First year DOSA carried out in September, 2001 Y Neither indicator nor target changed Yes for Year 2 Activities for Obj. 1 (a) Initial NGO registration made followed by research on new laws with consequent recent registration of HIRO Ltd. to carry out loans Prog. Dir. recruited and trained by Integra (b) Integra MIS for client info., FIS for loan tracking, Quickbooks and CIEL for financial management installed (c) Competent Financial Manager promoted from LO and trained Objective 2: Achieve significant outreach among low-income women Number of active borrowers 350 The target for Year 2 was 200 active borrowers. The achievement was 154 borrowers. Y HIRO experienced high loan defaults during its early loans and cut back on lending as one step in solving the problem No Number of loans disbursed 450 The target for Year 2 was 200 loans disbursed. The achievement was 187 loans. Y The reason is the same as explained above No Valuation of loan portfolio $185,500 The target for Year 2 was $101,800. The achievement was $66,856. Y ·Again, the cause of low portfolio stems from an early reduction in loans made. ·Target is measured in US$ but inflation & devaluation were significant cause of low result No Percent of women participants 100% The target for Year 2 was 100%. The achievement was 97%. Classification of borrowers by gender is somewhat arbitrary and was difficult to verify. The program no longer has women as an exclusive target. Women are focused on but men are not excluded. No Activities for Obj. 2 (a) Survey of applicants and analysis of secondary data for market research (b) Much time spent early on by LOs to clean problem loans and adjust borrowing requirements (c) LOs trained more to improve loan servicing and client follow-up and support Objective 3: Implement successful business training Number of women trained 370 The target for Year 2 was 225 women trained. The accomplishment was 219 women trained. Y The deviation from Year Two target was insignificant No Percent of business plans accepted 80% The target for Year 2 was 80%. The achievement was 89%. Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 3 (a) Integra training modified H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 47 Goal/Objective/ Activity Indicator End of Project Target Accomplishment Data verified? Explanation for Change in Indicator or Target, if any Target Fully Met? (b) Formulating Bus. Plans was made part of course and thus improved quality Objective 4: Achieve program efficiencies Percent of portfolio in default Less than 5% The target for Year 2 was less than 5% in default. The achievement was 7.3%. Y All borrowers in default were from the first two training sessions and are still reflected in overall figures No Loans per staff member 47 The target for Year 2 was 34 loans per staff member. The accomplishment was 19. Y Number of loan officers was increased from 4 to 5 in order to better service clients, review applications and verify data, assist in business plan preparation and reduce default No Activities for Obj. 4 (a) Analysis of initial high default rate and formulation of changes (b) Extra (5th) LO was hired (c) Additional requirements to approval added to process (d) Counseling of applicants Objective 5: Progress towards sustainability of MED institution Percent cost coverage 49% The target for Year 2 was 30%. The accomplishment was 42%. Y Neither indicator nor target changed Yes for Year 2 Interest income per staff member $2,700 The target for Year 2 was $1,100. The achievement was $3,381. Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 5 Objective 6: Achieve significant impact Percent of clients experiencing increases in business sales in the last 12 months of program 75% The target for Year 2 was 75%. The accomplishment was 64%. Y It is difficult, from the client surveys, to determine impact on sales. HIRO plans to ask clients directly in a uniform manner as part of reporting Also, de-valuation had a significant effect on impact No Percent clients experiencing increases in household income in last 12 months in the program 50% The target for Year 2 was 50%. The achievement was 64%. Y Neither indicator nor target changed Yes for Year 3 Activities for Obj. 6 (a) TA and training provided to clients from beginning of loan in development of business plan to advise throughout loan implementation H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc48 ANNEX C: EVALUATION SCOPE OF WORK * H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 49 STATEMENT OF WORK FOR Enterprise Development International (EDI) MATCHING GRANT EVALUATION MATCHING GRANTS PROGRAM OFFICE OF PRIVATE AND VOLUNTARY COOPERATION BUREAU FOR HUMANITARIAN RESPONSE U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc50 EVALUATION SCOPE OF WORK “Evaluation is a relatively structured, analytical effort undertaken selectively to answer specific management questions regarding USAID-funded assistance programs or activities.” (USAID ADS chapter 202.4). An evaluation scope of work (SOW) is a plan for conducting an evaluation. A good SOW provides clear directions to the evaluation team. PVC uses information from the evaluation of the programs it funds as part of a yearly results reporting process. In order to get more consistent information across all Matching Grants (MG) funded programs a standard evaluation format is used. The questions in this evaluation SOW are the questions that PVC is asking in all programs. EDI has reviewed this the template for this SOW and its comments have been added so that it reflects their specific information needs. The evaluations PVC is carrying out have two purposes: First, to assess the PVO’s compliance with the terms of its DIP, for which the PVO will be held accountable and, second, to learn lessons about the impact of the grantee’s activities without holding the PVO accountable for its achievements or lack of them regarding particular impacts. Where evaluation questions refer to impacts or lessons learned in areas which do not apply to the DIP of a PVO, the evaluation will make an appropriate notation in the text. I. PROGRAM IDENTIFICATION Enterprise Development International Matching Grant for Capacity Building for Microenterprise Development in Mexico, Romania and U.S. Headquarters Award No. FAO – A – 00-99-00055-00 Evaluation Date: September 26, 2001 to December 21, 2001 PROGRAM BACKGROUND The EDI program includes three activities: Mexico -- Avance Chalco is a microenterprise development organization which provides savings accounts, small loans, and business training to low-income residents of the Chalco Valley, outside Mexico City, Mexico. Avance Chalco began operations in 2000, and is a subsidiary of the Mexican Association for Rural and Urban Transformation (AMEXTRA), a local nonprofit development organization. Both AMEXTRA and Avance Chalco are partners of Enterprise Development International. Avance Chalco is AMEXTRA’s flagship microenterprise development program, and replication of its methodology in other areas of the country in which AMEXTRA works has already begun, most notably in Chiapas. Romania -- Help Integra Romania, or HIRO, is a microenterprise development organization which provides business training, mentoring, and small loans to low-income women in Oradea, Romania. HIRO is affiliated with the Integra Foundation, a network of institutions dedicated to the development of micro, small and medium-sized enterprises in Central and Eastern Europe. Both HIRO and Integra are partners of Enterprise Development International. HIRO’s mission is to advance the social, material and financial H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 51 well being of women with entrepreneurial spirit in Bihor County, in western Romania. HIRO’s offices are located in the city of Oradea (population 225,000), five miles from the Hungarian border. The Planning Matrices for headquarters, Mexico and Romania are shown in Attachment A. Indicate what information and data are available for the external evaluator. PVC already sent a document that will give you an excellent idea of the documents that should be assembled and preparation needed prior to an evaluation. ß Include documentation of any changes that have taken place since the initiation of the program. PURPOSE OF THE EVALUATION This section should contain two components --- (1) identify the evaluation audience and (2) establish a set of evaluation questions that are relevant to each audience. Outline the information needs of the evaluation audience (PVC, and local partners), and how each partner will use this information. ß Who wants the evaluation information, ß What do they want to know, ß What will the information be used for, ß When will it be needed, and ß How accurate must the information be? For example: The final (or mid-term) evaluation fulfills the requirements of the USAID/BHR/PVC Matching Grant (MG) Program. The MG program will use the information to: assess how well the MG is meeting its objectives; determine patterns and emerging issues across all MG funded programs; determine technical support needs for grantees shape new RFAs and to review of any follow-on proposals; develop internal and external documents to demonstrate the effectiveness of the MG program and to share lessons learned with the entire PVO community. PVC will use information outlined in the SOW template in its annual Results Report and in USAID's annual report to Congress. Achievements cited in the evaluation need to be supported by evidence and should be verifiable. Observations on data quality or constrains to interpretation should be stated as data from these evaluations is used for USAID reporting purposes and is subject to audits. Technical/program opinions and observations are an important element of the evaluation --- but should be stated as the evaluator estimate, opinion or forecast. THE EVALUATION QUESTIONS The following are a set of questions that the MG division is asking in all evaluations. These questions relate to the objectives of the MG division and PVC’s strategic plan. The evaluator team will assess the following program and institutional questions, provide evidence, criteria for judgment and cite data sources. The evaluator will assess both headquarters and the country-level programs. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc52 A. Program Implementation 1. Assess in qualitative terms the balance between the effectiveness of training activities as compared to the order-of-magnitude costs of delivering them. 2. Assess progress towards each major objective ß Based on the program’s planning matrices, determine if the program objectives have been met, partially met or were unattained. This is the single most important element the evaluation must document and discuss. In addition to the discussion of project results in the text of the evaluation, this information should also be put into matrix format. List each objective, and key outcomes at the effects and/or impact level. In the text: ÿ Identify major successes and constraints in achieving objectives and unanticipated effects. As part of this discussion comment on EDI and its local partners’ capacity to do program monitoring and evaluation. Note any constraints that prevented from measuring achievement of program objectives. If the program does not have “baseline” and end-of￾project data from which judgements can be made about the achievement of project objectives, this should be noted. (A more detailed discussion of monitoring and evaluation should be covered in Section ___ of the report) ÿ Identify if the project had a detailed implementation plan and the familiarity of field staff with the project design, implementation plan and monitoring and evaluation plan and data. ß Assess effectiveness of models, approaches or assumption that underlie the project. Has the approach been scaled-up in the project area or replicated elsewhere in country or in other countries? ß Has EDI engaged in program or policy advocacy? What was the focus of the advocacy and effects ß Discuss what EDI and its local partners have “learned” implementing this project. Identify if these “lessons learned” have been applied elsewhere (other projects or countries) 3. Assess the status of partnership(s) with NGOs, community based organizations or local level government. ß Include a chart that: ÿ Categorizes local level partners. Are the partners: NGOs, affiliates of EDI, private or commercial groups, cooperatives, community-based organizations, regional or local governments or intermediate service H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 53 organizations? ÿ Identify the type of mechanism employed with each partner, i.e. MOU, sub-grant, contract. ÿ Outline the roles, responsibilities and decision-making responsibilities of the partners. ÿ Identify the fiscal autonomy and amount of grant funds directly managed in past year. ß Assess the process that EDI used to build and maintain local partnerships. ÿ Does EDI have a partnership policy and approach to assess potential partners? ÿ Did EDI do a formal assessment of local partner capacity and develop plans to build their capacity? ÿ Document change in local partner capacity. ÿ What were the major constraints to effective partnerships? ÿ Has the project increased the local partners’ access to information technology? How? ß Assess the local level partners’ satisfaction with the partnership. ß Assess EDI and its local partners’ involvement in local networks or with intermediate service organizations. 4. What effect did participation in networks or service organizations have on the operational or technical capacity of the local partner? What would make it more effective? Cite the major implementation lessons learned and recommendations 5. If this is the case of a first grant to the recipient PVO by PVC, are there particular issues that can be identified for a new PVO and the development of its relationship with PVC. ß How “mature” has the PVO/PVC relationship gotten over the course of the grant? ß Is a three-year grant adequate for developing a working relationship such that the PVO’s objectives can be expected to be achieved in the given timeframe? ß Does the AID/PVC mechanism fit well with a new PVO grantee? ß What problems does the new PVO have in adjusting to these management mechanisms? B. Management Capacity/Institutional Strengthening The objective of the MG is to build EDI headquarters and field organizational and technical capacity. This section of the evaluation should assess change’s in operational and management capacity (organization, structure or quality of planning and management) as a result of PVC grant. ß Strategic Approach and Program Planning Have changes occurred in EDI headquarters capacity to: ÿ manage the planning process --- program renewal, strategy integration, project H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc54 design; ÿ address over-arching program issues of replicability, scale-up, sustainability, ÿ Use performance data to forecast emerging trends and develop strategic plans? ß Country Level Initiatives Identify and assess (if relevant), EDI contributions in the following areas: ÿ EDI cooperation and coordination with the USAID mission and other development partner programs including natl./local government agencies; ÿ EDI advocacy activities: issues, goals, partners and results (Has EDI used project data for advocacy with the public sector or consistently shared lessons learned with other EDI’s in country or with non-partner NGOs?); ÿ If the country or program area has a history of violent conflict, other man-made/natural disasters, or food insecurity: (a) EDI activities in conflict prevention, mitigation, resolution or post-conflict transition (b) EDI's contingency plan to ensure the safety of program staff and program continuity. ß Monitoring and Evaluation Has the project implemented a process and put into place a sustainable system to monitor project performance and collect results (effects or impact) data? Provide evidence that the project: ÿ Established results oriented objectives and valid indicators for the technical intervention and capacity building components in the project; collected valid baseline data, and made realistic plans to collect end-of-project data and analyze differences; analyzed performance data and used findings to manage the project. If this is a final evaluation, has EDI acted on recommendations from mid-term evaluation? ÿ Improved the knowledge and skills of field staff on how to measure performance and analyze data. ÿ Transferred monitoring and evaluation skills to local partners? - What changes have occurred in the capacity of the local partners to measure program performance and impact? - Have local partners increased M&E in their own activities (non-PVC￾funded programs) as a result of skills gained through this project? - What would accelerate the capacity of the local partners to document performance? Determine if EDI has used the MG to develop a sustainable capacity at headquarters and in the field offices to monitor project performance and measure effects and impact. Has EDI headquarters: ÿ fostered analysis and self evaluation in country programs, or conducted quantitative or qualitative analysis to refine interventions; H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 55 ÿ conducted periodic review of performance data by project personnel and taken actions as a result of review; ÿ institutionalized performance monitoring and impact evaluation systems developed with MG funds into other non-PVC grant funded programs, and; What were the biggest constraints to improving project monitoring and evaluation and what are the recommendations for PVC and EDI? ß Sustainability ÿ Does the project have a system for addressing financial or operational sustainability? ÿ Does the project have a business plan? ÿ Describe the program elements, financial or operational, that are intended to be sustained (objectives); the means for judging if the sustainability objectives have been achieved (indicators); and sustainability achievements and prospects for post-grant sustainability. ÿ Identify if the project has any cost-recovery mechanisms, i.e., local level financing or approaches to generate resources to support project operations. Describe the achievements of these mechanisms and provide an estimate of the magnitude of the system, for example, provide a ratio of costs recovered to operational expenses. · Other Management Systems Financial Management ÿ Are adequate financial monitoring systems in place? ÿ Has the program leveraged additional resources (beyond the match)? ÿ How cost-effective is the technical approach? · Information Management ÿ Comment on the utility and timeliness of EDI’s required reports. ÿ Has EDI developed, disseminated and used “lessons learned” from the project? ÿ Information Technology Logistics ÿ Comment on the adequacy and timeliness of EDI’s material inputs. ß Supervision/HRD ÿ Assess if there were sufficient staff with the appropriate technical and management skills to oversee program activity at both headquarters and in the field program ß USAID Management Comment on USAIDs oversight and backstopping of this cooperative agreement. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc56 Cite the major management lessons learned and recommendations V. EVALUATION METHODOLDOGY A. Approach EDI’s program was developed and funded prior to the Agency's emphasis on results￾oriented program designs and the development of PVC’s Strategic Plan. The data from all PVC-funded programs is critical to PVC's ability to report on achievements against the Office's Strategic Plan. Until all current PVC-funded programs have made the transition to a more results-oriented project plans, it will be necessary for the evaluator to conduct a team-planning meeting with and local partners to: ® refine and consolidate the purpose-level objectives and outputs into a set of results-oriented objectives; and ® Agree upon a set of appropriate indicators against which the evaluation will assess the achievement of project results outlined in the SOW and will be judged. And where necessary, identify criteria for judgement. B. Methodology The Evaluator will: ® explain the appropriateness of using the data collection approaches; ® use the Agency's microenterprise (ME) indicators to interpret the status of the ME intervention; ® document data sources (data constraints, quality, etc.); and ® Provide, a copy (electronic or paper) of all primary data collected and analysis performed. VI. TEAM COMPOSITION AND PARTICIPATION INSTRUCTIONS: Based on tasks outlined and the emphasis of each evaluation section determine skills needed and who will participate in the evaluation team ---- PVO, NGO and AID staff. Outline: - Roles and responsibility of team leader and members - Language requirements - Technical expertise, or country experience - Evaluation methods and data collection expertise VII. SCHEDULE INSTRUCTIONS: Determine: - Time needed at headquarters - Time needed in the field - Time necessary for report writing H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 57 VIII. REPORTING AND DISSEMINATION REQUIREMENTS INSTRUCTIONS: - The SOW will serve as the outline of the report - Delivery schedule - Review/revision policy H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc58 58 ANNEX D: SOLIDARITY GROUP SUMMARY INFORMATION FOR AVANCE CHALCO Debt Status Group Statistics Current loans Reasons for group initiation Group Measures Promoter Measures Solidarity Group Name Promoter In debt? (Frozen) Frequency of debt Problem Group Months in Program Number Members Loan Scale Economic Level Basis of member relationships Number Total Amount Loans typically takes out for what purpose New groups Spin-off Groups Attendance Savings commitment Loan repayment Promotion Follow-up Ferias Unidas Alejandra No Low No 1 9 0 High Business Neighbor 0 $0 Business Make $ for business New New New High Low Mirador Alejandra No High No 4.25 20 1 and 2 High Business Neighbor 22 $65,521 Business Make $ for business Low Medium High High Low Limite Alejandra No Low No 1.25 18 0 High Business Neighbor 0 $0 Business Make $ for business New New New High Low Pescadores Alejandra Yes Medium No 15 7 3 High Church Family 4 $12,440 Business Church/ Steev Low High Medium High Low 2001 (2) Irma Alejandra Yes High Yes 13 13 2 Medium 14 $9,000 Frozen Low Low Low Medium High Esmeralda Irma No Low No 3 9 2 Medium Family Friends 6 $14,788 Family and Business Medium High High Medium High Osiris Odi No New New 0.75 16 0 Medium Friends Business 0 $0 Will go for business Learned from other High New New Medium+ Medium+ Dario-2 Irma No Low No 12 12 Medium Family Friends 11 $35,904 Family and Business Left poor performing High High High Low High Chicos de Odi Odi No Low No 2 17 Medium Friends Business 8 $10,650 Will go for business Learned from other High High High Medium+ Medium+ Luna Odi No Low No 1.5 15 Medium Friends Business 0 $0 Will go for business Came from Chicos de Odi High New New Medium+ Medium+ Asusena Marie No New New 0.75 10 0 Low Friends Neighbors 0 $0 Will go for business Word of mounth New New New High New El Pino Marie No Low No 3 11 1 Low Friends Neighbors 4 $4,292 Family and Business Word of mounth New New New High New La Tormenta Marie No Low No 10 Low Family Friends 4 $10,500 Family and Business Word of mounth Medium New New High New No groups visited Laura Medium- New Definitions: Definitions: Frequency of debt: A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers Loan purpose: A categorization of the primary purposes for member loans based on evaluator questions and confirmed by Loan Officers Problem Group: Avance considers certain groups (10 as of this evaluuation) as "problems" New groups: Groups that started as "new" to Avance Loan scale: Avance allows groups to advance in borrowing limites, "scales", depending on various criteria Spin-off groups: Groups that started as "sin-offs", in one way or another, from othyer Avance groups Basis of relationships: A categorization of the primary bonds of members based on evaluator questions and confirmed by Loan Officers Group Measures: Three categorical measures used by Avance to measure the "quality" of groups regarding their performance Attendance: A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers Savings commitment: A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers Loan Repayment: A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers Promoter Measures: Two categorical measures used by Avance to measure the "quality" of loan officers regarding their performance A categorical measure assigned by the Avance Coordinator for each Loan Officers A categorical measure assigned by the Avance Coordinator for each Loan Officers H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc59 59 ANNEX E: CLIENT SUMMARY INFORMATION FOR HIRO Debt Status Client Statistics Business Information Loan Information Client Name Loan Officer Written Off? Frequency of loans Plans for more? Months in Program Woman/ Man Loan Repay-ment Economic Level Workers Employed Average Salary/ mon. Loan Amount (milliions) Loans typically takes out for what purpose? Geographic Location Nature of Business Maria Crai Anca No 1st Yes 1 Woman Beginning middle 2 33.00 But out partners Salonta Auto parts retailer Gavril Stranici Anca No 1st Yes 5 Man Good mid-high 35-40 $80 33.00 Salonta Shoe manufacturer Floare Boglut Monika No 1st Yes 6 Man Good mid 2 25.00 Buy supplies Salonta Funiture maker Sorin Capota Monika No 1st 2 Man w/wife Good mid 2 30.00 Buy raw wood Oradea Lumber retail Irina Mangra Carmen No 2nd Yes 14 Woman Good low/mid 0 15.00 Buy fruits and vegetables Oradea Open market sale Iuliana Birta Anca No 1st 3 Woman Good agric. 0 15.00 Buy pigs Hidisel Livestock Doina Maria Lukacs Monika No 1st Yes 5 Woman Good low/mid 0 15.00 Buy inventory Oradea Open market sale Halmagya Mirabela Alina Yes 1st 13 Woman Bad low 1 20.00 To pay for husband's funeral Oradea Flower shop now, Coffee Bar originally Gheorghe Vidican Dana No 1st Yes 6 Man w/wife Good mid 2 33.00 Inventory and repairs Santandrei Grocery then Internet Café Victor and Florica Bei Dana No 2nd 14 Man w/wife Good mid-upper 10 $70 25.00 Machines Cheresig Tailoring Marcela Silaghi Alina No 2nd 16 Woman Good agric. 2 Varies/ Low 15.00 Watering hose, Supplies Parhida Greenhouse/ Vegetables Coffee Bar H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc60 GROUP SUMMARY INFORMATION FOR HIRO Solidarity Group Name Promoter In debt? (Frozen) Frequency of debt Problem Group? Months in Program Number Members Loan Scale Economic Level Basis of member relationships Ferias Unidas Alejandra No Low No 1 9 0 High Business/Neighbor Mirador Alejandra No High No 4.25 20 1 and 2 High Business/Neighbor Limite Alejandra No Low No 1.25 18 0 High Business/Neighbor Pescadores Alejandra Yes Medium No 15 7 3 High Church/Family 2001 (2) Irma Alejandra Yes High Yes 13 13 2 Medium Esmeralda Irma No Low No 3 9 2 Medium Family/Friends Osiris Odi No New New 0.75 16 0 Medium Friends/Business Dario-2 Irma No Low No 12 12 Medium Family/Friends Chicos de Odi Odi No Low No 2 17 Medium Friends/Business Luna Odi No Low No 1.5 15 Medium Friends/Business Asusena Marie No New New 0.75 10 0 Low Friends/Neighbors El Pino Marie No Low No 3 11 1 Low Friends/Neighbors La Tormenta Marie No Low No 10 Low Family/Friends No groups visited Laura Definitions: "Frequency of debt" -- A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers "Problem Group" -- Avance considers certain groups (10 as of this evaluuation) as "problems" "Loan scale" -- Avance allows groups to advance in borrowing limites, "scales", depending on various criteria "Basis of relationships" -- A categorization of the primary bonds of members based on evaluator questions and confirmed by Loan Officers H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 61 Current loans Reasons for group initiation Group Measures Promoter Measures Solidarity Group Name Promoter Number Total Amount Loans typically takes out for what purpose? New groups Spin-off Groups Attendance Savings commitment Loan repayment Promotion Follow￾up Ferias Unidas Alejandra 0 $0 Business Make $ for business from Mirador New New New High Low Mirador Alejandra 22 $65,521 Business Make $ for business Low Medium High High Low Limite Alejandra 0 $0 Business Make $ for business from Mirador New New New High Low Pescadores Alejandra 4 $12,440 Business Church/ Steev Low High Medium High Low 2001 (2) Irma Alejandra 14 $9,000 Frozen Low Low Low Medium/Hig h High/Low Esmeralda Irma 6 $14,788 Family and Business Medium High High Medium High Osiris Odi 0 $0 Will go for business Learned from other CAME High New New Medium+ Medium+ Dario-2 Irma 11 $35,904 Family and Business Left poor performing group High High High Low High Chicos de Odi Odi 8 $10,650 Will go for business Learned from other CAME High High High Medium+ Medium+ Luna Odi 0 $0 Will go for business Came from Chicos de Odi High New New Medium+ Medium+ Asusena Marie 0 $0 Will go for business Word of mouth CAME New New New High New El Pino Marie 4 $4,292 Family and Business Word of mouth CAME New New New High New La Tormenta Marie 4 $10,500 Family and Business Word of mouth CAME Medium New New High New No groups visited Laura Definitions: "Loan purpose" -- A categorization of the primary purposes for member loans based on evaluator questions and confirmed by Loan Officers "New groups" -- Groups that started as "new" to Avance "Spin-off groups" -- Groups that started as "sin-offs", in one way or another, from othyer Avance groups "Group Measures": Three categorical measures used by Avance to measure the "quality" of groups regarding their performance "Attendance" -- A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers "Savings commitment" -- A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers "Loan Repayment" -- A categorical measure assigned by the evaluator through observation and confirmed by Loan Officers "Promoter Measures": Two categorical measures used by Avance to measure the "quality" of loan officers regarding their performance A categorical measure assigned by the Avance Coordinator for each Loan Officers A categorical measure assigned by the Avance Coordinator for each Loan Officers H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc63 ANNEX F: LIST OF PERSONS INTERVIEWED Name Organization.Title Juan A. Benitez President and CEO, EDI Kenneth W. Wesche Vice President, Programs, EDI Georgia H. Emory Technical Director, EDI Mark Petersburg Vice President, Development, EDI Donald Webster Director of Finance and Administration, EDI Lawrence Locklin Program Director, EDI Philip Chalk Director of Communications, EDI Omar Villagran Arjona Director Geneal, AMEXTRA Steev Hargrave Coordinator, Avance Chalco Eugenio Araiza Bahena Assist. Director for Operations, AMETRA Santiago Victor Leal Board Chairman, Avance Chalco Rosio Torreblanca Accountant, Avance Chalco Alejandra Lopez Loan Officer, Avance Chalco Irma Rosa Loan Officer, Avance Chalco Odilia Montalvo Loan Officer, Avance Chalco Marie Lourdes Vega Loan Officer, Avance Chalco LauraBlanca Loan Officer, Avance Chalco Howard D. Hood Vice President, North Am. Services, MEDA Florica Chereches Executive Director, HIRO Brett K. Howard Program Manager, Integra Diana Hanga Financial Manager Adriana Covaci Secretary and Cashier Alina Plumbas Loan Officer Dana Ghent Loan Officer Carmen Gavra Loan Officer Monika Mahr Loan Officer Anca Hapca Loan Officer Istvan Svabo HIRO Board of Directors Ferenc Pastor Executive Director, Transylvania Business Forum H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc64 ANNEX G: SIMPLIFIED ACTIVITY AND FINANCIAL REPORTS MEXICO AND ROMANIA Simplified Activity and Financial Statement: Romania In US$ (Exchange rate used @ US $1 =____________) Program and Country: FY00 FY01 FY02 year ACTIVITIES (thru 12/31/01) 1. Amount of loans outstanding, start of year 0 8,807 66,856 2. Amount of loans outstanding, end of year 8,807 66,856 71,799 3. Avg. amount of loans outstanding 734 57,682 72,498 4. Number of loans outstanding 9 154 143 5. Average loan size (line 2 divided by line 4) 979 434 502 6. Portfolio at Risk > than 30 days 0 15.8% 12.1% 7. Long run loss rate (line 16 divided by line 3) 0 7.3% 8.1% INTEREST RATES 8. Nominal interest rate charged by program 40% flat 40% flat 40% flat 9. 90 day CD rate 37% 31% 34% 10. Inflation rate 35% 27% 30% CLIENT REVENUES 11. Interest income from clients 139 22,823 8,906 12. Fee income from clients 11 1,259 168 13. Total client revenues (lines 11 + 12) 150 24,082 9,074 OPERATING EXPENSES 14. General Operating Expenses (salaries, rents, utilities, etc.) 34,127 73,586 20,582 15. Depreciation of fixed assets NR 2,503 1,132 16. Loan loss provision expense 0 4,239 2,116 17. Total Operating Expenses 34,127 80,328 23,830 ADJUSTED FINANCIAL EXPENSES 18. Adjusted financial expenses (line 3 multiplied by the higher of line 9 or 10) 272 17,881 6,162 (1st qtr. only) TOTALS 19. Total Expenses (line 17 plus line 18) 34,399 98,209 29,992 20. Financial Sustainability (line 13 divided by 19) 0.4% 25% 30% OTHER 21. Total number of voluntary savers 0 0 0 22. Total voluntary savings outstanding 0 0 0 23. Percent women borrowers (of line 4) 100% 97% 93% 24. Percent rural clients (of line 4)* 16% 16% 16% 25. Total Number of staff 5 8 8 26. Number of field officers (credit staff) 2 5 5 27. Number loans outstanding with initial balance under poverty loan level for your region.* *5 *128 106 *estimated Notes by line item 1. Amount of loans outstanding, start of year. Monetary volume of portfolio in US$. 6. Portfolio at Risk Over 30 days. Divide unpaid balance of loans with payments overdue more than 30 days by the amount of loans outstanding, end of year (line 2). Include as an attachment an aging of portfolio table: 1-30 days; 31-60 days; 61-90 days; over 90 days. See chart below. 7. Long run loss rate. Divide amount of loans written off during the past year (line 16) by average amount of loans outstanding (line 3). 16. Loan loss provision. Loans over one year delinquent should be added to write-offs. 19. Adjusted financial expenses. Cost of financing the portfolio at a 90 day CD cost of funds. This calculation avoids the need to adjust separately for various forms of subsidy. 27. Poverty lending levels vary by region. For Africa, Asia and the Middle East it is <$300. For LAC it is <$400, and for the Europe & Eurasia region it is <$1,000. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 65 INS INSTITUTION: Aging of Portfolio Report: Romania US$ Period: As of 12/31/01 # of Loans Outstanding Balance Percent Current Loans 98 48,961 68.2% 1-30 days past due 20 13,852 19.3% 31-60 days past due 6 1,758 2.4% 61-90 days past due 2 405 0.6% Over 90 days past due 17 6,823 9.5% TOTALS 143 71,799 100% H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc66 FINANCIAL RATIOS: Romania Data as of 12/31/01 Profitability Return on Equity (Net income/Average equity) -40% Return on Assets (Net income/Average assets) -24% Financial Structure Net loans/Total Assets 39% Investments/Total Assets 98% Earnings Portfolio Yield (Interest and Fee Income/Average gross portfolio) 45% Management Operating expenses/Average gross portfolio 33% First loans/Total number of loans in period 74% Percentage of first loans in period with initial loan balance at $300 or less 74% Number field officers/total staff 63% Number of borrowers/field officers 29 Percentage growth from one year ago in number of loans 83% Stratification of Loan Porfolio (on the basis of initial balance of loan) No. loans under $300 5 No. loans $301-$500 27 No. loans $500 - $1,000 74 No. loans $1,001- $1,500 36 No. loans $1,501- $2,000 1 No. loans $2,000 - $3,000 0 No. loans $3,000 + 0 Total 143 H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 67 Simplified Activity and Financial Statement: Mexico In US$ (Exchange rate used @ US $1 =____________) Program and Country: FY00 FY01 FY02 year year ACTIVITIES (thru 12/31) 1. Amount of loans outstanding, start of year 0 12,737 168,077 2. Amount of loans outstanding, end of year 12,737 168,077 348,594 3. Avg. amount of loans outstanding 3,842 92,120 281,820 4. Number of loans outstanding 35 433 803 5. Average loan size (line 2 divided by line 4) 364 388 434 6. Portfolio at Risk > than 30 days 0 21% 19% 7. Long run loss rate (line 16 divided by line 3) NR 9.4% 2.3% INTEREST RATES 8. Nominal interest rate charged by program 72% 58.5% 58.5% 9. 90 day CD rate 16% 14% 9% 10. Inflation rate 10.7% 7.3% 5.1% CLIENT REVENUES 11. Interest income from clients 2,240 48,580 24,232 12. Fee income from clients 0 0 0 13. Total client revenues (lines 11 + 12) 2,240 48,580 24,232 OPERATING EXPENSES 14. General Operating Expenses (salaries, rents, utilities, etc.) 31,516 70,492 27,283 15. Depreciation of fixed assets 624 988 252 16. Loan loss provision expense 350 8,621 6,486 17. Total Operating Expenses 32,490 80,101 34,021 ADJUSTED FINANCIAL EXPENSES 18. Adjusted financial expenses (line 3 multiplied by the higher of line 9 or 10) 615 12,897 6,341 (1st Qtr only) TOTALS 19. Total Expenses (line 17 plus line 18) 33,105 92,998 40,362 20. Financial Sustainability (line 13 divided by 19) 7% 52% 60% OTHER 21. Total number of voluntary savers 282 1,261 1,700 22. Total voluntary savings outstanding 14,642 67,518 108,262 23. Percent women borrowers (of line 4) 74% 69% 70% 24. Percent rural clients (of line 4) 0 0 0 25. Total Number of staff 5 8 8.5 26. Number of field officers (credit staff) 3 5 5 27. Number loans outstanding with initial balance under poverty loan level for your region. NA *190 *352 * estimated Notes by line item 1. Amount of loans outstanding, start of year. Monetary volume of portfolio in US$. 6. Portfolio at Risk Over 30 days. Divide unpaid balance of loans with payments overdue more than 30 days by the amount of loans outstanding, end of year (line 2). Include as an attachment an aging of portfolio table: 1-30 days; 31-60 days; 61-90 days; over 90 days. See chart below. 7. Long run loss rate. Divide amount of loans written off during the past year (line 16) by average amount of loans outstanding (line 3). 16. Loan loss provision. Loans over one year delinquent should be added to write-offs. 20. Adjusted financial expenses. Cost of financing the portfolio at a 90 day CD cost of funds. This calculation avoids the need to adjust separately for various forms of subsidy. 27. Poverty lending levels vary by region. For Africa, Asia and the Middle East it is <$300. For LAC it is <$400, and for the Europe & Eurasia region it is <$1,000. H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc68 INS INSTITUTION: Aging of Portfolio Report: Mexico US$ Period: As of 1/30/01 # of Loans Outstanding Balance Percent Current Loans 545 $197,497 71.2% 1-30 days past due 89 $27,163 9.8% 31-60 days past due 28 $13,094 4.8% 61-90 days past due 10 $10,532 3.8% Over 90 days past due 69 $28,833 10.4% TOTALS $277,119 100% H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc 69 FINANCIAL RATIOS: Mexico Data as of 12/31/01 (Loan Stratification through 9/30/01) Profitability Return on Equity (Net income/Average equity) -24% Return on Assets (New income/Average assets) -16% Financial Structure Net loans/Total Assets 95% Investments/Total Assets 66% Earnings Portfolio Yield (Interest and Fee Income/Average gross portfolio) 45% Management Operating expenses/Average gross portfolio 64% First loans/Total number of loans in period 37% Percentage of first loans in period with initial loan balance at $300 or less 80% Number field officers/total staff 63% Number of borrowers/field officers 161 Percentage growth from one year ago in number of loans 421% Stratification of Loan Porfolio (as of 9/30/01) (on the basis of initial balance of loan) No. loans under $300 135 No. loans $301-$500 202 No. loans $500 - $1,000 72 No. loans $1,001- $1,500 13 No. loans $1,501- $2,000 8 No. loans $2,000 - $3,000 3 No. loans $3,000 + 0 Total 433 H:\INCOMING\06-26-MSI-final-EDI-evaluation\PVC EDI Final Evaluation Report.doc70 ANNEX H: EDI CRITERIA FOR INTERNATIONAL PROGRAMS AND PARTNERSHIPS 1) We target the very poor. And there are lots of them: 1.3 billion living on less than $1 per day; half the world, more than 3 billion, living on less than $2 per day (based on data from The World Bank). While not all the world’s very poor are candidates for microenterprise, we seek to work with those who are economically active. These are people who have some skills and the will to work, but largely live without access to education, adequate health care, decent housing, a nutritious diet, or potable water. To a large degree, they are also people who often lack self-esteem, opportunity to advance, and hope. 2) We work through local partner organizations. Enterprise partners are locally registered, Christian non-profit organizations that have their own staffs and Boards of Directors. They are made up of people who speak the local language, understand the local culture, have ideas for appropriate solutions to local problems and truly "own" their organization and its work. They share our vision for how and why economic development should be done, and we join with them in addressing local problems. We typically provide technical assistance, training and funding; they implement. They are accountable to us through regular reports and good faith cooperation, but they are autonomous organizations – they do not report to us – rather, we serve them. 3) We implement sustainable solutions. Enterprise works with its partners to design and implement programs that will become self-supporting. Most use revolving loan funds through which funds are lent to local entrepreneurs, repaid, and lent again in the community. Some programs use other means of cost recovery but the principle of sustainability is designed into each one. Our programs do not give hand-outs. We believe hand-outs crush initiative and create dependency. We provide opportunities for people to support themselves and fulfill their dreams, thereby building self-esteem and self-reliance while strengthening the community. 4) We are a Christian organization. Our faith in Christ and his call to serve the poor are the motivations for our work. The life of Christ and his special attention to the poor and needy are the models for our outreach. Our programs never discriminate based on religion. On the contrary we purposefully reach out to needy people of any (or no) religious persuasion, seeking to demonstrate tangible Christian love. In short, we are in the business of providing people in chronically poor communities with what they need to transform their lives.