USAID/PVC Matching Grant Evaluation Series Institutional Development Services for Enterprise Development Organizations Matching Grant FAO-A-00-98-00049-00 Between Small Enterprise Education and Promotion (SEEP) Network and USAID/PVC April 2002 Conducted under USAID Evaluation Indefinite Quantity Contract # AEP-I-00-00-00024-00 Evaluation team members: James Dempsey, MSI Dana De Kanter, SEEP TABLE OF CONTENTS List of Acronyms Used .......................................................................................................................................ii Evaluation Identification Sheet .........................................................................................................................iii 1.0 Executive Summary......................................................................................................................................1 1.1 Overview ...................................................................................................................................................1 1.2 Key Findings, Conclusions and Recommendations...............................................................................1 3.0 Matching Grant Background........................................................................................................................5 3.1 Historical and Technical Context ............................................................................................................5 3.2 Matching Grant Goals and Objectives ....................................................................................................7 4.0 Purpose of the Evaluation.............................................................................................................................8 5.0 Program Implementation Evaluation Questions.........................................................................................9 5.1 The Detailed Implementation Plan........................................................................................................10 5.1.1 DIP Completion and Accuracy.......................................................................................................10 5.1.2 Quality of DIP..................................................................................................................................11 5.1.3 Familiarity with the DIP and Design..............................................................................................11 5.2 SEEP Matching Grant Performance ......................................................................................................12 5.2.1 The Process of Lateral Learning.....................................................................................................12 5.2.2 Outputs.............................................................................................................................................13 5.2.3 Institutional Development Services (IDS) Performance...............................................................15 5.2.4 Network Development Services.....................................................................................................21 5.3 Impact Results.........................................................................................................................................23 5.4 Assessment of Grant Model and Hypotheses .......................................................................................24 5.4.1 Appropriateness of Grant Hypotheses ..........................................................................................24 5.4.2 Replication and Scale-up of SEEP Approach................................................................................24 5.5 Advocacy Activities Under the Project .................................................................................................25 5.6 Implementation Lessons Learned ..........................................................................................................25 6.0 PARTNERSHIP QUESTIONS..................................................................................................................25 7.0 PROGRAM MANAGEMENT ..................................................................................................................26 7.1 Strategic Approach and Program Planning...........................................................................................26 7.2 Country Initiatives ..................................................................................................................................26 7.3 Conflict Management .............................................................................................................................26 7.4 Monitoring and Evaluation.....................................................................................................................26 7.5 Overall Management ..............................................................................................................................27 7.6 Sustainability...........................................................................................................................................28 7.7 Financial Management ...........................................................................................................................28 7.8 Information Management.......................................................................................................................29 7.9 USAID Management ..............................................................................................................................30 8.0 OVERALL CONCLUSIONS....................................................................................................................30 ANNEXES ANNEX A: Evaluation Surveys......................................................................................................................31 ANNEX B: Key Evaluation Documents ........................................................................................................38 ANNEX C: Statement of Work.......................................................................................................................39 ANNEX D: DIP and Business Plan Performance Matrixes..........................................................................46 ANNEX E: Budget...........................................................................................................................................50 ANNEX F: Member Agencies of the SEEP network ....................................................................................51 ANNEX G: Lateral Learning ..........................................................................................................................53 C:\12-20-working\New Folder\SEEP evaluation final report.doc ii LIST OF ACRONYMS USED AED Academy for Educational Development. AGM Annual General Meeting BDS Business Development Services BHR Bureau of Humanitarian Relief CGAP Consultative Group to Assist the Poorest CTO Cognizant Technical Officer DIP Detailed Implementation Plan FFP Food for Peace FY Fiscal Year HQ Headquarters ILO International Labor Organization ISA Institutional Support Assistance KM Knowledge Management MED Microenterprise Development M&E Monitoring and evaluation MF Microfinance MFI Microfinance Institution MG Matching Grant NDS Network Development Services NGO Non-governmental organization OMD USAID's Office of Microenterprise Development PL-480 Public Law 480 PLP Practitioner Learning Program PVC Private Voluntary Cooperation PVO Private Voluntary Organization (usually U.S.) SEEP Small Enterprise Education and Promotion Network UN United Nations USAID United States Agency for International Development VB Village Banking C:\12-20-working\New Folder\SEEP evaluation final report.doc iii EVALUATION IDENTIFICATION SHEET PVO name: Small Enterprise Education and Promotion (SEEP) Network Matching Grant Title: Institutional Development Services for Enterprise Development Organizations Cooperative agreement number: FAO-A-00-98-00049-00 Amount of Grant $450,000 - original $563,330 - as amended Period of Grant September 11, 1998 to September 29, 2001 Any (cost/no cost) extensions? None Current status of MG Completed USAID/PVC Grant Officer (s) Tom Kennedy Technical area of grant Microenterprise Development Date of the evaluation February/March 2002 Countries of program activity USA Country programs evaluated USA Evaluation Team Member (organization) James R Dempsey, MSI C:\12-20-working\New Folder\SEEP evaluation final report.doc 1 Institutional Development Services Matching Grant (FAO-A-00-98-00049-00) Between USAID's PVC Office and the SEEP Network 1.0 EXECUTIVE SUMMARY 1.1 Overview The Small Enterprise Education and Promotion (SEEP) Network is an association of 56 North American private and voluntary organizations which support micro and small enterprise programs in the developing world. USAID's Private Voluntary Cooperation Office awarded a $563,330 matching grant (MG) to the SEEP Network for a three-year period (9/1998 - 9/2001). The grant supported SEEP core activities in its Institutional Development Services (IDS) program. The objectives of the MG was to increase the effectiveness of microenterprise development practices on the part of North American private development organizations and their southern partners. SEEP included in the MG a set of objectives, indicators and targets to support developing country microenterprise development networks although no MG funding was provided for such activities. SEEP successfully raised funds to implement its Network Development Services (NDS) program. The general objective of this evaluation is to fulfill the condition of USAID/BHR/PVC’s Matching Grant program to complete a final independent review of grant effectiveness, results and lessons learned. The review was based on interviews with key informants and opinion surveys of SEEP members, the board of directors, selected working group participants and a group of developing country microenterprise development networks. The four surveys are attached as Annex A. The nature of SEEP as an association or network is an important structural characteristic to understand. Its structure is not hierarchical like a private firm or PVO but lateral. It is characterized by reciprocity and collaboration among people with no reporting relationship to each other, nor any formal contractual agreement. Thus, there are four important characteristics to understand about SEEP and the MG at the start: (1) as an association, SEEP focuses on member needs and interests at the practioner level; (2) the lateral network structure depends on volunteers for its operation; (3) most decision-making on programs comes through direct participation, and (4) SEEP and its members accept diversity of technical approaches. These are spelled out as objectives in SEEP documents, highlighted in previous evaluations and reinforced in the responses to the questionnaires sent out for this evaluations. That these characteristics existed in SEEP in the past and continue today is important as background and a finding for the present. 1.2 Key Findings, Conclusions and Recommendations Findings Sustainability · SEEP has met the sustainability targets set out in its Matching Grant. In diversifying its grant income, doubling its fee income and adding contract work as an income source, SEEP has strengthened its financial position and improved cost recovery. Lateral Learning C:\12-20-working\New Folder\SEEP evaluation final report.doc 2 · As a network driven by member needs and desires, the approach to planning by the SEEP staff and board was flexible, setting targets of achievement only at the higher level of objectives and results. The DIP and it companion document the Business Plan contained the framework used by the Board and Executive Director to guide the Network through the life of the grant. · The Matching Grant objectives are recognized by nearly all in SEEP as key ones to the success of SEEP's Institutional Development Services. These lay out the overall framework within which individual members contribute and make program decisions. Participating members who work as volunteers determine specific outcomes and activities. SEEP's goal of lateral learning would be diminished if a highly structure and rigid planning system were applied to its activities. The process of lateral learning in the SEEP Network is working well and is facilitated by the PVC Matching Grant. SEEP Network Member Benefits · There is a high level of participation in, and contributions to, SEEP activities by members. There is much member satisfaction with the SEEP programs and activities, which are meeting important needs of the member organizations. The types of member participation across all of SEEP activities is as varied as the needs of the sum of the individual members. · The learning agenda is member driven and responsive to member needs. SEEP products and activities are having an impact on member performance through the better understanding and application of best practices to their operations. · SEEP activities and structure as supported under the Matching Grant are effective in improving the understanding and application of microenterprise development (MED) best practices to member operations. The design of the SEEP products and the evaluation survey data show that SEEP activities are building some member capacity to transfer and apply best practices to field partners. Impact on Microenterprise Development · SEEP, through its research, tools and guides that are in wide use in MED institutions and programs, is having a major impact on standards, monitoring and evaluation in the general fields of microfinance and business development services (BDS). · The Working Groups are effective mechanisms for defining MED priorities as well as researching and disseminating MED principles, standards and practices. SEEP, primarily through its Working Groups (WG), is generating important products, training and understanding of MED principles, standards and practices in areas of high priority to practitioners. · Since the WG outputs are the products of the work of member volunteers, they are practical and field oriented, quickly disseminated among SEEP members, and benefit from inputs from multiple MED organizations. Network Development Services · The activities that SEEP has undertaken in the last three years to support network development have reached 18 organizations spread around the world. Its approaches have worked well as SEEP has linked with several organizations to leverage their resources. C:\12-20-working\New Folder\SEEP evaluation final report.doc 3 · There appears to be a high level of satisfaction with SEEP training, technical assistance and other support to networks and members. · The experiences and lessons learned by SEEP as a network are relevant to and useful for MED networks and their members in developing countries. SEEP is also particularly good in planning/strategy development and technical training to build network capacity because of its long experience with its own members doing the same. That SEEP can tap its member staffs to deliver NDS assistance makes the experiences more relevant and lower cost. Conclusions · SEEP has fully achieved the objectives set out in the MG for both the Institutional Development Services and the Network Development Services Programs. The impact of these on the development of MED best practices and their applications around the world is difficult to quantify, but its magnitude is substantial. · Activities under the SEEP matching grant have been technically varied, field oriented, practical and successful. They are based in a lateral learning network approach that is highly cost effective. · The success of SEEP rests on member motivation and skills as well as the incentives of a well functioning network. Bottom-up planning matched with shared vision and objectives are keys to the Network's success. · SEEP's success also rests on its shared vision to develop best or improved practices for microfinance and business development services in developing countries. · That SEEP members are practitioners has meant that the products are practical and ready for use by local MED institutions. The SEEP structure, especially its working groups, makes the most of the volunteers. Additionally, a small but strategically oriented staff has attracted the needed resources to complement the volunteer contributions and to maintain SEEP operations and structure. · SEEP and its Working Groups members have generally found the resources needed to complete important products that were beyond the capacity of its volunteer staff, but doing so has been, and continues to be a challenge. Many of its most important products, e.g. the impact handbook, the technical assistance guide, and BDS performance monitoring work, have required additional funding. · It is unrealistic to expect a high level of financial sustainability on the part of SEEP. It is not just a service organization to its members but a PVO in its own right with a development mission comparable to its members. Like its individual members, SEEP will be donor dependent but it also has the opportunity for substantial cost recovery from fees, sales and other income. · SEEP has been able to have a considerable impact on LDC national networks even though its interventions them have been modest in costs and scale. SEEP uses its own network experience to show fledging networks the right things to do in the right way. · Overall, this is a high impact, highly effective grant that reaches field organizations with useful tools and practices at a low cost to PVC. The cost effectiveness of the network approach and C:\12-20-working\New Folder\SEEP evaluation final report.doc 4 SEEP's successful use of it has resulted in a highly cost effective grant. USAID is getting excellent "bang for its buck". Recommendations · There is a need to further review the impact of SEEP IDS activities on member partners, service delivery organization in developing countries, especially to compare the tradeoffs and complementarity between the member assisted channels for field support and SEEP's own Network Development Services (NDS). · The SEEP Network approach offers a learning opportunity for successful operation of an association. As such, PVC may be able to use it to help other associations improve their structure, operations and impact. · The PVC Office is considering expansion of its support to networks such as SEEP and national associations like SEEP partner networks. PVC should consider a more complete review of SEEP and its partners as part of its research and background to inform its decision on expanded network support. This evaluation is primarily focused on the MG activities. · Although financial sustainability my not be a realistic objective for SEEP operations, increasing cost recovery and diversity in income sources should remain a financial objective for the organization. Increasing dependence on contract work for income should be approached with caution given that competitive awards are not well suited to SEEPs structure and operations as a volunteer dependent association. · SEEP should seek funding for a small Working Groups Fund that would be available to the WGs to complete, scale-up or disseminate activities or product that are beyond the capacity of the group to do by itself. · SEEP should complete a review of its communication and dissemination systems and strategy to take full advantage of the latest information and communication technologies. SEEP should explore the possibility of enhancing the communication capacity of its working groups to create a knowledge management community of practice that is able to reach more practitioners. Challenges From the interviews, responses on the questionnaires and especially SEEP leadership, a number of challenges surfaced that were either outside the evaluation scope of work or reflect circumstances that have arisen since the end of the for the matching grant that is being reviewed. These challenges centered around the question of maintaining the successful practices and products of SEEP as a member driven association in the face of growth, i.e. increased funding, new and expanded programs, more SEEP staff, expanding membership, greater diversity of member interests and greater knowledge of MED in general. They are included here as a response to USAID concerns about continued support to SEEP and its potential as a model for association support. There are several ways to express these challenges. Below they are grouped into three. Management of People, Programs and Funding: The SEEP budget is projected to reach $2.5 million in 2003 which is nearly $2.0 million more than the 1999 level of $545,000. The number of staff has doubled in the same period. New programs and grants are being added with more products and outputs planned. Directing and managing the staff, programs and outputs while keeping to its core work of meeting member needs is an operational challenge. Ensuring continued volunteer support as needed to C:\12-20-working\New Folder\SEEP evaluation final report.doc 5 keep SEEP products and outputs technically strong and practitioner relevant is a second major challenge coming from program growth. Increasing Diversity of Member Interests and Needs: SEEP membership is growing and the diversity of its member interests is expanding. Creating new working groups and other actions to respond to members and foster their participation in SEEP is a operational and motivational challenge. Knowledge Management: There is a enormous amount of MED information that is now available and needed by SEEP to make its programs and products successful and useful. Dissemination and communication technologies are advancing quickly to meet the challenge of knowledge management in the fast growing field of MED. The challenge for SEEP is to position itself in the center of this KM revolution in order to continue to be effective in responding to its member and sector needs for MED knowledge. 3.0 MATCHING GRANT BACKGROUND 3.1 Historical and Technical Context1 The Small Enterprise Education and Promotion (SEEP) Network is an association of 56 North American private and voluntary organizations which support micro and small enterprise programs in the developing world. The SEEP Network is incorporated as a non-profit organization in the state of New York, with 501-c-3 status. A list of its members is attached in Annex F. In existence since 1985, SEEP has built a community of members and member affiliates interested in and committed to a mutual process of learning. The Network's mission is to advance the practice of small and micro enterprise development among these organizations and other practitioners. In doing so, the SEEP Network provides a vehicle for the collective examination of member experience from which learning emerges. This lateral learning advances the professional development of its member staffs, increases program impact, fosters continuing methodological innovation, and informs policy and best practices. The SEEP Network goals have been to: · Define and promote professional standards among its membership; · Provide a forum for members to engage in joint research and training; · Develop and disseminate publications for field use; and · Serve as a communications center on sector issues and learning. SEEP has served as a center for best practice learning and dissemination through conferences, workshops and other meetings. It also completes technical manuals, training materials, computer tools, and working papers that its members have developed. Key technical areas of focus include: · Financial Services: design and management of financial services programs; financial projections; ratio analysis; and savings activities · Business Development Services: research on strategies for cost effective delivery of training, technical assistance, marketing, organization and other services · Poverty Lending: village banking principles and standards, strategies, and case experience; action research into credit dynamics of select programs worldwide · Institutional Development: frameworks and tools for developing effective enterprise development organizations · Evaluation: methods and tools for practitioner-led impact assessment at the client, business, and institutional level 1 Much of the background material on SEEP comes directly from its website, www.seepnetwork.org. C:\12-20-working\New Folder\SEEP evaluation final report.doc 6 · Information Services: reporting on the latest advances and issues in the field, as well as on network projects, NGO programs, resources, training opportunities, and employment opportunities; annual directories on member programs; periodic directories of member agency training materials · Network Development: diagnostic services, organizational development, and training of trainers for MED networks in developing countries An important structure for the work in these areas is the volunteer member Working Groups which presently include: (1) business development services, 2) client impact, (3) financial services, (4) poverty lending, (5) technical assistance, and the recent addition of (6) Microenterprise and HIV/AIDS. SEEP's member agencies in 1998 assisted over 5 million people world-wide, having expended roughly $112 million on enterprise development, and extending over $705 million in loans annually. Members provide financial services using individual, solidarity, village bank, credit union and cooperative methodologies; they offer training, technical assistance and technology services; they engage in strategies which link enterprise development to interventions in the fields of health, nutrition, vocational training, environment, and others. USAID's Private Voluntary Cooperation Office awarded a $450,000 matching grant (MG) to the SEEP Network for a three-year period running from September 11, 1998 through September 29, 2001. SEEP had submitted a MG proposal requesting $977,502 for a program consisting of two independent but complementary sets of activities: an enhancement of the existing Institutional Development Services (IDS) component and an addition of a Network Development Services (NDS) component. The IDS would build the capacity of the SEEP members and primarily take place in the US while the NDS would expand SEEP's proven capacity building services to the field through new partnerships with developing country MED networks. The overlap of the two programs create a synergy in the lateral learning and capacity building between members and SEEP-like networks in developing countries, made up of the affiliates and partners of the SEEP members. However, USAID funding was not adequate to cover the entire MG request. USAID provided approximately half of the funding, $450,000, which enabled SEEP to fully fund and implement the IDS component, with a substantial match from SEEP and its members for this component. No USAID funding was provided for NDS. SEEP did not have full funding for the NDS component at the time of the matching Grant proposal submission. It did have funding that could have been used for the match from the Banyan Tree foundation. Later SEEP was able to secure additional funding from the Ford Foundation and CGAP, the consortium of 28 donors for microfinance. The DIP budget included these funds in the match. The NDS program was implemented by SEEP in line with the requirements of other donors but essentially had the same objectives as set out in the Matching Grant proposal. The Matching Grant was modified twice to add funding and bring the total award to $563,330. In June 2000, $61,330 was added for increased funding towards capacity building strategies among members and staff and $52,000 was added in December 2000 for a conference and handbook on New Directions in Village Banking. There are four important characteristics to understand about SEEP at the start: (1) lateral network structure dependent on volunteers, (2) direct participation, (3) practitioner focus and (4) acceptance of diversity of technical approaches.2 The nature of SEEP as an association or network is an important structural characteristic to understand. Its structure is not hierarchical like a private firm or PVO, but lateral. It is characterized by reciprocity 2 The approach and ideas presented here were largely taken from the Evaluation of the SEEP Network: Marshall Bear, Jane Covey, and Beth Zwick, Institute for Development Research for USAID's Office of Private Voluntary Cooperation, December 9, 1994. C:\12-20-working\New Folder\SEEP evaluation final report.doc 7 and collaboration among people with no reporting relationship to each other, nor any formal contractual agreement. The evaluator does provide a caution here that the network structure per se is not enough to ensure its success and collaboration. Networks can be inefficient in completing tasks because of coordination and commitment lapses. Their reliance on personal relationships often restricts access. A few members may do most of the work and conflicts may grow over fairness of workload. When the network is successful, the information is high in quality and quantity within a context of trust and openness. Networks are a particularly effective form for joint learning and the transfer and quick adoption of technical knowledge. Whether positive or negative, in a network structure, the "processes of participation can be described as that of direct democracy, that is, individuals enjoy first hand involvement in decision making."3 Direct participation and decision-making by volunteers is key to the nature and operations of networks. SEEP focuses on practical issues and operations of microenterprise development programs and institutions. The member representatives are predominately senior level personnel, including several PVO CEOs while the typical staff member who participates in SEEP events is a technical officer often with substantial field experience. This fact coupled with the direct participation of the members in decisions on SEEP has meant that discussions, products and results are practitioner focused. A review of the SEEP products and agendas for its many meetings and trainings reveals that there is a great diversity of technical ideas included in its programs. There is acceptance among the members of the legitimacy and importance of a wide variety of technical approaches. For example, SEEP has established a new working group on Microenterprise and HIV/AIDS. Some in the field see the link of the commercial operations of microenterprise to the health and social welfare of HIV/AIDS programs as a threat to the market orientation and sustainability of microfinance institutions. SEEP is now on the cutting edge of finding ways to provide economic support to those affected by AIDS. Finally, knowing the nature of the SEEP member organization is helpful for understanding member reactions to and participation in SEEP. The 56 members are very diverse, but they generally fall in three categories with about a third of the membership in each. This is consistent with the finding of the 1994 evaluation by Bear et al. 4 The first category contains those NGOs that deal with microfinance institutions or the delivery of business development services for small and micro businesses. These organizations are focused almost exclusively on MF or BDS and are the technical leaders in the field. The second group of NGOs is made up typically of larger organizations that work in many sectors, an important one of which is microenterprise development. They often have a unit that works on MED. They are frequently the leaders in the integration of MED with other sectors. The third group is the least coherent. The members are usually smaller organizations that have a particular technical or geographic focus where microenterprise development or microfinance does play a role. Some examples of areas of focus are relief, conservation, poverty alleviation, cooperatives, training, health or education. These organizations are often coming to SEEP to learn the basics of microfinance and enterprise development in general or they have a particular approach to MED that would benefit from technical exchanges. 3.2 Matching Grant Goals and Objectives The Detailed Implementation Plan of May 1999 set out a hierarchy of objectives for the IDS and NDS components. SEEP included the NDS component in the MG agreement and Detailed Implementation Plan even though there was no funding provided by USAID/PVC. The NDS program was a priority for SEEP and a core piece of its strategic approach and mission. The following table summarizes the objectives taken from the Grant Agreement and Detailed Implementation Plan. 3 Ibid, page 4. 4 Ibid, pp 99-11 C:\12-20-working\New Folder\SEEP evaluation final report.doc 8 Table 1: Project Hierarchy of Objectives Goal: To increase the effectiveness of microenterprise development practices on the part of North American private development organizations and their southern partners Institutional Development Services Objective 1: Increase the institutional sustainability of SEEP Objective 2: Increase member participation in and contributions to SEEP's learning agenda Objective 3: Increase member performance through the application of best policies and practices Objective 4: Increase member capacity to transfer best practices to field partners Network Development Services Objective 1: Selected local field networks engaged in joint training/technical assistance activities with SEEP to strengthen field network structures, policies, programns and and traing skills Objective 2: Selected local field networks engaged in joint product development or contributing to regular SEEP publications Objective 3: Create a global communication infrastructure available to local field networks and their members Program Hypotheses A review of the program plans suggests that the following are the hypotheses underlying the goal and objectives of the SEEP Matching Grant. 1) SEEP as a member association can achieve a significant level of operational and financial sustainability. 2) SEEP member organizations and their staff benefit from SEEP activities such that significant volunteered staff time is given to sustain development and implementation of programs. 3) The outputs and products of SEEP programs and learning that flow from volunteer contributions are effective means to develop, understand and implement MED best practices. 4) SEEP-like associations of MED organizations can be established and sustained in a wide variety of developing countries. 5) SEEP can establish viable partnerships with MED networks in developing countries such that joint planning, training and other technical assistance results in stronger network and member programs and the application of MED best practices. See section 5.4.1 for a more detailed discussion of the hypotheses. 4.0 PURPOSE OF THE EVALUATION The general objective of this review is to fulfill the condition of USAID/BHR/PVC’s Matching Grant program to complete a final independent review of grant effectiveness, results and lessons learned. The purposes of this final evaluation are to: · Determine the degree to which initial objectives and outputs have been accomplished during the full five years of implementation. These outcomes are most clearly stated in the Detailed Implementation Plan. · Document the major dimensions and activities of the process of program implementation. C:\12-20-working\New Folder\SEEP evaluation final report.doc 9 · Examine the successes and failures of implementation, lessons learned, adaptations, modifications, targets of opportunity, discoveries, and unforeseen difficulties. · Make recommendations based on this experience for the sustainability, expansion, and improvement of the program approach and its accomplishments. In addition, together with other MG assessments, this evaluation is designed to assist PVC in: · Determining patterns and emerging issues across all MG funded programs; · Identifying the technical support needs for the PVC Office and its grantees; · Shaping new MG Request for Applications; · Developing internal and external documents to demonstrate the effectiveness of the MG program; and · Sharing lessons learned with the entire PVO community. PVC may use the evaluation information in its annual Results Report and in USAID's annual report to Congress. The second purpose of the evaluation is to help the SEEP Network assess and learn from its experiences in implementing the MG program over the past three years. Though technically a summative evaluation (in the sense that the MG is finished), it can actually be seen as a formative evaluation in that SEEP is implementing a follow-on Matching Grant and is continuing many of its activities into the future. This review should also help enhance SEEP understanding of member perspectives and priorities as well as strengthen program design, improve monitoring and evaluation and guide SEEP into the future. The scope of work for this evaluation appears in Annex C. USAID/PVC and the SEEP Executive Director developed it jointly. It does include a review of the NDS program that was funded by other donors. The review will be limited to the specific indicators and targets in the DIP and a survey of selected local country networks. 5.0 PROGRAM IMPLEMENTATION EVALUATION QUESTIONS Since many of the findings in this section are based on the four survey described in the methodology section above, some general information on the survey responses is provided here as background before moving on to the evaluation questions. Survey of: Total Number Sent Responding 1. Official Member Representatives 56 24 (43%) 2. Selected Working Group Members 24 16 (67%) 3. Board of Directors 8 4 (50%) 4. Directors/Coordinators of LDC Network 12 4(25%) As in any survey, the respondents did not answer all of the questions. Thus, the number of respondents for some of the questions was less than presented in the chart above. C:\12-20-working\New Folder\SEEP evaluation final report.doc 10 5.1 The Detailed Implementation Plan 5.1.1 DIP Completion and Accuracy Findings: The objectives and related performance plans found in the Planning Matrix of Development Implementation Plan of May 1999 set out baselines and end of project targets for both the IDS and NDS programs. The indicator column in some cases listed inputs rather than indicators but indicators were then specified under "baseline and targets". Annex D provides the Matrix with the actual EOP achievements added by SEEP. For the three of the four IDS objectives - participation, application of best practices and increased capacity to transfer best practices, SEEP has achieved the EOP targets. The achievement of these three objectives reflects a high level of results. Similarly, it achievements under NDS met the indicator targets but with other donor funding. The sustainability objective is problematic in that the three indicators and targets in the DIP were not useable. The first indicator was to seek higher levels of board member fund raising. The measurement of this objective is difficult since the fund raising is a joint undertaking with input often by members, board and almost always the Executive Director. To measure the Board Members direct contributions to increased SEEP funding was not feasible. The second indicator was increased SEEP fee income but the targets were set too low and measurement of income as a percent of the total budget proved an unreliable indicator as other income fluctuated year to year. The third sustainability indicator is on increased earning from publications by the introduction and implementation of a marketing plan. Earnings from the sale of publications are only in the $12,000 per year range and are not significant in relation to the total SEEP income. Nonetheless, publication sales have not changed significantly. SEEP did complete at the same time as the DIP a Business Plan that set out a strong vision, specific goals and objectives and a different set of sustainability targets and indicators for the organization. The relevant sections of the plan are also included in Annex D. The plan is summarized in the document as follows: Over the next 3 years, SEEP's goal and objectives to obtain more cost recoverable and financially viable programs will depend on increasing it fee-based income and continuing to diversify its grant support. NDS will be developed as an enterprise. Its activities will be cost recoverable by charging fees for services and products dispersed. At the same time, IDS will lessen its [dependence on] support on USAID/PVC by increasing its internal income generating activities and diversifying its grant resources. Specific to the sustainability objective in the MG, SEEP set out five indicators with targets. Four of the five are used in this evaluation to measure performance. The fifth is on Board Member assistance in fund raising which, as noted above, has a measurement challenge. SEEP did undertake efforts to use the Board Members to raise funds but the impact is not measurable. All four of the other indicator targets were achieved and are discussed in Section 5.2.2: Institutional Development Services. The targets are: · Increase IDS income by membership fees, publications and meeting fees by 100% · Have the MG represent no more than 50% of the IDS/NDS funding · All start-up and operating capital for NDS will come from non-MG sources · Secure three new sources of non-AID/PVC support SEEP set out to build a new fee-based services program for both the NDS and IDS programs. Over the life of the MG, SEEP succeeded in following a business model that first calculated break-even for all its C:\12-20-working\New Folder\SEEP evaluation final report.doc 11 courses and products and then SEEP set out to recover the 50-90% of the delivery costs from fees. SEEP will continue to use this approach to fairly price and recover its costs for services and technical assistance in the NDS program. As the capacity of regional trainers grows, SEEP and national level networks will see a greater ability to meet the cost of its services without donor assistance. In the NDS program, all networks expect to cost share but their income is minimal and thus SEEP and donors continued to provide for most of the expenses. 5.1.2 Quality of DIP Findings The DIP and its companion document, the PVC Matching Grant Business Plan, were useful in understanding the steps and activities needed to achieve the program objectives. The general quality of the documents was good. However, as will be discussed in more detail later in the document, the lateral learning, bottom up approach, found in many of SEEP's activities, had significant impact on the details of the planning decisions. The approach and objectives of the DIP and Business Plan recognized the special nature of the member- and practitioner-led nature of the SEEP Network. 5.1.3 Familiarity with the DIP and Design Findings The DIP and Business Plan prepared by SEEP identified four objectives in the Institutional Development Services Program. These are: Objective 1: Increase the institutional sustainability of SEEP. Objective 2: Increase member participation in and contributions to SEEP's learning agenda. Objective 3: Increase member performance through the application of best policies and practices. Objective 4: Increase member capacity to transfer best practices to field partners. As part of the evaluation, the organization representatives were asked: "In your experience at SEEP meetings and discussions, have the four objectives listed above been identified as key ones for SEEP?" Of the 17 member representatives responding, 15 responded in the affirmative. A similar question of the Board resulted in a positive response from all of the four that responded to the question. The survey of working group members resulted in positive responses to the question from all 16 respondents. However, three people did note the sustainability objective was not discussed or was thought to be less important to SEEP. Conclusions $ As a network driven by member needs and desires, the approach to planning by the SEEP staff and board was flexible. It sets targets of achievement only at the higher level of objectives and results. The DIP and its companion document, the Business Plan, contained the framework used by the Board and Executive Director to guide the Network through the life of the grant. $ The Matching Grant objectives are recognized by nearly all in SEEP as key to the success of SEEP's Institutional Development Services. These objectives lay out the overall framework within which individual members contribute and make program decisions. C:\12-20-working\New Folder\SEEP evaluation final report.doc 12 $ Participating members who work as volunteers determine specific outcomes and activities. SEEP's goal of lateral learning would be diminished if a highly structured and rigid planning system were applied to its activities. $ The success of SEEP rests on member motivation and skills as well as the incentives of a well functioning network. Bottom up planning matched with shared vision and objectives are keys to the Network's success. The planning process in a hierarchical organization was modified to match the lateral structure of the network. 5.2 SEEP Matching Grant Performance This section reviews the process of lateral learning in SEEP as well as the products, tools, training, research and other outputs of SEEP during the grant period. These products are listed by the Working Groups that largely developed them. Finally, the section assesses the two major programs of SEEP, IDS and NDS, in relation to their specific objectives as identified in the MG. 5.2.1 The Process of Lateral Learning Findings SEEP defines itself as an association of members that have created a lateral learning network. It has established a series of processes that nurture the participation of its members while at the same time providing a disciplined structure that yields products. Workshops, seminars, newsletter, websites and communications are the tools for networking, but above all else, the SEEP Working Groups form the backbone of the organization and its lateral learning. The MG provided much funding for the Working Group outputs as well as payment of a retainer/stipend for the group leaders. This was a major change from the all-volunteer arrangement and was initiated with this MG. Member and Board responses confirm that the facilitator retainer arrangement has greatly improved the management and productivity of the working groups. A sample of comments on the subject demonstrates the success of the arrangement. "In general, this has had a very positive impact, as this function requires considerable time organizing and communicating." "Very helpful. The leader plays an important facilitation role, ensures follow-up on major activities and contributes to the work of the group." "Products are developed more efficiently. The leader keeps the process on schedule and is able to do some of the busy work in getting products developed.'' A second productivity challenge for the WG is the need to find additional resources to complete a task, workshop or product that requires more time than is available to the WG members and leader. Many of the major WG outputs require such a level of effort. In the survey, the WG members responded to a specific question on this issue by noting the success of many of the WG leaders and the Executive Director in securing grant and contract support for the larger tasks. USAID's Office of Microenterprise Development’sIMS and MBP programs were identified as sources of additional assistance. One WG members noted, "SEEP leadership does a good job of sourcing funds." Another wrote: "The SEEP management has been doing a very good job of identifying different funding options and working with the groups to pursue them." However, some other WG members stated that their WGs are facing tasks beyond the group's capacity. One writes "it is always a struggle to get commitments." Conclusions C:\12-20-working\New Folder\SEEP evaluation final report.doc 13 · The initiation of a paid retainer arrangement for the WG facilitator has improved WG operations and outputs. The facilitators are able to organize the work process, assignments and follow-up to take the best advantage of the talents of the volunteers. There was some although not widespread concern about the fairness and balance in the use of paid individuals and volunteers on the same tasks. · SEEP and its Working Groups members have generally found the resources needed to complete important products which were beyond the capacity of its volunteer staff, but doing so has been and continues to be a challenge. Many of its most important products, e.g. the impact handbook, the technical assistance guide, and BDS performance monitoring work, have required additional funding. · The process of lateral learning in the SEEP Network is working well and is facilitated by the PVC Matching Grant. Recommendations Lateral learning may benefit from a more formal system of technical and operational exchange among members and their field entities. SEEP should explore the options for such an exchange. SEEP should seek funding for a small Working Groups Fund that would be available to the WGs to complete, scale-up or disseminate activities or product that are beyond the capacity of the group to do by itself. 5.2.2 Outputs Findings A listing of the key SEEP technical outputs produced during the MG is presented below, grouped under the six working groups and the AGM. WG are the main mechanism under which outputs result. 1. Business Development Services Working Group · Created and launched the website, SEEP Guide to Business Development Resources, which is an on-line learning product. It is a gateway for BDS learning that provides a range of basic training and technical information as well as links to other BDS sites. · Developed and conducted/tested a weeklong course, State of the Art in BDS. It is the first course to provide a comprehensive and practical step-by-step process to BDS program design that builds on the emerging market development model for BDS. · Completed research and learning that are helping to shape and develop the emerging BDS Market Development model. The focus is on work that is member driven and the education of members and donor staff. WG members have participated in numerous meeting and event to educate NGOs and others in this fast changing field. · Assisted the US Small Business Administration on the design and conduct of a seminar on business development in Cairo. 2. Client Impact Working Group · Conducted a series of client impact workshops to gather information on the assessment and monitoring practices of MFIs and MFI networks. The purpose of the on-going research and learning is to facilitate innovation in client assessment and monitoring practices as well as compile lessons learned on the subjects. C:\12-20-working\New Folder\SEEP evaluation final report.doc 14 · Worked with USAID and its contractor to help develop the SEEP/AIMS Impact Assessment Manual. This set of tools is helping to establish MFI client led programs. · Is producing an Impact Assessment and Monitoring Guide that will complement the SEEP/AIMS Impact Assessment Manual. This very practical guide includes case studies on MFI experiences in this area. · Developed and managed an on-line impact help desk to respond to practitioner questions. · Facilitated impact training for a large network of MFIs (INAFI) operating throughout Africa. · Developed and delivered a MF client impact training course. · Is collaborating with MicroSave Africa, a project of a group of donors, to offer training in market research tools, which complement the client assessment work. 3. Financial Services Working Group · Completed Financial Ratio guide and promoted its dissemination and use. The guide had significant impact on establishing standards and promoting their adoptions by MFIs. · Developed the Financial Ratios Spreadsheet Tool, which is providing a critical management instrument to monitor performance in relation to the Financial Ratio Guide. · Completed action research to identify best practices in performance monitoring. The results, descriptions of methods and lessons learned were recorded in document for practitioner use. · Developed and tested an MFI Board of Directors Training Manual. This new product provides a much needed set of materials and guides to train board members. 4. Poverty Lending Working Group · Undertook a consultative forum on village banking and completed a report on the State of the Art of Village Banking. It is a key reference for organizations and individuals who want to understand and plan village banking. · A second forum was completed and addressed some key developments in VB. It produced a Handbook on New Directions in Village Banking, which is becoming an important reference in the field. · Completed a widely distributed study on sustainability and Village Banking that suggest standards of performance and efficiency. It is a key study to further the process of setting standards for VB. · Developed and distributed the Village Bank Internal Account Management Tool Kit, which remains the best resource for VB, programs with internal accounts. · Planned and assisted with SEEP members a workshop in Rwanda on village banking and the basics of best practices for poverty lending. 5. Technical Assistance Working Group · Has completed extensive reviews with SEEP members and others on effectiveness of technical assistance and set out a conceptual framework for assessing TA. · Has completed a Handbook on technical assistance in the MF industry. It includes the principles for TA effectiveness, examples of different approaches, and useful references and resources. 6. Microenterprise and HIV/AIDS This is a new working group that has already run two successful seminars and has joined with a consortium of health and HIV/AIDS NGOs to explore needs and opportunities to help those individuals, households and communities affected by AIDS. C:\12-20-working\New Folder\SEEP evaluation final report.doc 15 7. Annual General Meeting (AGM) of SEEP Although not a working group, the AGM and the events scheduled around it provide important opportunities for professional exchanges. There is no professional association for microfinance or business development services, so the SEEP AGM is a key event where professionals can meet. Usually, 16 two hour workshops are presented at the AGM as well as longer meetings coordinated by the WGs, the most important of which are listed above. Conclusions $ The Working Groups are effective mechanisms for defining MED priorities as well as researching and disseminating MED principles, standards and practices. $ SEEP, primarily through its Working Groups, is generating important products, training and understanding of MED principles, standards and practices in areas of high priority to practitioners. $ Since the WG outputs are the products of the work of member volunteers, they are practical and field oriented, quickly disseminated among SEEP members, and benefit from inputs from multiple MED organizations. $ The MED products and outreach achieved by SEEP with matching grant support are significant and represent a high return on the PVC MG "investment". 5.2.3 Institutional Development Services (IDS) Performance IDS Objective 1: Increase the institutional sustainability of SEEP Findings The Business Plan set out four indicators and targets that measure the achievement of this objective. The first is to double the income from the sale of publications, membership and training fees. In Table 2 total income from fees and publication has in fact doubled from the 1998 level of $84,000 to $168,000 in 2001. Interestingly, the percent of the total budget from fees and publications is dropping as SEEP's total income has risen sharply. Table 2: SEEP Income and Fees ($000) SEEP 1998 1999 2000 2001 Total Income 506 545 730 1195 Fee/Service Sources: - Membership 28 32 36 41 - Training Fees 44 60 53 113 - Publication Sales 12 9 10 14 - Contract Work 85 75 200 227 Total Fees 169 176 299 395 Percent of Total Income 33% 32% 41% 33% Fees without Contract Work 84 101 99 168 Percent of Total Income 17% 19% 14% 14% One area not envisaged as a revenue source for SEEP was the contract and sub-contract work that it has undertaken with USAID, UNDP, CARE, ILO and African Development Bank. The largest of these have C:\12-20-working\New Folder\SEEP evaluation final report.doc 16 been the sub-contracts with Management System International and Development Alternatives, Inc in relation to their contracts with USAID's Microenterprise Development Office. SEEP has been able to identify contract work that complements its own mission and objectives. These resources have also helped SEEP to achieve a higher level of sustainability and diversify its income sources. One word of caution on donor contract work is that it is even more unpredictable than grant awards. USAID is presently in the process of awarding a completely new set of contracts for its Office of Microenterprise Development. The structure, arrangements and actual contractors are all changing. The member survey found that the fees were generally about right and that the value of the training and publications high. The one area of concern regarding pricing was the Membership fee for SEEP was thought to be high for about a third of the respondent. A summary of the actual responses from the Member survey follows. Summary of Responses from Member Survey The Board member survey yielded a similar view that the fees and pricing structure in SEEP is about right. Several mentioned that outreach in training and publications are more important than cost recovery. A sliding price scale for fees and publications was also mentioned to make the services and publications more affordable to LDC institutions and professionals. Several of the survey respondents pointed out that the SEEP members themselves are dependent on donations and grants to continue operations. This opens the question of whether SEEP should be expected to achieve a higher level of cost recovery than its members. One observation was that, “as long as donors prefer giving money directly to SEEP, rather than in unrestricted funding to a SEEP member that can use the funds to pay fees to SEEP, then we [SEEP] will be hard pressed to get much more than 30% of our funding from dues and fees.” On the membership fee, a common thought is that its purpose is to keep SEEP accountable to the members and the members accountable to SEEP. A second sustainability objective in the Business Plan is to have the Matching Grant represent no more than 50% of the IDS/NDS funding, which for all practical purposes was the entire SEEP budget from 1998 through 2000. Table 3 shows that SEEP has been able to attract substantial other donor grant support during the life of the MG, such that the PVC grant has dropped from 55% in 1999, the first full SEEP membership cost your organization $700 in1999; $750 in 2000; and $800 in 2001. In terms of value to your organization, the fee is: __8__High __15_About Right __1__Low Are the training and workshop fees and the cost of publications appropriate? What is there value? Training and Workshop Fees Cost of Publications __3__High __0__ High __17_About Right __23_ About Right __1__Low __0__ Low 3 No Answer __1__ No Answer Value of Training and Workshops Value of Publications _12__High _15__ High __9__Average __7__ Average __0__Low __1__ Low __3__No Answer __1__ No Answer C:\12-20-working\New Folder\SEEP evaluation final report.doc 17 year of the grant, to 29% in 2001 (combines income from MG I and II to SEEP). When the PLP Grant is subtracted for 2001 from income to leave just the NDS and IDS program, the matching grant still only represents 35% of the grant income. Table 3: SEEP: Income, Grants, and PVC Grant SEEP 1998 1999 2000 2001 2001* Income 506 545 730 1195 1052 Grant Income 333 361 422 794 651 PVG Grant I & II 8 198 245 228 228 PVC % of Total Grant 2% 55% 58% 29% 35% * The PLP Grant is subtracted from SEEP income to leave just IDS and NDS A third sustainability objective was to raise all start-up and operating capital for NDS from non-matching grant sources. This is in fact what SEEP did in raising money from CGAP, the Ford Foundation and the Banyan Tree Foundation. The final sustainability indicator was to secure three new sources of non-PVC support. Listed below are the awards obligated during the grant period of 1999-2001. USAID/OMD: subcontracts to DAI, 1996-01, $332.978 USAID/OMD: subcontracts to MSI, 1995-2001, $559,341 CGAP: 1997-2000, $326,125 Citigroup: 1999, $25,000 Ford Foundation: 1997-2000, $200,000 Thus, SEEP has secured five awards, two contracts and three grants. Conclusions In diversifying its grant income, doubling its fee income and adding contract work as an income source, SEEP has strengthened its financial position and improved sustainability. Given the grant dependence of its members and the economic development objectives that SEEP supports, financial sustainability is an unrealistic goal for SEEP. However, increasing cost recovery and diversity in income sources should remain financial objectives for the organization. IDS Objective 2: Increase member participation in and contributions to SEEP's learning agenda Findings This objective has two aspects in relation to members' participation/role in SEEP. One is a more passive role of just participating while the other is active, by contributing to the learning. Clearly there is a place for both in the same member organization and SEEP in general. The member surveys and especially the working group surveys consistently pointed out that the greatest benefits from SEEP came when the member staff most actively contributed to SEEP activities. Various versions of the saying "you get what you give" were often found in the surveys. In the member survey, the questions were asked about staff participating in setting the learning agenda and planning activities. 14 of the 24 members that responded reported on a specific action in which they were involved. The responses showed that they were not active in all areas but rather those most important to their organization. A single organization may contribute - help advance the learning agenda in one area; participate - educate itself - in several other areas; and not be involved with the remaining areas at all. Some organization that have large MED programs and staff have a higher profile - more volunteers working in SEEP. On the other hand, a small C:\12-20-working\New Folder\SEEP evaluation final report.doc 18 organization with only one or two staff in MED may have all staff involved with SEEP and thus have higher participation in terms of percent of staff involved. In terms of actual participation in SEEP activities, the member survey included a question and chart that listed the main forms of participation. The responses from the member survey showed a high level of participation. Level of Participation in SEEP Activities SEEP Activity 1999-2001 1. Member of SEEP Board 25% 2. Member of Working Group 96% 3. Authored all or part of publication 46% 4. Served as SEEP trainer, consultant or facilitator 17% 5. Purchased SEEP publication 79% 6. Attended the SEEP annual meeting 83% 7. Presented at SEEP annual meeting 54% 8. Attended other SEEP meetings/workshops 67% 9. Sent staff or partner/affiliate staff to field training 17% 10. Paid SEEP annual dues 100% In the Working Group survey on questions of participation, the responses identified a core of members that were active, some who attended meetings and contributed comments and some who just attended. This reinforces findings from the member survey. However, for many responding from the working groups the lower levels of participation were identified as a problem, in that they slowed or limited completion of tasks. There were many comments among the WG members that the WG were too big with too many members not contributing. There is obviously a desire to include members in the WG to broaden participation and experience, but if "recruitment' yields only passive members, working group efficiency and productivity will be reduced. Some level of dissatisfaction at the WG member level is to be expected, given the inherent challenge of combining participation and productivity in a volunteer organization for lateral learning. For the more general question of whether SEEP is meeting the needs of member organization, all 21 member representatives responding were positive but many did note that SEEP could meet only some of the member needs or should not try to meet all member needs. The responses on meeting the member needs icluded the following. Many good materials and jointly developed products have been born in SEEP. Even more [importantly] than the products is the way SEEP has led, consciously or not, in consolidating or bringing about a sharing/learning atmosphere among organizations that work in the area of microenterprise development. I feel that this sector has more collaboration than hardly any other thanks in part to SEEP. We [the member staff] get to participate with other experienced practitioers who are facing the same problems. Working with them, we get to incorporate their experience with our own and develop solutions that are much better and more complete than what any of us could come up with on our own. SEEP gives us the venue to constantly upgrade our learning and compare it to the best of what others are doing. SEEP as a network has met needs of our organization in that it has facilitated coordination with other similar institutions, and we have been able to participate in activities around themes which are very important to us. C:\12-20-working\New Folder\SEEP evaluation final report.doc 19 SEEP is meeting some of our training and MED orientation needs – I would not expect SEEP to meet them all. SEEP is one of the resources that we look to as a means of meeting our learning agenda. Conclusions $ There is a high level of participation in and contributions to SEEP activities by members. $ There is much member satisfaction with the SEEP programs and activities. They are meeting important needs of the member organizations. $ The type sof member participation across all of SEEP activities is as varied as the needs of the sum of the individual members. $ The learning agenda is member driven and responsive to member needs. IDS Objective 3: Increase member performance through the application of best policies and practices. Findings The finding in Section 5.2.1 that SEEP, primarily through its Working Groups, is generating important learning, products and training for MED best practices and policies does not automatically translate into the actual application of such policies and practices. Members were asked what activities of SEEP were most important to their understanding and use of best practices. The response showed that the working groups and the SEEP tools and publications were most important in achieving increased member performance but the AGM and the simple matter of associating with colleagues were also important. Every respondent identified at least one SEEP activity that was important or very important to their understanding and use of best practices. A summary of the results to the specific question to member representatives is below. Member Question: How important have the following been to increase your organization’s understanding and use of best practices? (We note that there is some overlapping activity within the listing.) Scale for Chart: 1 = very important; 2 = important; 3 = somewhat important; 4 = not important. Number of responses for each category Activity 1 2 3 4 NA* Working Groups 8 12 2 2 0 SEEP Tools and Publications 8 12 3 0 1 Annual General Meeting/Workshops 4 12 5 0 1 Technical Workshops and Training 1 8 6 2 6 Field Based Workshop 3 2 7 5 7 Associating with members at SEEP Sponsored Events 8 5 7 2 2 *NA: No Answer In asking the Working Group members to what extent their group is contributing to the development and transfer of best practices to SEEP members and others, the responses generally focused on the development of products rather than their transfer and application. There were only a few responses on changed performance, resulting from SEEP help on the application of best practices, but they are informative. C:\12-20-working\New Folder\SEEP evaluation final report.doc 20 When we first started developing our performance reporting system, I used things I had learned from [another PVO]. Later we got involved in the Financial Services Working Group (though we were not one of the main players) and used the SEEP ratios for our own system. We then used the Ratios publication to train our Implementing Partners in the use and importance of the ratios. And now our Implementing Partners are involved in local networks and helping to train other organizations in their countries in how to use the ratios. We are going through a similar process now with the 7Cs paper on Technical Assistance. We played a much more active role in this working group, and we have just redesigned our system for delivering technical services to make it consistent with the 7C principles. Conclusions $ SEEP products and activities are having an impact on member performance through the better understanding and application of best practices to their operations. $ SEEP activities and structure as supported under the Matching Grant are effective in improving the understanding and application of MED best practices to member operations. IDS Objective 4: To increase member capacity to transfer best practices to field partners through improved technical assistance strategies and methods Findings The field partners for members are their overseas offices, partner organizations and affiliates operating in developing countries. The following question was asked of the members with a summary of responses presented in the chart. Question: How important have the following been to increase your organization's capacity to transfer to field partners best practices? (We note that there is some overlapping activity within the listing.) Scale for Chart: 1 = very important: 2 = important; 3 = somewhat important; 4 = not important. Number of responses for each category Activity 1 2 3 4 NA* Working Groups 4 10 5 4 1 SEEP Tools and Publications 9 9 4 1 1 Annual General Meeting/Workshops 1 9 9 3 2 Technical Workshops and Training 1 5 10 2 6 Field Based Workshop 4 2 5 5 8 *NA: No Answer Only "Working Groups" and “Seep Tools and Publications" scored well and at a level comparable to the previous question on member understanding and use of best practicces. The extent to which SEEP activities build capacity to reach out to member partners and affiliates appears to be less than their impact on the members themselves - based on the responses to the two survey questions. The design and target audience for most of the tools are field organizations which may explain the relatively high score, but that does no explain the lower scores for the other activities. The matching grant objective is to increase the member capacity to transfer best proactices. There is no objective to have the field organizations adopt the best practices. Thus, we are left with questions on the nature of the impact on field operations as well as the need for more research on activity impact, the best channels for partner change, and the qualities of the local organization to best utilize SEEP assistance. C:\12-20-working\New Folder\SEEP evaluation final report.doc 21 Additional data from the survey is not avaiable to help clarify the result and better understand SEEP's impact on field partners. The survey does, however, provide some strong anectodal evidence about SEEP's impact on field partners. Survey respondents write: Over the period of the Matching Grant, SEEP has made important contributions to the development and transfer of best practices in the area of client assessment [to field based MFIs] The Poverty Lending WG is contributing learning, methods anmd tools to assist local institiutions to attain deep outreach in microfinance services while achieving scale and sustainability. The BDS Working Group has develped an on line resource for the field that provides both basic background information and hands-on technical tools for developing best practices programs…. Our patners are active in Networks associated with SEEP in Zimbabwe, Ghana, Eastern Europe and the Phillipines. Their participation has helped them not only incorporqate SEEP materials, but also learn how to teach then to others. Conclusions $ The design of the SEEP products and the evaluation survey data show that SEEP activities are building some member capacity to transfer and apply best practices to field partners. $ There is a need to further review the field impact of SEEP member activities on their partners, service delivery organization in developing countries, especially to compare the tradeoffs and complementarity between the member assisted channels for field support and SEEP's own NDS. 5.2.4 Network Development Services Based on its own successful experience as an association and lateral learnining organization, SEEP launched its Network Development Services Program in early 1999. Although it failed to receive matching grant funding, it found support from other donors. Also the Small Grants Program of USAID's Microenterprise Best Practices Program awarded grants to six networks to work with SEEP. In all 18 networks were assisted, twelve of which were in Africa. A list of networks appears in Annex F. To help run the NDS, SEEP has hired a full time manager for NDS in August 1999. The NDS vision is to promote sustainability with growth among national microfinance networks to foster best practices, national dialogue, and improved services to MED clients. The focus is on microfinance because its development over the past two decades has created in many countries a critical mass of potential members in a local network. The NDS program methodology comprises a three stage approach. 1. Building Relationships: SEEP first conducts site visits, assesses network capacity, and participates in network meetings and events. 2. Action Planning: With the network, SEEP helps assess member needs and reviews with the leaders and members the possible interventions that are part of the SEEP "toolkit". An action plan is developed jointly and set out in a Memorandum of Understanding committing both parties to a series of tasks. 3. Implementation and Follow-up: Activities in line with the plan are implemented. Monitoring, evaluation and follow-up are completed jointly. C:\12-20-working\New Folder\SEEP evaluation final report.doc 22 Most of the material and resources utilized in NDS were developed by SEEP Working Groups. Individuals from SEEP member organizations have helped in most cases to deliver the workshops, training sessions and technical assistance. Although funding was not part of the PVC Matching Grant, SEEP not only achieved the targets that were included in the DIP but in fact accomplished much more. Listed below are the activities and products arranged by the theree objectives set out in the Matching Grant for NDS. Objective 1: Organizational Development Assistance · Developed and tested an instrument to assess network capacity - the Network Assessment Capacity Tool. · Assessed 16 national networks who represent a total of 480 MFIs. · Conducted 20 network development and technical assistance assignments for capacity building and strengthening. The topics included staff retention, financial management, strategic planning, strengthening information and dissemination systems. · Provided intensive coaching to national networks in Ghana, Cameroon, Philippines and Mexico. · Representatives from 22 national networks attend the week of events and training around the SEEP Annual General Meeting in 1999 and 2000. Technical Training and Training of Trainers · Organized training of trainers on various topics for 10 national networks. · Jointly offereds financial analysis courses for 5 networks with 56 MFIs. · Completed imact training workshops for the members of the national networks in Kenya and Peru. · Fourteen national and regional African networks attended training in Zimbabwe on the Role of National Networks in Prerformance Monitoring. · Nine staff from national networks in Africa were trained at a workshop in Ghana on regulation, performance monitoring, capacity building and information dissemination. New Product Development · Developed a financial analysis course and translated it into French and Spanish · Translated the Lateral Learning document into Spanish and French · Produced a NDS profile document · Proceedings from the Zimbabwe conference published · Financial ratios Monograph translated into Russian (not funded by SEEP) Global Communication Infrastructure · Developed and moderated an electronic listserve for national networks with discussions on management of networks, financial performance monitoring and member services. · SEEP expanded its web site to include links or space for partner networks. Thus, for the frirst time siome networks had a site on the web and communications were improved among th network partners through the common linkages. An important document that SEEP uses in its general network development activies is Building Lateral Learning Networks: Lessons from the SEEP Network. This document is a self-assessment by SEEP. It identifies key characteristics and practices that support its programs and have been central to its success. The lessons learned are used with networks to guide them in their own development. The document is useful here not only as an example of a teaching devise but also as a complement to this evaluation of SEEP. As PVC considers expanding its assistance to networks in developing countries, the document is useful to understand key elements of network capacity building. It is attached as Annex G. C:\12-20-working\New Folder\SEEP evaluation final report.doc 23 Although SEEP has done much for MED networks in developing countries, much remains to be done. The typical network is still a youg organization, struggling to find resources and build its reputation with potential members SEEP assistance even to those that have benefitted from several SEEP interventions is small and can only meet some of the network needs. This does raise the question of the most appropriate tactic for the distribution of SEEP assistance. Should it spread its assistance out across many networks or concentrate on a smaller number. SEEP has been opportunistic to date in assisting networks. It has coordinated its work and partnered with other entities to stretch its resources. These linkages have helped SEEP assist some networks that would otherwise be outside of their reach. A survey was conducted of 12 of the networks to help get a picture of the strengths and weaknesses of SEEP assistance, the possible impact of the activities, the level of satisfaction with SEEP and priorities for the future. Only four networks responded so it is difficult to draw general conclusions. All four networks did benefit from training and exchanges with other networks and expressed satisfaction with these interventions. When asked about the impact of SEEP activity on their networks, three marked "considerable impact" and one felt there was a "major impact". In regards to SEEP's impact on network members, two reported "some impact" while two others saw "major impact". The SEEP staff provided for this evaluation, assessments of the MBP Microfinance Network Development grants and the training course on financial ratios. These also showed satisfaction with SEEP training and capacity building. In all the responses, the national networks identified needs in a wide range of areas and sought larger budgets and more of a long term relationship with SEEP. Conclusions $ The activities that SEEP has undertaken in the last three years to support network development are numerous and impressive in their breath of technical support to 18 organization spread around the world. There appears to be a high level of satisfaction with SEEP training, technical assistance and other support to networks and members. $ The expereinces and lessons learned by SEEP are relevant to and useful for MED networks and their members in developing countries. $ SEEP is a also particularly good in technical training and capacity building to the network members because of its long experience with its own members doing the same. That SEEP can tap its member staffs to deliver NDS assistance makes the experiences more relevant and lower cost. $ At this early stage of MED network development, SEEP has been able to have a considerable impact on those networks assisted even though the interventions have been small. This approach has worked well as SEEP has linked with several organizations to leverage their resources. Its apprach needs to be reviewed periodically as networks mature and donor and other funding for networks increases. (Note: SEEP has now focused on 10 core networks but with "mentor" status open to and existing with many.) 5.3 Impact Results The Matching Grant program objectives listed below are impact level achievements. Program Objectives Results (1) Increase member participation in and contribution to SEEP’s learning agenda (2) Increase member performance through the (1) EOPS: 75% increase of members involved as course facilitators and working group members (2) EOP: 50% of members contributed to best C:\12-20-working\New Folder\SEEP evaluation final report.doc 24 Program Objectives Results application of best policies and practices (3) Increase member capacity to transfer best practices to field partners practices products. (3) EOP: over 90% of SEEP members applied some or all of the SEEP MF ratios into reporting systems. 5.4 Assessment of Grant Model and Hypotheses 5.4.1 Appropriateness of Grant Hypotheses The SEEP Matching Grant Hypotheses are: 1) SEEP as a member association can achieve a significant level of operational and financial sustainability. 2) SEEP member organizations and their staff benefit from SEEP activities such that significant volunteered staff time is given to sustain development and implementation of programs. 3) The outputs and products of SEEP programs and learning that flow from volunteer contributions are effective means to develop, understand and implement MED best practices. 4) SEEP-like association of MED organizations can be established and sustained in a wide variety of developing countries. 5) SEEP can establish viable partnerships with MED networks in developing countries such that joint planning, training and other technical assistance results in stronger network and member programs and the application of MED best practices. The hypotheses are reasonable and appropriate but some clarifications are needed on the first three. The first hypothesis on sustainability needs to be seen in the context of SEEP members who are sustainable organizations but dependent on donations, grants and endowments to maintain operations. SEEP's mission in its most basic form is to enhance the capacity of its members. To expect that the costs of this economic and social development mission will be fully covered through SEEP fees, publication sales and other earned income would establish a financial requirement that none of SEEP members seek to or can achieve. SEEP as an association should seek donor support in line with the levels received to support the similar development objectives of its members. Nonetheless, SEEP should be able to achieve a significant level of cost recovery from fees, publication sales and other income. SEEP success, more than anything else, rests on its ability to attract member volunteers (Hypothesis 2). Its participatory, practitioner-led approaches draw top-notch staff from member organizations. The volunteeers are motivated by and learn from each other. Using working groups and other SEEP structures, they support the development, learning, and use of best practices through their work on SEEP tools, training and other activities (Hypotheses 3) . The practitioner relevance of SEEP and its outreach to so many MED organizations make it an attractive channel for donors to use to build MED capacity and extend best practices. 5.4.2 Replication and Scale-up of SEEP Approach C:\12-20-working\New Folder\SEEP evaluation final report.doc 25 Replication of SEEP in North America is not feasible since there is no room for a "second SEEP". However, other types of technical/developmental associations that are receiving USAID support can replicate the organizational and operational approaches of the SEEP Network. The findings, conclusions and general lessons learned about SEEP may be useful to other associations. The replication of SEEP-like networks in developing countries is in fact happening throughout the world. The findings and conclusions presented in the review of the NDS discuss the strengths and weaknesses of replication. 5.5 Advocacy Activities Under the Project SEEP does not undertake advocacy activities. It has a sister organization, the Microenterprise Coalition, which is a loose confederation of MED organizations that lobbies for reforms and MED assistance in the USA. It should be noted that some of the SEEP training, especially on microfinance standards, completed under the NDS does have an impact on the understanding of appropriate policies and government regulations and supervision in developing countries. A better understanding of best practices and policies has made training participants strong advocates for policy and practice reforms. Further, through the encouragement of local MED organizations, some government officials have participated in SEEP activities. After the SEEP training, they too are better informed on best practices and policies. Their position in government is helping the reform process in places like Uganda. 5.6 Implementation Lessons Learned $ The matching grant benefited from the fact that SEEP as a network reached out much further and deeper into the MED communities than any single PVO/NGO could do alone. SEEP success rests not just on the network structure but also on its effective operation, which includes openness and diversity in membership and technical approaches, member-driven activities and direct participation. USAID/PVC received a lot of "bang for the buck" with the SEEP matching grant. $ Small amounts of assistance are highly effective in the initial stages of the development of overseas MED networks. $ A lesson that SEEP identified as having learned from the matching grant was the importance of the grant in building its capacity. Program funding/grants have been easier for SEEP to obtain but they have had less of an impact on SEEP as an institution. $ Another lesson identified by SEEP was the importance of having NDS grantees prepare a business/implementation plan that included objectives, outcomes and activities as well as sustainability goals and income objectives. A detailed implementation plan alone was insufficient in that the network did not focus on its sustainability and income needs as a business once the grant was over. 6.0 PARTNERSHIP QUESTIONS Partnership questions are taken up under the NDS program in Sections 5.2.3. There is some limited discussion of SEEP member field partners and affiliates under IDS Objective 4 on building member capacity to transfer best practices (Section 5.2.2). The focus of the evaluation is on SEEP members rather than partners. No field visits were made to SEEP or member partners. C:\12-20-working\New Folder\SEEP evaluation final report.doc 26 7.0 PROGRAM MANAGEMENT 7.1 Strategic Approach and Program Planning Findings SEEP as a lateral learning network, uses a strategic approach that depends on participatory, member-led directions and actions. Clearly, its goal to support and build capacity in its members is enhanced through member leadership. That the member representatives are practitioners, strongly oriented to field programs/challenges, has kept SEEP working on innovative operational issues and the production of practical products to address these issues. As discussed previously the success of this bottom-up approach rests largely on the working group to help foster volunteer work and products. Equally important is the widespread understanding of SEEP's mission and general objectives not only by SEEP staff but by member staffs as well. This is evidenced by the survey results that showed the central place that the IDS objectives have among the board, staff, working groups and member representatives. The SEEP mission of developing and extending best or improved practices for microfinance and BDS is found throughout the network organization. The general mission is reinforced by the practice or policy that the findings and learning coming from SEEP work are quickly moved to the field and applied by members and others. That SEEP practice/policy is to accept diversity of technical approach means that members are not constrained in their bottom-up generation of issues and products. Conclusions $ The SEEP Network has a successful strategic approach that is dependent less on standard planning and hierarchies of objectives and more on a lateral learning process that flows from its network structure and participatory practices. $ SEEP's strategy and tactics to build and maintain lateral learning are highly successful and effective. 7.2 Country Initiatives SEEP does not have overseas offices and provides only limited periodic assistance to MED networks overseas under the NDS. This standard PVC evaluation section does not apply to this matching grant. 7.3 Conflict Management PVC is interested is learning if and how all of its matching grants relate to the problems of instability and conflict in developing countries. In relation to the SEEP matching grant, issues and questions of microenterprise development and microfinance related to HIV/AIDS and also natural disasters have been topics of SEEP member meetings. The issues, opportunities and challenges of MED in conflict and politically unstable countries have not been addressed. 7.4 Monitoring and Evaluation Findings As part of the PVC capacity building objectives for matching grants, the office is interested in building monitoring and evaluation into its partner PVO operations. SEEP has monitored and evaluated its matching grant in line with good practices. But the significant M&E outcomes lie in the tools, guides and frameworks it has helped to develop to evaluate microfinance and BDS practices. SEEP has been a leader in promoting standards for microfinance and the means to measure performance against those standards. C:\12-20-working\New Folder\SEEP evaluation final report.doc 27 The SEEP Financial Ratio Analysis for MFIs is a guide that can be found in practically every MFI. Other major SEEP supported guides and products for monitoring, evaluation and planning include: $ Learning from Clients: Assessment tool for Microfinance Practitioners $ Financial Performance Monitoring: A Guide for MFI Board Members $ SEEP’s Financial Services Working Group: Performance Monitoring Systems Project $ New Directions in Poverty Finance $ The Village Bank Internal Account Management Tool $ Seven Principles for Improving the Effectiveness of Technical Service Delivery in Microfinance: A Guide Financial Ration Analysis $ Financial Ratio Analysis of Micro-Finance Institutions: A Workshop For Managers, Facilitator’s Guide The State of Art in Business Development Services for Micro, Small and Medium Enterprises: Principles, Tools and Practices Conclusions $ SEEP has monitored and reviewed/evaluated the Matching Grant well. $ More important than the M&E of the MG itself, SEEP through its research, tools and guides that are in wide use in MED institutions and programs is having a major impact on standards, monitoring and evaluation in the general fields microfinance and BDS. 7.5 Overall Management In order to get a measure of the overall management of SEEP, member representatives and the board members were asked to rate management of activities. The results were: Board Responses Member Responses __4__Excellent __9.5_Excellent* __0__Good _13.5_Good* __0__Average __0___Average __0__Weak __0___Weak __0__Very Weak __0___Very Weak __2___No Answer * One member checked both excellent and good. The success of overall management rests primarily in (1) the small SEEP staff, (2) the Board of Directors and (3) the large group of volunteers. In 2000, SEEP estimated that volunteer time had a total value of just over $240,000. The staff, during most of the grant period, included the Executive Director, Dana de Kanter, the NDS Manager, Sharyn Tenn, and the Accountant/Finance Officer, Robin Munson who has worked 50-80% time. Interviews and responses to management questions on the surveys identify the critical importance of the Executive Director to SEEP's success and the high level of productivity of the staff in general. But rarely were positive comments made about the hard work of the staff without praise for the work of the SEEP volunteers. Comments include: I’m always amazed at what SEEP gets done with the size of staff it has. I have often used it as an example of what an effective network looks like. Of course, the reason it can get so much done is that most of the work is not done by staff, but by volunteers and consultants working together to produce research, documents and training manuals that address the common issues they face. I think SEEP has done a remarkable job with very limited resources and volunteers. I was motivated to volunteer because of the quality of the people who work with SEEP and the quality C:\12-20-working\New Folder\SEEP evaluation final report.doc 28 of the work they do. In addition, I learned a great deal more by working with the organization and its members than I ever could have otherwise. I certainly have received far more than I have given. It is absolutely amazing what SEEP has been able to accomplish with such a small staff and through help of volunteers and time/efforts of members. As described above, I think the volunteer/practitioner-driven mode of operation is one of SEEP’s major strengths. The level at which SEEP has been able to leverage its small staff is a genuine surprise and is commendable. The surveys also pointed out the importance of the paid WG facilitators to improved management and productivity. There were a number of suggestions to increase the size of the SEEP staff to strengthen management and technical capacity. These seem to come not from a dissatisfaction with the staff but a desire to build on its successes and strengths. In late 2001, SEEP did add one staff member to coordinate the activities of the working groups and their facilitators. Findings SEEP management is very strong and rests on the combined work of the staff and volunteers. 7.6 Sustainability The sustainability issue and performance were presented in the Section 5.2.2 discussion on the matching grant's IDS objective to increase the institutional sustainability of SEEP. 7.7 Financial Management Findings At about the time the grant started, SEEP added a new financial system (Peach Tree) and hired the part time financial manager. Up to that time, an administrative assistant handled finances. All financial reporting appears to have been submitted in a timely manner and met USAID standards. The final budget plans and actual expenditures are presented in Annex E and demonstrate the effective budget controls of SEEP. Grant Reporting to USAID was by input line items but SEEP also kept accounts by program activities to monitor costs and outputs. Outside audits of SEEP were completed each year of the grant. Total match for the SEEP grant was $1,001,368, which is essentially equal to the $1,022,431 set out in the MG agreement. The PVC grant proved to be core funding for SEEP that served to open the door to additional funding from other donors, including CGAP, Ford Foundation, the African Development Bank and USAID's Office of Microenterprise Development. While it is not possible to quantify the magnitude of the impact of the many SEEP products, its research/learning, capacity building and contributions to the extension of MED best practices are considerable compared to the total grant award of just $563,330. The high productivity of the small staff, the volunteer time of members and the outreach inherent in a member organization have meant that the cost effectiveness of the grant is extremely high. Conclusions $ The financial management of the SEEP grant was strong and met USAID requirements C:\12-20-working\New Folder\SEEP evaluation final report.doc 29 $ Matching amount for the grant substantially met the original grant plan. The MG provided SEEP with the core funds to leverage additional moneys. $ The cost effectiveness of the network approach and SEEP's successful use of it has resulted in a highly cost effective grant. USAID is getting excellent "bang for its buck" 7.8 Information Management Findings Two general sets of information are important within SEEP. The first is the operational information that keeps members and others informed of all meetings, formal, workshops, new programs and publications. The formal and informal networks within SEEP are effective in informing member staff and others of SEEP events. Word of mouth, the SEEP website and list serves, and other MED web sites were all identified as sources for operational information. The second information area is the broader one related to technical learning and product development. One of the planned results of the Matching Grant was to have an improved communications system that reaches out to developing countries. There is overlap between the two information areas, but many more members identified weakness in getting program information. Although there was no survey question asking to rate communications, the responses to open ended questions suggest that the communication systems are adequate, but several representatives pointed out the potential to do more. Not surprisingly communicating with field operations was the weakest area. List serves and e-mail were identified as effective in general. The BDS Working Group website that focuses on learning materials but also has links to other sources of information was identified by its members as an extremely useful communications and dissemination tool. Word of mouth and direct E-mails among WG members were also very effective communication tools. There were no specific suggestions in the survey responses on how to improve the communications except better and more use of e-mails and the SEEP website. Member responses to the request to rate SEEP's dissemination of its products and lessons learned to members and others were as follows. __7__ Excellent __10__ Good __6__ Average __0__ Weak __0__ Very Weak __1__ No Answer Conclusions $ Communication and dissemination of information by the SEEP Network and its various working groups are good. SEEP is meeting the communication and dissemination needs of members and others in the MED field. $ Although there is general satisfaction with the communication and dissemination of information in SEEP, the system could benefit from a professional review and direction. The present fast pace of change in electronic communication and its growing use in developing countries indicate that such a review would be beneficial now. $ The working groups are communities of practice as defined by the emerging knowledge management field. These technical networks offer the potential to be expanded through newer web based systems C:\12-20-working\New Folder\SEEP evaluation final report.doc 30 for broader knowledge management in their special fields. The extent to which they can be enhanced to reach a greater number of practitioners should be explored. Recommendations $ SEEP should complete a professional review of its communication and dissemination systems and strategy. $ SEEP should explore the possibility of enhancing the communication capacity of its working groups to reach more practitioners as a KM community of practice. 7.9 USAID Management USAID's oversight and backstopping of this grant has been timely and productive. The PVC Cognizant Technical Officer (CTO) knows SEEP well and has participated in numerous SEEP workshops and meetings. Interchanges between the CTO and SEEP staff and members, especially the Executive Director, are frequent. The CTO also receives informal feedback from SEEP members as part of his work with other PVOs. No issues were identified with the Grant Officer. 8.0 OVERALL CONCLUSIONS $ SEEP has fully achieved the objectives set out in the MG for both the Institutional Development Services and the Network Development Services Programs. The impact of these on the development of MED best practices and their applications around the world is difficult to quantify, but its magnitude is substantial. $ Activities under the SEEP matching grant have been technically varied, field oriented, practical and successful. They are based in a lateral learning network approach that is highly cost effective. $ SEEP's success rests on it base of member volunteers that share a vision to develop best or improved practices for microfinance and business development services. $ That SEEP members are practitioner has meant that the products are practical and ready for use by local MED institutions. The SEEP structure, especially its working groups, make the most of the volunteers. Additionally a small but strategically oriented staff has attracted the needed resources to complement the volunteer contributions and to maintain SEEP operations and structure. $ Overall, this is a high impact, highly effective grant that reaches field organizations with useful tools and practices at a low cost to PVC. C:\12-20-working\New Folder\SEEP evaluation final report.doc 31 ANNEX A: EVALUATION SURVEYS BOARD OF DIRECTORS SURVEY Name of Organization: Name and Position of Person Completing Survey: Phone number if need to be contacted re the survey: A. Sustainability A key objective of the USAID Matching Grant was to increase SEEP’s institutional sustainability. SEEP’s income comes from fees for meetings and the sale of publications. This income amounted to about a third of its total budgets in 2000 and 2001. Donor funding essentially provided the balance. To what extent should donors support SEEP as an agent to help build PVO capacity among SEEP members? What does your answer mean in terms of the level of donor support of the SEEP budget and programs? SEEP uses donor funds for most of its activities but its operations are driven by its members. Has it balanced donor and member needs and priorities? Are SEEP’s fees and pricing of publications appropriate? B. Strategic Objectives Has SEEP been strategic in setting its program and technical directions and priorities? What role has the board played in the planning process? How well have members participated in the planning? In your experience as a SEEP board member have the four objectives of the matching grant listed below been important for SEEP? 1. To increase SEEP’s institutional sustainability 2. To increase member participation in and contribution to SEEP’s learning agenda 3. To increase member performance through application of improved practices 4. To increase member capacity to transfer these best practices to field partners through improved technical assistance strategies and methods To what extent has SEEP been able to facilitate the transfer of best practices to members? SEEP has been good at capacity building and helping its members who are new to MED or still have much to learn. What should it do for it members who are now advanced technically? C. Management and Communications Please rate SEEP’s management of its activities? _____Excellent _____Good _____Average _____Weak C:\12-20-working\New Folder\SEEP evaluation final report.doc 32 _____Very Weak SEEP has operated with a very small staff and depended on volunteer work. Has this approach been adequate to achieve SEEP objectives? In your experience volunteering for SEEP what has motivated you? The MG funding has gone to strengthen the leadership of the working groups. How have the WG improved? What weaknesses still exist? Have the working groups addressed the priority needs of the members? To what extent has SEEP used program monitoring and evaluation to adjust and direct its plans and new activities? Assess SEEP’s communication systems/channels to reach members and local networks in developing countries. To what extent has SEEP disseminated its products and lessons learned to members and others? ________________________________________________________________________ SEEP MEMBER SURVEY Name of Organization: Name and Position of Person Completing Survey: Number of years with the organization: Number of Years participating in SEEP Activities Phone number if need to be contacted re the survey: A key measurement tool for the USAID Matching Grant (MG) was the Detailed Implementation Plan that set out objectives, indicators and targets for the life of the MG. Listed below are the objectives for the Matching Grant I (9/30/98 to 9/29/2001). For each of the objectives we ask you to respond to some questions to help in the assessment and evaluation of the MG. 1. To Increase SEEP’s Institutional Sustainability SEEP membership cost your organization in 1999 - $700; 2000 - $750; and 2001 - $800. In terms of value to your organization, the fee is: ____High ____About Right ____Low Are the training and workshop fees and the cost of publications appropriate? What is there value? Training and Workshop Fees Cost of Publications ____High ____ High ____About Right ____ About Right ____Low ____ Low C:\12-20-working\New Folder\SEEP evaluation final report.doc 33 Value of Training and Workshops Value of Publications ____High ____ High ____Average ____ Average ____Low ____ Low 2. To increase member participation in and contribution to SEEP’s learning agenda In the past three years (1999-2001), how does/has your organization participated in SEEP? Please fill out the following form. SEEP Activity YES NO 1. Member of SEEP Board 1 2. Member of Working Group 2 3. Authored all or part of publication 3 4. Served as SEEP trainer, consultant or facilitator 4 5. Purchased SEEP publication 5 6. Attended the SEEP annual meeting 6 7. Presented at SEEP annual meeting 7 8. Attended other SEEP meetings/WORKSHOPS 8 9. Sent staff or partner/affiliate staff to field training 9 10. Paid SEEP annual dues 10 11. Participated in other SEEP related activity not covered above. Specify: 11 What has been the most important activity or action undertaken by SEEP to increase your participation in and contribution to the SEEP Network and its learning agenda? For your organization, please rate SEEP’s learning agenda and program activities? (Check appropriate description) Learning Agenda/Program ____Very Strong ____Strong ____Average ____Weak ____Very Weak For all members of SEEP, please rate SEEP’s learning agenda and program activities? (Check appropriate description) Learning Agenda/Program ____Very Strong ____Strong ____Average ____Weak ____Very Weak To what extent have you had the opportunity to set a part of the SEEP learning agenda? Has your organization helped in designing and/or planning SEEP activities? What is the impact of different member needs and priorities on SEEP activities? Is SEEP meeting needs of your organization? C:\12-20-working\New Folder\SEEP evaluation final report.doc 34 3. To increase member performance through application of improved practices How important have the following been to increase your organization’s understanding and use of best practices? (We note that there is some overlapping activity within the listing.) Scale for Chart: 1 = very important 2 = important 3 = somewhat important; 4 = not important Activity 1 2 3 4 Working Groups SEEP Tools and Publications Annual General Meeting/Workshops Technical Workshops and Training Field Based Workshop Associating with members at SEEP Sponsored Events 4. To increase member capacity to transfer these best practices to field partners through improved technical assistance strategies and methods How important have the following been to increase your organizations capacity to transfer to field partner’s best practices? (We note that there is some overlapping activity within the listing) Scale for Chart: 1 = very important 2 = important 3 = somewhat important; 4 = not important Activity 1 2 3 4 Working Groups SEEP Tools and Publications Annual General Meeting/WORKSHOPS Technical Workshops and Training Field Based Workshop In your experience at SEEP meetings and discussions have the four objectives listed above been identified as key ones for SEEP? In your opinion which members have been the focus of SEEP’s Learning Agenda? ____Member organizations that are new to the field ____Member organizations that are leaders in the field ____All SEEP members – a balanced approach was used to reach all members. C:\12-20-working\New Folder\SEEP evaluation final report.doc 35 Comments: SEEP has operated with a very small staff and depended on volunteer work. Has this approach been adequate to achieve SEEP objectives? In your experience volunteering for SEEP what has motivated you? Please rate SEEP’s management of its activities? _____Excellent _____Good _____Average _____Weak _____Very Weak Assess SEEP’s communication system to reach members and local networks in developing countries. One of the objectives of the USAID matching grant (1998 – 2001) was to help SEEP improve its communications. How has communications changed over the life of the MG Please rate SEEP’s dissemination of its products and lessons learned to members and others? _____Excellent _____Good _____Average _____Weak _____Very Weak ________________________________________________________________________ WORKING GROUP SURVEY Name of organization: Name and position of person completing survey: Number of years with the organization: Member of which Working Group: Number of years participating in SEEP working groups: Phone number if need to be contacted re the survey: A key measurement tool for the USAID Matching Grant was the Detailed Implementation Plan that set out objectives, indicators and targets for the life of the MG. Listed below are the objectives for the Matching Grant I (9/30/98 to 9/29/2001). 1. To increase SEEP’s institutional sustainability 2. To increase member participation in and contribution to SEEP’s learning agenda 3. To increase member performance through application of improved practices 4. To increase member capacity to transfer these best practices to field partners through improved technical assistance strategies and methods In your experience at SEEP meetings and discussions have the four objectives listed above been identified as key ones for SEEP? To what extent do you believe that your working group is contributing to the development and transfer of best practices to SEEP members and others? C:\12-20-working\New Folder\SEEP evaluation final report.doc 36 What has been the impact on your working group of having the WG leader/facilitator on a contract that pays for some of his work? List and assess the importance of your WG products to the technical field. Describe WG strengths and weaknesses. Has your WG been strategic in setting its program and technical directions and priorities? How well have WG members participated in the planning? HOW ARE WG MEMBERS RECRUITED AND RETAINED? How do SEEP and the WGs find support for the completion of major tasks and products that are beyond the available time of the volunteer WG members? SEEP has operated with a very small staff and depended on volunteer work. Has this approach been adequate to achieve SEEP objectives? In your experience volunteering for SEEP what has motivated you? To what extent has WG disseminated its products and publications to members and others? ____________________________________________________________________ Survey of SEEP Assistance for Network Development: Background Name of network: Location: Number of members: Name and title of person completing the survey: Number of years with the organization: Phone number if need to be contacted re the survey Questions 1. How has your organization participated in the SEEP NDS? Name, date and location of SEEP training/workshops: Number of individuals attending? C:\12-20-working\New Folder\SEEP evaluation final report.doc 37 Main thematic areas of training/learning? SEEP publications used by Network? By members? Network staff that attended SEEP general meeting since 1999? Network staff that attended other SEEP meeting since 1999? Which ones? Any direct support from SEEP - Amount and purpose of grant from SEEP Other form of participation? 2. How would describe the overall impact of the SEEP assistance on your network operations? On your members? Please mark appropriate boxes. Your Network Your Members No significant impact Minimal impact Some impact Considerable impact Major impact 3. Has the SEEP NDS program been useful and effective in building the capacity of your network? If so how? Has SEEP had other impacts on your Network? 4. Assess your satisfaction with the SEEP’s NDS support to you and your members: Your Network Your Members Very Satisfied Satisfied Dissatisfied Very Dissatisfied 5. What were the major constraints, if any, to the NDS SEEP partnership for you? 6. What has worked well in the NDS program? 7. What additional services would you like SEEP to offer to your network? 8. What are your recommendations on how to improve the SEEP NDS program? 9. Do you have a monitoring and evaluation system in place to help determine the impact of your programs on members? (Surveys of members? self assessments? use of standards by members ?) Describe the M&E system briefly. 10. Budget and Sources of Support Approximately what is your network’s annual budget? What at is the approximate percentage of the overall budget supported by member contributions? What at is the approximate percentage of the overall budget supported by grants and donations? C:\12-20-working\New Folder\SEEP evaluation final report.doc 38 ANNEX B: KEY EVALUATION DOCUMENTS · PVC Matching Grant Program Proposal: Institutional and Network Development Services for Enterprise Development Organizations, SEEP, December 1997. · SEEP Matching Grant Agreement and Amendments for Institutional Development Services for Enterprise Development Organizations: USAID September 11, 1998. · PVC Matching Grant Program (1998): Detailed Implementation Plan, SEEP May 1999. · PVC Matching Grant Program (1998): Business Plan, SEEP May 1999. · An Assessment of the SEEP Network Survey Results Based on Interviews with Thirteen NGOs Receiving Matching Grants: Jeffrey Ashe and Julie Kelly Detweiler, AMA Technologies, March 2000. · Evaluation of SEEP Network: Marshall Bear and Institute for Development Research - Jane Covey and Beth Zwick, December 9, 1994. · SEEP Network Annual Reports, 1998, 1999, 2000 and 2000-2001. · Building Lateral Learning Networks: Lessons form the SEEP Network, undated. ANNEX C: STATEMENT OF WORK EVALUATION SCOPE OF WORK “Evaluation is a relatively structured, analytical effort undertaken selectively to answer specific management questions regarding USAID-funded assistance programs or activities.” (USAID ADS chapter 202.4). An evaluation scope of work (SOW) is a plan for conducting an evaluation. A good SOW provides clear directions to the evaluation team. PVC uses information from the evaluation of the programs it funds as part of a yearly results reporting process. In order to get more consistent information across all Matching Grants (MG) funded programs a standard evaluation format is used. The questions in this evaluation SOW template are the questions that PVC is asking in all programs. The PVO and their local partners will need to review this template and add sections or questions that reflect their specific information needs. ELEMENTS IN THE SOW I. PROGRAM IDENTIFICATION Include the following: PVO name Cooperative agreement number Date of the evaluation II. PROGRAM BACKGROUND Include the following information: ß Provide basic information on the program that will be evaluated Include a short statement on: - History of the MG program and SEEP - Current implementation status - Members and Network Partners - ß Provide program planning matrix and describe the following: ÿ Objective ÿ Indicators ÿ Data from baseline studies or description of the status of the intervention at the beginning of the project. ß Indicate what information and data are available for the external evaluator. PVC already sent a document that will give you an excellent idea of the documents that should be assembled and preparation needed prior to an evaluation. ß Include documentation of any changes that have taken place since the initiation of the program. Include decision by PVC not to support the NDS component of the SEEP proposal. III. PURPOSE OF THE EVALUATION This section should contain two components: 40 (1) To identify the evaluation audiences (2) To establish a set of evaluation questions that are relevant to each audience. Outline the information needs of the evaluation audience (PVC, the PVO and local partners), and how each partner will use this information. ß Who wants the evaluation information, ß What do they want to know, ß What will the information be used for, ß When will it be needed, and ß How accurate must the information be? For example: The final (or mid-term) evaluation fulfills the requirements of the USAID/BHR/PVC Matching Grant (MG) Program. The MG program will use the information to: assess how well the MG is meeting its objectives; determine patterns and emerging issues across all MG funded programs; determine technical support needs for grantees shape new RFAs and to review of any follow-on proposals; develop internal and external documents to demonstrate the effectiveness of the MG program and to share lessons learned with the entire PVO community. PVC will use information outlined in the SOW template in its annual Results Report and in USAID's annual report to Congress. Achievements cited in the evaluation need to be supported by evidence and should be verifiable. Observations on data quality or constrains to interpretation should be stated as data from these evaluations is used for USAID reporting purposes and is subject to audits. Technical/program opinions and observations are an important element of the evaluation --- but should be stated as the evaluator estimate, opinion or forecast. IV. THE EVALUATION QUESTIONS ß PVC EVALUATION QUESTIONS. The following are a set of questions that the MG division is asking in all evaluations. These questions relate to the objectives of the MG division and PVC’s strategic plan. The evaluator or evaluation team will assess the following program and institutional questions, provide evidence, criteria for judgment and cite data sources. The PVO will need to tailor the SOW to reflect their own and their local partners information needs by adding questions into each section, or adding additional sections if needed. A. Program Implementation 1. Assess progress towards each major objective ß Based on Detailed implementation Plan determine if the program objectives have been met, partially met or were unattained. This is the single most important element the evaluation must document and discuss. In addition to the discussion of project results in the text of the evaluation, this information should also be put into matrix format. List each objective, and key outcomes at the effects and/or impact level. In the text: ÿ Identify major successes and constraints in achieving objectives and unanticipated effects. 41 As part of this discussion, comment on SEEP capacity to do program monitoring and evaluation. Note any constraints that prevented SEEP from measuring achievement of program objectives. (A more detailed discussion of monitoring and evaluation should be covered in the M&E Section of the report) ÿ Identify if the project had a detailed implementation plan and the familiarity of staff and members with the project design, implementation plan and monitoring and evaluation plan and data. ß Assess effectiveness of IDS and NDS approaches and assumption that underlie the activities. To what extent has SEEP had replicating the NDS activities from country to country? ß SEEP has been good at capacity building and helping its members who are new to MED or still have much to learn. What should it do for it members who are now advanced technically? ß Discuss what SEEP, it members and the local networks have “learned” implementing this MG. Identify if these “lessons learned” have been used by members or others? 2. Assess the status of SEEP’s work with its members and their networks. Define local partner networks under Network Development Services and describes SEEP’s work with these local Networks. Describe the process that SEEP used to accept local network participation in NDS. (Note: MG funding was not provided for the NDS program. SEEP found other support and the program was implemented in line with other donor conditions.) Include a description that: ß Categorizes local level network partners. Are the partners: NGOs, affiliates of the PVO, private or commercial groups, cooperatives, community-based organizations, regional or local governments or intermediate service organizations? ß Identifies the type of mechanism employed with each partner, i.e. MOU, sub-grant, contract. ß Outlines the roles, responsibilities and decision-making responsibilities of the partners. ß Did the PVO do a formal assessment of local partner capacity and develop plans to build their capacity? ß To what extent can SEEP document change in it member’s capacity? ß What were the major constraints to effective partnerships with local Networks? 3. Describe member support and other donor support to local networks. 4. Assess the local Networks satisfaction with the partnership 6. What has been learned about local Network organization and management? B. Management Capacity/Institutional Strengthening The objective of the MG is to build SEEP’s organizational and technical capacity. This section of the evaluation should assess change in SEEP’s operational and management capacity (organization, structure or quality of planning and management) as a result of PVC grant. ß Strategic Approach and Program Planning Have changes occurred in SEEP to: 42 ÿ manage the planning process --- program renewal, strategy integration, project design; ÿ address over-arching program issues of member participation, technical focus and sustainability, ÿ Use performance data to adjust and develop strategic plans? What are the strengths and weaknesses of being member driven organization? What is the impact of different member needs and priorities on SEEP activities? Is SEEP meeting needs of all members? How have surveys of members been used to direct programming and member services and products? SEEP uses donor funds for most of its activities but its operations are driven by its members. How does it balance donor and member needs and priorities? Assess if there are systems in place to oversee program activity. (Consider related Supervision/HR question below.) ß Working Groups (WG) Describe the make-up and operations of WG. What is the impact of the MG on the way WGs operate? What is the impact of retainer contracts for WG facilitators? Describe WG products, strengths and weaknesses. Assess importance of the WG products to their technical field and members. How has SEEP expanded its technical areas/coverage in the Working Groups? Has the expansion or contraction been member driven? How do SEEP and the WG leaders keep the right people on the WG? How do SEEP and the WG leaders keep the WG members motivated? How do SEEP and the WGs find support for the completion of major tasks and products that are beyond the available time of the volunteer members? . ß Field Issues and Network Development Services Identify which local networks supported by SEEP are in USAID assisted countries. In USAID supported countries, assess local network cooperation and coordination with the USAID mission, the Microenterprise Development Office and other development partner programs. SEEP does not engage in policy and advocacy in the US. Overseas with its partners, its work can play a role through education and training. Assess SEEP provided education and training that assists local networks and members in advocacy activities: issues, goals, and results. If the country or program area has a history of violent conflict, other man-made/natural disasters, or food insecurity, consider SEEP’s contribution to: 43 ÿ conflict prevention, mitigation, ÿ resolution or post-conflict transition ÿ local partner contingency plan to ensure the safety of program ÿ Staff and program continuity. ß Monitoring and Evaluation Has SEEP implemented a process and put into place a sustainable system to monitor project performance and collect results (effects or impact) data? Provide evidence that the project: ÿ Established results oriented objectives and valid indicators for the technical intervention and capacity building components in the project; collected valid baseline data, and made realistic plans to collect end-of-project data and analyze differences; analyzed performance data and used findings to manage the project. If this is a final evaluation, has the SEEP acted on recommendations from mid-term evaluation? ÿ Improved the knowledge and skills of its staff and members on how to measure performance and analyze data. ÿ Transferred monitoring and evaluation skills to members? - What changes have occurred in the their capacity to measure program performance and impact? ÿ Have local Networks increased M&E in their own activities (non-PVC-funded programs) as a result of skills gained through NDS? - What would accelerate the capacity of the local partners to document performance? Determine if SEEP has used the MG to develop a sustainable capacity at headquarters to monitor project performance and measure effects and impact. Has SEEP: ÿ fostered analysis and self evaluation or conducted quantitative or qualitative analysis to refine interventions; ÿ conducted periodic review of performance data by activity and taken actions as a result of review; and ÿ institutionalized performance monitoring and impact evaluation systems developed with MG funds into other non-PVC grant funded programs. What were the biggest constraints to improving project monitoring and evaluation and what are the recommendations for PVC and SEEP? ß Sustainability Does the project have a system for addressing financial or operational sustainability? Does the project have a business plan? Describe the program elements, financial or operational, that are intended to be sustained (objectives); the means for judging if the sustainability objectives have been achieved (indicators); and sustainability achievements and prospects for post-grant sustainability. Identify the cost-recovery mechanisms. Describe the achievements of these mechanisms and provide an estimate of the magnitude of the system, for example, provide a ratio of costs recovered to operational expenses. Other Management Systems Financial Management ÿ Are adequate financial monitoring systems in place? 44 ÿ Has the program leveraged additional resources (beyond the match)? ÿ How cost-effective is the financial system? Information Management and Technology Assess SEEP’s communication system to reach members and local networks in developing countries. How has it been developed over the life of the CA? Comment on the utility and timeliness of PVOs required reports. To what extent has SEEP disseminated its products and lessons learned to members and others? ß Logistics Comment on the adequacy and timeliness of SEEP’s material inputs and office operations. ß Supervision/HRD Assess if there were sufficient staff with the appropriate technical and management skills to oversee program activity. ß USAID Management Comment on USAIDs oversight and backstopping of this cooperative agreement. ß Lessons learned Cite the major management lessons learned and recommendations V. EVALUATION METHODOLDOGY Give a brief description of the evaluation methodology use. - Evaluation approach - Methodology and instruments - Criteria used for judgment, data source, and data analysis. A. Approach The PVO’s program was developed and funded prior to the Agency's emphasis on results￾oriented program designs and the development of PVC’s Strategic Plan. The data from all PVC￾funded programs is critical to PVC's ability to report on achievements against the Office's Strategic Plan. Until all current PVC-funded programs have made the transition to a more results-oriented project plans, it will be necessary for the evaluator to conduct a team-planning meeting with the PVO and local partners to: ® refine and consolidate the purpose-level objectives and outputs into a set of results-oriented objectives; and ® Agree upon a set of appropriate indicators against which the evaluation will assess the achievement of project results outlined in the SOW and will be judged. And where necessary, identify criteria for judgment. B. Methodology The Evaluator will: 45 ® explain the appropriateness of using the data collection approaches; ® use the Agency's microenterprise (ME) indicators to assess the status of the ME intervention; ® document data sources (data constraints, quality, etc.); and ® Provide, a copy (electronic or paper) of all primary data collected and analysis performed. VI. TEAM COMPOSITION AND PARTICIPATION INSTRUCTIONS: Based on tasks outlined and the emphasis of each evaluation section determine skills needed and who will participate in the evaluation team ---- PVO, NGO and AID staff. Outline: - Roles and responsibility of team leader and members - Language requirements - Technical expertise, or country experience - Evaluation methods and data collection expertise VII. SCHEDULE INSTRUCTIONS: Determine: - Time needed at headquarters - Time needed in the field - Time necessary for report writing VIII. REPORTING AND DISSEMINATION REQUIREMENTS INSTRUCTIONS: - The SOW will serve as the outline of the report - Delivery schedule - Review/revision policy ANNEX D: DIP AND BUSINESS PLAN PERFORMANCE MATRIXES Program Goal: to increase the effectiveness of microenterprise development practice on the part of North American private development organizations and their southern partners by increasing its own sustainability Objectives Indicator Baseline & Targets Results Headquarters 1. Operational sustainability (a) Adoption of key operational changes: - set board funding targets (diversify funders) - establish fee for service products and activities - establish publications marketing scheme (1) Baseline: No board income revenue EOP: 10% board input of diversified donor funds (2) Baseline: <5% revenue generated from fees for service EOP: > 10% revenue from fees for service (3) Baseline: No marketing activities pursued EOP: >10% of R&D Business Plan indicators and targets used as measure of sustainability. Programs Objectives: 1.Enhanced IDS: (a) increase member participation in and contribution to SEEP’s learning agenda (b) increase member performance through the application of best policies and practices (c) increase member capacity to transfer best practices to field partners (a) begin international working groups (b) expanded product development (c) increase participation in annual meetings (d) increase capacity building activities by SEEP members on behalf of partners (1) Baseline:< 30% member agencies have sustained participation in SEEP EOPS: > 50% member agencies have sustained participation in SEEP (2) Baseline: <25% member agencies contribute to Best Practices EOP: > 33% member agencies contribute to Best Practice products (3) Baseline: <15% of member agencies applying MED best practices as promoted by SEEP training and products EOP: >75% of member agencies applying microenterprise development best practices as promoted by SEEP training and products (4) EOPS: increase of members involved as course facilitators and working group members increased by 75% (5) EOP: 50% of members contributed to best practices products (major works include 4 new books/manuals, 7 action research papers) (3) EOP: over 90% of SEEP members applied and integrated some or all of the SEEP MF ratios into reporting systems. training (b) develop new products (c) create a global communication infrastructure (b) provide regional workshops (c) develop manuals and technical papers (d) develop an electronic meeting place and document retrieval system (2) Baseline: 60 MFIs trained in 4 workshops EOP: < 50% trained in 6 workshops (3) Baseline: 3 products produced, published or translated annually EOP: <25% (4) Baseline: none EOP: >50% dependency on electronic communication (2) EOP: over 80 MFIs trained in 6 workshops (3) EOP: 5 products produced and/or published (4) EOP: 75% of program participation provided via electronic communicat 48 SEEP 1998 Matching Grant Business Plan Vision for Sustainability The overall vision for SEEP’s sustainability is to lessen the dependence on USAID/PVC funding over the next 10 years. This vision is three fold and emphasizes program development. 1. SEEP will create a new products and services. 2. SEEP will market its products and services. 3. SEEP will adopt a business model of operation. The NDS program has the greatest potential to bring longer term, bigger contracts and greater success to fulfill SEEP’s mission of sustainability. New products and services can be developed and marketed for the new NDS program. The NDS program potentially provides a broader more diverse constituency and a greater number of paying customers such as USAID/Missions, international MED networks, and other NGOs. Sustainability Goals and Objectives Over the next 3 years, SEEP’s goal and objectives to obtain more cost recoverable and financially viable program will depend on increasing its fee-based income and continue to diversify its grants support. NDS will be developed as an enterprise. Its activities will be cost recoverable by charging fees for service and products dispersed. At the same time, IDS will lessen its support on USAID/PVC by increasing its internal income generating activities and diversifying its grant resources. The potential for new or increased fee-based income include: · membership dues (increase the amount and increase the number of members). · training (offer more), · network development services: assisting new networks, · work with the domestic MED field through marketing SEEP publications, · publications: establishing distribution points beyond Pact Publications, · technical Assistance services (provide more), and · maintain a small staff size and hire consultants for specific tasks. Since SEEP maintains the double bottom lineæproviding its products and services to those who can afford to purchase them and providing its products and services to those who can’t afford to purchase themæSEEP will always need grant dollars from donors, foundations and corporate sponsors. Therefore, grant and donation diversification will be an important component to SEEP’s sustainability strategy. SEEP will also seek to increase its in-kind goods and services from outside sources such as rent, reproduction/printing, educational material development, board development, legal services, and corporate marketing. For a comprehensive strategy, SEEP will request the Board of Directors and its members to focus on SEEP’s own sustainability. The Board will address the issues of how to become a fundraising board. Members will be invited to participate on a task force that will look at the issues of SEEP’s own sustainability issues such as fundraising, marketing, and new product development. Sustainability Indicators, Benchmarks, and Targets The sustainable indicators and targets will not only pursue a cost recovery financial program but the indicators are also set up to achieve SEEP’s double bottom line. SEEP is concerned about getting its 49 products and services integrated into local NGOs of developing countries. SEEP’s critical indicators are based on outputs and summarized below. Objectives Indicators Benchmark Targets NDS: build new fee-based business organization 1) Organize development consultancies 2) Provide regional workshops 3) Develop manuals and technical papers 4) Develop an electronic meeting place and document 5) Develop a task force 6) 3 local networks receiving SEEP services 7) 2 new NDS contracts 8) Marketing materials created and disseminated 9) Sales of 3 TA to organizations during MG Diversify SEEP support 1)Adoption of key operational changes: -set board funding targets (diversify funders) -establish fee for service products and activities -establish publications marketing scheme 1) Increase IDS income by membership fees, publications, and meeting fees by 100% 2) Raise board mobilized resources by 15% 3) Secure 3 new sources of non￾AID/PVC support 4) Have MG represent no more than 50% of IDS/NDS funding 5) All start-up and operating capital for NDS from non-MG sources Business planning 1) Establish marketing scheme 2) Expand product development 3) Organize development consultancies 4) Develop an electronic meeting place and document retrieval system 1) Prepare a written business plan and make annual revisions 2) Appoint 3 new board members not of PVOs Board fundraising, targets, committee, tracking 3) Secure 2 or more proposals a year for NDS business services 50 ANNEX E: BUDGET The SEEP Network Cooperative Agreement Number: FAO-A-00-98-00049-00 10/1/98 - 9/30/01 Per Grant Budget Categories: Budget Actual Variance 1. Program 470,816 470,945 -0.03% 2. Training - - 0.00% 3. Procurement - - 0.00% 4. Indirect Costs 92,514 92,385 0.14% Total 563,330 563,330 Per Object Categories: Budget Actual Variance a. Personnel 223,824 227,506 -1.65% b. Fringe Benefits 76,100 72,923 4.17% c. Travel 35,406 33,681 4.87% d. Equipment e. Supplies f. Contractual 61,330 64,383 -4.98% g. Construction h. Other 74,156 72,451 2.30% i. Total Direct Charges 470,816 470,945 j. Indirect Charges 92,514 92,385 0.14% k. Totals 563,330 563,330 51 ANNEX F: MEMBER AGENCIES OF THE SEEP NETWORK ACCION International Action for Enterprise ADRA International Aga Khan Foundation ACDI/VOCA Aid To Artisans American Near East Refugee Aid Canadian Centre for International Studies and Cooperation CARE Catholic Relief Services Christian Children's Fund Coady International Institute Conservation International Cooperative Housing Foundation Counterpart International, Inc Development International Desjardins Enterprise Development International Enterprise Works Worldwide Feed the Children FINCA Food for the Hungry International Foundation for International Training Freedom from Hunger Foundation Friendship Bridge Grameen Foundation International Institute of Rural Reconstruction International Rescue Committee Katalysis Medical Ambassadors International Mennonite Economic Development Association Mercy Corps International National Cooperative Business Association OIC International, Inc. Opportunity International PACT, Inc. Plan International Pro Mujer Project HOPE Salvation Army World Service Office Save the Children, Inc. SOCODEVI SosteNica Strategies for International Development TechnoServe Trickle-Up Program Volunteers in Technical Assistance Wisconsin Coordinating Council on Nicaragua Women’s World Banking World Concern World Council of Credit Unions World Education World Relief Canada World Relief Corporation World Vision Canada World Vision Relief and Development 52 SEEP National Network Partners ASIA Philippines-APPEND SUB-SAHARAN AFRICA Benin- ALAFIA Burkina Faso-APIDEC Cameroon-ACIM Ethiopia-AEMFI Ghana-GHAMFIN Mauritania-APROMI Namibia-JCC Nigeria-CDMR Malawi-SEDOM Mozambique-MICRONET Uganda-AMFIU Zimbabwe-ZAMFI LATIN AMERICA Peru-COPEME Mexico -PRO-DESARROLLO EASTERN/CENTRAL EUROPE AND NIS Central and Eastern Europe and NIS – Microfinance Centre (MFC) BUILDING LATERAL LEARNING NETWORKS: LESSONS FROM THE SEEP NETWORK A Publication of The Small Enterprise Education and Promotion Network 1825 Connecticut Avenue, NW, Washington, DC 20009 The Small Enterprise Education and Promotion Africa, and Latin America with their enterprise (SEEP) Network is an association of more than 40 development assistance, expend roughly $ 112 million North American private and voluntary organizations on their behalf, and extend over $ 705 million in loans that support micro and small enterprise programs in on an annual basis. They are substantially the developing world. The Network provides a forum representative of the PVOs in this sector, both in terms for its members to engage in joint research and of the number participating, and in the diversity of training, and for the development and dissemination of approaches practiced. Members provide financial publications for field use. Through these endeavors, it services using individual, solidarity, village bank, credit promotes professional standards of practice, and serves union and cooperative methodologies; they offer as a center for communication and collaboration on a training, technical assistance and technology services; broad range of sector related issues. they engage in strategies which link enterprise development to interventions in the fields of health, In existence since 1985, SEEP has built a nutrition, vocational training, environment, and community of agencies interested in and committed to others. Attachment A provides a complete list of a collaborative process of learning. Begun initially as a member agencies. project under the umbrella of PACT, Inc., SEEP organized one of the first evaluation efforts in the From its perspective as an association of Northern enterprise development community, and articulated a NGOs with links throughout the South, SEEP has systematic approach to monitoring and evaluating the observed the growing need for strong and capable multiple dimensions - economic, social and institutions, and the emergence of lateral learning institutional -- of enterprise projects. The product of networks as one key mechanism for assisting with their that work, Monitoring and Evaluating Small Business development. The concept of NGO networks is Projects: A Step by Step Guide has sold over 6,500 certainly riot new. Nor are the challenges associated copies in English, Spanish, French, and bahasa with it. Over the years, dozens of networks have faced Indonesian, and remains in distribution today. difficult questions about how to galvanize members with limited resources to extend their efforts beyond Building on this effort, the Network has gradually the responsibilities of their individual programs. How expanded its focus to incorporate research, product can member organizations work together to concretely development and training in credit program design and respond to their collective needs and sustain that management, institutional development, poverty responsiveness over time? lending, business development services, training, and environmental linkages. It has initiated an information These questions are particularly relevant to lateral program that incorporates the systematic learning networks as opposed to affiliate networks or documentation of member programs and materials, apex institutions in which members are linked and a quarterly newsletter that reports on innovations, operationally and financially. This document debates, new programs, events and products relevant summarizes what SEEP has learned about the former. to practitioners. It presents first, a definition of networks based on their purposes and scope of operation; it identifies four Membership, too, has increased - from an initial critical areas of challenge that networks must address, 24 members to over 40 in 1996. Together, SEEP's and offers some learning based on SEEP's experience as member agencies assist over 5 million people in Asia, to how these challenges can be met. ANNEX G: LATERAL LEARNING 2 I. Defining Networks Often they are led by an apex institution that provides technical guidance and The rapid evolution of the enterprise sector and resources, as well as enforces adherence to the growing technical demands it places on the principles and goals set by the alliance. practitioners have generated renewed interest in Like the lateral learning networks, network structures. It is sometimes difficult to make operating alliances often support members sense of these networks given the diversity in levels at with technical training, systems which they operate, the purposes they serve and the development and policy advocacy. But relationships they cultivate among their members. they also tend to have more direct involvement in resource mobilization for To clarify a concept that can mean many things, members, channeling operating and/or loan the following definitions distinguish networks by three funds to them or functioning as financial critical features: their overall purpose, the nature of intermediaries (managing deposits, the members, and the formality with which they are investments, etc. on behalf of the group). structured. Broadly, networks can be divided into two classes based on purpose: Operating alliances are found at the national and international levels. · Lateral learning networks are associations International alliances are often North of microenterprise NGOs focused on South affiliate structures. improving the capacity of their members. Commonly, members are diverse in their The discussion of networks in this paper is focused target clientele and methodologies; they largely on the first type of association, the lateral operate independently of each other. What learning network. Nevertheless, some of the lessons brings them together is their interest in identified here may also apply to operating alliances, improving practice, sharing information, especially those that relate to strengthening member and coordination that will enhance the capacity and improving practice. policy and funding environment for the sector. Common services provided by lateral H. Four Areas of Challenge learning networks include information exchange, technical training, the As SEEP has studied the process of institutional development of best practice materials, development among microenterprise institutions, it has donor marketing and policy coordination. found that there are four key areas of challenge over which these organizations must develop mastery: Networks of this type are operating at the vision, capacity, resources and linkages. While, the international, regional and national levels. requirements for mastery in each area change as They may be constituted formally or institutions mature, direct service providers must informally, both in legal terms and in the nevertheless develop a vision that articulates target way they are structured to support the clientele, services and benefits to be attained. And they work of the membership. must develop the capacity - i.e. systems, structures and staff skills - to carry out this vision. Finally, they need · Operating alliances, on the other hand, are both the resources and linkages that will enable them formal affiliations of NGOs that share a to sustain their programs. common mission and are focused on a common clientele. As the name implies, an Similarly networks are called to be effective in the operating alliance constitutes a more same four areas, although the specific tasks before tightly knit group whose members have them are different than those that an implementing brought their operational strategies in line institution must undertake. These tasks are defined to achieve a common purpose. In many below: cases, they implement the same (or similar) methodology and use the same systems. 1. VISION, the ability of an institution to 3 articulate and generate commitment to the mission, aimed at information and advocacy, and through goals, and strategy that it pursues. For lateral learning alliance building with key actors in the policy and networks, the most critical challenge is to ensure that private sector arenas. member commitment and participation are constantly nurtured and supported through processes and Ill. Lessons from the SEEP Network structures that As SEEP has worked to master these challenges, ·define and update the network's vision and certain areas have been marked by greater success than goals; others. At the very least, the fact that the organization has sustained itself for eleven years suggests some ·establish programs, their goals and success in developing a viable institution, although the objectives; and issues of resource mobilization remain pressing, as for most non-profits. The lessons it has learned in the ·set association policy. process are summarized below, and discussed as they relate to the challenges inherent in the institutional 2. CAPACITY, the ability of an institution to development of any organization. Each is framed as a translate thinking into action. For a later-al learning recommendation with some discussion that derives network, this involves developing effective and efficient from SEEP practice and experience. mechanisms to improve members' technical skills, and advance the field, through b-dining and technical assistance services, systems development and support, the creation of " best practice " materials, and VISION information dissemination. It implies the creation of an institutional structure and methodology of work Lateral learning networks are associations of that can support practical, state of the art learning for members. Formally, they are owned by the practitioners. members that join them, and are constitutionally bound to respect their will. The challenge, 3. RESOURCES, the important capacity to however, is to take these "de jure" rights of mobilize resources for network activities, and in some ownership and translate them into the lifeblood of instances for member programs. In a field where self- the network. For SEEP, this has meant articulating sufficiency is much prized, networks are challenged to and adhering to strategic principles focused on the develop strategies that can lead to long term common good as well as establishing processes and sustainability Where the network does not serve as a structures that nurture participation. rnicrofinance channel, this Involves developing a mix of resources, internal and external, to support its continuity. Lesson 1: 4. LINKAGES, the ability to build productive Define the membership. relationships with a wide variety of organizations. They assist an institution in three areas: As a lateral learning network, SEEP values exchange among organizations actually implementing · strengthening organizational capacity programs. While it encourages participation from a through access to information, technical broad range of practitioner organizations, it does not assistance, staff training, and resources; accept government agencies, donors, research or consulting firms as members. This facilitates frank and · policy formation; honest exchanges among dime linked by the common experience, of carrying out programs in the South · increasing legitimacy. which Is highly valued by members. To be eligible for membership in SEEP, an applicant must be: For networks, the critical challenge is to expand the constituency for microenterprise through efforts - a not-for-profit development organizations 4 · located in North America project to strengthen its learning agenda, and to place upon grantees the obligation to "give back" to the · implementing small and microenterprise larger group through documentation of experience, a programs in a developing country sharing of tools, and training. In pursuing participation in several consortia programs, SEEP had to carefully · interested in improving small and define what elements seemed more appropriate for microenterprise development capabilities individual members to do; and which ones seemed better served by collective activity. Only in the latter · committed to contribute staff time and cases would the network negotiate for a role and travel costs to network activities, as well as funding. pay annual dues. Lesson 3: Lesson 2: Foster members' engagement in Set and adhere to guiding principles governance and policy formation. that acknowledge equality and community. SEEP's structure and processes provide a way for In establishing network policy, in deciding on SEEP to blend formal agency representation and active program directions and goals, in determining how engagement by many staff of member agencies. The activities will be carried out, SEEP consciously adheres structure has four components: to the following principles of operation: - the membership of agencies · All programs must be member-driven, responsive to their articulated needs - the Board of Directors · All members are equal partners, regardless - the Working Groups, and of the size of an agency's budget, or the expertise it has in microenterprise.' Each has - the staff. one vote; each pays the same dues. 7he membership. Each member organization is ·Learning is to be generated through the part of the "body" of SEEP, and designates an official pooling of experience across a variety of representative who acts as the principal point of agencies and strategies communication between itself and SEEP. Only official representatives can stand for election to the Board of ·Any funds distributed by SEEP to individual Directors, and only they can vote in board elections and members for research, writing, training, and on any other matter that might require the consent of technical assistance activities, must result in the membership. (However, other staff from member advances in learning or products that will agencies may participate in any SEEP-sponsored benefit the larger community, and activities.) The member's participate in setting policy through their participation at SEEP's Annual Meeting, ·SEEP will not enter into programs, or seek held every October. funds for activities that members can do directly. 7he Board of Directory. The Network is guided by a seven member Board of Directors elected from the Placing members at the center of network representatives of the member agencies. Six operations, these five principles are easy to state, but representatives hold three year terms, with two can be more difficult to follow. SEEP has faced positions rotated off each year A non-elective position instances in the past where interests of its members have is reserved for member agencies which have served as been unpopular with donor perspectives. In another SEEP's fiduciary agents for specific grants. The Board situation, a donor offer of funding that would provide meets four times a year and Is responsible for setting subgrants to some members posed a serious challenge to Network policy and program goals, for reviewing and SEEP's belief in equal status and treatment for all approving projects, for evaluating program outcomes, members. This was resolved by re-conceptualizing the and for developing relationships with external 5 institutions and groups. Board members provided The staff- The Network is staffed by a small core regular guidance and service to SEEP staff between of professional and administrative staff who support meetings, as well as provided representation to all the members in the execution of all program activities. Working Groups. The Executive Director and Program Assistant implement all Network programs in conjunction with The Working Groups- These groups of self- the relevant committee or working group, administer selected individuals, in many ways, represent the heart the budget, and fulfill all reporting obligations. They and soul of SEEP. Unlike the Standing Committees are supported in their efforts by a Senior Associate found in many association structures, they are ad hoc with responsibilities for Work Group facilitation, bodies established on the basis of member inter-est. information and training services. Occasionally, these Each is composed of between 5 and 15 members, and core staff have been supported by contract personal for serves as the vehicle for participatory research, applied short or long term projects, but in general, SEEP's learning, documentation and training on a particular work is managed by three. There are two lessons topic. Each designs its own learning process and SEEP has learned about staffing: implements it with the support of SEEP staff. The learning process may take several forms. It includes a Networks should be member driven, and sequence of events that starts with the formulation of not staff-driven. This requires that a staff issues to be studied, and then can include the remain small and not supersede the development of frameworks or other analytic tools, the decisions, or work, that can be done by the preparation of case studies, surveys of current members. practices, small group sessions, and one or more workshops. The Work Groups continue their efforts At the same time, networks don't "move" with the preparation of the written products that without some paid staff to coordinate, synthesize the content. motivate, and carry the ball for busy people. The core needs to be compact, yet Currently, the active working groups include: strong enough to maintain the momentum Financial Service Poverty Lending, Training, of the community. Evaluation, and Business Development Services. Past groups have included Institutional Development, Environment and Enterprise, and Nonfinancial Services (a pre-cursor to the current Business CAPACITY Development Services) - While many groups have existed for a long time, It is important to note that there is no statutory foundation for their existence. Lesson 4: Groups form, dissolve, and re-form on the basis of Focus on the practitioner member interest. The people who participate in SEEP are most As this description implies, Working Groups are often drawn from the ranks of program staff as the creative cauldron in which SEEP generates its opposed to management. They include experts in learning and crafts its products. They also are the place credit and savings, financial and Institutional where program directions are identified, and where management, training, technology, marketing, and critical Network issues are discussed. Because they are evaluation. Their conversations remain grounded in open to all interested staff of member agencies (and to field practice, and free of Institutional baggage. Within Friends of SEEP, individuals who have left their various venues of SEEP, an honest exchange of member agencies but are asked to remain engaged in information and analysis is expected. Images and ego SEEP because of their knowledge and experience), they must be left at the door. facilitate much broader involvement in network decision making. One of SEEP's mottos is that “program is decided by whoever comes to the table.” The Working Group provides a venue where many interested individuals can put in their two cents, and shape the course that the network will take. 6 Lesson 5: The value of these products can be seen in the number Start with one activity and do it well. of them that have been distributed or sold over the years. NEXUS, SEEP's newsletter, now reaches 1,300 SEEP started with a single purpose: to develop an readers worldwide. In addition, SEEP's publication list evaluation system that captured the multiple includes 23 titles, and approximately 1,500 units of dimensions of NGO enterprise programs. Then, in these are sold annually. 1989, it sponsored a workshop on credit management for members and thew field staff in response to SEEP has also learned that each product must growing NGO interest in credit programs. Its next have one or more champions: staff that are passionate major public event was a workshop on institutional in their interest in the topic, and willing to give their development that drew on the experience of several energy and commitment to moving the project member agencies. While several small subsets of forward. This type of leadership is essential to making members had formed to work on distinct issues (the things happen: individual members have stepped first working groups), in the early years, SEEP largely forward and back, depending on their interest in a devoted its limited resources to one major undertaking given topic. This volunteerism is also another way in at a time. which members set SEEP's agenda: SEEP will not apply or raise resources for projects that do not have Lesson 6: this level of commitment. It also serves in lieu of any Create mechanisms for learning that favor formal needs assessment for the definition of program collective analysis, and include opportunities services. Consistently, the projects that have had these for all to teach and to learn. champions have found great resonance among both the membership and the broader microenterprise As described above, the Working Groups are community. SEEP's chief vehicle for learning. Within those circles are represented a diversity of-technical approaches and Lesson 8: levels of experience. Whatever topic a group is Grow organically. Start small and expand as working on, the strength of the analysis lays in its needed. focus on extracting what has worked in a variety of settings and methodologies, and distilling that for For many years, SEEP was managed by a part others to learn from. At the same time, there is time coordinator who was paid as a consultant. As recognition that at any given time, there are some with needs surfaced among members, the coordinator greater knowledge and experience than others. SEEP's helped them define the need and develop a specific working groups -- and overall program of learning and strategy to address it, including the human and dissemination -- allow member institutions to financial resources required to get the job done. SEEP participate in various ways. Some start out as learners; responded to these ideas and needs on a case by case other as teachers. These roles may switch, possibly basis, mobilizing members for the tasks at hand and more than once, over time. hiring additional consultant help when absolutely necessary. As the network grew, the number of Lesson 7: activities increased, as did the amount of coordination Focus on products and their dissemination required. Nevertheless, even today, SEEP has less than 2.5 full-time staff supporting network activities. SEEP's Working Groups do just that. While a group's efforts may begin with an internal exchange of Lesson 9: information, the purpose of a SEEP group is to create a Establish the network as a formal product of value to the larger community The institution only when needed. product may take the form of a publication (monograph, manual, facilitator's guide, etc.) or Don't worry about setting up a formal institution. training workshop and design. Both publications and it is more important to invest early in getting an training events focus on communicating new products, activity going that will secure the commitment, and methods, systems or the state of the practice in ways support of members by addressing a relevant issue or that are accessible and relevant to field implementors. problem. Build on this. Try things. If they don't 7 work, move on to the next idea. Stay flexible and Donors have approached SEEP to serve a conduit responsive. And invest what will most probably be for grants to its members, or to administer a grant scarce resources in member services first (instead of making program. SEEP has learned that a network taking on the costs of formalization). SEEP only should not distribute grant resources to its members recently became a legally constituted, not-for-profit until a certain level of maturity has been reached in its (501 C-3) institution. For its first ten years, it was a relationship with them. project officially administered (for legal and fiduciary purposes) by a member. In fact, from 1985-1995, LINKAGES SEEP had three different institutional "homes". RESOURCES Lesson 12: Distinguish political activity from the research/learning agenda. Lesson 10: Keep core operations compact and expenses low. SEEP has been able to foster community and Depend on substantial contributions from member commitment to its active and learning agenda. members. But in the past, it has had greater difficulty achieving consensus among its diverse membership around SEEP's core budget is financed in three ways: by policy issues. Rather than invite divisiveness, SEEP has member dues, funds generated by the sale of chosen to leave the advocacy role to other entities. As publications and workshop fees and a grant from a result, in the United States, there is another coalition USAID. Members' dues have grown over the years of microenterprise agencies devoted to policy dialogue from a initial quota of $250 annually to $630. This with donors and government in which there is fee is significant for most members, and represents a substantial overlap with SEEP membership. However, strong vote of support for the. network. But equally it is important to note that this principle is rooted in important is the significant contribution of staff time the very specific context in which SEEP operates, its and travel to Board and working group meetings. All limited resources, and its priority to foster professional SEEP members underwrite their travel and time development among its members. If a network does associated with SEEP activities. The high level of find it useful to involve itself in policy initiatives that volunteerism contributes to an esprit de corps of SEEP affect its membership, it would be wise to invest in members that has been an emblem of the network building consensus before taking action. since its inception, and a defining part of its corporate culture. The strength of this cooperation allows SEEP For more information contact: to conduct a program far beyond what its budget The SEEP Network might suggest. 1825 Connecticut Ave, NW Washington, DC 20009 Lesson 11: Phone: (202)884-8392 Do not engage in competition with members for Fax: (202)884-8479 program funds. Email: seep@seepnetwork.org This principle emerges from observation of other networks and consortia that have broken apart over resource issues. SEEP does not engage in direct program implementation which helps it avoids competing with members for grant funds. This principle is slightly less clear in the case of competitive bids for USAID donor contracts for which teams of agencies bid together. SEEP consults extensively with members before committing itself to such a team. MEMBER AGENCIES OF THE SEEP NETWORK ACCION International Action for Enterprise Agricultural Cooperatives Development Int'l./VOCA ADRA International American Near East Refugee Aid Appropriate Technology International Calmeadow Canadian Co-operative Association CARE Catholic Relief Services Christian Children's Fund Cooperative Housing Foundation Development Int’l Desjardins (DID) Enterprise Development Enterprise Works Worldwide Feed the Children FINCA International Food for the Hungry International Foundation for International Training Freedom from Hunger Foundation International Institute of Rural Reconstruction Katalysis Mennonite Economic Development Association Mercy Corps Int’l National Cooperative Business Association Opportunities Industrialization Council, Inc. Opportunity International PACT, Inc. Pan American Development Foundation PLAN International Pro Mujer Project HOPE Salvation Army World Service Office Save the Children Seed Capital Development Fund SOCODEVI Strategies for Int’l Development TechnoServe Trickle Up Program Volunteers in Technical Assistance Women's World Banking World Council of Credit Unions World Concern World Education, Inc. World Relief World Relief Canada World Vision Canada World Vision Relief and Development