USAID/PVC Matching Grant Evaluation Series: SYNTHESIS REPORT of PVC MATCHING GRANT EVALUATIONS October 2002 Conducted under USAID/Evaluation Indefinite Quantity Contract # AEP-I-00-00-00024-00 Order No. 1 Prepared by: Joan M. Goodin 600 Water Street, S.W. Washington, D.C. 20024 USA 202/484-7170 Fax: 202/488-0754 i Table of Contents I. BACKGROUND..........................................................................................................................................1 II. FINDINGS....................................................................................................................................................2 A. Partnership...............................................................................................................................................2 B. Measuring Capacity Building ................................................................................................................9 C. Networking............................................................................................................................................13 D. Sustainability.........................................................................................................................................22 E. PVC Management of Matching Grants...............................................................................................26 Attachment: List of PVC Matching Grant Evaluations........................................................................................................34 1 SYNTHESIS REPORT of PVC MATCHING GRANT EVALUATIONS I. BACKGROUND In mid 2001, Management Systems International (MSI) was contracted by USAID’s Office of Private and Voluntary Cooperation (USAID/DCHA/PVC) to evaluate 12 of the Matching Grants it had awarded to U.S. Private and Voluntary Organizations (PVOs). The contract also stated: “In addition to carrying out evaluation activities as stipulated in the Matching Grant agreements, information from the 12 end-of-project evaluations needs to be synthesized to be used by PVC.” At the time of this writing, only the ten evaluations listed in the attachment to this report had been completed and were, therefore, the basis of this synthesis. The major purpose of this effort is to provide information that will be useful as PVC develops plans for the priorities outlined in its new strategic framework and studies the issues included in its Research, Development and Outreach Agenda. As stipulated in the contract, this synthesis is “to bring together information from the past two cycles of Matching Grant program evaluations and to put into place a series of small meetings to present this information and discuss the program implications with the PVO Community.” The contract also makes clear that this Synthesis Report is to serve basically as an internal document to provide raw material for those discussions with PVOs and “to provide technical input into the development of a series of communication pieces to be produced.” Evaluations were conducted in a participatory fashion; all teams included personnel from the respective PVOs, as well as MSI specialists. Each evaluation ended with a debriefing session, during which PVC personnel, evaluators and other representatives of the PVO discussed the major findings, conclusions and recommendations. Based on those discussions, PVC identified five crosscutting themes to be abstracted from evaluations and synthesized in this report. The themes selected were: 1. Partnership 2. Measuring Capacity Building 3. Networking 4. Sustainability 5. PVC Management Issues The remainder of this report is organized into five main sections corresponding to the themes chosen. Each section is then divided into various sub-themes, which present information on the major areas of interest emerging from a composite review of the ten evaluations. Sub-themes are followed by “Notes to PVC,” which are intended to highlight the relevance to PVC of the foregoing discussions. Analysis is based on review of the evaluation documents only. No supplemental interviews or data collection were conducted. 2 II. FINDINGS A. PARTNERSHIP Both current and past PVC Strategic Plans have stressed the importance of partnerships between U.S. and local organizations as vehicles for the effective and efficient achievement of sustainable development. Clearly, the message was heard. Of the ten Matching Grants evaluated, four included the creation or strengthening of partnerships at the goal or objective level, while four others focused on partnerships at the activity level and two addressed this issue indirectly. In examining these evaluations, five major sub-themes concerning partnership emerged: · Definition of Partnership & Partner Selection · Purposes & Types of Partnerships · Internal Implications of Partnership Management · Partnership Agreements – Formal vs. Informal · Growing Understanding of Partnership Principles & Factors Related to Successful Approaches The following sections present summaries of each sub-theme. 1. Definition of Partnership & Partner Selection “Partnership,” like “beauty,” appears to be in the eye of the beholder. In the grants evaluated there is no clear pattern with regard to the definition of partnership or the number and type of partners chosen by PVOs for grant-supported activities. They run the gamut from community￾based groups to national-level NGOs and international PVOs, village to national government agencies, and a wide variety of commercial enterprises. In addition to the definition provided above, the concept articulated by another PVO incorporates the major elements of partnership espoused by many PVC grantees: “a type of institutional relationship in which two or more organizations work together to achieve mutually defined goals on mutually accountable terms.” This organization further defined “strategic partnerships,” designed to increase the capacity and scale of its programs: “concurrent institutional relationships formed at community, national, or international levels in order to increase impact around a clearly defined policy or practice that incorporates two or more program areas.” The number of partnerships undertaken were shown to relate less to the size of the PVO than to its approach to development. One grantee with offices in 36 countries reported nearly 30,000 partnerships, while another registered 26 partnerships in 14 countries, and one organization with worldwide operations established only one partnership in each of the two countries targeted under the MG. A mutually beneficial alliance between organizations where roles, responsibilities and accountability are clearly defined. Partnerships are based on a shared vision regarding the objectives and purpose of the work undertaken together. Joint contributions of resources, shared risks, and shared control of program and financial information and planning identify partnerships. CARE 3 Among the factors mentioned most frequently as relating to the choice of partners were the following: · Existing relationships with members of other organizations; · Sector-specific experience in the area of project objectives; and · Local context and the presence of organizations with similar interests. Note to PVC: To contribute to the accomplishment of its strategic objectives, it would be important for PVC to include the issue of partnership in its Analytic Agenda. PVC could explore the link between how potential grantees define partnership and the types and number of partners anticipated for PVC-funded initiatives. This information could help to estimate more precisely the degree of coverage, scale-up and cost-benefit to PVC of the activities proposed. 2. Purposes & Types of Partnerships Clearly, partnership is not an end in itself. It must result in some greater development good that would not be attained by the partner organizations operating individually. The evaluation reports reviewed provided insights as to the purposes of the partnerships established by grantees. While they were designed to pursue a wide range of technical, sector-specific outcomes, their underlying purposes were to: · Leverage resources · Delivery of services · Increase coverage and impact · Achieve sustainable programs · Advocate for policy change · Improve efficiency · Access complementary expertise · Promote special initiatives or strategies of interest to the PVO grantee · Test and transfer new methodologies to local organizations In six cases, grantees have either ceased direct project implementation, or are in the process of shifting from direct service provision to indirect service delivery through partners. In one case, the PVO was committed to increasingly shifting from a service delivery mode to one of facilitation and capacity building for any interested public or private organization that might work within the priority geographic areas identified. It sought to capitalize on the advantages of working with partners of various types and at different stages of the project cycle for the purpose of increasing coverage and impact in terms of poverty reduction. For another PVO that works to strengthen the private sector, “the blurring of distinctions between partners and clients” was said to be the key to understanding its new strategic approach, which was termed “very partner-dependent: all the sustainable economic benefits that [the PVO] seeks to establish flow from the continued profitable operation of its partner/client businesses.” Activities under these Matching Grants revealed five types of relationships referred to as “partnerships” between recipient PVOs and other organizations. They differed mainly by the degree of shared decision-making and governance: 4 1. Sub-grants and contracts: The PVO awards a sub-grant or contract to a “partner” organization for the provision of specific services. The sub-grantee or contractor has virtually no role in the decision-making process or overall project management. 2. Dependent franchise: In this model, the PVO takes an ownership or major shareholder position in a local organization which then depends on the PVO for its sustenance and direction. 3. Spin-off NGO: The PVO either spins off staff from its own operations or motivates others to create a new, local NGO with which to partner. 4. Collaborating organization: The PVO engages other organizations with complementary expertise that work in the same area or sector to collaborate in the pursuit of goals and objectives of mutual interest. 5. Shared vision or co-equal arrangement: The PVO and its partners are committed to a mutually beneficial relationship based on a shared vision of the desired development outcome and agree to be held accountable for clearly defined roles and responsibilities, while contributing resources and equally sharing risks and project control. Note to PVC: The first two types of relationships listed are considered by many PVOs and development analysts to connote “clients,” “customers” or “sub-contractors,” rather than “partners.” Given the prevailing differences of interpretation as to just what constitutes a “partnership,” it would be useful for PVC to clarify its own understanding of the term when disseminating future information and requests for proposals. 3. Internal Implications of Partnership Management Perhaps the most important lesson about partnerships that emerged from these evaluations was that careful attention is required if they are to be successful. As stated in one report, “the partnership itself, including each organization’s role in it, needs to be managed, almost as a separate entity. It is not enough to simply assess each institution separately; the actual bonds, incentives, tensions, and structural issues that help or hinder effective partnership must also be examined.” In that case, it was recommended that responsibility for actively managing important partnerships be assigned to a specific individual, and that the partnership management process provide an “early warning system” to detect emerging problems. Of the ten MGs reviewed, six focused on micro-finance (MF). In most instances, these grants aimed at strengthening or expanding the PVO’s existing MF capacity, particularly by working with or through partners. In two cases the grant was to support the incorporation of MF into the organization’s traditional program portfolio. Of the latter, one PVO learned that some partnerships failed due to lack of effective internal communication and coordination systems. “Part of the problem has been the process of culture change within [the PVO] itself… Not only do personnel now need to think in terms of sustainable programming, but in terms also of losing autonomy over important community services. This has not come easy to some.” As a result, it was concluded that: “The art and process of partnering with independent MFIs (micro-finance institutions) has probably resulted in more learning… than the introduction of high performance MF into… country programs.” 5 Partnership management became a major issue at one PVO that had chosen to work with a single NGO in each of the two pilot countries selected – one of which was a spin-off NGO created by the grantee for MG activities. This PVO reported having learned a great deal about how to engage in partnerships as a result of the difficult lessons that emerged from partnership failures in both countries targeted. A lack of “ownership” of the project within headquarters, and between HQ and the field, plus the absence of participation by field staff and partners in project design, were cited as having “engendered ambiguity among country staff towards their partners and schizophrenia between HQ and the respective country programs on how to manage the partnerships.” In the end, the PVO also learned, among other things, that “it is not advisable to work with only one local partner, that a diversity of approaches to partnerships (joint work, contracts, grants, etc.) is useful.” Another PVO had no partners or formal partnership scheme, per se. Rather, it sought to establish a separate MF program within the organization - different from its traditional relief/social development activities. The design involved the addition of a regional office within the PVO to oversee creation of a regional bank structure, with MFI branches as partners in three countries. This was to become a model to be expanded over the longer term to other countries and regions. However, over the course of the MG, due to financial and management challenges, this proved to be unworkable and evolved into two autonomous country programs supported by a small HQ unit. The two commercial “partner” MFIs became independent limited liability companies (though not yet self-sufficient), with the PVO as their sole shareholder. Note to PVC: In reviewing grant proposals that involve the establishment of partnerships, it is important for PVC to learn the specifics of how the applicant intends to manage those relationships. Reviewers should gauge the degree to which field staff and partners are to participate in the planning and management of on-going activities. Applicants should describe whether the organization’s existing management system is structured to support such participation. If changes will be required, they should indicate how those changes are to be accomplished. 4. Partnership Agreements – Formal vs. Informal Among the lessons learned was that different interpretations of appropriate partnership agreements existed within PVO headquarters and between HQ and field offices. In one case, the PVO’s regional auditors had problems adapting to shared-vision, co-equal partnerships for which both parties took responsibility. They were accustomed to treating partners as contractors, hired to carry out a specific set of activities for a specified cost and at the PVO’s instruction. In another case, it was found that, “The donor-recipient relationship still lurks behind most partnerships, even where due diligence has discovered significant overlap of shared vision. When coupled with post-colonial sensitivities, this can become a critical problem.” One grantee reported that there was a tendency among some local NGOs to view partnering only as an opportunity to secure outside funds without regard to the specifics of the agreement. It was reported by another PVO that HQ felt that written agreements had to be in place to validate partnerships, and that these were then viewed as legal contracts. Meanwhile, field experience indicated that partnerships were more akin to a courtship in which the establishment of such relationships is an ongoing process and not an end in and of itself. Recognition of the 6 serious implications of cultural differences and local customs at the field level was found to be particularly important in the establishment of successful partnerships. Reportedly, written agreements were generally viewed by local partners as MOUs (Memoranda of Understanding), rather than as contractual agreements. And in some instances, where no transfer of funds was envisioned, partnerships rested on verbal agreements between partners at the local level. On the question of financial transfers, in one country the PVO gave a sub-award to its partner NGO, together with a sub-award agreement. However, there was no overarching MOU between the two organizations. Over the course of the grant, because of miscommunication and mutual suspicion, relations deteriorated drastically, leading to the PVO’s decision to cut MG funding to its partner. When questioned about this both partners reported that there had been uncertainty as to the intentions of the other, and suggested that “an MOU covering fundamentals of the relationship would have helped avoid problems.” The difficulties encountered in requiring partners to enter into written agreements included foot￾dragging, refusal to sign a binding agreement, disputes over sub-agreements, or differing interpretations as to whether agreements were legal documents or less formal MOUs detailing the responsibilities of each partner. Differences from country to country underscored the need for flexibility and for giving field managers responsibility for determining the precise nature of partnerships. “If partnering is viewed as the means to achieving greater capacity for carrying out tasks at the field level, then field managers need greater flexibility to enter into arrangements of varying rigor with organizations without written agreements that always pass legal muster in different cultural settings…,” though it was noted that there is a need to safeguard against “any potential liability or exposure for the PVO (and indirectly, USAID).” The major constraints to successful partnering encountered by the ten grantees may be summarized as follows: · The time required for the establishment of partnerships is substantial, and a minimum of 3-5 years is required to build a sustainable program. · Lack of internal cooperation contributed to delays and uncertainty. This included reticence to participate on the part of PVO administrative and field staff, especially when they had not been involved in partnership negotiation. · Commitment to partnership varied across the various levels of the PVO. In some instances, management and operational systems were not fully adapted to partnerships. · Absence of clearly understood and mutually acceptable agreements, whether formal or informal, written or verbal. · Lack of clarity regarding roles and responsibilities, both within the PVO and between the PVO and its partners. · Unequal financial status of partners. In many instances, this led to a “donor-client,” or “top-down” interpretation of the relationship. Note to PVC: When considering grant applications, PVC reviewers need to take into account the type of agreements that the recipient PVO intends to enter into with partner organizations and how and by whom those agreements are to be structured. Partnerships involving the transfer of funds to local organizations may require a specific type of written agreement in order to comply with USAID regulations. However, rather than prescribing a particular type of 7 agreement for all other partnerships as is now the case, PVC could exercise greater flexibility, allowing for input from field staff concerning local circumstances, customs and sensitivities. 5. Growing Understanding of Partnership Principles & Factors Related to Successful Approaches Shared interests or values provided powerful underpinnings for the development of some partnerships. For instance, for grantees involved in a specific sector, such as micro-finance or the environment, the identification of local organizations with like interests and the subsequent establishment of partnerships was relatively straightforward. In many cases, these organizations were already in contact with one another through the issue-based networks or associations to which they belonged. In a similar vein, one grantee was in contact with local organizations dedicated to the same Christian values, which was said to have instilled a sense of shared vision and a culture of harmony. Clearly, these factors facilitated the identification of local partners, and may have speeded the process of partnership development. In some cases, this may also have helped to ensure the success of these partnerships. But that was not so in all instances, since identification with a specific sector or cause was unrelated to the nature and management of the partnership itself. It is not surprising to learn that questions were raised by some local partner organizations as to who is driving the agenda and defining development. They did not necessarily feel part of that process, and some perceived PVOs as proxies of U.S. foreign policy, with little to distinguish them from donor agencies. So, the question remains, does identification with a specific issue or set of values reduce the time required for partnership development and management, and how does this impact the supervisory and oversight role of PVC’s grantees vis-à-vis their partners? Among the general principles mentioned most frequently as important for building and maintaining partnerships were: · Mutual trust and respect · Transparency · Mutual commitment to & responsibility for program outcomes · Clarity in objectives, roles of all parties and working relationships · Accountability to all stakeholders · Frequent communication and collaboration · Clear separation of financial transactions · Timely and creative problem solving and willingness to learn from difficulties · Open discussion of partnership challenges · Good working relationships between the PVO and partner staff · Agreements and relationships that transcend individuals · Long-term commitment to the partnership (minimum of 3-5 years) · Active commitment of country director and management team & country strategic plan that embraces the concept of partnership One grantee working in the area of micro-finance identified a six-step process for building partnerships: · Survey and assess potential partners 8 · Conduct feasibility studies · Prepare a business plan · Develop a written agreement · Set reporting standards and formats · Build in monitoring and learning tools Another PVO, while not attempting to partner with or contribute directly to other organizations as a function of the grant, generated insights for the promotion and operation of three-way partnerships among nonprofits, businesses and USAID. Activities under this grant helped to mainstream the concept of PVO-business partnerships as a new development paradigm, and contributed to the formulation of USAID’s Global Development Alliance, which is designed to foster such partnerships and is one of the Agency’s current priorities. Among the lessons learned through this experience were: · While accountability is important in any partnership, it is particularly critical in countries characterized by high levels of corruption. Representatives of the business sector held that, in such cases, U.S. PVOs enjoy a higher degree of trust than local NGOs, and thus are seen as providing a greater measure of “protection and ethics.” · Building three-way partnerships is a very time-consuming and complex process. It is necessary to learn the structures of all of the organizations involved in order to ensure sufficient internal support. The lack of institutional commitment at all levels of potential PVO, business and USAID partners led to a good number of “false starts.” The bureaucratic hurdles and contradictory views encountered at the various levels within USAID became a serious constraint to three-way partnerships. Note to PVC: While much has been learned about partnership principles and approaches, one issue that merits more in-depth examination is the cost-effectiveness of PVC’s investments in partnership development and management as compared with other less time-consuming and more direct approaches to the achievement of its strategic objectives. 9 B. MEASURING CAPACITY BUILDING In examining this theme, it should be noted that the grants under review dealt with capacity building at two distinct levels: the institutional capacity of the PVO itself, and the institutional or sector-specific capacity of its local partners. Eight of the ten MGs were designed to strengthen the institutional capacity of recipient PVOs to perform specific functions in pursuit of their development goals. The areas addressed ranged from micro-finance to rural development, conservation, and poverty at the household level. Six of these grants also sought to build the capacity of partner organizations. It was stressed that, for PVC, capacity building, like partnership, is not an end in itself; it should lead to improved service delivery. Here, three major sub-themes emerged: · The Starting Point · Approaches Employed · Unmeasured Progress 1. The Starting Point No evidence was found that any of the ten PVOs had established baselines against which to measure changes in institutional capacity within their own organizations or those of local partners. Nor had performance monitoring plans been formulated for that purpose. A number of grantees stated that they simply had not been able to find appropriate capacity building indicators, or that they had concentrated exclusively on tracking substantive results, and had therefore not attempted to measure changes in institutional capacity. A number of the PVOs had conducted some form of institutional capacity assessment of partner organizations. In most cases the assessment was conducted as part of the partner selection process, and was not used as a basis for the subsequent measurement of change. In one case, the assessment was conducted during a workshop with the partner NGO. With the resultant information, the PVO then developed a capacity building plan for the partner organization, but without its participation. This led to only limited ownership of the plan by the local NGO and, therefore, produced limited (though unmeasured) results. Another PVO explained that, because there was no provision in the MG for sub-grants to partners, no attempt had been made to measure their institutional capacity “beyond informal assessments based on local reputation.” One specific recommendation related to this topic was that PVC should require that the process by which capacity building under the MG will spread through and be measured by the PVO and its partners should be spelled out in the proposal. It was found in one case that the substantial field staff and other resources of the PVO were never actively engaged in the capacity building When selecting a measurement instrument, it is helpful to begin by clearly identifying what needs to be measured. The interven￾tion’s objective may be to strengthen the entire organization or only a specific function or component of the organization. Measurement, to be most useful, will capture only the information that is relevant to the intervention….Understanding and measuring institutional capacity is critical and often more complex than measuring the services and products that an organization delivers. Best Practices Paper, Measuring Institutional Capacity, CDIE, Sept. 1999 10 purpose of the grant. Furthermore, the evolution of the activities undertaken over the grant period, “made the vague capacity building objectives found at the start in the DIP even more inscrutable.” Note to PVC: For grants that include goals or objectives related to capacity building, either within the PVO itself or its partners, it would be in the interest of all concerned for PVC to ensure that mutually agreeable indicators and plans for regular performance monitoring are included in the DIP, beginning with the establishment of a baseline against which to measure change over the life of the grant. 2. Approaches Employed In one instance, the grantee introduced the Discussion-Oriented Organization Self-Assessment (DOSA) method to partner NGOs in the two pilot countries included in the Matching Grant. 1 The PVO facilitated one DOSA session for each of the two partners in order to “enhance local measurement of capacity.” Evaluators found that the PVO appeared “to follow and be more concerned with loan tracking and financial management (efficiency and sustainability) indicators and less concerned with organizational management and capacity building indicators.” This February 2002 evaluation found that, in one country, the DOSA was applied in September 2001, producing 12 “action items,” of which none had yet been addressed. In the other country, a DOSA session was conducted in June 2001, generating four “action items,” of which two had been implemented and one was being completed. No evidence was found of plans to repeat the DOSA exercise or otherwise monitor changes in capacity over time. Another PVO, working with MFIs in two countries to field test its new product in the credit arena, did not do a formal capacity assessment of either of the two local partners. However, it did employ a variety of organizational capacity assessment tools, including the SEEP Institutional Development Guide and framework, UNDP’s Sum Institutional Development checklist, and MEDA’s organizational assessment tool (Mennonite Economic Development Association). It also tracked progress by mapping information flows and data needs in the organization, which provided a basis for prioritizing information users and requirements, and for assessing where changes could be made. Evaluators concluded that, “Although many of the activities carried out under the MG undoubtedly strengthened the capacity of the two partner agencies, the fact that no baseline of capacity was undertaken is understandable given the focus of the partnership on product development and field testing. Moreover, it was felt that “given the nature of the relationship with both institutions, a formal capacity assessment may have been considered inappropriate unless requested by the local agency.” Finally, evaluators suggested that the mapping tool used for progress tracking “may provide a template for other types of institutional diagnostic tool.” In terms of strengthening performance monitoring capacity, another grantee reported that the tool passed on to local partners was a Planning Matrix together with its indicators. However, it was 1 DOSA was developed in 1997 for PVC. Using group discussion interspersed with individual responses to a 100- item questionnaire covering six capacity areas, two types of scores are produced: 1) a capacity score indicating how participants perceive their organization’s strengths and weaknesses; and 2) a consensus score indicating the degree to which participants agree on their evaluation of the organization’s capacity. 11 found that “there is not much training on how to use or modify them” and that changes in capacity were not monitored on an on-going basis. Note to PVC: To help transmit the importance attached to the measurement of capacity building interventions, PVC could require that all grantees working in that area include in their regular progress reports an update on the status of their performance monitoring plans and results to date. Reporting would be based on the indicators and plans included in approved DIPs, and would help to flag the need for adjustments. 3. Unmeasured Progress Two of the grants under review were designed specifically to institutionalize within recipient organizations new programmatic components or methodologies, thus increasing their capacity to achieve development impact. In both cases, these were PVOs with complex management systems encompassing large numbers of employees at headquarters and in countries around the world. Neither organization had developed plans for measuring changes in capacity within their organizations. Thus, no hard data were available to judge the degree of success in this area. Nevertheless, both evaluations found anecdotal evidence to suggest that significant progress had been made in terms of institutionalizing the innovations introduced through the MG. As stated in one case: “Since impact indicators were developed only for measuring change at the [beneficiary] level, it is not possible to measure precisely the impact of the overall MG, particularly with regard to changes in the institutional capacity of [the PVO] and its partners.” The grantee in question sought to incorporate throughout the organization three new methodologies for the achievement of its goals. It since recognized the need to develop capacity￾building measures, and reportedly made this a high priority for the future. Evaluators found that a start had been made when HQ asked all offices to assess their capacity in one of the three new areas, thus providing a baseline for the future measurement of organization-wide change in that category. The second grantee mentioned focused on adding a micro-finance (MF) program model to its traditional community-based services, working through partners in the six pilot countries selected for the MG. Evaluators found that, “the need to assist most MFI partners in capacity building has led to the development of a special section of [the PVO’s] MIS, which partners and National Credit Coordinators are supposed to review every six months.” The data included in the MIS measure changes in the capacity of the MFI partners in the six countries (though the idea is eventually to institutionalize the overall approach throughout the organization), but not within the PVO itself. Three categories of MF indicators are included, and a separate report is produced for each: a) outreach; b) financial/operational performance; and c) institutional development, including a checklist of key variables and an Institutional Development Report. Within this last category, five areas of core institutional capacity are assessed: governance and organizational structure; human resources; management systems; services and service delivery; and financial resources. A total of 26 elements make up these five areas, comprising a substantial set of variables that country offices are to monitor. It was noted that these indicators differ from other MIS measures, in that they must be reported largely in qualitative form. However, this category was made an “optional exercise,” and only two of the six countries were as yet reporting those 12 results back to the MF team. As explained: “Only the institutional development section appears not to have been very successful, and partners are given the option to report on these parameters or not.” In another case, although the grantee had made no provision for the measurement of changes in its institutional capacity, evaluators reported that its “increased ability to plan for the future, reinforced by significantly improved financial and administrative management systems allowing for better reporting and projecting, has been so remarkable that it is referred to among the significant successes of the MG.” For the MG-supported micro-enterprise development (MED) network, the technical areas of focus included: · Institutional Development: frameworks and tools for developing effective enterprise development organizations; · Network Development: diagnostic services, organizational development, and training of trainers for MED networks in developing countries; and · Evaluation: methods and tools for practitioner-led impact assessment at the client, business, and institutional level. A specific objective of the network was to increase member capacity to transfer best practices to field partners. The evaluation found that, “The extent to which [network] activities build capacity to reach out to member partners and affiliates appears to be less than their impact on the members themselves.” There was no objective aimed at having field organizations adopt the best practices, only that members be capable of transferring them. However, in terms of measuring changes in that capacity, the network stated: “We have not found an indicator to get at this information other than anecdotally.” Note to PVC: To ensure more measurable, concrete results from PVC’s investments in this area, proponents of grant-funded activities designed for capacity building should be asked to include in their proposals a description, not only of how they intend to build capacity, but also how they plan to measure change over time. To help potential grantees understand what is expected when developing such proposals, PVC could include in future requests for application a reference to the CDIE “Best Practices Paper” noted at the beginning of this discussion. This document provides a useful discussion of how USAID views institutional capacity building, as well as sections on: How to Measure Institutional Capacity; Measurement Issues; Institutional Assessment Tools; Measuring Individual Organizational Components; Developing Indicators; and Practical Tips for Busy Managers. 13 C. NETWORKING The strengthening of indigenous NGO networks is slated to be an important element of PVC’s new strategic framework. Since NGOs are frequently limited in capacity and reach, multi-organization initiatives that mobilize different groups around common concerns can expand NGO impact at the local and national level. Under past PVC strategies, networks have proven to be an effective means for helping PVOs and their partners to identify and address problems hindering program impact and to acquire the knowledge and skills needed to deal with programmatic challenges and policy issues. In addition to providing access to information, peer input and dialogue, networks have also been a platform for outreach to donors and governments. Of the ten Matching Grants evaluated, one provided direct support to a U.S. PVO network engaged in microenterprise development, while eight of the remaining nine grantees were members of a range of networks in the U.S. and overseas. Six sub-themes concerning networks emerged from the evaluations conducted: · Typology of Networks · Lessons for Lateral Learning Networks · The Network’s Role in Organizational Capacity Building · Network Sustainability · Advocacy · Networks as Mechanisms for Dissemination 1. Typology of Networks An assumption underlying support for networks is that linking NGOs together can enhance their problem-solving capacity, aggregate their interests, and magnify their ability to address issues that negatively impact beneficiaries. In examining this assumption, it should first be noted that there is tremendous variety in the types of development-oriented networks operating in the U.S. and overseas. They vary by the levels at which they operate, the purposes they serve, their operational structures, and the relationships they cultivate among their members. In general, a network can be defined as a set of relationships between and among organizations or individuals with common interests, goals and needs. The four most common kinds of networks are: · Generic NGO Networks or Consortia: These groupings are usually created at the national level and can include indigenous NGOs exclusively or international and local NGOs engaged in development, relief, or refugee assistance. The purpose of the network is to strengthen individual members and enhance their effectiveness, as well as the NGO sector by information sharing and dissemination, coordination of member activities, capacity building, research and, perhaps, fundraising. These networks also undertake advocacy on NGO￾specific issues as well as national or sectoral issues related to development. Examples of this type of network are InterAction, the Association of Development Agencies in Bangladesh and the Namibia Non-Governmental Organization Forum. For lateral learning networks, the most critical challenge is to ensure that member commitment and participation are constantly nurtured and supported through processes and structures that: define and update the network’s vision and goals; establish programs, their goals and objectives; and set association policy. Building Lateral Learning Networks: Lessons from the SEEP Network 14 · Networks of Community-Based Organizations: These networks, sometimes called federations or movements, are comprised of grassroots groups and associations and are created to encourage cooperation and collaboration among member groups, with more direct attention and involvement with development programs and projects than intermediary NGO networks. Examples include the Federation des ONG Senegalaises (FONGS) and the Organization of Rural Associations for Progress (ORAP) in Zimbabwe. · Sectoral Networks: These groupings focus attention on a particular sector and can undertake activities such as information sharing about the sector, capacity building in technical areas, program collaboration and coordination, joint research and training, and development and promotion of standards. Such networks can also engage in advocacy and policy dialogue with local or national government officials, bilateral and multilateral donors to promote policy change in the sector. These networks are sometimes created with donor resources and often at the urging of the donor. Examples of sectoral networks are the PVC-supported Small Enterprise Education and Promotion Network (SEEP) in the U.S., PROCOSI in Bolivia, and the African Forest Action Network in Cameroon. · Advocacy Networks: These groups, sometimes called alliances or coalitions, are created to engage with national or local government or international organizations for political, social or economic change. Created by organizations, networks and activist individuals, these alliances are often created in response to perceived opportunities or threats in the environment related to a particular issue such as women’s or children’s rights, agrarian reform, or democracy. Examples include the Civil Society Alliance (The Permanent National Forum for the Rights of Children and Adolescents) and the People’s Campaign for Agrarian Reform Network. In terms of organizational structure, two distinct types of networks were identified in the MGs evaluated. · Lateral Learning Networks: This type of network is sector specific and distinguished by its non-hierarchical structure and diverse membership. These networks tend to be composed of a range of independent organizations that do not necessarily share the same target clientele or development methodologies. What unites them is their interest in improving state-of-the-art practice, sharing information, and coordination to enhance the policy and funding environment in that particular sector. The services provided by lateral learning networks include information exchange, technical training, the development of best practice materials, donor marketing and policy coordination. 2 Networks of this type are operating at the international, regional and national levels. They may be informal, functioning largely on voluntary labor and in-kind contributions, or have a more formal structure, including a central secretariat with paid staff, membership requirements and dues. Where donor funding has been involved, as in the case of certain sectoral networks, these efforts have often started out as development projects with budgets, staff and a workplan intended to mobilize NGOs for particular activities, and have evolved into networks with formal structures. One of the ten MGs under review was awarded to a lateral learning network in the microfinance sector. 2 Building Lateral Learning Networks: Lessons from the SEEP Network (http://www.seepnetwork.org/LAT/SEEP_latlrn.html). 15 The network, known as the Small Enterprise Education and Promotion (SEEP) Network, is made up of 56 US PVOs that support micro and small enterprise programs in developing countries. 3 · Affiliate Networks: PVC grants have also been awarded to PVOs that have established “affiliate networks.” These are networks in which the members are operationally or financially linked. Typically these are formed around an individual PVO and the network comprises their country offices and affiliate organizations. The purpose of these networks can be to disseminate headquarters policy and harmonize technical approaches or to achieve scale and replication in service delivery. In contrast to a lateral learning network, members of affiliate networks tend to share the same methodology and are frequently led by an apex institution that provides technical guidance and resources, and enforces adherence to the principles and goals set by the alliance. Finally, affiliate networks are more directly involved in members’ resource mobilization. 2. Lessons for Lateral Learning Networks Only one of the ten MGs was in support of a network - a lateral learning network of MED￾related PVOs. Therefore, that evaluation provided the greatest detail on this topic. According to evaluators, the lateral approach “offers a learning opportunity for the successful operation of an association. As such, PVC may be able to use it to help other associations improve their structure, operations and impact.” Some of the lessons described below may also apply to other types of networks, particularly those aimed at strengthening member capacity and improving service delivery. A clear message that emerged from that evaluation was that this investment was cost-effective: “Overall, this is a high impact, highly effective grant that reaches field organizations with useful tools and practices at a low cost to PVC... it has resulted in a highly cost effective grant. USAID is getting excellent ‘bang for its buck’.” Bottom up planning matched with shared vision and objectives were identified as key to the network’s success. Limiting memberships to practitioners – government agencies, donors, consulting and research firms are barred from membership – has also encouraged the development of technical products that are ready for use by local MED institutions. Workshops, seminars, newsletters, websites, and communications are the tools of networking. But above all, working groups formed the backbone of this MED network and its lateral learning approach.4 The MG provided considerable funding for working group outputs as well as a retainer/stipend for group leaders. This was a major change from the original all-volunteer arrangement. In response to various member surveys, it was confirmed that the facilitator/retainer arrangement had “greatly improved the management and productivity of the working groups.” According to one respondent: “Products are developed more efficiently. The leader keeps the process on schedule and is able to do some of the busy work in getting products 3 The objective of the grant was to increase the effectiveness of MED practices on the part of U.S. PVOs and their southern partners through Institutional Development Services (IDS). The network also supports developing country MED networks, although there was no MG funding earmarked for this. Funding from other donors was used to implement a Network Development Services (NDS) program. 4 Presently there are seven working groups: business development services, client impact, financial services, poverty lending, technical assistance, and microenterprise and HIV/AIDS. 16 developed.” Member surveys and especially the working group surveys consistently pointed out that the greatest benefits from network membership had come when the staff of member organizations contributed most actively. Various versions of saying ‘you get what you give’ were often found in the surveys. Operational lessons learned by the network included: · Define the membership. · Set and adhere to guiding principles that acknowledge equality and community. · Foster members’ engagement in governance and policy formation. · Focus on the practitioner. · Start with one activity and do it well. · Create mechanisms for learning that favor collective analysis, and include opportunities for all to teach and to learn. · Focus on products and their dissemination. · Grow organically; start small and expand as needed. · Establish the network as a formal institution only when needed. · Keep core operations compact and expenses low; depend on substantial contributions from members. · Do not engage in competition with members for program funds. · Distinguish political activity from the research/learning agenda. Despite the ability to source additional funds for larger projects, some members stated that they were facing tasks beyond the working group’s capacity. There was also the recognition among members that, although one can learn a lot from one’s peers, networks cannot do it all. Evaluators cautioned that the network structure per se is not enough to ensure its success, noting that, “networks can be inefficient in completing tasks because of coordination and commitment lapses.” Their reliance on personal ties “often restricts access,” while “a few members may do most of the work.” It was also noted that lateral learning might benefit from a more formal system of technical and operational exchanges among members and field entities. The principle future challenges identified by evaluators center on the question of maintaining the successful practices and products of the network as a member-driven association while it grows and expands. These challenges include: · Management of people, programs and funding: The network’s budget is projected to increase five-fold in 2003 compared to its 1999 level, while the number of staff has doubled in the same period. New programs and grants are being added with more products and outputs planned. Directing and managing the staff, programs and outputs while keeping its core work of meeting member needs is an operational challenge. Ensuring volunteer support as needed to keep network products and outputs technically strong and practitioner-relevant is a second major challenge associated with program growth. · Increasing diversity of member interests and needs: Membership in the network is growing and the diversity of its member interests is expanding. Creating new working groups and other actions to respond to members and foster their participation is an operational and motivational challenge. 17 · Knowledge management: There is an enormous amount of MED information that is now available and needed by the network to make its programs and products successful and useful. Dissemination and communication technologies are advancing quickly to meet the challenges of knowledge management. The challenge for the network is to position itself in the center of this knowledge management revolution in order to continue to be effective in responding to its member and sector needs. 3. The Network’s Role in Organizational Capacity Building Clearly, networks can play a valuable role in getting newer PVOs up to speed by providing standards and disseminating best practices and tools. While the PVC-supported network has been effective in helping members who are new to MED, it has been more difficult to respond to the needs of technically advanced members. A challenge for networks is how to satisfy a broad spectrum of members, particularly when it comes to strengthening the capacity of individual member organizations. While the network did not collect data related to PVO capacity building, evaluators concluded that the network’s “products and activities are having an impact on member performance through the better understanding and application of best practices to their operations.” It was also reported that, “evaluation survey data show that [the network’s] activities are building some member capacity to transfer and apply best practices to field partners.” Nevertheless, the evaluators concluded that there is “a need to further review the field impact of [network] member activities on their partners, service delivery organizations in developing countries, especially to compare the tradeoffs and complementarity between the member assisted channels for field support and [its] own Network Development Services” (NDS). Overseas MED Networks: This grantee raised funds from other donors to implement its NDS program. This includes diagnostic services, organizational development, and training of trainers for MED networks in developing countries. This activity reached 18 organizations worldwide, 12 of which are in Africa. However, evaluators were “left with questions on the nature of the impact on field operations as well as the need for more research on activity impact, the best channels for partner change, and the qualities of the local organization to best utilize [this] assistance.” The NDS methodology employed a three-stage approach: · Building relationships; · Action planning using a ‘toolkit’ developed by the network; and · Implementation and follow-up. Evaluators concluded that, “The activities that [the network] has undertaken in the last three years to support network development are numerous and impressive in their breadth of technical support to 18 organizations spread around the world. There appears to be a high level of satisfaction with… training, technical assistance and other support to networks and members.” Another conclusion reached was: “At this early stage of MED network development, [the network] has been able to have a considerable impact on those networks assisted even though the interventions have been small. This approach has worked well as [the network] has linked with several organizations to leverage their resources. Its approach needs to be reviewed periodically 18 as networks mature and donor and other funding for networks increases… Small amounts of assistance are highly effective in the initial stages of the development of overseas MED networks.” Note to PVC: It is often assumed that networks are of considerable value in strengthening member PVOs and indigenous NGOs. However, there is little evidence in the evaluations conducted to substantiate this assumption. No visits to overseas networks were included in the SEEP evaluation, and only four of the 18 networks responded to the survey sent out by evaluators. Moreover, while important for PVC’s long term planning, there does not appear to be a great deal of information concerning the effectiveness of indigenous NGO networks. Related to this is the Network Capacity Assessment Tool produced by SEEP, which was used to assess 16 national networks representing 480 MFIs. The findings from these assessments and the utility of the tool are not yet fully explored. It would be in the interest of all concerned for PVC to address this gap and to explore in greater depth the effectiveness of indigenous NGO networks through its Analytic Agenda. 4. Network Sustainability An important consideration for PVC is the longer-term sustainability of the networks it supports. In the case of SEEP, it was found that expecting full financial sustainability is not realistic, since member organizations are not themselves financially independent. Networks that have a development mission are likely to remain dependent on third party funding. There is also a potential conflict of interest related to resource development, since the individual funding agendas of members may clash with that of the network – a dilemma endemic to cluster organizations. As evaluators pointed out, “It is unrealistic to expect a high level of financial sustainability on the part of SEEP. It is not just a service organization for its members, but a PVO in its own right with a development mission comparable to its members. Like its individual members, SEEP will be donor dependent but it also has the opportunity for substantial cost recovery from fees, sales and other income.” For SEEP, increasing dependence on contract work was seen as problematic, given that competitive awards are not well suited to the network’s structure and operations as a volunteer-dependent association. Nevertheless, networks need to strive for cost-recovery and the diversification of funding sources. There is also some concern that if a volunteer-based network raises membership fees substantially, it may undermine its volunteer base. This represents another potential conflict of interest, since the network could begin to compete with its members for the same pool of volunteers. Unfortunately, though the increased institutional sustainability of SEEP was an objective of the grant, no specific sustainability targets were included in the DIP. Therefore, evaluators were not able to gauge the precise degree of accomplishment in this regard. 19 Note to PVC: In any future grants designed to support the institutional sustainability of networks or alliances, in order to measure results it would be important for PVC to ensure that appropriate indicators and targets are described in the grant and included in the DIP. Such indicators should be designed to track trends that are critical to the network’s sustainability over time, such as board involvement in fundraising; changes in the governance structure; percentage of operating funds raised from donors and changes in the number of donors that provide those funds; changes in sources of income, such as membership and other fees, sale of publications, and so forth. 5. Advocacy As noted earlier in this report, PVOs are increasingly shifting from the provision of direct services, becoming facilitators for the building of local civil society organizations and multi￾sector alliances to solve local problems and deliver services. It is expected that this trend will continue, with indigenous NGOs taking greater responsibility for community and national development over time, and different relationships (partnerships, networks and coalitions) forming between PVOs and NGOs. PVC support has been important in building the advocacy-related skills of PVOs/NGOs, particularly through networks and coalitions. For example, while SEEP does not formally undertake advocacy activities, it has provided a platform to amplify the voice of PVOs in international policy discussion. A key role has been in collecting credible data and using its capacity to convene members for the purpose of influencing decision-makers. Five of the ten grantees evaluated used networks to develop advocacy strategies for sectoral policy reform in areas such as microfinance, the environment, and health. Networks have also been used to advocate for the adoption or use of particular program models or methodologies developed by the PVOs. A number of illustrative examples of the use of networks for advocacy are outlined below. · Improving the Enabling Environment: Networks have proven to be an effective mechanism for promoting regulatory reform and improving the enabling environment for NGOs. With MG support, this has been particularly relevant in the microfinance sector where networks have been the main channels for MF-related advocacy activities. One of the PVOs evaluated worked with local MFI associations in Kenya and Uganda to push for the regulation and supervision of the MF sector. In Uganda, the association began a dialogue with the government and the Central Bank on industry regulation. Evaluators reported that the network had “educated the Central Bank and generated interest about MF to such an extent that Central Bank employees received MF training in Boulder, Colorado.” The former general manager of the MFI was also closely involved with the development of a draft bill governing the sector which was pending in the legislature. In Kenya, the association has been involved in drafting a Microfinance Act that was pending in the Attorney General’s office awaiting action. In both cases, these associations were supported by the respective USAID Missions, and also received funding from USAID/W through the Microenterprise Development Office. 20 · Promoting Innovative Models, Scale-up and Replication: In some cases it was shown that peer networks can also play an important role promoting innovative program models. In the MF sector, one grantee and its local partners used network membership to advocate for a MF model that integrates health education with village banking services. Their main targets have been peer agencies, governments and donors. Despite resistance by some MFIs to this integrated approach, evaluators noted that participation in networks had been an important advocacy and dissemination strategy for the PVO and its partners. The careful analysis of case studies and the financial analysis of the credit-with-education experience in the field have given the PVO “an arsenal of research which it draws upon in its advocacy work.” However, by its own admission this PVO “has not impacted on the major players – donors, MFIs – some of whom are unaware of its work.” · Coalition building: Another PVO demonstrated what evaluators termed “significant progress” in developing coalitions to advocate for environmental issues in Jamaica and Indonesia. In the case of Jamaica, the grantee was instrumental in establishing a national network of local NGOs involved in protected area management. The network was effectively used to promote policy reform, and according to evaluators, “continues to provide a productive protective area management policy forum.” In Indonesia, the PVO and its local partner developed site-based coalitions as constituencies for conservation in two national parks and played an important role in helping aggregate community interests and facilitate community/park communications. Evaluators concluded that, in both countries, the policy and advocacy successes at the community, local and national levels might have been “the most significant on-the-ground impact of the project.” In one country, a major area of tension emerged between the PVO and a local NGO when the latter supported demonstrations against the government that were perceived by the PVO as being “overly confrontational.” This illustrates how differences in advocacy strategies can create discord between partners. It was recommended that establishing an ‘arms length’ distance between the NGO and PVO would provide greater latitude for the NGO to adopt advocacy tactics at variance with the PVO, while not jeopardizing the PVO’s status in the country by being associated with “internal” political issues. Evaluators concluded that, properly managed, there is no reason why the PVO “could not maintain its non￾confrontational approach (befitting an international NGO) while retaining the flexibility of allowing partner local NGOs the freedom to act more aggressively for change.” Note to PVC: PVC’s new strategic framework indicates that advocacy by local networks for the purpose of strengthening NGO support, creating enabling environments for NGOs and addressing service delivery issues will be an important part of its efforts aimed at NGO strengthening. With that in mind, PVC could capitalize on experience to date by gathering additional information and documenting the extent to which the networks to which grant-supported PVOs and NGOs belong have become effective advocates for improved practices or regulatory reform. 21 6. Networks as Mechanisms for Dissemination In addition to advocating for policy reform, PVO and NGO networks also have the potential for improving the efficiency of service delivery and increasing program impact and coverage. In the evaluations conducted, networks were found to be effective and inexpensive mechanisms for launching new initiatives and disseminating best practices, methodologies, and tools. A number of grantees had joined or helped create networks in developing countries in order to play a practical role in their programming. Under a World Bank contract, one grantee helped establish the Ghana MFI Network (GHAMFIN), and the PVO’s continuing efforts in building the institutional capacity of the network helped it to become an independent entity. Another PVO successfully launched a self-sustaining network of subsidiary MF practitioners in six pilot countries. Annual meetings of this Microfinance Network and a newsletter published by its MF technical team are important vehicles for exchanging information among the PVO’s country offices and between country offices and their microfinance partners. This network has proven to be an effective means for exposing partners to new methodologies in a range of different contexts. The PVO’s eventual goal is to expand the network to all 43 countries in which it has a presence. MF advisors in USAID missions are in close contact with the PVO through membership in informal or formal national networks and umbrella organizations. The PVO’s partners in all six pilot countries also belong to local networks of MF practitioner organizations. Note to PVC: Since support for a variety of national, regional and international networks has been provided through Matching Grants, PVC could seek the cooperation of grantees to compile a compendium of information on the networks with which they have worked, including a brief description of their background, purpose, location, point of contact, and so forth. This could then be made available to the broader development community as a contribution to in￾country programming. 22 D. SUSTAINABILITY Sustainability is clearly a matter of concern to donors and grantees alike. Of the MGs evaluated, one included the increased institutional sustainability of the PVO itself as an objective of the grant, while three others involved goals or objectives addressing the sustainability of new policies or program approaches within recipient organizations or their local partners. While sustainability was not addressed directly in other MGs, it was clearly an underlying concern. For this theme, three major sub-categories emerged from the evaluations reviewed: · Multidimensional Aspects of Sustainability · Financial Sustainability · Sustainability of New Approaches and Policies 1. Multidimensional Aspects of Sustainability One grantee defined sustainability as being comprised of four dimensions: a) institutional; b) financial; c) behavioral; and d) policy. A number of others, particularly those involved in micro￾finance, appeared to view sustainability almost entirely as a matter of financial viability. Though, for one PVO, a combination of “financial and operational sustainability of MF partners lies at the heart of the program model.” This grantee stressed that: “Organizational sustainability of MFI partners is necessary if financial services are to continue,” noting that this was especially important in areas where it intends to phase out (after 8-10 years), once a particular set of communities has reached a certain level of socio-economic development. Note to PVC: Given the different connotations associated with the concept of sustainability, it would be helpful for PVC to clarify for grantees its own interpretation of the word. 2. Financial Sustainability Economic viability was of special importance to grantees working in the field of micro-finance and economic development. Much attention was given to building and measuring the financial sustainability of partner organizations and businesses. However, with one exception, there appeared to be much less emphasis on the longer-term sustainability of the services provided by the PVOs themselves, which could present serious challenges in the future. As reported in one case, the PVO “needs a strategic cum business plan which sets partnership, regional and other targets and which articulates a plan for its own sustainability.” One view held that the cost of services should be recovered from the organizations and businesses assisted, while others felt that it is appropriate for these services to be subsidized, since their success is based on positive results at the beneficiary level. One report stated: “financial sustainability may not be a realistic objective,” stating flatly that the organization will Sustainability implies lasting positive change in institutions, behaviors, or policies affecting human well-being. Sustainability involves enabling individuals, communities and institutions to promote change and to adopt new behaviors and systems that endure during and after SC’s involvement. Save the Children 23 continue to be donor dependent. Indeed, though some development organizations become financially self-reliant (i.e., capable of raising funds from individuals or donor agencies) it is uncommon for them to produce sufficient income of their own to be self-sustaining. An example of such a dilemma involved a PVO whose MG focused on the development of a new product designed to strengthen micro-finance organizations, rather than on direct program implementation. Since this PVO ceased to be an implementing agency, it depends instead on partners to achieve its mission and contribute to the greater development good. Therefore, under the MG, it provided at no cost to partner MFIs capacity-building training and technical assistance aimed at achieving their financial sustainability through increased income from credit operations. This included the tools and materials produced by the PVO through the R&D work funded through the grant. While the PVO strives to help ensure the sustainability of its partners, it has no interest income stream or sustainability plan of its own. As reported, “In the past, such partnerships have relied on third party funding,” and the investment has apparently produced positive results. Clearly, however, these types of partnerships may not be replicable from a cost or human resource point of view. In another case, the goal of the MG included the promotion of equitable rural growth and the provision of sustainable services to the rural poor. Evaluators found that the model developed “may well be sustainable, but it is clearly based on high levels of US government support.” This PVO’s plan is increasingly to operate like a consulting firm, with the enterprises assisted paying for the services they receive. Currently, fees represent less than 1% of all costs, and evaluators found that: “In reality, few of the businesses assisted… actually pay fees and none pay at a rate that reflects the true costs of the assistance they receive… It would be impossible for businesses at their current state of evolution to pay these fees even if they wanted to.” In this case, the larger question posed was whether this PVO should be sustainable in the sense that a MF program is sustainable through internally generated income, i.e., interest on loans? If the answers is yes, “the PVO would be driven to focus its efforts on those few major businesses that can pay the hefty fees necessary to cover its costs.” This would quite likely result in a clear shift away from a focus on rural poverty, which is certainly not a desired result. Evaluators concluded that the PVO provides a valuable R&D service that links small farmers to markets and that, in this case, “charging token fees with no expectation of recouping the true development costs is probably justified.” It was then pointed out that, once the hard work of applying the PVO’s methodology has been demonstrated, “the co-optation of this experience by the private sector is a sign of success.” Evaluators suggested that an alternative might be to become an investor/shareholder in some of the businesses assisted, or possibly even in others not assisted. For yet another grantee, which sought to establish a regional program in a new approach to MF, evaluators found: “The MG objective of financial sustainability was not realistic and was not met” in either of the two pilot countries. The fact that the two experiences “have cost so much and the goal of financial sustainability remains years into the future does raise a question about the cost effectiveness of the approach.” It was also noted that there had been “a first time learning curve that has created extra costs.” With MG support, one PVO produced a Microfinance MIS, which was field tested in the six pilot countries selected for this grant and revised in line with experience and to expand reporting 24 formats to cover new types of MF programs and partners. The MIS corresponds with one of the key outputs described in the project logframe, and includes seven “sustainability indicators:” · Return on performing assets · Financial cost ratio · Loan loss provision ratio · Operating cost ratio · Imputed cost of capital ratio · Operating self-sufficiency · Financial self-sufficiency Finally, experience in the field of micro-finance raises the central question of impact vs. sustainability. If an MFI is to be sustainable and considered “successful,” it must produce sufficient income to cover costs. This invariably means making “safer” loans to those who can best afford them. The challenge is how to reconcile the need to serve the poorest of the poor and make safer loans at the same time. Note to PVC: The issues raised concerning the financial sustainability of the services provided by PVOs and consideration of whether or not to expect that those costs be recouped from the recipients of those services could be included in PVC’s Analytic Agenda, particularly since it bears significantly on the topic of partnership. 3. Sustainability of New Approaches and Policies In two cases, the goal of the MG involved the institutional sustainability of new policies or program approaches within grantee organizations. These addressed various innovations in each of the organizations, and included the adoption of partnership as a new program approach in both. Evaluators found in both instances that these new approaches had reached a high level of sustainability within grantee organizations. As concluded in one report: “It is clear that [the new] concepts have indeed been institutionalized and will be sustained by the organization as major programming principles.” In the other case, evaluators concluded that the “success of efforts to date seems to militate in favor of continued high-level support.” One of the objectives of another MG involved the PVO’s capacity to plan and support the implementation and monitoring of sustainable field programs. It was found that field offices “have been successful in sustaining the basic tenets of the Matching Grant.” Specifically, the programs initiated by partnerships in pilot countries “are all continuing, either directly with the same participants or through integration into the partners’ own programming.” Moreover, the substantive policy advocated by the PVO “has been adopted by all partners,” and, in addition, “most partners have accepted the importance of having M&E in their planning for assessing the performance of their programs.” Note to PVC: The adoption and institutional sustainability of new and effective policies or programs can turn entire organizations around and lead to longer-term, more far-reaching impacts. Therefore, PVC could make an important contribution to development planning by calling 25 for a more in-depth examination of the ramifications of support for this type of organizational change as compared with more traditional approaches to sustainability. 26 E. PVC MANAGEMENT OF MATCHING GRANTS The evaluations undertaken produced useful input for the management of future Matching Grants. They also led to important questions that merit consideration by PVC as it determines the types of investments that will best ensure the achievement of its own strategic objectives over the coming years. Findings for this theme are organized into the following three categories: · Grant Application, Review & Selection Process · Grant Management · Other Outstanding Issues 1. Grant Application, Review & Selection Process A number of PVOs felt strongly that PVC’s grant application process needs to be more efficient. They underscored that the cost of preparing Matching Grant proposals runs as high as $20,000- $30,000, which represents a significant investment, particularly for newer and smaller organizations. Combined with the risk of not being selected for an award, these costs represent a barrier to some organizations whose proposals might well be of interest to PVC. Based on experience with other donors, such as DFID, one concrete suggestion by several PVOs was that the application process be streamlined by dividing it into two phases. In the first phase, interested organizations would be asked to present 4-5 page concept papers, outlining the objectives, results to be achieved, and other necessary information about their proposals. If PVC determines that the proposed activities are of interest and fit within its strategic priorities, it then invites the PVO to present a full-blown proposal. If not, then at least the applicant is spared the high cost of full proposal development and PVC reduces the time and effort needed for the review process. A key finding that emerged with regard to proposal reviews was that, in several cases, a more careful examination of the technical feasibility of the new models proposed and of the proponent’s capacity to implement them could have avoided failed efforts. This, in turn, raised questions about PVC’s investment priorities, the role of review teams, selection criteria, and who is accountable for these decisions. Among the main issues that emerged were: · In some cases, the choice was to fund the development and testing of innovative models or methodologies by organizations with experience in a particular sector in order to move that sector forward. One lesson was that, in selecting such innovative grants, in addition to analyzing the feasibility of the new model, it behooves reviewers to consider how much capacity the PVO needs for successful implementation. In two instances, that capacity consisted of only one or two staff members, and produced less than ideal results. · In other cases, grants supported activities in a sector in which the PVO had no experience - a form of “institutional innovation” designed to strengthen the organization’s capacity or to For managers, the dynamics of knowledge impose one clear imperative: every organization has to build the management of change into its very structure…. On the one hand, this means every organization has to prepare for the abandonment of everything it does… On the other hand, every organization must devote itself to creating the new. Peter F. Drucker, Managing in a Time of Great Change, 1995 27 move it further along the relief-to-development continuum. One PVO explained that the purpose of its MG was not direct service delivery, but organizational change – described as a process by which USAID can affect the evolution of PVO policies, programs, and practices. And, since a match is required, the use of funds for serious organizational investment can be assured. As stated in one report: “The value of the MG… is that it allowed the organization to experiment programmatically and organizationally at reduced expense to itself.” While the PVO may have been able to find the extra $2.25 million, “the point is that bringing in additional monies from an important donor lends credibility and cachet to institutional innovation,” the scope of which was so great that it involved “significant culture change.” With one exception, the results of these grants were generally positive, though the degree to which these institutional innovations are sustained will only become clear over time. · Related to grantee selection is PVC’s desire to support new organizations in order to expand the overall capacity of the PVO community and help move the entire development field forward. This was reported to come at a high cost for PVC in terms of time, money and level of effort. Based on experience to date, it generally takes nine to ten years (two MG cycles) for organizations to get up and running. However, it is clear that in the current environment PVC’s support will need to be reduced to three to five years (one MG cycle). Therefore, the question is whether and how PVC should continue to support new organizations, while reducing the time needed for them to reach an adequate level of stability. It was suggested that, to avoid mistakes and false starts as new PVOs move along the learning curve, they could devise collaborative arrangements with technically strong organizations or consulting partners who could provide information or advice, even on an informal basis. This could be a particularly effective way to deal with needs and challenges related to the development of MFIs where, reportedly, there is a “culture of measurement.” It was noted that PVC is uniquely positioned to facilitate (but not fund or manage) such cross-fertilization among grantees by providing relevant information and timely referrals. · Questions were raised about PVC’s criteria for selecting “pilot countries,” and who should make those decisions. Reportedly, in some instances the countries finally approved by PVC from among those proposed were not the ones the PVO itself would have chosen as first priority. It was recommended that, to increase the effectiveness of MGs aimed at strengthening the capacity of entire organizations or institutionalizing new approaches, PVC should consider eliminating altogether the requirement that grantees identify at the outset the specific countries in which they will pilot grant-related activities. In one case, it was concluded that, while the PVO had been required to allocate a good portion of grant funds to the four pilot countries (62% of the total budget), greater progress had been made through more institution-wide initiatives, and that country-specific requirements had been a deterrent to even greater gains. For another grant, where major problems and few accomplishments were recorded, it was found that design issues had been prominent, along with the lack of “ownership” within the organization, and that country selection had negatively affected both. While the PVO had proposed to work in five countries (each with a constituency at HQ and in regional/field offices) to test and incorporate a new component into its traditional activities, PVC had reduced the number of pilot countries to two. Evaluators found that by cutting down the size of the award and the number of countries, PVC effectively hampered what was to be an organization-wide effect. As a result, the PVO implemented two country programs, instead 28 of a grant that was to institutionalize an organization-wide innovation, and ownership/interest in the effort among PVO officials and staff virtually evaporated. Of course, the PVO bears some responsibility for attempting the suggested revisions. But often it is difficult for PVOs to be as firm as they need to be in maintaining program integrity when millions of dollars are at stake. Note to PVC: The evidence presented in these evaluations argues for PVC consideration of improvements in the process of grant application, review and selection in order that it be less time￾consuming and more cost-effective for all concerned. The suggestions outlined in the foregoing discussion could help to guide that process. With its new strategy in place, PVC will need to balance its portfolio and develop written criteria for the types of organizations and activities it intends to support. This information can then be used to orient review teams as well as potential applicants. 2. Grant Management DIPs: Grant management implies monitoring activities and assessing investment performance over the life of the grant. This begins with a Detailed Implementation Plan (DIP), developed by the PVO in accordance with the matrix provided by PVC and approved by the CTO. The four columns of the matrix are: Objectives; Indicators; Measurement Method, Data Source & Frequency of Data Collection; and Major Planned Activities. In theory, the DIP is to serve as a plan for guiding and monitoring performance. However, this did not prove to be the case. Only one grantee, new to PVC, felt that the DIP had been useful in developing its first three-year plan and as a “roadmap for operations,” though it was not shared with or used by its two partners. In this case, questions were raised about the quality of the DIP, which contained 25 objectives – many of which were activities – and 51 indicators. As designed, the DIP combines elements of a strategic plan (similar to a logframe or results framework) with those of a performance monitoring plan – two very different tools for the management of most development activities. The resulting hybrid was shown to create confusion and, more importantly, to remain unused once approved. Virtually all evaluation reports contained conclusions and recommendations calling for improvements in the design and use of the DIP. The major points included the following: · The format of the matrix should be improved, incorporating the elements of a results framework – i.e., indicators that measure performance at all levels (including capacity building), and in terms of quality, quantity and time, as well as specific PACD targets related to those indicators. And DIPs should be subjected to greater technical review as part of the approval process. One recommendation was that, “Special attention should be given to the often-confusing area of capacity building,” and that, “In technical programs, such as micro￾finance, where there are industry-wide performance standards and targets, PVC should ensure that they are used in MG implementation planning.” In general, the DIP was seen as a requirement with which grantees needed to comply, but rarely referred to once approved. Typical of findings related to this were these statements: “The original DIP was overly ambitious and poorly designed, and lacked sufficient rigor to 29 objectively measure project outcomes and impact… USAID was derelict in failing to provide formal comments on the quality of the DIP and suggesting changes to improve its design logic… The DIP as submitted was not a useful tool for visualizing the implementation of this program. While institutionalization accomplishments are numerous and real, they were not outlined or described in the DIP. The document as submitted and accepted could not serve as a management tool.” · PVC should provide for the periodic review and possible revision of DIPs in order that the activities funded may maximize emerging opportunities or overcome unexpected obstacles. This is particularly important in cases of project designs that prove to be flawed, high staff turnover rates, and other unanticipated factors. In general, DIPs were treated as static documents; no periodic, formal reviews were conducted. It was asserted that PVC needs to provide formal and timely feedback by way of a DIP review if it is to hold PVOs accountable for implementing their programs in accordance with their DIPs. In two cases, based on discussions with the CTO, program strategies changed significantly in the early phase of the grants. However, no new or revised DIPs were written. Therefore, evaluators found no concrete evidence of strategic or work plans relevant to the revised designs, how performance was monitored or what the targets were and if they had been met. As stated in one report, once plans shifted, the “value of the DIP dropped even further; no revisions were made.” It was suggested that problems in DIP design/implementation be identified by PVO and PVC staff within the first 12-18 months, and that the DIP be revised accordingly. The point was also made that, when funding experimental initiatives, to facilitate the learning process, PVC should expect DIPs to be modified periodically, requiring that grantees establish performance monitoring plans designed to capture and document results at specific intervals. Information should then be analyzed and used to modify on￾going objectives and activities in accordance with the lessons that emerge and in concert with the CTO. · To instill a greater sense of ownership and to facilitate data collection and overall performance monitoring, grantees should involve their partners more actively in program planning, particularly the formulation of indicators for their respective parts of the activity, and they should be familiarized with the DIP and its use. · PVC should require that DIPs be used as a basis on which grantees prepare their Annual Reports. Thus, performance could be reported in relation to the objectives stated in the framework, and measured in accordance with approved indicators and targets. In several cases, it was suggested that PVC should make greater use of these reports as a basis for periodic, face-to-face meetings with grantees in order to address emerging problems collaboratively by highlighting implementation problems, as well as successes. Technical Oversight: Another issue discussed in various evaluations was the question of PVC’s technical oversight of grant operations and performance. The overall impression that emerged was that there is no specific policy or procedure that CTOs are to follow, and that the degree of oversight varies considerably from one grantee to another. A number of PVOs felt that greater oversight and technical skills were needed on the part of PVC, and would have helped to avoid or remedy problems. In other cases it was reported that the CTO, in addition to monitoring grant performance, had become intimately involved in decisions about on-going project operations. 30 As stated in one report, “Innovative technical approaches funded under an MG need to be followed closely by technically qualified staff.” It was stressed that, if the program is not working out as technically planned, as was the case in this MF-related activity, “the technical skills and judgment of the officer are critical to guide revisions and protect USG grant ‘investments’.” That led to the following recommendation: “PVC needs strong and experienced MF staff or technical advisors available to them to review implementation and identify MF technical challenges as they arise. Experienced grant managers are not enough in the fast changing field of MF. Technical skills are necessary.” Another report stated: “The key finding in USAID management was the lack of technical supervision and input to [the PVO] when they were shifting approaches and structure in the first two-and-a-half years of the grant. In fact, it was not until March 2000 that USAID approved the changes in the grant that had been discussed in 1996-1998 and implemented in 1997 through 1999… That PVC did not have a technical MF officer on staff from the start of the grant until the present MG manager and MF Specialist arrived in mid ’99 was a serious weakness.” It was further noted that the MG manager during the critical transition period “had a full load of other management responsibilities” to which this grant was added, and that, “He did not have MF technical background.” Relations with Grantees: These evaluations provide useful comments and suggestions about how PVC relates to and communicates with its grantees. In the majority of cases, relations were found to be cordial and helpful. One PVO considered “USAID a genuine partner in its activities,” and that it “felt supported by PVC during the MG.” In another case, evaluators concluded: “Relations between the project manager [at the PVO] and the MG manager at USAID were too cordial and allowed for a degree of informality that precluded a thorough and critical review of the DIP.” On the other hand, difficulties were reported in some areas. For instance, several PVOs reported long delays in PVC’s responses to their concerns or that they had received insufficient feedback from PVC on their annual reports . One stated that the realignment of grant activities had taken a full year to complete, and had been “complicated by USAID’s slow response.” It was urged that: “Greater attention should be paid to the original proposal and the subsequent design laid out in the DIP.” Reportedly, the lack of response from PVC made some grantees hesitant to transmit information of a negative nature to their CTOs. For that reason, in one case, evaluators recommended: “USAID should consider greater use of frank, face-to-face meetings between PVO project managers and their CTOs. These could be held twice per year and focus on implementation challenges, reconsidering fundamental design issues, agreeing and reviewing results indicators, and lessons learned.” In a similar vein, for one PVO it was reported that, in the early years, communication with PVC “was not supple.” It was obvious that “the process of DIP design suffered from lack of communication with PVC.” To facilitate on-going supervision and help ensure that PVC learns from emerging lessons, it was suggested that, “It would be useful within the MG Division of USAID/PVC for grant managers overseeing similar sectoral activities to meet regularly to discuss the accomplishments under 31 each grant. This would include sharing findings or issues provided in the annual reports. It would definitely mean comparing the results of midterm and final evaluations.” Note to PVC: Grant management within PVC could be greatly improved and lessons better captured by incorporating the suggestions and recommendations that emerged from these evaluations. Specific steps might include: · Re-design the DIP matrix to include PACD Targets for all goals and objectives. Or, in the alternative, abandon the DIP in favor of logframes or results frameworks. In either case, performance monitoring plans should also be required for all grants. · Ask grantees to provide evidence of participation by field staff and partners, as appropriate, in the formulation of indicators and the preparation of the DIP. · Conduct formal technical reviews of DIPs as part of the approval process, with particular attention to the inclusion of appropriate indicators to measure performance at all levels of the activity and in terms of quantity, quality and time. · Require grantees to use the DIP as a basis for the preparation of their annual reports, tracking performance in accordance with approved indicators and targets. · Conduct regular, periodic DIP reviews with grantees, and ensure the incorporation of appropriate adjustments in cases where realignments are necessary due to changes in the underlying program strategy or where the technical approach is not producing the desired results. · Ensure an appropriate level of technical oversight for all grants, engaging specialized advisors for areas where staff do not have relevant experience or skills. · Ensure prompt feedback from PVC program managers to all grantees they oversee upon receipt of annual reports and other project-related information. · Adopt as an official policy or procedure the convening of regular meetings of PVC grant managers to exchange information and discuss findings or issues raised in grantees’ annual reports, as well as the results of any evaluations undertaken. 3. Other Outstanding Issues: In addition to the foregoing, several other issues deserving of PVC’s attention emerged from these evaluations, and are briefly summarized below in the hope that they may be useful for the management of future grants: · Product Dissemination: With MG support, grantees have developed a variety of studies, tools and other products of interest to the broader PVO community and to local NGOs and other stakeholders. While some grants call for dissemination by the individual PVO, there is no concrete, overall dissemination plan for all of the new products or methodologies produced. Thus, PVC has not yet capitalized on this investment. Note to PVC: Since the new strategic framework announced by PVC calls for “leveraging intellectual resources,” a start could be made by creating a glossary of the products already developed with PVC funding, and working with grantees to devise a comprehensive plan to ensure their availability to and use by other interested parties. Special attention should be given to the 32 transfer of relevant products for the purpose of strengthening community-based NGOs, which is a key element of the strategic framework. · M&E Plans: All evaluation reports provided DIPs and other related materials for the grants reviewed. Thus, in addition to the information provided in the narrative, these documents were examined, particularly with reference to the second theme identified by PVC – “Measuring Capacity Building.” At the same time, this examination, together with the findings, conclusions and recommendations provided in the reports, led to the strong impression that the purpose, design and use of M&E systems as important tools for overall implementation and management are largely misunderstood by both PVC and PVO program managers. The formulation of such systems and Performance Monitoring Plans (PMPs) as an integral part of the design process, and their incorporation in DIPs and work plans, would help to ensure that the information collected is not only useful to but also used by grant managers. Note to PVC: For these reasons, it is suggested that PVC consider providing joint orientation or training sessions for its own grant managers and key PVO staff once new grantees are selected. This could cover the basics of PMP development and use, as well as an “indicator exchange,” where grantees share and discuss the indicators they have used for specific goals and objectives. This, too, would contribute to the “leveraging of intellectual resources” as envisioned in PVC’s “Research, Development and Outreach (RD&O) Agenda,” as well as key elements of its new strategic framework. · Strengthening Individual NGOs or the NGO Sector: PVC has stated that a key element of its new strategic framework is to strengthen local co-ops and community based NGOs to improve delivery of services. In that connection, PVC’s intention is to support “linkages with local governments, corporations and other networks,” through which advocacy “would be undertaken to strengthen NGO support, create enabling environments for NGOs and address service delivery issues.” During the debriefings held following these MG evaluations, questions were raised about the strategic implications of PVC support for initiatives designed to strengthen individual NGOs vs. those aimed at strengthening the NGO sector as a whole. Though, in some cases, building the capacity of a single influential NGO might well impact the entire sector, this was not seen to be the norm. A related issue was the choices involved in aiming for the broadest possible impact at the community level. One option is to fund the strengthening of NGOs or a network of NGOs in a particular technical sector, while another is to target a particular geographic area and work with the NGOs, local governments and businesses present in that location. Yet another option is to support advocacy at the policy level to improve the enabling environment, as outlined in PVC’s new framework. These approaches were employed to different degrees through the grants evaluated. Clearly, there were advantages and disadvantages associated with all of the options mentioned. 33 Note to PVC: To maximize the effectiveness of its investments in NGO strengthening, it will be important for PVC to develop priorities for the types of results it wishes to support and to announce these priorities to the PVO community so that future proposals include explanations of how NGO strengthening will be addressed and monitored, and the expected results. 34 PVC MATCHING GRANT EVALUATIONS ON WHICH THIS SYNTHESIS IS BASED Organization Title of Grant Grant No. CARE Partnership and Livelihood Security FAO-0158-A-00-6051-00 Enterprise Development International Capacity Building for Micro-enterprise Development in Mexico, Romania and U.S. Headquarters FAO-A-00-99-00055-00 Food for the Hungry Faulu Africa Regional Micro-Enterprise Loan Program FAO-0158-A-00-5011-00 Freedom from Hunger Improving Maternal and Child Health Impacts of Credit with Education FAO-A-00-99-00046-00 International Medical Services for Health Corporate Community Investment Service (CorCom) FAO-A-00-98-0071-00 PLAN International USA (Childreach) Institutional Strengthening of Credit and Microenterprise Programming FAO-0158-A-00-6047-00 Save the Children-USA Woman/Child Impact-II FAO-A-00-96-00003 Small Enterprise Education and Promotion (SEEP) Network Institutional Development Services for Enterprise Development Organizations FAO-A-00-98-00049-00 TechnoServe, Inc. Promoting Sustainable Rural Growth FAO-0158-A-00-6045-00 The Nature Conservancy Building Constituencies for Protected Areas FAO-A-00-97-00061-00