The International Eye Foundation Seeing 2000 Program: Expansion of Clinical and Surgical Eye Care Services for Children Through Support for NGOs Worldwide Final Evaluation December 2000 Project No. 938-0158 Cooperative Agreement No. FAO-A-00-95-00015-00 This report was prepared by AMA Technologies, Inc under contract number FAO-C-00- 95-00-00036 with the Office of Private and Voluntary Cooperation Bureau for Humanitarian Response, USAID Evaluation Team Leader: Roy Jacobstein, M.D., M.P.H. Evaluation Team Member: T. Otis Paul, M.D. ACKNOWLEDGEMENTS The evaluation team thanks the staffs of the International Eye Foundation and its grantee partner organizations with whom the team met (Aravind Eye Hospitals, Madurai, India; L. V. Prasad Eye Institute, Hyderabad, India; Lumbini Rana-Ambika Eye Hospital, Bhairahawa, Nepal; Tilganga Eye Centre, Kathmandu, Nepal; Bulgarian Eye Foundation, Sofia, Bulgaria; Hospital Elias Santana, Santo Domingo, Dominican Republic; Instituto de Educacion y Prevencion en Salud Visual, Lima, Peru) for their generosity and responsiveness to us, and more importantly, for their commitment to bettering the vision, and thus the lives, of the people in the communities they serve, especially children. TABLE OF CONTENTS List of Acronyms.................................................................................................................. i Executive Summary ............................................................................................................ii 1. INTRODUCTION AND BACKGROUND............................................................ 1 Brief Description of the International Eye Foundation .......................................... 1 Purpose and Context of the Matching Grant........................................................... 1 History........................................................................................................ 1 PVC Goals, Strategic Objective and Intermediate Results ......................... 3 Purpose of the Evaluation ....................................................................................... 4 Methodology ........................................................................................................... 4 2. THE SEEING 2000 PROGRAM: FINDINGS ....................................................... 6 Rationale.................................................................................................................. 6 Strategy ................................................................................................................... 6 Major Objectives..................................................................................................... 7 Dimensions.............................................................................................................. 7 Financial...................................................................................................... 7 Programmatic .............................................................................................. 7 Management................................................................................................ 8 Program Results ...................................................................................................... 9 Service Delivery.......................................................................................... 9 Capacity..................................................................................................... 10 3. CONCLUSIONS................................................................................................... 11 A. PROGRAM IMPLEMENTATION ................................................................. 11 Progress Toward Achievement of Major Objectives ............................................ 11 Service Delivery........................................................................................ 11 Capacity..................................................................................................... 12 Constraints................................................................................................. 13 Major Successes ........................................................................................ 15 Effectiveness of Models and Approaches ................................................. 16 Impact on Local Partners....................................................................................... 18 Progress Toward Sustainability of Subgrantees.................................................... 19 Status of Strategic Partnerships............................................................................. 21 B. PROGRAM MANAGEMENT......................................................................... 23 Change in IEF Management Capacity .................................................................. 23 Monitoring and Evaluation.................................................................................... 24 Financial Management .......................................................................................... 25 Information and Results Reporting ....................................................................... 26 Other Management Functions ............................................................................... 26 4. RECOMMENDATIONS ...................................................................................... 28 5. LESSONS LEARNED.......................................................................................... 32 6. UNRESOLVED ISSUES...................................................................................... 33 ANNEXES A. The Scope of Work for the Evaluation B. Evaluation Team Itinerary C. List of Persons Contacted D. References Consulted E. Tables and Charts 1. Partner Organizations, Grant Years and Amounts 2. HQ and Subgrant Expenditure by Categories 3. Results: Program Objective 1, Surgeries in Children 4. Results: Program Objective 2, Eye Exams in Children 5. Results: Program Objective 3, Training i LIST OF ACRONYMS AAO American Academy of Ophthalmology BHR Bureau for Humanitarian Response DIP Detailed Implementation Plan DOSA Discussion-Oriented Organizational Self-Assessment FY Fiscal Year HQ Headquarters IAPB International Agency for the Prevention of Blindness IEF International Eye Foundation IEPSV Instituto de Educacion y Prevencion en Salud Visual, Lima, Peru INGO International Non-governmental Organization IR Intermediate Results LAICO Lions Aravind Institute of Community Ophthalmology LVPEI L. V. Prasad Eye Institute, Hyderabad, India M&E Monitoring and Evaluation MG Matching Grant Program (of BHR/PVC) MIS Management Information System NGO Non-governmental Organization PVC Office of Private and Voluntary Cooperation PVO Private Voluntary Organization RFA Request for Applications ROP Retinopathy of Prematurity SO Strategic Objective SOW Scope of Work TA Technical Assistance USAID United States Agency for International Development WHO World Health Organization ii EXECUTIVE SUMMARY Pursuant to a continuing Congressional earmark, the Matching Grants Program of the Bureau for Humanitarian Response, Office of Private and Voluntary Cooperation (BHR/PVC) entered into a cooperative agreement with the International Eye Foundation (IEF) in 1995 to support Seeing 2000, a program of assistance in the area of surgery and related interventions for childhood blindness. PVC and IEF have a long-standing relationship, dating back to 1978. The Seeing 2000 Program was the second PVC-funded IEF program related to the earmark. A third, Seeing 2000: Revised and Expanded, is funded from 1999-2004. This is the report of the external final evaluation of the Seeing 2000 Program. (The program received a no-cost extension and is scheduled to run until February 2002.) The evaluation team was comprised of a pediatrician/public health physician who was a USAID employee for 13 years in the PHN Center of the Global Bureau, and a practicing pediatric ophthalmologist with wide international experience. FINDINGS AND CONCLUSIONS 1. Financial Dimensions • $1,571,990 aggregate funding from the Matching Grant Program. • No required match. • All field activities funded by small grants (up to $25,000 for 1-2 years) to local NGO partners, via an RFA process. • 57% of funds went to NGOs for grants; 43% to IEF HQ for management. • $943,000 was obligated for small grants; 61% of that was disbursed by 11/00. 2. Programmatic Dimensions of the Seeing 2000 Program • Grants were primarily for equipment (53%), staff (21%) and training (9%). • Grants were made in 14 countries, across all regions where USAID works. • Through 7/2000, 27 grants went to 22 partner institutions. Twenty-one of the 27 grants were made in 1996 and 1997, i.e., most of the small grant activity was early in the program. (See Annex E, Table 1.) • NGO partners ranged from mature, well-managed, well-staffed, large-volume institutions to smaller, more fragile institutions and programs. 3. Management Dimensions of the Seeing 2000 Program • Four IEF technical staff at HQ were supported: three core senior staff partially funded (5-15%) and the program coordinator fully funded. • A 1998 DOSA led IEF in 1999 to adopt its first overall strategic plan. This plan calls for IEF to give larger grants to fewer organizations and emphasizes IEF’s provision of technical assistance to enhance grantee capacity, management, and cost-recovery/financial sustainability. iii 4. Major Objectives of the Seeing 2000 Program Primary Objectives 1. Increase by 20% or more the number of children receiving needed surgery for correctable ocular conditions in the area being served. 2. Increase by 50% or more the number of children under age 6 receiving eye examinations in the areas being served by a project. 3. Identify at least one ophthalmologist or clinical officer within each project area and enhance their capacity to treat children clinically and surgically through additional training. Secondary Objectives 4. Increase by 10% or more the number of visually impaired children enrolled in blind schools who can be visually rehabilitated (spectacles, low vision aids, etc.) and integrated into a regular environment/school. 5. Support existing efforts to increase awareness of ocular disease in children, its management and referral, in the medical communities of regions being served. 6. Support existing efforts to increase awareness of ocular disease in children among parents and the general public. 5. IEF and Grantee Performance on Seeing 2000 Service Delivery/Training Objectives In the aggregate, IEF met all three primary objectives (See Annex E, Tables 3 and 4). IEF did not report on secondary objectives. With respect to surgery and eye exams there were aggregate increases of 63 % in surgery for children (5,741 operations) and 978% in eye exams for children (509,017 eye exams). With respect to capacity building (training), IEF generally met this objective, with almost all partner organizations that had training objectives meeting them, and/or themselves conducting training. In toto, 92 doctors and 121 other health providers were reported trained in the Seeing 2000 Program. Data were not collected systematically by IEF, but the team had the sense that almost all Seeing 2000 trainees stayed at their institutions and subsequently provided services. Most of IEF’s grantee partners were strong general eye care organizations already. They are well-managed, deliver high-quality, high-volume services, and have a moderate to high degree of cost-recovery, as well as relationships with other international NGOs. Thus individual grantees’ performance on these primary objectives for children’s services was generally very high. Twelve of the 14 partner institutions reporting on surgery met this objective, with some institutions reporting striking increases of 250% or more (range –32% to 276%). Similarly all but two of 15 partner institutions that reported on the eye exam objective met it (range –46% to 1148%). Finally, in the iv seven institutions visited by the evaluation team, quality of surgical service was judged by the team to be generally high. Results in numerical terms were not evenly distributed, however, but rather were skewed to those regions where the underlying disease burden is greatest. Three grantees––Aravind Eye Hospitals, Madurai, India, L.V. Prasad Eye Institute, Hyderabad, India, and Lady Reading Hospital, Peshawar, Pakistan––contributed 59% of overall surgery. Similarly, two institutions, Aravind and its “sister,” Lumbini Eye Hospital, Bhairahawa, Nepal, together performed 86% of the increased eye exams (437,299 of the increase of 509,017 children examined) during the project period, with Lumbini alone accounting for over 75% of the total. Other caveats include that data were not disaggregated by type of surgery, age or diagnosis (all of which have implications for visual outcome), nor was follow-up to determine quality of post-operative vision reported on. IEF’s tripling of its surgery objective and 20-fold rise in eye exams suggests those service objectives were too modest, as is one person trained per organization as a training target. Finally––and regrettably and unavoidably––Seeing 2000’s contribution to overall childhood blindness prevention is small (5,741 surgeries, 1.5 million blind children worldwide, 450,000- 750,000 non-Vitamin A-related, the focus of Seeing 2000.). 6. Successes of the Seeing 2000 Program Partnership Seeing 2000 had successes beyond its narrower service delivery and training achievements, particularly keeping in mind that IEF is a small organization (8-10 employees, overall annual budgets in range of $3-4 million) and Seeing 2000 was a small program (averaging $300,000 per year). The small grants program was an innovative, cost-effective, and successful way to increase surgical services for childhood blindness. A great deal of activity at many partner institutions was generated by a relatively small infusion of IEF funds. A number of the grant partners are true “centers of excellence” and almost all are continuing to provide increased services for children (relative to baselines). IEF fashioned a wide geographic network of committed, capable service organizations, which at international policy/program levels (e.g., WHO, IAPB) raised the profile of childhood blindness as a separate entity requiring separate services. IEF’s small grant program was successful for a number of reasons. The network was self-selected, so “good” organizations, like cream, rose to the top in the RFA process. Some of the partners, local NGOs, were very sophisticated (and had larger annual budgets than IEF itself). These mature partners became valuable training resources and offered varying useful models for IEF to learn from and draw upon. IEF was also able to foster South-South transfer of capability by partnering more nascent NGOs with more robust ones. There was also a sense of ownership within grantee partners because they had themselves identified needs, designed and implemented v activities, managed program resources, and tracked program results––all useful skills for organizational thrival. In short, this network confirmed PVC’s sense that “networks are important mechanisms to leverage resources and amplify influence of PVOs”. From the standpoint of PVO-NGO partnership, the Seeing 2000 Program was highly participatory and there was no micro-management from IEF, which was seen as a reliable and helpful partner, without any conflicting agendas of its own. All seven of the local partners visited by the evaluation team expressed high levels of satisfaction with the partnership with IEF, with the timeliness and utility of the IEF assistance given, and with what they, the grantees, accomplished with the grant funds. They all remarked that the Seeing 2000 grant enabled them to focus their institution on the needs of children, and to maintain that focus post-grant. Another noteworthy partnership aspect is that Aravind has not only become a technical resource partner for IEF as a site of South-South training, a model of service delivery, and a source of management technical assistance, it has also become a business partner. IEF’s own “social enterprise,” a component of Seeing 2000: Revised and Expanded, depends on Aravind’s products, which IEF then resells in other countries, in effect acting as a middleman. Because Aravind manufactures intraocular lenses and suture material of high quality at low prices, both IEF and Aravind make profits in this enterprise. In turn they use the funds to further their mission––in Aravind’s case, to cross￾subsidize services to the poor, in IEF’s to enlarge its program and diversify its funding. Finally, at the international and policy level, IEF has a “seat at the table.” It is a member of key blindness prevention coalitions. For example, it is a supporting member of Vision 2020: The Right to Sight, the WHO-led global initiative to eradicate avoidable blindness by the year 2020. IEF also co-sponsored the IAPB’s 1999 Scientific Meeting on the Prevention of Blindness (at which 9 of the 18 eye care organizations and 18 of the 30 individuals, were IEF partners in the Seeing 2000 Program). Capacity and Programmatic and Financial Sustainability, Post-grant These were not addressed systematically by IEF or its grantees. Nonetheless the evaluation team had the sense that prospects were good at many grantee institutions. For example, of the 14 institutions that reported increased surgery during the grant period, 12 are still providing surgical and other services for children at rates higher than baseline. Lumbini, which used grant funds to hire 12 outreach personnel, not only dramatically increased its outreach services in a sustained fashion, but was able to retain four with its own funds. IEPSV’s Seeing 2000 work attracted additional INGO support from Christoffel BlindenMission/Germany, and Mt. Sion Hospital in Papua New Guinea received $100,000 from New Zealand because of its Seeing 2000 Program. Five institutions have significant fee-for-service and are able to cross-subsidize over 50% of the services to children and the poor. Both Aravind and Tilganga are large volume manufacturers of products such as intraocular lenses and sutures. Partners also raise funds from private donors––L.V. Prasad Eye Institute has raised upwards of $10 million from vi private and corporate donors in its 14-year existence––and almost all IEF partners get support from other INGOs. 7. Areas That Need Improvement in Seeing 2000 Program Monitoring and Evaluation This area was generally weak in Seeing 2000. Weaknesses included a lack of a program MIS to consolidate, analyze and interpret data, and a lack of an overall M&E plan. Evaluation, to the extent that it happened, was largely retrospective, and indicators used measured service and training outputs not capacity or sustainability outcomes. Larger service-level and institution-level objectives, e.g., an institution’s capacity to provide quality, quantity and affordable services, or its degree of cost-recovery and self￾sufficiency, were not routinely monitored or evaluated by IEF, although IEF staff seemed generally aware of their status at partner institutions. Finally, post-grant assessment of grantees was not done in any systematic and/or written way. Information and Results Reporting As with the related area of M&E, IEF’s information generation and results reporting was generally weak. Annual reports to USAID have been of little usefulness because of timing (a year or more late) and format––long compendia of activities, not syntheses of individual program data or analyses of meaningful achievements. IEF indicators and results have not been linked to PVC’s SO and IRs. In particular, IEF has not reported on its own or its grantees’ enhanced capacity or sustainability (programmatic, technical and/or financial), nor has it used qualitative reporting, e.g., “vignettes” or “case studies”, which are also useful to PVC in its R4 process. Staffing IEF seems thinly staffed for the technical assistance mission it has embraced. MAJOR RECOMMENDATIONS Major Recommendations for IEF 1. IEF should continue to follow its new strategy, concentrating on increasing and diversifying its non-USAID funding, and management and financial sustainability. 2. IEF should continue to make fewer but larger grants to well-chosen partners. Grants should be for sustainability (not for, say, recurrent costs), and, where appropriate, for longer than 12-24 months. Also, the continuing existence and availability of the small grants program, in this new incarnation, should be made known in the international eye care community. vii 3. Criteria guiding partner selection should be written and should be a subject, if not a product, of dialogue with PVC. 4. IEF staff should––and will need to––provide significantly more technical assistance to its grantees in program management and service delivery than it has in the past. 5. Numerical goals that are used in the program should be more ambitious (while still realistic) and set in absolute terms as well as in percentage increases. 6. IEF should strengthen its M&E capability, including having its staff trained in the latest, field-useful, practical approaches to monitoring and evaluation. M&E should then be an integrated part of every technical staff person’s responsibilities. 7. IEF should have an overall M&E plan that articulates responsibilities of both IEF and local partner staff in the M&E process, data to be collected, timetables for collection, and ways that data will be used to inform subsequent programming. That is, M&E must become a management tool, an integral part of design and implementation, and prospective not retrospective, i.e., present from the start. 8. This attention to overall M&E, improved data generation and stronger outcome indicators needs to be immediate, with appropriate modifications made to existing sub-grants, and technical assistance in this area given to reporting institutions. 9. Useful, measurable, operationally defined and mutually understood benchmarks and indicators, related to capacity/sustainability, need to be part of agreements between IEF and partner grantees, and need to be tracked by IEF and regularly reported on. 10. As part of its management TA, IEF should provide a “results template” to its partners upon which these disaggregated and meaningful service delivery, capacity and sustainability results can be reported. 11. IEF should support the conduct of special outcome studies where needed. These studies would answer important “so what” questions related to key issues such as quality, cost (e.g., institutional strategies and experience with income generation and cross-subsidization), and partnership. (See p. 30 for fuller discussion.) 12. IEF needs to significantly strengthen the whole area of its results reporting. Results reporting, from local grantee to IEF and from IEF to PVC, needs to include baselines as well as ongoing, post-grant achievements and other measures of continuity of activities (indicators of programmatic sustainability). 13. IEF should routinely make post-grant organizational assessment reports of each grantee’s program, speaking to changes in service quantity and quality, increased capacity and other meaningful changes within the organization. As much as possible, viii such data needs to be comparable––same indicators, same time periods for baseline and for program activity (e.g., all information annualized). 14. IEF’s Annual Reports to USAID need to be syntheses of individual program data and accomplishments, and analyses (i.e., answering the “so what?” questions) rather than compendia of activity reports. Such reports need to speak to IEF’s contribution, in both qualitative and quantitative terms, to PVC’s SO and IRs. Major Recommendations for IEF-PVC Relationship 14. PVC should have more dialogue with IEF and greater involvement in discussing unresolved issues and/or strategic considerations. For example: • Should IEF’s programs emphasize geographic reach (i.e., a widespread network of partners) or absolute disease burden? • What should the criteria for working with a given institution be? • To what extent should PVC provide initial programmatic direction? • Should IEF focus on retinopathy of prematurity (many fewer patients served, in more-developed countries, but certain, lifelong gains) or more common problems needing lower-tech interventions (spectacles, low-vision devices)? • From the standpoint of IEF’s new strategic directions, will larger grants to fewer organizations, with a focus on management and cost-recovery “work,” i.e., lead to more, better, and sustained services? • Specifically, is the Aravind/Lumbini model of high quality, high volume, sustainable, cross-subsidized services, driven by “compassionate capitalism,” transferable/replicable in other regions? • Finally, are the numbers and skills of IEF’s staff, even with its recent augmentation, adequate to its reoriented strategy and new, more labor-intense modus operandi, with its emphasis on management TA and improved M&E? 15. In addition to the DIP Review, PVC should consider specifically approving the individual countries and institutions where the (fewer, larger) IEF grants will be given under Seeing 2000: Revised and Expanded. Such a process would promote PVC involvement and ensure that IEC was working in priority countries for PVC. 16. PVC should encourage IEF to interpret the earmark more broadly so that IEF can then make low vision services and spectacles part of its program. Such guidance would be according to the earmark’s spirit, to help restore vision to blind children, (rather than to its letter, increasing childhood surgery per se––which too often does ix not restore vision). Many children in schools for the blind could be mainstreamed with low-cost low-vision services. In this way, more children are likely to have meaningful vision restored or preserved for them, in a cost-effective manner. 17. PVC should engage in dialogue with IEF about which special outcome studies would be most useful to PVC, and then encourage IEF to conduct them. LESSONS LEARNED IEF’s implementation of its Seeing 2000 Program demonstrated a number of important points from the standpoint of development: 1. Overall a relatively modest level of funding to a PVO (in the $300-400,000/yr range) can be very effective and can result in measurable, meaningful achievements. 2. Small grants from the PVC-funded PVO to its NGO partners in developing countries can be quite an efficient and cost-effective mechanism for building networks, increasing leverage and adding capacity. 3. If the partners are “well-selected” (i.e., if they have strong services and management), and the program is relatively narrow and well defined, the management burden on the PVO granting the funds can be relatively low. 4. Nonetheless, larger grants to fewer organizations, if carefully targeted and designed, can potentially bring greater results in the long run in terms of strengthening an institution’s program management and financial sustainability. 5. Cost-recovery for affordable and accessible curative surgery can be achieved. In those situations involving high caseloads and empowered, committed organizations, cross-subsidization of services for children and/or the poor is quite feasible. For such “compassionate capitalism” to work, the right set of circumstances are needed, including: • a high prevalence of a condition (such as blindness). • people’s general willingness to pay for care for that condition. • strong organizational commitment and ability to deliver good quality services (a situation more likely in the private NGO sector than in the public sector). • high aggregate national wealth (e.g., India is a very wealthy country, although it has hundreds of millions of poor people). 6. South-South assistance can be real, significant and sustainable. In fact, South-North assistance can take place and meaningfully strengthen a grantor PVO’s own programs. 1 1. INTRODUCTION AND BACKGROUND 1.1 Brief Description of the International Eye Foundation (IEF) The International Eye Foundation is a small U.S.-based NGO with headquarters in Bethesda, MD. IEF was founded in 1961 and began receiving USAID funds in the early 1970s. Its mission is to “help people see” and its overall programmatic emphases include: • Expanding eye care services for those in need. • Prevention of avoidable blindness (cataract, trachoma, river blindness, childhood blindness). • Providing affordable ophthalmic supplies, equipment and medicines. • Enhancing eye care providers’ managerial capacity and financial self￾sufficiency to offer quality eye care services. Currently IEF has field offices in Bolivia, Cameroon, and Malawi as well as (unsalaried) country representatives in Albania, Bulgaria, Ethiopia, Guatemala, and Honduras. Overall, IEF worked in 21 developing countries at some point during 1995- 2000, the time Seeing 2000, the current program being evaluated, has been in effect. From the standpoint of staff and budget, during that same time period IEF had: • 8-10 employees at headquarters, 5-6 of whom have technical and/or programmatic backgrounds in public health and/or eye care services. • Annual overall expenditures of $3.4-4.3 million, with annual program service expenditures ranging from $2.9-3.8 million. These totals include in￾kind contributions of $1-2 million. Of this total program expenditure: • 37-54% represented the annual share of donated medical supplies. • 52-68% went annually to Malawi alone. • In FY99, 79% of total program expenditure went to two countries, Malawi and Cameroon (68% and 11% respectively). • Support from various units of USAID that in the aggregate ranged from $1.0- 1.5 million annually, representing 26-38% of IEF’s total annual revenues (including in-kind revenues). • Excluding in-kind revenue, 60% of IEF’s revenue is provided by USAID. 1.2 Purpose and Context of the Matching Grant 1.2.1 History IEF first received funding from the USAID 1976 and continued to receive funds from various USAID sources during the 1980s, including two matching grants from PVC, 2 one from 1981-1985 for $900,000 and one from 1985-1988 for $850,000. Typically these USAID funds went for various blindness prevention programs, such as Vitamin A promotion and distribution/child survival, onchocerciasis prevention and treatment, and clinical/surgical eye care programs. In its 1991 authorization of funds to USAID, Congress directed that “not less than $1 million shall be made available for private voluntary organizations to be used to finance operations for blind children”. This earmark has continued each year since, up to the present, at the same level of funding. The mandate was broadened somewhat for the 1999 Foreign Relations Appropriation Bill: “… at least $1,000,000 to assist children who are blind. Many blind children in developing countries can be cured of their disability through simple operations and inexpensive care.” In accordance with and as a portion of that ongoing earmark, the Matching Grants Program of PVC made funds available to IEF, from 1993 to the present, via three consecutive (and sometimes overlapping) cooperative agreements. These agreements supported the following three IEF programs: 1. SightReach: Eye Care for the Underserved and Children, 8/93-8/98. This program received $700,000 from the Matching Grant Program of PVC (of which $350,000 was funding from the newly authorized Congressional earmark for childhood blindness) and required a 100% match with non-U.S. Government funds. The program was never formally evaluated by USAID, although IEF itself conducted an external evaluation in Guatemala and Honduras as part of its October 1998 final report to USAID. The program had two components. One (“ResPack”) made equipment available to individual ophthalmology residents who agreed to work in underserved areas of 4 countries (Guatemala, Honduras, Ecuador and El Salvador. The other (“ChildSight,”) worked in Guatemala, Honduras, Albania, Bulgaria, Malawi and Eritrea, helping to conduct surveys, train eye care providers and develop referral networks. 2. Seeing 2000: Expansion of Clinical and Surgical Eye Care Services for Children through Support for NGOs Worldwide, 8/95-2/02. This program is the subject of this evaluation report, and is more fully described in Sections 2 and 3. Key aspects of the program include: • $1,571,990 aggregate funding from the Matching Grant Program. • No required match. • Provision of small grants (up to $25,000 for 1-2 years, with the possibility of a second grant) to applicant eye care institutions. • An 18-month no-cost extension, granted at the end of FY 2000 to allow activities to continue to be funded, at possibly higher levels, through 2/02. 3 3. Seeing 2000: Revised and Expanded, 9/99-9/04 This program is funded by the third cooperative agreement of the 1990s between the Matching Grants Program and IEF. It is for $2 million, requires a $725,000 match, and broadens the nature of the Seeing 2000 Program. Half of the money relates to the Congressional earmark (i.e., focus on children, and on sight restoration) and half does not (i.e., can be for adults and for a broader range of interventions). Beneath IEF’s overall programmatic rubric of “SightReach,” this program has three components: • Institutional support grants, now ranging from $25,000 to $100,000, to expand pediatric medical and surgical eye care services. This represents a continuation of the Seeing 2000 small grant program, and is part of IEF’s “SightReach Prevention” program. Fundamentally, the program objectives and reporting requirements and indicators did not change from Seeing 2000 to this program. • Technical assistance and training to improve grantee institutional management and financial sustainability (“SightReach Management”). The first three countries to be worked with are Malawi, Egypt and Guatemala. • “SightReach Surgical,” a social enterprise, whereby IEF will sell high quality ophthalmic supplies and equipment to providers at reduced cost, thus lowering the cost of service provision while ensuring the poor receive quality care. 1.2.2 PVC Goals, Strategic Objectives and Intermediate Results Funds made available to IEF are meant to contribute to the PVC Office’s five intermediate results (IRs) and to its one crosscutting strategic objective (SO), Increased Capability of PVC’s PVO Partners to Achieve Sustainable Service Delivery. The rationale for this SO, according to PVC’s 1996 Strategic Plan, is that “investments in capacity building to improve the planning and management systems of PVOs, as well as those of their local NGO partners, enhance the replicability and ‘scaling up’ of successful sustainable development programs initiated at the grassroots level”. Achievement of this SO in turn would contribute to BHR’s sub-goal of NGOs and Other Local Partners Strengthened and its goal of Sustainable Development Impacts in Priority Sectors. PVC’s five IRs, the achievement of which leads to achieving its strategic objective, are all germane to IEF’s work under the Seeing 2000 Program (to varying degrees: chiefly IRs 1-4, with IR 5 largely PVC’s domain, although IEF does raise public awareness at professional meetings and via newsletters). The five IRs are: • IR1: Operational and Technical Capacity of US PVOs Improved • IR2: Strengthened Partnership Between USAID and US PVOs • IR3: Strengthened US PVO and Local NGO Partnership • IR 4: Improved Mobilization of Resources by PVC’s PVO Partners • IR 5: U.S. Public Awareness Raised 4 Capability (or capacity) enhancement, increased service delivery, and sustainability of services post-assistance are thus central mandates for PVC-funded PVO partners. PVC also emphasizes: innovation, particularly the funding of creative approaches with widespread applicability, and especially those that enhance the financial survivability of PVOs and their NGO partners; increased credibility, visibility and participation of PVOs in international fora; and use of assessment tools and technical standards that increase service quality. 1.3 Purpose of the Final Evaluation This external final evaluation is meant to be of value to both PVC and IEF, and to further collaboration between them. From the perspective of PVC, the evaluation is meant to document results, particularly to address IEF’s contribution, via its Seeing 2000 Program, to PVC’s achievement of its IRs and SO. PVC will use the information in its annual results report, in ongoing reviews of IEF’s program, and in distilling lessons learned for broader application. After participatory discussions with Matching Grant Program project managers and IEF leadership, the evaluation team decided the to focus its analysis on both IEF’s program implementation (60%) and its program management (40%), particularly as they relate to the key considerations articulated in PVC’s SO: service delivery, capability (or capacity), and sustainability. Achievements in these two categories and three program dimensions are presented in Sections 2.4 and 2.5 and Section 3 of this report. (See Annex A for the Evaluation Scope of Work. Section 3––Conclusions–– answers the questions of this SOW, in a question-by-question format.) From the standpoint of IEF, the findings and recommendations of the evaluation team will be used to refine Seeing 2000: Revised and Expanded. IEF is particularly interested in evaluating its small grants mechanism, and has a number of questions relating to the efficacy of that small grants program and IEF’s own management and structuring of the program. IEF is also interested in an assessment of its local partners’ satisfaction with the program and of the extent to which service delivery improved and capacity expanded because of it. For both organizations this evaluation fulfills the requirements of the USAID/BHR/PVC Matching Grant Program. 1.4 Methodology The evaluation team was officially comprised of two persons with the following background and expertise: • Roy Jacobstein, M.D., M.P.H., a pediatrician and public health physician, and formerly Chief (from 1988-1999) of the Communication, Management and Training Division of the Global Bureau’s Office of Population, USAID, Washington. Dr. Jacobstein also participated in May 2000 in the DIP Review of the Seeing 2000, Phase Two follow-on Matching Grant. He served as team leader for the evaluation. 5 • T. Otis Paul, M.D., a practicing pediatric ophthalmologist, with experience in a broad range of developing countries in Africa, Asia, Latin America, and Eastern Europe. Dr. Paul is a Fellow of the American Association for Pediatric Ophthalmology and Strabismus, and of the American Academy of Ophthalmology. He was hired by IEF to participate in this evaluation, but was not earlier an IEF consultant. • Additionally, Lori Carruthers, M.P.H., IEF’s Seeing 2000 Program Coordinator since its inception, was an unofficial member of the team. She accompanied the team on all its field visits and provided additional information and perspective. The team reviewed numerous USAID, IEF, and partner NGO documents, as well as technical papers (See Annex D). A Team Planning Meeting in Washington was conducted on Oct. 26, 2000, at which both IEF and PVC participated and the Evaluation Scope of Work was finalized (See Annex A). The team then conducted key informant interviews with IEF HQ personnel and PVC project managers. A questionnaire was developed by the team to elicit input from partners who were not going to be visited; five of the 15 non-visited institutions responded. In the field the team met with IEF’s local partner NGOs and observed eye care service delivery at all the institutions visited (See Annexes B and C). Sites visited are all fairly long-standing and ongoing partners of IEF. The team also debriefed PHN personnel of USAID/India on its findings in India and Nepal. The team went together to Aravind Eye Hospitals in Madurai, India and L. V. Prasad Eye Institute in Hyderabad, India. Then Dr. Jacobstein and Lori Carruthers visited Lumbini Eye Hospital in Bhairahawa, Nepal, Tilganga Eye Centre in Kathmandu, Nepal, and the Pashev Center for Sight in Sofia, Bulgaria. Subsequently Dr. Paul visited Hospital Elias Santana in Santo Domingo, Dominican Republic, and the Instituto de Educacion y Prevencion en Salud Visual, Lima, Peru. Collectively, the team visited 35% (7 of 20) of the institutions that received grants from IEF and completed project activities. This represents 50% (7 of 14) of the countries where the Seeing 2000 Program was active. Regarding the remaining content and organization of this evaluation report: • Section 2 presents facts: Seeing 2000’s rationale, strategy and major objectives, its financial, programmatic and management dimensions, and its program results. • Section 3 assesses and judges those facts, and tracks the evaluation scope of work questions. Like the SOW, Section 3 is divided into two sections: Program Implementation and Program Management. Per instructions from PVC, the section on Program Management focuses primarily on three aspects: 1) changes in IEF’s management capacity as a result of the grant from PVC; 2) IEF’s monitoring and evaluation of the Seeing 2000 Program; and, 3) IEF’s information reporting to PVC. • Section 4 presents the team’s Recommendations. • Section 5 presents Lessons Learned that have wider program applicability. • Section 6 presents Unresolved Issues for further dialogue between IEF and PVC. 6 2. THE SEEING 2000 PROGRAM: FINDINGS (Empirical Facts) 2. 1 Rationale IEF’s Seeing 2000 Program was and is predicated upon the following facts: • There was an underlying Congressional mandate to address childhood blindness, specifically, to increase the quantity and quality of infant and child surgical services. • There are an estimated 1.5 million blind children worldwide, of whom 1 million are in Asia and 300,000 in Africa. (30-50% of cases, or 450,000-750,000, are due to non-Vitamin A Deficiency causes.) • Surgically correctable childhood blindness has multiple causes. Most cases occur infrequently, unpredictably and sporadically. To preserve vision, surgery generally must be done before age 6, and such surgery typically requires general anesthesia. • Eye care and anesthesia providers are in short supply in many developing countries. Of these providers, relatively few have skills in pediatric ophthalmology. • IEF is a small organization with relatively few staff. • In terms of mounting a major effort, the funds provided by PVC were relatively modest––averaging between $300,000 and $400,000 annually. • Until 1998, USAID required all subgrants above $25,000 to conduct outside audits that could cost the grantee up to $5,000. • There is an existing network of INGO’s and local NGOs worldwide delivering eye care services to a significant extent, and these organizations are known to each other. 2.2 Strategy Given the above set of facts, IEF’s strategy was to provide small grants. This might allow a small amount of IEF human and financial resources to leverage a great deal of increased services and capacity, by acting as a catalyst within existing, capable, eye care institutions and organizations. A small grant mechanism also might allow IEF to efficiently expand its own involvement with a network of local and international NGO partners. Finally, fewer “eggs” would be in any given “basket,” so poor performance in one or two sub-grants might not be as deleterious to overall program performance. The small grants program would make available up to $25,000 for a period of 12-24 months. It would focus on provision of needed surgical and/or anesthesia equipment for children, training of eye care providers in pediatric eye care service delivery, especially 7 surgery, and outreach, screening, referral, and education of providers, parents and communities, to increase the number of blind children being reached and served. 2.3 Major Objectives These were the objectives agreed upon between IEF and PVC. In turn, to receive a small grant, prospective local grantee partner organizations had to propose two of these three primary objectives that they would achieve. These objectives were: Primary Objectives: 1. Increase by 20% or more the number of children receiving needed surgery for correctable ocular conditions in the area being served. 2. Increase by 50% or more the number of children under age 6 receiving eye examinations in the areas being served by a project. 3. Identify at least one ophthalmologist or clinical officer within each project area and enhance their capacity to treat children clinically and surgically through additional training. Secondary Objectives: 4. Increase by 10% or more the number of visually impaired children enrolled in blind schools who can be visually rehabilitated (spectacles, low vision aids, etc.) and integrated into a regular environment/school. 5. Support existing efforts to increase awareness of ocular disease in children, its management and referral, in the medical communities of regions being served. 6. Support existing efforts to increase awareness of ocular disease in children among parents and the general public. 2.4 Dimensions 2.4.1 Financial (See Annex E, Table 2) • A total of $1,571,990 received by IEF from BHR/PVC/MG. • $943,000 obligated for small grants, 61% of that, $572,500, disbursed by 11/00. • 57% of IEF’s Seeing 2000 funds went to small grants to local NGO partners • 43% went to IEF headquarters for management of the grants and related activities. • Of the projected total of 35 small grants, all but 5 will have been for $25,000 or less. 2.4.2 Programmatic • All field activities funded by small grants. • Grants made in 14 countries, distributed across all regions where USAID works. • Through July 2000, 27 grants made to 22 partner institutions. (See Annex E, Table 1.) 8 • 21 of 27 grants made in 1996 and 1997, 1 made in 1998, 2 in 1999, 3 in 2000 (with 5 more approved). That is, most of the small grant activity was in the first two years. • Of the overall grant funds, 53% went for equipment and supplies, 21% for staff, 9% for training. • Seeing 2000’s NGO partners ranged from mature, well-managed, well-staffed, large-volume institutions to smaller, more fragile institutions and programs. • One of the mature partners, Aravind Eye Hospitals, became an important venue for South-South training of other IEF grantees, and an important model for IEF. 2.4.3 Management Four IEF technical staff at HQ were partially or fully supported, including the full-time program coordinator. A Review Board, comprised of three experienced pediatric ophthalmologists reviewed proposals and awarded grants. The board met three times in the first two years, and then not after April 1998. IEF participated but did not have an official vote. PVC participated in the first meeting. In the first year 35 proposals were received requesting funding. Program monitoring of grantees was primarily done by the Seeing 2000 program coordinator, who averaged one visit to each grantee partner institution. This interaction was augmented by e-mails, guidance and technical articles sent from HQ, and by international meetings at which IEF and many grantee partners were present. In mid-1998, IEF conducted a Discussion-Oriented Organizational Self￾Assessment (DOSA), which led IEF in 1999 to adopt its first overall strategic plan. This plan guides its strategic reorientation and emphasizes IEF’s own financial security and diversification of funding. According to IEF, this process of reorientation explains the fall-off in small grant awards after the first two years of the Seeing 2000 Program. The ensuing high pipeline led USAID to grant an 18-month no-cost extension until February 2002. At roughly the same time, USAID regulations changed, so that levels above which sub-grants needed an outside audit were raised from $25,000 to over $200,000. IEF has recently decided to give larger grants to fewer organizations, under both the Seeing 2000 Program and the Seeing 2000: Revised and Expanded Program. There is no RFP process or Review Board. Instead, IEF technical staff decides on grantees and levels of funding according to criteria that include the potential partner organization’s overall capabilities and the likelihood that investment in capacity building and financial sustainability will achieve results. The Seeing 2000 Program does not yet have annual workplans, monitoring and evaluation plans, routine organizational assessments of post-grant changes in service quantity and quality, capacity and sustainability of its local NGO grantees, or routine reporting of post-grant activity. Over the span of the Seeing 2000 Program, there have been three successive PVC project officers. The present project officer has backstopped the project since December 1999. Frequency of formal meetings between IEF and its project officers, 9 whether to review results or for other reasons, varied from quarterly to annually. Other meetings with the leadership of the Matching Grants Program and PVC Office also took place in one configuration or another, for various purposes, during the period of project activity. 2.5 Program Results The countries in which Seeing 2000 worked vary widely in socioeconomic, epidemiological, demographic and organizational characteristics. Some countries, such as Chile, Peru, Bulgaria, the Dominican Republic, and Guatemala, have relatively few children and relatively little childhood blindness. Others have many children and much blindness in both adults and children, such as India, Pakistan and Egypt. Some countries where IEF worked are rich in terms of health care resources and aggregate wealth, but have extremely large numbers of poor people and wide variations in individual income levels, e.g., India, Pakistan and Egypt. Others are poor in both health care and aggregate financial resources, e.g., Malawi, Ethiopia, Papua-New Guinea, and Nepal. These underlying characteristics influenced IEF’s achievement of results. 2.5.1 Service Delivery These are the Seeing 2000 Program’s two service delivery objectives: 1. to increase by 20% or more the number of children receiving needed surgery for correctable ocular conditions in the area being served; and, 2. to increase by 50% or more the number of children under age 6 receiving eye examinations in the areas being served by a project. These are the Seeing 2000 Program’s service delivery results: Surgery According to IEF data (See Annex E, Table 3), grantees realized increases from baseline levels of childhood surgery that ranged from –32% to 276%. In the aggregate, there was a 63% increase from baseline, representing an increase of 5,741 operations. The most active grantees were two Indian institutions, Aravind (reporting 3,595 pediatric surgeries, an increase of 1,555, or 76%, in 12 months) and L.V. Prasad (reporting 4,359, an increase of 1,102 surgeries over 24 months). The least active grantees were the LRBT Hospital of Pakistan and Robles Hospital of Guatemala––each of which reported decreases in the numbers of pediatric surgery––IEPSV of Peru, with an increase of 2 cases, from 27 to 29, and Elias Santana Hospital, with an increase of 15 cases, or 14%, from 105 to 120. Eye Exams of Children According to IEF data (See Annex E, Table 4), individual grantees realized increases from baseline levels of eye exams in children that ranged from –46% to 1148%. 10 In the aggregate, there was a 978% increase in eye exams of children representing an overall increase over baselines of 509,017 childhood eye exams at grantee institutions. Among the grantees, the most active institutions were Aravind (86,998 exams, an increase of 52,628 in 12 months) and Lumbini (384,671 eye exams in a two-year period, baseline of one exam reported). Together these two institutions accounted for 86% (437,299 of 509,017) of the reported increase in eye exams in children, with Lumbini alone accounting for over 75% of total program performance on this measure. 2.5.2 Capacity The Seeing 2000 Program has had one objective that relates to capacity: “to identify at least one ophthalmologist or clinical officer within each project area and enhance their capacity to treat children clinically and surgically through additional training.” (Strictly speaking, this objective is a training objective, since the indicator tracked and reported on is number of people trained, not capacity thereby built. It is clear, though, that the intent of the training was to augment institutional capacity to provide pediatric surgical eye care services.) Ten of IEF’s 20 partner organizations that have already implemented a program reported training doctors and/or clinical officers. In addition, the two organizations currently receiving small grant funds also have training providers as objectives. Ninety￾two doctors were trained in all, 51 of them––55% of the total providers trained––at Hospital Rodolfo Robles V in Guatemala. In addition, 121 other health providers were trained at 8 different institutions. (See Annex E, Table 5.) Finally, all but three of the partner organizations conduct their own training (in residency programs and the like) apart from the Seeing 2000 Program. Additionally, 11,408 teachers, students, and parents were “trained” by Hospital Elias Santana in the Dominican Republic. This education objective relates not to Seeing 2000’s Primary Objective #3, but to Secondary Objective #3, “support[ing] existing efforts to increase awareness … among parents and the general public …”. Except for this one (fairly striking) achievement, grantees––and IEF––did not report on their achievement of the three secondary objectives. Beyond the two service delivery objectives and one training objective, Seeing 2000 did not have other quantitatively expressed and measurable capacity objectives, nor did it otherwise systematically document and/or report on other aspects of capacity besides training. It also did not systematically report on sustainability or partnership. These additional objectives were clearly implicit in the Seeing 2000 Program, but nonetheless, it is difficult for the evaluation team to present overall accomplishments in an empirical way in the absence of data and other internal reports addressing enhanced capacity (at both HQ and in grantees) and/or sustainability. In the next section, though, this report does address and assess these additional aspects of capacity, as well as sustainability and partnership. 11 3. CONCLUSIONS (Interpretations and Judgments) On the basis of the set of facts and objective program results presented in Section 2, this section assesses the achievements of IEF and its grantees in the Seeing 2000 Program. This section is structured to closely track the organization and questions of the Evaluation Scope of Work (See Annex A). The questions from the SOW are indicated in italics in this section and are then answered. (Not all of the questions from the SOW are answered, because several were redundant.) Like the SOW, this section is subdivided into two major sections: Program Implementation and Program Management. 3A PROGRAM IMPLEMENTATION SOW Question A1. Assess progress toward each major objective. 3A.1 Progress Toward Achievement of Major Objectives SOW Question A.1.a. Based on the statement of program purpose from the proposal, have the objectives been met? 3A.1.1 Service Delivery (See also Section 2.5.1) IEF had two major service delivery objectives (See Annex E, Tables 3 and 4), one relating to childhood surgery, the other to eye exams. Through its grantees’ performance, IEF met––indeed, exceeded––both objectives. There were aggregate increases of: • 63 % in surgery for children (objective: 20% or more). • 777% in eye exams for children (objective: 50% or more). • 5,741 operations in children. • 509,017 eye exams in children. Individual grantees also generally met these objectives. Twelve of the 14 partner institutions reporting on surgery met this objective, with some institutions reporting striking increases (Tilganga Eye Hospital, Nepal, 250%; Bulgarian Eye Foundation, 276%). Similarly, 7 of 9 partner institutions reporting on the eye exam objective met it, with some institutions reporting increases of more than 200%. Eight other institutions reported increases in exams without giving baselines, so percentage increases are unknown (but likely to be considerable). The most active grantees in surgery were Aravind Eye Hospitals, L.V. Prasad Eye Institute, Lady Reading Hospital (of Peshawar, Pakistan), and Lumbini Eye Hospital (of Bhairhawa, Nepal). These four contributed 85% of overall increased surgery, with Aravind alone contributing 27% of the total. (This figure understates its overall contribution, since it reported for a 12-month period, and the other three were for 24 12 months.) Aravind and its “sister,” Lumbini Eye Hospital in Nepal were the most active in terms of eye exams, together accounting for 82% of the increase in eye exams (437,299 of the 509,917) during the project period. Lumbini alone accounted for over 75% of the total increase in eye exams of children (its figures were for a two-year period). Finally, in the seven institutions visited by the evaluation team, quality of surgical service was judged by the team to be generally high. Several caveats must be kept in mind, however: • Data were not disaggregated by type of surgery, age or diagnosis. This is important because, for example, usually only surgery under age 6 might be restorative of sight, yet the definition of “childhood” is typically 15. Also, surgery for monocular problems usually does not prevent or alleviate blindness, and different diagnoses imply different visual outcomes. • Follow-up to determine quality of post-operative vision was not reported on (for reasons of technical complexity and cost). It is likely that not all of the reported surgery was for correctable blindness (i.e., it may have been on one eye only), and also that not all binocular surgery resulted in correction. The extent of these situations, however, is necessarily uncertain. • IEF’s tripling of its surgical objective and almost 20-fold accomplishment of its eye exam objective suggests the objectives were too modest. Also, the results were skewed, in absolute terms, with only a few institutions producing most of the results. • Finally (and regrettably), the contribution of these activities to the overall prevention of childhood blindness is minuscule. (5,741 added childhood surgeries, the extent to which vision is restored somewhat uncertain, and the worldwide prevalence of childhood blindness is 1,500,000, of which 450,000-750,000 is due to non-vitamin A related causes, the focus of Seeing 2000.). 3A.1.2 Capacity (See also Section 2.5.2) IEF’s only stated capacity objective was “to identify at least one ophthalmologist or clinical officer within each project area and enhance their capacity to treat children clinically and surgically through additional training”. IEF generally met this objective. All partner institutions that had training objectives (10 of the 20) and completed the small grant program reported training doctors and/or clinical officers; the two current small grants also both have training objectives. In toto, 92 doctors and 121 other health providers were trained in the Seeing 2000 Program, with one institution alone, Hospital Rodolfo Robles V of Guatemala accounting for more than half of the total physician training (51 trained; 55% of total). (See Annex E, Table 5.) Data were not collected, but the team had the sense that almost all Seeing 2000 trainees stayed at their institutions and subsequently provided services. In addition, 19 of the 22 institutions conduct some sort of training themselves (e.g., residency programs), apart from Seeing 2000. 13 There are several problems and limitations with this objective and achievement, however. As an objective, training conducted does not equate to capacity built. The number of trainees is an activity/output measure, not an outcome/impact measure. It doesn’t focus on either individual or institutional behavior change. So it doesn’t answer the “so-what” question, i.e., what services are now being provided that weren’t before because of the training? What is different and better now in the institution? But IEF did not report on service activity of trainees post-grant, or, in a qualitative way, on what impact a newly trained person had on the institution. (The team visited three of the 10 hospitals that reported training, and children’s surgical services were being provided at all three.) Also, as a training target (approved by PVC), one trained person per institution is too modest an objective. More providers typically need to be trained, in more than clinical or surgical aspects of eye care––and other types of inputs may also be needed as well––to increase capacity. To be sure, training is often a valuable and necessary input. The lack of a trained provider(s), whether in pediatric surgical skills or anesthesia skills, is often a “rate-limiting step” to providing higher quality and quantity of services. But even when training is necessary, it may not be sufficient. From the standpoint of capacity, there are a number of other, larger, service-level and institution-level dimensions that are more meaningful objectives. At the service level, there is the institution’s capacity to provide good quantity and quality services at an affordable price. At the institution level, there is the ability to be self-managing and self-sufficient: the ability to plan, to raise and manage funds, and to provide a wide array of services. These observations also apply to other inputs that were not reported on as objectives in Seeing 2000, but which were central parts of the input side of assistance. For example, 53% of the all small grant monies went for anesthesia equipment, since the absence of such equipment can make providing surgery to infants and small children impossible. Clearly, then, such equipment can increase an organization’s capacity to provide childhood surgery, especially on infants and very young children. But there was no objective relating to the equipments being in use, and to what extent. In one instance the evaluation team encountered, the anesthesiologist upon whom the project was relying was no longer available. Thus, despite the purchase of needed equipment, the surgeries could not take place. There may have been similar instances. 3A.1.3 Constraints SOW Question A1b. Identify constraints encountered and assess IEF’s response to them. Constraints Encountered by IEF and Its Response to Them IEF encountered a number of significant constraints, i.e., aspects out of its immediate control that limited performance of the Seeing 2000 Program. These 14 constraints fell into financial, health system, institutional, and disease categories. Some of the most noteworthy constraints, and IEF’s responses, were: • IEF is a small organization with relatively few staff; funds for the overall program were modest (averaging between $300,000 and $400,000 annually, inclusive of headquarters expenses); and project monitoring and evaluation is labor-intensive. IEF responded to these constraints with a small grant program that would meet potential partners’ needs in the area of pediatric eye care services. This was a cost-effective way of building a partnership network, and a satisfactory modus operandi for the partners, who were having self-identified needs met rather than programs imposed on them. From a management perspective, this approach meant that half the activity involved procurement done in the U.S., and that project monitoring in the field could be minimized. (Greater frequency of monitoring in the field was felt to be infeasible because of the limited IEF staff, large number of grantee institutions, and costs involved in travel.) • Surgically correctable childhood blindness occurs infrequently, unpredictably and sporadically, but the Congressional mandate was somewhat narrowly written, relating to an increase in the quantity of infant/child surgery. To preserve vision, such surgery generally must be done on children below age 6, yet many of the cases typically present late, in which case surgery doesn’t equate to restoration of sight. IEF responded by expanding the dimensions of the program beyond surgery per se, to related activities, e.g., training, outreach and screening. IEF did not move as far as it might have, to areas like low vision or spectacle provision, which might have also “restored sight,” perhaps to more children. Nor did it engage in large-scale preventive activities with these funds (which might have had more impact), because of the Congressional language (training, e.g., in ROP, has a significant preventative aspect). Constraints Encountered by IEF ‘s Local Partners and IEF’s Responses SOW Question A1b, continued. How did IEF address constraints experienced by sub￾grantees? • Dedicated facilities/programs for pediatric ophthalmologic needs are relatively rare. Similarly, an individual or institutional commitment to eye care services for children is also relatively rare. IEF responded by supported training, providing equipment, and encouraging the establishment of dedicated eye services for children at its grantee organizations. • Eye care and anesthesia providers are in short supply in many developing countries, yet surgery for infant and child blindness typically requires distinct ophthalmologic skills as well as general anesthesia. 15 IEF responded by emphasized pediatric ophthalmologic training and provision of anesthesia equipment as key aspects of small grants. • In some countries, such as Bulgaria and Malawi, where health care is seen as a citizen’s right and a provider’s duty and has generally been provided free to the populace via the public sector, it has been difficult to structure services to promote cost-recovery. IEF responded by working successfully in Malawi institutions at the policy level to allow the Lions SightFirst Eye Hospital (a part of the public sector hospital) to charge fees and receive and use the funds it generates. On the other hand, despite the long-term relationship that IEF has had with its Bulgarian partner, the Pashev Center for Sight, no significant cost-recovery or fee-for-service activity has yet taken place there. (Although IEF has recently supported study tours to Aravind focusing on these issues.) • Data generation was of variable quality by grantee partners. IEF did not respond by guiding the grantees to provide more useful data relevant to capacity and sustainability. In general, IEF indicated that it felt its leverage in terms of inducing better grantee reporting was minimal because IEF funds were only a small part of most organizations’ operating budgets (and that such involvement might be construed as “micro-management”). The evaluation team does not agree, feeling that if IEF had made reporting forms or templates available, and had emphasized baselines and post￾grant reporting, with a focus on capacity built and sustainability achieved, the grantees would have supplied that information, and profited from doing so. 3A.1.4. Major Successes SOW Question A1c. Identify major successes and why these elements were successful. In many respects, IEF’s small grant program was an innovative, cost-effective and successful way to increase surgical services for childhood blindness. A great deal of activity in many countries and regions was catalyzed by a relatively small infusion of IEF funds. Many grant partners are true “centers of excellence” and almost all are continuing to provide increased services for children (relative to baselines). IEF itself also gained useful management skills in the process of implementing the small grants program. IEF fashioned a wide geographic network of committed and capable service organizations to focus on pediatric surgical and ancillary eye care needs. This wide network gave surgery for childhood blindness a global focus and helped to leverage resources. IEF itself helped, through its links to other INGOs, multilateral organizations such as WHO and coalitions like the IAPB, to raise the profile of childhood blindness at policy and program level as a separate entity requiring separate services. This small grant program was successful for a number of reasons. First, IEF was seen as a reliable and helpful partner, without any conflicting agendas of its own. 16 Second, there was a sense of program ownership within grantee partners because they had themselves identified needs, implemented activities, managed resources, and tracked program results. That is, the Seeing 2000 Program was highly participatory and there was no micro-management from IEF. Third, the network arose from self-selection, i.e., the “good” organizations, like cream, rose to the top in the RFA process. Thus some of the partners, while “local NGOs,” were very sophisticated and had larger annual budgets than IEF itself. These mature partners became valuable training resources and offered varying models of service delivery and organizational structure for IEF to learn from and draw upon. IEF was able to foster South-South transfer of capability by partnering nascent NGOs with more robust ones, and by itself partnering with those mature NGOs and other INGOs. In sum, PVC is correct that “networks are important mechanisms to leverage resources and amplify influence of PVOs”. 3A.1.5. Effectiveness of Models and Approaches SOW Question A1d. Assess effectiveness of models, approaches or assumptions that underlie the program. Comment on the possibility of plans to scale-up the approach. There are a number of models, approaches and/or assumptions that underlie both the Seeing 2000 Program and the Seeing 2000: Revised and Expanded Program. Some of the most important of these, and the evaluation team’s assessment of them, are: 1. The small grant model The characteristics and achievements of this model are the major part of this evaluation report. Scaling it up, i.e., making more small grants to more institutions, would depend either on the identification of other capable local-partner eye care organizations or on the identification of more need within existing partners. IEF maintains that in its small grants program it is already working with all the capable local￾partner eye-care NGOs, and thus the latter scenario is more likely to be fruitful. This in turn will be dependent on a broadened interpretation of the Congressional mandate to allow provision of low vision services and spectacles. Otherwise, the need for anesthesia equipment has been largely met and the desirability of funding further recurrent costs (e.g., for salaries for outreach workers) is limited. 2. Large-scale service delivery model. This model could also be termed the Aravind model, or “compassionate capitalism”. [Green, D., “Compassionate Capitalism,” IAPB News, #21, Jan. 1997.] It entails a strong sense of mission, excellent management, robust staffing, the provision of high quality, high volume services, a large amount of cost-recovery coupled to significant cross-subsidization of poor patients, and significant other internal sources of income generation. The service aspects of this model have already been replicated, with Aravind and Seva Foundation assistance, at Lumbini Eye Hospital in Bhairahawa, Nepal, which shares a similar sense of mission, and serves a similarly large catchment area. (Lumbini is 2 miles from the Indian border; 65% of its outpatients and 80% of its surgery 17 patients are Indian. Thus Lumbini is de facto largely a north Indian program, serving not only its Nepali patients, but the state of Uttar Pradesh as well, in a manner analogous to Aravind’s service to Tamil Nadu.) This model might well be transferable to other institutions in India––whose states all have 60 million or more people or Pakistan or Bangladesh––and Aravind is active in doing so. Whether this model is transferable/replicable in other regions (e.g., Africa) with different characteristics––fewer personnel, less aggregate wealth, predominantly public sector service provision, a lower eye disease burden, and significantly smaller population of adults and children, i.e., a smaller “market”––is an uncertain challenge. In any event, Aravind is a superb venue for staff from other regions to observe and learn. 3. The Western-oriented, state-of-the-art model. This could be called the “Prasad model”. The L. V. Prasad Eye Institute is a first￾rate “center of excellence”, and significant cross-subsidization to care for the poor occurs. It is a well-staffed, multi-disciplinary, research-oriented, training as well as tertiary-care institution, which receives much of its funds from wealthy individual donors (predominantly in-country and expatriate Indian physicians) as well as multinational companies (e.g., a $2 million grant from Bausch & Lomb). However this model is unlikely to be a useful one for replication to other institutions, most of which are poorer in both human and financial terms. 4. The Technical Assistance (TA) Model. IEF is in the process of implementing this model in the Seeing 2000 Program, Revised and Expanded. The model entails IEF’s provision of TA in management with a focus on financial sustainability (drawing on the Aravind model). Three institutions, in Malawi, Guatemala, and Egypt have been selected. Although this approach makes general sense from a development standpoint, it is too early to make judgments about its success or replicability, although the facility in Malawi, Lions SightFirst, working closely with Aravind/LAICO and IEF, has already seen a dramatic rise in services early in its program. • The ROP (Retinopathy of Prematurity) Model This model is in various stages of development at IEF grantee institutions, including IEPSV, Peru, Pashev Center for Sight, Bulgaria, and the All-India Institute for Medical Science, Delhi. ROP is a disease of very premature infants, a percentage of whom will develop a retinopathy that if untreated can soon lead to retinal detachment and blindness. The program entails education and training for pediatricians, neonatologists and nurses to emphasize appropriate oxygen use in premature infants and early referral to pediatric ophthalmologists for exams under anesthesia and surgery if needed. Numbers of children assisted in any country, typically an advanced developing country, will be small, but the gain in each child, in terms of blindness truly prevented and disability-adjusted life years gained would be very high on a per case basis. Furthermore, in optimal situations, 18 numbers of operations may be fewer because appropriate preventive care at both the neonatal and ophthalmological levels will have taken place. That is, in certain respects it is a preventive as well as curative program. 3A.2. Impact on Local Partners SOW Question A2. Assess the impact on the local partners in the implementation of the Matching Grant. What were the organizational benefits and challenges? [See Sections 2.4.2, 3A.1.4, and 3A.4 for related discussion] The impact on partners, in terms of benefits and challenges, took many forms. In addition to enhanced service capacity, the NGOs developed additional management capacity as well. The individual Seeing 2000 programs were fully conceived, designed and implemented by the local NGO grantees, with minimal if any modification or modulation by IEF and minimal programmatic oversight or intervention. Thus the program was highly participatory and was aimed at meeting felt needs in a “bottom￾up” fashion. Individual small grants reflected what the grantees wanted. Partners had to think about what they wanted, justify it, write proposals, track results, report on their achievements––all useful skills for organizational thrival. (One challenge for IEF was determining the content and frequency of required reports from grantees; conversely, generating these reports was often a challenge to the grantees). Even at the two most advanced, active, and financially and technically robust partner organizations, Aravind Eye Hospitals and the L.V. Prasad Eye Institute, the grant had a catalytic effect. Leaders of both organizations commented to the team to the effect that “we would have established separate [facilities for] child services eventually, but it would have taken time”. SOW Question A2., continued. Assess satisfaction of project beneficiaries. All seven of the local partners visited expressed high levels of satisfaction with the partnership with IEF, with the timeliness and utility of the IEF assistance given, with IEF’s responsiveness to their requests, and with what they, the grantees, accomplished with the grant funds. Leadership of each of the seven partner organizations visited by the evaluation team remarked that the Seeing 2000 grant enabled them to focus their institution on the needs of children, and to maintain that focus post-grant. For example the following are representative of all institutions visited • The director of the Bulgarian Center for Sight said, “The grant allowed us to promote the necessity of care for children and how important it is to make an early diagnosis.” • The director of the Tilganga Centre commented that, “Before the grant we hardly had focused on children’s eye disease. The effect of the grant was to give us a mandate to go there and exercise our brains to make child eye services better.” 19 • The project administrator of the Lumbini Eye Hospital said, “The grant really worked. Child eye health care, in my opinion and 15 years of experience, was never given a high priority [here]. Now it is.” 3A.3. Progress Toward Sustainability of Subgrantees SOW Question A3. Assess progress toward sustainability of grantees SOW Question A3a. Identify the program elements to be sustained. There are a number of types and dimensions of sustainability. Sustainability may be full, or partial to varying degrees. It may refer to technical, programmatic, financial, and/or organizational sustainability. With respect to pediatric eye care services, what needs to be sustained, at a minimum, at the grantee level, is an eye surgeon with pediatric skills and a commitment to providing services, availability of pediatric anesthesia, adequate funding, and appropriate infrastructure. Ideally included in this latter category is maintenance of equipment, presence of low vision and other ancillary services, and a continuing commitment to keeping services available, affordable and of good quality. SOW Question A3a., continued. Identify sustainability objectives and indicators. For the Seeing 2000 Program, the most salient immediate questions relate to grantee partners’ programmatic sustainability post-grant. These questions encapsulate the most important objectives and indicators for programmatic sustainability: • Are the pediatric eye care departments still intact and active? • To what extent are activities and benefits continued post-grant? • How many surgeries and eye exams on children take place post-grant? • Is the equipment still in use? • Is anesthesia for procedures on infants and children available? • Are the trained providers providing services, and if so, to what extent? These questions were not addressed in a systematic way by IEF, nor were they reported on by their grantees post-grant. However the evaluation team was able to gather some information during its visits about aspects of sustainability within visited grantee partners, as elaborated on below. SOW Question A3a., continued. Address prospects for post-grant sustainability. Prospects for both meaningful post-grant sustainability of significant service delivery to children and institutional self-reliance are very high in a number of institutions, chiefly those that were already strong general eye care institutions. (e.g., Aravind, Prasad, Lumbini, Tilganga, IEPSV, Elias Santana, Al Shifa, LRBT, and Lady Reading). Overall, in all of the grantee institutions, there are either dedicated facilities for children, or children receive priority in triage. Twelve of the 14 institutions reported 20 increased surgery during the grant, and most if not all of them are still providing surgical and other services for children, at a level higher than baseline. One institution, Lumbini, which used some of its small grant funds to hire 12 outreach personnel, not only dramatically increased its outreach services in a sustained fashion, but it was able to retain four of the personnel with its own funds. At IEPSV, the work on ROP attracted additional INGO support from Christoffel BlindenMission/Germany. Similarly, Mt. Sion Hospital, Papua New Guinea, received $100,000 from New Zealand because of its Seeing 2000 Program activities. It is too early to determine if post-grant financial sustainability, with significant cost-recovery and cross-subsidization of the poor, will take place in the institutions in Malawi, Egypt, and Guatemala, where IEF is working in depth (or planning to work) on management, cost-recovery, and financial sustainability. From the narrow standpoint of children’s services, the likelihood that these services would be financially sustainable anywhere is unfortunately low, given the relatively low volume of pediatric cases. “Success” will be dependent on cross-subsidizing pediatric services from more active adult services for paying patients, such as cataract surgery services. SOW Question A3b. Describe the existence and status of cost-recovery mechanisms, local level financing or other approaches to generate resources to support project operations. Ultimately, of course, financial sustainability underpins institutional and program sustainability. There are many components to financial sustainability, and just as different grantees had different programmatic structures, systems and results, so they had different approaches to cost-recovery, revenue generation and financial sustainability. Cost￾recovery and cross-subsidization of services exists substantially at a number of IEF’s grantee partners (e.g.,, Aravind and Prasad in India, Lumbini and Tilganga in Nepal, and Elias Santana in the Dominican Republic), and partially at three other institutions. Although cost-recovery has not been a focus heretofore of the Seeing 2000 Program, it is quite central to Seeing 2000: Revised and Expanded’s emphasis on management, cost-recovery, and financial sustainability. IEF has introduced financial sustainability planning and sliding fee scales that reflect costs and willingness/ability to pay in Malawi (Lions SightFirst Eye Hospital) and will be doing so at El Maghraby Eye Hospital (Egypt) and Roosevelt Hospital (Guatemala). Salient partner cost-recovery experiences include: • Fees for other related services. (For example, IEF contracted with the Aravind Eye Hospitals for provision of technical assistance to IEF’s Malawi partner.) • Fees for selling products such as intraocular lenses and sutures. (For example, Aravind-LAICO, with Seva Foundation assistance, and Tilganga, with Fred Hollows Foundation/Australia assistance, have both become manufacturers of ophthalmologic supplies, thus generating significant revenue streams.) 21 • Raising funds from private donors: (for example, L.V. Prasad Eye Institute has raised upwards of $10 million from private and corporate donors). • Securing international NGO donor funds: almost all of IEF’s partners also get support from other INGOs. 3A.4 Status of Strategic Partnerships SOW Question A4. Assess the status of strategic partnerships with NGOs, community￾based organizations, local level government, or charity hospitals.[Other parts of this SOW question were redundant; see Section 3A2 for related discussion.] From a certain perspective, IEF’s partnerships with its donors, grantees, and fellow INGOs are all “strategic partnerships”. At the most basic level, the partnership between IEF and each of its 22 small grant partner organizations creates benefits for each organization, as does the partnership between IEF and PVC. IEF’s Seeing 2000 resources allowed grantees to purchase equipment, conduct outreach, or train staff. In turn, the increased outputs helped IEF reach its own Seeing 2000 goals and objectives. Similarly, PVC funds and other inputs (such as encouragement to engage in the DOSA process) strengthened IEF, which in turn contributed to PVC’s own development objectives. This range of partners, in approximately 20 developing countries, has allowed IEF to solidify a wide partnership network. In the process of building this network, IEF has gained experience in assessing the strengths of different models of service delivery and management, in different regions. This has contributed to IEF’s own institutional learning and to the diversification of its program. For example, IEF’s own SightReach “social enterprise” depends on the Aravind Eye Hospital’s products that it then resells in other countries, in effect acting as a middleman. Because Aravind manufactures intraocular lenses and suture material of high quality at a very low relative price, both IEF and Aravind can make profits in this enterprise. In turn they can then use the funds to further their mission––in Aravind’s case, to cross-subsidize services to the poor, in IEF’s to enlarge its program, expand the availability of quality, low-cost consumable surgical supplies, and diversify its funding. Aravind is thus a valuable resource to IEF, in effect moving from grantee partner to “business partner”. That is only one way in which IEF profits from its association with Aravind. Another type of partnership is seen when the grantee partner becomes a technical resource for IEF. This has increasingly been the case with Aravind, especially for the Seeing 2000: Revised and Expanded, with its emphasis on financial and management aspects of programs rather than equipment, supplies and service delivery per se. Aravind has provided TA to Malawi and served as a training venue for staff from Albania, Bulgaria, Guatemala, Malawi and Pakistan. (With Seva Foundation assistance, Aravind has also partnered Lumbini and been a venue for significant training of its personnel.) On the service delivery front, in 1999, Aravind, located in Madurai, a city of 1 million people, had over 70 ophthalmologists on staff, performed 1.3 million outpatient 22 visits and 160,000 surgeries, of which 2/3 were free. This represented 1% of all the eye surgery in India and more eye surgery than in the entire U.K. Aravind not only serves a robust South-South function as outlined above, but it also serves a South-North function, with many developed country ophthalmologists coming there to work and learn. They are drawn by its high-quality, high-volume approach to delivering patient services and its sense of mission. (This approach was described to the evaluation team as “the airplane model: better meals and more legroom in 1st class, paid for by the passengers who can afford it, but all passengers get to the same destination at the same time.”) Another key grantee partner for IEF, and another (perhaps the preeminent) regional center of excellence, is the L.V. Prasad Eye Institute in Hyderabad, India. This institution conducts much first-rate research, and is modeled along the lines of multi￾specialty eye institutes in developed countries. It houses the secretariat for the IAPB and also edits and publishes a major scientific journal, the Indian Journal of Ophthalmology. In addition, IEF has a number of international NGO (INGO) partners, either at the local institutional level or at the international and policy level. Orbis International/USA is a partner in IEF’s Ethiopia program; Christoffel BlindenMission/Germany is a partner in Peru; SightSavers/UK is a partner in Zimbabwe; CBM/Germany is a partner in the Philippines; and Lions International, Seva Foundation, and Al-Noor Foundation are partners in Malawi. Finally, IEF’s grantee partners almost all have other partnerships, on other aspects of their programs, with other INGOs. Finally, at the international and policy level, IEF has a “seat at the table.” It is a member of key blindness prevention coalitions. For example, it is a supporting member of Vision 2020: The Right to Sight, the WHO-led global initiative to eradicate avoidable blindness by the year 2020. IEF also co-sponsored (with WHO and L.V. Prasad, which also hosted) the IAPB’s 1999 Scientific Meeting on the Prevention of Blindness. At this meeting 9 of the 18 influential eye care organizations, and 18 of the 30 individuals, were IEF partners in the Seeing 2000 Program. IEF’s other organizational links with WHO include IEF’s membership on the Partnership Committee of NGOs Dedicated to the Prevention of Blindness and Education and Rehabilitation of the Blind, of which IEF’s Executive Director was the Chair from 1993-1998. IEF also is a member of WHO’s Alliance for the Global Elimination of Trachoma, and its NGO Coordination Group for Ivermectin Distribution, as well as a member of the CORE Child Survival NGO Consortium. From a certain perspective, IEF’s most “strategic” and important partner is USAID. From the standpoint of IEF-PVC relations, the cooperative agreement for the Seeing 2000 Program calls for, or allows, substantial involvement from USAID. Although IEF’s executive director and director of programs have both served as PVO representatives on USAID’s Procurement Committee (during USAID’s streamlining process as part of the recent federal-wide re-engineering effort), on the whole, IEF and PVC have met relatively infrequently. According to IEF, frequency of formal meetings between IEF and its three successive PVC project officers, whether to review results or 23 for other reasons, has varied from quarterly to annually. (The present project officer has been backstopping the project since 12/99. Besides those meetings, occasional other meetings with the leadership of the Matching Grants Program and PVC Office have also taken place in one configuration or another, for various purposes, during the period of project activity.) 3B. PROGRAM MANAGEMENT 3B.1. Change in IEF Management Capacity SOW Question B1. Assess change in IEF’s management capacity/structure Seeing 2000 funds from PVC have allowed IEF to augment its staff and its collective technical capacity. The program coordinator is fully funded by the grant, and IEF’s senior staff is partially funded (5-15%). Collectively these four people––three of whom got their professional start in the U.S. Peace Corps––have over 40 years of tenure with IEF, and over 60 years of experience in international development. PVC funds have enabled IEF’s staff members to strengthen their skills at courses on quality assurance and other aspects of management, as well as their knowledge of the latest technical and programmatic developments in ophthalmology. Conversely, IEF staff has also presented papers and program results at various international fora, including one on sustainability planning at the General Assembly of the IAPB in Beijing in 1999, and several others at the American Academy of Ophthalmology meetings in 1999 and 2000. Under Seeing 2000: Revised and Expanded, an additional staff person with a strong background in management has recently come from Aravind Eye Hospitals to join IEF’s core technical staff as the sustainability specialist. A consultant who has done much of the seminal work in making surgical eye care affordable and financially sustainable in the developing world has also been used frequently. SOW Question B1, continued. What changes have occurred in IEF’s capacity for critical and analytical thinking regarding program design and impact? In mid-98, IEF conducted a Discussion-Oriented Organizational Self￾Assessment (DOSA). As a result, in late 1999 IEF adopted its first overall strategic plan to guide its own strategic reorientation. The plan identifies financial and programmatic priorities as well as targets and benchmarks. It is guiding the implementation of the remainder of the Seeing 2000 Program and the initiation of the Seeing 2000: Revised and Expanded Program, as well as IEF’s other activities implemented with other USAID and with non-USAID funds. The plan emphasizes IEF’s own financial security and diversification of funding, as well as its providing management technical assistance to its grantees, emphasizing cost-recovery and financial self-sufficiency. IEF understands that the most critical sustainability issue it itself faces is financial sustainability, including diversification of funding sources. The need to achieve financial sustainability and to reduce dependence on any one donor, in this 24 case, USAID, is a central feature of its newly devised strategic plan, with a goal that at least 60% of IEF’s income comes from private sources. (At the moment 40% of IEF funds, excluding in-kind donations, come from non-USAID sources.) For this reason, too, one of the three elements of Seeing 2000: Revised and Expanded is a social enterprise whereby IEF will sell supplies and equipment at a reduced cost, but at a profit. This program, “SightReach Surgical” began in mid-1999 and grossed $65,000 in its first fiscal year, working in 3 countries. The program is projected by year six to be working in 10 countries and to be generating $330,000 in gross revenue. SOW Question B1, continued. Are there changes in HQ capacity to manage the planning process, refine the RFA mechanism and award process, address replicability, sustainability, strategic planning, financial planning? Program design of IEF’s field programs has been modified to give larger grants to fewer organizations, under both the Seeing 2000 Program and the Seeing 2000: Revised and Expanded Program. This decision will allow IEF to focus and concentrate its resources on management and sustainability with those partners which might become significantly more sustainable with the help of IEF TA that emphasizes technical and management capacity-building leading toward financial self-sufficiency. The RFA process and Review Board have also been eliminated. Instead, IEF technical staff, now augmented in number, decides on grantees and levels of funding according to general criteria that include the potential partner organization’s overall capabilities and the likelihood of achieving significant results. The underlying thrust of the new direction is to draw from and try to duplicate the successful programmatic, planning, and financial sustainability experience of Aravind at other institutions. Specific criteria for selecting these potential grantee partners are currently being written, including the tools needed to assess potential-partner capabilities. These key dimensions (that IEF has already been considering in its decisions) include: clinical efficiency, effectiveness and competence; leadership, continuity, and adequacy of staff; management structure and expertise; service delivery characteristics; financing structures and degree of institutional control over resources; physical infrastructure, adequacy of equipment and supplies; and data-collecting, and reporting capabilities. 3.B.2. Monitoring and Evaluation SOW Question B2. Assess the capacity of the organization to monitor program performance and measure impact. Is there evidence of capacity building, sustainability objectives and indicators, baseline assessment and plans for final assessment, knowledge and use of impact and performance measurement techniques, an MIS to consolidate, analyze and interpret project data, transfer of skills and capacity to local partners? Notwithstanding that one of IEF’s own “guiding principles” in its new strategic plan is “indicators measure outcomes and impact rather than inputs,” there is to date little evidence of this orientation reflected in IEF program reports and documents. For 25 example, in evaluation team briefings and in IEF documents made available to the team, little documentation was given regarding outcomes and/or impact, in terms of capacities built and sustainabilities fostered. This area is generally weak within IEF, as IEF itself acknowledges. IEF staff seems to have a good sense of what is going on with their grantees, but much of it relies on impressionistic and anecdotal information. Weaknesses include a lack of a program MIS to consolidate, analyze and interpret data, and a lack of an overall M&E plan. Evaluation, to the extent that it happened in Seeing 2000, was largely retrospective. Furthermore, the Seeing 2000 indicators that were in use measured service and training outputs not capacity or sustainability outcomes. Field-based monitoring of projects was infrequent (because of the large number of small grants, the fact that the active period of project work was typically only 12 months, staff was limited and travel is expensive). Larger service-level and institution￾level objectives, e.g., an institution’s capacity to provide quality, quantity and affordable services or its degree of cost-recovery and self-sufficiency, were not routinely monitored or evaluated by IEF, although IEF staff seemed generally aware of the status of these variables at partner institutions. Finally, post-grant assessment of grantees was not done in any systematic and/or written way, and technical assistance to grantees in this area, in order to help generate more programmatically useful data, has not been given. (See also Section 3B. 4, below.) 3.B.3. Financial Management SOW Question B3. Were adequate financial monitoring systems in place? Has the program leveraged additional resources (beyond the match)? Is there an estimate of cost￾effectiveness of the program? Financial management was a relatively minor aspect of the evaluation team’s consideration, per consensus discussions with IEF and PVC at the Team Planning Meeting. The team had the sense though, that financial monitoring systems were working well, and that USAID funds were well tracked and were disbursed in a timely way. According to IEF, “our own annual audits focused on the Seeing 2000 Program have found no major discrepancies or weaknesses”. There was no match entailed in this agreement with IEF. However, over $1 million in additional funds, as well as over $1 million in in-kind resources were leveraged by IEF from various sources. IEF also leveraged human resources, with three experienced pediatric ophthalmologists serving voluntarily on the RFP Review Board, and many others conducting sub-specialty workshops in Bulgaria, Guatemala, Malawi, Pakistan, and Papua-New Guinea. There is no estimate of cost-effectiveness of the program. An extremely crude estimate of cost per childhood surgery, approximately $300 per child operated on, can be obtained by dividing the number of children operated on by the amount of the Seeing 2000 Grant. (On the one hand, this neglects the value of the screening exams of children, the ROP prevented by better oxygen use, the capacity built through training, study tours 26 and the like and other policy and program changes which IEF helps to foster. On the other hand, such an analysis only looks at IEF’s cost, i.e., there are costs involved to the NGO providing the services, and contributions of other donors.) 3B.4. Information and Results Reporting SOW Question B4. Comment on the utility and timeliness of IEF’s required reports. As with the related area of M&E, IEF needs to improve its information generation and results reporting. Results reporting, from local grantee to IEF and from IEF to PVC, has not routinely included baselines or post-grant achievements and other measures of continuity of activities/programmatic sustainability. Annual reports to USAID have been of little usefulness because of timing (a year or more late) and format––long compendia of activities, not syntheses of individual program data or analyses of meaningful achievements (i.e., answering the “so what?” questions). IEF indicators and results have not been linked to PVC’s SO and IRs and thus have not spoken usefully to IEF’s contribution to PVC’s achieving them. IEF has not reported on its own or its grantees’ enhanced capacity or sustainability (programmatic, technical and/or financial), nor has it used qualitative reporting, e.g., “stories,” “vignettes” or “case studies,” which are also effective ways to convey information, and are useful to PVC in its R4 process. SOW Question B4, continued. Comment on the organization’s public outreach activities. As a member of InterAction, IEF has raised awareness about the Global Initiative for the Elimination of Avoidable Blindness (Vision 2020: the Right to Sight). IEF has also reached out to various audiences about its program and USAID’s role and USAID programs’ development impact via a number of communication channels: international newsletters, direct mail, annual reports, and domestic professional meetings (e.g., AAO). 3B.5. Other Management Functions SOW Question B5. Logistics: Comment on the adequacy and timeliness of IEF’s material inputs. IEF’s material inputs, equipment, technical materials, supplies and the like, were quite adequate to the job, delivered in a timely fashion, and adjudged to be very useful to grantees. In particular, over 50% of the IEF assistance funds went for equipment and supplies desired by grantees. This equipment was procured and supplied in a timely fashion and helped the grantees meet their major objectives. An MIS is in use to track medicines distributed, equipment procured and shipped, and to otherwise manage the logistics of material distribution. Finally, while technical materials were supplied on a more ad hoc basis, a number of visited grantees commented on their utility. SOW Question B6. Supervision: Assess if there were sufficient staff and consultants with the appropriate technical and management skills to oversee program activity at both HQ and in field programs. 27 IEF seems thinly staffed for the mission it has embraced, although it is important to keep in mind that actually 60% or so of its overall budget goes to one country, Malawi. The technical skills within existing staff seem high, in that all are technically qualified and strongly committed to the organization and the field. Leaders of IEF are perceived as leaders in the field, as inferred from presentations made at high-level fora, participation in international (WHO and other) committees, task forces and the like, and presentation of programmatic papers at important meetings (e.g., IAPB, AAO). IEF has added another experienced person to its core staff, someone who formerly worked in a high management capacity at Aravind Eye Hospitals, and IEF is also drawing upon a very experienced and business-savvy consultant to help with aspects of its own and its grantees’ revenue generation and financial sustainability. Also, from a programmatic standpoint, IEF has focused and concentrated its mission, beginning to provide larger amounts of funds to fewer, carefully selected, institutions. It is still quite likely that IEF will feel thinly stretched and will need more personnel, because the new modus operandi whereby IEF itself provides technical assistance will require much greater staff time in the field. The Seeing 2000 modus operandi whereby a small grant partner is visited once, and then contact is maintained via e-mail and fax will not work when major organizational change is being attempted. (IEF itself has recognized this by budgeting for significant TA and field monitoring under Seeing 2000: Revised and Expanded.) Also, the greater attention to M&E and to results reporting, which IEF recognizes it needs to make, is a significant undertaking that will probably require outside assistance to IEF, at least initially. While this will represent effort very well spent in the long run, it requires significant staff time and commitment to bring about a results orientation and an ability to document those results in meaningful ways. And once such an orientation is inculcated in HQ, much work will need to be done to transfer that orientation to grantees. SOW Question B7. Human Resource Development: Did IEF address staff training needs and strengthen the organization and local partner professional or technical capacity? IEF addressed its own training and organization-strengthening needs via several courses on quality assurance at Johns Hopkins University, and by undergoing the DOSA process. Augmenting local partner professional and technical capacity was a major thrust of the Seeing 2000 Program, as this evaluation report has documented. It is clear that even if these dimensions were not measured and evaluated per se, the ability of partners to provide more and better surgical and ancillary eye care services to infants and children was significantly enhanced, in an ongoing manner. In sum, to both cite and agree with IEF: “IEF believes that staff members have been very good in addressing partner needs and developing capacities … [but] there is a need to improve certain analytical and reporting skills at HQ.” 28 4. RECOMMENDATIONS (proposed actions, based on conclusions) IEF’s Overall Program • IEF should continue to follow its new, DOSA-derived strategy, concentrating on: • increasing its non-USAID funding. • diversifying its funding sources. • strengthening and augmenting its staff. • emphasizing management in general and capacity enhancement and financial sustainability (of both itself and its partners) in particular. • providing technical assistance in management and financial sustainability. • continuing to be involved at the policy level at international eye care fora. Program Strategies and Priorities • IEF should continue its new programmatic directions and emphases, making fewer but larger grants to well-chosen NGO partners. Grants should be for sustainability (not for, say, recurrent costs), and, as appropriate, for longer time periods than 12-24 months. Fewer “eggs” in IEF’s “program basket” means that each NGO partner’s set of results comprises a larger proportion of IEF’s own desired results and objectives. Thus selecting optimal partners is even more of an imperative. Also key is the potential impact such a strengthened institution may have on the problem of blindness. It does little good to strengthen an institution which, when strengthened, will still make relatively little contribution to alleviating that problem. • Criteria guiding partner selection should be written and should be a subject, if not a product, of dialogue with PVC. Criteria IEF is using to select grantee partners are programmatically sound from the standpoint of selecting institutions. However they do not speak to the criteria for selecting countries. The success of the two Indian project sites and the two in Nepal––one of which serves mainly Indians–– is closely related to India’s population size, prevalence and amount of eye disease, number of children, and aggregate wealth, among other factors. • Numerical goals that are used in Seeing 2000: Revised and Expanded should be more ambitious (while still realistic) and set in absolute terms as well as in percentage increases. • The continuing existence of the small grants program, in this new incarnation, should be made widely known in the international eye care community. (A number of 29 grantees remarked to the evaluation team that they thought the small grant program for childhood blindness and related activities was no longer in operation.) • IEF staff should/will need to provide significantly more TA to grantees in program management and service delivery than it has in Seeing 2000. This TA, whether from HQ, consultants, or partners, has cost implications in terms of staffing and travel. Monitoring and Evaluation (M&E) • IEF should strengthen its M&E capability and performance, including having its staff trained in the latest, field-useful, practical approaches to monitoring and evaluation. • IEF should have an overall M&E plan that articulates responsibilities of both IEF and local partner staff in the M&E process, data to be collected, timetables for collection, and ways that data will be used to inform subsequent programming. That is, M&E must become a management tool, an integral part of design and implementation, prospective not retrospective, present from the start. • This attention to overall M&E, improved data generation, and stronger outcome indicators and goals, needs to be immediate, with appropriate modifications made to existing sub-grants, and technical assistance given to reporting institutions. At this point Seeing 2000’s relatively limited indicators and objectives are still in effect and have essentially been transferred to Seeing 2000: Revised and Expanded. For example, there are no requirements for NGOs’ post-grant reporting, and for disaggregation of service data into useful categories. There are no capacity or sustainability indicators. Finally, and fundamentally, the indicators do not get at the “so what” questions––how have service quality and quantity, capacity and sustainability improved? • Useful, meaningful, measurable, operationally defined, and mutually understood benchmarks and indicators, related to increased capacity and financial and programmatic sustainability, need to be part of agreements between IEF and partner grantees, and need to be tracked by IEF and regularly reported on. (IEF’s program may be only a small part of a grantee’s overall program, but goodwill exists, and IEF can reasonably expect cooperation in more substantive reporting. If organizations need initial TA in understanding reporting needs and formats, IEF should provide it.) • As part of its management TA, IEF should provide a “results template” to its partners upon which these disaggregated and meaningful service delivery, capacity and sustainability results can be reported. This template can be a management and evaluation learning tool for the grantee, addressing priority program aspects and providing feedback that helps guide institutional decisions. Ideally, annual reports using this template should continue even after formal IEF assistance has ended. 30 • IEF should routinely make post-grant organizational assessment reports of each grantee’s program, speaking to changes in service quantity and quality, increased capacity and other meaningful changes within the organization. • Given its size, IEF should not [now] have a separate evaluation officer, since that might only perpetuate the separation of M&E from program; rather, M&E should be an integrated part of every technical staff person’s responsibilities. • IEF should support the conduct of special outcome studies where needed. These studies would answer the kind of “so what” questions discussed above and in Section 3.A.3. For example, there is much rhetoric on quality of service in IEF documents, but quality is never defined or reported on. Is “quality” only affordable surgery, done well in sterile environments, or is it restoration of vision, especially in terms of childhood vision regained and/or saved? Similarly, special studies on cost aspects of the program, such as unit costs, health care financing, institutional income generation, and so on––will be needed and useful, as would case studies of individual institutions, on, for example, South-to-South experiences, or strategies for financial sustainability and experience to date. These studies could be of wide applicability and interest to the larger PVO community. Results Reporting • IEF needs to significantly strengthen the whole area of its results reporting. • Results reporting, from local grantee to IEF and from IEF to PVC, needs to include baselines as well as ongoing, post-grant achievements and other measures of continuity of activities (indicators of programmatic sustainability). • As much as possible, data need to be comparable––same indicators, same time periods for baseline and for program activity (e.g., all information annualized). • Annual reports to USAID––in fact all external reporting––need to be syntheses of individual program data and accomplishments, and analyses (i.e., answering the “so what?” questions) rather than compendia of activity reports. • Annual reports to USAID need to speak to IEF’s contribution, in both qualitative and quantitative terms, to PVC’s SO and IRs. This would lead grantor and grantee to focus on outcomes and impact, not simply inputs. Types of indicators would include measures of sustained service delivery and of increased institutional capacity. Measures would address quality, quantity and affordability of services, as well as an institution’s ability to be self-managing and self-sufficient, and to provide a wide array of services. Sample indicators include: 31 • Number and percent increase in surgical and other services (e.g., low vision), in both the immediate and post-grant periods. • Number and percent of grantees providing specified services. • Percent and amount of dollar increase from cost-recovery. • Number and percent trainees still providing services after training. • Percent of patients receiving reduced or free care. In addition to these quantitative variables, qualitative reports, i.e., “stories,” “vignettes,” or “case studies” can also be very effective ways to convey information, and are particularly useful to PVC in its R4 process. IEF-PVC Relationship • PVC should engage in more dialogue and greater involvement with IEF. IEF would welcome this, and it is also necessary to allow PVC to provide policy and program direction, and to address unresolved issues (See Section 6). Such greater involvement may itself strengthen IEF’s results reporting and ensure that such reporting is in a form most useful to PVC. • PVC should consider specifically approving the countries and institutions where the fewer, larger IEF grants will be given under Seeing 2000: Revised and Expanded (i.e., not just as part of a DIP Review). Such a process would promote PVC involvement, and ensure that IEF was working in priority countries for PVC. • PVC should allow and encourage IEF to interpret the earmark more broadly, so that IEF can then provide low vision services and spectacles as part of its programs (as it wants but feels constrained to do). Such guidance would be according to the earmark’s spirit, i.e., to help restore vision to blind children, rather than to its letter, i.e., increasing childhood surgery per se––which often does not restore vision. There are many children in schools for the blind, who could be mainstreamed with low-cost, low-vision services. In this way, more children are likely to have meaningful vision restored or preserved, in a cost-effective manner. (To quote IEF from its own documents: “In reality, there are not large numbers of children with congenital cataracts and glaucoma” [these are the major causes of childhood blindness that are potentially curable by surgery] … and even if they are identified and operated on, the outcomes are poor, even in the United States. …Non-vitamin A-related blindness is uncommon and usually difficult to identify and treat.”) • PVC should engage in dialogue with IEF about which special outcome studies would be most useful to PVC, and then encourage IEF to conduct them. 32 5. LESSONS LEARNED (broader implications for similar programs in different settings) • Overall, a relatively modest level of funding to a PVO (in the $300-400,000/yr range) can be very effective, and can result in measurable, meaningful achievements. • Small grants from the PVC-funded PVO to its NGO partners in developing countries can be quite an efficient and cost-effective mechanism for building networks, increasing leverage and adding capacity. • If the partners are “well-selected” (i.e., if they have strong services and management), and the program is relatively narrow and well defined, the management burden on the PVO granting funds can be relatively low. • Nonetheless, larger grants to fewer organizations, if carefully targeted and designed, can potentially bring greater results in the long run in terms of strengthening an institution’s program management and financial sustainability. • Cost recovery for good quality, affordable and accessible, curative surgery can be achieved. In those situations involving good quality of services, a high caseload, and an empowered, committed organization, cross-subsidization of services for children and the poor is quite feasible. • For such “compassionate capitalism” to work, the right set of circumstances are needed, including: • a high prevalence of a condition (such as blindness). • people’s general willingness to pay for care for that condition. • strong organizational commitment and ability to deliver good quality services (a situation more likely in the private NGO sector than in the public sector). • high aggregate national wealth (e.g., India is a very wealthy country, although it has hundreds of millions of poor people). • South-South assistance can be real, significant and sustainable. In fact, South-North assistance can take place and meaningfully strengthen a PVO’s own programs. • Meaningful data collection requires diligence but is feasible, even in situations where site visits are limited. • Data collection, analysis and synthesis are necessary to demonstrate quantitative and qualitative results, and the contribution of these results to PVC’s attainment of its IRs and SO. 33 6. UNRESOLVED ISSUES There are a number of unresolved “issues” or “strategic considerations” for IEF and/or PVC. They represent unknown aspects of the future program and potential areas for dialogue between IEF and PVC: • Should IEF’s programs emphasize geographic reach (i.e., a widespread network of partners) or absolute disease burden or priority countries? What should the criteria be? To what extent should PVC provide initial programmatic direction? Broad or deep? The number of children born each year in India, 25 million, is roughly equal to the total population of Peru, and is three times the total population of Dominican Republic or Bulgaria (where fewer than 200,000 children are born each year). Bombay alone has 7 times the population of Albania. Every state of India has 60-100 million or more people and a high disease burden, as does Bangladesh and Pakistan. Yet many of IEF’s grants to date have gone to much less populous countries. On the other hand, other eye care organizations are active in South Asia, and to concentrate exclusively there would reduce IEF’s “network effect”. • Should IEF focus on retinopathy of prematurity or more common problems that require lower-tech interventions? With ROP, small numbers of premature infants are served, in relatively developed countries, which may lack adequate means and equipment to monitor continuous oxygen provision (hence the ROP development). In these situations gains can be certain, meaningful and lifelong; on the other hand, spectacles and low vision devices are likely to help more children. • Is the Aravind/Lumbini model of high quality, high volume sustainable services, driven by a strong sense of mission and “compassionate capitalism,” cross￾subsidizing services and serving the poor, transferable/replicable in other regions with different characteristics? • For example, will IEF’s taking on the challenge of hoping to replicate the Aravind model in the much lower human and financial resource setting of Malawi (predominantly public sector services, fewer personnel, lower disease burden in absolute terms and thus smaller “market”) be successful? • Will IEF’s making larger grants to fewer organizations, with a focus on management and cost-recovery “work” (i.e., lead to more, better, and sustained services)? • Are the numbers and skills of IEF’s staff, even with its recent augmentation, adequate to its reoriented strategy and new, more labor-intensive modus operandi, with its emphasis on management technical assistance and improved M&E? 34 • Will the modest scope of IEF’s human and capital resources allow it to attain its desired niche as a technical assistance expert in eye care program management? • Will IEF’s “social enterprise” generate the funds it is projected to generate over the next 5 years? And in general, will IEF be able to diversify and increase its funding? • Will IEF be able to achieve and document service delivery, capacity enhancement and sustainability results of both a quantitative and qualitative nature and link them to PVC’s SO and IRs? ANNEX A. Evaluation Scope of Work FINAL EVALUATION INTERNATIONAL EYE FOUNDATION (IEF) MATCHING GRANT FAO-0158 -A-00-5015-00 I. PROGRAM IDENTIFICATION PVO: International Eye Foundation (IEF), cooperative agreement with USAID/Washington, Bureau for Humanitarian Response, Office of Private and Voluntary Cooperation, Matching Grants Division No. FAO-A-0158-A-00-5015-00 Activity Dates: 09/95 – 02/02 Countries: U.S. Headquarters; subgrants in 16 countries including Bulgaria, Dominican Republic, India, Nepal, and Peru. II. PROGRAM BACKGROUND History: In 1995, IEF was awarded a grant of $1,571,990 through PVC’s Matching Grant Division in support of its Seeing 2000 Program addressing childhood blindness (September 1995 to August 2000). As this grant came in the middle of IEF’s matching grant cycle (August 1993 to August 1996), the additional funds were awarded as a separate cooperative agreement and had no match requirement. When the IEF’s existing matching grant ended in 1996, the IEF carried on with the Seeing 2000 cooperative agreement. In July 2000, the IEF requested and was granted a no-cost extension for a period of 18 months through the end of February 2002. In September 1999, the IEF was awarded a new matching grant (Seeing 2000: Revised and Expanded- $2,000,000 over five years, September 1999 to September 2004) which supports sustainability programming and additional Seeing 2000 sub-grants which will be awarded once the current Seeing 2000 extension is completed in 2002. No external evaluation has been conducted for the current Seeing 2000 cooperative agreement in place from 1995 through 2002. Grant history: Name of Grant: SightReach Matching Grant - Latin America, Africa, E. Europe Dates: 8/31/93 - 8/30/96 extended no-cost 8/30/98 Grant Number: FAO-0158-A-00-3053-00 Source of Funds: USAID/BHR/PVC/MG Contact: Cathy Bowes; t - 703 351 0190; f - 351 0212 Amount: AID Share: $700,000; IEF Share: $563,300; TOTAL: $1,263,300 Description: SightReach “Respack” program provides young ophthalmologists with an ophthalmic equipment package for the establishment of private practices in undeserved areas of 4 nations: “ChildSight” program allows for the provision of sight-restoring surgery for blind and visually impaired children in 6 nations. Status: Completed Note: The “childhood blindness earmark” was established by Congress in 1992. IEF received funds through this matching grant to conduct childhood blindness programming in the six countries supported by the matching grant (Albania, Bulgaria, Eritrea, Malawi, Guatemala, Honduras). Name of Grant: Expansion of Clinical and Surgical Eye Care Services for Children through Support for NGOs Worldwide, Seeing 2000 - Worldwide Dates: 8/24/95 - 8/31/98 Grant Number: FAO-0158-A-00-5015-00 Source of Funds: USAID/BHR/PVC/CS Contact: Cathy Bowes; tel.(703) 351-0190; fax 351-0212 Joy Pentecostes; tel. (202)712-5737; fax (202) 216-3041 Amount: AID share: $1,571,990; TOTAL: $1,571,990. Description: Increase the quality and quantity of ocular surgery performed on children through PVOs/NGOs and charity hospitals. Status: Active on extension. Currently being evaluated. Note: A new concept for supporting childhood blindness activities called Seeing 2000 was submitted to PVC. IEF would award sub-grants of up to $25,000 to PVOs/NGOs and charity hospitals in developing countries through a small grants mechanism. IEF’s philosophy sees a wider universe of organizations in developing countries able to access funding through a small grants mechanism rather than IEF trying to absorb all of the funds through its own programs. Also, not all of IEF’s programs have surgical childhood blindness components and it would not be appropriate to add on such a sub-specialist activity. It would smarter to extend access to funding to programs with pediatric eye care services already in place. Name of Grant: Seeing 2000: Revised and Expanded – Worldwide Dates: 9/28/99 – 9/27/04 Grant Number: FAO-A-0158-A-00-5015-00 Source of Funds: USAID/BHR/PVC/CS Contact: Joy Pentecostes; tel. (202)712-5737; fax (202) 216-3041 Amount: AID share: 2,000,000 TOTAL: $2,717,043 Note: IEF’s new SightReach matching grant consists of several interrelated components supporting eye care institutions: 1) SightReach Management for sustainability planning in three to five countries (Malawi, most likely Guatemala, Egypt and others TBD) that are ready for sustainability planning and cost-recovery programming; 2) SightReach Surgical, IEF’s social enterprise, 3) support for IEF strategic planning, and 4) Seeing 2000for pediatric ophthalmology grants in countries TBD. This programming builds on, but is not limited to, institutions with which the IEF has worked through its earlier Seeing 2000 program (above). This cooperative agreement will continue to provide sub- grants ($25,000 up to $100,000) to pediatric eye care services when the previous Seeing 2000 cooperative agreement ends in 2002. Additional note: In 1992, a Congressional earmark mandated funding of $1 million per year to improve ocular surgery for children in developing countries. BHR/PVC was authorized to manage and distribute these funds. PVOs were requested to compete for the funds under the matching grant program. The earmark targets medical and surgical treatment and did not include vitamin A deficiency control. As pediatric ophthalmology is a sub-specialty and not found widely in developing countries, IEF felt it wise to utilize an RFA process and small grants mechanism allowing flexibility for fledgling pediatric units to design projects that would best support their work. Program goal and purpose: The Seeing 2000 program aims to increase the quantity and quality of ocular surgery performed on children in order to ameliorate childhood blindness. Through the Seeing 2000 Program, the IEF proposes to strengthen national and international non￾governmental organizations and charity hospitals in developing countries to expand and improve their clinical services to blind and visually-impaired children. Small one-to-two-year grants ($5000 to $ 25,000) are provided to qualified organizations whose proposals fit specific but flexible criteria. In the no-cost extension phase, sub-grant amounts are provided ranging from $25,000 to $100,000 depending on need. Availability of Data: Data reported from the following sources will be made available to the evaluation team: • Annual Reports • Internal evaluation reports • Data generated through the IEF's monitoring and evaluation system III. PURPOSE OF THE EVALUATION USAID: This evaluation fulfills the requirements of the USAID/BHR/PVC Matching Grant Program. PVC will use this information in its annual results report, the review of any follow-on proposals and in distilling "lessons learned" for broader application. IEF: This evaluation fulfills the requirements of the USAID/BHR/PVC Matching Grant Program. The PVO will use this information to refine Seeing 2000: Revised and Expanded and incorporate “lessons learned” into program design and strategies. IEF is particularly interested in evaluating the small grants mechanism as it relates to: • Solicitation of NGO partners and the sub-granting mechanism awards vs. traditional PVO managed programs • flexibility for sub-grantees in use of funds • capacity of small vs. medium Vs larger NGOs and charity hospitals to achieve expected results • common denominators which supported successes and/or posed obstacles • IEF’s ability to manage this approach • IEF’s ability to provide technical backstopping where needed • what types of PVOs/NGOs/hospitals should best be targeted to achieve results • what interventions should best be supported to impact on childhood blindness in the developing world • how have the achievements of “centers of excellence” such as the projects in Peru and India faired as opposed to specific activities in other types of programs Local partners (NGOs and charity hospitals): The evaluation process and results will provide local partners with guidance in their future decision-making, continued organizational development, and implementation of health programs. IEF is interested in learning how the grant recipients feel about: • the flexibility in designing their own program, given IEF expectations and guidelines • their capacity to use the funds • their capacity to achieve results • ability to improve services for children with eye disease • the impact this program has had on their general eye care service • the technical backstopping provided by IEF • their satisfaction with the Seeing 2000 program IV. EVALUATOR STATEMENT OF WORK The evaluation team, led by Drs. Jacobstein and Paul, will assess the Seeing 2000 program and institutional elements, providing evidence, criteria for judgement and citing data sources. Appendix A contains a list of specific evaluation questions to be addressed. Questions will be prioritized during the Team Planning Meeting. The team will assess both headquarters and country level programs in Bulgaria, Dominican Republic, India, Nepal, and Peru. An estimate of the percent emphasis or level of effort for each segment of the SOW will be determined during the Team Planning Meeting. A. PROGRAM IMPLEMENTATION 1. Assess progress towards each major objective: a.) Based on the statement of program purpose from the proposal, have the objectives been met? b.) Identify constraints encountered and assess IEF's response to them. PVC is particularly interested in how IEF addressed constraints experienced by sub-grantees in their programming. c.) Identify major successes and why these elements were successful. d.) Assess effectiveness of models, approaches or assumptions that underlie the program. Comment on the possibility or plans to scale-up the approach. 2. Assess the impact on the local partners (NGOs, charity hospitals, local government) in the implementation of the Matching Grant; assess satisfaction of project beneficiaries with project activities. a.) What were the organizational benefits of implementing the Matching Grant? Comment on experience gained, visibility and recognition developed, childhood blindness network developed, ability of IEF to better address childhood blindness within the WHO/IAPB strategies. b.) What organizational and technical skills were built or changed in the local partners? (Consider new skills acquired, new systems and procedures established, quality of relationships with communities, improved program ability to meet needs of beneficiaries, other team building aspects, etc.) c.) What were the organizational challenges in implementing the Matching Grant? 3. Assess progress towards sustainability of sub-grantees. a.) Identify the program elements to be sustained; the sustainability objectives and indicators; the achievements to date; and the prospects for post-grant sustainability. b.) Describe the existence and status of cost-recovery mechanisms, local level financing or other approaches to generate resources to support project operations. 4. Assess the status of strategic partnership(s) with NGOs, community-based organizations, local level government, or charity hospitals. a.) Characterize "partnerships" with local level partners • Roles, responsibilities (decision-making power) • Mechanisms employed (MOU, sub-grant, contract, etc.) • Fiscal autonomy and amount of grant funds directly managed in past year b.) How did the program plan to assess the quality and scope of partner relations? • What was the impact of the partnership on the program? • What change in capacity of local level partner was planned? • What was achieved? • What are the effects of training or resource transfer on the capacity of local partners? 5. Assess the local level partner’s satisfaction with the partnership a.) Has the interface and communications among PVO headquarters and partners been effective? b.) Were the funds transfers procedures effective? Cite the major implementation lessons learned and recommendations B. PROGRAM MANAGEMENT 1. Assess change in the PVO's management capacity (structure and quality of management) as a result of grant. a.) Strategic Approach and Program Planning. 1. What changes have occurred in the organization's capacity for critical and analytic thinking regarding program design and impact? Evidence that program has: • fostered analysis and self evaluation of country programs • conducted periodic review of performance data by project personnel and taken actions as a result of review 2. Are there changes in the capacity of the headquarters to: • manage the planning process • refine the RFA mechanism and award process • address over-arching program issues of - replicability - sustainability - forecasting and strategic planning - organizational development - financial planning and development C. MONITORING AND EVALUATION 1. Assess the capacity of the organization to monitor program performance and measure impact. a.) Is there evidence of: • capacity building • sustainability objectives and indicators • baseline assessment and plans for final assessment • knowledge of and use of impact and performance measurement techniques • a management information system (MIS) to consolidate, analyze and interpret project data • transfer of skills and capacity to local partners D. FINANCIAL MANAGEMENT 1. Were adequate financial monitoring systems in place to verify program, revenue, operating and financial expense, other inputs and outputs? 2. Has the program leveraged additional resources (beyond the match)? 3. Is there an estimate of cost-effectiveness of the program? E. INFORMATION 1. Comment on the utility and timeliness of PVOs required reports 2. Comment on the organization’s (headquarters) public outreach activities F. LOGISTICS Comment on the adequacy and timeliness of PVOs material inputs. G. SUPERVISION Assess if there were sufficient staff and consultants with the appropriate technical and management skills to oversee program activity at both headquarters and in the field programs. H. HUMAN RESOURCE DEVELOPMENT Did the PVO assess and address staff training needs and strengthen the organization and local partner professional or technical capacity? Cite the major lessons learned and management recommendations V. EVALUATION METHODS Evaluation approach, appropriate methodology, instruments and tools are to be determined with evaluator. Approach: Conduct a Team Planning Meeting (TPM) to refine and consolidate the purpose, objectives and outputs into a set of results-oriented objectives. Agree upon a set of appropriate indicators against which the evaluation will assess achievement of project results. Methodology: The evaluators will determine data collection approaches and instruments to be used, document data sources, and provide a copy (electronic or paper) of all primary data collected and analysis performed VI. TEAM COMPOSITION AND PARTICIPATION Team Composition - External Evaluator, pediatrician and public health physician (Team Leader hired by USAID) - External Evaluator, pediatric ophthalmologist (hired by IEF) - Matching Grant IEF HQ backstop - USAID/BHR/PVC/MG Program representative - Local project staff will participate in evaluation, as appropriate Participation: The evaluation team will consist of an external consultant who will also act as the team leader, another external consultant, IEF HQ staff members, and USAID/BHR/PVC/MG representative. IEF and Matching Grant field staff and local partners will be involved in discussions during an evaluation de-briefing (in each country) and will facilitate site visits. Translators will be provided to evaluation team members in each country (as necessary). The evaluators will collect and analyze data, and write the draft evaluation report. The team leader will have primary responsibility for revising the draft evaluation report. Other team members will facilitate the data collection process and be a source of information. The final report will be provided to AMA for distribution and feedback from the PVO and PVC. The team leader will lead the team in assessing the level of progress made by IEF and local partners in achieving results and will closely examine determinants of relative progress in each country. Progress toward objectives will be gauged according to expected achievement of selected indicators and benchmarks set by each country. Roles and responsibilities of the evaluators: - Review all project documents (to be provided by IEF and USAID) prior to site visits. - Participate in Team Planning Meeting (Collaborate with the evaluation team in preparing evaluation methodology and instruments prior to evaluation.) - Conduct interviews and focus group discussions and facilitate discussions and other assessment activities among selected stakeholders. - Lead follow-up discussions among evaluation team and Matching Grant Project staff. - Draft evaluation results and recommendations and present to evaluation team and Matching Grant Project staff. - Produce evaluation report as per USAID/BHR/PVC guidelines. - Incorporate comments from PVC and IEF. - Submit report by deadline. (Note: The team leader will have primary responsibility for revising the draft evaluation report) VII. TENTATIVE SCHEDULE The evaluation will be conducted during the months of October to December following an evaluation Team Planning Meeting (TPM) in late October. The Team Leader’s participation in this evaluation will be 39 days. Evaluation activities (including travel) will be planned as follows: Team Planning Meeting at IEF headquarters in Bethesda (One day- Oct.26) • Team planning/team briefing • Orientation and logistics • Evaluation methods and tools refinement Meeting with IEF staff Assessment of HQ program support functions (administrative and management review; headquarters activities under the matching grant) (One day- Oct. 27) Field Visits • Assessment of Project in India (2 sites) (Eight days) Conduct evaluation as planned. Debrief local project staff on evaluation findings • Assessment of Project in Country 2 (Six days) Conduct evaluation as planned. Debrief local project staff on evaluation findings • Assessment of Project in Country 3 (Six days) Conduct evaluation as planned. Debrief local project staff on evaluation findings: Debriefing Team debrief at IEF headquarters (One day) Reporting • Prepare Draft Report to submit to AMA Technologies, PVC and IEF (Ten days) • Report Review by PVC and IEF. (The evaluators are not involved at this time) • Final Report Preparation/Submission (team leader) (6 days). Evaluation leader reviews comments from PVC and IEF. Evaluation team leader submits final to AMA Technologies, PVC and IEF. Due date for final report is Jan. 1, 2001. Under no circumstance will the due date be extended beyond Jan. 22, 2001. VIII. REPORTING AND DISSEMINATION REQUIREMENTS The SOW will serve as the outline of the report. Delivery schedule: final report is submitted as scheduled. USAID/BHR/PVC will review documents as stated. APPENDIX A SEEING 2000 EVALUATION QUESTIONS I. PROJECT DESIGN 1. For the activities you proposed, how has the grant helped you achieve your goals? 2. Seeing 2000 was designed to support curative rather than preventive care, how did this fit in with your institution’s overall public health approach to eye care? 3. What elements that were not Seeing 2000 objectives, would you have like included in your project? II. PLANNING/MONITORING/EVALUATION 1. How successful do you think your project performed in meeting its objectives? 2. How was the project design developed? 3. Who was responsible for the project’s design, implementation and reporting? Were they appropriate for the task? 4. Were Seeing 2000 project reports used internally? Why, why not? III. SUSTAINABILITY 1. What are the critical factors needed to sustain pediatric eye care services? 2. What are your biggest obstacles to providing pediatric eye care services? 3. How would you define the term “sustainable” in relation to your eye care services? 4. Identify the program elements to be sustained; the sustainability objectives and indicators; the achievements to date; and the prospects for post-grant sustainability. 5. Are you seeing more children now than before Seeing 2000 funding? If yes, how has Seeing 2000 funding helped increase patient volume? If no, why? 6. Are you able to treat more ocular diseases in children now than before Seeing 2000 funding? Why? Why not? 7. Considering all eye care services within your clinic, where do pediatric services fall in your list of priorities? 8. If you were to get another Seeing 2000 grant, what would be your priorities? 9. Describe the existence and status of cost-recovery mechanisms, local level financing or other approaches to generate resources to support project operations. IV. PARTNERSHIPS 1. Has your organization’s credibility been enhanced by Seeing 2000 funding? (For example, within the government, the hospital, other local NGOs, other eye care providers, etc.) 2. Please give examples of how being in the Seeing 2000 network has been helpful. 3. How has the Seeing 2000 grant allowed you to strengthen partnerships with others (MOH, donors, etc.)? 4. Grant was locally designed and implemented, would you have liked more guidance structure from the IEF in project design and implementation? V. LESSONS LEARNED BY GRANTEES 1. How has the funding been helpful in developing pediatric eye care services at your institution? 2. Can you identify two specific activities that have been strengthened as a result of Seeing 2000 funding? 3. What new skills have you acquired because of this grant? Clinical? Surgical? Administrative? Financial? 4. With new expanded childhood blindness services, how has this affected other eye care services (positive and/or negative)? How has this affected other non-eye care services (use of beds, operating room, waiting area, etc.)? 5. Looking back, what would you do differently in designing your project? 6. If you had not received Seeing 2000 grant funding, where would your pediatric service be today? 7. Were there any unexpected benefits of the Seeing 2000 funding? VI. LESSONS LEARNED BY THE IEF 1. How did the IEF provide administrative support to the project? 2. The proposal guidelines emphasized that preference would be given to funding clinical training and equipment. In designing and implementing your project activities to increase the quantity and quality of pediatric eye care services, did you feel that unnecessary limits were placed by the proposal guidelines? 3. What recommendations do you have for the IEF in preparing future proposal guidelines for projects supporting pediatric eye care service delivery? 4. Do you feel the grant limit of $25,000 for 12-24 months of project activities was reasonable? ANNEX B. Evaluation Team Itinerary October 26-27, 2000 International Eye Foundation Headquarters, Bethesda Team Planning Meeting; Meeting with PVC, IEF Dr. Roy Jacobstein, Dr. T. Otis Paul, Ms. Lori Carruthers October 28-November 1, 2000 Evaluation site visit: Aravind Eye Hospital, Madurai, INDIA Dr. Roy Jacobstein, Dr. T. Otis Paul, Ms. Lori Carruthers November 2-4, 2000 Evaluation site visit: L. V. Prasad Eye Institute, Hyderabad, INDIA Dr. Roy Jacobstein, Dr. T. Otis Paul, Ms. Lori Carruthers November 4-6, 2000 Delhi, INDIA and Kathmandu, NEPAL November 7-10, 2000 Evaluation site visit: Lumbini Rana-Ambika Eye Hospital, Bhairhawa, NEPAL Dr. Roy Jacobstein, Ms. Lori Carruthers November 11, 2000 Evaluation site visit: Tilganga Eye Centre, Kathmandu, NEPAL November 12-13, 2000 Visit, debrief USAID Mission in Delhi, INDIA Dr. Roy Jacobstein, Ms. Lori Carruthers November 14-17, 2000 Evaluation site visit: Bulgarian Eye Foundation, Sofia, BULGARIA Dr. Roy Jacobstein, Ms. Lori Carruthers November 27 -29,2000 Evaluation site visit: Hospital Elias Santana, Santo Domingo, DOMINICAN REPUBLIC Dr. T. Otis Paul, Ms. Lori Carruthers November 30 - December 2, 2000 Evaluation site visit: Instituto de Educacion y Prevencion en Salud Visual, Lima, PERU Dr. T. Otis Paul, Ms. Lori Carruthers ANNEX C. List of Persons/Institutions Contacted Aravind Eye Hospitals, Madurai, Tamil Nadu INDIA Dr. P. Namperumalsamy, Director Dr. G. Natchiar, Joint Director Dr. P. Vijayalakshmi, Pediatric Ophthalmologist & Seeing 2000 Project Director Mr. R.D. Thulasiraj, LAICO, Executive Director Ms. Preethi Pradhan, LAICO, Training Faculty Mr. P. Bala Krishnan, Aurolab, Managing Director L. V. Prasad Eye Institute, Hyderabad, Andhra Pradesh, INDIA Prof. Balasubramanian, Acting Director/ Director - Research Centre Dr. Anil K. Mandal, Consultant - Glaucoma Services Dr. M. Sarfaraz A. Khan, Associate Director & Director Rehabilitation & Low Vision Services Dr. Lalit Dandona, Associate Director & Director, International Centre for Advancement of Rural Eyecare Dr. Taraprasad Das, Associate Director & Director, Eye Hospital Mr. D. Nagarajan, Honorary Consultant Dr. B.Venkateshar Rao, Pediatric Ophthalmologist Dr. Subhadra Jalali, Consultant - Retina & Vitreous Services Dr. Santosh Honavar, Consultant - Glaucoma Services Dr. Murali K. Aasuri, Consultant - Cornea & Anterior Segment Services Dr. C.V.M. Reddy, Anesthesiologist Mrs. Vijaya Ramam, Deputy Director - Administration Ms. Sharmila, Consultant - Rehabilitation Services Ms. Uma Nath, Administrator USAID, New Delhi, INDIA Victor Barbiero, Ph.D., Director, Office of Population, Health and Nutrition Bethanne Moskov, Team Leader, Infectious Disease, Office of PHN Pashev Center for Sight, University for Hospital "St. Anna," Sofia, BULGARIA Professor Dr. Petja Vassileva, Director, Center for Sight and Chairman, Department of Ophthalmology, Hospital "St.Anna" Dr. Dimitar Petrov, Senior Assistant Professor of Ophthalmology Dr. Rossitza Lolova, Assistant Professor of Ophthalmology Miss Daniela Sugareva, Secretary, Foundation "Sight for All" Lumbini Rana Ambika Eye Hospital, Bhairahawa, NEPAL Dr. S. K. Shrestha, Director Mr. Him Kant Adhikari, Hospital Administrator Dr. A. Deuja, Ophthalmologist Mr. C. Karn, Finance Officer Mr. H. L. Dhami, Outreach Coordinator Dr. S. Tenzing, Staff Ophthalmologist Tilganga Eye Centre, Kathmandu, NEPAL Dr. Sanduk Ruit, Medical Director Mr. B Bhagirath Baniya, Administrator Dr. Suman S. Thapa, Staff Ophthalmologist Hospital Elias Santana, Santo Domingo, DOMINICAN REPUBLIC Mrs. Janice Hunter, General Director Dr. Juan Batlle, Chief of Ophthalmology Ms. Bienvenida Suero, Blindness Prevention Program Coordinator Dr. Joaquin Sosa, Pediatric Ophthalmologist and Academic Coordinator Instituto de Educacion y Prevencion en Salud Visual, Lima, PERU Dr. Luz Gordillo, Pediatric Ophthalmologist Ana Cisneros, Health Educator Rosa Cisneros Canales, Social Communications Miguel Saravia, Administrator ANNEX D. References Consulted USAID Bureau for Humanitarian Response, Office of Private and Voluntary Cooperation. Matching Grant Program, Request for Applications, FY 2001, August 25, 2000. --. Results Review FY 1999 and Resource Request FY 2002, April 1, 2000. Center for Development Information and Evaluation. Performance Monitoring and Evaluation Tips, 1998, #12, “Guidelines for Indicator and Data Quality”. --. Performance Monitoring and Evaluation Tips, 1996, #6, “Selecting Performance Indicators”. Cooperative Agreement Between USAID and IEF, Seeing 2000, 8/95-8/98, August 25, 1995; Modification of Cooperative Agreement and Extension to 8/00, 9/26/96; Extension of Cooperative Agreement, through 2/28/02. Cooperative Agreement Between USAID and IEF, Seeing 2000: Revised and Extended, 9/28/99-9/27/04, September 28, 1999. IEF IEF Annual Reports, 1996, 1997, 1998 and 1999. --. Annual Report to USAID, #1, 8/25/95-10/31/96, March 12, 1997. --. Annual Report to USAID, #2, 11/1/96-10/31/97, January 1998. --. Annual Report to USAID, #3, 11/97-10/98, October 2000. --. Annual Report to USAID, #4, 11/98-10/99, October 2000. --. Fact Sheet – 2000. --. Final Report, Matching Grant of 8/93-8/98, October 1998. --. “‘Seeing 2000,’ Phase Two,” Detailed Implementation Plan. --. “Seeing 2000,” Request for Application [for grantees]. --. Strategic Business Plan, 3/2000. --. Strategic Plan, Annual Report, 10/1/99-9/30/00. IEF Grantees Aravind Eye Hospital: 1976-1996. Aravind Eye Hospitals Activity Report, 1999. Aravind Eye Hospitals: “Capacity Building Projects for Lions, Sight Savers & CBM Supported Eye Hospitals, Report, 1994-1999. LV Prasad Eye Institute: Into the Next Millenium. LV Prasad Eye Institute: Patient Care Statistics at a Glance, April 1987-March 2000. LV Prasad Eye Institute: Achievements, 1987-1998. LV Prasad Eye Institute: Annual Activity Report, 4/99-3/00. LV Prasad Eye Institute: Jast V Ramanamma Children’s Eye Care Centre, Activity Report, 12/97-5/99. “Lumbini Eye Care Project, Nepal and Northern India,” Green, D. and Le Clere, W., in Finanacial Sustainability Strategies for NGOs, Institute for Development Research, Boston, 1999. Malawi Eye Care Project, Report of LAICO Management Consulting Team (activities during 1999-2000), to IEF. Project Summary Sheets, all grantees. Tilganga Eye Centre, A Brief Insight. Tilganga Eye Centre, Corporate Profile. Other References Green, D. “Compassionate Capitalism,” IAPB News, #21, 1998. House of Representatives. Foreign Operations … Appropriation Bill, pp. 64-65, 6/12/91 [the “earmark” language, HR 2621]. --. Foreign Operations … Appropriation Bill,” pp. 16, 7/21/98. IAPB News. “Seeing 2000, A Program to Address Childhood Blindness,” Vol 23, 9/98. “Opthalmology in India,” Rao, G. N., Arch. Ophthalmol, 118, 1431-1432, Oct. 2000. “Preventing Blindness in Children,” Report of WHO/IAPB Scientific Meeting,” LV Prasad Eye Institute, Hyderabad, April 13-16, 1999. “A Retrospective Study of Low-vision Cases in an Indian Tertiary Eye-care Hospital,” Khan, A.A., Indian J. of Ophthalmol., 48,3, September, 2000. ANNEX E. 11/17/00 Table 1. Partner Organizations, Grant Years and Amounts Project Period Obligated Disbursed 1 Aravind 8/1/96-7/31/97 $25,000 $25,000 2 Univeridad Catolica de Chile 8/1/96-7/31/97 $11,000 $11,000 3 LV Prasad Eye Institute, India 8/1/96-7/31/97 $25,000 $25,000 4 Lions SightFirst, Malawi 8/1/96-7/31/97 $25,000 $25,000 5 Lumbini, Nepal 8/1/96-10/31/97 $25,000 $25,000 6 Lady Reading, Pakistan 8/1/96-7/31/97 $25,000 $25,000 7 Mount Sion, Papua New Guinea 8/1/96-7/31/97 $25,000 $25,000 8 Bulgarian Eye Foundation 8/1/96-1/31/98 $25,000 $25,000 9 ICEH South Africa 8/1/96-10/31/96 $17,000 $13,800 10 Roosevelt, Guatemala 2/1/97-1/31/98 $25,000 $25,000 11 Robles, Guatemala 3/1/97-8/31/97 $25,000 $6,400 12 Elias Santana, Dominican Republic 2/1/97-5/31/98 $25,000 $25,000 13 Al Shifa, Pakistan 6/1/97-5/31/98 $25,000 $25,000 14 Gwanda, Zimbabwe 6/1/97-5/31/98 $25,000 $25,000 15 IEPSV, Peru 6/1/97-5/31/98 $25,000 $25,000 16 Tilganga, Nepal 6/1/97-5/31/98 $25,000 $25,000 17 Lady Reading, Pakistan (2nd) 8/1/97-7/31/98 $25,000 $25,000 18 LRBT, Pakistan 9/1/97-8/31/98 $25,000 $25,000 19 Blind School Project, Egypt 9/1997-11/31/98 $25,000 $25,000 20 Lumbini, Nepal (2nd) 11/1/97-10/31/98 $25,000 $25,000 21 Foresight, Papua New Guinea 7/30/97-11/30/00 $15,000 $15,000 22 LV Prasad, India (2nd) 4/1/98-3/31/99 $10,000 $10,000 23 Lions SightFirst, Malawi (2nd) 2/1/99- $50,000 $15,000 24 AIIMS, India 9/03/99-5/2/01 $25,000 $25,000 25 Orbis, Ethiopia 5/1/00 - $25,000 26 Phillipines 5/1/00 - $25,000 $13,500 27 IEPSV, Peru 6/1/00 - $75,000 $4,500 28 Other Field $33,300 29 VOSA Guatemala* 8/1/00 $25,000 30 HFH Albania* 8/1/00 $25,000 31 Bulgaria* 8/1/00 $25,000 32 Fundacion Ayudamos A Ver Nicaragua* 8/1/00 $36,000 33 Lighthouse/ LVP, India* 8/1/00 $60,000 34 Estimated balance to disburse SGs 1/1/01 $69,000 Total Sub Grants obligated/ disbursed $943,000 $572,500 * Reviewed and programmed 2000/01