NGO Service Delivery Program (NSDP) Evaluation Report Evaluation Team Joy Riggs-Perla John Holley Ahmad A.N. Neaz March 2006 2 NSDP Evaluation Report Table of Contents Part I. Assessment of NSDP Page Table of Contents 2 List of Acronyms 4 Executive Summary 5 I. Brief Project Description and Background 7 A. History of NSDP 7 B. Summary Description of Project 8 C. Context and Role within USAID Program 9 D. Timing and Purpose of the Evaluation 9 II. Evaluation SOW and Methodology 9 A. Key elements of SOW 9 B. Summary of Methodology 10 III. Assessment of Progress 10 Objective 1: Expand Coverage and Quality of ESP Services 10 Offered by NGOs A. Family Planning Issues and Challenges 10 B. Child Health Issues and Challenges 12 C. Maternal Health Issues and Challenges 13 D. TB 14 E. HIV/AIDS/STI/RTI 14 F. Limited Curative Care 15 G. Rationalizing of Service Coverage 15 H. Summary of Service Coverage Issues 15 Objective 2: Increase Use of Service, Especially by the Poor 15 A. BCC and Creating Demand 15 B. Community Level Activities 16 C. Reaching the Poor 16 3 Objective 3: Increase NGO Capacity 17 A. Quality Improvement and Technical Capacity 17 Development B. Measuring Sustainability 18 C. Strategies and Activities Supporting Sustainability 20 1. Serving the Poor 20 2. Cost Recovery Lessons 21 3. Cost of Services 21 4. Financial Support Issues 22 D. Bottom Line on Sustainability 23 Objective 4: Influence GOB Policy, Coordinate with Donors, 24 Improve Environment for NGOs. NSDP Monitoring and Evaluation System 24 IV. Implementation Issues and Challenges 25 A. Adherence to Activities and Requirements of 25 Cooperative Agreement B. Efficiency Issues: Project Staffing and Use of Resources 26 C. Problems and Issues Affecting Implementation 28 D. Management Issues for USAID 28 V. Summary of Recommendations for Strengthening Implementation 29 Part II. Recommendations for the Future VI. Current Landscape 33 A. Realities and Challenges of Health Sector in Bangladesh 33 B. Positioning NGO Health Services in Bangladesh 34 C. USAID’s Niche 35 VII. Future Recommendations 35 A. Characteristics of New Project 35 B. Lessons from NSDP for the Future 30 Annexes 1 List of Individuals Interviewed, Sites Visited 2 Full SOW 3.Sustainability Analysis, JHolley 4 List of Acronyms ADB Asian Development Bank ARI Acute Respiratory Illness BAP Bangladesh AIDS Program BCC Behavior Change Communication BCC/M Behavior Change Communication/Marketing CA Cooperating Agency (USAID) BCCP Bangladesh Center for Communications Program DHS Demographic and Health Survey DOTS Directly Observed Therapy Short course EC Emergency Contraception EOC Emergency Obstetric Care ESP Essential Services Package ICDDR.B International Centre for Diarrheal Disease Research, Bangladesh GOB Government of Bangladesh HPSP Health and Population Sector Program HNPSP Health, Nutrition and Population Sector Program IMCI Integrated Management of Childhood Illness LA Least Advantaged LCC Limited Curative Care MCP Mexico City Policy MIS Management Information Systems MOCAT Modified Organizational Capacity Assessment Tool MOH Ministry of Health MR Menstrual Regulation NGO Non Governmental Organization NIPHP National Integrated Population and Health Program NSDP NGO Services Delivery Program NSV Non-Surgical Vasectomy OCAT Organizational Capacity Assessment Tool ORT Oral Rehydration Therapy PAC Post Abortion Care PLTM Permanent and Long Term Methods of Family Planning QMS Quarterly Monitoring System RDF Revolving Drug Fund RTI Reproductive Tract Infection SMC Social Marketing Company SOAG Strategic Objective Agreement STI Sexually Transmitted Infection TB Tuberculosis UPHCP II Urban Primary Health Care Project II URC University Research Co. USAID United States Agency for International Development WB World Bank WHO World Health Organization 5 Executive Summary The NSDP Evaluation is divided into two parts. The first section of this paper is an assessment of project implementation to date, with an emphasis on issues surrounding NGO capacity development and sustainability of the program. It includes an assessment of the following topics: technical capacity development, organizational and financial capacity, policy level achievements, the identification of project implementation issues, and recommendations for improving implementation during the final year. Part II of the paper looks the current landscape with respect to the role NGOs in the health care system, and makes recommendations for USAID’s future program with NGOs. The complete SOW is found in Annex 2. Part I: NSDP has demonstrated solid progress in expanding essential family planning and health services to the population in a catchment area of about 20 million poor urban and rural residents in the six Divisions of Bangladesh. Reliable data are available on service coverage from the MEASURE Evaluation surveys, two of which occurred during the NSDP project period. The current report draws on that information, NSDP service statistics and information gathered from the clinic visits and interviews to make judgments about the challenges facing the project in each of the Project’s Objectives. Key findings and recommendations are summarized below. NSDP needs to place high priority during the remaining period in the project on several programmatic issues. It should work proactively with CA partners, the MOH and its NGOs to address the problems that are causing a decline in utilization of long term and permanent family planning methods. Several recent pilot programs aimed at addressing more effectively ARI treatment and improving maternal and newborn health need to be expanded to have broader impact. Finally, the ideas in the Cooperative Agreement about “rationalizing” service coverage need greater attention, especially with the advent of the ADB-supported UPHCP II project and the upcoming WB contracting with NGOs. The “Smiling Sun” branding is excellent and provides the basis for expanding a true network of clinics offering high quality services. Clinic level marketing of services, however, is limited and needs more emphasis. Innovative community and clinic partnerships piloted by some NGOs should be shared and expanded if possible. NSDP NGOs have been successful in reaching the poor but need to expand efforts to identify the poor and expand creative financing mechanisms. Technical and managerial capacities have been strengthened among the NGOs. The one-size-fits-all approach to NGO staffing in the first part of the project has created high overhead costs. Greater efficiencies can be achieved with more flexibility in the final round of grants. In terms of financial sustainability, up to 30% of costs of NGO clinics may need to be subsidized from some source, not necessarily from USAID. Support already comes from government and should continue. At the same time, other donor financing opportunities are expanding. With managerial and financial flexibility, cost can be reduced, services expanded and program revenues used more effectively to 6 promote sustainability of NGO services. In order to stimulate increased cost-recovery and a sense of ownership, it is critically important for USAID to allow the NGO’s much greater managerial autonomy and financial flexibility, particularly in reference to the use of program income. The best avenue for engaging MOH at the policy level is within the context of the HNPSP. For the remaining period of NSDP, much can be done to share NSDP experience, technologies and systems with MOH staff who will be managing the new NGO contracts Part II: USAID must make it clear to NGOs that, as of the end of the next project, it will no longer be supporting the operating costs of NGOs. Dependence on USAID must end. Declining grants must be a feature of any follow on project. A new project should be developed to ensure the survival of a locally-managed NGO network of essential family planning and health services. The number of NGO’s in the network should be smaller than NSDP, preferably without losing health care delivery capacity. NGO’s which have proven unresponsive to NSDP assistance should be encouraged to seek other funding sources or simply dropped. Smiling Sun needs to become institutionalized as a Bangladeshi network that survives beyond USAID funding and is not associated with any particular donor in the future. Clinics and NGOs not previously associated with Smiling Sun should be provided the opportunity to join as a way of broadening the network. The NSDP network will be critical in the long-run since the large urban NGO’s should eventually become sustainable, and able to cross-subsidize the rural NGO’s. A key element of the new project should be the creation of a Smiling Sun franchise organization capable of providing the managerial and technical services currently being provided by NSDP. There must be an institutional incentive to belong to a network. The elements of a franchise should include the following: (1) the ability to monitor and enforce standards that ensure quality of services, (2) shared costs for marketing and maintaining the network of clinics, (3) a system to cross-subsidize services for the poor that is managed above the clinic and even NGO level, and (4) pooled procurement of drugs and commodities for bulk purchasing. TA should be integrated and feature strong project leadership under a simplified management structure. Beyond grant management, areas of focus should include continued and strengthened support in financial management and planning, marketing, both the Smiling Sun franchise, and individual NGO’s and clinics, quality assurance, management of revolving drug funds, and capacity development of the local franchise managing organization. The project management budget should not be tied to the amount of the grants. The new project should gradually replace grant monies by cost- 7 recovery monies. This allows USAID an exit strategy by stimulating NGO’s to increase cost-recovery. Part I. Assessment of NSDP I. Brief Project Description and Background A. History of NSDP History has played a significant role in the evolution of the NGOs currently involved in the NSDP project. Figure 1 depicts in part the various mechanisms which have led to the NSDP: Figure 1 Evolution of the NSDP Mar Sept 97 02 06 07 Year: Rural Service Delivery Project Existing NGOs NSDP Urban Family Health Partnership National Integrated Population & Health Program Pop Programs (USAID) GOB: Health & Population Services Program Health Nutrition Pop Prog USAID SOAG: In 1997, the Government of Bangladesh (GOB) created the Health and Population Sector Program (HPSP) which evolved into the Health, Nutrition and Population Sector Program (HNPSP) in 2004. USAID’s cooperation agreement SOAG (NIPHP) with that program began about the same time and continues as the defining agreement with the GOB. Most of the NGO’s participating in the NSDP existed prior to 1997, some 20-30 years earlier, created to pursue some socially useful purpose. Some participated in USAID’s community-based population programs. As part of the HPSP, USAID’s NIPHP consisted of two NGO support projects: the Rural Service Delivery Project and the Urban Family Health Partnership. The primary purpose of these projects was to 8 improve the service delivery of health and population programs through NGOs. The program included charging service fees and attention to quality of care. It is important to recognize that initially (1983-96) there were apparently over 100 NGOs involved. Their inability to respond and the difficulty of supporting so many individual organizations led to the reduction to 41 originally included in the NSDP project. That number has been further reduced to 33 due to non-compliance with the MCP and in two cases involving financial irregularities. In an effort better integrate and manage the health and populations services provided by NGOs, the NSDP was created to support the rural NGOs as well as many of the urban NGOs. The general purpose for the creation of the NSDP was to do as the name implies: make the NGOs and the services they offered more independent of donor support and more sustainable. Thus, following years of complete donor support, they began expanding their activities to include 18 of the 24 services in the Essential Services Package as defined by the GOB; and charging for some of those services. B. Summary Description of Project NSDP was awarded as a Cooperative Agreement to Pathfinder in 2002 with a total budget of $60 M from USAID and a cost share of $2.237 M from non-US federal sources. The NSDP is a consortium which also includes: CARE Bangladesh, Emerging Markets Group, IntraHealth International, Research Triangle Institute, Save the Children USA and University Research Co. BCCP is also a full partner although not originally listed as such in the Cooperative Agreement. To date Pathfinder International has reported spending roughly $ 41 M from USAID and $2.080 M in cost share as of March 2006. At the clinic level, GOB contributions to the project are substantial and in the form of contraceptives, vaccines, TB drugs, and some EPI equipment. These contributions also form part of the SOAG agreement with the Ministry of Health. The vision of NSDP as described in the Cooperative Agreement was to “enable Bangladeshi NGOs to become technically and managerially self-sufficient in the provision of essential health services and to maximize NGO access to non-USAID funding for essential service delivery”. (Cooperative Agreement, Program Description). The project has four primary objectives as follows: 1. Expand the range and improve the quality of ESP services provided by NGOs at the clinic and community level. 2. Increase the use of ESP services provided by NSDP NGOs, especially the poor. 3. Increase the capacity of NGOs to sustain clinic and community-based service provision, institutionally and financially. 4. Influence GOB policy in coordination with other donors, to expand the role of NGOs as providers of ESP services within the national health system. 9 NGOs supported by NSDP cover a catchment area of roughly 20 million people in urban and rural areas (17% of the population of Bangladesh) with 33 NGOs, 317 static clinics, 8,200 satellite clinics and 6,200 community workers (depot holders). C. Context and Role within USAID program The NSDP is a major portion of the current USAID program, roughly a third of the annual budget in 2005 for population health and nutrition. In addition to NSDP, USAID funds three other bilateral projects: Social Marketing Company (SMC), Family Health Research Project with ICDDR,B(which is soon coming to an end) and the FHI Bangladesh AIDS Program (BAP). USAID also supports a number of other important programs through field support including contraceptive procurement, and activities to help strengthen GOB family planning, health and nutrition programs in the areas of adolescent reproductive health, improvements in clinical family planning methods, strengthening drugs management systems, research and so on. D. Timing and Purpose of the Evaluation The NSDP was scheduled to end in September of 2006 but will be provided an extension until September 2007. The funding required will not exceed the amount originally authorized in the Cooperative Agreement. The purpose of this project assessment is to provide USAID with (1) an analysis of progress to date, with an emphasis on capacity development and sustainability of services provided by the NGOs, (2) recommendations for strengthening implementation during the remainder of the project life, and (3) recommendations regarding possible future programs involving NGO service delivery. II. Evaluation SOW and Methodology A. Key elements of SOW The evaluation scope of work was designed to look primarily at the issues surrounding NGO capacity development and sustainability of the program. The areas of focus included: technical capacity development, organizational and financial capacity, policy level achievements, the identification of project implementation issues, and recommendations for improving implementation during the final year. The SOW also requested a look the current landscape in terms of the role NGOs in the health care system, and recommendations for the future program with NGOs. The complete SOW is found in Annex 2. USAID has also been funding an on-going program with the MEASURE Evaluation Project which, independently of NSDP, conducts surveys since 1998 of service coverage within the USAID supported NGO project and non-project areas. Two of the most recent surveys were within the NSDP project period (2003, 2005). The last 10 survey report documented in detail the various accomplishments of NSDP on service delivery. B. Summary of Methodology The evaluation team was composed of three individuals. The first part of the evaluation, conducted from February 7 to March 7, was a detailed assessment of sustainability. This report is available as a separate document (see Annex 3). From February 27 to March 26, two additional evaluators completed the remaining elements of the evaluation. This document represents a consolidated report by the entire team. The team visited three of the six divisions in Bangladesh to talk with NGO headquarters staff and to view activities at the clinic and satellite clinic level, selecting NGOs and clinics with varying levels of capacity. In addition the team undertook an extensive list of interviews of USAID and NSDP project staff, MOH officials, NGO management and service delivery personnel, other donors and USAID CA partners. (See Annex 1 for a full list). A preliminary presentation of findings was made to USAID on March 19, after which this report was completed. The team also reviewed many existing documents and data sources including the 2004 DHS, the MEASURE Evaluation Survey reports, NSDP quarterly reports and service statistics, the NSDP 2006 Work Plan and many other publications. The Cooperative Agreement with Pathfinder International and the USAID strategic framework were also key documents used to assess progress. III. Assessment of Progress Given the reliable data that exist on service coverage by virtue of the MEASURE Evaluation surveys, the current report does not repeat that information but rather uses those data, NSDP service statistics and information gathered from the clinic visits and interviews to make judgments about the challenges facing the project in each of the Project’s Objectives. The information below should not be viewed as comprehensive but rather an attempt by the evaluation team to analyze lessons learned rather than describe and summarize the wealth of information that already exists. As requested by USAID, the focus is on problem areas and implementation issues, some of which are important for improving implementation during the remaining 18 months in NSDP and some of which are relevant to recommendations for the future. The following sections look at issues within each of the four project objectives. Objective 1: Expand Coverage and Quality of ESP Services Offered by NGOs A. Family Planning Issues and Challenges The family planning services offered at either static clinics, satellite clinics or depot holders include pills, condoms, IUDs, injectables, Norplant, NSV, tubectomy, and 11 emergency contraceptive pills. In terms of coverage of family planning services, the MEASURE Evaluation surveys, particularly in looking at the data from 2003 and 2005, show that the increases in contraceptive use in the project areas is greater than the non-project areas, led by increases in use of pills and injectables. The market share of clients receiving family planning services from NSDP clinics is also increasing over time and is evident in both urban and rural areas, although the total market share of NSDP NGOs is smaller in urban areas most likely due to the availability of many other providers. In urban areas, the NSDP project areas began with substantially lower use rates for modern contraceptives than the urban non-project areas, an indication that the project was targeted to serve more disadvantaged areas and is helping those areas to catch up. However, the most rapid growth in the place people obtain services in urban areas is pharmacies. The share of services provided by government facilities is declining in both rural and urban areas. Clearly, NSDP supported NGOs are expanding their coverage of the family planning services provided to the target population. This is also reflected in the increases in patient contacts reported by the project MIS for family planning services. However, this good news is tempered by a decline in use of clinical methods. The demand for long term methods and sterilization services is decreasing in general in Bangladesh for reasons not completely clear. (BDHS 2004) NSDP has made a substantial investment in ensuring that NSV (35% of clinics) and minilap (16% of clinics) are available. Tubectomy training alone costs the project an average of Taka 25,000 per trainee. There has been a significant loss of this training investment due to problems with physician retention, however even where services are available, the numbers of clients for permanent and long term methods is declining. The volume of services provided is surprisingly low. The figure to the left shows the trends in numbers of service contacts for each of the permanent and long term methods. Project staff attribute this decline to the client compensation provided by government facilities which is not available through NSDP clinics. However, TREND OF PLTM IN NSDP CLINICS 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 15,000 16,000 17,000 18,000 19,000 20,000 2003 2004 2005 FISCAL YEAR NO OF CLIENTS NSV Tubectomy IUD Norplant 12 IUD and Norplant use are declining in NSDP clinics despite the fact that little if any reimbursements to clients are provided by government. Among the clinical methods, only injectables are increasing at NSDP-supported clinics. Many experts believe that while knowledge about clinical methods is high, often women do not know specifically where to obtain services in places readily accessible to them. EngenderHealth, working with the MOH, shared data which indicate that use of long term and permanent methods is showing modest increases, which is encouraging. NSDP cannot afford to remain reactive on this important issue or leave behind a legacy of phasing out of clinical methods. They must work proactively with other partners such as Engender to develop strategies and take steps to increase services within their network of clinics. B. Child Health Issues and Challenges The picture is mixed when it comes to child health services. For immunizations, the rural project areas show that a majority of clients are getting their immunization services from NSDP sources. Increases in coverage are substantially the same in project and non-project areas. There have been remarkable improvements in immunization coverage in general over the past two years and clearly NSDP NGOs are doing their part. The record on ARI treatment is not good. While there is a decline between 2003 and 2005 in children not taken for treatment, there is no increase in either project or non￾project sites in children being treated by trained providers, either in NGO or public sector facilities. Source: 2004 DHS This is worrisome given the proportion of post-neonatal and child deaths caused by ARIs in Bangladesh (see figure above). Clearly a great deal more than offering treatment in static or satellite facilities is needed to have any significant impact on ARI mortality. The current NSDP Community-IMCI pilot program with depot holders trained to provide Pediatric Cotrimoxazole is a step in the right direction. The pilot is Causes of Neonatal, Post Neonatal & Child Mortality 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% Neonatal Post Neonatal Child Category % ARI Birth Asphyxia Premature/LBW Other Serious Inf. All other causes Diarrhea malnutrition Injury 13 being evaluated by ICDDR,B with results available in June 2006. Current project thinking is that “village doctors” are the more sought after treatment providers and should be the focus of the training programs. Whichever method is chosen, the project needs to determine the preferred approach and scale up the work during in the remaining time in the project. This may provide valuable experience for the national program. Treatment of diarrhea with ORT continues to increase in both project and non-project sites however the majority of people are getting those services from traditional doctors or pharmacies. NSDP NGOs do not appear to be significant sources of ORT as it may be more readily available over the counter or elsewhere. The high level of utilization, however, is a reflection of the earlier successful efforts to reach the population directly with education about the importance of ORT. This also offers lessons for the future about the need to reach mothers directly with messages about the danger signs of ARI’s and the need to seek treatment from qualified providers. Vitamin A coverage in both rural but especially urban areas has declined significantly between 2003 and 2005, probably due to a discontinuation of the National Immunization Days (NIDs) Again, the NSDP NGO network needs to be aware that coverage rates are declining and step up their identification of children and women needing supplementation. C. Maternal Health Issues and Challenges There has been a slow down during the NSDP project in the rate of increase of women receiving ANC services. In rural project areas, NSDP supported NGOs, however, are the predominant providers followed by the government. In 2005 only 21% in NSDP rural areas were getting three or more visits during the pregnancies (Measure Evaluation Survey). Much needs to be done to improve the coverage of ANC services for women. On the other hand utilization of ANC did increase in NSDP sites in urban areas. NSDP’s market share of ANC services in urban areas appears to be increasing. Safe delivery services are only offered in 16 clinics and of these only 6 provide emergency obstetric care according to the last quarterly report. There has been some caution about expanding these services too rapidly given the risks of mismanaging deliveries. Given that 90 % of deliveries in Bangladesh still happen at home with 13.4% attended by skilled attendants (2004 DHS), it is just as important to emphasize the fielding skilled birth attendants into the community to assist with deliveries at home rather an emphasizing institutional births. The project did pilot an SBA home￾delivery program from 10 clinics in 2005 and is planning to expand this to 30 additional clinics in 2006. Part of the challenge is finding a supply of paramedics with minimal level of training. Interestingly, Bangladesh, for reasons that are unclear, continues to have a gradually decreasing Maternal Mortality Ratio (MMR) despite little change in the level of births attended by SBAs. However, according to an analysis by WHO in 2004, the experience with SBAs in Bangladesh is very 14 encouraging in terms of their uptake of the necessary skills which could make SBA training a model for the region. There appears to be a very low demand for PAC services in the NGO clinics, perhaps because clients assume that services are more readily available at government facilities which are already equipped to provide MR. Experts estimate that as many as 20% of all MR cases need subsequent medical attention. Estimates on the numbers of MRs that are performed vary widely although the rate reported by the 2004 DHS is 5.9% for currently married women and 9% for currently married women in their 30s. Clearly there is a need for PAC services. On the other hand, only 3% (about 10) of NSDP clinics offer PAC services at the moment. NSDP is offering little by way of neonatal health care, mainly because so few deliveries are actually performed in the NGO clinics and follow up of neonates is not routinely done. Neonatal health will not improve until there are substantial improvements in maternal health, including making sure that births are attended by skilled personnel. Again facility based programs are probably not the best way of addressing this problem other than to make emergency obstetrical care available in as many clinics as feasible. USAID is also current developing a separate project to address this issue. D. TB The TB services offered through NSDP NGO facilities appear to be growing and showing solid progress. 17% of the clinics now are offering services in partnership with the National TB Program. The number of TB cases treated by the NGO clinics is increasing and case detection rates in NSDP catchment areas have improved from 30% in 2004 to 43% in 2005. Coordination with the national program is excellent. E. HIV/AIDS and STIs/RTIs NSDP NGO clinics are not VCT sites although a few seem to have the rapid test kits and conduct testing on request, with verification and follow-up referred to government hospitals. The NSDP clinics and satellite sites are not well-positioned to deal with clients currently at high risk for HIV/AIDS since their clientele are largely women and children. They are, on the other hand, active in STI and RTI treatment, although reported most of the cases are RTIs rather than STIs. It might be possible for NSDP NGOs clinics to establish satellite sites in collaboration with clinics assisted by the FHI project to offer the broader ESP services as needed by the clients or family members of the high risk groups. F. Limited Curative Care LCC is now a growth area in most NGO clinics. In the early project years, expanded LCC was said to be restricted by USAID due to fears that ESP services would receive less attention. In 2005, roughly 23% of all patient contacts were for LCC and the rate 15 of growth, higher than other services as shown below. This growth is important for reasons that will be explained later in this report. Services Customer Contacts Percent Change FY03 FY04 FY05* FY03 Vs FY04 FY04 Vs FY05 Total child immunizations 2,976,317 3,044,453 3,432,420 2% 13% CDD treatment 1,763,714 1,904,364 2,029,420 8% 7% Pneumonia treatment 137,168 149,420 167,333 9% 12% Limited Curative Care 3,626,708 3,839,688 4,842,438 6% 26% G. Rationalizing Service Coverage An important concept articulated in the Cooperative Agreement was an intention to “rationalize” service coverage within NGO catchment areas by analyzing alternative providers, understanding service gaps, developing referral system with government and other NGO providers as needed, and making well informed decision about service expansion based on these factors. Little has been accomplished in this area to date and yet it could be very useful for future USAID activities. (See Section IV A.) H. Summary of Service Coverage Issues · NSDP has demonstrated solid progress in expanding many ESP services to the population in its catchment area · NSDP needs to place high priority during the remaining period in the project on working proactively with CA partners, the MOH and its NGOs to address the problems that are causing a decline in utilization of long term and permanent family planning methods. · The current clinic, satellite clinic and depot holder program cannot deal effectively with ARIs, malnutrition, or neonatal health unless better connections are made with communities, especially in rural areas. Several programs started recently on a pilot basis to address mortality from pneumonia and expand the role of SBAs in home births need to be expanded to have broader impact. · The ideas in the Cooperative Agreement about “rationalizing” service coverage need greater attention, especially with the advent of the ADB-supported UPHCP II project and the upcoming WB contracting with NGOs. Objective Two: Increase Use of Services, Especially by the Poor 16 A. BCC and Creating Demand In general, it appears that NSDP, with BCCP as the primary partner, has done an excellent job of developing mass media as well as clinic level educational material. The Smiling Sun logo and promotional activities with Joya Ahsan and the TV drama series is clearly an innovative and high visibility program. Currently an evaluation of the impact of their mass media efforts is underway by Mitra and Associates, although the results were not available yet. The project reported that “based on preliminary results from the first two rounds of the survey, it was found that watching the ESH (TV drama series) significantly increased the chance of a person to seek essential care from NSDP clinics”. The Measure Evaluation survey also found a high level of awareness in both NSDP sites and non-projects areas of NSDP sponsored media messages. Leaflets and educational materials were available in the clinics visited by the team and videos were being used to show educational information in the waiting rooms. The evaluation team, however, noted very few clinic-level efforts to advertise or creatively market their services to the surrounding communities. Experimentation with discounts for a package of services (like ANC, delivery and post-natal care or advertising special days when sterilization or IUD services are available at the clinic) might help boost utilization and make people more aware of the specific services the clinics have to offer. B. Community Level Activities The team was impressed with some NGOs (e.g., PKS Khulna) which have established good ties with local business leaders and others willing to make contributions to off-set the cost of services for the poor. While the amounts of money are not large (nor can these be relied on for operating costs) the attempt to build ownership and involvement with influential local people is important for community support, involvement and ownership. Other NGOs (e.g., Swarnivar) were active with village development committees to map local conditions and resources and help connect the community to the clinic and its activities. The larger NGOs which have development projects other than their participation with NSDP, often have their own connections to these communities that benefit the clinics. These ways of strengthening NGO/clinics with local leaders and the communities are best practices that could be shared among the NGO network. Links with communities could be vital in future for ownership and increased utilization. More emphasis on community outreach in urban and rural areas might also be important for increasing demand for services. Ultimately, it will also help clinic staff understand the needs and problems of their client populations and provide improved access when community level interventions are needed (e.g., home deliveries, ARI treatment and referrals) C. Reaching the Poor: 17 Data from the MEASURE Evaluation surveys demonstrates that NSDP is serving the poor. For family planning services, 76.2% of clients at NSDP clinics were from the poorest three quintiles, while 71% were from the same quintiles in government and other NGO clinics. The number from the poorest quintile (roughly 25%) was the same in NSDP clinics as compared with government and other NGOs. In Bangladesh 44% population is estimated to be poor. NSDP clinics serve customers, of which 64% are reported to be poor, a good indication that the system is a pro-poor one. NSDP clinics are also located in areas that began the project with lower service coverage and mortality indicators than the national average. The project has introduced a method for identifying and providing Health Benefit cards to the “LA” (Least Advantaged or extremely poor segment of the population). Cards enable those identified through this mechanism to receive free services or in some cases services at a reduced cost. To date, 15 clinics have reported distributing roughly 24,000 cards and serving 78,300 LA clients. This program has the benefit of pre-determining clients who really cannot afford service fees and empowering them seek services knowing that fees will not be requested. But to have the intended impact, this program needs to be expanded substantially. Clinic and NGO staff visited by the team had a highly positive reaction to this Health Benefit card program. Summary of Issues · The “Smiling Sun” branding is excellent and provides the basis for expanding a true network of clinics offering high quality services. Clinic level marketing of services is limited and needs more emphasis. · Innovative community and clinic partnerships piloted by some NGOs should be shared and expanded if possible · NSDP NGOs have been successful in reaching the poor but need to expand efforts to identify the poor and expand creative financing mechanisms Objective Three: Increase NGO Capacity to Sustain Clinic and Community￾Based Services The key issues addressed by the evaluation team on improving NGO capacity were as follows: · Quality Improvement and Technical Capacity · Measuring Sustainability · Strategies and Activities Supporting Sustainability · Cost of Services · NGO Funding 18 A. Quality Improvement and Training Capacity Development Early in the project, NSDP staff worked directly with the clinics, overseeing quality and providing technical assistance. This approach shifted substantially in 2005 when the project redirected its attention to building the capacity of the NGOs themselves to undertake quality improvement activities. The current emphasis is on ensuring that the NGOs are using the tools developed by the project to do the monitoring and technical assistance themselves. All supervision visits by NSDP staff are now done with the appropriate NGO technical staff person responsible. Much of the quality monitoring system relies on NGO self assessments, using tools provided by the project. Central NSDP and NGO staff follow-up regularly to check on problems identified and steps taken to address the problems. In 2006, the project plans to do “program audits” which would essentially involved unannounced random visits with the NGO clinical monitoring staff to check on clinic performance in this regard. This is an important step to help institutionalize the system. While the quality assurance processes early in the project did little to build capacity among the NGO staff, the current system appears to be working well. During site visits the evaluation team noted that clinic staff was able to check their records for the date of the last QMS visit by NGO headquarters, and describe the problems identified at that visit. They also were able to explain the steps taken to address the problem, most of which were highly practical in nature. With respect to developing the training capacity, the strong points of the clinical skills training program have been the involvement of the government training centers as well as the project’s emphasis on skills building in a clinical setting with sufficient experience to ensure competence. The most problematic aspect of the training provided by the project is the issue of loss of the training investment due to the high staff turn-over, especially of clinical personnel. The evaluation team was not able to quantify these losses in financial terms but NSDP agrees that it is probably substantial. The NGOs need to experiment with creative ways to retain doctors and other key staff by either contracting as needed with physicians who are already employed elsewhere or providing other performance incentives that help retain the staff that have been trained. B. Measuring Sustainability There are many definitions of sustainability. The working definition applied by the NSDP’s team is: An organization’s ability to define a relevant mission, follow sound management practices, and develop diversified income sources to ensure the long-term continuity of quality, community-oriented services. 19 The evaluation team would add another dimension to this challenge: that health services provided through this project continue to be provided in achieved and preferably increased levels of volume, range and quality with diminished support from USAID. As implied in the NSDP team’s definition of sustainability, the long-term sustainability of the health services offered is dependent on the capacity of the NGO’s to develop and maintain those services. A critical question to be answered is whether the NGO’s have achieved this capacity or are likely to do so by the end of the project in September, 2007. The primary instrument for measuring organizational capacity is the MOCAT, the Modified Organizational Assessment Tool, an improved version of OCAT developed in South Africa and required by the Cooperative Agreement. The range of organizational elements considered in the MOCAT is quite broad. However, measuring sustainability of widely differing organizations is difficult and imprecise. It was not the purpose of this assessment to evaluate MOCAT itself, but interpretation of the scores requires some understanding of the limitations inherent in this instrument. First, all measurements are treated as equal – which may not be appropriate as some may be more important than others. Furthermore, there may be some over-riding characteristic of an NGO which is not included, but for which mitigate or even eliminate all chance of sustainability – such as compliance with the MCP or misuse of project funds. Secondly, the questions are both objective and subjective. As a consequence, different people could score questions differently. Key Findings on Measuring Sustainability · According to MOCAT, with the exception of a few relatively small NGO’s, management has improved somewhat since the beginning of the project. · MOCAT is provides a useful framework for quantifying organizational development, but interpretation should focus on specific areas and relative growth rather than precise scores. · Used alone, MOCAT is not always a good predictor of organizational capacity. Within NSDP its focus was on the NSDP health program, and particularly with well established NGO’s, fails to recognize the organizational strengths of the NGO itself. Thus, to label an NGO which has been around for decades and has multiple programs as “emerging”, paints a false impression. MOCAT must always be supplemented with first-hand knowledge of the character and context of the NGO, particularly its leadership. · MOCAT proved to be a useful tool for helping NGO’s understand where they needed strengthening. This also helped direct the focus of technical assistance. · MOCAT should be used as a rapid assessment tool, not a study framework. 20 Far too much time was dedicated to measurement with MOCAT. NSDP used MOCAT to produce 40-page studies on each NGO, which significantly distracted from the actual strengthening of the NGO’s themselves. C. Strategies and Activities Supporting Sustainability Before analyzing the strategies and activities supporting sustainability, there are four salient points to keep in mind. (1) The kinds of services offered at NSDP supported NGO clinics (mainly ESP services) have limited cost recovery potential because many are preventive in nature and the demand for them may be low among a poorly education population. (2) Curative services have been until recently restricted and therefore the ability for the clinics to increase cost recovery, constrained. (3) The NSDP and USAID’s mandate to focus on the poor also limits to some extent the potential for full cost recovery and (4) The restrictions on use of program income have also resulted in lost opportunities to encourage sustainability. These following are summary “lessons learned” from the NSDP experience on the topics listed below. (For a fully discussion and description of the issues, please refer to Annex D: Sustainability Analysis Report, JHolley, March 7, 2006.) 1. Serving the Poor NDSP services are reaching the poor. Community Funds can help support the poor, but probably only in communities with a mixed economic structure. Since they are voluntary, the sustainability of these Funds is questionable; and almost certainly not available in the poorest communities. Provision of drugs to the “Least Advantaged’ (LA) is a critical element to insuring their access to primary health care services. There is some anecdotal evidence that without access to free medicines, at least some LA’s will not attend clinic services, even if they are free-of-charge. Centralizing the Revolving Drug Funds (RDFs) at the NGO level would facilitate redistribution of excess RDF profits to help insure access to drug and services of all LA’s. Because of the differences in the populations served by NGO’s, particularly the rural NGO’s, a mechanism is required to redistribute funds across the entire Smiling Sun network to insure that no LA’s are left out. Health Cards have not proven to be a particularly useful mechanism for insuring services to the poor because of the small numbers to date. A great deal of effort was placed by the project in identifying the poor for this purpose. Now that it is done, and the mechanism is established, the task of updating should be identification of the non￾paying clients should be left to the communities, as is currently planned. Despite some short-comings, the Performance Enhancement Scheme has proven its potential for demonstrating the achievement of what are its two primary objectives: creating a mechanism to cross-subsidize across the Smiling Sun network; and stimulating higher levels of cost-recovery. Assuming that the Performance 21 Enhancement Scheme is eventually deemed a success, it will be important to maintain intact the participation of the stronger Smiling Sun NGO’s and clinics in order to be able to cross-subsidize those requiring financial support to serve the poor. In the long term, this probably requires a mechanism to lock all participating NGO’s, without which the stronger NGO’s might opt out. At least 30% of all Smiling Sun costs must be subsidized in one manner or another, and most likely through a combination of mechanisms. These include donor funding, but also include cross-subsidies within and across NSDP NGO’s, support from communities, in-kind provision of commodities, and other sources. The more non￾donor support is stimulated, the easier and safer it will be for USAID to withdraw without losing the tremendous investment already made. 2. Cost Recovery Lessons Cost-recovery is increasing among most NSDP NGO’s. It is, of course, most difficult for the rural NGO’s who serve relatively poorer populations. Because of the nature of NSDP targeted services, cost-recovery potential is limited for two reasons: the services themselves are largely preventive, and will never generate large amounts of income, particularly if significant numbers of poor clients are served without service fees; and the currently narrow range of services excludes much curative care which has a high demand, and thus, for those curative and other services not currently offer in Smiling Sun clinics, people seek services elsewhere. The capacity for cost-recovery varies between NGO’s and clinics. Mechanisms must be established to insure an equitable distribution of resources, particularly to serve the poor, within NGO networks, and between the NGO’s in the Smiling Sun network. Offering a wider range of services particularly curative services will attract more clients to Smiling Sun clinics, generate more cost-recovery to improve sustainability, and at the same time, generate increased demand for NSDP priority services. Particularly where competition is present, quality of care is critical to attract and retain a large clientele. As services expand, this component must be strengthened, and supervision from the NGO level enhanced. Improved quality includes the constancy of well-trained, supervised staff, particularly doctors and paramedics who have developed a strong clientele. It is more cost-effective to pay them at competitive levels than to forego services, have reductions in quality, and costs of recruiting and training. Well established NGO’s might well experiment with innovative payment schemes. Efforts to brand and attract clients to the Smiling Sun network have been hugely successful, and should be continued. This type of social marketing requires a continual presence, and is worth the investment. Little attention has been paid to marketing the services of individual clinics. This needs to be strengthened to attract a wider range of clients for all services. 3. Cost of Services 22 USAID expenditures at the depot holder level are negligible. Most of the value at that level is supported by the GOB and the NGO’s. 22 NGO’s provide 90% of the patient contacts at the static and satellite clinic level, expending about 86% of grant and program income funds. This suggests that theoretically, a full third of the NGO’s could be eliminated with minimal impact on the numbers of patient contacts. This said, the impact on the communities served by any NGO’s in the bottom third could be serious with their elimination, and obviously incorporation of services into the stronger NGO’s could be preferable. The cost per contact does not significantly reduce with increases in volume of services in NSDP supported clinics. Part of the reason that fixed costs are relatively high, particularly for the smaller NGO’s. This is a result of two factors: first, the staffing pattern has been standard for all clinics regardless of volume; and secondly, each NGO must have its own administrative structure to support the service delivery structure. Recent rationalization of staffing patterns will help cost-effectiveness of service delivery. Reduction of the number of NGO’s served would reduce the fixed administration costs. 4. Financial Support Issues A relatively small number of NGO’s provide most of the service contacts and consume most of the budget. It would be more cost-effective to work with fewer NGO’s both in terms of diminishing costs per service contact as the volume of services grows, and in terms of the ability of the technical assistance to support NGO’s. At present, recognizing this last fact, the NSDP concentrates on the 14 “focus” NGO’s to enhance the cost-effectiveness of their own services – leaving the others much on their own. Total expenditures during the first three years of the project remained relatively constant, driven in part by a restricted focus of support on limited range of services designated by the project. Expenditures have risen in FY 06 due to a gradual liberalization of that policy, but the increase is largely financed through utilization of program income. The mix of project expenditures in terms of grants and program income is not based on an effort to gradually reduce dependency on grants and replace grants with generated income, but is completely an artifact of project design. Restrictions on the use of generated income contributed, but there is a contractual incentive for the USAID NGO Expenditures 0 100 200 300 400 500 600 700 1 2 3 4 Project Year Millions of TK Generated Income Grants 23 Recipient to use grant monies, as replacement of monies by generated income diminishes the total amount available for management and technical assistance due to the 35%-65% ratio of TA to grants described in the Cooperative Agreement. This will become more important in the future as the capacity to generate income increases. The appropriate measure of cost to USAID for supporting NGO’s is therefore, not grants plus program income, but simply grants. The use of program income was overly restrictive during the first years of the project. Use of those funds to expand services beyond those designated by the project will increase the volume of paid services and cost-recovery while increasing the volume of services the ESP package by broadening the number of clients attending services. This expansion can be financed by using program income rather than grant income, and is thus essentially free for USAID in all but contractual legal terms. Furthermore, the definition of generated income = program income is inherently incorrect because other sources of resources are used to support cost-recovery efforts. The calculation of program income should take this fact into account. The use of program income should be further liberalized to stimulate additional cost-recovery and growth of all services, as well as increase the sense of ownership of the NGO’s of their health systems. This can now be accomplished through a broad policy statement from NSDP to USAID; and through a simple justification process to allow NGO’s access to the use of program income accompanied by appropriate planning, monitoring, and auditing. The accounting system of the project and consequently, the NGO’s, tallies all expenditures in the year actually paid. This distorts expenditures, cost accounting, and cost-recovery, and could be easily remedied by keeping operating expenses separate from a “capital account”. This would permit the use of depreciation for purposes of program income calculation and expenditures, service costing and pricing policies, and eventually calculating the value of assets transferred from one NGO to another. While USAID holds a strong claim on generated income, other donors will be reluctant to fund NSDP-related activities. Attracting additional donor funding is an important reason for modifying the policy related to program income. G. Bottom Line on Sustainability Two highly important general conclusions emerge from the overall analysis of sustainability. · Up to 30% of costs of NGO clinics may need to be subsidized from some source, not necessarily from USAID. Support already comes from government and should continue. At the same time, other donor financing opportunities are expanding. 24 · With managerial and financial flexibility, cost can be reduced, services expanded and program revenues used more effectively to promote sustainability of NGO services Objective Four: Influence GOB Policy, Coordinate with Donors, and Improve NGO Environment The Cooperative Agreement was probably too ambitious in some of the policy level accomplishments expected from this project. NSDP interactions with the MOH have been directed at resolving problems or issues that have arisen in the MOH-NGO interaction at the field level. There is not much evidence of other major policy analysis or dialogue which, in any case, would be difficult to accomplish within a project framework. NSDP, nevertheless, has a good working relationship with various counterparts within the MOH, most of who seemed to understand and respect the project’s contribution. Formal meetings are held with the Steering Committee annually and more frequently with the Working Committees, primarily to review the annual work plans. Field issues are dealt with as they arise. Accomplishments reported in internal evaluation document focus practical issues aimed at improving coordination and compliance of NGOs to government standards and requirements for reporting (e.g., affiliation, ability to charge for certain services, etc.). HPNSP has allocated 15% of the total pooled funding for NGOs and 25% of the total HPNSP (including non-pooled funds). USAID’s contribution to supporting NGOs is included under this umbrella. At a matter of policy, there appears to be a growing level of acceptance of the NGO’s role in health system. Key Conclusion: The best avenue for engaging MOH at the policy level is with USAID (supported by NSDP) within the context of the HNPSP. For the remaining period of NSDP, much can be done to share NSDP experience, technologies and systems with MOH staff who will be managing the new NGO contracts. Two agencies will be contracted by the MOH, the PSA to manage the grants and the PMA to provide technical and quality oversight. Accordingly the World Bank representative interviewed by the team, both agencies and their MOH managers in the Planning Unit would benefit enormously from sessions conducted by NSDP staff on lessons learned about topics such as NGO financial capacity building, service quality improvement and oversight, programs for identifying the poor and so on. This will also permit NSDP and USAID to stay well informed about the plans and progress of the NGO contracting, and make suggestions that might facilitate the NSDP NGOs to become part of that larger system at some point. NSDP Monitoring and Evaluation System A huge amount of data is collected and reported by the project. NSDP staff believe that the program could be managed with a more streamlined and less cumbersome system. . Major changes in the system at this point are probably not wise, particularly since the data requirements of both USAID and those requested by the MOH would 25 need to be protected. It is not clear how much of this data is used actively by NGOs for decision making or monitoring individual clinics. Clinic staff and especially those involved in delivering services are certainly spending too much time filling up records. Modifications made on original M&E plan and indicators suggested in Cooperative Agreement. Changes appear to be reasonable. Some targets were very ambitious and will not be reached (e.g. 100% of clinics using IMCI and CYP short falls mentioned in Section IV) Some of the studies undertaken by the project have been time consuming, complex and expensive. For some, such as the cost study, it is too early to tell whether the investment will yield the expected benefits. For remainder of project, emphasis needs to be on consolidated lessons learned for the future, not undertaking additional studies unless they are essential for decision making. Quick assessments of what has worked, hasn’t and why, would be a good legacy for the next project. IV. Implementation Issues and Challenges A. Adherence to activities and requirements of the Cooperative Agreement Rationalizing Service Delivery Networks In Attachment 2 (Program Description) of the Pathfinder Cooperative Agreement, there is emphasis on the need to work with other NGOs and partners to map service coverage and plan NSDP supported services based on need. Special attention was to be given to developing referral systems with other facilities that allow for follow up of referred patients. The idea was to have NSDP clinics and community workers relate to a broader system, including the WB and ADB funded activities with the government as well as the MOH service delivery facilities. Looking at the environment and other services providers and rationalizing package of services seems not to have been emphasized early in the project. Few of these plans actually materialized although some attempt has been made to do “coverage mapping”, which is a reasonable first step, and keeping records of referrals. Discussions have also occurred with the ADB to avoid overlapping clinic coverage in urban areas. Given the other donor-supported projects (mainly ADB and WB) are now expanding into coverage areas near many NSDP sites, giving more attention to this task in the final year could provide useful guidance for any follow on programs. It will also provide the information needed for USAID to engage more actively in coordinating with ADB and the WB to discuss issues of coverage, overlap, referrals, and share experience. 26 Project Targets An area of substantial concern related to the annual project performance targets is the CYP accomplishments. The declining level for permanent and long term methods was clearly not anticipated. The following is a comparison of planned and actual CYP performance. CYP (millions) FY 2003 FY 2004 FY 2005 FY 2006 Planned 1.636 1.702 2.246 3.010 Actual 1.046 1.139 1.205 0.315 (1 st Qtr) While it is unlikely that at this stage in the project the original targets can be achieved, much could be done to at least reverse the downward trend with a more coordinated and proactive approach as mentioned in section III A. BCCP The only current NSDP partner organization not mentioned in the Cooperative Agreement is BCCP. However, a sub-agreement with BCCP was apparently negotiated early on, and it has been an active and useful organization with unique capacities that have served the project well. As of this evaluation, the staffing for NSDP included 38 from BCCP which was felt by project management to be out of proportion to BCCP’s input on project specific activities. Some of the BCCP staff are, however, part time. There is an effort underway currently to adjust these levels to reflect a more accurate and programmatically justifiable level of salary support to BCCP which makes sense at this juncture. In the future, however, it maybe better to contract with organizations such as BCCP on an output or task order basis to avoid problems of this kind. B. Efficiency Issues: Project Staffing and Use of Resources Organization and Staffing The NSDP project is implemented by a consortium of organizations led by Pathfinder International. Obvious questions arise as to the impact and need of having so many partners. The number of partners was not based on project requirements, but was a consequence of the bidding process when during the bid preparation some major shifts occurred among prime bidders. The consequences of having such a large group have impacted the project in many ways. First of all, it has resulted in less integration of activities and skills and more fragmentation than would have been the case with fewer partners implementing the same scope of work. This has been particularly true since the various teams were not guided well to work in a unified fashion. As a consequence, each team set its own direction, and while complementary, there was a tendency to work somewhat 27 independently of all others. Perhaps this was stimulated by the fact that each partner is responsible for specific “deliverables” – rather than the overall project being responsible for a single set of deliverables to be accomplished by the project team, irrespective of their organizational affiliation. The current project leadership is now strengthened, and the teams more closely integrated through the creation of regional teams cutting across function and organizational boundaries. Where there may have once been some friction between groups, we sensed little at this point in time. Another consequence is the increase complexity of management of the project. Each of the partners’ sub-contracts must be managed separately, relations established between the partners maintained, head office concerns addressed, etc. Human resource policies for each of the partners are different, and so Pathfinder has recently and somewhat successfully tried to smooth them out to avoid conflicts among individual staff members. Finally, multiple partners greatly increase the costs without necessarily adding much value to the project. Each partner charges overhead, and the prime contractor charges an additional overhead for managing each partner. The NSDP Financial Director estimates that the additional cost of each partner is between $200,000 to $400,000 per year, including incremental overhead, the costs of overseas visits by supervisors, additional administrative costs, and so on. The project still has a rather large staff concentrated in Dhaka but changes at this juncture would be disruptive and impractical. There will be an inevitable attrition of staff as the project begins in wind down and the challenge for management will be to attempt to retain those who are essential while letting others move on. Lessons Learned on Organization and Staffing: · In general, the important element in putting together a technical assistance team is the individual, not the organization. While organizations may have strong desirable in-house capacity, that capacity may or may not be readily available to the project, particularly if the individuals assigned are inappropriate for one reason or another. · A large group of partners makes management by the prime contractor more difficult as the administration of multiple contracts is much more complex and time-consuming when multiple partners are present. · Multiple partners add significantly to the costs without necessarily adding much value to the project. 28 C. Problems and issues affecting implementation There is little point dwelling on the disruptions and problems caused by the MCP violations other than to say that they most certainly had a negative impact on the project during 2005 while they were being sorted out. Valuable time was lost and staff from both NSDP and USAID were distracted as they struggled to preserve the project while taking the required actions to discontinue the NGOs who were found to be out of compliance on MCP. Unfortunately, some of the strongest and most capable NGOs were lost to the project in this process. Of the 55 clinics affected by NGOs which were discontinued, 51 clinics were successfully ‘adopted’ by other NGOs. A side-effect of the efforts to deal with the MCP is a noted reluctance on the part of NSDP staff to initiate potentially useful project interventions without the expressed approval of USAID. This has hampered the success of the project to date, but appears to be resolving itself as a result of dialogue stimulated by this evaluation. A rigorous system is now in place for monitoring compliance with an emphasis on the NGOs themselves taking the lead for monitoring. All clinics are also supposed to be visited at least once a year by NSDP staff. One of the unfortunate consequences, however, is that many NGOs and to some extent the NSDP staff themselves are reluctant to engage on issues that they fear will put them at risk of Tiahrt problems by association. This view, for instance, was evident when the team questioned staff about why they were not taking more proactive steps to address the problem of declining PLTM use. The other major event in the project was the replacement of entire US senior team, which began with the departure of the former COP for health reasons. For a period of nine months, the COP position was vacant until the current COP was identified. Any change of this nature is disruptive but the evaluation team noted that, while leadership changes have naturally been difficult, the new NSDP leadership team is functioning well and the outcome has been a number of very positive changes in the project. D. Management Issues for USAID The project design itself did not make this activity especially management intensive for USAID, and in fact combining two earlier projects should have reduced the amount of management time required. However due to several problems that arose during implementation, it became very intensive with a large burden on USAID. Most of the management burden on was due to the reported MCP violations and the need to get those problems resolved in ways that were acceptable to all. 29 Another was the issue of the use of program income (explained in Section IIIC). The lack of clear written agreement at the beginning of the project about how those funds could be spent caused a set back on dealing with financial sustainability, but it also resulted in USAID becoming involved in approving each individual proposal for use of the funds. Because there was generally a lack of clarity, and a perception that USAID was highly restrictive about how such funds could be used, time was wasted. Pathfinder headquarters should have intervened earlier to make a proposal in writing for how the funds could be used and to seek formal concurrence from USAID. The team understands that this is now in process. Lessons Learned on Management and Implementation · In order to stimulate increased cost-recovery and a sense of ownership, it is critically important for USAID to allow the NGO’s much greater managerial autonomy and financial flexibility, particularly in reference to the use of program income. · Having transparent and rational systems in place to monitor compliance to MCP and Tiahrt requirements, and taking timely action on reported violations, will continue to be critically important in Bangladesh. V. Summary of Recommendations for Strengthening Implementation The following summary points are presented in bullet form to facilitate discussions about follow-up with the NSDP team. Programmatic · NSDP needs to place high priority during the remaining period in the project on increasing the utilization of permanent and long term methods by (1) working proactively with CA partners and the MOH to address the problems that are causing a decline in these methods and (2) ensuring that there is sufficient local promotional activities at the clinic level to be sure that the population is aware of the specific services available in each clinic. o NSDP should meet with EngenderHealth to strategize and coordinate on activities related to increasing the availability and use of the full range of clinical methods. More can be achieved if USAID partners work together. o Service utilization may improve if special activities are arranged in the catchment area of NSDP supported clinics to ensure that the population knows about the availability of the services (NSV or minilap days when qualified doctors are on hand, etc). Clinics should ensure that the full 30 range of family planning services are available and clients counseled properly. · The current clinic, satellite clinic and depot holder program cannot deal effectively with ARIs, maternal or neonatal health unless better connections are made with communities, especially in rural areas. Several programs started recently on a pilot basis to address mortality from pneumonia and expand the role of SBAs in home births need to be expanded to have broader impact and to learn lessons for the future. · The ideas in the Cooperative Agreement about “rationalizing” service coverage needs greater attention, especially with the advent of the ADB II project and the upcoming MOH contracting with NGOs. o NSDP staff should review the suggestions on rationalization and work with their NGOs to gather information on coverage of services by MOH, contracted NGOs and existing private or non-governmental organizations. Progress should be made on establishing more formal referral and follow-up systems, and coordinating schemes for subsidizing poor clients within their various catchment areas. Information gathered during the remainder of the NSDP project could be very useful for future USAID-supported work and policy dialogue in this area. o NSDP, with USAID colleagues, should maintain regular contacts with key staff from ADB and the MOH/World Bank projects to talk about coverage issues, share best practices on working with NGOs, and planning future work. NSDP should share the systems and procedures it has developed to strengthen NGO capacity in the areas of finance, management, quality oversight, targeting the poor and other topics with MOH NGO contract managers. This can also help USAID better influence what happens with other large donor supported activities. · Innovative community – clinic partnerships piloted by some NGOs should be shared and expanded if appropriate. This includes the linkages some have created with local business leaders (community funds) to support services for the LA population or linkages with village development committees to help understand the needs of the population. These activities will help increase local ownership and involvement, and facilitate the NGO’s ability to reach into the communities for expansion of programs such as ARI awareness and treatment programs. NGO Capacity Development Technical 31 · There is evidence of technical capacity development among the NGOs, especially with use of the QMS and ability to identify and deal with practical quality problems at the clinic level. The planned “program audits” will be an important way of further strengthening this process with an external review. NSDP should continue to explore ways of institutionalizing the capacity and motivation for NGOs to provide quality oversight of their clinic services. Managerial/Administrative · The one-size-fits-all approach in the first part of the project to NGO staffing has created high overhead costs. Greater efficiencies can be achieved with more flexibility in the final round of grants. Financial · A written agreement between the project and USAID on the acceptable use of program income is important to give NGOs greater flexibility to use that income for expanding services and improving cost recovery. This should be followed-up immediately. · NGOs need strong encouragement to offer more curative services and services that are in demand from the community (with the exception of those prohibited by USAID) but at the same time preserve their ability to serve the poor with ESP services. NGOs need to understand that this expansion is necessary as a way of increasing cost recovery as well as identifying more clients in need of ESP services · Judicious use of proceeds from the RDF needs to be encouraged, with the management of those funds at the NGO rather than clinic level, to enable some cross subsidization among clinics. NSDP should hire a consultant who can focus on this issue immediately and provide financially sound advice to the NGOs. · The “performance enhancement scheme” is showing strong potential to help the poor and should be refined and expanded during the remainder of the project. · Stimulating cost consciousness and improving clinic specific marketing is important and should ge t additional emphasis. These marketing efforts should emphasize the high quality, affordable services available at NSDP￾supported clinics, with practical information about how to reach the clinics and what specific services are available. NSDP Project Level Issues 32 · There was “stove piping” of various units at the Dhaka office early in the project but the geographic teams currently in place have improved the situation substantially. NSDP should continue to work on strengthening the ability of the NSDP regional staff to provide on-site support to the NGOs. · NSDP has started and should complete the steps currently underway to rationalize the NSDP supported staff at BCCP relative to their contribution to the project. BCCP is developing plans of their own for sustaining their work in the future which is important to protect USAID’s investment in this unique and important organization. · There have been a fair number of studies conducted by the project and various partners. Many have been conducted quickly and the results applied to practical steps to improve operations. Others have been protracted, expensive and it is less clear how the study outputs have been used. Emphasis during the remainder of the project needs to be on practical application of findings and increasing coverage of pilot programs rather than initiating new studies. · MCP- related issues have posed major disruptions and institutional trauma in the past. Hopefully the current compliance and oversight system will help promote a clear and rationale way of verifying violations and taking appropriate and timely action. NSDP needs to work with the NGOs, however, to ensure that clinic and NGO staff do not over-react by avoiding PAC services or efforts to increase the availability and use of clinical FP methods due to generalized apprehensions about MCP or Tiahrt. · The original 41 NGO’s have been reduced to 33, but that is still an unwieldy number to support. As noted earlier, some NGO’s could be cut loose at the end of NSDP without a significant loss of coverage should consolidation into other stronger NGO’s not prove feasible. This should be an immediate focus of attention, requiring at least one staff member specifically responsible for analyzing potential “mergers and acquisitions”. 33 Part II: Recommendations for the Future VI. Current Landscape A. Realities and Challenges of Health Sector in Bangladesh The Government health system in Bangladesh is weak, and fraught with leadership and management problems that make it extremely difficult for USAID to invest too heavily or exclusively on working with the MOH. The current civil service system requires that many professional staff rotate frequently within the system and therefore the effort to develop capacity in any one unit of the MOH would be challenging at best. Therefore the current approach, which is a balanced one of working on service delivery with NGOs and on selective technical assistance to the MOH, is sound. NGOs will continue to be an important part of health care system in Bangladesh. In fact, they are probably more important now that the HNPSP (and the ADB project with the Ministry of Local Government), is contracting NGOs to provide ESP services on behalf of Government. Internationally, the view of NGOs in the health system is changing in many countries. Governments seem more willing to try contracting with NGOs for service provision or to develop pilots to work more closely with the private medical sector. Contracting with NGOs has been tested in Cambodia, for instance, with impressive results in terms of expanding service coverage for an essential package of services (much like those in ESP). The challenge for such governments is to ensure adequate allocations from the government’s own budget to cover the costs once donor assistance is no longer available. USAID’s role and contribution to building NGO capacity in Bangladesh has been incredibly important given the duration of support provided and the well-documented achievements in making high quality services available, especially for the poor. USAID is respected by the MOH and other donors for this contribution, and is therefore well- positioned to make substantial future contributions to the policy dialogue process on the issue of the role of NGOs in the health care system. As noted earlier, USAID does bring to the PNPSP, the wealth of experience it has accumulated over the 30 history with NGOs in Bangladesh, including innovative pilot activities to provide services for the very poor and cross-subsidizing those services across NGO clinics. Undoubtedly, there will be a growing need within the MOH to have the capacity to undertake policy analysis and build in-house expertise to oversee the partnership between NGOs and the MOH. This is likely to be an important function in the future, especially if the MOH is eventually less involved in direct service delivery and more in overseeing the entire health system with a multiplicity of service providers. At the moment, interviewees reported that many policy studies are conducted by donor￾funded consultants but rarely read, understood or acted on by government officials. 34 Donors contributions amount to about half of the MOH’s annual development budget so the MOH is receptive to advice but they are not in a good position to accept advice selectively based on their own internal analysis. USAID could play a role in supporting this capacity development but may need to wait until the conditions are such that it would be worthwhile. In the meantime, working with the HNPSP structure to share USAID’s experience is valuable. B. Positioning NGO Health Services in Bangladesh Many of the individuals interviewed by the evaluation team were skeptical about the HNPSP NGO contracting plans given prior difficulties the MOH has had managing World Bank and other donor funded projects, including using the resources available. The soon-to-be launched NGO contracting program is funded at the level of $130 M over a five year period. It is an important development in the health sector, and there should be a great deal more interaction among HNPSP partners to share best practices and learn about how the role of NGOs can grow in Bangladesh. Of particular interest for USAID would be how to expand the Smiling Sun network of clinics beyond those supported only by USAID. There are also opportunities for cross fertilization of ideas and innovations between NSDP and the UPHCP II. For instance, the ADB-supported UPHCP project is using performance incentives to stimulate higher levels of achievement by contracted NGOs. NSDP and USAID may learn from that experience. Likewise NSDP also has lessons that might benefit UPHCP. Furthermore NSDP should be a good training ground for NGOs which can then apply for UPHCP II contracts. The UPHCP II project’s use of “performance guarantees” of Tk 2.5 million to eliminate weaker NGOs has unfortunately caused several USAID-supported NGOs with excellent capacity to withdraw from competition. Other NGOs, which previously received support from USAID, have won contracts probably in part due to the capacities they have developed from USAID support. The role of NGOs in the health care system may evolve over time as experience is gained with the partnership between government and NGOs. The stronger USAID￾supported NGOs need to be encouraged to compete for contracts as this represents good potential for transitioning to a longer term source of funding for their activities. Current reluctance is due, in part, to a lack of confidence in the selection process as well as not knowing which organization will be awarded the PSA contract. Hopefully as the HNPSP program comes on board, and if the selection process is transparent, more NGOs will be willing to sign on. Individual private providers are the largest part of the current health care system in Bangladesh. In fact, the use of private medical care in Bangladesh is highest among the neighboring countries and higher than the average for 45 other developing countries. (Private Sector Assessment for HNP in Bangladesh, World Bank, 2004) This private sector is largely unregulated with serious deficiencies in the quality of services among the informal and untrained providers. It is currently very difficult to 35 work with these individual practitioners without policy guidance or regulatory support from the MOH. Developments in this area should, however, be encouraged. C. USAID’s Niche USAID should continue its focus on NGOs (in addition to other programs) but work proactively and purposefully to make an exit as the chief financier of NGO services. NGOs must be pushed to increase their cost recovery and find other sources of support for elements of their programs such as government and/or cross subsidizing between services that can generate revenue and those that do not. USAID should protect its considerable investment by continuing to work with NGOs but use the lessons from NSDP to accelerate the process of graduating NGOs from dependence on USAID. This is especially important at this juncture given the declining USAID resources for Bangladesh and the uncertainties about future levels of funding for population and health programs. In addition to working with NGOs, USAID has a compared advantage over other donors in areas such as social marketing and public education campaigns to reach rural families directly with messages on improving health and changing health seeking behavior. Along with working on service delivery, broadening the impact of USAID’s health work in the country will require greater efforts to reach people directly in their homes and communities. These efforts need to proceed in parallel with service delivery. NSDP has made a good start on this front but more is needed. Finally, adherence to MCP and Tiahrt requirements will continue to require vigilance but the risks that can be managed using the kind of system now established by NSDP. Previous experience and the residual trauma may help avoid problems in future VII. Future Recommendations A. Characteristics of a New Project The many lessons learned from the NSDP project, both positive and negative, can be used to develop a follow-on program that preserves USAID’s considerable investment in NGOs and yet moves purposefully toward ending NGO dependence on USAID financial support. Not to finish the task begun by NSDP would result in a collapse of the Smiling Sun network, and effectively in a loss of the tremendous effort and financial investment made to date. For this reason, the team strongly supports the development of a follow￾on 5-year project with the following characteristics: · The new project should encourage expansion into primary curative care, not simply for a humanitarian viewpoint, but principally because this is the best way to make the NSDP health services reasonably sustainable. People seek health services primarily for curative care, and are generally willing to pay for 36 those services. As the volume of those services increase, more people can also be offered the ESP services. · The number of NGO’s in the NSDP network should be reduced, preferably without losing health care delivery capacity. This means consolidating smaller NGO’s into the larger ones where feasible. One key to sustainability is volume of paid services. Another is cost-effective management and TA support. Presently each NGO has a management structure supported by NSDP which is both costly and time-consuming for a TA team to support, and dilutes the support offered. Consolidation of NGO’s will improve sustainability on all counts. There is no magic number of NGO’s to be supported, but it would be good to have a maximum of 25 which will be much easier to support than the original 41 of NSDP. · NGO’s which have proven unresponsive to NSDP assistance should be encouraged to seek other funding sources or simply dropped. · The NSDP network should be kept intact. This may be critical in the long-run since the large urban NGO’s should eventually become sustainable, and able to cross-subsidize the rural NGO’s. · Smiling Sun needs to become institutionalized as a Bangladeshi network that survives beyond USAID funding and is not associated with any particular donor in the future. Clinics and NGOs not previously associated with Smiling Sun should be provided the opportunity to join as a way of broadening the network. · USAID must make it clear to NGOs that, as of the end of the next project, it will no longer be supporting the operating costs of NGOs. Dependence on USAID must end. Declining grants must be a feature of any future project. · Since by definition, “projects” are not sustainable, a key element of the new project should be the creation of a Smiling Sun franchise organization capable of providing the managerial and technical services currently being provided by NSDP. There must be an institutional incentive to belong to a network. Those elements include the following: o The ability to monitor and enforce standards that ensure quality of services. Best practices can be shared through such a network. o Future costs of marketing and maintaining the network can be shared among clinics. o Ideally, there should be a system to cross-subsidize services for the poor that is managed above the clinic and even NGO level. This will allow clinics who serve a larger proportion of the poor to benefit from 37 clinics within the network who serve a higher proportion of paying clients. This must be managed across the entire Smiling Sun network. o A network of clinics can take advantage of pooled procurement of drugs and commodities and therefore take advantage of bulk purchasing. · TA should be integrated and be led by very strong project leadership. Beyond grant management, areas of focus should include continued and strengthened support in: o Financial management and planning, including strategies for pricing, payment mechanisms, cost-effectiveness, diversification of funding, and management of the Equity Fund. o Marketing, both in terms of the Smiling Sun franchise, and individual NGO’s and clinics. o Quality of Services. A strong component should be established in every NGO. o Management of the Revolving Drug Funds o Consolidation of NGO’s and the creation of the Smiling Sun franchise. · The next project should encourage services offered by a single organization rather than a consortium of partners. · The communications component should be better linked to project objectives. · The project management budget should not be tied to the amount of the grants for reasons outlined earlier. A better model would be to encourage the gradual replacement of grant monies by cost-recovery monies. This allows USAID an exit strategy by stimulating NGO’s to increase cost-recovery. The following options are ones that would need to be investigated in greater detail before a final decision is made about how to structure, and what to include in, the follow-on project. The team is most inclined to recommend Option One given the need for simplicity, although Option Two also has some attractive elements. The following would be the basic management structure of either option. 38 USAID Bangladesh Cooperative Agreement US Organization Local NGO Franchise Manager NGO Grants Program 20-25 Large NGOs · Provide TA Declining grants · Conduct period QMS reviews Clinic consolidation · Assist affiliation process Expand services · Assist w/ GOB commodities Expand use of income, RDF Expand cross-subsidies for the poor, seek other donor support Option One : Under this option, grants would be awarded to a smaller number of larger, more established, and high performing NGOs with broader service coverage. Criteria for inclusion would also include availability of ESP services, and commitment to serving the poor. The lessons learned from NSDP about sustainability would be applied to ensure that the NGO operations are as efficient as possible and that financially, they are moving toward more sustainable program. A certain level of partial support would be needed in the form of commodities, for instance, from government beyond the USAID project period. NGOs would be encouraged to compete for contracts from the GOB under the ADB or WB projects but should remain within the Smiling Sun network. The US CA staff would be much leaner. The CA would manage the declining grants program with the stipulation that no further USAID support will be forthcoming. At the same time, the CA would work to develop the capacity of a local entity which would function as the network or franchise manager during the period of the project and beyond. The franchise manager (with help from the CA) would assist NGOs to expand sustainability initiatives piloted in NSDP. Option Two : This option would essentially have the same characteristics as Option One but the grants would not be traditional but rather be performance-based using criteria which can be verified independently. These might include ESP services provided to the poor and or strengthening of outreach and community programs. Reimbursement would be provided for predetermined outputs. Grants would not include overall payment for salaries or operating expenses to encourage sustainability and local decision making. It would provide for yearly certification, based on quality, 39 to continue with Smiling Sun franchise. Option Two would require that an independent entity evaluate accomplishment of performance benchmarks and would require careful analysis of the costs in advance of negotiating the grants. While this option has some advantages, it may too new and complex for current conditions in Bangladesh. Franchise Manager One of the critical elements of any transitional strategy will be not merely the survival of individual NGO’s, but the continuation of at least most of the Smiling Sun network. Given the fact that the glue which holds the present 33 NGO’s together is the NSDP project, should there be no follow-on project, the Smiling Sun network would quickly disintegrate as there would be nothing to hold them together. This would be a tragedy as it is the network itself which will potentially make the services to the poor sustainable through a system of redistribution of income across NGO’s to cross￾subsidize those NGO’s and clinics which are serving a greater share of the poor, and are thus less sustainable standing alone. Particularly if no longer supported by donor subsidies, NGO’s will inevitably cut back on services to the poor in order to enhance and protect their overall sustainability. Clearly what is required is an on-going organization to provide support to the Smiling Sun network which will itself be sustainable. For this reason, we propose that one of the principal objectives of any follow-on project be to develop such an organization. The term “franchise” is used because it should fulfill a number of functions benefiting all members, and be supported directly by the members themselves through annual membership fees. In return, members obtain valuable support, prospering more than if they were on their own. Those functions include those already mentioned: management of the Performance Enhancement Scheme to cross-subsidize services to the poor; the marketing of the Smiling Sun brand; the establishment and enforcement of quality services; and continued TA in support of better services and sustainability Since eventual sustainability of the franchise will depend largely on voluntary membership and fees, it is critical that members of the franchise perceive benefits from their membership. In this sense, the franchise management organization must be able to ‘sell itself”. This is a process which will require time to develop, and for this reason, the development of this organizational capacity must not be seen as an after-thought, but rather as critical to the survival of the network, and begun early in a follow-on project. B. Lessons from NSDP for the Future (1) Graduating NGOs from USAID dependence must be explicit from the beginning and the project structured and managed in ways that provides optimal conditions to encourage the graduation process. Given the mandate of serving the poor and offering ESP services is important and may require some level of subsidy from government and other donors in the future, most likely through the current donor supported NGO contracting projects. 40 (2) The technical assistance provided in the past by USAID CAs needs to be replaced by a local organization which can assist with the quality oversight and technical assistance functions previously provided by the USAID project structure. (3) Clinics, and even satellite clinics, alone will not be able to make much of a contribution to reducing ARI, maternal or neonatal mortality in Bangladesh unless facility based services are accompanied by education programs that reach the communities, both through mass media and through person-to-person contacts at the household level. Depot holders are playing an important role in selling contraceptives and ORS but, except on a small scale, they are not addressing other causes of mortality. Even in urban areas, outreach to communities to link them to the services and follow-up on cases is essential. (4) The next project should include stronger and more proactive programs to ensure that permanent and long term family planning methods are made accessible to those couples who have already achieved desired family size. (5) If feasible, USAID should consider how to strengthen policy analysis and formulation activities within MOH, particularly as they relate to the role of NGOs in service delivery. The current challenges are daunting but USAID could play an important role given its history and credibility in this sector.