IFPS II Evaluation September 2007 This publication was produced for review by the United States Agency for International Development. It was prepared by Joy Riggs-Perla, Anuradha Bhattacharjee, Paula Quigley, and Venkat Raman through the Global Health Technical Assistance Project, and by Sarah Harbison and Mihira Karra of USAID. IFPS II Evaluation DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 07-001-25) is available in printed or online versions. Online documents of GH TECH public reports can be located on the GH Tech web site at www.ghtechproject.com/resources/. Public documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. CONTENTS ACRONYMS .....................................................................................................................iii EXECUTIVE SUMMARY ..................................................................................................1 Evaluation Objectives and Methodology...........................................................................3 Part One: Review of IFPS II..............................................................................................5 Technical and Programmatic Findings..........................................................................5 Project Management Findings ....................................................................................25 Policy, Sustainability, and Institutional Findings..........................................................29 Project Outcomes and Measurement Issues ..............................................................36 Part Two: Recommendations for the Future ...................................................................39 General Strategic Approach........................................................................................39 Priority Technical Areas and Geographic Coverage...................................................40 Relationship with Major Partners ................................................................................42 Management Recommendations ................................................................................42 Annexes Annex A: Evaluation, Scope of Work, and Team Members Annex B: Team Itinerary and People Interviewed Annex C: Documents Reviewed Annex D: Assessment Tools Annex E: BCC Detailed Assessment Annex F: PPP Model Analyses IFPS II Evaluation i ii IFPS II Evaluation ACRONYMS ANC Antenatal care ANM Auxiliary nurse midwife ASHA Accredited Social Health Activist BCC Behavior change communication BISR Birla Institute of Scientific Research BPL Below poverty line CBD Community-based distribution CHACS Comprehensive Health and Counseling Sessions CHC Community Health Center CMO Chief Medical Officer CPR Contraceptive prevalence rate CTO Cognizant Technical Officer (USAID Project Officer) DAP District action plans DM District Magistrate DPMU District Program Management Unit EAG Empowered Action Group FP Family planning GOI Government of India GOJ Government of Jharkhand GOUK Government of Uttarakhand GOUP Government of Uttar Pradesh HLFPPT Hindustan Latex Family Planning Promotion Trust IEC Information, education, and communication IFA Iron folic acid IFPS Innovations in Family Planning Services IMR Infant mortality rate IPC Interpersonal communication IPHS Indian Public Health Standards ITAP IFPS Technical Assistance Project IUD Intra-uterine device JH Jharkhand JHS Jharkhand Health Society JSY Janani Suraksha Yojana LAM Lactational amenorrhea method LMO Lady Medical Officer LOE Level of effort MHC Mobile health clinic MIS Management information system IFPS II Evaluation iii MoHFW Ministry of Health and Family Welfare MMR Maternal mortality rate NFHS National Family Health Survey NGO Nongovernmental organization NHSRC National Health Systems Resource Center NRHM National Rural Health Mission ORV Outreach volunteer PBD Performance-based disbursement PC Project coordinator PIP Performance Implementation Plan PMU Project Management Unit PMP Performance Monitoring Plan PPP Public-private partnership PMSG Program Management Support Group PS Program Support Office PSI Population Services International RCA Recipient-contracted audit RCH Reproductive and child health RCO Regional Contracting Office RFMO Regional Financial Management Office RH Reproductive health SDM Standard days method SHRC State Health Resource Center SIFPSA State Innovations in Family Planning Services Agency SPMU State Program Management Unit TA Technical assistance TFR Total fertility rate TOT Training of trainers TT Tetanus toxoid UK Uttarakhand UAHFWS Uttarakhand Health and Family Welfare Society UP Uttar Pradesh VISTAAR USAID MCH Project VMA Voucher management agency iv IFPS II Evaluation EXECUTIVE SUMMARY USAID/India has supported family planning programs since 1992 through the Innovations in Family Planning Services (IFPS) I Project in Uttar Pradesh (UP) and more general maternal and child health activities since 2002 in UP, Uttarakhand (UK), and Jharkhand (JH). The IFPS II project, which began in September 2004, is scheduled to end on September 30, 2008. In August of 2007, an evaluation was commissioned to assess IFPS II progress, determine whether any changes are needed, and make recommendations about future USAID programming in reproductive health and family planning. Among IFPS II achievements during the first three years are the following: • USAID’s program has evolved to achieve policy consistency and convergence with the objectives of the National Rural Health Mission (NHRM). Activities planned in coordination with development partners in support of NHRM, such as support for the National Health Systems Resource Center (NHSRC), have led to USAID being regarded as an important partner at the national level as well as in the states of UP, UK, and JH. • Beginning with IFPS I, substantial capacity has been built in the State Innovations in Family Planning Services Agency (SIFPSA) in UP, the Society in UP which as envisioned in the NHRM may become a state Project Management Unit (PMU), a State Health Resource Center (SHRC), or both. Among the programs piloted by SIFPSA during IFPS I that have been adopted and expanded as part of the National Rural Health Mission (NRHM) are reproductive and child health (RCH) camps, quality improvement programs, district action plans, and the use of community-based volunteers (now called Accredited Social Health Activists, ASHAs). With expansion, however, some quality issues have emerged that need attention. • Changes in national policy related to public-private partnerships (PPPs) enabled IFPS II to initiate innovative pilot activities and establish flexible mechanisms to test and learn lessons from the pilots for eventual extension through NRHM. Programs of nongovernmental organizations (NGOs) that continued on a reduced scale are still producing useful results. • IFPS II also initiated excellent behavior change communication activities, including national mass media support and state campaigns in support of components of the program. • ASHAs and outreach volunteers (ORVs) within NGO programs are demonstrating that, with good training, they can function well as community motivators and service providers. • Social marketing efforts are producing good results and making pills and condoms more readily accessible to wider population groups in UP. The evaluation team has developed recommendations for the current IFPS II project to ensure that its objectives are achieved and government is more likely to scale it up: IFPS II Evaluation 1 • The PPP pilot work must be allowed to continue for a full three years to allow for the learning, analysis, and documentation necessary to convince state governments to scale up the efforts. This means IFPS II should be extended for two years beyond the scheduled end date of September 30, 2008. Both performance-based disbursement (PBD) and the IFPS Technical Assistance Project (ITAP) need to continue until September 2010. • Recommendations for each of the current PPP pilots are provided to ensure that lessons already learned from some pilots are applied by others. For those that have not yet started, areas deserving closer monitoring are suggested. The ITAP contract needs to be used to analyze and document experience, determine cost￾effectiveness, and estimate costs of scaling up the models. USAID will need to advocate proactively with state governments as the necessary information becomes available. Coordination and oversight at the district level between PPP project staff and district government counterparts need improvement. • Behavior change communication should be strengthened by linking messages and materials among the components of the IFPS II program and analyzing the impact of efforts more closely. • Other technical assistance (TA) activities supported under the program should also be reinforced by increased technical inputs from ITAP. Scaled-up models from IFPS I and II require continuous monitoring for quality to ensure that the impact of activities is not diluted. ITAP should be held accountable for TA and capacity development results rather than for reaching general program targets, such as contraceptive prevalence or total fertility rates. • Continuing support to state-level societies is critical. SIFPSA is at a crossroads. USAID should continue its support through 2010 and help implement institutional reforms that will wean SIFPSA away from dependence on USAID. In UK and JH, USAID and its development partners should work to build the capacity of the societies chosen by the state governments to support NRHM but avoid creating dependence. For the future, the evaluation team recommends that USAID move to a mode of technical collaboration with the Government of India (GOI) at both national and state levels through mechanisms like as ITAP but without the PBD feature of IFPS I and II. PBD should be continued only long enough to finish out the PPP pilots. There is a continued demand for USAID technical support and capacity development at the state level. Given the potential of India’s extensive private sector institutions and providers, work on new PPPs should continue, but carefully selected TA for public programs is also recommended because of USAID’s comparative advantage in such areas as quality improvement, training and skills development in family planning (FP), and behavior change communication. USAID should also continue to work at the national level to ensure that FP and reproductive health (RH) remain at the center of the NRHM. 2 IFPS II Evaluation Evaluation Objectives and Methodology The IFPS II evaluation, undertaken in September 2007, one year before the project is scheduled to end, was intended to assess progress, suggest any adjustments needed, and offer recommendations to USAID about future directions in RH and FP. Another team, working closely with USAID staff in late September, is designing the follow-on program. This report is also intended as background material for the design team. Specific objectives for the evaluation listed in the scope of work were to • Assess program strategies and technical approaches for reaching results in FP and RH • Review how successful the project was in demonstrating innovative schemes and in advocating for adoption and scale up of those schemes • Review implementation mechanisms and assess the effectiveness of project management systems • Identify lessons learned • Recommend any changes needed in IFPS II and technical approaches and strategies for the future The analytic framework for the evaluation (see Annex D) was based on the project outcomes, objectives and results, and components identified in the New Activity Description for IFPS II dated May 2004. The evaluation team used the following methodology: • Interview of stakeholders, using a question guide (see Annex D) • Analysis of data from the project and other sources • Review a variety of descriptive and analytical documents related to the project • Evidence gathered during field visits The evaluation team consisted of a Team Leader and five others with expertise in: public health impact assessment, FP and RH, PPPs, behavior change communication, and health policy and systems. Field visits to view activities and interview stakeholders were conducted in UP, UK, and JH; the six –person team was organized into three pairs that conducted simultaneous field trips to maximize exposure to field activities. (Notes from these field visits are available from USAID as a separate document.) IFPS II Evaluation 3 4 IFPS II Evaluation Part One: Review of IFPS II TECHNICAL AND PROGRAMMATIC FINDINGS Public Sector Technical Activities Quality Improvement: The IFPS I summary evaluation report (April 2003) highlighted the quality of sterilization and FP/RH services generally as an area that needed additional support. A specific area of concern has always been the lack of quality assurance processes within public health facilities. In response, in IFPS II USAID made strengthening the public sector to manage quality assurance a specific objective. As a result, SIFPSA has created a quality assurance tool to be used by public sector staff in collaboration with the health facility and the community with TA from SIFPSA. The intent is to highlight quality improvement (QI) issues in primary health centers and community health centers (PHCs/CHCs) so that supervisors can make improvements. The SIFPSA trial provides initial funds to make these improvements. The range of issues covered includes equipment, supplies, and facilities. The concept also calls for repeated use of the tool to highlight new problems as old ones are resolved. The evaluation team visited one CHC in the Sitapur district of UP to view QI activities. Accomplishments: • The QI concept is excellent and sorely needed. SIFPSA has designed a collaborative process that involves facility staff, supervisors, and community leaders. • A simple checklist has been drafted and piloted. • Interest and buy in from the Government of UP (GOUP) and GOI is high. • This initiative complements the GOI Quality Improvement for RCH services being launched in UP and several other states. Together these two activities reflect increased GOI commitment to broad and systematic improvement of the quality of public facilities. Issues: • The tool developed by SIFPSA is focused on equipment and infrastructure. Quality of provider counseling and other behavior is not addressed. Specific emphasis on the quality of provision of FP/RH services is absent. • The tool does not appear to be informed by current international experience in QA or by the excellent tool developed at the national level by the GOI and UNFPA with TA from the Population Council, EngenderHealth, and PATH. IFPS II Evaluation 5 • The SIFPSA process fails to include any guidelines for supportive supervision of providers. • The process has not yet been systematically linked to the untied funds available at facilities for QI under NRHM/RCH II. • Although some start-up funding may be necessary for QI, any process that includes special funds upfront for equipment and infrastructure raises issues of sustainability and the potential for scaling up. Recommendations: • The state government should be encouraged to take a more comprehensive and systematic approach to QI, which should be reflected in the State Performance Implementation Plan (PIP). • SIFPSA should consider revising the tool and the process in consultation with the TA partners who developed the tools and processes for the national QI initiative. The tool needs to include sections on client provider interaction, client feedback, provider needs, availability and quality of information, education, and communication (IEC) materials, and specific sections on FP/RH content of these interactions and materials. • SIFPSA should reconsider how to structure funding for any required quality improvements and specifically consider how to link the quality concerns raised during these processes with the resources available under the NRHM. IUD Training: Intrauterine devices (IUDs) have been part of the Indian FP program from its inception, but the method is grossly underutilized due to the usual issues of public myths and misperceptions, provider biases and lack of skills, and until recently lack of availability of the CuT380A, the most effective IUD. The GOI is very committed to increasing awareness and use of this method; several activities to revitalize its use throughout the country have been started by the GOI and the IFPS II project. These include branding IUD 380 A as Suvidha for markets in UP and working with the GOI to ensure that it is available in UP; creating the Suvidha social marketing campaign; SIPFSA IUD trainings in IFPS I and II; and NGO provision of IUDs through programs supported by SIFPSA. More recently, the GOI, deciding to take a more comprehensive national approach to increase the availability of the IUD, approached the USAID Mission to provide TA for national state-of-the-art training of trainers. USAID and GOI, along with a core technical team, drafted an Alternative Training Methodology for repositioning the IUD in the National Family Planning Program. JHPIEGO was contracted to provide this training to trainers from 12 states. They used the latest techniques for improving IUD insertion skills, such as the Zoe model, skills-based training methodologies, and new IEC materials and tools. 6 IFPS II Evaluation Accomplishments: • GOI has made a commitment to revitalize the IUD. • JHPIEGO provided training of trainers (TOTs) for 60 public sector staff from 12 states. The quality of the training was excellent and the response to the GOI TA request was very prompt. • A comprehensive cascade training is planned shortly to have trained providers for the IUD in one district in each of the 12 states. • A comprehensive review of all IEC and training materials related to the IUD was completed through an extensive participatory process as a precursor to designing national IEC tools and guidelines. Concerns: • The GOI recognizes that demand creation will lag behind the training of providers. There is a concern that provider skills will erode if there is no client flow due to the lack of demand. • The GOI is quite clear that the IUD should be provided through static facilities, not through a “camp” setting. The experience of NGOs suggests that a range of nonstatic modes of providing IUD is feasible and would increase IUD use through quality services. Recommendations: • Demand creation should be coordinated with provider training. The behavior change communication (BCC) strategy for promoting IUDs should include both public and private provision of services and community-level outreach through ASHAs, auxiliary nurse midwives (ANMs), and NGO outreach workers. • IUD training offers the opportunity to upgrade provider knowledge of all other methods. As in standard days method (SDM) training, it is important to have a day of contraceptive technology updates (CTUs) for all providers no matter which method is the focus of the particular training event. • There is experience from around the world that suggests that in countries where IUD use is very low, the Centers of Excellence model, which takes a phased approach to training of providers, has a higher chance of retaining the skills of providers due to higher client loads in a few select facilities. This approach needs to be explored in the context of the GOI’s plan for broad scale IUD training. RCH Camps: IFPS I piloted the RCH camps with the objective of providing integrated services at the community level, including FP/RH services and intensive tetanus toxoid (TT) and iron folic acid (IFA) campaigns. SIFPSA contracted with lady medical doctors and ANM paramedical providers in these camps to provide some clinical services, such as sterilization. In IFPS II the RCH camp concept was adopted and adapted by the GOUP. IFPS II Evaluation 7 Currently, the camp is set up at the CHC once a month (twice in winter). On that day the full complement of staff are present to provide comprehensive services with an emphasis on sterilization. This move to have the camp at a CHC was a result of concerns about sterilization quality in a more temporary setting, and the lack of trained government personnel at the CHCs on a regular basis. The evaluation team visited an RCH camp in a CHC in Sitapur district in UP. Accomplishments: • The concept is very good and the GOUP’s commitment to funding this innovation after the IFPS project is commendable. • SIFPSA is tasked with monitoring the quality of these RCH camps in 70 districts, which will provide the GOUP with an independent audit. • The community health and counseling sessions (CHACS) are well located (in schools), are very well attended and well staffed, and regularly offer a broad mix of services. Collaboration with the community workers is excellent. FP/RH spacing methods are being provided, with referrals for sterilization and IUDs where insertion is not feasible. Both free and socially marketed products are available. Concerns: • While the reasons for moving the RCH camps back to a static facility (CHC) are understandable, the concept of using a more accessible location within the community is lost. • The CHC should be able to provide routinely, throughout the month, all the services that are provided on the day of the camp. While this is outside the mandate of IFPS II, the GOI should be upgrading CHCs to Indian Public Health Standards (IPHS) as an essential component of the NRHM program. Recommendations: • The PPP concept can be exploited to move the RCH camp location back into the community. Contracting out the RCH camps to NGOs or other private organizations to follow the CHACS model should be considered until fully staffed CHCs can mobilize their own staff to go into the field periodically to provide outreach services. • Independent organizations like SIFPSA should continue monitoring and evaluation (M&E). • USAID, with other partners in UP, should help the GOUP, perhaps through ITAP, to develop a realistic plan for staffing all CHCs and bringing them to IPHS status over time. DAP/PIP Capacity Development: The concept of drafting district action plans (DAPs) based on specific local context and circumstances was formulated by SIFPSA during IFPS I. From 1997–2004, SIFPSA worked with district public health officials to draft plans for the 33 districts where SIFPSA was implementing programs. Under IFPS II the 8 IFPS II Evaluation GOI picked up the action planning process as part of NRHM and RCH II. SIFPSA’s role shifted to providing TA to district management units in developing the plans and has guided the drafting of 70 district PIPs in partnership with the GOUP and stakeholders for the UP NRHM. In JH, ITAP contract staff have provided TA for drafting the state PIP (ITAP Jharkhand presentation) and DAPs in 22 districts. The process included collection of primary and secondary data for the district, consultations with stakeholders, a workshop on the DAP process, draft planning, and final plan development, including budget allocations. Similarly, in UK, ITAP staff provided TA to the Government for the drafting of the state PIP and 13 district RCH II plans. Accomplishments: • The systematic planning process started in IFPS I has been institutionalized and expanded by RCH II. • Stakeholder involvement at the district level seems to have improved significantly in some places, as is particularly evident in the JH documentation. • Many plans are based on analysis of district survey data, service site mapping, availability of commodities, and service statistics. Concerns: • How involved stakeholders are in preparing DAPs varies. They seem to have been more involved in JH than in UP, according to the process documentation. • Village action plans (a mandate of the NRHM) have yet to materialize. • The priority given to FP/RH services is very variable. For example, the DAP for Dehradun has output and outcome indicators specified for many services related to malaria, immunization, etc., but fewer and not comprehensive indicators for FP/RH. There are no outcome/output indicators for use of different methods, such as pills and condoms (Dehradun DAP). • Where objectives for a district relate to population and FP, the level of detail for DAP strategies and process/outcome indicators related to FP/RH are less than those for other health areas (Saharanpur district PIP). • Until the DAPs are implemented and M&E processes and units are set up, it will be impossible to estimate the impact of the DAPs and draw up future plans. Recommendations: • Village-level planning should be the starting point for future DAPs. • More consistent stakeholder involvement at all levels would enhance the DAP process. • TA for the FP/RH content of DAPs/PIPs needs to be improved. IFPS II Evaluation 9 • ITAP can help the states to monitor implementation of the DAPs and improve participation in the next cycle of planning Maternal and Infant Death Audit: The NFHS 3 identified the high infant and maternal mortality rates as major issues for JH. As a result, the IFPS II project drew up plans with the GOJ to conduct a maternal and infant death audit. GOJ, CARE, and ITAP signed an agreement to implement this audit in five JH districts. The objectives of the study are to identify medical and social causes of mortality; map medical, systemic, and social issues that may be contributing to mortality; improve district planning and prioritization based on information from the audit; and improve coordination among agencies involved in maternal and child health services. Comprehensive stakeholder involvement was integral to the study (ITAP Jharkhand presentation). The JH Secretary of Health, Director of Health, Secretary of Social Welfare, other senior officials and representatives of CARE, ITAP, UNICEF, and VISTAAR reviewed data from the NFHS, planned the study, and provided resources. The District Collector, Civil Surgeon, District Panchayat Officer, representatives from other departments, CARE, ITAP, UNICEF and other stakeholders will be monitoring progress during the audit, reviewing audit reports, and initiating appropriate responses. At the community level, grassroots workers, such as CARE change agents, Anganwadi workers, other community volunteers, school teachers, and opinion leaders, will help identify cases of infant and maternal deaths during the last three years and advise on actions to follow up the audit. Accomplishments: • Pilots were completed earlier this year and the results were used in drawing up study instruments and procedures. • The state ethics committee has approved methodologies and instruments. • Training of interviewers was scheduled for August 2007. • Stakeholder commitment at all levels seems to be excellent. • The plan to train one male and one female interviewer in each block is very sound. Concerns: • There is great pressure to jump to solutions to urgent problems before the audit is complete. • Large studies like this require intensive follow-up and monitoring of data collection and analysis. Recommendations: • Solutions to problems should be identified with the help of all stakeholders after all data has been collected and thoroughly analyzed. • The buy-in from stakeholders should be maintained as interventions are designed and implemented. 10 IFPS II Evaluation • A more accurate method for collecting routine birth and death information at the village and district level can be found based on audit results. The project could train community-based field workers to improve their data collection skills and collaborate with district birth/death registration officials to improve supervision of community workers (usually Anganwadi workers and possibly ASHAs in the future). • All partners should consider incorporating mechanisms into the public health system to continue verification of causes of death. Public-Private Partnerships A major objectives of IFPS II is to design, implement, and document models of public￾private partnerships to improve access to and use of integrated RH services, child care, and family planning services. A comprehensive package of services would include antenatal care (ANC), safe motherhood, institutional deliveries, newborn care and support, treatment of sexually transmitted diseases, promotion of contraceptives, access to emergency care services, promotion of referral systems, and behavioral change through IEC, BCC, and interpersonal communication (IPC). IFPS-II has initiated six PPP models: • Access to RCH services through camps organized by NGOs (UP) • Provision of services through mobile health clinics (MHCs) in Ramnagar (UK) • Access to private services through voucher systems in Haridwar (UK) and in Agra and Kanpur city (UP) • Access to services in designated franchised clinics (UP) • Access to contraceptives through social marketing (UP) • Contracting out government health facilities to the private sector (UK and UP). While some of the PPP initiatives are underway, others have yet to be started. The following is a brief description of each model with recommendations emerging from the evaluation (see Annex F for a more complete analysis of the PPP projects). MODEL 1: NGO Projects (Uttar Pradesh–Agra and Kanpur) One of the major challenges in providing health services in rural UP is reaching remote villages and urban slums. In UP 80 percent of people live in villages and about 5 percent live in slums. The IFPS I project recognized the importance of NGOs in reaching these areas and initiated NGO projects to pilot and expand the CBD concept in UP through SIFPSA. These projects validated the feasibility and impact of providing FP/RH services through volunteers. In IFPS II the NGO projects were designed to cover a wider range of services, design approaches tailored to the locality, and reach more people. Major themes for these projects were expected to be • promoting informed choice • increasing the mix of methods • mobilizing community support IFPS II Evaluation 11 • creating demand for FP/RH • providing spacing and terminal methods • providing maternal health services, including emergency obstetric care (EOC), institutional deliveries, and postpartum care • partnerships with private clinics for services • improving sustainability through cost recovery measures. As the new ASHA cadre was instituted by NRHM, the NGO projects were revised to include linkages with the ASHAs and withdraw financial support to private hospitals for clinic-based services. The IFPS II began its one-year NGO projects in October 2006. The 24 projects were to cover 45 rural blocks and two urban slum areas in 11 districts of UP. The evaluation team visited two NGOs in Kanpur—the Amin Charitable Trust, which covers urban slums, and another that provides services in rural areas near the city—and one in Agra, NIRPHAD. The Amin Charitable Trust has worked in the FP/RH field with SIFPSA since IFPS I. Its staff have experience in working with male and female volunteers in FP/RH in employment-based areas, such as tanneries. NIRPHAD, which has been active in development for over 25 years, runs three hospitals in Agra district and is active in the urban slums there. All three NGOs have a similar strategy of using either their own outreach volunteers (ORVs) in urban areas or ASHAs in rural areas to help motivate clients, provide spacing methods such as pills and condoms, and refer for clinical services to an ANM or doctors in private and public clinics. The ORVs and ASHAs also provide ANC counseling and IFA. Where feasible they escort clients to clinics for institutional deliveries and for other clinical services such as IUD insertions and sterilization. They also refer clients for childhood immunizations and TTs. These NGOs also hold monthly comprehensive health and counseling sessions (CHACs) led by a contracted female doctor and ANM. CHACs are expected to cover a population of about 20,000 in rural areas and one slum each in urban areas. The range of services they provide are counseling, IEC and services for ANC, TT vaccinations, IFA distribution, promotion of institutional deliveries, immunization of children, distribution of pills and condoms (either free or social marketing bands), IUDs and referral for sterilizations, screening and referral for STDs/HIV, basic curative services, and where appropriate partnerships with private clinics for the new voucher schemes. ASHAs or ORVs trained and supervised by the NGO also escort or refer clients to these CHACS for services. Their NGO supervisors are in turn supervised by the NGO assistant project coordinator (APC) and project coordinator (PC). SIFPSA provides technical assistance in training, supervision, management systems, and M&E to the NGOs. The PC is expected to coordinate with the chief medical officer (CMO) and/or local PHC/CHC for supplies of vaccines and contraceptives, oral rehydration salts, etc. 12 IFPS II Evaluation Accomplishments: • Building on lessons from IFPS I SIFPSA has created a good mechanism and process for involving NGOs, both new and experienced. • NGO staff seem to be committed to the objectives of the program. ASHAs and ORVs seem to be consistently motivated and knowledgeable. • CHACs are being held regularly in good locations, such as schools, and are drawing large crowds. • Although only the PCs, APCs, and supervisors have been trained in the SDM, many ASHAs have become aware of the method and are eager to provide it. Concerns: • The quality of management information systems (MIS) and other management systems, such as supervision, is a concern. • So far the State Institute for Health and Family Welfare (SIFHW) has trained ASHAs only in module 1 out of 5. SIFPSA was allowed to use its own curriculum for ORVs and do minimal additional training for ASHAs in the SIFPSA NGO blocks on SDM and family planning methods. Neither SIFPSA nor SIFHW training appears to be of the same quality as the ITAP training of ASHA Plus in UK. • ASHAs and ORVs have not yet received any IEC materials or tools. In response to a GOUP recommendation that all ASHAs in UP receive uniform training materials and job aids, SIFPSA developed flip books that were ready during the week of the evaluation but had not yet been distributed. Moreover, the flip book needs some revision because of technical errors/omissions. For example, the pictorial depiction of all services provided by ASHAs on the first page does not have FP. The amenorrhea criterion, the most important one for the lactation amenorrhea method (LAM), is missing from the pictorial representation of LAM, and postpartum services do not include FP in the pictures. • There appears to be some variation in how ASHAs are remunerated across states. In UK the ORVs appear to be remunerated more along the lines of the ASHA Plus program. • Remuneration for the IUD and sterilization far outweigh any minimal margin that the ORVs and ASHAs are allowed for pills and condom sales. This may lead to a lack of attention to promotion of the short-term spacing methods. • There are some general problems with ASHAs getting reimbursed by the public sector. • A major issue for the ASHAs is lack of transportation for institutional deliveries for their clients. • The relationship between the ANMs and ASHAs may be a problem. IFPS II Evaluation 13 Recommendations: • The quality and impact of ASHA training in all three states needs to be assessed once all the modules have been implemented. IPFS II can do much to improve the quality and impact of ASHA training and should explore the feasibility of doing so by having SIFPSA and ITAP work with SIHFW on this issue. • There is a serious need for male volunteers in all projects. SIFPSA should help NGOs build on past experience with male volunteers and add them to any future program. • IEC materials and other tools need to be supplied to the ASHAs and ORVs. The flip book needs to be revised in the next phase. • SDM and LAM should be added to all NGO projects in all three states. • Innovative ways are needed for reimbursing and giving credit to ASHAs and ORVs for pill and condom clients. • There are many opportunities to include a broader spectrum of postpartum services, especially FP/RH, in NGO programs. Postpartum programs should be streamlined and should all include FP/RH. • Innovative mechanisms for providing transport for institutional deliveries are needed. • There should be a follow up of the serious issues and recommendations identified in ITAP’s recent assessment of NGO projects and SIFPSA management. MODEL 2: Mobile Health Clinic (Van), Ramnagar, Nainital (Uttarakhand) The MHC model is being piloted in Nainital District, UK. This model is particularly appropriate for providing basic RCH services in hard-to-reach areas like the hilly villages in Ramnagar, Nainital. The Birla Institute of Scientific Research (BISR) has a partnership arrangement with the IFPS II project to implement this scheme. BISR’s past experience in providing diagnostic services through this model was the primary reason they were chosen for this pilot. The plan is for BISR to use the MHCs to provide diagnostic, RCH, and referral services including provision of spacing methods of FP; ANC and PNC services; immunizations; etc. This pilot will be operational as soon as the van is fitted with equipment and staff are hired and trained in providing all the specified services. The referral system calls for close collaboration between the van and public and private sector static facilities so that clients can be referred for sterilizations and other clinical services requiring more intensive follow-up. Accomplishments: • This model has potential for reaching geographically isolated populations. • It is based on past successful experience in providing diagnostic services. • The GOI and GOUK are highly supportive and there is a plan to expand the model once the pilot is operational and providing data for decision making. 14 IFPS II Evaluation Concerns: • The launch of the pilot was delayed due to faulty equipment. • There seems to have been no communication between BISR and the district officials, so planning was not coordinated. • Staff have yet to be hired and trained on all the needed preventive services. • Mechanisms for referrals between the mobile clinic and static facilities have not yet been established. • No plans have been made for outreach to communities to be affiliated with the mobile clinic, an essential element for increasing utilization of its services. • While there are reporting forms, BISR does not yet have an MIS system to track and report service use. Reporting forms at present apply only to diagnostic services, not RCH/FP services. • The plan for use of the revenue from user fees is unclear. Recommendations: • It is necessary to create a mechanism for close coordination between block and district health personnel and the mobile clinic. • Training and reorientation of project managers and the mobile clinic staff on FP/RH and maternal and child health services is urgently needed. • There should be a system for utilizing ASHAs and other community volunteers. It is essential to have some male volunteers associated with the mobile clinic. • Referral systems between the MHC and static facilities need work. • An MIS system needs to be developed and all staff need to be trained in its use. • A channel for getting supplies from the CMO is needed immediately. • The MHC should design a flexible staffing pattern so that services, such as IUD insertions, can be provided. Most of the services can be provided through lower￾level staff, but some require female medical officers or ANMs. • If IUDs are to be inserted, a plan is needed to address women’s privacy and equipment sterilization. MODEL 3: Voucher Scheme (at Haridwar, UK, and Agra, UP) The IFPS II project is piloting a voucher scheme in Haridwar, UK, and Agra and Kanpur in UP. The evaluation team visited pilot sites in Haridwar and Agra. The government’s objective is to enable lower-income families (below poverty line, BPL) to access quality services in the private sector. BPL families are given vouchers to cover the costs of specific services such as ANC, delivery, postpartum checkups, childhood immunizations, FP, IFA tablets and TT immunizations. IFPS II Evaluation 15 Some essential components of the scheme are formulation of accreditation, regulation, and quality control systems for the private health facilities involved; design a system for working with village officials to identify BPL families; and increasing the pool of service providers. The proposal for the pilot scheme clearly defines the roles of each partner: the private sector helps government and local officials identify families; and the NGOs manage and monitor the scheme and the volunteers, such as ASHAs and NGO outreach workers. Basically, the scheme calls for the public sector to issue vouchers to NGOs, which in turn distribute the vouchers to the BPL families through ASHAs and outreach volunteers. The BPL families submit the vouchers at participating health facilities when they use a particular service. There is a separate voucher for each type of service. The facilities then submit the vouchers back to the public sector to get reimbursed. In UK the NGOs also monitor the quality of services and the equity of voucher distribution; in UP the SIFPSA district offices monitor the scheme. Accomplishments: • In Agra, the Government of UP, NGOs, and Agra Medical College working as partners have successfully launched the scheme in seven rural blocks. Ten private facilities have been accredited and are providing services. • In Haridwar, the model of using an NGO to monitor the scheme and provide feedback appears to be highly effective in maintaining quality and equity. • In both states ASHAs have been well trained on the voucher scheme and are highly enthusiastic about it. • Client satisfaction in both states is very high, and demand for services is growing rapidly. Concerns: • A smaller experiment in Agra is examining whether the public CMO office could manage the scheme rather than NGOs. The roles and responsibilities of the CMO office do not seem to be clear in this block. • The rate of cesarean deliveries in Agra is excessively high (approximately 40% of all deliveries). • The IEC/BCC component of the scheme is weak. • District officials in Agra are supportive of the program but unaware of the details. • Due to the high demand, the number of vouchers printed appears to be insufficient and is causing some delays in Agra. • In Agra there is delay in getting supplies to the facilities and ASHAs from the public sector. • Panchayat members in UP are putting pressure on ASHAs to issue vouchers to families who do not qualify as BPL. 16 IFPS II Evaluation • The vouchers do not cover some essential postpartum services in UP, where extra blood requirements and medications are not covered. • Patient feedback is not as well implemented in UP as in UK. • There is poor coordination between the private facilities and public referral sites for services other than those covered by vouchers. Clients have been refused public services because they have used the private sector for services covered by the vouchers. • In some private facilities, the staff have been instituting separate hours for BPL clients and their regular clientele, raising concerns about quality and equity. Recommendations: • There is good potential to use client waiting time for BCC/IEC activities in the health facilities. Materials developed by the IFPS II project for other parts of the program should be made available for this purpose. • Concerns to be followed up on include high proportions of cesarean deliveries, long waiting periods at some sites, segregation of BPL patients, and inadequate supplies. • The NGO model for monitoring the scheme seem to be more efficient in UK than that in UP. Lessons learned should be shared. • Demand for this scheme is high and rapidly growing; in response the project should consider accrediting more private facilities or expanding existing facilities. • The types of services covered by the vouchers may need to be expanded to meet all the needs of postpartum clients. • Reimbursement to ASHAs needs to be streamlined promptly. • The current scheme of remunerations to ASHAs places a higher value on some FP methods, such as IUDs and sterilizations, which could skew the method mix. A revised remuneration plan that provides incentives for counseling and use of other spacing methods should be considered. • Before expanding the voucher scheme, a more comprehensive inventory of private facilities is recommended. MODEL 4: Social Franchising, Life Spring Hospital, UP Another interesting model the IFPS II project is piloting is social franchising of private health facilities to increase access. The Hindustan Latex Family Planning Promotion Trust (HLFPPT) is the implementing partner. The organization owns two hospitals in Agra and Kanpur, the Life Spring Hospitals, that will be considered the top tier (L0 level) of the model. The L0 level is expected to have 20 beds and provide comprehensive FP/RH, maternal and child health (MCH), and other preventive and curative services, including all clinical services, such as deliveries and sterilizations. At the next level down are the L1 hospitals. These will be branded as MerryGold facilities. HLFPPT expects to franchise 70 such facilities with as many new entrepreneurial doctors. HLFPPT will facilitate loans for these entrepreneurs from institutions partnering in the scheme to IFPS II Evaluation 17 establish the facilities and provide technical assistance to the L1 facilities in such areas as management, technical competence, monitoring and evaluation. The L1 facilities will mirror the L0 facilities in services and will be located at the district level in the outskirts of the city or in peri-urban areas. At the third level, below the L1, the scheme calls for fractional franchising of 700 existing private clinics run by allopathic doctors that can provide most of the services that L1 facilities can provide except for cesarean sections and other complicated clinical procedures. They should be able to provide all routine preventive MCH and FP/RH services, including IUD insertions. These facilities will be branded as MerrySilver and located at the block level. They will have a system for referrals to MerryGold facilities. Finally, at the lowest level, the project will train 10,500 Tarang agents. Of these, AYUSH practitioners, who have formal training in indigenous systems of medicine, will be marketed as MerryTarang agents and RMPs will be considered Tarangs. These practitioners are expected to be the first line of contact for clients and will be trained to provide services such as condoms and pills, IEC and IPC for FP/RH and RCH, test kits for pregnancy, and malaria and DOTS medications. They will be expected to refer their clients to higher levels in the system as appropriate. The scheme is expected to charge fees at all levels. Financial projections from HLFPPT show the scheme breaking even in the fourth year. USAID is expected to fund the scheme for three years. User fees will be placed in a project fund to cover costs after USAID funding ends until the scheme breaks even. Accomplishments: • IFPS II undertook extensive preparation before awarding the contract to HLFPPT. Activities included literature reviews, consultations with experts and all stakeholders, an international workshop, and a review of similar schemes elsewhere in India. Concerns: • Staff confirm that one of the L0 hospitals in Agra is underutilized. The hospital has been in operation for a year and does not seem to have been marketed adequately to increase clients. • Staff at this hospital were not aware of the new scheme. • Staffing patterns and overheads of staff salaries in L0 and L1 facilities are a concern for sustainability. Recommendations: • The project has enormous scope for increasing access to critical services. Location of facilities selected needs to be carefully considered to ensure maximum utilization. • Linkages to voucher schemes, Janani Suraksha Yojana (JSY) and other such programs are essential to success. The only way the IFPS II project will achieve its goal of reaching BPL clients is through such linkages. 18 IFPS II Evaluation • To ensure success the scheme will require intense supervision, monitoring, and technical assistance from HLFPPT and ITAP during the first few years. MODEL 5: Contracting out UK Public Health Facility Contracting out is one of the most common forms of PPPs around the world. This model involves handing a public facility over to a private partner to manage. The government gives the private partner, usually an NGO, a budget for staff, supplies, and maintenance costs. The private partner must then provide all the services the public sector would have provided. Karnataka and Gujarat have successfully piloted this model in India. Under IFPS II, UK decided to pilot it in the Yamkeshwar block of Pauri district. Accomplishments: • ITAP has completed a preliminary assessment of facilities in the block for potential contracts and prepared a draft contract. • A study tour for government officials to other states is being planned before the model is implemented. Concerns: • UK government officials are still hesitant to start implementation because of the political sensitivity of health care and the state’s lack of experience with this approach. Recommendations: • The study tour planned needs to be completed as soon as possible and ITAP should continue its technical assistance throughout the process. MODEL 6: Commercial Social Marketing (CSM), UP One of the main objectives of IFPS II is to build on the positive outcomes of IFPS I social marketing activities. The goal is to increase access to and demand for pills and condoms in rural villages in UP. DKT is the implementing partner under direct contract from ITAP. The types of activities implemented by DKT include mass media promotion of methods and branded products; press releases, advertisements, bus panels, billboards, and wall paintings; and market town activities, such as street plays, magic shows, and community meetings. DKT also trains and closely supervises marketing agents across the state. DKT is a highly experienced social marketing firm of international repute. The project is expected to reach 69,000 category C and D villages in rural UP between April 2007 and March 2008. DKT is expected to sell 145 million condoms and 2.82 million cycles of pills; hold 350 market town activities; place 234 TV spots and 1,350 radio spots; and hold 1,200 community and 50 gram panchayat meetings by December 2007. In addition to its sales representatives, DKT also uses ASHAs, Anganwadi workers, medical practitioners, and retailers to sell their products. IFPS II Evaluation 19 Accomplishments: • The market town activities appear to be well-designed and contain all the appropriate messages FP/RH as well as other health issues. More than 142 such activities have already been completed, and the events are well-attended. • 66 gram panchayat meetings have so far been held, with nearly 800 participants. More than 2,500 community meetings have been conducted—well above the target. • Sales figures seem to be consistent with the project objectives. Concerns: • As with any social marketing project, it is difficult from the project data to understand whether the products are reaching the targeted beneficiaries. Recommendations: • Special studies to understand whether the products are truly reaching beneficiaries are recommended. • Continued technical assistance from DKT and ITAP is essential for any scaling up of social marketing efforts. Behavior Change Communication The Assessment Summary Report on Innovations in Family Planning Services, Phase I (IFPS I), covering 1992–2002, stated: Communication support for service delivery initiatives and behavior change has been the weakest project activity. When campaigns were developed and implemented (such as for the contraceptive social marketing effort and for auxiliary nurse-midwife insertion of IUCD), they were initiated relatively late in the project. Frequently, communication campaigns were not developed and implemented in support of project service delivery activities. The BCC assessment covers IFPS II activities to date in the three states. Besides assessing the contribution of BCC in increasing demand for and uptake of RH/FFP products and services in the states, it also looked at contributions to the objectives of the national RCH-II program and the NRHM, the expanded IFPS II project focus of increasing PPPs, and ITAP assistance in developing communication strategies and campaigns by ITAP. Accomplishments: • There is broad commitment to developing communication support for the awareness-building objectives related to specific health issues, such as institutional deliveries, maternal and child health, family planning, age at marriage, and immunization, as well as the promotion of specific products and services, such as condoms, oral contraceptives IUDs, and sterilization. • Integrated communication strategies have been developed for each state, in line with the project objectives related to communication for behavior change. These 20 IFPS II Evaluation were finalized after consultations with Government officials and other stakeholders, with technical assistance from the Johns Hopkins University Center for Communication Programs (JHUCCP) and ITAP, and after formative research into media habits and exposure, and attitudes and behavior related to RH/FP issues. • For national BCC campaigns to support the NRHM, ITAP created a series of TV spots for the IEC division of the Ministry of Health and Family Welfare (MoHFW). These were supported by radio spots based on the sound track of the TV spots. The TV spots were aired from May 2006 through October 2007 over a range of satellite channels and the terrestrial channel, Doordarshan. The radio spots were aired a few months after the films. • In January–February 2007 ITAP carried out a population-based survey through an independent research agency to gauge the visibility, message recall, and comprehensibility of the TV spots. The study confirmed that “nearly three-fifths of the respondents in each of the target groups who were exposed to the TV spots found them to be effective and to convey the desired message.” • The NRHM films and radio spots have clear messages and good production values, and information about the different health issues were communicated as intended. • In UP: o The Suvidha Campaign (IUD), Female Sterilization Campaign, and two Radio Enter-Educate Serial Programs built awareness of the range of RH/FP and MCH health issues, promoted use of products and services by the general public, and improved the image of ANMs as service providers. o Folk Media, which is being used successfully in UP, has been incorporated into the communication plans for the other two states. The evaluation team observed two magic shows, one of the crowd pullers in the IFPS II folk entertainment genre and also used very successfully in IFPS I. Magic shows, along with other forms of folk media, are a proven communication channel for taking messages to the deepest rural areas and facilitating engagement with the target audience. The performers successfully incorporate messages on FP, RCH, and MCH into their repertoire in a way that is much appreciated by the target audience. • BCC in UK: o Two campaigns are being created, on institutional delivery and immunization, after baseline studies on health and behavior change issues. o For the PPP pilot, a brand and logo have been designed for the voucher scheme, with accompanying IEC material for the ASHAs. • BCC in JH: o Two campaigns are being created, on institutional delivery and birth spacing, with clear communication objectives and expectations of behavior change. IFPS II Evaluation 21 (Detailed information can be found in Annex E.) Concerns: • The NRHM tracking study showed that recall and comprehension of the TV spots vary across different spots and target segments. Because the target audience is not homogeneous, different groups within it have different communication needs. Better client segmentation is required in designing and placing TV spots. Though TV is an important medium and the most efficient mass media channel, the reach among the rural audience, although growing, is limited. • IFPS II has is no overarching communication strategy that covers all the different project components, such as the NGO projects, MHCs, social marketing, social franchising, and the voucher scheme. Such a strategy is necessary even though the components are just at a pilot stage. If the communication strategy does not cover all the project components from the beginning, it will be difficult to ascertain their success later in terms of correlating communication with behavior change. • Though the BCC campaigns used some formative research, baseline studies, and pre-testing of communication messages, more KAPB studies and continuous behavior change tracking and impact assessment of all BCC activities are required. • For the Suvidha campaign in UP, the GOUP representatives were not aware of the developments in branding or packaging of the product and had no authorization to circulate such a brand among their portfolio of IUDs. This may hamper the campaign if some products are available in a nonbranded pack while the branded product is being promoted simultaneously. Recommendations: National • The NRHM TV and radio spots need to be aired to saturation point, with on￾ground reinforcement through posters, billboards, and illuminated signs near district hospitals, CHCs, RCH camps, panchayat meeting halls, and private clinics. • Because mainstreaming RCH issues is an important part of the RCH II Project Implementation Plan, the TV spots and related IEC materials need to be developed and made available for use as content in Government departments, railways, public sector undertakings, and corporations, with consultations and workshops to explain why it is important to have a comprehensive, multisector roll-out of such messages. • The Government’s plan for the NRHM to provide the ASHAs with refresher training modules that will enable them to hold meetings and events is a good initiative. The training module might be expanded to include record-keeping. If they can collect data on community-level attitudes to FP/RCH/MCH, indications of behavior change, and so on, the ASHAs can be a useful link in the M&E process. 22 IFPS II Evaluation • All BCC products from state-level initiatives should be shared with the MoHFW and NHSRC to facilitate dissemination and learning across all states. Jharkhand • To sustain campaign and message recall for the BCC campaigns on institutional delivery and birth spacing, TV spots need to be supported with a simultaneous roll-out of radio spots, posters, wall paintings, and billboards. • The Sahiyyas (ASHA-level workers) need IEC materials connected with the campaigns to facilitate interaction with the target audience. • There is a need to plan communication outreach and activities with community influencers through meetings and sensitization workshops with family members, political leaders, teachers, village pradhans, etc. • There is need for a communication strategy that uses IEC materials and IPC in activities for orienting and motivating service providers at the PHC, CHC, and district hospitals and in government departments (for bureaucrats and officials). Uttarakhand • To sustain message recall for both the immunization and institutional delivery campaigns, there needs to be a link with the IFPS II PPP pilot initiatives, the voucher scheme and the MHCs. • There needs to be an integrated strategy for achieving such a linkage. For example, posters accompanying the two campaigns can be placed in the private clinics that are accredited to the voucher scheme; thematic billboards, posters, films, and leaflets for takeaway can be part of the MHCs and Immunization Day campsites; and ASHA and ASHA+ workers can be given the IEC materials developed with the campaigns for their IPC interactions. • PHC, CHC, and district hospitals; government offices; and block development offices can screen the TV spots or play the radio spots and display posters. • Good IPC materials that communicate simply and effectively have been created for the ASHAs. They should be included with ASHA training programs in JH and UP and at the national level. They could also be useful in capacity-building programs for women in communities, and self-help groups, mahila mandals, etc. Uttar Pradesh • The Suvidha and female sterilization campaigns are complete, with TV and radio spots supported by other IEC materials to help sustain message recall. They need to be disseminated to saturation point and used as content for wide distribution in the public sector healthcare system (PHC, CHC, and government hospitals), NGO project sites and clinics, RCH camps, the private clinics in the voucher scheme, and the MerryGold, MerrySilver, and MerryTarang facilities in the social franchising scheme. • SIFPSA during IFPS I worked with Indo-Gulf Fertilizers to incorporate FP initiatives into the company’s corporate social responsibility program; there is IFPS II Evaluation 23 merit in starting similar partnerships with other companies both for wider dissemination of BCC efforts and for stimulating PPPs for FP service delivery. • The radio dramas can be better leveraged to both encourage and monitor behavior change through additional response mechanisms in the second round of airing. Currently, they include an invitation to listeners to share their feedback on program episodes through letters. Interaction might be enlarged by o motivating listeners to improve health indicators in their community: have more mothers delivering in hospitals, more babies being immunized, more men involved in FP, etc. Letters that which detail a minimum number of such indicators can provide data for recognizing “model communities” through government prizes or certificates; o creating a third series of radio episodes that feature real stories of the people from the model communities; and o involving ASHAs and ANMs to sustain ground-level interest in the radio dramas and in creating model communities. In this way, the radio dramas can be self-perpetuating through involvement of the target audience (like Soul City). • The radio dramas and their messages can be made into audio CDs and cassettes for van publicity, public meetings, PHC, CHC, private clinics, district hospitals, village melas, panchayat listener groups, and women’s groups for wider dissemination of the messages. They can also be used in the other two states. The audio content can be supplied (along with a 2-in-1 cassette player) to ASHAs and ANMs to help them interact during community meetings and IPC sessions. • The audio content can also be used by other ICT programs, such as eChoupal and GramSat/Community Radio Programs. • The rerun of the Darpan series should include an episode where the ASHAs are shown as helpers of ANMs, becoming a vital link between the community and healthcare providers. • Since the Ao Baatein Karein logo was re-used in the female sterilization campaign, perhaps an umbrella branding exercise can be tested using the same logo in all other communication activities (including the radio dramas, the Suvidha campaign, and EMOC). • The PVC poster and wall charts accompanying the Suvidha campaign (along with other posters displaying a complete range of FP methods) are an innovation that should be replicated in other campaigns and in other states. • The SIFPSA IEC team has introduced a standard guidelines book for wall paintings and hoardings that uses grids, color codes, and precise measurements to ensure that the design value of the message is not eroded in execution; this can be adopted by all IFPS II campaigns. • A well-designed flipbook was created to help with IPC for ASHAs and for use in their training programs. The high pictorial content makes the flipbook an 24 IFPS II Evaluation attractive and effective piece of communication. There is an almost incidental inclusion of FP messages (after RTI, HIV, etc) and no graphic denoting FP on the cover. While it is accepted that IFPS II priorities include NRHM health issues, FP, RH, and MCH should be given equal weight, with clearly demarcated sections, followed by HIV, TB, ORS, AYUSH, etc. Color pages separating the sections would help the demarcation. • For folk media, a budget to equip the performers with a radio that can play the messages, radio dramas, jingles from the NRHM spots would help them entertain the crowds without diminishing the value of their own performance. • For social franchising, branded training material for doctors and healthcare providers is needed, as well as branded IPC materials for the Ayush and ISMP doctors to use with patients. Posters and wall charts with details of the services offered by the MerryGold/Silver and Tarang Clinics are needed. The HLFPPT hospitals can make use of all communication materials created in UP for IFPS II. PROJECT MANAGEMENT FINDINGS Three areas of management are assessed in this section: issues surrounding the ITAP technical assistance contract, issues related to the bilateral performance-based disbursement system, and overall USAID project management. Accomplishments: There are a number of substantial accomplishments to be noted on project management: • SIFPSA: In a relatively short period of time in IFPS I and II, a new institution, SIFPSA, was created that now represents a well-acknowledged resource for the NRHM. The institutions in UK and JH are very new; issues related to those institutions will be dealt with below. • Bilateral PBD: A successful mechanism has been established through SIFPSA for USAID to fund innovative activities with the UP state government in support of the objectives of NRHM. Despite some shortcomings, the PBD mechanism has demonstrated its ability to accelerate management of a diverse set of activities agreed upon by the UP Government and USAID. • Expanded Program Activity: SIFPSA has reprogrammed the savings generated by implementing PBD for additional activities that have helped USAID achieve more than originally envisioned. Essentially, when benchmarks were reached using less money than budgeted, the savings were used to extend the coverage of services or activities in ways that have benefited the whole program. • TA Contract: A highly useful system provides high quality TA, mostly from Indian experts but as needed from off-shore consultants. This consolidated TA under the ITAP contract with Constella/Futures and its partners is a substantial improvement over the IFPS I system of multiple USAID cooperating agencies assisting SIFPSA. ITAP has contributed generally high quality TA and performed an enormous amount of excellent work over a 28-month period (April 2005 to August 2007). Having ITAP in place has also enabled USAID to work on IFPS II Evaluation 25 activities in JH and UK even before the bilateral mechanisms were established and functioning, providing valuable assistance to both states. • Partnerships: USAID has solid partnerships with the stakeholder institutions in IFPS II. USAID’s relationships with national and state health and family welfare officials generally seem to be highly consultative and productive, and its relationships with SIFPSA and ITAP are equally strong and characterized by mutual respect and teamwork. Concerns: • TA Component: Because of the time it takes for USAID to award a new TA technical assistance contract, ITAP was not started until April 2005, eight months after the four-year IFPS II project began. The ITAP contract start-up was rapid because Constella Futures was already operational in India, and the contract has proven to be a flexible instrument that can respond to unanticipated requests from both national and state governments. Nevertheless, there are a number of findings and some recommendations that could improve the strategic use of ITAP in support of the program: • Work Plans: Annual work plans submitted to USAID appear to be simple lists of activities with a schedule rather than meeting the requirements for work plans described in Section A.4.V.of the contract, which calls for a “detailed description of the life-of-project expected results, the benchmarks toward achieving those results, and planned activities geared toward achieving the benchmarks.” Ideally, work plans should also contain some analysis of constraints and factors affecting the work. USAID stated that quarterly progress reports are submitted from headquarters, but these do not contain much analytic information. • Contract Performance Monitoring Plan (PMP): The ITAP PMP, originally developed by QED, one of the ITAP consortium partners, was updated by ITAP staff in May 2007 in consultation with USAID to feed into USAID’s own PMP. Many of the benchmarks and indicators in the PMP are reasonable and are a good reflection of the capacity development and SIFPSA support functions needed, but the PMP does not seem to be used as a real performance monitoring tool. The Performance Data Table in Appendix A has few baseline, target, or actual achievement data filled in for 2006 and earlier. It would be beneficial for ITAP to use the PMP more actively to manage their priorities, and for USAID to more proactively hold ITAP accountable to document progress toward certain targets. • Consortium Management: Often USAID contracts with multiple subcontractors are fraught with institutional conflict and make it harder to function as a smoothly integrated team. Fortunately, there do not appear to be such problems in ITAP, perhaps because only one subcontractor, JHU-CCP, has a long term, in-country presence on the project; Sibley International, Bearing Point, and QED have been providing short-term technical consultants in a timely manner with reportedly high quality TA. The only issue appears to be related to the turnover of BCC staff, which is JHU-CCP’s responsibility, and the delays in filling vacated positions. Since JHU-CCP has an office in India, it is not clear why positions have not been 26 IFPS II Evaluation filled more quickly. Because BCC is such an important part of the IFPS II project, these delays have not been helpful. • Bilateral PBD Mechanism: As noted earlier, the bilateral PBD mechanism has been highly useful as an instrument to work closely with the governments in all three states, particularly in UP, where this mechanism has been in use for about 12 years in IFPS I and II. It has increased ownership by the state government of the USAID- funded work and has made it possible to scale up IFPS innovations into the broader NRHM program of activity. In general, the team believes that the PBD mechanism should be continued through the end of the period needed to complete the pilot work with PPPs (essentially another three years). But the PBD mechanism has changed over time, largely because of USAID’s difficulties in defining it as an acceptable and well-defined mechanism of assistance. In response to questions from USAID auditors in 2001, a legal opinion from USAID/Washington (documented in a memo to PPC dated April 18, 2002) suggested that because the PBD mechanism is more similar to a “Fixed Amount Reimbursement” (FAR) than to program assistance, it cannot be used to provide incentives for future performance but only to reimburse for reasonable estimates for the upfront costs of carrying out activities. This interpretation led to a shift away from true performance-based payments linked to outcomes and to a focus on activities and inputs as the basis for calculating and justifying the value of benchmarks. Nevertheless, even within these constraints, the PBD system has demonstrated that results can be achieved faster and at a lesser financial risk to USAID than with a more traditional cost￾reimbursement system. Emphasis on benchmarks that better reflect the quality of activities might be useful (e.g., skills retention measures six months post training programs). Another problem with the PBD system is that it has always been very staff- and management-intensive for USAID. The layers of negotiation, review, and approval within the State governments and with USAID have resulted in delays and many person-hours of work on both sides and within the Societies at the state level. Some steps have been taken to simplify review and approval systems, but it remains staff-intensive. It is also staff-intensive from the audit perspective. In UP, the team found that there are essentially three levels of audit; those conducted by the Supreme Government Audit Agency of India (reportedly rigorous); external balance sheet audits by a CPA firm contracted by SIFPSA; and audits conducted according to USAID’s Recipient Contracted Audit (RCA) guidelines. Yet no external audit has been conducted of the management costs (indirect costs, currently at 18%) of SIFPSA, which matter for the long-term sustainability of SIFPSA as an institution. The team feels that a periodic audit of SIFPSA’s management costs (and the Societies in UK and JH) needs to be substituted for the RCA audits, which do not provide any added value to USAID. USAID is not at risk because it pays based on accomplishment of pre-costed benchmarks, not expenditures, and there are two other audit levels. In addition, while the USAID RFMO is involved in reviewing the costing of benchmarks, broad standardized IFPS II Evaluation 27 parameters to be used as the basis for valuating benchmarks might be useful to simplify and facilitate the review process. • USAID Management: USAID management of the IFPS II Project is generally very strong, but there are areas that could be improved. Because of staffing shortages due to a temporary hiring freeze, only two professional staff members in the RH/FP Division have oversight responsibility for a complex project. A matrix management approach using staff from other divisions has helped the division cope and has brought in valuable expertise on project content areas. The evaluation team believes that this temporary arrangement, while feasible in the short term, may cause confusion and miscommunication over time because there are too many potential decision makers. A better model might be for the Division Chief to provide strategic oversight and management of all projects in the division and function as the unifying “brain” of IFPS II, with one designated manager for the PBD component and one CTO for the ITAP contract, both supervised directly by the Division Chief. State coordinators from other divisions, or staff who provide technical review of particular elements of IFPS II, can function in an advisory capacity to those who have line management authority. In this way, it is clear to partners who the decision makers are and it avoids the problem of decision making by those who do not have a full picture of the project. There have been such problems. The management of ITAP could be more proactive, using tools like the annual work plans and the PMP to bring more forward planning and accountability. While no serious problems have been noted, there are opportunity costs for being too reactive to ad hoc requests at the expense of work that will lead to the specific results needed by the project as a whole. The evaluation team noted a number of areas where ITAP involvement in improving the quality of field activities would be advantageous. An example would be to continue to help SIFPSA oversee the quality circle expansion phase of the quality improvement pilot work from IFPS I. Issues and challenges were identified in the ITAP publication “Ideas, Insights, and Innovations: Achievements and Lessons Learned from the IFPS Project, 1992–2004,” but problems are still evident (see below). Some scope to respond to new requests and new areas of work should certainly be protected, because flexibility is important, especially given the focus on innovation. ITAP has done a good job of preparatory analysis on developing PPPs (e.g., the international work shop of social franchising) but as time goes on the focus will need to be on monitoring the quality of projects and making necessary revisions, as well as, documenting lessons learned, outcomes, cost implications and advocacy required for the government to adopt and scale-up PPPs. This work must be planned and scheduled systematically, given the importance of ITAP’s role in helping USAID advocate for expansion of these models beyond the period of USAID assistance. 28 IFPS II Evaluation Recommendations: • Improve the annual work plans submitted by ITAP and ensure that the PMP is used to manage for results. • Continue the PBD mechanism in its current form for the duration of the PPP pilots to ensure that they are successful completed, and introduce more quality benchmarks. • Commission periodic audits of the management costs (indirect cost rate) of SIFPSA and the societies in UK and JH to improve institutional sustainability in lieu of continued RCAs. • Unify management of IFPS II in the RH/FP Division but continue to use staff from other divisions as advisors to enhance the technical quality of activities and sharing of strategic approaches across the PHN office. • Manage the ITAP contract more proactively to ensure that sufficient attention is given over time to analytic documentation of PPPs and improving the quality of activities. POLICY, SUSTAINABILITY, AND INSTITUTIONAL FINDINGS National-Level Activities The policy environment in the health sector has changed considerably over the past few years. Following the elections of May 2004, the new coalition government launched the NRHM to carry out “architectural correction of the basic health care delivery system.” The overall goal of the NRHM is to improve the availability of, and access to, quality health care especially for those residing in rural areas, the poor, women, and children. The focus is on reducing mortality and fertility rates through improved RCH care (including immunization) and communicable disease control. Governing principles are to promote of synergism between the various determinants of health, decentralize program management to district level and below, implement quality standards, and enhance flexibility in funding to foster innovations. Recognizing that there would be difficulty in achieving results at all levels with the current systems, a number of institutional and support mechanisms have been established or are being set up, among them a National Mission Steering Committee, a Program Management Support Group (PMSG) and a National Health Systems Resource Centre (NHSRC); State Health Missions, State Program Management Units (SPMU) and State Health Systems Resource Centres (SPSRC); District Health Missions and DPMUs; and, in some cases, block-level PMUs. While the management support units at all levels are envisaged as functioning as a secretariat to government departments and mission directors, the resource centers are intended to facilitate sharing of good practices, promotion of innovations, and provision of consultant expertise for capacity building. All development partners contributing to the health sector have been encouraged to align their programs and projects with the NRHM through pooled funding mechanisms or other complementary approaches. IFPS II Evaluation 29 Accomplishments: USAID has welcomed and supported the movement of the Government toward a more comprehensive and holistic health system and has been a responsive partner for the MoHFW. The Delhi office has provided both financial and technical assistance for activities related to policy development as well as implementation in the areas of BCC, PPP, M&E, convergence, urban health, and quality assurance, either directly through USAID staff or through ITAP. It gave significant support when the NHSRC was being established by providing 23 consultants, and also supported the latest National Family Health Survey (NFHS-3) in collaboration with other development partners. While much of this support has been channeled through the health systems division of USAID, there is a considerable contribution from IFPS-II and the RH division, as well as MCHUH (the health systems division picks up experiences from specific projects to feed into the national level). This has improved the sharing of good practices and innovations and their inclusion into the national policy framework, contributing . to the PPP framework document, the upcoming National Urban Mission, and the Quality Assurance pilot scheme for RCH-2 services. USAID also has good relations and solid partnerships with other important Indian institutes contributing to health sector development, such as including the National Institute of Health & Family Welfare (NIHFW), the International Institute for Population Sciences (IIPS), the Public Health Foundation of India (PHFI), and the Indian Clinical Epidemiology Network (IndiaCLEN). Concerns: While having a clear and appropriate policy is an essential step on the long road to having better health outcomes, implementation clearly needs to be continuously monitored and supported at all levels. USAID is ideally placed to strengthen the entire cycle with a strong in-house team to complement the ITAP staff in Delhi and the three States and SIFPSA in Lucknow. Much of the support provided at national level has been in response to government requests, which the central government much appreciates. However, there is an opportunity for USAID to be more strategic in its support now that the NHSRC has a director. The coming months will be a critical period for USAID to help influence the direction the NHSRC takes and align its support accordingly. The 23 consultants funded by USAID through ITAP need to be integrated into the new NHSRC structure as soon as feasible. They were brought in as an interim measure to fill gaps in the system until the NHSRC would be fully functional with a director. Though highly valued, their contribution has clearly been reactive; USAID now needs to move in a more strategic direction concerning what they can contribute to strengthening health systems in a way that is more focused on USAID areas of strength and expertise. For example, one priority areas the NHSRC has identified for urgent action is improving child health outcomes. USAID can advocate strongly for high priority to be given to FP spacing methods because birth spacing and child survival have mutually reinforcing outcomes. USAID will be better able to do this role if the vacancies in the divisions are filled. Staff members are currently struggling to cope with the workload involved in monitoring projects and translating lessons learned into policy. 30 IFPS II Evaluation Recommendations: • USAID should continue to strongly advocate to keep FP high on the national political and strategic agenda through its influence with the MoHFW, the NHSRC, and the NIHFW, particularly given the extremely broad framework of the NRHM. • USAID should provide evidence to GOI/NHSRC of the benefits of all FP programs, not just those the government has currently approved, and the positive interactions between FP and other areas that are GOI priorities, particularly MCH. • USAID should ask for an early decision on mainstreaming the 23 consultants into the new NHSRC set-up so that their contribution can be maximized and sustained. • USAID should ensure that all IFPS-II experiences are made available to all national stakeholders, particularly the NHSRC, to facilitate greater learning and sharing with other States. • The USAID/India PHN Office should be allowed to recruit high-caliber individuals to fill the vacant positions as soon as possible so that the desired advocacy and supportive roles at national level can be achieved. Capacity Building and Sustainability IFPS-II has two primary capacity building and sustainability objectives: 1. Strengthen public sector capacity to provide public and private RCH services through appropriate policies, monitoring and evaluation, quality assurance; and 2. Strengthen the capacity of SIFPSA so that it may more effectively promote collaborative partnerships with Indian & U.S. institutions to support the delivery of integrated RCH services. The task of the evaluation team was to examine the extent to which IFPS has helped build the capacity of state societies to deliver and manage FP/RH programs and the degree to which it has contributed to sustainable, quality societies that can implement the vision of the NRHM. The context for this varies tremendously in the three states. In UP SIFPSA has been in existence for almost 15 years, although it took some time to be fully functional. Since the IFPS-I evaluation, there has been a complete shift in approach away from implementation toward facilitation and capacity building with development of partnerships in response to the second objective. UK and JH are newer states, both formed in 2000, so they have no legacy or entrenched systems. This has enabled both to start with a clean slate and build more flexible structures and systems. The States have responded differently to the demands and opportunities of the NRHM; UP lags behind the other two states in establishing the necessary structures because of to the complex political environment, state elections, and frequent transfer of senior officials. In the past year there has been more stable and strategic leadership from the Principal Secretary for the health sector, but SIFPSA has had continuing struggles with IFPS II Evaluation 31 frequent changes and lapses in its own leadership. UP has some of the worst indicators in the country, although the contraceptive prevalence rate has improved. In UK, the State made quick progress in the first year of the NRHM but is reported to have slowed down during the second year and use of funds is low. However, there has been progress in outcomes measured in NFHS-3: an increase in institutional deliveries and a decrease in unmet need for both spacing and permanent FP methods. The IMR is also the lowest among the Empowered Action Group (EAG) States. JH has some of the worst RCH indicators in India and apart from immunization rates these have not improved much over the past few years. While some innovations are taking place, the State has been relatively slow in progressing with the institutional arrangements and here, too, use of funds is quite low. The state has identified the lack of qualified human resources as a major challenge, but the state PIP has no clear strategies for addressing this. In assessing IFPS-II, the accomplishments, concerns, and recommendations are dealt with separately for each state. Uttar Pradesh Accomplishments: In UP, SIFPSA is a well established agency with considerable experience in improving access to FP services. The positive trends in indicators show that there has been some impact on health outcomes, although UP still has a long way to go to catch up with other States and achieve the NRHM goals for the State. Among the strengths of SIFPSA perceived by State officials are the systems put in place, its contribution to good practices and interesting innovations, and its flexibility compared to the government system. SIFPSA has shown that partnerships with the private sector, especially NGOs, can succeed, and its contribution to IEC/BCC materials and campaigns, improved training methodologies, and district action plans has been recognized. The ITAP team supports the activities of SIFPSA with expertise that is not available within the society. It has complemented and enhanced the work of SIFPSA considerably and contributed significantly to achievement of benchmarks. They have in particular supplemented areas where SIFPSA was lacking expertise, including development of new models of working with the private sector and negotiating processes, strategy design for BCC campaigns, research and evaluation, and design of new training modules. Concerns: The extent to which both SIFPSA and ITAP are facilitating work in all these areas rather than doing it themselves is not clear. In many areas there is still a lack of clear and consistent guidelines the government could use to implement the initiatives themselves. Because both agencies are monitored more on the outputs and achievements of benchmarks, rather than on the processes used to reach that point, there is limited pressure on them to perform a facilitative role and significant pressure on them to achieve the benchmarks. 32 IFPS II Evaluation The Principal Secretary has put forward a proposal to the UP Cabinet nominating SIFPSA as the state PMU or the State Health Systems Resource Center for the NRHM program. The eventual decision will clearly have an enormous impact on the future of SIFPSA. If it is approved, that would be a very positive acknowledgement of USAID’s investment over the years. Whether or not SIFPSA becomes the SPMU, it will have to change its way of working considerably. The future role of SIFPSA has been under discussion since it was first mooted in 2003 for RCH-2, and the uncertainty about it has resulted in the departure of some of the most qualified experts. A reorganization study was undertaken by ITAP (subcontracted to KPMG) that recommended a number of restructuring processes to enhance the profile and efficiency of SIFPSA. Implementing the suggestions, which has not yet been done, will be critical for SIFPSA to fit into the new state environment. If SIFPSA is appointed as the SPMU, the SHSRC, or both, its expertise will have to be expanded to cover all the areas covered by the NRHM. If it is not appointed as either body, its future is less certain, but it may have a role as a TA agency specializing in FP/RH. Either way, SIFPSA clearly still needs to become more proficient. At the district level, the DIFPSAs will not be allowed to continue working separately, according to the Principal Secretary, but must be integrated into the DPMU on merit and according to district requirements. SIFPSA needs to take this into consideration in future planning. ITAP is not involved in all of SIFPSA activities; its role seems to be more reactive than strategic in terms of influencing the quality of SIFPSA’s work. Although success is being claimed due to the fact that experiences from IFPS-1 have been taken over by the government, for example, in RCH camps, CBD projects, quality improvement, and DAPs, observations during evaluation team field visits did not inspire confidence in the quality of some of these activities. For example, RCH camps, which initially were undertaken in remote areas, are now being held in fixed facilities. While this was a State decision based on concerns about the quality of procedures in camps, it is unlikely to realize the intention of reaching the most remote areas with RCH services. The full list of activities provided to the team to show what has been taken over by the GOUP is a simple list that gives no indication of the quality of the activities. Recommendations: • ITAP and SIFPSA need to be monitored on how they are performing their facilitative role as well as on achievement of results. Process indicators should be incorporated into the PMP for ITAP and into the benchmarks for SIFPSA. • USAID should work to convince the UP government to jointly insist on implementation in principle of the recommendations of the SIFPSA reorganization study. • If SIFPSA is nominated as SPMU or SHRC, details of the reorganization of SIFPSA should be agreed on with all stakeholders to ensure that SIFPSA is able to perform its new role adequately. IFPS II Evaluation 33 • The reorientation of SIFPSA should also take into consideration areas where support from ITAP is still required, with a clear plan for enhancing SIFPSA’s ability to gradually take over these functions or installing mechanisms for SIFPSA to contract for that expertise. • In any case, the innovations supported by SIFPSA and ITAP, including the voucher schemes and the social franchising model, need to be continued and carefully documented so that they can be scaled up as desired. • USAID, ITAP, and SIFPSA should negotiate with the GOUP to allow monitoring of activities taken over by the government to maintain or where necessary enhance their quality. Uttarakhand Accomplishments: The contribution of USAID is greatly appreciated and seems to be well-integrated into state plans. State officials have actively sought TA, and ITAP has responded appropriately. Some fascinating innovations are being undertaken and the State is active in seeking inputs from USAID. The ASHA+ and voucher schemes seem to be particularly noteworthy and will provide valuable guidance for States with similar conditions. Concerns: As already mentioned, some weaknesses were noted in specific areas, particularly regarding the MHCs and BCC activities. These must be addressed to ensure continuing credibility with the state government; the TA requests offer excellent opportunities for USAID to increase its influence. Notably, however, within these activities FP is given relatively little priority. This is an area where USAID can clearly provide more strategic guidance to the State so that the gains in improved indicators are not lost. It will take time for some of the innovations to take root and show success, particularly the voucher scheme and ASHA+ training and follow-up. This means that support from USAID/ITAP should not be withdrawn too early. Recommendations: • USAID should ensure support to UK for long enough to allow for a reasonable chance of adequate learning in implementing innovations because this State is more open to them, and other states will be able to adapt the models. • USAID should respond quickly and comprehensively to the new requests for TA in health insurance and contracting out, bringing in other partners if it cannot meet all the demands, some of which will likely extend beyond the scope of IFPS. • FP should be a central focus in all activities supported. • USAID/ITAP should draft a clear plan to address the weaknesses observed and integrate the modifications required into the benchmarks and the PMP for ITAP. 34 IFPS II Evaluation Jharkhand Accomplishments: The situation has been difficult in JH because a complete change in the senior officials dealing directly with USAID has required multiple reorientations from USAID staff. A protracted effort has begun to ensure that a society is in place that meets all USAID financial and management requirements. Nevertheless, there is considerable appreciation of the support provided by IFPS. The present and planned activities seem to be responsive to state plans, demands, and needs. The ITAP team is respected and their location in the compound of the Jharkhand Health Society (JHS) is facilitating communication and acceptance. The compound also houses the new Public Health Institute; it is proposed that it become the State Health Systems Resource Centre. Concerns: While the JHS was originally intended to be the integrated society for all funds coming to the State, and a Memorandum of Understanding had been drafted to that effect by the GOI, GOJ, USAID, and ECTA (the European Commission Technical Assistance which had been active in the State), the latest Governing Board meeting approved use of the JHS only for USAID-funded activities. Other funds from the GOI for NRHM will be routed through the Jharkhand Health & Family Welfare Society. Because this does not respond to the GOI recommendations for merging all state and district societies under the NRHM umbrella, USAID will need to decide whether or not its funds should continue to be channeled separately. The new approach makes it less likely that activities will truly complement other NRHM initiatives and will be sustainable. The Public Health Institute is not yet functional because ITAP has only just begun recruiting staff. The institute has been designed to act as the main training institute for the State (there is no SIHFW), a research centre, and the SHSRC. Staff positions will be funded by ITAP for the first year, and it is envisaged that the State government will take them over thereafter. However, because there was no clear business plan for the institute, it is not clear how this will happen. If the institute is to be autonomous, more than 50 percent of its funds should come from nongovernmental sources. If it is designated as the SHSRC, and if USAID support is short-term, it is very likely that most of the funding will come from government, which will limit its independence. It will take time for the new staff to build capacity in delivering services in training and research, thus generating income. Most other activities are just starting because approval of benchmarks was delayed by the frequent staff transfers and the resulting need for repeated orientation of new officials on program objectives and the complex benchmarking process. The State took considerable time to set up the society and establish the procurement and HR procedures, and will now need time to undertake the benchmarked activities; that means it will require extended assistance from ITAP because the State as little expertise available. As in UK, FP does not seem to be receiving priority attention. If the many vacancies and the acute lack of expertise in State government system are not addressed, interim support from USAID will be less effective and sustainable. While IFPS II Evaluation 35 there is a State PMU, also located in the JHS, its staff seem to be underutilized. There are also more demands than USAID will be able to meet within the mandate of IFPS-II. While the Vistaar project, also funded by USAID, is also active in JH, there seems to be limited collaboration or convergence of support. The offices are separate, limiting opportunities for communication, although synergy would have considerable potential. Recommendations: • USAID needs to make and communicate to the State government a clear decision on whether to continue routing funds through a separate society. • USAID/ITAP should encourage the State government to draft a comprehensive business plan for the Public Health Institute so that it can move toward becoming fully autonomous. • All activities supported by IFPS should have a clear link to FP/RH so that funds are not widely used for nonrelated areas. • All activities supported should be integrated into the State PIP for NRHM and be supportive of other strategies, for example, regarding training and subsequent placement of people. • USAID should enhance the opportunities for synergy between its various projects by sharing work plans, building communication channels, and providing strategic guidance on collaboration. PROJECT OUTCOMES AND MEASUREMENT ISSUES Selection and monitoring of project results: Discussion of the objectives and targets of IFPS I is essential for understanding how IFPS II objectives and targets were set. IFPS I was originally designed as a 10-year (later extended to 12-year) $325 million family planning service project to increase contraceptive prevalence and reducing total fertility rates (TFR) in UP. In 1992/1993, in UP as a whole modern-method contraceptive prevalence was 17.8 percent, and in 1998/9 it was 20.8 percent as measured by NFHS II. The project focused on increasing access to, improving the quality of, and expanding demand for FP/RH in a significant portion of UP. The strategy was to craft innovative approaches to the improvement of both public and private sector FP services and then to support their broad-scale implementation in about half the districts in the State (the “SIFPSA districts”). The project supported a range of data collection activities to measure progress toward project objectives, including the PERFORM surveys, which compared CPR and TFR, as well as other indicators, in SIFPSA and non-SIFPSA districts. Data for UP as a whole were also available from the RHIS and the National Family Health Survey. By 2003 (one year before the end of the expanded project), contraceptive prevalence in SIFPSA districts was 27.8 percent and in non-SIFPSA districts 23.9 percent. By 2005/6 contraceptive prevalence in UP as a whole had reached approximately 30 percent. There was a major shift from its predecessor in the approach, strategy, and breadth of issues addressed by IFPS II (2004–2008) (see NAD, IFPS II, 2004). The executive 36 IFPS II Evaluation summary of the founding document states that “during the extension period (04–08), project activities will be reoriented toward development, demonstration, documentation, and leveraging expansion of working models of public-private partnerships, for provision of integrated reproductive and child health services in UP, Uttarakhand, and Jharkhand.” Major differences include the shift from development of new approaches and direct implementation to development of innovations and promotion and technical assistance for scaling up—a shift from an exclusive FP focus to a broad RCH approach—wider geographic focus, a much shorter timeframe, and a much smaller budget, though that was complemented by the large increase in GOI funds for NRHM/RCH II. This refocused approach applies to both the SIFPSA and the new consolidated TA components of the project. IFPS II objectives for achievement by 2008 were therefore set as follows: • CPR at 30.4 percent • TFR at 4.0 percent • Condom sales (rural UP): over 420 million • Oral pill sales: over 14 million • IFA coverage: 43.2 percent The NFHS data for 2005 show that statewide in UP modern-method CPR had reached about 30 percent and TFR had dropped to 3.8—so the 2008 targets for the project had already been achieved. This may suggest that the project endpoint indicators need to be revised, but in doing so a critical consideration is the appropriate level of aggregation of the indicators. In both IFPS I and II, the evaluation team believes, selecting indicators of CPR and TFR at the State level was inappropriate because most of the interventions were concentrated in specific districts. Although some activities, such as social marketing, are state-wide, the only indicators appropriate for such activities are sales figures. It would take a very complicated study to be able to analyze the impact of such activities on state CPR and TFR levels. What was done during IFPS I, appropriately, was to gather more detailed data from SIFPSA districts to compare them with controls, because the inputs into SIFPSA districts were extensive enough to warrant such comparisons even for CPR and TFR. In IFPS II, however, the project moved to a TA mode in three states and the only interventions directly implemented by the project are NGO activities in UP and social marketing and other pilots of PPPs, which have a very broad technical RCH mandate. Therefore, it is reasonable and advisable for the project to attempt to collect baseline and end line data only in UP districts where SIFPSA is supporting NGO projects, because their geographic coverage of these districts is quite extensive, and possibly in the other two states for similar activities, if the geographic coverage is sufficient. Furthermore, a TA-focused project might reasonable be directed to analyzing the process and output indicators selected by the project (see the IFPS II project paper), for example, numbers of condoms and pills sold for social marketing, three state proposals for RCH II developed and funded with local resources, UK public health directorate operational and funded by GOUK, and at least one working model of PPP funded by other agencies for scale up. IFPS II Evaluation 37 The likelihood of the project achieving the selected process and output level results is good if the end date is extended to allow for completion of the PPP pilots and dissemination of results. The strong and focused TA activities of the project and the good relationships it has with both public and private sectors in all three states is a solid foundation for its future success. For example, the DAP and PIP development process, funded by GOI, has been extensive and successful. Three new PPP models—social franchising, voucher schemes, and MHCs—have been initiated, which is a critical first step to providing evidence for scaling up. Condom and pill sales are on track to achieving the annual targets. With good follow-up of the ITAP assessment recommendations and enough time for further implementation, the NGO projects that have started have the potential to contribute significantly to the CPR and TFR in their districts. As for USAID, the new draft FP/RH strategy (4/5/07) highlights changes in state CPR and TFR in UP and JH (with an emphasis on spacing methods) to be expected over the next five years. Again, the evaluation team would recommend that USAID accept responsibility for results such as policy changes, institutional capacity results, and utilization of innovative approaches by X number of organizations and states. Here, CPR and TFR are inappropriate indicators at the state level, though they are useful for project areas like the SIFPSA districts if a geographical focus for actual service delivery is continued. Because some of the PPP models in the works have tremendous potential for increasing CPR and decreasing TFR and at a future stage USAID could help state governments in broad application and measurement of impact without their holding themselves responsible for actual implementation. Measurement issues: Some of the indicators specified in the 2007 FP/RH draft strategy cannot be measured through routine MIS systems or RHIS or community surveys. Important indicators like rates of discontinuation of methods and delay of first pregnancy require special studies. Similarly, once adolescent projects are launched in the three states or nationally, selection and measurement of indicators at the policy and process levels may be more feasible, depending on the activities implemented. 38 IFPS II Evaluation Part Two: Recommendations for the Future GENERAL STRATEGIC APPROACH A fundamental decision for USAID’s work in RH/FP is how it can invest its limited resources so as to achieve the maximum impact given the context in which it operates in India. Because considerable resources are now available at the State level to implement RCH programs, USAID does not need to directly fund implementation. It is clear from the evaluation team interviews that both state and national officials welcome USAID as an NRHM partner because of both the flexible TA it provides and its ability to take risks on innovations. This is in many ways USAID’s comparative advantage. The TA can go far beyond just hiring of domestic or off-shore consultants to apply to building the capacity of state institutions in selected technical areas, continued experiments with PPPs, and contributions to national policy development. Another major decision is how much emphasis to put on public, private, and PPP activities in RH/FP programs. The team strongly believes that USAID has created an important niche for itself in work with PPPs and experiments on how state governments can best tap the private sector to accomplish their RH/FP objectives. At the same time, USAID still has much to offer in certain areas of public sector RH/FP programs (see below). In future, USAID’s work with PPPs may be more acceptable to state governments if it is packaged with assistance on improving the quality of purely public programs. This may be particularly true if USAID should choose not to include a more direct form of assistance to government, such as a PDB mode of collaboration. The team believes that besides working with PPPs, USAID should select a few areas within public health programs where it would target work; for instance. • At the national level, for MOHFW and ministries like the Ministry of Social Welfare, TA on policy change affecting implementation of the NRHM/RCH II program should continue. Major themes for continued USAID TA might include national policies addressing the role of the ASHAs, remuneration for ASHAs for various services, policies on provision of IUDs at various levels of the health system, the role of the private versus the public sector in providing FP methods, and addition of injectables and other methods (e.g., LAM and SDM). TA for building the capacity of specific national institutions should concentrate on just a few, such as the NHSRC and NIFHW. USAID could, for instance, help the GOI to clarify the mandate of each institution, draft clear scopes of work (SOWs) for each; and work out guidelines for how the two could collaborate and complement each other. This will also help clarify how their state affiliates would work together. Any USAID support for actual staffing of these institutions should be very short–term, and the GOI should be encouraged to pay for staff as soon as possible. • Staff from an ITAP-like unilateral TA mechanism should be used to place a minimum number of TA staff within the MOHFW, NHSRC, and NIFHW to help their build capacity over the next few years and should also help coordinate TA across all the entities. The TA staff should never be considered staff of the IFPS II Evaluation 39 organizations, and their positions should not be picked up in the long term by the host organizations. They are simply there to provide shorter-term TA (up to five years or as needed) to organization staff. The GOI should fully staff up the organizations early on so that the TA staff can help build the capacity of the entire team. Types of TA that may be useful are curriculum development, strategic planning, policy guidelines, and formulation of M&E systems, depending based on the mandate of each organization and its role in the NRHM/RCH II program. • In UP, JH, and UK, TA might be similar. Once the SPMUs and SHRCs are identified, the ITAP-like mechanism could be used to place TA staff at these institutions and within the State Directorates of Health (if they are themselves not the SPMU or SHRC) to provide and coordinate TA. Again, as at the national level, the TA could be on topics like for strategic planning, policy development, systems development, identifying best practices, or piloting new activities. • Continued support for PPP innovations at the state level is essential. The pilots started under IFPS II need to be completed, and support for NGO projects should continue until the services have been established long enough (at least another year) to demonstrate the impact of the model in the 38 SIFPSA districts . Further, lessons learned from the pilots in social franchising, voucher schemes, MHCs, NGO projects, etc. should be disseminated and TA should be provided in the longer term to these states and at the national level for widespread scaling up. PRIORITY TECHNICAL AREAS AND GEOGRAPHIC COVERAGE Family Planning: FP/RH should continue to be the major focus of any extension of a project or initiation of a new project. Although much progress has been made, the FP situation in UP and JH continues to be very poor and TA is sorely needed. In UK, although the aggregate CPR at the state level is relatively high, there are wide variations across the state. Moreover, in all three states there continues to be a need for improving postpartum FP and the method mix to add more spacing methods, such as IUDs, SDM, LAM, and injectables, as well as continued work on improving access to and the quality of permanent methods. Adolescent RH: The commitment of the USAID Mission and state government to adolescent FP/RH is very strategic and should be a priority for future efforts. Moreover, male involvement through use of male volunteers and other strategies suggested by state and national stakeholders is highly recommended. Increasing work with adolescents also offers opportunities to address HIV/AIDS prevention and dual protection, as well as ensuring that FP is an important part of HIV/AIDS counseling for youth as well as adults. Working on adolescent RH offers opportunity for cross-divisional collaboration within USAID/PHN to help districts in the three northern states that NACO has are identified as high prevalence (A and B), and address HIV/AIDS and FP/RH in ways that reinforce USAID programs in both areas. Integrated Programs: The expansion of IFPS II to cover ANC, institutional deliveries, immunizations, IFA, and TT provision fits in well with the national NRHM/RCH II program and deserves continued support. The evaluation team recommends that the 40 IFPS II Evaluation USAID Mission try to expand the range of funds used for this program and to leverage other project, donor, and private funding for these activities. Quality Improvement: Providing TA for better quality improvement (QI) programs is a good role for USAID given the long and successful history the agency has with these systems all over the world. It should be funded by RCH II with technical inputs, including monitoring and capacity development, from USAID. QI is also a critically important area of systems development for private facilities and providers as government strengthens its role in accreditation and oversight of private providers. BCC: Behavior change programs, including those related to seeking services at private facilities through PPPs, is an area that continues to need TA and innovation. There is still much work to be done in promoting behaviors that are independent of the health system itself, such as delaying age of marriage, spacing births, breastfeeding, etc. Also, various IPC and nontraditional BCC approaches need to be evaluated for cost-effectiveness and scalability. Again, USAID’s contribution should be on the creative side, with state-wide implementation funded by the government. UP, because of its large population and relatively poor PHN indicators compared to other states, should remain the major target of USAID RH/FP programs. USAID already has an excellent track record there and can be influential without necessarily investing huge resources. Both public and private programs need improvements to make a broad range of FP spacing methods more available and to ensure that the quality (skills development and training, client counseling, M&E, etc.) of these FP programs rises. UK, which has relatively better indicators than UP, is undoubtedly less in need of assistance from USAID, but the team recommends that USAID continue its work there, primarily because UK appears to be fertile ground for testing innovations and creating service delivery models that may be of use in other states. For example, if the use of mobile teams to reach geographically isolated areas in UK proves to be cost-effective, the model could be replicated by other state governments. Health and Family Welfare officials in Uttarakhand seem to be deeply committed to expanding PPPs, including a possible “universal health insurance” system that if successful, could be highly useful to other states. With continued involvement in UK, USAID could help analyze and document lessons learned and encourage the cross-state learning that should be occurring in the NRHM. It can also continue to help develop the capacity of nascent technical resource agencies like the State Health Resource Centers, which have not yet taken off. JH is a state where RCH indicators are still poor and institutions relatively weak because it is a new state and its government has had difficulty recruiting and retaining qualified staff. USAID can continue to play an important role, both by helping improve capacity in selected RH/FP technical program areas and helping take advantage of private providers and facilities through PPPs. As in UK, strengthening the capacity of the State Health Resource Center is a worthwhile effort because it will help sustain improved technical capacity in the state. IFPS II Evaluation 41 RELATIONSHIP WITH MAJOR PARTNERS USAID already has excellent relations with other development partners in the health sector and can strengthen these further by continuing to systematically engage in collaborative networks. The NHSRC offers a good opportunity to enhance the ways in partners can benefit from each other’s comparative advantage and produce synergistic effects. There is more than enough room for a variety of approaches and modalities of working in India, and USAID can benefit from the physical presence of partners in other states to expand their experiences without itself having to engage in those places. MANAGEMENT RECOMMENDATIONS With respect to mechanisms for development cooperation in the future, the evaluation team believes that USAID, after the extended IFPS II project ends, should use the more traditional TA mechanisms now employed by other PHN divisions. The PBD model that served a very important purpose in the past is not longer so important given the resources of the NRHM. Within the agreement with the GOI, there is a demand for the kind of flexible technical inputs and state institutional capacity development that USAID does so well. If future USAID projects are planned in a highly consultative and inclusive manner with the GOI and its development partners, and if the assistance flows from the strategic framework of the NRHM, the evaluation team believes that a PBD mechanism does not necessarily add value at this stage. 42 IFPS II Evaluation For more information, please visit http://www.ghtechproject.com/resources/ Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com