EVALUATION: USAID/India Innovations in Family Planning Services Project Final Evaluation Report May 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Michele Andina, Dipanjan Roy, Rahul Bhawsar, and Soumitro Ghosh of Social Impact and Jenny Ruducha of Management Systems International. USAID/INDIA INNOVATIONS IN FAMILY PLANNING SERVICES FINAL EVALUATION REPORT May 2013 AID-386-TO-10-00003 RAN-I-00-09-00019-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government. Photo: © Ray Witlin / World Bank TABLE OF CONTENTS ACRONYMS........................................................................................................................................................................ i ACKNOWLEDGEMENTS...............................................................................................................................................iv EXECUTIVE SUMMARY..................................................................................................................................................1 EVALUATION PURPOSE & EVALUATION QUESTIONS......................................................................................9 Evaluation Purpose ..................................................................................................................................9 Evaluation Questions...............................................................................................................................9 PROJECT BACKGROUND........................................................................................................................................... 10 EVALUATION METHODS AND LIMITATIONS.................................................................................................... 22 FINDINGS AND CONCLUSIONS ............................................................................................................................. 30 Impact on Reproductive Health Key Indicators (SOW #1) ............................................. 30 Strategies, Operational Contributions and Lessons Learned (SOW #2 & #3) ........ 51 IFPS Public Private Partnership (PPP) Performance (SOW #4)...................................... 57 ITAP TA Effectiveness (SOW Question #5) ........................................................................... 68 Legacy of IFPS (SOW Question #6) ......................................................................................... 73 Final Comments/Conclusions ........................................................................................................... 76 RECOMMENDATIONS................................................................................................................................................ 78 Annex I: Evaluation Statement of Work............................................................................................................... 79 Annex II: Evaluation Methods and Limitations.................................................................................................. 89 Annex III: Data Collection Instruments................................................................................................................. 96 Annex IV: Persons Contacted.................................................................................................................................103 Annex V: Documents reviewed.............................................................................................................................109 Annex VI: Assessment of PPP Interventions under IFPS..............................................................................115 Annex VII: IFPS III: Districts Visited and Coding..............................................................................................117 Annex VII: BCC Interventions under IFPS II & IFPS III...................................................................................119 Annex IX: Supporting Charts..................................................................................................................................121 Annex X: IFPS Document Review Checklist: Average Scores.....................................................................122 Annex XI: Team Member CV’s ...............................................................................................................................122 Annex XII: Summary of Indicator findings 1995-2010 .................................................................................155 BOXES, FIGURES, AND TABLES Box 1: SIFPSA Mission Statement Box 2: JSY Scheme Box 3: TA Consortium Partners Box 4: PPP Framework for Assessment Box 5: MGHN Goals Box 6: Sambhav Voucher Goal Box 7: Mobile Health Van Goal Figure 1: IFPS Benchmark Achievements by States Figure 2: IFPS Total Expenditure 1992-2012 Figure 3: CPR Trends in India, Bihar, and Uttar Pradesh 1992-2011 Figure 4: IFPS Interventions in Uttar Pradesh Figure 5: IFPS Interventions in Uttar Pradesh – Training/Capacity Building Figure 6: IFPS Interventions in Uttarakhand Figure 7: IFPS Interventions in Jharkhand Figure 8: Site Selection and Visits for IFPS Evaluation Figure 9: Trends in TFR in Select States of India Figure 10: Contraceptive Prevalence Rate in Large EAG States of India Figure 11: Contraceptive Prevalence Rate in Small EAG States of India Figure 12: Contraceptive Prevalence Rate in UP High and Low Intensity Districts and Bihar Figure 13: Trends in Usage of Modern Methods among Currently Married Women in Large EAG States Figure 14: Trends in Usage of Modern Methods among Currently Married Women in Small EAG States Figure 15: Trends in Modern Methods Prevalence Rate in UP High and Low Intervention Districts and Bihar Figure 16: Trends in Condom Usage by Partners of CMW in Large EAG States of India Figure 17: Trends in Contraceptive Prevalence Rates in Uttar Pradesh High and Low Intervention Districts and Bihar Figure 18: Trends in Usage of OCP in Large EAG States of India Figure 19: Trends in Usage of OCP in Small EAG States of India Figure 20: Oral Contraceptive Prevalence Rates in UP High and Low Intervention Districts and Bihar Figure 21: Trends in Usage of IUDs in Large EAG States of India Figure 22: Trends in Usage of IUDs in Small EAG States of India Figure 23: Trends in IUD Prevalence in Uttar Pradesh High and Low Intervention Districts and Bihar Figure 24: Trends in Female Sterilization Pattern in Large EAG States of India from NFHS, DLHS and AHS Figure 25: Trends in Female Sterilization Pattern in Small EAG States of India Figure 26: Trends in Female Sterilization Rates in Uttar Pradesh High and Low Intervention Districts and Bihar Figure 27: Trends in Unmet Need for Family Planning in Large EAG States in India Figure 28: Trends in Unmet Need for Family Planning in Small EAG States in India Figure 29: Trends in TT Coverage of Pregnant Women in Large EAG States in India Figure 30: Trends in TT Coverage of Women in Small EAG States in India Figure 31: Basket of contraceptives Figure 32: Mobile Health Van Figure 33: ITAP Spending Figure 34: Development Partners in Jharkhand Figure 35: ITAP Spending by States Figure 36: Degree Centrality - National and States Table 1: Overview of Phases 1-3 Table 2: IFPS Interventions at a glance Table 3: National, District and IFPS Data Sets Table 4: Trends in CPR Between IFPS I-III Table 5: IFPS Strategies, Operational Contributions and Lessons Table 6: CSMP Sales in UP Table 7: Sambhav Reimbursements Table 8: Proposed ITAP Activities and ITAP Activities Implemented Table 9: ITAP Documentation Review - Ranking of Documents Reviewed i ACRONYMS AHS Annual Health Survey ANC Antenatal Care ANM Auxiliary Nurse Midwife ARSH Adolescent Reproductive and Sexual Health ASHA Accredited Social Health Activist AWW Anganwadi worker AYUSH Ayurveda, Yoga & Naturopathy, Unani, Siddha and Homoeopathy BCC Behavior Change Communication BMGF Bill and Melinda Gates Foundation BPL Below Poverty Line BS Birth Spacing CBD Community Based Distribution CEDPA The Centre for Development and Population Activities CHETNA Centre for Health Education, Training and Nutrition Awareness CHC Community Health Center CHV Community Based Volunteer CII Confederation of Indian Industries CMO CMW Chief Medical Officer Currently Married Women CPR Contraceptive Prevalence Rate CSMP Contraceptive Social Marketing Program CSR Corporate Social Responsibility CTO Cognizant Technical Officer (USAID Project Officer) DAP District Action Plans DCTC Divisional Clinical Training Centre DEC Development Experience Clearinghouse DLHS District Level Household and Facility Survey DPMU District Program Management Unit EAG Empowered Action Group ECP Emergency Contraceptive Pill FHI Family Health International FP Family Planning FY Fiscal Year GOI Government of India GOJH Government of Jharkhand GOUK Government of Uttarakhand GOUP Government of Uttar Pradesh HLFPPT Hindustan Latex Family Planning Promotion Trust HLL Hindustan Latex Limited IEC Information, Education and Communication IFA Iron Folic Acid IFPS Innovations in Family Planning Services IMR Infant Mortality Rate IPC Interpersonal Communication ii IPHS Indian Public Health Standards ITAP IFPS Technical Assistance Project IUD Intra-uterine Device JHUCC John Hopkins University Center for Communications JHPIEGO The Johns Hopkins Program for International Education in Gynecology and Obstetrics JHS Jharkhand Health Society JH Jharkhand JSK Jansankhya Sthirta Kosh JSY KGMC Janani Suraksha Yojana King George Medical College LAM Lactational Amenorrhea Method LMO Lady Medical Officer LOE Level of Effort KGMC King George Medical College, Lucknow MCH Maternal and Child Health MDG Millennium Development Goals MGHN Merrygold Health Network MHC Mobile Health Clinic MHV Mobile Health Van MIS Management Information System MoHFW Ministry of Health and Family Welfare MMR Maternal Mortality Rate MMU Mobile Medical Units NFHS National Family Health Survey NGO Non-Government Organization NHSRC National Health Systems Resource Center NIHFW National Institute of Health and Family Welfare NRHM National Rural Health Mission NSSO National Sample Survey Organisation NSV Non Scalpel Vasectomy OB/GYN Obstetrician/Gynecologist OCP Oral Contraceptive Pills ONA Organizational Network Analysis PATH Program for Appropriate Technology in Health PBD Performance-Based Disbursement PC Project Coordinator PEPFAR President’s Emergency Plan for AIDS Relief PERFORM Program Evaluation Review for Organizational Resource Management (Survey) PFI Population Foundation of India PMU Project Management Unit PNC Post Natal Care PPIUCD Postpartum Intrauterine Contraceptive Device PPIUD Postpartum Intrauterine Device Insertion PPP Public-Private Partnership PSI Population Services International QA Quality Assurance iii RCH Reproductive Child Health RH Reproductive Health RHIS Reproductive Health Indicator Survey RTI Reproductive Tract Infections SARC State ASHA Resource Centre SBMR Standard Based Management Recognition SDM Standard Days Method SES Socio-Economic Status SF Social Franchising SI Social Impact SIFPSA State Innovations in Family Planning Services Project Agency SIHFW State Institute of Health and Family Welfare SHSRC State Health System Resource Center SHS State Health Society SMOs Social Marketing Organizations SO2 Strategic Objective #2 SOW Scope of Work SPMU State Program Management Unit TA Technical Assistance TFR Total Fertility Rate TT Tetanus Toxoid UKHFWS Uttarakhand Health and Family Welfare Society UDAAN Understanding, Delivering and Addressing Adolescent Needs UHI Urban Health Initiative UHI/FHI Urban Health Initiative/Family Health International UK Uttarakhand UNFPA United Nations Population Fund UP Uttar Pradesh USAID United States Agency for International Development USHA Urban Health Social Activists VISTAAR USAID MCH Project VMA/U Voucher Management Agency/Unit WB World Bank iv ACKNOWLEDGEMENTS The IFPS evaluation team would like to acknowledge the USAID/India team for their assistance and guidance during this evaluation. A special thanks to, Ms. Sheena Chhabra, Team Leader, Health System Strengthening Division; Mr. Vijay Paulraj, Project Management Specialist (Family Planning and Reproductive Health); Ms. Ekta Saroha, Project Management Specialist (Strategic Information and Policy); Mr. Gulshan Bhatla, Program Development Specialist; Ms. Charushila Lal, Program Development Specialist, Monitoring and Evaluation. Former USAID/India employees, some dating back 20+ years, including Mr. John Dumm, Mr. Robert Clay, and Ms. Kerry Pelzman, to mention just a few, were generous with their time and we thank them for the important historical perspective and context they provided. The team also had the honor and privilege to interact with early IFPS Directors including Ms. Aradhana Johri and Mr. JS Deepak whose strong and efficient leadership was still evident today in their presentations and discussions with the IFPS evaluation team. In UP, the SIFPSA team, including field staff and NGO partners demonstrated their commitment to the IFPS project as well as to curbing fertility in the State. A special thanks to Mr. BK Jain, SIFPSA General Manager for guiding the evaluation team through 20 years of IFPS work. To Mr. Amit Kumar Ghosh, Executive Director SIFPSA and NRHM UP Mission Director, the team thanks you for your leadership and desire to improve the health and well-being of UP. In the public and private health facilities visited, we would also like to acknowledge the inspiring work of doctors, nurses, counselors, lab technicians, pharmacists, ANM, ASHAs and sweepers who work day in and day out, often in very challenging environments, to care for the women, men and youth who seek family planning services. In Uttarakhand, Uttarakhand Health and Family Welfare Society (UKHFWS) officials in Dehradun included: Dr. Abhay Kumar, State Project Manager; Dr. R.K. Pant, Additional Director; Dr. Saroj Naithani, Joint Director Reproductive Health, Mr. Piyush Singh, FD and MD; Mr. Raj Kamal, Project Support Officer. In Jharkhand: Mr. Abubaker Siddique, Mission Director NRHM, and Dr. Praveen Chandra, Director of Health Services. All of these individuals have dedicated considerable energy, resources and talent to lowering fertility and maternal mortality in their respective states. The team would also like to acknowledge and thank those representatives of the various cooperating agencies and organizations involved with IFPS over the past 20 years, with whom the team interacted at the national and state level. In the past 20 years Family Planning has been absorbed into a broader women's reproductive public health approach. This team would like to thank and acknowledge both the governments of India and the United States for this significant 20 year commitment to lowering fertility rates in one of the most populous countries in the world. The team believes that it is through this type of leadership, financial commitment, collaboration and long-term approach to family planning that demographic transitions are occurring. v A final word of thanks goes to Dr. Robertino Mera, Mr. Rahul Dutta, and Ms. Carolyn Mann for their tireless efforts with data analysis and to the Social Impact Team, especially Mr. James Fremming, Ms. Paige Mason, Ms. Erin Crossett and Dr. Ash Pachauri for their invaluable support and assistance. 1 EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of this evaluation was to conduct both an impact evaluation and a legacy evaluation of Innovations in Family Planning Services (IFPS) I–III, a $325 million bilateral agreement operating from 1992–2012. PROJECT BACKGROUND The IFPS project, initiated in 1992, represents 20 years of United States Agency for International Development (USAID)-supported family planning innovations and services in the state of Uttar Pradesh (UP), and nearly eight years of interventions in the states of Uttarakhand (UK) and Jharkhand (JH). “Designed to serve as a catalyst for the GOI in reorienting and revitalizing the country's family planning services”—a program geared toward sterilization with limited contraceptive choice—$325 million was initially committed for 10 years.1 In 1992, Uttar Pradesh, India's most populous state, with the highest total fertility rate (TFR, 5.2)2 and the lowest contraceptive prevalence rate (CPR, 19.8%) was selected as the site both of greatest need and where a significant impact could be achieved.3 IFPS II–III (2005–2012) expanded to include interventions in Uttarakhand State (13 former UP districts) and the newly created state of Jharkhand (former Bihar districts). IFPS I interventions focused on five primary areas: 1) clinical training/institutional capacity building 2) Behavior Change Communication (BCC) 3) Community Based Distribution (CBD) of Family Planning/Reproductive Health (FP/RH) services 4) Public Private Partnerships (PPP) and 5) Evidence Generation. IFPS II & III focused primarily on PPP, Evidence Generation and Technical Assistance. Throughout the life of IFPS the overarching goals were to significantly reduce total fertility rates, increase contraceptive prevalence rates, and to improve women's health. A unique element of the IFPS project design was the creation of an autonomous parastatal organization,4 the State Innovations in Family Planning Services Agency (SIFPSA), as a mechanism “to provide flexibility and avoid bureaucratic delays endemic to government systems.”5 With a governing body that includes representatives from the governments of India 1 Project Agreement IFPS, Annex I, Amplified Project Description, IFPS Project, pg. 2 - September 30, 1992. $225 million was a bilateral agreement and $100 million was from USAID/Washington specifically for technical assistance. 2 Sample Registration System (SRS), 1992 3 20 Years of the Innovations in Family Planning Services Project in Uttar Pradesh, India, April 2012, pg. 3 4 A “parastatal organization” is defined as one that is owned or controlled wholly or partly by the government. 5 ibid 3 - pg 3 2 and Uttar Pradesh, USAID, and private sector experts, SIFPSA’s executive committee provides policy guidance and makes decisions on the procurement of personnel, products and services.6 Implementation of project activities began in 28 districts in 1994.7 The existence of SIFPSA allowed for greater innovation in project activities, the ability to hire technical experts with salaries higher than government agencies could offer, and enhanced control of project finances. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The evaluation questions include: 1. To what extent did the overall IFPS project make an impact on reproductive health behaviors and outcomes for men, women, youth, and vulnerable populations in targeted areas, based on the “evaluable” key indicators and comparison groups identified? 2. What are the key lessons learned from IFPS, including the strategies it introduced such as performance-based financing systems, district action planning, working with the private sector, and establishment of entities like SIFSPA? 3. What are IFPS’s most significant operational contributions to the field of RH/FP? 4. What performance or impact related conclusions can be made regarding specific innovative sub-projects, such as the PPP for ASHA support system, Networks – Voucher Schemes, Merrygold, Social Franchisee services, and what factors most contributed to their relative success or failure? 5. How effective has the project’s technical assistance been in promoting technical and program priorities, and improving the capacities of local institutions under NRHM to deliver RH programs in USAID-supported states, and what lessons can be drawn for future designs by GOI and USAID/India? 6. What is the enduring legacy of IFPS? How close are we to achieving the original objectives, noting the major milestone events and challenges of IFPS over 20 years? The evaluation methodology included an extensive desk review of all project-related documents, interviews with key informants and field visits to the three IFPS intervention states—UP, JH and UK—where team members met National Rural Health Mission (NRHM) government officials, district health officers and other government and non-governmental organization (NGO) informants; visited public and private hospitals, clinics, health centers and NGOs; and met with service providers, including medical officers (MO), nurses, auxiliary nurse midwives (ANMs) and accredited social health activist (ASHA)/community health volunteers (CHVs). To assess the impact of IFPS, a separate analysis using national data was conducted. The IFPS evaluation team used organizational network analysis (ONA) to aid in the determination of the IFPS legacy and 6 ibid 3 7 The original number of districts was 28, but many districts split and by 2000 the original 28 had become 33. 3 conducted a detailed review of documents prepared by the IFPS Technical Assistance Project (ITAP). Following the desk review, the team determined that assessing changes-over-time of the original project outcome indicators (CPR, TFR, use of modern methods and changes in method mix) using primary national data sets (e.g., National Family Health Survey [NFHS], District Level Household and Facility Survey [DLHS], and Annual Health Survey [AHS]) would yield the most reliable information about project impact. The team also sought access to primary project data (including the 1995 Performance Indicators [PERFORM] survey determining baseline values) to verify the national-level findings and for greater clarity regarding changes in the original 288 intervention districts. Unfortunately, numerous challenges were encountered in obtaining access to primary data sets, original questionnaires and to their respective data dictionaries for Reproductive Health Indicator Survey (RHIS), PERFORM, Strategic Objective #2 (SO2) surveys, NFHS, and DLHS.9 Because each of these surveys used a different survey protocol and measured different variables, the team made adjustments in response to these changes and used only the three officially recognized National survey data sets (NFHS, DLHS, and AHS) in its analysis. To provide the most comprehensive picture of the impact of IFPS, project outcome indicators were analyzed at the state level as well as at the district level. Per the request of USAID, the findings compare trends in the three IFPS intervention states to the Empowered Action Group States (EAG).10 The District Level Analysis used DLHS 1, 2, and 3 and created two separate data sets for analysis - one with individuals (i.e. women 15-49 years of ages) from UP and Bihar as the unit of analysis and the other with districts from UP, UK and JH as the unit of analysis. Bihar was selected for the district level comparison due to its demographic, health and economic similarities to UP.11 Generalized linear models (Logistic or Gaussian links) with baseline and time dependent adjustments were used to analyze each project outcome indicator (TFR, CPR etc.) based on exposure to IFPS interventions. In UP the "high-intensity intervention districts" include the original 28 (33 after redistricting) IFPS districts and the "low-intensity intervention districts" are the remaining 42 districts that were only exposed to statewide IFPS interventions (e.g. media campaigns, CSM, technical assistance etc.) 8 These original 28 became 33 primary intervention districts overtime due to re-districting. 9 Access to NHFS and DLHS data sets was obtained by the Evaluation Team through personal contacts. Despite repeated requests and attempts the team was unable to access project data sets including PERFORM, RHIS and SO2 surveys. 10 Empowered Action Group States (EAG), for analysis of state level trends were separated into two groups; the five larger states (UP, Bihar Madhya Pradesh, Rajasthan, Odisha) and the three smaller states (Jharkhand, Uttarakhand, Chhattisgarh), 11 Bihar was the state suggested and approved by USAID as the most appropriate state to use for comparison purposes. 4 FINDINGS AND CONCLUSIONS Impact of IFPS Using national survey data sets (NHFS, DLHS, AHS) from 1992-2010 to analyze EAG state and India level trends and to compare the original 2812 UP IFPS high-intensity intervention districts to 42 low-intensity intervention districts the team concludes:  IFPS I and II (1995-2007) had a significant impact on the uptake of modern contraceptive methods in the original 28 (33 after redistricting) districts. Other EAG states kept similar pace with this trend and, since 2005/6, Rajasthan has outpaced UP in the use of modern methods.  By 2007/8 (IFPS II) condom use in IFPS high-intensity intervention districts was significantly greater than in low-intensity intervention districts. On the whole, condom use in UP paralleled trends in Rajasthan but exceeded Bihar condom use throughout the life of IFPS. Attribution for this trend may need to be shared with the National AIDS Control Program (NACP) which has emphasized the use of condoms to stem the transmission of HIV/AIDS.  Over time, other modern methods, including OCP and IUDs, showed no significant trends, but in 2007/8 (IFPS II) female sterilization showed a significant difference and declining use in UP high-intensity intervention districts when compared to Bihar. Based on observations as well as consumer and provider interviews, it appears that in the original (high-intensity) intervention districts couples have greater awareness, broader choice, and wider availability of modern spacing methods. In contrast, states like Bihar show sterilization trends increasing since 2002 and low utilization rates of other modern methods (condoms, IUD, OCP).  During IFPS I, in all EAG states TFR continued to decline. The three IFPS intervention states had significant declines in TFR between 1998 and 2008 (p < 0.01); however, no significant difference in TFR can be attributed to IFPS interventions at the district level.  During IFPS I, from 1992-2004, CPR made notable gains, especially in UP compared to other EAG states. Between 2002/4-2007/8 CPR rate of growth slowed to 2.6% in high intensity districts versus low-intensity districts that slowed to 2%. In 2005 (beginning of IFPS II) CPR trends declined (with the exception of Rajasthan) as NRHM, (incorporating a 12 28 original districts became 33 districts due to redistricting. 5 number of IFPS innovations, e.g. district action plans, ASHA) was rolled out with a broad reproductive health mandate and an emphasis on meeting Millennium Development Goals (MDGs) related to lowering maternal and infant mortality rates. Since 2007 CPR trends have improved, probably due to the maturing of NRHM,  Since 2005/6 (IFPS II), with the exception of Rajasthan, there has been an upward trend in "unmet need." Based on field observations, the emphasis on safe motherhood interventions13 as a critical aspect for lowering maternal and infant mortality provides a unique and under-utilized opportunity to emphasize birth spacing and limiting.14 IFPS Phase Assessment  Phase 1, (1992-2004) was characterized by intensive interventions15 in 28 (33) districts of UP and demonstrated a greater rate of CPR increase (19.8-43.6) than other EAG states, a decreasing TFR (4.8-3.8) and significantly increased use of modern methods in the high￾intensity intervention districts.  IFPS II (2005-2008) added UK (former UP districts) and JH and focused primarily on technical assistance, especially for the creation of PPP and other innovative pilot strategies. During this period, indicators across the EAG states made relatively little progress in most instances.  IFPS III (2009-2012) continued the work of IFPS II, with increased focus on the documentation and dissemination of PPP models. Based on AHS data it would appear that progress has been made on the majority of indicators. IFPS Operational Contributions The most notable IFPS innovations that have been adopted by NRHM and scaled-up throughout the country include District Action Plans, Quality Assurance Programs, the use of village level health workers (ASHA),16 engaging village level leadership (e.g. Panchayat Raj or Pradhan) and the creation of Divisional Clinical Training Centers. Based on field observations, the early work of IFPS and SIFPSA is evident in hospital quality-assurance programs; regional training centers for ANMs; NRHM’s use of district action plans in all states; and the work of ASHAs in rural and urban settings, to mention a few examples. IFPS has served as a pilot testing ground for many approaches and, while more rigorous models for testing approaches (e.g., using control groups or districts) could have been utilized, NRHM uptake of these innovative approaches has helped to sustain a vibrant FP program in UP. 13 Most notably promotion of institutional delivery 14 The IFPS Post Partum IUCD (PPIUCD) intervention is currently attempting to take advantage of the increase in hospital deliveries by inserting IUDs immediately postpartum. 15 More detail on IFPS interventions can be found in the body of the report 16 A number of respondents reported that many others have demonstrated the effectiveness of village level workers 6 Public Private Partnerships The piloting, testing and documentation of innovative PPPs have been primary foci of IFPS II and III. Greater clarity as to the definition, implementation and evaluation of these types of initiatives is still required.17 The Contraceptive Social Marketing Program (CSMP), the oldest and most successful PPP model, has expanded the availability of contraceptives in rural UP. Other models, such as the Sambhav Vouchers, while effective in some states (e.g., UP), have been discontinued by the government in Uttarakhand because the state prefers to utilize their resources on supporting government institutions rather than private facilities. The distrust of the private sector among government officials remains a powerful barrier to the long-term success of PPP. Other initiatives like the Merrygold Health Network Franchise require more attention to meeting the needs of the franchisees and to evaluating the network’s impact beyond process indicators. ITAP and Technical Assistance From 1992–2012, technical assistance accounted for approximately 46% of total IFPS project spending and was provided by multiple U.S. cooperating agencies. This evaluation focused primarily on the technical assistance provided by the Futures Group ($28.1 million) during IFPS Phase II and III (2005–2012). Working in three states and with the national government, ITAP supported national behavior-change communication (BCC) campaigns and facilitated the implementation and documentation of the PPP Initiatives. In the current India health and economic context, projects such as ITAP, providing technical assistance without “financial strings” to the government, would benefit from a clear set of mutually desired and agreed-upon programmatic objectives and indicators for success. Technical and managerial oversight from USAID also requires the delegation of financial and human resources with the necessary technical and diplomatic skills to serve the Indian government effectively in this challenging capacity. Role of SIFPSA SIFPSA, like many organizations, faces the challenge of staying relevant over a 20-year period in a changing socio-economic, political and demographic environment. Organizational divisions— such as those between private- and public sector interventions—will not serve an organization that is testing new models for collaboration—such as private-public partnerships. Depending on SIFPSA’s intended focus, having the highest quality staff and capacity to accomplish their chosen goals will be critical to provide the “unique selling point” required for credibility and effectiveness. Strong and stable leadership has also been shown to be a critical factor for effectiveness and institutional respect. The legacy of SIFPSA includes its reputation as a leader in 17 This same point has been made in the following publication: Taneja, Udita, Bharti Birla Research Scholar, “Public Private Partnerships for Healthcare Delivery in India, “ The Internet Journal of World Health and Societal Politics ISSN:1540-269x 7 family planning, a strong and competent alumni network, close working relations with key reproductive-health players throughout the state and a strong NGO network in UP. All of these factors combine to support the continued role of SIFPSA in UP. Concluding Comments The final evaluation question is, “How close are we to achieving the original project objectives?” Given the changes in IFPS objectives – from a focus on family planning to a broader reproductive health mandate, and from providing direct services and technical assistance (TA) to only providing TA – UP’s progress from a CPR of 19.8% to 49.9% in 20 years is a notable achievement.18 UP TFR has also decreased from 4.8 to 3.6, which represents a 25% decline since 1992.19 More women are using modern spacing methods and (except for injectables) most methods are readily accessible in rural as well as urban settings. In conclusion, many IFPS/SIFPSA innovations like district action plans, the use of ASHAs, quality assurance programs, and others have been expanded throughout India. SIFPSA has the potential to serve as a "laboratory for innovation" with rigorous testing of ideas that could help India to further expand contraceptive choice beyond traditional female sterilization and thereby achieve these important MDGs. RECOMMENDATIONS Recommendations for SIFPSA 1. Reassess the organizational structure and align to a refocused mission relevant to the current context; 2. Rebuild core technical competence in accordance with redefined mission; 3. Use central position within FP/RH network to offer technical support to Govt. of UP/GOI, eventually serving as an "emersion learning site" or center of excellence for innovations. Recommendations for the Government of UP 1. Focus on Health Systems Strengthening by exploring options for PPP for health in order to expand quality care and services; 2. Explore potential role for a State level technical support unit to facilitate NRHM planning and implementation. 18 The original goal was for a CPR of 49%. The 50% CPR achieved includes use of all methods (modern, limiting and traditional) for women ages 15–49 years, and is based on AHS published results. 19 Based on NHFS 1 and AHS data 8 Recommendations for USAID 1. Advocate and support SIFPSA to realize its potential in the current context of RH/FP; 2. To augment NRHM's successful programming and broad focus on reproductive health, USAID should actively advocate for continuous, strong, and effective family planning interventions and utilize the platform of institutional delivery to promote family planning initiatives; 3. Refocus on family planning and serve as major advocate for FP/birth spacing in both the private and public sector; 4. To provide quality technical assistance in the current health environment, ensure that sufficient internal technical capacity, especially for project development, management and evaluation, is available; 5. Re-engage with GOI to create an appropriate PPP framework for health; 6. Support the development of external rigorous evaluation methodologies in synchrony with program start-ups to measure program impact. 9 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this evaluation was to conduct both an impact evaluation and a legacy evaluation of IFPS I-III, a $325 million bilateral agreement operating from 1992–2012. The Indian Ministry of Health and Family Welfare requested that a final evaluation be conducted by an external, independent agency. The governments of India and Uttar Pradesh, USAID/India and USAID Headquarters in Washington, D.C., other government and non-government stakeholders and development partners constitute important audiences for this evaluation. The report begins with a brief history of IFPS and what made IFPS unique. An overview of the 20-year period, from 1992–2012 offers insight into the political, socio-demographic and fertility context in which IFPS was implemented. A description of the IFPS phases, spending, activities and management will be followed by the evaluation purpose, methodology and impact findings. Per the scope of work (SOW), additional findings to be discussed include identifying and assessing: IFPS strategies and lessons learned; interventions involving public private partnerships; and the role of technical assistance especially during IFPS II and III. The legacy of IFPS and recommendations to the Government of India (GOI) and USAID conclude this evaluation report. EVALUATION QUESTIONS As per the IFPS Evaluation Scope of Work (SOW), the questions to be addressed include: 1. To what extent did the overall IFPS project make an impact on reproductive health behaviors and outcomes for men, women, youth, and vulnerable populations in targeted areas, based on the “evaluable” key indicators and comparison groups identified? 2. What are the key lessons learned from IFPS, including the strategies it introduced such as performance-based financing systems, district action planning, working with the private sector, and establishment of entities like SIFSPA? 3. What are IFPS’s most significant operational contributions to the field of RH/FP? 4. What performance or impact related conclusions can be made regarding specific innovative sub-projects, such as the PPP for ASHA support system, Networks – Voucher Schemes, Merrygold, Social Franchisee services, and what factors most contributed to their relative success or failure? 5. How effective has the project’s technical assistance been in promoting technical and program priorities, and improving the capacities of local institutions under NRHM to deliver RH programs in USAID-supported states, and what lessons can be drawn for future designs by GOI and USAID/India? 6. What is the enduring legacy of IFPS? How close are we to achieving the original objectives, noting the major milestone events and challenges of IFPS over 20 years? 10 SIFPSA seeks to facilitate, through innovative means and partnerships with government and other agencies, the goal of health for all by improving the quality, demand, access and delivery of family planning and MCH services and also improving related quality of life parameters, including the status of women. PROJECT BACKGROUND IFPS HISTORY AND CONTEXT In 1992, 40 years after beginning India’s first government-sponsored national family-planning program, the state of Uttar Pradesh (UP) was home to one-sixth of India's total population (131.9 million20) and slightly smaller than the sixth largest country in the world (Brazil).21 The demographic trends of the southern India states were progressing, but in many northern states, including UP, progress was slow and the country’s population of 846 million was quickly approaching one billion people. Given the size and scope of UP’s population-related issues, USAID saw the potential for demographic global impact and, following an extended period of design and negotiation, signed the IFPS bilateral agreement on September 30, 1992.22 IFPS UNIQUE ELEMENTS A unique element of the IFPS project design was the creation of an autonomous parastatal organization,23 the State Innovations in Family Planning Services Project Agency (SIFPSA), as a mechanism “to provide flexibility and avoid bureaucratic delays endemic to government systems.” 24 With a governing body that includes representatives from the governments of India and Uttar Pradesh, USAID, and private sector experts, SIFPSA’s executive committee provides policy guidance and takes decisions on the procurement of personnel, products and services.25 Established in Lucknow in 1993, implementation of IFPS project activities began in 28 districts in 1994.26 The existence of SIFPSA allowed for greater innovation in project activities, the ability to hire technical experts with higher salary expectations, and enhanced control of project finances. Another unique design feature of IFPS was the channeling of funds through performance-based disbursements (PBD), which provided funds based on project outputs, rather than inputs. Working in close collaboration, USAID and SIFPSA set mutually agreed-upon project outcomes, 20 Census of India, Provisional Population Total, 1991 21 ibid 22 Personal communication, Mr. John Dumm, Advisor to the President, Pathfinder International, October 11, 2012, Delhi 23 A “parastatal organization” is defined as one that is owned or controlled wholly or partly by the government. 24 ibid 3 - pg 3 25 ibid 26 The original number of districts was 28, but many districts split and by 2000 the original 28 had become 33. Box 1: SIFPSA Mission Statement 11 called “benchmarks”—measureable indicators—that were to be achieved within a specific timeframe and designated financial value.27 The use of benchmarks, coupled with comprehensive procurement and management systems, has enabled SIPFSA to operate transparently and without accusations of corruption.28 IFPS 20 YEAR CONTEXT: 1992–2012 The 20-year span of IFPS represents a period of significant change in India in terms of family planning (FP), India’s economic transition, and its position within global politics. During this evaluation, many respondents identified external factors that they felt impacted the original 1992 IFPS intention to expand FP choices beyond sterilization methods. In the early 1990s, FP services in India were primarily delivered via “sterilization camps” and availability and uptake of methods other than sterilization remained limited. In 1996, following the 1994 Cairo International Conference on Population and Development (ICPD), the Government of India (GOI) eliminated method-specific targets (e.g., for male and female sterilization) and changed the Family Planning Program to promote a wider choice of methods that address broader reproductive health (RH) issues.29 The contraceptive prevalence rate (CPR) was on the rise and the total fertility rate (TFR) was decreasing (especially in the southern states), but progress at the national level masked important differentials among sub-groups of the population: rural-urban, rich-poor and between the educated and the uneducated.30 Since 1996, the National Family Welfare Program31 has focused more on RH issues.32 In 1997 India launched the Reproductive and Child Health Program with a focus on reducing MMR. During this time the socio-demographic and political climate in India was relatively passive towards family planning. In the United States, similar trends emerged with a change of leadership in 2000 followed by the September 11 attacks, which drew world attention away from over-population and toward the threat of terrorists. In 2000, the UN creation of the Millennium Development Goals (MDGs) and the focus on decreasing maternal and infant mortality rates set 27 ibid 3 28 This achievement was noted by multiple respondents especially in light of the accusations of the alleged corruption scandal, in which top politicians and bureaucrats are alleged to have siphoned off an estimated US$1.82 billion from the National Rural Health Mission. At least five people are said to have been murdered in an attempt to cover-up large-scale irregularities. February 17, 2012 29 Jain, Anrudh K, Ph.D., and Jain, Aparna, M.P.H, UNFPA - ICOMP REGIONAL CONSULTATION, Family Planning in Asia and the Pacific, Addressing the Challenges, December 8–10, 2010, Bangkok, Thailand, Family Planning and Fertility in India, pg. 6 30 ibid 31 The name of India's National Family Planning Program was changed to National Family Welfare Program in 1978. 32 ibid "Benchmarks are like real life . . . first you do the work . . . then you get paid." –UK Government Official 12 the bar for achievement of major public health indicators by all countries. At the same time, India developed the National Population Policy 2000, with specific goals—including universal access to quality contraceptive services—in order to lower the TFR to 2.1; reduce infant mortality rates (IMR) and the maternal mortality ratio (MMR); achieve zero marriage of girls below the age of 18; increase deliveries conducted by trained persons to 100%; and et cetera.33 By 2004, the growing HIV/AIDS epidemic, and its potential impact in India resulted in the initiation of The President’s Emergency Plan for AIDS Relief (PEPFAR), drawing both attention and resources further away from FP. For IFPS, especially phases II & III, the most significant event was the initiation of the National Rural Health Mission (NRHM) (2005– 2012) under the Ministry of Health and Family Welfare/Department of Health and Family Welfare (MOHFW/DHFW). The goal of NRHM was to improve the availability of and access to quality health care, especially by those residing in rural areas: the poor, women and children.34 The main objective of the program was to bring about a change in three critical health indicators, i.e., reducing total fertility, infant mortality and maternal mortality rates.35 With a focus on 18 states (including UP, UK and JH), the NRHM was an articulation of the commitment of the GOI to increase public spending on health from less than one percent (0.9%) of GDP to two to three percent of GDP.36 IFPS piloted a number of innovations in the late 1990s to early 2000s—including district action plans, the use of community-based volunteers and working directly with Village Health and Sanitation Committees—which were incorporated into the NRHM action plan. In 2005, with the initiation of NRHM, the UP State Health Society (SHS) was created and the role of SIFPSA as an autonomous parastatal entity, now led by the NRHM mission director, became unclear. 33 accessed November 13, 2012 34 , National Rural, Health Mission (2005-2012) Mission Document, pg. 2. 35 accessed November 13, 2012 36 The 18 States are Arunachal Pradesh, Assam, Bihar, Chhattisgarh, Himachal Pradesh, Jharkhand, Jammu & Kashmir, Manipur, Mizoram, Meghalaya, Madhya Pradesh, Nagaland, Orissa, Rajasthan, Sikkim, Tripura, Uttarakhand and Uttar Pradesh. Box 2: Janani Suraksha Yojana Scheme Janani Suraksha Yojana (JSY), a safe motherhood intervention under the NRHM, is being implemented with the objective of reducing maternal and neonatal mortality by promoting institutional delivery among poor pregnant women. JSY integrates cash assistance with delivery and post-delivery care. The success of the scheme is determined by the increase in institutional delivery among poor families. All mothers irrespective of age, birth order, or income group (BPL & APL) will get cash assistance of Rs 1400 in a lump sum at the time of delivery. ASHAs receive Rs 600 for accompanying a rural delivery and Rs 200 for an urban delivery. If the mother or her husband, of their own will, undergoes sterilization immediately after the delivery of the child, compensation money of Rs 600 for tubectomy and Rs 1,100 for vasectomy is available under the existing family welfare scheme and should also be disbursed to the mother at the hospital itself. 13 UTTAR PRADESH CONTEXT The challenges of working in UP, especially related to the delivery of health services must also be noted. In addition to the alleged NRHM corruption scam,37 the availability of trained medical manpower is lacking. A recent survey found that 78% of government posted doctors were working in private nursing homes and the remaining had set up their own clinics, spending just 2-3 hours in their official posts.38 This practice, high levels of poverty, illiteracy and significant changes in political leadership have characterized the 20 years of IFPS and created an especially challenging work environment. IFPS Phases Table 1 provides an overview of IFPS Phases I-III. In 1998, to comply with GOI requirements, the project’s original emphasis on FP was broadened to focus on RH and included antenatal care (ANC) and institutional delivery indicators. In Phase II, (2005–2008) the project expanded beyond UP at the request of the GOI to include a new state, Jharkhand (with districts separated from Bihar), and continued its activities in the newly formed state of Uttarakhand, which was created in 2000 from 13 former UP districts, adding another layer to project complexity. An additional reason for expanding the geographic scope beyond UP was the availability of Phase I carry over funds (US$117.4 million) and the need to address Jharkhand's poor FP and health indicators.39 Phases II and III (2005–2012) aligned with USAID's shift in program emphasis from direct implementation to TA and focused on the creation of private-public-partnership models for FP and RH. 37 Please refer to footnote 28 38 "UP Health Sector is Critically Ill" - the Sunday Guardian, Lucknow 15 July 2012 39 Personal communication, Mr. J.S. Deepak, SIFPSA Executive Director June 2003-July 2004, Delhi, India October 30, 2012. IFPS I: TA Consortium Partners (1995-2004) 1. The Futures Group 2. CARE 3. Engender Health 4. JHPIEGO 5. JHUCCP 6. Intra Health 7. Population Council 8. CEDPA 9. PATH 10. John Snow Inc. 11. Michigan Fellows Box 3: TA Consortium Partners 14 40 The numbers shown here were provided after further analysis by USAID, and differ from the numbers the team collected in the field. The funding information gathered by the evaluation team suggests that the full obligated amounts were not spent; therefore the funding information provided in the evaluation report is inconclusive. An analysis of financial data was outside of the SOW for the evaluation, and further inquiry would be needed to determine funding utilization. 41 Originally 28, became 33 after redistricting 42 Eighteenth Amendatory Agreement to the Project Grant Agreement Between the President of India and the USA for Innovations in Family Planning Services, August 24, 2004, pg 7 43 State Action Plan Innovations in Family Planning Services Project April 2009 - March 2012, pg 4 PHASE I40 PHASE II PHASE III YEARS 1992(5)-2004 2005-2008 2009-2012 GEOGRAPHIC COVERAGE 33 districts Uttar Pradesh (UP)41 UP, UK, JH (focus on three districts) + national UP, UK, JH + national OBJECTIVES  Increase access  Promote method use  Improve quality of FP services  1998 expanded to include antenatal care (ANC) tetanus toxoid (TT) and iron and folic acid (IFA) and institutional deliveries  Develop, demonstrate, document and leverage working models of public private partnership for improved delivery of integrated reproductive and child health services  Training institutions and capacity of providers strengthened for improved delivery of quality FP/RH services.  Improved demand, awareness and use of family planning/reproductive health services and products  Community-based delivery of FP/RH services and counseling strengthened for increased awareness and use of FP/RH services  Documentation and dissemination of PPP models introduced in Phase II and scale-up of successful models FUNDING APPROACH Performance Based Disbursements (PBD - based on benchmarks) + Technical Assistance (TA) PBD + TA PBD + TA FUNDING Obligation Expenditure Obligation Expenditure Obligation Expenditure PBD: $108.074m TA: $100 m PBD: $108.07m TA: $100 m PBD : $26.073 m ITAP TA: $17.47 m JHPIEGO TA: 0.8m PBD: $25.6 m ITAP TA: $17.47 m JHPIEGO TA: 0.8m PBD:$11.43m ITAP TA: $11.15 m JHPIEGO TA: 9.85m PBD: $8.18 m ITAP TA: $11.15 m JHPIEGO TA: $ 9.85 m EXPECTED END OF PROJECT OUTCOMES  Increase CPR from 35 to 50%  Decrease TFR from 5.4 to 4.0  Increase method mix UP42  Modern method CPR increased to 30.6%  Percentage of women receiving sufficient quantity of IFA tabs during last pregnancy increased to 43.2%  Over 420 million condoms sold in rural areas  Over 14 million cycles of oral pills sold in rural areas UP, UK, JH  Three state proposals for RCH II developed and funded with local resources  Over $60 million leveraged over the UP and JH43  Increased contraceptive prevalence  Increase use of modern spacing methods UP, UK, JH and National  Scale-up of at least 3 RCH innovations  Capacity of at least two institution enhanced to provide TA on RH in two states  Basket of contraceptives expanded  Capacity of institutions at national level enhanced in FP/RH, training (i.e., NIHFW) and BCC (i.e., BCC unit of MOHFW) 15 During IFPS Phase I (1995-2004)44 a consortium of US cooperating agencies with $100 million directly from USAID Headquarters, provided technical assistance to IFPS (See Box 3).45 In 2005 the IFPS II Technical Assistance Project (ITAP), managed by Futures, was created with an additional $28 million. It's important to note that throughout the 20 years of IFPS, increasing CPR and decreasing TFR have remained key performance indicators of project impact. IFPS Financials IFPS project achievements were measured in terms of benchmarks completed, also referred to as PBD. For each state, the process of creating the benchmarks was an intensive, time￾consuming process, with opportunities for revision and addition of benchmarks over time, as 44 The IFPS project was signed in 1992 but project activities, including the work of the Technical Assistance Consortium officially began in 1995. 45 Phase 1 TA was not part of the evaluation SOW and is therefore only included to offer the broader historical context life of project from GOI and other development partner resources  By end of project, at least one working model of PPP funded by other agencies for wider implementation UK only  Uttarkhand Public Health Directorate staffed, five-year plan Figure 1: IFPS Benchmark Achievements by States 25 31 10 20 28 5 32 0 5 10 15 20 25 30 35 40 45 50 Uttar Pradesh Uttarkhand Jharkhand Percantage Benchmark Achievement in IFPS I - III by States IFPS I IFPS II IFPS III Total Total Benchmarks Achieved: 434/605 Total Benchmarks Achieved: 57/60 Total Benchmarks Achieved: 42/62 44 36 16 needs and conditions changed. All work was focused around completing the benchmarks and their results-oriented nature allowed for flexibility and creativity in achieving them. As seen in Figure 2, UK dropped three of their 60 benchmarks and government representatives expressed satisfaction with this mechanism.46 In JH, a different picture emerged, with many benchmarks dropped (or not met) and consistent delays in meeting benchmark deadlines. In UP, the percentage of benchmarks dropped increased with each phase of the project. Figure 247 shows total IFPS spending, PBD by state, as well as the total funds spent for technical assistance. The total bilateral project spending for the 20 years was $141.85 million of the $145.58 million originally obligated plus $139.27 million for technical assistance. About half (50%) of total project funds were spent on PBD. IFPS MANAGEMENT AND LEADERSHIP During IFPS I, management involved three key players: USAID/India, the GOI/UP and SIFPSA. From 1998–2003, SIFPSA was led by dynamic Indian Administrative Service (IAS) Officer Ms. A. 46 An Office of Inspector General USAID Audit report found the following shortcomings with the benchmarking process: 1) project was behind in achieving benchmarks 2) significant amounts of funds were unused 3) methodology to value the benchmarks lacked accuracy 4) project lack transition/sustainability plan for FP activities. "Audit of Phase III IFP," August 25, 2011. 47 The numbers shown here were provided after further analysis by USAID, and differ from the numbers the team collected in the field. An analysis of financial data was outside of the SOW for the evaluation, and further inquiry would be needed to determine funding utilization. 100, 37% 26.62, 10% 132.8, 49% 7.4, 3% 1.8, 1% Total Expenditure in IFPS - PBD and TA 1995- 2012 Consortium TA ITAP TA PBD UP PBD UK PBD JH Total Expenditure: $ 281.12 million in 20 years Figure 2: IFPS Total Expenditure 1992-2012 17 Johri, representing the Ministry of Health/Department of Family Welfare. Working closely with her was another acknowledged leader, Mr. J.S. Deepak, who served as Executive Director from 2003-2004. This six-year period was referred to by every respondent as the “SIFPSA Golden Era,” a time of intensive interventions and strong leadership. Staffing of SIFPSA ranged from more than 100 employees during this period, as compared to 2012 staffing of 60, which includes just one medical doctor. At USAID/India, the 1990s were defined by a strong technical team with extensive FP/RH knowledge (both medical and management). Simultaneously, the GOI was highly motivated to improve their RH performance and they needed U.S. financial assistance to achieve their objectives. By 2005, when NRHM began, there appeared to be a lack of clarity as to the exact role of SIFPSA vis-à-vis a newly created Uttar Pradesh State Health Society, whose mandate was to focus on rural health, especially for women and children. NRHM is fully funded by the GOI and during its early years many states, including UP, had difficulty spending the funds allocated. From 2004–2012, a total of 15 NRHM directors were assigned to lead SIFPSA, with limited focus or success.48 As Figure 3 illustrates, a notable CPR increase in Uttar Pradesh appears to have been achieved during 1998–2004, a period of strong, stable leadership, numerous project interventions and strong IFPS spending. 48 SIFPSA noted that 15 directors were assigned to lead; however, ITAP notes that not all 15 of these executive directors were mission directors of NRHM. Figure 3: CPR Trends in India, Bihar and Uttar Pradesh, 1992–2011 18 IFPS INTERVENTIONS IFPS aimed to increase CPR and decrease TFR throughout UP by reaching out with a broad mix of contraceptive choices beyond the traditional sterilization option specifically to rural, underserved populations. Using innovative strategies and public-private partnerships (PPP), both leadership and management saw the need to raise demand (through behavior-change communication [BCC]), improve service delivery (through training of providers and capacity building of institutions) and to increase access to services (through CBD of FP/RH messages and services). The need for tracking and documenting work was also acknowledged and implemented. IFPS interventions fall under the above mentioned five broad categories: (1) Clinical training/institutional capacity building, (2) BCC, (3) CBD of FP/RH Services, (4) PPP and 5) Evidence Generation. Table 2 provides a sampling of the specific activities associated with each intervention. INTERVENTION CATEGORY EXAMPLES OF INTERVENTION Training and Capacity Building District Action Plan (DAP) support Postpartum Intrauterine Device Insertion (PPIUD) trainings Contraceptive Training Updates (CTUs) for Medical Orderlies and Nurses Infection Control Trainings Dairy Cooperative Engagement and Trainings Auxiliary Nurse Midwife (ANM) Trainings Indian System of Medical Practitioners (ISMP) trainings Emergency Obstetric Care Training Support for Nurses Skilled Birth Attendant training (Dai training) Quality Assurance – Standard Based Management Recognition (SBMR), etc. Facility Up-gradation Divisional Training Centers Panchayat Raj Institution trainings Non Scalpel Vasectomy (NSV) training and facility support Reproductive Child Health (RCH) camp support Trainings for Village Health and Sanitation Committee NGO CBD Interventions49 Community Based Distribution (CBD) Interventions Clinic Based Interventions PPP Interventions Contraceptive Social Marketing Program (CSMP) Private Sector Employees intervention Merrygold Clinics Vouchers Mobile Vans 49 NGO/CBD Interventions were considered to be part of the PPP interventions but have been included as a separate category in this table to highlight their importance. 19 INTERVENTION CATEGORY EXAMPLES OF INTERVENTION Family Life Education/Adolescent Health Initiative Accredited Social Health Activist (ASHA) Support System BCC Interventions Tetanus Toxoid Campaign Aao Baten Karen - “Come Let's Talk” Marital Age Increase Program Support to MoHFW for FP Campaign, National IEC/BCC Workshop Evidence Generation Program Evaluation Review for Organizational Resource Management (PERFORM) Survey Strategic Objective 2 (SO2) survey Reproductive Health Indicator Survey (RHIS) Policy Uttar Pradesh (UP) Population Policy Table 2: IFPS Interventions at a Glance The PERFORM Survey, designed to establish project baselines, was the first intervention conducted in 1995 in 28 districts of Uttar Pradesh. Over time, these 28 districts were subdivided into 33 districts where the majority of interventions occurred. BCC campaigns and Contraceptive Social Marketing were conducted statewide, so IFPS/SIFPSA interventions have touched every district of UP (Figure 4 and Figure 5). The maps below show all interventions conducted in UP during IFPS Phases I-III. Please note that in the charts below, the symbols represent the different types of intervention, and the color of the symbols represents the different phases of IFPS. Figure 4: IFPS Interventions in Uttar Pradesh 20 The maps of Uttarakhand and Jharkhand (Figure 6 and Figure 7) illustrate interventions during IFPS Phases II and III. In Jharkhand, IFPS was asked by the GOI to work specifically in three districts: Simdega, West Singhbhum and Giridih. Some of the PPP projects (e.g. Mobile Health Units, Sambhav Vouchers) were scaled up in each of these states, but only the pilot projects are reflected in the map. PPP will be discussed in more detail later in this report. Figure 5: IFPS Interventions in Uttar Pradesh – Training/Capacity Building Figure 6: IFPS Interventions in Uttarakhand 21 IFPS III In the above detailed description and maps of IFPS I–III, one observes the reduced scale and number of interventions during Phase III. The final three years of IFPS (Phase III 2009–2012) focused on “strengthening health systems for delivery of quality FP and RH services.”50 Efforts concentrated on demonstrating the effectiveness of innovative models with the goal of handover and adoption by the states by the end of IFPS III. Four broad areas of focus included: 1. Training institutions and the capacity of providers strengthened for improved delivery of quality FP/RH services; 2. BCC activities for improved demand, awareness and use of FP/RH services and products; 3. CBD of FP/RH services and counseling (addressing myths and misconceptions on FP methods) strengthened through NGO projects; 4. Existing PPP projects completed, evaluated and documented. 50 Uttar Pradesh State Action Plan - IFPS April 2009-2012, pg 5 Figure 7: IFPS Interventions in Jharkhand 22 Table 1 (IFPS I–III Overview table) includes the IFPS III performance targets, which will be addressed in the “Findings” section. The Phase III PBD spending was only $ 8.18 million (6% of total IFPS PBD spending) split between three states, plus $11.15 million for ITAP technical assistance (TA) and an additional $9.85 million to The Johns Hopkins Program for International Education in Gynecology and Obstetrics (JHPIEGO) for strengthening of FP services, pre-service education and implementation of the Postpartum Intrauterine Contraceptive Device (PPIUCD). EVALUATION METHODS AND LIMITATIONS METHODOLOGY USAID contracted Social Impact, Inc. (SI) and its partner, Management Systems International (MSI), to assemble a five-member team (two international and three national specialists) to conduct an impact and legacy evaluation of IFPS during a seven-week period from September 17 – November 7, 2012. The evaluation team included Team Leader and Senior Reproductive Health Specialist Dr. Michele Andina, Evaluation Methods Specialist Dr. Jenny Ruducha, Demographer Dr. Rahul Dev Bhawsar, Senior Population Analyst Dr. Dipanjan Sujit-Roy and Public Private Partnership Specialist Mr. Soumitro Ghosh. The evaluation methodology included (1) team planning meetings between the team and USAID/India (teleconference and in person); (2) an extensive desk review of all project-related documents; (3) interviews with key informants in Delhi and around the world, both in person Figure 8: Site Selection and Visits for IFPS Evaluation 23 and via Skype; (4) field visits to the three IFPS intervention states (UP, JH, UK) where team members met NRHM government officials, district health officers and other government and NGO informants, visited public and private hospitals, clinics, health centers and NGOs and met with service providers, including MOs, nurses, auxiliary nurse midwives (ANMs) and accredited social health activist/community based volunteers (ASHA/CHV); (5) midterm briefings with SI and USAID through teleconference; and (6) presentations and discussion of findings with members of the IFPS team and USAID. To determine the impact of IFPS, a separate analysis using national data was conducted. The IFPS evaluation team employed organizational network analysis (ONA) to aid in the determination of the IFPS legacy and conducted a detailed review of ITAP prepared documents. The respective methodologies for each analysis will be presented in the appropriate sections of this report. The evaluation team conducted field visits to understand the legacy of IFPS and to inform subsequent quantitative impact findings with qualitative data. Key informant interviews further enhanced the findings of the IFPS evaluation team and provided additional insights. All five IFPS evaluation team members spent six days in UP and, subsequently, a team of two members visited either JH or UK (five days), while the fifth team member remained in UP for three days of additional interviews, then returned to Delhi for data analysis. Figure displays the specific districts visited by the team. Criteria for the selection of districts visited in UP was based on observing a sample of the “high￾intensity intervention districts”—i.e., the 28 original (33 after redistricting), 1995 IFPS intervention districts—and “matching” these with observations in “low-intensity intervention districts”— i.e. the 42 remaining districts (primarily exposed to statewide IFPS interventions such as BCC campaigns and CSM programs).51 The team selected four high-intensity (Shahjahanpur, Kanpur Naga, Allahabad, Sitapur) and two low-intensity districts (Hardoi, Faizabad) in Uttar Pradesh (UP) based on travel time and the availability of Phase III PPP interventions. For the PPP assessment, the evaluation team held a group meeting of Merrygold Health Network franchisees in Kanpur Nagar, UP.52 In JH and UK, the team held meetings with government officials in Ranchi and Dehradun, respectively; in UK, the team visited one intervention district (Nainital,) and one non-intervention district (Pauri Gharwal). In JH, the team visited two of the three IFPS intervention districts (Simdega and West Singhbhum). 51 “Matching” here was based on similar CPR, TFR rates and demographics. 52 This was the first time that this group of Kanpur Merrygold franchisees had come together for discussion. 24 FINDINGS ANALYSIS During the 20-year period of IFPS, multiple national surveys were conducted to collect relevant state- and district-level data. IFPS collected additional project data for baseline determination and to study progress toward benchmarks and project indicators. (See Table 3). DATASET TYPE YEAR National Family Health Survey (NFHS) - Phase 1,2,3 National level Based on standardized Demographic and Health Survey (Measure) 1992-93 1998-99 2005-06 District Level Household Survey (DLHS) - Phase 1,2,3 State level survey Rapid Household Survey 1998–99 2002–2004 2007–2008 Annual Health Survey (AHS) 2010 State level dataset for select states 2010 PERFORM Survey Project level baseline data for 28 original districts 1995 Reproductive Health Indicator Survey (RHIS) Project level dataset 2003, 2005, 2010 Strategic Objective 2 (SO2) Survey Project level annual dataset 1998, 1999, 2000, 2001, 2002 Table 3: National, District and IFPS Data Sets Following the IFPS Evaluation desk review, the team determined that assessing changes-over￾time of the original project outcome indicators (CPR, TFR, use of modern methods and changes in method mix) using national data sets (NFHS, DLHS, AHS) would yield the most reliable information about project impact. The team also sought access to project data to verify the national-level findings and to gain greater clarity of changes in the original 28 high-intensity intervention districts, 53 for which the 1995 PERFORM survey determined baseline values. Numerous challenges were encountered in obtaining access to primary data sets and to their respective data dictionaries for Reproductive Health Indicator Survey (RHIS), PERFORM, Strategic 53 These original 28 became 33 primary intervention districts over time due to re-districting. 25 Objective #2 (SO2) surveys, NFHS, and DLHS.54 Some additional challenges included the absence of original survey questionnaires, inconsistent district coding and differing sampling methodologies in the same data sets over time. Due to inaccessible and inconsistent data from project surveys and time constraints, the findings presented are drawn from the national datasets of India: the internationally accepted Demographic and Health Survey (DHS) (NFHS 1, 2 and 3); the government equivalent DHS (District Level Household and Facility Surveys -DLHS 1, 2 and 3); and the Annual Health Survey (AHS), 2010-11, a rapid survey done by government of India to inform the planning commission on preparation of the XII Five Year Plan for India. To provide the most comprehensive picture of the impact of IFPS, the team analyzed performance indicators including TFR, CPR, use of modern methods, method mix (condoms, OCP, IUD, female sterilization), coverage of TT and "unmet need" at the state and district levels. Per the request of USAID, the findings compare trends in the three IFPS intervention states to the Empowered Action Group States (EAG), separated into two groups; the five larger states (UP, Bihar Madhya Pradesh, Rajasthan, Odisha) and the three smaller states (Jharkhand, Uttarakhand, Chhattisgarh) all referenced against overall India rates. The District Level Analysis used DLHS 1, 2, and 3 and, to adjust for the data inconsistencies, created two separate data sets for analysis. One has subjects (i.e. women 15-49 years of ages) from UP and Bihar as the unit of analysis and the other uses districts from UP, UK and JH as the unit of analysis. The team selected Bihar for district level comparison due to its health, socio￾demographic and economic similarities to UP.55 The advantage of subject-level data is that more statistical power is available for multivariate models when the individual is the unit of analysis. District level analysis rates presented throughout the findings section were obtained from separate multivariate analyses. Statistical Methods The team analyzed state level data56 to determine trends over time for each of the following indicators: (1) TFR; (2) CPR; (3) Modern CPR (includes only modern methods such as Male and Female Sterilization, IUDs, OCPs and Condoms); (4) Method Mix – OCPs, Condoms, IUDs, Male and Female Sterilization, (5) Coverage of Tetanus Toxoid in pregnant women and (6) unmet 54 Access to NHFS and DLHS data sets was obtained by the Evaluation Team through personal contacts. Despite repeated requests and attempts the team was unable to access project data sets including PERFORM, RHIS and SO2 surveys. 55 The selection of Bihar as a suitable UP comparison state was approved by USAID/India. 56 Using NFHS 1, 2, and 3, DLHS 2 and 3, and AHS data. 26 need for family planning. The team compared trends to national rates and calculated significance (if any) using a chi squared test. The team made adjustments to the DLHS data to make it comparable to the NFHS datasets,57 then entered the data into MS Excel 2013 and used STATA 12 SE to complete the significance calculations. The team analyzed the district level data for UP using DLHS 1, 2 and 3 to assess the impact of IFPS interventions in high-intensity and low-intensity districts of UP. The dependent variables used for this evaluation are (a) for prevalence rates, Total Fertility Rate (TFR) Contraceptive Prevalence Rate (CPR), and Use of Modern Methods (MM); and (b) for method mix, Oral Contraceptive Use, Female Sterilization, Condom Use, and Inter-uterine Device Use. The main independent variable is whether an IFPS intervention occurred in a given district.58 In UP the "high-intensity intervention districts" include the original 28 (33 after redistricting) IFPS districts and the "low-intensity intervention districts" are the remaining 42 districts that were only exposed to statewide interventions (See Annex VII). This exposure variable is time dependent as it changes over time. To analyze each of the dependent variables (TFR, CPR, etc.), the multivariate model used two adjustments - baseline and time dependent. The baseline (i.e. 1998/9) adjustment made the districts similar so that they were comparable. The time dependent adjustment accommodated changes over time in the districts that may have biased the results (e.g. changing urban/rural proportion over time). The team used generalized linear models (Logistic or Gaussian links) coupled with a mixed model for the longitudinal data. All dependent variables were assigned binary codes in the subject-level analysis (logistic link) or rates in the district-level analysis (Gaussian link). The team used STATA 12 for all district- and subject-level analyses. Limitations Though the datasets use similar methodologies for obtaining data and are considered to be comparable, there are inherent differences that allow for a variety of interpretations of the same findings. For example, there are differences in respondent age groups for NFHS and DLHS. NFHS uses a DHS approved age group of 15–49 years of currently married women (CMW) and 57 Adjustments to CPR for modern methods included removing the emergency contraception rates from DLHS 3 and AHS data. The respondents for DLHS 2 were currently married women 15–44 years and not 15–49 years as calculated for NFHS 1-3 and AHS. This was adjusted to be comparable to the other data set. 58 The interventions were classified as high-intensity and low-intensity intervention districts. However the major difference was the presence of direct interventions in high-intensity districts and indirect interventions (e.g. statewide BCC, CSM, TA) in the others. The reader is referred to the fact that IFPS direct interventions took place in 28/33 districts in IFPS I (1995-2004) and shifted to TA mode in IFPS II and III, (2005-2012) but TA was provided across UP beginning in 1995. 27 DLHS (1 and 2) used CMW ages 15-44 years. Further, there are minor differences in sampling and attributions such that DLHS sampling allows for district level analysis of data, whereas NFHS is more appropriate for state and national level analysis. The calculation of confidence intervals for the AHS data was not possible due to the lack of access to primary data. Due to inaccessibility of project-specific data and the absence of randomized control groups, full attribution to IFPS for changes over time is not possible, though the team can identify some causal links. While trends can be observed and the DLHS District Level analysis may indicate some differences between the high- and low-intensity UP intervention districts, the lack of comparable "control districts," i.e. those without any IFPS interventions, is a limitation. The presence of other donors and projects in UP from 1992-2012 is also not quantifiable, further limiting the potential for full attribution. ITAP Documentation Review A detailed content analysis of ITAP documentation was undertaken to assess the quality and standards of the documents. A total of 115 documents (29 published and 86 unpublished) were prepared and produced during the seven years of ITAP (2005–2012). The team reviewed 15/29 (52%) published documents and 11/86 (13%) unpublished documents, representing 22% of total ITAP documents. Using a standardized checklist specially prepared for the purpose (attached as Annex III) 14 criteria were assessed including: organization of the report, executive summary, design and methodology, findings, lessons learned, etc. A rating scale of one to five was used (one being the minimum and five the maximum score) with a maximum possible score of 70 for each document reviewed. Please see Annex X for document scoring. Typically, each sample document was reviewed by two reviewers from the team and an average of the two reviewer's scores was taken to avoid observer bias. A non-probability sampling technique (heterogeneity sampling) was used with the goal to include at least one document representing each type of IFPS intervention from published as well as non-published reports, with a stated bias toward including published reports. Analysis of the results was carried out using rankings for different documents based on their score. Limitations of Document Review Despite using a standardized review questionnaire, this type of review has a high level of subjectivity related to the individual reviewer’s expertise, personal bias and perspective. The large volume of materials to be reviewed within a specific time frame limited the number of reviewers to two per document, thereby only generating a mean score for each document reviewed. 28 Organizational Network Analysis (ONA) Network analysis is a new field of study that employs a set of powerful tools for understanding and visualizing (mapping) the patterns of complex interactions that occur in government departments and organizations. ONA also provides quantitative data that substantiate the maps and their underlying patterns.59 The purpose of the IFPS network analysis was to establish the relative position and influence of SIFPSA within the FP and reproductive networks at the national, state and district levels. Key informants were identified by USAID and further refined by the IFPS evaluation team to target the most essential departments and organizations involved in UP. An instrument (see Annex III) was developed with a list of relevant organizations to systematically determine whether a connection existed between the organizations; if it existed, a series of four types of connections were examined.60 The quality of that connection was determined based on a Likert scale (poor, fair, good, or excellent). ONA interviews were conducted with representatives from National, UP State and UP district-level organizations. Network analysis utilized UCINet61 software and plot formation was completed using NetDraw,62 and then converted to JPEG files for visualization. ONA Limitations Time constraints prevented the IFPS evaluation team from interviewing all the national-level organizations, beyond the subset directly knowledgeable about the IFPS project or enjoying a direct relationship, as a coordinating government department or partner organization. Therefore, the analysis reflects “unconfirmed ties,” that is, the relationship identified by one department or organization was not confirmed by the other relevant department. General Limitations This evaluation highlighted the challenges of evaluating a 20-year large scale, multi-state project. This includes processing hundreds of documents; attempting to compare results from multiple national and project surveys, conducted over time with different sampling methodologies and coding systems; reconstructing lost institutional memory; tracking 59 Further information on ONA may be found at http://www.byeday.net/ona.htm accessed 2 Dec 2012 60 The four connections were (1) sharing information; (2) coordinating programs or services; (3) involvement in joint BCC/IEC; and (4) training or capacity building 61 UCInet: http://www.analytictech.com/default.htm 62 Netdraw: http://www.analytictech.com/downloadnd.htm 29 programmatic expenditures; and, finally, determining amidst all the implementing agencies if the work done and money spent has made an impact. 30 FINDINGS, CONCLUSIONS & RECOMMENDATIONS FINDINGS AND CONCLUSIONS Impact on Reproductive Health Key Indicators (Evaluation Question #1) To what extent did the overall IFPS project make an impact on reproductive health behaviors and outcomes for men, women, youth, and vulnerable populations in targeted areas, based on the “evaluable” key indicators and comparison groups identified? Throughout IFPS, "end of project outcomes" (evaluable key indicators) have remained decreased TFR, increased CPR, and improved method mix, specifically increased use of modern spacing methods (e.g. OCP, condoms and IUD) versus the historical dependence on limiting methods such as sterilization.63 To provide a broader picture of IFPS "impact," these indicators will be analyzed at the State and at the District level. State Level trends in unmet need for contraception and coverage of tetanus toxoid for pregnant women, one of two indicators for antenatal care (ANC), are also presented.64 The State Level analysis compares data for the Empowered Action Group States (EAG), separated into two groups; the five larger states (UP, Bihar Madhya Pradesh, Rajasthan, Odisha) and the three smaller states (Jharkhand, Uttarakhand, Chhattisgarh), all referenced against overall India rates. As discussed in the methodology section on limitations, despite the differences that exist between the NHFS, DLHS and AHS data sources, in the absence of other credible data these were used, with minor adjustments,65 to provide the findings below. DLHS 1, 2, and 3 serve as the data source for the District Level analysis comparing "high-intensity intervention districts" and "low-intensity intervention districts" to control-group districts in Bihar (deemed the most comparable state to UP on the basis of demographic and socio-cultural indicators). “High￾intensity Intervention Districts" include the 2866 original 1995 IFPS intervention districts and “Low-intensity Intervention Districts” are the 42 remaining districts that were primarily exposed 63 Modern spacing methods available in India include oral contraceptive pills, IUDs, condoms, and injectables. Limiting methods include sterilization for males (vasectomy and non-scalpel vasectomy) and for females (tubectomy). 64 Requested by USAID/India. 65 Adjustments were made to DLHS 2 and 3 dataset findings to make it comparable to NFHS 1-3 and AHS. The adjustments were done for modern methods, where emergency contraceptives were deleted from DLHS 3 and AHS. TT coverage was taken uniformly as ‘at least one TT injections taken during the pregnancy’ etc. 66 The original 28 became 33 over time as a result of division of districts. 31 to statewide IFPS interventions such as BCC campaigns, CSM programs, and technical assistance.67 This section will conclude with a summary of the findings and discussion of potential contributing factors, including the possible impact of IFPS. TFR Trends Comparing TFR trends for UP to other large EAG states, one sees why in 1992 UP, with the highest TFR,68 was an appropriate site for IFPS. The initial accelerated rate of decline in TFR for UP slowed by 1998 and was then comparable to India and other states. By 2010 UP's TFR declined below Bihar's and, over 18 years, UP reduced TFR by 25%, compared to TFR reduction in MP by 21%, Rajasthan by 11% and Bihar by 8% . 67 See Annex VII for coding of high-intensity and low-intensity districts 68 The 5.4 TFR in the original proposal, written in 1992 was taken from Sample Registration System data. NFHS results were only published in 1994/5. In any case, in both data sets UP had the highest TFR. NFHS 1 1992-93 NFHS 2 1998-99 NFHS 3 2005-06 AHS 2010 India 3.4 2.9 2.7 Uttar Pradesh 4.82 4.1 3.8 3.6 Bihar 4 3.7 4 3.7 Madhya Pradesh 3.9 3.4 3.1 3.1 Rajasthan 3.6 3.8 3.2 3.2 Odisha 2.9 2.5 2.4 2.3 0 1 2 3 4 5 6 TFR Trends in TFR in select states of India Data from NFHS 1-3 and AHS Figure 9: Trends in TFR in Select States of India 32 For the district level analysis, a proxy of TFR was calculated from all three DLHS surveys.69 Using adjusted rates, TFR significantly declined in UP from 3.2, [95% CI 3.1 – 3.3} in 1998-99 to 2.6, [95% CI 2.5 – 2.7] in 2007/8 (p < 0.01). JH and UK also showed significant declines over time in both states, with JH going from 3.6, [95% CI 3.5 – 3.7] in 1998-99 to 3.4, [95% CI 3.3 – 3.5], and UK declining from 3.3, [95% CI 3.1 – 3.4] to 2.8, [95% 2.7 – 2.9]; however, there was no significant difference in TFR in any district attributable to IFPS interventions. CPR70 Trends In 1992 CPR, representing the use of modern and traditional contraceptive methods71 in UP, was the lowest of the five major EAG states (19.8%), further justifying the need for the IFPS intervention. During IFPS 1 (1995-2004), UP showed a greater rate of increase in CPR as compared to other EAG states (≈ 2.0%, but not significant when compared to the reference value for India – see Table 4 for details). From 2005-2007,72 all EAG states and India show a declining or level CPR trend (see Figure 10 between A & B line). A notable exception is Rajasthan with a steep rise in CPR.73 Post-2008 CPR for UP regained ground with a rate of increase comparable to that during IFPS 1. During the same period, the other large EAG states demonstrated similar trend increases.74 69 DLHS surveys do not provide specific TFR estimations . A proxy defined as "the children born in the reproductive history of women 15-44 years old" was used. The TFR is a summary measure, based on the Age-Specific Fertility Rates (ASFRs), that indicates the number of children a woman would bear during her reproductive years if she were to experience the ASFRs prevailing at the time of the survey. The ASFR for any specific age group is calculated by dividing the number of births to women in that age group during the period 1-36 months preceding the survey by the number of woman-years lived by women in that age group during the same three-year time period. [Ref. page 77 section 4.1 NFHS 3 Report Vol 1] 70 Contraceptive prevalence rate (CPR) is defined as: the percentage of women of reproductive age who are using (or whose partner is using) a contraceptive method at a particular point in time, almost always reported for women married or in sexual union. Generally, the measure includes all contraceptive methods (modern and traditional). The indicator is calculated as follows: (# of women 15-49 using a contraceptive method / total # of women 15-49) x 100. 71 Modern methods include both limiting and spacing methods such as male and female sterilization, pills, condoms and IUDs. Traditional methods include rhythm methods, safe period and withdrawal techniques. 72 Synchronous with launch of NRHM in 2005, marking a reproductive health focus shift from contraceptives to safe deliveries 73 Rajasthan was not an IFPS intervention site and no information was collected about this state. 74 Please note that data after 2007-08 was collected for states (AHS mostly EAG and Assam) and no consolidated dataset is available for India as a whole. Thus, the evaluation team is unable to comment on the reference value for India. 33 Table 1: Needs Title State NFHS 1 1992- 1993 NFHS 3 2005- 2006 Difference in CPR (total over 12 years) Rate of Increase (per year for 12 years) Significance (Chi squared test) India 40.7 56.3 15.6 1.3% Reference Uttar Pradesh 19.8 43.6 23.8 1.98% P >0.05, Not significant Bihar 23.1 31.4 8.3 0.7% P >0.05, Not significant Madhya Pradesh 36.5 55.9 19.4 1.62% P >0.05, Not significant Rajasthan 31.9 47.2 15.3 1.3% P >0.05, Not significant Odisha 36.3 50.7 14.4 1.2% P >0.05, Not significant Table 4: Trends in CPR between IFPS I-III NFHS 1 1992-93 NFHS 2 1998-99 DLHS 2 2002-04 NFHS 3 2005-06 DLHS 3 2007-08 AHS 2010 India 40.7 48.2 52.5 56.3 54 Uttar Pradesh 19.8 27.1 34.2 43.6 38.4 49.9 Bihar 23.1 23.5 27.8 31.4 32.4 37.6 Madhya Pradesh 36.5 44.1 49.9 55.9 56.2 61.2 Rajasthan 31.9 40.3 45.9 47.2 57 64.5 Odisha 36.3 46.8 52.4 50.7 47 56.2 0 10 20 30 40 50 60 70 CPR(%) Contraceptive Prevalence Rate (%) in Select States of India Data from NFHS 1-3, DLHS 2-3 and AHS A B Figure 10: Contraceptive Prevalence Rate in Large EAG States of India 34 Among the smaller EAG states (see Figure 11) both Uttarakhand and Jharkand were officially included in IFPS II & III (2005-2012) and the majority of Uttarakhand districts75 were covered under IFPS I until 2000 when they were originally part of UP. Beginning in 1998 with a CPR of 43.1%, Uttarakhand has shown remarkable progress and by 2010 surpassed the national CPR target of >60%.76 In this chart one should also note that Jharkhand, which separated from Bihar in 2000, had a leveling off of its CPR rate from 2002-2008. However, since 2009 (IFPS III) the CPR trends for both the large and small EAG states show remarkable but non-significant increase. The table in Annex IX details the rate and pace of CPR statewide trend77 over time. 75 Pauri Gharwal was never an IFPS district. 76 National Socio-Demographic Goals of India. National Population Policy 2000. Government of India. Available at http://populationcommission.nic.in/npp_obj.htm. Accessed on 9th April 2013. 77 The end point has been taken as DLHS 3 as data for AHS is not available for India used consistently as reference for analysis. Figure 11: Contraceptive Prevalence Rate in Small EAG States of India 35 Looking at the district level CPR trends for UP high intensity and low intensity districts, it is important to note that in 1998/9 (DLHS 1 - mid IFPS 1) baseline rates were similar (30.3% versus 29.2%). The rate for Bihar was significantly lower at 25.1%. CPR among all districts in UP and Bihar increased significantly over time and, in a multivariate model, UP high intensity districts had significantly higher CPR in 2007/8 (IFPS II) than UP low-intensity districts, p < 0.001. For the period between 2002/4 and 2007/8 (end of IFPS 1 and IFPS II), the rate of growth slowed to 2% except in the UP high-intensity districts, where it was 2.6% during the same period. By 2007/8 the control districts in Bihar also had a significantly higher CPR, but still lower than the rate for UP (see Figure 12). Over the 10 year period (1998-2008) the difference favoring high-intensity and low-intensity UP intervention districts grew significantly on the order of 4.4% [95% Confidence Interval (CI) 1.9-6.8]. Figure 12: Contraceptive Prevalence Rate in UP High and Low Intensity Districts and Bihar 36 Trends in Usage of Modern Contraceptive Methods Figure 13 above, the usage of modern contraceptive methods in these states, shows similar trends as noted with CPR. From 1992-2005/6 one notes a steady increase for the five large EAG states with comparable slopes. Post 2005, with the exception of Rajasthan,78 all states shows a declining trend in the usage of modern contraceptive methods. From 2007 onwards the pace of using modern methods improved. The smaller EAG states show similar trends with the exception of Uttarakhand, which by 2005 had a higher usage of modern contraceptive methods (Figure 14 below). Jharkhand's usage remained unchanged from 2002-2008 but increased after 2008.79 78 This evaluation team is unable to make any comments regarding the situation in Rajasthan 79 DLHS 3 and AHS included emergency contraception and injectable contraceptives as part of their survey methods. These methods were not available in India prior to 2000 and have not been captured in earlier surveys. Figure 13: Trends in Usage of Modern Methods among Currently Married Women in Large EAG States 37 At the district level, the usage of modern contraceptive methods among women 15-49 grew significantly over time in both UP and Bihar, with Bihar increasing by 6%. In contrast, UP high￾intensity intervention districts had an increased rate of nearly seven percentage points from 21.7-28.5%, while UP low-intensity intervention districts had an increase of less than four percentage points, with no significant growth after 2004. In fact, at baseline in 1998 there was no significant difference in the use of modern methods between high-intensity and low￾intensity districts, but by 2007/8 there was a statistically significant difference (p < 0.01) in an analysis that compared average rates in 2007/8 after adjusting them for their values at baseline. Using a multivariate model80 and adjusting for differences in baseline value confirms a) the significant growth of modern methods prevalence over time in both Bihar and UP districts, b) the lack of difference between UP high-intensity and low-intensity districts at baseline, and c) the significant difference between high-intensity and low-intensity intervention districts in UP in DLHS 3 as the prevalence of modern method usage grew faster among high-intensity districts. 80 The team will provide the data sets accompanied by specific details of the multivariate model upon request from USAID. Figure 14: Trends in Usage of Modern Methods among Currently Married Women in Small EAG States 38 In the same multivariate model and after adjusting for the effect of time (all rates were increasing everywhere) and other covariates, UP high-intensity and low-intensity intervention districts differed at a rate of 14%, 95% CI [12-16%]. See Figure 15 below. Method Mix Expanding method mix was an important area of focus for IFPS. This was done through behavioral change communication, contraceptive social marketing (condoms and OCP) as well as through training and the improved quality of service delivery, especially for IUDS. The findings will therefore focus on these three spacing methods and look at trends for female sterilization, the most popular limiting method. Trends in Condom Use Figure 16, below, indicates that in the large EAG states, UP condom use exceeds the rate of India condom use. However, it must be noted that Rajasthan also had a high condom usage pattern and actually overtook UP sometime during the 2006/07 period with UP again exceeding Rajasthan post-2008. In the smaller EAG states, Uttarakhand has a history of high condom usage. Despite the decline noted in 2006/7, today Uttarakhand has one of the highest rates of condom usage in India comparable to non-EAG states like Tamil Nadu, Maharashtra, Kerala and Gujarat. In Jharkhand, the trend in condom usage has only made small gains since 2007 (Figure 17). Figure 15: Trends in Modern Methods Prevalence Rate in UP High and Low Intervention Districts and Bihar 39 Figure 17: Trends in Contraceptive Prevalence Rates in Uttar Pradesh High and Low Intervention Districts and Bihar Figure 16: Trends in Condom Usage by Partners of CMW in Large EAG States of India 40 At the district level, condom usage has increased over time in UP from 3.9%, [95% CI 3.8, 4.1%] in 1998/9 to 7.2%, [95% CI 6.9, 7.4%] in 2007/8, while it has remained very low in Bihar reaching just 1.4%, [95% CI 1.3, 1.5] in 2007-8. At baseline there was no difference between high-intensity and low-intensity intervention districts. By 2002, there was a non-significant trend toward greater condom use in the high￾intensity districts, which became statistically significant (p=0.003) in a multivariate model in the years 2007/8, at which point high-intensity districts had a rate of condom use of 7.3%, 95% CI [7.0, 7.6] in 2007/8, while the low-intensity districts had a rate of 6.8%, 95% CI [6.5 – 7.0%]. See Figure 17 above. Trends in Use of Oral Contraceptive Pills Figure 18, below, shows that from 1992-2010 the rate of OCP use in UP increased from 1% in 1992/3 to 2.7% in 2010. In a similar time period (approximately 16 years), India's rate went from 1.2% to 4.2% while Bihar moved slowly from 1.1% to a high of 1.5% in 2002/4 declining and then returning in 2010 to 1.5%81 In UP the trend follows similar lines as CPR and shows a leveling during the 2005/6 period. The progress of Odisha is notable, climbing from 0.9% in 1992/3 to 11.1% in 2010, but beyond the scope of this evaluation. Figure 19, below, shows OCP trends for the smaller EAG states. The use of OCP is similar in Uttarakhand and Jharkhand ( IFPS II & III intervention states) except that UK shows an early increase in OCP use and remains stable between 2004-2010. In contrast, Jharkhand fluctuates over time and converges with Uttarakhand in 2009/10. 81 AHS does not provide an estimate for India. This survey was limited to nine states and was contracted to an independent agency, ORG, by the Registrar General of India (census bureau). 41 Figure 18: Trends in Usage of OCP in Large EAG States of India Figure 19: Trends in Usage of OCP in Small EAG States of India 42 At the district level, the prevalence of OCP use among women 15-49 is less than 2% in both UP and Bihar from 1998-2008. Between 1998 and 2002 (IFPS I) there was a marginal increase, but those gains were erased by 2008 (IFPS II). UP high-intensity intervention districts had higher prevalence over time than low-intensity districts, but that is a non-statistically significant difference. See Figure 20 below. Trends in Use of IUDs Trends in IUD usage show wide variability among the larger EAG states. UP in particular shows an increase between 1998/9-2002/4 but then decreasing use after 2005/6, as can be seen in Figure 21. In the smaller EAG states, trends also show wide variance. While in 1998/9 Uttarakhand is closer to the India national average, Jharkhand remains in the lowest decile for India.82 Between 2002- 2010, Uttarakhand shows a steep decline in IUD use. 82 Jharkhand has among the lowest IUD usage in India. As per DLHS 3, IUD usage in India varies from 0.3 (Andhra Pradesh) to 6.7 (Sikkim). Jharkhand has the third lowest rate of 0.5 after Andhra Pradesh – lowest at 0.3; Orissa & Bihar – 0.4. District Level Figure 20: Oral Contraceptive Prevalence Rates in UP High and Low Intervention Districts and Bihar 43 Household and Facility Survey (DLHS3) 2007-08. International Institute of Population Sciences and Ministry of Health and Family Welfare, Govt. of India, 2010. Table 6.7, page 122. Figure 21: Trends in Usage of IUDs in Large EAG States in India 44 Figure 22: Trends in Usage of IUDs in Small EAG States in India District level IUD use was very low overall, averaging around 1.2% in UP and 0.5% in Bihar. Rates of IUD use did not change significantly over time in either state, with prevalence in 2007/8 no different than in 1998/9. There was no statistically significant difference between UP high-intensity and low-intensity districts at any point in time. However, a notable (though non-significant) difference was found in DLHS 3 where high-intensity intervention districts had an IUD prevalence rate of 1.2%, 95% CI [1.1-1.3], while the low-intensity intervention districts had a rate of 1.0%, 95% CI [0.86-1.1%]. See Figure 23 below. 45 Figure 23: Trends in IUD Prevalence in Uttar Pradesh High and Low Intensity Intervention Districts and Bihar Female Sterilization Trends Figure 24 and Figure 25 show trends in female sterilization in the EAG states as compared to India. Traditionally, India had high female sterilization rates and in NFHS 1 the contribution towards CPR was almost 67%.83 In recent years sterilization rates have declined and the contribution to CPR has decreased to about 50-65% across various states.84 UP shows a flattened trend with female sterilization rates markedly lower than the values for India and lower than all comparison EAG states. Among the small EAG states, female sterilization rates in Jharkhand follow the national pattern and show a modest rise since 2007. In contrast, Uttarakhand shows a decline in rates since 2007. 83 Total CPR for India in NFHS 1 was 40.7% and absolute contribution of Female Sterilization was 27.4%. This amounts to 67%. In NFHS 2, the contribution went up to 71% and has declined to 66% in NFHS 3. The details are presented in Annex XI. 84 ibid 1.3% 1.4% 1.2% 1.2% 1.3% 1.0% 0.4% 0.7% 0.5% 0.0% 0.2% 0.4% 0.6% 0.8% 1.0% 1.2% 1.4% 1.6% 1.8% DLHS 1 (1998-1999) DLHS 2 (2002-2004) DLHS 3 (2007-2008) IUD Prevalence Rate District Level Trends in IUD Prevalence Rate in UP High and Low Intervention Districts and Bihar UP High Intervention UP Low Intervention Bihar 46 Figure 24: Trends in Female Sterilization Pattern in Large EAG States in India Figure 25: Trends in Female Sterilization Pattern in Small EAG States in India 47 At the district level the patterns of female sterilization are very different with Bihar having the highest prevalence in 1998/99 (19.3%, 95% CI [18.8, 29.6%]), with a slight decline in 2002 and a rebound in 2007/8 to levels of 27%, 95% CI [26.6, 27.5%]. In contrast, the rate of female sterilization was the same between UP high-intensity and low-intensity districts at baseline, with no change in 2002 and a significant increase in 2007-8 favoring the low-intensity districts. See Figure 26 below. Figure 26: Trends in Female Sterilization Rates in Uttar Pradesh High and Low Intervention Districts and Bihar 48 Trends in Unmet Need for Contraception As discussed in the methodology limitations, varying definitions makes comparisons a challenge, so only four data points are used for this analysis: (NFHS 1, NFHS 2, NFHS 3 and DLHS 3 (definition II)).85 Like the other large EAG states, unmet need in UP declined from 1992 to 2005 (IFPS I) and shows an increase in unmet need from 2005 till 2007/8. Consistent with CPR prevalence rates, Rajasthan remains an outlier with a continuing decrease in unmet need. See Figure 27 below. 85 DLHS 1 and 2 and AHS use different definitions for unmet need and therefore could not be compared to the NHFS data. Figure 27: Trends in Unmet Need for Family Planning in Large EAG States in India 49 For the smaller EAG states, similar trends and are presented in Figure 28 below. The rate of decline for unmet need for Uttarakhand during the period of 1998/99-2005/6 was higher than the reference value for India; however, it was still not significantly different due to the upsurge after 2006. The unmet need for Jharkhand was the exception in this comparison showing an increasing trend during the 1998/99 to 2005/6 and thereafter showing a decreasing trend. Figure 28: Trends in Unmet Need for Family Planning in Small EAG States in India 50 Trends in Coverage of TT for Pregnant Women The coverage of TT, an important indicator of antenatal care (ANC), shows wide variation across the large and small EAG states. Explanation of the large swings seen in states like Bihar is beyond the scope of this evaluation. It's worth noting that in 2002/4 when Bihar experienced a significant drop in coverage, UP under IFPS/SIFPSA remained stable. Post 2005/6, coverage trends are increasing. Figure 29: Trends in TT Coverage of Pregnant Women in Large EAG States in India 51 Figure 30: Trends in TT Coverage of Women in Small EAG States in India Discussion of Findings One of the objectives of this evaluation was to analyze the impact of IFPS, with specific reference to other EAG states. The summary of findings (see Annex XIII) shows that from 1992-2010, indicators including TFR, CPR, use of modern methods (condoms, OCP, IUD, sterilization) in UP and other EAG states have made steady progress. Given the extent of the limitations86 presented earlier, the challenge is to attribute this progress to the work of IFPS I, II and III. In this discussion, the evaluation team aims to present the facts and observations, thereby acknowledging the participation of all players. Throughout the life of IFPS, specifically from 1998-2008 TFR declined significantly in the three IFPS states, UP, JH, UK, although there was no significant difference in TFR due to IFPS interventions at the district level. The findings show decreasing TFR and increasing CPR trends across India and in all EAG states. UP's performance on both indicators has kept pace with the overall trends in India and exceeded Bihar's performance on TFR, closing the difference in TFR that existed with other EAG states in 1992 when IFPS began. 86 Limitations are specifically the limited comparability of the National data sets and the absence of "control districts" without interventions 52 During IFPS I (1992-2004), CPR in UP increased, with a significant difference noted in the high￾intensity intervention districts. Referred to by many respondents as the "SIFPSA Golden Era", the years 1998-2004 showed CPR increasing at a faster rate than in other EAG states. In 2005, as IFPS shifted away from direct service delivery interventions to focus more on providing technical assistance to the State and National government, a downward CPR trend is noted. The introduction of NRHM in 2005, using many innovations that were piloted under IFPS, was met with several challenges, including supply chain and manpower issues. These challenges potentially slowed the CPR trends in UP as well as in other states.87 A number of respondents acknowledged SIFPSA's role in facilitating the NRHM roll out, working with the National Government and sharing IFPS innovations including district action plans, village level workers, quality assurance programs, etc. During this NRHM roll out, only Rajasthan appears to have maintained its increasing CPR. By 2009 the CPR trends for all EAG states were once again on an upward trajectory, which many respondents credited to the maturity of NRHM and the RCH II program. Progress in UK CPR has been especially strong, attributed by many to strong state government leadership and political will. Because there were a number of donors and projects working on RH issues at the same time as IFPS, attribution for these changes cannot be made to any one program. Looking at "unmet need" across all EAG states, progress is noted from 1992-2005, but the post￾2005 rise in "unmet need" is of concern. During IFPS evaluation field visits, the team was impressed by the success of institutional delivery initiatives (most notably JSY). The evaluation team observed crowded hospital delivery rooms, over extended medical personnel, and many ASHAs accompanying women for delivery. With increased demand for institutional delivery, only limited staff time and manpower is available to provide family planning information and services, resulting in a potential opportunity to combine quality, comprehensive FP services with institutional deliveries.88 The greatest impact of IFPS can be seen in the increased use of modern contraceptive methods with a significant difference in UP high-intensity intervention districts by 2007/8 (IFPS II). At the same time, it's important to note that comparable districts in Bihar kept pace with this trend. One factor contributing to this significant increase in the 28 (33 after re-districting) UP high￾intensity intervention districts may well have been the intensive interventions under IFPS I including: trained village level health workers, district action plans that mobilized all constituents from villagers to politicians, infrastructure enhancements to primary care centers and hospitals, and intensive training of medical, laboratory and pharmacy personnel. This high intensity intervention package was further enhanced by state level BCC and strong contraceptive social 87 Personal communications from SIFPSA staff members, UP, UK and JH government officials to IFPS evaluation team members 88 The IFPS Post Partum IUCD intervention is currently attempting to take advantage of the increase in hospital deliveries by inserting IUDs immediately postpartum. 53 marketing (CSM) interventions, which were received by the remaining 42 UP low-intensity districts. Improvements in method mix are noted in 2007/8 (IFPS II), where the same UP high-intensity intervention districts showed a significant difference in condom use.89 By 2010 the use of condoms in UP (10%) surpassed Bihar (1.6%), and kept pace with Rajasthan (9.3%). Of the EAG states, Uttarakhand demonstrated among the highest rates of condom use (15.9% in 2010). While part of this increased condom use may well be attributed to IFPS, the role of the National AIDS Control Program (NACP) should also be considered. The use of OCP in the IFPS intervention states shows no significant increase, nor was there a significant increase in Bihar, with both UP and Bihar having rates under 2%. This is in contrast to India overall and to Odisha which increased OCP use from 0.9% in 1992 to 11.1% in 2010. IUD usage was highly variable across all EAG states with the rates for UP(1%) and Bihar (0.5%) similar at baseline (1992) to AHS results in 2010 thereby showing little progress. The trends in female sterilization help to explain why Bihar's rate for usage of modern methods has kept pace with UP. Here one sees that while the use of female sterilization has risen slowly in UP from 11.7 (1992) to 17.5 (2010), in Bihar the trend has increased from 17.3 (1992) to 29.4 (2010). A significant difference in female sterilization was noted in UP high-intensity and low￾intensity intervention districts indicating that the focus on modern spacing methods and the availability of a broader basket of contraceptive choice may have contributed to the lower rates of female sterilization in these UP high-intensity intervention districts. IFPS Phase Assessment IFPS Phase 1 (1992-2004) was characterized by intensive interventions90 in 28 (33) districts of UP and demonstrated a greater rate of CPR increase (19.8-43.6) than other EAG states, a decreasing TFR (4.8-3.8) and significantly increased use of modern methods in the high-intensity intervention districts. Improvements in the use of condoms and OCPs are noted while the trend in female sterilization remained essentially stable. 89 Condom use has been a very important area of focus for the National AIDS Control Program, and therefore attribution to IFPS or any family planning program is difficult. 90 Please refer to Table 2: IFPS Interventions at a Glance 54 IFPS II (2005-2008) added UK (former UP districts) and JH and focused primarily on technical assistance, especially for the creation of PPP and other innovative pilot strategies. Assisting the National Government with the roll-out of NHRM was another important IFPS mandate including the scale-up of IFPS innovations such as district action plans, ASHA, quality assurance programs, etc. During this period, indicators across the EAG states made relatively little progress in most instances. IFPS III (2009-2012) continued the work of IFPS II, with increased focus on the documentation and dissemination of PPP models. Based on AHS data it would appear that progress has been made on the majority of indicators, but field respondents offered little credit to IFPS. Strategies, Operational Contributions and Lessons Learned (Evaluation Questions #2 & #3) What are the key lessons learned from IFPS, including the strategies it introduced such as performance-based financing systems, district action planning, working with the private sector, and establishment of entities like SIFSPA? What are IFPS’s most significant operational contributions to the field of RH/FP? During the IFPS evaluation team’s fieldwork, a number of strategies were identified and observed at state and district government hospitals, health centers, training institutions, NGOs and private medical facilities. Table 5 below provides a framework for identifying the most significant IFPS strategies, the innovative operational contributions developed and the specific lessons learned from each. 55 91 Referred to as the “Golden Era” by respondents because of the large number of interventions and strong leadership of SIFPSA. IFPS STRATEGIES, OPERATIONAL CONTRIBUTIONS & LESSONS LEARNED STRATEGY OPERATIONAL CONTRIBUTIONS/ YEAR INITIATED LESSONS LEARNED Innovative Organizational Model to Support Government Health Programs State Innovations in Family Planning Services Agency (SIFPSA) 1993  During IFPS Phase I, SIFPSA made FP a higher priority in UP  Strong, stable leadership enhances organizational effectiveness (e.g., Johri-Deepak, “Golden Era” 91 1998-2004)  The management structure of an organization has an impact on program integration, as demonstrated by the division within SIFPSA between “private” and “public” programming  Complete “buy-in” from the state government is essential for success  To be effective high level technical competence is required  This type of organization could serve as an appropriate mechanism for channeling external funding, e.g. from other non-governmental donors  SIFPSA was a pioneering concept in 1992, but today a new innovative model for effective collaboration with GOI would be required Results Oriented Financing Performance Based Disbursements (Benchmarks) 1993  Offers a transparent mechanism for financing project activities  Benchmarking process is highly subjective and labor intensive, but created a shared commitment for implementation  Many challenges in creation of measurable indicators, resulting in more process than output level indicators being used  Has potential for use with accurate alignment (of costs/ results) and should be evidence based  Has potential to include incentives for achieving targets Decentralized Participatory Planning District Action Plans 1998  “Bottom up planning” and involving local communities (Panchayat Raj) and health workers results in a greater buy-in for intervention planning  Design and implementation requires capacity building at the grassroots level and ongoing supportive mentoring for success  Requires a significant time investment for quality process and should only be “scaled-up” in a phased manner  For real effectiveness the actual operational process needs to be followed and implemented in each district, not merely copied from one district to another  DAPs look great on paper but may face implementation constraints and challenges (money, political and administrative will, human resource restraints, etc.) 56 Table 5: IFPS Strategies, Operational Contributions and Lessons 92 Informants and documentation attributed this to JHPIEGO. Comments on the draft report from the ITAP team attribute this to Engender Health. IFPS STRATEGIES, OPERATIONAL CONTRIBUTIONS & LESSONS LEARNED (CONT.) STRATEGY OPERATIONAL CONTRIBUTIONS/ YEAR INITIATED LESSONS LEARNED Quality Assurance Standard-Based Management Recognition (SBMR) JHPIEGO/Engender Health92  Health Systems Strengthening comprehensive approach to QA (including needs assessments, trainings, facilities upgrading, accreditation etc.)  District Quality Assurance Committee and Quality Manager can work with effective leadership  Implementation requires on-site, qualified technical support Community Mobilization Community Based Workers (CBW), Community Health Volunteers (CHV), Accredited Social Health Activist (ASHA), Urban Social Health Activists (USHA) 1998  Incentives to voluntary workers may create challenges in the long run  Community facilitators have been able to convert - awareness and knowledge into action  A peer-based approach enables socially sanctioned delivery of PH/BCC information  Community leadership (Panchayat Raj - Pradhan) buy-in is important for increased use of FP/RH services  Role clarity, including remuneration between multiple community-based mobilizers (e.g. ASHA/USHA/CHV/others) will be important for success Demand Generation BCC 1995  BCC has created an increased demand for MCH services (e.g. through radio dramas and TV serials) - especially evident with Tetanus Toxoid campaign  Personalized IEC has had positive effect on uptake of PPIUCD (e.g. counselor for PPIUCD in hospitals)  Mid media campaigns where messages are integrated into local cultural events have potential and were especially well utilized in IFPS I. (e.g. folk media activities, sensitization and involvement of religious, political and community leaders)  IPC - through peer-led approach and coupled with action has led to an increased uptake of services (e.g., increased institutional deliveries due to ASHAs, increased knowledge of adolescents through UDAAN) Leveraging Private Resources for Sustainable Public Health Services Public Private Partnerships 1998  Requires a clear definition of who is “private” and what constitutes “partnership” and “sustainability” that can be evaluated within an acceptable PPP evaluation framework 57 It is worth noting that since beginning in 2005, NRHM has adopted a number of IFPS operational contributions, including district action plans, quality assurance, BCC and the use of community health volunteers (called ASHA since 2005). The following section will highlight the use of PPP as a mechanism for delivering quality RH services. IFPS Public Private Partnership (PPP) Performance (Evaluation Question #4) What performance or impact related conclusions can be made regarding specific innovative sub￾projects, such as the PPP for ASHA support system, Networks – Voucher Schemes, Merrygold, Social Franchisee services, and what factors most contributed to their relative success or failure? Background of PPP The first official PPP in India occurred in the late 1960s, when the GOI engaged organizations like Hindustan Lever, Union Carbide, ITC Ltd.,93 Lipton, Brooke Bond and Dey’s Medical to distribute condoms under the national Contraceptive Social Marketing Program (CSMP). Realizing that addressing India’s healthcare challenges required the resources and expertise of both the public and private sectors, in 2002 India formally included PPP as a key strategy of the National Health Policy and 10th Five Year Plan.94 These PPP are being viewed as social experiments that can supplement state-run health services by engaging private stakeholders.95 Under IFPS II and III, the use of innovative PPP was envisioned as an important strategy to expand the use and availability of contraceptive information, commodities and services.96 SIFPSA (in 2002)97 and the USAID-funded POLICY Project (in 2004)98 took the lead in assisting the national government to draft a PPP strategy that was included in the overall RCH program. 93 ITC, originally an acronym for Imperial Tobacco Company, later changed to Indian Tobacco Company, is a leading Indian multinational company with a diverse range of products and services including cigarettes, toiletries, cosmetics, snacks, confectionaries, apparel, hotels and papers. 94 “The Emerging Role of PPP in Indian Healthcare Sector,” CII in association with KPMG, pp. 8–9. 95 Taneja, Udita, Bharti Birla Research Scholar. “Public Private Partnerships for Healthcare Delivery in India,” The Internet Journal of World Health and Societal Politics, ISSN:1540-269x 96 The GOI’s Report (2006–2007) by The Task Force on Medical Education for the NRHM states that the private sector provides 58% of the hospitals, 29% of the beds in the hospitals and 81% of the doctors. Nearly 78% of the rural and 81% of the urban population is provided medical treatment by private healthcare players. Also, according to the National Sample Survey Organisation (NSSO) 60th Report (2004), use of public healthcare is lowest in the rural areas of the states of Bihar (up to 89% in urban and 95% in rural areas) and Uttar Pradesh. ibid 97 20 Years of the Innovations in Family Planning Services Project in Uttar Pradesh, India, p. 38. Contraceptive Social Marketing Program (CSMP) Goal To increase sales of condoms and oral contraceptive pills in rural Uttar Pradesh through social marketing efforts, especially in villages with a population size of 1,000 to 4,999. 58 The strategy defined PPPs, identified mutual benefits for the private sector and the government, listed different PPP models and services that could be brought under the program, and provided guidelines for capacity building to manage PPPs.99 In IFPS II and III (2005–2012), SIFPSA piloted several PPP interventions in UP: (1) CSMP in villages with a population of 1,000–4,999 (C and D villages, with populations less than 5000); (2) Merrygold Health Network (MGHN); (3) Sambhav Vouchers; (4) Mobile Medical Units (MMU)—mobile vans; (5) ASHA Plus; and (6) “Understanding, Delivering and Addressing Adolescent Needs” (UDAAN). Each intervention involved private￾sector partners in the delivery of FP and RH information, commodities and services. Framework for Assessment PPP interventions, with sustainability as a key requirement, may vary in terms of their goals, with some offering financial viability and others offering sustainable health outcomes.100 To assess the IFPS PPP interventions, the IFPS evaluation team undertook reviews of published and unpublished documents, as well as a rapid assessment of interventions in the field using a framework proposed by the Confederation of Indian Industries (CII) to the GOI for PPP initiatives in health. The 98 Review of draft report, comments from ITAP team 99 20 Years of the Innovations in Family Planning Services Project in Uttar Pradesh, India, p 38 100 The IFPS evaluation team felt that the IFPS PPP definition of “sustainability” was also vague, with adoption by the GOI of an NGO￾led initiative considered to be a successful PPP in light of the government's prior reluctance to collaborate with the NGO/private sector. Traditional PPP evaluation frameworks offer little scope for including this as a criteria for success and tend to focus more on fiscal, rather than the behavioural, aspects. Box 4: PPP Framework for Assessment PPP Key Success Factors Efficiency Effectiveness Equity Financial Sustainability 59 framework includes four primary areas to assess the success or failure of a PPP model. 1. Effectiveness, i.e., the appropriateness and ability of the program to meet the objectives it was originally set out to achieve. An important element of this assessment is clarity of the objectives laid down by the program and ability to measure the result through identified and measurable outcomes. It also takes into account the potential scalability and local buy-in for the program. 2. Efficiency, or the cost efficiency of the program in achieving its objectives. This evaluates the financial consequences to the public sector vis-à-vis risk transfer achieved, including government willingness to partner with the private sector as well as the extent of reforms and policy changes required to make such partnerships work. 3. Equity relates to evaluating whether the benefits of the program accrue to the government and political priorities—i.e., low-income, below-the-poverty line (BPL), targeted sections of the society—and does not subsidize the cost of service provision to wealthier sections of the country. 4. Financial Sustainability deals with the financial viability of the program, including investment potential and attractiveness to the private sector in delivery of the program. Assessment of IFPS PPP The IFPS evaluation team assessment of the various PPP initiatives is subjective in nature and intended to serve as the basis for broader discussions with all partners. Based on the above framework, two of the IFPS PPP initiatives, ASHA Plus (linking ASHAs to an NGO and facilitators for increased support and improved effectiveness) and UDAAN (life skills training for adolescents through peer learning) are implemented by NGOs and have limited potential for financial sustainability. Therefore, both fall outside the scope of official PPP models. A number of respondents reported that SIFPSA’s organizational structure, with a complete separation between the private and public health sectors, allowed for little integration between the two organizational divisions. Therefore, SIFSPA considered that working with any type of NGO, (as per the ASHA and UDAAN interventions) was a private sector initiative, irrespective of whether it fit an acknowledged PPP model. For the purpose of this evaluation, to best utilize the proposed framework, the assessment will focus on the four PPP initiatives that met the criteria for a public-private partnership: (1) CSMP; (2) Merrygold Health Network; (3) Sambhav Vouchers; and (4) MMUs (mobile health vans).101 It is important to note that, although introduced 10 years ago, formal approval of this assessment framework is still pending with the GOI. The absence of an approved PPP regulatory 101 Similar sentiments are expressed in the previously mentioned, “Public Private Partnerships for Healthcare Delivery in India” 60 framework limits the clarity, objectivity and consistency of PPPs in the Indian healthcare sector. (See Annex VII for Assessment of PPP Public and Private Sector Roles). (1) Contraceptive Social Marketing Program (CSMP) Background: Enthused by the success of urban CSMP interventions during IFPS I, SIFPSA supported contraceptive social marketing in 2002 in UP to expand access to contraceptives, including oral contraceptive pills and condoms, to rural areas, especially in C and D villages (population less than 5000).102 Simultaneously, the Venture Rural Project in Moradabad Division piloted the concept of a “rural salesman,” offering a basket of products, e.g., branded OCP, condoms and other products, including disposable delivery kits, oral rehydration salts, and iron and folic acid tablets. During 2003 and 2006, in addition to the statewide CSMP project, which continued until 2011, additional regional projects were implemented by private social marketing agencies with the aim of saturating the state. It is also important to note that the National Aids Control Program’s third phase (2007–2012) made significant efforts to promote condom use through free or discounted supplies and social marketing.103 Results: During IFPS II, CSMP, through various activities, reached out to an estimated 1.5 million men, 700,000 women, over 20,000 Anganwadi workers (AWW), 30,000 retailers and 6,000 gram panchayat104 members. In UP, the condom market more than doubled from 2005 to 2011 and contribution of the rural market to total sales increased to 64 percent. The presence of both condoms and OCP in at least one outlet in small villages (population between 1,000 and 5,000) increased from 41 percent in 2006 to 70 percent in 2011.105 102 IFPS Phase I CSMP was implemented by Hindustan Latex Family Planning Promotion Trust (HLFPPT), one of the most significant players in leading India’s PPP strategies including CSMP, Merrygold Health Network, Sambhav Vouchers and mobile vans. HLFPPT is officially an NGO subsidiary of Hindustan Latex Limited (HLL) a large, public sector enterprise and therefore not truly a private sector organization, as delineated in the guidelines. 103 More information on NACP-III can be found here: www.nacoonline.org/ 104 Gram Panchayat are local self governments at the village or small town level in India. 105 Ibid 98 p 41-42 CSMP SALES UTTAR PRADESH 2005 & 2011 (in million units) URBAN RURAL 2005 2011 2005 2011 Condoms 213 439 124.84 257.51 Oral Pills 8.49 12.54 3.99 6.89 Table 6: CSMP Sales in UP Figure 31: Basket of contraceptives 61 Assessment: During field visits, the IFPS evaluation team visited multiple shops and observed a range of contraceptive choices. The biggest seller was reported to be “emergency contraception,” which is not tracked in the SIFPSA management information system (MIS).106 The low per-unit margin on condoms and oral contraceptives was noted as a challenge, but high sales volume and the revenue potential offset such challenges. Currently, the financial attractiveness of CSMP to private partners has been adversely affected by government schemes that incentivize institutional delivery and permanent FP methods. Eventually, with a willing private sector, the potential market is large enough. Especially with effective and sustained demand-generation efforts, third-party revenues could be sufficiently sustainable to merit private investment. Conclusions at a Glance: High-Medium Medium Medium Medium High-Medium Positives Negatives  Over 50 years’ experience  Large market, low unit price and strong commercial pharmaceutical and non-pharmaceutical trading channels  Public and Private sector buy-in exists  Inadequate demand in smaller rural areas  Strong incentive for institutional delivery as well as long acting and permanent methods (2) Merrygold Health Network (MGHN): Background: Merrygold Health Network, a flagship PPP intervention of IFPS II and III, aims to create access to quality, low-cost maternal and child health (MCH) services by networking with private health service providers as franchisees. Supported by a multi-media communication strategy promoting internal and external branding of facilities that promised consumers a consistent, quality experience, MGHN was designed to cater to lower-middle￾and working-class residents of UP urban and peri-urban areas with limited access and capacity to pay. The Merrygold Health Network is a hub and spoke design. Level 1 franchisees (Merrygold Hospitals) established at 106 Emergency contraception was not part of the IFPS expanded basket of contraceptives. Effectiveness Efficiency Equity Financial Sustainability Overall Merrygold Health Network (MGHN) Goals To increase access to equitable, affordable and quality healthcare services for low income groups and the working class by engaging the private sector through sustainable partnerships and developing a network of franchised hospitals. Box 5: MGHN Goals 62 the district level serve as the hub connected to Level 2 fractional franchisees (Merrysilver Clinics), established at the subdivision and block level.107 Level 3 (Merrytarang) includes providers like auxiliary nurse midwives (ANMs), ASHA workers and AYUSH practitioners and acts as the first point of contact with the community.108 Level 3 is also designed to provide referral support to Merrysilver and Merrygold hospitals. The network’s emphasis is on affordable pricing, quality assurance, customer servicing and efficient service delivery through standardized operating protocols.109 Results: Begun in October 2007, by February 2012 the Network included 67 Merrygolds, 367 Merrysilvers and nearly 10,000 Merrytarang spread across 36 districts of UP. These facilities reportedly provided 756,100 antenatal check-ups, 133,900 deliveries, 10,600 sterilizations and 38,200 IUCD insertions, together with more than one million couple-years of protection through distribution of condoms and oral contraceptive pills.110 According to a 2009 Nielsen study, client satisfaction with the network was high.111 Thirty-eight percent of the women rated that they were ‘very satisfied’ and 53 percent as ‘satisfied’ (N=66) with services received at the network.112 Of the women familiar with MGHN, 65 percent (N=474) reported they would visit the facilities again (presumably for a future birth) and 90 percent would recommend the network to a friend. Similarly medical audits conducted from September 2009 to August 2011 indicate that the facilities improved their quality of services and met more than 80% of the criteria. According to the franchisor Hindustan Latex Family Planning Promotion Trust (HLFPPT), “[S]upport to the model for another few years with a focus on revenue generation, while meeting its social objectives, will enable a sustainable network of health facilities in rural UP.” 113 Assessment: While the strengths of social franchising, including those of MGHN, are well documented,114 IFPS evaluation team field visits revealed critical gaps in implementation of the model, which if not addressed on a priority basis could affect its sustainability as a PPP model. The gaps identified include: 107 The administrative divisions of India compose a nested hierarchy of country subdivisions. Indian states and territories frequently use different local titles for the same level of subdivision (e.g., the mandals of Andhra Pradesh correspond to tehsils of Uttar Pradesh and other Hindi-speaking states and taluka of Gujarat and Maharashtra). The smaller subdivisions (villages and blocks) exist only in rural areas. In urban areas, these rural subdivisions are replaced by Urban Local Bodies. 108 Department of Indian Systems of Medicine and Homoeopathy (ISM&H) was created in March, 1995 and renamed the Department of Ayurveda, Yoga & Naturopathy, Unani, Siddha and Homoeopathy (AYUSH) in November, 2003 with a view to providing focused attention on the development of education and research in Ayurveda, yoga and naturopathy, Unani, Siddha and homoeopathy systems. 109 http://healthmarketinnovations.org/program/merrygold-health-network accessed November 14, 2012 110 “Social Franchising as a Public-Private Partnership Model”, MGHN MIS, February 2012 p. xi 111 Social Franchising as a Public-Private Partnership Model, p - xi 112 ibid 113 Ibid 111 114 ibid 63  MGHN’s targeting strategy lacked focus and objectivity o 12 Merrygold L1 units are based in Kanpur City, whereas there is only one Merrygold L1 in Allahabad district. o Primary focus was on obstetric care and NOT on FP, indicating that commercial considerations may be superseding social objectives.115  MGHN lacked strong marketing skills and strategy o MGHN’s fee structure and its lack of revision over time restricted clinic income, causing financial losses for the franchisees. o Visibility of MGHN over mass media and mid media was found to be low. o Absence of a clear strategy and plan for the effective use of the Merrytarang as key referral agents (a backbone of the model), has adversely affected client mobilization.  MGHN lacked “buy-in” among local government administrators and government health officials in general o MGHN has not adequately addressed the issue of lack of trust and credibility between the Indian public and private sectors, in particular the government's perception of the ability of private providers to deliver health impact at scale among the poor. o The government's decision (in 2008) to introduce JSY (encouraging institutional delivery) in urban areas (as well as rural) has affected MGHN’s ability to serve the poor/BPL population, because many women are choosing to have babies in public facilities (thereby earning Rs 1,400), rather than seeking FP services in a private facility and paying for that service.  MGHN lacked franchisee motivation o Lower than anticipated client load, especially in areas such as Kanpur City, where many private providers already existed; establishing 12 Merrygold facilities there is of questionable value. High competition among providers has resulted in a loss of income. o Lack of communication and information sharing between the franchisor and franchisees. o Lack of guidance and mentoring by the franchisor. After six years, the IFPS Merrygold investment totals $6.11 million.116 The above factors lead the IFPS evaluators to question the commercial sustainability of this enterprise as implemented. In fact, given the success in India of other private healthcare provider networks (e.g. Apollo Clinics) one wonders why this type of initiative should be managed by a public-sector undertaking. 115 The hospital generates more money for obstetric services than for FP services 116 Ibid 111 64 Conclusions at a glance: 3) Sambhav Vouchers Scheme Background: From 2006–2012, SIFPSA worked with the government of UP to extend Sambhav vouchers to BPL populations, thereby offering the poor a choice of private providers and access to quality, private FP/RCH services. Piloted in four districts of UP, the vouchers covered a range of services, including ANC, institutional delivery, postnatal care (PNC), neonatal care, and FP.117 Voucher management units (VMUs), headed by the district chief medical officers (CMO), coordinated with local NGOs (working with ASHA and CHV) to: (1) identify BPL households; (2) offer information on available services; (3) distribute the vouchers; and (4) accompany clients to seek services. Only accredited private facilities participated in the program. Results: Within a period of one to two years, the vouchers facilitated the delivery of nearly 12,500 babies in private health facilities, promoted maternal health by supporting approximately 44,000 ANC visits and 10,300 PNC visits, and approximately 9,500 vouchers were redeemed for a range of FP methods by women and men.118 Assessment: In UP and UK, different reimbursement schemes directly impacted the schemes effectiveness. 117 Four UP districts included Agra (seven blocks), Kanpur Nagar (368 urban slums), Haridwar (two blocks), and Gumla (two blocks). 118 Sambhav: Vouchers Make High-Quality Reproductive Health Services Possible for India’s Poor, p-xiv Effectiveness Efficiency Equity Financial Sustainability Overall Medium Medium Medium Low Medium Positives Negatives  Opportunity for private providers to serve the BPL population  Financially feasible reimbursement structure in Uttarakhand  Opportunity to target BPL with needed RH/FP services  Lack of government will to partner with the private sector  Financially non-feasible reimbursement structure for providers and CHVs in UP  Too much paper work required of the private providers  Coexistence with JSY Scheme offering financial incentives to CHV/ASHA and women for institutional deliveries impacts on use of FP services Sambhav Voucher Goal To provide affordable, accessible, and high-quality FP/RH services through accreditation of private facilities and to empower BPL families to choose and access a provider through a voucher distribution system. Box 6: Sambhav Voucher Goal 65 In UP, with the assistance of SIFPSA, the government expanded Sambhav to 1,562 slums in five cities, but the IFPS evaluation team found that a number of private providers no longer wanted to continue, due to the limited reimbursements that, in most instances, did not cover their costs.119 The Government of Uttarakhand, assisted by ITAP (2006– 2012), made all BPL cardholders, regardless of their district of residence, eligible under Sambhav for services provided by accredited private nursing homes in five districts.120 However, reported misuse of vouchers by the private sector, the challenge of going to scale in partnership with private doctors and facilities,121 and the lack of support from the GOI122 led the government of Uttarakhand to discontinue the Sambhav Vouchers and reverted to its own infrastructure and schemes to serve BPL consumers. A number of accredited private providers in UK expressed unhappiness over the abrupt withdrawal of the scheme. Given the severe shortage of medical staff in public-sector facilities, the IFPS evaluation team questions this decision. In both UP and UK, JSY was seen as a threat to the Sambhav Vouchers, especially its use by the urban poor to access private FP services, when delivering a baby in an institution provides a larger financial return. Conclusions at a Glance: 119 Five UP cities were Kanpur, Agra, Varanasi, Allahabad, and Lucknow. The IFPS evaluation team visited facilities in Lucknow, Kanpur and Allahabad 120 Five UK districts were Almora, Dehradun, Haridwar, Nainital, and Udham Singh Nagar 121 As reported by the key informants of the government of UK 122 As reported in the “ITAP Team comments to the IFPS draft review. “ SAMBHAV REIMBURSEMENTS Payments to UP UK Provider for Delivery Rs 1,850 Rs 3,500 Provider for C-Section Rs 1,850 Rs 7,500 CHV/ASHA (motivation) Rs 350 Rs 600 Table 7: Sambhav Reimbursements Effectiveness Efficiency Equity Financial Sustainability Overall Medium Medium Medium Low Medium Positives Negatives  Opportunity for private providers to serve the BPL population  Financially feasible reimbursement structure in Uttarakhand  Opportunity to target BPL with needed RH/FP services  Lack of government will to partner with the private sector  Financially non-feasible reimbursement structure for providers and CHVs in UP  Too much paper work required of the private providers  Coexistence with JSY Scheme offering financial incentives to CHV/ASHA and women for institutional deliveries impacts on use of FP services 66 4) Mobile Health Van (MHV) Operation Background: In 2006, IFPS/ITAP supported a pilot MHV model in Ramnagar block of Nainital District, UK. To address the lack of government health services in the remote hilly areas, the MHV model, implemented by a local NGO, adopted a comprehensive, three-pronged approach to provide diagnosis, treatment and follow-up services. The team traveling with the van included a project coordinator, medical officer (MO), lady medical officer, radiologist, electrocardiograph technician, pharmacist, lab-technician, health coordinator, lady health worker, utility worker and two drivers. RH/FP services were delivered through a regular series of camps that rotated through eight locations around the district, on a fixed-date, fixed-route schedule. Results: Since 2009, with funding from NRHM, positive elements of the Ramnagar MHV model were scaled up throughout Uttarakhand. In 2012 a total of 30 MHVs with 140 qualified staff operate at least two vans in each of the 13 UK districts, working in close collaboration with local communities and ASHAs along the routes.123 In 2007, IFPS/ITAP also provided support for Jharkhand’s Mobile Medical Units (MMU) program, in the form of an assessment of the MMU operation in Jharkhand, and suggested changes as per the Uttarakhand model. Currently in Jharkhand, 103 MMU are operating in difficult-to-reach and insurgency-hit areas of all 24 districts.124 Assessment: In UK, the IFPS evaluation team met with the NGO Project Coordinator operating the Arogya Van, a government-owned, mobile diagnostic laboratory with the capability to treat common ailments.125 The project coordinator reported serving 600–700 clients per month (80– 85% BPL) in the remote hilly terrain of Uttarakhand. This van collaborates with another van, operated by Jain Videos, to offer FP/RH services. Coordination is performed by the district CMO, with effective collaboration. Given the severe shortage of medical staff, especially in the upper Himalayan region, the mobile vans offer an effective, efficient and sustainable strategy to serve the underserved, BPL population in remote areas of UK. It offers a high potential to attract 123 Reaching Underserved Communities through Mobile Health Vans in Uttarakhand, India p. xii 124 ibid 125 NGO is The Society of People for Development Mobile Health Van Goal To increase accessibility of preventive, diagnostic and curative services, with an emphasis on RH services to the rural poor, especially in remote, hard to reach and poorly served areas. Box 7: Mobile Health Van Goal Figure 32: Mobile Health Van 67 private sector investors interested in corporate social responsibility (CSR) activities that serve the underserved. To improve its sustainability, the possibility of generating revenues from those not holding BPL cards could also be explored. Conclusions at a Glance: Conclusions for PPP The piloting, testing and documenting of innovative PPPs has been a primary focus of IFPS II and III. Greater clarity as to the definition, implementation and evaluation of these types of initiatives is still required.126 CSMP, the oldest and most successful PPP model, has been successful in expanding the availability of contraceptives in rural UP. Other models, such as the Sambhav Vouchers, while effective in some states (e.g., UP), have been discontinued by the government in UK because they prefer to utilize their resources to support government institutions, rather than private facilities. The distrust of the private sector among government officials remains a powerful barrier to the long-term success of PPP. Other initiatives, like the Merrygold Health Network Franchise, require greater attention to meeting the needs of the franchisees and to evaluating the network’s impact beyond process indicators, such as numbers served, instead ascertaining if they are actually providing quality, affordable care to the appropriate target audience in a sustainable fashion. 126 Taneja, Udita, Bharti Birla Research Scholar. "Public Private Partnerships for Healthcare Delivery in India" The Internet Journal of World Health and Societal Politics ISSN:1540-269x Effectiveness Efficiency Equity Financial Sustainability Overall High High High Low Medium Positives Negatives  Ability to serve the underserved, BPL population in remote areas  Cost effective compared to total cost of hiring/retaining full time medical staff and running medical facilities in remote locations with low daily client utilization  Encourages community involvement and local buy-in  Limits nature and quality of services that can be offered  Challenges of coordination between two types of vans - one diagnostic and the other providing services  Low capacity of clients to pay, limits model's potential for financial sustainability 68 ITAP TA EFFECTIVENESS (Evaluation Question #5) How effective has the project’s technical assistance been in promoting technical and program priorities, and improving the capacities of local institutions under NRHM to deliver RH programs in USAID-supported states, and what lessons can be drawn for future designs by GOI and USAID/India? Objectives and Expenditures: The technical and program priorities identified in the IFPS II Technical Assistance Project (ITAP), dated April 4, 2005 included: “[d]evelop, demonstrate, document and leverage working models of public private partnership for improved delivery of integrated reproductive and child health services.” Figure 33: ITAP Spending UP $11.7 42% UK $4.65 16% JH $2.54 9% National $6.8 24% Management $2.49 9% ITAP Spending in Million USD 2005-12 69 Funds were originally obligated from 2005–2008, then extended to cover IFPS III from 2009– 2012. Implemented by The Futures Group, total spending was $28.1 million and was disbursed as shown in Figure 33. 128 Table 8, above, includes illustrative examples of originally proposed ITAP activities and examples of activities implemented from 2005-2012. As documentation of IFPS project activities was an important objective for ITAP, multiple, innovative project activities were documented and published. Using the evaluation methodology described earlier in this report and the data instrument included as Annex III, the evaluation team identified and reviewed 26 relevant documents/reports. The scores of the top ten ranked documents, in descending order, are presented in Table 9. Please see Annex X for more detailed score breakdown. 127 ITAP Contract, 1 April 2005, pg 11 128 The evaluation team made multiple attempts to procure ITAP programmatic funding expenditures but only the above-mentioned spending allocations were available. ILLUSTRATIVE EXAMPLES OF PROPOSED ITAP ACTIVITIES 2005127 EXAMPLES OF ITAP ACTIVITIES (2005-2012) NATIONAL Assistance for newly established Empowered Action Group (EAG) cell within MoHFW  Funding support for technical staff for NIHFW RH/FP cell  Funding Support for MOHFW BCC/ICC technical staff UP, UK, JH  Operations research  Design and development of social franchising models  Design and development of social marketing programs  Design and development of BCC  Assistance to operationalize technical cells and Health Directorate (UK only) SIHFW &FW Directorate (JH only)  RHIS (2005, 2010) and other project surveys  Produced 115 project documents (29 published 86 unpublished)  Mobile Vans program  SAMBHAV voucher program  Merrygold Hospital Franchise scheme  BCC campaigns National and in all three states  Creation of Institute of Public Health (JH only) Table 8: Proposed ITAP Activities and ITAP Activities Implemented 70 Table 9: ITAP Documentation Review - Ranking of Documents Reviewed Report quality varied greatly; the top score, of 46/70, was obtained by an unpublished document. In the top ten, 40% were unpublished and 60% were published. Few documents included information on cost structure or the scalability of the intervention, and many offered only a limited analysis of intervention effectiveness. Where a methodology was included, limitations were rarely mentioned. In most of the publications, the implementation process was clearly documented (geographic coverage, quantitative and qualitative findings, lessons learned), but objective evaluation of what worked, what did not work, and why, lacked scientific rigor. Based on the 14 assessment criteria, all documents achieved a high score of 75.5/130129 for the criterion, “Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading).” The lowest score, of 43.5/130, was obtained for the criterion, “If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness?” This suggests that scalability and cost-effectiveness ranked lowest in the hierarchy of those documents reviewed. Delivery of Technical Assistance: ITAP's technical assistance to PPP initiatives, as discussed in the above findings, demonstrates The Futures Group’s role in establishing guidelines, building 129 A total of 26 documents were reviewed yielding a maximum score of 130 (5x26) for any one of the 14 assessment criteria. DOCUMENT/REPORT YEAR PUBLISHED/UN PUBLISHED SCORE (OUT OF POSSIBLE 70) Reach of Social Marketing Products in Cat C & D - Villages of UP 2011 Unpublished 46 Community Based Workers Improve Health Outcomes in Uttarakhand 2012 Published 43 SAMBHAV: Vouchers Make High-Quality RH Services Available for Poor 2012 Published 40.5 Formulation of Population and Health Policies in Indian States (1997-2004) 2006 Published 39 Promoting Adolescent RH UK and UP 2012 Published 39 Social Audit of Infant and Maternal Deaths in Jharkhand 2010 Published 39 Assessment of Condom/OCP Social Marketing in India 2008 Unpublished 39 Audit of Infant deaths in Uttarakhand 2008 Unpublished 37.5 Rapid Assessment of the Functionality of FRUs and 24x7 PHCs in Uttar Pradesh 2008 Unpublished 37 Health Issues and Health Seeking Behavior of Tribal Populations 2009 Published 36.5 71 human capacity for implementation and strengthening the management capacity of implementing partners, both governments and NGOs. ITAP support to BCC campaigns also proved effective in developing a wide range of programs for mass media, mid-media and IPC/community-level programs at the national level, as well as in the three states (UP, JH, UK). Many of the campaigns focused on demand generation, particularly for the voucher and mobile van programs. It is especially worth noting that, working with private, local advertising agencies, ITAP effectively helped to build local capacity for delivering social communication for health.130 In general, ITAP appeared to lack a comprehensive strategic vision for evaluating programmatic achievements. Opportunities for meaningful operations research were missed in interventions like the ASHA Plus PPP, where including a comparison district in the project design (rather than only in a rapid assessment) would have greatly enhanced the findings of this approach. Given the innovative and pilot nature of many IFPS interventions, a more rigorous operations-research methodology, especially during Phases II and III, should have been a critical component of the technical assistance. Observations by the evaluation team in the field, especially in JH and UK, revealed the limited presence of ITAP/The Futures Group.131 One government official, regarding slow implementation of the vouchers program, noted, “They could have been more effective. They mainly drafted letters, but offered no motivation, no monitoring and limited analysis.” While the official acknowledged that ITAP might have been instructed not to interfere with the government, the technical assistance was described as “very high up” and “rarely reaching down to the field.” 132 130 Some of the advertising agencies working with ITAP included: BEI Confluence for the Vouchers; Subidha CuT, Ogilvy & Mather with SIFPSA; Independent Film Makers, Film Rajendera, Anand Kumar worked with the National level NRHM advertisements. 131 The evaluation team acknowledges that ITAP concluded its work approximately 5 months prior to the evaluation field visits. 132 Personal IFPS evaluation team communication with GOI officials Figure 34: Development Partners in Jharkhand 72 In JH, the presence of multiple USAID-supported initiatives, without coordination and collaboration between them, struck the IFPS evaluators as a missed opportunity for ITAP and USAID to consolidate their ground presence (See Figure 34). Figure 35133 demonstrates limited ITAP spending in Jharkhand during Phase II, which may have contributed to JH’s inability to complete their benchmarks in a timely fashion. These preceding examples stand in direct contrast to the presence of another technical assistance partner, JHPIEGO, whose work in the three states was observed by the evaluation team in JH and UK. Additionally, respondents in these two states expressed their appreciation for their effective technical inputs and for introducing PPIUCD in JH and UK. During the evaluation fieldwork, an important dialogue developed with different government officials regarding the nature of donor technical assistance not directly linked to programmatic funding. Since 2005, GOI/ NRHM has been the source of the majority of implementation funding, with ITAP providing “only technical assistance.” While in some instances, this type of external technical assistance may have been welcomed, it appeared that, in other situations, it may have been viewed as unnecessary and even unwelcome. J.S. Deepak, former SIFPSA executive director (2003–2004), reminded the evaluation team of an important Indian saying that he learned in elementary school: “The chap with the pot of gold is the one who makes the rules.” This is an important consideration when providing technical assistance within the current Indian health context, where external donor funding may not be required or wanted. 133 USAID provided alternative figures for ITAP spending; however, these data do not feature disaggregation by state. The graph represents financial data gathered by the evaluation team during fieldwork. Figure 35: ITAP Spending by States $7.7 $ 3.1 $ 0.9 $ 3.9 $1.6 $1.6 0 2 4 6 8 10 UP UK JK ITAP Spending by States and Phases of IFPS (millions of dollars) IFPS II IFPS III 73 One of the respondents stated that USAID and SIFPSA should advocate for FP in the prevailing focus on maternal and child health for the national program so that the long-term goal of population stabilization for India may be met. Conclusions for TA Effectiveness The role of projects, such as ITAP, that provide technical assistance without “financial strings” in the current India health and economic context, would benefit from a clear set of mutually desired and agreed-upon programmatic objectives and indicators for success. Technical and managerial oversight from USAID also requires the delegation of financial and human resources with the necessary technical and diplomatic skills to serve the Indian government effectively in this challenging capacity. Legacy of IFPS (Evaluation Question #6) What is the enduring legacy of IFPS? How close are we to achieving the original objectives, noting the major milestone events and challenges of IFPS over 20 years? S IFPSA, as an organization, and its vast network of human resources constitutes the enduring legacy of USAID's 20-year investment in IFPS. Using organizational network analysis, SIFPSA was found to have the highest-degree centrality score, representing the most connectedness to other organizations and departments.134 SIFPSA (followed by UP NRHM) maintains a central position in the FP network of UP and plays an important broker role in connecting other organizations to each other.135 The extent of SIFPSA’s relationships with international NGOs provides access to learning and training resources that can translate into state-level capacity enhancement. In addition, the number of people who have been trained by or worked at SIFPSA has created a “SIFPSA alumni” network with strong connections to FP and RH who now are spread, not only throughout India, but around the globe.136 Anyone who has worked in the field of FP in India knows of SIFPSA and acknowledges its innovative role in FP. This branding, coupled with a vast and competent network, is a significant legacy. 134 In network analysis, there are various types of measures of the centrality of a vertex within a graph that determine the relative importance of a vertex within the graph (i.e., how influential a person/organization is within a social network). 135 Note that the SIFPSA Executive Director is simultaneously the NRHM Director. 136 Personal communication with Mr. J. S. Deepak, SIFPSA ED 2003-2004. Interviewed in Delhi October 30, 2012. THE IFPS LEGACY  SIFPSA  “Alumni network “  Skilled NGOs  Focus on quality  History of training & capacity building  Focus on BCC  History of innovations 74 Figure 36: Degree Centrality - National and States State level (UP) National level (N) 75 In UP, where 86.6% of IFPS funds (PBD & TA) were invested, the introduction of NRHM in 2005 was enhanced by the presence of SIFPSA. The creation of District Innovations in Family Planning Services Agencies, implementation of decentralized planning and the regular use of field-based workers, supported by NGOs, were piloted in UP between 1997 and 2001, facilitating the uptake of these NRHM initiatives. The DIFPSAs slowly transformed into the NRHM District Project Management Units (DPMU) and in UP, the 18 divisional PMUs are all headed by SIFPSA-funded staff.137 NGO capacity also has been built by SIFPSA, providing an additional legacy of which IFPS can be very proud. For this reason, donors like the Gates Foundation have been able to identify appropriate local partners for health-related interventions.138 A number of NGOs expressed their increased ability to attract projects and attributed it to their experience working with SIFPSA. Having worked over the years with hundreds of NGOs throughout UP, as well as UK and JH, the SIFPSA network and legacy continues to grow. Focus on quality is another important legacy of SIFPSA. Understanding that spacing children and maintaining health for the mother and child are all about making choices, giving couples different options for contraception and for accessing services, are important factors that health providers need to consider. Building the skills and capacity of providers, from hospitals to villages, is another important legacy of SIFPSA and one that will require continuous upgrades and reinforcement. Finally, the use of BCC for demand generation of all health related services is a critical element in the growing Indian consumer environment. The final evaluation question is, “How close are we to achieving the original project objectives?” Given the changes in objectives - from a focus on family planning to a broader reproductive health mandate, and from providing direct services and technical assistance (TA) to only providing TA – UP’s progress from a CPR of 19.8 to 49.9% in 20 years is a notable achievement.139 UP TFR has also decreased from 4.8 to 3.6, which represents a 25% decline since 1992.140 More women are using modern spacing methods and (except for injectables) most methods are readily accessible in rural as well as urban settings. However, if family planning and safe delivery initiatives do not work in tandem, this progress may be threatened by a declining interest and emphasis on family planning and greater attention to maternal and infant mortality rates (MMR & IMR). 137 The state of UP currently has a total of 75 districts and these districts have been grouped into 18 divisional units, which may contain three to five districts. 138 Personal IFPS evaluation team communication with Rochak Bhardwaj, city manager, Urban Health Initiative, Allahabad 10/16/2012 139 The original goal was for a CPR of 49%. The 50% CPR achieved includes use of all methods (modern, limiting and traditional) for women ages 15–49 years, and is based on AHS published results. 140 Based on NHFS 1 and AHS data. 76 FINAL COMMENTS/CONCLUSIONS Using national survey data sets (NHFS, DLHS, AHS) from 1992-2010 to analyze EAG states and India level trends and to compare the original 28141 UP IFPS high-intensity intervention districts to 42 low-intensity intervention districts the evaluation team concludes:  From 1995-2007 IFPS had a significant impact on the uptake of modern contraceptive methods in the original 28 (33 after redistricting) districts. Other EAG states kept similar pace with this trend and since 2005/6 Rajasthan has outpaced UP in the use of modern methods.  Condom use in IFPS high-intensity intervention districts was significantly greater than in low-intensity intervention districts. On the whole, condom use in UP paralleled trends in Rajasthan but exceeded Bihar condom use throughout the life of IFPS. Attribution for this trend may need to be shared with the National AIDS Control Program (NACP), which has emphasized the use of condoms to stem the transmission of HIV/AIDS.  Other modern methods, including OCP and IUDs, show no significant trends, but female sterilization shows a significant difference and declining use in UP high-intensity intervention districts when compared to Bihar. Based on observations as well as consumer and provider interviews, it appears that in the original high-intensity intervention districts, couples have greater awareness and broader choice and availability of modern spacing methods. In contrast, states like Bihar show sterilization trends increasing and low utilization rates of other modern methods (condoms, IUD, OCP).  In all EAG states TFR continues to decline and CPR has made notable gains from 1992- 2005. The three IFPS intervention states had significant declines in TFR between 1998 and 2008; however there was no significant difference in TFR in any district attributable to the IFPS interventions. In 2005 CPR trends declined (with the exception of Rajasthan) as NRHM was rolled out with a broad reproductive health mandate and an emphasis on meeting Millennium Development Goals (MDGs) related to lowering maternal and infant mortality rates. Since 2007 CPR trends have risen, probably due to the maturing of NRHM, but the concurrent upward trend in "unmet need" is of concern.142 Based on field observations, the early work of IFPS and SIFPSA is evident in hospital quality￾assurance programs; regional training centers for ANMs; NRHM’s use of district action plans in all states; and the work of ASHAs in rural and urban settings, to mention a few examples. IFPS has served as a pilot testing ground for many approaches and, while more rigorous models for testing approaches, (e.g., using control groups or districts) could have been utilized, NRHM uptake of these innovative approaches has helped to sustain a vibrant FP program in UP. 141 28 original districts became 33 districts due to redistricting. 142 The IFPS Post Partum IUCD (PPIUCD) intervention is currently attempting to take advantage of the increase in hospital deliveries by inserting IUDs immediately postpartum. 77 However, factors contributing to the recent slowdown of CPR rates will need to be carefully assessed and tracked so that gains made in the past 20 years are not lost. SIFPSA, like many organizations, faces the challenge of staying relevant over a 20-year period in a changing socio-economic, political and demographic environment. Organizational divisions, such as those between private- and public-sector interventions, will not serve an organization that is testing new models for collaboration, such as PPPs. Depending on SIFPSA’s intended focus, having the highest quality staff with the knowledge, experience and capacity to accomplish those goals will be critical to providing the unique selling point required for credibility and for delivering effective technical assistance. Strong and stable leadership has also been shown to be a critical factor for effectiveness and institutional respect as well as for creating collaborative partnerships with government and private institutions. The legacy of SIFPSA includes its reputation as a leader in FP, a strong and competent alumni network, close working relations with key RH players throughout the state, and a strong NGO network in UP. All of these factors combine to support the continued role of SIFPSA in UP. As we have already noted, the IFPS record in increasing CPR and reducing TFR in UP is commendable.143 However, there remains an under-utilized opportunity to promote family planning through existing safe delivery platforms. In conclusion, NRHMs successful programming and broad focus on reproductive health provides the ideal opportunity for strong and effective family planning interventions. In order to achieve the MDGs of decreasing maternal and infant mortality, coupling effective birth spacing and limiting with safe birth is critical. Many of the innovations created by IFPS and SIFPSA in UP have been expanded throughout India. SIFPSA has the potential to serve as a "laboratory for innovation" with rigorous testing of innovative ideas that could help India to further expand contraceptive choice beyond traditional female sterilization and thereby achieve these important MDG. 143 The original goal was for a CPR of 49%. The 50% CPR achieved includes use of all methods (modern, limiting and traditional) for women ages 15–49 years, and is based on AHS published results. 78 RECOMMENDATIONS FINDINGS/ISSUES RECOMMENDATIONS SIFPSA  Question relevance of current organizational structure in today's environment where NRHM is the major driver behind RCH services and funding  Took a detour from original mission (innovations in family planning) - organization appears to lack clarity in the current health environment  Technical capacity of the organization has weakened over time  Network analysis reveals that SIFPSA has a central role in FP/RH network in UP  Reassess the organizational structure and align to a refocused mission relevant to the current context  Rebuild core technical competence in accordance with redefined mission  Use central position within FP/RH network to offer technical support to Govt. of UP/GOI, eventually serving as an "emersion learning site" or center of excellence for innovations GOVERNMENT OF UP  The health system - infrastructure and human resources - is overwhelmed by the current daily patient institutional delivery load  Shortage of trained managers with technical skills to guide and implement NRHM initiatives  Focus on Health Systems Strengthening by exploring options for PPP for health in order to expand quality care and services.  Explore potential role for a State level technical support unit to facilitate NRHM planning and implementation USAID  Creation of SIFPSA has had an impact on FP indicators in UP  Lack of coordination between USAID projects with limited synergy achieved and visible overlaps144  USAID technical capacity is not aligned with the context of NRHM changing priorities  Engagement with partners and Government described by many as limited  Advocacy for FP/BS at all levels appears to be missing  Disparity in types of private partners in PPP, level of engagement and expectations for sustainability  Numerous IFPS surveys without a broader evaluation strategy and comparison groups have detracted from measuring program effectiveness over time  Advocate and support SIFPSA to realize its potential in the current context of RH/FP  Actively advocate for continuous, strong, and effective family planning interventions and utilization of the platform of institutional delivery to promote family planning initiatives  Refocus on family planning and serve as major advocate for FP/birth spacing in both the private and public sector  To provide quality technical assistance in the current health environment, ensure that sufficient internal technical capacity, especially for project development, management and evaluation, is available  Re-engage with GOI to create an appropriate PPP framework for health.  Support the development of external rigorous evaluation methodologies in synchrony with program start-ups to measure program impact 144 Especially evident in Jharkhand (see pg. 66) 79 ANNEX I: EVALUATION STATEMENT OF WORK USAID/INDIA Office of Program Support DELIVERY ORDER STATEMENT OF WORK Evaluation of Innovations in Family Planning Services Project PROGRAM PROJECT INFORMATION Program Project Title: Evaluation of the Government of India-USAID Innovations in Family Planning Services (IFPS) bilateral project Start-End Dates: September 1992-March 2012 (with one year extension to March 2013 in Uttar Pradesh only for limited close-out activities) Budget: IFPS I-II: $225m, IFPS III: $70m Program/Project Description USAID’s commitment to improving family planning and reproductive health (FP/RH)1 in India is represented by its support for the bilateral Government of India (GOI)-USAID bilateral IFPS project, which began in 1992 with the formation of the State Innovations in Family Planning Services Project Agency (SIFPSA) as an independent society in Uttar Pradesh, and has continued over the past 20 years. In an environment in which fertility rates were high and significant challenges were identified in the state government’s ability to effectively address family planning needs, IFPS I was launched, laying out ambitious results-level indicators in family planning and other reproductive health services. The IFPS I project contributed not only to improvements in contraceptive prevalence rates (CPR) and reductions in the total fertility rate, but many innovations of IFPS were pioneered for the first time in the country and have yielded substantial results in the most populous state of India. The assessment of IFPS I highlighted the project’s wide range of 1 For the purposes of this evaluation, RH also encompasses adolescent RH, age of marriage, female empowerment, and male participation. 80 achievements, particularly service access and training. Based on the assessment, this project was recommended for an extension and addition of two further states (Jharkhand and Uttrakhand) with an increased focus on developing a technical assistance strategy and an expansion of social marketing and behavior-change-communication approaches. The rollout of IFPS II coincided with the launch of the GOI’s ambitious National Rural Health Mission (NRHM), and the next phase was designed to address emerging reproductive health (RH) needs. Given delays in the rollout of NRHM, IFPS II provided responsive, targeted technical assistance to states and the national government to catalyze and facilitate implementation of the national scheme. Technical assistance filled critical gaps as NRHM was being rolled out and also helped states effectively program NRHM resources in a strategic manner. Given the tremendous magnitude of NRHM, IFPS II also shifted, in part at the GOI’s request, to develop and pilot innovative public-private partnership (PPP) models. As NRHM and the pooled funding for the Reproductive and Child Health (RCH II) program increased focus on the public sector, IFPS II was reoriented to catalyze opportunities to engage with the private sector. The vision was for these piloted innovations, developed in consultation with state and national governments, to scale up with NRHM resources. The evaluation of IFPS II () recommended that the project shift its mode of technical collaboration with the GOI at both national and state levels as well as continue PPP pilots to allow continued learning, analysis and documentation of the project in order to take it scale. Therefore, IFPS was designed to refocus support on building the capacity of state and national institutions that provide the technical assistance necessary for health program implementation in their states. In full partnership and consultation with the government, IFPS III identified FP/RH innovations while also supporting the key institutions, systems and structures of NRHM. IFPS III addresses family planning more comprehensively through a life-cycle approach that includes antenatal care, institutional delivery and post-natal care (also in line with core NRHM principles). Since 2004, the IFPS project provided Technical Assistance (TA) at the national and state levels, placing 21 technical experts/consultants in Ministry of Health and Family Welfare (MOHFW) in various divisions such as IEC, Family Planning Statistics, National Rural Health Mission (NRHM), Social Marketing, Health Insurance, and Donor Coordination. These technical experts provided a wide range of support including developing a number of mass media spots for promoting RH/FP; conceptualizing alternate methodology for Intra-Uterine Contraceptive Device (IUCD) training and initiating expansion of IUCD training and services, and supporting the roll out of new systems. The project also supported the establishment of the National Health Systems Resource Center (NHSRC) by supporting staff positions and providing some office equipment until NHSRC received funding from the GOI. The priority focus for the final three years of IFPS was to strengthen health systems for the delivery of quality family planning and reproductive health services. One key area of investment was to build the core technical competencies of SIFPSA in behavior change communication, evaluation, public-private partnerships, FP/RH training and community-based programs. IFPS III 81 included a three-year phase out plan for USAID support to SIFPSA, with the aim of developing a highly qualified, responsive, sustainable agency that is able to support the Government of Uttar Pradesh (GOUP) in the implementation of NRHM and, ultimately, to serve the health needs of the state. Additionally, the efforts in this phase concentrated on demonstrating the effectiveness of innovative models and processes with the goal of complete handover and adoption by the state at the end of the three years. IFPS III priorities focused on four broad areas in the targeted areas of Uttar Pradesh, Uttrakhand and Jharkhand: Training institutions and capacity of providers strengthened for improved delivery of quality FP/RH services Behavior Change Communication (BCC) activities for improved demand, awareness and use of family planning/reproductive health services and products Community-based delivery of FP/RH services and counseling (addressing myths and misconceptions on FP methods) strengthened through NGO projects Existing PPP projects completed, evaluated and documented STATEMENT OF WORK EVALUATION PURPOSE The purpose of this complex final evaluation is outlined below: Part One: The Desk Review will set the parameters for the Impact and Legacy Evaluation that covers IFPS I-III. The Desk Review will determine what conclusions can be reasonably drawn from available data, and develop the impact evaluation research design and methodology, including identifying potential comparison groups and key indicators. The Desk Review will also determine if any follow-on data collection is required to address critical data gaps. Part Two: This Impact and Legacy Evaluation will review and analyze the different systems, strategies, processes and activities of the IFPS project as a whole (IFPS I-IFPS III), which resulted in improving scale and quality of interventions; strengthened systems, evidence-based interventions including operations research and institutional capacity; and enhanced coordination and ownership of project activities in the states of Uttar Pradesh, Jharkhand and Uttarakhand (the latter two for IFPS II and IFPS III). The evaluation will rigorously analyze key indicators for impact and attribution to USAID/India interventions, as well as provide a legacy overview of the program that documents its evolution over time. Intended Uses or other Audiences for the Evaluation The Indian Ministry of Health and Family Welfare (MOHFW) has requested a final evaluation by an external, independent agency. The findings, recommendations and conclusions could be disseminated through workshops and meetings attended by relevant GOI and other 82 stakeholders (e.g., development partners) at the national level. The purpose of these workshops will be to further share IFPS innovations, interventions and approaches. USAID/India will use this evaluation to inform its future designs and comply with USAID Forward policy on evaluations. USAID as an agency has an interest in documenting the legacy of a 20-year FP/RH program. EVALUATION QUESTIONS This complex evaluation will answer the following questions, in priority order: Desk Review: What is the “evaluability” of IFPS, and what statistically sound conclusions can reasonably be drawn from the available data on CPR, TFR, contraceptive method mix, institutional deliveries/safe deliveries, ANC, IFA, birth spacing and TT? Outline the most appropriate quasi-experimental impact evaluation methodology/techniques for a rigorous impact evaluation, including valid comparison groups that take into account shifting district and state borders. What are the major data gaps, and does any follow-on primary research need to be conducted? Is there sufficient data to conduct a modeling exercise that would postulate the effect on key indicators if IFPS intervention had not been implemented? Impact and Legacy: IFPS I-III How effective has the project’s technical assistance been in promoting technical and program priorities, and improving the capacities of local institutions under NRHM to deliver RH programs in USAID-supported states, and what lessons can be drawn for future designs by GOI and USAID/India? To what extent did the overall IFPS project make an impact on reproductive health behaviors and outcomes for men, women, youth, and vulnerable populations in targeted areas, based on the “evaluable” key indicators and comparison groups identified during the Desk Review? What are the key lessons learned from IFPS, including the approaches it introduced such as performance-based financing systems, district action planning, working with the private sector, and establishment of entities like SIFSPA? What performance or impact related conclusions can be made regarding specific innovative sub-projects, such as the PPP for ASHA support system, Networks – Voucher Schemes, Merrigold, Social Franchisee services, and what factors most contributed to their relative success or failure? What are IFPS’s most significant operational contributions to the field of RH/FP? What is the enduring legacy of IFPS? How close are we to achieving the original objectives, noting the major milestone events and challenges of IFPS over 20 years? TECHNICAL REQUIREMENTS FOR EVALUATION: 83 Data Collection and Analysis Methods: Considering that IFPS is a 20-year, multi-phased and complex program with four separate geographic hubs, USAID anticipates a quasi-experimental approach that uses both quantitative and qualitative methodologies. Extensive documentation, including previous evaluations of Phase I and II, various baseline survey data sets, assessments, and reports over the life of the program, as well as solid secondary data exist for this exercise. (Note: See the list of IFPS-generated studies in Attachment 1.) Baseline data is available through reports such as the National Family Health Survey (NFHS) and other RH surveys. USAID/India will provide the team with all relevant country and project specific documents including proposals, evaluation reports and other relevant documents for conducting this Desk Review. The evaluation team is also expected to collect and collate relevant international documents, reports, and data. Extensive end of project documentation from the IFPS Technical Assistance Project (ITAP) will also be available. Desk Review: The parameters of this evaluation will be set during the Desk Review that will select the most appropriate indicators, comparison groups, and evaluation design and methodology based on data availability, quality, and reliability. If required, sample surveys or statistical sampling could be introduced that respond to key evaluation questions. Data collection methodologies will be discussed with, and approved by, the USAID/India Health Office and Program Support Office team prior to the start of the assignment. We would also like to explore introducing quasi-experimental methods to increase the overall rigor of this evaluation, if valid comparison groups can be identified that will allow the evaluation to make sound conclusions about programmatic impact. Impact and Legacy Evaluation of IFPS I–III: This mixed methods quantitative and qualitative study will rigorously analyze key indicators over time; in the state of Uttar Pradesh, for the full IFPS life of project, for the other two states, solely since the start of IFPS II (or the appropriate start date for activities by state, noting that some districts in Uttarakhand were involved in IFPS I when it was part of Uttar Pradesh). The analysis could also include modeling to demonstrate outcomes for these indicators without the IFPS intervention, as well as a broad-based overview of the 20 year program that documents the major milestones, challenges, and accomplishments. Additional data may need to be collected during this part of the evaluation. This approach will also incorporate standard qualitative methods such as key informant interviews, group interviews, focus groups, structured observations, or illustrative case studies, as well as additional quantitative analysis of existing primary and secondary data. Data sources: Data sources that the team will be expected to utilize, review and analyze include project design documents, project proposals, annual work plans, M&E data, state annual action plan, NFHS, RH surveys, NGO evaluation reports, and other project-related documents and reports. Additional relevant documents related to reproductive health in India may be utilized as supporting documents, as well as relevant international standards. Dissemination: The team will conduct a one day Delhi-based dissemination event at the conclusion of this evaluation, in coordination with various stakeholders including the MOH. 84 Composition, Technical Qualifications and Experience Requirements of the Evaluation Team: USAID seeks up to a five member evaluation team (two international and three local members) comprised of a Team Leader/Senior Technical Advisor (RH), Evaluation Methods Specialist, Demographer, Senior Population Analyst, and Public Private Partnership Specialist. All team members must have extensive RH program management, technical or implementation experience, familiarity with USAID’s objectives, approaches, and operations, and prior evaluation/assessment experience. The team will have collective experience in health systems strengthening, behavior change communication and public-private partnerships. Prior experience in India is essential. Collectively, the team must have experience in evaluating RH programs worldwide, as well as health policy analysis. In addition, individual team members should have the technical qualifications and required experience identified for the specific position below: Team Leader/Senior Technical Reproductive Health Analyst (international): This Team Leader/Senior Technical Advisor (RH) has an excellent understanding of global RH strategies and knowledge of the Indian context and programs, as well as an advanced or professional degree in a relevant field. Specifically, s/he should have an excellent understanding of RH barriers, with prior work experience in designing, monitoring and evaluating RH programs. Additionally, s/he should have proven experience in leading and managing large-scale evaluations of various RH programs throughout the world, and have demonstrated success leading multi-phased complex evaluation teams responsible for multiple deliverables. S/he should have knowledge and experience on technical support strategies for strengthening the state’s and private sector’s capacity for providing RH services. S/he should be familiar with the functioning of large donor funded programs in India. The person must have the ability to lead a diverse team of technical and management experts, and to interface with various stakeholders ranging from governmental to non-government organizations and donors, beneficiaries, etc. A minimum of 15 years of experience in the design, management and evaluation of complex programs is required (LOE up to 56 days). Evaluation Methods Specialist (international): This expert will have deep knowledge of impact evaluation methodologies, including quasi-experimental designs, and their practical applications. A minimum of ten years of experience in strategic planning, management, operations research, and/or monitoring and evaluation of global and national RH programs is required. S/he should have strong experience in designing and conducting comparison group analysis, statistical modeling techniques, understanding of secondary literature reviews and developing sampling/survey methodologies. Experience in presenting research publications will be an added advantage (LOE up to 56 days). Demographer (local): This Senior Demographer should have extensive experience with statistical analysis and modeling in the RH/FP field, and hold an advanced or professional degree in a relevant field. S/he should have professional expertise analyzing complex sets primary and secondary data. (LOE up to 56 days) 85 Senior Population Analyst (local): This Senior Population Analyst should have extensive experience with USAID project design, implementation, and evaluation. The person should be considered an expert in integrated public health programming. A good understanding of health systems strengthening, particularly in institutional capacity development of government health systems is desirable. Knowledge and experience in behavior change communication and public private partnerships would be an added advantage. A minimum of 15 years of experience in the design and management of public health programs, including health systems strengthening, is required. Having knowledge and understanding of UP, national-level, and global RH programs and government systems would be an added advantage (LOE up to 30 days). Public Private Partnership Specialist (local): The Public Private Partnership Specialist should have at least 15 years of experience in commercial private sector, at least some of which has been in the area of RH/FP. This expert will be responsible for assessing private commercial sector involvement in the project and assess the innovations piloted by the project. Experience in working with social marketing, behavior change communication, and RH/FP innovations will be an added asset. Experience of working with private provider associations will also be an advantage. S/he should have an understanding of marketing, promotion and consumer research. (LOE up to 30 days) EVALUATION MANAGEMENT ROLES AND RESPONSIBILITIES Overall Guidance: The Health Evaluation Specialist in conjunction with the Evaluation COR, the IFPS AOR and Activity Managers, other key Health Office team members and the Contracting Officer (CO), will provide overall direction to the evaluation team. The Contractor will be responsible for obtaining visas and country clearances for travel for consultants. The Contractor will be responsible for coordinating and facilitating assessment-related TPM, field trips, interviews, and meetings in conjunction with USAID and the IFPS Project. The Contractor will be responsible for submitting an illustrative budget for all estimated costs incurred in carrying out this review. The proposed cost may include, but not be limited to: (1) international and in-country travel; (2) lodging; (3) M&IE; (4) in-country transportation; and (5) other office supplies and logistical support services (i.e., laptop, communication costs, etc.) as needed. The Contractor will be responsible for in-country logistics including transportation, accommodations, communications, office support, etc. For the field work phase of this evaluation, the Contractor will hire an on-the-ground POC for scheduling and other logistics support, given that the IFPS Technical Assistance Project (ITAP) agreement ends in May 2012. 86 Schedule: The duration of the Desk Review and the subsequent Impact and Legacy Evaluation (IFPS I-III) will be for up to four months in duration starting from September 2012. The evaluation team is expected to provide a schedule (in a tabular form) defining when specific steps in the evaluation process will occur and when deliverables will be submitted. Team Planning Meetings (TPM): Before the Desk Review begins, the evaluation team will have either a conference call or DVC with USAID/India to discuss the parameters and expectations for the next month. Once the Desk Review is completed, the evaluation team leader will lead a more formal DVC during which the team will present and discuss their proposed research design and methodology developed during the Desk Review, a draft set of tools and guidelines, preliminary work plan and a preliminary itinerary for the proposed evaluations, taking USAID/India feedback into account. Government of India counterparts from the Ministry of Health and Family Welfare will be invited to participate in this DVC, as well as SIFPSA. A full two-day TPM will take place when the full team has assembled in New Delhi to begin the field work, and will: Clarify team members’ roles and responsibilities for the field work Finalize evaluation questions Establish the timeline and set out the agreed evaluation design and work plan in writing Finalize the methodology guidelines including sampling plan for interviews and site visits, data collection tools and questionnaires, and the data analysis plan to be used by the team Set the date for the mid-term debriefing, final presentations, and other dissemination activities Site Visits and Interviews: Conduct a thorough review of the Project through site visits and interviews. Interviewees will be selected through random or purposeful sampling, with efforts to reduce selection bias, and will include key members from all stakeholder groups, including Ministry of Health and Family Welfare (MOHFW), the governments of Uttar Pradesh, Uttarakhand and Jharkhand, SIFPSA, other state societies, donors and partners in Reproductive Health and Family Planning, USAID and beneficiaries. An interview questionnaire will be prepared in advance and finalized during the TPM, in addition to other evaluation tools identified during the Desk Review that may be required. Site visits will be selected through random or purposeful sampling methods and will take into consideration factors like geographical diversity, representation of various beneficiary groups, and scale of interventions. If random sampling is used, care will be taken to ensure samples are of sufficient size to draw statistically valid conclusions. Reports and Deliverables Desk Review DVC and TPM: The evaluation team will develop a detailed evaluation research design, methodology and work plan prior to departure from Washington, D.C. The team will 87 also meet with USAID/India and a MOHFW counterpart on the evaluation for at least two working days prior to departure for the field. Mid-Point Review/Briefing: The evaluation team will provide a mid-point briefing to the USAID/India and MOHFW team, including evaluation and technical members, to clarify any outstanding queries that may have emerged since the initiation of the evaluation process. The mid-point review for the Impact and Legacy Evaluation should be scheduled after the data analysis phase has taken place. Oral Presentation/Dissemination: The evaluation team will provide an oral briefing on its findings and recommendations to relevant staff in the field, to GOI and state government officials, and to USAID staff at the conclusion of the visits to the various implementing partners. The evaluation team will be required to debrief the USAID/India Mission Director and Deputy Mission Director separately on the observations and recommendations, as well as the MOHFW. A D.C.-based presentation of findings and recommendations is also envisioned, as well as a New Delhi-based dissemination activity in coordination with various stakeholders including the MOHFW. Reports: The evaluation will be required to submit the following drafts and reports: Evaluation Design and Workplan: At the conclusion of the Desk Review, the evaluation team will submit a detailed evaluation design and methodology and workplan for the Impact and Legacy Evaluation of IFPS I-III; for USAID/India review, with a turnaround time of 5 working days after the DVC. Draft Report: The evaluation team will send separate draft reports of its findings and recommendations for the Impact and Legacy Evaluation of IFPS I-III to the USAID/India’s IFPS AOTR and Activity Managers, Health Evaluation Specialist and Evaluation COTR, and other key Health and Program Support Office staff before the oral de-brief and return to the United States. USAID staff will arrange for a review of the draft, to include the implementing partner and other stakeholders as appropriate, and provide comments within five working days. The Evaluation Team will consider all comments, and decide if and how to incorporate the comments, as the team finds appropriate. Any data collection and analysis will disaggregate by gender where appropriate (as required by USAID policy), and other characteristics as relevant and depending on data availability (i.e., age, geographic region). Final Report: The final report should meet the criteria for evaluation reports as stated in Appendix 1 of the USAID Evaluation Policy (). The final report, with executive summary and in electronic form, should be sent to the Evaluation COR, Health Evaluation Specialist and USAID/India IFPS AOR within seven working days after receiving the final comments on the draft evaluation report from USAID/India team. The final report should include an executive summary of no more than three pages that captures key outcomes and recommendations, and can serve as a stand-alone document for policy makers and others who will not read the whole report. The main report with conclusions and recommendations will not exceed 30-40 pages, and annexes to include at a minimum a copy of this scope of work, the final evaluation design, evaluation questionnaires and other data collection tools used to collect 88 information on each of the program components, and lists of sources of information, including persons and organizations contacted. Data sets and records collected in the course of the evaluation should be provided in a separate file to the Evaluation COR for warehousing, if possible. If there are any differences among the Evaluation Team, USAID Staff, the Implementing Partner, or other stakeholders, a Statement of Differences can be included as an annex to the Final Report. Other Deliverables: These could include short video clips to be disseminated on USAID’s You Tube channel, webinar to present Evaluation findings more broadly, local briefing of stakeholders, or other products to be proposed as needed. Evaluation LOE and Budget: Level of Effort: Desk Review LABOR CATEGORY LEVEL MAXIMUM LOE Senior Technical Expert (RH) /Team Leader 1 12 Evaluation Methods Specialist 1 12 Demographer (local) 2 12 Senior Population Analyst (local) 1 3 Public Private Partnership Specialist 2 3 Impact Evaluation2 LABOR CATEGORY LEVEL MAXIMUM LOE Senior Technical Expert (RH) /Team Leader 1 40 Evaluation Methods Specialist 1 37 Demographer 1 34 Senior Population Analyst 1 34 Public Private Partnership Specialist 1 34 2 Enumerators may need to be included in the budget if additional data collection is identified through the Desk Review. The Impact Evaluation LOE may also need to be adjusted based on the outcome and recommendations of the Desk Review. 89 ANNEX II: EVALUATION METHODS AND LIMITATIONS METHODOLOGY USAID contracted Social Impact, Inc. (SI) and its partner, Management Systems International (MSI), to assemble a five-member team (two international and three national specialists) to conduct an impact and legacy evaluation of IFPS during a seven-week period from September 17 – November 7, 2012. The evaluation team included Team Leader and Senior Reproductive Health Specialist Dr. Michele Andina, Evaluation Methods Specialist Dr. Jenny Ruducha, Demographer Dr. Rahul Dev Bhawsar, Senior Population Analyst Dr. Dipanjan Sujit-Roy and Public Private Partnership Specialist Mr. Soumitro Ghosh. The evaluation methodology included (1) team planning meetings between the team and USAID/India (teleconference and in person); (2) an extensive desk review of all project-related documents; (3) interviews with key informants in Delhi and around the world, both in person and via Skype; (4) field visits to the three IFPS intervention states (UP, JH, UK) where team members met NRHM government officials, district health officers and other government and NGO informants, visited public and private hospitals, clinics, health centers and NGOs and met with service providers, including MOs, nurses, auxiliary nurse midwives (ANMs) and accredited social health activist/community based volunteers (ASHA/CHV); (5) midterm briefings with SI and USAID through teleconference; and (6) presentations and discussion of findings with members of the IFPS team and USAID. To determine the impact of IFPS, a separate analysis using national data was conducted. The IFPS evaluation team employed organizational network Figure 8: Site Selection and Visits for IFPS Evaluation 90 analysis (ONA) to aid in the determination of the IFPS legacy and conducted a detailed review of ITAP prepared documents. The respective methodologies for each analysis will be presented in the appropriate sections of this report. Team Planning Meeting and Desk Review Upon kickoff of the evaluation, the team had a teleconference with the Mission to clarify any questions on the SOW and finalize a calendar for evaluation activities. The evaluation team met with USAID/India upon arrival to finalize the workplan. The desk review included an extensive review of all project related documents, found in Annex V. The team also conducted a mid-term briefing during data collection and presented preliminary findings with USAID/India prior to departure. Interviews with Key Informants and Field Visits The evaluation team conducted field visits to understand the legacy of IFPS and to inform subsequent quantitative impact findings with qualitative data. Key informant interviews further enhanced the findings of the IFPS evaluation team and provided additional insights. All five IFPS evaluation team members spent six days in UP and, subsequently, a team of two members visited either JH or UK (five days), while the fifth team member remained in UP for three days of additional interviews, then returned to Delhi for data analysis. Figure displays the specific districts visited by the team. Criteria for the selection of districts visited in UP was based on observing a sample of the “high￾intensity intervention districts”—i.e., the 28 original (33 after redistricting), 1995 IFPS intervention districts—and “matching” these with observations in “low-intensity intervention districts”— i.e. the 42 remaining districts (primarily exposed to statewide IFPS interventions such as BCC campaigns and CSM programs).1 The team selected four high-intensity (Shahjahanpur, Kanpur Naga, Allahabad, Sitapur) and two low-intensity districts (Hardoi, Faizabad) in Uttar Pradesh (UP) based on travel time and the availability of Phase III PPP interventions. For the PPP assessment, the evaluation team held a group meeting of Merrygold Health Network franchisees in Kanpur Nagar, UP.2 In JH and UK, the team held meetings with government officials in Ranchi and Dehradun, respectively; in UK, the team visited one intervention district (Nainital,) and one non-intervention district (Pauri Gharwal). In JH, the team visited two of the three IFPS intervention districts (Simdega and West Singhbhum). Presentation to USAID 1 “Matching “ here was based on similar CPR, TFR rates and demographics. 2 This was the first time that this group of Kanpur Merrygold franchisees had come together for discussion. 91 As mentioned previously, the evaluation team presented preliminary findings to USAID/India prior to departure. In January 2013, the team leader presented the findings, conclusions, and recommendations to USAID Headquarters in Washington, DC. FINDINGS ANALYSIS During the 20-year period of IFPS, multiple national surveys were conducted to collect relevant state- and district-level data. IFPS collected additional project data for baseline determination and to study progress toward benchmarks and project indicators. (See Table below). 92 DATASET TYPE YEAR National Family Health Survey (NFHS) - Phase 1,2,3 National level Based on standardized Demographic and Health Survey (Measure) 1992-93 1998-99 2005-06 District Level Household Survey (DLHS) - Phase 1,2,3 State level survey. Rapid Household Survey 1998–99 2002–2004 2007–2008 Annual Health Survey (AHS) 2010 State level dataset for select states 2010 PERFORM Survey Project level baseline data for 28 original districts 1995 Reproductive Health Indicator Survey (RHIS) Project level dataset 2003, 2005, 2010 Strategic Objective 2 (SO2) Survey Project level annual dataset 1998, 1999, 2000, 2001, 2002 Table 3: National, District and IFPS Data Sets Following the IFPS Evaluation desk review, the team determined that assessing changes-over￾time of the original project outcome indicators (CPR, TFR, use of modern methods and changes in method mix) using national data sets (NFHS, DLHS, AHS) would yield the most reliable information about project impact. The team also sought access to project data to verify the national-level findings and to gain greater clarity of changes in the original 28 high-intensity intervention districts, 3 for which the 1995 PERFORM survey determined baseline values. Numerous challenges were encountered in obtaining access to primary data sets and to their respective data dictionaries for Reproductive Health Indicator Survey (RHIS), PERFORM, Strategic Objective #2 (SO2) surveys, NFHS, and DLHS. Some additional challenges included the absence of original survey questionnaires, inconsistent district coding and differing sampling methodologies in the same data sets over time. Due to inaccessible and inconsistent data from project surveys and time constraints, the findings presented are drawn from the national datasets of India: the internationally accepted Demographic and Health Survey (DHS) (NFHS 1, 2 and 3); the government equivalent DHS (District Level Household and Facility Surveys -DLHS 1, 2 and 3); and the Annual Health Survey 3 These original 28 became 33 primary intervention districts over time due to re-districting. 93 (AHS), 2010-11, a rapid survey done by government of India to inform the planning commission on preparation of the XII Five Year Plan for India. To provide the most comprehensive picture of the impact of IFPS, the team analyzed performance indicators including TFR, CPR, use of modern methods, method mix (condoms, OCP, IUD, female sterilization), coverage of TT and "unmet need" at the state and district levels. Per the request of USAID, the findings compare trends in the three IFPS intervention states to the Empowered Action Group States (EAG), separated into two groups; the five larger states (UP, Bihar Madhya Pradesh, Rajasthan, Odisha) and the three smaller states (Jharkhand, Uttarakhand, Chhattisgarh) all referenced against overall India rates. The District Level Analysis used DLHS 1, 2, and 3 and, to adjust for the data inconsistencies, created two separate data sets for analysis. One has subjects (i.e. women 15-49 years of ages) from UP and Bihar as the unit of analysis and the other uses districts from UP, UK and JH as the unit of analysis. The team selected Bihar for district level comparison due to its health, socio￾demographic and economic similarities to UP.4 The advantage of subject-level data is that more statistical power is available for multivariate models when the individual is the unit of analysis. District level analysis rates presented throughout the findings section were obtained from separate multivariate analyses. Statistical Methods The team analyzed state level data5 to determine trends over time for each of the following indicators viz. (1) TFR; (2) CPR; (3) Modern CPR (includes only modern methods such as Male and Female Sterilization, IUDs, OCPs and Condoms); (4) Method Mix – OCPs, Condoms, IUDs, Male and Female Sterilization, (5) Coverage of Tetanus Toxoid in pregnant women and (6) unmet need for family planning. The team compared trends to national rates and calculated significance (if any) using a chi squared test. The team made adjustments to the DLHS data to make it comparable to the NFHS datasets,6 then entered the data into MS Excel 2013 and used STATA 12 SE to complete the significance calculations. The team analyzed the district level data for UP using DLHS 1, 2 and 3 to assess the impact of IFPS interventions in high-intensity and low-intensity districts of UP. The dependent variables used for this evaluation are (a) for prevalence rates: Total Fertility Rate (TFR) Contraceptive Prevalence Rate (CPR), Use of Modern Methods (MM), and (b) for method mix, Oral 4 The selection of Bihar as a suitable UP comparison state was approved by USAID/India 5 Using NFHS 1, 2, and 3, DLHS 2 and 3, and AHS data 6 Adjustments to CPR for modern methods included removing the emergency contraception rates from DLHS 3 and AHS data. The respondents for DLHS 2 were currently married women 15–44 years and not 15–49 years as calculated for NFHS 1-3 and AHS. This was adjusted to be comparable to the other data set. 94 Contraceptive Use, Female Sterilization, Condom Use, and Inter-uterine Device Use. The main independent variable is whether an IFPS intervention occurred in a given district.7 In UP the "high-intensity intervention districts" include the original 28 (33 after redistricting) IFPS districts and the "low-intensity intervention districts" are the remaining 42 districts that were only exposed to statewide interventions. This exposure variable is time dependent as it changes over time. Dependent variables are prevalence rates such as CPR, modern methods, proportion method use by category, TFR. Main comparison groups are time varying intervention membership (UP, JK, UK no-intervention; UP, JK, UK intervention; and Bihar districts as an additional control). The main independent variable is whether an IFPS intervention occurred in a given district. This intervention group membership (28 initially in UP, and 33 later as some districts got divided) may change over time, but the comparison groups is those districts in UP without interventions. This exposure variable is time dependent. There are two kinds of adjustments for the individual multivariate models (one per dependent variable), baseline adjustment and time dependent adjustment. The former will be used to make districts similar at baseline so they are comparable, and the latter will be used to adjust for changes over time in the districts that may bias the results (such as a changing urban/rural proportion over time). Not all covariates are chosen for every model since there may be variables whose association with some outcome may need to be highlighted and not “adjusted”. The statistical method used was generalized linear models (Logistic or Gaussian links) coupled with a mixed models for the longitudinal data. All dependent variables are binary variables in the person level analysis or rates in the district level analysis. The logistic link is used for the former and the Gaussian link for the latter. STATA 12 was used for all analyses. Limitations Though the datasets use similar methodologies for obtaining data and are considered to be comparable, there are inherent differences that allow for a variety of interpretations of the same findings. For example, there are differences in respondent age groups for NFHS and DLHS. NFHS uses a DHS approved age group of 15–49 years of currently married women (CMW) and DLHS (1 and 2) used CMW ages 15-44 years. Further, there are minor differences in sampling and attributions such that DLHS sampling allows for district level analysis of data, whereas NFHS 7 The interventions were classified as high-intensity and low-intensity intervention districts. However the major difference was the presence of direct interventions in high-intensity districts and indirect interventions (e.g. statewide BCC, CSM, TA) in the others. The reader is referred to the fact that IFPS direct interventions took place in 28/33 districts in IFPS I (1995-2004) and shifted to TA mode in IFPS II and III, (2005-2012) but TA was provided across UP beginning in 1995. 95 is more appropriate for state and national level analysis. The calculation of confidence intervals for the AHS data was not possible due to the lack of access to primary data. Due to inaccessibility of project-specific data and the absence of randomized control groups, full attribution to IFPS for changes over time is not possible, though the team can identify some causal chains. While trends can be observed and the DLHS District Level analysis may indicate some differences between the high- and low-intensity UP intervention districts, the lack of comparable "control districts", i.e. those without any IFPS interventions, is a limitation. The presence of other donors and projects in UP from 1992-2012 is also not quantifiable, further limiting the potential for full attribution. 96 ANNEX III: DATA COLLECTION INSTRUMENTS Checklist for Assessing ITAP Documents1 Title of Study Being Reviewed: __________________________________ EVALUATION REVIEW FACTOR 1 2 3 4 5 REVIEWER COMMENTS Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? Regardless of the type of evaluation, does the evaluation report reflect use of sound social science research methods? Does the Executive Summary concisely state the main points of the evaluation: summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation? Does the report provide a clear description of the evaluation’s design, sample selection, where data was collected, and when the evaluation was conducted? Does the report state the type of intervention that was being evaluated, its geographic coverage and project/intervention time span (reference period) covered by the evaluation? Does the report contain key research questions and address each key question around which the evaluation was designed? In answering the evaluation questions, does the report appropriately use comparisons made against baseline data? If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the 1 Adjusted by IFPS evaluation team: based on USAID (2011), Evaluation Policy, Good practices in evaluation reporting have also been drawn from: Linda Morra Imas and Ray C. Rist. (2009), The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC.: The World Bank; Michael Scriven. (2005), Key Evaluation Checklist tufflebeam, Daniel L. 1999. Program Evaluations Metaevaluation Checklist. 97 EVALUATION REVIEW FACTOR 1 2 3 4 5 REVIEWER COMMENTS intervention, as well as its effectiveness? Is there a clear description of the evaluation’s data collection methods (summarized in the text with the full description presented in an annex)? Does the evaluation report contain a section describing the limitations associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc)? Are data appropriately analyzed with percentages, ratios, cross-tabulations and statistical testing? Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? Does the evaluation report give the appearance of a thoughtful, evidence-based, and well organized effort to objectively evaluate what worked in the project, what did not and why? TOTAL SCORE:_______ Summary Comments: (few sentences): 98 NATIONAL LEVEL NETWORK ANALYSIS QUESTIONNAIRE Network Analysis Survey Instrument NATIONAL LEVEL i. Respondents Name:_________________________ ii. UNIQUE ID: ___ ___ ___ ___ ____ iii. Respondent’s organization________________ iv. LEVEL: National _____iii. State: (circle) 1. Uttar Pradesh; 2. Uttarkhand; 3. Jharkhand v.. District (Write in): ______________________ vi. Interviewer(s): _______________Date of Interview: __________ 1 What departments or organizations do you currently work for? What are your position(s)/title(s)? Of these which is the key department/organization you work for? List all department(s)/organization(s) respondent is working for and the title of the position: A. Department/Organization Name B. Respondent’s position C. Key departmen t No. Months/Years Worked 1 1=Yes 0=No _____Mo ______Yrs 2 1=Yes 0=No _____Mo ______Yrs 2 Did you or your organization ever have a relationship with IFPS Program? 1 = Yes 0 = No 3 Have you ever worked for SIFPSA? 1 = Yes 0 = No 4 How long have you worked at SIFPSA (IF LESS THAN 30 DAYS, WRITE AS 1 MONTH) No. Months: ______ No. Years: ______ 6 What years have you worked at SIFPSA? 19___ ____ until ___ ___ ___ ___ (fill in the years) 7 What position(s) in the organization did you have? 1. ____________________ 2. _____________________ 8 Did you ever receive training from SIFPSA? 1 = Yes 0 = No 99 TABLE 1: National Level Organizational Networks Relationship (Column 2) Did YOU or YOUR organization ever have a relationship on FP & RH with (name organization)___? 1=YES; 0=NO; 3=Respondent ORG TYPES of Family Planning and Reproductive Health LINKS (1=YES; 0=NO) 1=Poor 2=Fair 3=Good 4=Excellen t Department or Organization (SHOW CARD) Relationshi p Sharing Informatio n Coordinating programs or services Joint BCC/IEC Program s Training or Capacity Building Relationsh ip Quality National Level 1. FP-MOHFW 1 2 3 4 2. RCH-MOHFW 1 2 3 4 3. IEC/BCC - MOHFW 1 2 3 4 4. NRHM 1 2 3 4 5. NHSRC 1 2 3 4 6. NIHFW 1 2 3 4 7. Futures 1 2 3 4 8. JHPIEGO 1 2 3 4 9. HLFPPT 1 2 3 4 10. PFI 1 2 3 4 11. FHI-360 1 2 3 4 12. PSI 1 2 3 4 13. BMGF 1 2 3 4 14. USAID 1 2 3 4 15. World Bank 1 2 3 4 16. Intrahealth / Vistaar 1 2 3 4 17. Population Council 1 2 3 4 18. Pathfinder 1 2 3 4 19. JSK (Jana Sankhya Sthiratha Kosh) Population Stabilization Fund 1 2 3 4 20. Engender Health 1 2 3 4 21. CEDPA 1 2 3 4 100 22. UNFPA 1 2 3 4 TABLE 2a: UTTAR PRADESH STATE LEVEL NETWORK UTTAR PRADESH STATE LEVEL (INSERT INTO COLUMN 2) Did YOU or YOUR organization ever have a relationship on FP & RH with (name organization)___? 1=YES; 0=NO; 3=Respondent ORG TYPES of Family Planning and Reproductive Health LINKS (1=YES; 0=NO) 1=Poor 2=Fair 3=Good 4=Excellen t Department or Organization Relationsh ip Sharing Informatio n Coordinatin g programs or services Joint BCC/IEC Programs Training or Capacity Building Relationsh ip Quality 1. NRHM/SPMU 1 2 3 4 2. RCH-MOHFW 1 2 3 4 3. SIHFW 1 2 3 4 4. State Health Society 1 2 3 4 5. HLFPPT 1 2 3 4 6. PSI 1 2 3 4 7. JHPIEGO 1 2 3 4 8. UHI/FHI-360 1 2 3 4 9. Intrahealth / Vistaar 1 2 3 4 10. MBPH/DIMPA / Abt 1 2 3 4 11. Population Council 1 2 3 4 12. Institute of Reproductive Health / Georgetown Univ. 1 2 3 4 TABLE 2b: JHARKHAND STATE LEVEL NETWORK JHARKHAND STATE LEVEL ( INSERT INTO COLUMN 2) Did YOU or YOUR organization ever have a relationship on FP & RH with (name organization)___? 1=YES 0=NO 3=Respondent ORG TYPES of Family Planning and Reproductive Health LINKS (1=YES; 0=NO) 1=Poor 2=Fair 3=Good 4=Excellen t Department or Relationsh Sharing Coordinatin Joint Training Relationsh 101 Organization ip Informatio n g programs or services BCC/IEC Program s or Capacity Building ip Quality 1. NRHM/SPMU 1 2 3 4 2. RCH-MOHFW 1 2 3 4 3. RIMS Training Ctr 1 2 3 4 4. State Health Society 1 2 3 4 5. HLFPPT 1 2 3 4 6. JHPIEGO 1 2 3 4 7. IntraHealth / Vistaar 1 2 3 4 8. MBPH/DIMPA/ABt 1 2 3 4 9. Population Council 1 2 3 4 10. Institute of Reproductive Health / Georgetown Univ. 1 2 3 4 11. Engender Health 1 2 3 4 TABLE 2c: UTTARKHAND STATE LEVEL NETWORK UTTARKHAND STATE LEVEL (INSERT INTO COLUMN 2) Did YOU or YOUR organization ever have a relationship on FP & RH with (name organization)___? 1=YES; 0=NO; 3=Respondent ORG TYPES of Family Planning and Reproductive Health LINKS (1=YES; 0=NO) 1=Poor 2=Fair 3=Good 4=Excellent Department or Organization Relationship Sharing Information Coordinating programs or services Joint BCC/IEC Programs Training or Capacity Building Relationship Quality 1. NRHM/SPMU 1 2 3 4 2. MOHFW 1 2 3 4 3. State Health Society (UKHFWS) 1 2 3 4 4. Population Council 1 2 3 4 5. JHPIEGO 1 2 3 4 102 6. HIHT (Himalayan Institute of Health Technology) 1 2 3 4 We would like to know departments or organizations you most admire for doing an especially good job on improving family planning and reproductive health. Please list up to five such departments or organizations below (please indicate level of the Department: National, State or District) a. ____________________________________ (please circle: National or State or District) b. ____________________________________ (please circle: National or State or District) c. ____________________________________ (please circle: National or State or District) d. ____________________________________ (please circle: National or State or District) ____________________________________ (please circle: National or State or District) 103 ANNEX IV: PERSONS CONTACTED Organization Name of person interviewed Position/title Allahabad, UP ASHA Hospital (Sambhav) Dr Sujit Singh MD Allahabad CHWs for Sambhav Ms. Madhubala, Sakuntala, Sashi Singh Allahabad DIFPSA K.S. Bisht Division Project Manager, Allahabad DIFPSA Vinod Kumar Singh Dist Prog Manager, Allahabad DIFPSA Sashank Singh Dist Community Mobilizer, Allahabad Indian Institute for Development Studies & Research Dr B Sharma Director MLN Medical College (NSV) Dr Dilip Chourasia Prof Orology, Allahabad State Government Dr Padmakar Singh CMO Allahabad State Government Dr Sarveswari Natiyal Ditrict Women's Hospital Allahabad State Govt Dr R.C. Tripathi CMS, CHC, Phoolpur Swarg Dr Anil Yadav Director UHI Rochak Bharadwaj City Manager, Allahabad Vatsalya Hospital (MG) Dr Niraj Agrawal MD Allahabad Faizabad, UP Addl. Directorate Office Dr R N Mishra Addl Director, Faizabad Division ANM TC Ms Kalavati Devi I/C ANM TC ANM TC Ms Indirawati Devi Cook ANM TC Ms Kalavati Mishra Helper ANM TC Ms Bichitra Tara Sweepress CHC Pura Bazar Dr Ashok Kumar Singh I/C CHC Dept of Medical and FW Dr Suresh Pataria Ex Addl. CMO RCH Dept. of Medical & FW Dr Rakesh Kumar Yadav CMO District Women's Hospital, Faizabad Dr Neerja Mala CMS Div Clinical Training Centre Mr Deshbandhu Acting DPM Faizabad Div Clinical Training Centre Mr V N Singh Warden DCTC Div Clinical Training Centre Mr R P Singh I/C DCTC Div Clinical Training Centre Mr R G Pandey Senior Instructor RFPTC 104 Organization Name of person interviewed Position/title Div Clinical Training Centre Mr P R Kanojia Instructor Div PMU Mr Harit Saxena Divisional Project Manager DPMU Dr A C Tripathy Dy. Director, Faizabad Division Sub-centre, Tehsinapur Ms Kanak Lata ANM I/C Sub-centre, Tehsinapur Ms Sarika Tiwari Contractual ANM Sub-centre, Tehsinapur Ms Vineeta Gupta Contractual ANM Sub-centre, Tehsinapur Ms Upma Rai Contractual ANM Haldwani, UK Astha Seva Sansthan, Haldwani CMO, Nainital Dr S.C. Pant CMO Nainital DPMU, NRHM Mr Madan Mehra Dist Project Manager, NRHM, Nainital Haldwani Base Hospital Dr B. C. Bhat Chief Medical Superintendent Haldwani Base Hospital Dr. D.S. Mankoti State Master Trainer NSV Haldwani Base Hospital Dr Savitri Singh Sr Consultant and Master Trainer Haldwani Base Hospital Mr Abhay Kumar Quality Manager Himalayan Seva Samity, Haldwani Mr Dhiren Prasad Joshi NGO Head Himalayan Seva Samity, Haldwani Mr Chandan Singh Bisht UDAAN Project Coordinator Himalayan Seva Samity, Haldwani Mr. Digdarshan Floria Male Mobilizer Himalayan Seva Samity, Haldwani Ms. Usha Rekwal Female Mobilizer LR Khanna Maternity Home & FP Centre, Haldwani Dr Meena Bhat Medical Superintendent Hardoi, UP Acting DPM Mr. Ajay Kumar Sharma District Community Mobilizer Community Health Centre, Kachuna Dr R P Dixit MO I/C CHC District Medical and Famaily Welfare Dept, Hardoi Dr R P Rawat Addl CMO, Hardoi District Medical and Famaily Welfare Dept, Hardoi Dr A Bhargava CMO, Hardoi District Women's Hospital, Hardoi Dr Ranjana Srivastava CMS Kanpur, UP Amin Welfare Trust, Jajmau, Kanpur Dr. Shahi Project Director CHC, Kalyanpur Dr Mohanty Medical Officer 105 Organization Name of person interviewed Position/title DIFPSA Harit Saxena Divisional Project Manager District Women's Hospital, Kanpur Dr Madhu Lal Chief Superintendant, Kanpur Div Clinical Training Centre Dr Kajali Gupta I/C Hari Merrygold Hospital, Kanpur Dr Satinder Lamba Owner MerryGold Clinic, Kanpur Dr Sylva Shukla GD MO Rajrani Hospital, Kanpur Dr Preeti Verma Owner Tirupathi Hospital, Kanpur Dr S.K. Shukla Owner Dr Kiran Lady Medical officer Kotdwar, UK State Govt, Medical College , Kotdwar Dr I.S. Sawant Medical Superintendant, State Govt, Medical College , Kotdwar Dr Sangeeta Sharma Lady Medical officer State Govt, Medical College , Kotdwar Tarunam Jahan FP Counsellor State Govt, Medical College , Kotdwar Ms Usha ASHA Lucknow, UP King George Medical University Dr D Dalela Professor of Urology, KGMU King George Medical University S Kumar PCDF Mary Stoppes International Dr Sulabha Swaroop Ex DGM - Public Sector SIFPSA SIFPSA Amit Ghosh ED SIFPA and MD of NRHM SIFPSA Devesh Tripathi SIFPSA Maya Preeti SIFPSA Digvijay Trivedi Asst Project Manager (Private Sector) SIFPSA S. C. Joshi G.M. Finance SIFPSA Savita Chauhan G.M. Private Sector Division SIFPSA B.K Jain G.M. R&E/FPIS SIFPSA Dr Meena Bajpai G.M. Public Sector Division SIFPSA S. P. Khare Consultant (R & E) SIFPSA Seema L. George Project Coordinator (R&E) SIFPSA Sanjay Srivastava SIFPSA Sanjay Sengupta DGM, HR SIFPSA Suman Chandrabhan SIFPSA Saheen Khan 106 Organization Name of person interviewed Position/title SIFPSA Rajesh Bangia DGM DAP, HR under NRHM SIFPSA O. P. Singh Consultant National Programmes SIFPSA R.C. Chadha Consultant National Programmes SIHFW Dr Usha Saxena Asst Prof. SIHFW Dr N L Srivastava Professor Naukuciyatal, UK Grameen Utthan Samity, Naukuchiyatal Mr Saryu Nandan Joshi Community Mobilizer (ASHA) Grameen Utthan Samity, Naukuchiyatal Mr Prakash Pande Block Coordinator (ASHA) Society of People's Development, Naukuciyatal Mr Arun Pokriyal Mobile Van Coordinator, Kumaon Mondal New Delhi BD Manoj Gopalakrishna Managing Director Department of Commerce, GOI J S Deepak Addl Secretary FHI Bitra George Country Director Futures Group Suneeta Sharma Managing Director Futures Group Shuvi Sharma Deputy Country Director Futures Group Dr. G. Narayana Former CEO Futures Group Shuvi Sharma Deputy Country Director HLFPPT Dr, R, Vasanthi Technical Consultant MOHFW Dr. S K Sikdar Director of Communication - Family Planning MOHFW Aradhana Johri Addl. Secretary NACO NIH&FW Sanjay Gupta Associate Professor Pathfinder John Dumm Special Advisor to the President PFI Sona Sharma Joint Director, Advocacy and Communications PFI Sainath Banerjee Chief of Party, Health of the Urban Poor Project PSI Pritpal Marjara Managing Director PSI Sanjeev Dham Senior Director-Programs US Gov - HHS Loveleen Johri Senior Technical Officer USAID Delhi Ms. Sheena Chabra Project Management Specialist USAID Delhi Mr. Vijay Paulraj Project Management Specialist (Family Planning/Communication) USAID Delhi Ms. Ekta Saroha Project Management Specialist (Strategic Information and Policy) USAID Delhi Mr. Gulshan Bhatla Development Program Specialist 107 Organization Name of person interviewed Position/title USAID Delhi Ms. Charu Lal Evaluation Specialist USAID Delhi Kerry Pelzman Former Director Health Ranchi, JH Engender Health Mr Vivekanand Pandey State Program Manager Intra Health - Vistaar Dr Manju Shukla State Director, Intra Health IRH Mr Vikas Program Associate IRH Ms Rashmi Program Associate JHPIEGO Dr Kamlesh Lalchandani State Program Manager JHPIEGO Dr Dinesh Singh State Advisor, Clinical Services and Training JHPIEGO Mr Alok Data Assistant Sadar Hospital, Ranchi Ms Shalini Counselor FP Sadar Hospital, Ranchi Dr A K Chaudhary Civil Surgeon SHS Gunjan Khalco State Coordinator, FP Cell SHS Mr Abubaker Siddique MD NRHM SHS Dr Praveen Chandra Director Health Services, JHK Vikas Bharati Ms Ranjana Kumari Team Lead Vikas Bharati Ms Nipa Das Health Lead Vikas Bharati Mr Sudipto Banerjee Nutrition Specialist Vikas Bharati Mr Ram Kumat Chaudhary Dist Lead Godda Vikas Bharati Dr C K Sharma Dist Lead Latehar Vikas Bharati Ms Santoshi Beneficiary of PP IUCD at Sadar Hopsital Vikas Bharati Mr Ajay CINI - Vistaar Program Manager Shahjahanpur, UP District Medical and Famaily Welfare Dept, Shahjahanpur Dr A K Srivastava Chief Medical Officer District Medical and Famaily Welfare Dept, Shahjahanpur Dr Shiv Om Addl. CMO RCH District Medical and FW Dept. Mr Rajesh Kumar Bhatnagar District Health Information Education Officer District Women Hospital Dr Shashi Gupta Gynaecologist Div PMU, Bareilly Mr Shahid Hussain Div. Program Manager DPMU Mr K K Sharma District Program Manager DPMU Ms Rita Verma Community Mobilizer Dr B N Behl Memorial Hospital (Merri Gold) Dr Siddhart Behl Director Dr B N Behl Memorial Hospital (Merri Gold) Dr Rashmi Behl Co-Director and Chief Gynaecologist 108 Organization Name of person interviewed Position/title Jain Maternity Hospital (Merri Silver) Dr Ramesh Jain Director Rashtriya Yuva Vikas Shodh Sansthan Ms Sandhya Saxena Director Rashtriya Yuva Vikas Shodh Sansthan Mr Brijesh Saxena Secretary Rashtriya Yuva Vikas Shodh Sansthan Mr Subhash Kumar Member and Ex APC, NGO CBD workers project Rashtriya Yuva Vikas Shodh Sansthan Mr Anand Prakash Ex Community Mobilizer NGO CBD workers project Rashtriya Yuva Vikas Shodh Sansthan Ms Kanchan Gupta, Ms Shafiya Begum, Ms Rekha Dixit NGO CBD Workers Ravi Om Maternity Home Dr R K Singh Owner Shri Siddhi Vinayak Hospital (Other Pvt sector) Dr Richa Director Vinoba Sewa Ashram Shri Ramesh Bhaiya Chairman Simdega, JH ANM Training Center Ms Agnes Xaxa Principal ANM TC, Simdega ANM Training Center Ms Savitri Devi ANM Student ANM Training Center Ms Binti Kumari ANM Student ANM Training Center Ms Hemlata Kumari ANM Student Labor Room, District Hospital Ms Kusum Sokhey ANM I/C Labor Room, District Hospital Ms Lily Kujur ANM Labor Room, District Hospital Ms Subhadrani Gudiya ANM Sitapur, UP DIFPSA Raja Ram Yadav Divisional Project Manager, Sitapur State govt Dr Surendra Shahi Superintendant cum Consultant Surgeon, CHC Sitapur State govt Ms Nazar Jahan ANM, Suraicha Sub centre + 2 ASHAs and 1 DAI State govt Dr Arun Kumar Gautam Sr Consultant Eye. EX Dy CMO, Sitapur State govt Dr. Suresh Kumar Chouhan Addl CMO, Sitapur State govt Dr Sushma Karanwala Superintendant, Dist Hospital, Sitapur Uttarakhand Futures Group Mr Ashutosh Kandwal State Manager, Uttarakhand Himalayan Insititute of Dr V.D. Semwal Project manager, ASHA Resource 109 Organization Name of person interviewed Position/title Hospital Trust Training Centre, Dehradun Himalayan Insititute of Hospital Trust Dr Ruchira Nautiyal Associate Professor, Ob & Gyn dept, Dehradun SHS Dr Abhay Kumar State Project Manager, Dehradun SHS Dr R.K. Pant Addl Director, National Program, Dehradun SHS Dr. Saroj Naithani Jt Director RCH, Dehradun SHS Mr Piyush Singh ED & MD, Dehradun SHS Mr Raj Kamal Project Support Officer Washington, DC USAID HQ Mr. Robert Clay Deputy Assistant Administrator, Global Health Bureau West Singbhum, JH District Hospital, Chaibasa Dr A D N Prasad Civ Surgeon JHPIEGO Dr Arunav Program Officer, West Singhbhum MCH Centre, District Hospital Ms Silvy Laurence ANM MCH Centre, District Hospital Ms Rajlaxmi Amma B ANM I/C ANNEX V: DOCUMENTS REVIEWED IFPS Project: 1st Set of Benchmarks IFPS Project Benchmarks (JH) IFPS Project Benchmarks (UA) Performance Indicators for IFPS: 1995 PERFORM Survey in UP, State Seminar Report Coverage Evaluation Survey of TT Campaign and Estimation of Neo-Natal Mortality Rate 2000 Report on Coverage Evaluation Survey of TT Campaign and Estimation of Neo-Natal Mortality Rate 2001 Coverage Evaluation Survey of TT Campaign, November-December 2001 Coverage Evaluation Survey of TT Campaign and Estimation of Neo-Natal Mortality Rate 2002 Endline Evaluation of District Action Place Intervention Under IFPS in UP Endline Evaluation Report – Low Sewa Sansthan, Mirzapur 110 Final Report of Community-Based FP/RCH Project in Bansdih Block of Ballia, Implemented by World Vision Final Report of Impact Evaluation of Traditional Birth Attendant Training Programme in Agra District Endline Assessment of the IFPS Summary Report Endline Evaluation of Strengthening RH/FPS of Bareilly District, Implemented by IFFCO Endline Evaluation of PEACE Project in Saharanpur District, UP Final Report of Promotion of Reproductive and Child Health and Family Planning Counselling, Motivation and Informed Choice in Mahu Block of Banda District Final Report of Training of ISM and Homeopathic Practitioner in Kanpur Nagar District, Implemented by NPSS Final Report of RCH Services in Urban Slums of Meerut District, Implemented by DUDA Final Report of Endline Evaluation of Strengthening ASHA Support System for Promoting Reproductive and Child Health in Agra District, Implemented by Foundation for Social Care Final Report of A Clinical Based Reproductive Health Services to the Community of Nuyamtabad and Chakiya Blocks of Chandauli District, Implemented by IID Final Report of Improving Clinical Reproductive and Child Health Services in Two Blocks of Kaushambi District, Implemented by Jan Kalyan Maha Samiti Endline Evaluation of Agra Voucher Project Draft Report, 2009 Endline Evaluation of Allahabad Family Life Education (FLE) Project, Final Report Endline Evaluation Report of Centre of Excellence (COE) Project SIFPSA Evaluation of CSM Project in Uttar Pradesh, 2006 Mid-Term Evaluation of Social Franchises Project in Uttar Pradesh, Final Report Evaluation of Skilled Birth Attendant Training of Medical Officers in Uttar Pradesh, Final Report Evaluation of Village Health and Sanitation Committee (VHSC) Project, Final Report Assessment of the Innovation Family Planning Services Project, 2003 IFPS II Evaluation Report, 2007 Jharkhand State Action Plan, IFPS Project Uttar Pradesh State Action Plan, IFPS Project National State Action Plan, IFPS Project Uttarkhand State Action Plan, IFPS Project IFPS Project Grant Agreement, Seventeenth Amendatory Agreement Project Grant Agreement Between President of India and USA for IFPS 111 IFPS Project Grant Agreement, Eighteenth Amendatory Agreement IFPS Project Grant Agreement, Twenty-Fourth Amendatory Agreement Office of Inspector General, Audit of Phase III of USAID/India IFPS Project Annual Work Plan for IFPS II Technical Assistance Project Quarterly Indicator Report, IFPS Technical Assistance Project (ITAP), Jan 2011-March 2011 Consolidated Quarterly Report, IFPS ITAP Project Portfolio Review Project/Activity Summary Sheet for IFPS in Jharkhand Portfolio Review Project/Activity Summary Sheet for IFPS in Uttarkhand Portfolio Review Project/Activity Summary Sheet for IFPS in Uttar Pradesh Portfolio Review Project/Activity Summary Sheet for IFPS ITAP FY 2010 Portfolio Review Project/Activity Summary Sheet for IFPS FY 2010 Portfolio Review Project/Activity Summary Sheet for IFPS in Jharkhand FY 2010 Portfolio Review Project/Activity Summary Sheet for IFPS in Uttarkhand FY 2010 Portfolio Review Project/Activity Summary Sheet for IFPS in Uttar Pradesh FY 2010 Portfolio Review Project/Activity Summary Sheet for IFPS ITAP FY 2011 Portfolio Review Project/Activity Summary Sheet for IFPS FY 2011 Portfolio Review Project/Activity Summary Sheet for IFPS in Jharkhand FY 2011 Portfolio Review Project/Activity Summary Sheet for IFPS in Uttarkhand FY 2011 Portfolio Review Project/Activity Summary Sheet for IFPS in Uttar Pradesh FY 2011 Portfolio Review Project/Activity Summary Sheet for IFPS ITAP Formulation of Population and Health Policies in Indian States, 1997-2004 (2006) District Action Plans- Implementing Decentralized Health Planning (2006) "IDEAS, INSIGHTS, AND INNOVATIONS: Achievements and Lessons Learned from the Innovations in Family Planning Services (IFPS) Project, 1992–2004" (2006) "IDEAS, INSIGHTS, AND INNOVATIONS: Achievements and Lessons Learned from the Innovations in Family Planning Services (IFPS) Project, 1992–2004- A Summary" (2006) Behavior Change Communication Activities and Achievements (IFPS/ITAP) (2010) Performance Based Disbursement- Innovations in Family Planning Services Project (2010) Injectable Contraceptives in India: Past, Present and Future (2010) Promoting Adolescent Reproductive Health in Uttarakhand and Uttar Pradesh, India (2012) 112 Behavior Change Communication Activities and Achievements: Lessons Learned, Best Practices and Promising Approaches (2012) SAMBHAV: Vouchers Make High-Quality Reproductive Health Services Possible for India’s Poor (2012) Capacity Building of Institutions in the Health Sector: Review of Experiences in Uttar Pradesh, Uttarakhand and Jharkhand (2012) Uttar Pradesh Reproductive Health Indicator Survey-2005 (2006) Reproductive and Child Health Status in Slum, Non Slum and Rural Areas of Agra (2006) Reproductive and Child Health Status in Slum, Non Slum and Rural Areas of Kanpur Nagar (2006) Report on Maternal Death Audits in Uttar Pradesh (2006) International Workshop on Social Franchising in the Health Sector (2006) Designing a Social Franchising Initiative in the Health Sector (2006) Knowledge about RCH, services provided and media exposure: A study of grass root level health workers (2007) Knowledge and Perceptions about Reproductive Health- Issues, and Media Reach among Mothers-in-Law (2007) Towards Equity in health: A study of barriers in accessing RCH services in Uttar Pradesh (2010) Reproductive Health Indicator Survey- Uttar Pradesh 2010 (2010) Social Franchising as a Public-Private Partnership Model: Lessons Learned from the Merrygold Health Network of Uttar Pradesh, India (2012) 20 Years of the Innovations in Family Planning Services Project in Uttar Pradesh, India: Experiences, Lessons Learned and Achievements (2012) Uttaranchal National Rural Health Mission - Objectives, Strategies and Implementation Mechanisms (2005) Evaluation of Mobile Clinics in Uttaranchal (2006) Qualitative assessment of Mobile Health Vans (2010) Reaching Underserved Communities through Mobile Health Vans in Uttarakhand, India (2012) Community-based Workers Improve Health Outcomes in Uttarakhand, India (2012) Jharkhand Health Communication Strategy (2008) Health Issues & Health Seeking Behavior of Tribal Population (2009) Social Audit of Infant and maternal Deaths in Jharkhand (2010) Reproductive Health Indicator Survey, 2010 (2010) Reproductive Health Indicator Survey-2005 113 Private Health Facility Mapping Survey- Agra Commercial Social Marketing- Back Check Commercial Social Marketing- Baseline survey Maternal Health and Vaccination Expenditures in UP Household RCH Expenditure- NSSO Analysis A Study On Identifying Interventions For Improving Motivation Of Health Workers In Four Districts Of Uttar Pradesh Behaviour Change Communication- Women Rapid Assessment of School Health Program In Five Districts Of Uttar Pradesh Household Health Expenditure- NSSO Analysis Assessment of Clinic Based NGO Projects in Uttar Pradesh Government Health Facility Mapping Survey- Agra Pre testing of health care cost study tools Commercial Social Marketing (DKT) Evaluation 2008 Health Insurance Need Assessment in Bahraich district Rapid Assessment of the Functionality of FRUs and 24x7 PHCs in Uttar Pradesh FGD Report- Need Assessment Study -SACH in UP Behaviour Change Communication Strategy for National Rural Health Mission in Uttar Pradesh Assessment of The Social Franchise Project in Uttar Pradesh, India Rapid Assessment of Rogi Kalyan Samitis in Uttar Pradesh Evaluation of condom/OCP Social Marketing Program in Uttar Pradesh, 2009 Beneficiary satisfaction survey of voucher scheme in Kanpur Nagar Listing of health service provider/clinic in A & B villages in UP Baseline Survey of Nutrition and Reproductive Health Intervention of Saloni Project in UP Evaluation of Condom/OCP Social Marketing Program In Uttar Pradesh Increase in Access of Contraceptives in C and D Category Villages in Uttar Pradesh Medical Audit of Nursing Homes under Agra Voucher Scheme Reach Of Social Marketing Products In Category C & D Villages, Up (Penetration Study) Re-validation study note- SM Verification study of MGHN in UP Baseline Survey in Haridwar 114 Baseline survey in 6 project blocks of Uttarakhand (ASHA+Project) Baseline Survey in 6 Control Blocks Of Uttarakhand (ASHA+Project) Assessment of CHC for Public Health Standard Compliance Draft Model for adapting ASHA like worker scheme for Uttarkashi Assessment of ASHA Trainings BCC Needs Assessment in Uttarakhand Assessment of District level health facilities for Public Health Standard Compliance Rapid Assessment of the Functionality of FRUs and 24x7 PHCs in Uttarakhand Beneficiary Satisfaction Survey of Voucher Scheme In Haridwar Audit of Infant Deaths in Uttarakhand Medical Audit of Nursing Homes under Hardwar Voucher Scheme Assessment of Health Facilities in Chakrata & Yamkeswar Blocks For Contracting Out Baseline Survey in Yamkeshwar & Chakrota for Contracting Out Program Endline Survey in Haridwar for Voucher Program Baseline Survey in Hardwar for Voucher Scaling-Up Program Baseline Survey in Dehradun for Voucher Scaling-Up Program Baseline Survey in Nainital for Voucher Scaling-Up Program Baseline Survey in Almora for Voucher Scaling-Up Program Baseline Survey In Udham Singh Nagar For Voucher Scaling-Up Program Customer Satisfaction Survey for Voucher System-HARIDWAR Voucher Scheme for Improving RCH outcomes in Haridwar District, Uttarakhand Baseline Qualitative Research on Adolescent Health in Uttarakhand Baseline Quatitative Research on Adolescent Health in Uttarakhand Developing communication for Adolescent Health Programs in Uttarakhand Communication Needs Assessment for Scale up of Voucher Scheme (Qualitative) Assessment of ASHA Resource Centre in Uttarakhand Cost Effectiveness of Haridwar Voucher Scheme, Uttarkhand Assessment of MHVs in Uttarakhand Midline Assessment Of Adolescents Health Client Satisfaction Study Voucher Scheme Medical Audit Voucher scheme Endline Voucher Scheme In Uttarakhand 115 NRHM - Media Tracking Strategy Assessment of Condom/OCP Social Marketing Programs In India WHO SAGE study (International Institute for Population Sciences, Mumbai) Public Private Partnership In Health Care- Guidelines For Contracting Out To Private Organizations National Strategy for Social Marketing Agra Baseline Survey Household Questionnaire, 2006 Agra Baseline Survey Women’s Questionnaire, 2006 Agra Baseline Survey Children Age 0-4 Years, 2006 Quick Assessment of CHCs/BPHCs in Agra, 2007 Private Health Facilities Having In-Patient Facility in Agra District, 2006 Mapping Of Private Health Facilities in Agra District, 2006 Evaluation of Condom/OCP Social Marketing Program in Uttar Pradesh, February-March, 2009 Evaluation of Condom/OCP Social Marketing Program in Uttar Pradesh, February-March, 2010 Condom/OCP Penetration Survey in Uttar Pradesh, Aug-Sept 2006 Survey of Private Health Providers In A and B Type Villages In Uttar Pradesh, 2009 Evaluation of Condom/OCP Social Marketing Program In Uttar Pradesh, February-March, 2008 Kanpur Nagar Baseline Survey-2006 Household Questionnaire Kanpur Nagar Baseline Survey – 2006 Woman’s Questionnaire Kanpur Nagar Baseline Survey – 2006 Children Age 0-59 Months (0-4 Years) RH Indicator Survey –2003 IFA Woman’s Questionnaire, Uttar Pradesh IFPS II Baseline Survey (Up)-2005 Household Questionnaire IFPS II Baseline Survey (Up) – 2005 Woman’s Questionnaire Reproductive Health Indicator Survey, 2009-10 Woman’s Questionnaire ANNEX VI: ASSESSMENT OF PPP INTERVENTIONS UNDER IFPS PROJECT INITIATIVES ROLE OF PUBLIC SECTOR ROLE OF PRIVATE SECTOR 3RD PARTY INVOLVEMENT FINANCING RESPONSIBILITY 116 CONTRACEPTIV E SOCIAL MARKETING PROGRAM (CSMP) Procurement and provision of branded commodities Packaging and marketing subsidies Project oversight and financing Registered Social Marketing Organizations (SMOs For or Not for Profit) Marketing / distribution of contraceptives using commercial and community based channels Commercial pharmaceutical and FMCG trading channel Sell SM branded contraceptives to end beneficiaries Government: Costs of commodity, subsidies, and demand generation SMOs : Costs of sales overheads, brand promotion 3rd Party: Costs of space, stock holding and sales overheads MERRYGOLD SOCIAL FRANCHISING NETWORK (SF) Development of quality standards and guidelines for clinical protocol Generate generic demand for FP and MCH services Project oversight and financing Project management and financial controls Development and management and marketing of branded franchisee network Community mobilization Quality assurance Monitoring & Evaluation Network of registered, accredited private providers (medical practitioners, paramedics and non￾clinical health workers) Provision of FP & MCH services at subsidized fee Client mobilization for the network through community mobilizers Government: Costs of generic demand creation Franchisor: Costs of network management, marketing and organizational overheads 3rd Party: One time network joining fee, service royalty per client, partial overheads SAMBHAV VOUCHER SCHEME Development of quality standards and guidelines for clinical protocol and quality assurance Generate generic demand for FP and MCH services Project oversight and financing Project management Management of targeted voucher distribution Community mobilization for effective utilization n of the vouchers 3rd party reimbursements Registered, accredited private providers (medical practitioners, paramedics) with provision of FP & MCH services for subsidized fee Community health volunteers/ASHA/ USHA for client mobilization Government: Demand side financing Private Party: Nil 3rd party: Partial overheads MOBILE VAN Procurement of Van Development of quality standards and guidelines for clinical protocol and quality assurance Generate generic demand for FP and MCH services Project oversight and financing, Procurement of Van Management of van logistics and clinical team Community mobilization for effective of the van None Government: Partial to full costs Private Partner:– Van cost and partial overheads ASHA PLUS & ARSH UDAAN Project oversight and financing Development of training modules, Managing ToT and training logistics Provision of supportive supervision None Government: Full costs Private partner: Partial overheads 117 Monitoring ANNEX VII: IFPS III: DISTRICTS VISITED AND CODING UP DISTRICTS HIGH-INTENSITY INTERVENTION LOW-INTENSITY INTERVENTION CONTROL (BIHAR DISTRICTS) Code = 2 Code = 1 Code = 0 Agra Lucknow Araria Aligargh Ambedkar Nagar Arwal Allahabad Bahraich Aurangabad Auriya Barabanki Banka Azamgarh Basti Begusarai Baghpat Bhadohi Bhagalpur Balia Bijnor Bhojpur Balarampur Badaun Buxar Banda Bulandshahar Darbhanga Bareilly Deoria East Champaran Chandauli Etah Gaya Chitrakoot Faizabad Gopalganj Etawah Farrukhabad Jamui Fatepur Gautam Buddha Nagar Jehanabad Firozabad Ghaziabad Kaimur Gonda Ghazipur Katihar Gorakhpur Hamirpur Khagaria Hathras Hardoi Kishanganj JP Nagar Jalaun Lakhisarai Jhansi Jaunpur Madhepura Kanpur Nagar Kannauj Madhubani Kaushambi Kanpur Dehat Monghyr Maharajganj Lakhimpur Kheri Muzaffarpur Meerut Kushinagar Nalanda Mirzapur Lalitpur Nawada Moradabad Mathura Patna Rampur Muzaffarnagar Purnea Shahjahanpur Pilibit Rohtas Saharanpur Rae Bareilly Saharsa Sitapur Mahoba Samastipur Sultanpur Mainpuri Saran Unnao Mau Shiekhpura Varanasi Mirzapur Sheohar Pratapgarh Sitamarhi Sant Kabir Nagar Siwan Districts visited by team Sant Ravidas Nagar Supaul 118 Shravasti Vaishali Siddhartnagar West Champaran Sonbhadra Sultanpur Prabhuddha Nagar Panchsheel Nagar Bhim Nagar 119 ANNEX VIII: BCC INTERVENTIONS UNDER IFPS II & IFPS III1 STRATEGY DEVELOPMENT BCC CAMPAIGNS/MATERIAL DEVELOPMENT BCC CAPACITY BUILDING BCC IN SOCIAL MARKETING/ FRANCHISING Mass media Mid-media IPC/ Community Level NATIONAL Content Brief for the ‘Swasth Bharat’ Health Magazine Program FP Campaign 2004-09 FP Campaign 2009-10 (3 phases) NRHM advocacy film Atmajaa Tele Series Promotion of JSK Call Centre Menstrual Hygiene Campaign IUCD Campaign ‘ASHA se maango’ campaign - National IEC/BCC Workshop – 1 IFPS Technical Assistance Project (ITAP). 2012. Behavior Change Communication Activities and Achievements – Lessons Learned, Best Practices and Promising Approaches. Gurgaon, Haryana: Futures Group, ITAP. 120 STRATEGY DEVELOPMENT BCC CAMPAIGNS/MATERIAL DEVELOPMENT BCC CAPACITY BUILDING BCC IN SOCIAL MARKETING/ FRANCHISING UTTARPRADESH NRHM BCC Strategy 2008 Multimedia Sterilization Campaign 2004 Radio Series –Sanwarte Sapne-Sunahri rahen Comprehensive Poster on Family Planning ASHA News Letters Immunization – jachcha￾bachcha raksha card Folk Media - Street plays, puppet and magic shows Saloni Teachers Manual and Diary Distance Learning Program NRHM Flipbook for ASHA Saloni Teachers’ Training Manual Family Welfare Counselors Training Module Regional BCC capacity building Communication Plan for MGHN Brand Equity and Barrier Analysis Study Voucher Scheme Campaign UTTARAKHAND BCC strategies for specific programs: Mobile Health Vans Voucher Scheme Adolescent Health Institutional Delivery Campaign Immunization Campaign Sambhav Voucher Scheme Promotional Campaign UDAAN-Adolescent Health Campaign Voucher Scheme Demand Generation Mobile Health Van Demand generation ASHA Plus Program – – Jharkhand Health Communication Strategy Health Issues and Health Seeking Behaviors of Tribal Populations BCC Strategy for Voucher Scheme Spacing Campaign Institutional Delivery Campaign Voucher Scheme Demand Generation Street Theatre Campaign IPC toolkit for Sahiyas (ASHAs) Strategy for Intra￾communication - 121 ANNEX IX: SUPPORTING CHARTS CPR Significance Values State NFHS 2 1998- 99 DLHS 3 2007- 08 Difference in CPR (total over 10 years) Rate of Increase (per year for 10 years) Significance (Chi squared test) India 42.8 54 11.2 1.12 Reference Jharkhand 27.6 34.9 7.3 0.73 P >0.05, Not significant Uttarakhand 43.1 60.1 17 1.70 P >0.05, Not significant Chhattisgarh 45 49.7 4.7 0.47 P >0.05, Not significant 122 ANNEX X: IFPS DOCUMENT REVIEW CHECKLIST – AVERAGE SCORES Document Number (Refer to Reading Assignments) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Total 1. Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? 3.5 2.5 3 3 3 1.5 3.5 3 1 3 2 3.5 3.5 3.5 3 3 4 4 2 3.5 4 3 1.5 2.5 2 3.5 75.5 5. Does the report state the type of intervention that was being evaluated, its geographic coverage and project/interventio n time span (reference period) covered by the evaluation? 3.5 3 3 3 3 1.5 2.5 2 4 3 3 3 2 3.5 3 3 3 3 1.5 3 1 4 2.5 3 4 2 73 4. Does the report provide a clear description of the evaluation’s design, sample selection, where data was collected, and when the evaluation was conducted? 2 2 3 3 3 1.5 2 2.5 3 3 2 2.5 2 3.5 3 3 3 2 1.5 2 3 4 3 3 3.5 4.5 70.5 12. Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? 3 1.5 3 3 3 1.5 3.5 2 3 3 2.5 2 2 3 3 3 3 3 1.5 2.5 2 5 2.5 3 3 2.5 70 123 Document Number (Refer to Reading Assignments) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Total 13. Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? 3.5 3 3 3 3 1.5 3.5 1.5 3 3 3 3.5 2.5 3.5 3 2 3 3 1. 5 2.5 1 2 3 2.5 3 3 70 14. Does the evaluation report give the appearance of a thoughtful, evidence -based, and well organized effort to objectively evaluate what worked in the project, what did not and why? 3 2 3 3 3 1.5 1.5 1.5 3 3 2.5 2.5 2.5 3 3 3 3 3.5 1.5 2.5 1 4 3 3 1.5 1.5 65.5 2. Regardless of the type of evaluation, does the evaluation report reflect use of sound social science research methods? 2 2 3 2 2 1 1.5 2.5 3 3 2.5 2.5 2.5 3 3 2 3 3 1.5 2.5 2 3 2.5 2 3.5 4.5 65 11. Are data appropriately analyzed with percentages, ratios, cross -tabulations and statistical testing? 2.5 1.5 3 3 3 1.5 0.5 3 2 3 3 2.5 2 3 3 3 3 2.5 1.5 2.5 1 5 2 2.5 2.5 2 64 6. Does the report contain key research questions and address each key question around which the evaluation was designed? 3 2.5 2 2 2 1.5 2.5 2 3 3 2 2 2 3 3 3 3 2 1.5 2 1 4 2 3 3 3.5 63.5 124 Document Number (Refer to Reading Assignments) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Total 9. Is there a clear description of the evaluation’s data collection methods (summarized in the text with the full description presented in an annex)? 2.5 2.5 1 1 1 1 2 2 2 3 2.5 2 2 3 3 3 3 2 1.5 1.5 3 5 2 3 3 4 61.5 3. Does the Executive Summary concisely state the main points of the evaluation: summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation? 1 1 3 3 3 1.5 3.5 1 1 1 1 1 3.5 3.5 3 3 3 4 1.5 3.5 2 3 2 2 3 2.5 60.5 7. In answering the evaluation questions, does the report appropriately use comparisons made against baseline data? 2.5 2.5 2 2 2 1.5 2.5 2 3 3 1 2.5 2 3.5 3 2 3 3.5 1.5 2 1 2 2 3 1 1.5 57.5 125 Document Number (Refer to Reading Assignments) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Total 10. Does the evaluation report contain a section describing the limitations associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc)? 2 2 2 2 2 1 3 1.5 2 3 1 1.5 1 2 2 1 1 1.5 1.5 2 1 1 1 2 3.5 2.5 46 8. If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness? 2 1 3 2 2 1 4 0.5 3 2 1 1 2 2 1 1 1 3.5 1.5 2 0 1 1.5 2 1 1.5 43.5 EVALUATION REVIEW FACTOR 36 29 37 35 35 19 36 27 36 39 29 32 31.5 43 39 35 39 40.5 21.5 34 23 46 30.5 36.5 37.5 39 126 ANNEX XI: TEAM MEMBER CV’S SOCIAL IMPACT MICHÈLE M. ANDINA R.N., PH.D. Team Leader/Senior Technical Reproductive Health Analyst Dr. Michèle Andina, a nurse-anthropologist with more than 20 years of global experience with reproductive health and HIV/AIDS programs, was appointed in 2010 as a member of the Global Fund for AIDS, TB and Malaria Technical Review Panel. This year the panel processed and made recommendations on RH proposals from 150+ countries totaling $4.3 billion. Recently, she served as director of Pathfinder International’s Mukta Project, an $8.5 million HIV/AIDS prevention and care project in Maharashtra funded by the Bill and Melinda Gates Foundation, where she was directly responsible for project design, implementation and monitoring. In this capacity, she also worked with numerous public and private groups to enhance their capacity to deliver RH services. Dr. Andina has evaluated both large and small scale technical assistance programs throughout Africa, Asia and the Caribbean for USAID, The Population Council, and International Planned Parenthood Federation, among others. Possessing expertise in global RH strategies in the Indian context, she has served as Team Leader for an APAC Evaluation in Tamil Nadu, HIV/AIDS Specialist for a SAMASTHA Evaluation in Karnataka, and Public Health Specialist for SAMASTHA Lessons Learned in UP. Her extensive medical background (trained as a Pediatric Nurse Practitioner and Public Health Nurse) is augmented by strong statistical and financial analytic skills. Dr. Andina holds an MA in Nursing Education and a Ph.D. in Applied Anthropology from Columbia University. PROFESSIONAL EXPERIENCE 2011 Public Health and HIV/AIDS Expert, Global Health Tech, Final Evaluation of the USAID Samarth Program, India  Team member on the USAID/India Evaluation of the Samarth program  Focus on Technical Assistance to Government of India for HIV/AIDS prevention, care and support 2011 Health and HIV/AIDS Analyst, Social Impact, Final Evaluation of the USAID Samastha Program, Karnataka, India  Team member on the USAID/India Evaluation of the Samastha program  Developed data collection instruments, methodology and collected data in India  Authored final report 2010 Global Fund for AIDS, TB, Malaria, Technical Review Panel, Geneva  Served as member of Round 10 TRP reviewing and making recommendations on 150+ proposals 2005-2008 Director, Mukta Project funded by Bill and Melinda Gates Foundation, Pathfinder International, Pune, India 127  Directed and managed a team of 30+ individuals working with fifteen NGOs that included 200 staff and 250+ peer educators for a $8.5 million STI/HIV/AIDS prevention project in ten districts of Maharashtra State which was home to an estimated total of 25,000 at-risk individuals  Worked to develop the capacity of local NGOs to deliver and distribute behavioral change communication and to expand outreach and medical services  Empowered MARPs within the region through collectivization and health seeking behavior 2001-2005 Founder/President, Jagriti International, Santa Barbara, CA  Created, developed and implemented cultural and educational programs that supported the work of over a hundred grassroots women’s groups (CBOs) worldwide with a special focus on South Asia (Afghanistan, India, Nepal, Pakistan)  Responsible for the development of a website (http://www.jagriti-international.org) and developing additional methods of outreach to stakeholders  Responsible for guiding the vision and providing leadership to a variety of international non-profit organizations seeking to expand and refine their missions. 2000-2002 Program Consultant, Refugee Women in Development (REFWID), Washington, DC  Developed and enhanced the institutional capacity of non-governmental organizations in Afghanistan as a part of REFWID’s Long-Term Strategy for Rebuilding Afghan Civil Society.  Proposal development in collaboration with Afghan NGOs and worked to develop their missions; evaluated and trained Afghan-led NGOs in Pakistan and Kabul in organizational development 1997-2000 Project Director, JAGRITI: The Awakening, Santa Barbara, CA  Severed as leader and director for an independently-funded project, “JAGRITI: The Awakening - Women without Borders - A Trek to Nepal.”  Produced a documentary film about the empowerment of Nepalese women  Responsible for the development of project including raising funds, coordinating two expeditions to Nepal and organizing meetings with local organizations 1995-1997 Project Co-Director, Pacific Institute for Women's Health, Los Angeles, CA  Directed a multi-country evaluation to assess impact of activities of The Global Fund for Women on the empowerment of women and their demand for family planning.  Drafted proposals prepared and administered budget and managed the project for evaluation of women’s organizations in eight countries including Pakistan, Nepal, Turkey and Philippines. 1991-1995 Independent International Health Consultant  World Health Organization— Prepared and delivered report on Maternity Waiting Homes,1995  International Planned Parenthood Federation (IPPF) — Evaluated Haitian Family Planning Association, 1995  Ciba-Geigy Foundation/IAMANEH — Evaluated Sikasso, Mali Maternal Child Health (MCH) Project, 1994-1995 128  USAID- Private Voluntary Organizations for Health —Served as USAID Team Leader and performed midterm evaluation in India, 1994  USAID- Family Planning Management Development — Served as USAID Team Member and evaluated program in Kenya and Bangladesh, 1994  The Population Council — Safe Motherhood Demonstration Project, proposal development for World Bank Grant and preformed qualitative assessment in Vietnam, 1993-1994  REDSO/USAID-Family Health and AIDS — Served as Family Planning & HIV/AIDS Specialist for West and Central Africa Design Project, 1994  Global Health Action (INSA) – Instructed a course in project development and methodologies, 1993  World Health Organization – Evaluated Faisalabad Obstetric Flying Squad, Pakistan, 1993 PUBLICATIONS “Women’s Empowerment, Family Planning and Civil Society - Lessons Learned from Research with Women’s NGOs,” Dr. Barbara Pillsbury, Pacific Institute for Women’s Health and Dr. Michèle M. Andina R.N, 1997 "Is Two Better Than Too Many? - Reproductive Behavior of Rural Jamaican Women” (PhD dissertation) “Needle Park, Zürich, Switzerland, - Combating AIDS or Politicians?” presented at Annual Meeting of American Anthropological Association, 1990 “Is The Child Safe? - Collaborative Decision Making in Jamaica,” presented at Annual Meeting of Society for Applied Anthropology, 1989 “Is Two Too Many? - Discourse in a Family Planning Program in Jamaica,” presented at Annual Meeting of American Anthropological Association, 1987 “The Economics of Fertility: The Decision to Contracept in Rural Jamaica,” presented at Annual Meeting of Society for Applied Anthropology, 1985 “Pregnant Women and Ganja in St. Thomas, Jamaica,” Columbia University, 1985 LANGUAGES Fluent in Fluent in French, German, Swiss German and Italian; Conversant in Spanish EDUCATION Ph.D. —Applied Anthropology, Columbia University, New York, NY, 1993 M.A. — Nursing Education, Columbia University, New York, NY, 1985 B.S. — Public Health Nursing, University of California at San Francisco, San Francisco, CA, 1974 Pediatric Nurse Practitioner 129 130 SOCIAL IMPACT DIPANJAN SUJIT ROY, MD DNB MNAMS Senior Population Analyst Dr. Dipanjan Sujit Roy is a senior public health specialist, physician and epidemiologist evaluator, with 18 years’ experience in US government agencies (CDC and USAID), multi-lateral organizations (WHO,UNAIDS and UNICEF), Govt. of India (Armed Forces, ICMR, MOH&FW and NACO) and NGO sector (AHF, CINI, Alliance, FPAI and IPPF). Having started his career as medical officer in Armed Forces of India, he diversified his field to work in Maternal and Child Health programs with CINI and UNICEF in innovations for IUD use and spacing methods and emergency obstetric care in West Bengal. He is currently serving an advisory role for NRHM 2 and RCH 3 on epidemiological analysis and evidenced based programming, and is analyzing the maternal health data for Uttar Pradesh and Jharkhand. He has also been associated with advisory work on innovations approach with newer contraceptives such as intelligent IUDs (AIMU My Cu IUDs) and Persona® through ICMR. Currently, he is the lead evaluator for a multi-centric Japan Trust Fund funded study for Family Planning Association of India (FPAI) on access to maternal health services for FSWs in the four Indian states of Maharashtra, West Bengal, Nagaland, and Tamil Nadu. He possesses expertise in the design, start-up, and scaling of large, donor-funded programs in reproductive health, family planning, and HIV/AIDS. He is familiar with USAID programming, having participated in the APAC review as a member of the technical expert group for FSW TIs, as well as a senior technical advisor on the SAMARTH lessons learned. Dr. Roy has strong management experience, including direct implementation projects in 15 states of India, and has served as a member of the design and implementation teams for the National AIDS Control Program Phase III and now Phase IV in India. Dr. Roy has contributed to various manuals on HIV/AIDS, TB-HIV, TB and RCH for the Government of India. Dr. Roy contributed towards health systems strengthening in India with the development of the 2012 Indian Public Health Standards, has been associated with Human Resources for Health through Public Health Foundation of India since 2008, and has been appointed the Principal Investigator for Evaluation of Heath Systems in the Kingdom of Bhutan. He is proficient in epidemiological modeling and quantitative methods of statistical analysis, including use of numerous statistical software packages. In 2009 he was awarded the “Global Health Achievement” award by CDC in 2009 for his contributions towards public health and has received the “Extra Mile Award” in 2011 from US Embassy, New Delhi. 131 EDUCATION 2003: Diplomat of National Board (DNB), Social and Preventive Medicine & MCH, National Board of Examinations, India 2002: MD (Social and Preventive Medicine), All India Institute of Hygiene and Public Health, Kolkata, University of Kolkata, India 1991: MBBS, Armed Forces Medical College, Pune, India QUANTITATIVE PROFICIENCIES Expert level in statistical software –SPSS, SAS, STATA, Spectrum, Analyze It Advanced level in epidemiological modeling with MATH LAB PROFESSIONAL EXPERIENCE Sept 2011 – present Independent Public Health Consultant  Lead evaluator, Family Planning Association of India's project on addressing stigma for key population and PLHIVs and access to reproductive health services.  Principle Investigator, Health Systems Strengthening and Evaluation for Kingdom of Bhutan, Alliance Technical Hub for South Asia.  Programmatic and M&E consultant on HIV and TB for LFA Price Waterhouse for The Global Fund to fight AIDS, Tuberculosis and Malaria.  Senior Lecturer for Institute for Advanced Medical Studies in Epidemiology, Infectious Diseases and Biostatistics.  Evaluation, lessons learned and implementation work for various USAID contractor viz. GHTECH, Deloitte and KPMG and Social Impact. Evaluation experience includes: o Joint Implementers Review for NACP III, Member since 2007 o Technical Expert, "Lessons Learned - SAMARTH Project" of FHI (HIV Technical Assistance Project to Govt. of India and other partners) o Technical expert, APAC review of FSW TIs  Freelance work for WHO, Geneva on HIV-TB coordination  National AIDS Control Program Phase III and design for phase IV, contributor for Care and Support, M&E, TIS and RCH convergence Feb 2010 – Aug 2011 Public Health Specialist, Centers for Disease Control and Prevention, Department of Health and Human Services, US Embassy, New Delhi, India  Technical assistance for multi-centric research studies on influenza burden in India (Implemented by ICMR and AIIMS).  Provided epidemiological support for surveillance activities on Influenza in India. 132  Liaison with Govt. of India, State Govts, UN agencies, Bilaterals and Private Sector  Handled a grant of 3.7 million for various projects in India Sept-Dec 2009 Director Operations – Asia Pacific Region (Short Term Assignment), AIDS Healthcare Foundation Global, New Delhi  Streamlined AHF operations in Asia Pacific region viz. India, China, Thailand, Cambodia, Vietnam and Nepal  Supervised 15 country coordinators along with staff in six countries.  Handled a budget of 1.5 million dollars in Asia Pacific region. 2004 – 2009 Medical Epidemiologist, Centers for Disease Control and Prevention, Department of Health and Human Services, US Embassy, New Delhi, India  Provided technical, managerial and coordination support for HIV, TB and other infectious diseases.  TB-HIV review – RNTCP, member review panel since 2002  Technical assistance for TI among FSWs (CINI, Jharkhand), APAC (USAID), Bill and Melinda Gates Foundation’s Avahan project for Northeast; IDU intervention in Manipur and MSM interventions in Dimapur, Nagaland  CARE SAKSHAM review – Technical member for evaluation team  Lead Consultant, Public Health Foundation of India’s study on Maternal deaths, Child malnutrition and Child immunization  Lead evaluator, Child-in-Need Institute’s ICCHAA & DASTAC projects (reproductive health projects on family planning and HIV).  Technical advisor for convergence with sexual and reproductive health program for Ministry of Health & Family Welfare, Govt. of India (ICMR and Dept. of Family Welfare, RCH division)  Technical assistance for FSW targeted intervention project, Indian Health Organization/People’s Health Organization 2003-2004 Medical Consultant, HIV-TB, World Health Organization, India  Designed, adapted, piloted and scaled up TB-HIV coordination in high prevalence HIV states in India.  Overall in-charge for TB control implementation for fourteen states of India. 2000-2003 Research Officer and Senior Research Fellow, Indian Council of Medical Research, India  Clinical researcher and epidemiologist in HIV/AIDS (NARI).  Technical Advisory board members for evaluations for Family planning innovations with CINI. 133  Epidemiologist for Phase II and III trials for 'female condoms', AZT trials and HPTN 052 and 047 studies and vaginal microbicides.  “Sonagachi” project – Evaluator and Technical assistance 1997-2000 Medical Officer and Post Graduate Trainee, Satyanand Hospital and All India Institute of Hygiene and Public Health, India  Post graduate training in Public Health (MD)  Pilot project to roll out Basic Emergency Obstetric Care in South 24 Parganas of West Bengal (Kolkata) with UNICEF and CINI  Assessment of child nutritional  STD HIV Intervention Project – medical officer 1992-1996 Medical Officer, Indian Army, India  Clinical and public health provider for armed forces personnel across India RELEVANT CONSULTANCIES July 2012 Lead external evaluator FPAI/IPPF  Japan Trust Fund project, "Addressing stigma and positive prevention among PLHIV and key population" - a convergence project on Reproductive health and HIV in four states of India Feb - Mar 2012 Public health management consultant SI/USAID  Evaluation team - APAC project by VHS, Chennai (USAID) supported Nov 2011 Proposal writer, Family Planning Association of India/IPPF  "Migrant workers - a comprehensive HIV and Reproductive health intervention in Chennai Metro Rail project", to Japan Trust Fund (USD 170,000) Sept-Nov 2011 Senior HIV Care and Support Team Member, GHTECH, USAID/India  Lessons Learnt Team - SAMARTH Project by FHI (USAID supported) July 2011 Public Health Consultant, WHO, Geneva  Developed report and presentation for “The inclusion of HIV/TB Collaborative Activities in Global Fund Proposals: An examination of select Round 8 -10 Global Fund funded HIV and TB proposals” 134 Mar 2010 – present Program and M&E Consultant GFATM with LFA Price Waterhouse  Monitoring, Performance Updates, On Site Data Verification and Rapid Service Quality Assessment for GFATM HIV and TB projects in India Dec 2010 – Jan 2011 Technical Consultant, India HIV/AIDS Alliance  Created baseline assessment report and policy briefs for Sexual and reproductive health and rights for marginalized population. 2009 Technical Consultant, Public Health Foundation of India  Review of Maternal and Child Health situation in India SELECTED PUBLICATIONS Contributed two chapters in MoH&FW’s book – Epidemiology and TB-HIV Co-infection in “Tuberculosis Control in India”. Can be accessed at http://www.tbcindia.nic.in/pdfs/Tuberculosis%20Control%20in%20India-Final.pdf. Broor S, Krishnan A, Roy D S et al. Dynamic patterns of circulating seasonal and pandemic 2009A(H1N1) influenza viruses from 2007-2010 in and around Delhi, India. PLoS One. Submitted June 2011. Roy D S. Advocacy and Policy Analysis Programme for Public Health in India - Review of Policies and Programmes on Maternal Mortality and Child Health with focus on Malnutrition and Immunization – A Report. Public Health Foundation of India 2010. 135 SOCIAL IMPACT RAHUL DEV BHAWSAR, PH.D. Demographer Dr. Rahul Bhawsar is an accomplished demographer with over 15 years’ experience in conducting statistical analyses and modeling in the RH/FP field for both the public and private sector. He currently serves as an HMIS expert with DFID-funded Madhya Pradesh Technical Assistance Support Team, where he provides technical support to the state government on M&E, the use of HMIS data, and the improvement of data quality through third party data validation and triangulation. Previously, he worked with government and large donor-funded programs to conduct statistical analyses of RH/FP health indicators among various populations throughout India. He also has extensive experience working closely with government departments (both at state and national level) to provide support to health systems strengthening initiatives, as well as the establishment of MIS and M&E systems for the improved planning, implementation and monitoring of programs focused on underserved areas. A skilled demographer, Dr. Bhawsar has a strong foundation in quantitative analysis of primary and secondary data sets such as DLHS, census data, baseline/evaluation studies, and large-scale surveys for RH programs funded by USAID, UN, UNICEF, UNFPA, World Bank, Oxfam, and MacArthur Foundation, among others. He gained fluency in applied statistical analysis of large-scale survey data through his use of multiple regression analysis on India’s DLHS-3 data, and presented his findings at the 6th Asia Pacific Conference on Adolescent Sexual Reproductive Health and Rights. Dr. Bhawsar has conducted over 40 research studies for national and international stakeholders, holds 10 publications in national health and development journals, and has trained hundreds of development professionals in demography and M&E methodologies. He is highly knowledgeable in SPSS, EpiInfo, and CMIS (under NACP III of NACO) statistical software. RELEVANT PROFESSIONAL EXPERIENCE 2008 – 2012 HMIS Expert, Madhya Pradesh Technical Assistance Support Team (MP TAST), DFID Health Sector Reform Program, Madhya Pradesh, India  Provided technical expertise to the department of public health and family welfare and department of women and child development to improve the M&E system and build institutional evaluation capacity  Developed hospital MIS reporting format; operationalized HRDMIS software system for reporting by physicians  Designed and implemented ICT tools and survey instruments, including rapid assessments, baselines and evaluation studies in the areas of MIS, reproductive health, and nutrition  Analyzed data generated from routine MIS and other data sources; developed and supported innovative applications for data collection tools (e.g. mobile phone tracking of malnourished children)  Projects completed/ongoing: Study on Data Verification of information generated from Routine MIS of Department of Women and Child Development, Govt. of Madhya Pradesh, 2012; Rapid Assessment of Functionality of CEmONC/BEmONCs in the state of Madhya Pradesh, Govt. of Madhya Pradesh, 2011; Evaluation of Deendayal Mobile Health Clinic Scheme in Madhya Pradesh, Govt. of Madhya Pradesh, 2009 136 2005 – 2008 Manager of Research and Evaluation, EPOS Health Pvt Ltd, New Dehli, India  Coordinated country-wide research and M&E initiatives of EPOS India, including large-scale, multi-state surveys  Developed and conducted training program on research methods for network of partner organizations  Provided technical support for the design, implementation and monitoring of projects, provided technical expertise to ongoing public health projects under National Rural Health Mission (NRHM)  Prepared technical reports and developed research protocols using SPSS  Projects completed: District Level Household and Health Facility Survey (DLHS 3) in Jammu & Kashmir, Govt. of India, 2008; Study on Validity Checking of Data for Hospital Performance Indicators in Rajasthan, Govt. of Rajasthan, 2008; NRHM Punjab – Preparation of State Action Plan and District Action Plans, Govt. of Punjab, 2007 1997 – 2005 Senior Research Officer, Indian Institute of Health Management Research (IIHMR), Jaipur, India  Coordinated "End line/Baseline Survey in 5 districts of M.P. under Country Program-6", sponsored by UNFPA, to generate data on RCH indicators including accessibility of RCH services for adolescents  Supervised VIKALP-MP project in Madhya Pradesh that integrates family welfare projects into broader district-wide RH programs; provided technical support, project implementation, and M&E  Instructed staff in demography, research methodology and M&E methods  Coordinated projects throughout the health and population sector, including project design, implementation, research, monitoring and supervision of large-scale surveys for family health programs  Conducted independent evaluation studies of various MCH/FP service delivery projects implemented by SIFPSA (USAID-funded) and the MacArthur Foundation  Trained staff members and organization partners in demography, research methodology, and M&E 1994 – 1997 Research Executive, Centre for Operations Research and Training (CORT), Gujarat, India  Developed M&E tools and developed methodology for project design and implementation  Conducted data management and analysis for the development of technical reports and proposals  Conducted project M&E  Supervised and managed research team RELEVANT CONSULTANCIES  UNFPA India — Served as Resource Person for training of Data Managers and Statistical Officers in service design and M&E under the U.N. Joint Convergence Program, 2012  UNFPA India — Analyzed data and produced report for baseline and end line study of health-seeking behavior of adolescents under CP6 program (district-wide RH program), 2007 137  SUTRA (Social Upliftment Through Rural Action) — Served as Resource Person for training of health workers and mid-level managers in quantitative and qualitative research methods in Himachal Pradesh, 2003  World Vision — Conducted nutritional assessment and data analysis of Bangladeshi migrant rag￾picking community in Jaipur, 2003  SIFPSA – Conducted surveys, analyzed data, and developed technical reports for IFPS project funded by USAID, Uttar Pradesh, India, 1999-2004 RELEVANT PROJECTS  DFID India (2009): Review of Using of PDA/Mobile Phone Approach for improving health and nutrition systems  OXFAM (INDIA) TRUST (2007): Behaviour Surveillance Survey for Evidence based planning leading to district level response to HIV/AIDS in Bharatpur District of Rajasthan  The World Bank (2007): Review of HMIS under World Bank Financed Sector Projects in India  UNICEF (2006): A Study on Institutional deliveries: Why Women in Madhya Pradesh Choose This Option  UNICEF (2006): A Study on Process Documentation of Dhanwantari Block Scheme in Madhya Pradesh  UNICEF (2006): Routine Immunization Coverage Evaluation Survey in Bihar  CARE India (2005): Estimating Outcome and Process Indicators Under INHP-II Through Rapid Assessment in Madhya Pradesh, Jharkhand and West Bengal  UNFPA (2004): Endline/Baseline Survey for IPD Districts in M.P. and Rajasthan under UNFPA Country Programme-6  MAC ARTHUR FOUNDATION, USA (1999): Meeting Information Needs of Men and Women on Sexual Health in Ajmer And Udaipur Districts of Rajasthan, Implemented by IIHMR.  UNICEF (1995): Multi-Indicator Cluster Survey in Gujarat - An Evaluation Study of Child Survival and Safe Motherhood Indicators in Six Districts  FAMILY PLANNING ASSOCIATION OF INDIA (1994): Baseline Survey for Innovations in Family Planning Program in Madhya Pradesh - A Case Study of Bhopal, Sagar and Vidisha Districts SELECTED PUBLICATIONS  Health Information System in India: Issues of Data Availability and Quality, Demography India, Vol. 39, No. 1, 2010, pp. 111-128.  Determinants of RTIs/STIs among Women in Punjab and Their Health Seeking Behaviour, The Journal of Family Welfare, Vol. 51, No.1, June 2005.  Population Ageing in India : Demographic and Health Dimensions, Indian Journal of Gerontology, Vol. 15, Nos. 3 & 4, special issue, 2001. RELEVANT SKILLS SPSS, EpiInfo, and CMIS (under NACP III of NACO) statistical software; MIS design and implementation; analysis of complex sets of primary and secondary data LANGUAGES 138 Fluent in English and Hindi EDUCATION Ph.D. — Demography and Population Studies, International Institute for Population Sciences (IIPS), Mumbai, India, 2004 M.Phil. — Population Studies, IIPS, Mumbai, India, 1994 M. P.S. — Post-PG Specialization in Population Studies, IIPS, Mumbai, India, 1993 M.Sc. – Statistics, Vikram University, Ujjain, India, 1991 B.Sc. – Vikram University, Ujjain, India, 1989 139 SOCIAL IMPACT SOUMITRA GHOSH Public Private Partnership Specialist Mr. Soumitra Ghosh is a Public Private Partnership Specialist with 28 years of experience in program management, social marketing and behavior change communications, strategy planning, and research with for-profit and non-profit multinationals—namely Marie Stopes International, Population Services International (PSI), J Walter Thompson (JWT), McCann Erickson, BEI PRISM, ADMAR and Gfk-MODE. Highlights include12 years as Country Director and Technical Services Director for USAID, CDC, DFID funded SRH programs with over 400 staff in South Asia and Africa. Mr. Ghosh has worked extensively in South Asia, Pacific Asia, Africa and Latin America on sexual and reproductive health, maternal and child health, child survival and malaria prevention including designing and implementation of UNICEF’s national WATSAN (Water & Sanitation and personal hygiene; diarrhea prevention and management) project in Bangladesh. He was global head of Marie Stopes International for Integrated Marketing leading MSI’s Social Marketing and Behavior Change Communication initiatives across 44 country operations covering family planning and safe abortion with special emphasis on Long Acting and Permanent Method (LAPM), Medical Abortion (MA) and Post Abortion Care (PAC); demand generation for MSI’s clinics and its Outreach activities and Blue Star Social Franchising network. Mr. Ghosh also served as MSI Representative in India overseeing MSI’s Delhi based regional office with 4 expatriates as well as the Founder, Director, and Chairperson of Marie Stopes India, an Indian NGO registered under section 25, with a total staff strength of 185 (100 of them being clinical and technical staff) delivering over a million CYPs in 2010 alone. PROFESSIONAL EXPERIENCE Current Independent Consultant Consultants  Part time (10 working days per month) Chief Advisor of Profam Healthcare, private healthcare marketing company with annual turnover of INR 80 million, responsible for leading the 60 member team to develop, implement and achieve strategic goals.  Part time (10 working days per month) Chief Marketing Advisor of the Consumer Product Division of Dey’s Medical, an Indian pharmaceutical and consumer product company with annual turnover of INR 1500 million, responsible for leading the 100 member strong sales and marketing team to develop, implement and achieve strategic goals for its Keo Karpin range of products.  STTAs in RH sector: a. Assessment to identify and prioritize appropriate business strategies in India for Abt Associates USA (30 working days spread over 3 months starting August 2012) b. Rapid assessment of RH unmet needs, government priorities, challenges, role of private sector in Ghana for Abt Associates USA (15 working days spread over 2 months starting September 2012) 140 Feb 2009 – Dec 2011 Associate Director & Global Head of Integrated Marketing, MSI Representative in India  Building, nurturing and leading technical and operational excellence in social marketing and behavior change communications across 44 country operations for family planning and safe abortion services of MSI -- 10 senior managers reporting directly of which 4 are expatriates.  Managing a for-profit global sales organization, Profam Healthcare, with annual turnover of USD 2 million and 60 sales staff in South Asia and Africa regions.  Overseeing MSI’s Delhi based regional office with 4 expatriate staff and as the Founder Director and Chairperson of Marie Stopes India, an Indian NGO registered under section 25, with a total staff strength of 185 (100 of which is clinical and technical staff).  Undertaking innovative projects - like Total Market Initiative (TMI), conducted for Madagascar in collaboration with The Futures Group and UNFPA with support of Innovations Fund of Reproductive Health Supplies coalition (RHSC)  Social marketing and BCC training under USAID funded SIFPO (Support for International Family Planning Organizations) project. Jun 2007 – Jan 2009 Director, Global Operations and Chief Execitive of India, BEI PRISM  Responsible for setting up and overseeing PRISM’s business operations in India and the Middle East specializing in Marketing Research, Strategy Planning and Strategic as well as Social Marketing consultancy for USAID, PSI, NACO/BMGF, Pathfinders International, MSI, Godfrey Phillips and many others.  For USAID, conducted pre-feasibility assessment of reach, capacity and motivation of existing and potential commercial sector organizations to integrate market based approaches in expanding SRH provision among the BOP population segments in India. Mar 2004 – Apr 2007 Country Director, Botswana, Population Services International (PSI)  Managed PSI’s Botswana program that focused on HIV/AIDS Prevention (covering targeted Condom Social marketing, Tebelopele VCT social franchising network and multi-media BCC initiatives promoting delayed sexual onset and partner reduction as well as high risks of HIV with alcohol and workplace intervention), Safe Water, Malaria Prevention funded by US-CDC, ACHAP (BMGF), Dutch, EC and PSIDF, with a team strength of 72 staff of which 10 were senior management staff;  Responsible for handling donor and stakeholder relations, fund raising and management, providing technical guidance to the team, meeting donor deliverables, ensuring documentation and showcasing of health impact made by the program.  Collaborated with the public and private sector to harness the strengths of each to ensure consistent access to quality SRH products and services. 141 Aug 2001 – Mar 2004 Technical Advisor & Director Technical Services, Zimbabwe, Populations Services International (PSI)  Building, nurturing and leading technical and operational excellence of USAID and DFID jointly funded PSI’s flagship Zimbabwe program (under a cooperative agreement) with a team strength of over 400 staff comprising: o New Start VCT social franchising network of 20 centres o Social marketing of Protector Plus male condom and Care female condom o Behavior change communications for HIV prevention among youth o PROFAM Social franchising network of 1100 medical service providers and pharmacies offering family planning, MCH o Workplace intervention for SRH services  Supporting local capacity building in the areas of marketing research, monitoring and evaluation  Handling donor and stakeholder relations, fund raising, meeting donor deliverables, ensuring documentation and showcasing of health impact made by the programs  Managing USAID funded regional project Corridors of Hope, targeting truckers, young women, cross border traders along the Durban – Lusaka corridor covering South Africa, Zimbabwe, Namibia, Swaziland, Lesotho and Zambia.  Collaborated effectively with the US Ambassador’s initiative and World Food Programme (WFP) activities to ensure access to remote rural areas of Zimbabwe.  Collaborated with both the public sector as well as private sector including an agreement with The Coca Cola Company to carry condoms to remote rural areas in their trucks as part of a CSR agreement signed between Coke and UNAIDS to ensure consistent access to quality SRH products and services in the midst of unprecedented economic crisis. May 1999 – Aug 2001 General Manager Marketing, India  Managing Social Marketing, Behavior Change Communications, Marketing Research, Monitoring and evaluation and Public Affairs functions for PSI’s India program with over 1000 professional staff that focused on Family Planning, Child Survival and HIV/AIDS Prevention funded by DFID, KfW, Packard Foundation;  Providing technical guidance to the technical services team, handling donor and stakeholder relations, fundraising, managing funds, meeting donor deliverables, ensuring documentation and showcasing of health impact made by the platform.  Collaborated with USAID’s Goli Ke Humjoli project with CMS (Commercial marketing Strategies) in India. Mar 1997 – May 1999 Director Strategy Planning, Thompson Social, J. Walter Thompson, (JWT), India 142  Managing strategy planning, implementation and monitoring & evaluation of all Behavior Change Communication projects within and outside India; new business development within and outside India (South-East Asia).  Designing and implementation of UNICEF’s national WATSAN (Water & Sanitation and personal hygiene; diarrhea prevention and management) project in Bangladesh. May 1995 – Mar 1997 Director and Chief Executive Officer, ADMAR Research, India  Managing India operation of ADMAR a Market Research company with 120 staff spread across 5 branch offices and 8 field offices.  Providing leadership to the organization through a long term vision and strategic plan to achieve business goals and objectives; setting annual goals and objectives and providing opportunity, ability and motivation among the staff to help achieve them.  Managing and overseeing financial and statutory/legal aspects of the business, handling public relations, new business development, providing technical guidance to the team. Oct 1988 – May 1995 Director of Strategy Planning and Head of Research Responsible for managing strategy planning, implementation and research of all key national clients within India; new business development within India. Apr 1984 – Oct 1988 Manager of Calcutta Branch, MODE, India Starting as a Research Executive handling research project executions, went through various levels of Sr. Research Executive, Project Director to become the Branch Manager responsible for managing business generation, supervising execution of all research projects, ensuring budgetary control to meet profitability targets, training and development of 100 strong staff compliment. RELEVANT CONSULTANCIES Oct – Dec 2007 Pre-feasibility assessment of and identification of potential partners for USAID’s Market Based Partnership approaches to promote SRH in India, PSI 2 month assignment for assessing reach, capacity and motivation of existing and potential commercial sector organizations to integrate market based approaches to expand SRH provision among the BOP population segments in India. Aug – Dec 2008 Assessment and development of National Social Marketing Strategy for condoms in India, Hindustan Latex Family Planning Promotion Trust for National AIDS Control Organization (NACO), Ministry of Health & Family Welfare, Govt. of India 143 2 month assignment to undertake SWOT of national condom social marketing program in India and to develop a robust strategy framework to meet GoI’s 2015 goals Feb 2008 – Feb 2009 Develop and introduce a tool for Evidence Based Decision Making across MSI global partnership, Marie Stopes International (MSI), London A year-long assignment to develop a management tool to prioritize, collate and integrate evidences related to clinical as well as non-clinical performance, finance, productivity from various channels of product and service delivery through a pilot in 8 countries across MSI’s global partnership. Feb 2008 – Feb 2009 Research, Monitoring & Evaluation support to MSI’s country programs in Africa and Latin America for evidence based decision making, Marie Stopes International (MSI), London A year-long assignment to assist MSI’s country programs in Africa and Latin America region in identifying, designing, implementing, reporting clinical and non-clinical research to aid programmatic decision making as well share best practices internally and externally. Sept 2008 Assessing and redesigning social marketing program in Madagascar, PSI, Madagascar 2 week long rapid assessment to restructure and redesign PSI Madagascar’s social marketing program Jul – Oct 2007 REsulTS Capacity Building for Health Impact: Sales and Distribution Course development, PSI, Washington DC 4 month long assignment for designing and writing three sessions on sales and distribution as part of the evidence based social marketing course aimed at developing capacity of Sales and Marketing personnel of PSI and its partner organizations. May 2007 Botswana: Program Management and Oversight, PSI, Washington DC 2 week long assignment involving management and oversight of PSI’s Botswana country program as well as designing and launching of PR support for a behavior change communication program on condom use in high risk behavior e.g. alcohol and sex, intergenerational sex, concurrent partnership and serial monogamy. Apr – Jul 2007 Zambia: Social Marketing of Condoms, Hormonal 144 Contraceptives, Safe Water and ITNs, PSI, Washington DC 2 week long assignment involving a thorough review of PSI/Zambia’s sales and distribution infrastructure, management process, monitoring tools, physical distribution system, transportation system; diagnosing the bottlenecks; recommending solutions to overcome such bottlenecks so as to improve operational efficiency as well as overall sales volume, distribution coverage and quality of trade servicing. Apr – Jul 2007 Lesotho: Social Marketing of Condoms, PSI Washington DC 1 week long assignment involving management and oversight of PSI’s Botswana country program as well as designing and launching of PR support for a behavior change communication program on condom use in high risk behavior e.g. alcohol and sex, intergenerational sex, concurrent partnership and serial monogamy. Apr – Jul 2007 Swaziland: Social Marketing of Condoms, New Start VCT services, PSI, Washington DC 2 week long assignment to a) review PSI/Swaziland’s sales and distribution targeting strategy, management and control tools, make specific recommendations to improve sales volume, distribution coverage and quality of servicing and b) provide technical guidance for designing New Start VCT follow up communication campaign. Jan 2004 Botswana: Social Marketing of Condoms, PSI, Washington DC 2 week long assignment reviewing PSI/Botswana’s sales and distribution infrastructure, strategy, management process, monitoring tools, physical distribution system, transportation system; diagnosing the bottlenecks; making recommendations to overcome such bottlenecks so as to improve sales efficiency as well as sales volume, distribution coverage and quality of servicing. Feb – Mar 1999 India: Strategic Sustainability Planning Exercise, PSI, India 4 weeks’ assignment to conduct internal and external stakeholder interviews, analyze and present SWOT outputs and recommended alternative approaches to plan long term sustainability by achieving better health impact, building organizational capacity and ensuring financial viability among both PSI India staff as well as external stakeholders. EDUCATION 1983: Masters in Business Administration (MBA), Indian Institute of Social Welfare & Business Management, University of Calcutta, India 1981: Masters in Economics, Birla Institute of Technology & Science, Pilani, Rajasthan, India 145 146 Jenny Ruducha Management Systems International A subsidiary of Coffey International, Ltd. 600 Water Street, SW Washington, DC 20024 (202) 484-7170 Proposed Position: Evaluation Methods Specialist (international) Experience Summary: Dr. Jenny Ruducha brings 34 years of public health, development and evaluation experience, including two dozen years of experience with maternal, newborn, child health (MNCH) including reproductive health (RH) interventions. She has conducted numerous assignments in India, including living in India for over eight years. Most recently in Uttar Pradesh, she has led quantitative evaluations and operations research of MNCH and RH interventions, using experimental and quasi-experimental design methodologies. She is well-versed in USAID’s objectives, approaches, and operations, having served three years as USAID/India’s Research and Evaluation Specialist and consulted for a variety of USAID/India-funded projects, including under GH-TECH. Dr. Ruducha also has extensive knowledge about USAID operations in India including public-private partnerships. For example, she evaluated the Government of India National AIDS Control Organization initiative for Children Affected by AIDS and assessed the effectiveness of a new Integrated Care Model of Drop-in Centers in Maharashtra. She also conducted a Mid-Term Review of MCH-STAR, involving a public-private partnership between USAID and State Innovations in Family Planning Services Project Agency (SIPFSA) to build capacity in Indian institutions. In addition, Dr. Ruducha’s health policy work includes a review of India’s national health policy on improving maternal and newborn health programming and an evaluation of Vermont’s health care reforms on women’s access to health care as part of state legislation on health insurance. Dr. Ruducha holds a Dr.PH in Health Policy from Johns Hopkins University. Specifics of her experience relevant to her proposed position on the IFPS evaluation are presented below. Experience with Impact Evaluation Methodologies, including Quasi-experimental Designs (QED) Dr. Ruducha has led nine impact evaluations using QED methods, in addition to an experimental evaluation. Most recently, she served as the Technical Lead directing the development and implementation of six applied research and evaluation projects in four states in southern India and Delhi. She also developed impact evaluation methodologies, tools, quality assurance of survey implementation, analysis plans and reports for the project, To Change Health Behaviors and Improve Coverage of Health Services by Activating Social Platforms for the Poor in Uttar Pradesh. Dr. Ruducha also designed a QED evaluation of the Program in Appropriate Technology for Health in Indonesia and India, which focused on maternal and newborn health. She developed a randomized field trial impact evaluation of the Sure Start program, which focused on newborn care in Uttar Pradesh. The evaluation involved 7 districts, 18,000 mothers from 700 villages, and 2,000 qualitative interviews with ASHAs, ANMs and AWWs (AAAs), including a verbal autopsy of maternal and newborn deaths. In 2005- 2006, Dr. Ruducha used the QED approach to evaluate Save the Children’s Saving Newborn Lives Program in India, focusing on neonatal interventions. Experience in Strategic Planning, Management, Operations Research, and/or Monitoring and Evaluation of Global and National RH Programs Dr. Ruducha has been conducting M&E and other research on RH programs for 24 years. Earlier in her career, 147 she served as Research and Evaluation Specialist in USAID, New Delhi for 3 years, designing and planning bilateral project evaluations of health projects such as, Integrated Child Development Scheme, Family Planning, Communication and Marketing. Later, she evaluated a US-based RH program, the Family Home Visiting Program for high-risk, minority pregnant women and newborns in 23 health services delivery areas throughout Massachusetts. She also conducted assessments and research to support strategic planning for the New Hampshire Department of Health and Human Services’ Bureau of Maternal and Child Health. More recently, Dr. Ruducha conducted Mid-Term Reviews of the USAID/India-funded Maternal Child Health Sustainable Technical Assistance Research Project and the Intrahealth Vistaar Program in Maternal, Neonatal, Child Health and Nutrition. Experience in Designing and Conducting Comparison Group Analysis, Statistical Modeling Techniques, Secondary Literature Reviews and Developing Sampling/Survey Methodologies As a published expert with more than 30 years of research experience, Dr. Ruducha is well-versed in both quantitative and qualitative research methods. She designed and conducted comparison group analysis for the following studies:  Ongoing impact evaluation and operations research of the To Change Health Behaviors and Improve Coverage of Health Services by Activating Social Platforms for the Poor in Uttar Pradesh  Six applied research and evaluation projects for India’s Orphans and Vulnerable Children’s Project  Impact evaluation of the Program in Appropriate Technology for Health’s Sure Start program in Uttar Pradesh  Evaluation of the Saving Newborn Lives I Program in India  Mid-term assessments of the Saving Newborn Lives II Program in Bangladesh and Pakistan Dr. Ruducha used statistical modeling techniques to analyze data for the evaluations listed above, as well as for the following studies. She typically uses STATA and conducts regression analysis, t-tests, chi-square tests, and propensity-score matching as well as UCINet for organizational network analysis  Maternal and infant health needs assessment, including county variation in health indicators by analyzing the New Hampshire Birth Registry  Evaluation of Family Home Visiting Program in Massachusetts  Post-Doctoral research on the social determinants of health and use of health services of migrant children, using original migrant data  Evaluated the efficiency and cost-effectiveness of programs serving the elderly for the Maryland Department of Health and Mental Hygiene Dr. Ruducha has used secondary literature reviews in all of her research, dating back to 1981. She has developed sampling/survey methodologies for the following studies:  Ongoing impact evaluation and operations research of the To Change Health Behaviors and Improve Coverage of Health Services by Activating Social Platforms for the Poor in Uttar Pradesh  Six applied research and evaluation projects for India’s Orphans and Vulnerable Children’s Project  Impact evaluation of the Program in Appropriate Technology for Health’s Sure Start program in Uttar Pradesh  Evaluation of the Saving Newborn Lives I Program in India and Saving Newborn Lives II in Bangladesh and Pakistan  Developed child health surveys for the New Hampshire Bureau of Maternal and Child Health  Evaluation of Family Home Visiting Program  Developed a health status and parasitology survey for the Delmarva Migrant Health Project in Virginia Experience in Presenting Research Publications 148 Dr. Ruducha is comfortable presenting research results to a wide variety of audiences. As Technical Lead of six applied research and evaluation projects for India’s Orphans and Vulnerable Children’s Project, she organized a national dissemination meeting in India. She also provided a debriefing on the findings of a Mid-Term Review of the Saving Newborn Lives II Program in Bangladesh and Pakistan for Save the Children. She presented the results of her report on the Mid-Term Review of Intrahealth’s Vistaar Program in Maternal, Neonatal, Child Health and Nutrition for USAID/India. Dr. Ruducha also contributed testimony to the State of Virginia legislature and other state departments on the status and needs of migrant farm workers. Education: Post-Doctoral Fellowship in Society and Health, Harvard School of Public Health and the Health Institute, Tufts Medical Center, Boston, MA Doctor of Public Health (Health Policy), Johns Hopkins University, Bloomberg School of Public Health, Baltimore, MD Master of Public Health, Johns Hopkins University, Bloomberg School of Public Health, Baltimore, MD Bachelor of Science in Nursing, Fairfield University, Fairfield, CT Employment History: Braintree Global Health, Cambridge, MA November 2011 – Present Partner  Launched a global health research and evaluation organization to provide technical assistance to design and conduct applied research and evaluation studies to improve the evidence base for accelerating public health improvements. Boston University, Boston, MA October 2010 – Present Research Scientist, Center for Global Health and Development a) Co-Principal investigator: To Change Health Behaviors and Improve Coverage of Health Services by Activating Social Platforms for the Poor in Uttar Pradesh, India (February 2012 - 2016)  Providing technical assistance to develop maternal and newborn health interventions through women’s self-help groups and developing a framework for improving linkages to functional health services  Working with consortium to develop impact evaluation methodologies, tools, quality assurance of survey implementation, analysis plans and reports  Developing operations research study using network analysis to trace changing health provider access and use of health services linked to program interventions b) Technical Lead: India’s Orphans and Vulnerable Children’s Project (October 2010 - April 2012, 18 month project)  Directing the development and implementation of six applied research and evaluation projects on orphans 149 and vulnerable children in four states in southern India and Delhi  Principal Investigator of two projects: 1) an evaluation of a new Government of India National AIDS Control Organization (NACO) initiative for Children Affected by AIDS (CABA) in five districts using Organizational Network Analysis and; 2) assessing the effectiveness of a new Integrated Care Model of Drop-in Centers in three districts in Maharashtra  Organized a national dissemination meeting and monitoring and evaluation capacity building workshop in India Program in Appropriate Technology for Health (PATH), Washington, DC, Indonesia and India June – September 2010, May - December 2011, March 2012 – Present Research and Evaluation Consultant  Directed the analysis of a newborn asphyxia study in Indonesia based on community interventions by midwives and neurodevelopmental outcomes at age 2 and wrote the report  Contributing technical inputs into the end-line impact evaluation survey of the Sure Start program, consisting of large scale community maternal and newborn interventions in 7 districts in rural Uttar Pradesh, India as part of the Sure Start Evaluation Team  Leading the data analysis of determinants of performance of village health workers and the village health sanitation committees in relationship to selected maternal and newborn health indicators  Contributing to development of analysis plan and on-going discussions of methods and measurement of the impact of Sure Start interventions on newborn mortality USAID-India Office, Washington, DC and New Delhi April – June 2010 Mid-Term Review Team Member - Maternal Child Health Sustainable Technical Assistance Research (MCH-STAR) Project led by Cardno Emerging Markets  Reviewed a USAID program designed to provide capacity building and institutional strengthening support to 5 major Indian institutions that would enhance their ability to perform quality research and evaluation, engage in advocacy and technical assistance services to the government  Interviewed senior government officials and leaders of major Indian institutions, analyzed data and documents to establish critical conclusions and recommendations for the life of the project and wrote major sections of final report Save the Children, Washington, DC April – November 2009 Team Leader for Mid-Term Review of Saving Newborn Lives II Program (Bangladesh and Pakistan)  Led the technical and managerial oversight of mid-term review (MTR) programs in each country (10 members of mid-term review team)  Planned and led meetings, prepared teams’ terms of reference, instruments, procedures and data collection methods  Responsible for successful documentation, qualitative interviews, data analysis, debriefing and lead authorship of reports USAID-India Office, New Delhi July - September 2009 Mid-Term Review Team Member - Intrahealth’s Vistaar Program in Maternal, Neonatal, Child Health and Nutrition 150  Assessed USAID funded Intrahealth’s technical assistance to the government health care delivery of maternal, adolescent, child and newborn health interventions through: state and district level strengthening of health policy and planning; and performance improvement of frontline health providers (micro-planning, supportive supervision, reliability of drugs and supplies)  Major contributor, writer and presenter of debriefing PPTs and report Program in Appropriate Technology (PATH), New Delhi, India August 2006 – April 2008 Monitoring and Evaluation Consultant  Engaged in development of Sure Start strategy, operational framework, Common Minimum Program, project log-frame, monitoring and evaluation indicators and implementation guidelines for scaling up maternal and newborn health interventions in 25 million population  Developed evaluation design and methodology for PATH’s Sure Start Program in Uttar Pradesh (UP) and Maharashtra in collaboration with key PATH staff and external experts.  Led the development of a successful Gates Foundation proposal to enable the implementation of a randomized field trial impact evaluation in UP consisting of 7 districts, 18,000 mothers from 700 villages, 2,000 qualitative interviews including verbal autopsy of maternal and newborn deaths  Managed and provided technical input to Sure Start evaluation partners including randomization procedures, quantitative and qualitative instrument development, quality assurance, pre-testing, initial analysis structure and plan  Contributed to PATH and IndiaClen evaluation of essential newborn care needs and services in the National Rural Health Mission Priority States in India Save the Children, New Delhi, India November 2005 - May 2006 Consultant  Evaluated components of the Saving Newborn Lives I (SNL I) Program in India  Developed a methodology and conducted research resulting in a strategy document for planning Saving Newborn Lives II (SNL II) program  Explored gaps and opportunities for scaling up successful neonatal interventions  Conceptualized and organized a large SNL II workshop in Delhi to assess the political and technical environment for neonatal health with major stakeholder input into Save the Children program priorities  Contributed to India country planning team’s assessment for SNL2 programs in India Management Sciences for Health, New Delhi, India February - March 2006 and May – 2006 Consultant  Reviewed India’s national policy and program environment as well as state level challenges and opportunities towards improving maternal and newborn health programming in UP and Jharkhand  Assessed partnership opportunities which informed the development of a major USAID proposal  Provided technical and logistical input in proposal development  Evolved into taking on leadership role for writing the proposal and integrating individual partner comments and contributions to the final product Johns Hopkins University, Bloomberg School of Public Health, New Delhi, India, March - May 2006 Co-Investigator 151  Contributed to the Johns Hopkins University School of Hygiene and Public Health and the IndiaClen team to develop a methodology for a multi-state qualitative evaluation of CARE’s RACHNA program  Provided assistance in formulating the analysis component  Provided technical input and editorial support for the production of the final evaluation report EPOS Health Consultants, New Delhi, India April - November 2005 Public Health Specialist  Developed a decentralized planning toolkit for implementing a major provision of the National Rural Health Mission  Evaluated and contributed to the development of the RCHII state and district plans  Provided technical assistance in the implementation of research initiatives including the Health Resource Infrastructure Rationalization Project in Himachal Pradesh New Hampshire Department of Health and Human Services, Concord, New Hampshire 2000 – 2003 Senior Health Policy Advisor - Bureau of Maternal and Child Health  Conducted maternal and infant health needs assessments including county variation in health indicators by analyzing the New Hampshire’s Birth Registry  Provided technical support to the Department in reevaluating MCH program priorities and developed tools for information based decision making  Developed child health surveys and provided technical support in their execution Consultant on Short Term Projects: Based in Cambridge, Massachusetts 1993 - 1999 1. Campaign to Enroll Poor Children in Health Insurance Developed and implemented multi-pronged strategies to enroll poor children in available health insurance in the city of Cambridge 2. Evaluation of Academic Advisory Program Developed conceptual model of positive functioning for high risk adolescents in health and social relations related to academic performance and supported the implementation of interventions with an associated evaluation study 3. Evaluation of Family Home Visiting Program Designed and executed a plan for evaluating the effectiveness of Family Home Visiting Program for high risk minority pregnant women and newborns in 23 health services delivery areas throughout the state of Massachusetts 4. Development of National Curriculum on Migrant Health Prepared curriculum on the health and health care utilization of migrant children to improve technical knowledge and sensitivity to minority health issues in schools of medicine, nursing and public health 5. Evaluation of Health Reform Proposals Evaluated Vermont’s health care reforms on women’s access to health care as part of state legislation on health insurance Department of Health and Social Behavior, Harvard School of Public Health and the Health Institute of the New England Medical Center, Boston, MA, 1992 - 1994 Post-Doctoral Fellow and Research Associate  Conducted research focusing on the social determinants of health and use of health services of migrant 152 children using original migrant data collected during doctoral program  Conducted qualitative study on health status and behaviors of five ethnic minority communities in Boston  Participated in working group on Women and Violence resulting in publication of a report  Co-instructor of Society and Health Course at Harvard School of Public Health USAID, New Delhi, India 1988 - 1991 Research and Evaluation Specialist  Designed and planned bilateral project evaluations in many sectors with special emphasis on health, population and nutrition projects, i.e., Integrated Child Development Scheme, Family Planning, Communication and Marketing projects  Developed computerized evaluation tracking systems and program performance indicators Bureau of Services to the Aging, Maryland Department of Health and Mental Hygiene, Baltimore, MD, 1982 – 1984 Evaluation Consultant  Evaluated the efficiency, cost-effectiveness and quality of programs serving the elderly  Conducted research project of the Geriatric Evaluation Service, which led to a legislative mandate to expand the program to all counties in Maryland Delmarva Migrant Health Project, Nassawadox, VA 1981 – 1984 State Coordinator  Directed the planning and administration of a federally funded health project for migrant farm workers and their families  Managed all aspects of staffing (recruitment, training, assessment) and state level budget  Coordinated a health status and parasitology survey  Developed a clinical training program for medical and nursing students through the Area Health Education Center  Contributed testimony to state of Virginia legislature and other state departments on status and needs of migrant farm workers Pan American Health Organization, Washington, DC 1978 – 1980 Consultant  Developed a computerized disaster information system including the indexing and abstracting of materials for Latin America and the Caribbean  Assisted in the development of country disaster action plans, training modules and production of a regional newsletter Public Health Nurse, Mercy Southern Health Center, Baltimore, Maryland, 1979 - 1980  Developed community health education programs to improve access and quality of services Clinical Nurse, Loeb Center at Montefiore Hospital and Medical Center, Bronx, NY, 1978 – 1979  Provided comprehensive primary care clinical nursing services Awards:  University of Vermont, Women’s Studies Department, 1993 153  Harvard University School of Public Health Post-Doctoral two-year fellowship, 1992  John Hume Award for relevance of doctoral research, Johns Hopkins University, Baltimore, Maryland, 1989  Ruth B. Freeman Award for academic excellence, Johns Hopkins University, Baltimore, Maryland, 1989  Milbank Memorial Fund, 1987  U.S. Department of Health and Human Services, Office of Migrant Health, 1986 Selected reports and Publications: India’s HIV Orphans and Vulnerable Children, Generating Evidence for Policy and Practice, USAID, Boston University Center for Global Health and Development, New Delhi, January 2012. The Effectiveness of an Integrated Model Drop-in-Center in Improving Access to Services and Selected Social and Health Outcomes of Orphans and Vulnerable Children in Maharashtra, Boston University Center for Global Health and Development, Boston, MA, December 2011. The Children Affected by AIDS Pilot Scheme: An Organizational Network Analysis, Boston University Center for Global Health and Development, Boston, MA, November 2011. Resuscitation at Home Birth and Neurodevelopment Outcomes at 2 Years of Age: A Cohort Study, PATH and University of Indonesia, June 2010. Maternal and Child Health – Sustainable Technical Assistance and Research Initiative (MCH-STAR) Mid Term Review, GH Tech, Washington, DC and USAID, New Delhi, India, May 2010. Pakistan: SNL II Mid-term Review Report, Save the Children, Washington, DC, November 2009. Mid-Term Review of the Vistaar Program, GH Tech, Washington, DC, September 2009. Bangladesh: SNL II Mid-term Review Report, Save the Children, Washington, DC, June 2009. Saving Newborn Lives: A Summary of Accomplishments and Future Opportunities, Save the Children, New Delhi, India, February 2006. Rapid Assessment of Essential Newborn Care Services and Needs in NRMH Priority States of India, IndiaClen Program Evaluation Network and PATH, New Delhi, India, September 2006. A Toolkit for Decentralized Planning under National Rural Health Mission, EPOS Health Consultants, New Delhi, India, November 2005. New Hampshire: Minority Maternal and Infant Health Report, New Hampshire Department of Health and Human Services, Concord, NH, 2002. Maternal and Infant Needs Assessment in New Hampshire, New Hampshire Department of Health and Human Services, Concord, NH, 2001. 154 The Performance of New Hampshire Funded Prenatal Care Agencies, New Hampshire Department of Health and Human Services, Concord, NH, 2001. Migrant Child Health: The Role of Social, Cultural and Economic Factors, National Migrant Resource Program, Austin, TX, 1994. Migrant Child Health: A Bibliography of the Role of Social, Cultural and Economic Factors, National Migrant Resource Program, Austin, TX, 1994. A Review of Family Planning Communications and Marketing Project, U.S. Agency for International Development, New Delhi, India, September 1989. Determinants of Health Status and Use of Health Services by Migrant Children, A Doctoral Dissertation, Johns Hopkins University School of Hygiene and Public Health, Baltimore, Maryland, May 1989. Disaster Preparedness Update: A Computerized Index of an Emergency Preparedness and Disaster Relief Bibliography of Interest for Latin America and the Caribbean, Volume I, Pan American Health Organization, Washington, D.C., 1983. Volume II published in 1983; Volume III published in 1984. Other activities:  Reviewer of manuscripts for Biomed Central Public Health and Journal of Tropical Pediatrics  Presenter at major conferences (American Public Health Association, International AIDS Conference, Massachusetts Public Health Association, American Sociological Association, Association of Maternal and Child Health Programs, New Hampshire Health Department, Society and Health Program at Harvard School of Public Health, Save the Children)  Yale University Workshop on Randomized Field Experiments – Impact Evaluation, Summer 2007  Member of American Public Health Association since 1979  Member of White Ribbon Alliance for Safe Motherhood in India since 2004  Board Member of HelpLife, an India based NGO improving the lives of disabled girls 155 ANNEX XII: SUMMARY OF INDICATOR-BASED FINDINGS 1995-2010 TFR  Declining in all EAG States  During IFPS I-III, UP decrease TFR by 25% (4.82-3.6) vs Bihar decrease of 8% (4-3.7)  District Level for all 3 IFPS States shows significant decrease from 1998-2007 (IFPS I-II)  No significant TFR difference in any district attributable to IFPS CPR  During IFPS 1 (1995-2004) greater rate of increase in UP compared to other EAG states  Post 2005 (IFPS II) all EAG states, except Rajasthan have declining CPR - until 2008 when CPR in UP and other states regains lost ground  Among all districts in UP and Bihar CPR increased significantly over time, with UP high-intensity districts having a significantly higher CPR in 2007/08 than low-intensity districts  Between 2002/4 and 2007/8 rate of growth slowed to 2.6% in high-intensity districts versus low-intensity districts slowed to 2.0% Use of MM  Increase seen in all EAG states 1992-2005  Post 2005 (except for Rajasthan) declining trend noted in all EAG states - after 2007 increasing trend noted  By 2005 (IFPS II) Uttarakhand useof MM is high (55.5%)  JH improvement noted post 2008 (end of IFPS II)  District level shows significant growth of use of MM in UP and Bihar  From 1998-2004 UP high intensity intervention districts have signficant increase in use of MM despite being similar at baseline to low-intensity districts METHOD MIX - CONDOMS  UP, Rajasthan, UK with increasing, high condom use vs JH and Bihar with low use and little change over time  At district level UP condom use from 3.9-7.2% (1998-2007 IFPS I & II) remained low in Bihar reaching 1.4% in 2007/8  Non-significant trend toward greater condom use in high-intensity districts in 2002 (IFPS 1) which became signficant in 2007/8 (IFPS II) METHOD MIX - OCP  1992-2010 UP OCP increase from 1-2.7% compared to Bihar increase from 1.1-1.5 - - - Odisha increased from 0.9-11.1 in same time period  Districts level use of OCP in UP and Bihar is less than 2% from 1998-2008 (IFPS I & II) - between 1998-2002 (IFPS I) marginal increase but erased by 2008 (IFPS II)  Over time, UP high-intensity intervention districts had non-significant higher prevalence over low-intensity districts METHOD MIX - IUD  Wide variability of use among EAG States  UP shows increase use between 1998/9-2002/4 (IFPS 1) decrease noted after 2005/6  UK shows steep decline in IUD use between 2002-2010 (IFPS I-III)  District level use low approx 1.2% in UP and 0.5% Bihar - little change in rates over time and little difference between 1998/9 and 2007/8  No significant difference between UP high-intensity and low-intensity districts METHOD MIX - STERILIZATION  UP shows flattened trend with female sterilization lower than values for all comparison EAG states (1992-2010)  District level shows Bihar with highest sterilization prevalence  UP High intensity and low-intensity districts show similar rates in 1998/9 (IFPS 1), no change in 2002/4 and significant increase in sterilization in 2007/8 in the low-intenstiy districts  Declined in UP and other large EAG states from 1992-2005 (IFPS I) UNMET NEED  Increasing unmet need from 2005-2007/8 in all EAG states, except Rajasthan  Wide variation in coverage across large and small EAG states TT COVERAGE1  UP coverage increased from 37.4 (1992/3 - IFPS I) to 80.9 (2010 - IFPS III) - Bihar showed similar increase despite a drop from 1998-2004 1 Unmet need and TT coverage were only analyzed at the state level 156 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 157