FAMILY PLANNING AND REPRODUCTIVE HEALTH FINAL EVALUATION OCTOBER 15, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Sandra Wilcox and Dr. Javeriah Shamsy with Zahra Fatima Moulvi (assignment manager), Management Systems International, a Tetra Tech Company. FAMILY PLANNING AND REPRODUCTIVE HEALTH FINAL EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Family Planning and Reproductive Health—Final Evaluation iii ACKNOWLEDGMENTS The evaluation team would like to acknowledge the generous support and assistance it received in planning, organizing, and conducting the evaluation from the Marie Stopes Society of Pakistan. Dr. Imam Baig, the FP/RH chief of party, Dr. Shabnum Sarfraz, the deputy chief of party, and Mr. Zulfiqar Mithani, director of operations, gave generously of their time and programmatic support answering various questions and providing insight on the FP/RH project’s experience and results. The team would also like to thank Dr. Rukshana, the government liaison officer who facilitated meetings with district and provincial level officials at the Population Welfare Department and the Department of Health. Additional thanks are due to Dr. Waqas Hamid, general manager of research, for his assistance with interpretation of MSS’s extensive data sources. Special thanks are also due to Monica Villanueva, USAID/Pakistan health and population officer, and to Dr. Shabir Chandio, the USAID/Pakistan health officer who provided insight and support to the evaluation from the donor’s perspective. The team would also like to thank the MSS Sindh district health coordinators, directors, and providers; SSF health providers; Sindh PWD and DOH district and provincial directors and staff; HANDS directors; field health educators; lady health workers; and numerous clients who were interviewed for this report and who took a real interest in sharing their FP/RH experience with the team. Finally, the team thanks Zahra Moulvi, Babur Arif, Douglas Krieger, and Christine Mulligan from Management Systems International for all their assistance with the team’s organization, assignment preparation, logistics, and final organization and editing of the report. Despite the best efforts of the FP/RH project, Management Systems International, and USAID, factual errors may persist in the report. They must be considered the responsibility of the principal authors who tried to grasp the complexity of the environment in a short period of time. Family Planning and Reproductive Health—Final Evaluation iv CONTENTS Acknowledgments........................................................................................................................ iii List of Abbreviations and Acronyms .......................................................................................... vi Project Summary .......................................................................................................................... 1 Executive Summary...................................................................................................................... 2 Evaluation Purpose and Questions.................................................................................................................2 Summary of Key Findings and Conclusions..................................................................................................2 Recommendations..............................................................................................................................................3 Evaluation Purpose and Questions.............................................................................................. 4 Evaluation Questions.........................................................................................................................................4 Project Background ...................................................................................................................... 4 Family Planning in Pakistan and Sindh Province...........................................................................................4 The FP/RH Project .............................................................................................................................................6 Evaluation Methods and Limitations........................................................................................... 7 Data Collection Methods and Sources..........................................................................................................7 Data Analysis........................................................................................................................................................9 Data Limitations................................................................................................................................................10 Findings and Conclusions for Question 1: Effectiveness.........................................................................11 Findings and Conclusions for Question 2: Cost Effectiveness...............................................................24 Findings and Conclusions for Question 3: Best Practices and Lessons Learned...............................26 Other Findings and Conclusions ................................................................................................ 29 New Users.........................................................................................................................................................29 Sustainability.......................................................................................................................................................29 General Conclusions ................................................................................................................... 30 Recommendations ...................................................................................................................... 31 Annexes........................................................................................................................................ 33 Annex 1: Evaluation Scope of Work............................................................................................................33 Annex 2: Assignment Work Plan..................................................................................................................37 Annex 3: Data Collection...............................................................................................................................62 Annex 4: List of Individuals Interviewed .....................................................................................................63 Annex 5: Data Collection Instruments........................................................................................................64 Annex 6: List of Documents Reviewed.......................................................................................................98 Annex 7: Conflict of Interest Statements...................................................................................................99 Annex 8: Client Growth Analysis and Cost per CYP........................................................................... 100 Family Planning and Reproductive Health—Final Evaluation v List of Tables Table 1: Project Summary................................................................................................................................................1 Table 2: Breakdown of Suraj Social Franchise Clinics...............................................................................................7 Table 3: Distribution of Sample by District.................................................................................................................9 Table 4: Providers’ Test Scores Before and After Training ..................................................................................14 Table 5: Trends in Cost Effectiveness ($/Output)...................................................................................................24 Table 6: Helpline Cost Fluctuations, January 2104 – April 2016..........................................................................26 List of Figures Figure 1: Change in Service Quality ............................................................................................................................12 Figure 2: Constraints to Accessing Family Planning Services from Non-SSF Facilities....................................13 Figure 3: Growth in Client Numbers, 2013–2015...................................................................................................19 Figure 4: Comparative Costs of Generating CYPs..................................................................................................25 Family Planning and Reproductive Health—Final Evaluation vi LIST OF ABBREVIATIONS AND ACRONYMS BEmONC Basic Emergency Obstetric and Newborn Care CMW Community Midwife CNIC Computerized National Identity Card CoP Chief of Party CPR Contraceptive Prevalence Rate CYP Couple Years of Protection DCO District Coordination Officer DHS Pakistan Demographic and Health Survey DoH Department of Health DPWO District Population Welfare Officer FHE Field Health Educator FP Family Planning FP/RH Family Planning and Reproductive Health FWW Family Welfare Worker HANDS Health and Nutrition Development Society IEC Information, Education, and Communication IUCD Intrauterine Contraceptive Device JHU/CCP Johns Hopkins University Center for Communication Programs LAPM Long-Acting and Permanent Methods LHV Lady Health Visitor LHW Lady Health Worker M&E Monitoring and Evaluation MCH Maternal and Child Health MCHIP Maternal and Child Health Integrated Program MDT Medical Development Team MMR Maternal Mortality Ratio MNCH Maternal, Newborn, and Child Health MSS Marie Stopes Society MWRA Married Women of Reproductive Age NGO Non-Governmental Organization OMU Outreach Mobile Unit PPFP Post-Partum Family Planning PWD Population Welfare Department Family Planning and Reproductive Health—Final Evaluation vii RH Reproductive Health RME Research, Monitoring and Evaluation SBA Skilled Birth Attendant SF Social Franchise SFS Senior Field Supervisor SIFPO Support for International Family Planning Organizations SOP Standard Operating Procedure SPSS Statistical Package for the Social Sciences SSF Suraj Social Franchise TFR Total Fertility Rate TL Tubal Ligation TPW Team Planning Workshop UNICEF United Nations Children’s Fund USAID United States Agency for International Development Family Planning and Reproductive Health—Final Evaluation 1 PROJECT SUMMARY Table 1 summarizes basic information about the Family Planning and Reproductive Health Services Project. TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers AID-391-A-13-00007 Contracting/Agreement Officer’s Representative (COR/AOR) Start date October 2, 2013 Completion date Original: October 1, 2018 Modified: September 30, 2016 Location Pakistan Implementing partner(s) Marie Stopes Society (prime) Health and Nutrition Development Society (HANDS) (sub￾contractor) Creative Social Marketing (sub-contractor) USAID/Pakistan Mission Strategic Framework objectives addressed Increased utilization of quality FP and MCH services Budget Family Planning and Reproductive Health—Final Evaluation 2 EXECUTIVE SUMMARY Evaluation Purpose and Questions The final evaluation of the USAID/Pakistan Family Planning and Reproductive Health (FP/RH) project focused on the project’s effectiveness in achieving its goals and objectives. The Mission expects the evaluation to identify lessons learned, innovations, and best practices and make recommendations for improving future family planning (FP) and reproductive health programming. The evaluation addressed three questions: 1. To what extent has the project been effective in meeting its four major objectives? 2. To what extent has the project been cost-effective and efficient? 3. What best practices, innovations, and lessons learned can be applied to other existing or future programming in family planning and reproductive health? Summary of Key Findings and Conclusions The FP/RH project either had met or was on track to meet its targets by the original 2018 end of project date. By the end of its second year, the project had completed its scale-up and trained and certified the local paramedic providers called social franchises (SFs). The client load was increasing at project-supported facilities and outreach camps and was projected to continue growing. Question 1: Objective 1: Increasing the size and capacity of the health workforce: At the time of the evaluation, the project had trained 767 field health educators (FHEs) and senior field supervisors (SFSs), 361 SF providers, and 485 public providers and trained and operationalized 14 outreach camp teams and 4 outreach mobile units (OMUs). Interview respondents attributed the project’s success to improving the capacity of providers and quality of care and establishing more accessible provider locations. Cost was very important to clients, and the project provided vouchers that made services available to some clients free of charge, but most clients interviewed said they would still go to the SF even if they did not have vouchers because of the high quality of service and their awareness of the need for family planning. Question 1: Objectives 2 and 3: Scaling up facility-based, outreach, and community-based service delivery: During a briefing with the evaluation team, project staff reported generating 1,600,000 couple years protection (CYPs),1 increasing the contraceptive prevalence rate (CPR) by 4.5 percent, 2 and delivering services that resulted in 650,000 family planning users, including 470,000 long-acting and permanent method (LAPM)3 users. The project has distributed 417,740 family planning vouchers, between 78 percent (2013) and 81 percent (2015) to users living on less than $2.50 per day, and between 20 percent (2013) and 22 percent (2015) to users living on less than $1.25 per day.4 The 1 CYP is a measure that estimates the protection from pregnancy provided by contraceptive methods during a one-year period (Marie Stopes International). 2 CPR is calculated as the percentage of married women of reproductive age who are using a contraceptive method at any given point in time in the given intervention area. See N. Corby, T. Boler, and D. Hovig, “The MSI Impact Calculator: Methodology and Assumptions,” London: Marie Stopes International (2009). 3 Long-acting FP methods include intrauterine devices and implants, while permanent methods include male and female sterilization. These are all clinical methods and are the most effective types of contraception as they provide continuous protection and greater continuation of family planning. The Marie Stopes Society (MSS) uses the Marie Stopes International impact calculator to estimate the number of new family planning users, including LAPM users. 4 Data presented at FP/RH project briefing workshop, June 2016. Family Planning and Reproductive Health—Final Evaluation 3 evaluation team’s interviews suggest that men strongly influence family planning decisions. The decision not to engage men was a missed opportunity. Question 1: Objective 4: Strengthening public-private partnerships: USAID removed this objective at the end of the project’s second year, but the evaluation found that the project had engaged well with the public sector. District government officials believed the project, especially the camps, provided a useful service and brought more people to public services. However, Marie Stopes Society (MSS) senior management believed it would not be possible to reach the 2020 goals to which the Government of Pakistan has committed without both the public and private sectors working together to achieve them. Question 2: To what extent has the project been cost-effective and efficient? Over the project period, MSS was able to reduce its costs per CYP from about $13 to $10, largely due to the SF and outreach activities reaching scale. Likewise, the project was able to reduce the costs of its helpline from $4.00 per call to $1.51 per call. The helpline seems to be fulfilling a service referral and information need for family planning and reproductive health in areas beyond the project intervention communities. Question 3: What best practices, innovations, and lessons learned can be applied to other existing or future programming in family planning and reproductive health? Key informants identified vouchers as a useful innovation that improved access to family planning services. Linking free services to enhanced awareness of family planning amplified the project’s impact, although improving access to LAPM to address limiting and spacing births is an ongoing need. Interview respondents also emphasized the importance of good quality of care to satisfy clients and retain them as family planning users. These findings coincide with the project’s overall strategy. Recommendations  Pakistan’s low CPR, high total fertility rate (TFR), cultural barriers, and large rural population require the kind of family planning-focused strategies and service delivery model employed by the FP/RH project. To address Pakistan’s 2020 goal of moving from 35 percent to 55 percent CPR, USAID/Pakistan’s family planning programming should continue its vertical focus and, if family planning is to be integrated with other health programs, the focus should be on family planning.  Until the CPR increases, USAID/Pakistan’s family planning programming should continue to rely on the private sector to reach remote rural populations. However, USAID/Pakistan should find a way to engage the public sector to effectively address the low CPR and related maternal health risks in these areas. This may entail negotiations on how improvements and credit for meeting CYPs, CPR, and other standards will be attributed to both the public and private sectors.  In future family planning programming, USAID/Pakistan should work with the private sector and government to improve advocacy and policies for family planning access in remote rural areas. Advocacy and policy work could focus on supporting efforts to certify paramedic providers to provide LAPM in remote rural areas or supporting lady health workers (LHWs) and other community educators in such areas.  Future USAID/Pakistan family planning programming should include components that focus on family planning awareness activities that include men. Men are key decision makers, and most women will seek family planning services only if they have their husband’s support. Future programming could include educational activities specifically directed to men, such as men’s community education groups held at hours convenient for working men and facilitated by male health educators. Family Planning and Reproductive Health—Final Evaluation 4  If USAID/Pakistan requires more detailed cost analysis of family planning programs, it should specify that implementing partners collect and disaggregate data appropriately. In the case of this evaluation, that would have meant collecting additional data on costs per unit of output (e.g., CYP by method and service delivery point) and the cost of providing services through vouchers relative to not using vouchers, disaggregated by method. EVALUATION PURPOSE AND QUESTIONS The final evaluation of the FP/RH project focuses on assessing the extent to which the project achieved its goal and objectives. The FP/RH project is one component of a five-component maternal and child health (MCH) program in Pakistan. USAID/Pakistan is reorganizing the FP/RH component and anticipates using lessons learned from the evaluation to guide future programming in family planning (FP) and reproductive health. Evaluation Questions The scope of work (Annex 1) posed three evaluation questions. The specific questions were: 1. To what extent has the project been effective in meeting its four major objectives? 2. To what extent has the project been cost-effective and efficient? 3. What best practices, innovations, and lessons learned can be applied to other existing or future programming in family planning and reproductive health? PROJECT BACKGROUND Family Planning in Pakistan and Sindh Province Pakistan is a populous and geographically diverse nation, facing a heavy burden of maternal and neonatal mortality. Pakistan has not made sufficient progress to reach Millennium Development Goals 4 and 5. 5 Key challenges to progress include insufficient investment in the health sector as well as broader issues such as political instability, insecurity and conflict, vulnerability to natural disasters, low female education, and the lack of empowerment of women in rural areas. Marked inequities in health outcomes are present between rural and urban areas and between lower and higher socioeconomic levels.6 Shortages of health professionals, particularly of nurses and midwives, are a significant constraint on the provision of FP/RH services.7 Pakistan has the highest population growth and birth rates in South Asia, and its population is expected to surpass 300 million by 2050. The overall maternal mortality ratio (MMR) in 2006–2007 was 276 per 100,000 live births, with a much higher ratio in rural areas (319) compared to urban areas (175), 5 The Pakistani government committed to achieving a reduction in the infant mortality rate (goal 4) from 72 to <55 per 1,000 live births, the newborn mortality rate from 55 to <40 per 1,000 live births, and the maternal mortality rate (goal 5) from 276 to 140 per 100,000 live births by 2015. The reduction in maternal mortality will be achieved, in part, by increasing the contraceptive prevalence rate and reducing the unmet need for family planning. 6 Pakistan Demographic and Health Survey (DHS) 2006-7, National Institute for Population Studies (Islamabad, Pakistan), ICF International (Calverton, Maryland, 2008). 7 Maternal and Child Health (MCH) Program Indicator Survey 2013 Sindh Province, Jhpiego Corporation (Johns Hopkins University, 2013). Family Planning and Reproductive Health—Final Evaluation 5 reflecting disparities in availability of, and access to, skilled birth attendants (SBAs).8 World Bank data indicates that the MMR dropped to 178 in 2015, but the rural or urban breakdown of this statistic is unavailable. 9 Pakistan’s 2012–2013 Demographic and Health Survey (DHS) notes that the national CPR is 35 percent, with 26 percent of the population using modern methods. In Sindh, the provincial CPR is 30 percent, with 25 percent using modern methods. These rates have not changed significantly since the 2007 DHS. About 24 percent of Pakistan’s population lives in Sindh province, where the FP/RH project is focused. Sindh includes Karachi and 22 predominantly rural districts, which lack adequate health infrastructure. To date, the most complete source of population-level health data for Sindh province is the Pakistan DHS 2012–2013. This study found that the neonatal mortality rate in Sindh was 44 per 1,000 live births, the infant mortality rate was 81 per 1,000 live births, and the under-5 mortality rate was 106 per 1,000 live births (unchanged since 2006–2007). Sindh had a total fertility rate (TFR)10 of 3.9, the second-highest in the country after Balochistan (4.2). The 2012–2013 DHS found that 37 percent of Pakistani children are born less than 24 months after a previous birth, a birth interval considered “too short,” 11 and this was only 3 percentage points better than the proportions measured in 2006–2007. Although knowledge of birth control methods was nearly universal, only 25 percent of currently married women—and only 17 percent of rural women—in Sindh used modern contraceptive methods in 2012–2013.12 Although it has the highest fertility rate in South Asia, Pakistan has had one of the longest running family planning programs in Asia. According to previous DHSs and UNICEF Multiple Indicator Cluster Surveys, knowledge of family planning has steadily increased from 90 percent in 1994–1995 to 94 percent in 1996–1997 and 98 percent in the most recent DHS. In the past, Pakistan’s family planning program focused on limiting childbearing through voluntary surgical contraception. In recent years, the focus of the program has shifted toward spacing births through reversible contraceptive methods. The DHS reported that quality of care has been a major impediment to the use of contraception, with 37 percent of women discontinuing contraceptive use within a year, often because of side effects or other health issues. The most recent DHS also showed no change in the use of the three main reversible methods over six years. Both the 2012–2013 and 2006–2007 reports found that 6.7 percent of married women used intrauterine contraceptive devices (IUCDs) or oral or injectable contraceptives. Condom use, which has continued to lead as the main method of contraception, was 8.8 percent in 2012–2013, followed by female sterilization (8.7 percent). Use of traditional methods was at 8.5 percent, a higher rate than IUCDs and oral and injectable contraceptives combined. The preference for less effective methods such as condoms and withdrawal is believed to be an indicator of poor quality family planning and follow-up services available in Pakistan. Pakistan’s preference for relatively large families may also suppress contraceptive use. The 2013 MCH Indicator Survey in Sindh13 found that most women wanted more than three children and often had closely spaced births. Less than one-third of the women had ever used contraceptives, and only one￾quarter were currently using them. Women who used sterilization, IUCDs, injectables, and pills were likely to be high parity. The private sector was the main provider of family planning methods for all women. 8 Pakistan Demographic and Health Survey (DHS) 2012-13, National Institute for Population Studies (Islamabad, Pakistan) MEASURE DHS, ICF International (Calverton, Maryland, 2013). 9 World Bank Country MMR data. Data.worldbank.org 10 The TFR is the number of children who would be born per woman (or per 1,000 women) if she were to pass through the childbearing years bearing children according to a current schedule of age-specific fertility rates (MEASURE Evaluation). 11 Pakistan Demographic and Health Survey (DHS) 2006-7. 12 Ibid. 13 Ibid. Family Planning and Reproductive Health—Final Evaluation 6 The 2012–2013 DHS found that 24 percent of married women in rural areas of Sindh had an unmet need for family planning (12 percent for spacing and 12 percent for limiting). The 2012 Research and Development Solutions policy brief14 noted that 25 percent of married women in Pakistan had an unmet need for family planning and that most of the unmet need was for limiting. The DHS also noted a higher demand nationally for limiting (11 percent) than spacing (8.8 percent). The policy brief further noted that many women were not able to access services. About 3.3 million women had a need for limiting families, but permanent or long-term services reached only 300,000 women. The brief recommended meeting the unmet need by increasing the capacity of government facilities and creating linkages with the private sector. It also noted that private sector and NGO models are available for spacing and limiting births and could be scaled up through donor funding or contracting with government services. The FP/RH Project The Marie Stopes Society (MSS) of Pakistan, a subsidiary of Marie Stopes International, is a service delivery organization that focuses on providing high-quality family planning services to poor and remote populations. MSS has been operating in Pakistan for 20 years. In response to the high unmet need for family planning services and lack of trained providers, particularly in remote rural areas, MSS developed a strategy to expand family planning services to remote areas of Pakistan. In 2012, the Support for International Family Planning Organizations (SIFPO) pilot project used this strategy. USAID/Pakistan, which funded the pilot, folded it into the FP/RH project in 2013. The FP/RH project’s aim was to respond to the unmet demand for contraception that leads to child and maternal deaths and unsafe abortions. The objective was to provide high-quality family planning services to poor and remote populations in Sindh by increasing the supply of services through a large network of local paramedic providers called social franchises (SFs)—MSS’s SFs in Pakistan are branded Suraj Social Franchises (SSFs)—and mobile outreach services. Due to the lack of qualified medical providers in remote areas, this SF model is the result of MSS’s advocacy for task-shifting to allow lower-level health workers—such as nurses, midwives, and lady health visitors (LHVs)—to perform the highest level of procedures they are qualified to perform. MSS has adopted a strategy of high-impact couple years of protection (CYPs),15 targeting poor and remote populations with a preference for long-term and permanent contraceptive methods. MSS’s approach aims to minimize overlap with other services, such as short-term methods offered by lady health workers (LHWs), and to maximize impact. To overcome cultural barriers and increase the demand for long-term and permanent contraceptive services, MSS added a component of field health educators (FHEs) whom the project trained and assigned to each SSF provider to educate, promote, and refer community women to the family planning services that the assigned provider offered. To further increase demand, the FHEs provided vouchers for free services to women who could not afford to pay. MSS delivered ongoing training and monitoring of its SF providers; it gathered comprehensive data on the quality of its services through its management information system, conducted regular exit surveys of clients, and used this data in the day-to-day management of the FP/RH project. Because surveys identified quality of service and issues with side effects as the major obstacles to family planning, MSS implemented this rigorous quality assurance activity to regularly monitor clinical and counseling services. Table 2 shows the breakdown of SSF clinics by geographic location, including urban, semi-urban, and rural providers. 14 What Unmet Need for Family Planning Means in Pakistan, Research and Development Solutions (2012). 15 CYP is a measure that estimates the protection from pregnancy provided by contraceptive methods during a one-year period (Marie Stopes International). Family Planning and Reproductive Health—Final Evaluation 7 TABLE 2: BREAKDOWN OF SURAJ SOCIAL FRANCHISE CLINICS Count Regions Urban Semi-Urban Rural Total Sukkur 6 34 61 101 Hyderabad 10 19 79 108 Karachi 10 21 21 52 Multan/Bahawalpur 8 0 31 39 Total 34 74 192 300 Percentage Regions Urban Semi-Urban Rural Total Sukkur 5.9% 33.7% 60.4% 100.0% Hyderabad 9.3% 17.6% 73.1% 100.0% Karachi 19.2% 40.4% 40.4% 100.0% Multan/Bahawalpur 20.5% 0.0% 79.5% 100.0% Total 11.3% 24.7% 64.0% 100.0% In sum, to respond to the unmet need for family planning services in rural Sindh province, MSS increased the supply of rural paramedic providers by making them part of its SSF network and training them on family planning counseling, clinical, and practical skills and followed up this training with monitoring, coaching, and supplies. MSS also provided FP/RH services through its mobile outreach clinics and vans to remote areas that do not have access to facility-based services. To address the cultural and other barriers to family planning in these remote areas and generate demand for the services, MSS complemented the SSF and outreach providers with FHEs who provided education, counseling, and referrals to the local providers. To further increase demand, FHEs distributed vouchers for free services to these populations. The project was to continue until 2018, but USAID decided to end it at the mid-term in 2016 due to budget constraints. Initially the project was to be implemented in 40 districts in both Punjab and Sindh provinces, but this was modified to 29 in Sindh and 3 in Punjab, with activities closed in Punjab during year 2. EVALUATION METHODS AND LIMITATIONS The evaluation employed a variety of methods, data sources, and data types to triangulate findings at multiple levels and generate credible and robust evidence to answer the evaluation questions. The data included quantitative data from the project and other databases and reports, and qualitative data from individual and group interviews. (For the list of individuals interviewed, see Annex 4.) The quantitative data documented what happened, while the qualitative data helped explain how and why. Data Collection Methods and Sources Prior to beginning the fieldwork, the evaluation team conducted a team planning workshop (TPW) during which it developed a data analysis plan, designed a data collection plan and instruments consistent with the analysis plan, planned the fieldwork, and presented the evaluation plan to the Mission. (For the data collection instruments, see Annex 5.) During the TPW, the team identified the three primary data collection methods and a variety of sources. Family Planning and Reproductive Health—Final Evaluation 8  Document review—The evaluation team reviewed available project documents and reports, including the cooperative agreement, annual work plans and reports, performance management plan, and assessment data and reports. The document review helped the team develop a thorough understanding of project goals and objectives and planned and actual activities, outputs, and results. Annex 6 lists the documents the team reviewed.  Secondary data—The team also harvested quantitative secondary data from project databases and reports, including annual and quarterly performance reports, assessment reports, baselines, client exit interview reports, client follow-up reports, voucher validation reports, audit reports, facility and service provider records, and data and information about the website and helpline established by the project. It also drew from national surveys such as the DHS.  In-depth interviews—The team conducted individual in-depth interviews with beneficiaries of FP/RH training, relevant public and private health service providers, SSF staff, mobile outreach teams, MSS staff, partners, district and provincial Department of Health (DoH) and Population Welfare Department (PWD) officials, LHWs, and FHEs. The team also interviewed USAID staff and individuals from the project implementation organizations to develop a thorough understanding of project objectives, implementation mechanisms, and the evaluation purpose and context.  Direct observation—The team collected qualitative data from direct observation of health facilities and facility staff. Sampling The evaluation team worked closely with USAID to develop a sampling approach that balanced the requirements of the evaluation with the time and financial limitations of the fieldwork. Requirements included exploring differences between facilities in rural, semi-rural, and urban areas; differences between facilities engaged in the project at different times; adequate coverage of facility-based and outreach activities; and activities in Bahawalpur, Punjab. To ensure that results are as representative as possible, the evaluation used random sampling approaches to the greatest extent practical. The team started by identifying the locations where it could lodge and work productively and clustered the sample around these locations. These sites were Karachi, Sukkur, and Hyderabad in Sindh and Bahawalpur in Punjab. These locations gave the team access to districts with both SSF-A and SSF-B facilities;16 old and new SSF facilities; urban, semi-urban, and rural locations; and active outreach teams in 21 districts that were within a two- to three-hour driving distance of the cluster center. This became the district sample. In the third stage of sampling, the team randomly selected six SSF and three public facilities from the district samples in each cluster. It also arranged interviews with a district population welfare officer (DPWO), an LHW, and MSS staff at the regional offices in each cluster. At each sampled facility, the team selected a health care provider (a doctor or a paramedic, depending on the type of facility), an FHE, and at least two clients (voucher and non-voucher) for interviews. Given the sensitive nature of the topic, the team asked MSS staff in each cluster to arrange interviews with LHWs, DPWOs, and DoH personnel from public hospitals. The interview questions focused primarily on respondents’ experience working with MSS. The team also interviewed an FHE from the 16 In SFF-A facilities, paramedic providers offer short-term and long-acting (IUCD only) methods. In SFF-B facilities, doctors provide obstetric and newborn care and comprehensive family planning services. For details on the categories of clinics, see the findings for Question 1, Objective 1. Family Planning and Reproductive Health—Final Evaluation 9 outreach team in each cluster. Throughout the fieldwork, the team took all possible measures to protect the confidentiality of interview subjects. At the time of the evaluation, the project had ceased activities in Punjab and no longer had connections with the government. It was therefore not possible for the evaluation team to obtain the permissions necessary for the fieldwork. The team selected a replacement sample of districts and facilities from Sukkur, which was the cluster most similar to Bahawalpur in terms of the urban and rural mix of districts. Table 3 summarizes the data collection by cluster. TABLE 3: DISTRIBUTION OF SAMPLE BY DISTRICT Source Cluster Karachi Hyderabad Sukkur I Sukkur II* Total Service provider interviews (SSF provider, outreach camp team, outreach mobile unit, public provider) 8 10 9 6 33 FHE/SFS interviews 8 8 8 6 30 MSS staff interviews (DCOs, project office) 5 1 1 0 7 Regional and district level government interviews (PWD, DPWOs, and DoH) 5 5 4 0 14 Client interviews 12 12 12 12 48 Partners 2 0 0 0 2 Total by location 40 36 34 24 134 * The team had already interviewed MSS staff and regional and district level government officials during its first visit to Sukkur. Fieldwork The Assignment Work Plan (Annex 2) specified a team that included two sector specialists, each of whom would lead a three-person field team. Due to last-minute issues, however, one sector specialist was unable to join the team. Consequently, the team conducted the fieldwork as a single unit under the direction of the remaining sector specialist. The sector specialist conducted key informant interviews with all the government staff, while the field researchers completed the facility checklist, conducted individual interviews as necessary, and transcribed the facility-level data. The field team spent seven to eight days in each cluster and visited as many of the randomly selected facilities as possible in the allotted time. The team conducted approximately 34 to 40 interviews in each cluster (except for the Sukkur II cluster, where it conducted 24 interviews; the team had already interviewed government officials and MSS staff during its first visit to Sukkur, i.e., in the Sukkur I cluster). While the field team visited SF facilities, outreach teams, and public facilities, the expatriate team leader conducted interviews with USAID personnel, MSS staff, and provincial DoH and PWD staff in Karachi. Annex 3 documents the fieldwork. Data Analysis The quantitative data provided information on what the project had accomplished. For example, it provided evidence of the number and volume of services provided at supported facilities, CYPs delivered, and measures of service quality. The team used the Statistical Package for the Social Sciences (SPSS) to produce frequencies and cross-tabulations for the quantitative analysis. The qualitative data provided detail on how and why the project produced, or failed to produce, anticipated results. To conduct the qualitative analysis, the team used MAXQDA, a software package for processing and analyzing qualitative data. In interview notes, the team marked segments of text that Family Planning and Reproductive Health—Final Evaluation 10 illustrated key themes relevant to answering the evaluation questions, e.g., the mechanisms through which project activities increased the capacity of health care providers. The team then used the software to produce frequencies and cross-tabulations of the codes. Data Limitations The mixed methods approach utilizing a wide variety of quantitative and qualitative data from multiple sources makes this a particularly strong evaluation design. However, the analysis must consider several data limitations.  Findings from the primary data are not generalizable to the entire project since the data came from a non-random sample of 24 of 367 facilities. Even though the results do not necessarily represent the entirety of the project, the sample adequately covers the dimensions of interest (e.g., project phase and urban, semi-urban, and rural locations) and will produce useful findings along these dimensions. Furthermore, the data collection instruments covered all key questions from all the objectives of the project to ensure that findings from the non-random sample adequately addressed all key intervention strategies.  The project ended unexpectedly early due to USAID budget constraints, which could have affected MSS staff motivation to participate in interviews and willingness to respond to additional data requests. However, MSS staff cooperated very well with the evaluation. The evaluation team made every effort to minimize the demands on their time and did not get the sense that the project’s status affected staff’s response to questions.  It was not possible to include Punjab in the evaluation sample because the team did not obtain the government permissions necessary to conduct the fieldwork. Although the team replaced the interviews with additional interviews in Sukkur, the evaluation findings do not represent the experience in Bahawalpur (Punjab).  Facilities did not maintain client records in a form that allowed the team to identify repeat clients. The team obtained and manually reviewed service provider diaries to try to identify repeat users. However, because the diaries did not include unique identifiers (e.g., a CNIC number) or consistent name formats, the team could not identify new users. It was therefore not possible to determine the number of unique or “new” family planning users from these records. Consequently, the team relied on MSS-reported estimates at a single point in time which may not represent the experience of the entire project lifecycle.  The report relies on data collected and reported by MSS for several key results—usually outputs. Since the evaluation team did not collect these data, it cannot vouch for the quality. The report presents project-supplied data under the heading “Secondary Data” and notes the evaluation team’s perceptions of the data’s quality. To mitigate this limitation, the team triangulated project-reported results against primary data collected from other sources whenever possible.  The issues covered in the evaluation are sensitive, especially to clients, which could contribute to selection and response biases in interviews with clients. To mitigate this potential bias, the team relied on trusted community contacts (LHWs and FHEs) to identify for interviews clients who would be willing to talk about family planning and conducted the interviews outside their homes. While this approach did not eliminate the potential for selection and response bias, and introduced possible bias if LHWs and FHEs did not select representative individuals, it was the best the evaluation team could do in the circumstances. Family Planning and Reproductive Health—Final Evaluation 11 FINDINGS AND CONCLUSIONS Findings and Conclusions for Question 1: Effectiveness Question 1: To what extent has the project been successful in meeting its four major objectives? The project’s four major objectives were (1) to increase the size and capacity of the health workforce, (2) to scale up facility-based service delivery, (3) to scale up outreach and community-based service delivery, and (4) to strengthen public-private partnerships. This section presents findings for each objective separately. Objective 1: Increasing the Size and Capacity of the Health Workforce The FP/RH project has categorized the social franchise (SF) and outreach providers it uses in the following way:  Category SF-A: Paramedics—including community midwives (CMWs), LHVs, family welfare workers (FWWs), and nurses—providing short-term and long-acting (IUCD only) methods.  Category SF-A+: Doctors providing comprehensive family planning services, including short￾term, long-acting (IUCD and implant), and permanent (tubal ligation [TL]) methods.  Category SF-B: Doctors providing safe motherhood and basic emergency obstetric and newborn care services and comprehensive family planning services, including short-term, long-acting (IUCD and implant), and permanent (TL) methods. The FP/RH project also developed a cadre of community health workers assigned to each provider who generated demand for the provider’s services in the surrounding community. These were the field health educators (FHEs) and senior field supervisors (SFSs). On average, 8 to 10 FHEs reported to 1 SFS. The project built the capacities of providers and community staff by providing the following trainings:  Module 1.1 – Family planning theory and business planning  Module 1.2 – Counseling and interpersonal communication skills  Module 2.1 – Practical skills training – given per level of provider All providers (A, A+, B) received each of the three modules. The FHEs and SFSs received only modules 1.1 and 1.2. Secondary Data The project’s workforce size and capacity commitment, which USAID formally modified in year 2, was to train 600 new FHEs and SFSs, and train and operationalize 300 new SF providers and 18 new outreach teams. The project also agreed to train 450 (originally 200) public providers. It reports having either met or exceeded these targets by the time of the final evaluation. By April 2016, the project had trained 767 FHEs and SFSs (target 600). It had also trained a total of 361 SF providers, of which 300 were accredited (on target). However, in April 2016 USAID asked MSS to cut back on its activities due to USAID budget reductions, so MSS reduced the number of active SF providers to 169. The project trained and operationalized the targeted 18 outreach teams (14 outreach camp teams and 4 outreach mobile units [OMUs]). However, from October 2015 to June 2016, MSS cut back to 12 outreach teams Family Planning and Reproductive Health—Final Evaluation 12 (10 outreach camp teams and 2 OMUs) because of USAID budget reductions. The project also exceeded its target (450) and trained 485 public providers by the end of year 2. 17 Because surveys had identified poor quality of service as one of the main reasons for stagnation of contraceptive use in Pakistan (see “Project Background”), MSS focused not only on training a large number of providers but also on extensive monitoring, follow-up supervision, and mentoring of providers to improve quality. Once MSS staff identified gaps, they mentored or referred the provider for more training. The project briefing stated that it took MSS staff about nine months to bring an SF provider up to the MSS quality standards. If providers could not meet the standards by the end of nine months, then MSS dropped them from the project. The project was working with 367 SSF facilities, of which 218 were inactive (i.e., dropped from the project) at the time of this evaluation, which puts the dropout rate at 59 percent. Marie Stopes International has provided quality assurance and quality technical assistance to MSS since 2008 to ensure provider compliance with Marie Stopes International standards. Marie Stopes International conducted quality audits once a year throughout the FP/RH project. As a result, MSS improved its quality assurance scores between 2008 and 2015 and instituted a clinical governance structure with accreditation criteria in place. Project data shows evidence that there has been a gradual improvement in quality assurance scores over the project period (Figure 1). Project documents attributed improved quality standards at facilities to regular monitoring, supportive supervision, refresher trainings, and assistance with equipment and supplies. FIGURE 1: CHANGE IN SERVICE QUALITY A review of the MSS client exit survey results between 2013 and 2015 points to client satisfaction with the quality of provider services.  80–93 percent of clients stated they were given clear instructions on method use.  84–92 percent of clients stated they had been informed about potential side effects.  83–92 percent of clients stated they were informed about follow-up.  96–100 percent of clients were satisfied with services (2013–2015). 17 The evaluation team verified the end results—which had increased slightly from the ones mentioned in the project reports— with MSS staff. These are the final numbers that the evaluation team received. Family Planning and Reproductive Health—Final Evaluation 13  77–100 percent of clients would return for MSS services at the facility (2013–2015).  95–100 percent of clients would recommend the MSS services to others (2013–2015).  53–55 percent of clients said they would still use services without vouchers in 2015.  45–50 of clients said they would use family planning if the facility did not exist in 2015.  38 percent said it would be a problem for them to travel for services if the facility did not exist (2014) (most frequent response out of six possible responses). 18  32 percent said that a higher cost of services would cause a problem for them if the facility did not exist (2014) (second most frequent response out of six possible responses). 19 In sum, these figures show that while there is a high level of satisfaction with the SF provider services, just over half of clients said they would continue to use the services without vouchers, and less than half said they would continue to use family planning if the SF facility did not exist. This information triangulates with MSS’s client survey question about problems accessing family planning services (Figure 2). Here the largest proportion of respondents cited travel distance to other facilities and cost as the major impediments to seeking other family planning services if the current SF provider did not exist. It is important to note that MSS selected the clients for the surveys from about 150 SSF facilities, so answers are not indicative of the entire list of project SSF facilities. The evaluation team received a list of 92 facilities (31 percent of the 300 SSF facilities) that were dropped from the project at USAID’s request because they were too close to public facilities (within 5km). 20 FIGURE 2: CONSTRAINTS TO ACCESSING FAMILY PLANNING SERVICES FROM NON-SSF FACILITIES Source: MSS Client Exit Survey 2013, 2014. 18 At the time of the final evaluation, 2015 client exit survey data was not yet published. This was one of two most frequent responses to a question with six possible responses. 19 Ibid. 20 The team received this information for only 300 facilities, and not the full list of 367. Hence the percentage of facilities that were operating within 5km of public facilities is likely to be a bit higher or lower than 31 percent (92 of 300). Family Planning and Reproductive Health—Final Evaluation 14 The evaluation team analyzed a sample of 77 of 19721 training reports to assess pre- and post-test scores over 2013–2015 to get an understanding of the knowledge and skills of providers before and after training (Table 4). The team found that on average post-test scores for providers increased by 90 percent over pre-test scores.22 TABLE 4: PROVIDERS’ TEST SCORES BEFORE AND AFTER TRAINING Region 2013 2014 2015 Type A Type B Type A Type B Type A Type B Average Score % with Score ≥ 85% Average Score % with Score ≥ 85% Average Score % with Score ≥ 85% Average Score % with Score ≥ 85% Average Score % with Score ≥ 85% Average Score % with Score ≥ 85% Karachi 34.9 0.0% 71.0 0.0% 59.4 0.0% - - 84.0 50.0% 89.0 100.0% Sukkur 30.7 0.0% 58.9 0.0% 61.3 0.0% 73.3 0.0% 83.0 41.3% 87.4 100.0% Hyderabad - - - - 52.9 0.0% - - 80.4 34.0% 90.2 83.3% Multan 66.6 6.3% 70.00 33.3% 65.8 0.0% 74.5 50.0% 87.7 68.4% 95.1 100.0% Source: MSS client exit interview data. MSS and Marie Stopes International operate from a premise of providing high quality family planning services for poor, remote populations. To assure continued quality of service after SSF and outreach staff have received clinical and counseling training, MSS gathers data on its services through its management information system and client exit surveys and then uses the data in the day-to-day management and follow-up of its programs. MSS’s quality standards are assisted and overseen by Marie Stopes International’s Quality Technical Assurance System, which regularly audits clinical programs and clinics and provides outreach teams with regular technical assistance and updates. Although the project reports having worked hard to maintain quality standards for services and has documented a high level of satisfaction among clients, additional indicators, such as family planning method discontinuation rates and complication rates, would provide further evidence of quality. Discontinuation rates may reveal how satisfied the clients were and whether they trusted the providers to continue using methods even if they experienced side effects or other issues. Complication rates could indicate problems in the quality of care from a procedure (e.g., infections or the need for follow-up care). Due to time constraints, the evaluation team did not compute these two rates from client interviews. Primary Data Seventy-four percent of 86 respondents23 stated that the number of family planning providers had greatly or somewhat increased over the life of the project. The most frequent reason respondents cited for the increase was the project’s focus on building the capacity of existing providers in the project districts (26 percent), which allowed local providers to acquire new family planning skills and behaviors. Fifty-eight percent of 88 respondents also believed that increased awareness of family planning services had increased demand, and 60 percent of 20 respondents believed the increased demand had increased 21 Not all 197 reports could be analyzed, as average test scores were not a reporting requirement of the project, hence MSS did not maintain an electronic database of pre- and post-training scores. 22 This is an aggregate finding, not disaggregated by training topic or individual participants. 23 Respondents included health care providers, DoH/PWD staff, FHEs, and MSS staff and partners. Family Planning and Reproductive Health—Final Evaluation 15 the number of family planning providers by encouraging potential providers to start offering services.24 The following comment illustrates the increasing availability of family planning services. “Many people get the opportunity to use family planning services. There are 32 clinics working as the SSF here through MSS support. They all were previously present but did not provide the family planning services. After MSS, they provided free family planning services. Family planning providers, except for this provider, charged much for the family planning services. For IUCD, the rates are 3,000 Rs, 6,000 Rs, 5,000 Rs, or 2,000 Rs. Many people cannot afford this. MSS offered free vouchers due to this. People came to the clinic and they got services. MSS gave family planning project to these clinics. I have worked here for seven months. MSS has given trainings for four to five days. Due to this, there was ease in work related to family planning.” – SFS/FHE When asked about the capacity of providers, 91 percent of 86 respondents25 stated that the providers’ capacity had greatly or somewhat increased. Eighty-five percent of health care providers and 50 percent of FHEs thought the capacity of providers had greatly increased. The evaluation team asked providers and FHEs (the people who had received the training and the most frequent respondents to this question) to identify which provider capacities had improved. Capacity improvement areas included interpersonal communication and counseling (19 percent), infection prevention (12 percent), IUCD insertion (12 percent), and quality assurance and monitoring (11 percent). When asked which training was most useful, SF providers did not identify one particular training module. Because all trainees received the first two modules, 26 these received the highest percentage of responses (81 and 87 percent respectively). The other modules were for selected professionals depending on their training level. Comments made clear that participants appreciated the practical aspects of the training and the opportunity to gain new knowledge and skills for their practice, as shown in the following comment. “They did clinical practicum for copper T insertion, and asked us to practice on real clients in Jacobabad. Specifically, the madam asked me to do it on a real client—not a dummy. First she showed us, then she took us to Jacobabad Center. Then it was done on a real client. This helped me to correct my errors. I never knew that measuring the uterus was a requirement before the insertion. I learned this through these trainings.” – Health care provider in Sukkur All of the 33 providers interviewed stated that their quality of service had improved after the training. Infection prevention was the area where the greatest proportion of providers reported that quality improved (52 percent of 33 respondents), followed by technical/clinical skills (42 percent of 33 respondents) and counseling (27 percent of 33 respondents). When the evaluation team asked clients about the quality of care that they received from SF providers, all 48 respondents said they received good quality services. The most frequent reason (24 percent of 72 responses) was the good attitude and behavior of the providers (i.e., a friendly and reassuring “bedside manner”). 24 The denominator for these three results varied because respondents interpreted one of the questions differently and essentially answered different questions. The evaluation team analyzed the question the respondents understood instead of the question the team asked. 25 Respondents included health care providers, DoH/PWD staff, FHEs, and MSS staff and partners. 26 Module 1.1 was on family planning theory and business skills. Module 1.2 was on interpersonal communication and counseling skills. Family Planning and Reproductive Health—Final Evaluation 16 “Yes, I did. I have a government hospital closer than this but in the government facility, doctors’ attitude was not good. They do not bother to spend time with the patient and do not give any information, which is my right. Here, the behavior of the doctors was excellent.” – Client from an SSF clinic in Karachi Other notable reasons respondents thought they received good quality service were good infection control (18 percent of 72 responses); good clinical care (15 percent of 72 responses); and good counseling, i.e., “informing the client about side effects in detail and answering questions skillfully” (13 percent of 72 responses). Consistent with MSS’s client exit survey findings, when the evaluation team asked clients during interviews how easy or difficult it was to reach the provider, 94 percent of 47 respondents said it was easy or convenient to get to the provider’s facility. Interestingly, in the client interviews, 70 percent of the 50 clients who had received vouchers said that they would still come to the SF services even if they were not given vouchers. The most frequent reasons given were that they were satisfied with the provider (26 percent) and that they needed birth spacing services (22 percent). These responses indicate increased awareness of family planning and quality services. When the evaluation team asked clients in interviews why they preferred to go to the SSF provider instead of the public provider, they gave the following reasons: they received good quality care from the SF (24 percent of 71 responses), because they lived close to the SF (17 percent of 71 responses), because the public services were of poor quality (15 percent of 71 responses), because they lived far from the public facilities (13 percent of 71 responses), because they had not known about family planning before the FP/RH project (12 percent of 71 responses), and because the FHE helped them go to the facility (10 percent of 71 responses). Conclusion By the end of year 2, the project had either met or exceeded its targets to train and operationalize SF providers (300), OMUs and outreach camp teams (18), public providers (485), and FHEs (767). The evaluation team’s interviews showed that the number of providers increased in project areas in part due to the project’s activities to increase awareness of family planning in the communities which increased demand for local providers. Moreover, the number of providers offering family planning increased due to the project’s training of providers and offer of free vouchers to increase client use of services. The most useful skills acquired through the training included interpersonal communication and counseling. Providers also indicated that their quality of service improved due to the clinical skills training for selected professionals, and regular quality assurance monitoring helped ensure that all SSF providers met MSS quality standards. The evaluation team’s interviews and MSS’s client exit surveys also confirmed the importance of accessible provider locations. Interestingly, although clients interviewed during the final evaluation considered cost important, just over half said they would go to the SF even without vouchers because of the quality of service and an awareness of the need for family planning. Objective 2: Scaling Up Facility-Based Service Delivery Secondary Data As noted in in the “Secondary Data” section for Objective 1, the project target for facility-based service delivery was to train and operationalize 300 new SF providers. The project had trained 361 by December 2015. Of these, 300 were accredited, but due to funding cutbacks during year 3, as of July 2016 only 169 remained active. Overall the project documented that it generated 1.6 million CYPs between 2013 and April 2016. The objective was to generate 3.15 million—which USAID later modified to 3.7 million—by the end of the project in 2018. The project had planned to increase the contraceptive Family Planning and Reproductive Health—Final Evaluation 17 prevalence rate (CPR) by 3.8 percent (modified to 14 percent). By June 2016, the CPR increase was 4.5 percent according to the Marie Stopes International impact estimator. The project planned to deliver services to 940,000 family planning users, including 575,000 long-acting and permanent method (LAPM) users by the end of the project (modified to 1.5 million users and 600,000 LAPM users). By the time of the final evaluation, the project reported having delivered services that resulted in 650,000 new family planning clients and 469,042 LAPM users.27 Because the project had scaled up and trained the end of project-targeted number of SF providers and FHEs by the end of year 2, MSS staff stated in interviews that they were confident that they would be able to meet the modified CYP and other targets by the end of the project. The original project target was to distribute 645,000 family planning and MCH vouchers by the end of the project. USAID modified this to 800,000 in year 2 but later removed the MCH vouchers because they were costly and not generating enough clients. By July 2016, the project had distributed 417,740 family planning vouchers. The project used the World Bank tool for determining income/poverty levels and provided vouchers for individuals living on less than $2.50 a day. According to project data, in 2013, 78 percent of voucher users were living on less than $2.50 per day, and 20 percent and were living on less than $1.25 (the extreme poverty line). In 2015, 81 percent of voucher users were living on less than $2.50 per day, and 22 percent were living on less than $1.25.28 The aim of providing vouchers to poor clients was to boost access/demand and augment choice for this group of marginalized women seeking family planning services, thus providing them more opportunity to access modern contraception. During year 2 (2014–2015), the project made an effort to focus on distribution and redemption of vouchers. Throughout this period, 35 percent of the married women of reproductive age (MWRA) the FHEs contacted were qualified and interested in using the vouchers for services, and of these, 93 percent redeemed the vouchers. Although the project reports did not document abuses, the evaluation team heard from MSS field staff that there had been some initial abuse of the voucher system in which providers counted more voucher clients than they actually had. The quality assurance team discovered and corrected these abuses, and the providers were dismissed from the program. One of the dilemmas that MSS faced was that the voucher system was not sustainable. At the beginning of the project, MSS and USAID recognized this but felt it was important to stimulate demand from the poorest sectors of the population. During the final evaluation briefing, MSS staff expressed concern that they were being pulled in opposing directions. They noted that USAID’s modifications to the project were asking MSS to target more and more of the poorest strata of the population (those earning less than $1.25 per day) and at the same time to reduce the number of voucher clients. They stated that they could not reach the poorest strata without vouchers. They gave the example of Greenstar, a social marketing firm for health and family planning products, which had tried this strategy. Greenstar asked poor clients to pay 10 percent of commodities, and the result was that the clients then preferred to buy short-term methods (pills and condoms). They also noted that reducing vouchers increases the cost per CYP for outreach. This is because the outreach clinics are in remote locations, and if clients are asked to pay, fewer will come. While the overhead costs remain the same, the cost per CYP increases when SFs have fewer clients and a larger proportion of clients asks only for short-term methods. 27 Project staff explained that they used Marie Stopes International’s impact calculator to estimate the number of new users of family planning services. 28 Data presented at project final evaluation briefing, June 2016. Family Planning and Reproductive Health—Final Evaluation 18 Primary Data When the evaluation team asked interview respondents29 about the SSF and outreach services, 95 percent of 82 indicated that access to services had become somewhat or much better in their communities. Forty-two percent of those who indicated that access to services was somewhat better and 49 percent of those who indicated that it was much better stated that this was because of the door￾to-door visits, community mobilization, and follow-up by FHEs in the communities. Nineteen percent of those who thought access to services was somewhat better and 25 percent of those who thought it was much better thought that it was because of the vouchers for free services. Only 18 percent of 85 interviewees said they thought the need for family planning in their communities had been met. When asked why, 38 percent of respondents who indicated their needs had not been met said it was because they needed more geographic coverage of family planning services, and 56 percent said it was because they needed a more sustainable mechanism for family planning services. The evaluation team conducted these interviews during or shortly after the period when MSS was closing out the FP/RH project and so had withdrawn support for SSF and outreach services. The comment below illustrates the impact of this withdrawal. “Family planning is still a need of this community because clients are still asking us to help them, but as the project is over, we don’t have contraceptive supplies and are unable to offer our clients family planning services and methods. People of this area are very poor, they can’t afford these family planning services on cash and they are asking us to continue the provision of family planning services. This indicates that they still need family planning services.” – SSF provider in Hyderabad region One hundred percent of 29 health care providers and 87 percent of 30 FHEs agreed that vouchers had helped increase the number of family planning clients. The obvious reason was that the services were free. MSS performance data from 2013–2015 also shows that the growth rate of voucher clients has been over 150 percent (Annex 8). During the same period, the growth rate of overall clients was over 170 percent. Figure 3 shows the increase in the number of both total clients and voucher clients over 2013– 2015. 29 Providers, DoH/PWD officials, MSS staff and partners, and FHEs Family Planning and Reproductive Health—Final Evaluation 19 FIGURE 3: GROWTH IN CLIENT NUMBERS, 2013–2015 Objective 3: Scaling Up Outreach and Community-Based Service Delivery Secondary Data Initially, the project planned to develop 18 outreach teams. By the end of year 2, the project had met this objective by creating 14 outreach camp teams that regularly held clinics in the districts and 4 outreach mobile units (OMUs) that moved into remote areas with a mobile van containing equipment and supplies for services. However, due to USAID budget reductions, by the time of the final evaluation, the number of outreach teams had been reduced to 12: 10 outreach camp teams (6 in Sukkur, 3 in Hyderabad, and 1 in Karachi) and only 2 OMUs, which both operated in Sindh. The outreach teams provide all modern methods, including long-acting and permanent methods (LAPMs), to remote rural areas. MSS included the OMUs in year 2 to expand geographic coverage of FP/RH service delivery points and reach the most remote areas with high unmet need and lack of access to other public or private facilities. The outreach teams usually operate out of public clinics in remote areas that do not have family planning methods available (particularly LAPMs). If a public clinic is not available, the project will use a private clinic, usually an SSF facility. The FP/RH project also planned to train over 320 FHEs. By the end of the project, FP/RH had trained 737 FHEs. The FHEs were assigned to specific providers or outreach facilities, and their task was to map the surrounding communities and visit the families of MWRA to counsel and promote the family planning services being offered. These FHEs received interpersonal communication and counseling skills training as well as training on the different family planning methods offered by the project-supported providers. This training allowed them to visit and carefully explain family planning to the women and families and discuss levels of protection, how to manage side effects, possible complications, and available resources, including the provider and the MSS helpline. The FHEs provided the women with educational materials and a brochure with helpline information. The project planned for the FHEs to reach 1.5 million men and women through home visits and community education activities by the end of the project. As of June 2016, the FHEs had contacted and promoted family planning to 2,135,391 individuals, surpassing the target. In addition to the door-to-door Family Planning and Reproductive Health—Final Evaluation 20 visits to their catchment populations, FHEs also conducted “Mohalla” (community) and “Mashwara” (community with provider) meetings to disseminate information about family planning methods and services available at the nearby service delivery point. The year 2 annual report (2014–2015) notes that, during the reporting period, FHEs reached 1.04 million MWRA through household visits and Mohalla and Mashwara meetings. By the end of year 2, the project expected the number of clients to increase significantly over years 3–5 because all the providers and FHEs were trained and making inroads in their communities. Primary Data All this educational activity resulted, as seen above, in an increase in the numbers of clients at the SFs and outreach clinics, as well as over 95 percent satisfaction of clients with the quality of services. During field interviews, the evaluation team found that 47 percent of 47 respondents knew about the MSS helpline. Five individuals had used the helpline and said they were satisfied with the information they received about family planning methods and side effects from procedures. Ninety-four percent of 48 clients stated they had received educational materials (cards, posters, and booklets) from the FHEs and providers. When asked what they thought about the materials, 70 percent of 33 respondents said the materials helped them better understand the family planning messages, and 18 percent said the visual aspect of the materials was helpful. Sixty-four percent of 47 clients said that the FHE had also educated their husband and/or family, and 98 percent of 48 respondents said that the husbands had been supportive of family planning. The most common reason cited for husbands’ support was to limit family size (42 percent), followed by economic reasons (21 percent) and mother’s health (19 percent). When asked to explain the obstacles to community acceptance of family planning, FHEs mentioned objections from the husband or family (22 percent of 105 responses), distance to the service provider (17 percent), lack of knowledge (16 percent), expense (16 percent), objections of religious leaders (12 percent), and lack of services in the area (10 percent). When the FHEs were asked whether they had promoted family planning to men as well as women, 93 percent said yes. When asked about what percentage of men were interested in family planning, the most common responses were one half (31 percent of respondents) and one quarter (20 percent of respondents). All of the FHEs thought family planning should be promoted to men; the most frequent reason given (67 percent) was because men are the decision makers in the family. Although the project’s promotional activities focused on women and did not target men, the evaluation team asked in interviews about men’s support. Almost all the clients interviewed said that their husbands had been supportive of family planning, which goes along with the observation by FHEs that the men are the decision makers. However, the FHEs also identified husbands as the biggest obstacle to family planning. When FHEs were asked how they overcame the husband’s objections to family planning, the most frequent answer (51 percent of 45 responses) was through counseling and communicating with the husband; the next most frequent answer (27 percent of 45 responses) was that the services were free, and husbands were concerned about cost. Interestingly, although the project had recently added a few male FHEs, only 13 percent of 45 responses stated that the male FHEs were able to address the husband’s concerns. It is therefore easy to conclude that a major role and success of the FHEs was convincing the husbands and families to seek and use family planning services. “The main thing that women usually told me is that first they will ask permission from their husbands and then they will use the service. Some of them use the services in private, and some said that they will first seek permission from their husband and ask his point of view. Then their husbands said to them that ‘I did not know about the family planning but you got the information through FHE.’ That is why it is necessary to give education on family planning to men. It is necessary to make them aware of family planning, that it is good for Family Planning and Reproductive Health—Final Evaluation 21 you and for your health and for your children and for your family. It is good for you.” – FHE from Hyderabad FHEs promoted the helpline as a family planning information resource as part of their community education activities. Forty-seven percent of 47 clients interviewed stated that they were aware of the helpline service. Nineteen clients were informed of it by the FHE, two by the provider, and one saw it on the brochure. One who said she did not know of it knew that the number was on the brochure but said she had not paid attention to it. Sixteen percent of 31 respondents said they had used the helpline. These women had called to get information about family planning methods, complications from tubal ligation surgery, and problems with their menstrual cycle. All the clients said they had been greeted and treated well by the health educators or doctors who responded to their concerns. Only two of the respondents who said they had not used the helpline gave a reason—both said that women were not allowed to use mobile phones in their communities. Conclusion for Objectives 2 and 3 The FP/RH project met most of its targets, and those it had not met probably would have been met by the end of the project, given that the system was in place, staff and providers were trained, and relationships with communities had been built through FHEs by the end of year 2. MSS staff had anticipated a big expansion in services throughout years 3–5 of the project, which is why they agreed with USAID to raise many of their indicators during project modifications. Although the project did not target men, all the women who received services from the project’s providers had obtained the husband’s support, and in some cases encouragement. This, along with the fact that all FHEs interviewed thought men should be targeted for family planning education because they are the “decision makers in the family,” points to a missed opportunity. The scaling up of both facility-based and community-based service delivery shows how important the work of the FHEs has been in increasing awareness about the importance of family planning and in addressing concerns about family planning effectiveness and side effects from clients and their families. As pointed out in the “Project Background” section, 98 percent of the population in Pakistan knows about family planning, but only 26 percent are using modern methods. The main reasons stated for not using family planning had to do with quality of service and side effects or other health concerns. The FP/RH project has been making progress in this area because the FHEs are addressing these obstacles through education, and clients are responding by seeking and receiving family planning services. Objective 4: Strengthening Public-Private Partnerships Secondary Data At the time of the final evaluation briefing, MSS informed the team that this objective had been removed from the project during year 2 through a modification to the cooperative agreement. However, the first two annual reports document some of the activities discussed in this section. The reports and MSS staff interviews note that a long-term public-private partnership strategy is necessary to ensure a cohesive health sector and for the country to meet its 2020 MCH goals. During the first two years, FP/RH contributed to advocacy and policy initiatives at both the provincial and national levels. FP/RH teams participated in technical discussions and policy and health system dialogues and provided technical inputs to inform policy. Activities included participation in the LHW program, development of an MCH program community support package with the Maternal and Child Health Integrated Program (MCHIP), development of messages and communication strategy for DoH and PWD with John Hopkins University Center for Communication Programs (JHU/CCP), and development of the Post-Partum Family Planning (PPFP) framework and implementation plan for Sindh under the Sukh initiative. At the district level, collaborations include joint coordination for the outreach camps to be housed in public facilities. Family Planning and Reproductive Health—Final Evaluation 22 Referrals from public sector community workers such as LHWs and FWWs are also a form of local public-private collaboration. In its 2014 annual report (year 1), the project highlighted these referrals as an example of district-level collaboration and partnership, stating that 68 percent of referrals to MSS outreach camps and 25 percent of referrals to SSFs came from LHWs and LHVs. This shows that at the local level, the project is working with the public sector. Recognizing the lack of health workers in the remote rural areas where a large part of the population has access only to mid-level providers and almost 70 percent seek access through the private sector, MSS has worked with government agencies and professional organizations to develop policies to train and expand the range of family planning providers. In November 2014, FP/RH hosted a meeting in Lahore on “Increasing Access to family planning through Task Sharing.” The meeting was attended by senior representatives from DoH, PWD, the Midwifery Association of Pakistan, and the Pakistan Nursing Council as well as representatives from MCH partners, development agencies, and NGOs. The key outcome of the meeting was to develop competency-based pre-service and in-service curricula for implant insertion and roll this out to mid-level providers, including nurses, LHVs, FWWs, and CMWs. The year 2 annual report notes that the project, continuing its close liaison with the LHW program, trained the LHW supervisors on interpersonal communication and counseling skills. The project provided technical assistance with updating the family planning section of the LHW manual and the family planning interpersonal communication toolkit. It also provided technical assistance for revision of family planning curricula for the LHW and CMW training manuals. The annual reports indicate that the FP/RH project followed PWD guidelines. PWD staff reviewed five health manuals in year 1, and FP/RH staff shared training and family planning curricula with them. PWD Sindh trained and certified FP/RH outreach doctors and SSF-B providers on tubal ligation and insertion/removal of implants. The project also planned to refurbish 88 public sector facilities in Sindh. FP/RH procured supplies and equipment for the facilities but then discovered that it could not install the equipment without making improvements in the infrastructure (e.g., electrical, plumbing) which would have required a special waiver from USAID. MSS was unable to get this waiver before the project ended. This task will now be shifted to the other MCH partners. Primary Data Six of the 14 DoH and PWD officials interviewed30 said that LHWs and FWWs worked with the FHEs during the FP/RH project. Most of them were involved in referring clients to SSFs or outreach camps (consistent with the finding from the FP/RH first annual report). MSS district staff also noted the important collaboration between FHEs and LHWs: “When our FHEs go to those areas, LHWs provide them support. There were also lady health supervisors, as well as an attendant, because for FHEs, that is a new village, while an LHW is already working there, in that village. In the community, people do not recognize FHEs, they listen to LHWs. The community trusts LHWs. Then when an FHE goes there and tells the community who she is, what work she does, it builds the trust of the community for FHEs. FHEs were introduced through LHWs.” – Member of MSS DCO staff in Sukkur Because USAID told the evaluation team that the PWD was concerned that the location of the SF providers was too close to public clinics, the evaluation team asked DoH and PWD staff about this 30 Four out of the 14 government respondents were from PWD, and 10 were from DoH. Family Planning and Reproductive Health—Final Evaluation 23 concern. The largest group (43 percent of 14 respondents) said they did not know if it made any difference, and the next largest group (36 percent of 14 respondents) said that the location of the SF provider facilities made no difference to the quantity of the family planning services they provided. Only 1 out of 14 said the services at public clinics had decreased because of the SF provider location, while 2 said they had increased. When the DoH and PWD staff were asked whether the FP/RH outreach activities had increased the use of the public facilities, 57 percent of respondents said yes. They noted that there had been increased use of facilities, increased follow-up visits, and increased coverage. When asked for their general observations about the FP/RH project, 64 percent of 14 respondents said the project had been good and served an important family planning need. Two additional respondents said that the project needed to be sustained and that the outreach camps needed to be made permanent. One respondent from PWD said that coordination with MSS was not good. Two senior PWD officials said they were not happy with MSS’s work in Sindh. They stated that they did not know if there had been any improvements in services or number and capacity of providers. They indicated that this was because communication between themselves and MSS had not been good. They also said that the outreach camps at their facilities were not scheduled correctly and that there had been cancellations of planned clinics. The four MSS senior managers stated during the initial final evaluation briefing and later during interviews that they regretted not being able to work more effectively with PWD and believed MSS had a key role to play in national family planning policy formulation. As evidenced in the quote below, they were disappointed that USAID removed this objective from their agreement and gave it to another USAID contractor. “FP/RH needed to have a strong advocacy role. Advocacy was not seen as a [Marie Stopes International] role by USAID—I think that working on policy was the only way for MSS to improve its relationship with the [Government of Pakistan]. It’s also crucial for sustainability. This work would have mitigated the barriers we have with the government. When USAID told us to drop the fourth objective [policy] and gave this task to another partner—this was a problem. No one can build MSS’s relationship with the [Government of Pakistan]—not another organization—this should have been MSS’s role. Though challenging, if you want to build a legacy network, it’s necessary to invest and work with the PWD and DoH through USAID so that activities are aligned and delivered in part through shared objectives. Would have had [Government of Pakistan] ownership from the beginning. Would be a challenge as the [director general] of health [services] changes all the time, but it needs to be done.” – MSS senior manager, Karachi Conclusion Although both MSS central staff and Sindh PWD provincial level staff expressed difficulties in working with each other, it appears that at the local level the relationships were better. The project worked well with district level staff to arrange outreach camps, and PWD Sindh provided implant and tubal ligation training to the outreach and SF-B providers. Most of the PWD and DoH staff thought the FP/RH project, especially the camps, provided a useful service and brought more people to public services. However, MSS senior management commented during interviews that a big lesson for them was the need to work better with the public sector and that MSS needed to do more at the policy level. They stated that it will not be possible to reach the 2020 goals to which the Government of Pakistan is committed without both the public and private sectors working together to achieve them. It seems this was a missed opportunity and, while challenging, it would have been worth the effort for MSS to continue its policy objective. Family Planning and Reproductive Health—Final Evaluation 24 Findings and Conclusions for Question 2: Cost Effectiveness Question 2: To what extent has the project been cost-effective and efficient? Cost effectiveness in health is expressed as a ratio where the denominator is a change in a health outcome (years of life, premature births averted, CYPs) and the numerator is the cost associated with the health gain (cost/unit of output).31 Because the evaluation team did not collect any direct cost effectiveness data, the responses to this question come from secondary data provided by MSS and secondary analysis. Costs per CYP As Table 5 illustrates, MSS was making progress in reducing its costs per CYP. In 2013 and 2014, costs were higher because the project was in its start-up phase and was training new providers. In 2015, the scaling up was completed and client numbers were increasing, so costs started coming down. Given the evidence provided, the evaluation team believes that the costs would have continued to decline had the project continued to 2018. For details on costs per CYP, see Annex 8. TABLE 5: TRENDS IN COST EFFECTIVENESS ($/OUTPUT) Impact Indicator Costs Over the Life of the Project (in U.S. Dollars) Outputs Year1: Oct 2013– Sept 2014 Year 2: Oct 2014– Sept 2015 Year 3: Oct 2015– May 2016 Average Cost/client 32 37 26 30 Cost/CYP 13 15 10 12 Source: FP/RH project performance data provided by MSS. Note: Both figures of cost/client and cost/CYP include a component of supplies and equipment, which may mean family planning commodities. As Figure 4 illustrates, MSS generated CYPs more efficiently than the average in Pakistan32 and the Asia and Africa regions. 33 31 M.R. Gold, J.E Siegel, L.B. Russell, and M.C. Weinstein (eds), Cost-effectiveness in Health and Medicine (New York: Oxford University Press, 1996), xviii. 32 K. Abbas, A. A. Khan, A. Khan, “Costs and Utilization of Public Sector Family Planning Services in Pakistan,” Journal of the Pakistan Medical Association 63 (2013), S3-10. 33 Ibid, and: Michael Vlassoff, Susheela Singh, Jacqueline E. Darroch, Erin Carbone, and Stan Bernstein, Assessing Costs and Benefits of Sexual and Reproductive Health Interventions (New York: The Alan Guttmacher Institute, 2004). Family Planning and Reproductive Health—Final Evaluation 25 FIGURE 4: COMPARATIVE COSTS OF GENERATING CYPs The higher public sector costs in Pakistan were attributed to underutilization of fixed facilities, weak outreach, and non-functioning referral systems.34 Unfortunately, there was not enough data for the evaluation team to calculate a comparison of CYP costs between SF providers and outreach teams. Likewise, the cost data was not broken down between voucher and non-voucher clients, so the team was unable to compare the CYP costs for the two groups. However, MSS staff indicated that, as part of their agreement, the SF providers charged non￾voucher clients the same rates that the project paid for voucher clients. This indicates that the only difference in cost between voucher and non-voucher clients is the cost of reimbursing providers for the redeemed vouchers. Helpline Costs MSS has operated its helpline since 2008. The helpline was originally open 24 hours a day, 7 days a week. It responds to calls and offers information and counseling on family planning methods, reproductive health, referrals, and managing side effects. According to MSS reports, this call-in mechanism is viewed as a modality for improving access to resources for hard-to-reach areas. In 2013, as a partner with the MCH program, MSS expanded the scope of the helpline to address MCH issues. At the time of the final evaluation, according to MSS briefing documents, the helpline had reached 128,005 clients. More than 55 percent of callers were under age 30, 47 percent of calls were for family planning, 43 percent were for reproductive health, and 10 percent were for MCH concerns. Sixteen percent of callers were men. According to the annual reports, in year 1, 27 percent of 31,000 calls received were from FP/RH project areas, and in year 2, 15 percent of 36,500 calls were from project areas. The reports did not make clear why fewer calls came from the project areas in year 2, but data presented during the briefing shows that in April – June of year 1, a big promotion of the helpline increased calls from an average of 10,000 per month to 40,000 (Table 6). FHEs were active in this promotion, so it makes sense that more calls would originate from project areas, and this may account for the higher number of calls from intervention districts during year 1. One concern of project staff was that free access calls were limited to landline users and did not extend to mobile phones. They pointed 34 Abbas, “Costs and Utilization.” Family Planning and Reproductive Health—Final Evaluation 26 out in the briefing that only 7 percent of calls came from landlines, which are generally not available in the project areas. They noted that during the big helpline promotion effort, which was mainly done through mass media, most of their calls came from the urban and peri-urban areas of Karachi. In January 2014, MSS staff undertook a cost analysis of the helpline services (Table 6). They found the average cost per call to be too high, so they launched an internal (for project staff) and external (IEC materials with call information) promotion of the helpline, leading to an increase in the volume of calls and a lower cost per call. They could not maintain the volume after the promotion, however, and the cost per call increased again. To reduce the cost per call, in December 2015 MSS instituted a value-for￾money strategy: it reduced the number of helpline staff by 50 percent and discontinued the night shift. Cost per call dropped but then rose again, and project staff attributed this increase to the cost of returning missed calls. During final evaluation briefings, MSS staff stated that they had hoped to reduce the cost per call to a dollar or less but did not know if that would be possible with FP/RH ending. TABLE 6: HELPLINE COST FLUCTUATIONS, JANUARY 2104 – APRIL 2016 Date Cost/Call Explanation January 2014 $4.00 MSS undertook a cost analysis of the helpline services and the found cost per call to be too high. June 2014 $0.28 Promotion campaign increased call volume from between 10,000 and 14,000 per quarter to 41,000 per quarter. June 2015 $2.30 Effects of promotion campaign wore off and call volume dropped to between 8,000 and 10,000 per quarter. December 2015 $1.25 MSS conducted financial analysis and instituted value-for-money initiative, which reduced staff by 50 percent and eliminated the night shift. April 2016 $1.51 Costs increased to deal with call-backs from calls missed overnight. Source: Presentation from MSS personnel. Conclusion As these results indicate, the FP/RH project was making progress in reducing its cost per CYP and cost per helpline call. In addition, the helpline appears to fulfill a need for service referrals and information in areas well beyond the project intervention communities. One factor to consider, however, is how available telephones are in remote areas, and whether women will use them for helpline calls, especially since mobile phone calls are not covered by the free helpline service. Findings and Conclusions for Question 3: Best Practices and Lessons Learned Question 3: What best practices, innovations, and lessons learned can be applied to other existing or future programming in family planning and reproductive health? The information in this section comes entirely from interviews conducted by the evaluation team. Best Practices and Innovations The evaluation team asked health care providers, DoH/PWD staff, FHEs, SFSs, and MSS staff and partners to identify best practices and innovations that emerged from the experience of the FP/RH Family Planning and Reproductive Health—Final Evaluation 27 project. The respondents were asked to provide three or more best practices and innovations. In general, more responses addressed best practices than innovations. The most common best practice identified was the introduction of long-term methods (35 percent of 86 respondents). “Provision of this family planning method was a good practice for those clients who wanted to have a long-term method because they were not having complications with it and they remained safe and happy for a long time.” – Health care provider in Sukkur Other frequent responses included: providing free services or vouchers (22 percent of 86 respondents), conducting door-to-door visits and mobilization of communities (20 percent of 86 respondents), strengthening quality assurance and monitoring (20 percent of 86 respondents), building the capacity of service providers (19 percent of 86 respondents), strengthening infection prevention skills (16 percent of 82 respondents), increasing the awareness of family planning (13 percent of 86 respondents), introducing outreach camps and OMUs (10 percent of 86 respondents), and strengthening monitoring and evaluation systems (7 percent of 86 respondents). Not all interviewees recognized the strengthened monitoring and evaluation (M&E) as a key aspect of the FP/RH project, but the MSS staff interviewed considered this to be a key organizational strategy that was significantly strengthened during the project period. “Created a culture where evidence was reviewed regularly regarding performance trends. Decisions are made based on dashboards. This is challenging as many decisions in Pakistan are not based on this. MSS is proud of its optimal use of data and team decision making for interventions. Allows efficient and effective research and problem solving. Because of their success, MSS was given the lead of the MCH (5 partners) M&E working group by USAID. Two data quality assessments were reviewed and recognized by USAID.” – MSS senior manager Lessons Learned The evaluation team asked health care providers, DoH and PWD staff, FHEs, SFSs, and MSS staff and partners about lessons learned. Out of 14 response categories, the most frequently mentioned lesson learned (23 percent of 168 responses) was the importance of improved quality of care and standards of practice at the FP/RH facilities. Respondents generally noted that good patient care, good counseling skills, and updated professional knowledge and skills are necessary for a successful FP/RH practice. Below are comments regarding the project’s focus on training and follow-up. “Yes, it is important for our practice purpose as well. Like I was afraid of performing tubal ligation when [the] project was started. Doctor [trainer] said that she does not have time so I should come later. Again I waited for 40-45 days. Such training was important for my own skills development.” – Health care provider “We developed interest that it is not a superficial monitoring. They really take interest. They conduct pre-test and post-test. That also generate our interest.” – Health care provider “MSS has provided us training due to which we learned lots of new things and practices, and our skills to provide IUCD, injection, and implant are improved, and we learned about the infection prevention which we never observed before this program. So, it was good for us to improve our capacities.” – Health care provider Family Planning and Reproductive Health—Final Evaluation 28 Other frequent lessons learned included the importance of community awareness activities to generate demand for family planning services (14 percent of 168 responses) and of providing transport from remote areas to the outreach camps (7 percent of 168 responses). Providers most frequently (27 percent of 83 responses) recognized the importance of the demand generation and community mobilization activities that the FHEs provided for both SF providers and outreach camps. Family Planning Needs The evaluation team asked professionals (health care providers, DoH/PWD staff, FHEs, SFSs, and MSS staff and partners) what they saw as the top three family planning needs in their areas. The most frequent responses were increased awareness and education about family planning among communities and providers (19 percent of 195 responses), more availability of the IUCD (12 percent of 195 responses), and free services (19 percent of 195 responses). The need for more tubal ligation services was also frequently mentioned (9 percent of 195 responses). In fact, 33 percent of 195 responses identified a need for more LAPMs. The following comment illustrates this point. “Most of the people already have many children, so it is needed to motivate them for some permanent family planning method, and some people want to keep [a] long time space [before having the] next baby, so permanent and long-term family planning methods should be provided to the clients according to their needs.” – DoH official in Hyderabad When asked further why free services were so important, all respondents said because the clients were poor and had other priorities such as food. The comment below illustrates this. “Why: As people of this area are very much poor and family planning is their secondary priority. Their earning is [much] less so they should have free of cost family planning services.” – DoH staff, Karachi Although the evaluation team did not ask clients about their family planning needs, 35 a review of the most frequent responses from the FHEs who spent the bulk of their time in the communities talking with clients shows consistency with the above findings. The FHEs believed that free services (25 percent of 138 responses) and increased education and awareness raising in the communities about family planning (22 percent) were key. The IUCD (16 percent) was the next most frequent need identified. The evaluation team identified needs for more awareness and counseling about family planning methods and benefits and for long-term methods. This is consistent with the DHS finding that women seem unaware or unmotivated about family planning. According to the DHS, women in Sindh have a 10 percent unmet need for spacing and an 11 percent unmet need for limiting pregnancies. The MCH survey points out that only 30 percent of women who have reached their desired family size are using contraception. The DHS notes that 20 percent of women nationally (and the same percent in Sindh) have an unmet need for family planning, and it points out that if all the women who said they wanted to limit or space their births were to use a contraceptive method, the CPR would increase to 56 percent. Conclusion Best practices that can be applied to other family planning or reproductive health programming include focusing on LAPMs for clients who want to limit their family size or space their pregnancies farther apart, and providing free services (or vouchers) for clients for whom family planning may not be a financial priority. The evaluation did not, however, identify ways to make free services sustainable. 35 Due to cultural sensitivities and to protect the privacy of the clients around a difficult topic such as family planning, the evaluation team decided not to ask clients about their family planning needs. Family Planning and Reproductive Health—Final Evaluation 29 Quality assurance and monitoring also play a key role in improving a project’s effectiveness. Building the capacity of service providers leads to improved quality of care and standards of practice, which in turn can generate demand for family planning services. Community outreach, education, and awareness activities are also key to meeting current unmet needs and generating more demand. Employing FHEs for demand creation was the cornerstone of MSS’s community outreach component. The strategy that MSS employed was effective but not necessarily sustainable once funding ended. OTHER FINDINGS AND CONCLUSIONS In addition to the final evaluation questions, USAID asked the evaluation team to address questions related to new users and sustainability. New Users MSS and PWD disagreed on how to define a “new user.” Both agreed that someone who had never used a modern method of contraception before should be considered a new user, but the two institutions used different cut-off points to determine whether a returning client who had stopped using her former method should be considered “new” or “continuing.” MSS used three months as a reference point to define a new client, and the PWD used one year. In an attempt to address this disagreement, USAID asked MSS to calculate the percent of “new users” using six months as the reference point. MSS did this for year 3 and noted in its briefing that 50 percent of clients had not used a modern method of contraception in the previous 6 months. In other words, half of its clients were new users. The implication of this is that the FP/RH project has made significant progress addressing unmet family planning needs, and if FP/RH had continued for the entire life of project, it probably would have met its targets. Sustainability Another question is how to make a project that was just beginning to make inroads in remote rural areas sustainable. To address the lack of awareness about family planning and lack of demand, the project invested in training FHEs to conduct home visits and community mobilization activities. The project also offered vouchers for poor clients to obtain free services from the newly trained providers in the community or outreach site. As noted above, these factors were key in attracting family planning clients. While many of these clients used LAPMs, which require less frequent follow-up, others used short-term methods. It takes time and motivation for clients to decide to use LAPMs, and having access to free services that do not require long travel is an important factor in their decisions. Given the remoteness and extreme poverty of the target population, it is unlikely that they will be easily convinced to use their scarce resources to pay for family planning services. According to project data, 72 percent of clients have either no education or less than primary level, indicating that most of the clients are poor and not literate. Despite this, the project was able to increase the proportion of non-voucher clients from 15 percent to 26 percent from year 1 to year 2. The project attributes this to its success in generating demand through the FHEs and the provision of safe, quality services at the SF clinics. The evaluation team also found that 70 percent of 40 clients interviewed stated they would continue with local providers even if the vouchers were discontinued. The most frequent reason cited (26 percent of 44 clients) was because of the quality of care. The second most frequent response (22 percent) was because the clients were now aware of the availability and importance of birth spacing services. However, despite this, because sustainability of services is such an important issue for USAID (to ensure continuation of results) one does wonder if the increase in paying clients was in part due to a push to Family Planning and Reproductive Health—Final Evaluation 30 increase the number of non-voucher clients, and if this inadvertently would reduce the proportion of poorest clients. One of the dilemmas that MSS presented to the team during the final evaluation briefing (see findings for Question 2) concerned mixed directives from USAID. USAID encouraged MSS on one hand to target more of the poorest clients (earning less than $1.25 per day) and on the other to further sustainability, which meant reducing the number of voucher clients. While MSS staff recognized the need for sustainable services, they also believed that to reach the poorest populations with LAPMs, it was necessary to offer free services. Charging clients for services also indirectly influences the choice of methods. If clients cannot afford LAPMs, then the cost reduces their ability to choose them. One positive, sustainable result is that trained providers in the project areas are now able to provide family planning services. Their service levels, however, will be impacted because clients will have to pay for services and supplies. Moreover, it is unlikely that the FHE demand generation work will continue in the communities. Only 30 percent of 20 providers said that they would continue to work with the FHEs and pay them a salary with their own resources because they valued their support. As noted in responses to Questions 1 and 3, FHEs have been important in increasing the number of clients. As noted in the “Project Background” section of this report, one of the constraining factors for family planning in Pakistan is that although 98 percent of people are aware of family planning, only 26 percent use modern methods. The DHS reports that the public mistrusts promotional messages about family planning, and the main reasons for this are poor quality of care and side effects. This is an area that the FHEs can address in their communities. They were successful in addressing these concerns with potential clients by convincing them that they would receive good quality care from the provider and that the provider and FHE would help them with any side effects or other family planning concerns. Because the target population is remote and poor, the evaluation team expects that they will require free health and family planning services for some time. Unlike the public sector, MSS has been successful in reaching this part of the population because of its rural strategy and focus. However, unless donors are willing to support free services, it is not likely that this model will be sustained by private NGOs. The public sector clinics do not reach many of these areas, and staff availability is uneven. The LHWs work in the communities, but they distribute only short-term methods, and as USAID noted in a meeting with the evaluation team, cover only 40-50 percent of the rural areas. The evaluation team therefore concludes that addressing the lack of family planning services in remote areas will require an family planning-focused program with significant budget, trained staff, free services, a great deal of promotion, and a sustained commitment from the public and/or private sectors. GENERAL CONCLUSIONS More detailed conclusions and summaries are noted at the end of the findings for each evaluation question or objective. Following are some overall conclusions.  Almost all objectives were met. The few indicators that were not met (vouchers distributed, public facilities refurbished, CYPs, CPR increase) were on target to be met by the original end of the project in 2018.  The MSS model of service delivery deserves credit for much of the success: (1) improved access through the availability of numerous SF clinics and OMUs in remote rural areas; (2) high level of quality assurance and monitoring of services; (3) demand generation activities Family Planning and Reproductive Health—Final Evaluation 31 through the FHEs that included making services affordable to impoverished populations through vouchers.  The project was improving cost efficiency in its activities. By 2015, costs per CYP were coming down, as were helpline costs. (The project was not able to provide the data for a more informative analysis of costs, such as CYP by method.)  Client exit interviews showed 95–100 percent satisfaction with family planning services.  The biggest obstacles to seeking services if an SSF facility was not available were travel distance and cost. MSS strived to address both these obstacles in the project strategy. To further explore obstacles and potential solutions, it would have been useful to have additional evidence of service quality, particularly if the client exit surveys or research department documented discontinuation rates and method complication rates.  Lessons learned and family planning needs identified call for more awareness generation about family planning and free services to improve access. There is also a need for LAPMs to meet limiting and spacing needs. The best practices support these findings and also stress the importance of good quality of care to satisfy clients and keep them as continuing family planning users.  One of the requirements for the evaluation was to explore differences between facilities in rural, semi-rural, and urban areas. The evaluation team found no differences in the responses of the interview respondents when data was disaggregated on this dimension. RECOMMENDATIONS  Pakistan’s low CPR, high TFR, cultural barriers, and large rural population require the kind of family planning-focused strategies and service delivery model employed by the FP/RH project. To address Pakistan’s 2020 goal of moving from 35 percent CPR to 55 percent CPR, future USAID family planning programming should continue to have a vertical focus and, if integrated with other health programs, the focus should be on family planning. (See the “Sustainability” section for findings and conclusions related to this recommendation.)  Until the CPR increases, USAID family planning programming should continue to rely on the private sector to reach remote rural populations, given the scarcity of accessible public facilities and the uneven availability of trained staff, supplies, and equipment. An appropriate solution would be to find a way for USAID to support the public and private sectors to work together to address the low CPR and related maternal health risks in remote rural areas. This requires that the government and donor agencies be willing to work with the private sector, and may entail negotiations regarding how improvements and credit for meeting CYPs, CPR, and other standards will be attributed to both the public and private sectors.  USAID/Pakistan should also work with the private sector and government to improve advocacy and policies for family planning access in remote rural areas. Some potential areas for cooperation include (1) supporting efforts to certify paramedic providers to provide LAPMs in remote rural areas and (2) increasing family planning efforts by LHWs and other community educators in remote rural areas. Family Planning and Reproductive Health—Final Evaluation 32  Future USAID/Pakistan family planning programming should include components that focus on family planning awareness activities with men. Men are the key decision makers, and most women will seek family planning services only if they have their husband’s support. One way to address this issue would be for future programming to include educational activities specifically directed to men, such as men’s community education groups held at hours convenient for working men and facilitated by male health educators.  If USAID/Pakistan requires more detailed cost analysis of family planning programs, it should specify that implementing partners collect and disaggregate data appropriately. In the case of this evaluation, that would have meant collecting additional data on costs per unit of output (e.g., CYP by method and service delivery point) and the cost of providing services through vouchers relative to not using vouchers, disaggregated by method. Family Planning and Reproductive Health—Final Evaluation 33 ANNEXES Annex 1: Evaluation Scope of Work Family Planning and Reproductive Health—Final Evaluation 34 Family Planning and Reproductive Health—Final Evaluation 35 Family Planning and Reproductive Health—Final Evaluation 36 Family Planning and Reproductive Health—Final Evaluation 37 Annex 2: Assignment Work Plan Family Planning and Reproductive Health—Final Evaluation 38 Family Planning and Reproductive Health—Final Evaluation 39 Family Planning and Reproductive Health—Final Evaluation 40 Family Planning and Reproductive Health—Final Evaluation 41 Family Planning and Reproductive Health—Final Evaluation 42 Family Planning and Reproductive Health—Final Evaluation 43 Family Planning and Reproductive Health—Final Evaluation 44 Family Planning and Reproductive Health—Final Evaluation 45 Family Planning and Reproductive Health—Final Evaluation 46 Family Planning and Reproductive Health—Final Evaluation 47 Family Planning and Reproductive Health—Final Evaluation 48 Family Planning and Reproductive Health—Final Evaluation 49 Family Planning and Reproductive Health—Final Evaluation 50 Family Planning and Reproductive Health—Final Evaluation 51 Family Planning and Reproductive Health—Final Evaluation 52 Family Planning and Reproductive Health—Final Evaluation 53 Family Planning and Reproductive Health—Final Evaluation 54 Family Planning and Reproductive Health—Final Evaluation 55 Family Planning and Reproductive Health—Final Evaluation 56 Family Planning and Reproductive Health—Final Evaluation 57 Family Planning and Reproductive Health—Final Evaluation 58 Family Planning and Reproductive Health—Final Evaluation 59 Family Planning and Reproductive Health—Final Evaluation 60 Family Planning and Reproductive Health—Final Evaluation 61 Family Planning and Reproductive Health—Final Evaluation 62 Annex 3: Data Collection Cluster District Number of Facilities Selected Number of Individual Interviews Hyderabad Hyderabad 4 14 Nawabshah 1 4 Umerkot 0 2 Matiari 2 9 Tando Allahyar 0 3 Karachi Karachi 8 40 Sukkur I Sukkur 1 6 Shikarpur 3 11 Khairpur 1 4 Larkana 1 6 Mirpurkhas 3 4 Qambar Shahdadkot 2 4 Sukkur II Kashmore 2 9 Ghotki 5 18 Family Planning and Reproductive Health—Final Evaluation 63 Annex 4: List of Individuals Interviewed This annex has been removed from the public report to protect the confidentiality of respondents. Family Planning and Reproductive Health—Final Evaluation 64 Annex 5: Data Collection Instruments Introduction Section for all Respondents (except clients)36 S1. Date (day/month/year): ___________________________________________ S2. Interviewer name: _______________________________________________ S3. Interview location (city/village): _____________________________________ S4. Interview type (CIRCLE ONE NUMBER) 1 DoH government officials 2 PWD government officials 3 MSS staff 4 Sub-awardee (Marie Stopes International, CSM, HANDs) 5 Health care providers – SSF 6 Health care providers – others 7 Health educators/SFS 8 Field Health Educators (FHEs) 9 Outreach team member(s) S5. Interviewee name: ______________________________________________ S6. Organization of interviewee: ______________________________________ S7. Interviewee title: _______________________________________________ S8. Interviewee profession: _______________________________________________ My name is ______________________. I work for a research organization based in Islamabad. We are conducting a study to assess the benefits of family planning in the cities and villages of Karachi, Hyderabad, Sukkur Multan, and Bahawalpur region. We would like to ask you a few questions. This interview will take approximately 30-45 minutes to conduct. Do I have your permission to record interview? (Yes/No ____________) 36 The first 14 introduction questions were part of each respondent type’s data collection tool except for the clients. Hence additional questions that were asked from the government, MSS staff, MSS partners, health care providers, and FHEs start their numbering from 15. Family Planning and Reproductive Health—Final Evaluation 65 Questions for all interviewees (except clients): 1: To what extent has the project been successful in meeting its four major objectives? Obj. 1: Increasing the size and capacity of the health workforce. 1. In your opinion or experience, to what extent, if any, has the FP/RH project impacted the number of health care providers delivering family planning services? Would you say it has greatly increased, somewhat increased, or not affected, the number of family planning service providers? (CIRCLE ONE NUMBER) 1 Greatly increased 2 Somewhat increased 3 Stayed the same 4 Don’t know 5 Refused 2. Can you tell me how the FP/RH project increased the number of family planning service providers or, if it did not, why not? 3. To what extent, if any, has the FP/RH project affected the capacity of providers to deliver family planning services? Would you say the project greatly increased capacity, somewhat increased capacity, or that capacities did not change? (CIRCLE ONE NUMBER) 1 Greatly increased 2 Somewhat increased 3 Stayed the same 4 Don’t know 5 Refused 4. If the project increased the capacity (knowledge, skills, or behaviors) of service providers, what capacities did it build? If it did not change capacities, why do you think it failed to do so? 5. (IF CAPACITY INCREASED) In your opinion, has the increased capacity of FP providers improved/benefited the communities in the project areas? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know Family Planning and Reproductive Health—Final Evaluation 66 Objective 2: Scaling up facility-based service delivery 6. We would like to know the extent, if any, to which the MSS FP/RH activities have changed access to FP services for communities in the project areas. Would you say access to FP services has become much better, somewhat better, somewhat worse, much worse, or stayed the same? (CIRCLE ONE NUMBER) 1 Become much worse 2 Become somewhat worse 3 Stayed the same 4 Become somewhat better 5 Become much better 6 Don’t know 7 Refused 7. (SKIP IF RESPONDENT SAID “STAYED THE SAME” OR “DON’T KNOW”) In what specific ways did access to FP services change? (Probe for any direct links between the project and improvements in services) (WRITE A RESPONSE) 8. We would like to know the extent, if any, to which the MSS FP/RH activities have changed the quality of FP services for communities in the project areas. Would you say the quality of FP services has become much better, somewhat better, somewhat worse, much worse, or stayed the same? (CIRCLE ONE NUMBER) 1 Become much worse 2 Become somewhat worse 3 Stayed the same 4 Become somewhat better 5 Become much better 6 Don’t know 7 Refused 9. (SKIP IF RESPONDENT SAID “STAYED THE SAME” OR “DON’T KNOW” OR “REFUSED”) In what ways did the quality of FP services change? (Probe for any direct links between the project and improvements in services) (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 67 10. In your opinion, are the family planning needs of people living in the project areas/ communities met? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 11. Please explain why you believe the FP needs of the people living in the project areas/communities are being met. (WRITE A RESPONSE) 3. What best practices, innovations, and lessons learned can be applied to other existing or future programming in family planning and reproductive health? 12. In your opinion, what best practices and/or innovations have been created by the FP/RH project, in order of importance? (ORDER THE LIST FROM MOST TO LEAST IMPORTANT) Importance Best practices and/or innovation identified? (WRITE RESPONSES FOR EACH PRIORITY) Why is it important? (WRITE RESPONSES FOR EACH PRIORITY) 1. 2. 3. 13. In your opinion, what are the three most important lessons that have been learned from the FP/RH project? (ORDER THE LIST FROM MOST TO LEAST IMPORTANT) Importance Lessons learned? (WRITE RESPONSES FOR EACH PRIORITY) Why is it important? (WRITE RESPONSES FOR EACH PRIORITY) 1. 2. 3. Family Planning and Reproductive Health—Final Evaluation 68 14. In your opinion, what are the top three FP needs in the area where the project was implemented, in order of priority, and why? (ORDER THE LIST FROM MOST TO LEAST IMPORTANT) Importance Most important FP needs? (WRITE RESPONSES FOR EACH PRIORITY) Why is it important? (WRITE RESPONSES FOR EACH PRIORITY) 1. 2. 3. Family Planning and Reproductive Health—Final Evaluation 69 Additional Questions for DOH/PWD 15. In your opinion has the FP/RH project contributed to improving the capacity of public health care providers to provide high-quality family planning services? Has it greatly improved, somewhat improved, or not affected the capacity of public health care providers? (CIRCLE ONE NUMBER) 1 Greatly improved 2 Somewhat improved 3 Stayed the same 4 Don’t know 5 Refused 16. (IF GREATLY OR SOMEWHAT IMPROVED) Please explain how the FP/RH project contributed to improving the capacity of public health care providers? (WRITE A RESPONSE) (ASK FOR DOH ONLY) 17. Has the FP/RH project provided equipment to the public health facilities? (CIRCLE ONE NUMBER) 1 Yes 2 No IF YES, 17a. Did the project provide instruments and equipment? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 Family Planning and Reproductive Health—Final Evaluation 70 17b. Did the project provide furniture? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 17c. Did the project provide fixtures? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 17d. Did the project provide consumable supplies? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 Family Planning and Reproductive Health—Final Evaluation 71 17e. Did the project provide contraceptive supplies? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 (ASK FOR DOH ONLY) 18. Were public facility staff involved in the outreach camps at their facilities? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 18a. If yes, how were they involved? 19. In your opinion, how effective were the project’s efforts to scale up outreach and community￾based service delivery? Were they very effective, somewhat effective, or did they not make any difference in scaling up community based service delivery? (CIRCLE ONE NUMBER) 1 Very effective 2 Somewhat effective 3 Made no difference 4 Don’t know 5 Refused 20. Please explain why you thought the project was effective or ineffective. (WRITE A RESPONSE) 21. Do the LHWs/FWA work with FHEs? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused Family Planning and Reproductive Health—Final Evaluation 72 22. IF YES, What kind of cooperation, if any, is there between them and how, if at all, has it affected access to services? (WRITE A RESPONSE) 23. In your opinion, did the voucher system used for outreach make any difference in the number of patients seeking FP services? 1 Yes 2 No 3 Don’t know 4 Refused 23a. IF YES, why did it make a difference? (WRITE A RESPONSE) 24. In your opinion, did MSS administer the voucher system well? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 25. Please explain why? (WRITE A RESPONSE) 26. Is the supply of contraceptives managed by the DoH/PWD adequate or have there been problems with stock outs? (CIRCLE ONE NUMBER AND FILL IN BLANKS IF RESPONDENT CHOOSES OPTION 2) 1 Adequate 2 Problems with stock outs of ______________ (WRITE IN PRODUCT) at _______________ level (WRITE IN JURISDICTIONAL LEVEL e.g., facility level, district level, regional level, provincial level) 27. Can you explain how the government is planning to manage the contraceptive supply chain after USAID support ends? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 73 28. Who provides the contraceptive supplies for the outreach camps? (CIRCLE ALL THAT APPLY AND FILL IN BLANKS IF RESPONDENT CHOOSES OPTION 4) 1 MSS 2 DOH 3 PWD 4 Other_________ 29. Who provides the contraceptive supplies for the LHW program activities? (CIRCLE ALL THAT APPLY AND FILL IN BLANKS IF RESPONDENT CHOOSES OPTION 4) 1 MSS 2 DOH 3 PWD 4 Other_________ (ASK FOR NATIONAL, PROVINCIAL, AND REGIONAL DOH/PWD STAFF) 30. Please explain what, if any, contributions MSS has made to the interpersonal communication (IPC) tool kit for the national program? (WRITE A RESPONSE) (ASK FOR NATIONAL, PROVINCIAL, AND REGIONAL DOH/PWD STAFF) 31. Please explain what, if any, contributions MSS has made to the revision of the FP curricula (mid￾level providers such as midwives and LHVs) and CHW training manual? (WRITE A RESPONSE) 32. In your opinion what, if any, effect does the distance between SSF and public facilities have on the FP services being delivered at the public health facilities? Have FP services at the PH facility greatly decreased, somewhat decreased, not changed, somewhat increased, or greatly increased? (Probe: Has there been an impact on FP service delivery at the public health facilities (PWD/DoH)?) (CIRCLE ONE NUMBER) 1 Greatly decreased 2 Somewhat decreased 3 Stayed the same 4 Somewhat increased 5 Greatly increased 6 Don’t know 7 Refused Family Planning and Reproductive Health—Final Evaluation 74 33. In your opinion, why do women who choose to seek family planning services from an SSF provider choose the SSF facility over the public facility or why do they choose the public facility over the SSF? (Probe: does it depend on the type of service they seek, quality, provider?) 34. In your opinion, have the outreach activities affected the utilization of public health facilities? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 35. IF YES, In what way? (WRITE A RESPONSE) 36. How does the DoH/PWD define a “new” family planning client? (WRITE A RESPONSE) 37. Do you have any other comments or observations about the MSS FP/RH project? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 75 Additional Questions for CSM, Marie Stopes International, and HANDS 15. In your opinion, to what extent, if any, has the FP/RH project contributed in improving the capacity of health care providers? Would you say their capacity has greatly increased, somewhat increased, or stayed the same? (CIRCLE ONE NUMBER) 1 Greatly increased 2 Somewhat increased 3 Stayed the same 4 Don’t know 5 Refused 16. Can you explain the nature and extent of your participation in the FP/RH project? (WRITE A RESPONSE) 17. How would you rate your working relationship with MSS? Compared to your working relationships with other organizations would you say it was above average, average, or below average? (CIRCLE ONE NUMBER) 1 Above average 2 Average 3 Below average 4 Don’t know 5 Refused 18. Please explain the rationale for your rating. (Probe: Why did you rate the relationship above average/average/below average?) (WRITE A RESPONSE) 19. (ASK ONLY OF CSM AND HANDS) Why was the partnership discontinued? (WRITE A RESPONSE) 20. What is your impression of the way that MSS is implementing the FP/RH project? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 76 21. In your opinion has MSS made any significant contributions to FP/RH through this project? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 22. Please explain your answer. (Probe as appropriate: What contribution has it made? Why did it not make any contribution?) (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 77 Additional Questions for MSS Staff (COP, Admin & Operations, MDT, M&E, Finance (SSF, Vouchers & OR), Helpline, Outreach Teams) 15. How would you describe your relationship with USAID? Would you rate it as excellent, good, fair, or poor? (CIRCLE ONE NUMBER) 1 Excellent 2 Good 3 Fair 4 Poor 5 Don’t know 6 Refused 16. Please explain the rationale for your answer. (Probe as appropriate: Why did you rate the relationship as excellent/good/fair/poor?) (WRITE A RESPONSE) 17. Why do you think USAID is closing the program? (WRITE A RESPONSE) 18. In your opinion, how, if at all, did MSS benefit from its work on the FP/RH project? (WRITE A RESPONSE) 19. In your opinion, what, if any, challenges did MSS encounter working on the FP/RH project? (WRITE A RESPONSE) 20. In your opinion, which activities are likely to continue or be sustained beyond the life of the project? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 78 20a. Why do you think these activities will continue? (WRITE A RESPONSE) (ASK ONLY FROM ADMIN, OPERATIONS, AND FINANCE STAFF) 21. Please explain what steps, if any, MSS took to upgrade public health facilities? (WRITE A RESPONSE) 22. In your opinion, is it possible to decrease the use of vouchers without reducing utilization of FP services in marginalized communities? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Maybe under certain conditions 23. Please explain your answer. (Probe if necessary: How important are vouchers to increasing utilization of FP services? IF NO, why not? IF MAYBE, Under what conditions?) (WRITE A RESPONSE) (ASK OF CENTRAL MSS STAFF ONLY) 24. Are any of the other components of MNCH supporting the MSS hotline? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused (ASK OF CENTRAL MSS STAFF ONLY) 25. IF YES, Who is providing the support and what is the nature of the support? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 79 (ASK OF CENTRAL MSS STAFF ONLY) 26. Will any USAID project or other donor support the hotline after the FP/RH project ends? (WRITE A RESPONSE) 1 Yes 2 No 3 Don’t know/refused (ASK OF CENTRAL MSS STAFF ONLY) 27. If yes, who are the programs/ donors? (WRITE A RESPONSE) 28. ASK ONLY OF THE DCO. Please describe the working relationship, if any, that has been developed at the district level between FHEs and the LHW program network? (WRITE A RESPONSE) (ASK OF CENTRAL MSS STAFF ONLY) 29. Was the purpose of refurbishing the public health facilities to make them suitable to support outreach camp activities or was it for some other purpose? (CIRCLE ONE NUMBER) 1 Support outreach camps 2 Some other purpose 3 Don’t know 4 Refused (ASK OF CENTRAL MSS STAFF ONLY) 30. How many clinics have been refurbished? ___________ (WRITE THE NUMBER) 30a. Why not more? (WRITE A RESPONSE) 31. Do you think the public sector facilities where MSS has provided outreach services will continue to provide the same family planning services after the MSS outreach teams stop coming/ project ends? 1 Yes 2 No 3 Don’t know 4 Refused Family Planning and Reproductive Health—Final Evaluation 80 31a. Please explain which services will continue and why. (WRITE A RESPONSE) 31b. Please explain which services will not continue and why. (WRITE A RESPONSE) 32. Do you have any other observations about the project or suggestions you would like to share? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 81 Additional Questions for HCP (SSF, Public HCPs, outreach team providers) 15. What FP services did you provide prior to the FP/RH project and which FP services do you provide now? FP services 68a. Services you offered prior to 2012? (CIRCLE NUMBERS OF ALL THAT APPLY) 68b. Services you offer now? (CIRCLE NUMBERS OF ALL THAT APPLY) Condoms (CD, RC) 1 1 Oral pills (COCs, POPs) 2 2 Injectable – Depo (3 months) 3 3 Injectable (1 month) 4 4 IUCD 5 5 Implant 6 6 Tubal ligation 7 7 Counseling 8 8 Other (SPECIFY)___________ 9 9 Other (SPECIFY)___________ 10 10 Other (SPECIFY)___________ 11 11 16. What is the most common FP service/ method you provide? (CIRCLE ONE NUMBER) 1 IUCD 2 Injectable (3 month) 3 Injectable (1 month) 4 Pills 5 Condoms 6 Other ________________________ (FILL IN) 17. COLLECT A COPY OF THE REPORT LOGS TO VERIFY FP SERVICE PROVISION AND TYPE OF SERVICE. DO THE REPORT LOGS SUPPORT THE ANSWERS TO THE PREVIOUS TWO QUESTIONS? EXPLAIN DISCREPANCIES IF ANY. 18. How do you report the number of IUCD insertion procedures to MSS? (CIRCLE ONE NUMBER) 1 Monthly – on a form provided by MSS 2 On your own form, ledger used by your office 3 Other (SPECIFY) _____________________ Family Planning and Reproductive Health—Final Evaluation 82 19. Please explain how you maintain your contraceptives supply. (WRITE A RESPONSE) 20. ENUMERATOR: CHECK STORAGE AREA FOR FP SUPPLY. CHECK DO THEY HAVE A SYSTEM FOR STORING AND ORDERING CONTRACEPTIVES I.E. BIN CARD, HOW TO STORE, RECORDING NUMBERS ETC? WRITE YOUR OBSERVATION. NOTE ANY DISCRIPENCIES BETWEEN ANSWER TO QUESTION ABOVE) 21. When did you join the SSF network? (Enumerator: If they joined before 2012 please ask them to explain what program this was?) (WRITE A RESPONSE) 22. What kind of training you have received through MSS under the FP/RH project? (CIRCLE ALL NUMBERS THAT APPLY) 1 Module 1.1 (FP theory and business plan) 2 Module 1.2 (IPC, CS, FP Counseling) 3 Module 2.1 (FP Clinical Practicum as per level of HCP) 4 BEmONC 23. Which training(s) did you find useful? (CIRCLE ALL THAT APPLY) 1 Module 1.1 (FP theory and business plan) 2 Module 1.2 (IPC, CS, FP Counseling) 3 Module 2.1 (FP Clinical Practicum as per level of HCP) 4 BEmONC 24. Please tell us which of the trainings you said were useful was most useful and second most useful and why 77a. Training? (ENTER APPROPRIATE NUMBER FROM PREVIOUS QUESTION) 77b. Why was this training useful? (WRITE RESPONSES) Most useful Second most useful Family Planning and Reproductive Health—Final Evaluation 83 25. Did the type or quality of the services you provide change as a result of the trainings? (CIRLCE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 26. IF YES, Please explain the changes you made in your services? (WRITE A RESPONSE) 27. Did the training you received from MSS affect your skill level? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 28. IF YES, Please list the new skills and tell us how those changed your practice. (LIST NEW OR IMPROVED SKILLS THEN WRITE RESPONSE TO HOW THE PRACTICE WAS CHANGED) IF NO, Why did the training not improve skills? 29. Did you work with an FHE through the MSS FP/RH project? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 30. IF YES, What was the role of the FHE and how effective was the work of the FHE for your practice? (WRITE A RESPONSE) 31. Were there challenges in working with the FHEs? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused Family Planning and Reproductive Health—Final Evaluation 84 32. IF YES, What challenges did you experience working with the FHEs? (WRITE A RESPONSE) 33. What is your opinion of the voucher system? Do you think it helped increase, somewhat increase, or not make any difference in the number of patients seeking FP services in your practice? (CIRCLE ONE NUMBER) 1 Increase 2 Somewhat increase 3 No change 34. What was your opinion regarding how the voucher system was administered by MSS? (WRITE A RESPONSE) 35. What is the difference, if any, in the price for services/methods you charge to non-voucher clients and what you receive from MSS for voucher clients? (Probe: Focus on IUDs but ask about all MSS supported services) (FILL IN COSTS FOR EACH OPTION BELOW) 1 What MSS pays for voucher service ________________________________ 2 What you charge for the same service to non-voucher clients __________ 36. Did you work with the LHW network in your area? (CIRCLE ONE NUMBER) 1 Yes 2 No 37. IF YES, What was the role of the LHWs and how effective was their work for your FP practice? (WRITE A RESPONSE) 38. Did you experience any challenges working with the LHWs? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused Family Planning and Reproductive Health—Final Evaluation 85 39. IF YES, What challenges did you experience working with LHWs? (WRITE A RESPONSE) 40. Has there been a change in the number of FP clients in your practice as a result of MSS’s FP/RH project support? Have the numbers greatly increased, somewhat increased, or not changed since you started working with MSS? (CIRCLE ONE NUMBER) 1 Increased 2 Somewhat increased 3 No change 4 Don’t know 5 Refused 41. Please explain why the project increased or did not change the number or clients you serve in your practice. (Enumerator: If increased ask for numbers if possible) (WRITE A RESPONSE) 42. IF INCREASED, Is your practice continuing to grow by adding new clients at this time? Please explain how and why? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 86 43. Has the MSS FP/RH project provided equipment and supplies to the SSF? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused IF YES, 43a. Did the project provide instruments and equipment? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 43b. Did the project provide furniture? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 43c. Did the project provide fixtures? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 Family Planning and Reproductive Health—Final Evaluation 87 43d. Did the project provide consumable supplies? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 43e. Did the project provide contraceptive supplies? (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 43f. List any other instrument or supplies provided (if any) (LIST AND VERIFY) Item (LIST ITEMS) Verification (CIRCLE ONE NUMBER) Yes No 1 2 1 2 1 2 1 2 1 2 1 2 1 2 44. How, if at all, will your practice/services change after the support from the MSS- FP/RH project is discontinued? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 88 45. Will you continue to support and work with the FHE after support from the MSS-FP/RH project is discontinued? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 46. IF YES, Please explain how and why you will continue to support or work with the FHEs. IF NO, Why not? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 89 Additional Questions for the Field Workers (SFS & FHEs) 15. When did you join the SSF network? (Enumerator: If they joined before 2012 please ask her to explain what program this was?) (WRITE A RESPONSE) 16. What is the role of an FHE? Please describe your work in the community? (WRITE A RESPONSE) 17. Please describe what kind of training you have received through MSS under the FP/RH project? (list topics) (WRITE A RESPONSE) 18. Did these trainings have an impact on your support to the SSF HCP/ or outreach camp? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 19. Please explain how? (WRITE A RESPONSE) 20. What skills did you learn as a result of the training you have received from MSS? (List skills) (WRITE A RESPONSE) 21. How would you describe your relationship with the SSF? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 90 22. How would you describe your relationship with the LHW network working in your area/s? (WRITE A RESPONSE) 23. Are you going to continue working with the SSF after the support from FP/RH project is discontinued? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 24. If yes, please explain how? (WRITE A RESPONSE) 25. In your opinion, what have been the biggest obstacles for women in your area to seek FP care? (CIRCLE ONE NUMBER) 1 Knowledge 2 Distance too far to provider 3 Method and service were too expensive 4 Family/ husband objected 5 Religious leaders 6 Service was not available in the area 7 Other _________________________ 26. Please explain how you have overcome or not overcome this/ these obstacles? (WRITE A RESPONSE) 27. Do you think the voucher program affected the number of patients that obtained FP services? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused Family Planning and Reproductive Health—Final Evaluation 91 28. IF YES, Please explain how the voucher program affected the number of patients that obtained FP services? (WRITE A RESPONSE) 29. In our opinion, how well did MSS administer the voucher program? (WRITE A RESPONSE) 30. Do you have many non-voucher clients? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 31. IF YES, About what proportion or percentage of your patients are non-voucher clients? (ENTER PERCENT) __________________ 32. In your opinion, about what proportion or percentage of your voucher clients would be willing to pay for the cost of FP services without vouchers after the end of the project? (CIRCLE ONE NUMBER) 1 None 2 About 10% or less 3 About 25% (one-quarter) 4 About 50% (one half) 5 About 75% (three-quarters) 6 About 100% (all) 7 Other (FILL IN PERCENTAGE)_____________________________ 8 Don’t know/refused 33. Please explain why you think they would be willing to pay or not pay for the services? (WRITE A RESPONSE) 34. Do you believe that the number of FP clients in your community have increased as a result of your work through MSS’s FP/RH project support? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused Family Planning and Reproductive Health—Final Evaluation 92 35. IF YES, Please explain how and why you think your work through the FP/RH project increased the number of FP clients in your community. (WRITE A RESPONSE) 36. At this time are the numbers of FP clients that you visit in your community/ies continuing to grow? (How and why?) (WRITE A RESPONSE) 37. Have you promoted FP services to male family members as well as women? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 38. IF YES, about what percentage of men were interested in FP? (CIRCLE ONE NUMBER) 1 About 10% or less 2 About 25% (one-quarter) 3 About 50% (one half) 4 About 75% (three-quarters) 5 About 100% (all) 6 Other (FILL IN PERCENTAGE)_____________________________ 7 Don’t know/refused 39. Do you think more FP education should be provided to men? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 40. IF YES, Please explain why you think more FP education should be provided to men. (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 93 Questions for the Clients 1. What motivated you to seek family planning services? (WRITE A RESPONSE) 2. What did your husband think about your desire to seek FP services? (WRITE A RESPONSE) 3. Have you ever used a FP method before? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 4. IF YES, Which methods have you used? (CIRCLE ALL THAT APPLY) 1 Condoms 2 Pills (COCs, POPs) 3 IUCD 4 Injectable (1 month) 5 Injectable - Depo (3 month) 6 Implant 7 Other____________ (WRITE METHOD) 5. IF YES, Who provided the services? (CIRCLE ALL THAT APPLY FOR PUBLIC AND/OR PRIVATE AND PROVIDERS AND FILL IN DISTANCE AND TIME) 130a. Public 130b. Private Lady health worker (LHW) 1 1 Community midwife (CMW) 2 2 Family welfare worker (FWW) 3 3 Lady health visitor (LHV) 4 4 Nurse midwife 5 5 General practitioner 6 6 Other 7 7 Family Planning and Reproductive Health—Final Evaluation 94 6. IF YES, which method did you use last? 1 Condoms 2 Pills (COCs, POPs) 3 IUCD 4 Injectable (1 month) 5 Injectable - Depo (3 month) 6 Implant 7 Other____________ (WRITE METHOD) 6a. How far was the provider from your home? ____________ (ENTER DISTANCE IN KM) 6b. How did you get there? (WRITE RESPONSE) 1 On foot 2 Motorcycle 3 Car/taxi 4 Public transport (QinQi, rickshaw, bus, etc.) 5 Others, (SPECIFY) ____________________ 7. What methods are you using now from the SSF provider? (CIRCLE ALL THAT APPLY) 1 IUCD 2 Injectable 3 Pills 4 Condoms 5 Other (SPECIFY) ____________________________ 8. If different from the method you used before going to the SSF, Why did you not use this method before? (CIRCLE ALL NUMBERS THAT APPLY) 1 Knowledge 2 Distance too far to provider 3 Method and service were too expensive 4 Family/ husband objected 5 Religious leader objected 6 Service was not available in the area 7 Other (SPECIFY) _________________________ 9. Please explain further why you did not use this method before and what changed. (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 95 10. Have you received information from the FHE? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 11. IF YES, Did the FHE educate your husband/family as well? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 12. If yes, please describe the education that you, or your husband or family, have received? (WRITE A RESPONSE) 13. How easy is it for you to get to the SSF provider? Would you say it is easy, somewhat easy, somewhat difficult, or difficult? (CIRCLE ONE NUMBER) 1 Easy 2 Somewhat easy 3 Somewhat difficult 4 Difficult 14. Please explain how you get to the facility, why is it easy or difficult? (WRITE A RESPONSE) 15. Why did you select the SSF facility instead of a DOH/PWD facility? (WRITE A RESPONSE) 16. Did the SSF provider offer you several different FP methods? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused Family Planning and Reproductive Health—Final Evaluation 96 17. In your opinion, did you receive good quality services from the SSF provider? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 18. Please explain why you thought the quality of services was good or bad? (WRITE A RESPONSE) (FOR RETURN CLIENTS ONLY) 19. Have you noticed any change in the quality of service after your provider started using the voucher system? If yes, please explain the change you noticed. (WRITE A RESPONSE) 20. Have you ever received a voucher from an FHE that you used for FP services? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 21. IF YES, would you still come to the SSF for FP methods if the vouchers are discontinued? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 22. Please explain why or why not? (WRITE A RESPONSE) Family Planning and Reproductive Health—Final Evaluation 97 23. Are you aware of the MCH helpline? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know 4 Refused 24. IF YES, Please explain how you heard about it? (WRITE A RESPONSE) 25. Have you used it? (CIRCLE ONE NUMBER) 1 Yes 2 No 3 Don’t know/refused 26. IF YES, Please tell us what you thought of the response you received from the operator? (WRITE A RESPONSE) --------------------------------------------------------THE END---------------------------------------------- Family Planning and Reproductive Health—Final Evaluation 98 Annex 6: List of Documents Reviewed  Cooperative Agreement No. AID-391-A-13-00007 Family Planning and Reproductive Health (FPRH) Services Project. USAID, 2013.  Cooperative Agreement No. AID-391-A-13-00007 Family Planning and Reproductive Health (FPRH) Services Project Modification No.3. USAID, 2016.  Monitoring and Evaluation Plan Component One: Family Planning and Reproductive Health. Marie Stopes Society, 2014.  The MSI Impact Calculator: methodology and assumptions. Marie Stopes International, 2009.  USAID MCH Program Component 1: Family Planning/Reproductive Health Year Three Work Plan. Marie Stopes Society, 2015.  USAID MCH Program Component 1: Family Planning/Reproductive Health Annual Report. Marie Stopes Society, 2016.  USAID MCH Program Component 1: Family Planning/Reproductive Health Annual Report. Marie Stopes Society, 2014.  USAID MCH Program Component 1: Family Planning/Reproductive Health Annual Report. Marie Stopes Society, 2015.  USAID MCH Program Component 1: Family Planning/Reproductive Health Quarterly Progress Report January-March. Marie Stopes Society, 2016.  USAID MCH Program Component 1: Family Planning/Reproductive Health Quarterly Progress Report January-March. Marie Stopes Society, 2015.  USAID MCH Program Component 1: Family Planning/Reproductive Health Quarterly Progress Report April-June. Marie Stopes Society, 2015.  USAID MCH Program Component 1: Family Planning/Reproductive Health Quarterly Progress Report April-June. Marie Stopes Society, 2014.  Assessing interpersonal care and client satisfaction survey: findings of internal routine monitoring: Research, Monitoring and Evaluation Department FP/RH. Marie Stopes Society.  Client exit interview survey: FPRH Component 2013. Marie Stopes Society, Socio Economic Development Concern (SEDCO) Associates and Development Strategies, 2014.  Client exit interview (CEI) survey report 2014: FPRH Project. Marie Stopes Society. Others:  FP/RH Overview presentations  Helpline strategy, data and reports  Research and training documents provided by MSS RME department  Data Sets: o Client Exit Interview Datasets (2013-2015) o External Voucher Validation (& Data) o Outreach Verification Data o SOP Strategies o Performance Data o Facility Mapping and Selection  Services Data (of the last three months from selected facilities) Family Planning and Reproductive Health—Final Evaluation 99 Annex 7: Conflict of Interest Statements Conflict of interest disclosures have been removed from the report to protect confidentiality. They are available from Management Systems International on request. Family Planning and Reproductive Health—Final Evaluation 100 Annex 8: Client Growth Analysis and Cost per CYP Client Growth Analysis COUNT Regions Total Clients Served Total Voucher Clients Served 2013 2014 2015 2013 2014 2015 Sukkur 46,324 79,534 79,510 36,217 58,948 66,584 Hyderabad 6,152 46,183 80,972 5,487 34,194 58,910 Karachi 6,156 27,388 29,831 5,406 18,369 19,881 Multan 16,288 14,228 13,390 15,080 12,371 12,488 Total 74,920 167,333 203,703 62,190 123,882 157,863 PERCENT VOUCHER CLIENTS Regions Total Clients Served 2013 2014 2015 Sukkur 78.2% 74.1% 83.7% Hyderabad 89.2% 74.0% 72.8% Karachi 87.8% 67.1% 66.6% Multan 92.6% 86.9% 93.3% Total 83.0% 74.0% 77.5% GROWTH RATES Regions Growth Rate of Total Clients Growth Rate of Voucher Clients 2013–14 2014–15 2013–15 2013–14 2014–15 2013–15 Sukkur 71.7% 0.0% 71.6% 62.8% 13.0% 83.8% Hyderabad 650.7% 75.3% 1216.2% 523.2% 72.3% 973.6% Karachi 344.9% 8.9% 384.6% 239.8% 8.2% 267.8% Multan -12.6% -5.9% -17.8% -18.0% 0.9% -17.2% Total 123.3% 21.7% 171.9% 99.2% 27.4% 153.8% Family Planning and Reproductive Health—Final Evaluation 101 Cost per CYP – Key Performance Indicators FAMILY PLANNING CLIENTS SERVED Indicators Year 1 Oct 2013– Sep 2014 Year 2 Oct 2014– Sep 2015 Year 3 Oct 2015– Jun 2016 Total to Date SSF Network 78,478 198,105 217,973 494,556 Outreach 22,377 40,389 49,949 112,715 OMU 2,205 8,205 10,410 HANDS 908 545 1,453 MCH 782 339 1,121 Total 100,855 242,389 277,011 620,255 CYPS GENERATED Indicators Year 1 Oct 2013– Sep 2014 Year 2 Oct 2014– Sep 2015 Year 3 Oct 15– Jun 16 Total to Date SSF Network 168,059 419,001 521,562 1,108,622 Outreach 82,522 168,437 152,511 403,470 OMU 1,632 8,384 10,016 HANDS 1,432 2414 3,846 MCH 1,662 530 2,192 Total 250,581 592,164 685,401 1,528,146 MATERNAL DEATHS AVERTED Indicators Year 1 Oct 2013– Sep 2014 Year 2 Oct 2014– Sep 2015 Year 3 Oct 2015– Jun 2016 Total to Date SSF Network 108 197 167 472 Outreach 60 90 50 200 Total 168 287 217 672 UNINTENDED PREGNANCIES AVERTED Indicators Year 1 Oct 2013– Sep 2014 Year 2 Oct 2014– Sep 2015 Year 3 Oct 2015– Jun 2016 Total to Date SSF Network 75530 164,699 181,321 421,550 Outreach 42192 75,806 58,319 176,317 OMU 518 2442 2,960 HANDS 525 955 1,480 MCH 607 190 797 Total 117,722 242,155 243,227 603,104 Family Planning and Reproductive Health—Final Evaluation 102 COST Particulars Year 1 Year 2 Year 3 Total Oct 2013– Sep 2014 Oct 2014– Sep 2015 Oct 2015– May 2016 $ $ $ 1. Personnel/Salaries and Wages 1,559,256 2,762,705 2,408,307 6,730,269 2. Fringe Benefits 99,581 198,963 185,337 483,880 3. Travel, Transportation and Per Diem 359,010 958,714 792,839 2,110,563 5. Equipment and Supplies 135,804 765,877 591,400 1,493,081 6. Contractual 372,687 2,453,992 1,139,810 3,966,489 7. Training 195,851 366,119 243,558 805,528 8. Other Direct Costs 412,462 1,217,080 1,315,096 2,944,638 9. Security Costs 49,815 125,297 117,560 292,672 Total costs 3,184,467 8,848,746 6,793,906 18,832,356 - Costs include MCH and Safe Motherhood costs - Capital expenditures not included to keep consistency with Cost per CYP calculation. COST PER KEY PERFORMANCE INDICATOR Key Performance Indicator Year 1 Year 2 Year 3 Total Oct 2013– Sep 2014 Oct 2014– Sep 2015 Oct 2015– May 2016 $ $ $ Cost/Client Served 31.6 36.5 24.5 30.4 Cost/CYP generated 12.7 14.9 9.9 12.3 Cost/Maternal Deaths Averted 18,955.2 30,831.9 31,308.3 28,024.3 Cost/Unintended Pregnancies Averted 27.1 36.5 27.9 31.2 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523