SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS MARIE STOPES INTERNATIONAL MIDTERM PROJECT EVALUATION AUGUST 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Sarah Castle, PhD, and Pandora Hardtman, RN, CNM, DNP through the Global Health Technical Assistance Bridge III Project. Cover Photo by Pandora Hardtman, May Day Clinic, Accra Ghana EVALUATION Support for International Family Planning Organizations Marie Stopes International Midterm Project Evaluation August 2013 Global Health Technical Assistance Bridge III Project (GH Tech) USAID Contract No. AID-OAA-C-13-00032 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 13-03-007) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge 3 Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Engility-IRG to the United States Agency for International Development under USAID Contract No. AID-OAA-C-13-00032. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION i ACKNOWLEDGMENTS The authors would like to thank USAID staff in Washington, Antananarivo, and Accra for their guidance. We are also grateful to MSI staff in London as well as in Madagascar and Ghana who were most helpful and collaborative. Our thanks also go to representatives from SIFPO partner organizations, the numerous project beneficiaries, peer educators and field staff who gave us their invaluable opinions about the project. ii SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION iii CONTENTS Acronyms .............................................................................................................................................. v Executive Summary .......................................................................................................................... vii I. Introduction................................................................................................................................... 1 Evaluation Purpose ....................................................................................................................... 1 Evaluation Questions ................................................................................................................... 1 II. Project Background ..................................................................................................................... 3 Country Participation and Country Buy-in ............................................................................ 4 Methodology ................................................................................................................................. 5 Field Visits to Madagascar and Ghana ..................................................................................... 6 Limitations ..................................................................................................................................... 8 III. Findings ........................................................................................................................................... 9 Satisfaction of Stakeholders with SIFPO Assistance ............................................................. 9 Organizational Strengthening and Capacity Building ..........................................................11 Program Strengthening Under SIFPO ....................................................................................12 Strengthening Tools ...................................................................................................................15 Strengthening Research, Monitoring, and Evaluation .........................................................17 Publications ..................................................................................................................................19 Sustainability of Local Health Systems ...................................................................................20 Networks for Sustainability .....................................................................................................22 Gender and Youth Activities in SIFPO Countries to Date ..............................................24 Gender and Youth Documentation .......................................................................................25 Barriers to the Delivery of SRH to Young People .............................................................25 Sociocultural Barriers to SRH Services .................................................................................26 IV. Conclusions and Recommendations ......................................................................................29 Recommendations .....................................................................................................................31 Annex I. Scope of Work .................................................................................................................35 Global Health Technical Assistance Bridge Project ...........................................................35 Appendix A: USAID Criteria to Ensure the Quality of the Evaluation Report ...........47 Annex II. Key Informants ................................................................................................................49 USAID/Washington ...................................................................................................................49 USAID Country Missions .........................................................................................................50 iv SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Other Stakeholders ...................................................................................................................50 Annex iii. References .......................................................................................................................53 PowerPoint Presentations ........................................................................................................53 Reports .........................................................................................................................................53 Annex Iv. Evaluation Methods and Limitations ..........................................................................57 Limitations ...................................................................................................................................57 Annex V. Persons Interviewed ......................................................................................................59 Annex VI. Data Collection Instruments.......................................................................................63 QUESTIONNAIRE FOR MSI COUNTRY OFFICES .........................................................65 TABLES Table 1: MSI-SIFPO Support Summary........................................................................................... 7 Table 2: Interviews and Activities Undertaken during Site Visits in Madagascar and Ghana ..................................................................................................................................................... 7 Table 3: Principal Activities and Outputs Undertaken by MSI’s SIFPO Partners during Period of Funding ..............................................................................................................................10 Table 4: Urban and Rural Differentials in Median Age at First Marriage and Median Age at First Intercourse in Ghana and Madagascar ...........................................................................25 Table 5: Adaptation of Table on Social Norms from Gender and Youth Assessment Tool with Specific Examples from Ghana and Madagascar ......................................................27 Task LOE ............................................................................................................................................42 FIGURES Figure 1: SIFPO Funding to MSI by Intermediate Results up to April 2013........................... 4 Figure 2: Increase in Total Number of Franchisees and LARC Clients since SIFPO Started in Late 2010 .........................................................................................................................13 Figure 3: USAID Bilateral Agreement Monetary Ceilings Emerging from SIFPO Country Buy-ins .................................................................................................................................................22 Figure 4: A Framework for Norms that Impact SRH Supply and Demand ........................26 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION v ACRONYMS AHME African Health Markets for Equity AIDS Acquired immune deficiency syndrome BSE Breast self-exam CO Clinical officer CHE Community health educator CHW Community health worker CLIC Client information center COI Conflict of interest CPR Contraceptive prevalence rate CYP Couple year of protection DfID Department for International Development DHS Demographic health survey FP Family planning FP/RH Family planning/reproductive health FP2020 Family Planning 2020 GBV Gender-based violence GHI Global Health Initiative HIV Human immunodeficiency virus ICRW International Center for Research on Women IPPF International Planned Parenthood Federation IUD Intrauterine device LARC Long-acting and reversible contraceptive LAPM Long-acting and permanent method M-Health Mobile Health MARPs Most at-risk populations M&E Monitoring and evaluation MDG Millennium Development Goal MSG Marie Stopes Ghana MSI Marie Stopes International MSK Marie Stopes Kenya MSM Marie Stopes Madagascar MST Marie Stopes Tanzania MSU Marie Stopes Uganda vi SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION MOH Ministry of health NGO Non-governmental organization OCPs Oral contraceptive pills ODA Organizational development assessment ODT Organizational development tool PMP Project management plan P4P Pay for performance PRH Population and reproductive health PSI Population Services International PSZ Population Services Zimbabwe QTA Quality tool assessment RME Research, monitoring, and evaluation SHOPS Strengthening Health Outcomes through the Private Sector SIFPO Support for International Family Planning Organizations SMS Short Message Service SOW Scope of work SRH Sexual and reproductive health STI Sexually transmitted infection TL Tubal ligation USAID United States Agency for International Development VCT Voluntary counseling and testing WHO World Health Organization SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION vii EXECUTIVE SUMMARY The Support to International Family Planning Organizations (SIFPO) project has a period of performance from September 30, 2010, until September 30, 2015, and funds Marie Stopes International (MSI) and Population Services International (PSI). This evaluation pertains to Marie Stopes International’s performance only. Marie Stopes International collaborates with four implementing partners: the Population Council, the International HIV/AIDS Alliance, the International Center on Research for Women (ICRW), and EngenderHealth. SIFPO’s mission is to increase the use of family planning (FP) services globally through strengthening selected international FP organizations and an extensive, multicountry network of FP clinics, in order to achieve maximum program impact and synergies. The current SIFPO projects are working toward the following four results areas:  Result 1: Strengthened organizational capacity to deliver quality FP services to target groups  Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level  Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance  Result 4: Gender-sensitive FP services targeting youth strengthened at a global level A special focus of the SIFPO projects has been on capacity building and systems strengthening to build upon and leverage the organization’s extensive network of country programs that are often funded by other donors. These capacity-building areas include improving clinical and counseling quality, standardizing and sharing best practices (e.g., in social franchising or mobile outreach programs), improving and standardizing metrics, and improving health management and information systems. The evaluation’s overarching questions seek to ascertain how satisfied various stakeholders, including USAID missions, have been with the work done by and assistance from SIFPO. The questions also seek to identify what existing gaps and future technical directions or issues need to be addressed in any follow-up , as well as what kinds of inputs—specific to organizational strengthening—are no longer needed. The technical evaluation questions focus on the use of core resources by each recipient organization to strengthen country-level platforms, and their effect on improved organizational capacity, sustainability, performance, and management. They also address the additional support and strengthening needed for improved sustainability at the country level. Lastly, they seek to identify the areas for improvement or strengthening family planning/reproductive health (FP/RH) service delivery and quality assurance. The questions comprise, but are not limited to: What evidence exists that core resources invested in organizational strengthening have improved country-level platforms and programming? What has been the effect on organizational capacity? Is there evidence that there is increased sustainability? How have management practices within the organization been affected? Is there evidence that the internal quality assurance standards have been disseminated viii SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION to strengthen FP service delivery and performance at a global level? To what extent is the MSI project’s portfolio of service delivery activities meeting the needs of stakeholders? The midterm evaluation of MSI-SIFPO took place between June and August 2013, and concerns only the Marie Stopes International’s SIFPO project. An evaluation of the PSI SIFPO project also occurred and is reported on elsewhere. The lead evaluators were a demographer/public health monitoring and evaluation (M&E) specialist and a doctoral-level certified nurse–midwife with international experience in training, professional standards, and capacity building. The methodology involved “triangulating” different research methods, including document reviews and in-depth interviews with USAID/Washington and mission staff and the international project partners. All USAID missions and MSI country offices in the eight SIFPO countries were sent an online survey asking about their views on what the funding had achieved. In addition to meetings with key USAID and MSI personnel in Washington, the evaluators were comprehensively briefed by MSI/London. They then carried out fieldwork with two MSI country programs in Madagascar and Ghana. These sites were chosen because they had a complementary range of services. In both settings, in-depth interviews took place with project staff, project beneficiaries, providers, community health workers, and peer educators, as well as with local partners and stakeholders. Clinical checklists were administered and observations of service provision were made in a number of service delivery settings. In Ghana, interviews were carried out with government representatives, because USAID has not worked with the Malagasy government since the political crisis in 2009. The evaluation concluded that MSI-SIFPO has provided an exciting and unique opportunity for USAID to fund an international FP organization. USAID itself benefited in many ways from the collaboration with MSI. In particular, this project catalyzed new reflection within the Agency regarding direct programming and integrating gender issues into service delivery. In general, USAID/Washington appears to be extremely satisfied with MSI’s performance via SIFPO. This satisfaction was expressed during the interviews evaluating four main areas: cost-effectiveness, metrics and M&E, quality of services, and models of service delivery. Involving MSI made for a very lean project. USAID did not need to recruit many new staff but rather used MSI’s existing locally available staff. USAID recognized that MSI is extremely good at metrics and M&E and could provide evidence of impact as well as the cost of service delivery using different approaches. USAID also articulated that they were impressed with the quality of services delivered and with the overall systems of quality assurance. The new modes of service delivery and sensitization—for example, using tuktuks for outreach in Tanzania or mobile money in Madagascar—have substantially improved USAID’s reach. The fact that local MSI staff provide on-the-job training and mentoring, particularly for government counterparts, was seen by the Agency as an effective way of strengthening local capacity and sustainability. Out of the four missions who returned the questionnaire, three countries expressed great satisfaction with MSI-SIFPO-funded activities. Direct exchanges took place with mission staff in Madagascar and Ghana, which were the host countries for the field observation as part of the evaluation. In Madagascar, USAID noted that the two large USAID-funded community health projects only provided short-term methods. USAID worked with them to catalyze referrals to outreach and other services, and MSI made the full range of long-acting reversible contraceptives (LARC) and permanent methods (PM) available. The fact that MSI provided quality services via local infrastructure that was already in place meant that the project was cost￾effective for USAID, which did not have to set up a service delivery infrastructure from scratch. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION ix MSI’s collaboration with SIFPO partners—the Population Council, ICRW, EngenderHealth, and the HIV/AIDS Alliance—not only enabled them to develop innovative new tools for programming but also influenced how these four internationally recognized institutions looked at FP service delivery. This lasting legacy within four influential and established partners is likely to have a global impact in years to come. Due to USAID requirements, the SIFPO award necessitated setting up new systems of accountability and compliance that benefited MSI as a whole, at both an institutional and a country level. The capacity strengthening was evident in all sectors of MSI’s work and quantified via the organizational development tool (ODT). This improvement was evident in areas such as financial management and technical capacity related to human resources. Under SIFPO, the number of social franchisees increased significantly and the quality of their services improved, as measured by the quality technical assessments (QTAs) which comprise a south-to-south intraorganizational partnership for quality assessment. In addition, MSI’s mobile outreach services were expanded, especially for reaching the poorest communities. This significantly averted maternal deaths and led to considerable savings in the health sector. For example, in Madagascar, savings to the health sector via outreach activities totaled $7,600,000. These cost savings were estimated using MSI’s Impact 2 tool, developed with support from SIFPO. The figures generated by country use of this tool serve as useful advocacy tools for governments and other interested parties. Another particularly important contribution made by MSI and disseminated under SIFPO is additional M-health technology. MSI has worked with mobile phone technology to both increase access to and provide support of FP providers. This aspect of capacity strengthening has used SIFPO funds to initiate Short Message Service (SMS) and youth projects in Senegal and Madagascar and to establish mobile voucher reporting and reimbursement in Madagascar. The CLIC software was also developed under SIFPO and is now being rolled out. This software will enable longitudinal following of client services and provide other information about new FP adopters or FP switchers. This and other innovative tools, such as the poverty grading tool, to allow better documentation and dissemination of information. MSI is now better—and more productively—integrated with the international FP community. For example, since receiving SIFPO funding, MSI is now a member of technical working groups related to M-health and high￾impact practices. MSI representation within these technical working groups, as one of the largest global clinical service providers will serve to move the field of FP forward considerably. MSI will provide access to important resources for other partners working toward realization of the agendas of Family Planning 2020 (FP2020) and Millennium Development Goals (MDGs) 3, 4, and 5. In the area of research and M&E, MSI-SIFPO (with the Population Council) has funded important training in operations research and data analysis, among other things. MSI recently offered qualitative research training in Kenya at the request of country offices. Focus on qualitative research has appeared only recently in the MSI portfolio. However, it can help the organization better understand its client base and develop locally and culturally appropriate initiatives, especially for youth. In addition, MSI’s exit interviews provide important data to inform programmatic decision-making and to enable awareness raising activities to be tailored to different client segments However, there is likely to be some selection bias among exit interview respondents, as dissatisfied clients may be less willing to reply to questions. For example, more than 20% of 210 clients in South Sudan refused exit interviews. MSI may not x SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION adequately recognize biases and selectivity in the exit interview data, although this challenge may become less of an issue thanks to CLIC. With regard to quality assurance, MSI prides itself on delivering quality services, even in challenging circumstances. All MSI-SIFPO countries have achieved increases in their QTA scores since receiving SIFPO funding. The QTA ensures minimum standards of care in the service delivery area to aid decision-making, focus resources, identify risks, and ultimately lead to internal accreditation. The evaluation concluded that overall quality of care is good, largely as a result of the regular support and monitoring QTAs provide. A selection of items from the QTA checklist were observed during the evaluation in the field and found to be satisfactory. For example, in Ghana, there are exemplary examples of many QTA line items regarding infection control. MSI is mindful of sustainability issues at a local level and has taken important practical steps to ensure long-term access to FP and sustainability of health systems. Most importantly, MSI’s expansion of service provision has induced a “culture” of demand for FP in some countries. In Malawi and probably in Tanzania, Madagascar, Zimbabwe, and Uganda, MSI has contributed significantly to the national modern contraceptive prevalence rate (CPR). One of the most important ways MSI ensures sustainability is via its approach to supporting, including, and training local ministries of health. For example, in Cambodia, MSI has trained government health workers to carry out voluntary tubal ligations and participates in discussions with UNFPA and USAID about the procurement of FP commodities. MSI has also trained government health workers in Cambodia on intrauterine device (IUD) and implant insertions and removals. In Ghana, ministry of health (MOH) staff assist Marie Stopes Ghana (MSG) personnel with the insertion of implants and IUDs during outreach and have been trained by MSI to manage removals and complications. Ghanaian public sector providers also carry out voluntary tubal ligations and vasectomies. It is theorized that MSI can further contribute to long-term country capacity and sustainability in FP services through integration of services into pre-service nursing, midwifery, and medical education. With multiple, often high-volume, sites available, MSI is ideally placed to embrace clinical mentoring and instruction of students. Those students who will undergo supplemental training through MSI as quality FP providers can then go on to strengthen the public sector and provide a future pool of staff for MSI. Similarly, the task-sharing study in Uganda assessed postoperative complications and patient satisfaction associated with tubal ligations in non-clinical settings with clinical officers (COs) expanding access to this method in areas where doctors are not available. The study found that the procedures were classified as “good” or “very good,” and that most clients would recommend them to a friend. This resulted in MSI engaging with Ugandan policymakers to advocate for the scale-up of tubal ligations (TLs) in non-clinical environments by a CO. For example, in Ethiopia, much of the work around delivering LARC is carried out by lower-level personnel. SIFPO has also been critical in enabling MSI to identify alternative funding mechanisms to enhance local ownership and sustainability. Many BlueStar clinics provide services under national insurance schemes, again ensuring their sustainability and integration with government services. For example, in recent years Ghana has implemented the National Health Insurance Scheme (NHIS), a national social health insurance program that generally covers patient care. Many BlueStar clinics are registered as preferred providers, allowing them to be reimbursed under the social health insurance scheme. The government announced free FP services under NHIS last SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION xi June. MSIG and other partners are working with the government of Ghana to formulate an implementation strategy that will strengthen the franchisees with regard to FP provision. MSG believes that this will support the future sustainability of MSIG’s centers and outreach, since— under the PPP framework—the government will subcontract some of these services to the private sector. In other settings, MSI has used vouchers to make sure that the poorest and most vulnerable can access services. For example, in Madagascar, vouchers directed to the very poor subsidize voluntary LARCs and short acting methods, while users are referred for PMs. In Zimbabwe, vouchers are directed at youth to facilitate young people’s access to FP services. In both settings this serves to increase the reach and inclusivity of national programs. BlueStar franchisees are given sufficient training and resources so that they could operate independently if MSI withdrew. They have access to discounted equipment and have their clinics refurbished and rearranged to maximize their attraction to clients, increasing patient load and the franchisees’ overall financial sustainability. It appears that MS has room for further development with regard to gender and especially youth programming. MSI views “youth” as those under 25, with some country variation. For example, Madagascar defines “youth” as those under 20. However, much of the data for MSI’s evaluation of young people comes from the exit interviews. This data may be biased, because youth tend not to use fixed clinics or specialist services. More importantly, there is a need to recognize that young people require sexual health information and services from an early pre-adolescent age. Early preventive intervention can lead to healthier lives during adolescence and early adulthood. In addition, MSI does not sufficiently recognize the heterogeneity of young people. It operates on a “one size fits all” model, which tends to be heavily medicalized and does not capture young people’s “multiple vulnerabilities,” including exposure to HIV risk or encouraging delayed childbearing. Nevertheless, important advances have been made with regard to targeting youth in Tanzania, Zimbabwe, Madagascar, and Ghana. The Kayayei project in Accra (funded with Ghana Mission FP/RH and HIV resources) is groundbreaking and can be used as a model for other youth￾oriented services MSI may want to deliver in the future. The strength of the program lies in its focus on the vulnerable and poor migrant market workers, providing them with FP and integrated gender-based violence (GBV) services, along with HIV/AIDS prevention and testing. In Madagascar, the youth work included a voucher scheme to attract young people to BlueStar franchisees. However, it appeared to be heavily geared toward literate youth and was hampered by a lack of real consultation with local youth organizations. As an organization, MSI needs to build their youth services based upon formative research rather than exit interviews. Qualitative research should involve young people to help identify youth leaders and appropriate locations for outreach. In this way, the multiple vulnerabilities of the most at-risk youth can be identified and successful interventions designed. MSI’s work in Cambodia with the most at-risk populations (MARPs) goes some way in doing this, but future integration work—of prime importance in youth populations—may require USAID to reallocate STI/HIV funding into FP budgets. In conclusion, SIFPO provided MSI with funds to carry out trailblazing work that has strengthened not only their own systems but also those of their local and international partners. For USAID, it has proved to be a cost-effective, high-impact investment yielding cutting-edge tools, systems, and technologies. SIFPO funding has also increased the reach of high-quality service provision in a sustainable manner through a variety of original and appropriate channels. xii SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 1 I.INTRODUCTION EVALUATION PURPOSE The Support to International Family Planning Organizations (SIFPO) cooperative agreement with Marie Stopes International (MSI) was made in 2010 and, in theory, will continue to 2015, although the ceiling ($40 million) has nearly been attained. The midterm evaluation presented here was commissioned by the United States Agency for International Development (USAID) and aims to examine the impact of SIFPO funding on systems strengthening and capacity building within MSI. It also assesses MSI’s quality assurance standards and the organization’s gender- and youth-sensitive services. It must be emphasized here that this is not an evaluation of MSI’s impact on family planning (FP) service delivery nor of FP uptake resulting from activities associated with the grant. Neither is it an evaluation of country programs in Madagascar and Ghana, where the fieldwork was carried out. It is an examination of MSI’s internal system and structures and the impact SIFPO has had upon them. It also examines MSI’s relations with SIFPO partners and the joint activities they have undertaken. The audience for this report is MSI’s staff at headquarters and in-country offices, as well as USAID staff with an interest and expertise in population and reproductive health. It will also be of interest to MSI-SIFPO’s sub-awardees (Population Council, EngenderHealth, HIV/AIDS Alliance, and International Center for Research on Women [ICRW]). The findings will inform decision-making about future ways USAID can partner with and support MSI and other large international service delivery organizations. EVALUATION QUESTIONS The overarching questions framing the evaluation seek to ascertain how satisfied various stakeholders, including USAID missions, have been with the work done by and assistance from MSI-SIFPO. The technical evaluation questions focus upon the use of core resources for each recipient organization to strengthen country-level platforms, and their effect on improved organizational capacity, sustainability, performance, and management. They also address the additional support and strengthening that are needed for improved sustainability at the country level. Lastly they seek to identify the areas for improvement or strengthening family planning/reproductive health (FP/RH) service delivery and quality assurance. The questions include but are not limited to:  What evidence exists that core resources invested in organizational strengthening have improved country-level platforms and programming?  What has been the effect on organizational capacity?  Is there evidence that there is increased sustainability?  How have management practices within the organization been affected? 2 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION  Is there evidence that the internal quality assurance standards have been disseminated to strengthen FP service delivery and performance at a global level?  To what extent is the MSI project’s portfolio of service delivery activities meeting the needs of stakeholders? SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 3 II.PROJECT BACKGROUND USAID awarded MSI a cooperative agreement to implement the SIFPO project. The period of performance is from September 30, 2010, until September 30, 2015. MSI implements their activities with four international partners including the Population Council, the International HIV/AIDS Alliance, the ICRW, and EngenderHealth. The MSI-SIFPO project has a ceiling of $40 million. To date, $33.76 million has been obligated to the MSI project, of which 73% is field funding and 27% is core funding. A parallel SIFPO award was made by USAID to Population Services International (PSI), but this is not discussed in detail here. The SIFPO mission is to increase the use of FP services globally through strengthening selected international FP organizations and an extensive, multi-country network of FP clinics, in order to achieve maximum program impact and synergies. The current project is working toward the following four results areas:  Result 1: Strengthened organizational capacity to deliver quality FP services to target groups  Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level  Result 3: Increased organizational sustainability of country-level programs, including internal South-to-South support and technical assistance  Result 4: Gender-sensitive FP services targeting youth strengthened at a global level A special focus of the SIFPO projects has been on capacity building and systems strengthening to build upon and leverage MSI’s extensive network of country platforms, programming, and innovative service delivery channels. These capacity-building areas include improving clinical and counseling quality; standardizing and sharing best practices, in areas like social franchising or mobile outreach programs; improving and standardizing metrics, such as for equity; improving health management and information systems; testing new approaches for service delivery and creating efficiencies in existing approaches, such as for mobile outreach or clinic-based approaches; creating a cadre of technical RH leaders within the organization; and increasing evaluation and operations research skills of field staff. The vision is that by strengthening and streamlining the procedures and systems of an international FP organization such as MSI, these improvements will cascade down to local affiliates in developing countries, thereby enhancing sustainability of private sector partners to contribute to the overall health system. The work described here is a midterm evaluation and, as such, does not have a conceptual framework or log frame. However, MSI does have a performance monitoring plan (PMP). The main global indicators within the PMP are the following: 1.Estimated number of maternal deaths that will be averted through the use of FP provided by MSI 2.Estimated number of pregnancies averted through use of FP provided by MSI 4 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 3.Total number couple years of protection (CYPs) provided through FP provided by MSI (disaggregated further by the single category comprised of CYPs generated by all long-acting and permanent methods (LAPMs) 4.Total number of FP visits at MSI service delivery points (percentage under/over 25 years of age, percentage male/female) Additional sub-indicators are linked to each of the four results described above but are not presented in detail here. All SIFPO project activities are organized into two categories: core-funded and field-funded activities. Core- and field-funded activities are predominately comprised of FP activities. Although some funds were initially available for integrated HIV prevention, the limited ceiling that remains on the project was prioritized for FP activities implemented with field support. Figure 1 shows that up until April 2013, over half of all funding dispersed went to strengthening organizational capacity for FP services. One-fifth went to increasing organizational sustainability, 18% to internal quality assurance, and 8% to gender-sensitive FP services aimed at youth. Figure 1: SIFPO Funding to MSI by Intermediate Results up to April 2013 COUNTRY PARTICIPATION AND COUNTRY BUY-IN Over $9.6 million has been allocated from SIFPO core funds for in-country and headquarters￾based activities. Field buy-ins total $24.2 million and have been disbursed via USAID missions in Tanzania, Zimbabwe, Ethiopia, Ghana, South Sudan, Pakistan, Cambodia, and Madagascar. Three buy-in countries have transitioned to bilateral programs with the local MSI affiliate: Pakistan, Tanzania, and Zimbabwe. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 5 In summary, core investments catalyze and build upon field buy-in and success by  Increasing and improving sexual and reproductive health (SRH) technical leadership  Piloting and documenting innovation  Strengthening organizational systems  Building capacity with regard to human resources  Developing cutting-edge SRH tools Field buy-ins leverage core system investments and buy-ins to  Take innovative best practice FP service delivery to scale  Demonstrate national FP impact  Attract long-term funding outside of SIFPO Analysis of the main country-specific SIFPO core-funded MSI activities to date showed that they are strongly aligned with the U.S. Government Global Health Initiative (GHI) principles, the Global Health Strategy for the GH Bureau, and USAID FORWARD. The GHI principles are linked to the Agency’s commitment to Paris Declaration aid effectiveness principles, including alignment with country strategies and priorities, strengthened health systems, new partnerships and innovations, and strengthened monitoring and evaluation (M&E) for accountability and results. The GHI principles also include a focus on women and girls and integrated services, which are a strong focus of MSI’s SIFPO-funded activities. They also conform to additional performance measures of the Population and Reproductive Health (PRH) Office, which are to increase the percent of births spaced more than three years apart, reduce the percent of births order five or higher, and increase the percent of demand satisfied through modern contraception. The MSI-SIFPO project contributes to all of these objectives, as well as to Millennium Development Goal (MDG) 4 to reduce child mortality; MDG 5a, improved maternal health; and MDG 5b, universal access to RH and also to MDGs for poverty, gender, and HIV/AIDS. In addition, MSI-SIFPO-funded activities are also aligned with USAID’s Policy on Youth (2012) which seeks to 1) strengthen youth programming, participation, and partnership in support of Agency development objectives; and 2) mainstream and integrate youth issues and engage young people across Agency initiatives and operations. MSI-SIFPO activities are also aligned to USAID’s policy on Gender Equality and Female Empowerment (2012), which seeks to reduce gender disparities and gender-based violence (GBV) as well as increase women’s capacity to realize their rights. The project also takes into account the FP2020 summit follow-up actions. The goal of the FP2020 summit is to reduce unmet need and enable 120 million women and girls to use contraceptives by 2020. METHODOLOGY The evaluation used both quantitative and qualitative methods to “triangulate” evidence for greater validity. Each evaluator brought a complementary skill set to the evaluation. Sarah Castle has a background in demography, epidemiology, and program evaluation, and focused on the 6 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION project’s tools and impact assessment as well as the perspectives of clients, non-clinical field staff, and volunteers. Pandora Hardtman is an experienced nurse–midwife and focused on clinical service provision, personnel, services, and quality assessments. Document review: First, the team reviewed extensive background documentation pertaining to the project. This included the SIFPO project agreements, the PMP, semiannual and annual reports, country reports and workplans, SIFPO strategies (youth, gender, and integration), country-level and global curricula, training materials, and M&E tools. The review also included USAID technical reports, compliance training materials, and materials relating to the Agency’s youth and gender strategies, as well as to the GHI and other initiatives. Briefings and interviews with USAID and MSI in Washington: The evaluation team met with senior USAID staff in Washington, including the MSI-SIFPO Agreement Officer Representative (AOR) and MSI-SIFPO senior technical advisor. They also interviewed, among others, the office director for PRH and the GH/PRH/SDI division chief. A visit was made to MSI’s Washington, DC, office, where the director of MSI-US and MSI’s U.S.-based demographer/M&E office were interviewed. In addition, a telephone interview was carried out with the senior health and development specialist at ICRW—one of MSI-SIFPO’s major partners. Development of workplan and research instruments: During their time in Washington, the team developed a workplan and submitted it to USAID for approval. They also developed research instruments for use with MSI country officers and local MSI providers. In addition, one SurveyMonkey questionnaire was developed for USAID missions with country-level buy-in to SIFPO, and a second for MSI country directors in SIFPO countries. An additional questionnaire was developed for the three remaining partners (HIV/AIDS Alliance, Population Council and EngenderHealth). Briefings with MSI London: Over two days, Hardtman and Castle received a very extensive briefing from MSI London. This briefing covered topics such as partnerships, social franchising, mobile outreach, costing, task sharing, M&E, M-Health, and sustainability. In-depth discussions took place with, among others, the interim CEO, the vice president and international programs director, the global policy advisor, the associate director of the medical development team, the vice president and health systems director, the M&E advisors, and the head of research. FIELD VISITS TO MADAGASCAR AND GHANA The team spent approximately one week in each country and visited several facilities and outreach sites. Madagascar and Ghana were chosen as field visit sites because they represented complementary aspects of MSI’s SIFPO-funded programming and thus would enable the team to get a complete picture of SIFPO activities. The table below shows that Ghana provides mobile outreach, FP/HIV integration, and activities to counter GBV. Madagascar provides mobile outreach, social franchising, and vouchers for subsidized FP methods. Ghana also provides social franchising, but this is not SIFPO funded. Ghana is a newer program for MSI (established in 2007), while MSI has been in Madagascar for over 20 years. More information about the demographic and fertility contexts, SIFPO activities, and impact of SIFPO activities can be found in the following Table. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 7 Table 1: MSI-SIFPO Support Summary Mobile Clinic Outreach Social Franchising FP Vouchers Youth FP/HIV Integration GBV Cambodia X X X Ethiopia X X X Ghana X X X X Madagascar X X X X Pakistan X X X South Sudan X Tanzania X X X X Zimbabwe X X During the field visits, interviews were conducted with USAID mission staff, both MSI country directors, and senior staff involved with service provision, quality assurance, social franchising, and M&E. In Madagascar, three BlueStar clinics were visited and providers, peer educators, and clients interviewed. An outreach site at Ankisiramena (seven hours from Antananarivo) was also visited and interviews also took place with MSI providers, community health workers, and clients. In Ghana, the evaluation team visited the Kayayei project in Ghana’s central Agobloshie market. A sensitization session about domestic violence by an officer from the Domestic Violence Victim Support Unit was observed. The team interviewed Kayayei and one peer educator, and participated in a film screening about FP methods in the Tema market area. An outreach site based three hours into the Central region was also visited. Interviews took place with MSI providers and Ghana Health Service Personnel. Table 2: Interviews and Activities Undertaken during Site Visits in Madagascar and Ghana MADAGASCAR GHANA Clinical checklists formally administered 5 5 Large group counseling sessions 2 1 Individual pre-counseling 10 10 Individual post-counseling 6 10 Marie Stopes ligation procedures 2 n/a FP method technical service provision 10 10 BlueStar providers 3 n/a BlueStar community health educators 7 n/a BlueStar clients 3 n/a Outreach providers 3 2 Outreach driver 0 1 Outreach community health workers 1 0 8 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION MADAGASCAR GHANA Outreach clients 2 ? Kayayei n/a 2 Kayayei peer educators n/a 3 Ghana Health Service providers n/a 4 Ghana Police (DOVVSU) n/a 2 Local implementing partners 1 2 Regional health directorate n/a 1 Table 2 lists the individuals and organizations interviewed in each country. The distribution of interviewees reflects the emphasis of SIFPO funding for BlueStar and outreach in Madagascar and for outreach and the Kayayei project in Ghana. The Kayeyei project works with the police in combating GBV. In Madagascar, MSI partners with MAHIFA, also funded by USAID for community-based distribution. In Ghana, some of the HIV/AIDS integration activities were carried out by the Society for Women and AIDS in Africa (SWAA) via the Kayayei association. Thus, representatives from both were interviewed. In-depth interviews in both country settings pertained to service quality, service training, method choice, peer-educator training, motivation/remuneration, and client satisfaction (See Annex VI for copies of interview guides). The interviewees gave informed consent and were left a copy of the consent form in case they subsequently had questions. Those who were photographed also signed a form authorizing use of the photographs. In all clinical settings visited, Pandora Hardtman (RN, CNM, DNP) carried out clinical observations and administered a clinical checklist to address the quality of service provision and counseling (Annex VI). LIMITATIONS The first major limitation of this evaluation was the short time allocated for the country visits considering the enormous amount of data that needed to be collected and analyzed. Secondly, MSI country offices chose the field visit sites, which may have introduced some bias to the evaluation. All providers and peer educators at the sites visited were interviewed either formally or informally. Clients were mainly chosen by project staff depending on their availability, another possible source of bias. It should be noted that there may have also been a self-selection of clients willing to be interviewed being skewed toward those who are happy with the services and are more willing to talk. It should be noted that in Madagascar, since the 2009 coup d’état, USAID does not work with the Malagasy government and thus USAID/SIFPO funding cannot fund MSM’s activities in this sector nor engage with local health authority personnel. The subsequent unavailability of government personnel for interview is likely to give a somewhat incomplete picture of health￾care provision. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 9 III.FINDINGS SATISFACTION OF STAKEHOLDERS WITH SIFPO ASSISTANCE USAID/Washington In general, USAID/Washington appears to be extremely satisfied with MSI’s performance via SIFPO. Their satisfaction was expressed during the evaluation interviews around a number of main areas including cost-effectiveness, metrics and M&E, quality of service, and service delivery models. First, involving MSI made for a very “lean” project, as USAID did not need to recruit many new staff but rather used MSI’s existing local staff. This made USAID’S investment extremely cost-effective. Second, USAID recognized that MSI is extremely good at metrics and M&E, and could provide robust evidence of impact as well as of the cost of service delivery using different approaches. It was felt that these metrics and their application improved through the period of SIFPO funding. Third, USAID articulated that they were impressed with the quality of services delivered and with the overall systems of quality assurance. Fourth, the innovative modes of service delivery and sensitization—for example, using tuktuks in Tanzania and mobile money in Madagascar—substantially improved USAID’s reach. Finally, the fact that local MSI staff provide on-the-job-training and mentoring, particularly for government counterparts, was seen by USAID as an effective way of strengthening local capacity and sustainability. SIFPO allowed USAID to provide financial resources to MSI, which delivered local FP services directly through their well-established country programs. Interviewees at USAID noted that this move from providing technical assistance for capacity building to the public sector to actually providing funds for implementation was particularly important for improving uptake in low CPR settings. A few minor issues were also raised by USAID/Washington. The sustainability of MSI’s mobile outreach approach was questioned, but many informants said the issue could be addressed by increasing the training and mentoring of government employees on the job. USAID Missions Of the eight USAID missions that received the questionnaire as part of this evaluation, only four completed it—despite numerous reminders. One of these included South Sudan, which explained, “Activities under this project never started. MSI signed an MOU with the Host Country State Ministry where activities were expected to be carried out but the MOU was later cancelled by the Government and the efforts by MSI to reinstate it did not yield any fruits.” However, the three other countries expressed great satisfaction with SIFPO-funded activities. Direct exchanges took place with mission staff in Madagascar and Ghana, which were visited as part of the evaluation. In Madagascar, USAID noted how the two large USAID-funded community health projects only provided short-term FP methods. By working with them to catalyze referrals to outreach and other services, MSM had made the full range of LAPMs available. The fact that MSM provided” world-class clinical services and had the local infrastructure in place” meant that it was cost-effective for USAID, which did not have to set up a service delivery infrastructure from scratch. MSM was said to be the “’ideal partner’ with regard to responsiveness, technical competence, roots on the ground and quality documentation.” It was noted that despite restrictions on working with the Malagasy MOH, “Marie Stopes has adopted an innovative mixed methodology to use other donor funds to work 10 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION with health centers which compliment USAID investments.” Similarly, in Ghana, the USAID mission was extremely satisfied with MSI-SIFPO, noting that “the level of empowerment exhibited by the beneficiaries is evident of the investments made.” Project Partners All outputs relating to MSI-SIFPO project partners are listed in Annex VI. Table 3: Principal Activities and Outputs Undertaken by MSI’s SIFPO Partners during Period of Funding Year 1 Year 2 Year 3 EngenderHealth Peer review of protocols and trainings Technical review of vasectomy and tubal ligation training packages Methods of tubal ligation comparative exchange with field exchange in Malawi and Kenya Technical review of tubal ligation consent form HIV/AIDS Alliance Self-assessment tool for HIV integration Self-assessment tool for HIV integration Clinical quality assurance tools revised for HIV/SRH integration Development of a training package on HIV/SRH integration for MSI clinicians from Africa and Asia Development of a best practice paper on FP/HIV integration ICRW Gender and youth assessment strategy tool Gender and youth assessment strategy tool Gender and youth assessment strategy tool Population Council Operations research Pay for performance workshop Service delivery model workshop Operations research Evaluation of MSI mobile delivery service Willingness to pay study Operations research Evaluation of MSI mobile service delivery Literature review of performance￾based pay in community-based FP programs There has been ongoing attendance at technical working groups and various meetings across all partner relationships during the three years of SIFPO funding. EngenderHealth EngenderHealth said that the collaboration with MSI had been “enormously fruitful.” It left a legacy in terms of repositioning their organization with regard to FP programming. Their respondent said that it had strengthened their awareness of the differences between FP provisions and programming between the public and private sectors and the complementary roles that each can play. It had also reinforced the importance of a strong public sector to ensuring an enabling environment for private sector partners, and provided the opportunity to explore and manage perceived differences related to EngenderHealth’s and MSI’s technical approaches at the field level. HIV/AIDS Alliance The self-assessment tool was piloted in Zambia and Tanzania and then rolled out via a webinar to MSI country programs globally. The HIV/AIDS Alliance also coordinated two country SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 11 partnerships in Cambodia and Kenya between Alliance partners and MSI country programs where staff have carried out peer-to-peer assessments of each other’s programs. The Alliance noted that it had “been involved in integration work for over 10 years, but since the global MOU and the SIFPO project there has been increased commitment to integration in our strategy and in our support to partner capacity in this technical area. SIFPO has helped us to develop our thinking around creating better links between our community networks and services, which are being realized in new programs and funding bids.” ICRW ICRW’s role was to help MSI look at issues of gender and youth and to develop a gender strategy. This programmatic tool took into account interactions between gender and age, allowing MSI to see how gender may shape both services offered and client base served. The tool—currently in draft form—was first based on assessments in Ethiopia and Zambia and then tested in Bangladesh. ICRW also reviewed MSI’s GBV policy. For both tasks, they encouraged a life-cycle approach to FP, for example, engaging with women using antenatal and maternity services. ICRW felt that MSI did not integrate gender approaches into their programming and that at the start of SIFPO, institutionally speaking, gender was not a priority issue for the organization. Their informant said that ICRW’s input, which integrated gender issues into FP provision, had made MSI’s global strategies more effective. For example, in restrictive policy environments such as Bangladesh, MSI was able to help ICRW think about ways to deliver services to unmarried women. ICRW spoke of MSI as “an open and engaging partner on a very steep learning curve.” Population Council During the first year of the project, the Population Council held a workshop for selected MSI staff to examine MSI’s service delivery models that use community health workers (CHWs) and the extent to which pay-for-performance (P4P) 1approaches affect MSI’s ability to program effectively to reach the poorest clients. The Population Council representative who responded to the evaluation questionnaire said, “Overall, the relationship worked well”. It has encouraged us to consider more thoroughly the benefits—and challenges—of mobile outreach for meeting the need for long-acting methods, and of P4P as a mechanism for motivating CHW programs.” ORGANIZATIONAL STRENGTHENING AND CAPACITY BUILDING Result 1: Strengthened organizational capacity to deliver quality FP services to target groups. Management Systems, Reporting, and Compliance Before receiving SIFPO funding, MSI had very little experience of working with USAID. An initial aspect of organizational strengthening was therefore to ensure compliance with USAID’s 1 Pay-for-Performance (P4P) is an approach to match payment or incentives to achievement of health results or outputs. In the case of family planning, P4P is done in the context of ensuring voluntarism and informed choice. P4P approaches might include providing non-monetary incentives to CHWs for providing FP information to rural women, selling low cost vouchers that could be redeemed for a FP method at private provider clinics, or providing non-monetary incentives to health clinics for provision of good quality counseling to clients. 12 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION administrative and program procedures. Thus, manuals were produced on USAID compliance and staff at both headquarters and the country office level, and staff were trained in its implementation. In fact, this exercise required an internal review of both reporting and programmatic practice. In addition, other administrative procedures were put in place by MSI to comply with SIFPO reporting to USAID. These included timesheets for SIFPO-funded staff. All of these requisite initiatives, perhaps initially daunting, in fact enabled the internal strengthening of MSI as an institution. This is recognized at a country level. For example, the MSI Cambodia office noted that their organizational capacity and sustainability had been improved: “MSI Cambodia team members have benefited from SIFPO supported workshops in M&E, finance and USAID compliance training.” In Ethiopia, the MSI country office noted that “in order to track the progress achieved under SIFPO, MSIE had to develop stronger systems for data collection and processing. This had a positive influence on MSIE’s capabilities in this area from which all our activities benefit.” One challenge articulated by USAID/Washington, a number of missions, and MSI itself was that, at first, MSI had trouble communicating in the very specific language of USAID. However, MSI considered USAID a very engaged donor, coaching them as to optimal reporting formats and presentation. Organizational Development Assessment Each country program has an organizational development assessment (ODA) once every two or three years, based on the standard MSI organizational development tool (ODT). The purpose of the ODT is to improve organizational capacity and accountability by assessing country programs against MSI’s shared global standards. Capacity is assessed in seven main categories, including strategic management and leadership, financial management, and technical performance. Marie Stopes Madagascar (MSM) was last assessed in December 2012 and received an overall classification assessing them as “a partner who shows a robust capacity in this domain with strong systems in place to maintain quality and to facilitate growth.” MSM has the potential to manage several programs in the context of a wider portfolio. This was a marked improvement on the pre-SIFPO score in 2009. PROGRAM STRENGTHENING UNDER SIFPO Strengthening the Private Sector through Social Franchising MSI started BlueStar social franchising in Zimbabwe and Pakistan in 2012 and 2009, respectively. USAID mission field support through SIFPO for social franchising started in 2012 in both countries. SIFPO funding currently supports over three hundred franchisees worldwide. The franchisees are normally already practicing in the private sector and are trained and supported by MSI to provide quality services. Figure 2 indicates that the global growth of franchisees since 2010 corresponds with an increase in SIFPO funding and a sharp upward trend in the uptake of long-acting and reversible contraceptives (LARCs) along with ongoing provision of short term methods’ – the majority of services provided by franchisees in most SIFPO-supported countries remains short term methods. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 13 Figure 2: Increase in Total Number of Franchisees and LARC Clients since SIFPO Started in Late 2010 SIFPO has provided added value to social franchising and strengthened MSI’s organizational capacity in the sector by funding a business training roll-out; adding new service capacity, such as IUD training; and strengthening youth-friendly services, such as the youth voucher system in Madagascar. SIFPO has also supported the documentation and dissemination of learning from this sector by providing core funding for the annual social franchising managers’ workshop and for the production of guidance for USAID voucher-funded programs. 14 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Strengthening Mobile Outreach for Family Planning There is a mobile outreach component in every country buy-in, and over 350,000 women have been supported by outreach activities conducted under MSI-SIFPO to date. Based on exit interview samples, on average, about 75% of these live on less than $2.50 per day, and 42% were new adopters of FP. Forty percent of MSI’s CYPs are delivered through mobile outreach. MSI articulates that SIFPO’s support to mobile outreach has helped the organization demonstrate effectiveness and the scalability of the model at a country level. For example, buy-ins via missions in Tanzania and Madagascar means that, by accessing remote rural areas, MSI mobile outreach was able to increase access, which led to higher demand. The result is that one in five FP users in these countries received their contraceptive method from MSI in 2012, with an estimated $31,000,000 saved in the health sector in both countries ($11 million in Madagascar and $20 million in Tanzania). This illustrates how MSI’s improved technical capacity can directly improve health outcomes and strengthen the local health sector to facilitate long-term sustainability. Mobile outreach models are adapted for the specific setting in which they are used and the skill set available. For example, in Ethiopia, an emphasis on task shifting has meant that clinical officers can safely and effectively perform TLs. In Tanzania, cervical cancer screening has been integrated into outreach provided by tuktuks, and MSI staff members in Ghana mentor government providers with regard to LAPMs. Observations of this during the evaluators’ field visits revealed GHS’s satisfaction with the collaboration. As one health worker put it, “If MSIG stops coming, the pregnancy rate will go up!” MSG has been providing outreach in her district since 2011 when the CPR was 12.9%. In 2012, it was 33.9% (Source: Gomoa East District, Heath Directorate Report). This nearly threefold increase was attributed to MSIG outreach by the Regional Director of Health. Strengthening M-Health Recognizing that over six billion people have access to mobile phones, MSI has used mobile phone technology to both increase access to FP information and services and support FP providers. This aspect of capacity strengthening has used SIFPO funds to do the following:  Carry out a global mapping exercise of where MSI uses M-health technology  Provide salary funding for MSI technical expert within Innovation and Best Practice team to lead and coordinate MSI’s M-health work  Share M-health best practices and lessons externally via conference papers and abstracts  Carry out a clinical training pilot in Nigeria and India (underway)  Initiate SMS and youth projects in Senegal and Madagascar  Establish mobile voucher reporting and reimbursement in Madagascar The mobile money service in Madagascar has been particularly innovative and carefully documented. Providers report their indicators and voucher use by SMS. Providers are then reimbursed for the vouchers via money transfers. Capacity Building of Clinical Services Observations of clinical practice exhibited an overall strengthened organizational capacity to deliver quality FP service to target groups. There is evidence of an organizational culture of SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 15 effective multidisciplinary models of training, education, and implementation of a fully functional service delivery package. Client-focused care is provided by state-licensed personnel, including physicians, midwives, and nurses. There are also non-licensed support personnel such as the CHWs or peer educators and drivers who have had FP tasks shifted toward basic provision of care by a lower cadre of workers. According to one of the SIFPO missions that has bought into MSI programming, “MSI FP programs are sustainable with or without SIFPO because of its prior structure and the long-term vision of MSI for organizational building.” All new service providers undergo a similar induction plan based upon individualized needs identified by service delivery channels. According to the medical development team in Ghana, in order to secure an interview with MSIG, all new employees are expected to pass with a score of better than 50% in a pre-employment examination prior to receiving an interview with MSI. If a potential employee fails the written examination, he or she may take an oral test to obtain a passing score. During the field evaluation, team members had the opportunity to observe clinical training of new MSI service delivery providers under the direction of the physician in charge of training and development for the region. While the providers under observation were not slated to enter a SIFPO-funded service delivery channel, the observation provided a direct immediate assessment of core baseline training applicable to all service delivery channels. The providers demonstrated sound theoretical baseline knowledge related to FP physiology, mechanisms of action, management of side effects, and infection control. The providers also demonstrated procedural competency for LARCs. Client counseling skills were the area found to be most in need of further support. STRENGTHENING TOOLS Quality Assessment Tools According to the seven MSI-SIFPO country programs who have received support, the assessment of the quality of the training and tools used to implement organization-wide system changes has improved. The overall support has translated into improved in-country capacity. Under SIFPO, there have been more frequent trainings using tools that increase the engagement of all MSI teams, including those indirectly affected by SIFPO funding. Our survey results showed that five out of seven MSI-SIFPO country programs think that the quality, accessibility, and dissemination of training tools have improved during the term of SIFPO support. Two out of seven countries remain undecided about improvements in the frequency of training but stated that trainings have become more targeted and focused. All countries surveyed report improvement in quality assurance. A country representative of a MSI-SIFPO supported country expressed the following: SIFPO support has been instrumental in building organizational capacity. The QTA tool checklist has become more comprehensive by addition of USG indicators. The improved and updated Standard Operating Procedures (SOPs) for Social Franchise and outreach have improved service quality, including follow-up of all clients. Training for implants has increased the range of methods FP/RH can offer to its clients; therefore, increasing clients’ range of method choice. MSI-SIFPO countries were also asked to rate the quality of specific areas of training/tools that have evolved under SIFPO support. In general, the majority of the materials were rated as high 16 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION quality (see figure 3). Training programs proposed by the MSI team have been implemented in a manner consistent with the objectives of the proposed and approved workplans. The development of a soon-to-be-implemented postpartum intrauterine contraceptive device (IUCD) training manual provides evidence of the MSI-SIFPO team’s commitment to responding to the needs of its clinical service providers. It is noteworthy that the need for training on postpartum insertion of IUCD was frequently voiced by providers while on field observations, as a self-identified priority need. The draft curriculum for a three-day CHW training course is also in process. There is also ongoing adaptation of job aids for use by community educators to specifically address those with a lower literacy level. All tools and informational resources are disseminated to all MSI staff, including those in the field, via the best practice gateway, an online internal organizational library for technical resources that discuss, inform, or evaluate FP/RH programmatic models, approaches, or tools designed to aid ongoing education. The gateway includes MSI-produced documents as well as key external documents, for example, journal articles and other organizations’ publications. The gateway serves as a critical knowledge repository for MSI’s staff members. The MSIG country director stated, “The gateway has been instrumental in assisting the program to produce client chart forms that are easy to adapt for their programs specific needs.” The MSIG client outreach/in-reach intake medical record form is a solid example of the utilization of the gateway in ensuring best practice. The presence of the knowledge gateway assists with the SIFPO end results of improved knowledge, adoption of best practices, and dissemination of resources and innovations by MSI country programs and by external partners and implementers. Gender and Youth Self-Assessment Tool (Draft) The tool developed with ICRW enables MSI programs to answer three broad questions that they may not have been able to adequately address previously: 1.What gender and age factors affect programming and how? 2.What are we currently doing to address these key gender and age factors? 3.What more could we be doing to ensure that gender and age factors do not limit current or potential clients from using our services? The tool is usefully oriented around the pillars of The Power of Ten (MSI’s strategic plan): access, equity, quality, and efficiency. The tool will be extremely useful in better targeting services to the most vulnerable. A discussion with USAID/Washington’s gender specialist revealed that, in her view, “Thinking about gender at an institutional level and systemizing and documenting it was new to MSI. However, thinking about gender at a service delivery level was new to USAID, so there was a kind of synergy.” She also saw that gender approaches would lead to greater sustainability, especially if a gender approach could be institutionalized by local partners and affiliates. To some extent this is happening with the Kayayei project in Ghana that reaches vulnerable young market traders with SRH services and also works with the police to provide recourse for reporting GBV. Measurement of its impact can be effectively linked as a package with the high-impact CYPs described below that identify poor and young clients. It is important to quantify exactly what a SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 17 gender approach adds to impact in order to encourage country buy-in and engage local stakeholders, including ministries. Self-assessment Tool for the Integration of HIV and SRH Services Under the SIFPO award, the HIV/AIDS Alliance developed an integration tool for MSI. It was piloted in Zambia and Tanzania and then rolled out via webinar to MSI country programs globally. The tool provides a comprehensive set of checklists to measure the stage and quality of different country settings. Using observation, provider interviews, exit interviews, and grids to record statistics, the tool provides a comprehensive record of integration and recommends a minimum package of activities that countries aim to put in place. Challenges to implementing integration may include a lack of commodities—for example, in Ghana, the MOH made an error with regard to condom procurement, resulting in condom stock-outs. In Cambodia, MSI has taken a lead on integration and responded during our survey that “MSIC’s leadership on national assessment of FP/HIV integration has led to work with key stakeholders to improve national policy.” Their work with MARPs provides an example of how mission support and engagement of local technical partners and government can combine to provide effective programming among hard-to-reach groups as well as more mainstream clients. Poverty Grading Tool A poverty grading tool was developed using financing by DfID and USAID. It serves to identify the poorest clients and—in some countries, such as Madagascar—offer them subsidized methods of FP. The tool allows for a better understating of clients’ profiles and orients programs to the most vulnerable. SIFPO contributed to team members’ time during the conceptualization of the tool and funded country programs’ uptake and use. STRENGTHENING RESEARCH, MONITORING, AND EVALUATION Research Trainings One way of developing and maintaining capacity is through international trainings and workshops for country-level staff. To date, those funded by SIFPO include: Operations Research (2011), Data Analysis Using SPSS (2011), M&E and ToT (2011 and 2012), and Qualitative Research (2012). Some of the subsequent training includes south-to-south initiatives. For example, a researcher from India trained staff in Yemen, while staff in Ghana, Uganda, and Tanzania trained their colleagues in-country. This enhances the local ownership of results and the country capacity of local MSI researchers. Strengthening Research Activities As described above, the Population Council was MSI’s main partner for research-oriented outputs. The collaboration resulted in a number of important operations research activities, which formed the basis of subsequent program design. For example, the Population Council examined MSI’s service delivery models for CHWs and the extent to which P4P approaches affect MSI’s ability to program to ensure maximum reach. The task-sharing study in Uganda assessed postoperative complications and patient satisfaction associated with tubal ligations in non-clinical settings with clinical officers (COs). The study found that the procedures were classified as “good” or “very good” and that most clients would 18 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION recommend them to a friend. This resulted in MSI engaging with Ugandan policymakers to advocate for rolling out tubal ligations (TLs) in non-clinical environments by COs. These studies provide a good example of how, with SIFPO funding, MSI can develop and test innovative research ideas and then translate them into practice on a global level. However, all findings should be peer reviewed and replicated before going to scale. It is notable that the Ugandan task-sharing study has been accepted for publication in the International Journal for Gynecology and Obstetrics and the Kenya vocal-local study has been submitted to BMC Women’s Health. Strengthening Research and M&E Tools A number of research and M&E tools have been developed during the period of the SIFPO grant. These include but are not limited to the following: Impact 2 Impact 2 was launched in 2012 with core support from SIFPO. The newly updated version released in July 2013 includes new features and updated data sources. It has also been redesigned, based on user feedback, to make it easier to obtain and interpret results. Impact 2 estimates a program’s past, current, and future contributions to national FP use and contraceptive prevalence. In addition, Impact 2 can be used to estimate the wider health, demographic, and economic impacts of these services, including savings to national health-care systems due to births and unsafe abortions averted. This tool has moved the field forward on both programmatic and policy levels. For example, the Zambian MOH used the tool to plan their eight-year scale-up strategy, and the Zambian first lady used data produced by the tool in a speech she made at the Women Deliver Conference in May 2013. The International Planned Parenthood Foundation (IPPF) is rolling the tool out to all countries, and the DfID is using it widely for business case development and reporting. MSI is also working in Tanzania with the government and other partners to cross-check Impact 2 estimates against demographic health survey (DHS) data. This helped the Government of Tanzania understand which sources of FP were driving CPR increases. MSI is also in discussion about using the tool for FP2020 monitoring and measurement. It should be noted that MSI/Washington’s metrics expert is a member of the FP2020 metrics working group. The Exit Interview Tool The exit interview tool was developed, tested, and then implemented in 2010. Its scaling up was carried out with MSI-SIFPO funding. It comprises a 15-minute individual survey that enables MSI to better know who their clients are and to see if they are reaching their target populations. It enables MSI to report progress back to donors and to build the capacity of local researchers. It is currently used in 28 MSI countries (including all MSI-SIFPO countries except Zimbabwe). It therefore provides an example of a research tool that was rolled out with SIFPO funding to now benefit a larger number of MSI’s country programs. MSI in Zimbabwe found that the tool gave them “a better understanding of clients with regard to national norms.” MSI in Madagascar used the tool and observed: “Through the use of exit interviews we know who our clients are, which has informed our marketing and market segmentation exercise. It has also enabled us to better communicate our program to other external stakeholders. For example, MSM’s 2012 national impact report was very well received by Government and donors alike.” Although extremely useful for a range of activities, such as client profiling in relation to young or poor clients, the exit interview instrument is largely quantitative and subject to selection bias. This shortcoming needs to be more widely acknowledged. Research elsewhere indicates that SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 19 exit interviews may be biased toward those who are satisfied with the services, because those who are unhappy with the way they were treated may be unwilling to report their dissatisfaction (Onitsha, Gupta, and Peters 2010). MSI country programs work hard to reduce the number of those unwilling to report dissatisfaction through educational materials about the exit interview and informed consent of clients (which have been approved by MSI’s Ethics Review Committee). MSI’s Research, Monitoring, and Evaluation (RME) team acknowledges these limitations when discussing findings with colleagues. It should be noted that all these limitations are noted in the standard exit interview protocol and are a part of the standard report template. Nevertheless, in South Sudan for example, the clinic refusal rate for exit interviews was 22% and the outreach refusal rate was 15%. This represents a significant proportion of the clientele who may have different characteristics and points of view than those who agreed to be interviewed. Client Information Center The client information center (CLIC) is an informational tool that serves to strengthen MSI’s capacity for robust, evidence-based decision-making. This on-site software package helps MSI understand who its clients are and what services they are using over time. This will enable longitudinal follow-up, eliminate double registration, and facilitate the tracking of clients who switch providers or methods. After being piloted in Uganda, South Africa, and India, it is now also being used in Tanzania, Zambia, and Malawi. Cost Calculator Cost calculator is an Excel-based tool that allows programs to differentiate service costs depending on the type of service. For example, it can cost out provider times, commodity amounts and delivery modes, office support, and overhead. It was piloted in Madagascar and then rolled out in Ghana and Nigeria. High-impact CYPs MSI tracks performance using CYPs, in other words, the protection provided by a specified method for a couple over the course of a year. This is a standard metric rooted in formal demographic analysis that was recently updated with USAID funding. MSI has, however, recently taken the field forward by focusing on ”high-impact CYPs,” which focuses on and tracks those who are most vulnerable and have the greatest FP need. These include new adopters, the poor (living on < $1.25 a day), and the young (<19 years old). It is an extremely important advance in the field of reproductive health research and programming, and fits well within the framework of FP2020. PUBLICATIONS An extensive list of publications produced during the SIFPO grant period can be seen in Annex VI. Result 2: Quality assurance—Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level MSI utilizes several mechanisms of quality assurance. However, the QTA checklist is the tool that is universally referenced in terms of assessments of clinical service quality. Six out of eight MSI-SIFPO buy-in countries believe that internal quality assurance mechanisms have been disseminated to strengthen FP service delivery and performance in their country. Every country surveyed discussed how the elements of the QTA process contributed to improvements in quality assurance as measured by improvements in their country program scores. All “buy-in” 20 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION countries, with the exception of Pakistan, have exhibited increases in the overall QTA scores. (Pakistan’s result may have dipped due to rapid scale-up with USAID funding, but remains high.) Quality assurance is addressed using the QTA system to assure minimum standards of care, aid decision-making, focus and standardize resources, identify risks, and ultimately lead to internal accreditation. This system encompasses a lengthy multi-area item checklist that is administered via random sampling by a clinical member of the MSI team who may or may not be locally based. In many cases, QTAs are carried out by other MSI country office staff—for example, staff from Mali did the QTA for Madagascar. This demonstrates a good model of south-to-south support. If red flags are raised, an individually designed system of supportive follow-up is developed and implemented. The tool is modified annually to reflect system changes and sent out to programs in advance of visits. The sites are given ample opportunity to self-assess and remedy any perceived problem areas prior to the formal assessment. Future plans for the organization include a formalization of the self-assessment process for the providers, the need for which has become more evident under SIFPO. The current QTA reference manual reflects an update from March 2012 under MSI-SIFPO’s core support system. The manual introduces the QTA, explains how to conduct a QTA and the follow-up, and provides a guide to the QTA checklist. Result 3: Increasing organizational sustainability at a country level Sustaining Demand MSI is mindful of sustainability issues and seeks to address these in a number of practical ways, such as providing south-to-south technical assistance; increasingly influencing FP policy discussion; looking at the role of the informal private sector; and focusing on leveraging SIFPO through continued diversification of funding, including looking at bilateral agreements. Importantly, MSI appears to successfully create and sustain a demand for their FP services. In many settings they have created a “tipping point” with regard to behavioral norms. They are supporting a “culture” of FP, enabling women to identify and articulate links between lower fertility and better maternal health and child survival. A marketing strategy in Madagascar provides visual evidence of this culture of FP. In a made for television and radio commercial, MSM utilizes a popular singer and actors to show the lifestyle impact of smaller and larger families in comparison/contrast format. National-level changes relating to the sustainability of demand are also evident in the SIFPO￾supported countries of Madagascar, Zimbabwe, Tanzania, and Uganda. In Tanzania, for example, MSI provides approximately 20% of the modern contraceptives used, most of which are LARC and PM methods. In Malawi, MSI’s local affiliate is the source of more than 43% of voluntary female sterilizations. SUSTAINABILITY OF LOCAL HEALTH SYSTEMS Supporting Ministries of Health In all SIFPO countries, MSI collaborates closely with the local MOH at national, regional, and local levels (although in Madagascar MSI are precluded from working with the MoH with USAID funding). For example, in Cambodia, MSI has trained government health workers to provide IUDs and implants and participates in discussions about the procurement of FP commodities with UNFPA and USAID. In Ghana, MOH staff assist MSIG personnel with the insertion of implants and IUDs during outreach and have been trained by MSIG staff in removals and SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 21 complications. This training was very much appreciated by both local health-care workers and the MSIG staff with whom they collaborate. The public sector staff could identify gaps in their training, which they sought to counter with MSIG training. One MSG outreach nurse said, “We have a very good collaboration with Ghana Health Services. We train two of them, then during our next visit we work with them. After that, we set up a bed where they can do their own insertions but we watch them carefully.” An important aspect of sustainability is task sharing and, within the framework of recent WHO recommendations, MSI has been at the forefront of many initiatives to enable lower-level staff to safely and effectively provide services normally reserved for physicians. This has involved new pilots—such as COs performing tubal ligations in Uganda— global technical follow-up, and policy momentum. In Ghana, many BlueStar clinics are registered as preferred providers, allowing them to be reimbursed under the national social health insurance scheme. MSIG and other partners are working with government to formulate a strategy for providing free FP services. MSIG believes that this will support the future sustainability of its centers and outreach as, under the P4P framework, the government will subcontract some of these services to the private sector. Supporting the Private Sector Through its BlueStar network, MSI strengthens the private sector to enhance sustainability. The BlueStar program is, by nature, scalable and sustainable, as it does not need to hire new health workers or build new clinics. It works with existing providers and infrastructure and ensures delivery of quality services. Taking a health systems approach to strengthening capacity building, MSI allows BlueStar providers to access commodities, provides them with business training, and monitors quality. Thus, in theory, if MSI had to pull out of a country or scale down a country program for any reason, the private sector would still be able to function and provide quality services. This potential for sustainability would be strengthened if there were greater links between the franchisees and local professional associations. In collaboration with African Health Markets for Equity (AHME), MSI worked on issues associated with accurate pricing and cost recovery. The partnership, financed by The Gates Foundation and DfID, includes both MSI and PSI and operates in Ghana, Nigeria, and Kenya. The approaches developed under the AHME to enhance sustainability, however, have been extended to other countries using SIFPO funds. The initiative seeks to not only enhance social franchising in the private sector but also to address ICT development, policy, demand-side financing, access to capital, and quality improvement—all of which enhance sustainability. It should be noted that, prior to the design and launch of the AHME project, ABT did business training with MSI through the Strengthening Health Outcomes through the Private Sector (SHOPS) project. The design of the AHME project drew strongly on USAID’s work with health loans and credit and business skills training, which were core elements of the Agency’s Banking on Health project set up nearly ten years ago. Cost sharing for Sustainability Figure 3 shows that SIFPO has acted as a catalyst for securing bilateral funding in three countries. In Pakistan, this award exceeds $70 million). In Zimbabwe, Through SIFPO, PSZ (MSI’s affiliate in Zimbabwe) is expanding into a total of 56 districts that build on the 23 districts served under SHOPS. This indicates that, in many settings, SIFPO acts as leverage for consolidating MSI’s relationship with USAID to secure longer-term commitments and partnerships. 22 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Figure 3: USAID Bilateral Agreement Monetary Ceilings Emerging from SIFPO Country Buy-ins SIFPO has also helped pilot innovative FP ideas that can then be brought to scale by other donor funding. For example, in Tanzania, the delivery of outreach services by tuktuks and a single nurse was funded by SIFPO with USAID core funds and then scaled up using DfID funding. NETWORKS FOR SUSTAINABILITY Some of the benefits of SIFPO are that it has catalyzed MSI’s involvement in international best practice working groups, forged new partnerships, and heightened the organization’s visibility. Referring back to the Power of 10 (MSI’s strategic plan), networks and partnerships are the basis for the organization’s growth and expansion. MSI is also a key participant in Social Franchising for Health which documents best practices in social franchising and disseminates case studies, including one on BlueStar in Bangladesh and Ghana. In addition, MSI participated in a quality sub￾group at the World Community of Practice in Toronto in 2011. MSI participated in the 2011 International Conference on Family Planning in Dakar, Senegal, delivering 21 presentations, panels, and posters at this esteemed global conference. Since being awarded SIFPO funding, MSI has been able to join USAID’s networks and technical working groups, which they were unable to access previously. This has facilitated new learning and partnerships as well as provided channels for the dissemination of MSI’s work and tools. For example, mobile money was disseminated via USAID’s M-health working group. MSI also benefited from being part of the International Best Practice working group and from participating in another group focusing on high-impact practices. Through these groups, MSI has been able to raise their profile and meet other potential partners and donors. With regard to FP2020, MSI is on committees for task shifting and metrics and has been able to share Impact 2 through the Committee on Performance, Monitoring, and Accountability. MSI also participates in the FP2020 country engagement rights and empowerment working groups. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 23 Barriers to Sustainable Programming Some major barriers to sustainable clinical programming centers are too few service delivery providers and the challenge of retaining clinical staff. MSI staff numbers have increased under SIFPO but are still not sufficient to provide long-term program implementation. It had also been determined that, in some instances, salary structures for paramedical staff were not in line with current standards in the communities of context. In Ghana for example, MSIG previously found it difficult to recruit staff due to low salary structures. However, in 2012, the program underwent a review of compensation packages, and revised the organizational salary structure to ensure that competitive packages are now offered. Nevertheless, with high and increasing patient volumes and long hours, care providers may also suffer from burnout, limiting the effectiveness of services provided. In Madagascar, the salaries paid to the CHEs were equivalent to <$2.50 a day (i.e. the target population of poor clients). It is important to note, however, that MSM’s CHEs are not full time salaried employees but part time community-based workers who typically engage in other local activities and CHW work. Remuneration provided to them is designed to be in line with levels provided by government and other NGOs to this cadre. A significant finding of the Kayayei project was that the majority of the FP services reported have been provided by one midwife. She reports providing services seven days per week since the project execution over a year ago. There was the addition of one nurse’s aide in May 2012, which has minimally assisted in the overall function of the program. During the course of the normal week, the registered midwife provides clinical services in one of three market sites, serving 35 or more clients per day and then showing the FP film two nights per week: “I usually get home around 10 p.m. and leave by 5 a.m. to get to the clinic on time. My children are grown so the girls (Kayayaei) have become my children.” The midwife also reports receiving 30 to 50 uncompensated phone calls per day. In fact, at the end of each FP visit, she gives clients her personal number on their return-to-clinic date card. The anecdotal report is that few to none of the subsequently received calls centered around FP services or management. Most addressed the social situations of the clients and immediate threats or concerns about personal safety or well-being. Another potential barrier to sustainable programming centers around MOH involvement in FP service delivery. Often MSI is filling LARC/PM gaps, while the MOH does provide short-acting methods. In the long term, the question could be asked regarding the MOH’s ability to maintain the capacity to provide these methods. It is evident that the various ministries involved are pleased with the ability of MSI to contribute to the increase in FP choice of the country. However, there is the potential for ministerial overdependence on MSI services. The specific request of one MOH was that MSI mobile outreach teams “leave some commodities behind” at the various health centers, so health center staff could utilize them during the one-month periods between MSI servicing of the area. In Ghana, the MOH reports that, due to cost, very few FP services are delivered outside of the MSI context. This may present a challenge for the retention of new skills obtained by the MOH staff in providing LARCs/PMs. The MOH staff in Ghana report that prior to MSI intervention and training there was no level of competency in providing these methods and that they generally do not offer FP services when MSI is not around. 24 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Result 4: Gender and Youth-Gender-Sensitive FP Services Targeting Youth Strengthened at a Global Level In the 2012 annual report, MSI notes that “of all the clients served by Marie Stopes International across our country programs, 7% are currently under the age of 20 and 23% under the age of 24.” Most developing countries have extremely young populations and so, while this is a promising start, it indicates that MSI still has considerable work to do in order to reach significant numbers of young people. In particular, MSI’s programming does not acknowledge at its core that FP needs, STI/HIV risk, and economic insecurity are all disproportionally found among the younger age groups. Young people tend to exhibit multiple vulnerabilities and therefore require integrated, specialist programming. There appears to be a tendency to adapt existing models of service provision—for example, adding youth services to BlueStar’s work in Madagascar—to attract young people. But it may be more effective to actively move services to the places they frequent and recruit peers from among the target group (such as the Kayayei project in Ghana). Research indicates young people often require specialist services, for example, outreach to places where they naturally congregate such as bars, youth groups, football grounds, and so on. GENDER AND YOUTH ACTIVITIES IN SIFPO COUNTRIES TO DATE Tanzania Youth Project MST implemented this project with funding from USAID/SIFPO to meet “the sexual and reproductive health (SRH) needs of adolescents and youth in Lindi and Mtwara, Tanzania,” in the coastal regions of Lindi and Mtwara from September 2011 to August 2012. The primary goal of the project was to improve the SRH of females and males aged 10 to 24 in the Mtwara and Lindi regions through increased voluntary uptake of SRH services. Over the course of the project, it is estimated that 3,350 people, predominately youth, attended these events. A total of 266 youth under 18 and 482 youth up to 24 accessed FP and voluntary counseling and testing (VCT) services during the project period. Zimbabwe Youth Social Franchising Training Through SIFPO, MSI piloted youth-friendly FP information and services in Zimbabwe, with a view to learning from this training and scaling up to multiple programs. As part of this activity, PSZ supported training 20 BlueStar social franchises in Matabeleland, Bulawayo Province, in two phases (March and May 2013). Of the 200 redeemable vouchers distributed, 190 were redeemed by the end of June, mostly for implants (148), then pills (37), IUD insertion (1), and implant removals (4). The Kayayei project in Ghana: The Kayayei are a highly marginalized group of people (predominantly young women) working in poverty and incurring great risks to their health and safety in the markets in Accra. The project delivers FP counseling and services, VCT for HIV, and referrals for HIV-positive Kayayei to receive treatment in government health facilities. It also seeks to reduce GBV by raising awareness and developing reporting structures within Kayayei communities. The project was observed as part of the evaluation and more detailed comments are presented throughout the report. Youth services in Madagascar: To date, seven outreach teams, as well as one clinic and one MS lady, have been trained in “youth-friendly” services. This included training on techniques for counseling young people, life skills training, and an emphasis on confidentiality. A video was SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 25 made for centers encouraging youth to attend them. In the near future, a new strategy will be implemented using both core and buy-in funds to promote E-Vouchers aimed at youth. In addition, MSM works with SISAL, a local NGO working with marginalized youth on the street. SISAL treats them for sexually transmitted infections (STIs) and refers them to MSM for FP. Male Involvement Activity During the year 3 workplan period (July 2012–June 2013), MSI selected Uganda and Kenya to commence implementation of male involvement activities with SIFPO core funding. Both countries have a high unmet need for SRH services and have documented low levels of male involvement. Marie Stopes Uganda (MSU) launched a vasectomy campaign to address the myths and misconceptions related to this FP method in two districts of Uganda, working in coordination with a marketing agency to design and develop key messaging. Marie Stopes Kenya (MSK) will focus on assessing vasectomy knowledge, attitudes, and practices among the urban population of reproductive age in Nairobi County, Kenya. GENDER AND YOUTH DOCUMENTATION GBV clinical guidelines have been produced. These guidelines will be supported by an implementation guide, to be developed in coming months. Gender assessments were undertaken in Ethiopia and Zambia, resulting in the Gender and Youth Toolkit being drafted and tested in Bangladesh. Once USAID approval has been secured, this will be rolled out across the partnership—a significant step for MSI. SIFPO contributed to the production of a best practice paper on MSI and youth. This is currently in draft form and awaiting feedback from internal team members. BARRIERS TO THE DELIVERY OF SRH TO YOUNG PEOPLE Heterogeneity among Youth The gender and youth assessment tool guided by ICRW recommends the use of DHS data to ascertain the demographic and social profile of potential clients. Young people’s sexual behavior and need for services can be extremely heterogeneous depending on, for example, educational status or place of residence. Table 6 uses data from the 2008 Ghana DHS and 2009 Madagascar DHS, which show significant differences in age at first marriage and age at first intercourse depending on place of residence. Table 4: Urban and Rural Differentials in Median Age at First Marriage and Median Age at First Intercourse in Ghana and Madagascar Median Age at First Marriage Median Age at First Intercourse Urban Rural Urban Rural Ghana 21.3 18.7 18.8 17.9 Madagascar 20.4 18.6 18.3 17.1 *Among women aged 25–29 at time of survey Thus, programs aimed at “youth” need to take account of this and acknowledge, for example, that urban Ghanaian women may spend more than two years being sexually active before marriage, whereas their rural counterparts spend less than one. This indicates that services 26 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION should be targeted differently among the youngest age groups. Currently, the Madagascar program’s exit interviews group 15–19 year olds together when, in reality, there is a large difference in the needs and behavior of a 15 year old and a 19 year old. In addition, many young people are sexually active below this age and have a need for information before they become teens. The new voucher campaign by MSM does, to some extent, recognize the heterogeneity of youth and has adapted its marketing plan accordingly. The campaign is heavily worded and seems to be aimed at middle-class literate youth, which will bypass the most vulnerable who are likely to be illiterate. However, it is unclear what proportion of at-risk youth can read, since many of the really marginal will be uneducated or poorly educated. They are also likely to have multiple vulnerabilities, possibly linked to drugs and alcohol, which may need to be addressed. In general, it seems that MSI, and MSM in particular, may know the general demographic profile of their clients but seem to prefer to target those who are easier to reach, literate, and probably less at risk of a range of SRH problems, including HIV/AIDS. This may be because strategies were built around exit interviews, which show who is using services rather than who is not. SOCIOCULTURAL BARRIERS TO SRH SERVICES The gender and youth tool places great emphasis on understanding the social norms regarding SRH supply and demand and yet, in the field, social norms do not seem to be adequately recognized as barriers to service use. Figure 4, taken from the gender and youth tool, provides an important framework for delivering gender- and youth-oriented services within the four pillars of the Power of 10 (access, equity, quality, and efficiency). It focuses on 1) individual and couple identity, 2) SRH information and service access, and 3) SRH information and service utilization that go on to create supply and demand. Figure 4: A Framework for Norms that Impact SRH Supply and Demand (Source: Kanesathasan and J McClearly-Sills [ICRW]) The following table has been adapted from the norms presented in ICRW’s gender and youth assessment tool based on figure 4 to illustrate selected sociocultural aspects of MSI programming in Ghana and Madagascar witnessed by the evaluation team. In many cases, the SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 27 recognition of the importance of sociocultural factors seems to be somewhat absent from MSI’s youth and gender programming to date. Examples from our evaluations have been added to show how such factors need to be considered to effectively shape service delivery, particularly to women and young people. Table 5: Adaptation of Table on Social Norms from Gender and Youth Assessment Tool with Specific Examples from Ghana and Madagascar Individual and Couple Identity SRH Information and Service Access Example from Evaluation Fieldwork Implication for Service Delivery Norms on masculinity and femininity Norms on health￾seeking and being informed about SRH Madagascar: Unmarried female peer educators unable to talk about own experience of FP use Better match peer educators to client base Norms on social status Norms on SRH decision-making Madagascar: Women usually stay at their mother’s house for several months after giving birth. Postpartum IUD insertion and follow-up may need to take into account that new mothers may switch place of residence Norms on relationships and concurrent partnerships Norms on communication about SRH issues Ghana: Concurrent relationships and unprotected intercourse affirm masculinity Need greater emphasis on integration of HIV services when targeting young men Norms on acceptance on intimate partner violence (within sexual relationships) Norms on mobility Norms on GBV Ghana: Kayayei require shelter in safe space Work with Kayayei association and police to find shelter and ensure safety Norms on male involvement in SRH Madagascar: High amount of secret use due to male disapproval (may govern method choice) Support secret use if necessary (not necessarily couple counseling or communication) The examples above are not exhaustive but shed some light on how understanding cultural SRH norms may better guide service provision. First, social norms in many countries, including Madagascar, may assume that unmarried women are not sexually active. Even through the reality may be very different, this norm may act as a barrier to providers’ and CHEs’ communication and sensitization. For example, an unmarried CHE had difficulty talking about contraception to married women and said, “They do not ask my advice as they assume I do not use family planning.” It may therefore be better to target married women with peers who are also married and to better train unmarried women to work with youth. Secondly, norms on social status in Madagascar dictate, particularly in the rural areas, that a woman return to her mother’s home to give birth and stay there for several months afterwards. This may mean, for example, that postpartum IUD insertions may not be done in a woman’s place of residence. This has implications for follow-up. Furthermore, in Ghana, informal chats with young men affirmed that multiple partners and unprotected intercourse assert masculinity among urban youth. This underscores the nature of multiple risks among youth and the importance of including HIV integration in any outreach work. Lastly, norms on male involvement with SRH may dictate that 28 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION a woman has to keep her FP use secret if her husband disapproves. In such settings, providers may need to support secret use and offer methods that a woman can use clandestinely, such as injectables and IUDs. In FP counseling, couple communication is usually encouraged. In such cases, however, it may lead to anger, disapproval, or even GBV. The limited examples above illustrate the need to understand the whole picture, including the social context of reproductive health, when providing gender-based and youth-friendly services. In particular, as described above, this requires actively involving youth in shaping the programs that target them. It was noticeable that both the youth program in Madagascar and MSM’s research team were run by individuals with clinical training as physicians. Young people can be engaged as researchers as well as informants to influence programmatic orientation and impact. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 29 IV.CONCLUSIONS AND RECOMMENDATIONS SIFPO has provided an exciting and unique opportunity for USAID to fund MSI as a service delivery organization in the field of FP. As USAID is the leading donor in the domain of FP, it provided the agency with a new mechanism for funding that is highly cost-effective. MSI has large networks of service providers in place and uses innovative channels to increase reach and is thus very economical in terms of achieving impact. USAID has benefited in many ways from the collaboration with MSI. In particular, it catalyzed new reflection within the Agency on direct programming and about integrating gender issues into service delivery. The SIFPO funding arrived during a period of great growth for MSI, and the organization was able to capitalize on this momentum to expand their services. SIFPO enabled the development of new management practices and internal accountability. In addition, it facilitated greater emphasis on quality assurance and enabled the development of an innovative range of evaluation instruments and metrics. MSI’s collaboration with SIFPO partners—the Population Council, ICRW, EngenderHealth, and the HIV/AIDS Alliance—not only enabled them to develop new tools for programming but also changed the way in which these four institutions looked at FP service delivery. This legacy within four influential and established partners is likely to have a global impact for years to come. With regard to organizational strengthening and capacity building, the SIFPO award required the setting up systems to ensure accountability and specific areas of compliance, which benefited MSI at both an institutional and country level. Voluntarism and informed choice are cornerstones of MSI’s work, thus, carrying out these mandates of USAID support did not require any institutional change. The ODT indicates great improvements in capacity. The capacity strengthening was evident in all sectors of MSI’s work. The number of franchisees increased significantly under SIFPO and mobile outreach was strengthened to have even more of an impact, especially on the poorest communities. This led to considerable savings in the health sector in terms of maternal deaths averted. These costs could be easily calculated with the Impact 2 tool developed by MSI under SIFPO and the figures presented to governments as an advocacy tool. A particularly important contribution to the field made by MSI and disseminated under SIFPO is that of M-health initiatives. As previously described, providers in Madagascar submit data relating to key indicators by mobile phone and are reimbursed by SMS. This and other innovative tools and practices including CLIC and high-impact CYPs, respond to the global agendas of FP2020 and MDG and can now be better documented and disseminated. This will take the field forward considerably and provide important resources to other partners to improve impact at a global level. Turning to research and M&E, SIFPO has funded important training in two other domains: operations research and data analysis. MSI recently held regional qualitative research training in Kenya at the request of country offices. Qualitative research has appeared rather late in MSI’s toolkit. However, greater implementation of qualitative research can help the organization better understand its client base and develop locally and culturally appropriate initiatives, especially for youth. The new CLIC initiative that follows clients longitudinally is groundbreaking and will provide important new information about contraceptive behavior to better inform 30 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION service provision. Qualitative research will also better enable MSI programs to determine who they are not currently reaching and the barriers their programming should address. To obtain the most accurate picture of clients’ profiles, behaviors, and satisfaction, MSI encourages programs to triangulate data from different sources. For example, this may mean looking at client satisfaction measures from more regular surveys of satisfaction (which are usually faster to fill out and self-reported), looking at data from mystery client surveys (where data collectors act as clients and evaluate service performance), and even looking for more anecdotal data (for example, from data collectors’ field notes). This is to be encouraged, particularly with regard to youth programming, which also requires more innovative and participatory methods. MSI prides itself on delivering quality services, even in challenging circumstances. All SIFPO countries have increased their QTA scores since receiving SIFPO funding. The evaluation concludes that, particularly due to the QTAs that provide regular support and monitoring, overall quality is good. The QTA is an example of south-to-south collaboration that can disseminate best practices across MSI country offices. There is some concern about if or when the updates to the clinical standards handbook suggested by EngenderHealth were incorporated. It is acknowledged that MSI often works with personnel who have had poor or limited training before joining the organization. Many field staff requested more ongoing support which would serve to identify such problems and improve quality further. MSI is very mindful of sustainability issues at a local level and has taken important practical steps to ensure their programs’ longevity. Most importantly, they have supported a culture of demand for FP in some countries. In Malawi (and probably in Tanzania, Madagascar, Zimbabwe, and Uganda), MSI has contributed significantly to the national modern CPR. One of the most important ways in which sustainability is ensured is via MSI’s approach to supporting, including, and training local MOHs. In addition, important innovations with regard to task sharing mean that much of the work of safely and effectively delivering LARCs and PMs can be carried out by appropriately trained and supported lower-level health personnel. Many BlueStar clinics provide services under national insurance schemes, again ensuring their sustainability and integration with government services. BlueStar franchisees are given sufficient training and resources so that they could operate independently if MSI withdrew. Importantly, cost sharing with other donors also leads to sustainable programming. In many instances, MSI-SIFPO has been used to pilot initiatives that are then taken to scale by other donors. This was the case of the outreach via tuktuks in Tanzania, which was piloted with core funds from USAID through SIFPO by MSI and then eventually scaled up with funding from DfID. It should be noted that the USAID networks and working groups in which MSI is now embedded thanks to SIFPO have provided opportunities to create links and forge partnerships with other potential donors. This can only further enhance sustainability. Additional efforts need to be made to improve south-to-south approaches to sustainability, but the intraorganizational administering of the “external” QTAs is likely to enhance south-to-south sharing of best practices within MSI. With regard to gender and youth programming, MSI has some more work to do, especially with regard to the latter. Nevertheless, important advances have been made in targeting youth in Tanzania, Zimbabwe, Madagascar, and Ghana. The Kayayei project in Accra is groundbreaking and can be used as a model for other youth-oriented services MSI may want to deliver in the future. Its strengths are the targeting of the very vulnerable and providing FP with integrated GBV services and HIV/AIDS prevention and testing. In Madagascar, the youth work included a voucher scheme to attract young people to BlueStar franchisees. However, it appeared to be SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 31 heavily geared toward literate youth and hampered by a lack of real consultation with local youth organizations. It was also very medicalized and physician-driven and did not really recognize the heterogeneity of young people. As an organization, MSI needs to better develop their youth services, building on formative research (as opposed to exit interviews) and identifying youth leaders and appropriate locations for outreach. In this way, the multiple vulnerabilities of the most at-risk youth can be identified and successful interventions that engage young people as both project beneficiaries and stakeholders can be designed. MSI’s work in Cambodia with MARPS goes some way to doing this, but future integration work—of prime importance in youth populations—will require USAID to free up additional HIV/AIDS funding. RECOMMENDATIONS For USAID  Increase ceiling to encourage multicountry buy-in and longevity of funding.  Attain and obligate more HIV funds for STI and HIV work, including integration of HIV with FP, especially in programs aimed at youth.  Consider methods that better align MSI’s and USAID’s fiscal program schedules to allow more effective long-range MSI programming. For MSI2: Organizational capacity  Better disseminate and share methods, tools, and findings.  Better disseminate clinical protocols and procedures from headquarters, possibly using M&E health approaches to avoid delays in updating staff.  Submit the results of studies that subsequently influence global practice for peer review.  Recognize both limitations and strengths of MSI’s tools.  Instigate better mentoring and coordinated capacity building of MSI’s local partners in￾country for more effective collaboration.  Identify ways in which MSI can strengthen the capacity of other international organizations and communities of practice. Quality assurance  Better disseminate clinical protocols and procedures from headquarters.  Follow up qualitatively on individual cases, acknowledging their social context.  Continue to ensure that provider counselling covers all methods, to ensure client understanding of the range of methods available.  Make “external” QTAs truly external using outside, objective evaluators rather than MSI staff from other countries. 2 It should be noted that the recommendations for MSI fall closely in line with the Power of 10 strategy. 32 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION  Supervise franchisees’ clinical skills and quality of service provision more closely, especially in FP counseling skills. – Provide ongoing support of franchisees by local professional bodies to create sustainable networks. – Investigate the possibility of investing in pre-service training to improve base capacities and clinical skills. – Increase inter- and intra-country communication and exchanges for all levels of providers, including CHEs and BlueStar providers. – Investigate the feasibility of SIFPO-specific MS Lady outreach mechanism for FP service provision. – Support CME mechanisms outside of MSI structures. Sustainability Better identify target populations and barriers to reaching them.  Increase support of private providers by expanding BlueStar out of clinics (e.g., chemical shops in Ghana).  Increase support of MOHs through joint service delivery, training, and exploration of task￾shifting mechanisms.  Explore contracting out/in with MOH for continuing nursing/medical education.  Consider pre-service arena with MSI as educator, clinical trainers/instructors, or clinical site.  Assist with policy development to encourage access to full method choice and task sharing.  Devote additional physical and financial support to provider recruitment and retention.  Encourage MSI to seek more bilateral funding resulting from SIFPO buy-ins (e.g., pilot projects brought to scale).  Increase south-to-south learning exchanges outside QTAs.  Tap into supply- and demand-side financing through insurance schemes and vouchers. Gender and youth  Encourage greater integration of FP, STI/HIV, and GBV services.  Better divide youth programming, including communication and service delivery, to address age-specific needs and vulnerabilities.  Differentiate between different youth segments socially, culturally, and based on behavior and risks.  Adopt WHO definition of youth (10–24 years) for better segmentation of youth programming. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 33  Improve communication and service delivery to address age-specific needs and better differentiation between different youth segments socially and culturally based on age-specific behavior and (perceptions of) risk. – Increase considerably the targeting of unmarried young men and women. – Use qualitative research to increase knowledge of the sociocultural context of FP service delivery and tailoring of delivery channels accordingly (e.g., workplace, bars/clubs, sports grounds, etc.). – Better recognize the heterogeneity of young people and matching the CHEs accordingly. – Increase targeting of the poorest youth and modify publicity and information material accordingly. – Further de-lexicalize approaches to working with youth. – Increase involvement of youth leaders and members of youth organizations (as peer educators, informants, and researchers). – Ensure security of young clients and peer educators/CHEs. 34 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 35 ANNEX I. SCOPE OF WORK GLOBAL HEALTH TECHNICAL ASSISTANCE BRIDGE PROJECT GH Tech Contract No. AID-OAA-C-13-00032 FINAL SCOPE OF WORK June 10, 2013 I.TITLE: Support for International Family Planning Organizations (SIFPO) Midterm Project Evaluation – Maries Stopes International (MSI) II.CONTRACT: Global Health Technical Assistance Project Bridge III (GH Tech) III.PERFORMANCE PERIOD Start date: o/a early June 2013 Completion date: o/a early August 2013 IV.FUNDING SOURCE PRH Core funding into the GHTECH Bridge III Mechanism V.OBJECTIVES AND PURPOSE OF THE ASSIGNMENT Overall Purpose:  To assess the MSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement  To gather information that will help to improve the management of the MSI project for the remainder of its implementation  To gather information that will result in useful recommendations for a potential future project External Technical Evaluation  To evaluate whether or not MSI project activities are leading to the results and outcomes outlined in the agreement  To identify if there have been any technical gaps that have prevented achieving intended results of the MSI project; and  Based on accomplishments toward results as well as the current/anticipated environment, identify potential technical future directions 36 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION VI.BACKGROUND The two Support for International Family Planning Organizations Projects with a period of performance of September 30, 2010–September 30, 2015 are cooperative agreements. The two projects are implemented by 1) Maries Stopes International with four implementing partners, including the Population Council, the International HIV/AIDS Alliance, the International Center on Research for Women, and Engender Health; and 2) PSI with two implementing partners, including Stanford Program for International Reproductive Education and Services (SPIRES) and Intra Health. The MSI-SIFPO project has a ceiling of $40 million and the SIFPO –PSI project has a ceiling of 39.9 million. To date, $32 million has been obligated to the MSI project with $2,250,000 in pending FY12 obligations,3 72% of which is field funding and 28% is core funding. A total of $21.5 million has been obligated to the PSI project, of which 61% is field funding4 and 39% is core funding. The SIFPO project’s mission is to increase the use of FP services globally through strengthening selected international FP organizations which have a global reach and an extensive, multicountry network of FP clinics, in order to achieve maximum program impact and synergies. The current SIFPO projects are working toward the following four results areas: Result 1: Strengthened organizational capacity to deliver quality family planning services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level A special focus of the SIFPO projects has been on capacity building and systems strengthening within the two recipient international FP organizations to build upon and leverage both organizations’ extensive network of country platforms of programming that are oftentimes funded by other donors. These capacity-building areas include improving clinical and counseling quality; standardizing and sharing best practices, such as around social franchising or mobile outreach programs; improving and standardizing metrics, such as for equity; improving health management and information systems; testing new approaches for service delivery and creating efficiencies in existing approaches, such as for mobile outreach or clinic-based approaches; creating a cadre of technical reproductive health leaders within the organization; and increasing evaluation and operations research skills of field staff. The vision is that by strengthening and streamlining procedures and systems of international FP organizations, these improvements will cascade down to local affiliates in developing countries, thereby strengthening and enhancing sustainability of these private sector partners to contribute to the overall health system. All project activities are organized into two categories: core-funded and field-funded activities. 3 Ethiopia, Cambodia, and South Sudan field support or sub-obligations were pending and in process as of January 15, 2013. Since these are pending yet in the USAID system, the amounts are included in the field support total. 4 As of January 15, 2013, Cambodia and Benin were in negotiations for additional sub-obligations to amount to approximately [missing something from this note?] SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 37 Core-funded activities include predominately element activities in FP with a very small amount of HIV/AIDS funding in the Marie Stopes agreement. Most field support funding is FP, with a limited amount of funding in HIV/AIDS and maternal and child health for select integrated service delivery activities. The following is an illustrative list of information sources that the evaluators should consult for the MSI project prior to conducting the evaluation: the annual program statement, the proposal, the agreement, annual workplans, annual reports, PMPs, financial reports, technical reports and papers, and other referential or historic documents. The SIFPO management team will provide a more comprehensive list of reference documents. This midterm MSI project evaluation should follow the Bureau of Global Health Guidelines for Management Reviews and Project Evaluations (2007), as well as the Agency’s new evaluation guidelines (2011). In addition, the evaluation should take into account relevant U.S. Government/USAID initiatives, policy developments, and reform efforts, such as the U.S. Government Global Health Initiative (GHI) especially the GHI principles, the Global Health Strategy for the GH Bureau, and USAID FORWARD. The GHI principles are linked to the Agency’s commitment to Paris Declaration aid effectiveness principles such as alignment with country strategies and priorities, strengthening and use of health systems, new partnerships and innovations, and strengthened monitoring and evaluation for accountability and results. The GHI principles also include a focus on women and girls and integrated services. The GHI goals for PRH are: 1) to prevent 54 million unintended pregnancies, 2) to increase contraceptive prevalence by 2 percentage points each year, and 3) to reduce first births to women under 18 by 15%. Additional performance measures of the PRH office are to increase the percent of births spaced more than three years apart, 2) reduce the percent of births order five or higher, and 3) increase the percent of demand satisfied through modern contraception. The two SIFPO projects contribute to all of these objectives and also to the Millennium Development Goal (MDG) 4 to reduce child mortality; MDG 5a Improve Maternal Health; MDG 5b universal access to reproductive health; and also to MDGs for poverty, education, environment, gender, and HIV/AIDS. The midterm evaluators should also take in account the FP2020 Summit follow-up actions as they relate to this MSI project and future designs. The goal of the FP2020 summit is to reduce unmet need and enable 120 million women and girls to use contraceptives by 2020. USAID’s GH priority, “A Promise Renewed: Ending Preventable Child and Maternal Mortality,” also strives to decrease maternal mortality and infant and under-5 mortality. FP service delivery is a key intervention to achieve both of these goals. VII.SCOPE OF WORK (TASKS TO BE PERFORMED BY THE CONSULTANT TEAM) The technical evaluation will focus strategically on big-picture and overarching questions as well as four of the MSI project’s technical areas. Big-picture and overarching questions can be divided into the following two categories: 1) questions about the existing MSI project and (2) questions relevant to the design of potential future project(s). The three technical areas that will be evaluated are: (1) The evaluation will examine the inputs of core resources intended for strengthening each recipient organization as a whole, with the intention that those inputs would strengthen country level platforms, and their effect on improved organizational capacity, sustainability as well as performance and management. (2) The evaluation should provide insight 38 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION into what additional support and strengthening are needed for improved sustainability at the country level. 3() In addition, while MSI project level output data and service delivery statistics reveal high quality performance on FP/RH service delivery to date, an external examination will contribute to validating those findings and suggesting areas for improvement or strengthening in FP/RH service delivery and quality assurance. Big-picture and Overarching Questions Questions for evaluation of SIFPO 1.How satisfied have various stakeholders been with the work done by and assistance from SIFPO including the following? – Missions (The MSI project management team will provide the evaluation team with the results of a Mission survey that has been recently done to help answer this question.) – GH – Other stakeholders, including other donors such as DfID 2.Issues for the follow-on project(s) – What existing gaps and future technical directions/issues need to be addressed in the follow-on that are not currently being addressed in SIFPO? – What kinds of inputs, specifically organizational strengthening, are no longer needed? Technical Evaluation Questions 1.Use of core resources for each recipient organization to strengthen country-level platforms, and their effect on improved organizational capacity, sustainability as well as performance and management. – What evidence exists that core resources invested in organizational strengthening have improved country-level platforms and programming? – What has been the effect on organizational capacity? – Is there evidence that there is increased sustainability? – How have management practices within the organization been affected? – To what extent has the MSI project’s internal organizational strengthening activities contributed to improved capacity at the country level (including at local and affiliate NGOs, government service sites)? – What are the facilitators and barriers to achieving the intended results? – What is the quality of the trainings and tools used to roll out some of these organization-wide system changes (based on the available evidence, for example, evaluations by the participants, including headquarters and field staff)? 2.Additional support and strengthening that are needed for improved sustainability at the country level. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 39 – What is the experience with the different approaches to achieve sustainable programming? (Please include any instances of governments contracting out for mobile outreach, different mobile outreach strategies, training public sector workers on the job, as well as social franchising approaches with vouchers, in the analysis. Also, please include cost-share and the organization’s leveraged and own funding in the analysis.) 3.Areas for improvement or strengthening FP/RH service delivery and quality assurance. – What evidence exists that stakeholders have found the MSI project’s service delivery effective? – How effectively is the MSI project partnering and collaborating with other CAs and global partners involved with service delivery activities? – Is there evidence that the internal quality assurance standards have been disseminated to strengthen FP service delivery and performance at a global level? – To what extent is the MSI project’s portfolio of service delivery activities meeting the needs of stakeholders? VIII.METHODOLOGY Data Collection The evaluation team will work collaboratively with the USAID management team to develop a detailed workplan as well as a data collection strategy, including data collection instruments. For the technical evaluation, it is envisioned that a select number of countries with moderate to high investments (money and time) would be selected for field visits and the three technical areas of focus. The evaluation team will consult with and receive approval from the USAID SIFPO management team as to the selection of countries for field visits. The primary methodologies for this evaluation will include (1) document review, (2) in-depth key informant interviews, (3) focus group discussions, (4) surveys, and (5) direct observation. The specific methodologies for each of the evaluation areas are identified and described below; however, where feasible, methods should be combined to address multiple questions at once. 1.Document review for MSI project: – Big-picture, overarching questions, and specific focus areas:  APS Solicitation document  Project agreement(s)  Semiannual and annual reports  Performance monitoring plan  Workplans  SOWs for field-funded activities  Results reporting (mission & HQ) 40 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION  Management review presentations and memos  U.S. Government Global Health Initiative (GHI) strategy  Global Health Bureau GHI strategy  USAID FORWARD reform agenda  Review “use” of products/methods/tools/papers, including website downloads and dissemination of products via CD, print copy, etc.  Participant evaluations of trainings, workshops, other country-level activities  Examination of the curricula and training objectives 2.Key informant interviews – in-depth, semi-structured interviews, in person when possible (for example, during country visits, at USAID/Washington and MSI headquarters office), alternatively via phone or video conference. – Big-picture and overarching questions:  Project staff, including those from the field and headquarters  BGH stakeholders  USAID missions  USAID/Washington staff  Project partner organizations and other CAs 3.Focus group discussions – Representatives of local ministries (clinical staff) – Project field staff and SIFPO staff – Clients 4.Surveys – Survey (email/web-based/phone) with USAID missions that have used, and those that have not used, the project’s services (Some of this exists in the existing Mission survey, but further follow-up may be needed as the response rate was low) – Survey SIFPO staff – Survey partner organization staff on collaboration and communication – Survey key stakeholders—ask if their feedback was requested, if future interactions reflected any of the changes suggested – Email/web-based survey to community of practice participants, including LAPM working groups SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 41 5.Direct observation – Big-picture and overarching questions:  Interview all relevant staff at HQ and in the field of the international NGOs.  Observe activities in countries for specific focus areas IX.TEAM COMPOSITION, SKILLS, AND LEVEL OF EFFORT Team Composition There will be two separate and staggered evaluations for each international NGO (MSI and PSI). Both evaluations will use this scope of work (SOW). Each evaluation team will consist of two professionals that have demonstrated knowledge and experience in the areas described below. Depending on consultants identified, it is tentatively suggested that the team consist of the following professionals: one team leader with organizational development expertise and one clinical FP/RH specialist. It is expected that each of the skills and qualifications described below are covered in their entirety by the evaluation team; however, it is understood that specific skills may fall differently across each of the two job descriptions than what is listed below. Team leader/M&E specialist will oversee all aspects of the evaluation. The team leader will liaise with the other consultants and with USAID/GH, oversee data collection and analysis, write sections of the report, and meld contributions of the technical consultants into a coherent set of responses and present conclusions and recommendations to USAID. The team leader should have prior experience and expertise in program evaluation and assessment, understanding of USAID program processes, and experience in monitoring and evaluation of global health programs. Qualifications include:  Track record of successful oversight of the evaluation of complex international technical assistance projects, preferably in health and family planning  Excellent oral and written communication skills in English, including the ability to facilitate groups and present complex material  Demonstrated knowledge of USAID’s policies and priorities in PRH and other health experience working in developing countries  A background and experience in organizational development Skills in designing qualitative and survey research instruments and methodologies:  Knowledge of monitoring and evaluation in the area of international health (FP/RH or other health)  Must be available for travel FP/RH clinical services and quality specialist will have specialized evaluation experience and expertise in clinical programming in family planning in the international health and/or development sector. This individual will bring the lens of his/her subject matter expertise and experience to bear on all aspects of the SOW. S/he will work closely with the team leader to assess the progress, quality, and relevance of approaches of the family planning activities of the 42 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION project. S/he will work seamlessly with the team leader to interview key informants, conduct data collection and analysis, and write sections of the final report. Qualifications include:  Demonstrated ability to implement and evaluate FP/RH clinical service delivery programming and quality standards in developing countries  Some understanding of integrated health programming (FP/RH and HIV and/or FP/RH and MCH)  Experience with evaluating different service-delivery models and their potential sustainability at both the program and country level  Demonstrated ability to evaluate programming that serves underserved populations  Must be available for travel Illustrative Level of Effort Table Task LOE Tasks LOE Team Leader FP/RH Specialist Document review 5 days 5 days Team planning meeting 2 days 2 days Meeting with USAID/Washington 1 day 1 day Meetings at each international NGO with HQ staff 2 days 2 days Creation of data collection instruments 3 days 3 days Data instruments review by USAID (2 days) N/A N/A Revision of data collection instruments 2 days 2 days Data collection includes interviews with key informants, field visits, and email/telephone surveys 21 days 21 days Data analysis 4 days 4 days Report writing 5 days 5 days Debrief 1 day 1 day USAID and MSI provide comments on first draft (10 days). N/A N/A Consultant team revises report based on USAID comments 5 days 3 days USAID reviews second draft (2 days) Team leaders incorporates final edits, if any 1 day N/A USAID provides final sign off of second draft (2 days) N/A N/A Final presentation at USAID 1 day 1 day Travel days 6 days 6 days Total # of days (estimated) A six-day work week is approved for in-country work. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 43 X.LOGISTICS The SIFPO management team will provide overall direction to the evaluation team, identify key documents and key informants, and liaise with the international NGOs headquarters’ staff and USAID missions to ensure logistical support for field visits prior to the initiation of field work. The SIFPO management team shall be available to the team for consultations regarding sources and technical issues, before and during the evaluation process. Roles and Responsibilities Before Work 1.Consultant conflict of interest (COI): To avoid COIs or the appearance of a COI, review previous employers listed on the CVs for proposed consultants, and provide additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates. 2.Documents: Identify and prioritize background materials for the consultants and provide them, preferably, in electronic form. 3.Site visit preparations: Provide a list of site-visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. 4.Lodgings and travel: Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation) and identify a person to assist with logistics (i.e., visa letters of invitation, etc.). 5.Work closely with two international NGOs to develop and finalize the in-country schedule and logistics with all transportation costs funded by GHTECH Bridge III. During Work The two international NGOs will assist the team to identify and arrange (although the costs for this activity may be covered by funding from the GHTECH Bridge III) 1.Formal and official meetings: Arrange key appointments with national and local government officials. 2.Other meetings: If appropriate, assist in identifying and helping to set up meetings with local professionals relevant to the assignment. 3.Facilitate contacts with partners: Introduce the team to project partners, local government officials, and other stakeholders, and—where applicable and appropriate—prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After Work The evaluation team will provide 1.Timely reviews: Provide timely review of draft/final reports and approval of the deliverables. XI.DELIVERABLES AND PRODUCTS  Final workplan and data collection instruments: The evaluation team will prepare a detailed workplan in response to SOW requirements and evaluation questions. The detailed 44 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION workplan should identify the countries for site visits, the individuals and stakeholders for surveys and in-depth interviews and should include each of the proposed data collection instruments (i.e., structured interview guides, surveys, observation forms, etc.). A draft of the detailed workplan and data collection instruments should be submitted to the SIFPO management team for input prior to finalization.  Draft report: This report should describe the findings from the technical evaluation as well as findings related to the big-picture and overarching issues spanning the evaluation. The report should separately and comprehensively address each of the objectives and questions listed in the SOW as well as the findings, interpretations, conclusions, and recommendations, which should be clearly supported by the collected and analyzed data. Findings should be presented graphically where feasible and appropriate, using graphs, tables, and charts. The final report should make recommendations for future action, including recommendations that may be relevant to the implementation of the second half of the existing MSI project as well as for the redesign of the future project(s) in either technical and/or managerial aspects. The report should not exceed 40 pages in length (not including appendices, list of contacts, etc.). The final report should contain an executive summary; table of contents; and main text, including findings, conclusions, and recommendations. Annexes should include the scope of work, description of the methodology used, lists of individuals and organizations consulted, data collection instruments (i.e., questionnaires and discussion guides, etc.), and bibliography of documents reviewed. The executive summary should accurately represent the report as a whole and should not exceed two pages in length. – Evaluation findings should focus primarily on those issues within USAID technical and management staff’s manageable interest. Discussion of those issues that are outside of the realm of influence of these staff, such as, but not limited to, issues of a political nature, funding constraints, or limitations with the Global Health/Child Survival Account, and so forth, should be reserved for, if at all, the limitations section. – Forward-looking report /memo (internal use only): The evaluation team will prepare a memo which will provide an assessment on future thinking among interviewees and other relevant knowledge sources for the purposes of future project design. The internal memo will include recommendations from what interviewees and other knowledge sources believe are the priority areas and opportunities USAID should consider investing in the future. This portion of the report will not be made public and will be for USAID internal use only.  Final Report: After receiving the draft version of the report, USAID will have 10 days to respond with one set of comments. The team will then have 5 days to revise the report and submit it to USAID. USAID will have up to 4 days to review second draft and provide final sign off. An electronic version of the final report should be submitted to the SIFPO management team along with 15 hard copies. GH Tech Bridge III will provide the edited and formatted final document approximately 30 days after USAID provides final approval of the content. The report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) (http://dec.usaid.gov) and the GH Tech project website www.ghtechproject.com). SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 45  Final Presentation: The final report is to be accompanied by a PowerPoint presentation that aims to debrief selected stakeholders on the results and recommendations stemming from the midterm evaluation. A draft of the final presentation should be submitted to the SIFPO management team prior to finalization. XII.RELATIONSHIPS AND RESPONSIBILITIES This evaluation will be a participatory external review, in the sense that the GH Tech evaluation team will work collaboratively with the USAID management team throughout the duration of the evaluation. The evaluation team will consult with the USAID SIFPO management team regarding the methodology, approach, and data collection instruments, but will be primarily responsible for data collection, analysis, and report writing. XIII. MISSION AND/OR WASHINGTON CONTACT PEOPLE/PERSON USAID management team points of contact: Marguerite Farrell, AOR; Elaine Menotti, senior technical advisor MSI-SIFPO project point of contact: James Harcourt PSI-SIFPO project point of contact: Jennifer Pope Mission points of contact: Robert Kolesar (Madagascar); other country, TBD XIV.COST ESTIMATE (EXCLUDED) XV.REFERENCES (PROJECT AND RELEVANT COUNTRY DOCUMENTS)  Guidelines for Management Reviews and Project Evaluations, Bureau of Global Health, 2007  USAID Evaluation Policy, 2011  APS  Project proposals  Cooperative agreements  Project workplans (years 1–3)  Project semiannual and annual reports  PMPs  SOWs for field-funded activities  Trip reports  Financial reports  DfID evaluation of MSI Tanzania joint work with USAID  Case study of Tanzania DfID Gift to USAID for MSI 46 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION  Management review memo and presentations  Checklist for evaluation reports  U.S. Government Global Health Initiative (GHI) strategy  USAID FORWARD reform agenda  FP2020 Summit website  Global Health GHI strategy  Project papers and case studies SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 47 APPENDIX A: USAID CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT  The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why.  The evaluation report shall address all evaluation questions included in the SOW. (Although the report should not answer each question directly in the report but should thematically and in an integrated fashion in the narrative address the evaluation questions.)  All modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the AOR.  The evaluation methodology shall be explained in detail and all tools used in conducting evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology.  Evaluation findings should be presented as analyzed facts, evidence, and data, and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 48 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 49 ANNEX II. KEY INFORMANTS USAID/WASHINGTON USAID SIFPO Management Team  Marguerite Farrell, GH/PRH/SDI  Elaine Menotti, GH/PRH/SDI GH Leadership  Jim Shelton GH/PRH Leadership  Ellen Starbird  Jeff Spieler GH/PRH  Bev Johnston, Liz Bayer Policy and Compliance GH/PRH/SDI, Division Chief  Kathryn Panther GH/PRH/SDI Division  Patricia MacDonald, LAPM technical priority champion  Victoria Graham, community FP priority champion  Alex Todd, repositioning FP technical priority champion  Carolyn Curtis, PAC technical priority champion GH/PRH/PEC  Carmen Coles, repositioning FP technical priority champion  Shelley Snyder, FP point person Benin country team  Michal Avni, gender technical priority champion GH/PRH/RTU  Tabitha Sripipitana (also works on FP/HIV integration)  Linda Sussman GH/PRH/CSL  Mark Rilling, division chief GH/OHA  Lindsey Miller 50 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION AFR  Ishrat Husain AME  Jen Mason or Katie Qutub USAID COUNTRY MISSIONS (Telephone/email interviews for countries not visited by the evaluation team) Tanzania  Aly Cameron Madagascar  Robert Kolesar Zimbabwe  Jo Keating  Reena Shukla Ethiopia  Zewditu Kebede Ghana  Susan Wright (with Abt. Jordan now)  Salamatu Fata  Vandana Stapleton Cambodia  Robin Martz South Sudan  Basilica Modi  Heather Smith (in PRH/SDI now) Pakistan  Anna McCrerey  Kate Crawford OTHER STAKEHOLDERS Partner Organizations under MSI-SIFPO International HIV/AIDS Alliance MSI to provide key point(s) of contact SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 51 EngenderHealth  MSI to provide key point(s) of contact Population Council  MSI to provide key point(s) of contact PSI  Jennifer Pope  Maxine Eber DfID  Liz, Tanzania  Nels Druce Gates  Guy Stallworthy University of California at San Francisco, Global Health Program  Dominic Montague 52 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 53 ANNEX III. REFERENCES POWERPOINT PRESENTATIONS MSI-SIFPO Midterm Evaluation – MSI Overview MSI-SIFPO Midterm Evaluation – SIFPO Overview Social Franchising – Harnessing Private Sector Capacity to Strengthen Health Sectors MSI – Outreach MIS Transformation – Client Information Center (CLIC) MSI-SIFPO – Madagascar and Ghana MSI-SIFPO Management Review Task Sharing Policy Model: Showing the Potential Impact of FP Policy Change U.S. Government FP Statutory and Policy Requirements: Training Presentation Local Successes: Global Leadership on Task Sharing Clinical Quality at MSI Championing Evidence-based Decisions at MSI: Building Capacity and Strengthening Evidence mHealth at MSI The Cost Calculator: Breaking down MSI’s Costs Longer-term Perspectives and Looking Forward QTA Scores – SIFPO Buy-ins REPORTS Ali, Mohamed M., Rachael K. Sadler, John Cleland, Thoai D. Ngo, and Iqbal H. Shah. Long-term Contraceptive Protection, Discontinuation and Switching Behaviour: Intrauterine Device (IUD) Use Dynamics in 14 Developing Countries. London: World Health Organization and Marie Stopes International, 2011. Approche Jeune 2012 et 2013: Elaboration Plan Marketing Jeune en 2012. 2012. Cambell, J. and N. Corby. Increasing Family Planning Access and Choice: Key lessons from Marie Stoped International’s Clinical Outreach Programmes. London: Marie Stopes International. (Undated) Corby, N. Using Mobile Finance to Reimburse Sexual and Reproductive Health Vouchers in Madagascar. London: Marie Stopes International. (Undated) Hanitriniaina, O. Enquete Voucher Tracing: Projet SIFPO-Madagascar 2011–2012. Support for International Family Planning Organizations and Marie Stopes International, 2012. Jacobstein, R. “Lessons from the Recent Rise in Use of Female Sterilisation in Malawi.” Studies in Family Planning 44(1): 85-95. 2013. Jacobstein, R., C. Curtis, J. Spieler, and S. Radloff. “Meeting the Need for Modern Contraception: Effective Solutions to a Pressing Global Challenge.” International Journal of Gynecology and Obstetrics (in press). 54 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION MSI. Becoming a Community Health Worker (CHW) within the MSI Partnership: Initial CHW Training – Trainer’s Guide (Draft). London: Marie Stopes International, 2013. MSI Business Case Outline: For the Consideration of Expanding MSI’s Interventions into Niger (Draft). London: Marie Stopes International. (Undated) MSI. Draft Self-Assessment Tool for the Integration of HIV and SRH Services in our Programmes. London: Marie Stopes International, 2012. MSI. Family Planning Outreach Services: Reaching the Underserved with Family Planning Services in Ghana. Workplan submitted to USAID for SIFPO funding. London: Marie Stopes International, 2012. MSI. Ghana Factsheet: 2012 Exit Interviews. London: Maries Stopes International, 2012. MSI. Incident Notification Form: Appendix A. London: Marie Stopes International, 2012. MSI. Incident Reporting, Investigation and Management: Version 1 January 2013. London: Marie Stopes International, 2013. MSI. Journey of an Incident: Appendix C. London: Marie Stopes International. (Undated) MSI. Logique d’intervention projet E voucher for youth. London: Marie Stopes International, 2013. MSI. Marie Stopes International Clinical Standards. London: Marie Stopes International, 2009. MSI. Partnership with Private Clinics to Expand Access to Contraceptive FP Services, Ethiopia: Work Plan Year 1, October 2011–September 2012. London: Marie Stopes International, 2011. MSI Partnership with Private Clinics to Expand Access to Contraceptive FP Services, Ethiopia: Work Plan Year 1, October 2012–September 2013. London: Marie Stopes International, 2012. MSI. Postpartum IUCD Curriculum (Draft). London: Marie Stopes International, 2013. MSI. Profil clients et satisfaction clients MSM: tableau de synthese. London: Marie Stopes International. (Undated) MSI. Programmatic Brief: Integrating Voluntary Medical Male Circumcision into Sexual and Reproductive Health Programming. London, Marie Stopes International. (Undated) MSI. Programme Investigation Report Proforma – Appendix D. London: Marie Stopes International, 2011. MSI. Protocol for Client Referral/Transfer to Another Health Facility. London: Marie Stopes International, 2013. MSI. QTA Checklist 2013 –Outreach Summary. London: Marie Stopes International, 2013. MSI. The Power of Ten: A Strategy to Achieve Greater results in Family Planning and Safe Abortion 2011–2015. London: Marie Stopes International. (Undated) MSI Stratégie Jeune MSM. London: Marie Stopes International, 2012. MSI. Table of Obligations by Country. London: Marie Stopes International. (Undated) MSI Cambodia. Rapid Assessment of Family Planning HIV Integration in Cambodia. Marie Stopes International, 2012. MSI Ghana. Family Planning Outreach Services: Reaching the Underserved with Family Planning Services in Ghana. Workplan submitted to USAID for SIFPO funding. Marie Stopes International Ghana, 2012. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 55 MSI Ghana. Minor Incident Report Form. Marie Stopes International. (Undated) MSI-SIFPO. Addendum to Start-up Workplan: Request for Reprogramming. Support for International Family Planning Organizations and Marie Stopes International, 2011. MSI-SIFPO Addendum to Year 3 Workplan – Request for Reprogramming. Support for International Family Planning Organizations and Marie Stopes International, 2013. MSI-SIFPO. Clinical Guidelines for Sexual and Gender-based Violence. Guidelines for MSI Country Programmes in the Provision of Clinical Services to Survivors of SGBV. Support for International Family Planning Organizations and Marie Stopes International, 2013. MSI-SIFPO. Draft Gender and Youth Self-Assessment Tool. Support for International Family Planning Organizations and Marie Stopes International, 2013. MSI-SIFPO. Family Planning Outreach Services: Reaching the Underserved with Family Planning Services in Ghana. SIFPO Quarterly Report, January–March 2013. Support for International Family Planning Organizations and Marie Stopes International, 2013. MSI-SIFPO. Implementing Marie Stopes International Family Planning Voucher Programmes with USAID Support: A Guide for MSI Country Programmes in Designing and Implementing Family Planning Voucher Schemes, in Compliance with United States Government FP Statutory and Policy Requirements. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. Increasing Access to Sexual Reproductive Health Services and Reducing Gender-Based Violence within the Kayayei Community. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. Increasing Access to Sexual Reproductive Health Services and Reducing Gender-based Violence within Kayayei Community: Workplan for the Period of October 2012–June 2013. Support for International Family Planning Organizations and Marie Stopes International, 2012. MSI-SIFPO. First Annual Performance Monitoring Report (30th September 2010–30th June 2011). Support for International Family Planning Organizations and Marie Stopes International, 2011. MSI-SIFPO. First Semiannual Report (30th September 2010–31st March 2011). Support for International Family Planning Organizations and Marie Stopes International, 2011. MSI-SIFPO. MSI-SIFPO Management Review Questions: Year 3, July 2012–June 2013. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. Piloting Innovations to Increase Access to Family Planning Choices and Integration of Family Planning and HIV/AIDS Services. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. PRH Template for Results Reviews. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. Report on Baseline Study on Family Planning, HIV/AIDS, STI and Gender-based Violence in Selected Kayayei Communities in Accra. Support for International Family Planning Organizations and Marie Stopes International, 2013. MSI-SIFPO. Research, Publications, and Tools. Support for International Family Planning Organizations and Marie Stopes International. (Undated)MSI-SIFPO. Revised Start-up Workplan: September 30th 2010–June 30th 2011. Support for International Family Planning Organizations and Marie Stopes International, 2011. 56 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION MSI-SIFPO Second Annual Performance Monitoring Report (1st July 2011–30th June 2012). Support for International Family Planning Organizations and Marie Stopes International, 2012. MSI-SIFPO. Second Semiannual Performance Monitoring Report (1st July–31st December 2011). Support for International Family Planning Organizations and Marie Stopes International, 2012. MSI-SIFPO. MSI-SIFPO Management Review Questions: Year 2, July 2011–June 2012. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI￾SIFPO. Start-up Workplan Period: 30th September 2010–30th June 2011. Support for International Family Planning Organizations and Marie Stopes International, 2011. MSI-SIFPO. Strengthening Delivery of Comprehensive Family Planning and Sexual and Reproductive Healthcare in Madagascar. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. Strengthening Delivery of Comprehensive Family Planning and Sexual and Reproductive Healthcare in Madagascar: Year 2 Workplan October 2012–September 2013. Support for International Family Planning Organizations and Marie Stopes International. (Undated) MSI-SIFPO. USAID Memorandum from the MSI-SIFPO Management Review. Support for International Family Planning Organizations and Marie Stopes International, 2012. MSI-SIFPO. Year 2 Proposal and Workplan: January–September 2013. Support for International Family Planning Organizations and Marie Stopes International, 2012. MSI-SIFPO. Year 2 Workplan: 1st July 2011–30th June 2012. Support for International Family Planning Organizations and Marie Stopes International, 2011. MSI-SIFPO. Year 3 Workplan: 1st July 2012–30th June 2013 (revised). Support for International Family Planning Organizations and Marie Stopes International, 2012. USAID. USAID Evaluation Policy: Learning from Experience. Washington, DC: USAID, 2011. USAID. USAID Policy on Gender Equality and Female Empowerment. Washington, DC: USAID, 2012. WHO. Responding to Intimate Partner Violence and Sexual Violence Against Women: WHO Clinical and Policy Guidelines. Geneva: WHO, 2013. SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 57 ANNEX IV. EVALUATION METHODS AND LIMITATIONS The evaluation methods are fully described in the body of the report. LIMITATIONS In both settings, there was a distinct amount of “hovering” by MSI staff during our interviews, which were supposed to be confidential. We asked that MSI staff not be present, and generally they were not, but in some instances, such as at the Kayayei project, staff members were in the close vicinity while clients and educators were interviewed. In this and in other settings where it occurred, staff presence may have biased interviewees’ or clients’ responses. In the Kayayei project, there was also a misunderstanding about who constituted “clients,” which were chosen by the project director. The two interviewed turned out to have gained their family planning from other sources outside the project. For each woman, this was not discovered until mid￾interview, but the interviews continue nevertheless, as they elicited interesting socioeconomic information about the Kayayei’s as well as their views on other aspects of the project, such as GBV prevention. In the Ghana outreach setting, drivers play an active part in the sensitization of women in the waiting room. However, in several instances, the driver was present during clinical consultations. The clients were not asked if this was acceptable to them, and it may have biased some of their interactions with the provider, which the evaluator was observing. 58 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 59 ANNEX V. PERSONS INTERVIEWED USAID/Washington Maggie Farrell MSI-SIFPO AOR Elaine Menotti MSI-SIFPO Technical Advisor Andrew Park MSI-SIFPO Program Assistant Michal Avni Gender Specialist Kathryn Panthyer GH/PRH/SDI Division Chief Trish MacDonald LARC/LAPM champion Carolyn Curtis Post-Abortion Care Champion Liz Bayer Policy fellow Ellen Starbird Office Director, PRH Jim Shelton Senior Science Advisor Alex Todd Repositioning FP Advisor MSI Washington Nomi Fuchs Montgomery Director MSI, US Michelle Weinberger Demographer, MSI US MSI London Michael Holscher Interim CEO Claire Morris VP and International Programmes Director James Harcourt Director, MSI-SIFPO Anna Mackay Deputy Director, MSI-SIFPO Brendan Hayes Head of Social Franchising Sue Holland Regional Director, West Africa Rachael Saddler MSI Programme Manager Sam Guy Executive Office Director John Worley Global Policy Advisor Grethe Peterson Regional Director, East Africa Martyn Smith Deputy Regional Director, Asia Jennifer Tuddenham SIFPO Field Support Officer Heidi Quinn Former SIFPO Senior Technical Advisor Anne O’Connor Associate Director, MDT Patricia Atkinson VP and Health Systems Director Barbara Reichwein M&E Advisor Thoai Ngo Head of Research Olivia Nuccio M&E Advisor 60 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Judy Gold Innovation Analyst Kenzo Fry Research Analyst Alex Brown VP and Global Finance Director Sally Hughes Deputy Regional Director, Asia Dana Hovig Former MSIO Executive Director USAID/Madagascar Robert Kolesar Senior Health Advisor Volontsoa Raharimalala M&E Manager MSI Madagascar Nicole Raatgever Country Director Lalaina Razafinirinasoa Projects, Innovations, and Evidence Director Hasinara Rasoloharimahefa Channel Manager Olinirina Randrianantoandro Operations Director Dr. Thierry Ramanantsoa Quality Assurance Manager, Outreach Dr. Jasmin Channel Manager Rasolofonirina Rijalaina M&E Manager Dr. Odile Hanitriniaina Research Manager Dr. Joelle Rabesoa Clinical Training Manager Tovo Ranaivomino Integrated Marketing Director Franchisees CHEs Outreach team CHWs Clients Other Madagascar Agnès Duban Economist, EU Dr. Yvette Ribaira, JSI/MAHEFA Deputy Chief of Party USAID/Ghana Salimatu Futa Maternal and Child Health Specialist MSI GHANA Faustina Fynn-Nyame Country Director Andrea Fearney Deputy Director Simeon Archeampong Outreach Coordinator Harriet Asomani Head of Finance and IT Abass Dagbui Senior Finance Officer Patrick Tweneboah Finance Officer, Compliance Abu Mahamadu Kayayei Project Coordinator SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 61 Fred Gbagbo MDT Manager Esther Worae Kayayei project peer educators Kayayei project beneficiaries Outreach team Clients Other Ghana Isaac Mohamed I Salifu Kayayei Association President Elizabeth Dassah National Coordinator DOVVSU Lydia Agyapong Programme officer DOVVSU Aku Xornam-Adzraku SWAA National Coordinator Benjamin Grant SWAA Accountant Cecelia Senoo Lodrunu SWAA Former President and Board Member Wilburforce Adade Regional Health Director (Gomoa East) Crystal Clottey District Nurse (Gomoa East) Veronica Amoako Assistant District Nurse (Gomoa East) GHS staff 62 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 63 ANNEX VI. DATA COLLECTION INSTRUMENTS MSI Country Questionnaire Question Guidelines for Partners Questionnaire for USAID Missions in SIFPO Countries Questionnaire for Voucher or Outreach Clients Questionnaire for Community Educators FP Clinical Checklists 64 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANING ORGANISATIONS (SIFPO) MIDTERM PROJECT EVALUATION - MARIE STOPES INTERNATIONAL You are being contacted in order to answer some brief questions which will assist with the midterm evaluation of the MSI-SIFPO program and help to build additional knowledge about family planning services. The overall purpose of the evaluation is:  To assess the MSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement  To gather information that will help to improve the management of the MSI project for the remainder of its implementation  To gather information that will result in useful recommendations for a potential future project  All questions posed in the questionnaire will assess the following results categories: Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level It would be helpful if your answers could reflect the overall expected program results. It is kindly requested that you answer all of the questions proposed completely providing sufficient detail for analysis. Your written comments can be made in the text boxes provided. Please feel free to add extra comments at the end of the questionnaire. This questionnaire is for evaluation purposes and was commissioned by USAID, therefore review by an institutional review board was not required. Results of the questionnaires will be shared and will be presented in a format that protects the specific identity of respondents while providing country-specific information. Responding to the survey is voluntary and implies consent. The questionnaire will take about 25 minutes to complete. We are asking that you complete the question survey by July 5, 2013, at the close of business. If you wish to have a more detailed discussion via telephone or Skype, please provide contact details and indicate this on your returned survey. Thank you for your assistance with this evaluation. Regards, Sarah Castle, PhD, and Pandora Hardtman, RN, CNM, DNP SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 65 QUESTIONNAIRE FOR MSI COUNTRY OFFICES Name of respondent…………………………………………………… Email of respondent……………………………………………………. Telephone number of respondent………………………………….….. Position of respondent………………………………………………… Please note that the above details will be kept confidential DEMOGRAPHIC DATA Country Table Please tick Cambodia Ethiopia Ghana Madagascar Pakistan UK USA South Sudan Tanzania Zimbabwe Please indicate which organization you represent: USAID Mission MSI Country Office --------------------------------------------------------------------------------------------------------------------- 66 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Q1. How satisfied are you and your team with MSI’s work in the following areas? Very Satisfied Satisfied Not Satisfied Please expand upon your response Mobile Clinic Outreach Social Franchising FP Vouchers Youth FP/HIV Integration Gender-based Violence Q2. How have SIFPO core resources impacted upon MSI’s organizational capacity and sustainability as well as on the performance and management of your country program? Please give detailed examples. Q3. What evidence exists that core resources invested by SIFPO in the organizational strengthening of MSI and its local partners have improved country-level platforms and programming? Please give specific examples from your country. Q4. How have management practices within MSI been affected by SIFPO? (For example, positively or negatively?) Please give detailed examples to support your response. Q5. In your country, to what extent has MSI’s internal organizational strengthening activities (carried out within the SIFPO framework) contributed to improved capacity, including that of local and NGOs and government services? Please give detailed examples to support your response. Q 6. Based on the available evidence, what is your assessment the quality of the trainings and tools? SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 67 Tools and trainings are: High quality Average quality Poor quality Please give examples of tools and trainings and explain the reasons for your assessment Staff continual professional development Staff recruitment and retention Data management and use Monitoring and evaluation Quality assurance Evidence-based clinical protocol implementation Post-training participant observations Procurement and logistics Other (please specify) Q7. Under SIFPO, has the quality, accessibility, and dissemination of training tools improved? YES NO Please give specific examples Quality Accessibility Dissemination Q8. Under SIFPO, has frequency of training improved? Yes No Unchanged Please give specific examples. 68 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Q9. Have the updates and the revision of tools been ongoing and improved? Yes No Please Give Specific Examples Q10. What additional support and strengthening is needed for the improved sustainability of SIFPO activities at the country level? Please give specific examples. Q11. In your country programme, what kind of approaches best achieves sustainable programming? Please choose all that apply Best approach for sustainable programming Please expand upon your reasoning. How has SIFPO strengthened this approach? Yes No Mobile Outreach Strategies Social Franchising Public Sector on-the-job training Other approaches￾please specify Q12. Please identify the best strategy to achieve sustainable programming in your country. Describe in detail any organizational and personnel challenges and how these were overcome. How did SIFPO help overcome these challenges? Please include details of cost-share between MSI’s leveraged and own funding? Q13. Has SIFPO improved quality assurance? Yes No SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 69 Q14. If SIFPO has improved quality assurance, how was this achieved? If quality has not improved, why was this the case? How can quality assurance be improved further? Q15. Have MSI’s internal quality assurance standards been disseminated to strengthen FP service delivery and performance in your country? Yes No Q16. Please elaborate as to the effect of the dissemination of quality assurance standards and any changes brought about because of them. Q17. How effective is MSI’s portfolio of service delivery activities in meeting the needs of the Ministry of Health? Very Effective Effective Minimally Effective Please expand upon your response Q18. What evidence exists that the Ministry of Health has buy-in, support, and investment in the MSI/SIFPO’s method of service delivery? Q19. How effective is MSI’s portfolio of service delivery activities at meeting the needs of local stakeholders including community-based organizations? Very Effective Effective Minimally Effective Please expand upon your response Q20. What evidence exists that local community-based organizations have found the MSI project’s method of service delivery effective? Please give specific examples. Q21. If there were a follow-on to SIFPO, in your view, what programmatic gaps and future technical directions/issues would need to be addressed? Q22. What aspects of organizational strengthening (for example, with regard to training, quality assurance, and M&E) provided by SIFPO to MSI and its local partners are no longer needed? Why are these aspects of organizational strengthening no longer needed? Q23. Additional comments, remarks, and recommendations regarding SIFPO related activities 70 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION SUPPORT FOR INTERNATIONAL FAMILY PLANING ORGANISATIONS (SIFPO) MIDTERM PROJECT EVALUATION - MARIE STOPES INTERNATIONAL You are being contacted in order to answer some brief questions which will assist with the midterm evaluation of the MSI-SIFPO program and help to build additional knowledge about family planning services. The overall purpose of the evaluation is:  To assess the MSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement  To gather information that will help to improve the management of the MSI project for the remainder of its implementation  To gather information that will result in useful recommendations for a potential future project All questions posed in the questionnaire will assess the following results categories. Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level It would be helpful if your answers could reflect the overall expected program results. It is kindly requested that you answer all of the questions proposed completely providing sufficient detail for analysis. This questionnaire is for evaluation purposes and was commissioned by USAID, therefore review by an institutional review board was not required. Results of the questionnaires will be shared and will be presented in a format that protects the specific identity of respondents while providing country-specific information. Responding to the survey is voluntary and implies consent. The questionnaire will take about 20 minutes to complete. We are asking that you complete the question survey in its entirety. Please respond to the following survey monkey link or open attachment to begin the questionnaire. If you choose respond via hard copy, 1) save document with your country name, 2) respond to questionnaire, and 3) resave document and forward to sarah@sarahcastle.co.uk. and phardtmancnm@gmail.com. If you wish to have a more detailed discussion with us via telephone or Skype, please provide contact details and indicate this on your returned questionnaire. Thank you for your assistance with this evaluation. Regards, SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 71 Sarah Castle, PhD, and Pandora Hardtman, RN,CNM, DNP QUESTIONNAIRE FOR USAID MISSIONS IN SIFPO COUNTRIES Name of respondent…………………………………………..……….. Email of respondent……………………………………………………. Telephone number of respondent……………………………….……. Position of respondent………………………………………………… Please note that the above details will be kept confidential. Country Table Please tick Cambodia Ethiopia Ghana Madagascar Pakistan UK USA South Sudan Tanzania Zimbabwe Please indicate which organization you represent: USAID Mission MSI Country Office ------------------------------------------------------------------------------------------------------------------------- 72 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Q1. How satisfied is your mission with the work done by SIFPO in the following areas? Very Satisfied Satisfied Not Satisfied Please expand upon your response Mobile Clinic Outreach Social Franchising FP Vouchers Youth FP/HIV Integration Gender-based Violence Q2. What evidence exists that core resources invested in MSI and local partner(s) organizational strengthening have improved country-level platforms and programming? Q3. Do you feel that additional support and strengthening is needed for the improved sustainability of SIFPO activities at the country level ? Yes No Q4. If yes, why? And for which activities? If no, why not? Please give specific examples Q5. In your country, how effective is MSI’s portfolio of service delivery activities in meeting the needs of the Ministry of Health? Very Effective Effective Minimally Effective Please expand upon your response Q6. What evidence exists of long-term Ministry of Health buy-in, support, and investment in the MSI/SIFPO project’s method of service delivery? Please specify. Q7. How effective is MSI’s portfolio of service delivery activities in meeting the needs of local stakeholders, including community-based organizations? Very Effective Effective Minimally Effective Not Effective SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 73 Please give specific examples in relation to your response above. Q8. If there were a follow-on to SIFPO, in your view, what programmatic gaps and future technical directions/issues would need to be addressed? Q9. What aspects of organizational strengthening (for example, with regard to training, quality assurance, and M&E) provided by SIFPO to MSI and its local partners are no longer needed? Why is the organizational strengthening no longer needed? Q10. Please give a detailed example of a successful programmatic/service delivery aspect of SIFPO or of lessons learned in your country. Q11. Additional comments, remarks, suggestions, and recommendations 74 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION QUESTION GUIDELINES FOR PARTNERS International and national representatives of technical partners - Population Council, EngenderHealth, ICRW, and HIV/AIDS Alliance, as well as DfID You are being contacted in order to assist with the midterm evaluation of the MSI-SIFPO programming and build additional knowledge about family planning services. The overall purpose of the evaluation is to:  Assess the MSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement  Gather information that will help to improve the management of the MSI project for the remainder of its implementation  Gather information that will result in useful recommendations for a potential future project All questions posed about your technical collaboration/role as co-funder with SIFPO are open￾ended and assess the following results subcategories: Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level Your participation in the evaluation is being solicited via direct observation of clinical services or interview. Results of the interview will be shared and will be presented only in aggregate form, thereby protecting the identity of respondents. Observations of clinical service delivery will in no way impact any work-related performance appraisals. Data will be de-identified and analyzed in the aggregate to assure confidentiality and maintain anonymity of those responding. Responding to the questions is voluntary and implies consent. Please fill in the attached set of questions and give as much evidence and information as possible to back up your statements. The phrases and questions in italics are intended to serve as guidelines for your responses. Please kindly return the form to sarah@sarahcastle.co.uk by xx/xx/xx (date). Please do not hesitate to contact us for further clarification if you have questions or if you would like to set up a Skype or telephone conversation to discuss SIFPO further. Thank you for your assistance with this evaluation. Regards, Sarah Castle, PhD, and Pandora Hardtman, RN,CNM,DNP SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 75 QUESTIONS FOR MSI PARTNERS AND DONORS (EMAIL QUESTIONNAIRE ) 1.Please give an overview of your involvement with SIFPO? (Did the program meet your expectations? What was the aim of your organisation’s involvement with SIFPO? Did you achieve this aim? Did the nature or degree of your organisation’s involvement with SIFPO change over time? If so, why? 2.How has your organisation helped MSI at an institutional level through SIFPO support? (Please outline the nature of any central capacity strengthening and evidence for its effectiveness? 3.How has your organisation helped MSI country-level programmes through your input into SIFPO-funded activities? (Please specify any activities that have been initiated, developed, or reoriented due to your collaboration with MSI via SIFPO) 4.Has your involvement in SIFPO resulted any changes in the way your own organisation operates or thinks? (Has the collaboration with SIFPO led to an increased awareness about family planning or service delivery? Will these new perspectives alter the way your own organisation operates in the future?) 5.Were there any difficulties or barriers with regard to your partnering/funding SIFPO? (Please describe any logistical, programmatic, or financial barriers? Was communication with MSI and USAID conducive to optimal collaboration?) 6.If SIFPO were to be replicated in the future, what recommendations would you make from the point of view of a partner/funder? (What could be done differently? What could be changed or dropped?) Donors only: 1.What are the advantages of SIFPO as a funding mechanism for your programme? (Please discuss any advantages that pertain to the aim and scope of SIFPO, financial deadlines, reporting procedures, policy and legislative issues, etc.) 76 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 2.What are the disadvantages of SFPO as a funding mechanism for your programme? (Please discuss any advantages that pertain to the aim and scope of SIFPO, financial deadlines, reporting procedures, policy and legislative issues, etc.) 3. What “added value” did SIFPO bring to your existing programmes via the co-funding mechanism? SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 77 Dear Clinical Service Provider: You are being contacted in order to assist with the accurate midterm evaluation of the MSI￾SIFPO programming and build additional knowledge about family planning services. The overall purpose of the evaluation is to:  Assess the MSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement  Gather information that will help to improve the management of the MSI project for the remainder of its implementation  Gather information that will result in useful recommendations for a potential future project All questions posed or service delivery procedures observed will assess the following results subcategories: Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level Your participation in the evaluation is being solicited via direct observation of clinical services or interview. The interview or observation is for evaluation purposes and was commissioned by USAID, therefore review by an institutional review board was not required. Results of the interview will be shared and will be presented only in aggregate form, thereby protecting the identity of respondents. Observations of clinical service delivery will in no way impact any work-related performance appraisals. Data will be de-identified and analyzed in the aggregate to assure confidentiality and maintain anonymity of those responding. Responding to the questions is voluntary and implies consent. The observation or interview process will take approximately 60 minutes of your time. Thank you for your assistance with this evaluation. Regards, Pandora Hardtman, RN, CNM, DNP 78 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Clinical Service Delivery Based Potential Guided Questions 1.How long have you been with MSI as a clinical service provider? 2.What is your title within the organization? 3.*Have you undergone in-service training in the last six months? On what topics? What is your experience with the quality of the training received? Has any of this training been received or provided in conjunction with MOH employees? 4.In your opinion, where does the MIS-FP program need to improve service delivery? 5.What are the priority clinical service delivery needs? (i.e., skills, staff, patient related) 6.What are your thoughts about client load and the ability to thoroughly counsel your clients? 7.What are the facilitators to clinical service implementation? 8.What are the barriers to clinical service program implementation? 9.+Tell me more about the relationship of MSI with the MOH? Other collaborating agencies? 10.What would you change/do differently regarding clinical service delivery mechanisms? 11.+In brief, tell me basics about what you know about USAID FP legislation? 12.Do you think SIFPO funding has improved clinical service delivery? 13.What are the lessons learned from SIFPO and recommendations for the future? SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 79 14.+*Are all methods of FP readily available on client request? Including equipment and supplies? (most of the time, often, not at all) 15.+*Have you experienced stock-outs in the last 6 months? How many? 1-3 4-6 7-10 16.*Adapted from USAID family planning sustainability checklist 17.+ Adapted HIP Physician/Nurse/Midwife-specific Guided Open-ended Questions *Tell me about your experience with FP task shifting and supervision. +Are you able to practice full-scope client care in the context of the cultural and legal regulatory framework in-country? 80 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION +Client-specific questions 1.+Were you previously on a method of FP prior to your visit to MSI? 2.+Had you chosen a method prior to your current visit? 3.+Did you receive counseling or information on the following? Yes No Do Not Recall Additional Comments OCP LARC/LAPM BARRIER METHODS HIV TESTING/COUNSELING GBV Side effects of chosen method Informed consent for method chosen 4. +Do you think that all FP methods are accessible and affordable thru MSI-SIFPO? SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 81 Service Delivery Observational Checklist Observed=Yes Not Observed=No Infection Control Case 1 Case 2 Case 3 Case 4 Case 5 Date Site Date Site Date Site Date Site Date Site Bio–hazard disposal procedures Hand washing Sterile techniques or clean techniques (as applicable) Personal protective equipment (PPE) Fixed or Mobile Clinic Setting/Infrastructure Observed=Yes Not Observed=No Case 1 Case 2 Case 3 Case 4 Case 5 Date Site Date Site Date Site Date Site Date Site *Clinical guidelines/reference materials available, including MSI, organizational, and MOH *Job aids/algorithms available for reference and client management, i.e., contraceptive eligibility wheels Written or posted information available on GBV Written or posted information on HIV Written or posted information on FP method—Mix 82 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION Clinical Service Delivery and Family Planning Counseling Checklist Observed=Yes Not Observed=No Case 1 Case 2 Case 3 Case 4 Case 5 Date: Site Date: Site Date: Site Date: Site Date: Site Appropriate introductions/respectful care Counseling at appropriate time Privacy/confidentiality ensured Communication skills effective Patient-specific data collection –demographic, medical eligibility Contraceptive history assessment Contraceptive method Information given re full method mix Assessment of mitigating factors, i.e., religion, social, fears concerns Screening for co-morbid conditions, i.e., risk for GBV, HIV, and ability to offer first-line support as indicated Non-coercive assistance with method choice Referral for medical examination if indicated Side effects management and follow-up Functional referral system for intersectoral collaboration (CBO, NGO, MOH) Integrated services SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 83 QUESTIONNAIRE FOR THE COMMUNITY HEALTH EDUCATORS (CHES) Informal In-depth Interview Name Location Age Sex Educational level Duration they have been a CHE 1.0Background 1.1What motivated you to become a CHE? How were you chosen? 2.0 Training 2.1Please describe the training you received. (when did it happen, duration, topics, numbers of trainees, role play, practical experience) 2.2What was the best bit of the training? 2.3What did you learn that was new? 2.4Which part of the training was most useful in your work as a CHE? 2.5Which part was least useful? 2.6What was not included in the training that you would have liked to see included? 2.7Have you had any opportunities for retraining (in-service training)? If so, please describe (Have you requested any?). 2.8What topics would you like some more training about? 3.0Work as a CHE 3.1Please describe your average day as a CHE. 3.2What are the enjoyable parts of your work? 3.3What are the difficult parts of your work? 3.4How do you carry out your sensitization? (Where do you meet people, what materials do you have?) 3.5How are you received in the communities? (most receptive and most non-receptive community members?) 3.6Who are the groups you sensitize? (married women, unmarried women, youth, men) 3.7Are there groups you would like to sensitize but currently do not reach? 3.8When you do sensitizations, do you ever talk about HIV (why or why not, how, what do you do if someone needs HIV testing?) 84 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 3.9Do you ever talk about gender-based violence? (What do you do if you learn that someone has experienced GBV?) 3.10 Has SIFPO (USAID) changed anything about the way that you work? 4.0Sensitization around family planning 4.1How do you assess a woman’s family planning needs? 4.2Please describe exactly how you present family planning to women (for spacing, stopping, health, better employment, and education opportunities). 4.3Which methods do you present first? Which do you present last? 4.4Do you have all methods available to show her? 4.5Are there any methods you think are better than others? (for older women, for younger women) 4.6Do women come with ideas what methods they want to use? (What is their choice based upon?) 4.7How do you present the voucher system? 4.8What are the advantages of the voucher system? 4.9What are the disadvantages of the voucher system? 4.10 How could the voucher system be improved? 4.11 What is your relationship like with the BlueStar provider you work with? 4.12 How could this relationship be improved? 5.0Remuneration 5.1Please tell me how much you earn each month and where this money comes from? (salary, incentives, etc.) 5.2Please tell me how the voucher system is linked to your monthly income. 5.3What other financial or material benefits do you receive from your work as a CHE? 5.4Has SIFPO (USAID) changed anything about the way you are paid? 6.0Comment and recommendations 6.1About BlueStar 6.2About family planning methods and service delivery 6.3About vouchers 6.4About future directions SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 85 QUESTIONNAIRE FOR VOUCHER/OUTREACH CLIENTS ABOUT FP CONSULTATION/SENSITIZATION Informal In-depth Interview Name Location Age Educational level Parity 1.0Sensitization around family planning 1.1How did you first hear about family planning? 1.2How did you first hear of BlueStar/MSM outreach? 1.3What are the benefits of family planning (for spacing, stopping, health, better employment, and education opportunities) 1.4What are the disadvantages of family planning? 1.5Does your husband/partner support your family planning? 1.6Does anyone give you different (conflicting) advice about family planning? (compared with that given by the CHE/outreach worker ) 2.0Sensitization by CHE or MSM outreach worker 2.1Before the sensitization session, had you already used family planning? 2.2When you came into the sensitization session, was there a particular method you had in mind? 2.3Did you end up using this method? (If so, why; if not, why not?) 2.4 Please describe the sensitization session. (In group? Singly? How could it have been approved?) 2.5 Which methods were presented to you first? Which were presented to you last? 2.6 During the sensitization, did you learn about some methods you had never heard of before? 2.7 Did they CHE/outreach worker have all the methods to show you? 2.8 Did she discuss some methods in more detail than others? 2.9 Did she recommend a specific method to you? (Which one? Why this method?) 2.10 Did you have any questions during the consultation? (Was she able to answer them?) 2.11 Do you think you will return to the CHE/outreach worker? (why/why not?) 2.12 How long do you think you will use your current method of contraception? 86 SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS: MSI MIDTERM PROJECT EVALUATION 2.13 Why will you stop/switch? 2.14 How could the CHE/outreach worker sensitization be improved? For voucher users only: 2.15 How did the CHE present the voucher system to you? 2.16 What are the advantages of the voucher system? 2.17 What are the disadvantages of the voucher system? 2.18 Did the voucher system influence your choice of method? (Would you have chosen another method if there had not been a voucher?) 2.19 How could the voucher system be improved? 2.20 What is your relationship like with the BlueStar provider? 2.21How could this relationship be improved? 3.0Comment and recommendations 3.1About BlueStar 3.2About family planning methods and service delivery 3.3About vouchers 3.4About future directions ANNEX VII. DISCLOSURE OF ANY CONFLICTS OF INTEREST For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge III Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com