SUPPORT FOR INTERNATIONAL FAMILY PLANNING ORGANIZATIONS POPULATION SERVICES INTERNATIONAL MIDTERM PROJECT EVALUATION March 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Sarah Castle, PhD, and Pandora Hardtman, RN, CNM, DNP Cover Photo by Pandora Hardtman RN, CNM EVALUATION Support for International Family Planning Organizations Population Services International Midterm Project Evaluation March 2014 Global Health Technical Assistance Bridge 4 Project (GH Tech) USAID Contract No. AID-OAA-C-13-00113 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. GH Tech Bridge 4 Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Engility-IRG to the United States Agency for International Development under USAID Contract No. AID-OAA-C-13-00113. ACKNOWLEDGMENTS The authors would like to thank USAID and PSI staff in Washington, Guatemala and Benin who were most helpful and collaborative. Our thanks also go to representatives from SIFPO partner organizations, the numerous project beneficiaries and field staff who gave us their invaluable opinions about the project. PSI-SIFPO: MIDTERM PROJECT EVALUATION i CONTENTS Acronyms ........................................................................................................................................................................... iii Executive Summary .......................................................................................................................................................... v I. Introduction .................................................................................................................................................................... 1 II. Project Background ...................................................................................................................................................... 2 III. Evaluation Methods and Limitations........................................................................................................................ 4 IV. Findings........................................................................................................................................................................... 6 V. Conclusions and Recommendations ...................................................................................................................... 22 Annex I. Scope of Work ................................................................................................................................................. 26 Annex II. Persons Interviewed ..................................................................................................................................... 40 Annex III. Sources of Information ................................................................................................................................43 Annex IV. Data Collection Instruments ..................................................................................................................... 46 Annex V. Disclosure ........................................................................................................................................................ 66 PSI Management Response to the SIFPO Mid-term Evaluation ........................................................................... 68 ii PSI-SIFPO: MIDTERM PROJECT EVALUATION ACRONYMS AE Adverse Event AHME African Health Markets for Equity AIDS Acquired Immune Deficiency syndrome ABMS Association Béninoise pour le Marketing Social BSE Breast Self-Exam CO Clinical Officer CHE Community Health Educator CHW Community Health Worker CLIC Client Information Center COI Conflict of Interest CPR Contraceptive Prevalence Rate CYP Couple Year of Protection DfID Department for International Development DHIS2 District Health Information System DHS Demographic Health Survey FP Family Planning FP/RH Family Planning/Reproductive Health FP2020 Family Planning 2020 GBV Gender-Based Violence GHI Global Health Initiative HIS Health Information System HIV Human Immunodeficiency Virus ICRW International Center for Research on Women IPPF International Planned Parenthood Federation IUD Intrauterine Device LARC Long-acting and Reversible Contraceptive LAPM Long-acting and Permanent Method M-Health Mobile Health MARPs Most at-Risk Populations M&E Monitoring and Evaluation PSI-SIFPO: MIDTERM PROJECT EVALUATION iii MDG Millennium Development Goal MOH Ministry of Health NGO Non-Governmental Organization OCPs Oral Contraceptive Pills PASMO Pan-American Social Marketing Organization PMP Project Management Plan PRH Population and Reproductive Health PSI Population Services International PSZ Population Services Zimbabwe QA Quality Audit QTA Quality tool assessment RME Research, Monitoring, and Evaluation SHOPS Strengthening Health Outcomes through the Private Sector SIFPO Support for International Family Planning Organizations SPIRES Stanford Program for International Reproductive Education and Services SOW Scope of work SRH Sexual and Reproductive Health STI Sexually Transmitted Infection TL Tubal Ligation USAID United States Agency for International Development VCT Voluntary Counseling and Testing WHO World Health Organization iv PSI-SIFPO: MIDTERM PROJECT EVALUATION EXECUTIVE SUMMARY The Support for International Family Planning Organizations (SIFPO) project has a period of performance from September 30, 2010, until September 29, 2015, and funds Cooperative Agreements with Population Services International (PSI) and Marie Stopes International.This evaluation pertains to PSI’s performance only. PSI collaborates with IntraHealth for clinical guidelines, gender and youth work and with The Stanford Program for International Reproductive Education and Services (SPIRES) for quality assurance.. SIFPO’s mission is to increase the use of family planning (FP) services globally through strengthening selected international FP organizations in order to achieve maximum program impact and synergies.The current SIFPO projects are working toward the following four results areas: • Result 1: Strengthened organizational capacity to deliver quality FP services to target groups. • Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level. • Result 3: Increased organizational sustainability of country-level programs, including internal south-to-south support and technical assistance. • Result 4: Gender-sensitive FP services targeting youth strengthened at a global level. A special focus of the SIFPO project has been on capacity building and systems strengthening to build upon and leverage the organization’s extensive network of country programs.The capacity-building areas include improving clinical and counseling quality, standardizing and sharing best practices (for example, in social franchising or mobile outreach programs), improving and standardizing metrics, and improving health management and information systems while strengthening in-country leadership and management skills. The evaluation’s overarching questions seek to ascertain how satisfied various stakeholders, including USAID missions, have been with the work done by SIFPO and with assistance from the program.The questions also seek to identify what existing gaps and future technical directions or issues need to be ad￾dressed in any follow-up.The technical evaluation questions focus on the use of core resources by each recipient organization to strengthen country-level platforms and their effect on improved organizational capacity, performance, and management.The report also addresses the additional support and strength￾ening needed for improved sustainability at the country level. Lastly, the evaluation seeks to identify the areas for improvement or strengthening of family planning/reproductive health (FP/RH) service delivery and quality assurance. The questions comprise, but are not limited to: What evidence exists that core resources invested in organizational strengthening have improved country-level platforms and programming? What has been the effect on organizational capacity? Is there evidence that there is increased sustainability? How have management practices within the organization been affected? Is there evidence that the internal qual￾ity assurance standards have been disseminated to strengthen FP service delivery and performance at a global level? To what extent is the PSI project’s portfolio of service delivery activities meeting the needs of stakeholders? The midterm evaluation of PSI-SIFPO took place between December 2013 and February 2014.The lead evaluators comprised a demographer/public health monitoring and evaluation (M&E) specialist and a doctoral-level certified nurse–midwife with international experience in training, professional standards, and capacity building.The methodology involved “triangulating” different research methods, including document reviews and in-depth interviews with USAID/Washington and mission staff and the interna￾tional project partners. All USAID missions and PSI country offices in seven existing PSI-SIFPO coun￾PSI-SIFPO: MIDTERM PROJECT EVALUATION v tries (Benin, Cambodia, Guatemala, Kenya, Madagascar, Malawi and Pakistan) were sent an online survey asking about their views on what the funding had achieved. In addition, a comprehensive briefing took place with key USAID and PSI personnel in Washington.The evaluators then carried out fieldwork with two PSI-SIFPO country programs in Guatemala and Benin.These sites were chosen because they had a complementary range of services. In both settings, in-depth interviews took place with project staff, proj￾ect beneficiaries, providers, and community health workers as well as with local partners and stakehold￾ers. Clinical checklists were administered and observations of service provision were made in a number of service delivery settings. Overall, PSI has made solid contributions to FP service delivery and supported technical capacity build￾ing with an array of impressive training materials and online resources. Capitalizing on their rich history of social marketing and social franchising,1 PSI has also sought to strengthen public sector delivery in a sustainable manner. Over 3000 new franchisees have been added to PSI’s network during the period of SIFPO funding.This has resulted in a significant increase in the provision of long-acting and reversible contraceptives (LARCs) and permanent methods which provided over 6,500,000 couple years of protec￾tion (CYPs) in 2013 and represents a significant contribution to national modern contraceptive preva￾lence rates (CPR) and expansion of method mix.The work PSI has been able to do through SIFPO has not only allowed the organization to do direct implementation through mission buy-ins but also allowed PSI to develop tools and to strengthen capacity-building activities throughout the organization. However, it is sometimes difficult to say which impacts seen in result areas are due to SIFPO and which are due to other donors. PSI has received obligations for both core and field funding which reaches a combined total of $32.4 mil￾lion of which 73 percent is field funding and 27 percent is core funding In some cases, field buy-ins were pragmatically framed by missions’ need to continue existing funding and fill shortfalls or gaps as well as to address the specific SIFPO result areas. Scopes of work were in line with mission strategic priorities for the countries. Missions indicated a general satisfaction with the SIFPO funding mechanism and viewed PSI as cost-effective. The organization builds upon strengthening a decentralized system and addresses challenges related to the circular flow of information by using information technologies, including creating an online university to increase access to learning materials and exchange (including south-to-south support). PSI has also instigated some clinical and procedural innovations such as the postpartum interuterine contraceptive device (PPIUD) inserter, for which they received seed funding from a Saving Lives at Birth Challenge Fund Grant.2 All clinical guidelines and practice occur within a framework of informed choice. PSI has addressed issues of quality assurance by integrating quality checks into every aspect of its service delivery systems whilst at the same time providing supportive supervision to enhance timely and on-the￾spot learning. Challenges relating to the standardization of quality among large numbers of franchisees (private clinics) are addressed in a manner that is mindful of striving to balance quality and cost-effective￾ness.The “Business in a Box” initiative successfully fuses elements of quality and business training. Never￾theless, some of the franchisees visited during the fieldwork in Benin (which, it should be noted, repre￾sented a small percentage of total franchises) had not reported ‘added value’ of adhering to the franchise system, yet still chose to join it. However, in this setting, franchise outlets were perceived by clients to be of higher quality than public services. 1 Social franchises are networks of private health providers that use commercial franchising methods to achieve health and other social goals. Building upon existing exper tise in poor communities, social 2 Saving Lives at Bir th: Challenge Fund for Development, a joint initiative of USAID, the Government of Norway, the Bill & Melinda Gates Foundation, Grand Challenges Canada, and DFID The PPIUD inser ter is described here http://www.ghspjournal.org/content/1/3/428.full.pdf+html vi PSI-SIFPO: MIDTERM PROJECT EVALUATION PSI is to be commended on its approach to integrated maternal, newborn and child health (MNCH) services. For example, adding FP to immunization not only increases cost-effectiveness, but responds to the cultural need to hide contraceptive use in settings where male disapproval is high and creates conve￾nient access to FP services for clients. Quality audits (QA) and supportive supervision have been increased and enhanced the technical competence of providers in both the public and private sectors. PSI has also worked to regionalize quality assurance support through the creation of a cadre of QA auditors from within its country program network. During a field visit in Benin, it was noted that network providers could continue to focus more on patient privacy and comfort, for example, by minimizing the time a patient is undressed or by providing cover, though it is recognized that these practices are associated with Ministry of Health (MOH) clinical norms in place. It is also recommended that PSI play a stronger role in adovcating for system-wide improvements with the MOH in this area and intensify its provider behavior-change work to address remaining challenges. PSI recognizes the importance of task shifting and realizes the need to more proactively encourage countries to apply appropriate approaches to service delivery models. The Reproductive Health and Social Franchise Leadership Trainings effectively served to transfer knowl￾edge, increase skills, and enhance local capacity and suitability. Follow-up to training occurs via the various online media which enables PSI in-country staff to promote south-to-south learning quickly and effectively. IntraHealth has also provided guidance on several tools including research considerations for gender-based violence (GBV) and their contribution should be acknowledged accordingly. With regard to research, PSI has standardized indicators across programs and carried out cutting-edge work on measures of equity that will take the field forward.The wealth index will be incorporated into client exit interviews measuring satisfaction, perceived quality and other indictors, and PSI’s TRaC surveys (a population-based cross-sectional survey mechanism). In the future, it is hoped that PSI will monitor the economic profiles of franchising clients and public sector clients where appropriate to help individual platforms assess whether they are reaching the intended market segment. In Benin, social franchise services were clearly appealing to a more middle class population while outreach workers said that 7 out of 10 women they encountered could not afford the price of PSI’s socially marketed products (‘Laafia’) nor those made available through the franchises. PSI is piloting a new client record monitoring system based on the DHIS2 HIS.This system is used in the public sector in several countries, and use of the same system can improve the opportunities for collaboration with the public sector, and lessen the time needed for routine reporting. It was noted in Guatemala, in the three public health clinics visited in San Marcos department, that investments had been made in improving the available equipment and surface infrastructure for local government health ser￾vices which subsequently received extremely small numbers of clients for LARCs – averaging, less than two per month in the newly refurbished clinics. However, LARCs were previously unavailable in these clinics so this is a new service that is being offered, and has broadened the method mix. By contrast, in the Quiche department, the annual number of LARC clients in 12 out of 13 clinics increased signfi cantly within the context of informed choice, as LARCs had never been offered routinely in these clinics. For example, in Sacapulas, the number of voluntary LARCs provided increased by 150 percent and in San Miguel Uspatan, voluntary LARC provision increased by 2,450 percent from 2012 to 2013.3 On-going cost-effectiveness monitoring combined with analysis of individual-level records would be helpful in as￾sessing the cost of method delivery by channel and to enable the better matching of demand and re￾sources. It is acknowledged however, that within USAID, many missions and some PRH staff are rightly concerned with health systems strengthening of public sector services which will generally not be as cost-effective or have as much productivity as other channels such as mobile outreach. Low uptake at the beginning of investments in such services, especially those delivering LARCS for the first time, is to be expected in under-served settings such as those visited. 3 This finding is based on ser vice statistics and does not reflect or represent any target or quota setting by clinics or providers. PSI-SIFPO: MIDTERM PROJECT EVALUATION vii The research carried out under SIFPO has been substantial, but is constrained by a lack of dedicated full-time human resources and the poor communication of findings.The current allocation of time for the Research Advisor is insufficient.Within the DC office, more staff should be allocated to research to build upon the obvious good capacity of local researchers at the country level and to enable a broader dissemination of findings. In this way, PSI can test innovations, scale up and provide inter-country com￾parisons and share knowledge promoting best practice more widely. PSI should also capitalize on global networks facilitated by SIFPO to further disseminate fi ndings. PSI is mindful of sustainability issues and seeks to build long-term impact and capacity within health systems in a number of practical ways, such as providing south-to-south technical assistance; increasingly infl uencing FP policy discussion; strengthening the private sector and creating public-private partnerships. In addition, PSI has strongly focused on successfully leveraging additional funds through continued diversifi cation, includ￾ing obtaining bilateral agreements and instigating cost-sharing with other donors.The clearly productive relationships within the MOHs, resulting, for example in Guatemala, in the support of disseminating laws on reproductive rights to disadvantaged populations in the Western Highlands, and an increasing investment in commodities, will enable PSI to help local stakeholders attain and maintain the goals of FP2020 and beyond. However, care should be taken not to induce a dependence on PSI’s technical assistance which seemed to be the case in Guatemala. In addition, attention may be needed to ensure the real engagement and ongoing technical learning among MOH personnel who have been trained by PSI in the insertion and removal of LARCs. Observations and interviews in Benin found the government providers to be used as administra￾tors rather than as service providers during clinical consultations. However, it is recognized that this obser￾vation is not generalizable and that the service delivery model may differ by country. For example, in Benin, PSI is primarily providing services rather than technical assistance. In addition, within-country variation may exist depending on the time allocation for government staff to assist with PSI service delivery. The youth initiatives undertaken during SIFPO have mainly focused upon building the capacity of health care providers within social franchise networks to deliver youth-friendly health services.The Youth Friendly Services Initiative addresses an area that is complex and challenging but requires more atten￾tion to be given to improving the engagement of young people at the community level. Under SIFPO, PSI involved youth in the training of health care providers and in the design of youth-friendly services in Liberia and Malawi. As a next step, PSI should ensure further involvement of youth in the implementa￾tion, monitoring, and evaluation of FP services. In Guatemala, the “Plan de Vida” or Life Plan sensitization sessions were helpful in situating FP in a rights-based longer-term context. Progress has been made in gender now that PSI has a gender staff focal point. Gender-awareness activities, especially those pertaining to GBV and constructive male engagement have been strengthened with the aid of IntraHealth. Initiatives such as the no-scalpel vasectomy (NSV) assess￾ment in Benin have sought to resonate culturally with gender norms affecting potential clients. Gender issues relating to clients’ perceptions of the relative benefit of male or female providers may require further attention if, as recommended, task-shifting is to become a priority. Detailed recommendations are provided and include the following: • Instigate real mobile services to reach women in remoter areas as opposed to investing in ‘in-reach’. • Integrate and monitor respectful care practices gender awareness for all elements of patient care and FP service delivery. • Better price-product match and/or integration with income generation mechanisms for poor women. • Improve data for decision making and micro-level analysis of client flow, client profiles, and method mix and instigate the use of routinely collected data for improved service delivery and targeting. viii PSI-SIFPO: MIDTERM PROJECT EVALUATION I. INTRODUCTION The SIFPO Cooperative Agreement with PSI was made in September 2010 and will continue to Sep￾tember 2015.The report presented here therefore comprises a mid-term evaluation of multi-country programs and was commissioned by USAID.The aim of the evaluation is to examine the impact of SIFPO funding on system strengthening and capacity building within PSI. It is also to assess PSI’s quality assur￾ance standards and the organization’s gender and youth-sensitive services. It is emphasized here that this is not an evaluation of PSI’s impact on FP service delivery nor of FP uptake resulting from activities associated with the Cooperative Agreement. Neither is it an evaluation of the country programs in Benin and Guatemala where the fieldwork was carried out. It is an examination of PSI’s internal systems and structures and the way that SIFPO has impacted upon them. It also examines PSI relations with SIFPO partners and the joint activities they have undertaken. The audience for this report comprises PSI’s staff in its headquarters and in-country offices as well as USAID staff with an interest and expertise in population and reproductive health. It will also be of inter￾est to PSI’s SIFPO partners.The findings will inform decision making with regard to future ways USAID may choose to partner with and support PSI and other large international service delivery organizations. EVALUATION QUESTIONS The overarching questions framing the evaluation seek to ascertain how satisfied various stakeholders, including USAID missions, have been with the work done by and assistance from SIFPO. The technical evaluation questions focus on the use of core resources for each recipient organization to strengthen country-level platforms, and their effect on improved organizational capacity, sustainability, and performance and management.They also address the additional support and strengthening that are needed for improved sustainability at the country level. Lastly, they seek to identify the areas for improv￾ing or strengthening FP/RH service delivery and quality assurance. The questions comprise but are not limited to: • What evidence exists that core resources invested in organizational strengthening have improved country-level platforms and programming? • What has been the effect on organizational capacity? • Is there evidence that there is increased sustainability? • How have management practices within the organization been affected? • Is there evidence that the internal quality assurance standards have been disseminated to strengthen FP service delivery and performance at a global level? • To what extent is the PSI project’s portfolio of service delivery activities meeting the needs of stakeholders? PSI-SIFPO: MIDTERM PROJECT EVALUATION 1 II. PROJECT BACKGROUND The United States Agency for International Development (USAID) awarded Population Services Inter￾national (PSI) a Cooperative Agreement as part of the Support for International Family Planning Orga￾nizations (SIFPO) project.The period of performance is from September 30, 2010 until September 30, 2015. PSI implemented their activities with two international partners namely IntraHealth International (referred to here as IntraHealth) and Stanford Program for International Reproductive Education and Services (SPIRES). The SIFPO-PSI project has a ceiling of just under $40 million. As of February 2014, over $32.4 million has been obligated and just over $18 million expended, of which approximately two-thirds is field funding and one-third is core funding. A parallel SIFPO award was made by USAID to Marie Stopes International but is not discussed in detail here. The SIFPO projects’ mission is to increase the use of family planning (FP) services globally through strengthening selected international FP organizations which have a global reach and an extensive, multi￾country network of FP clinics, in order to achieve maximum program impact and synergies.The current project is working toward the following four results areas: Result 1: Strengthened organizational capacity to deliver quality FPservices to target groups. Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level. Result 3: Increased organizational sustainability of country-level programs, including internal south-to￾south support and technical assistance. Result 4: Gender-sensitive FP services targeting youth strengthened at a global level. A special focus of the SIFPO projects has been on capacity building and systems strengthening to build upon and leverage PSI’s extensive network of country platforms of programming and social franchise ser￾vice delivery channels.These capacity-building areas include improving clinical quality oversight; standard￾izing and sharing best practices, such as around social franchising ; improving and standardizing metrics, such as for equity; improving internal health management and information systems; and creating a cadre of technical reproductive health leaders within the organization’s country platforms. Additional sub-indi￾cators are linked to each of the four results described above but are not presented in detail here. The overarching goal is that by strengthening and streamlining procedures and systems of international FP organizations, these improvements will cascade down to local affiliates in developing countries, thereby strengthening and enhancing sustainability of private sector partners to contribute to the overall health system. The work described here is a midterm evaluation, and as such does not have a conceptual framework or log frame. However, PSI does have a Performance Management Plan (PMP) with indicators, a selection of which are referred to in this report. All SIFPO project activities are organized into two categories: core-funded and field-funded activities. 2 PSI-SIFPO: MIDTERM PROJECT EVALUATION Figure 1:Year Three: July 2012 - June 2013 Result 4 16% Result 1 31% Result 3 29% Result 2 24% Figure 1 above shows that from July 2012 - June 2013, around one-third of core funding went into strengthening organizational capacity and just less than one third on increasing organi￾zational stability. Around one-quarter went on quality assurance standards and the rest on gender and youth sensitive services. SIFPO-funded activities are aligned with US￾AID’s Policy on Youth (2012) which seeks to 1) strengthen youth programing, participation, and partnership in support of Agency development objectives and 2) mainstream and integrate youth issues and engage young people across Agency initiatives and operations. PSI-SIFPO activities are also aligned with USAID’s policy on ‘Gender Equality and Female Empowerment’ (2012) which seeks to reduce gender disparities and gender￾based violence as well as increasing women’s capacity to realize their rights.The project also takes into account the FP 2020 Summit follow-up actions which seek to reduce unmet need and enable 120 million women and girls to use contraceptives by 2020. It also responds to the new USAID initiative to address preventable maternal and child deaths. PSI-SIFPO: MIDTERM PROJECT EVALUATION 3 III. EVALUATION METHODS AND LIMITATIONS METHODOLOGY The evaluation used a number of both quantitative and qualitative methods to ‘triangulate’ evidence for greater validity. Each evaluator brought a complementary skills set to the evaluation. Sarah Castle has a background in demography, epidemiology, and program evaluation and focused on the project’s tools and information systems as well as the perspectives of clients and of non-clinical field staff and volunteers. Pandora Hardtman is an experienced Nurse-Midwife and focused on clinical service provision, training, and quality assessments. Document review: The team reviewed extensive background documentation pertaining to the project.This included the SIFPO project agreements, the PMP, semi-annual and annual reports, country reports and work-plans, country-level and global curricula, training materials, and monitoring and evalua￾tion tools.The review also included USAID technical reports, compliance training materials, and materials relating to the Agency’s youth and gender strategies as well as to the GHI and other initiatives. Development of work plan and research instruments: The team developed a work plan and research instruments which were submitted to USAID for approval. In addition, using ‘Survey Monkey’, a questionnaire was developed for USAID missions with country-level ‘buy-in’ to SIFPO and a second was used for PSI country directors in SIFPO countries. An additional questionnaire was developed for PSI’s SIFPO partners. Briefings and interviews with USAID and PSI in Washington: The evaluation team met with senior USAID staff in Washington including the SIFPO Agreement Officer’s Representative (AOR) and SIFPO-PSI Technical advisor.They also received a very extensive briefing from PSI-Washington including exchanges with the CEO; Director and Deputy Directors of SIFPO; the Global Medical Director;Techni￾cal Advisors; and M & E,Youth, and Gender specialists. Field visits to Guatemala and Benin: Benin and Guatemala were chosen as field visit sites because they represented complementary aspects of PSI’s SIFPO-funded programming and thus would enable the team to get a complete picture of SIFPO activities. Both PSI country programs operate through local affiliates – the Pan-American Social Marketing Organization (PASMO) in Guatemala and the Associatin Beninoise pour le Marketing Social (ABMS) in Benin. During the field visits, which comprised four days in Guatemala and three days in Benin, interviews were undertaken with USAID mission staff, both PSI country directors, and senior staff pertaining to service provision, quality assurance, and social franchis￾ing. In Guatemala, the evaluators visited an outreach site at San Marcos, three hours into the Western Highlands, which served rural, indigenous populations.They also visited urban sites in Cotonou and Porto Nuovo in Benin. Interviews took place with PSI providers in public and private sectors, and MOH officials along with local partners and project beneficiaries. 4 PSI-SIFPO: MIDTERM PROJECT EVALUATION Table 1: Interviews and activities undertaken during site visits in Benin and Guatemala Benin Guatemala Clinical checklists formally administered 5 3 Large group Counseling Sessions 1 1 Individual Pre-Counseling- fixed clinic and home visits 6 4 Individual Post Counseling 6 4 FP Method Technical Service Provision 6 3 Social Franchise providers 5 0 Community Health Educators 5 0 Clients individual interviews/assessments 10 5 MOH service providers 2 1 Sub-agreement partners (including local implementing partners/service 0 2 providers) Commodity Distribution Center 1 0 Mobile Clinic Site 1* 1 *Mobile service delivery in Benin is not funded by SIFPO but was visited to provide a holisitc view of other in-country activities many of which may face some of the same challenges as those funded by SIFPO. Table 1 above shows the individuals and organizations interviewed in each country. In-depth interviews (Annex II) in both country settings pertained to service quality, service training, method choice, relationships with MOH and other providers, client follow-up, future directions/needs and motivation/remuneration and client satisfaction (copies of interview guides are available in Annex IV). The interviewees gave informed consent and were left a copy of the consent form in case they subse￾quently had questions.Those who were photographed also signed a form authorizing use of the photo￾graphs. In all clinical settings visited Pandora Hardtman (RN, CNM, DNP) carried out clinical observations and administered a clinical checklist to address the quality of service provision and counseling (see Annex IV). LIMITATIONS The first major limitation of this evaluation was the short time allocated for the country visits given the enormous amount of data that needed to be collected and analyzed. Secondly, PSI-country offi ces chose the field visit sites, which may introduce some bias to the evaluation. It should be noted that there may be a self-selection of both providers and program and services as those agreeing to interview may be more willing to talk positively about the program. Additionally, bias may have been introduced through the use of language interpreters for clients and providers speaking French, Spanish, Mayan, and Fon languages. PSI-SIFPO: MIDTERM PROJECT EVALUATION 5 IV. FINDINGS SATISFACTION OF STAKEHOLDERS WITH SIFPO ASSISTANCE USAID/Washington In general, USAID/Washington appeared to be satisfied with PSI’s performance under SIFPO.Those in￾terviewed felt that PSI had made progress in increasing CYPs, with a large number attributable to LARCs and permanent methods.They saw PSI as an active and valued partner in contributing to USAID’s role in achieving FP2020 goals. In particular, PSI’s ability to deliver a variety of clinical services in one setting was seen as an advantage for increasing access and integration. PSI has had a lot of direct USAID bilateral funding and has existing infrastructure and established coun￾try programs and did not thus need to recuit a lot of new staff. Nevertheless, there were significant con￾cerns at the beginning of the project about PSI’s capacity to manage the Cooperative Agreemeent and implement the activities with limited staff.This may have accounted for the slow start in disbursement, delayed responses, and limited exchanges with USAID with regard to implementation. It was remarked that at first PSI ”was not involving USAID enough in conceptualization –for example, in the reading of early drafts of work plans, of documenting case studies etc.We needed more technical briefings … at the beginning but now it is better“. Most USAID contacts interviewed agreed that these issues have now been resolved with the addition of technical advisors to the core team as the mission buy-ins expanded. PSI has also increased the technical information shared during its bimonthly meetings with the USAID SIFPO AOR team.The SIFPO manager currently works on the project for 95 percent of her time and four other senior staff at 100 percent. Numerous brown bag presentations as well as formal reporting have keep USAID aware of the progress of PSI’s SIFPO-related activities. USAID MISSIONS Responses to the questionnaires sent to the missions indicated a general satisfaction with the SIFPO funding mechanism.The Guatemala Mission saw it as “an extremely impactful, cost-effective intervention that will result in improved health outcomes across several health areas including nutrition and MCH. We are working well with this mechanism”. It was felt that SIFPO had also helped strengthen the pub￾lic sector as “PSI products have contributed to alleviating the contraceptive stock-outs in public health centers”. In Cambodia, SIFPO was seen as an effective way to implement cost-sharing via both bi-laterals to PSI and with other donors, resulting in reinvestment increasing cost-effectiveness. All but one of the missions who replied would recommend SIFPO to other mission as a funding mechanism. In some cases, SIFPO was seen by missions as an easy way to continue the funding of existing projects as well as supporting initiatives that specifically responded to the four SIFPO result areas. In Benin, SIFPO has been used to carry on the work of a previous six-year bilateral grant (IMPACT) and to strengthen social franchise outlets. PROJECT PARTNERS PSI’s main project partners under SIFPO are IntraHealth International and SPIRES.The table below shows the nature of the activities during the first three years of these collaborations. 6 PSI-SIFPO: MIDTERM PROJECT EVALUATION Table 2: Partner’s SIFPO contributions Partner Year 1 Year 2 Year 3 Intra- Health SPIRES Protocols for community￾based oral pills and inject￾able Quality Assurance Audits: Togo, Kenya PSI Gender Training report & resources QA Audits Quality Assurance system development and imple￾mentation; clinical capacity building Permanent methods (NSV and BLT) protocols Coaching/mentoring guidelines Quality Assurance Audits: DRC, Benin Gender and Youth work￾shop for RHL training Gender and Youth integra￾tion into PSI Delta tool No- Scalpel Vasectomy As￾sessment Quality Assurance Auditor Training (Benin) Quality Assurance Audits: Madagascar ,Togo Quality Assurance Auditor Training Youth Friendly services guide & Training Malawi QA Audits Quality Assurance system development and imple￾mentation; clinical capacity building QA Audits Quality Assurance system development and imple￾mentation; clinical capacity building There has been ongoing attendance at technical working groups and various meetings across all part￾ner relationships during the three years of SIFPO funding. PSI has also leveraged other partnerships to further advance the goals of SIFPO. IntraHealth Input provided by Intrahealth related primarily to PSI’s internal tools such as the refinement of its quality assurance manual and private sector site selection tools to include gender considerations. IntraHealth saw the joint activities as a positive experience although they had some concerns about intellectual prop￾erty and dissemination.They remarked that “an important lesson learned for us is around intellectual property agreements for jointly developed products and more discussion with PSI about how to share SIFPO products and lessons learned with partner organizations, including our own“. SPIRES Dr. Paul Blumenthal is PSI’s Global Medical Director and head of SPIRES which provides technical sup￾port for improved access to family planning. Dr. Blumenthal allocates 50 percent of his time to PSI and 50 percent to SPIRES. In collaboration with PSI, although not under SIFPO funding, SPIRES has developed a new post-partum IUD inserter.Through SIFPO, SPIRES and PSI have improved quality assurance and audit tools, for example, including better guidance about eligibility and screening and improved syndromic management of STIs.Through SIFPO, SPIRES has consolidated the quality audit system and made FP qual￾ity assurance an institutionalized programmatic concern within PSI.There are 34 external auditors and 27 regional auditorsthat have been established and strengthened by SPIRES, thus increasing the autonomy of PSI’s regional networks PSI-SIFPO: MIDTERM PROJECT EVALUATION 7 In addition, since July 2012, through the Richard T Clark Fellowship Program, three rounds of Merck Fel￾lows have been affiliated with PSI to primarily enhance the quality and scalability of its franchise operat￾ing model. Specifically, the fellows worked to help establish standards and procedures in areas such as brand management, field support, and supervision together with finance mechanisms.These have been incorporated into the “Business in a Box” initiative that is used by PSI to improve the quality and scalabil￾ity of country operations. RESULT 1: STRENGTHENED ORGANIZATIONAL CAPACITY TO DELIVER QUALITY FP SERVICES TO TARGET GROUPS. Organizational Development PSI’s PMP states that “By the end of the SIFPO project, 90% of top management positions in the PSI plat￾form will be filled by local PSI staff“. At the start of the project, this figure was 47 percent and at the end of year three, 96 percent of PSI’s FP program positions were filled by in-country nationals.This indicates a transfer of capacity and provides a base for sustainable programming. Particularly pertinent to organi￾zational development is the Reproductive Health Leaders Program which takes place over five weeks in Washington DC.The Anglophone training in 2011 was for 16 staff from 14 platforms and a Francophone training in 2012 involved 13 staff from 11 platforms.The Knowledge Transfer Checklist helped PSI and the participants (“Reproductive Health Leaders“) to track knowledge acquisition and application dur￾ing on-going evaluations. Six months after training, 93 percent of participants were better able to solve problems using innovative techniques and creativity. PSI is currently piloting a mentoring program whereby five RH Leaders are matched with five senior RH TAs and over one year they will establish goals and a professional development plan. In all organisational development and capacity strengthening activities, PSI makes full use of online technologies and instigates on-going guidance and feedback, even after the programmes have ended.This is likely to be highly impact￾ful and cost-effective and illustrates innovative approaches to both learning and resource management. Other programs such as the Social Franchise Leaders (SFL) program also have helped to increase or￾ganisational capacity at a country level as well as providing clinical and marketing guidance. As PSI had already had a significant number of bilateral grants with USAID, they were already well informed about issues relating to compliance. However, the importance of compliance was also reinforced under SIFPO via the SFL program. For example, PSI’s partner in Pakistan, Greenstar, reported that “the social franchise leaders program helped key senior staff to learn more about informed choice and compliance with USG regulations. From senior medical staff to interpersonal communications officers, Greenstar staff is now more attuned to the requirement to inform each beneficiary of the range of family planning choices as well as the benefits and side effects of each method.” Strengthening the private sector through social franchising and business training During the period of SIFPO funding, the number of countries with PSI FP social franchisees increased from 15 in 2009 to 23 in June 2013.The total number of franchisees has increased from 11,057 in 2009 to 12,362 in 2011 to 14,211 in 2013. During the same period, the number of LARCs and permanent methods ( PMs) across PSI also increased substantially as shown in Figure 1 below (which is for all types of provider), with many also being provid￾ed in the public sector due to improved and rolled out training by PSI.The figure does not differentiate between LARCs provided under SIFPO and via other funding streams. However, given that the nature of SIFPO is to strengthen organizational support, this is likely to increase health impact both due to direct implementation as well as to organization-wide improvements to systems, the introduction of innovative practices, and the development of staff capacity. 8 PSI-SIFPO: MIDTERM PROJECT EVALUATION Figure 2: LARC and PM CYPs 2009-2013 8 6 4 2 0 VSC IUD Implant In million 2009 2010 2011 2012 2013 All SIFPO country platforms now offer a range of contraceptive methods , including LARCs.This has therefore fulfilled the SIFPO PMP indicator of “offering three or more modern FP methods including LARCs and PMs” which is likely attributable to the franchising sector. However, to date, PSI does not offer disaggregated data of method uptake by type of service delivery modality at a global level, only at a country level. PSI is known as a world leader in social franchising (SF) and has implemented many franchising activi￾ties under SIFPO.These include the first ever PSI global SF strategic planning meeting, providing ongoing global and regional support, conducting a SF leaders’ training, establishing SF centers of excellence and, with PSI, developing a course on SFfor the USAID Global Health E-Learning site. Additionally with SIFPO funds, PSI has produced case studies that provide an in-depth assessment of SF networks in Tanzania, Mali, and Madagascar.These contribute to the High Impact Practice series. In addition, in year three, PSI partnered with Banyan Global to develop a business skills training module to improve the sustainability of FP services.This was subsequently tested with PACE - a PSI affiliate in Uganda with the assistance of the Merck Fellows. Eight out of the 20 modules were field tested and PSI is now revising the modules to make them available to other countries. Perhaps the most innovative output developed under SIFPO (in collaboration with Merck) was the Social Franchise ‘Business in a Box’ initiative. Faced with the challenge of each country having a different busi￾ness model for their social franchise, this toolkit enables country offices to align their work with PSI’s corporate business model for SF. Fully financed by SIFPO, the development of the tool kit allowed PSI to draw on their substantive global experience to see what strategies worked and what did not with regard to SF. Twenty seven people including participants from five SIFPO mission buy-in countries attended the initial training in 2013 and training for Francophone countries is scheduled for early 2014. In the evalu￾ation questionnaire, the PSI Cambodia country office reported that “The Business in a Box initiative, including trainings and resources, has strengthened the organisation’s capacity to respond to platforms’ needs to bring strong and sustainable franchise networks.The comprehensive toolkit, available on the or￾ganisation’s web-based resource centre KIX, hosts key resources and guidance tools needed by platforms as they progress from one stage of franchise development through to the next. Moreover, this key initia￾tive has established clear and measurable markers for moving through the various stages of franchise development and ultimately achieving the gold standard. Strengthening outreach and mobile service provision Most of PSI’s mobile services work has been funded through donors other than SIFPO.With support from SIFPO, PSI/Togo added a full-time dedicated mobile team to provide services within the interior of the country. A SIFPO-funded case study noted that the majority of women who received LARCs via PSI-SIFPO: MIDTERM PROJECT EVALUATION 9 these mobile services were first time users. In Guatemala, a ‘mobile’ service delivery site was visited as part of the evaluation.This was run by the Asociacion Pro-Bienestar de la Familia Guatemala (APROFAM -the IPPF affiliate in Guatemala). Although APROFAM mobile services are currently funded by SIFPO they appear to operate more or less independently of PASMO (PSI). However, the APROFAM-staffed public clinics were refurbished by PASMO (for example, improvements were made in paint, privacy, and new equipment given) and the physicians trained in IUD insertion. In-reach to existing fixed services occur every month to an average of 20 clinics on a rotational basis. Clients are referred by both APROFAM and PASMO outreach workers. APROFAM is experienced in service delivery and was allowed to ex￾pand under SIFPO with pre-established mechanisms of communication and community mobilization.The APROFAM physician interviewed was part of a team of two doctors and three nurses. He reported that during an average quarterly visit they saw around 8 patients for ligations, 10 for implants and no IUDs, and that this number reflected only a slight increase given the additional investment. Although nurses and midwives can place implants and IUDs (and indeed do so during regular service days in CAPs /primary health clinics), they did not do so in the mobile clinics and the concept of ‘task shifting’ seemed to be unfamiliar to the APFOFAM providers interviewed.This may reflect local government policy. Community outreach, BCC and IPC PSI has a lot of expertise in community-level interventions and is extremely experienced in demand-side approaches (including Behavior Change Communication ( BCC), Information, Education and Communica￾tion (IEC), Interpersonal Communication (IPC), social marketing, mass media, edutainment). Every field buy in has had a demand side component which draws on this expertise and links it to the supply side - that is to say, actual services that people can go to to receive a variety of methods in the context of informed choice.These explicit linkages between demand and supply are exemplified by, for example, the Provider Behavior Change toolkit (a major SIFPO investment which PSI developed in partnership with Merck), and the Pakistan Mission buy-in which strengthens mass media activities. In addition, the Guate￾mala Mission buy-in has a strong focus upon community outreach with community health workers going door-to-door to reach prospective clients as well as upon community outreach with youth. In addition, the Kenya Mission buy-in strengthened significant behavior change work around youth. SIFPO supported the development of an IPC toolkit that includes monitoring and supportive supervision tools and structures.As needed,TA support can be provided in-country whether through SIFPO or through other donor resources. PSI also makes these IPC resources available on SIFPO-funded internal online media. In Benin, unfortunately, the SIFPO funding for IPC had been cut but some workers paid by other funding streams were carrying on with the outreach in their communities. Many clients were sensitized while attending immunization sessions for their young children during which the outreach workers referred them to family planning services.This is indicative of both good integration and addresses a practical way to enhance clandestine use in the face of male disapproval. In general, the outreach workers were satis￾fied with their training, which focused on the clinical aspects of family planning. However, many clients experienced financial barriers in meeting the costs of FP and outreach workers reported that 7 of 10 women sensitized could not afford PSI’s ‘Laafia’ brand. TOOLS FOR LEARNING AND PROFESSIONAL DEVELOPMENT Under SIFPO, PSI either developed or expanded an array of online tools for in-house learning and pro￾fessional development.These appeared to not only improve skills and knowledge at the country level, but also greatly strengthened management systems, administration, technical support, and service provision. They also improved communication and fostered south-to-south partnerships and exchange and moti￾vated staff to adhere to the PSI brand enhancing corporate cohesion.The e-learning tools and their use are constantly monitored and their impact assessed. Importantly, barriers to the application of the knowl￾edge applied are identified and addressed. 10 PSI-SIFPO: MIDTERM PROJECT EVALUATION The main tools developed or enhanced during the SIFPO funding period were: 1) PSI University, 2) SocialCast, 3) KIX, and 4) Worksmart. SIFPO initiatives, such as the RH and Social Franchise Leadership trainings, utilized all of these tools as complimentary methods to improving and sustaining the transfer of knowledge and skills among local PSI staff. PSI University: Through the online university, PSI staff in any country can enrol in a course, access the materials, and eventually receive a certificate. In all, 4,463 PSI staff have enrolled in the University, which offers 93 courses (30 by PSI). Nearly 11,000 courses have been completed and 95 percent have used skills acquired from the University on the job. KIX: This is a kind of Wikipedia for PSI established before SIFPO. It focuses on technical areas as well as issues relating to finance, administration, and branding. Eighty-seven percent of field staff have consulted it during over half a million visits. Nearly 80 percent have used knowledge acquired from KIX on the job. SocialCast: This is similar to an internal Facebook for PSI where users can post messages describ￾ing their work or ask for advice or tools. Eighty-eight percentof field staff say they had used it via over 18,000 posted questions. WorkSmart: SIFPO funding supported the development and the piloting of Worksmart, an online, open-source platform for developing resources that support program implementation and easily fosters south-to-south collaboration as resources can be co-developed through this platform. STRENGTHENING RESEARCH, MONITORING, AND EVALUATION Research design Under SIFPO, PSI has carried out some important studies which are ground- breaking methodologically and in terms of their findings.They have made important contributions to the field and elicited new knowledge which should be shared widely with academics, implementing organizations, and donors. For example, in Kenya an innovative study using a quasi-experimental design is currently being carried out to assess if franchising improves access to FP, equity of care seeking, and how it impacts upon provider rev￾enue and volume.This will provide important new information about whether franchises attract existing users away from the public sector and/or create demand among new clientele. Its findings will have impli￾cations for using the Total Market Approach both within and beyond PSI.They will contribute to USAID’s High Impact Practices series as well as serving to refine franchise recruitment by better understanding if franchising improves provider revenue. Data collection and management The PMP addresses improved FP data management and the most recent quarterly report has an indica￾tor relating to the number of PSI platforms reporting on standardized data indicators. At baseline this was ‘0’ and is now at 100 percent. However, the report notes that ‘currently PSI platforms report on a range of indicators but specific indicator definitions may vary from platform to platform’. Standard indica￾tors for country programs were developed in response to an inability to standardize reporting within or between countries and because of inaccuracies in the measurement of common metrics and due to cost￾inefficiencies.These were integrated into population-based cross-sectional surveys where appropriate. This has contributed greatly to inter- and intra-country and regional comparison and will enable better service definition and targeting as well as assessing demand generation and potential. PSI is currently enhancing its Global MIS system as part of its on-going approach to continuously improve and strengthen monitoring, decision making, and program quality.This corporate-wide initiative has been funded through a variety of sources including SIFPO. More attention however needs to be given to the collection of socio-economic data to better target services to poorer clients within the Total Market Approach, per￾haps using vouchers or subsidies if funds permit. PSI-SIFPO: MIDTERM PROJECT EVALUATION 11 Training and capacity building There has been a variety of training and capacity building activities under SIFPO around research and M&E. A number were included in the 2011 RH Leadership Training including sessions on Management Informa￾tion Systems (MIS), Metrics, Research Ethics, Publishing Research and Research Methods. In addition, the Delta process to inform social marketing includes a research component as part of the situational analysis. A global research training in October 2013, was attended by all PSI regional researchers, DC research staff, and research managers from 20 countries.These initiatives and opportunities appear to have had an impact at a country level. For example, in Guatemala, the evaluation survey response elicited that “Research proto￾cols have been updated to include considerations of GBV and more rigorous health impact measurements”. Research at PSI receives strong regional support as well as from the DC office. For example, the survey response from Cambodia’s country office noted that “PSK’s Strategic Information Director receives regular support from PSI’s Regional Researcher based in Bangkok”. In addition, there is good integration across programing with research well embedded in SF as well as in situation analyses for formative pro￾gram design. Considerable feedback on research methods and findings are available due to PSI’s SIFPO￾sponsored leadership participation in various working groups. For example, PSI co-leads the Social Fran￾chising Metrics Working Group (SFMWG) which comprises social franchisors, academics, and bilateral donors. Research and M& E receive ad hoc support from the Director of Research and Metrics but more directly from the RH Research Advisor (who works across several RH projects to ensure synergies). It is recommended below that more human resources be allocated to research to better facilitate the dis￾semination of findings, lesson learned, and best practices. Measurement of impact One of the greatest challenges to strategic programming is measuring cost-effectiveness.Through other donor funding, PSI is piloting work in the African Health Markets for Equity (AHME) projectto estimate costs specific to the impact of SF. However, from the evaluators’ field visits it would seem that additional work to estimate cost-effectiveness is urgently required and needs to take place around PSI’s support to both the public and private sectors.To date, PSI’s in-country programmatic analyses do not yet take into account the cost-effectiveness of each channel of delivery (franchisee, clinic, mobile unit, outreach) for each method.Through the establishment of the new MIS system (DHIS2), PSI will be able to determine cost-effectiveness by channel for FP provision. However, for logistical and financial reasons, further disag￾gregation by method or user characteristic, while desirable, is impractical. PSI’s research division has also spent considerable time in carefully investigating and testing different mea￾sures of economic status and has contributed significant and innovative methodological advances that will take the field forward. After discussions with the SFMWG, and testing and comparing the PPI and Wealth Index used in the DHS, it was decided to adopt the latter.This is now being scaled out across countries and will be routinely used in exit interviews, among franchise clients, and in TRaC surveys.TRaC surveys collect demographic, attitudinal, and behavioral information including measures of socio-economic status (SES) from a representative sample of target populations. For example, in Benin a TRaC study revealed that new strategies were needed to promote condoms among poorer youth in the informal sectors. PSI has been investigating the use of DHIS2 software which is currently used by many MOHs.This allows for the tracking of individual clients and can integrate information about commodities and service provi￾sion, demographic characteristics, and even allow for geographical mapping and the submission of data by mobile phone. PSI, through a variety of donor funding sources, aims to support the launch of DHIS2 in selected countries in 2014 so that they will report in a standardized manner and be better able to share data with MOHs. DHIS2 will allow for a unique client identifier and permit tracking over time and allow better market segmentation and the development of marketing plans.This will, in turn, improve quality of care and follow-up. In this way, PSI can better respond to funding stream requirements (including those associated with SIFPO) whose brief is underpinned by facilitating access to services for poorest women. 12 PSI-SIFPO: MIDTERM PROJECT EVALUATION PSI measures client satisfaction in exit interviews which give extremely high rates of satisfaction with services. During a technical meeting in October 2013, PSI country-level researchers from 20 countries and regional researchers covering all of PSI’s programs were trained in how to measure equity using a client exit interview tool. PSI acknowledges that there may be social desirability biases in the measuring satisfaction via client exit interview but notes that the multi-dimensional measures of satisfaction and perceived quality can effectively capture variation in client satisfaction across settings. PSI has also used mystery client techniques in a number of countries, including Cambodia and Mali to assess perceived quality and observed aspects of technical quality in a more objective manner. Data for Decision Making Aiding providers around understanding data for decision making is an area that requires further strength￾ening.The field visits to Guatemala and Benin revealed that there was very little notion of ‘Data for Deci￾sion Making‘ among either local country staff or their partners either in franchises or public health struc￾tures. Data collected and collated via registers or via PSI’s own forms were rarely transmitted back to providers, particularly those in MOH facilities, or to staff in franchises, other than the owners. As a result, those required to fill in forms or registers felt that there was little motivation to do so as they did not see the results of this work, nor understand its purpose. Providers also had other reporting obligations to the MOH and other donors which increased their administrative burden. PSI M&E staff in Guatemala and in Benin felt that once DHIS2 was operationalized it would enable the more rapid collation of data pertaining to service delivery and commodities and facilitate, for example, an understanding of the domi￾nance of one method (e.g., injectables in Guatemala) or stock-outs.The collection of socio-economic data would allow for an analysis of client profiles to ascertain if the poorest and most in need were really being served.This can be continued and expanded by the use of cross-sectional surveys rather than rou￾tine monitoring which would be costly and time-consuming. In addition, using DHIS2, selected variables can be submitted by mobile phone allowing for more timely and accurate reporting. Knowledge sharing It was recognised by both USAID and PSI that SIFPO has allowed PSI a ‘place at the table’ that they did not formerly occupy in terms of membership of groups for knowledge sharing and policy formulation. These positions allow PSI to share the findings of projects and research implemented under SIFPO and to draw on best practice.Table 3 below outlines the memberships directly attributable to SIFPO. Table 3 : Group membership attributable to SIFPO support USAID HIP Technical Advisor Group,Task Team and Partners Group Market Dynamics Working Group under FP2020 Steering Committee Member of the International Conference of Family Planning (2011 and 2013) Sayana Press injectable contraceptive ETAG, Global Health Group’s Social Franchising Metrics Working Group (Co-Chair) Youth Health and Rights Coalition (Co-Chair) Implementing Best Practices Consortium International Consortium for Emergency Contraception, Private Sector Working Group WHO Technical Consultations PPIUD Working Group Steering Committee of Women Deliver FP/Immunization Working Group Coalition for Adolescent Girls (Steering Committee and Effective Practices Working Group Co-Chair) PSI-SIFPO: MIDTERM PROJECT EVALUATION 13 Under SIFPO, PSI has also participated in and organized several technical consultations. For instance, PSI collaborated with USAID partners EngenderHealth and SHOPS to plan a series of technical consulta￾tions on bringing LARCs closer to the client. The first consultation, held in September 2013, attracted 71 participants from 18 organizations and focused on mobile services and dedicated providers. A second consultation focusing on LARCs in the private sector is scheduled for March 2014, and a third, focusing specifically on LARCs and social franchising, will take place in the second quarter of 2014. In addition to leadership and participation in multiple technical working groups, PSI has collaborated with multiple partners to support learning and dissemination. For example, PSI presented or served as discus￾sant at FHI 360’s PROGRESS and the Institute for Reproductive Health’s Fertility Awareness Project’s End-of-Project Meetings, and presented lessons learned at the 2012 PSI-organized meeting to dissemi￾nate new WHO guidance on task shifting. RESULT 2: QUALITY ASSURANCE - INTERNAL QUALITY ASSURANCE STAN￾DARDS AND RESULTS QUANTIFIED AND DISSEMINATED TO STRENGTHEN FP PERFORMANCE AT A GLOBAL LEVEL Quality Assurance Under the guidance of the Global Medical Director, PSI undertakes a thoughtful approach to quality improvement tools and management. As an organization, there is the belief that quality has to be built by the implementers from the inside out within the confines of supportive structures. SIFPO funding has allowed the organization to pay stricter attention to quality indicators as it shifted into global recognition as a FP service delivery institution. During the duration of SIFPO funding, a QA audit tool has been strengthened and all countries have undergone Quality Audits by a team of auditors.The team of auditors comprises at minimum two people with one devoted to clinical evaluation.There is a standardized comprehensive checklist for internal/ external auditors which assesses the knowledge, action, and skill levels of providers. Proficiency or competency levels are noted with areas for remediation identified. Auditors are asked to give feedback to country teams and technical assistance where indicated, prior to the end of the audit timeframe.The focus of the external audits are to assess and improve PSI’s country-level quality improvement teams, and provide them with the systems and skills to address issues identified during the audit in a sustainable manner.The QA tool is easy to use and addresses major technical areas and infection control in ad￾dtion to PSI’s Quality Assurance Standards. The checklist used in the “document review” portion of the audit could benefit from greater specificity. The results of the audit and action plans are reviewed by the Global Medical Director. The Independent Audit QA Score card tool is based on 21 standard indicators of clinical quality, which is coupled with the audit to appraise the performance of country programs.The QA scorecard, which is designed to act as a template for prioritization of needs, was piloted under SIFPO and taken to scale in 2014. Step-wise instructions for following technical competency are clearly laid out for LARC and PM methods.The audit may also be used as an opportunity to supervise a supportive supervision visit and to address the root causes of weaknesses observed. Covered under the clauses of contract specifics, franchises are aware that a failure to comply with quality standards may result in de-franchising.Through the independent external audits conducted bi-annually, PSI strives to reach a representative sample of clinics in different regions of the country, including remote areas and all clinic styles and service delivery mechanisms, but recognizes the limitations inherent in a two-week audit.The independent external audits complement internal audits conducted annually with a larger sample of clinics, and are meant to identify common problems and their systemic causes and work with country QA team to address them. The QA audit system rotates the evaluation of countries depending on the size of the total SF network in a 14 PSI-SIFPO: MIDTERM PROJECT EVALUATION country. PSI remarks that “effective audits are difficult in larger countries” with the audit team reaching “however many it is able to reach in the two-week time frame”.This may detract from examining a full range of services and from assessing quality in remoter areas and PSI recognizes these weaknesses and notes that PSI audit teams strive to reach the maximum number of service delivery sites they can, with representation of all modes of service delviery in a country (e.g., mobile, seconded providers, franchised clinics, etc.) and cover a range of rural and urban sites in a range of regions. The logisitical limitations of external audits underscore the need for regular, more extensive internal audits as part of the role of the external audit is to help validate internal audit fi ndings. PSI audit systems also allow for simulations of patient interactions to be used in lieu of actual patient encounters due to inability to predict patient flow during the time of audits.To assess such simulation ex￾ercises, the evaluators undertook a simulated patient scenario with a designated FP champion performer in Guatemala which revealed that additional support systems are needed to ensure accurate side-effects management and counseling. It was stated by PSI headquarters that SIFPO had allowed for a deeper, more thorough QA audit system and analysis than was previously allowed due to the challenges of time, need, and trust of the country platform SF network. It is planned that 18 of 23 countries’ FP programs have independent QA audits for 2014.Through SIFPO, PSI has made significant progress in the regionalization of quality improvement, developing a cadre of experts from within its country programs to improve the performance and sus￾tainability of these programs. To date, PSI has trained 27 host-country national staff to become external auditors and 17 have taken part in independent audits. In addition to auditor trainings, PSI uses a variety of approaches to increase QA capacity of staff across the organization including medical and program￾matic meetings, often organized around international conferences; and contraceptive technology updates via webinars and other online platforms. There are also global mechanisms in place to track and record adverse events that may occur. Overall, based upon field observations (which are limited to the two countries visited and findings are not gener￾alisable to PSI as a whole), it can be assumed that private sector providers need additional support to ad￾here to PSI mechanisms for Adverse Event (AE) reporting.The providers interviewed were confident in the ability to refer individuals through the in-country management systems but did not know about the format for PSI compliance. PSI developed an “AE Checklist” to help ensure countries identify any weak￾nesses in their AE system. This checklist is disseminated during audits and during any regional or global meetings of QA and program staff. PSI headquarters recognizes vulnerabilities in any AE reporting sys￾tem, especially in a context where reporting requirements may be new to providers, and the checklist is an important step in building PSI platform capacity to address this. SIFPO funded an AE video focused on complications following an IUD placement.The video is now translated into English, French, and Spanish and available on the KIX platform. During a field visit in Benin, it was noted that network providers could continue to focus more on patient privacy and comfort, for example, by minimizing the time a patient is undressed or by providing cover, though it is recognized that these practices are associated with MOH clinical norms in place. It is also recommended that PSI play a role in advocating for system-wide improvements with the MOH in this area and intensify its provider behavior-change work to address remaining challenges. PSI recog￾nizes the importance of task shifting and realizes the need to more proactively encourage countries to apply appropriate approaches to service delivery models. Respectful care is central to service quality and patient rights. PSI SIFPO-assisted systems strengthening has contributed to the scale-up of LARCs and PM delivery channels. During the three voluntary laparoscopic tubal ligation procedures that were observed via APROFAM in Guatemala, the in-reach clinic was converted into a fully operational theater utilizing three licensed personnel for laparoscopic procedures.The physicians used a full laparoscopic set-up for pro￾PSI-SIFPO: MIDTERM PROJECT EVALUATION 15 cedures that could be carried out by the less costly mini-lap in which all Guatemalan physicians have been trained. SIFPO is sponsoring a physician from APROFAM to attend the PSI and EngenderHealth led technical clinical symposium in March 2014 on provision of permanent methods in low resource settings, focused on NSV and mini-lap female sterilization, as a way to improve their knowledge and understand￾ing of this procedure in hopes they will change their approach to be more cost-effective. Clinical training and capacity building PSI has capitalized on the addition of SIFPO funds to initiate several trainings in multiple areas ranging from infection control to business skills throughout its service delivery channels. Comprising a major focus of its clinical capacity building, these trainings generally use a cascade competency-based approach for future large scale-up.These competency-based trainings fill an important gap for providers who may not have had sufficient clinical pratice in their pre-service training. PSI endorsement and training is deemed to give added value to services of private sector FP providers.The institutional alignment of sup￾port systems and training within PSI led to the greater recognition of higher quality services and elevated provider status in the community. Selected providers interviewed in Guatemala and Benin indicated that there would be no LARCs or PMs in their communities if not for PSI support. SIFPO invited team members to attend the PSI and EngenderHealth led technical clinical symposium in March 2014 on provision of permanent methods in low resource settings, focused on NSV and mini lap female sterilization, as a way to improve their knowledge and understanding of this procedure and to participate in repositioning PM as important methods in the method mix for those women and men who want them. One of the most notable workshops centered on PPIUD insertion. In partnership with MCHIP, the PSI SIFPO team organized a regional meeting in Africa to bring together international and regional experts to advance integration of PPIUD services into maternal health services. Participants from 10 countries included MOH staff, donors, and implementing partners.They actively engaged in south-to-south learning, sharing successes and challenges based on their country experiences.The objective of the meeting was to help existing PPFP/PPIUD programs accelerate their integration of the service into maternal health services by: providing a forum for participants to observe PPIUD services in one country, and share successes and discuss challenges to implementing quality PPIUD programs from initiation to scale-up; discussions on the role of advocacy, community engagement and service delivery strategies within PPIUD programs, grounded in the use of evidence and guided by the use of program data; and, providing an op￾portunity to develop tools for country-team action plans. A second workshop for Francophone pro￾grams is scheduled to take place in Burkina Faso in February 2014. The field settings highlighted the two differing aspects of the challenges surrounding the provision of PPIUD in the public sector outside of PSI’s SIFPO funded program. In Benin, the MOH uptake of the training of the PPIUD lead to a greater ability to offer the service, yet due to high volume of deliveries at the national hospitals, providers were unable to offer the method. Service delivery for FP is complicated by rapid discharge planning after delivery.Through its social franchise mechanism’s intra-partum services, PSI is uniquely positioned to add PPIUD service to its lineup. PSI recognizes the importance of including PPIUD information as part of post-partum FP counselling that takes place in antenatal care and as part of pre-discharge counselling to help women make an informed choice. Capacity building among PSI staff is reinforced by the availability of PSI University and online support ser￾vices for providers to continue to access support once training sessions are completed. PSI could explore similar on-line support for franchise members. However, limited connectivity continues to pose a problem for access to the online support for clinical services. All providers in Benin reported recent face-to-face training in the areas of infection control.This correlates with the ABMS reports indicating that modalities of infection control was one of the areas most in need of support due to initial poor quality scores and reflects an appropriate and rapid organizational response. 16 PSI-SIFPO: MIDTERM PROJECT EVALUATION Visual job-aid materials were in widespread use throughout all observations in all sites. Job aids assist in providing both patient information and step-wise skill reminders to providers.Theses aids also served in the branding of PSI products for public awareness and also seemed to instill a great deal of provider pride. PSI is conscious of provider concerns and strives to develop tools, processes, and support networks to minimize challenges in the field. PSI seems to have an overall grasp on the needs of its providers and has taken step to address gaps pertaining to quality and provision. For instance, providers have expressed a felt need to improve their business skills to ensure the sustainability of FP services. As such, through SIFPO funding, PSI has developed business skills modules and is in the process of piloting and translating them for broader dissemination. PSI appears to have an overall grasp of the needs of its providers and has taken steps to address gaps pertaining to quality and provision. RESULT 3: INCREASING ORGANIZATIONAL SUSTAINABILITY AT A COUNTRY LEVEL Sustaining demand As described above, PSI has explicitly linked demand creation to service provison in a number of in￾novative and sustanable ways. In general, by working with local networks and supporting Ministries, PSI appears to successfully create and sustain demand for FP services. In many settings they have created a “tipping point” with regard to behavioral norms which has resulted in national impact. Integrated ser￾vices create a “culture” of FP, enabling women to identify and articulate links between lower fertility and better maternal health and child survival. In Madagascar, PSI is expected to contribute to changes in national CPRs. In Cambodia, over 50 percent of all CYPs in the country are provided by PSI. Supporting Ministries of Health The PSI SIFPO funding has been utilized to further integrate PSI systems of training and supportive supervision into the MOH mechanisms in partner countries.There appear to be strong relationships that have been built and a level of respect between all parties. In Guatemala, the policy work on legal aspects of reproductive rights which framed FP activities (including those observed with youth at a local level) was impressive. However, due to the severe human resources for health constraints within the regions where PSI works, the potential for long-term sustainability may be of concern and PSI’s capacity-building efforts among MOH plays a role in addressing these gaps. A key to attaining and maintaining sustainable programming centers around MOH involvement in FP service delivery mechanisms In certain countries, PSI is filling LARC/PM gaps directly for the MOH. In the long term, the MOH will need to maintain the capacity to provide these methods. In countries with extremely low CPR and shortages of health care workers, PSI believes that it is essential to address high unmet need while working toward a more sustainable solution. In Guatemala, the MOH staff report that, prior to PSI, training around LARCs and PMs was absent and that significantly lower numbers of long-act￾ing methods are offered outside facilities supported by PSI. Now, due to PASMO, the Ministry is ordering IUDs as part of its stock of commodities for the first time.This example demonstrates SIFPO’s success with increasing access to LARCs and PMs. PSI protocols follow MOH guidelines.When MOH guidelines are not in alignment with PSI protocols or standard practice, the Global Medical Director reviews inconsistencies to ensure that the differences do not compromise patient care. The inconsistency of local policy with international best practice perhaps highlights how PSI and other partners must continue to advocate to change policy which, in the long run, may serve to decrease the cost and effort within the public health sector. However, PSI is mindful of local MOH norms at all times. PSI-SIFPO: MIDTERM PROJECT EVALUATION 17 In many cases, PSI provides supportive supervision and opportunities for training and learning for govern￾ment health workers to ensure long-term sustainability and returns to investment in human resources. For, example, in Cambodia, PSK implements Practical Training Days “Plus” (PTD+) at select public Health Centers.These consist of two days of community outreach among women of reproductive age to gener￾ate clients for FP counselling and services. Coached and supervised by PSK’s Supportive Supervision team, public sector providers practice their counselling and IUD insertion/removal skills.This, in turn, leads to greater confidence and an increased likelihood that IUD service provision will be available to women seeking FP services in the public sector. Also in Cambodia, PSK’s cervical cancer screening and treatment pilot program is also consistent with the MOH’s Guidelines.The MOH has committed to providing a na￾tional trainer to lead training sessions confirming evidence of buy-in and of the sustainability of PSI’s activi￾ties. However in clinical settings in Benin, it was observed that MOH staff who had been trained in LARC/ PM procedures acted in an administrative rather than clinical role during consultations. For example, they filled in charts or opened packaging instead of actually doing procedures when paired with PSI- affi liate staff. Further investigation revealed that this was a common occurrence. If trained staff do not practice or apply their recently acquired clinical competencies, this does not support capacity building or sustainability and may present a challenge for the retention of new skills with regard to providing LARCs/PMs. In addition, it was noticeable in Guatemala, that PSI’s investments in the interior government clinics, as requested by the MOH, has resulted in only a very few voluntary LARC/PM users in nearly all the three facilities visited. Despite these overhauls, together with the provision of training and technical sup￾port, voluntary LARC users averaged less than two clients a month.The extremely low uptake of these methods underscores the importance of the need for cost-effectiveness studies of method provision by channel. However, it should be recognised that these services are relatively new and that until now LARCs were unavailable. Demand in these locations may therefore initially be low, especially in a context where socio-cultural factors frame myths and rumours which discourage their use.. Yet, PSI is starting to see increases. For instance, in the San Miguel Uspatan SIFPO supported clinic, voluntary LARC uptake increased from 4 LARC method users in 2012 to 102 new users in 2013. Supporting the private sector The ongoing research that PSI is currently undertaking in Kenya is likely to provide important results that will shed light on the motivation, activities and impact of private sector providers and how they directly affect sustainability.The Business Training for franchisees is invaluable for increasing their sustainability and thereby their motivation and adherence to PSI’s principles and practice as network members.The in￾novative work developed by the Merck Fellows and Banyan Global has developed business skills training modules for improving social franchisees’ business operations in a sustainable manner. Historically, the private sector for health care, has received far less attention in efforts to scale up and improve the quality of global health care in resource constrained areas. PSI has filled this gap and inte￾grated its services to focus on the needs of the private sector franchises which may be under-regulated and under-served.This benefit is recognized by providers. A ProFam Franchise owner in Benin said “Without PSI I wouldn’t have received any other training in the last two decades since finishing medical school.With PSI I am more respected amongst my peers and active in my Association’s activities” In Benin, 5 out of the network’s 57 franchisees were interviewed. Some of the larger (already profitable) franchisees acknowledged the clinical improvements and skills acquisition from which they had benefit￾ted. However, some did not see the added value of belonging to PSI’s network and thus require additional business training to explicitly show them the links between quality, demand, and income. Sustainability can also be enhanced by integrating other services beyond FP. For instance in Benin, social franchises also offer HIV counseling and testing, antenatal care, prenatal care, safe delivery, and malaria prevention and treatment. However, where this occurred in Benin our observations noted that although FP appeared to ‘flourish’ because of PSI’s excellent branding and publicity materials for FP, it tended to hide the fact that other services, such as immunization, were on offer. 18 PSI-SIFPO: MIDTERM PROJECT EVALUATION Leveraging and cost sharing PSI has reported almost $6 million in cost sharing and has therefore achieved the 15 percent cost-share requirement. PSI consistently identifies ways to leverage SIFPO funds to maximize investments. In Cam￾bodia, the Mission noted that “The Social Marketing/Behavior Change Interventions award funded by USAID and DfID during 2007-2013 earned income by PSI that has been reinvested in activities that are conducted under the PSI-SIFPO award.Thus, the cost share by PSI is larger than the USAID $1.5 million that the Cambodia USAID Mission has invested for the initial year of our buy-in to the central SIFPO mechanism”. In addition, PSI is rolling out the Business in a Box toolkit to five different countries with other corporate donor sources, with the aim of improving the effectiveness and efficiency of the SF net￾work. In addition, SIFPO’s support of PSI’s participation in FP2020 initiatives has led to additional funding through the Jadelle Access Initiative. Internal south-to-south support and technical assistance The mentoring program encourages exchange and fosters support both between the DC office and the field staff and between in-country staff.These exchanges are enhanced by PSI’s capacity-building tools avail￾able online as well as by personal contact as discussed above. Additional south-to-south support comes via the Quality Audits. For example, the QA audit in Benin was conducted by PSI staff from Mali. On a regional level, there is also strong inter-country collaboration, for example, via the regional workshops associated with the PPIUD Working Group. Fruitful south-to-south exchanges have also occurred with PSI. For example, via the Addis family planning conference, a representative from the Ghana MOH became interested in urban outreach and PSI Ghana has reached out to PSI/Mali to organize an exchange visit. RESULT 4: GENDER SENSITIVE FP SERVICES TARGETING YOUTH STRENGTHENED AT A GLOBAL LEVEL Youth The PMP indicator for youth relates to the number of PSI service delivery platforms that formally pro￾vide gender and youth-friendly services. At the baseline, the number of platforms was ‘0’ and the target was ‘5’.There are now nine platforms with youth-specific programs and two have been trained with the new Youth Friendly Health Services (YFHS) guide.Youth programs are designed around the Delta mar￾keting tool to be evidence-based and audience-centered and to promote knowledge and services as well as life skills and experiential learning.The value SIFPO brought to youth-related activities was in creating new tools – namely, ‘Making Your Health Services Youth Friendly’ (with IntraHealth) and a ‘Sexual and Reproductive Health Programing Brief for Young People’. Existing tools were also upgraded to include youth-specific approaches.This included integrating elements of the YFHS into the Business in a Box and updating RH Standard Indicators to support M&E for youth programs. The ‘Making your Health Services Youth Friendly’ guide was developed and piloted in Malawi.The guide was based upon a careful review of best practices and current thinking in effective service delivery for adolescents and young people.While the guide does not bring new approaches to the field, it provides a compact, user-friendly and action-oriented tool for health providers who have not been trained in YFHS and have little time to do their own research on best practices.The guide’s key messages center around the unique needs of adolescents and young people,WHO’s standards for youth-friendly health services, quality assurance, and the importance of training and sensitizing providers. However, it is not clear how its impact will be measured. Over 20 health providers, young people, and the PSI country program staff in Liberia were trained in implementing youth-friendly services using the guide which contains an important gender component. The “Plan de Vida” or Life Plan sensitization sessions observed in Guatemala were well-liked by a se￾lected number of participants who were briefly interviewed.The holistic approach, which allows young people to articulate their personal visions and aspirations has been very well received by all sectors. Positioning FP in this life-skills agenda and providing relevant legal information (for example, about how PSI-SIFPO: MIDTERM PROJECT EVALUATION 19 intercourse is illegal below age 14), as well as information about how to report abuse, situates young people’s sexual health knowledge acquisition in a useful framework of rights and empowerment. PSI/Madagascar was able to expand its youth program and service delivery within franchised clinics with support from SIFPO especially focusing on FP/HIV integration.They reported that 98.9 percent of youth clients were satisfied with the advice they received in the Top Reseau clinics. Gender Prior to SIFPO, PSI did not have a gender focal point. Staff dedicated to gender and the partnership with IntraHealth has helped mainstream gender issues in all aspects of service provision. PSI-SIFPO partici￾pated in Gender Cooperating Agencies’ meetings, and shared gender results in USAID bi-annual reports. A number of gender indicators based on USAID and WHO standards requiring services to be ‘gender equitable’ were developed under SIFPO. Gender indicators were incorporated into tools used in all programming areas.Tools specific to family planning that were adapted during SIFPO to reflect gender equity include the QA manual and the Delta tool for the planning and marketing of services.The tools have been disseminated via KIX and SocialCast.The KIX page on gender was started by SIFPO in 2012 to engage PSI staff around the world in examining and addressing the ways gender inequities, biases, and norms affect reproductive health Indicators in the client satisfaction tools are framed around questions such as “Did the facility require your parent or partner’s consent for you to receive services today?” The tools also look at whether clients would have preferred to see a provider of a different gender. Guatemalan clients who were in￾terviewed as they were waiting for FP services said they would prefer to see a male doctor rather than a female nurse as they thought they would be more competent, indicating that culturally constructed ‘world views’ of quality also have important gender biases. Importantly, male perspectives are also given weight in the gender-sensitive approaches. For example, in Guatemala, PASMO has been involved in a ‘Responsible Fatherhood’ campaign and has set up Fathers’ Day events, recruited male IPC agents, and initiated FP discussions around film screening for men. Male involve￾ment in Kenya has been focused around the ‘C-word’ campaign to promote contraceptives among youth. Under SIFPO, trainings on gender for PSI/Washington staff included workshops for RH technical, pro￾grammatic, and research staff in 2011 and 2012; and values exploration exercises and training on gender analysis, gender continuum, and gender integration throughout the program cycle. In addition, trainings were offered to 56 host country national (HCN) staff including 29 HCNs trained during the Anglophone and Francophone RH Leadership trainings and 27 HCNs trained during the Social Franchise Leaders Training. Under SIFPO, additional trainings have been held for PSI-Washington and field staff on GBV. These include a session on GBV at the Research Technical Meeting (30 HCN, 15 PSI-Washington) and a day-long GBV capacity-building workshop for the Sexual Reproductive Health and TB Department, in which USAID participated. In addition, under SIFPO, PSI developed GBV Research Guidance with IntraHealth.This comprised ethi￾cal and technical guidance, information about qualitative and quantitative research techniques, and GBV indicators. Subsequently, gender approaches, particularly those relating to GBV, have featured significantly in PSI’s country-level programing since SIFPO’s inception. Guatemala reports that “there is an increased empha￾sis on incorporating gender issues into all aspects of our FP/RH programming” and that “indicators have been developed and introduced for gender sensitive and youth-friendly indicators.These have been in￾tegrated into RH log frame,TRaC survey questionnaire, and client exit interviews “. In Benin, with SIFPO and other donor support, ABMS runs a toll-free health hotline which recently started to respond to 20 PSI-SIFPO: MIDTERM PROJECT EVALUATION calls about GBV.The information compiled by the PSI-Washington team and placed on KIX on GBV has helped the ABMS management team better respond to the needs of its target population. The gender work PSI has conducted under SIFPO has also had an effect on other health program areas within PSI. For example, gender aspects are also addressed in VMMC demand- creation tool kit via the ‘Gender Continuum’.This aims to orientate programs to be gender transformative and to promote eq￾uity to reach health outcomes.The recent study of acceptability of NSV in Benin, conducted with SIFPO funding, also provides useful gender-orientated material related to service provision. For example, biases against NSV were evident among many providers interviewed although the majority said that they would provide the service if it were offered in their clinic. However, importantly, it was found that the current organization of services at certain clinics in Benin is not conducive to serving men.This finding is relevant within and beyond PSI’s networks as most FP settings are highly feminized and men may feel uncom￾fortable or stigmatized if using them. An additional study result with wider cultural relevance (and one reiterated by the MOH representatives interviewed in Benin) was that often women were not in favor of NSV as they thought it would give their partners a free license to go off with other women without the risk of getting them pregnant. Although these findings are anecdotal they illustrate the need to routinely address perspectives from both sexes when targeting messages or services that on the surface appear to be for either men or women. PSI-SIFPO: MIDTERM PROJECT EVALUATION 21 V. CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS Overall, PSI has made solid contributions to FP service delivery and developed important and accessible resources for capacity building and technical support. Lessons from PSI’s rich history of social marketing and franchising have been capitalized upon with the simultaneous strengthening of the public sector. Dur￾ing the period of SIFPO funding, a considerable number of new franchises have been added to PSI’s net￾work and a concurrent increase in CYPs has been recorded. However, it is not possible to quantify the contribution of SIFPO-funded initiatives to this expansion.This is because given that the nature of SIFPO to strengthen organizational support, increases in health impact are due to both direct implementation as well as improvements to systems, introduction of innovative practices, and staff capacity development. PSI has focused largely on support for internal structures and extensive capacity-building mechanisms in order to affect change.The organization is wise to acknowledge the challenges of a decentralized system and is taking steps to ensure the circular flow of information through the use of online technologies.The emphasis on quality and informed choice permeates all levels. PSI has also brought about some significant clinical contributions to the global field of FP such as the PPIUD. PSI has addressed issues of quality assurance in a mindful manner attempting to integrate quality checks into every aspect of its franchising systems. As expected, PSI is challenged by the very nature of its role as a non-employing overseer of a large number of private sector providers who are constantly balanc￾ing quality with the financial and practical concerns related to running a business.The Business in a Box initiative successfully fuses elements of quality and business training and draws on the important support of the Merck Fellows as well as other global partners. PSI is to be commended on its approach to integrated MNCH services which serve to broaden the global impact of FP delivery and increase both cost-effectiveness and cultural efficacy, especially in set￾tings where male disapproval is high. Quality Audits and supportive supervision have been increased, and enhanced the technical competence of providers in both the public and private sectors. PSI has made significant progress in the regionalization of quality improvement, developing a cadre of experts from within its country programs to improve performance and sustainability throughout the organization. PSI provides excellent capacity-building opportunities and, most importantly, ensures that long-term feedback and supervision are given to participants. On-going exchange and sharing of tools occurs via KIX, SocialCast and the PSI University thus strengthening technical skills and quality and enhancing south-to-south support.The Reproductive Health Leaders Training and the Social Franchise Training both serve to transfer knowledge, increase skills, and enhance local capacity and suitability. Innovative research tools, especially pertaining to gender, were incorporated into recent research training, although issues of accreditation and ownership need to be resolved with IntraHealth. With regard to research, PSI has standardized indicators across programs and has introduced equity measures into exit interviews of client satisfaction and into the TRaC cross-sectional surveys mechanism. However, insufficient attention is given to the economic background of both public and private sector cli￾ents. Clarity is needed on the socioeconomic status of clients that PSI social franchises serve in order to understand the type of value added by the franchising mechanism. It is important to learn whether social franchises expand FP access for the poorest clients or relieve some of the burden on the public sector by serving middle-income clients. Given that it would be burdensome for providers to routinely collect data on clients’ socioeconomic status, PSI should collect this information using surveys generalizable to the service points affiliated with PSI.The information would allow PSI to better understand its contribu￾tion to Total Market Approaches and identify where voucher systems or integrated microfinance activi￾22 PSI-SIFPO: MIDTERM PROJECT EVALUATION ties for the poor may be needed. Another rationale for working with the private sector is getting those who can afford to pay out of pocket to do so leaving MOHs to serve poor. Current data collection systems (which include those for PSI, the Ministry, and other partners) tend to be burdensome for local health personnel. Often they do not see the point of data collection as it is not fed back to them or used to reorient services. A greater emphasis on Data for Decision Making is required so that services can accurately reflect client need. PSI hopes to introduce the DHIS2 system which may lesson health workers’ reporting obligations. It will also increase the potential for data for decision-making, ensure enhanced transparency between MOH, PSI, and other partners and allow for ongoing cost-effectiveness monitoring.This is much needed as a greater emphasis needs to be put upon value for money for both SIFPO and for the beneficiaries. The research carried out under SIFPO has been well applied.The findings of the Kenya social franchising study currently underway will take the field forward in terms of understanding both user and provider decision making.This study has great potential as it is multi-donor funded, adds to a very limited body of literature, and comprises quality research on social franchising. However, USAID and other partners noted that, often, research findings are not written up or disseminated, nor published in peer-reviewed journals.The current allocation of time for the Research Advisor is insufficient and should be increased in order for PSI to share knowledge and promote best practice more widely. It is highly recommended to increase research support to aid in the recording and wider dissemination of PSI project findings, lessons learned, and best practices. PSI is mindful of sustainability issues and seeks to address these in a number of practical ways, such as providing south-to-south technical assistance; facilitating intensive, practical leadership trainings for local staff; increasingly influencing FP policy discussion; strengthening the private sector including by creat￾ing public-private partnerships, and focusing on leveraging SIFPO through continued diversification of funding, including bilateral agreements and cost-sharing with other donors.The productive relationships within the MOH, for example, formulating and re-orientating policy and increasing investment in com￾modities, represent a strength that can be further built upon to attain and maintain the goals of FP2020 and beyond. PSI has integrated gender and youth perspectives into a number of tools and programs, and there re￾mains room to advance in this area. Building the capacity of PSI’s social franchise providers, in addition to some of its public sector provider partners, to strengthen youth-friendly services is commendable. Fu￾ture resources should be dedicated to piloting new and innovative delivery mechanisms to young people, as well as targeting particularly vulnerable populations such as married girls.With regard to gender, prog￾ress has been made now that PSI has a gender staff focal point. Gender-awareness activities, especially those pertaining to GBV, have been strengthened with the aid of IntraHealth, and initiatives such as the NSV assessment in Benin have sought to resonate culturally with gender norms affecting potential clients. The client satisfaction tool developed under SIFPO will allow PSI to analyze client preferences regarding provider gender and accommodate those preferences to the extent possible. Gender issues relating to clients’ perceptions of the relative benefit of male or female providers require further attention if task￾shifting is to become an even greater priority as recommended below. It is anticipated that PSI will continue to contribute to the field of FP and leave a significant mark within the domains of service delivery, particularly within the private sector. RECOMMENDATIONS Organizational capacity • Improve the sharing of e-learning tools to document and measure improvements in internal capacity and explore opening up access to e-learning to the provider network. PSI-SIFPO: MIDTERM PROJECT EVALUATION 23 • Improve the dissemination of the latest clinical protocols and procedures from headquarters to avoid delays in updating staff, including local providers in PSI or partner facilities. • Submit the results of studies that subsequently influence global practice for peer review • Where approariate ( i.ee., in line with MOH and Misson policy), initiate ongoing cost-effective moni￾toring of method delivery by channel including appraisals of returns on investments in infrastructure, equipment, and training. Adjust delivery mechanisms, if feasible, to improve cost-effectiveness. • Identify additional ways (beyond TAGs,Working Groups and Technical Consultations) in which PSI can strengthen the capacity of other international organizations and communities of practice, for example by sharing tools and research findings with PSI and other international FP agencies. Quality assurance • Continue the streamlining of health systems management and clinic (mobile or fixed) functioning to increase productivity and provider and client satisfaction. • Working wth context-specific mandates, priorities, and resource availability, instigate real mobile services to reach women in remoter areas as opposed to investing in ‘in-reach’. • Leverage PSI’s wide network of in-country partners for service delivery to more proactively advo￾cate for changes in MOH service delivery protocols that are not consistent with global standards. • Integrate and monitor respectful care practices and gender awareness for all elements of patient care and FP service delivery. • Instigate better mentoring and coordinated capacity building of PSI’s local partners for more effective collaboration to use advocacy to effect policy-level changes in FP services that reflect international standards . • Carry out advocacy with ministries for alignment of policies to adhere to international standards, in areas such as respectful patient care, follow up on provider training to sustain MOH capacity. • Expand the time frame and increase the systemization of quality audits to reflect more widely repre￾sentative samples of country programs. • Improve price-product match and/or integration with income generation mechanisms for poor women. • Explore increase in role of subsidies and vouchers to enable poor to access methods within the lens of a Total Market Approach pending the availability of resources. • Revision of QA document review audit tool to reflect a higher level of specificity in auditor checklists. • Greater supportive supervision of franchisees’ clinical skills and of the quality of service provision related to FP counseling skills and side-effects management (for example, with regard to the manage￾ment of menstrual disruption). Sustainability • Better disseminate PSI’s work on advocacy and rights within regulatory frameworks highlighting suc￾cessful approaches such as Guatemala’s work with FP law. • Provide ongoing support of franchisees via improving links with formal accreditation programs, such 24 PSI-SIFPO: MIDTERM PROJECT EVALUATION those providing Continuing Professional Development credits through the national licensing bodies or national associations. • Improve data for decision-making for providers and micro-level analysis of client flow, client profiles (including socioeconomic status) and method mix and instigate the use of routinely collected data for improved service delivery and targeting. • Increase the support for DHIS2 roll-out and better coordination of PSI, ministry and partner data collection tools (including exploring the possibility of submitting selected data by mobile phone). • Increase support of MOHs through policy development, continued joint service delivery, training, and exploration of task-shifting mechanisms. • In contexts with low levels of schooling among women, ensure that branding and marketing mecha￾nisms are aimed at the non-literate population. • Provide support to clinic owners to address issues related to their provider recruitment and reten￾tion strategies. • Continue emphasis on south-to-south learning exchanges in addition to QAs, webinars, workshops, and training sessions. Gender and youth • Additional testing of Youth Friendly Health Services Guide and training materials with greater focus on youth participation and ownership. • Pilot additional youth activities in underserved groups such as girls at risk for early marriage or mar￾ried adolescents wishing to delay or space early pregnancies • Emphasize choice of delivery channels for youth based on recent studies and peer reviewed litera￾ture. • Place greater emphasis on alternative non-fixed service delivery models of care focusing on youth outreach to areas where they congregate organically. • Demedicalize inteventions with youth and encourage the active participation of young stakeholders ‘in situ’. • Encourage greater rights-based integration of FP, STI/HIV, and GBV services if funding is available. • Better focus IPC training to address age-specific needs, vulnerabilities behaviors, and risks with re￾gard to communication and service delivery. PSI-SIFPO: MIDTERM PROJECT EVALUATION 25 ANNEX I. SCOPE OF WORK Global Health Technical Assistance Bridge Project GH Tech Contract No. AID-OAA-C-13-00113 SCOPE OF WORK 11.7.2013 I. TITLE: Supporting International Family Planning Organizations (SIFPO) Midterm Project Evalua￾tion—Population Services International Contract: Global Health Technical Assistance Bridge IV Project (GH Tech) II. PERFORMANCE PERIOD Work is set to begin on/about November 20, 2013 with the completion of a second draft report and presentation concluded by approximately February 21, 2014 III. FUNDING SOURCE PRH Core funding into GH Tech Bridge IV IV. PURPOSE OF ASSIGNMENT Overall Purpose: • To assess the PSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement • To gather information that will help to improve the management of the PSI project for the remainder of its implementation • To gather information that will result in useful recommendations for any potential future projects External Technical Evaluation • To evaluate whether or not PSI project’s activities are leading to the results and outcomes outlined in the agreement; • To identify if there have been any technical gaps that have prevented achieving intended results of PSI project; and • Based on accomplishments toward results as well as the current/anticipated environment, identify potential technical future directions V. BACKGROUND The PSI SIFPO Project mission is to increase the use of family planning (FP) services globally through strengthening selected international FP organizations which have a global reach and an extensive, multi￾country network of FP clinics, in order to achieve maximum program impact and synergies.The current SIFPO project is working toward the following four results areas: • Result 1: Strengthened organizational capacity to deliver quality family planning services to target groups 26 PSI-SIFPO: MIDTERM PROJECT EVALUATION • Result 2: Internal quality assurance standards and results quantified and disseminated to strength￾en FP performance at a global level • Result 3: Increased organizational sustainability of country level programs, including internal South to South support and technical assistance • Result 4: Gender-sensitive FP services targeting youth strengthened at a global level A special focus of the SIFPO project has been on capacity-building and systems strengthening within the recipient international family planning organization to build upon and leverage the organization’s exten￾sive network of country platforms of programming that are oftentimes funded by other donors. Capacity building areas include improving clinical and counseling quality; standardizing and sharing best practices, such as around social franchising or mobile outreach programs; improving and standardizing metrics, such as for equity; improving health management and information systems; testing new approaches for service delivery and creating efficiencies in existing approaches, such as for mobile outreach or clinic based approaches; creating a cadre of technical reproductive health leaders within the organization; and increasing evaluation and operations research skills of field staff.The vision is that by strengthening and streamlining procedures and systems of international FP organizations, these improvements will cascade down to local affiliates in developing countries, thereby strengthening and enhancing sustainability of these private sector partners to contribute to the overall health system. • All project activities are organized into two categories: core-funded and field-funded activities. • Core-funded activities include predominately element activities in family planning with some HIV/AIDS funding in the PSI agreement. Most field support funding is family planning, with a limited amount of funding in HIV/AIDS and maternal and child health for select integrated service delivery activities. This midterm PSI project evaluation should follow the Agency’s Evaluation Policy (2011). http://www. usaid.gov/evaluation/policy In addition, the evaluation should take into account relevant U.S. Government/USAID initiatives, policy developments, and reform efforts, such as the U.S. government Global Health Initiative (GHI) especially the GHI principles, the Global Health Strategy for the GH Bureau and USAID FORWARD. The GHI principles are linked to the Agency’s commitment to Paris Declaration aid effectiveness principles such as alignment with country strategies and priorities, strengthening and use of health systems, new partner￾ships and innovations, and strengthened monitoring and evaluation for accountability and results. The GHI principles also include a focus on women and girls and integrated services.The GHI Goals for PRH are to: 1) prevent 54 million unintended pregnancies, 2) increase contraceptive prevalence by 2 percentage points each year, 3) reduce first births to women under 18 by 15 percent. Additional performance measures of the PRH Office are to: 1) increase the percent of births spaced more than three years apart, PSI-SIFPO: MIDTERM PROJECT EVALUATION 27 2) reduce the percent of births order 5 or higher, and 3) increase the percent of demand satisfied through modern contraception. The PSI SIFPO project contributes to all of these objectives and also to the Millennial Development Goal (MDG) 4 to reduce child mortality, MDG 5a Improve Maternal Health, MDG 5b universal access to reproductive health and also to MDGs for poverty, education, environment, gender, and HIV/AIDS. The midterm evaluators should also take into account the FP 2020 Summit follow-up actions as they relate to this PSI project and any design for any potential future project.The goal of FP2020 is to reduce unmet need and enable 120 million women and girls to use contraceptives by 2020. USAID’s GH prior￾ity A Promise Renewed also strives to decrease maternal mortality and infant and under five mortality. Family planning service delivery is a key intervention to achieve both of these goals. VI. SCOPE OF WORK (SOW) The technical evaluation will focus strategically on big picture and overarching questions as well as four of the PSI project’s technical areas. Big picture and overarching questions can be divided into the follow￾ing two categories: (1) questions about the existing PSI project, and (2) questions relevant to the design of potential future project(s).The three technical areas that will be evaluated are: (1) The evaluation will examine the inputs of core resources intended for strengthening each recipient organization as a whole, with the intention that those inputs would strengthen country level platforms, and their effect on im￾proved organizational capacity, sustainability as well as performance and management. (2) The evaluation should provide insight into what additional support and strengthening is needed for improved sustainabil￾ity at the country level. (3) In addition, while PSI project level output data and service delivery statistics reveal high quality performance on FP/RH service delivery to date, an external examination will contrib￾ute to validating those findings and suggesting areas for improvement or strengthening in FP/RH service delivery and quality assurance. Big Picture and Overarching Questions Questions for Evaluation of SIFPO 1. How satisfied have various stakeholders been with the work done by and assistance from SIFPO including the following? • Missions (The PSI project management team will provide the evaluation team with the results of a Mission survey that has been recently done to help answer this question.) • Global Health Bureau • Other stakeholders, including other donors such as DFID 2. Questions relevant to potential future projects • What existing gaps and future technical directions/issues need to be addressed in the follow-on that are not currently being addressed in SIFPO • What kinds of inputs, specifically organizational strengthening, are no longer needed? 28 PSI-SIFPO: MIDTERM PROJECT EVALUATION Technical Evaluation Questions 1. Use of core resources for each recipient organization to strengthen country level platforms, and their effect on improved organizational capacity, sustainability as well as performance and manage￾ment. • What evidence exists that core resources invested in organizational strengthening have improved country level platforms and programming? • What has been the effect on organizational capacity? • Is there evidence that there is increased sustainability? • How have management practices within the organization been affected? • To what extent has the PSI project’s internal organizational strengthening activities contributed to improved capacity at the country level (including at local and affiliate NGOs, government service sites)? • What are the facilitators and barriers to achieving the intended results? • What is the quality of the trainings and tools used to roll out some of these organizational wide sys￾tem changes (based on the available evidence (for example, evaluations by the participants, including headquarters and field staff)? 2. Additional support and strengthening that are needed for improved sustainability at the country level. • What is the experience with the different approaches to achieve sustainable programming? (Please include governments contracting out for mobile outreach, different mobile outreach strategies, training public sector workers on the job, as well as social franchising approaches with vouchers in the analysis. Also please include cost share and each organizations leveraged and own funding in the analysis.) 3. Areas for improvement or strengthening FP/RH service delivery and quality assurance. • What evidence exists that stakeholders have found the PSI project’s service delivery effective? • How effectively is the PSI project partnering and collaborating with other CAs and global partners involved with service delivery activities? • Is there evidence that the internal quality assurance standards have been disseminated to strengthen FP service delivery and performance at a global level? • To what extent is the PSI project’s portfolio of service delivery activities meeting the needs of stake￾holders? PSI-SIFPO: MIDTERM PROJECT EVALUATION 29 VII. METHODOLOGY Data Collection The evaluation team will work collaboratively with the USAID management team to develop a detailed work plan as well as a data collection strategy including data collection instruments. For the technical evaluation, it is envisioned that a select number of countries (approximately two) with moderate to high investments/money and time would be selected for field visits and the three technical areas of focus.The evaluation team will consult with and receive approval from the USAID SIFPO man￾agement team as to the selection of countries for field visits. The primary methodologies for this evaluation will include (1) document review, (2) in-depth key infor￾mant interviews, (3) focus group discussions, (4) surveys, and (5) direct observation. The specific methodologies for each of the evaluation areas are identified and described below; however, where feasible, methods should be combined to address multiple questions at once. 1. Document Review for PSI project: Big picture, overarching questions, and specific focus areas: • APS Solicitation document • Project agreement(s) • Semi- and Annual reports • Performance Monitoring Plan • Work plans • SOWs for field-funded activities • Results reporting (Mission & HQ) • Management review presentations and memos • U.S. Government Global Health Initiative (GHI) Strategy • Global Health Bureau GHI Strategy • USAID FORWARD reform agenda • Review “use” of products/methods/tools/papers, including website downloads and dissemination of products via CD, print copy, etc. • Participant evaluations of trainings, workshops, other country-level activities • Examination of the curricula and training objectives 2. Key Informant Interviews – in-depth semi-structured interviews, in person when possible (for example during country visits (Guatemala, Benin), at USAID Washington and at PSI headquarters office), alter￾natively via phone or video conference 30 PSI-SIFPO: MIDTERM PROJECT EVALUATION Big Picture and Overarching Questions: • Project staff, including those from the field and headquarters • BGH stakeholders • USAID Missions • USAID Washington staff • Project partner organizations and other CAs 3. Focus group discussions • Representatives of local ministries (clinical staff) • Project field staff and SIFPO staff • Clients 4. Surveys • Survey (email/web-based/phone) with USAID Missions that have used, and those that have not used, the project’s services (Some of this exists in the existing Mission survey but further follow-up may be needed as the response rate was low.) • Survey SIFPO staff • Survey both CAs and project staff on collaboration and communication • Survey key stakeholders—ask if their feedback was requested, if future interactions reflected any of the changes suggested • Email/web-based survey to community of practice participants, including LAPM working groups 5. Direct observation Big Picture and Overarching Questions: • Interview all relevant staff at PSI -- HQ and in the field (Guatemala, Benin). • Observe activities in countries for specific focus areas The USAID management team selected the two countries for field visit and direct observation, but we will require Mission concurrence for evaluation activities to occur in these sites. USAID/Guatemala has provided field support to the PSI SIFPO project since 2011, and represents the longest and largest field program of the PSI SIFPO project since its inception. Because of this, the USAID management team feels it is an important and relevant portion of the PSI SIFPO portfolio to understand and examine more deeply. Benin, while not a long country program, it has been in place for one year and will continue again this year, is representative of the type of programming that the PSI SIFPO project has been engaged in globally. Importantly, the Benin country program is the recipient of many of the core-led efforts to stan￾dardize and improve tools and approaches for social franchising, completed in the first two years of the PSI SIFPO project.The Benin in country program, in addition to the Guatemala programming, together provide a representative picture of the type of programming covered under the PSI SIFPO program.The PSI-SIFPO: MIDTERM PROJECT EVALUATION 31 table below illustrates the programmatic approaches present in each country program: TABLE 1: PSI SIFPO Programmatic Approaches by Country Guatemala Benin Type of programming Mobile outreach in public X facilities Public sector capacity building X Social Franchising with private X providers Social Marketing X IEC / BCC X X VIII. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) A. Team Composition The evaluation team will consist of two professionals that have demonstrated knowledge and experience in the areas described below. Depending on consultants identified, it is tentatively suggested that the team consist of the following professionals: One team leader with organizational development expertise and one clinical FP/RH specialist. It is expected that each of the skills and qualifications described below are covered in their entirety by the Evaluation Team; however, it is understood that specific skills may fall differently across each of the two job descriptions than what is listed below. Team Leader/Monitoring and Evaluation Specialist will oversee all aspects of the evaluation.The team leader will liaise with the other consultants and with USAID/GH, oversee data collection and analysis, write sections of the report, and meld contributions of the technical consultants into a coherent set of responses and present conclusions and recommendations to USAID.The team leader should have prior experience and expertise in program evaluation and assessment, understanding of USAID program pro￾cesses, and experience in monitoring and evaluation of global health programs. Qualifi cations include: • Track record of successful oversight of the evaluation of complex international technical assistance projects, preferably in health and family planning. • Excellent oral and written communication skills in English, including the ability to facilitate groups and present complex material • Demonstrated knowledge of USAID’s policies and priorities in PRH and other health experience working in developing countries • Background and experience in organizational development. Skills in designing qualitative and survey research instruments and methodologies • Knowledge of monitoring and evaluation in the area of international health (FP/RH or other health) • Must be available to travel to Guatemala and Benin 32 PSI-SIFPO: MIDTERM PROJECT EVALUATION FP/RH Clinical Services and Quality Specialist will have specialized evaluation experience and expertise in clinical programming in family planning in the international health and/or development sec￾tor.This individual will bring the lens of his/her subject matter expertise and experience to bear on all aspects of the Scope of Work. S/he will work closely with the team leader to assess the progress and quality, and relevance of approaches of the family planning activities of the project. S/he will work seam￾lessly with the team leader to interview key informants, conduct data collection and analysis, and write sections of the final report. Qualifications include: • Demonstrated ability to implement and evaluate FP/RH clinical service delivery programming and quality standards, in developing countries • Some understanding of integrated health programming (FP/RH and HIV and/or FP/RH and MCH) • Experience with evaluating different service delivery models and their potential sustainability at both the program and country level • Demonstrated ability to evaluate programming that serves underserved populations. • Must be available to travel to Guatemala and Benin B. Illustrative Level of Effort Table PSI-SIFPO: MIDTERM PROJECT EVALUATION 33 TABLE 2:Task LOE Tasks LOE Team Leader FP/RH Specialist Illustrative POP Document Review 5 days 5 days November 25 - 29 Remote TPM /Instrument Creation 2 days 2 days December 2 - 3 Instrument Review 2 days 2 days December 4 - 5 Instrument Revision 1 day 1 days December 6 Remote kick off meeting with USAID/ 2 day Washington 2 days December 11 - 12 Off n/a n/a December 16 – Januar y 3 US-Based Data Collection 4 days 5 days Januar y 4 – 8 Field-based Data Collection Guatemala 8 days 8 days Januar y 9 - 17 (includes inter views with key informants, field visits, and email/telephone sur veys) Field-based Data Collection – Benin (in­ 8 days 8 days Januar y 18 - 26 cludes inter views with key informants, field visits, and email/telephone sur veys) Data Analysis 4 days 4 days Januar y 27 - 31 Repor t Writing 5 days 5 days Februar y 3 - 14 In-person Debrief with USAID & PSI(with Travel)/First Draft Due 4 days 5 days Februar y 16 - 20 Travel Days 6 days 8 Days Assignment Ends n/a n/a Februar y 21 Total # of days (estimated) 51 55 November 20 – Februar y 21 A six-day work week is approved for in-country work. IX. LOGISTICS The USAID SIFPO Management Team will provide overall direction to the evaluation team, identify key documents and key informants, and liaise with PSI Headquarters staff and USAID Missions to ensure logistical support for field visits prior to the initiation of field work.The Evaluation contractor will be re￾sponsible for all travel arrangements, scheduling of meetings, translation services and secretarial support. The USAID SIFPO Management Team shall be available to the team for consultations regarding sources and technical issues, before and during the evaluation process. The USAID SIFPO Management team will reach out to relevant Missions to explain evaluation, confirm interest in the evaluation and in in brief and/or debrief meetings with the consultant team. Roles and Responsibilities: GH Tech will take the lead role in the following key items: 1. Consultant Conflict (COI) of Interest:To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CVs for proposed consultants, and provide additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates. 2. Documents: Identify and prioritize background materials for the consultants and provide them, prefer￾ably, in electronic form. 3. Site Visit Preparations: Provide a list of site-visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. 4. Lodgings and Travel: Handle all logistics for international travel. Provide guidance on recommended secure hotels and methods of in country travel (i.e., car rental companies and other means of trans￾portation) and identify a person to assist with logistics (i.e., visa letters of invitation etc.) 5.Work closely PSI to develop and finalize the in country schedule and logistics. 34 PSI-SIFPO: MIDTERM PROJECT EVALUATION During Field Work in the US PSI will take the lead in the following activity: 1. Prepare a schedule and agenda for the evaluation team to be briefed on the PSI SIFPO project, bring￾ing in relevant staff to brief evaluation team on key activities, country programs, and overall informa￾tion on PSI organizational structure and function. 2. Facilitate contact with sub awardees on the SIFPO project or other so evaluators can meet with them to discuss project activities. During Field Work in Countries (Guatemala, Benin) PSI will assist the Evaluation Contractor and the consultant team to identify and arrange: 1. Formal and Official Meetings. Arrange key appointments with national and local government officials. Where applicable and appropriate, prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. Arrange for translation services, if needed. 2. Assist in arranging a time for in-brief and/or debrief with relevant USAID mission staff in countries. USAID SIFPO Management team will have briefed the Mission in advance and will have secured com￾mitment to and concurrence for the evaluation. 3. Logistics for local in country travel. Assist with logistics and recommendations, and reservations for safe, local lodging. Provide local transport to sites if needed, to be reimbursed by Consultant team. 4. Other Meetings. If appropriate, assist in identifying and helping to set up meetings with other local stakeholders or contacts relevant to the assignment. Arrange for translation services, if needed. After Work: The USAID SIFPO Management team and PSI SIFPO leadership will provide: Timely Reviews. Provide timely review of draft/final reports. USAID will ensure timely approval of the deliverables at hand. X. DELIVERABLES AND PRODUCTS • Workplan and data collection instruments: The evaluation team will prepare a detailed work plan in response to SOW requirements and evaluation questions.The detailed work plan should identify the countries for site visits, the individuals and stakeholders for surveys and in-depth inter￾views and should include each of the proposed data collection instruments (i.e. structured interview guides, surveys, observation forms, etc.). A draft of the detailed work plan and data collection instru￾ments should be submitted to the SIFPO Management Team for input prior to finalization. • First Draft Report: This report should describe the preliminary findings from the technical evalua￾tion as well as findings related to the big picture and overarching issues spanning the evaluation.The report should separately and comprehensively address each of the objectives and questions listed in the Statement of Work as well as the findings, interpretations, conclusions, and recommendations which should be clearly supported by the collected and analyzed data. Findings should be presented graphically where feasible and appropriate using graphs, tables and charts.The report should make recommendations for future action, including recommendations that may be relevant to the imple￾mentation of the second half of the existing PSI project as well as for potential future project(s) in either technical and/or managerial aspects. PSI-SIFPO: MIDTERM PROJECT EVALUATION 35 o The draft report will not be copy-edited, formatted, made 508 compliant or posted to the DEC for public consideration. o The report should not exceed 40 pages in length (not including appendices, list of contacts, etc.). It should contain a draft executive summary, table of contents, main text including findings, initial conclusions, and recommendations. Annexes should include the Scope of Work, description of the methodology used, lists of individuals and organizations consulted, data collection instru￾ments (i.e. questionnaires and discussion guides etc.), and bibliography of documents reviewed. The executive summary should accurately represent the report as a whole and should not exceed two pages in length. o Evaluation findings should focus primarily on those issues within USAID technical and manage￾ment staff ’s manageable interest. Discussion of those issues that are outside of the realm of influ￾ence of these staff such as, but not limited to, issues of a political nature, funding constraints or limitations with the Global Health/Child Survival Account, and so forth, should be reserved for, if at all, the limitations section. • Debrief Presentation(s) and Power Point:The draft report is to be accompanied by a PowerPoint presentation that aims to debrief selected stakeholders of the results and recommendations stem￾ming from the midterm evaluation. A draft of the Final Presentation should be submitted to the SIFPO management team prior to finalization. A version of this presentation should be presented to PSI, as an in-person debrief and sharing of key findings from the evaluation. XI. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the evaluation team and will undertake the following specific re￾sponsibilities throughout the assignment: • Recruit and hire the evaluation team. • Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: Before Field Work • SOW. Respond to queries about the SOW and/or the assignment at large. • Consultant Conflict of Interest (COI).To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional in￾formation regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. • Documents. Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment. • Local Consultants. Assist with identification of potential local consultants, including contact information. 36 PSI-SIFPO: MIDTERM PROJECT EVALUATION • Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. • Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation). During Field Work • Mission Point of Contact.Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. • Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). • Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. • Facilitate Contact with Implementing Partners. Introduce the evaluation team to implementing part￾ners and other stakeholders, and where applicable and appropriate prepare and send out an intro￾duction letter for team’s arrival and/or anticipated meetings. After Field Work • Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. This evaluation will be a participatory external review, in the sense that the GH Tech [or other mecha￾nism] evaluation team will work collaboratively with the USAID management team throughout the dura￾tion of the evaluation. The evaluation team will consult with the USAID SIFPO management team regarding the methodology, approach, and data collection instruments, but will be primarily responsible for data collection, analysis, and report writing. XII. MISSION AND/OR WASHINGTON CONTACT PEOPLE/PERSON USAID Management team points of contact: Marguerite Farrell, AOR; Elaine Menotti, Senior Technical Advisor SIFPO PSI Project point of contact: Jennifer Pope Mission points of contact: Guatemala: Yma Alfaro, Eric Janowsky Benin: Michelle Kouletio, Milton Amayun, Cheryl Combest XIII. COST ESTIMATE (EXCLUDED) XIV. REFERENCES (PROJECT AND RELEVANT COUNTRY DOCUMENTS) • Guidelines for Management Reviews and Project Evaluations, Bureau of Global Health, 2007 PSI-SIFPO: MIDTERM PROJECT EVALUATION 37 • USAID Evaluation Policy, 2011 • APS • Project Proposals • Cooperative Agreements • Project Workplans (years 1-3) • Project Semi- and Annual Reports • PMPs • SOWs for field-funded activities • Trip Reports • Financial Reports • Management review memo and presentations • Checklist for Evaluation Reports • U.S. Government Global Health Initiative (GHI) Strategy • USAID FORWARD reform agenda • FP 2020 Summit website • Global Health GHI Strategy • Project papers and case studies APPENDIX A: USAID Criteria to Ensure the Quality of the Evaluation Report • The evaluation report should represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not, and why. • The evaluation report shall address all evaluation questions included in the Statement of Work (SOW). (Although the report should not answer each question directly in the report but should thematically and in an integrated fashion in the narrative, address the evaluation questions.) • All modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the AOR. • The evaluation methodology shall be explained in detail and all tools used in conducting evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the fi nal re￾port. 38 PSI-SIFPO: MIDTERM PROJECT EVALUATION • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limita￾tions associated with the evaluation methodology. • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. APPENDIX B. KEY INFORMANTS TBD with evaluation team PSI-SIFPO: MIDTERM PROJECT EVALUATION 39 ANNEX II. PERSONS INTERVIEWED 40 PSI-SIFPO: MIDTERM PROJECT EVALUATION USAID/Washington Maggie Farrell SIFPO PSI AOR Elaine Menotti SIFPO PSI technical advisor Andrew Park SIFPO PSI program assistant Michal Avni Gender specialist Kathr yn Panther GH/PRH/SDI division chief Trish MacDonald LARC/LAPM champion Carolyn Cur tis Post-abor tion care champion Liz Bayer Policy fellow Ellen Starbird Office director, PRH Jim Shelton Senior science advisor Jeff Spieler Senior Advisor for FP/RH Shelley Snyder Technical Advisor, Benin Countr y Representative Mar y Vanderbroucke Technical Advisor, Guatemala Countr y Representa￾tive Alex Todd-Lippock Repositioning FP Advisor Shawn Malarcher Technical Advisor, Benin Countr y Representative Diana Santillan Gender Advisor PSI Washington Karl Hoffman President and CEO Jen Pope Director, SIFPO Marie-Laure Curie Deputy Director, Learning and Performance Dr Krishna Jafa Vice_president SRHT Dr Paul Blumenthal Global Medical Director Maxine Eber SIFPO, Deputy Director Christine Bixiones SIFPO,Technical Advisor Julie McBride Senior Social Franchising Fellow Rober t Dribbon Former Merck Fellow Mariah Preston SIFPO Program Manager Dr Kim Longfield Research and Metrics Director Dr Nirali Chakrabor ty RH Research Advisor Daniel Messer Global Business Systems Director Rene Greifinger SRHT Technical Advisor Ashley Jackson SIFP Technical Advisor Judi Heichelheim LAC Regional Director Marcie Cook Asia and EE Regional Director USAID/Guatemala Yma Alfaro USAID Alternate Agreement Officer Representative (by phone) PSI Guatemala Pilar Sebastian: SIFPO/PlanFam COP/PSI Countr y Representative Dr Morales Medical Director, PlanFam Norber t de Anda RH/FP Technical Advisor Dr. Rossana Cifuentes Inter-Institutional Coordinator, PlanFam Dr. Carlos Leonel Gomez SIFPO/PlanFam Medical Coordinator Amilcar Rivera SIFPO/PlanFam Community Coordinator Karen Steel SIFPO/PlanFam M&E and Research Manager Haydee Lemus SIFPO/PlanFam Communication Specialist Franchisees CHEs Outreach team CHWs Clients Other Guatemala Miriam Lopez APROFAM, Social Program Manager Mirna Montenegro Obser vatorio de Salud Reproductiva Eric Rivas Consejo Nacional de la Juventud Snra Mar tinez Secretaria contre la Violencia Explotacion y Trata de personas Ludy Rodas Ministr y of Health Carlos Contreras Ministr y of Health Julieta Flores Ministr y of Health Erik Alvarez Ministr y of Health Dr Molina Ministr y of Health Gustavo Batres Ministr y of Health Dr Nehemias Santizo Area health Director , San Marcos, MoH Mrs Silvia Jaurez, RN Programs Director, San Marcos, MoH Dr Juan Pablo Velazquez Technical Advisor, San Marcos, MoH San Marcos Depar tmental Committee San Marcos Local PlanFam Team MoH Providers (San Marcos) APROFAM mobile outreach team Community health Workers Clients USAID/Benin Kevin Armstrong Mission Director, USAID ,Benin PSI Benin Megan Wilson Countr y Director, ABMS/PSI Benin Dr Margeurite Ndour Medical Coordinator, ABMS/PSI Benin PSI-SIFPO: MIDTERM PROJECT EVALUATION 41 Cyprien Zinzou Research Coordinator, ABMS/PSI Benin Ghyslain Guedegbe Data Analyst, ABMS/PSI Benin Dr Moutiatou Toukourou Tidjani Director of “l’Abattoir’ Pharmacy, Member CA,, ABMS ABMS Warehouse Manager Other Benin Dr Hounkpatin Ministr y of Health Dr Justin S. AKOHA AGBATODE, Zone Coordinator(Zones 1 and 2, Cotonou) , MoH ABMS/PSI Mobile Clinic staff MoH providers Franchisees in Cotonou and Por to Nuovo Community health Workers Clients Amour & Vie Youth Center staff, Abomey-Calavi, Cotonou 42 PSI-SIFPO: MIDTERM PROJECT EVALUATION ANNEX III. SOURCES OF INFORMATION Bixiones, C., and Pope, J. (2013). Positioning local leaders to reposition family planning:The importance of lead￾ership and management skills for improved health impact. Poster presented at the International Conference on Family Planning, Addis Ababa, Ethiopia. November 12-15 2013. Bloom, S., Fortune-Greeley, H. and Kincaid, M. (no date). GBV Research Guidelines: Considerations for con￾ducting research on gender-based violence. Washington, DC. PSI. Blumenthal, P., Shah, N., Jain, K. et al. (2013). Revitalizing long-acting contraceptives in settings with high unmet need: a multi-country experience matching demand creation and service delivery. Contraception, 87: 170-175. Chakraborty, N., Firestone, R., Bellows, N. (2013). Equity monitoring for social marketing: use of wealth quintiles and the concentration index for decision making in HIV prevention, family planning, and malaria programs. BMC Public Health, 13(Suppl 2): S6 Chakraborty, N., Fry, K., Sudhinaraset, M. (2013). Measuring equity – without tears. Presentation at the Private Sector in Health iHEA pre-conference symposium, Sydney, Australia. July 6th 2013. IntraHealth International and Support for International Family Planning Programmes (2013). Enquete d’acceptabilité et de faisabilité de la vaséctomie sans scalpel au Benin:Technical Report. September 2013. Murphy, C., and Chakraborty, N. (2013). Understanding Client Satisfaction and Perceived Quality of Care within Reproductive Health (RH) Services: A Literature Review. Washington, DC. PSI. PASMO/PSI (2014). Resultados de proyecciones de planificacion familiar, das San Marcos. Powerpoint presen￾tation. 2014. Population Services International (2009). Implant Delivery Service Protocols. In: PSI Quality Assurance Manual. May 2009.Washington, DC. PSI. Population Services International (2009). IUD Service Delivery Protocols. In: PSI Quality Assurance Manual, Part 3. February 2009.Washington, DC. PSI Population Services International (2011). Female Sterilization Service Delivery Protocols. In: PSI Quality Assur￾ance Manual. November 2011.Washington, DC. PSI. Population Services International (2011). Profile: PSI Reproductive Health Leader. Washington, DC. PSI. Population Services International (2011). PSI Quality Assurance Auditor Training for 28th February to 3rd March 2011, final agenda. Washington, DC. PSI. Population Services International (2011). PSI Reproductive Health Leadership Training Schedule for 5th July – 5th August 2011, Washington DC. PSI. Population Services International (2011). Reproductive Health Leadership Training 2011, Needs Assessment for Participants. Washington, DC. PSI. Population Services International (2011). Reproductive Health Leadership Training Program. Washington, DC. PSI. Population Services International (2011). Social Franchise Case Study: Madagascar’s Top Réseau Network. PSI-SIFPO: MIDTERM PROJECT EVALUATION 43 Washington, DC: PSI. Population Services International (2011). Total Market Approach to Family Planning in Cambodia: A Case Study. Washington, DC: PSI. Population Services International (2011). Vasectomy Service Delivery Protocols. In: PSI Quality Assurance Manual. August 2011. Population Services International. November 2011.Washington, DC. PSI. Population Services International (2012). ProFam Urban Outreach: A High Impact Model for Family Planning. Washington, DC: PSI. Population Services International (2012). Social Franchising: Strengthening Local Capacity to Deliver Better Health Care. Washington, DC. PSI. Population Services International (2013). OC & Injectable Protocols with Community-based Distribution Guide￾lines. In: PSI Quality Assurance Manual. February 2013.Washington, DC. PSI. Population Services International (2013). PSI Community-Based Access to Injectables (CBA21) Advocacy and Program Guide. Washington, DC. PSI. Population Services International (2013). Social Franchising Training 2013, Needs Assessment for Participants. Washington, DC. PSI. Population Services International (2014). Making your Health Services Youth Friendly: A guide for program planners and implementers. Washington, DC. PSI. Population Services International (no date). Global Level Performance Monitoring Plan for SIFPO. Washington, DC. PSI. Population Services International (no date). Independent QA Audit: Guidelines and Tools for PSI Programmes. Washington, DC. PSI. Population Services International (no date). PBCC Toolkit: Capacity Building Resources. Washington, DC. PSI. Population Services International (no date). Provider Behaviour Change Communications Program – Self As￾sessment Tool. Washington, DC. PSI. Population Services International (no date). QA Audit Report Matrix (Excel spreadsheet). Washington, DC. PSI. Population Services International (no date). QA Audit Tools. Washington, DC. PSI. Population Services International (no date). Reproductive Health Service Delivery Standards. Washington, DC. PSI. Population Services International (no date). Reproductive Health Leadership Mentoring Program. Population Services International. 2012-13. PSI. Population Services International (no date). RH Client Satisfaction and Perceived Quality Questionnaire Guide. Washington, DC. PSI. Population Services International (no date). Social Franchise Leaders’Training Schedule for 18th April - 3rd May, Washington DC. PSI. 44 PSI-SIFPO: MIDTERM PROJECT EVALUATION Population Services International (no date). Social Franchising Leader Profile. Washington, DC. PSI. Population Services International (no date). Social Franchising: Business in a Box. Washington, DC. PSI. Population Services International (no date). The DELTA Companion: Marketing Planning Made Easy. Wash￾ington, DC. PSI. Population Services International (no date). WC A QA Auditor Training Programme for 22nd-26th October, no year given. Washington, DC. PSI. Population Services International and IntraHealth (2013). Summary Report: Assessment of the acceptability and feasibility of non-scalpel vasectomy in Benin. November 2013. Washington, DC. PSI. Population Services International and Support for International Family Planning Associations (2014). SIFPO Semi-Annual Year 4 Report. January 2014. Washington, DC. PSI. Shah, N., Fry, K., Sudhinaraset, M. (2012). Piloting the equity assessment of social franchises. Presentation at the 2nd Global Symposium on Health Systems Research, Beijing, China. 31st October-3rd November 2012. Shah, N.M. (2011). Review of PSI research on private sector providers: Key findings, lessons learned and recom￾mendations. Washington, DC: PSI. The RESPOND Project. 2013. Bringing Long-Acting Reversible and Permanent Contraceptive Methods and Services Closer to the Client: Innovative Approaches—Meetings Highlights. New York:The RESPOND Project/ EngenderHealth. PSI-SIFPO: MIDTERM PROJECT EVALUATION 45 ANNEX IV. DATA COLLECTION INSTRUMENTS PSI Country Questionnaire Question Guidelines for Partners Questionnaire for USAID Missions in SIFPO Countries Questionnaire for Voucher or Outreach Clients Questionnaire for Community Educators FP Clinical Checklists SUPPORT FOR INTERNATIONAL FAMILY PLANING ORGANISATIONS (SIF￾PO) MIDTERM PROJECT EVALUATION – POPULATION SERVICES INTERNA￾TIONAL You are being contacted in order to answer some brief questions which will assist with the midterm evaluation of the PSI-SIFPO program and help to build additional knowledge about family planning ser￾vices.The overall purpose of the evaluation is: • To assess the PSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement • To gather information that will help to improve the management of the PSI project for the remainder of its implementation • To gather information that will result in useful recommendations for a potential future project • All questions posed in the questionnaire will assess the following results categories: Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south￾to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level It would be helpful if your answers could reflect the overall expected program results. It is kindly requested that you answer all of the questions proposed completely providing sufficient detail for analysis.Your written comments can be made in the text boxes provided. Please feel free to add extra comments at the end of the questionnaire. This questionnaire is for evaluation purposes and was commissioned by USAID, therefore review by an institutional review board was not required. Results of the questionnaires will be shared and will be presented in a format that protects the specific identity of respondents while providing country-specific 46 PSI-SIFPO: MIDTERM PROJECT EVALUATION information. Responding to the survey is voluntary and implies consent. The questionnaire will take about 25 minutes to complete.We are asking that you complete the question survey by July 5, 2013, at the close of business. If you wish to have a more detailed discussion via tele￾phone or Skype, please provide contact details and indicate this on your returned survey. Thank you for your assistance with this evaluation. Regards, Sarah Castle, PhD, and Pandora Hardtman, RN, CNM, DNP QUESTIONNAIRE FOR PSI COUNTRY OFFICES Name of respondent…………………………………………………… Email of respondent……………………………………………………. Telephone number of respondent………………………………….….. Position of respondent………………………………………………… Please note that the above details will be kept confi dential DEMOGRAPHIC DATA Country Table Please tick Cambodia Ethiopia Ghana Madagascar Pakistan UK USA South Sudan Tanzania Zimbabwe Please indicate which organization you represent: USAID Mission PSI Countr y Office PSI-SIFPO: MIDTERM PROJECT EVALUATION 47 Q1. How satisfied are you and your team with PSI’s work in the following areas? Mobile Clinic Outreach Social Franchising FP Vouchers Youth FP/HIV Integration Gender-based Violence Very Satisfi ed Satisfi ed Not Satisfied Please expand upon your response Q2. How have SIFPO core resources impacted upon PSI’s organizational capacity and sustainability as well as on the performance and management of your country program? Please give detailed examples. Q3.What evidence exists that core resources invested by SIFPO in the organizational strengthening of PSI and its local partners have improved country-level platforms and programming? Please give specific examples from your country. Q4. How have management practices within PSI been affected by SIFPO? (For example, positively or negatively?) Please give detailed examples to support your response. Q5. In your country, to what extent has PSI’s internal organizational strengthening activities (carried out within the SIFPO framework) contributed to improved capacity, including that of local and NGOs and government services? Please give detailed examples to support your response. Q 6. Based on the available evidence, what is your assessment the quality of the trainings and tools? Tools and trainings are: High Average Poor Please give examples of tools and trainings and quality quality quality explain the reasons for your assessment Staff continual professional development Staff recruitment and retention Data management and use Monitoring and evaluation 48 PSI-SIFPO: MIDTERM PROJECT EVALUATION Quality assurance Evidence-based clinical protocol implementation Post-training par ticipant obser vations Procurement and logistics Other (please specify) Q7. Under SIFPO, has the quality, accessibility, and dissemination of training tools improved? YES NO Please give specific examples Quality Accessibility Dissemination Q8. Under SIFPO, has frequency of training improved? Yes No Unchanged Please give specific examples. Q9. Have the updates and the revision of tools been ongoing and improved? Yes No Please Give Specifi c Examples Q10.What additional support and strengthening is needed for the improved sustainability of SIFPO ac￾tivities at the country level? Please give specific examples. PSI-SIFPO: MIDTERM PROJECT EVALUATION 49 Q11. In your country programme, what kind of approaches best achieves sustainable programming? Please choose all that apply Best approach for Please expand upon your reasoning. How has SIFPO sustainable programming strengthened this approach? Yes No Mobile Outreach Strategies Social Franchising Public Sector on￾the-job training Other approaches￾please specify Q12. Please identify the best strategy to achieve sustainable programming in your country. Describe in detail any organizational and personnel challenges and how these were overcome. How did SIFPO help overcome these challenges? Please include details of cost-share between PSI’s leveraged and own funding? Q13. Has SIFPO improved quality assurance? Yes No Q14. If SIFPO has improved quality assurance, how was this achieved? If quality has not improved, why was this the case? How can quality assurance be improved further? Q15. Have PSI’s internal quality assurance standards been disseminated to strengthen FP service delivery and performance in your country? Yes No Q16. Please elaborate as to the effect of the dissemination of quality assurance standards and any changes brought about because of them. Q17. How effective is PSI’s portfolio of service delivery activities in meeting the needs of the Ministry of Health? Very Effective Effective Minimally Effective Please expand upon your response 50 PSI-SIFPO: MIDTERM PROJECT EVALUATION Q18.What evidence exists that the Ministry of Health has buy-in, support, and investment in the PSI/ SIFPO’s method of service delivery? Q19. How effective is PSI’s portfolio of service delivery activities at meeting the needs of local stakehold￾ers including community-based organizations? Very Effective Effective Minimally Effective Please expand upon your response Q20.What evidence exists that local community-based organizations have found the PSI project’s meth￾od of service delivery effective? Please give specific examples. Q21. If there were a follow-on to SIFPO, in your view, what programmatic gaps and future technical directions/issues would need to be addressed? Q22.What aspects of organizational strengthening (for example, with regard to training, quality assur￾ance, and M&E) provided by SIFPO to PSI and its local partners are no longer needed? Why are these aspects of organizational strengthening no longer needed? Q23. Additional comments, remarks, and recommendations regarding SIFPO related activities SUPPORT FOR INTERNATIONAL FAMILY PLANING ORGANISATIONS (SIF￾PO) MIDTERM PROJECT EVALUATION – POPULATION SERVICES INTERNA￾TIONAL You are being contacted in order to answer some brief questions which will assist with the midterm evaluation of the PSI-SIFPO program and help to build additional knowledge about family planning ser￾vices.The overall purpose of the evaluation is: • To assess the PSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement • To gather information that will help to improve the management of the PSI project for the remainder of its implementation • To gather information that will result in useful recommendations for a potential future project PSI-SIFPO: MIDTERM PROJECT EVALUATION 51 All questions posed in the questionnaire will assess the following results categories. Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south￾to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level It would be helpful if your answers could reflect the overall expected program results. It is kindly requested that you answer all of the questions proposed completely providing sufficient detail for analysis. This questionnaire is for evaluation purposes and was commissioned by USAID, therefore review by an institutional review board was not required. Results of the questionnaires will be shared and will be presented in a format that protects the specific identity of respondents while providing country-specific information. Responding to the survey is voluntary and implies consent. The questionnaire will take about 20 minutes to complete.We are asking that you complete the question survey in its entirety. Please respond to the following survey monkey link or open attachment to begin the questionnaire. If you choose respond via hard copy, 1) save document with your country name, 2) respond to questionnaire, and 3) resave document and forward to sarah@sarahcastle.co.uk. and phardt￾mancnm@gmail.com. If you wish to have a more detailed discussion with us via telephone or Skype, please provide contact details and indicate this on your returned questionnaire. Thank you for your assistance with this evaluation. Regards, Sarah Castle, PhD, and Pandora Hardtman, RN,CNM, DNP QUESTIONNAIRE FOR USAID MISSIONS IN SIFPO COUNTRIES Name of respondent…………………………………………..……….. Email of respondent……………………………………………………. Telephone number of respondent……………………………….……. Position of respondent………………………………………………… 52 PSI-SIFPO: MIDTERM PROJECT EVALUATION Please note that the above details will be kept confi dential. Country Table Please tick Cambodia Ethiopia Ghana Madagascar Pakistan UK USA South Sudan Tanzania Zimbabwe Please indicate which organization you represent: USAID Mission PSI Country Office Q1. How satisfied is your mission with the work done by SIFPO in the following areas? Mobile Clinic Outreach Social Franchising FP Vouchers Youth FP/HIV Integration Gender-based Violence Very Satisfied Satisfi ed Not Satisfied Please expand upon your response Q2.What evidence exists that core resources invested in PSI and local partner(s) organizational strengthening have improved country-level platforms and programming? Q3. Do you feel that additional support and strengthening is needed for the improved sustainability of SIFPO activities at the country level ? Yes No PSI-SIFPO: MIDTERM PROJECT EVALUATION 53 Q4. If yes, why? And for which activities? If no, why not? Please give specific examples Q5. In your country, how effective is PSI’s portfolio of service delivery activities in meeting the needs of the Ministry of Health? Very Effective Effective Minimally Effective Please expand upon your response Q6.What evidence exists of long-term Ministry of Health buy-in, support, and investment in the PSI/ SIFPO project’s method of service delivery? Please specify. Q7. How effective is PSI’s portfolio of service delivery activities in meeting the needs of local stakehold￾ers, including community-based organizations? Very Effective Effective Minimally Effective Not Effective Please give specific examples in relation to your response above. Q8. If there were a follow-on to SIFPO, in your view, what programmatic gaps and future technical direc￾tions/issues would need to be addressed? Q9.What aspects of organizational strengthening (for example, with regard to training, quality assurance, and M&E) provided by SIFPO to PSI and its local partners are no longer needed? Why is the organiza￾tional strengthening no longer needed? Q10. Please give a detailed example of a successful programmatic/service delivery aspect of SIFPO or of lessons learned in your country. Q11. Additional comments, remarks, suggestions, and recommendations 54 PSI-SIFPO: MIDTERM PROJECT EVALUATION QUESTION GUIDELINES FOR PARTNERS International and national representatives of technical partners You are being contacted in order to assist with the midterm evaluation of the PSI-SIFPO programming and build additional knowledge about family planning services.The overall purpose of the evaluation is to: • Assess the PSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement • Gather information that will help to improve the management of the PSI project for the remainder of its implementation • Gather information that will result in useful recommendations for a potential future project All questions posed about your technical collaboration/role as co-funder with SIFPO are open-ended and assess the following results subcategories: Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south￾to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level Your participation in the evaluation is being solicited via direct observation of clinical services or inter￾view. Results of the interview will be shared and will be presented only in aggregate form, thereby protecting the identity of respondents. Observations of clinical service delivery will in no way impact any work-re￾lated performance appraisals. Data will be de-identified and analyzed in the aggregate to assure confidentiality and maintain anonymity of those responding. Responding to the questions is voluntary and implies consent. Please fill in the attached set of questions and give as much evidence and information as possible to back up your statements.The phrases and questions in italics are intended to serve as guidelines for your responses. Please kindly return the form to sarah@sarahcastle.co.uk by xx/xx/xx (date). Please do not hesitate to contact us for further clarification if you have questions or if you would like to set up a Skype or telephone conversation to discuss SIFPO further. Thank you for your assistance with this evaluation. Regards, Sarah Castle, PhD, and Pandora Hardtman, RN,CNM,DNP PSI-SIFPO: MIDTERM PROJECT EVALUATION 55 QUESTIONS FOR PSI PARTNERS AND DONORS (EMAIL QUESTIONNAIRE ) 1. Please give an overview of your involvement with SIFPO? (Did the program meet your expectations? What was the aim of your organisation’s involvement with SIFPO? Did you achieve this aim? Did the nature or degree of your organisation’s involvement with SIFPO change over time? If so, why? 1. How has your organisation helped PSI at an institutional level through SIFPO support? (Please out￾line the nature of any central capacity strengthening and evidence for its effectiveness? 2. How has your organisation helped PSI country-level programmes through your input into SIFPO￾funded activities? (Please specify any activities that have been initiated, developed, or reoriented due to your collaboration with PSI via SIFPO) 3. Has your involvement in SIFPO resulted any changes in the way your own organisation operates or thinks? (Has the collaboration with SIFPO led to an increased awareness about family planning or service delivery? Will these new perspectives alter the way your own organisation operates in the future?) 4. Were there any difficulties or barriers with regard to your partnering/funding SIFPO? (Please de￾scribe any logistical, programmatic, or financial barriers? Was communication with PSI and USAID conducive to optimal collaboration?) 5. If SIFPO were to be replicated in the future, what recommendations would you make from the point of view of a partner/funder? (What could be done differently? What could be changed or dropped?) Donors only: 1. What are the advantages of SIFPO as a funding mechanism for your programme? (Please discuss any advantages that pertain to the aim and scope of SIFPO, financial deadlines, reporting procedures, policy and legislative issues, etc.) 2. What are the disadvantages of SFPO as a funding mechanism for your programme? (Please discuss any advantages that pertain to the aim and scope of SIFPO, financial deadlines, reporting procedures, policy and legislative issues, etc.) 56 PSI-SIFPO: MIDTERM PROJECT EVALUATION 3. What “added value” did SIFPO bring to your existing programmes via the co-funding mechanism? Dear Clinical Service Provider: You are being contacted in order to assist with the accurate midterm evaluation of the PSI-SIFPO programming and build additional knowledge about family planning services.The overall purpose of the evaluation is to: • Assess the PSI project’s performance to date and to assess whether or not the project’s activities are achieving the intended results as outlined in the agreement • Gather information that will help to improve the management of the PSI project for the remainder of its implementation • Gather information that will result in useful recommendations for a potential future project All questions posed or service delivery procedures observed will assess the following results subcatego￾ries: Result 1: Strengthened organizational capacity to deliver quality FP services to target groups Result 2: Internal quality assurance standards and results quantified and disseminated to strengthen FP performance at a global level Result 3: Increased organizational sustainability of country-level programs, including internal south￾to-south support and technical assistance Result 4: Gender-sensitive FP services targeting youth strengthened at a global level Your participation in the evaluation is being solicited via direct observation of clinical services or inter￾view. The interview or observation is for evaluation purposes and was commissioned by USAID, therefore review by an institutional review board was not required. Results of the interview will be shared and will be presented only in aggregate form, thereby protecting the identity of respondents. Observations of clinical service delivery will in no way impact any work-related performance appraisals. Data will be de￾identified and analyzed in the aggregate to assure confidentiality and maintain anonymity of those responding. Responding to the questions is voluntary and implies consent. The observation or interview process will take approximately 60 minutes of your time. Thank you for your assistance with this evaluation. Regards, Pandora Hardtman, RN, CNM, DNP PSI-SIFPO: MIDTERM PROJECT EVALUATION 57 Clinical Service Delivery Based Potential Guided Questions 1. How long have you been with PSI as a clinical service provider? 2. What is your title within the organization? 3. *Have you undergone in-service training in the last six months? On what topics? What is your expe￾rience with the quality of the training received? Has any of this training been received or provided in conjunction with MOH employees? 4. In your opinion, where does the MIS-FP program need to improve service delivery? 5. What are the priority clinical service delivery needs? (i.e., skills, staff, patient related) 6. What are your thoughts about client load and the ability to thoroughly counsel your clients? 7. What are the facilitators to clinical service implementation? 8. What are the barriers to clinical service program implementation? 9. +Tell me more about the relationship of PSI with the MOH? Other collaborating agencies? 10. What would you change/do differently regarding clinical service delivery mechanisms? 11. +In brief, tell me basics about what you know about USAID FP legislation? 12. Do you think SIFPO funding has improved clinical service delivery? 13. What are the lessons learned from SIFPO and recommendations for the future? 58 PSI-SIFPO: MIDTERM PROJECT EVALUATION 14. +*Are all methods of FP readily available on client request? Including equipment and supplies? (most of the time, often, not at all) 15. +*Have you experienced stock-outs in the last 6 months? How many? 1-3 4-6 7-10 16. *Adapted from USAID family planning sustainability checklist 17. + Adapted HIP Physician/Nurse/Midwife-specific Guided Open-ended Questions *Tell me about your experience with FP task shifting and supervision. +Are you able to practice full-scope client care in the context of the cultural and legal regulatory frame￾work in-country? +Client-specifi c questions 1. +Were you previously on a method of FP prior to your visit to PSI? 2. +Had you chosen a method prior to your current visit? 3. +Did you receive counseling or information on the following? OCP LARC/LAPM BARRIER METHODS HIV TESTING/COUNSELING GBV Side effects of chosen method Informed consent for method chosen Yes No Do Not Recall Additional Comments PSI-SIFPO: MIDTERM PROJECT EVALUATION 59 4. +Do you think that all FP methods are accessible and affordable thru PSI-SIFPO? Service Delivery Observational Checklist Observed=Yes Not Observed=No Infection Control Bio–hazard disposal procedures Hand washing Sterile techniques or clean techniques (as applicable) Personal protective equipment (PPE) Case 1 Date Site Case 2 Case 3 Case 4 Case 5 Date Site Date Site Date Site Date Site Fixed or Mobile Clinic Setting/Infrastructure Observed=Yes Not Observed=No *Clinical guidelines/reference materials available, including PSI, organizational, and MOH Case 1 Date Site Case 2 Case 3 Case 4 Case 5 Date Site Date Site Date Site Date Site *Job aids/algorithms available for reference and client management, i.e., contraceptive eligibility wheels Written or posted information available on GBV Written or posted information on HIV Written or posted information on FP method—Mix 60 PSI-SIFPO: MIDTERM PROJECT EVALUATION Clinical Service Delivery and Family Planning Counseling Checklist Observed=Yes Not Observed=No Appropriate introductions/re￾spectful care Counseling at appropriate time Privacy/confi dentiality ensured Communication skills effective Case 1 Date Site Case 2 Case 3 Case 4 Case 5 Date Site Date Site Date Site Date Site Patient-specific data collection – demographic, medical eligibility Contraceptive histor y assessment Contraceptive method Informa￾tion given re full method mix Assessment of mitigating factors, i.e., religion, social, fears concerns Screening for co-morbid condi￾tions, i.e., risk for GBV, HIV, and ability to offer first-line suppor t as indicated Non-coercive assistance with method choice Referral for medical examination if indicated Side effects management and follow-up Functional referral system for intersectoral collaboration (CBO, NGO, MOH) Integrated ser vices PSI-SIFPO: MIDTERM PROJECT EVALUATION 61 QUESTIONNAIRE FOR THE COMMUNITY HEALTH EDUCATORS (CHES) Informal In-depth Interview Name Location Age Sex Educational level Duration they have been a CHE 1.0 Background 1.1 What motivated you to become a CHE? How were you chosen? 2.0 Training 2.1 Please describe the training you received. (when did it happen, duration, topics, numbers of train￾ees, role play, practical experience) 2.2 What was the best bit of the training? 2.3 What did you learn that was new? 2.4 Which part of the training was most useful in your work as a CHE? 2.5 Which part was least useful? 2.6 What was not included in the training that you would have liked to see included? 2.7 Have you had any opportunities for retraining (in-service training)? If so, please describe (Have you requested any?). 2.8 What topics would you like some more training about? 3.0 Work as a CHE 3.1 Please describe your average day as a CHE. 3.2 What are the enjoyable parts of your work? 3.3 What are the difficult parts of your work? 3.4 How do you carry out your sensitization? (Where do you meet people, what materials do you have?) 3.5 How are you received in the communities? (most receptive and most non-receptive community members?) 3.6 Who are the groups you sensitize? (married women, unmarried women, youth, men) 3.7 Are there groups you would like to sensitize but currently do not reach? 3.8 When you do sensitizations, do you ever talk about HIV (why or why not, how, what do you do if someone needs HIV testing?) 62 PSI-SIFPO: MIDTERM PROJECT EVALUATION 3.9 Do you ever talk about gender-based violence? (What do you do if you learn that someone has experienced GBV?) 3.10 Has SIFPO (USAID) changed anything about the way that you work? 4.0 Sensitization around family planning 4.1 How do you assess a woman’s family planning needs? 4.2 Please describe exactly how you present family planning to women (for spacing, stopping, health, better employment, and education opportunities). 4.3 Which methods do you present first? Which do you present last? 4.4 Do you have all methods available to show her? 4.5 Are there any methods you think are better than others? (for older women, for younger women) 4.6 Do women come with ideas what methods they want to use? (What is their choice based upon?) 4.7 How do you present the voucher system? 4.8 What are the advantages of the voucher system? 4.9 What are the disadvantages of the voucher system? 4.10 How could the voucher system be improved? 4.11 What is your relationship like with the BlueStar provider you work with? 4.12 How could this relationship be improved? 5.0 Remuneration 5.1 Please tell me how much you earn each month and where this money comes from? (salary, in￾centives, etc.) 5.2 Please tell me how the voucher system is linked to your monthly income. 5.3 What other financial or material benefits do you receive from your work as a CHE? 5.4 Has SIFPO (USAID) changed anything about the way you are paid? 6.0 Comment and recommendations 6.1 About BlueStar 6.2 About family planning methods and service delivery 6.3 About vouchers 6.4 About future directions PSI-SIFPO: MIDTERM PROJECT EVALUATION 63 QUESTIONNAIRE FOR VOUCHER/OUTREACH CLIENTS ABOUT FP CONSULTATION/SENSITIZATION Informal In-depth Interview Name Location Age Educational level Parity 1.0 Sensitization around family planning 1.1 How did you first hear about family planning? 1.2 How did you first hear of BlueStar/MSM outreach? 1.3 What are the benefits of family planning (for spacing, stopping, health, better employment, and education opportunities) 1.4 What are the disadvantages of family planning? 1.5 Does your husband/partner support your family planning? 1.6 Does anyone give you different (conflicting) advice about family planning? (compared with that given by the CHE/outreach worker ) 2.0 Sensitization by CHE or MSM outreach worker 2.1 Before the sensitization session, had you already used family planning? 2.2 When you came into the sensitization session, was there a particular method you had in mind? 2.3 Did you end up using this method? (If so, why; if not, why not?) 2.4 Please describe the sensitization session. (In group? Singly? How could it have been approved?) 2.5 Which methods were presented to you first? Which were presented to you last? 2.6 During the sensitization, did you learn about some methods you had never heard of before? 2.7 Did they CHE/outreach worker have all the methods to show you? 2.8 Did she discuss some methods in more detail than others? 2.9 Did she recommend a specific method to you? (Which one? Why this method?) 2.10 Did you have any questions during the consultation? (Was she able to answer them?) 2.11 Do you think you will return to the CHE/outreach worker? (why/why not?) 2.12 How long do you think you will use your current method of contraception? 2.13 Why will you stop/switch? 2.14 How could the CHE/outreach worker sensitization be improved? 64 PSI-SIFPO: MIDTERM PROJECT EVALUATION For voucher users only: 2.15 How did the CHE present the voucher system to you? 2.16 What are the advantages of the voucher system? 2.17 What are the disadvantages of the voucher system? 2.18 Did the voucher system influence your choice of method? (Would you have chosen another method if there had not been a voucher?) 2.19 How could the voucher system be improved? 2.20 What is your relationship like with the BlueStar provider? 2.21 How could this relationship be improved? 3.0 Comment and recommendations 3.1 About BlueStar 3.2 About family planning methods and service delivery 3.3 About vouchers 3.4 About future directions PSI-SIFPO: MIDTERM PROJECT EVALUATION 65 ANNEX V: DISCLOSURE 66 PSI-SIFPO: MIDTERM PROJECT EVALUATION PSI-SIFPO: MIDTERM PROJECT EVALUATION 67 Disclosure of Conflict of Interest for USAID/GH Consultants Name Pandora Hardtman Tl tie Family Planning/Reproductive Health Specialist Organization GH Tech Bridge 4 Consultancy Position Award Number (contract or other Contract Number: AID-OAA-C-13-00113 instrument) USAID Project(s) Evaluated (Include PSl-SIFPO ( SIFPO 3) project name(s), Implementer name(s) and award number(s}, if applicable) I have real or potential conflicts of 0 Yes Ox No interest to disclose. If yes answered above, I disclose t he following facts: Real or potential conflicts of interest may Include, bul ore not llmi1ed to: 1. Close family member who is on employee of tfle USAID operating unit managing the projcct{s) being evaluated /Jr the lniplementlng organi1ation(s) whose profect(s) ore being evaluated. 2. flnoncial Interest that ls direct, or ls significant though indirect, In the lmplcmcntlng orgonizotion(s) whose projects ore bein(I evaluated or in the outcome of the evaluotlan. 3. Culfcnt or previous