i The STRENGTHENING HEALTH STRENGTHENING HEALTH OUTCOMES THROUGH THE PRIVATE SECTOR (SHOPS) FINAL PERFORMANCE EVALUATION July 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by consultants Pamela Putney, Nedjma Koval-Saifi, Huda Murad and Wisam Qarqash on behalf of Management Systems International, with the support of Integrated Solutions. STRENGTHENING HEALTH OUTCOMES THROUGH THE PRIVATE SECTOR (SHOPS) FINAL PERFORMANCE EVALUATION July 2015 USAID/Jordan Monitoring and Evaluation Support Project (MESP) Contracted Under AID-278-C-13-00009 Management Systems International Corporate Offices 200 12th Street, South Arlington, VA 22202 USA DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Acronyms ........................................................................................................................................................... ii Executive Summary .......................................................................................................................................... 1 Evaluation Purpose and Project Background .......................................................................................... 1 Evaluation Questions, Design, Methods and Limitations ..................................................................... 1 Selected Findings and Conclusions ........................................................................................................... 2 Recommendations ........................................................................................................................................ 7 Evaluation Purpose and Evaluation Questions ........................................................................................ 10 Evaluation Purpose .................................................................................................................................... 10 Evaluation Questions ................................................................................................................................ 10 Project Background ....................................................................................................................................... 12 Evaluation Methods and Limitations .......................................................................................................... 15 Findings and Conclusions ............................................................................................................................. 16 Findings: JAFPP ........................................................................................................................................... 16 Conclusions: JAFPP ................................................................................................................................... 20 Findings: Effectiveness ............................................................................................................................... 21 Conclusions: Effectiveness ....................................................................................................................... 32 Findings: Gender ........................................................................................................................................ 33 Conclusions: Gender ................................................................................................................................ 35 Findings: Sustainability ............................................................................................................................... 35 Conclusions: Sustainability ....................................................................................................................... 36 Findings: Learning ...................................................................................................................................... 37 Conclusions: Learning ............................................................................................................................... 39 Recommendations ......................................................................................................................................... 41 ANNEXES ....................................................................................................................................................... 44 ANNEX I: Evaluation Statement Of Work ................................................................................ 45 ANNEX I1: Evaluation Design Report ........................................................................................ 57 ANNEX III: Performance Monitoring Plan ............................................................................... 108 ANNEX IV: Contact List .............................................................................................................. 112 ANNEX V: Bibliography ............................................................................................................... 116 ii ACRONYMS AJA Al Aman Jordanian Association AMEP Activity Monitoring and Evaluation Plan AOR/COR Agreement Officer’s Representative/Contracting Officer’s Representative BCC Behavior Change Communication BoD Board of Directors CAT Critical-Appraised Topics CCA Circassian Charity Association CHW Community Health Worker CMIS Clinic Management Information System COCs Combined Oral Contraceptives CPR Contraceptive Prevalence Rate CYP Couple Years of Protection DCOP Deputy Chief of Party DO Development Objectives DOS Department of Statistics DHS Demographic and Health Survey DMPA Depot Medroxy Progesterone Acetate EBM Evidence-based Medicine FP Family Planning FCR Findings, Conclusions, Recommendations FGD Focus Group Discussion GOJ Government of Jordan GP General Practitioner GUVS General Union of Voluntary Societies HCAC Healthcare Accreditation Council HH Household HLC Al Hussein Labor Clinics HMIS Health Management Information System HPC Higher Population Council HR Human Resources HRM Human Resources Management HSS II Health Systems Strengthening II ICCS Islamic Charity Center Society IEC Information, Education, and Communication IFH Institute for Family Health (Noor Al Hussein Foundation) IP Implementing Partner IUD Intra-uterine Device JAFPP Jordan Association of Family Planning and Protection JCAP Jordan Communication Advocacy and Policy Activity JFDA Jordan Food and Drug Administration JHCP Jordan Health Communication Partnership JPA Jordan Pharmacists Association KII Key Informant Interview M&E Monitoring and Evaluation MESP Monitoring and Evaluation Support Program MOH Ministry of Health iii NGOs Non-Governmental Organizations NWD Network Doctor OCP Oral Contraceptive Pills POPs Progesterone-only Pills PPFP Post-Partum Family Planning PPP PSP Public Private Partnership Private Sector Project for Women’s Health QA Quality Assurance RH Reproductive Health SDP Service Delivery Point SOP Standard Operating Procedures SHOPS Strengthening Health Outcomes through the Private Sector STTA Short-term Technical Assistance TFR Total Fertility Rate UNRWA United Nations Relief and Works Agency 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND PROJECT BACKGROUND Jordan’s population growth rate is constraining the country’s ability to achieve socio-economic progress and maintain stability. If not curbed, the population will double by 2047. Increasing the use and demand for voluntary modern family planning and reproductive health services is critical to curbing population growth. USAID’s Strengthening Health Outcomes through the Private Sector (SHOPS) was a five-year, $38 million activity implemented by Abt Associates in collaboration with Banyan Global to address this issue. As a follow-on to USAID’s Private Sector Project for Women’s Health (PSP), SHOPS concluded implementation in September 2015. The purpose of the final performance evaluation of SHOPS is to support USAID decision-making for future program design and implementation, and for effective allocation of resources, particularly as they relate to investments in the Jordan Association of Family Planning and Protection (JAFPP) and with private sector doctors. The evaluation is intended to provide evidence-based analysis of how effective the project was in meeting its objectives of increasing the demand for, access to, and quality of family planning services; to assess the sustainability of project outcomes and practices; to identify factors contributing to outcomes and sustainability; and to provide recommendations on how to make future projects with similar objectives more effective and more sustainable. SHOPS began in 2010 with the primary goal of expanding the access, quality and utilization of Family Planning (FP) services by partnering with the private and non-governmental sectors. SHOPS interventions were designed to overcome the challenge of Jordan’s plateauing contraceptive prevalence and total fertility rates. According to SHOPS’ Cooperative Agreement, family planning challenges were to be addressed by SHOPS as follows:  Increasing the use of existing methods, particularly underutilized methods such as injectable contraceptives and implants;  Increasing the range of product options in the injectable/implant category;  Developing marketing and behavior change strategies to improve the acceptance of hormonal methods;  Maintaining and expanding current collaborative relationship with pharmaceutical companies while exploring new partnership opportunities; and  Removing medical barriers (provider bias). SHOPS worked closely with six grantee partners, building capacity in 78 service delivery points (clinics) across Jordan, four private hospitals, 300 network doctors, and 300 pharmacists, supported by a team of community health workers (CHWs) referring women to SHOPS’ family planning outlets. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The SHOPS evaluation employed multiple sources of data including desk review, in-depth interviews, focus groups, and clinic observations. Surveys were conducted with 133 network doctors, 141 pharmacists, and 375 CHW-referred clients of SHOPS family planning services. Pharmacists were selected at random within the north, center, and south regions. Focus groups were conducted with 19 2 clients and 25 CHWs in order to gain further perspectives. A purposive sample of 16 grantee clinics was selected to cover all grantee organizations in the north, south and central areas. In addition, JAFPP clinics were selected based on performance in cost recovery, representing the top three performers, three of the bottom five performers, and two average performers. Due to reasons of client confidentiality, the evaluation was able to access clients only through CHW records at CCA and GUVS. As such, client responses may be biased (positively or negatively) by CHW interaction. Data collected through the survey, key informant interviews, and focus groups is self￾reported, and as such presents possible limitations. SELECTED FINDINGS AND CONCLUSIONS JAFPP Question 1: To what extent has SHOPS assistance strengthened JAFPP’s management, financial and governance systems to operate sustainably? SHOPS’ assistance built JAFPP systems to the point where they can operate sustainably however JAFPP leadership is not inclined toward sustainability. Addressing issues uncovered in SHOPS’ assessment of JAFPP in 2010-2011 SHOPS provided on-the-job training and technical assistance to update bylaws, improve management by documenting standard operating procedures, improving staff morale and reducing turnover with performance incentives, and upgrading dilapidated facilities with equipment, renovation, and purchase of new clinics. JAFPP’s organizational performance has improved with respect to internal systems and operations, as evidenced by key performance indicators, adherence to procedures, and recognition from independent entities such as the Health Care Accreditation Council (HCAC) and the King Abdullah Award. However, JAFPP has not made significant progress towards financial sustainability. Despite SHOPS’ recommendations for three price increases, JAFPP senior management raised prices only once, citing fear of losing clients. While the price increase raised revenue by 40 percent, cost recovery remained at 58 percent due to management decisions to raise salaries and overhead costs. Nonetheless, JAFPP’s clinics are moving towards cost recovery. On average, 17 clinics (excluding six new clinics established in the last year) reached 77 percent cost recovery in 2014, up from 66 percent the previous year. However, JAFPP headquarters expenses represent one third of JAFPP overall expenses. Despite clinics’ progress towards cost recovery, the headquarters costs are not supported by clinic revenue at current prices, capacity utilization, nor client levels. Clinics would have to reach targets that greatly exceed 100 percent cost recovery in order to support JAFPP headquarters operating costs. JAFPP senior management is not proactively pursuing options to meet the deficit; JAFPP will continue to require an infusion of funds to continue operations. Effectiveness Question 2: How effective were the various interventions/mechanisms and approaches in achieving project objectives of increasing demand for, access to and quality of FP services with targeted clients and providers? Question 3: How did SHOPS’ internal operations and organization assist or hinder achieving project targets and objectives? Question 4: To what extent where project partners effective in increasing access to, demand for and the quality of family planning products and services as a result of SHOPS support? SHOPS increased demand for, access to, and quality of FP services with targeted clients, however, cultural norms and practices and norms that were not addressed through SHOPS are likely to have diluted the results to effect of reducing fertility to slow population growth. SHOPS worked with JAFPP, 3 UNRWA, local NGOs with clinics, pharmacists and network doctors for FP service provision. Interventions/mechanisms such as CHW outreach and referrals were most effective in increasing demand, while health fairs and mass media campaigns increased temporary demand for modern family planning services and commodities. Clinic renovations and service provider capacity building, improved quality of FP service provision. Interventions such as increasing the number of clinics, vouchers, and coupons, post-partum FP counseling, and expanded FP method mix increased access to FP, especially for low-income women in underserved areas. Demand: The combined CHW outreach, referral and voucher interventions were effective in increasing demand for family planning, increasing family planning visits, clients, and new acceptors of modern FP. Vouchers provided by CHWs referred women to private doctors and health clinics for cost-covered consultations for family planning decision-making, resulting in a 22 percent uptake in modern methods. The number of new outreach clients accepting new methods exceeded 89,000, while those accessing private sector FP consultations with voucher subsidies totaled 33,260, exceeding SHOPS’ annual targets. Without the CHW outreach referral/voucher system, it is likely that uptake among low-income women would have been far less. While SHOPS campaigns for oral contraceptives and Intrauterine Device (IUDs) boosted the uptake of IUD insertion services, the effect on demand was transient. The least effective mechanism for increasing demand was coupons for highly discounted contraceptives, as redemption of coupons required multiple trips to doctors and pharmacies, consuming time and money that negated the cost savings. Access: SHOPS increased access for women in areas previously unserved by purchasing clinics for partners. SHOPS facilitated the introduction of a wider method mix to partners that in turn enabled providers to offer clients access to a wider range of FP commodities. However, obstacles posed by FDA limited the introduction of methods such as Cyclofem. SHOPS’ pilot initiative offering post-partum FP counseling through private hospitals demonstrates significant potential for increasing FP access to women at a critical juncture in their reproductive lives. Quality: Clinical training and evidence-based medicine (EBM) raised the awareness, knowledge and caliber of services offered by FP providers, especially within JAFPP clinics and Network Doctors (NWDs). These interventions enabled doctors to observe and practice IUD and Implanon insertions and better counsel women on a wider variety of FP methods. Results of other interventions were varied. Upgrades to CMIS was successful with JAFPP and the Institute for Family Health (IFH), but was problematic, and hindered data reporting with Islamic Charity Center Society (ICCS). Operations: SHOPS internal operations and organization, for the most part, assisted in achieving project objectives. For example, performance-based grants may be credited with raising performance at JAFPP clinics, as they provided significant financial motivation. However, when awarding performance￾based grants to new NGO grantees, additional time and assistance may be required to ensure monitoring and evaluation (M&E) systems are in place to sufficiently report on verifiable targets. While M&E systems were able to track project performance, client tracking among grantees may require technical assistance to ensure data integrity. The lack of data tracked by SHOPS on discontinuation makes it challenging to address the issues of long-term modern FP and impact on Couple Years of Protection (CYP) and total fertility. SHOPS’ inability to contract with UNRWA (due to incompatible contracting protocols) resulted in a missed opportunity for engagement with ten percent of the target population for FP services, dampening project potential reach in UNRWA catchment areas. Partners: The synergies created among CHWs, partnering clinics and NWDs through the referral and voucher systems were particularly effective in increasing FP demand, exceeding SHOPS’ projected targets. As CHWs are trusted community members who are able to meet with women in their homes and often with other family members who influence FP decisions, they were effective in generating demand for FP services and products among low-income populations. CHW visits further increased 4 demand and access for IUDs and FP services by distributing vouchers for free consultation with NWDs. In fact, NWDs’ impact on CYP is close to that of JAFPP. Due to contracting obstacles, support to UNRWA from SHOPS was largely limited to UNRWA doctor participation in EBM training and furniture/equipment provision. Although SHOPS support to the NGO clinics is likely to have increased demand and access, support to these clinics took place too late to collect time-series data and interpret the relative effectiveness. Gender Question 5: To what extent were steps taken to address gender differentials and gaps? Despite extensive awareness-raising by CHWs that increased acceptance of FP among low-income women, misconceptions about modern contraceptives remain pervasive. These misconceptions have a significant effect on choice and sustained use. SHOPS effectively addressed the preference among Jordanian women for female health care providers by working predominately with female providers: JAFPP and CHWs are all female, and over 70 percent of NWDs and pharmacists partnering with SHOPS were female. Nonetheless, the gender issue pertaining to the cultural preference for male children, which often results in families continuing to have children until a sufficient number of male children are produced, was not part of SHOPS’ programming or design. Changing deeply-seated social preferences for male children may require considerable social research to identify messaging that is culturally acceptable and not perceived to be based on a foreign agenda. Although men are recognized as significant influencers in family planning decisions, SHOPS lack of attention to gender differentials in messaging and engagement may have diluted achieving objectives. Sustainability Question 6: To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation? Question 7: What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue, and which are unlikely to continue and why? Question 8: Which of SHOPS’ partners (such as private doctors working through the voucher system) are most likely to continue practices that contribute to sustaining outcomes of increased a) access to; b) utilization; and c) quality of family planning services? While demand-based interventions such as coupons, vouchers, CHW outreach and referral were not designed for financial sustainability of intervention, clinic purchases and renovations, capacity building and systems instituted within partnering organizations were designed to be maintained, used, and sustained by partners. SHOPS partners will continue to operate and contribute to increased access to, use of, and quality of family planning services, albeit likely at a lower rate of growth without vouchers/referrals. JAFPP is not financially sustainable without donor funding, nor is the CHW program. Initiatives such as EBM and hospital-based post-partum counseling were effectively adopted by and integrated into local institutions, and will likely continue. Design of Interventions: Intended sustainability of SHOPS interventions was varied. Demand-based interventions such as coupons, vouchers, CHW outreach and referral were not designed for financial sustainability of intervention. On the other hand, clinic purchases and renovations, capacity building and systems instituted within partnering organizations were designed to be maintained, used, and sustained by partners. For example, clinic renovations were designed to ensure long-term ownership of renovations and remove rent burdens from JAFPP. HR, Health Management Information systems (HMIS) and finance systems were designed to be institutionalized by JAFPP. Some interventions, such as the EBM training, were integrated into local institutions for sustainability of programming post project 5 closeout. Other interventions such as Public Private Partnerships (PPP) do not have a tangible partnership mechanism for sustainability built into the design. Sustainability of Interventions: Among interventions designed for sustainability, partners indicate an intent to for continuity. For example, JAFPP has integrated the SHOPS-initiated operational and management systems into their regular operations. Similarly, EBM and clinical training and toolkit were designed for and have been integrated into medical institutions’ training protocols; as EBM is designed for self-directed learning and critical thinking, the learning results will continue to deliver benefit. Due to the ownership and integration of EBM within multiple Jordanian institutions, the EBM is likely to continue without further donor support. Less sustainable are the mass media campaigns that were not conceived for sustainability and unlikely to continue without donor support. Similarly, the coupon initiative and voucher system are donor dependent, financially unsustainable, and are unlikely to continue after SHOPS program closeout. As a result, the client visits supported by these mechanisms are likely to diminish in the absence of referral and voucher support. Sustainability of Partners and Practices: While SHOPS partners will continue to operate and contribute to increased access to, use of, and quality of family planning services, gains will likely taper. Moreover, results from partner to partner also differ. Based on current financial models and practices, JAFPP is not sustainable without donor funding, nor is the CHW program. While all NGO partner clinics will likely continue to provide modern FP services, the level of clinic visits and clientele growth rate are unlikely to continue at the same rate as with SHOPS support, due to unsustainability of the uptake in FP services associated with voucher redemption. However, UNRWA is likely to continue to provide services at the same rate of growth as it did without SHOPS support. Network doctors are very likely to continue EBM training and use, as EBM was effectively adopted by and integrated into local institutions. Likewise, private hospitals such as Specialty Hospital recognized intrinsic benefit through increased business from post-partum counseling, and will likely continue. Learning Question 9: To what extent were the theory of change (assumptions, pre-conditions) and project design used for the development approach appropriate to achieve USAID’s intended results? Question 10: What recommendations/suggestions does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? SHOPS’ M&E plan did not have a documented theory of change, nor did it list assumptions upon which the design was based. However, the project was able to meet most (81percent) of its targets by the beginning of Year 5, and positively impact CYP. Future programming needs to consider the cultural context – norms and practices that are contrary to or could dilute intended results programming. Simultaneously systematic collaboration between the private sector and MOH could improve sustainability of FP gains in the private sector. Theory of Change: SHOPS’ M&E plan did not have a documented theory of change to tell the story of how activities will lead to outcomes (at multiple levels), nor did it list assumptions upon which the design was based. As a result, determining the appropriateness of design is challenging. Based on the results framework, it can be surmised that the theory of change is based on the premise that if access to, demand for and quality of family planning services are increased, then CYP will increase, which should have an overall impact on the contraceptive prevalence rate (CPR) and/or the total fertility rate (TFR). However, CPR and TFR were not higher-order indicators found within the SHOPS results framework. According to available indicator results reported, the project was able to meet most (81percent) of its targets by the beginning of Year 5, and positively impact CYP. 6 An overarching assumption of SHOPS appears to be that by increasing access to and quality of family planning products and services, products will be used long-term, without interruption, and that CYP will increase which in turn will positively impact the TFR. However, cultural norms and practices such as the preference for male children, large family sizes (at least four children) and the practice of removing IUDs during Ramadan, do not to appear to have been considered. These norms and practices must be addressed to have an impact on reducing TFR. Partnerships: While SHOPS engaged private sector actors, facilitating relationships between the Ministry of Health (MOH) and the private sector could significantly contribute to sustainability. Cultural Norms and Practices: Programming needs to consider the cultural context – norms and practices that are contrary to or could dilute intended results, such as the cultural preference for large families and male children, and during religious holidays. 7 RECOMMENDATIONS Based on the Findings and Conclusions, the following recommendations are offered to USAID for future program design, implementation, and effective allocation of resources in family planning and reproductive health services. JAFPP 1. If JAFPP sustainability is a priority for USAID, a Project Implementation Letter (PIL) or Memorandum of Understanding (MOU) stating this agreement should be signed with JAFPP. Terms of the PIL/MOU should include intent for financial sustainability, agreement to reducing Head Quarter (HQ)/clinic expense ratios, and agreement to set prices and services based on market research for each location. 2. Assistance to JAFPP should focus on clinic decentralization and facilitating clinics to operate as individual business units (profit centers). 3. Financial assistance to JAFPP should be predicated on meeting performance targets, and funding should be provided in tranches conditional upon meeting benchmarks such as cost recovery targets. Support for HQ costs should be minimal with a phase-out plan. 4. USAID should support market research to determine whether JAFPP could increase cost recovery and its client base by offering a wider range of medical services for families as a one￾stop shop for all maternal and child health needs. Effectiveness 5. If USAID wishes to ensure increased access to FP, commodities and services should continue to be subsidized through vouchers for FP consultation products until a critical mass has been built for sustained social change in attitudes and decision-making for long-term, continuous use of FP. 6. Coupons for FP products should be continued but the process should be streamlined to eliminate the need for women to make multiple trips. A directory of participating pharmacists should be readily available in print or through mobile technology to eliminate uncertainty of travel and locations. Relationships should be facilitated between clinics and pharmacies; coupon expiration dates should be eliminated. 7. USAID should continue to fund CHWs and leverage them as a key resource to expand community outreach. USAID should strengthen CHW-clinic collaboration and create direct and active CHW linkages to UNRWA and other NGO clinics in the target catchment areas in order to increase FP demand. 8. Expand services of CHWs to include pre-marital girls, preconception, extended family members living in the same house and male relatives (husbands, fathers, brothers), and Syrian communities. 9. Health programming should focus heavily on addressing misconceptions about the side effects of contraceptives, targeting medical practitioners, clinic staff, pharmacists, CHWs, and the general public in order to increase the use of long-term methods such as IUDs, and to reduce discontinuation. Given cultural practices and norms in which information is delivered in-person, a cost/benefit analysis should be conducted of mass media vs. social means such as CHWs and if possible, the impact of each on CYP. 10. Post-partum counseling through private hospitals should be expanded, following the model SHOPS supported at Specialty Hospital. 11. Strengthen PPPs by establishing a PPP grant fund for implementation of joint activities by private￾public FP actors. 8 12. All grantees should be required to use a Management Information System such as that used by JAFPP and UNRWA for evidence based decision-making and standardization of reporting protocols. Provide technical assistance to grantees for collecting client data and tracking clients to determine FP behavior, FP purchases, fertility and FP discontinuation. Bar-coded or chip￾verified cards could be used for tracking client FP behavior by swiping or scanning the card with each visit to record the frequency and types of services and products accessed, discontinuation (such as IUD removal), and demographic information. 13. Performance based grants should have a one-year grace period to allow grantees to build adequate capacity and systems to measure performance and meet targets. 14. Performance based grants should continue to be used as a contracting mechanism, directly tying key benchmarks to activity/project objectives of increased FP demand, access and cost recovery. 15. Set long-term FP use as a clear project target and track discontinuation. Gender 16. Continue to focus on female providers to address cultural preference for female doctors and pharmacists. 17. Support a study on FP use in Ramadan and build an education/counseling program around findings, targeting religious leaders, clients, community members, men, and FP providers. Link activities to a Family Planning Fatwa issued by the Jordanian Iftaa Department to supplement work conducted on FP with religious leaders under Jordan Health Communications Partnership (JHCP). 18. Programming objectives, activities, and indicators need to acknowledge cultural norms and practices relative to family size and preference for male children. Such acknowledgement should include behavior change communication to change these norms and practices; identify other stakeholders who are conducting behavior change communication to change these norms and practices; or lower expectations to account for cultural norms and practices. 19. If behavior change communications is conducted it should be preceded by research to carefully identify messaging and target markets, and recognize sensitivities relative to the perception of imposing Western agendas. 20. Programming objectives and activities, and indicators need to acknowledge husbands’ role in family planning decisions; gender sensitive interventions and messaging should be designed accordingly. Sustainability 21. The approach of integrating interventions into local institutions should be continued and replicated in future programming. For example, the model of integration of training programming within Jordanian institutions, as seen with EBM integration into the Jordanian Medical Association, Jordan University for Science and Technology, Mu’ta University, Jordan University, and Hashemite University, lends itself to local ownership and sustainability. 22. Recognize that although not financially sustainable, CHWs can be leaders of behavior change; reinforce and expand geographic coverage of messages to reach a tipping point for long-term social change. Meanwhile, explore possibilities for financial sustainability of CHW programming by linking them to pharmaceutical companies whose products they market. 23. Recognize that although not sustainable, funding free access to FP services and products through voucher and coupons can contribute to CYP and will contribute to reaching a tipping point of social change. 9 Learning 24. Program design should include a theory of change describing the causal linkages among results and activities. Assumptions should be explicitly documented, as should issues that are out of the sphere of control or influence of the implementing partner. 25. Research and follow successful practices In Muslim countries for reducing TFR and taking a holistic approach that includes integrating all factors influencing family planning decision-making. 26. Require IPs to track discontinuation. 27. Produce highly targeted messaging and approaches to including men, mothers-in-law, and religious leaders in family planning attitudes and behavior. 28. Enlist and build a broad base of support among the Royal Court, Ministries, and other thought leaders in campaigns to change preferences for at least four children and for male children to impact total fertility rate. 29. Incorporate work done within Iftaa to ensure Islamic contributions to combating misconceptions of FP. 30. Identify compatible sub-grantee mechanisms for UNRWA. 10 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE Jordan’s population growth rate is constraining the country’s ability to achieve socio-economic progress and maintain stability. If not addressed, the population will double by 2047. Increasing the use and demand for voluntary modern family planning and reproductive health services is critical to curbing population growth. The purpose of the final performance evaluation of SHOPS is to support USAID decision-making for future program design and implementation of family planning initiatives, and for effective allocation of resources, particularly as they relate to investments in JAFPP and with the private sector. The evaluation is intended to provide an evidence-based analysis of how effective the project was in meeting its objectives of increasing the demand for, access to, and quality of modern family planning services; to assess the sustainability of project outcomes and practices; to identify factors contributing to outcomes and sustainability; and to provide recommendations on how to make future projects with similar objectives more effective and more sustainable. EVALUATION QUESTIONS JAFPP 1. To what extent has SHOPS assistance strengthened JAFPP’s management, financial and governance system to operate sustainably? Effectiveness 2. How effective were the various interventions/mechanisms and approaches in achieving the project’s objectives of increasing demand for, access to and quality of family planning services with targeted clients and providers? 3. How did SHOPS’ internal operations and organization (SHOPS strategic planning, project management, financial management, communications, grant-making process, grant management, M&E systems and indicators, staffing structure) assist or hinder achieving project targets and objectives? 4. To what extent were project partners (JAFPP, IFH, ICCS, UNRWA, private providers, etc.) effective in increasing access to, demand for and/or the quality of family planning products and services as a result of SHOPS support? Gender 5. To what extent were steps taken to address gender differentials and gaps? Sustainability 6. To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation? 11 7. What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue, and which are unlikely to continue and why? 8. Which of SHOPS’ partners (such as private doctors working through the voucher system) are most likely to continue practices that contribute to sustaining outcomes of increased a) access to; b) utilization; and c) and quality of family planning services? Learning 9. To what extent were the theory of change (assumptions, pre-conditions) and project design appropriate to achieve USAID’s intended results? 10. What recommendations/suggestions does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? 12 PROJECT BACKGROUND Program: USAID/Jordan Population and Family Health Office Project Title: Strengthening Health Outcomes through the Private Sector (SHOPS) Award Number: 278-A-00-10-00434-00 Award Dates: 20 July 2010 – 23 September 2015 Funding: $38,062,336 (amended from $24,362,336 in 2010 to $33,362,336 in 2011 and an additional $4,700,000 in field support in 2012) Implementing Partner: Abt Associates Inc. The combination of a high fertility rate and an influx of refugees contribute to a population growth rate that is curbing the country’s socio-economic potential and is putting the Kingdom’s stability at risk. If not addressed, the population will double by 2047. Increasing the demand for and use of voluntary modern family planning and reproductive health services is critical to mitigating population growth and resulting impacts on Jordan’s future. Despite a conducive policy environment for family planning (FP) and birth spacing that contributed to reducing the fertility rate to 3.5 percent in 2012, modern contraceptive use has experienced a plateau in recent years, stabilizing at 42 percent.1 The MOH provides the majority of family planning (FP) services (42 percent), followed by private physicians and private hospitals/clinics (20 percent), pharmacies (15 percent), the Jordanian Association for Family Planning and Protection (JAFPP) (11 percent), and UNRWA (ten percent).2 To address the population growth, USAID/Jordan issued an Associate Award to Abt Associates in 2010 to implement the five-year Strengthening Health Outcomes through the Private Sector (SHOPS) activity to expand the access, quality and utilization of modern FP services through engagement with the private and non-governmental sectors in Jordan, and particularly with JAFPP. SHOPS assistance to JAFPP follows decades of support to the organization from USAID/Jordan, most recently through the PSP for women’s health, and Cost Recovery & Sustainability of Jordan Association for Family Planning & Protection (JAFPP) Activity (2001-2006). JAFPP was established in 1964 to provide family planning and reproductive health services to low and middle income women through clinics with an all-female staff. JAFPP’s FP market share has experienced a steady decline over the years, falling from 30 percent3 in the 1990s to 14 percent in 20094 and 11 percent in 2012.5 According to SHOPS’ Cooperative Agreement, the anticipated outcomes of SHOPS were as follows:  Strengthened JAFPP management and governance systems; 1 The Higher Population Council (HPC), ‘National Reproductive Health/ Family Planning Strategy 2013-2017’ 2 Ibid 3 USAID ‘USAID-SHOPS Agreement: Associate Cooperative Agreement No. 278-A-00-10-00434-00.’ USAID, July 20, 2010 4 USAID ‘A Trend Analysis of Family Planning in Jordan. Informing Policy and Program Planning.’ Health Policy Initiative, USAID, 2009 5 DOS ‘Jordan Population and Family Health Survey 2012’ Department of Statistics., Department of Statistics 2012, p. 76 Source: Jordan Population and Family Health Survey 2012. Department of Statistics., p.64. 13  Increased JAFPP financial sufficiency;  Improved quality of FP/RH services at JAFPP and UNRWA clinics;  Increased demand for and access to private sector/NGO services;  Expanded method mix and product choice in the private/NGO sectors;  Developing marketing and behavior change strategies to improve the acceptance of hormonal Methods; and Removing medical barriers (provider bias). 6 SHOPS’ initial design was to focus on supporting JAFPP to reach 100 percent financial sustainability; to work with JAFPP and UNRWA to improve quality of services; and to work with private doctors and pharmacists to increase access to family planning service and availability of modern contraceptive methods. These efforts were to be supported through behavior change mass media campaigns and social marketing through community health workers to reach women who have limited FP exposure outside the home by going to their homes. Project Modifications SHOPS’ Cooperative Agreement was modified in 2011 and 2012, increasing the budget from $24,362,336 to $38,062,336 and broadening the scope to include private sector partners. The 2011 modification SOW “incorporates recommendations from the PSP assessment as well as recommendations related to the sustainability of efforts with JAFPP and UNRWA.” The amendment expanded SHOPS’ activities to include:  Expanded geographical access to family planning/reproductive health (FP/RH) services through purchasing and renovating JAFPP clinics;  Improved Quality of Family Planning Services at UNRWA;  Increased demand for family planning through NGO outreach;  Improved access to quality services through expansion of provider network to include clinics operated by NGOs (through grants), and private hospitals; and  Awarding performance-based grants for sub-grantees SHOPS simultaneously expanded its focus to employ more innovative methods including the use of evidence-based medicine (EBM) and Critical-Appraised Topics (CAT) to support its efforts to improve quality and access. SHOPS Results Framework SHOPS developed a results framework, performance monitoring plan and associated indicators upon project start, forming the basis for performance measurement. In addition to the main outcome indicators tracked throughout the lifetime of the project reflected below, the plan also included output indicators for JAFPP deliverables in governance, management, and financial sustainability, primarily described as “presence of” plans.” By 2013, the plans had been developed and the indicators no longer tracked; implementation of plans were not part of these indicators. With the modification in 2012, 6 USAID ‘USAID-SHOPS Agreement: Associate Cooperative Agreement No. 278-A-00-10-00434-00’ USAID, July 20, 2010. 14 SHOPS added a number of indicators to measure performance related to NWDs and pharmacies, improved quality of services and outreach, expanded FP market and product choice. The 2014 AMEP is reflected below: Table 1: List of indicators and outcomes SHOPS AMEP 2014 Project Objective: To expand the access, quality and utilization of family planning services in Jordan  Impact 1: Couple Years of Protection (CYP) achieved through JAFPP, UNRWA, private network doctor FP voucher redemption, and private market contraceptive sales  Impact 2: % of target women who take up a modern FP method as a result of in-house FP counseling by CHWs Result 1: Strengthened management and governance systems and increased financial sustainability at JAFPP (and other NGOs) Result 2: Increased access to and improved quality of private sector FP services Result 3: Increased Demand for FP Products and Services in the Total Market Strengthened management and governance systems at JAFPP Outcome 1 % adherence to management authority policies/procedures Overall improvement of quality Output 1 Number of people trained in FP/RH with USG funds Increased demand nationally Outcome 2 % annual change in OCP and IUD market Outcome 2 % adherence to Human Resources (HR) policies/procedures Improved quality of NGO FP services Outcome 1 Number of returning women per year using FP services at JAFPP Outcome 4 Number of new modern FP methods successfully introduced through USG￾supported programs Outcome 2 % client satisfaction at JAFPP Output 2 Number of counseling visits for FP/RH per year as a result of USG assistance Increased financial sustainability at JAFPP Outcome 5 % adherence to financial management policies/procedures Output 2 Number and location of health facilities rehabilitated, including those newly purchased Increased demand for projects sub￾grantee NGO FP services Outcome 5 Number of new women per year using project’s sub-grantee NGO FP services Outcome 7 % cost recovery, excluding project grant Increased access to and improved quality of private network doctor and pharmacist services Output 5 Number of doctors participating in private network that provide family planning services (cumulative) Output 3 Number of clinic visits for FP/RH per year with USG assistance (through NGO sub-grantees) Outcome 8 % cost recovery, including project grant Increased demand through community outreach Outcome 11 Number of new outreach clients using private sector network doctor family planning services to obtain modern FP method Outcome 9 % of revenue from non-client-fee sources (rent, grants, etc.) Output 7 Number of CATs developed, updated and shared with doctors and pharmacists as a result of the EBM program (cumulative) Outcome 12 Number of acceptors of modern contraceptive methods generated among target women Output 4 Number of women reached through outreach visits by CHW 15 EVALUATION METHODS AND LIMITATIONS The SHOPS evaluation employed multiple sources of data, including desk review, in-depth interviews with SHOPS staff, partners, grantees, and other stakeholders, focus groups with community health workers (CHWs) and clients, clinic observations, and a survey of pharmacists, clients, and private doctors. A complete list of interviewees is contained in the Contact List in Annex IV. A purposive sample of 16 grantee clinics was selected to cover all grantee organizations in the north, south and central areas. In addition, JAFPP clinics were selected based on performance in cost recovery, representing the top three performers, three of the bottom five performers, and two average performers. In-depth interviews were guided by semi-structured questions to capture perspectives and activities of each type of informant. The interview guides were designed to preserve the potential for a relatively free-flowing conversation, while creating a standardized format to facilitate a reliable, comparative analysis of data pertaining to the evaluation questions for triangulation of information from multiple stakeholder perspectives. While conducting interviews, clinic observations were conducted using a structured checklist to facilitate triangulating data. Table 2: Evaluation Interviews, Surveys, FGDs and Clinic Observations Total Male participants Female participants Interviews (KIIs) 46 15 31 Grantees 17 10 7 SHOPS Personnel 10 3 7 Key External Stakeholders 10 2 8 Private Hospitals 9 1 18 Focus Group Discussions 7 0 44 CHWs and Staff 3 0 25 Clients 4 0 19 Surveys 674 86 588 Clients 400 0 400 Pharmacists 141 47 94 Network Doctors (NWDs) 133 39 94 Clinic Observations 16 - - JAFPP 8 - - ICCS 2 - - UNRWA 4 - - HLC 1 - - IFH 1 - - Structured surveys were conducted with 133 network doctors, 141 pharmacists, and 400 CHW￾referred clients of grantee family planning services. Pharmacists were selected at random within the north, center, and south regions. Among pharmacists surveyed 28 percent participated in the coupon initiative. Focus groups were conducted with 19 clients and 25 community health workers (CHWs) in order to gain further perspectives. Network doctors were selected at random within each region. Survey details are contained in Annex II. Interviews and focus groups were conducted in Arabic by evaluation team members. Surveys were conducted by local subcontractor Mindset. Data collection was 16 conducted from June 1 through June 18, 2015. The evaluation design, data analysis approach and evaluation tools are included in the Evaluation Design Report in Annex II. Limitations Due to reasons of client confidentiality, the evaluation was able to access clients only through CHW records at CCA and GUVS. As such, client responses may be biased (positively or negatively) by CHW interaction. This limitation was mitigated by interviewing other partners in the network on CHW performance. Clients were selected at random and representationally from the north (25 percent), south (nine percent), and center (66 percent). Surveys were conducted by phone and in-person. Data collected through the survey, key informant interviews, and focus groups is self-reported, and as such presents possible limitations. This limitation was mitigated through data triangulation. The evaluation was conducted by an independent, external team of consultants that included Team Leader Pamela Putney, Senior Evaluation Specialist Nedjma Koval-Saifi (through local subcontractor Integrated Solutions), Jordanian Family Planning Specialist Huda Murad, Jordanian Health Specialist Wisam Qarqash, and data Quality Manager Rand Milhem (through local subcontractor Integrated Solutions). All electronic source data files are on the MESP file server and hard copies are warehoused with MESP. Upon request from USAID or closure of MESP, both electronic and hard copy data files will be transferred to USAID as per USAID Data Policy. Prior to conducting the evaluation, all evaluation team members signed Conflict of Interest forms indicating that they had no conflicts of interest related to the evaluation; these forms are on file with Management Systems International (MSI) home office and are available upon request. FINDINGS AND CONCLUSIONS FINDINGS: JAFPP Question 1: To what extent has SHOPS assistance strengthened JAFPP’s management, financial and governance systems to operate sustainably? Governance: In 2010-2011 SHOPS conducted baseline assessments of JAFPP’s HR and financial management, quality of care needs, service quality, and brand image. The assessments revealed obsolete bylaws, weak management and financial systems, no standard operating procedures, marketing plan, low staff morale, high staff turnover, and dilapidated clinic environments. In 2012 and 2013, SHOPS began capacity building with JAFPP’s Board of Directors (BoD) and executive staff to enhance their ability to use data for decision-making and to improve clinic management procedures through direct technical assistance and workshops on governance. These resulted in the development, approval and implementation of updated bylaws, restructured Board committees, and Delegation of Authority (DOA). Management: SHOPS worked closely with JAFPP to jointly develop clinic management guidelines, standard operating procedures, targets for clinic visits, employee incentive systems, and management information systems to schedule appointments and to track data for clinic-level management and service delivery. At the clinic-level, SHOPS provided technical assistance to establish salary scales, improve management processes, and improve quality of care through clinic-level supportive supervision training. SHOPS also provided technical and financial assistance to automate systems and computerize records, establish an internet connection between clinics and headquarters, and upgrade IT infrastructure and 17 capacity at all clinics and at JAFPP headquarters. IT assistance supported the Health/Clinic Management Information System that integrates HR, financial, procurement and operations, including client service data, across all of JAFPP’s clinics and offices.7 The clinic-level data contributed to operationalizing the staff incentive system, as staff receive bonuses when clinic revenue, clinic visits, and FP visits reach specified targets. Clinic observation through the evaluation confirms the use of CMIS in decision-making, as well as staff awareness of the incentive system and their interest in meeting targets in order to receive bonuses. Staff in two clinics expressed concern about their ability to sustain the client growth rates indefinitely and suggested that growth targets may need to be periodically revisited. Table 3 provides a summary of the changes to systems and practices as a result of SHOPS assistance. Table 3: Changes in JAFPP Governance and Management Systems At SHOPS Inception (2010) At SHOPS Completion (2014) HQ Level  Obsolete By-laws  Weak Governance, Financial & Management Systems  Imbalance between BoD & Management  Inadequate Supervision System  Weak HMIS System  No Explicit Standard Operating Procedures (SOPS)  No Marketing Plan  Poor HR Policies & Management  New By-laws Enacted & Executed  DOA & Code of Ethics In Place  Institutional Strategic Plan Developed  One-Year Business Plan Developed  Improved Structure & Composition of BoD  Management Support Systems to Clinics Operational  HR System Developed & Implemented  Improved Financial Systems & Management  Marketing Plan Clinic Level  Poor Clinic Infrastructure  Lack of Essential Equipment & Supplies  Some Clinics Closed due to Staff Vacations  Difficulty Recruiting & Retaining Female Doctors  Low Staff Morale  High Staff Turnover  New Salary Scale  Incentive System in Place  Hardship Allowance for Female Doctors  FP Service Quality Improvement  Improved Infrastructure  Adequate Equipment & Supplies 7 When SHOPS began, JAFPP operated a clinic management information system that collected basic information from clients for antenatal care and FP methods at each clinic. The IT clerk at JAFPP HQ collected the information at each clinic and compiled it. According to the Cooperative Agreement, SHOPS interventions were designed to update the system for collection of information on pregnant mothers and FP clients, and to build capacity for using data in decision-making. 18 Organizational Performance: In 2014, JAFPP achieved 97 percent compliance with HR procedures, 83 percent adherence to management policies/procedures, and 100 percent adherence to financial management policies/procedures. When assessed by the Health Care Accreditation Council (HCAC) at the start of SHOPS, JAFPP scored 39 percent on a variety of quality of care, management, and organizational criteria, and increased its score to 92 percent in 2013. JAFPP was awarded the King Abdullah II Center for Excellence’s Mark of Best Practice award in 2013, based on governance and management criteria. Fourteen out of 17 JAFPP clinics surpassed a target of 85 percent adherence to the association’s clinical policies and guidelines for FP services. JAFPP implemented a quality scoring system for clinic supervision, supported by updated performance checklists, as part of the SHOPS-upgraded CMIS. SHOPS also supported the development of procedural manuals such as medical waste management standards and a procurement manual. With the exception of financial sustainability, most targets in key performance indicators (KPIs) related to organization performance, as measured by SHOPS and JAFPP, have reportedly been met. Sustainability: JAFPP had not raised prices for ten years prior to the start of SHOPS. SHOPS subcontractor Banyan Global8 conducted research to compare prices charged by JAFPP to that of the private sector and concluded that JAFPP could raise prices (although a specific willingness to pay survey was not conducted). As a result, SHOPS set an initial goal for JAFPP to recover 100 percent of its costs through three sets of price increases, and recommended simultaneous reductions in overhead costs. In 2012, with the launch of a financial sustainability and pricing strategy, JAFPP raised prices for services and implemented KPIs for clinics as a measure toward controlling clinic expenses. As a result, revenue increased by 40 percent, however, cost recovery remained at 54 percent in 2012 due to management decisions to raise salaries and overhead costs simultaneous to the price increases. While SHOPS 8 Subcontractor Banyan Global was tasked with capacity building of JAFPP. Figure 1: Targets for 2014 Achieved in 2014 19 sustainability plans envisioned three price increases in the project period, JAFPP management was unwilling to raise prices more than once, citing their belief that clients would be unwilling to pay more. When JAFPP did not raise prices again, SHOPS repeatedly repealed cost recovery targets, first to 72 percent, then 65 percent, and finally to 59 percent. In 2013, an audit of SHOPS by the Office of the Inspector General (OIG) found that: “JAFPP did not make progress toward its financial sustainability target. While JAFPP offers services at a steep discount, it loses money on each client. Increasing the number of clients thus accelerated JAFPP’s revenue losses, putting the sustainability of project achievements at risk.” OIG recommended that USAID/Jordan require JAFPP to formally commit to specific sustainability measures (e.g., price increases) as a condition of receiving additional assistance under the agreement. In response, SHOPS provided JAFPP with a performance-based grant that included performance bonuses based upon JAFPP meeting sustainability benchmarks. In Q1 2015, JAFPP reported that 13 of 23 clinics had received performance bonuses for meeting targets that created competition among clinics to improve. However, the JAFPP performance incentive system criteria do not include cost recovery as an incentive target. To further support sustainability objectives, SHOPS developed an 18-month business plan in 2013 which included feasibility assessments and suggestions for improved cost recovery through expansion of services (a laboratory9, mammogram exams, pharmacy,10 and a wider range of family planning products), increased clinic service capacity/efficiencies, and sub-letting its unused spaces for laboratory and pharmacy services. According to evaluation findings, JAFPP does not appear to have followed these recommendations for cost recovery, with the exception of renting out pharmacy space in the Aqaba clinic. An assessment of performance against the business plan reveals that JAFPP “increased revenues from clinics by 14 percent due to the 27 percent increase in the number of FP visits compared to 2013, and as a result of the 13 percent increase in new clients. For the more strategic and adaptive initiatives such as marketing campaigns and tools, inclusion of new services, and development of new business lines there has not been progress.”11 SHOPS also provided business development technical assistance to build JAFPP capacity in fundraising and attracting donor support, with proposals for funding circulated to 11 donors with no results to date.12 As of 2014, JAFPP is at 58 percent cost recovery, with four clinics approaching or exceeding 100 percent cost recovery, and 13 clinics received performance bonuses by meeting or exceeding incentive targets. While SHOPS’ purchase of clinics for JAFPP removed the burden of rent, other costs such as an increase in electricity and operating costs at new facilities diluted the savings. Clinic observation during 9 Legal analysis of laboratory services proposed by SHOPS included a review of the legal implications of JAFPP adding products and services, and the associated licensing and registration requirements. 10 Legal analysis by Banyan Global revealed that, “the option to open a pharmacy was removed at the legal analysis step because of legal and technical constraints.” 11 SHOPS ‘Final Assessment Report of JAFPP Business Plan Execution’, 2015 12 Proposals submitted covered JAFPP marketing, outreach, business plans, expansion of services (labs, mammogram) for funding from donors such as UNFPA and embassies. “We do not feel that we can increase prices. If we do, fewer women will come because they can get services elsewhere at a lower price…there is no way that JAFPP will reach financial sustainability.” JAFPP management 20 the evaluation pointed to the collaboration between clinics and CHWs increasing client visits through referrals, such as in Irbid where clinics alerted CHWs when client visits dipped. Lower performing clinics such as in Aqaba and Zarqa did not work as closely with CHWs, were not as motivated by the incentive program, and had unused space not generating revenue. On average, clinics (excluding six new clinics established in the last year) reached 77 percent cost recovery in 2014, up from 66 percent the previous year. JAFPP headquarters expenses represent approximately one third of total expenses.13 Overall, cost recovery figures from 2013 to 2014 show 58 percent recovery including headquarters expenses and 68 percent without headquarters expenses. While organizational performance has improved with respect to internal systems and operations, JAFPP senior management stated that they do not want to raise prices because they fear losing clients; furthermore, they do not have immediate plans to follow SHOPS’ recommendations for sustainability through expansion of services, through rental income, or through alternate streams of donor funding. As stated in a June 2015 SHOPS Trip Report on Strengthened Management and Governance Systems at JAFPP, “While JAFPP has several ideas they are pursuing to generate additional funding, the effort is not grounded in a systematic plan. The senior management team and the Board of Directors assume a donor-dependent mentality.” During interviews, JAFPP senior management shared its sustainability idea of building a commercial profit center that could subsidize FP services; these ideas were predicated on receiving additional donor funding. JAFPP senior management is not proactively pursuing options to meet the deficit. CONCLUSIONS: JAFPP SHOPS assistance has clearly strengthened JAFPP management and financial systems, as evidenced from implementation of systems to track key performance indicators, adherence to procedures, and recognition from independent bodies such as HCAC and the King Abdullah Award. Despite clinics’ overall progress towards cost recovery, the headquarter costs are not supported by clinic revenue at current prices, capacity utilization and client visits. Clinics would have to reach targets that greatly exceed 100 percent cost recovery in order to support JAFPP headquarters’ operating costs. The current model of JAFPP offering only FP services may not be financially viable. At 58 percent cost recovery, JAFPP’s reserves will be exhausted in four to five years at the current rate of revenues and expenses; JAFPP cannot survive without an infusion of funds to maintain the current levels of services. SHOPS assisted JAFPP to the extent that they could very likely operate sustainably before exhausting their funds if recommendations for cost recovery such as price increases and service expansion were followed; ultimately, the interest and willingness of JAFPP leadership to use the organization’s newfound capacity will determine its sustainability. In the meantime, JAFPP remains dependent on donor funding and/or reserves for approximately 50 percent of its expenses. 13 SHOPS Annual Report 2011-2012’ and ‘Financial Strengthening of JAFPP Report’, January 2014 21 FINDINGS: EFFECTIVENESS Question 2: How effective were the various interventions/mechanisms and approaches in achieving project objectives of increasing demand for, access to and quality of FP services with targeted clients and providers? Question 3: How did SHOPS’ internal operations and organization assist or hinder achieving project targets and objectives? Question 4: To what extent where project partners effective in increasing access to, demand for and/or the quality of family planning products and services as a result of SHOPS support? SHOPS employed a wide variety of approaches to increase the availability and use of modern contraceptive methods, reduce discontinuation of contraceptives, and increase access and availability. In so doing, SHOPS worked through JAFPP, UNRWA, four clinics operated by local NGOs, pharmacists and network doctors for service provision through referral, vouchers and coupons for family planning services and contraceptive products, CHWs through social marketing, health fairs, and mass media campaigns. SHOPS also funded clinic renovation for some NGOs, and purchased clinics for JAFPP in situations where rented properties were substandard or at risk of extraordinary rental increases. Technical, organizational and clinical capacity building was integrated into work with network doctors, pharmacists, and grantees. Table 4: SHOPS Interventions Areas of Intervention Interventions Grants for NGOs  Renovation of 31 clinics  Capacity building  Organizational Development (Business plan, admin and management manuals/guides)  Quality assurance  Equipment and furniture provision  13 Family Fairs  Training for 210 CHWs  Implement outreach FP visits Private Network Doctors (NWDs)  Referral (vouchers)  FP advocacy on TV for FP use  1,200 Academic detailing visits (with pharmacists)  Training for 300 NWDs14  EBM capacity building Pharmacists  Coupon Pilot Initiative with 110 pharmacists  Training for 300 Pharmacists15  1,200 Academic detailing visits (with NWDs)  12 EBM seminars Community Health Workers (CHWs)  200 CHWs trained for Behavior Change and Outreach Community level beneficiaries  1.4 million Household Outreach visits  678,595 women reached and 89,600 new acceptors of FP 14 120 doctors were from the PSP Activity and 180 private doctors were added. 15 Pharmacists were identified with Jordan Pharmacists Association based on geographic criteria. 22  IUD ‘Edutainment’ Lectures for 12,000 women Private Hospital Initiative  Post-Partum Contraceptive Initiative at four hospitals Public Private Partnerships (PPP)  Two PPP round tables Demand Referral System: (CHW Outreach, Vouchers, Service Delivery Points and Network Doctors): SHOPS introduced a referral system in which CHWs conducted outreach to women in their homes to discuss modern family planning methods, and referred acceptors16 of modern FP methods to SHOPS service delivery points (SDPs) and network doctors. “The referral system itself bridges women in the community to the 78 SDPs” according to SHOPS’ DCOP. CHWs gave vouchers that were redeemable for no-charge family planning services (primarily through NWDs and JAFPP as other partners were not added until late in SHOPS implementation) to low-income women. 16 Family planning acceptors are married women of reproductive age who received counseling and accepted a modern contraceptive method at a SHOPS service delivery point (partner pharmacy, hospital, clinic). 23 CHWs Feedback CHWs form the foundation of the SHOPS network, with unique insight into dynamics of family planning decision-making and uptake in the various communities they serve. Agents of Social Change: “The most effective way to increase FP demand is the couples counseling during household visits. CHW visits had more impact than the IUD campaign. We go door to door. Outreach to households has been the most effective since we are generating social change in the community. We convince women but we still need to convince the mothers-in-law and husbands. We should also target pre￾marital girls and counsel the whole community.” Syrians: “We work very hard to make Syrians accept the concept of FP.” Coupons and Vouchers: “Cost becomes an obstacle to FP uptake without vouchers. We are restricted to distribute vouchers to only 50 percent of the women we visit. The free IUD helped increase IUD demand because it encouraged women to try it. However the coupons were not useful because of the cost of transportation and time added to securing the coupon savings.” CHWs fielded by the Circassian Charitable Association (CCA) and General Union of Voluntary Services (GUVS) conducted 1.4 million household visits reaching 678,595 low￾income women, resulting in 89,600 acceptors of modern family planning methods.17 About one third of acceptors use IUDs, followed by condoms (23 percent) and COCs (22 percent. CHWs distributed vouchers (covering all costs of consultation) to low-income women who CHWs determined could not afford family planning services offered by private sector providers18, redeemable at JAFPP clinics and with network doctors. CHWs are themselves from low￾income communities, most of whom are educated and call themselves “leaders of social change within their communities.” During home visits, CHWs addressed misconceptions about FP modern methods, and included husbands in family planning counseling whenever possible. SHOPS’ Deputy Chief of Party reported that a “randomized controlled trial demonstrated that in-home family planning counseling and free vouchers, either to women alone or to couples, had a strong and positive impact on uptake of modern methods. Counseling women alone increased modern method uptake by about 48 percent, while counseling couples increased it by about 59 percent, compared to those who received no counseling.” 17 Huda Khayame ‘Achieving Results Through Integrating Supply and Demand’, June 16, 2015 18 JAFPP clientele are mainly women who cannot access MOH due to lack of insurance coverage, and are not eligible for other free services such as UNRWA or RMS. Figure 2: Modern Method Adopters 24 Vouchers for free family planning counseling were redeemed at a rate of 58 percent.19 The number of new clients using private sector NWD services to obtain modern contraceptives totaled 33,260,20 exceeding SHOPS’ annual targets.21 Most (four out of seven) JAFPP clinics visited noted that JAFPP vouchers for free services were effective in increasing demand for modern FP services. According to CHWs interviewed, vouchers were instrumental in helping low-income women access family planning services because it allowed them to use a modern method they could not otherwise afford and/or encouraged them to try a new modern method. In 2014, couple years protection (CYP) rates attributed to NWDs were almost triple those of 2011, and closely matched the CYP results of JAFPP in 2014 as illustrated in Figure 3. Among CHW clients22 surveyed, 78 percent cited the CHW as their primary source for FP information.23 CHWs were cited by most SHOPS partners as critical to facilitating FP uptake and demand. Coupon Pilot and Pharmacists: 19 SHOPS ‘SHOPS Year 4 Annual Report.’ October 15, 2014 20 This refers to the number of outreach clients accessing a private doctor, and therefore emerge from a pool of clients referred by the CHWs. 21 S Number of new outreach clients using private network doctor family planning services to obtain modern FP method (indicator based on voucher redemption database) exceeded Year 2 targets by 14%, Year 3 targets by 24% and year 4 targets by 28%. Source: SHOPS ‘SHOPS Activity Monitoring and Evaluation Plan (AMEP) Year 5’, 2015 22 Clients of SHOPS are defined as the beneficiaries of demand-generating interventions and programs, either approached by CHWs or existing clientele of the partner NGO clinics. Clients surveyed by the evaluation focused only on those clients served by CHWs, as that was the only data set available to the evaluation team. CHW clients represent the majority of SHOPS clients. 23 Client contact information was provided through CHWs. Figure 3: CYP by source 25 The coupon initiative involved 110 of SHOPS’ 300 partner pharmacies who were identified with the Jordan Pharmacists Association (JPA) based on NWD geographic locations. Coupons provided a 30 percent discount on modern FP products such as oral contraceptives, IUDs, injectables, and vaginal rings. Partners interviewed had low awareness of the coupons. Forty-one percent of pharmacists participating in the coupon initiative indicated that the coupon pilot had little effect on increasing demand. CHWs reported that in order to use coupons women had to first secure a stamp from an FP provider, then proceed to a pharmacy to secure the contraceptive. If the contraceptive required medical intervention (such as IUD insertion) the woman would have to return to the doctor. According to CHWs and clients, the time and cost of transportation to redeem coupons diluted the benefit. As one observed, “oral contraceptives are cheaper and more feasible to buy from pharmacies than to go through the coupon process.” In order to facilitate communication between clients and pharmacists, CHWs wrote “prescription” notes24 to pharmacists indicating the FP method the woman desired. Although SHOPS produced a directory of participating pharmacies, none were found at clinics visited by the evaluation team. In the South, participating pharmacies were geographically dispersed and limited in number, thus limiting access further. Mass Media Campaigns: SHOPS conducted a series of mass media campaigns for oral contraceptives and IUDs through radio, newspaper, television, and through information, education and communication (IEC) materials. A SHOPS study on the impact of the IUD campaign revealed a boost in national CYP in 2014 driven by a spike in the uptake of IUD insertion services following a promotion of the IUD as a long-acting reversible contraceptive. A SHOPS manager stated that, “An evaluation of the campaign showed that the number of IUD insertions at private network doctor clinics increased by 36 percent in the month after the campaign’s start, compared to the same month in the previous year, while the year-on-year increase was only six percent the month before campaign start.” Among doctors surveyed who were aware of the campaigns, 75 percent indicated that these campaigns were very effective in increasing demand to FP methods and 69 percent stated they were very effective in increasing demand, while only 32 percent of pharmacists who were aware of the campaigns felt they were very effective in increasing demand. According to SHOPS’ studies, the increase in demand was immediate but temporary, as evidenced by spikes in FP immediately after health fairs. IEC materials were the fourth most-cited source of influence on FP choice among clients interviewed. 24 CHWs counseled women and jointly decided on a preferred FP method. CHWs sometimes wrote notes to pharmacists to explain the desired FP product as women are sometimes too shy to speak to the (usually male) pharmacists, as recorded in CHW focus groups. Figure 4: National CYP, by method 26 Misconceptions and FP Demand: Misconceptions about contraceptive methods have long been cited as an important influence on women and men's decisions to adopt and to continue or discontinue contraception. A common practice in Jordan is for women to remove IUDs before Ramadan in order to prevent spotting, as cultural norms do not allow women to fast if they are spotting. Despite a booklet produced by Iftaa,25 which addresses religious misconceptions about family planning, including Ramadan issues with spotting, prayer, and fasting, SHOPS partner clinics appear to be unaware of its teachings and existence.26 This booklet could dispel religion-based misconceptions about family planning. A further misconception among clients and CHWs revealed a common belief among both men and women that IUDs cause cancer and can move around the body. CHWs and providers stated that women mistakenly believe that the IUD will move in the belly or get lost during sexual intercourse, the pill will cause cancer, or they may get pregnant despite the IUD. In answer to an evaluation survey question about their perception of why women do not use modern methods of contraception, 66 percent of pharmacists and 59 percent of network doctors listed the fear of side effects as the first or second most important reason. Surveyed clients confirmed that side effects - both real and believed – affected long￾term use of modern FP methods. Of the 31 percent of respondents who reported that they faced barriers in the prolonged use of modern FP methods (users and non-users), 55 percent cited actual side effects as the main barrier while 24 percent listed fear of side effects as the most important barrier. Observations at clinics such as JAFPP, Jerash, and Aqaba reveal that sometimes misconceptions about contraceptives are not addressed with clients. Syrian Population Differentials: With a population exceeding one million, Syrian refugees living in Jordan affect population growth and fertility rates. CHWs visited Syrian women and referred them to the SHOPS network, but as Syrians receive similar services gratis through the Syrian humanitarian relief effort, Syrians rarely accessed the SHOPS network. CHWs report that Syrian women are even more reluctant to accept modern methods of FP than Jordanian women; according to CHWs they are less exposed to FP and have lower FP awareness. A Syrian client in Mafraq stated, “My uncle has ten children and he does not mind having more.” As the major influx of Syrians occurred after SHOPS began, SHOPS was not designed to address this particular issue. Nevertheless, Syrians are intermarrying with Jordanians and are now part of the new population of Jordan, affecting the FP landscape, fertility choices, and fertility rates in the near future. 25 Iftaa is the General Fatwa Department of Jordan with authority equal to a Ministry, dealing with matters relating to all Islamic community needs and legislation. http://www.aliftaa.jo/ 26 K4 Health, ‘Iftaa Booklet – Family Health Training Manual’ Section 5, p49. https://www.k4health.org/sites/default/files/Wo3ad.pdf 27 Reproductive Lifecycle Opportunities: The population targeted by SHOPS was married women between the ages of 18 and 35 (reproductive age). As SHOPS’ design focused on use of contraceptives, SHOPS’ mandate did not include females at puberty, pre-marital, and single stages. However, CHWs interviewed stated that there is a need to target unmarried women for FP counseling, representing an opportunity to increase demand for FP commodities and services. Access As part of a comprehensive assessment of JAFPP clinics, SHOPS examined each clinic in terms of physical infrastructure, equipment, staff processes, and staff training needs, identifying necessary equipment and supplies at 17 clinics. Many existing facilities were too small for the volume of patients, poorly laid-out in terms of patient flow and infection prevention, and unattractive to potential clients. Findings highlighted urgent needs for the relocation, renovation, and construction. While SHOPS’ initial design and work plan called for the renovation of JAFPP’s original 17 clinics, an assessment conducted in the first year indicate that nine of the 17 were unsuitable for renovation, and the other eight had landlords who refused to allow the renovations. In addition, the assessment highlighted the potential for substantial increases in rental costs for JAFPP due to potential legislative changes. A decision was made to renovate only JAFPP-owned clinics, as JAFPP’s intent was to eventually own all its clinic properties. With SHOPS support, 15 JAFPP clinics were relocated to newly purchased clinics. Due to cost and availability, new clinics were sometimes in less desirable locations such as the Mahata clinic, resulting in a drop in client visits. Despite SHOPS’ documented coordination with JAFPP management, this individual clinic claims that SHOPS did not include them in decision-making for the clinic move. Table 5: SHOPS Service Delivery Points Clinics 78 JAFPP 23 UNRWA 24 ICCS 17 FHI 9 HLC 3 AJA 4 CHWs 196 CCA 125 GUVS 71 NWDs/Pharmacists/Hospitals Private Network Doctors 300 Pharmacists 300 Private Hospital post-partum pilot 4 28 In an attempt to expand geographic access to family planning, SHOPS’ contract modification included technical and physical support to NGOs beyond JAFPP. SHOPS conducted an assessment of potential partners to expand its NGO partner clinic network, and selected ICCS with 17 clinics serving moderate to low-income highly populated and refugee communities, AJA with four clinics, HLC with three clinics, and IFH with nine clinics serving refugee and Jordanian communities. Similar to renovations for JAFPP, SHOPS renovated and equipped four ICCS clinics, two HLC clinics, and four IFH clinics. By the end of SHOPS’ activity, 78 service delivery points throughout the country had received support from SHOPS, including physical renovation, partial renovation, or furnishing and equipping, in addition to other technical support that varied from partner to partner based on partner needs and duration of agreement.27 Interventions with local NGOs varied according to clinic need and capacity to absorb assistance. Not all renovations were complete at the time of writing this report, and performance data from SHOPS support is not yet available. From 2010 through 2014 SHOPS-supported clinics from all partners reported 215,000 visits for family planning and reproductive health. JAFPP increased its clientele by 53 percent during SHOPS implementation, averaging 11,500 new clients per year, with an increase of returning clients by over 13 percent.28 By comparison, during 2011-2014 UNRWA increased its new clients by an average of 7,000 clients per year with the rate of returning clients increasing by three percent.29 Unlike other grantees, UNRWA did not receive systematic referrals from CHWs, and did not participate in the voucher program as UNRWA services are free and UNRWA has its own outreach program. UNRWA’s client growth rate is lower than that of JAFPP; it did not participate in SHOPS’ outreach activities. FP Commodities: SHOPS supported the expansion of the available FP method mix by improving and expanding individual FP provider practices, and facilitating the introduction of commodities such as Implanon (implant rods), Mirena (type of IUD), and injectables30 new to private sector partners through training, vouchers, and bridging and facilitating FP supply between MOH and SHOPS partners. The private sector serves 56 percent of contraceptive users.31 IUDs are the contraceptive most used by clients surveyed for the evaluation (52 percent of current users) and were reported to be prescribed by 95 percent of the surveyed network doctors, followed by progesterone-only pills (POP) and combined oral contraceptives (COCs). Although IUDs were one of multiple products supported by SHOPS, IUDs were the best known and most widely used contraceptive in Jordan in 2012, used by 21 percent of married contraceptive users; 50 percent of married women using modern methods; and representing 62 percent of modern methods distributed by the private sector. Among modern methods distributed by the private sector in 2012, 13 percent were distributed by doctors and one percent by pharmacists.32 SHOPS supported the uptake of modern method mix through coupons for FP products, vouchers for FP services, and mass media campaigns for POPs and IUDs. SHOPS tracked percent change in the IUD 27 Examples of renovation include new clinic facilities with JAFPP, new clinic rooms for partners such as ICCS and IFH, furniture (office desks, chairs, waiting rooms), equipment such as IUD examination tables, sterilizers. Technical assistance included competency-based training, EBM, counseling on family planning, organizational capacity building and CMIS support. 28 JAFPP Client database 29 UNRWA ‘UNRWA Health Report 2011-2014’ 30 Cyclofem is pending approval with FDA. Options explored by SHOPS included emergency contraceptive pills, DMPA, implanon, hormonal intrauterine system (IUS). 31 DOS. Jordan Population and Family Health Survey 2012. Department of Statistics., Department of Statistics 2012, p. 69. 32 Department of Statistics ‘Jordan Population and Family Health Survey 2012’, DOS, Amman, Jordan. September 2013 29 market, as measured by market growth in sales based on data received from MOH, RMS, UNRWA, JAFPP and others) meeting project targets, and exceeding project targets in Year 4.33 SHOPS also tracked percent annual change in the Oral Contraceptive Pill (OCP) market; targets were exceeded in Year 3 but not met in Year 4. However, direct attribution of IUD sales growth and OCP market growth to SHOPS alone is not possible. SHOPS expanded the mix of modern methods offered by participating pharmacists: among pharmacists surveyed, nine percent reported offering additional commodities as a result of SHOPS, with six percent adding IUDs to their available methods. Logistics of continuous supply was a problem, as some UNRWA, ICCS and JAFPP clinics visited cited shortages in FP commodity supply which disrupted FP service provision;34 vaginal suppositories were only available at UNRWA. Private Hospitals: With 99 percent of women in Jordan delivering babies in hospitals, SHOPS decided to test private hospitals as an entry point for family planning counseling. SHOPS conducted a six-month pilot test in which women delivering at private hospitals in Amman would also receive family planning counseling. The hospitals included in the pilot deliver 250 to 300 babies per month; private sector hospitals in Jordan serve almost 13 percent of the population. According to an unpublished assessment conducted by SHOPS (control vs. treatment) regarding the post-partum pilot, “uptake of modern contraceptives was higher among the counseled women at 40 days and at three months postpartum, the differences did not reach statistical significance, likely due to a small sample size and counseling treatment contamination (moderate adherence to the randomized assignment by counselors).”35 Prior to SHOPS, these hospitals provided family planning counseling via private doctors. Within the six-month pilot timeframe, SHOPS introduced provision of family planning counseling through nurses. The evaluation team found that one hospital (Specialty) is continuing family planning counseling through nurses, and one hospital (Israa) is planning to continue counseling through a designated post-partum family planning counseling office. Key informants at the Specialty Hospital suggested that future interventions should include pro￾active follow-up with women who have been counseled and offer referrals for continued counseling. Quality EBM/CAT: SHOPS instituted the Jordan Evidence-Based Medicine/Reproductive Health (JEBMRH) Group that conducted seminars on Critically Appraised Topics (CATS). As part of EBM, SHOPS conducted 1,200 academic detailing36 visits providing one-on-one educational sessions to 300 physicians and 300 pharmacists in Amman, Irbid, Karak and Aqaba. The visits focused on increasing doctors’ and pharmacists’ awareness and knowledge of modern contraceptive methods, including injectables (DMPA) and IUDs. Seventy-one percent of network doctors surveyed stated that EBM was very effective in increasing access for FP while 16 percent stated it was somewhat effective; 64 percent stated they would continue EBM after SHOPS closeout. Network doctors attribute effectiveness of EBM to raising 33 SHOPS Activity Monitoring and Evaluation Plan (AMEP), Year 5, 2015. 34 UNRWA cited that vaginal suppositories had been out of stock for a few months. JAFPP Irbid Oula, Jarash and Aqaba clinics cited disruptions of a few days of miscellaneous FP products, citing a need for reliable, uninterrupted supply. ICCS Jarash clinic had no stock for family planning commodities at time of visit. 35 Timothy Ingrens, SHOPS COP, email dated August 18, 2015. 36 Academic detailing is defined as face-to-face education of prescribers by trained health care professionals, typically pharmacists, physicians, or nurses. Within SHOPS this translated into a regular or quarterly visit to doctors and pharmacists to update them and provide them with new information of FP methods based on evidenced based medicine. 30 their level of knowledge of FP products and services, and thereby increasing the repertoire of services and products they offered to the target population. NGO Capacity Building: SHOPS collaborated with HCAC to establish quality assurance systems for clinical training and supportive supervision, organizational capacity building, quality monitoring and CMIS installation and utilization with partner NGOs. According to SHOPS data, client satisfaction rose from 69 percent in Year 1 to 90 percent in Year 4 at JAFPP, exceeding SHOPS’ Year 4 target of 76 percent. JAFPP clinic observation for the evaluation as well as SHOPS’ assessments indicate that learning is being applied, as evidenced by standard operating procedures in place, application of client satisfaction procedures, continued supportive supervision visits, and application of HCAC quality standards related to documentation and policies adopted, as illustrated in Table 6. Table 6: Quality of Care Interventions  Infrastructure improvements  Essential medical equipment & supplies  Job aids (e.g. uterus models for FP counseling, pocket reference guides)  Quality assurance & improvement systems at NGOs - Provider competency - Compliance with clinical guidelines - FP method choice - Client satisfaction - Infection prevention and control  Client satisfaction monitoring systems  FP service quality assurance training for Network Doctors through HCAC  Quality assessment of NWD FP services  Clinical counseling training for private sector providers  Clinical training on FP and other related topics - IUD and Implanon insertion/removal and counseling - Counseling for oral contraceptives, DMPA and menopause - Health maintenance - Infection prevention & control - Post-partum FP at hospitals - EBM integration into health care system - Academic detailing for pharmacists & NWDs Clinic Renovations: In order to compete with other FP service providers, SHOPS renovated and equipped clinics to enhance the types of service NGOs were able to provide, and to improve the comfort of the environment. According to clinic staff and JAFPP management, new equipment and training on equipment use have enabled clinic staff to provide better care, upgrade service delivery, and improve clinic performance. Five out of eight clinics renovated cited clinic renovations and medical equipment as the most effective intervention in increasing FP access and/or demand. Site visits confirm that JAFPP’s clinic environment is on par with private sector clinics: modern, inviting and uniformly branded. Clinic clients and staff have noted the difference, which according to SHOPS reports has helped to increase demand at JAFPP clinics. According to staff at JAFPP Irbid Oula, “SHOPS opened a lot of doors to the private sector. We challenge them with the same level of quality at JAFPP.” SHOPS also renovated eight ICCS clinics and four IFH clinics, some of which were nearing completion during the time of evaluation. Among renovated clinics visited by the evaluation team, the renovations, according to clinic staff, supported increased supply of modern family planning services and client satisfaction. Internal Operations Alignment of SHOPS Objectives with Grantee Performance: Following OIG recommendations for conditional funding to JAFPP, SHOPS modified JAFPP funding from an expense reimbursement model to performance based grants in 2013. The performance based reimbursement model establishes performance benchmarks for increased family planning (FP) visits while ensuring service quality and client satisfaction, and offers incentives when benchmarks are met. Performance based grants reimburse costs contingent upon acceptance of revenue goals and financial sustainability targets, and performance￾based incentive payments for JAFPP meeting targets. The shift to a performance based grant introduced competition among JAFPP and other NGO clinics, and according to SHOPS, appeared to motivate JAFPP 31 to improve its organizational performance.37 JAFPP’s incentive program for increased clinic revenue and FP visits were linked to the performance based grants targets, resulting in increased revenues of 14 percent due to the 27 percent increase in FP visits, and a 13 percent increase in new clients (2013 to 2014). However, performance based grant targets did not include cost recovery. In 2014, SHOPS awarded performance based grants to four other NGO clinic partners, tying performance to FP visits, quality assurance and client satisfaction. NGOs are eligible to receive performance incentive payments on a quarterly basis based on reported results, with support from SHOPS to develop new or improved clinic management information systems to accurately capture FP visits and other data. Among grantees interviewed, ICCS was unable to sufficiently verify that it met targets (attributed to weak monitoring systems); as a result, ICCS was not paid an incentive bonus. According to the ICCS Director, targets were not set appropriately and they had to forego JD 140,000. In addition to more capacity, SHOPS staff believe that ICCS may have been able to meet targets if given more time, such as a one-year grace period. Contracting: UNRWA provides FP services to ten percent of the population of married women of reproductive age, and serves 39,000 women annually. UNRWA utilizes a MIS to track clients and clinic data including discontinuation rates, and targets its clientele through community outreach and through family planning teams at each UNRWA clinic. Although SHOPS attempted to support UNRWA with clinic renovations and capacity building, SHOPS and UNRWA were unable to resolve incompatible contracting mechanisms. As a result, SHOPS support was limited to in-kind furniture procurement and participation in EBM training. While UNRWA was designed to be included in CHW outreach referrals, only six percent of SHOPS clients were UNRWA clients. As UNRWA services are free, the voucher program did not apply to them. Project Management Systems and Approaches: SHOPS’ approach to project management was built on a foundation of baseline studies followed by action plans. For example, JAFPP organizational assessments were conducted at project start, followed by feasibility studies for price increases. Following the price increase, SHOPS conducted further studies on alternate avenues for sustainability, market studies for non-FP services, and alternate sources of revenue, followed by assessments of performance against plans. Staffing: SHOPS used a limited pool of consultants to provide recurring short-term technical assistance (STTA) through the life of the project, thus increasing efficiency by minimizing the learning curve and increasing effectiveness through depth of experience and relationships built with partners. Clinic Management Information System (CMIS) - Client Data and Tracking: SHOPS supported the upgrade of CMIS systems with partner NGOs. Within JAFPP, the evaluation team found that all visited clinics are using the CMIS, all are adhering to HR systems and KPIs generated by the CMIS, and data is used for decision-making. However, CMIS installation at NGO partner clinics was unsuccessful due to bugs in the system, IT provider-related issues, and the relatively short period of implementation of system. CMIS negatively impacted ICCS, whose reporting was hindered by CMIS complications. However, IFH was able to modify the software and integrate the CMIS into its own system. Maintaining current client contact information was challenging for SHOPS and partners. Client tracking at the sub- 37 SHOPS ‘SHOPS Year 3 Annual Report’ October 10, 2013. 32 grantee level created gaps for client verification due to incorrect or changed phone numbers. While the CMIS tracked returning clients, it did not track discontinuation, even though reducing discontinuation was part of the original design. Although continuation among new contraceptive users was checked at the three-month point through the Careline; discontinuation beyond three months was not tracked. SHOPS acknowledged that discontinuation should be tracked; time remaining in the project was insufficient to modify the software. UNRWA tracks FP discontinuation rates through its MIS system across five UNRWA countries. CONCLUSIONS: EFFECTIVENESS SHOPS increased demand for, access to, and quality of FP services with targeted clients, however cultural norms and practices that were not addressed through SHOPS are likely to have diluted the results for the ultimate goal of reducing fertility to slow population growth. Interventions Without the outreach/voucher initiative, it is likely that uptake among low-income women would have been far less. Although UNRWA’s participation was limited in this intervention it serves a significant target population; deepening involvement of UNRWA in this activity could substantially increase FP uptake and demand. Although the pharmacist coupon initiative was not considered effective in increasing FP demand, it could be made more effective by eliminating the multiple steps for women to procure commodities, and ensuring each clinic has a directory of participating FP commodity suppliers within a close radius. Steps could be eliminated by doctors dispensing coupons directly or by eliminating the need for an FP provider stamp on the coupon. While the post-partum family planning counseling at hospitals increased awareness of FP among women at the time of delivery, it has further potential to increase FP acceptance and use if it were supported by a FP provider referral list, a point of FP commodity distribution, and FP follow-up. Hospitals offer a key entry point to reach women of reproductive age, at a critical juncture of a woman’s reproductive lifecycle when she may be more receptive to FP after delivery. While mass media campaigns positively influenced FP awareness and uptake, the transient influence on uptake in family planning resulting from mass media points to less efficient use of resources than outreach and marketing through CHWs and health fairs. SHOPS was unable to introduce new modern family planning contraceptive methods, such as Cyclofem, due largely to the intransigence of MOH and FDA. Nevertheless, demand for modern FP methods indicates that a wider method mix would be well￾received and adhered to with products such as intra-uterine system, which would also help address FP discontinuation. Increasing the number of clinics undoubtedly increases access for women previously unserved; however, increasing access is not guaranteed, as some purchased properties required relocating to areas less convenient for women to access. Clinic renovations contribute to a positive image of JAFPP, which likely influenced an increase in FP access and/or demand. Internal Operations Performance based grants may be credited with raising performance at JAFPP, attributable to financial motivation. The operational expense reimbursement model with JAFPP in the first phase was at odds with financial sustainability objectives, as it perpetuated subsidization of JAFPP; performance based grants are better aligned with sustainability objectives. However, when awarding performance based grants to 33 new NGO grantees, additional time and assistance may be required to ensure M&E systems are in place to sufficiently report on verifiable targets. While M&E systems were able to track project performance, client tracking among grantees may require technical assistance to ensure data integrity. The lack of data tracked by SHOPS on discontinuation makes it challenging to address the issues of long-term FP and the impact on CYP and total fertility. Inability to contract with UNRWA resulted in a missed opportunity to engage with ten percent of the target population for FP services, dampening project potential in UNRWA catchment areas. Partners Because CHWs are trusted community members who are able to meet with women in their homes and often with other family members who influence FP decisions they were effective in generating demand for FP services and products among low-income populations. CHWs formed the backbone of the SHOPS referral network, and supported the results of NWDs and JAFPP in particular. As UNRWA and JAFPP serve a similar low-income clientele through a similar network of clinics and have a similar market share, the two organizations bear parallels for comparison. Without SHOPS support JAFPP client growth may more closely reflect UNRWA’s client growth figures. The referral network was effective in increasing demand for modern family planning products, particularly IUDs. The combined voucher and referral system was critical for increasing IUD demand and access, and had the greatest impact on CYP. However, the impact of the referral network could have been more effective with active use of the pharmacy referral booklet, increased proximity of pharmacies, and a more efficient coupon redemption procedure. Moreover, a more comprehensive CMIS that tracked discontinuation could have enabled the SHOPS network to seize missed FP opportunities. Although expansion of SHOPS to support additional NGOs is likely to have increased demand and access, interventions took place too recently to collect and interpret relative effectiveness. As ICCS has 17 clinics and a positive presence in low-income and predominantly Islamic communities, ICCS could serve as an Islamic voice for FP to a large audience. Supported by the Ministry of Awqaf and Iftaa, Islam can be a supportive voice for FP, and should be incorporated into future FP programming to increase effectiveness of family planning uptake, long-term use. FINDINGS: GENDER Question 5: To what extent were steps taken to address gender differentials and gaps? Female Provider Preference: In order to address the preference for women to receive medical services from females, SHOPS took steps to focus on including female providers in its activities, resulting in females representing 71 percent of doctors and 70 percent of pharmacists. A DOS survey reveals that, “the IUD is a provider-based method and women in Jordan prefer to receive IUD insertions from a female 34 provider. The majority of IUDs were inserted by female doctors 80 percent doctors and nine percent midwives), and only ten percent male doctors.”38 Reaching Women in their Homes: Recognizing the oral and communal nature of Jordanian society, SHOPS enlisted a cadre of trusted female community member CHWs who brought family planning to women inside their homes. The information delivered by CHWs additionally addressed the preference of women interacting with other women (rather than men) for health services. While CHWs attempted to include male family members in family planning discussions, data was not collected on the number of men included in such discussions, or the effectiveness of including them in discussions. Although CHWs distributed vouchers for free family planning visits to medical professionals and coupons for discounts on contraceptives, the process of redeeming the coupons often required multiple trips to the doctor, the pharmacy, and back to the doctor for contraceptives such as IUD or injectables that require doctors’ intervention. Women in focus groups pointed to the challenges of locating pharmacies and the multiple trips required as deterrents, specifically citing issues with child-care, cost, unavailability of transportation, and lack of familiarity with medical provider locations. Women also raised the issue of their reticence in telling pharmacists the exact type of contraceptive they wanted, as such direct discussions about intimate topics, particularly with male pharmacists, made them uncomfortable. To address this, CHWs began writing “prescription notes” that women could simply give to the pharmacist. CHWs are credited with recruiting almost 90,000 new acceptors of modern FP methods. Gender Differentials in FP Decision-Making: FP decisions in Jordan are not individual decisions, but rather communal decisions that are heavily influenced by family members and social norms. Female clients in focus groups raised the issue of husbands dictating use of contraceptives. Given women’s circumscribed movement outside the home, family members are typically aware of visits for family planning purposes. A client in Aqaba stated that she had to get her IUD during a visit to her family in Amman so that her husband would not know. Some women commented that husbands’ misconceptions about contraceptives are a deterrent to use, while others point to their husbands wanting more children. The evaluation did not uncover whether these misperceptions were more prevalent among men than women, or if the desire for more children was more prevalent among men than women. SHOPS created family-focused messaging with images of both men and women through the IUD and POP mass media campaigns; however, messaging was not differentiated for men and women. Preferences for Male Children: The social pressure to have at least one male child, and in some families, at least as many male as female children, contributes to reduced CYP and elevated fertility. Furthermore, more than 63 percent of women surveyed by DOS state a desire for at least four children.39 While SHOPS’ design does not acknowledge these preferences, they are fundamental factors in achieving SHOPS’ objectives. Other USAID projects such as JHCP and JCAP are directing efforts toward behavior change communication. 38 DOS. ‘Jordan Population and Family Health Survey 2012’. Department of Statistics., Department of Statistics, 2012, p. 77. 39 DOS. ‘Jordan Population and Family Health Survey 2012’. Department of Statistics., Department of Statistics, 2012, p. 64. 35 CONCLUSIONS: GENDER The steps SHOPS took to eliminate barriers to accessing services and increasing demand through female providers and CHWs undoubtedly contributed to achieving objectives, and met cultural preferences for female health care providers. As family planning decisions are heavily influenced by social norms and pressure from extended family and community, such norms must be at least considered if not directly addressed in order to influence family planning decision-making. Changing social preferences for male children may require considerable social research to identify messaging that is culturally acceptable to influencing family planning decisions. There is a need to continue to address the gender differentials to unravel the complexities of FP decision-making around male preference. While SHOPS family fairs, health fairs and social marketing campaigns targeted both men and women, there is a need to strengthen the involvement of the extended family and target gender differentials among groups influencing FP choices such as husbands, mothers and mothers-in-law. Outside of mass media campaigns, SHOPS did not systematically address men in influencing FP decision-making. Although men are recognized as significant influencers in family planning decisions, SHOPS’ lack of attention to gender differentials in messaging and engagement may have diluted achieving objectives. FINDINGS: SUSTAINABILITY Question 6: To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation? Question 7: What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue, and which are unlikely to continue and why? Question 8: Which of SHOPS’ partners (such as private doctors working through the voucher system) are most likely to continue practices that contribute to sustaining outcomes of increased a) access to; b) utilization; and c) quality of family planning services? Technical Interventions: SHOPS supported the Higher Population Council (HPC) in convening a task force for advancing public-private partnership (PPP), thus strengthening the engagement between MOH and the private sector on advocacy for continued public financing of contraceptive supply. One SHOPS NGO partner stated, “Partnership with the private sectors is critical. We never thought of what we could offer them, we always expected them to do things for us… this is the trick to moving forward.” SHOPS supported round tables for development of action plans, but to date no public private partnerships have culminated. As MOH stated “We recommend strengthening PPP in terms of knowledge sharing”, while HPC stated, “SHOPS needs to better strategize its planning and interventions with HPC in order to meet national needs. If they engaged with us more we could have contributed more to meeting SHOPS objectives such as the PPP initiative.” The CHW program was built and operated over the past 12 years with USAID funding, and was not designed for financial sustainability. Currently CHW activities have been assumed by JCAP; according to CCA, the CHW scope has been narrowed from working with husbands and family members to focusing only on women of reproductive age who are using traditional FP methods or none at all. JCAP plans to diminish funding for CHWs over the course of the project with the goal of increasing CHW financial sustainability. However, given CHWs’ focus on low-income populations (in Jordan and globally) it is unlikely that they will be able to generate income to continue serving this population. 36 SHOPS supported the upgrade and use of HMIS across all partner clinics. At JAFPP the HMIS is institutionalized and at HQ and clinic levels, driving evidence-based decision-making and supporting data for the incentive program, clinic-level revenue generation, and clinic visits. While the system is in use at JAFPP, the clinic staff depend on JAFPP HQ to analyze the data. The HMIS system in the new NGO grantees is not fully functioning due to problems with vendor performance and design challenges. SHOPS has attempted to resolve the problems with the new HMIS systems through the use of excel sheets. SHOPS activities such as coupons, vouchers, and media campaigns were not designed with the intent of sustainability but rather focused on having a direct effect on increasing demand for family planning and contraceptives. The voucher and referral system facilitated increased access to FP providers. However, maintaining client levels may not be sustainable without CHW outreach or vouchers after project closeout. Partner Practices: Under the post-partum counseling pilot with private hospitals, SHOPS conducted capacity building training for nurses, midwives, and physicians in FP counseling and distributed IEC materials, service provider tools, and equipment for post-partum contraception in four participating hospitals. The post-partum program was piloted for six months; while the pilot was not designed for sustainability, Specialty Hospital recognized intrinsic value of the activity and is continuing family planning counseling through nurses. Israa Hospital is considering continuing counseling through a designated post￾partum family planning counseling office and nurse. The Specialty and Issra Hospitals (two of the four partner hospitals) intend to institutionalize the post-partum family planning program “because it’s good PR and good for business.” In collaboration with NWDs, SHOPS designed and produced an evidence notebook, posting the content on professional society and medical information websites, including the Electronic Library of Medicine (ELM). In addition, a condensed Arabic version of the EBM/Reproductive Health (RH) notebook for community pharmacists was prepared. EBM was adopted by private doctors and integrated into local institutions such as the Jordanian Medical Association, affiliated British Medical Society and within the Jordan University for Science and Technology, Mu’ta University, Jordan University, and Hashemite University, and was published on the JPA website. SHOPS capacity building with pharmacists through academic detailing was designed for learning outcomes rather than sustainability of the intervention of training (outputs). Academic detailing entailed quarterly visits to 300 pharmacists in select high-volume pharmacies in Amman, Jerash, Madaba, Balqa, Irbid, Zarqa, Ma’an, and Aqaba. However, the JPA has stated that they will not continue this training without outside funding, SHOPS renovations and capacity building for clinics was designed to be sustained by the respective institutions. For example, clinic renovations, training and capacity building unified JAFPP into a premier Jordanian FP service provider - all clinics look identical, all are operating using the CMIS, all are adhering to HR systems, and KPIs. The culture of competition among clinics built through the incentive program has been institutionalized. JAFPP clinics and HQ HR, governance, IT, HMIS, and finance systems are in daily use. CONCLUSIONS: SUSTAINABILITY SHOPS activities were designed to focus on partners’ organizational sustainability rather than sustainability of technical interventions. For example, demand-based interventions such as coupons, vouchers, CHW outreach and referral were not designed for financial sustainability of intervention. On the other hand, clinic purchases and renovations, capacity building and systems instituted within partnering organizations were designed to be maintained, used, and sustained by partners. Some 37 interventions, such as the EBM training, were integrated into local institutions for sustainability of programming post project closeout. Other interventions need more support for local institutions to continue to implement activities once SHOPS ends. For example, the Public Private Partnership (PPP) program, while appreciated by the public and private sectors, does not have a tangible partnership mechanism for sustainability built into the design. Sustainability of Interventions: As management systems have been fully integrated into JAFPP, they will continue to be used. Likewise, EBM, clinical training approaches, and toolkits lend themselves towards continued application as they have been adopted by leading medical institutions, and training was delivered by Jordanians. Due to the ownership and integration of EBM within multiple Jordanian institutions, the EBM is likely to continue without further donor support. Conversely, interventions such as mass media campaigns, the coupon initiative, and voucher systems are donor-dependent and financially unsustainable, and will continue only with external funding. Likewise, CHWs are unable to continue without external funding. In order for demand for family planning services and products to continue, stakeholders such as pharmacists, doctors, clinics and MOH will need to recognize an inherent or financial benefit. Sustainability of Partners and Practices: SHOPS activities were designed to focus on partners’ organizational sustainability rather than sustainability of technical interventions. Although systems have been introduced and strengthened that could support sustainability, JAFPP leadership does not appear willing or interested in a self-sustaining financial model. In working with JAFPP, donors need to formally acknowledge JAFPP’s interests and goals, perhaps through formalized agreements such as Project Implementation Letters (PILS). The CHW program is not sustainable without continued donor funding. While CHWs continue to operate under the Jordan Communication, Advocacy and Policy Activity (JCAP), their current contracting mechanism places the program at risk of folding due to narrowed scopes of work and cost share/sustainability requirements. While SHOPS partners will continue to operate and contribute to increased access to, use of, and quality of family planning services, the level of gains will likely taper. While all NGO partner clinics will likely continue to provide modern FP services, the level of clinic visits and clientele growth rate are unlikely to continue at the same rate as with SHOPS support, due to unsustainability of the uptake in FP services associated with voucher redemption. However, UNRWA is likely to continue to provide services at the same rate of growth to its clientele, as the SHOPS intervention with UNRWA did not significantly influence new client growth due to minimal/nominal participation in the SHOPS referral and voucher program. Post-partum counseling is likely to continue as it is recognized by participating medical facilities as having inherent benefit and likely to continue. Nonetheless, when IEC materials run out, the hospitals are unlikely to spend their own fund to reprint materials. FINDINGS: LEARNING Question 9: To what extent were the theory of change (assumptions, pre-conditions) and project design used for the development approach appropriate to achieve USAID’s intended results? Question 10: What recommendations/suggestions does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? Theory of Change: SHOPS’ M&E plan did not have a documented theory of change to tell the story of how activities will lead to outcomes (at multiple levels), nor did it list assumptions upon which the 38 design was based. Although USAID best practices point to including a Theory of Change in AMEPs, SHOPS’ AMEP did not have one. As a result, determining the appropriateness of design is challenging. Nevertheless, based on the results framework and Cooperative Agreement, it can be surmised that SHOPS’ theory of change is based on the premise that if access to, demand for and quality of family planning services are increased, then CYP will increase, which should have an overall impact on CPR and/or TFR.40 The SHOPS results frameworks and design documents appear to follow the above theory of change thinking. An overarching assumption of SHOPS appears to be that by increasing access to and quality of family planning commodities and services, products will be used long-term, without interruption, and that CYP will increase and TFR will decrease. However, CPR and TFR were not higher-order indicators found within the SHOPS results framework. Alignment of M&E: According to results generated to date, SHOPS met most (81 percent) of its targets by the beginning of Year 541 and results have positively affected CYP. While discontinuation was identified within project objectives42, SHOPS did not track long-term use and discontinuation and thus the impact of discontinuation on CYP cannot be assessed. Assumptions: It appears that an assumption within SHOPS’ design is that once women begin using modern contraceptives they will be used long-term without interruption. These assumptions are not valid in Jordan, given cultural preferences for at least four children and the “necessity” for each family to have at least one male child (if not an equal number to females), and the practice of removing IUDs during Ramadan. “Overall, 48 percent of contraceptive users discontinued an episode within 12 months of starting its use [in Jordan].”43 Furthermore, a study of seven Demographic and Health Survey (DHS) countries identified the key determinants of a decline in fertility as an increase in contraceptive use; a decrease in the desired number of children; a reduction in unwanted fertility; and an increase in socio￾economic indicators. Countries where fertility rates have plateaued well above replacement level will need declines in preferences for children to complete their fertility transition.44 Cultural Norms: Cultural norms and practices can dilute the results of increased demand and access to quality FP services. Cultural factors need to be considered when designing activities and setting targets. Future programming in family planning and reproductive health should consider the overall impact of cultural norms and practices during the design of activities, taking into consideration the effect 40 SHOPS Cooperative Agreement. “Jordan’s stagnant Contraceptive Prevalence Rate (CPR) is the product not of one determining factor but rather of a set of variables and barriers that include market dynamics (for example, limited choice and limited access to female providers) and cultural factors (among the most obvious, the preference for large families, negative attitudes toward counseling, and provider bias). No one solution can address all these variables, and all stakeholders (private, public, and nongovernmental) will have to act strategically and in a complementary manner to remedy current conditions. Expanding family planning methods and the range of products available to Jordanians is paramount in addressing current population dynamics. Expanding FP/RH services in the private and non￾governmental sectors is one way to address these challenges that is likely to make a positive impact.” 41 All targets set for Year 4 were met according to SHOPS Year 5 AMEP were met with the exception of 3 indicators: number of new family planning methods successfully introduced through USG-supported programs, number of facilities rehabilitated for UNRWA, and adherence to management authority procedures/policies at JAFPP. Targets for indicators such as CYP, were not set as SHOPS is not the sole contributor to the indicator, while indicators such as percent of women counseled who take up a modern FP method did not have set target as the Tiahrt Amendment prohibits setting targets for numbers of acceptors of FP. 42 “The program will contribute to increased availability and use of modern contraceptive methods, a reduction in the current high rates of discontinuation and a reduction in unmet need.” SHOPS Cooperative Agreement. 43 DOS. Jordan Population and Family Health Survey 2012. Department of Statistics., Department of Statistics 2012, p. 69. 44 John Bongaarts, ‘The Causes of Stalling Fertility Transitions’, The Higher Population Council (HPC), June, 2005 39 of cultural preferences such as family size and male children when establishing targets. The impact of these cultural norms such as the influence of mothers-in-law was underestimated and their influence on family planning/family size practices was not considered during the activity design. This is particularly relevant to family planning decision-making and its ultimate impact on total fertility and contraceptive prevalence rates. According to international experience, influencing CPR or TFR after a plateau requires a decline in preferences for children to complete their fertility transition.45 This cannot happen in the absence of a discussion of cultural attitudes, preferences and norms. Partners: Among other partners in the SHOPS network, SHOPS outcomes were curtailed with ICCS and UNRWA due to design of contracting mechanism. While performance based grants were effective in improving outcomes with JAFPP, with ICCS it became clear to SHOPS management that grantees may need a one year grace period to ensure that compliance with performance is possible, before implementation of conditionalities. Contracting obstacles with key partners such as UNRWA, which serves a critical market share, limited collaboration to UNRWA doctor participation in EBM, limited CHW referral to UNRWA clinics, distribution of IEC materials to UNRWA, and provision of furniture to UNRWA (in lieu of clinic upgrades) in 2015. SHOPS referrals to UNRWA represented only six percent of all clients within the SHOPS referral network. Contracting with UNRWA limited the depth to which UNRWA was engaged. Relative to JAFPP, it appears that the theory of change was that if governance and management systems were sufficient, JAFPP leadership would use the systems. This theory is predicated on the assumption that JAFPP leadership has willingness and intent for financial sustainability within the timeframe of the project. As discussed earlier, this assumption is not valid. The private sector serves 56 percent of contraceptive users, while the public sector serves the rest.46 SHOPS focused on engaging private sector actors in family planning for improved access to, demand for, and quality of family planning services, and engaged with the public sector (MOH, HPC) on establishing public private dialogue. Approaches to family planning in the public sector and private sector differ, with respect to availability of methods, data collection, among others. Influencing measures such as CYP and TFR also requires coordination in measurement, unifying definitions, standards and data collection among all stakeholders. Other groups working on family planning such as Iftaa, focus on dispelling misconceptions about FP through Islam. CONCLUSIONS: LEARNING Documenting a theory of change can assist activity designers and implementers in identifying gaps in inputs, outputs, and outcomes that are necessary to achieve objectives. Furthermore, explicitly stating assumptions assists in identifying factors over which the activity does not have control or influence. Of particular importance are cultural norms and practices. As SHOPS worked on service delivery, behavior change communication and demand generation, it would have been helpful to define the extent to which cultural norms and practices influence these activity outcomes and targets. 45 John Bongaarts, ‘The Causes of Stalling Fertility Transitions’, The Higher Population Council (HPC), June, 2005 46 DOS. ‘Jordan Population and Family Health Survey 2012’. Department of Statistics., Department of Statistics 2012, p. 69. 40 Moreover, cultural norms which often negate these assumptions were not accounted for. Cultural norms and practices can dilute the results of increased demand and access to quality FP services. Cultural factors need to be considered when designing activities and setting targets. When achieving objectives depends on counterpart behavior and will, ensuring congruity of implementing partner and counterpart objectives is critical. Documenting this alignment through an agreement that acknowledges obligations and conditionality of funding could help to avoid such situations in the future. Contracting mechanisms can both facilitate and hinder project progress and outcomes. In the case of an institution like UNRWA where bureaucracy poses contracting obstacles, flexibility in contracting and innovative approaches to engagement and collaboration may need to be considered. While SHOPS focused on engaging private sector actors in family planning for improved access to, demand for, and quality of family planning services, there is a need to both expand within the private sector and integrate with the larger family planning service delivery stakeholders, such as MOH, and more actors in the private sector. There is a need to ensure that public and private sector approaches to family planning are unified and can benefit from synergies. For example, work completed by Iftaa on Islamic misconceptions about FP could have been better integrated into SHOPS outreach and clinic interface with clients. Coordination in influencing measures such as CYP and TFR through measurement, unifying definitions, standards and data collection among all stakeholders, could form the basis for collaboration in any follow-on programming. 41 RECOMMENDATIONS JAFPP 1. If JAFPP sustainability is a priority for USAID, a Project Implementation Letter (PIL) or Memorandum of Understanding (MOU) stating this agreement should be signed with JAFPP. Terms of the PIL/MOU should include intent for financial sustainability, agreement to reducing Head Quarter (HQ)/clinic expense ratios, and agreement to set prices and services based on market research for each location. 2. Assistance to JAFPP should focus on clinic decentralization and facilitating clinics to operate as individual business units (profit centers). 3. Financial assistance to JAFPP should be predicated on meeting performance targets, and funding should be provided in tranches conditional upon meeting benchmarks such as cost recovery targets. Support for HQ costs should be minimal with a phase-out plan. 4. USAID should support market research to determine whether JAFPP could increase cost recovery and its client base by offering a wider range of medical services for families as a one￾stop shop for all maternal and child health needs. Effectiveness 5. If USAID wishes to ensure increased access to FP, commodities and services should continue to be subsidized through vouchers for FP consultation products until a critical mass has been built for sustained social change in attitudes and decision-making for long-term, continuous use of FP. 6. Coupons for FP products should be continued but the process should be streamlined to eliminate the need for women to make multiple trips. A directory of participating pharmacists should be readily available in print or through mobile technology to eliminate uncertainty of travel and locations. Relationships should be facilitated between clinics and pharmacies; coupon expiration dates should be eliminated. 7. USAID should continue to fund CHWs and leverage them as a key resource to expand community outreach. USAID should strengthen CHW-clinic collaboration and create direct and active CHW linkages to UNRWA and other NGO clinics in the target catchment areas in order to increase FP demand. 8. Expand services of CHWs to include pre-marital girls, preconception, extended family members living in the same house and male relatives (husbands, fathers, brothers), and Syrian communities. 9. Health programming should focus heavily on addressing misconceptions about the side effects of contraceptives, targeting medical practitioners, clinic staff, pharmacists, CHWs, and the general public in order to increase the use of long-term methods such as IUDs, and to reduce discontinuation. Given cultural practices and norms in which information is delivered in-person, a cost/benefit analysis should be conducted of mass media vs. social means such as CHWs and if possible, the impact of each on CYP. 10. Post-partum counseling through private hospitals should be expanded, following the model SHOPS supported at Specialty Hospital. 11. Strengthen PPPs by establishing a PPP grant fund for implementation of joint activities by private￾public FP actors. 12. All grantees should be required to use a Management Information System such as that used by JAFPP and UNRWA for evidence based decision-making and standardization of reporting protocols. Provide technical assistance to grantees for collecting client data and tracking clients to determine FP behavior, FP purchases, fertility and FP discontinuation. Bar-coded or chip- 42 verified cards could be used for tracking client FP behavior by swiping or scanning the card with each visit to record the frequency and types of services and products accessed, discontinuation (such as IUD removal), and demographic information. 13. Performance based grants should have a one-year grace period to allow grantees to build adequate capacity and systems to measure performance and meet targets. 14. Performance based grants should continue to be used as a contracting mechanism, directly tying key benchmarks to activity/project objectives of increased FP demand, access and cost recovery. 15. Set long-term FP use as a clear project target and track discontinuation Gender 16. Continue to focus on female providers to address cultural preference for female doctors and pharmacists. 17. Support a study on FP use in Ramadan and build an education/counseling program around findings, targeting religious leaders, clients, community members, men, and FP providers. Link activities to a Family Planning Fatwa issued by the Jordanian Iftaa Department to supplement work conducted on FP with religious leaders under Jordan Health Communications Partnership (JHCP). 18. Programming objectives, activities, and indicators need to acknowledge cultural norms and practices relative to family size and preference for male children. Such acknowledgement should include behavior change communication to change these norms and practices; identify other stakeholders who are conducting behavior change communication to change these norms and practices; or lower expectations to account for cultural norms and practices. 19. If behavior change communications is conducted it should be preceded by research to carefully identify messaging and target markets, and recognize sensitivities relative to the perception of imposing Western agendas. 20. Programming objectives and activities, and indicators need to acknowledge husbands’ role in family planning decisions; gender sensitive interventions and messaging should be designed accordingly. Sustainability 21. The approach of integrating interventions into local institutions should be continued and replicated in future programming. For example, the model of integration of training programming within Jordanian institutions, as seen with EBM integration into the Jordanian Medical Association, Jordan University for Science and Technology, Mu’ta University, Jordan University, and Hashemite University, lends itself to local ownership and sustainability. 22. Recognize that although not financially sustainable, CHWs can be leaders of behavior change; reinforce and expand geographic coverage of messages to reach a tipping point for long-term social change. Meanwhile, explore possibilities for financial sustainability of CHW programming by linking them to pharmaceutical companies whose products they market. 23. Recognize that although not sustainable, funding free access to FP services and products through voucher and coupons can contribute to CYP and will contribute to reaching a tipping point of social change. Learning 24. Program design should include a theory of change describing the causal linkages among results and activities. Assumptions should be explicitly documented, as should issues that are out of the sphere of control or influence of the implementing partner. 43 25. Research and follow successful practices in Muslim countries for reducing TFR and taking a holistic approach that includes integrating all factors influencing family planning decision-making. 26. Require IPs to track discontinuation. 27. Produce highly targeted messaging and approaches to including men, mothers-in-law, and religious leaders in family planning attitudes and behavior. 28. Enlist and build a broad base of support among the Royal Court, Ministries, and other thought leaders in campaigns to change preferences for at least four children and for male children to impact total fertility rate. 29. Incorporate work done within Iftaa to ensure Islamic contributions to combating misconceptions of FP. 30. Identify compatible sub-grantee mechanisms for UNRWA. 44 ANNEXES Annex 1: Evaluation Statement of Work Annex 11: Evaluation Design Report Annex 111: Performance Monitoring Plan Annex 1V: Contact List Annex V: Bibliography 45 ANNEX I: EVALUATION STATEMENT OF WORK Strengthening Health Outcome through the Private Sector (SHOPS) Performance Evaluation Statement of Work June 3, 2015 I. INTRODUCTION USAID/Jordan requires an external final evaluation of the USAID/ Strengthening Health Outcomes in the Private Sector (SHOPS) Activity. The objective is to provide USAID with overall feedback and strategic recommendations related to expanding the access, quality and utilization of Family Planning (FP) services through partnership with the private and non-governmental sector in Jordan. The evaluation results will be used in guiding design and implementation decisions in future programming. The purpose of the final performance evaluation of SHOPS is to assess how effective the project was in meeting its objectives of increasing the demand for, access to, and quality of family planning services; to assess the sustainability of project outcomes and practices; to identify factors contributing to outcomes and sustainability; and to provide recommendations on how to make future projects with similar objectives more effective and more sustainable. The external evaluation will use quantitative and qualitative approaches to assess how effective the project was in meeting its objectives of expanding the access, quality, and utilization of family planning services in the private sector; to assess the sustainability of project outcomes and practices for NGOs and the commercial sector; to identify factors contributing to outcomes and sustainability; and to provide recommendations on how to make future projects with similar objectives more effective and more sustainable. The evaluation will focus particular attention on the Jordan Association for Family Planning and Protection (JAFPP), the country’s only non-governmental entity providing woman-to-woman family planning services, which has received significant financial and technical support from USAID. Details of the project to be evaluated: Program: USAID/Jordan Population and Family Health Office Project Title: USAID – Strengthening Health Outcomes in the Private Sector (SHOPS) Award Number: 278-A-00-10-00434-00 Award Dates: 20 July 2010 – 30 August 2015 Funding: $30,000,000 (amended to $38,000,000 in 2012) Implementing Partner: Abt Associates Inc. AOR: Dr. Nagham Abu Shaqra II. BACKGROUND A. Project description 46 SHOPS is funded through a Leaders with Associates Cooperative Agreement under USAID’s Strengthening Health Outcomes through the Private Sector Project. The local project name is “Strengthening Family Planning”, in Arabic, “Ta’ziz Tanzim Al Usra” or “Ta’ziz” in short. This local activity name is used for all of the initiatives activities, briefing materials, invitations, announcements and all external and public communications funded by USAID through this Associate Award. SHOPS began in 2010 with the primary goal of expanding the access, quality and utilization of Family Planning (FP) services by partnering with the private and non-governmental sector in Jordan. The various interventions within this activity were designed to overcome the challenge of Jordan’s plateauing contraceptive prevalence rate and total fertility rate. Jordan’s most pressing family planning challenges that were to be addressed by SHOPS included the following:  Increasing the use of existing methods, particularly underutilized methods such as injectable contraceptives and implants;  Increasing the range of product options in the injectable/implant category;  Developing marketing and behavior change strategies to improve the acceptance of hormonal methods;  Maintaining and expanding current collaborative relationship with pharmaceutical companies while exploring new partnership opportunities; and  Removing medical barriers (provider bias). In supporting JAFPP, SHOPS intended JAFPP to benefit from outreach expertise through the Jordan Health Communication Project (JHCP) and Private Sector Project (PSP)/Jordan’s FP informational materials for placement in clinics and distribution to catchment areas. In addition, the SHOPS project team would help JAFPP collaborate with PSP/Jordan’s outreach staff to refer clients to JAFPP clinics and to connect with HSS II/MOH voluntary community health committees. SHOPS also worked with the United Nations Relief and Works Agency (UNRWA) to strengthen its ability to provide FP services in its 24 clinics which serve approximately 1.9 million Palestinian refugees in Jordan. The original expected outcomes of the project were as follows:  Strengthened JAFPP management and governance systems  Increased JAFPP financial sufficiency Improved quality of FP/RH services at JAFPP and UNRWA clinics  Increased demand for and access to private sector/NGO services  Expanded method mix and product choice in the private/NGO sectors B. Project Modifications SHOPS was awarded in FY2010 with the original scope to include a) strengthened JAFPP management and governance systems; b) increased demand for and access to private sector/NGO services; c) improved quality of FP/RH services at JAFPP and UNRWA clinics; d) expanded method mix and product choice in the private/NGO sectors; and e) increased JAFPP financial self-sufficiency. In October 2011, SHOPS’ Cooperative Agreement was modified and the program budget increased by $8 million to reach a total of $38 million. The original program and project objective to increase access, quality and utilization of family planning services in Jordan remained the same. However, the amendment broadened the scope, the budget and the private sector partners contributing to project outcomes. The 47 amended SOW incorporated recommendations from the PSP assessment as well as recommendations related to the sustainability of efforts with JAFPP and UNRWA. The amended SOW added the following activities to SHOPS’ existing SOW:  Expanded geographical access to family planning/reproductive health (FP/RH) services through purchase and renovation of JAFPP clinic property  Improved quality of family planning services at UNRWA  Increased demand for family planning through NGO outreach  Improved access to quality services through expansion of provider network In Year 3 SHOPS expanded its focus to employ innovative methods including the use of evidence-based medicine and Critical-Appraised Topics to support its efforts to improve quality and access by attempting to reduce provider bias and encourage informed choice. These interventions were designed to support USAID’s objectives to increase the access and use of modern FP methods, reduce discontinuation rates, and address unmet needs for FP services. C. SHOPS Results Framework D. Review by the Office of the Inspector General (OIG) In 2012 SHOPS worked on the financial sustainability of JAFPP including recommendations to increase the price of JAFPP services. In 2014, the OIG assessed Ta’ziz (SHOPS) with a particular focus on financial sustainability of JAFPP. The audit contained a finding and recommendation relative to JAFPP, and addressed with a management comment from USAID: Finding: JAFPP did not make progress toward its financial sustainability target. While JAFPP offers services at a steep discount, it loses money on each client. Increasing the number of clients thus accelerated JAFPP’s revenue losses, putting the sustainability of project achievements at risk. Recommendation: USAID/Jordan require the Jordanian Association for Family Planning and Protection to formally commit to specific sustainability measures (e.g. price increases) as a condition of receiving additional assistance under the agreement. IR 3: Increased demand for FP products and services in the total market IR 2: Increased access to and improved quality of private sector FP services IR 1: Strengthened management and governance systems and increased financial sustainability at JAFPP (and other NGOs) SHOPS Goal: To expand the access, quality and utilization of family planning services in Jordan 48 Management Comment: The mission introduced performance-based grants and awarded such a grant to JAFPP on January 13, 2014. The new agreement included sustainability benchmarks that the organization must meet, and it becomes effective February 28, 2014. Therefore, we acknowledge that the mission made a management decision. III. Evaluation Questions In addressing the stated purpose, the evaluation will explicitly answer the questions stated below. JAFPP: 1. To what extent has SHOPS’ assistance strengthened JAFPP’s management, financial and governance systems to operate sustainably? Effectiveness: 2. How effective were the various interventions/mechanisms and approaches (such as NGOs organizational development capacity building, grants programs, and behavior change communication campaigns) in achieving the project’s objectives of increasing demand for, access to and quality of family planning services with targeted clients and providers? 3. How did SHOPS’ internal operations and organization (SHOPS strategic planning, project management, financial management, communications, grant-making process and grant management, M&E systems and indicators, staffing structure) assist or hinder achieving project targets and objectives? 4. To what extent were project partners (JAFPP, IFH, ICCS, UNRWA, private providers, etc.) effective in increasing access to, demand for and/or the quality of family planning products and services as a result of SHOPS support? 5. To what extent were steps taken to address gender differentials and/or gaps? Sustainability: 6. To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation? 7. What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue without additional support, which are unlikely to continue and why? 8. Which of SHOPS’ partners (grantees, doctors, community health workers) are most likely to continue practices that contribute to sustaining outcomes of a) increased access; b) utilization; and c) quality of family planning services? Learning: 9. To what extent were the Theory of Change (assumptions, pre-conditions) and project design used for the development approach appropriate to achieve USAID’s intended results? 49 10. What recommendations/suggestions does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? IV. EVALUATION Methodological APPROACH In answering the evaluation questions particular emphasis will be placed on the objectives as stated earlier: 1. Strengthening the management and governance systems and increasing the financial sustainability of partner NGOs, particularly JAFPP; 2. Increasing access to and quality of private sector family planning services; and 3. Increasing demand for family planning products and services among the general population as a result of the interventions. The detailed evaluation design will consider the following types and numbers of grantees with interventions as described in the table in Annex I. Grantees: 8 Service Delivery Point (SDP): 80 Network Doctors (NWD): 300 Pharmacists: 300 Community Health Workers (CHWs): 170-200 Community level beneficiaries (women): 259,177 Private Hospitals: 4 The evaluation will use the following approaches for data collection: Desk Review (Q1, Q2, Q3, Q4, Q5, Q6, Q8): An in-depth review will be conducted of SHOPS’ contract and modifications, AMEP and indicators, implementation and work plans, Annual and Quarterly Reports, grantee reports, JAFPP financial and management documents, research and special studies conducted by SHOPS, training curricula, and databases of information maintained by SHOPS and grantees. In-Depth Interviews (Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q10): In-depth interviews will be conducted with SHOPS management and staff, grantees (JAFPP, ICCS, GUVS, IFH) Ministry of Health, JDFA Focus Group Discussions (Q3, Q5): Group discussions will be conducted to explore issues in-depth, to triangulate data, and to solicit the input of private providers, pharmacists, general practitioners, and community health workers. Survey (Q3, Q6: Given the large number of clients, pharmacists, general practitioners and private providers that SHOPS touched, surveys will be conducted with these groups to provide quantitative information. All people-level questions will be gender disaggregated. The team will also identify the questions that will require examination of gender specific or gender differential effects. 50 V. TEAM COMPOSITION In accordance with guidance provided in USAID ADS 203 for composition of evaluation teams, the team will be composed of experts with significant knowledge of health and family planning in developing countries and in Jordan in particular, with skills and experience in the following areas:  Experience in evaluation and assessment design methodologies;  Experience implementing and conducting USAID assessments and evaluation;  Expertise in family planning, behavior change communications, health care cost recovery, and evaluation of organizational “fitness;”  Experience in managing evaluations;  Excellent writing and communication skills with experience in producing team-based, collaborative reports that are learning-oriented;  Skills in qualitative and quantitative data analysis;  Local language skills; and  Familiarity with USAID evaluation policy. In order to meet the requirements of team composition, ensure data quality, and contribute to building capacity of local evaluation specialists, the following is suggested for team composition: 1. Team Leader/Evaluation Specialist: Primary point of contact for assignment with responsibility for assigning team duties, managing activities, resources, and team member performance to meet objectives; leadership role in analysis, final reporting and presentation. Leads meetings with USAID; leads in creating design methodology and instruments; conducts literature review; participates in interviewing and data collection; assigns team activities and facilitates smooth team operations; ensures that findings, conclusions, and recommendations answer evaluation and assessment questions and meet USAID purposes; produces/finalizes assessment and evaluation tools and final report; ensures final reporting meets USAID requirements. 2. Family Planning/Health Sector Technical Specialists (2-3): Focuses on his/her particular areas of specialization (behavior change communication, grants, NGO capacity building, service quality, cost recovery) in conducting desk review, design methodology and data collection instruments; co-leads training and pilot testing for data collection; participates in data collection, analysis and interpretation; produces report sections as assigned by Team Lead. 3. Senior Local Evaluation Expert (through local sub-contractor Integrated Services): Leads in production of Evaluation Design Report; ensures evaluation instruments appropriately address the evaluation questions; ensures data analysis follows rigorous process of linking findings, conclusions and recommendations; leads in structuring final presentation and report to address evaluation questions; ensures final report meets USAID evaluation requirements. 4. Data collectors (through sub-contractor Mindset): Trains data collectors and conducts quantitative and qualitative data collection; conducts data entry and data cleaning; provides data in formats required by evaluation team. 51 5. Data Quality Manager (through local sub-contractor Integrated Services): Oversees quality of quantitative and qualitative data collection; supervise training; oversee pilot testing; ensure revisions to tools based on pilot testing; ensure data quality and adherence to USAID requirements; standardize qualitative data collection; build data quality capacity of Mindset. 6. MESP Technical Specialist: Ensures quality and timeliness of evaluation deliverables; serves as main point of contact between evaluation team, USAID, MESP, and subcontractors; supports evaluation team with local health sector knowledge; resolve evaluation implementation challenges. 7. MESP Senior M&E Advisor: Provide guidance on evaluation methodology, process, reporting, and USAID evaluation requirements; and ensure quality and timeliness of deliverables. Depending on the individual role, team members may be employed for the full or partial duration of the evaluation in order to efficiently and effectively address needs. The Team Leader will be engaged for the full duration of the assignment. VI. MANAGEMENT AND REPORTING A. Management The Evaluation Team will be supervised by MESP who will manage communications with USAID through the Program Office. The Evaluation Team meet with USAID prior to commencing the evaluation to clarify outstanding issues, and will brief USAID at critical points such as after pilot testing instruments, upon completion of field work, and after data analysis to present a final briefing. The evaluation will be carried out in its entirety in Jordan; a draft report is required prior to the Team Leader’s departure from Jordan. B. Reporting Evaluation reporting will include the following: 1. Periodic updates to USAID on progress of the evaluation. 2. Debriefing presentation using PowerPoint for USAID on initial evaluation findings, conclusions, and recommendations. 3. A workshop with project stakeholders to share initial evaluation findings, conclusions, and recommendations after the incorporation of USAID feedback (if requested by USAID). 4. A final report not to exceed 30 pages (excluding annexes) organized as follows: a. Executive summary b. Table of contents c. Statement of the evaluation purpose d. Background of the project e. Main evaluation questions f. Evaluation methodology (summary in evaluation body with full methodology included in the Evaluation Design Report) g. Evaluation findings, conclusions, recommendations, and lessons learned (if applicable) of the evaluation. 5. The following are required in the report: a. All evaluation questions must be explicitly answered; b. Recommendations must be based on findings; 52 c. Recommendations must be highly specific and actionable, clearly defining who does what and how; d. Limitations on data must be clearly stated; e. Copies of the evaluation statement of work, sources of information, data collection instruments, and aggregated data used in analysis must be included as appendices in the final report. 6. Approved evaluation report uploaded to USAID’s Development Experience Clearinghouse at https://dec.usaid.gov. Data collected during the course of the evaluation will be securely stored by MESP and will be available to USAID upon request. VII. DELIVERABLES AND TIMELINE The evaluation is expected to take place from mid-May through mid-August, with data collection completed in June and reporting conducted in late July and early August due to the Eid holiday and scheduling conflicts of team members. An attempt will be made to complete field work prior to the beginning of Ramadan in mid-June; although field work may be conducted during Ramadan it is likely to be at a slower pace. The Evaluation Team will work a six-day workweek although the formal work week in Jordan is Sunday through Thursday. The Evaluation Team shall budget for all travel and administrative support costs within Jordan as needed. The Evaluation Team is expected to arrange all logistics needed for the evaluation. Deliverable Timeline (working days after start of evaluation)* Level of Effort (LOE) Estimated Timeline Weekly progress reports or meetings with USAID to provide updates, verify and/or clarify information, and address any logistical issues As requested Background preparation, work plan, document review 6 6 May 10-May 16 Detailed evaluation design report (design, methodology, tools). 12 8 May 17-May 25 Data collection training and pilot testing 7 May 26-June 4 Field data collection 15 June 6-June 21 Data analysis 9 June 22-June 30 Debriefing presentation for USAID in PowerPoint on evaluation findings, initial conclusions and recommendations 42 1 June 30 Presentation to project stakeholders 43 July 7 Draft report incorporating comments from the USAID debriefing and presentation 6 August 14 USAID provides written comments on the draft evaluation report 18 (from day of receipt) Revised report 24 (from day of receipt) 3 Total 55 53 * Based on a 6-day work week 54 ANNEX I: SHOPS INTERVENTIONS SHOPS Interventions Total # Intervention/s Number /intervention Location 1. SHOPS Grantees (NGOs) 8 Grantees  JAFPP 23 Clinics Clinic renovation 22 clinic Capacity building 23 clinics Family fairs 8 Organizational development (Business plan, admin and management Manuals/guides)  UNRWA 24 Clinics Capacity building 24 clinics Provide equipment and furniture 24 clinics Quality assurance  ICCS 17 Clinics Clinic renovation 3 clinics Organizational development Capacity building Quality assurance  AJA 4 Clinics Family fairs 2 FF Business plan Organizational development Capacity Building  HLC 3 Clinics Clinic renovation 2 clinics Family fairs 3 FF Organizational development (HR policy procedures manual) Capacity Building 55 Total # Intervention/s Number /intervention Location Quality assurance  FHI 9 Clinics Clinic renovation 4 clinics Capacity Building Quality assurance  CCA 0 Clinics HCWs Training Implement outreach visits 120 HCWs  GUVS 0 Clinics HCWs Training Implement outreach visits 70 HCWs 2. Private GP Network doctors (NWD) 30047 Referral (vouchers) 185 FP advocacy 85-90 Academic detailing visit 1200 visit for 300 doctor/year EBM (Y4 report) round table discussion 11 session for 305 doctor in y4 and 10 for 211 in y3 Training 300 3. Pharmacists 300 Coupon Pilot Initiative 110 pharmacy Training 300 Academic detailing visit 1200 for 300 Pharmacist/year EBM seminars 12 seminars (981 attendees) 4. Community Health 170-200 Behaviour Change 200 47 25 doctors of the NWD were not active 56 Total # Intervention/s Number /intervention Location Workers (CHWs) 5. Community level beneficiaries (women) Household outreach visit 259,177 (Y4 report)48 Edutainment Lectures (IUD) 12,000 women 6. Private Hospital Initiative 4 Postpartum Contraceptive initiative 4 48 Since September 30, 2014 outreach visits moved to JCAP 57 ANNEX I1: EVALUATION DESIGN REPORT INTRODUCTION Jordan’s high population growth rate is constraining the country’s ability to achieve socio-economic progress and maintain stability. If not addressed, the population will double by 2040. Increasing the use and demand for voluntary family planning and reproductive health services is critical to the achievement of this objective. The Strengthening Health Outcomes in the Private Sector (SHOPS) Activity, a significant contributor to family planning in Jordan, is ending operations in August 2015. The objective of the final evaluation of SHOPS is to provide USAID with strategic recommendations related to expanding the access, quality and utilization of Family Planning (FP) services through partnership with the private and non-governmental sector in Jordan. The evaluation will help USAID obtain information to more effectively allocate resources for future programming. The evaluation results will be used to inform the new design especially with investments related to the Jordan Association of Family Planning and Protection (JAFPP) and commercial doctors. EVALUATION PURPOSE AND SCOPE The purpose of the final performance evaluation of SHOPS is to assess how effective the project was in meeting its objectives of increasing the demand for, access to, and quality of family planning services; to assess the sustainability of project outcomes and practices; to identify factors contributing to outcomes and sustainability; and to provide recommendations on how to make future projects with similar objectives more effective and more sustainable. The external evaluation will use quantitative and qualitative approaches. . The evaluation will focus particular attention on the Jordan Association for Family Planning and Protection (JAFPP), the country’s only non-governmental entity providing woman￾to-woman family planning services, which has received significant financial and technical support from USAID. The ten key evaluation questions that will guide the final performance evaluation are: JAFPP:  To what extent has SHOPS’ assistance strengthened JAFPP’s management, financial and governance systems to operate sustainably? Effectiveness:  How effective were the various interventions/mechanisms and approaches (such as NGOs organizational development capacity building, grants programs, and behavior change communication campaigns) in achieving the project’s objectives of increasing demand for, access to and quality of family planning services with targeted clients and providers?  How did SHOPS’ internal operations and organization (SHOPS strategic planning, project management, financial management, communications, grant-making process and grant management, M&E systems and indicators, staffing structure) assist or hinder achieving project targets and objectives? 58  To what extent were project partners (JAFPP, IFH, ICCS, UNRWA, CCA, GUVS, private providers, etc.) effective in increasing access to, demand for and/or the quality of family planning products and services as a result of SHOPS support?  To what extent were steps taken to address gender differentials and/or gaps? Sustainability:  To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation?  What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue, which are unlikely to continue and why?  Which of SHOPS’ partners (grantees, doctors, community health workers) are most likely to continue practices that contribute to sustaining outcomes of a) increased access; b) utilization; and c) quality of family planning services? Learning:  To what extent were the Theory of Change (assumptions, pre-conditions) and project design used for the development approach appropriate to achieve USAID’s intended results?  What recommendations/suggestions does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? EVALUATION DESIGN In answering the evaluation questions particular emphasis will be placed on the objectives as stated earlier:  Strengthening the management and governance systems and increasing the financial sustainability of partner NGOs, particularly JAFPP;  Increasing access to and quality of private sector family planning services; and  Increasing demand for family planning products and services among the general population as a result of the interventions. The evaluation will consider the following types and numbers of grantees with interventions as described in Annex I.  Grantees: 8 (Jordan Association for Family Planning and Protection (JAFPP), Islamic Center Charity Society (ICCS), Aman Jordan Association (AJA), Hussein Labor Clinics (HLC), Institute for Family Health Institute (IFH), Circassian Charity Association (CCA) and General Union of Voluntary Societies (GUVS)  Technical Assistance: United Nations Relief Works Agency (UNRWA)  Service Delivery Points (SDP): 50-80  Network Doctors (NWD): 300 59  Pharmacists: 300  Community Health Workers (CHWs): 170-200  Community level beneficiaries (women): 259,177  Private Hospitals: 4 DATA COLLECTION METHODS The evaluation team will conduct the desk review, interviews, focus group discussions, and SDP observations. This will allow team members to probe further into issues raised by stakeholders. Surveys will be conducted by the local contractor Mindset. Desk Review (Q1, Q2, Q3, Q4, Q5, Q6, Q8): An in-depth review will be conducted of SHOPS’ contract and modifications, Activity Monitoring and Evaluation Plan (AMEP) and indicators, work plans, Annual and Quarterly Reports, grantee reports, JAFPP financial and management documents, research and special studies conducted by SHOPS, training curricula, and databases of information maintained by SHOPS and grantees. Special studies reviewed are listed in Annex V1. In-Depth Interviews (Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q9, Q10): Between 35-37 interviews will be conducted with individuals from government, the USAID Health Office, USAID health implementing partners, SHOPS staff, SHOPS grantees, Jordan Pharmacists Association (JPA), and the Jordan Food and Drug Administration (JFDA). Interviewees will be selected based on participation and implementation in key interventions under evaluation, as well as sector expertise. The in-depth interviews will be guided by semi-structured questions covering the evaluation topics to capture perspectives and activities of each informant category, revolving around the evaluation questions relevant to each stakeholder. The guides are designed to preserve the potential for a relatively free￾flowing conversation, while creating a standardized format to facilitate a reliable, comparative analysis of data pertaining to the evaluation questions for triangulation of information from multiple stakeholder perspectives. While questions are based on the evaluation’s overarching questions, they may vary depending upon the identity of the informants. For example, questions asked of some informants may not be asked to others due to varying levels of involvement in the various interventions or knowledge of specific interventions, while relative importance of questions will vary by the type of stakeholder. Interview guides will take into account the need to capture gender differentials. Where possible and relevant, answers will be gender disaggregated. Interview guides will be tailored to each of the stakeholder clusters, with each interview lasting 45-60 minutes. Most interviews will be conducted in Arabic; some interviews conducted by the team leader will be conducted in English/Arabic with translation. Tools will be tested and refined as necessary. Draft guides and tools are contained in Annex III. To take advantage of the extensive knowledge about family planning that exists within SHOPS and to kick-start the data gathering process, the evaluation will begin by conducting in-depth meetings with SHOPS staff. In addition to SHOPS, the evaluation team will consult with the USAID Health team and implementing partners with significant ongoing family planning programming. Government Officials: Ministry of Health (3) Higher Population Council (1) USAID and IPs: USAID Health DO (1) JCAP, HSSII (2) 60 Professional Associations: Jordan Pharmacist Association (1) SHOPS Personnel: Chief of Party, former and current (2) Deputy Chief of Party (1) Component Leads (5) M&E Manager (1) Banyan Global (1) SHOPS Grantees: ICCS (3 – HQ and field) UNRWA (1) JAFPP (5 – HQ and field) AJA (1) HLC (1) FHI (1) CCA (1) GUVS (1) Private Hospitals: All 4 private hospitals engaged with SHOPS (4) Focus Group Discussions (Q3, Q4, Q5): Group discussions will be conducted to explore issues in￾depth, to triangulate data, and to solicit the input of clients, pharmacists, and community health workers. Focus group discussions will be conducted at SDPs as available.  Community Health Workers (CHWs): 3 FGDs consisting of 6-8 CHWs each in North, and Center (2), drawn from CCA and GUVS CHWs.  Pharmacists: 1 focus group following pharmacist survey in Amman. Focus groups will target the six pharmacies participating in the FP coupon pilot intervention, if feasible depending on pharmacist availability.  Clients: 4 FGDs (North, South, Center based on client distribution) with 30-35 clients. Clinic Observation (Q1, Q6, Q7): An observation checklist will be used during visits to field clinics while interviewing staff at 12 clinics (representing 15 percent of the SHOPS supported Service Delivery Points) with priority given to those receiving the most support; geographic distribution; overlap with focus groups; and what is possible within the time frame of the evaluation. Observation aspects will include Clinic Management Information System (CMIS), counseling, coupon pilot, renovations. ICCS (2 clinics – Zarqa, Jarash) UNRWA (4 clinics – Balqa, Amman, Aqaba, Jarash) JAFPP (4 clinics – Amman, Aqaba, Irbid, Mafraq) HLC (1 clinic - Amman) IFH (1 clinic – Amman/Balqa) Survey (Q3, Q6): Surveys will be conducted with clients by phone (if possible) and with pharmacists and private providers in-person. Client contact information may be difficult to obtain as clients are assured confidentiality by providers. Clients will be surveyed by phone at random. Sampling is as follows:  150 Pharmacists (50 percent)  15 in the North (at random)  104 in Center (at random) 61  21 in South (at random)  150 Network Doctors (50 percent)  17 in the North (at random)  86 in Amman (at random)  28 in other central governorates (at random)  17 in the South (at random)  385 Clients (95 percent confidence level based on population of 259,000)  At random (for example, every 25th client in database, if phone number is working) All people-level questions will be gender disaggregated. The team will also identify the questions that will require examination of gender specific or gender differential effects. Table 1: Data Collection Methods Evaluation Questions Data Collection Methods KIIs Focus Groups Clinic/SDP Checklist Desk Review Survey 1. To what extent has SHOPS assistance strengthened JAFPP’s management, financial and governance system to operate sustainably? X X X 2. How effective were the various interventions/mechanisms and approaches in achieving the project’s objectives of increasing demand for, access to and quality of family planning services with targeted clients and providers? X X X 3. How did SHOPS’ internal operations and organization (SHOPS strategic planning, project management, financial management, communications, grant-making process, grant management, M&E systems and indicators, staffing structure) assist or hinder achieving project targets and objectives? X X X 4. To what extent were project partners (JAFPP, IFH, ICCS, UNRWA, private providers, etc.) effective in increasing access to, demand for and/or the quality of family planning products and services as a result of SHOPS support? X X X X 5. To what extent were the steps taken to address gender differentials and/or gaps? X X X X 6. To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation? X X X X 62 7. What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue, and which are unlikely to continue and why? X X X 8. Which of SHOPS’ partners (such as private doctors working through the voucher system) are most likely to continue practices that contribute to sustaining outcomes of increased a) access to; b) utilization; and c) quality of family planning services? X X X 9. To what extent were the theory of change (assumptions, pre-conditions) and project design used for the development approach appropriate to achieve USAID’s intended results? X X 10 What recommendations/suggestions does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? X X X X X Sampling Plan: The sampling of pharmacists, general practitioners (GPs) and clients will be based on geographic location and involvement in specific interventions. To the extent possible, respondent selection will be random. However, it is anticipated that client sampling will depend on accuracy of contact information, while pharmacist and GP sampling will be depend on availability/willingness of respondents to complete the survey. Sampling is as follows:  150 Pharmacists (50 percent, random sample)  385 Clients (95 percent confidence level, based on population of 259,000)  150 Network Doctors (NWD) (50 percent, random sample) The selection of clinics/SDPs will be based on the level of support they received from SHOPS, geographic location, and involvement in specific interventions under review with this evaluation. The evaluation will cover the program interventions in three regions of the North, Central, and South of Jordan. All SHOPS partners will be interviewed as key informants at their respective HQs. DATA ANALYSIS METHODS Data analysis will be structured by the evaluation questions, and will apply the following data analysis methods: Table 2: Data Analysis Methods Evaluation Questions 63 Comparison Analysis Q1, Q2, Q3, Q4, Q5, Q9 Frequency Response/ Trend Pattern Analysis Q2, Q3, Q4, Q5, Q6, Q7, Q8, Q10 Triangulation Q1, Q2, Q3, Q4, Q5, Q6, Q8, Q10 Sustainability Self Assessment Quantification and Analysis Q1, Q6, Q7, Q8, Q10 As the issue of sustainability of the partner institutions is of particular importance to this evaluation, questions that explore sustainability have been integrated into all surveys and interviews. Conslusions regarding sustainability will result from triangulation of a variety of data sources including observation, self-reporting by partners, partner and SHOPS reports on partner sustainability, and JAFPP financial data on revenues and expenses. 64 ANNEXES: I. Getting to Answers (G2A) II. Workplan III. Instruments IV. List of Key Informant Interviews and Focus Groups V. Evaluation Team: Roles & Responsibilities VI. List of Special Reports Consulted 65 ANNEX I: GETTING TO ANSWERS Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods JAFPP 1. To what extent has SHOPS assistance strengthened JAFPP’s management, financial and governance system to operate sustainably? Descriptive Define SHOPS support and recommendations Evidence of sustainability models adopted, new management and governance practices implemented JAFPP Key staff: CEO, Technical Manager, Administrative/ operational Manager HR Manager, Finance manager, Service Delivery officer KIIs HQ staff, SHOPS relevant staff Triangulation of data Effectiveness 2. How effective were the various interventions/mechanisms and approaches in achieving the project’s objectives of increasing demand for, access to and quality of family planning services with targeted clients and providers?  Were results tracked by intervention?  Were expenditures tracked by intervention? Numerical/statistical analysis and description of which interventions intended to achieve which objectives and the results: Interventions:  Clinic renovation  Org development  CHW training  QA training  GP referral vouchers  Social marketing/BCC  Capacity building QR, Annual reports, Strategic plan Document Review 1. Increase in # of clients, # of visits, #service delivery points, #FP products, QA measures, and client quality perception over time 2. Comparison of outcomes for various interventions M&E Databases Special Reports Quantitative analysis SHOPS staff: program/component, field, management KIIs, FGDs GOJ: MOH, JFDA, JPA Contacts supplied by SHOPS 66 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods  Pharmacist coupons  Academic detailing  HH outreach  New/renovated clinic operations Objectives:  Access to quality service  Increased demand  JAFPP/NGO capacity Evidence: # of clients, # of visits, #service delivery points, #FP products, QA measures, client quality perception Grantees: JAFPP, ICCS, UNRWA, AJA, HLC, FHI, CCA, GUVS KIIs All 8 HQ offices, purposive sample of 15% of the 80 SDPs – geographic distribution 3. Triangulation of data Databases on # clients, #visits, # FP products distributed, grant M&E data Quantitative analysis Private providers (GP 300, pharmacists 300) In-depth interviews Random sample (50%) by geo area 200 CHW Survey, FGDs Random sample (10%) by geo area Clients 3. How did SHOPS’ internal operations and organization (SHOPS strategic planning, project management, financial management, communications, grant-making process, grant Descriptive Internal structures:  Alignment of SHOPS activities and participatory SHOPS staff KIIs, FGDs 1. Response frequency 2. Correlation of response and achievement 3. Triangulation of data AMEP, indicator drop/loss table Document Review 67 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods management, M&E systems and indicators, staffing structure) assist or hinder achieving project targets and objectives? planning with partners  SHOPS HR structure and ability/appropriate  USAID branding requirements for grantees  Ability of grantees to meet requirements  Usefulness of grantee monitoring data for results  SHOPS indicators appropriate to activities Objectives:  Access to quality service  Increased demand  JAFPP/NGO capacity Evidence: # of clients, # of visits, #service delivery points, #FP products, QA measures, client quality perception SHOPS and grantee databases Quantitative analysis Grantees KIIs All 8 HQ, 15% of clinics/SDPs 4. To what extent were project partners (JAFPP, IFH, ICCS, UNRWA, private providers, etc.) effective in increasing Descriptive analysis Baseline and current - #clients, SHOPS and grantee databases Quantitative analysis 1. Before/after intervention comparison #clients, # visits, new 68 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods access to, demand for and/or the quality of family planning products and services as a result of SHOPS support? # visits, new services/service delivery points, products distributed/sold Grantees KIIs All 8 HQ services/service # delivery points, # products distributed/sold 2. Partner to partner comparison 3. Private provider, CHW and client perception Private providers (GP 300, pharmacists 300) FGD, survey Random sample (50%) by geo area 200 CHW FGD, survey Random sample (10%) by geo area FP clients Survey 5% confidence interval, random survey sample 5. To what extent were the steps taken to address gender differentials and/or gaps? Descriptive  Define gender constraints  How were they identified  Define steps taken  Define objectives of steps and measurement Strategy docs, annual and QR reports, AMEP, achievement reports Document review Comparison of objectives and results; triangulation of data Project staff: component/program/fiel d staff KIIs, FGDs Partners KIIs, clinic visits All 8 grantee HQ, 15% grantee clinics 69 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods by geo area Sustainability 6. To what extent did SHOPS integrate sustainability of interventions and institutions into design and implementation? Descriptive Technical interventions:  Clinic renovation  Org development  HCW training  QA training  GP referral vouchers  Social marketing/BCC  Capacity building  Pharmacist coupons  Academic detailing  HH outreach  New/renovated clinic operations Describe sustainability intent, partner/intervention sustainability plans, actions Strategy docs, annual and QR reports, AMEP, achievement reports, grantee reports Document review Triangulation of evidence in documents and informants Project staff: management/component /program/grant KIIs Partners KIIs, clinic visits 100% grantee HQ offices, 15% field offices/SDPs 70 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods 7. What technical interventions (such as increasing demand through outreach activities, training, EBM, supportive supervision, quality improvement, HMIS system in place and data utilized for decision making) of the project can USAID expect to continue, and which are unlikely to continue and why? Descriptive Technical Interventions:  outreach activities  training  EBM  supportive supervision  quality improvement  HMIS system  New facilities Evidence of continuation beyond funding; expression of intent to continue; expression of demand Strategy docs, annual and QR reports, AMEP, achievement reports Document review SHOPS staff: management/component /program/ field KIIs Grantees KIIs, clinic visits 100% HQ, 15% field offices Pharmacists, GPs, CHWs Survey (pharmacists, GPs), FGDs (CHW) Random survey sample of 50% by geo area 8. Which of SHOPS’ partners (such as private doctors working through the voucher system) are most likely to continue practices that contribute to sustaining outcomes of increased a) access to; b) utilization; and c) quality of family planning services? Practices:  outreach activities  BCC  training  EBM  supportive supervision  quality improvement  HMIS system SHOPS staff: management/component /program/ field KIIs, clinic visits Triangulation of responses regarding intent and resources Grantees KIIs, clinic visits 100% HQ, 15% field offices 71 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods  GP voucher system  Pharmacist coupons Evidence:  Level of interest in continuation  availability of financial resources to continue Pharmacists, GPs, CHWs Survey, FGDs of CHW random survey sample of 50% by geo area, CHWs in FGDs (10%) Learning 9. To what extent were the theory of change (assumptions, pre￾conditions) and project design used for the development approach appropriate to achieve USAID’s intended results? Descriptive  Documented TOC; TOC as described by staff, partners and USAID  In absence of TOC narrate logic flow, assumptions and required pre-conditions  Describe intended results:  Access to quality service  Increased demand  JAFPP/NGO capacity increased Project description, AMEP Document Review Identification of link between preconditions and assumptions articulated or absent in the TOC and stakeholder comments on effectiveness SHOPS management, USAID Health DO members KIIs Grantees KIIs 100% of HQ 10. What recommendations/suggestions Descriptive SHOPS staff and management KIIs Logic and reasonableness, credibility and success of 72 Data Collection Evaluation Question Type of Answer/Evidence Source Method Sampling/ selection Data Analysis Methods does the evaluation propose for future programming to be more effective in achieving and sustaining USAID’s intended family planning outcomes? Intended results:  Access to quality service  Increased demand  JAFPP/NGO capacity GOJ: MOH, JDFA, JPA KIIs Contacts identified by SHOPS recommender, team experience and knowledge of context Grantees KIIs 100% HQ, 15% field offices/SDPs GPs, pharmacists, CHW Surveys, FGDs with CHWs (10%) 50% random survey sample  Sustainability Reports  Pre/post Capacity reports  Profit/loss statements SHOPS staff and management 73 ANNEX II: WORK PLAN SHOPS Performance Evaluation Work Plan DESCRIPTION DELIVERABLE RESPONSIBLE Dates (May- August 2015) Inception Phase Desk review of project documents  Desk review and gaps analysis  List of evaluation reference documents Team Leader Evaluation Team May 17 Work plan submitted to USAID for approval  Work plan/schedule MESP May 21 USAID approves Work Plan May 26 Design of the evaluation methodology and tools  Design report  Draft instruments in English Team leader Evaluation team May 21-26 Quality Control Checkpoint Design Document Review MESP May 24 Design report submitted to USAID  Final Design report MESP May 26 USAID approves evaluation design report May 28 Finalizing and translating data collection tools Training data collectors Piloting tools  Final tools in Arabic  Training delivered  Piloting completed Team Leader Evaluation team MESP May 28- June 3 74 DESCRIPTION DELIVERABLE RESPONSIBLE Dates (May- August 2015) Check in with USAID Mindset (MESP Sub-contractor) Implementation Phase Quality Control Checkpoint Weekly Update MESP Weekly Data Collection through Key Informant Interviews, Focus Groups, and Surveys  Interview notes and summary report  Survey data (statistical analysis tables and graphs)  Focus group reports  Analysis report  (soft & hard copies for all deliverables) Evaluation Team June 1- 18 Analysis, De-briefing and Reporting Phase Analysis of data collected and drafting of reports  FCR Table Team Leader & Evaluation team June 20-27 Quality Control Checkpoint FCR Review MESP June 27 Refine FCR, draft report, draft PPT  Draft PPT presentation  Draft Report Team Leader & Evaluation team June 25-30 Quality Control Checkpoint PPT Review MESP June 25 De-briefing of findings to USAID  PPT presentation Team Leader, Evaluation team and MESP June 29 Presentation for project Stakeholders  PPT presentation Team Leader, Evaluation team and MESP June 30 75 DESCRIPTION DELIVERABLE RESPONSIBLE Dates (May- August 2015) Quality Control Checkpoint  Draft Report Review  Review Report Against Evaluation Checklist MESP July 1 Draft report incorporating USAID comments and stakeholders feedback from de-briefing and validation workshops  Draft evaluation report to USAID MESP, Team Leader and Evaluation team July 7 USAID comments on draft evaluation report  Draft report with comments USAID July 30 Response to USAID comments and update report  Updated report and response to comment table Team Leader, Evaluation team and MESP August 9 Quality Control Checkpoint  Response to USAID comments table review MESP August 9 Final evaluation report incorporating USAID comments  Final evaluation report MESP, Team Leader and Evaluation team August 16 76 ANNEX III: DISCUSSION GUIDES KII Guide for IPs – JCAP/HSSII (DRAFT) Name: Date: Title: Gender: 1. Which interventions or activities have been the most effective in increasing family planning demand? 2. Which interventions or activities have been the most effective in increasing access to FP services? 3. Which interventions or activities are the most likely to be sustained without donor funding? 4. In your opinion, to what extent has SHOPS assistance strengthened JAFPP’s management, finaincial and governance systems to operate sustainably? 5. What could have improved SHOPS effectiveness? 6. In your opinion, would involving men more have resulted in increased success? If yes, how? 7. What would you recommend for future investments and programming for family planning in the private sector to increase utilization and continuation of family planning? 77 KII Guide for MOH (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: 1. What do you think is the role of MOH within SHOPS? How did you collaborate or coordinate with SHOPS? 2. Which interventions/activities were you involved in? Why were they effective/not effective? Involved? Y/N Very Effective Somewha t Effective Not at Effective Sustainabl e Partially sustainable Not sustainable EBM Training PPP Roundtable Coalition Contraceptive Mix and Expansion Pilot (cyclofem, implanon) IUD Campaign DMPA Training IEC Materials Other - cite 3. Which interventions or activities have been the most effective in increasing family planning demand? 4. Which interventions or activities have been the most effective in increasing access to FP services? 5. Which interventions or activities are the most likely to be sustained without donor funding? 6. What could have improved SHOPS effectiveness? 7. What would you recommend for future investments and programming for family planning in the private sector to increase utilization and continuation of family planning? 78 KII Guide for HPC (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: 1. What do you think is the role of HPC within SHOPS? How did you collaborate or coordinate with SHOPS? 2. Which interventions/activities were you involved in? Why were they effective/not effective? Involved? Y/N Very Effective Somewha t Effective Not at Effective Sustainabl e Partially sustainabl e Not sustainabl e PPP Roundtable Coalition IUD Campaign 2011 FP/RH Symposium Contraceptive Insurance IEC materials 3. Which interventions or activities have been the most effective in increasing family planning demand? 4. Which interventions or activities have been the most effective in increasing access to FP services? 5. Which interventions or activities are the most likely to be sustained without donor funding? 6. What could have improved SHOPS effectiveness? 7. What would you recommend for future investments and programming for family planning in the private sector to increase utilization and continuation of family planning? 79 KII Guide for JPA (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: 1. What do you think is the role of JPA within SHOPS? How did you collaborate or coordinate with SHOPS? 2. Which interventions/activities were you involved in? Why were they effective/not effective? Involved? Y/N Very Effective Somewhat Effective Not at Effective Sustainabl e Partially sustainabl e Not sustainabl e Coupon Pilot Initiative EBM 3. Which interventions or activities have been the most effective in increasing family planning demand? 4. Which interventions or activities have been the most effective in increasing access to FP services? 5. Which interventions or activities are the most likely to be sustained without donor funding? 6. What could have improved SHOPS effectiveness? 7. What would you recommend for future investments and programming for family planning in the private sector to increase utilization and continuation of family planning? 80 KII Guide for Grantees (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: 1. What do you think is the role of your organization within SHOPS? How did you collaborate or coordinate with SHOPS? 2. Which interventions/activities were you involved in? Why were they effective/not effective? Involve d? Y/N Very Effectiv e Somewha t Effective Not at Effectiv e Sustainabl e Partially sustainable Not sustainable Coupon Pilot Initiative EBM Clinic renovations Coalition IUD Campaign Contraceptiv e Insurance IEC materials Training Household Outreach 3. Which interventions or activities have been the most effective in increasing family planning demand? 4. Which FP methods that you offer are most in demand? Methods Rate (1,2,3) COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms IUDs (intra-uterine device) 81 DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 5. What do you think are the most significant reasons that prevent women from increasing the use of FP methods and products? (Choose top 3: 1-most significant reason; 2-second most significant reason; 3- next most significant) Reason Given Rate (1, 2, 3) Side effects Fear of side effects Cost No access to pharmacy/provider No access to family planning services Want more children Attitude of husband Other? 6. Which interventions or activities have been the most effective in increasing access to FP services? 7. Which SHOPS interventions or activities are the most likely to be sustained without donor funding? 8. What could have improved SHOPS effectiveness? 9. Were men involved in interventions? Would involving men more have resulted in increased success? If yes, how? 10. What would you recommend for future investments and programming for family planning in the private sector to increase utilization and continuation of family planning? 82 KII Guide for JAFPP (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: 1. What do you think is the role of JAFPP within SHOPS? How did you collaborate or coordinate with SHOPS? 2. What do you think is the role of JAFPP with respect to family planning in Jordan? 3. Which interventions/activities were you involved in? Why were they effective/not effective? Involved? Y/N Very Effective Somewh at Effective Not Effectiv e Sustainable Partially sustainable Not sustainabl e JAFPP Specific Interventions Equipment purchase Renovation of clinics Clinical Training Capacity Building of BOD, executive staff Marketing for newly ren.clinics Market study and price increase Supportive supervision QA Financial Sustainability support (new services feasibility, investment plan, etc) HR management Support SHOPS Overall Interventions Coupon Pilot Initiative EBM Clinic renovations Coalition IUD Campaign 83 Contraceptive Insurance IEC materials Training Household Outreach 4. Which interventions or activities have been the most effective in increasing family planning demand for JAFPP? 5. Which interventions or activities have been the most effective in increasing access to JAFPP FP services? 6. What could have improved SHOPS effectiveness in these interventions? 7. Were men involved in interventions? Would involving men more have resulted in increased success? If yes, how? 8. Which interventions or activities is JAFPP most likely to sustain without donor funding? 9. How does JAFPP plan to sustain itself and its services after SHOPS program closeout? 10. With regard to financial sustainability, what could SHOPS have done differently to improve financial sustainability of JAFPP? What could JAFPP have done differently to improve financial sustainability? 11. What would you recommend for future investments and programming for family planning in the private sector to increase utilization and continuation of family planning? 84 KII Guide for Grantee/Private Hospital (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: 1. When did the intervention begin? 2. Who was trained in PPFP counseling? 3. Which FP methods were offered/Revised during counseling? COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 4. Which methods were provided prior to leaving the hospital? COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 5. What was the impact of the program on post-partum family planning uptake? 6. What, if any of the program, are you planning to sustain with or without a donor? 85 CLINIC CHECKLIST (DRAFT) Date: Location: Name of Clinic: 1. Which interventions did the clinic participate in? (Check all that apply) Intervention Yes No Clinic renovation & medical equipment Organizational Development CHW Training QA Training GP Referral vouchers Social Marketing/BCC Pharmacist Coupon Academic Detailing HH Outreach New Renovated clinic Operations 2. In the opinion of the clinic staff, did this increase demand for FP services? Yes/No If yes, which interventions were most effective in increasing demand for FP services? (Choose top 3: 1- most effective; 2 – second most effective; 3 – third most effective) Intervention Rate (1,2,3) Clinic renovation & medical equipment Organizational Development CHW Training QA Training GP Referral vouchers Social Marketing/BCC Pharmacist Coupon Academic Detailing HH Outreach New Renovated clinic Operations 3. In the opinion of the staff, did this increase access to FP services? Yes/No 86 If yes, which interventions were most effective in increasing access for FP services? Intervention Rate (1,2,3) Clinic renovation & medical equipment Organizational Development CHW Training QA Training GP Referral vouchers Social Marketing/BCC Pharmacist Coupon Academic Detailing HH Outreach New Renovated clinic Operations Did the clinic distribute contraceptive coupons? Yes/No Did the clinic track coupon use? Yes/No Does the clinic have a pharmacy directory for clients for coupon redemption? Yes/No Are the SHOPS danglers and other materials displayed? Yes/No Does the clinic conduct client satisfaction surveys? Yes/No Is the CMIS system in place? Yes/No If yes, is it in use? Yes/No What does the clinic track with respect to family planning? Open Ended 87 Client Survey Questionnaire (DRAFT) Name: Date: Title: Gender: Age: # of children: 1. Are you aware of the Ta’ziz program? Yes/No (if no skip to Q4) 2. Did you participate in Ta’ziz program? Yes/No (if no skip to Q5) 3. Which Ta’ziz activities were you involved in/aware of? (check all that apply) Yes No IUD and COC National Campaign Health Fair Awareness Sessions IEC Materials Contraceptive Coupons Care line 4. Did any of the above influence your family planning choice for contraception? Yes/No 5. If yes, which ones? Yes IUD and COC National Campaign National IUD campaign Health Fair Awareness Sessions IEC Materials Contraceptive Coupons Care line 6. Do you use FP? Yes/No If Yes, which method? (Check all that apply – current or past) Methods Current Past COCs (combined oral contraceptives) 88 POPs (progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 7. Do you currently use a modern method of family planning ? Yes/No If No, go to Q9 8. If Yes, which method? (Check all that apply – current or past) Methods Current Past COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 9. If no, Cite reasons why? Check all that apply (rank in order of importance- 1 most important; 3-least important) Reason Given Rate (1, 2, 3) Side effects Fear of side effects Cost No access to pharmacy/provider No access to family planning services Want more children Attitude of husband Other (Cite) 10. How did you learn about the FP methods? (check all that apply) Source 89 Pharmacist Doctor Clinic/ Service provider Health Fair Care line Brochures /IEC material Community Health Worker Other - cite 11. Which is your preferred point of contact for your FP services? Point JAFPP UNRWA Clinic MOH clinic Private doctor Royal Medical Service NGOs Pharmacists Community health worker Other – cite 12. Has the information you received increased the probability of continuing use of FP? Yes/No 13. Which information or contact point had the most influence on your choice for FP? Rank them in order of importance with 1 being most important and 3 being least important Source Rank 1,2,3 Pharmacist Doctor Clinic/Service provider Health Fair Careline Brochures/IEC material Community Health Worker 90 Other - cite 14. Do you face obstacles to prolonged use of FP? IF yes cite them Reason Given Rate (1, 2, 3) Side effects Fear of side effects Cost No access to pharmacy/provider No access to family planning services Want more children Attitude of husband Other (Cite) 15. Do you have any information on JAFPP services? (Yes/No) If No go to 17 16. Do you use JAFPP services? (Yes/No) If yes go to 17 17. If No why Not? (check all that apply) Reason Not available in geographic area Quality of care Diversity of services Cost Other (Cite) 18. What would make you increase your use of Services/methods? Open ended Researchers Notes: 91 Pharmacists Survey Questionnaire (DRAFT) Name: Date: Title: Gender: Duration of Involvement with SHOPS: Name of Pharmacy: 1. Are you aware of SHOPS? Yes/No 2. When did you become involved in SHOPS? (Check appropriate box) 1 year or less 2-3 years 1-2 years Above 4 years 3. Which interventions are you aware of? Yes No Coupon Pilot Initiative Training Academic Detailing EBM IUD and COC national campaign 4. Which interventions were you involve in? Yes No Coupon Pilot Initiative Training Academic Detailing EBM IUD and COC national campaign 5. Which FP methods and/or products do you offer? Methods Yes No 92 COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms Hormonal contraceptive IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 6. Which methods or products do you sell the most of? (Choose top 3: 1-best selling; 2- second best selling; 3- next best selling) Methods Rate (1,2,3) COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms Hormonal contraceptive IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) Has your product offering expanded due to the SHOPS intervention? Yes/No 7. If Yes-what additional products are you offering due to SHOPS Methods/Products Check all that apply COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 93 8. Have FP product sales increased since your involvement with SHOPS? Yes/No If yes, of what and by how much? Methods/Products 0-5% 6-10% 10-20% 21- 35% Above 35% COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectables) Vaginal Ring (Nuva Ring) Other (cite) 9. Do you counsel and/or refer women to health clinics/doctors for family planning services? Yes No Counseling customers on FP Referring customers on FP 10. Which of the interventions (if any) have been most effective in increasing demand for FP? (Very Effective -3; Somewhat effective -2; Not effective – 1) Methods Rate (1,2,3) Coupon Pilot Initiative Training Academic Detailing EBM Seminars 11. Which interventions have been the most effective in increasing access to FP methods and products? (Very Effective -3; Somewhat effective -2; Not effective – 1) Methods Rate (1,2,3) Coupon Pilot Initiative Training 94 Academic Detailing EBM Seminars 12. What do you think are the most significant reasons that prevent women from increasing the use of FP methods and products? (Choose top 3: 1-best selling; 2-second best selling; 3- next best selling) Reason Given Rate (1, 2, 3) Side effects Fear of side effects Cost No access to pharmacy/provider No access to family planning services Want more children Attitude of husband Other? Researchers notes: 95 NWD Survey Questionnaire (DRAFT) Name: Date: Title: Gender: Name of Clinic: Age: 1. Are you aware of SHOPS project? (Yes/ No) If no, go to 5 2. How long have you been involved with SHOPS? (Check appropriate box) 1 year or less 2-3 years 1-2 years 3-4 years Above 4 years 3. Which interventions were you aware of? (check all that apply) Activity/Intervention Yes No Coupon Pilot Initiative Training Academic Detailing EBM TV/Radio Talk Show Advocates 4. Which interventions were you involved in? (check all that apply) Activity/Intervention Yes No Coupon Pilot Initiative Training Academic Detailing EBM TV/Radio Talk Show Advocates 5. Which FP methods and/or products do you prescribe? (Check all that apply) 96 Methods Yes No COCs (combined oral contraceptives) POPs (progesterone only pills) Condoms Hormonal contraceptive IUDs (intra-uterine device) Hormonal injection) Cervical Capsules) Other (cite) 6. Which SHOPS interventions do you think were the most effective in increasing demand for FP methods? Activity/Intervention Very Effective Somewhat Effective Not Effective Coupon Pilot Initiative Training Academic Detailing EBM TV/Radio Talk Show Advocates 7. Which SHOPS interventions do you think were most effective in increasing access to FP methods? Activity/Intervention Very Effective Somewhat Effective Not Effective Coupon Pilot Initiative Training Academic Detailing EBM TV/Radio Talk Show Advocates 8. Which of the SHOPS activities/interventions are you most likely to continue without support from SHOPS? 97 Activity/Intervention Very Likely Somewhat Likely Not Likely Coupon Pilot Initiative Training Academic Detailing EBM TV/Radio Talk Show Advocates 9. What do you think are the most significant reasons that prevent women from increasing the use of FP methods and products? (Choose top 3: 1 – most significant reason; 2 – second most significant reason; 3 next most significant reason) Reason Given Rate (1, 2, 3) Side effects Fear of side effects Cost No access to pharmacy/provider No access to family planning services Want more children Attitude of husband Other? Notes : 98 Community Health Worker (CHW) FGD Guide (DRAFT) Date: FDG Leader: Note taker: Name of Participant Male Female Age Group Clinic Affiliation Geo. Area of HH visits Org. Affiliation Duration of involvement with SHOPS 1. Which interventions/ Activities were you involved in ? Were they effective? Why or Why Not? Are they sustainable? Why or Why not? Interventions and Activities Involved Y/N Very effective Somewhat effective Not effective Sustainable Partially sustainability Not sustainable IUD campaign Household visit IEC materials DMPA training Coupon pilot initiative 2. Which intervention or activities have been the most effective in increasing family planning demand? 3. Which FP methods do you provide counseling for? Method Y N COCs(combined oral Contraceptive ) Pops(progesterone only pills) Condoms IUDs (intra-uterine device) DMPA (injectable)vaginal ring (Nuva Ring ) Other (cite ) 4. (In your opinion) Did your household visits increase FP access? If yes, how so? 99 5. (In your opinion) Did your household visits increase FP uptake and demand? If yes, how so? 6. What are the challenges you faced in encouraging women to access FP services and use FP methods? 7. What do you think are the most significant reasons that prevent women from increasing the use of FP methods? Reason Given Rate (1, 2, 3) Side effects Fear of side effects Cost No access to pharmacy/provider No access to family planning services Want more children Attitude of husband Other? 8. What could have been done to increase the impact of your work? 9. What would you recommend for future investments and programing for FP Particularly for HH visits intervention? 100 ANNEX IV: LIST OF KIIS AND FGDS Key Informant Interviews Government Officials: Ministry of Health (3 KIIs) Higher Population Council (1 KII) USAID and IPs: USAID Health DO (1 KII) JCAP, HSSII (2 KIIs) Professional Associations: Jordan Pharmacist Association (1 KII) SHOPS Personnel: Chief of Party, former and current (2 KIIs) Deputy Chief of Party (1 KII) Component Leads (5 KIIs) M&E Manager (1 KII) Banyan Global (1 KII) SHOPS Grantees: ICCS (3 KIIs) UNRWA (1 KII) JAFPP (5 KIIs) AJA (1 KII) HLC (1 KII) FHI (1 KII) CCA (1 KII) GUVS (1 KII) Private Hospitals: All 4 private hospitals engaged with SHOPS (4 KIIs) Focus Group Discussions: Eight focus groups will be conducted with the following groups:  Community Health Workers (CHWs): 3 FGDs consisting of 6-8 CHWs each in North, and Center (2), drawn from CCA and GUVS CHWs.  Pharmacists: 1 focus groups following pharmacist survey, targeting up to 10 pharmacists, if feasible depending on pharmacist availability.  Clients: 4 FGDs (North, South, Center based on client distribution) targeting up to 45 clients. Clinic/SDP Checklist: The evaluation team will develop a clinic checklist to assess use and impact of interventions such as CMIS, counseling, coupon pilot, renovations. The team will visit 12 clinics as follows, representing 15% of the Service Delivery Points (or 25% of project document confirmed-clinics – the difference between SDPs and clinics is to be determined in the course of the evaluation) supported by SHOPS: ICCS (2 clinics – Zarqa, Jarash) UNRWA (4 clinics – Balqa, Amman, Aqaba, Jarash) JAFPP (4 clinics – Amman, Aqaba, Irbid, Mafraq) HLC (1 clinic - Amman) IFH (1 clinic – Amman/Balqa) Survey: GPs, pharmacists, and clients will be surveyed by phone and/or in person. The sampling of pharmacists, GPs and clients will be based on geographic location and involvement in specific interventions. To the extent possible, respondent selection will be random. However, it is anticipated that client sampling will depend on accuracy of contact information, while pharmacist and GP sampling will be depend on availability/willingness of respondents to complete the survey. Sampling is as follows:  150 Pharmacists (50% of pharmacists)  385 Clients (5% confidence interval, based on population of 259,000) 101  150 Network Doctors (50% of NWDs) 102 ANNEX V: EVALUATION TEAM: ROLES & RESPONSIBILITIES Pamela Putney, Team Leader/Evaluation Specialist: Primary point of contact for assignment with responsibility for assigning team duties, managing activities, resources, and team member performance to meet objectives; leadership role in analysis, final reporting and presentation. Leads meetings with USAID; leads in creating design methodology and instruments; conducts literature review; participates in interviewing and data collection; assigns team activities and facilitates smooth team operations; ensures that findings, conclusions, and recommendations answer evaluation questions and meet USAID purposes; produces/finalizes evaluation tools and final report; ensures final reporting meets USAID requirements. Nedjma Koval-Saifi, Senior Local Evaluation Expert: Leads in production of Evaluation Design Report; ensures evaluation instruments appropriately address the evaluation questions; ensures data analysis follows rigorous process of linking findings, conclusions and recommendations; leads in structuring final presentation and report to address evaluation questions; ensures final report meets USAID evaluation requirements. Wisam Qarqash and Huda Murad, Family Planning/Health Sector Technical Specialists: Focuses on his/her particular areas of specialization (behavior change communication, grants, NGO capacity building, service quality, cost recovery) in conducting desk review, design methodology and data collection instruments; co-leads training and pilot testing for data collection; participates in data collection, analysis and interpretation; produces report sections as assigned by Team Lead. Rand Milhem Quality Manager: The Quality Manager will develop the logistical plan, ensure data collection and data entry protocols are followed; ensure integrity of focus group transcripts and translation; and participate in data collection. 103 ANNEX VI: SPECIAL REPORTS SHOPS – Evaluation and Assessments Title Month/Year Type Description Family Planning Focus Group Discussions among Married Women in Jordan June/2011 Formative assessment Presents qualitative findings related to: Family planning perceptions, motivation to switch from traditional to modern FP methods, FP decision making and general information about FP JAFPP financial management assessment findings report January/2011 Formative assessment Findings from a thorough assessment of JAFPP’s financial systems, procedures and capacity (via in-country assessment visits and desk research), resulting in a report of findings. These findings were used to update JAFPP’s Financial Management Procedures Manual based on assessment results. JAFPP Financial sustainability Assessment Findings April/2011 Formative assessment Recommendations for improving sustainability based on findings from quantitative analysis of the Association’s financial health, assessment of JAFPP’s position within the competitive landscape of FP service provision in Jordan and qualitative findings from interviews with clients and staff. JAFPP HRM Assessment December/2010 Formative assessment Assessment of the Association’s human resource management policies and procedures. JAFPP Quality of Care Needs Assessment January/2011 Formative assessment Assessment of the Association’s physical infrastructure, equipment and supplies, human resources, training, clinical guidelines, clinic processes, clinical supervision, clinic management practices, and client perception of the quality of services. JAFPP Service Quality and Brand Perception Final Report.pdf June/2011 Formative assessment Assessment of current and potential JAFPP clients’ perceptions of service quality though FGDs. Jordan FP Users and Market Segmentation Analysis February/2011 Formative assessment The overall objective is to inform the development of marketing and strategic plans to increase demand of FP products and services in Jordan. The researcher used DHS data to determine the size and distribution of the FP market in Jordan, study profiles of users and non-users of FP methods and review factors - demographic, socioeconomic, cultural, and other - that affect FP method use, review existing research on providers’ profiles and supply side factors affecting FP in Jordan, and divide the Jordanian FP market into segments and estimate the size of potential demand by method and source. Jordan. 104 Title Month/Year Type Description Private Health Insurance Coverage of Contraception Report March/2011 Formative assessment This report presents a situational analysis of FP coverage in private insurance companies and presents a strategy towards the successful introduction of FP coverage. Trip Report_Cuellar_July 2012 JAFPP governance & management assessment July/2012 Evaluation Baseline assessment of the Association’s adherence to governance and management policies and procedures. Voucher redemption analysis February/2012 Formative An examination of the redemption rate of free FP service vouchers distributed through the outreach program OCP Campaign Phase I Tracking Survey Report October/2012 Evaluation Quantitative survey using non-probabilistic sampling through household interviews in Amman, Irbid, and Zarqa to gauge reach and effectiveness of the OCP campaign after the first wave. EBM Impact Evaluation Report (DMPA) October/2013 Evaluation Randomized controlled trial approach in order to evaluate the effectiveness of the evidence based medicine approach on physicians’ knowledge, attitudes and practices JAFPP Client Satisfaction Report_Jan 2014 January/2014 Formative assessment A survey of JAFPP clients’ satisfaction at a sample of clinics using two different methodologies: Face-to-face exit interviews and telephone interviews. JAFPP Institutional Perception Study December/2 013 Formative assessment A qualitative study utelizing in-depth interviews with key stakeholders of family planning and reproductive health in Jordan in order to assess the Association’s position nationally and to identify its strengths and weaknesses with relation to its role as a leader and advocate. OCP Campaign Endline Survey Report December/2013 Evaluation Quantitative survey using non-probabilistic sampling through household interviews in Amman, Irbid, and Zarqa to gauge reach and effectiveness of the OCP campaign after its final wave. Trip Report_Cuellar_June 2014_JAFPP management & governance June/2014 Evaluation Mid-line of the Association’s adherence to governance and management policies and procedures. Careline Pilot Evaluation April/2015 IUD Campaign Y4 FY14 Evaluation December/2014 Evaluation Quantitative survey using non-probabilistic sampling through household interviews in Amman, Irbid, and Zarqa to gauge reach and effectiveness of the 105 Title Month/Year Type Description IUD campaign. Ongoing evaluations: Evaluation Description Status Family planning couple counseling Study Longitudinal cohort study with a randomized control trial approach. Study is divided into two phases; each phase includes baseline screening, recruitment and interviewing, randomization, outreach intervention implementation, and endline interviewing. ☒ Study prep ☒ Data collection start: September 2013 ☒ Data collection end: Phase 1and Phase 2 surveys completed. ☒ Analysis: Ongoing ☒ Report writing initiated ☐ Local dissemination Careline – outreach program Randomized control trial in order to assess the effectiveness of telephone follow-up calls in increasing voucher redemption and method uptake and in decreasing discontinuation after uptake. ☒ Study prep ☒ Data collection start: 7 April 2013 ☒ Data collection end: 28 August 2013 ☒ Analysis: Completed ☒ Report writing: Completed ☐ Local dissemination FP insurance study Cost analysis using claims data Cohort study using telephone interviews to gage beneficiary uptake of modern FP method due to insurance coverage. Cohort of beneficiaries interviewed prior to their ☒ Study prep ☒ Data collection start: Baseline 28 September – 7 Oct 2013 106 Evaluation Description Status knowledge of the FP coverage and prior to the activation of the coverage, and towards the end of the coverage period. ☒ Data collection end: Beneficiary survey cancelled. Claims data collected ☒ Analysis ☐ Report writing ☐ Local dissemination Implanon method acceptability study Quantitative study through telephone interviews with women who received the Implanon at NWDs through the Outreach Program. The study ☒ Study prep ☒ Data collection start: 14 Sep 2014 ☒ Data collection end: 21 Sep 2014 ☒ Analysis: Ongoing ☒ Report writing: Report drafted. Sent for quality assurance review ☐ Local dissemination Postpartum counseling at private hospitals - pilot Randomized control trial using the same methodology used by HSS-II in Al Bashir Hospital to test if postpartum counseling affects modern method uptake and continuation. ☒ Study prep: Commenced during Q2 ☒ Data collection start: 14 Dec 2014 ☐ Data collection end: May 2015 ☐ Analysis ☐ Report writing: ☐ Local dissemination Assessment of the training program and the EMB Mystery clients will be sent to a sample of consenting physicians and pharmacists. The study will explore whether ☒ Study prep: Commenced during Q1 ☒ Data collection start: May 2015 107 Evaluation Description Status approach with physicians and pharmacists through mystery clients there is a dose effect with relation to the dose of trainings in association with the quality of counseling or information dissemination on a set number of modern FP methods. ☐ Data collection end: May 2015 ☐ Analysis ☐ Report writing: ☐ Local dissemination Endline assessment of JAFPP management & governance End-line of the Association’s adherence to governance and management policies and procedures. ☒ Study prep: May 2015 ☒ Data collection start: June 2015 ☐ Data collection end: June 2015 ☐ Analysis ☐ Report writing: ☐ Local dissemination 108 ANNEX III: PERFORMANCE MONITORING PLAN Indicators FY 2011 FY 2012 FY 2013 Project Objective: To expand the access, quality and utilization of family planning services in Jordan Impact indicators Couple years of protection (CYP) achieved through JAFPP and UNRWA FP services and private market contraceptive sales (1) √ √ √ Percentage of in-union women of reproductive age using, or whose partner is using, a modern method of contraception (2) √ √ √ Result 1. Strengthened Management and Governance Systems at JAFPP Outcome indicators % adherence to management authority procedures/policies (3) √ √ √ % adherence to HR procedures/policies (4) √ √ √ % employee satisfaction (5) √ √ √ % adherence to financial management procedures/policies (6) √ √ √ % of quarterly JAFPP budget appropriation reviews with satisfactory findings (7) √ √ √ % of JAFPP procurements done according to procedures and guidelines (8) √ √ √ % cost recovery √ √ √ Output indicators Presence of capacity building plan for governance, HR, financial, and organizational management √ √ Number and gender of people trained/coached in governance, management systems, and financial and grant management topics. √ √ Presence of approved bylaws, Delegation of Authorities Charter, and code of conduct √ √ Presence of a performance management and reporting system including a self-assessment tool √ √ Presence of a 3-year strategy-based business plan √ √ Presence of a proper organizational structure and manual of organizational roles and responsibilities √ √ % of quarterly MOU milestones met by JAFPP √ √ √ Number of quarterly technical and financial reports and plans completed by JAFPP on time √ √ √ Result 2: Increased Demand and Access to Quality JAFPP and UNWRA Services Outcome indicators 109 Indicators FY 2011 FY 2012 FY 2013 Change in number of returning women using JAFPP and UNRWA FP services (9a) √ √ √ Change in number of new women using JAFPP and UNRWA FP services (9b) √ √ √ % client satisfaction (10) √ √ √ % of clinics offering minimum package of modern contraceptive methods (11)/implant services √ √ √ Output indicators Presence of plan for clinic expansion and improvements √ √ Number and location of health facilities rehabilitated √ √ √ Number and location of new clinics built and opened √ √ √ Number of clinics with new equipment to match needs assessment (JAFPP and UNRWA) √ √ √ Number of people trained in FP/RH with USG funds √ √ √ Presence of marketing strategies for JAFPP, products, and services and brand positioning √ Number of detailing visits to doctors and pharmacists √ √ Number of doctors participating in private network that provide family planning services √ √ Completed assessment of project-assisted clinic facilities’ compliance with clinical standards √ Result 3: Improved Quality of Services Outcome indicators Number and gender of providers with improved knowledge and attitudes towards modern hormonal methods (12) √ √ % of quarterly clinic data updates received on time by JAFPP HQ (13) √ Number and type of referrals made to other sites (e.g. hospitals) using formal referral system (14) √ Number of new outreach clients using private sector network doctor family planning services to obtain modern FP methods (17) Number of new outreach clients using private sector network doctor family planning services to obtain modern FP methods (17) √ √ Number of FP counseling visits (JAFPP, UNRWA, and outreach) (18) √ √ % of target (poor, high risk) women that comply with referrals for FP counseling visits to network doctors (20) √ √ % of target (poor, high risk) women that act upon free FP vouchers (21) √ √ % of MWRA who agree that OC pills are safe (measuring attitudes for other contraceptive methods TBD) (22) √ √ 110 Indicators FY 2011 FY 2012 FY 2013 % of new clients in JAFPP clinics with implemented marketing activities (Aqaba clinic as of June 2011, others to follow) (23) √ % female physician vacancies at JAFPP (15) √ √ Output indicators Completed assessment of project-assisted clinic facilities’ compliance with clinical standards √ Number of people trained in FP/RH with USG funds √ √ √ Existence of a formal clinical referral system √ Result 4: Expanded FP Market and Product Choice Outcome indicators % annual growth in OC pill market (measuring market growth for other contraceptive methods TBD) (16) √ √ √ Amount of in-country public and private financial resources leveraged by USG programs for FP/RH (17) √ % market share for modern FP methods, by each of three methods (IUD, OC's, and injectables), held by private hospitals, clinics and pharmacies (25) √ √ % market share for modern FP methods, by each of three methods (IUD, OC's, and injectables), held by JAFPP (26) √ √ % market share for modern FP methods, by each of three methods (IUD, OC's, and injectables), held by UNRWA (27) √ √ Number of acceptors of modern contraceptive methods generated among target (poor, high risk) women (19) √ √ Output indicators Number of new methods successfully introduced through USG-supported programs √ √ √ Number of FP promotional campaigns supported by project √ √ √ Number and type of partnerships established with private sector pharmaceutical firms √ √ Number of FP counseling outreach visits among poor, high risk women √ √ Number of target (poor, high risk) women reached through outreach visits by CHW √ √ Number of target (poor, high risk) women referred by CHW for FP services √ √ Number of target (poor, high risk) women receiving vouchers for FP methods from CHW √ √ Presence of marketing strategies for clinics and JAFPP institutional and brand positioning √ Result 5: Increased Financial Sustainability 111 Indicators FY 2011 FY 2012 FY 2013 Outcome indicators Revenue to cost comparisons (18)/cost recovery √ √ √ % of revenue from non-client-fee sources (rent, grants, etc) (19) √ √ √ Output indicators Presence of sustainability plan √ √ Presence of pricing and financing strategy √ √ Presence service costing/pricing mechanism √ √ Number and gender of JAFPP staff trained in financial and grant management topics √ √ √ 112 ANNEX IV: CONTACT LIST Organization Position(s) Name(s) Contact Information Location Total Interviews Team Members IP (SHOPS staff) Previous COP (through February 2015) Vietnam 10 Eval Team Leader, Tech Expert/s COP (as of March 2015) Amman DCOP Communication and Information Dissemination officer Grant Manager Service Delivery Program Manager Monitoring, Evaluation and Research Officer Organizational Development Program Manager Clinical Renovation and Construction Advisor Private Health Market Coordinator J-CAP, HSSI JCAP COP Amman 3 Eval Team Leader Tech Expert/s JCAP DCOP HSSI COP Counterparts/Stakeholders 113 Organization Position(s) Name(s) Contact Information Location Total Interviews Team Members 1. MOH Director, Women and Child Health Directorate Amman 3 Eval Team Leader Tech Expert/s Technical Manager, Women and Child Health Directorate Supplies Manager, Women and Child Health Directorate 2. HPC General Secretary DCOP Amman 3 SHOPS grantee/Partners 1. JAFPP Executive director Amman 5 Eval Team Leader Senior Evaluation Expert Tech Expert/s Head of maintenance and transportation section Social marketing manager Financial Manager Medical Services Manager Field clinic head (sample): A list of clinics was provided separately by Social marketing director 2. Islamic Charity Center Society (ICCS) Director of Social and Health Department Amman 3 Senior Senior Finance Assistant Evaluation 114 Organization Position(s) Name(s) Contact Information Location Total Interviews Team Members Clinics Coordinator Expert Tech Expert/s 3. Circassian Charity Association (CCA) Amman 2 Senior Evaluation Expert Tech Expert/s 4. General Union of Voluntary Societies (GUVS) Amman 1 Senior Evaluation Expert Tech Expert/s 5. UNRWA Field Family Health Officer Amman 1 Senior Evaluation Expert Tech Expert/s 6. Aman Jordanian Association (AJA) Director Amman 2 Senior Evaluation Expert Tech Expert/s 7. Hussein Labor Clinics (HLC) General Manager Amman 2 Senior Evaluation Expert Tech Expert/s Coordinator for shops and HR of the HLC 8. Institute for Family Health Director Amman /Swieleh 1 Senior Evaluation Expert Tech Expert/s 115 Organization Position(s) Name(s) Contact Information Location Total Interviews Team Members Jordan Pharmacists Association Project coordinator Amman 1 Senior Evaluation Expert Tech Expert/s Private Hospital Initiative Al-Essra Hospital Director of Gynaecology Section Amman 2 Tech Expert/s SN/MW Al-Amal Hospital Public Relations PR and Coordinator QC&ICP Amman 3 Tech Expert/s PP unit head nurse Al-Hayat Hospital Laboratory Director Amman 1 Tech Expert/s The Speciality Hospital Nurse Amman 3 Tech Expert/s 116 ANNEX V: BIBLIOGRAPHY Bongaarts, John. The Causes of Stalling Fertility Transitions, The Higher Population Council (HPC), June, 2005 Department of Statistics (DOS) ‘Jordan Population and Family Health Survey 2012. Amman, Jordan. DOS, 2012 K4 Health ‘Iftaa Booklet – Family Health Training Manual’ https://www.k4health.org/sites/default/files/Wo3ad.pdf Khayame, Huda. Achieving Results Through Integrating Supply and Demand, June 16, 2015 Jordan Association for Family Planning and Protection (JAFPP), Action Plans Report for the Execution of the Business Plan of The Jordanian Association for Family Planning and Protection, March, 2015 Jordan Association for Family Planning and Protection (JAFPP), Client Satisfaction Report, Amman, Jan 2014 Jordan Association for Family Planning and Protection (JAFPP), Financial Management Assessment Report, Amman, Jan 2011 Jordan Association for Family Planning and Protection (JAFPP), Financial Sustainability Assessment, Amman, April 2011 Jordan Association for Family Planning and Protection (JAFPP), Governance and Management Assessment Trip Report, Amman, Jun 2014 Jordan Association for Family Planning and Protection (JAFPP), Governance and Management Baseline Assessment Trip Report, Amman, July, 2012 Jordan Association for Family Planning and Protection (JAFPP), HRM Assessment, Amman, Dec 2010 Jordan Association for Family Planning and Protection (JAFPP), Institutional Perception Study, Amman, Dec, 2013 Jordan Association for Family Planning and Protection (JAFPP), Needs Assessment PSP, 2008 Jordan Association for Family Planning and Protection (JAFPP), New Products and Service Feasibility Study Synthesis Report, Amman, Aug, 2012 Jordan Association for Family Planning and Protection (JAFPP), Quality of Care Needs Assessment, Amman, Jan 2011 Jordan Association for Family Planning and Protection (JAFPP), Service Quality and Brand Perception Final Report, Amman, June, 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Activity Monitoring and Evaluation Plan (AMEP), Amman, 2010-2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Activity Monitoring and Evaluation Plan (AMEP) Year 5’, 2015 Strengthening Health Outcomes through the Private Sector (SHOPS), Annual Report, Amman, 2010-2011 117 Strengthening Health Outcomes through the Private Sector (SHOPS), Annual Report, Amman, 2011-2012 Strengthening Health Outcomes through the Private Sector (SHOPS), Annual Report, Amman, 2012-2013 Strengthening Health Outcomes through the Private Sector (SHOPS), Annual Report, Amman, 2013-2014 Strengthening Health Outcomes through the Private Sector (SHOPS) Careline Pilot Evaluation, April, 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Dashboard, May, 2015 Strengthening Health Outcomes through the Private Sector (SHOPS) EBM Impact Evaluation Report (DMPA), Oct, 2013 Strengthening Health Outcomes through the Private Sector (SHOPS) Family Planning Focus Group Discussions among Married Women in Jordan, June, 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Final Assessment Report of JAFPP Business Plan Execution, 2015 Strengthening Health Outcomes through the Private Sector (SHOPS) IUD Campaign Evaluation Year 4, 2014 Strengthening Health Outcomes through the Private Sector (SHOPS) OCP Campaign Endline Survey Report, Dec, 2013 Strengthening Health Outcomes through the Private Sector (SHOPS) OCP Campaign Phase 1 Tracking Survey Report, October, 2012 Strengthening Health Outcomes through the Private Sector (SHOPS), Performance Monitoring Plan (PMP), Amman, 2010-2011 Strengthening Health Outcomes through the Private Sector (SHOPS) Private Health Insurance Coverage Contraception Report, March, 2011 Strengthening Health Outcomes through the Private Sector (SHOPS). STTA to the Strengthened Management and Governance Systems Results Area - Trip Report, Oct, 2012 Strengthening Health Outcomes through the Private Sector (SHOPS), Year 1 Work Plan, Amman, 2010- 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Year 2 Work Plan, Amman, 2010- 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Year 3 Work Plan, Amman, 2010- 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Year 4 Work Plan, Amman, 2010- 2011 Strengthening Health Outcomes through the Private Sector (SHOPS), Year 5 Work Plan, Amman, 2010- 2011 The Higher Population Council (HPC), National Reproductive Health/ Family Planning Strategy 2013-2017 United Nation Relief and Works Agency (UNRWA) UNRWA Health Report 2011-2014 United Nation Relief and Works Agency (UNRWA) UNRWA Health Report 2014 UNRWA, 2014 118 USAID ‘A Trend Analysis of Family Planning in Jordan. Informing Policy and Program Planning.’ Health Policy Initiative, USAID, 2009 USAID ‘USAID-SHOPS Agreement: Associate Cooperative Agreement No. 278-A-00-10-00434-00’ USAID, July 20, 2010. 119 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523