EVALUATION END-OF-PROJECT PERFORMANCE EVALUATION OF USAID/CAUCASUS SUSTAINING FAMILY PLANNING AND MATERNAL AND CHILD HEALTH (SUSTAIN) PROJECT IN GEORGIA December 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez, England & Associates END-OF-PROJECT PERFORMANCE EVALUATION OF USAID/CAUCASUS SUSTAINING FAMILY PLANNING AND MATERNAL AND CHILD HEALTH (SUSTAIN) PROJECT IN GEORGIA Final Report 12 December 2014 Prepared under RFTOP No. SOL-114-14-000011 Submitted to: USAID/Caucasus Submitted by: Jean Capps, BSN MPH (Team Leader) Dr. Boris Sergeyev (Evaluation Expert) Dr. Rusudan Chkhubianishvili (Local Expert) Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS EXECUTIVE SUMMARY ................................................................................................... i Evaluation Purpose ............................................................................................................................... i Project Background ............................................................................................................................. i Evaluation Design, Methods and Limitations ................................................................................ii Evaluation Questions with Findings ...............................................................................................iii 1.0 Evaluation Purpose & Evaluation Questions ....................................................... 1 1.1 Evaluation Purpose .................................................................................................................. 1 1.2 Evaluation Questions .............................................................................................................. 1 2.0 Project Background ................................................................................................ 2 3.0 Evaluation Methods & Limitations ....................................................................... 4 3.1 Evaluation Methodology ........................................................................................................ 4 3.1.1 Quantitative Research and Analysis......................................................................5 3.1.2 Qualitative Research and Analysis.........................................................................6 3.2 Evaluation Limitations.............................................................................................................7 4.0 Findings, Conclusions and Recommendations ..................................................... 8 4.1 Question 1: How flexible and effective was the project in adjusting to the radical changes in the political environment and did those changes affect the key outcomes? ............................................................................................................................................. 8 4.1.1 Findings ....................................................................................................................... 8 4.1.2 Conclusion ................................................................................................................. 9 4.1.3 Recommendations ................................................................................................... 9 4.2 Question 2: What were significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted from the project interventions? ........................................................................................................................ 9 4.2.1 Findings ....................................................................................................................... 9 4.2.2 Conclusions ............................................................................................................ 18 4.2.3 Recommendations ................................................................................................ 19 4.3 To what extent has the project contributed to private sector-led service delivery development (in particular, as it relates to the provision of evidence based Effective Perinatal Services (EPC) in the privately owned and operated health care facilities? .............................................................................................................................................. 20 4.3.1 Findings .................................................................................................................... 20 4.3.2 Conclusions ............................................................................................................ 23 4.3.3 Recommendations ................................................................................................ 24 4.4 Are the key stakeholders satisfied with the project (with the special emphasis on medical schools with regards to curriculum and education practices at medical schools)? ............................................................................................................................................. 25 4.4.1 Findings .................................................................................................................... 25 4.4.2 Conclusions ............................................................................................................ 25 4.4.3 Recommendations ................................................................................................ 26 4.5 Question 5: How sustainable are project activities? .................................................... 26 4.5.1 Findings .................................................................................................................... 26 4.5.3 Recommendations ................................................................................................ 28 5.0 Lessons Learned ....................................................................................................29 ANNEXES ANNEX 1: EVALUATION STATEMENT OF WORK ANNEX 2: EVALUATION MATRIX ANNEX 3: SUMMARY OF MINI-SURVEY’S RESULTS ANNEX 4: LIST OF INTERVIEWEES ANNEX 5: SUMMARY OF RESULTS OF FOCUS GROUP DISCUSSIONS AMONG WOMEN OF REPRODUCTIVE AGE (AGE 15 TO 49) ACRONYMS ACOG American Congress of Obstetricians and Gynecologists AMSTL Active Management of the Third Stage of Labor BCC Behavior Change Communication BF Breastfeeding CDCH Center for Disease and Public Health (check) CoP Chief of Party CS Caesarian Section EPC Effective Perinatal Care EPC Effective Perinatal Services ET Evaluation Team FGD Focus Group Discussion FP Family Planning FSU Former Soviet Union GoG Government of Georgia GRHS 2010 Georgia Reproductive Health Survey 2010 HMIS Health Management Information System HWG Health Women in Georgia project IMR Infant Mortality Rate IUD Intrauterine Device JSI John Snow, Inc. KI Key Informant KII Key Informant Interview LAM Lactational Amenorrhea Method MCH Maternal and Child Healthcare MDG Millennium Development Goal ME&A Mendez, England & Associates MMR Maternal Mortality Ratio MNCH Maternal, Newborn and Child Health MoLHSA Ministry of Labor, Health, and Social Affairs OB/GYN Obstetrics and Gynecology OCP Oral Contraceptive Pills OSCE Objective Structured Clinical Examination PAC Post-abortion Care PEPC Promoting Essential Perinatal Care PHC Primary Health Care PPH Postpartum Hemorrhage RH Reproductive Health SDM Standard Days Method SUSTAIN Sustaining Family Planning and Maternal and Child Health Services TL Tubal Ligation TSMU Tbilisi State Medical University UHC Universal Health Care UNFPA United Nations Fund for Population Assistance UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG US Government WHO World Health Organization WRA Women of Reproductive Age (15–49 years old) i EXECUTIVE SUMMARY EVALUATION PURPOSE This is a report on the end-of-project performance evaluation of the Sustaining Family Planning and Maternal and Child Health (SUSTAIN) project funded by the United States Agency for International Development (USAID) Mission in Georgia. The project is implemented by John Snow Incorporated (JSI) between October 2009 and September 2014. By the time of the submission of this report, the project had been extended to continue a limited number of activities for one additional year and is scheduled to end on September 30, 2015. The evaluation of SUSTAIN project took place during the period September-October 2014, and was conducted by a team assembled by Mendez, England & Associates (ME&A) that consisted of three specialists (international and local) with experience in the Eurasia region and health sector, in general, and Family Planning (FP) and Maternal, Newborn and Child Health (MNCH), in particular. The purpose of the evaluation was to examine the project’s effects on advancement of FP and MNCH services in target facilities throughout Georgia, as well as to provide an independent view of how the project has been implemented, what results have been achieved in the target regions, and the external and internal challenges and any impact they have had on the project. The evaluation also was to address the sustainability of any positive changes that might have occurred as a result of the project. An integral part of the evaluation mission was to answer a set of five pre-determined questions, which are detailed in the body of this report. The questions concern: 1. How flexible and effective was the project in adjusting to the radical changes in the political environment, and did those changes affect the key outcomes? 2. What were the significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted from the project interventions? 3. To what extent has the project contributed to private sector-led service delivery development [in particular, as it related to the provision of evidence-based Effective Perinatal Services (EFS)] in the privately owned and operated health care facilities? 4. Are the key stakeholders satisfied with the project? 5. How sustainable are project activities? The results of the evaluation will be used to summarize the results of SUSTAIN, provide basis for future programs in Georgia, and share information about experiences in FP/MNCH within USAID’s health community across the region and globally. PROJECT BACKGROUND The SUSTAIN project began in October 2009 and was originally scheduled to end in September 2014. In mid-20014, USAID awarded a one-year extension for a limited number of specific activities at the national level, primarily related to assisting the national government in their regionalization of perinatal services. The original purpose of SUSTAIN was to improve access and quality of women’s health services and information, especially family planning/reproductive health (FP/RH) and evidence-based, ii family-friendly MNCH. SUSTAIN was designed to be in line with the national health strategy at the time that stressed market-based, private sector service delivery amid minimum State regulation. Quality supervision was also part of the original program. Government responsibilities in 2009 were largely limited to providing a "safety net" of services for the lowest economic groups. SUSTAIN's original objectives were to: 1) launch new private sector-led service delivery, health insurance and product-specific social marketing models for FP/MNCH services; 2) build strong commercial sector distribution of contraceptives and public/private partnerships to finance Behavior Change Communication (BCC) campaigns; 3) catalyze the Georgian health insurance sector to “become the vanguards of FP/MCH services”; and 4) incorporate FP/RH modules and practicum into medical and nursing school pre-service curriculum. SUSTAIN was built upon a foundation of a package of evidence-based MNCH health services promoted by the World Health Organization (WHO) that had been introduced in the previous JSI-implemented Healthy Women in Georgia (HWG) project (2003-2009). Georgia had already made significant improvements in FP and MNCH services; however, as of 2010, reductions in maternal deaths1 were still not on track to reach national Millennium Development Goals (MDG) targets by 2015 and newborns represented the majority of infant deaths2 . The Maternal Mortality Study3 identified preventable causes of maternal deaths that could be addressed using existing evidence-based interventions if implemented at scale. With the change of government that took place after the 2012 elections, the Government of Georgia (GoG) assumed the responsibility for providing health services to the entire population through Universal Health Care State Program (UHCSP) to pay the (still private) health facilities on a per capita basis for essential health services. This resulted in changes in the project strategy from assisting the private sector to take leadership of FP/MNCH as was envisioned in the original project design, to working directly with the national government to set policies, protocols and regulations to monitor quality of health services in FP and MNCH throughout the country. SUSTAIN has been supporting the Ministry of Labor, Health, and Social Affairs (MoLHSA) to formulate policies and implementation plans for regionalization of perinatal services, which will improve MNCH service quality. This will include certification and accreditation of different levels of facilities and develop procedures for referrals for complicated cases. SUSTAIN’s recently-awarded one-year extension4 focuses on technical support to the MoLHSA to complete this process. EVALUATION DESIGN, METHODS AND LIMITATIONS The performance evaluation of SUSTAIN used a mixture of quantitative and qualitative methods, and was based on a quasi-experimental design. Quantitative methods included a mini- 1. Georgia Reproductive Health Survey (GRHS), 2010. 2 Ibid. 3 Berduli, N, et al Maternal Mortality Study Georgia, 2011, National Center for Disease Control and Public Health, Ministry of Labor Health and Social Affairs, SUSTAIN Project, and USAID, 2011. 4 Now scheduled to end in September 2015. iii survey of health workers that assessed the knowledge and practices of SUSTAIN-supported capacity building participants and compared their responses with similarly qualified health workers that had not participated. Health workers that had participated in SUSTAIN trainings were also surveyed about their perceptions of the value and quality of the trainings they received. Qualitative methods consisted of review of project and secondary data, including the 2010 Georgian Reproductive Health Survey (GRHS), the 2011 Maternal Mortality Study, secondary data from international organizations such as UNFPA, UNICEF, as well as special studies in RH and MNCH conducted in Georgia and other countries in the region5 . The Evaluation Team (ET) made site visits to both target and non-target health facilities and conducted 42 Key Informant Interviews (KII) with stakeholders and beneficiaries, and 15 Focus Group Discussions (FGDs) with 73 Women of Reproductive Age (WRA) across the three regions of the country that had been selected to be evaluation sites because the majority of the Georgian population lives in those areas. The evaluation attempted to mitigate known limitations of each of the selected methods, including self-selection bias, halo bias, availability of key informants, and refusals. The limitations were addressed by randomization of health facilities and participants, guaranteeing confidentiality to respondents, providing privacy for discussion on sensitive topics (such as contraception), and using methods that allowed input from hundreds of individuals representing multiple perspectives on key components of the project. The ET was able to interview almost all of the important key informants. Results of quantitative and qualitative findings were triangulated, compared, and analyzed. Preliminary findings were presented at a meeting with USAID/Georgia for feedback and additional input, and compiled in a draft report for comments from USAID. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Question 1: How flexible and effective was the project in adjusting to the radical changes in the political environment and did those changes affect the key outcomes? In answering this question, without exception, all stakeholders that were interviewed by the ET praised SUSTAIN's flexibility in making a shift and effectively adapting to the changes after 2012 so that positive project impacts were still achieved. Most stakeholders said they either barely noticed this shift or did not notice it at all because SUSTAIN had already established strong working relationships with heath service providers and key health leaders in the country prior to the shift. Accordingly, it was easier for the project to face changes in the political environment. Private pharmaceutical and insurance networks played a smaller role in the second half of the project; however, insurance companies remained engaged with the program as the country moved forward with developing the regionalization program. Government officials with new responsibilities in managing health care said they already had good relationships with SUSTAIN staff and respected their competence; therefore, they were very pleased to have their technical 5 Hill, K. et all, Assessment of Best Practices in Maternal and Newborn Care in Albania, Armenia, Georgia and Russia, Technical Report, Health Care Improvement Project, University Research Corporation, June 2012. iv support as they move to institutionalize Evidence-Based Perinatal Care (EPC) within the national health quality improvement program through regionalization, accreditation, and certification of health facilities. Both MoLHSA and SUSTAIN say that they estimate it will take until 2016 to complete this process. Conclusion Due to strong professional relationships and their reputation for high quality, evidence-based technical assistance that existed both before and after the change of government, and since the overall purpose of the program did not change, SUSTAIN was able to be extremely flexible and effective in adapting to the major health system changes that resulted from the 2012 elections. The shift to increased collaboration with the national government corresponded with decreased focus on private insurance companies, other private owners, and pharmacies in making changes in health service delivery. National efforts in regionalization of perinatal care are intended to institutionalize the improved practices that came from the capacity building in the SUSTAIN project. Recommendation The ET supports USAID’s decision to extend SUSTAIN’s technical assistance to the MoLHSA and, if funds become available, to consider a second one-year extension to finish the regionalization process that would contribute to sustaining and scaling up the changes the project supported. Question 2: What were significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted from the project interventions? Based on meetings with stakeholders and health service providers, as well as the quantitative survey that ET conducted with health providers that received SUSTAIN’s technical assistance, the ET found that SUSTAIN built upon a foundation of applying global, evidence-based RH and MNCH best practices, which had first been introduced to Georgia at scale by the HWG project. HWG, in turn, was based on a package of MNCH interventions promoted by the WHO specifically to be used to update MNCH practices in Former Soviet Union (FSU) countries. SUSTAIN extended much of the institutional strengthening capacity building to pre￾service training institutions and professional associations, and introduced “hands on” training in additional life-saving, evidence-based technical interventions, such as Active Management of the Third Stage of Labor (AMSTL). Without exception, stakeholders interviewed in KIIs provided examples of multiple positive health outcomes, including decreased Postpartum Hemorrhage (PPH). They also said that a very important impact of the program was a major change in professional culture away from traditional practices to “evidence-based” practices founded on scientific research. The consensus was that the health providers are now eager to incorporate and adapt best practices at every level in the future, even without SUSTAIN's support. According to them, decreased cases of PPH due to implementation of AMSTL, decreased complications when PPH occurs, and improved newborn outcomes, have reduced OB/GYN and neonatologist workloads and resulted in obvious improved birth outcomes. During KIIs and FGDs, stakeholders and clients mentioned that now health facilities are more hygienic, doctors provide more information to clients, and that women received more care aimed at improving their comfort when they were in labor. Data included in SUSTAIN’s monthly reports, as well as v some data supplied by health service providers working in health facilities, indicated increases in interventions and behaviors that are known from the evidence-base to result in positive maternal and newborn outcomes. Outcomes themselves, such a decreased maternal and infant mortality, were not tracked by the project and could not be measured in a performance evaluation. To see outcome results at scale, a reproductive health survey such as the (proposed) 2015 Georgia Reproductive Health Survey (GRHS), or another population-based quantitative survey would be needed. But since AMSTL and EPC were introduced in more than 50% of maternity hospitals in Georgia, the ET, based on interviews and data collected, was able to draw conclusions related to the effectiveness of SUSTAIN’s technical assistance. SUSTAIN also expanded access to FP services and contraceptive choice. The project introduced three new contraceptive methods, including more affordable oral contraceptive pills (OCPs) and a new, long-term reversible method (implants), as well as trained health workers and pharmacists on FP counseling. SUSTAIN also provided supplies of contraceptives for free distribution to district polyclinics/Primary Health Care (PHC) and women consultation clinics that provide FP services. As a result, uptake of contraceptives has increased; however, there is still more to be done to overcome religious and cultural barriers as well as provider and client misconceptions about some (primarily hormonal) methods. In addition, the long-term supply of affordable and accessible contraceptives is not guaranteed. Conclusion SUSTAIN contributed to positive clinical outcomes in target maternity clinics by providing support for implementation of new policies, protocols, and health worker professional skills to implement proven interventions to reduce maternal and infant mortality. The positive outcomes included decreased rates of episiotomy and PPH through AMTSL in mothers, and increased application of neonatal resuscitation and effective neonatal care in babies. The project also increased access to affordable contraceptives and a new long-term reversible method (implants) but it is unclear if the impact of the contributions regarding contraceptives can be sustained. Recommendation In addition to the extension mentioned in Question 1, the ET recommends that the regionalization, accreditation, and certification process include standards for periodic training updates to health workers in MNCH and FP. The ET also supports repositioning FP as an essential component of quality MNCH services in national quality standards. Question 3: To what extent has the project contributed to private sector-led service delivery development [in particular, as it relates to the provision of evidence based Effective Perinatal Services (EPC) in the privately-owned and operated health care facilities]? SUSTAIN worked on scaling up EPC in privately-owned health facilities throughout the project. Before 2012, this was done directly through networks of private owners of health facilities. After 2012, the government took the lead even though facilities remained privately owned. By the end of the program, EPC was implemented in 56 out of 95 facilities, representing over 50% vi of facilities conducting deliveries in the country. Measurement of many EPC practices, such as AMSTL, partographs, reduced routine episiotomies and routine skin to skin contact, and immediate initiation of breastfeeding showed increased usage of these methods. On the other hand, there was no reduction in Cesarean Section (CS) rates. Evaluators found multifactorial explanations about the causes for this, which would require policies and procedures on multiple levels to address both client perceptions and demand, and provider resistance before reductions in CS will occur. SUSTAIN was the only large technical-support program focused on perinatal care from 2009 to 2014 in Georgia. SUSTAIN is now providing technical assistance in regionalization to the national government that has responsibility for monitoring and regulating the quality of care services (see Question 5 on Sustainability). Technical assistance in EPC in medical training institutions is addressed in Question 4. Conclusion SUSTAIN effectively scaled-up EPC to cover a majority of facilities conducting deliveries in Georgia. To do this, the project collaborated with all targeted health facilities, which remained privately-owned and managed. The regionalization process will assist the national government with the scaling up of EPC to private facilities not targeted by the program. Recommendations The possibility of scaling up EPC to the health care facilities that remain after the regionalization process currently underway is another reason why ET agrees that the extension of SUSTAIN’s technical assistance to MoLHSA was warranted. Since CS rates remain high, if Georgia wishes to reduce them and allow the execution of only CSs that are necessary, additional measures will need to be taken to develop appropriate national policies. Discussions focused on implementing policies aimed at reducing unnecessary CS should be an explicit component of scaling up EPC within the regionalization, certification, and accreditation process. Quality assurance monitoring should include methods to determine whether clinical indications for CS (or any clinical procedure) were present. To address the non-clinical factors contributing to CS, including client perceptions and demand and conflicting factors motivating provider behavior, a strong commitment at the national level to engage in this complex approach is needed by the GoG in partnership with the professional associations and private-sector implementers in order to overcome these barriers. These approaches should explicitly address women’s fear of pain in childbirth as well as provider and health facility management resistance to reducing CS rates. A clear shared vision should be developed and indicators that measure successful progress towards that vision will be needed. Question 4: Are the key stakeholders satisfied with the project (with special emphasis on medical schools with regards to curriculum and education practices at medical schools)? Evaluators interviewed a wide variety of health providers, educators, department heads, trainees, medical students and clients about multiple topics related to curriculum and education practices, both in medical school curricula as well as in-service training on specific FP and EPC vii topics. KIIs with medical faculty indicated profound satisfaction with the new training methods, equipment, and curricula related to Obstetrics and Gynecology (OB/GYN). Faculty said they will apply the improved teaching techniques they learned from SUSTAIN to other medical disciplines. SUSTAIN’s assistance led to the first ever offering among post-Soviet countries of the internationally-recognized Objective Structured Clinical Examination (OSCE) in OB/GYN in Georgia in 2014. The exam was passed by 250 students. Stakeholders said this assistance will lead to high quality new-practitioners because they will not have learned outdated information and practices, which they will then have to “unlearn.” According to USAID, this component of the program represented a small portion of the budget. The survey that the ET conducted with the participants in SUSTAIN-sponsored training also found high satisfaction in the quality of training and ability to make practical use of what they learned6 . Master Trainers - some of whom are also medical professors - that were interviewed in KIIs also expressed a high level of satisfaction with the quality of the training they received from SUSTAIN as well as the usefulness of the training methods they were taught. Conclusion Stakeholders, including medical faculty, were extremely satisfied with the quality and usefulness of SUSTAIN’s support with regard to medical school curricula and improved practices in medical education. Given the small portion of the project’s budget devoted to this component of the program, it represents very good value for money. Recommendation Since SUSTAIN will work for another year supporting the national government health reforms focusing on quality of health services, discussion of standards and regulations during the regionalization process should also include how Georgia will provide continued support for quality pre-service and continuous health worker professional development. Question 5: How sustainable are project activities? Stakeholders interviewed by the ET said that the experiences of the effects of evidence-based approaches to FP and MNCH were so positive that improved practices in FP and EPC will be continued “because we see the results” (expression used by many respondents). They also said they felt return to maternity practices of Soviet period, where women were subjected to unnecessary drugs and procedures, would be “highly unlikely.” In addition to much more positive health outcomes, EPC has been shown to reduce costs for deliveries. The regionalization process currently underway at the national level is intended to ensure these evidence-based practices are part of the national health care quality standards. 6 Detailed information, including number of participants and the questions that were asked in the mini-survey are included in the Evaluation Methods and Findings, Conclusions and Recommendations sections of the report as well as in the Mini-survey report included in Annex 3. viii When SUSTAIN ends, free distribution of contraceptives that they have conducted will also end (UNFPA has already stopped). Sustainable access to affordable contraceptives for many women will be in doubt unless the national government includes FP and access to affordable contraceptives in the national standards for MNCH with which all private facilities will need to comply. Pharmacists are uncertain whether their networks will continue to provide low-cost OCPs introduced in the project, and implants cost over $100 on the open market. SUSTAIN’s position is that FP is an essential component of quality MNCH services. Many Parents Schools (similar to preparation for childbirth classes) were established during HWG and continued by SUSTAIN project. The on-site schools in health facilities are not functioning as well as had been envisioned. KIs said that this is due to the limited time and financial incentives for physicians facilitating them as well as the lack of willingness of some families to attend the Parents Schools’ sessions. The online PS introduced by SUSTAIN to address these challenges and expand access to the valuable parent’s education showed 7,000 new visitors to the site per month as of August 2014. However, the site will need to be maintained and updated to continue and new strategies for reaching rural populations are still needed. This will require financial and technical support over time. Conclusion Changes in professional medical culture with almost universal acceptance of the improved evidence-based practices of EPC, expanded access to affordable FP and improved medical education techniques, and inclusion in the national regionalization, certification and accreditation process, indicate that, with the exception of the uncertain continuation of access to affordable contraceptives, most of the major SUSTAIN activities (examples include EPC, contributions to quality management and regionalization, and improved medical training practices) will be sustained if they are formally adopted (as planned) into the national system. Sustainability of the Parents Schools is also unclear and requires more attention from professional associations to ensure that the course is expanded to be both operational and usable by families and receives continued oversight. As of now, the course covers many important aspects of the Continuum of Care in Promotion of MNCH supported by WHO, such as pre-conception, pregnancy, delivery and post-partum care, and addresses topics not covered elsewhere. At the national level, but not specifically within the scope of the SUSTAIN project, progress in reduction of maternal and newborn mortality and morbidity will depend on uptake of evidence￾based practices in the other components of WHO Continuum of Care (that include antenatal care, child health, reproductive health and adolescents) for mothers, babies and children. Recommendations Support for the national regionalization process has been mentioned in earlier recommendations. Sustainability and continued progress in uptake of evidence-based practices that have been proven to reduce maternal and newborn mortality will depend on whether the other components of the WHO Continuum of Care for mothers, babies and children (that include antenatal care, child health, reproductive health and adolescents) are also strengthened. The ix additional MNCH components were outside of the scope of the SUSTAIN project, but as part of the regionalization of perinatal services, improved antenatal care, access to affordable FP, and strengthened linkages for newborns with child health services should receive attention in the national health agenda. More of the Georgian population could benefit from Parents’ Schools by positioning them closer to rural communities and involving other types of health professionals, such as Village Doctors and/or as facilitators. Community volunteers might be recruited to become engaged to support and extend the impact of the health promotion messages and behaviors included in the Parent’s Schools’ curriculum. 1 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE The purpose of the performance evaluation of the Sustaining Family Planning and Child Health (SUSTAIN) project was to examine the project’s effects on advancement of family planning (FP) and maternal and child healthcare (MCH) services in target facilities throughout Georgia, and determine how likely it is to sustain the quality services in these facilities after the end of the project. SUSTAIN was funded by the United States Agency for International Development (USAID) Mission in Georgia, and implemented by John Snow Incorporated (JSI) between October 2009 and September 2014. The project has been granted a one year extension for a limited number of activities and is now scheduled to end in September 2015. The goal of the SUSTAIN evaluation was to provide an independent view of how the project has been implemented and what results have been achieved in the target regions by determining: 1) actual progress toward achieving key expected results; 2) accomplishments, delays, external and internal challenges, and their impact on the project; and 3) the project’s flexibility and effectiveness in adjusting to the changes in the political environment. The evaluation was conducted during the period of September-October 2014, by a team assembled by Mendez, England & Associates (ME&A). The team consisted of three key experts: Ms. Jean Capps, Team Leader; Dr. Boris Sergeyev, Evaluation Specialist; and Dr. Rusudan Chkhubianishvili, Local Health Expert. The intended audience of the evaluation includes USAID/Georgia, as well as JSI as implementing agency. The results of the evaluation may also be shared with project partners and other local stakeholders, including the Ministry of Labor, Health, and Social Affairs (MoLHSA), the Donor Coordination Unit under the Chancellery of the Government of Georgia (GoG), the National Center for Disease Control and Prevention, the National Maternal and Child Health Council, and other donors working in this area including UNICEF and UNFPA. USAID/Georgia may share findings and lessons learned with other USAID Missions and partners working in FP and MNCH. 1.2 EVALUATION QUESTIONS The Evaluation Team (ET) was asked to answer a number of specific evaluation questions, outlined below. The methods and sources used to research each question are described in the Evaluation Matrix in Annex 2. 1. How flexible and effective was the project in adjusting to the radical changes in the political environment and did those changes affect the key outcomes? 2. What were significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted from the project interventions? 3. To what extent has the project contributed to private sector-led service delivery development (in 2 particular, as it relates to the provision of evidence based Effective Perinatal Services (EPC) in the privately owned and operated health care facilities? 4. Are the key stakeholders satisfied with the project (with special emphasis on medical schools with regards to curriculum and education practices at medical schools)? 5. How sustainable are project activities? 2.0 PROJECT BACKGROUND Georgia has a population of 4.5 million people and a declining birth rate. As of 20107 , Georgia's Maternal Mortality Ratio (MMR) was at 27.7 per 100,000 live births and not on track to reach the Millennium Development Goal (MDG) of 12 per 100,000 live births by 2015. The under-five child mortality rate had already reached MDG targets in 2010, but infant mortality still remained high, with the majority of deaths in the early neo-natal and neo-natal period. Fetal and newborn deaths during the perinatal period constituted to 77% of all infant mortality in the country. These data pinpointed the need for improvements in the quality of perinatal care8 . In 2009, USAID funded the five-year SUSTAIN project, which was intended to build upon the previous Healthy Women in Georgia (HWG) project that had introduced evidence-based reproductive health (RH) and perinatal services (EPC) in a select number of facilities in Georgia. SUSTAIN was intended to extend and sustain those results while also adding new technical approaches, such as Active Management of the Third Stage of Labor (AMSTL); management of preeclampsia and eclampsia intended to prevent specific causes of maternal mortality, such as Postpartum Hemorrhage (PPH) and Pregnancy-Induced Hypertension; and also introduce techniques for neonatal resuscitation and management of other critical maternal and newborn conditions. The project worked in public and private partnership with health care providers and insurers to improve the quality of care in each of these areas. Georgia has been through several health sector reforms since independence. Shortly after the change of government in 2003, massive hospital privatization was initiated and the majority of state hospitals were sold to private investors. In 2007, State Medical Insurance was introduced to provide health vouchers to the poorest sectors of the Georgian population to purchase private insurance. To finalize hospital privatization, private insurance companies participating in State health care programs were required to have built hospitals in the regions by 2012. This increased access to modern, well-equipped facilities and health services to most of the population. Full responsibility for health services, including public health, was handed over to the private insurance companies. 7 More recent information from the Georgian NCDCH says that “as of 2013 the maternal mortality ratio was 26 per 100,000 live births”. If this is the case, then there has been slightly more additional reduction in maternal deaths. 8 Health care - Georgia, Short statistical highlights, National Center of Disease Control and Public Health (NCDC), 2013. 3 Prior to 2012, the role of the public (national government) sector was largely limited to supporting a "safety net" that only covered services for the poorest economic groups. However, this still left out a large percentage of the population (more than 50%) who could not afford to pay for private insurance. Previous Georgian government health policy encouraged competition between service providers through privatization, liberal market policy, and reduction of regulations to an essential minimum. Within the de-regulation framework, re￾certification and continuous professional development of medical personnel were abolished, and health specialists were not required to update or refresh their skills and qualifications. Due to the strong market-oriented policies of the existing government at the beginning of the SUSTAIN project, SUSTAIN focused MNCH capacity building through the networks of private facilities and providers. FP activities had a strong social-marketing component through networks of private pharmacies. Pharmaceutical networks were engaged to introduce some new contraceptive methods into the market. Oral contraceptive pills (OCP) could be purchased over the counter without a prescription at the time SUSTAIN started, but prices were high and the selection was limited. Little or no counselling on selection of OCP or side effects was offered. Fear of the side effects of hormonal methods has long prevented many women from trying to use them. Working directly with pharmacies was intended to provide high-quality affordable OCPs, along with training pharmacists (through their networks) on counselling women how to use them. Half-way through the project, the government changed as a result of the 2012 election. This resulted in a re-orientation of health service policies, protocols and standards. Changes included reform of existing health financing arrangements and national level efforts to improve the quality of health services. At the same time, regulations changed and OCPs that were over the counter could now be sold by prescription only, limiting the potential impact of a social marketing approach to increasing access to affordable contraceptives. The new government assumed a stronger role in financing and oversight of health services. As a result, in 2013, it introduced the Universal Health Care State Program (UHCSP) for core medical services to the Georgian citizens without any insurance packages, and mandated that the program be managed by a public entity, the Social Services Agency (SSA). The Minister of Health, however, emphasized that UHCSP was not a State insurance package; it was funded by the State budget and not by payment of premiums, did not require an insurance policy, and any person enrolled could participate. This represented a major shift away from the previous government’s voucher-based financing limited to target groups. Although State health programs are now administered by SSA, and private insurance companies are no longer in charge of State-financed health care programs, ownership and management of health facilities has remained in private hands. But they operate with very few regulations. The State Regulation Agency of Medical Activities of MoLHSA has responsibility for the quality of physicians and other health care professional services; however, it primarily deals with patients’ complaints. UHCSP reimburses facilities for services on a per capita basis, based on established fees for covered services. Private insurance schemes are still in place and remain an option for those 4 who can pay for them or are covered by their employers. The government has now assumed more responsibility for regulating and monitoring the quality of health care, even for the privately-owned and managed health facilities. Regionalization, certification, and accreditation of perinatal services that includes criteria to qualify for various levels of perinatal services, is under development by the national government and is expected to be fully in place within two years. SUSTAIN has been a key partner to MoLHSA in developing this process. UNFPA and USAID/SUSTAIN (and USAID/HWG prior to 2009) have been the lead agencies providing support for national RH human resource capacity building. 3.0 EVALUATION METHODS & LIMITATIONS 3.1 EVALUATION METHODOLOGY To conduct the evaluation, the ET used a quasi-experimental design. Randomized control approach, typical of experimental methods, was not considered applicable9 given the fact that this was a performance evaluation and the time and resources allocated to conduct it were limited. Another reason why randomized control approach could not be used was that the ET had no control over assigning regions to treatment (where project activities are implemented) and control (where no project activities are introduced) groups, where they could ensure similarity in key aspects relevant to SUSTAIN. The quasi-experimental design called for carrying out the evaluation in those regions and facilities where SUSTAIN’s activities were implemented (intervention group), as well as in those where SUSTAIN did not operate (comparison group). As the project works in just above 50% (56 out of 95) facilities throughout Georgia, non-participating facilities are located in the same regions where SUSTAIN operates. Both participating and non-participating facilities were sampled in the regions selected for this evaluation. To conduct the evaluation, the ET selected those regions that were implementing a full range of activities - effective perinatal care (EPC), contraceptive distribution, social marketing to pharmacies, in-service training, and promotion of effective perinatal services – and were introducing the new FP and MNCH-related curriculum and education practices at medical schools. In addition, selection of regions took into account the 201310 birth, maternal mortality, and abortion rates to ensure areas were comparable. Other considerations for site selection included geographical access of the population to improved services and the extent of their involvement in the project’s activities. The three regions selected for this evaluation are presented below (see Table 1). 9 USAID’s Evaluation Policy. 10 Statistical Yearbook, National Center for Disease Control and Public Health, 2013. 5 Table 1: Regions Selected to Conduct the Evaluation Region Number of births / Birth rate per 1,000 population MMR per 100,000 live births Neonatal mortality per 1,000 live births Abortion per 100 live births Contracepti on / Abortion ratio Tbilisi 24,248 / 20.7 20.6 12.0 52.7 0.4 Imereti 8,477 / 12.0 59.0 13.4 61.0 1.6 Samegrelo￾Zemo/ Svaneti 3,820 / 8.0 26.2 1.0 34.6 2.4 Georgia 57,688 / 12.9 26.0 8.4 64.2 1.0 The regions included Tbilisi, Imereti and Samegrelo-Zemo Svaneti. In Imereti and Samegrelo￾Zemo, the ET collected data in regional capitals (Kutaisi and Zugdidi, respectively) as well as in four randomly selected districts. In Imereti, these randomly selected districts included Chiatura, Samtredia, Tskhaltubo and Zestafoni; in Samegrelo-Zemo Svaneti, the districts included Abasha, Khobi, Senaki and Tsalenjikha (see Annex 3 and Annex 4 for details about sites visited and informants interviewed). To gather data, both in intervention and comparison sites, the ET used a mixed methods approach which included: 1) review of project documents and secondary data/background documents, project monitoring data, and local health statistics; 2) semi-structured questionnaires for interview of key informants (KIs); 3) surveys of physicians and nurses (see Annex 3); 4) key informant interviews (KIIs) with health workers and health facility managers, private network representatives, and ministry representatives; and 5) focus group discussions (FGDs) with project beneficiaries and women of reproductive age (WRA) who were the primary clients of SUSTAIN-supported health services. This mixed methods approach, which relied on the use of both qualitative and quantitative methods, helped the ET capture the diversity of opinions and perceptions of beneficiaries and stakeholders about SUSTAIN’s strengths and weaknesses, challenges and barriers faced, as well as personal satisfaction from participation in the project. Given both budget and time constraints, the selection of methods was based on the best evidence-based and feasible approaches to obtain the desired information. A detailed description of the respective quantitative and qualitative methods is provided below. 3.1.1 Quantitative Research and Analysis As part of the quantitative research and analysis, the ET conducted a mini-survey of health workers at 23 participating and 15 non-participating health facilities in the Tbilisi, Imereti, and Samegrelo-Zemo Svaneti regions. Regions, districts within regions (Samtredia and Tskhaltubo in Imereti, and Abasha and Senaki in Samegrelo-Zemo Svaneti), and health facilities were selected randomly, using the respective data provided by SUSTAIN, supplemented with information on health facilities available on the GoG website. In each of the 23 health facilities whose staff took part in the SUSTAIN-sponsored training seminars, three to five medical professional participants were sampled using the list of names provided by the project. Although in some cases the selected respondents could not be located due to staff turnover, the ET was able to reach most of them. By matching characteristics of the participating facilities, 15 non￾participating medical institutions were selected in the respective districts where 63 health 6 professionals, with qualifications similar to those trained by SUSTAIN, were sampled and asked comparable questions related to recent trainings they had received. Overall, 140 respondents were targeted for interview and 12711 interviews were successfully completed, resulting in a response rate of 91%. Sampling design ensured that the mini-survey’s results were representative of medical professionals’ views in the selected regions, falling within 5% of the respective population’s parameter at the 95% confidence level. Questions for the survey were first developed in English, translated into the Georgian language, and then checked for accuracy by the Georgian health specialist and ET member. The questions captured the respondents’ involvement in SUSTAIN’s activities and the knowledge, expertise, and skills gained through participating in SUSTAIN’s training programs; issues where providing additional support would be appropriate; and perspectives regarding whether these activities have (or will have) made RH services more accessible and sustainable for WRA in Georgia. Respondents’ participation in mini-surveys was voluntary and they were advised that their identity would remain anonymous to encourage frank responses. An informed consent statement conforming to USG guidance on protection of human subjects was provided before the start of interview. Interviews were conducted face-to face by interviewers trained by IRMS, the local sub-contractor responsible for the data-collection phase of the mini-survey. Data were entered, cleaned, and analysed using data entry software. The respective univariate distributions are available in Annex 3. 3.1.2 Qualitative Research and Analysis Review of secondary data/background documents Collection of secondary data comprised reviewing information on SUSTAIN’s activities aimed at promoting RH and FP services in Georgia, project-related sources, and third-party data on the quality of maternal health and FP services, as well as capacity-building needs of service providers. In addition, the ET collected data from documents such as studies conducted by the HWG project, UNFPA, UNICEF, the GRHS, as well as publications from MoLHSA and international organizations providing services to women and their families. Key informant interviews KIIs were held at national, regional, and facility levels, and included pharmacies. The list of KIs was developed in cooperation with USAID, the project, and the national partners; respondents were selected for a wide perspective on SUSTAIN's performance. Policy-makers, administrators of medical facilities, and doctors from secondary and primary health facilities were targeted for in-person interviews. Semi-structured protocols covering issues such as perceived changes in the range and quality of MCH and FP services provided at health institutions, availability of contraceptives, the role of SUSTAIN in promoting these changes, assessments of its capacity-building activities, information materials, and awareness campaigns were also developed. In total, the ET completed 42 interviews at the national, regional and 11 There were 13 non-respondents in the comparison group. 7 district levels. Some informants filled multiple roles, such as being a provider as well as a SUSTAIN trainer. Focus group discussions (FGDs) To assess the quality of maternal and FP services available to Georgian women, FGDs were held in the three regions that participated in SUSTAIN. In Imereti and Samegrelo-Zemo Svaneti, the districts selected for mini-survey were also covered with FGDs. Qualifications for FGD participants included women that had delivered a baby, had an abortion, or sought FP services in the past five years. The IRMS staff made preliminary phone calls asking women a number of screening questions (whether they had delivered a baby, had an abortion, or sought FP services in the past five years). Eligible participants were then invited to take part in the FGD on the conditions of anonymity and confidentiality. Those who agreed also signed the informed consent form and completed a registration form including demographic information, whether the participant had used MCH and/or FP services since 2009, and rating their opinion of health service accessibility and quality. On average, each FGD took about one hour and included a maximum of eight patients. In total, 73 people participated in FGDs and discussed topics ranging from MNCH and FP services available to familiarity with various contraceptives and sources of information on FP and MCH. The summary of FGDs’ results is included in Annex 5. 3.2 EVALUATION LIMITATIONS Limited time and resources. Limited time and resources for the evaluation prevented a population-based randomized survey of WRA. While a randomized population-based survey (like the GRHS) would be the best method of determining changes in health behaviors and outcomes, those types of studies require large numbers of people and months to collect and analyze data. The time and budget provided for the performance evaluation would not allow for this type of study to be done. Selection bias. The evaluation selected respondents from both participating and non￾participating facilities and sought qualitative perspectives from a wide variety of respondents. However, a few KIs declined to be interviewed, resulting in a self-selection bias where respondents who chose to be interviewed might differ from those who did not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experiences. Some trainee lists turned out to be outdated and some respondents were removed because they did not fit the criteria for the survey. Recall bias. SUSTAIN’s activities were launched in October 2009 and covered a period of 5 years. Therefore there was a risk that some respondents could find it difficult to accurately compare situations before and after the project. In addition, some key informants were involved in the previous HWG project (2003-2009). However, they were able to respond to questions about SUSTAIN when reminded that the evaluation only covered activities after 2009. Although FGD respondents were not able to name the specific project that resulted in changes in health services, the events such as births, abortions, or use of FP are relatively rare and specific enough that this did not prove to be a problem in providing time frames for comparisons. 8 Halo bias. The tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm, was addressed by using female interviewers with guarantees of anonymity as well as asking the same questions to several groups of WRA in multiple locations representing a wide area of the project. By selecting different types of appropriate methods and triangulating findings from various sources of information to determine common responses, these limitations were mitigated. 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Note: SUSTAIN is an integrated program, meaning that various components are designed to support and reinforce the effects of other components and some activities are designed to impact multiple objectives. In addition, some KII respondents played multiple roles in the project, e.g. training participant, trainer, private health facility owner, officer of professional association, and medical school faculty. To avoid unnecessary repetition, some answers include cross references to responses to other questions. Some overlap in answers, however, is likely. 4.1 QUESTION 1: HOW FLEXIBLE AND EFFECTIVE WAS THE PROJECT IN ADJUSTING TO THE RADICAL CHANGES IN THE POLITICAL ENVIRONMENT AND DID THOSE CHANGES AFFECT THE KEY OUTCOMES? 4.1.1 Findings All key respondents familiar with the SUSTAIN project, as well as review of project documents both before and after 2012, indicated that SUSTAIN proved to be exceptionally flexible in adapting from its private-sector, market-driven approach to working with the MoLHSA in assuming the leadership role in bringing health system oversight, policy decisions, and adoption of the UHCSP. The project’s strategy shifted to a much larger focus on providing technical assistance directly to the national government. The primary purpose was to help institutionalize evidence-based perinatal care through regionalization, accreditation, and certification of perinatal services. Up until the change in government, private sector facility owners delivering maternity services were not accountable to any single regulatory entity in the country that would guarantee adherence to quality of care and health professional standards. USAID’s one-year extension of some SUSTAIN activities will enable them to focus support on completing this adaptation through a system of national regulation and accountability for perinatal health service delivery standards. Aside from government officials, informants said that any adjustments that had to be made within SUSTAIN were not noticeable because capacity building activities continued, although less emphasis was directed towards insurance networks and pharmacies. Although roles changed, many of the same health professionals were involved both before and after the change in government. MoLHSA officials said that they found SUSTAIN “very flexible and effective” in adapting to the changes. SUSTAIN has already helped standardize maternity care across Georgia by developing 9 ten obstetric and neonatal clinical practice guidelines and protocols, which have been approved and put into practice by the MoLHSA12 . Both SUSTAIN managers and MoLHSA officials added that they estimate it will actually take two years for MoLHSA to fully institutionalize the national-level reforms through regionalization, accreditation, and certification of perinatal services that SUSTAIN has helped to develop. 4.1.2 Conclusion SUSTAIN was both flexible and effective in adapting to the radical changes that took place in government after 2012. SUSTAIN’s objectives, as stated at the beginning of the project, were achieved with increased uptake of health care practices and behaviors, although the emphasis of the implementing partners shifted from private sector networks to working more closely with the national government. Strong professional relationships with partners before and after the change in government in 2012, as well as with current and former government officials, made the transition smooth while retaining strong professional relationships with the private sector. The one-year extension provided by USAID was well justified. 4.1.3 Recommendations To support long-term sustainability of the changes made during the SUSTAIN project, within available funding and policy options, USAID could consider supporting the second year extension to complete the regionalization process if sufficient progress has been made by the end of the first year. 4.2 QUESTION 2: WHAT WERE SIGNIFICANT POSITIVE CHANGES IN THE TARGET SITES, EVIDENCED BY THE IMPROVED HEALTH OUTCOMES THAT MAY HAVE RESULTED FROM THE PROJECT INTERVENTIONS? 4.2.1 Findings To answer this question the ET focused on facility-level changes. Target sites included health facilities [maternity hospitals, clinics, and Primary Health Care (PHC) centers] and pharmacies (included in the private sector social marketing of contraceptives component in the original strategy). Some facilities provide both maternity and FP services, while others (such as clinics) were targeted only for FP services. The next Georgia Reproductive Health Survey (GRHS) will be needed to objectively identify positive health outcomes and changed behaviors at the human level. Findings related to specific capacity-building components of the program, such as EPC in privately-owned facilities and medical school pre-service training, are addressed in more depth under questions 3 and 4. SUSTAIN strengthened the quality of private sector provision of maternal/neonatal, FP, and RH services across the country. EPC was scaled up to over 50% of facilities that conduct deliveries in the country. To improve the quality of FP and RH services, the project has trained 1,865 12 USAID Georgia SUSTAIN SOW for one year extension, 2014. 10 health care providers and pharmacists. As a result, 800 primary health care facilities now offer FP services13 . Maternal, Newborn and Child Health Stakeholders’ opinions Government officials, health facility managers, and representatives of OB/GYN professional networks (who are also facility managers, practitioners and trainers) were unanimous in their praise for SUSTAIN’s positive impact on MNCH and FP health services at target sites through their support for introducing and institutionalizing evidence-based health interventions. Several contrasted SUSTAIN’s capacity building approaches with the former Soviet training models where long time practitioners or professors lectured without providing any scientific basis for what they taught and new approaches and scientific findings were not integrated as they were developed. They also specifically mentioned that it was not only the content but also the training methods that they felt were very effective. SUSTAIN combined scientific evidence and information with “hands on” practical experience, including use of “The Simulator,” and simulator-based training, a model and approach where most conditions encountered in complicated and uncomplicated deliveries can be simulated. Government MNCH officials said that SUSTAIN was one of the best partners they have had in helping them improve the quality of health services in facilities across the country (the other was UNICEF for immunization services). These positive assessments were corroborated in the survey that the ET conducted with health professionals, in which 51 respondents from SUSTAIN’s participating medical institutions, vs 39 from non-participating facilities, reported that effective newborn care is available at their facilities (Chi-Square 4.83, p<.03)14 . Health providers’ feedback The survey conducted by the ET found that there was overwhelming satisfaction with the SUSTAIN-sponsored training courses and technical assistance among their medical practitioner participants (N=64). When asked the question “What is your overall impression of the training programs and technical assistance provided by SUSTAIN? Are you very satisfied, satisfied neutral, dissatisfied or very dissatisfied with them?” 44.2% of participants were “very satisfied” while 48.1% were satisfied with these programs15 . Non-participants were not asked this question. Figure 1, next page, indicates the participants’ overall impression of SUSTAIN-sponsored training programs. Participants were also asked the question “What is your opinion about the quality of capacity building or health education technical assistance provided by SUSTAIN? How satisfied are you with them?” An overwhelming majority of participants stated that they are either “very satisfied” 13 USAID Georgia, SUSTAIN Extension Document, 2014. 14 Evaluation mini-survey, October 2014 15 Evaluation mini-survey of SUSTAIN training participants, October 2014. 11 (44.2%) or “satisfied” (49.3%) with the quality of capacity building or technical assistance provided by SUSTAIN (see Figure 2)16 . Figure 1. Overall Impression of SUSTAIN-Sponsored Training Programs among Participants17 Figure 2. Level Satisfaction with the Quality of SUSTAIN-Sponsored Capacity Building and Technical Assistance18 Through those trainings, SUSTAIN built on the foundation of evidence-based practices introduced in HWG, but also added on-the-job training in key specific evidence-based EPC practices, such as AMSTL, management of PPH, neonatal resuscitation, Post Abortion Care (PAC), and extensive use of the partograph to track progress of women in labor. In addition, 61% of survey respondents confirmed that since 2010, new methods of contraception have been introduced in their facility.19 16 Evaluation min-survey, October 2014. 17 Ibid. 18 Ibid. 19 Evaluation mini-survey results, 2014. See Annex 3 for tabulations and statistical analysis. 12 Health care providers interviewed by evaluators also referred to both provider and client satisfaction. Neonatologists specifically mentioned that introduction of skin to skin contact training and thermal protection of newborn, neonatal resuscitation and breastfeeding support resulted in many positive neonatal outcomes and reduced infant deaths. Examples of positive clinical outcomes that they mentioned included reduced complications of PPH, more babies adequately and effectively resuscitated, and ability to intervene by tracking progress of labor using the partograph. Evaluators asked for specific data to support these impressions from providers and received data from some but not all of them. In many cases, facilities documented data continuously from 2003 or 2006 and included data from the previous HWG project that continued until the present. The ET did not have access to complete data sets and therefore could not make generalizations about significance. A facility-level reproductive health survey based on the health management information service (HMIS) would be needed to measure changes in clinical outcomes, specifically from 2009 until 2014, the time period covered by the SUSTAIN project. This was beyond the scope of this performance evaluation. Caesarian Section (CS) rates did not decrease as hoped; in fact, rates increased somewhat from the beginning of the project. This is discussed in more detail in response to Question 3 of the evaluation. Providers interviewed in KIIs stated that, from their perspective, abortion rates continue to decline in their facilities but data from individual facilities was difficult to interpret. WRA FGD participants confirmed that it was their impression that the number of abortions is decreasing. Both sources of information are qualitative in nature and would require a population-based RH survey to provide objective abortion data for the entire population. Some facilities said that they no longer provide abortion services. Some OB/GYN health facility providers attributed this decrease to increased availability and uptake of contraceptives but the ET was unable to verify this. Other health providers, however, think this decline may be due to some facilities refusing to perform abortions, or charging high prices when they are performing them. They were concerned that this may be leading to increases in illegal abortions. A number of providers in Samegrelo reported that women were using over the counter misoprostol to induce unsafe abortions and were ending up in hospitals with bleeding complications. New government policy makes misoprostol available only by prescription. OB/GYNs interviewed by the ET believe that the new policy would decrease this type of unsafe abortion. Family Planning Health providers To increase the number of methods available and access to affordable contraceptives, SUSTAIN introduced and promoted longer term reversible methods [e.g. implants and Intrauterine Device (IUD)], provided supplies, and trained health professionals on how to insert them. They also provided contraceptive supplies for free distribution to some facilities, including PHC facilities, and trained Village Doctors in contraceptive counseling and oral contraceptive pills provision. The ET confirmed that participating health facilities had contraceptives available when the ET visited them; comparison sites either had no contraceptives or limited supplies. The OB/GYNs interviewed had a wide variety of opinions about introducing new contraceptive methods such as implants. Some health providers (OB/GYNs) have become enthusiastic 13 "adopters" and have already implemented what they have learned with many clients (e.g. implant insertion). Two participating providers (who are also SUSTAIN Master Trainers) have already inserted over 100 implants each. They said that from their perspective, demand was increasing and appeared to be coming largely from positive “word of mouth." Other OB/GYNs from participant sites said that they do not have time or incentives to counsel women on FP and contraceptive use, and a few cited their own religious objections to artificial means of contraception. Lactational Amenorrhea Method (LAM) has been introduced in Georgia and is part of the FP method mix promoted by SUSTAIN. Standard Days Method (SDM), aka "The Beads," was also introduced in some trainings. Both LAM and SDM are acceptable methods to other religions opposed to artificial contraception methods. LAM is thought to be popular, but the beads are not readily available and have not been widely promoted. KII respondents could not recall if LAM and SDM have ever been discussed with religious leaders that are opposed to other methods of contraception. Results of the mini-survey also indicated that new contraceptive methods such as subdermal implants are now offered at 31 SUSTAIN-sponsored medical institutions as opposed to 9 of non-participating facilities (Chi-square 17.16, p< .000). Respondents from participating facilities are also more likely to report that “new methods of contraception have been introduced here since 2010” (53 as opposed to 25 from non-participating facilities, Chi-square 24.92, p <.000). The mini-survey also revealed statistically significant differences in availability of newborn care and subdermal implants at participating medical institutions as opposed to non-participating ones20. The former also introduced new methods of contraception at higher rates21 . Some SUSTAIN components, such as educational materials and family centered delivery, have been adopted by non-target facilities (to some extent) but the reasons for adoption in non￾target sites are not known and were not measured by the project. One non-target facility said that they are now providing family-centered care due to “increased client demand.” This was possibly due to increased public awareness from SUSTAIN’s mass media activities that promoted FP and family centered delivery practices. They also said they thought that “word of mouth” from clients of participating facilities was impacting on the clients in their (non￾participating) facility. Behavior Change Communication Campaigns SUSTAIN organized three communication campaigns. One was devoted to FP, the second covered family centered delivery, and the third campaign included a combination of FP, Parents Schools, and birth partner delivery. Until September 2014, the video spots aired as Public Service Announcements (PSAs) on public and commercial TV channels as well as the USAID SUSTAIN YouTube page (https://www.youtube.com/user/USAIDSUSTAIN), which had 523 views as of the time of the evaluation. Some elements of EPC and other health information materials printed by SUSTAIN were found at non-target facilities. This is probably a result of the communication campaigns, which were intended to reach the overall population and not only the target facilities. The increased number of visitors to electronic version of Parents 20 SUSTAIN End of Project Performance Evaluation mini-survey, October 2014. 21 See mini-survey and statistical analysis results in Annex 14 Schools is also assumed to be related to the third communication campaign that promoted parents education. In January 2014, during the last year of the project, SUSTAIN revamped its project Facebook page in an effort to provide better and more timely information to project stakeholders and general public interested in the work of the project. The number of Facebook users who "like" USAID SUSTAIN's Facebook page rose from 94 to 6,281 between January and October 2014. Primary Health Care (PHC) Providers With the introduction of family medicine in Georgia, FP has been included in the list of competencies for the family physicians. However, FP services at the PHC level have not become part of the package of services provided under the state Universal Health Care Program and Village Doctor’s Program. FP services are also not covered by private insurance plans. IUD and implant insertion services are currently provided only by OB/GYNs and ”Reproductologists” (a sub-specialty of OB/GYNs) according to the strict specialization system for doctors in Georgia. The curriculum used by SUSTAIN for FP training of family physicians includes provision of Contraceptives by Family Doctors (FD). FD currently can only provide OCPs and condoms based on their officially approved competency. The curriculum also includes relevant modules that are designed to ensure the knowledge and skills required for FD to provide quality FP counseling and services. These modules include safety, managing side effects and dispensing of oral contraceptives and condoms. SUSTAIN says that they consider this to have been a major breakthrough of the project22 . Through the targeted capacity building interventions, HWG and SUSTAIN projects, as well as UNFPA, reached out to hundreds of health providers, including primary health care service providers both at district and rural areas. In district centers, FP services are provided at PHC centers, polyclinics and women consultations (clinics) by OB/GYNs. Interviews conducted by the ET at family medicine centers confirmed that providers were quite knowledgeable and effective in providing contraception counseling and advice on appropriate contraceptive selection. Some of them had stock of free Implanon and did Implanon™ and IUD insertion for their clients. Currently, these services are not allowed to be provided by FD. FP promotional materials were available and they said that they were handed out to patients on a regular basis. Village doctors - who are trained in family medicine - are private providers but they do not do deliveries. Although not in their job description and not required as part of the services they provide, village doctors were trained by SUSTAIN to provide FP counseling, services (oral contraceptive pills and condoms) and referral, and provide contraceptives free of charge to villagers when they have them. SUSTAIN says that they trained approximately 740 village doctors and 1,346 nurses in FP/PAC, FP refreshers, follow-up and supportive supervision, and distributed free contraceptives to participating PHC providers. The ET found a 6-month stock of oral contraceptives (Microlut and Microgynon) and condoms in SUSTAIN-supported PHC centers that were visited. Even though according to their job descriptions, they are not 22 SUSTAIN project, 2014. 15 required to provide FP services, respondents expressed much enthusiasm for providing both counseling and contraceptives and they said that they eagerly provide free information materials about FP to village residents. Non-participating village doctors that were interviewed in Chiatura and Tselennjikha had been trained in Family Medicine (FM) and went through a three-day module on FP as part of FM curricula. Additional trainings on modern contraceptives and FP counseling took place with support from UNFPA in 2006 and 2009 (prior to the SUSTAIN project). In the past, UNFPA used to supply “reproductology cabinets” (outpatient offices where the sub-specialty of OB/GYNs, the so-called “reproductologists” provide services) and women consultation centers at district level and above with contraceptives, but no longer does so. According to UNFPA, since Georgia is now a middle-income country, they are not able to provide free contraceptives to the country. However, they are willing to assist the MoLHSA to purchase them at favorable prices, if requested. They did not provide contraceptives to PHCs where village doctors work. The ET was surprised to find that some village doctors from non-SUSTAIN supported ambulatories in Chaitura also possessed booklets and information materials printed by SUSTAIN that were provided to them through the Municipal Center of Public Health. SUSTAIN booklets were also seen in a non-participating maternity in Samegrelo and some members of their OB/GYN staff had attended SUSTAIN training. It is possible that could also be the case in other regions due to turnover of staff, but the ET was not able to confirm this. Both participating and non-participating village doctors said that these materials were of high quality, very useful, and they would try to replicate them (probably with lower cost versions) after the project is over. When asked about linkages from maternities to PHC services after the birth of a baby, all village doctors that were interviewed said that they are not officially notified about the birth in order to follow up with the family and provide child health services after the mother returns home. Parents Schools Parents Schools were started during HWG project and continued under SUSTAIN. Women consultations or childbirth education at Parents Schools located within maternity hospitals were conducted by OB/GYNs using a Parents School curriculum. MNCH providers interviewed for KII in Tbilisi said they had Parent Schools in their facilities and that they were well attended. But KII with health providers outside of Tbilisi said that attendance at Parents Schools was limited and had been discontinued at some facilities. Some of the reasons they gave included lack of staff motivation, lack of time and financial issues. SUSTAIN had revised their strategy to promote the Parents Schools concept through social marketing and advertisement of e-Parents Schools. The social media campaign for Parents School e-course was made available at www.mshobeltaskola.ge and promoted using Facebook. Parents School promotional posters, calendar and books were distributed and they were seen by the ET at participating maternity hospitals and primary health care facilities that provide FP services. Among 73 WRA FGD participants specifically asked about Parents School, the majority said that they were aware of existence of Parents Schools, but had never actually attended one. 16 Out of 73 WRA, only one woman said that she participated in Parents School course23 . SUSTAIN tracking data for the online site documented over 6,000 new visitors by April and May 2014 and over 7,000 in August 2014. Analysis of new visitors found 70.2 % of new visitors to the Parents School web site were from Tbilisi, 23.4% were from undefined locations, 2.4 % from Kutaisi (Imereti), 3.6 % from Batumi (Achara), and 0.1% from Rustavi (Kvemo Kartli)24 , indicating that uptake of online services was much higher in urban rather than rural areas. This could explain why providers and FGD participants outside of Tbilisi were less knowledgeable and supportive of conducting PS25 . Reproductive Health Hotline Service RH hotline services were provided through a sub-contract to an NGO, Claritas. The hotline services covered both FP and MCH components, operated 12 hours every day, and promoted the online Parents School course. Callers could also obtain a free consultation on the long￾lasting FP method Implanon™ and discuss other FP and MCH topics. By August 2014, the callers who were current contraceptive users (WRA) were using pills (49%), condoms (38%), withdrawal (22%), IUD (18%), spermicides and natural methods (16%), and 2% were not using any method. Calls placed by youth were mostly related to emergency contraception. Most of the calls were from Tbilisi residents. The hotline number was advertised on all TV spots in the SUSTAIN’s communication campaigns and also the education materials and the SUSTAIN Facebook page. However, the ET could not validate the value of the hotline as it was not mentioned by KIs or WRA during FGDs. Aside from the neonatologist working for the NGO managing the hotline, no other respondents interviewed by the ET mentioned the hotline number and services as an important component of the SUSTAIN project. There is currently no funding available for the program after the SUSTAIN project ends; another source of funds will be needed for it to continue. Pharmacists The original private-sector strategy of the project introduced training for pharmacists to use a social-marketing approach to increase access to low-cost, high quality OCPs. For the purposes of the evaluation these pharmacies were considered target sites. Low cost OCPs promoted through SUSTAIN were available at pharmacies that belong to networks supported by SUSTAIN's training. SUSTAIN also provided training to pharmacists on FP counseling. The new 23 See FGD report in Annex 5 24 SUSTAIN monthly report, August 2014. 25 SUSTAIN provided additional figures from Google Analytics through October 2014 that was not available to the ET at the time of the evaluation: if SUSTAIN monthly data show consistent increase of online visitors. According to Google analytic data to compare first six month October 1, 2013 (e-course launch date) – April 1, 2014 and the last six months of the PY5, April, 2014 – October 1, 2014 there was 20 fold increase in cumulative number of online course visitors, 1, 248 vs 21, 382. 17 government policy requiring prescriptions for OCPs has decreased the relevance of social marketing for increased contraceptive uptake. The ET interviewed pharmacists that had been trained with support from SUSTAIN and those who had not. Due to high turnover of pharmacist trainees and the time elapsed since the training took place, the ET could not obtain a list of pharmacy trainees from the project. However, the ET was able to visit random participating pharmacies that were identified by the network's logo outside of the shop. SUSTAIN’s training included RH, FP, contraception methods, and counseling. Pharmacists said that their understanding and knowledge on FP counseling and methods have been improved but pharmacists will have little opportunity to use the skills they acquired in their daily practice. Therefore, they said that there had been no significant changes in their behavior related to promoting contraceptive products. Reasons for this included limited demand by customers for counseling from pharmacists and reluctance of clients to discuss contraceptives in the pharmacy because there is no privacy or confidentiality. As of September 2014, women must have a prescription to buy OCPs, so advice/counseling provided by pharmacist on subsidized contraceptive products would not impact use. This will be a consideration for any FP program design from now on. Non-participating pharmacists in Samegrelo, some of whom were trained in FP counseling, also said they do not consider it “as part of their job.” Earlier in the project, the regulated prices for OCPs promoted by SUSTAIN made them accessible and affordable. Since the move away from market-oriented services and the prescription requirements for oral contraceptives have been introduced, additional investments in pharmacy training have not been made. Client Feedback FGDs with 73 WRA were conducted in both urban and rural areas across all three regions targeted by the evaluation. All participants in these FGDs said that they had noted specific improvements in FP/MNCH services since 2009 when SUSTAIN began. Women said that facilities were more hygienic, quality of care from providers was better, and that interpersonal communication had been improved. Knowledge of methods of FP had increased and many were aware of newer methods, such as implants. Misperceptions about side effects of some methods, such as OCPs (weight gain) and fear of hormonal methods still remain and were cited as deterrents to trying some of these methods, though there were some women that knew of other women who had used implants. Several participants also listed opposition from the Church as a barrier for their using contraceptives. Cost of contraceptives was also mentioned among limiting factors. Some facilities in Tbilisi that conduct deliveries provide a box to receive written feedback from patients. Managers indicated that feedback is almost always positive about their experiences since they have started EPC in their facilities. During field visits, ET members also conducted brief KIIs with maternity clients that had just delivered in the past few days. EPC, by definition, includes allowing families to be present during delivery and “rooming” of babies in their mother’s rooms rather than in nurseries. Mothers who had babies before EPC and after said they are much happier with the later experience, especially since their family could be with them during the delivery and they could keep the baby with them all the time. 18 4.2.2 Conclusions SUSTAIN’s interventions resulted in increased availability of evidence-based health care practices, which in turn have resulted in positive clinical outcomes and reduced mortality. Perceptions of health care providers and clients, as well as data from selected facilities and from the project, also support this positive trend. Quantitative assessments undertaken during the evaluation measured high degrees of satisfaction in the usefulness and quality of clinical training provided. Mortality measurement, however, was beyond the scope of the performance evaluation. To determine if health outcomes and behaviors have changed on a population level, the GRHS will be needed to measure them. Specific perspectives on SUSTAIN’s performance are provided below.  Health providers are universally supportive of the evidence-based approaches and say that these changes will continue after the project and are confident they will continue to result in positive health outcomes. There remain some barriers to full use of some practices promoted by the project (such as reduction in CS or increased use of hormonal contraceptives) but evidence has shown that these types of changes need several years to take effect. The exact timing cannot be pre-determined, but they should improve if institutionalized in pro-active national policies that are enforced and supported from the top by the professional and regulatory authorities.  Working with the medical training institutions will introduce most future providers of perinatal and FP services to the best clinical practices in these areas from the beginning of their careers. The Dean and OB/GYN faculty from the largest medical school in the country in Tbilisi (TSMU) said that they plan to continue the training practices and content introduced by SUSTAIN and would like to continue their linkages with the global health community. This is good evidence that they value what SUSTAIN has done and would like to build upon it.  SUSTAIN succeeded in introducing low-cost and affordable contraceptives into pharmacies. Pharmacists, however, were never significantly proactive in providing FP counseling and contraceptive selection advice to women. Environmental factors inside commercial pharmacies do not provide sufficient privacy for respectful FP counseling and many pharmacists do not consider it part of their duties. Women’s reluctance to accept that advice due to the lack of privacy at pharmacies and cultural considerations were major barriers to the success of this part of the project. Price-checks confirmed prices that were significantly lower for those OCPs than the commercial market prices, indicating the intervention reduced cost as an inhibiting factor. New regulations requiring prescriptions for OCPs make health providers, rather than the pharmacists, the more appropriate target for capacity building in FP counseling. These new policies make it impractical to continue a social marketing approach for anything other than with condoms in future programs. It is unclear whether pharmaceutical chains will choose to include these OCPs after SUSTAIN ends if other products are more profitable.  Village doctors have the potential to play a major role in increasing access for WRA to contraceptives in rural areas. Their current job descriptions, however, do not permit them to provide implants and IUDs so government regulations and medical training would need 19 to be changed for them to provide these services. (This is unlikely to happen in the short term). Even though there are barriers (mentioned elsewhere) to uptake of hormonal methods (OCP) and the popularity of IUDs among WRA, SUSTAIN says that it is their opinion that the focus should be on continuing access to and availability of free contraceptives (OCP and condoms) in rural areas, continuous medical education system for updating village doctors with FP knowledge and skills, and inclusion of FP services into UHC and Village Doctor’s program.  Behavior change communication campaigns were useful to some extent but the ET found it hard to measure their impact. SUSTAIN said they measured the coverage of their television campaigns based on the normal number of viewers assumed to be watching when the spots were aired but the ET did not find this compelling evidence that the campaigns were effective. On the other hand, presence of high-quality SUSTAIN materials was pervasive and, with Georgia’s high literacy rate, seemed to have had wide-spread reach. There were also some health providers and WRA who recalled hearing messages from recent Public Service Announcements.  Attendance at Parents Schools that were conducted at maternity hospitals turned out to be difficult since participation was voluntary and physicians were not paid for the time they devote to the schools. The project turned to electronic media and posted content online. This seemed to be a more effective approach because visitors to online courses can take modules at their own pace and select those topics that are of interest to them. This approach may not be as effective when compared to facility-based programs since facilitation is not provided and site visitors are not monitored to see if they have received all of the information that they need to be prepared for childbirth.  Effectiveness of the RH hotline services could not be determined, though it is possible that they were also a useful source of information for general public. This was a carry-over activity from another project. KIIs with some health professionals determined that the service was valuable to WRA and youth. On the other hand there was insufficient evidence of significant impact. 4.2.3 Recommendations USAID should support the country to conduct the GRHS survey (tentatively scheduled for 2015 after the results from a national census are available) to find evidence of positive outcomes on a population basis. Discussions about regionalization should include specific attention to means for overcoming the remaining barriers/challenges to achieving desired FP outcomes. This will probably include additional capacity building in contraceptive counseling, especially related to safety and side effects. MoLHSA should recognize both enhancing and inhibiting factors towards scaling up and continuing SUSTAIN's contributions to improving FP (including religion, remaining provider prejudice, competition between facilities and financial incentives) when formulating new health policies and reimbursement protocols (see Question 5). Village doctors across the country should have their job descriptions updated to include the FP services, if they are already providing them. To increase access to a broader method mix for 20 WRA, village doctors should be considered for expansion to provide services in additional methods, such as IUD and Implanon insertion, which they are currently not allowed to provide. Barriers to this expansion are related to the specialization system for doctors in Georgia. Even though additional FP methods were included in their FM training, there is a perception that they lack sufficient knowledge and skills to perform these services safely. FP services should also be available at the non-specialized PHC level such as rural ambulatories. If FP service provision by the health providers trained by SUSTAIN were formalized in the national health system, it would also be more likely that RH practice updates would be included in continuous medical education as the national government develops policies in that area. SUSTAIN should undertake a more detailed assessment of the Parents Schools and its web site to determine if it should be further developed and promoted by the government and professional associations. If Parents Schools are determined to be valuable, the advantages of the model should be documented and shared with the health facilities and professional networks and become a formal part of perinatal service provision in the country. Parents Schools are likely to be more effective if they were able to address the time constraints of clients and health care providers and be organized in rural areas. In this case, they could be facilitated by village doctors. Community health volunteers could be engaged for promoting the non-clinical aspects such as healthy nutrition, keeping appointments, avoiding smoking and alcohol, etc.). They can be trained in the curriculum and can facilitate Parents School classes26 . Similar types of volunteers have been effective in promoting MNCH practices in other low and middle-income countries.27 Strengthening community-level approaches goes beyond the SUSTAIN’s scope of work, but in the future it could support reducing the barrier of the need to travel into the cities only for health education purposes. If the online content is considered effective, it could be integrated into the rural-based schools where internet connections are available. Village-level facilitators can be trained when to refer specific clinical questions or problems related to pregnancy to OB/GYNs. 4.3 TO WHAT EXTENT HAS THE PROJECT CONTRIBUTED TO PRIVATE SECTOR-LED SERVICE DELIVERY DEVELOPMENT (IN PARTICULAR, AS IT RELATES TO THE PROVISION OF EVIDENCE BASED EPC) IN THE PRIVATELY OWNED AND OPERATED HEALTH CARE FACILITIES? 4.3.1 Findings All SUSTAIN target facilities remain privately-owned and operated, as are almost all health facilities in the country. They are owned by insurance companies, pharmaceutical companies, private companies, individuals, or groups of doctors. Health care facilities and their professional 26 At the time of the evaluation, UNICEF said there were discussions of nurses playing a larger role in community MNCH . But details about the strategy, including whether sufficient human resources or budget support would be provided, were not yet available. 27 See information about USAID’s NGO/PVO Child Survival and Health Grants Program that includes several community-based strategies in FP and MNCH: http://www.usaid.gov/what-we-do/global-health/child-survival￾and-health-grants-program 21 staff received capacity-building support in specific EPC components and built upon practices introduced under the HWG project. Private-sector owners, managers and providers are institutionalizing EPC in their facilities. Clinically-proven and cost-saving EPC is now being implemented in more than half (56/95) of Georgia’s private hospitals and SUSTAIN trained over 600 health care providers in EPC28 . Even though responsibility for national health strategy and payment changed during the life of the project, partners targeted for EPC capacity building stayed the same despite the fact that facilities sometimes changed ownership. KIIs with managers of these facilities provided the ET with specific examples of how SUSTAIN trained, monitored, and supported their facilities to implement and measure evidence-based EPC. Support was considered very high quality both in terms of training methods, job aids provided, and follow-up support after the training. Many SUSTAIN Master Trainers also head departments in facilities and some are officers in their professional organizations. Hence, the support for provider behavior change came from multiple sources and impacted multiple levels. KIIs across various types of health practitioner trainees and trainers at MNCH target sites, found that the overwhelming majority of respondents acknowledged SUSTAIN's "incredible" (their words) contribution to their improved attitudes and practices through the EPC training they received. SUSTAIN’s training that specifically focused on Pre-Eclampsia/Eclampsia (PE/E), AMSTL and management of PPH - and built on EPC training provided by HWG - reduced PPH, the major cause of maternal death in Georgia, from 1% in 2009 to only 0.6 % in 2014. Routine episiotomy has essentially ended and rates have decreased to 5-8% as of 2014 after EPC training, both under HWG and SUSTAIN29 . EPC training was not offered in comparison sites. One non￾participating maternity facility provides family centered delivery, AMSTL, and PPH interventions. However, the OB/GYN head of the hospital was trained by, and is an international Master Trainer in Promoting Essential Perinatal Care, or PEPC, the WHO program that pre-dated EPC but included most of the same components. EPC, as an intervention package, has replaced PEPC for the most part because it is perceived to be more “user friendly.”30 Data for the Tbilisi maternity facility headed by the above OB/GYN showed rates of PPH have also decreased; however, the time frames measured did not make a comparison with SUSTAIN data possible31 . According to SUSTAIN, AMSTL training provided by the project was more “hands on” and “on￾the-job” than that provided under HWG32. Although rates of PPH in HWG-participating facilities, (many of those facilities continued as SUSTAIN facilities) dropped significantly prior to 28 USAID Georgia, SUSTAIN project extension SOW, 2014. 29 SUSTAIN Monthly Report #11, August 2014. 30 KII interview with SUSTAIN management staff, October 2014. 31 Site visit and KII with Maternity #1, Tbilisi Georgia, October 2014. 32 KII with SUSTAIN CoP, October 2014. 22 2009, further decreases in PPH continued during the timeframe of the SUSTAIN project and included data from many facilities that did not participate in the HWG project (see Figure 3). Figure 3: Post-Partum Hemorrhage Rate in Relation to AMTSL Coverage. August, 201433 0% 0% 0% 82% 96% 98% 99% 98% 98% 98% 9% 9% 98% 8% 4% 2% 2% 1% 0.8% 0.9% 1.% 0.8% 0.6% -4% -2% 0% 2% 4% 6% 8% 10% 0% 20% 40% 60% 80% 100% 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 AMTSL SUSTAIN The use of the partograph, while introduced in HWG, is now "standard" practice in deliveries in SUSTAIN-supported facilities and practitioners testified as to how important it has been as a tool for monitoring the progress of labor and improving birth outcomes. In the past, they had nothing comparable to use to monitor the progress of women in labor. Another major change is that neonatal resuscitation training is now available from neonatologists. Coupled with the skin to skin contact, rooming-in, early initiation of breastfeeding and reduced use of general anesthesia, neonatologists say the condition of newborns is significantly better than in the past. WRA in FGDs in all areas cited the quality of maternity care has noticeably improved since 2009 though they were not aware that SUSTAIN was responsible for the changes. In FGDs conducted by the ET, women gave examples of better interpersonal communication and education from providers, cleaner and more hygienic facilities, comfort measures to reduce pain, the ability of family members to be present (even if they did not avail themselves of this service) as noticeably better than prior to 2009. Practitioners cite evidence of the effectiveness based on the positive clinical outcomes they see in mothers and babies as well as decreases in the number and severity of perinatal complications. Although both HWG and SUSTAIN discouraged unnecessary CSs, rates that dropped somewhat in HWG, did not come down during the implementation of SUSTAIN. CS rates remain high even in SUSTAIN supported facilities (33% before and 35.7% after EPC interventions against the target of <25%34, see Figure 5), despite SUSTAINS’s efforts to reduce it. Causes appear to be multi-factorial and many of the contributing factors, including some mixed opinions about the importance, were outside of project’s control. Until the recent changes in 33 Ibid. 34 Data from 38 target maternity hospitals, August 2014. 23 GoG policies, women had a “right” to choose the method of delivery, including CS. OB/GYN department heads of two maternity facilities in Samegrelo interviewed by the ET said that women will "shop" for a facility that will give them a CS, even if they are not clinically indicated and there is some pressure from hospital administrators not to turn away clients that want CS. These KIs questioned some of the clinical “justifications” for CS provided by some other providers. According to the SUSTAIN Chief of Party, vaginal birth after CS is not widely supported by OB/GYNs35 . Figure 5. Key interventions during delivery: C-section rates before and after EPC interventions. Data from 38 target maternity hospitals. August, 201436 OB/GYNs interviewed by the ET said that women often “demand” CS because they “want to avoid the pain.” It is not clear if they believe the procedure is safer for the mother and/or the baby. One high-performing SUSTAIN OB/GYN department head in Samegrelo told the ET that during HWG, she had succeeded in reducing CS to the target of 25%, but her facility is now owned by an insurance company and they had already laid off 5 of the 15 OB/GYN staff. The director of her hospital told her that if they did not offer CS, the women would “go to another hospital anyway” and that would mean a loss of revenue for the hospital.There is a higher cost for CS delivery as compared to vaginal delivery. But whether this amount is significant enough to be a deterrent for the patient, or provides financial incentives for providers to continue to conduct CS, is unclear since the CS rate remains high in spite of the additional cost. 4.3.2 Conclusions SUSTAIN’s approach to scaling up EPC for improvement of perinatal care quality in 56 target privately-owned and operated health facilities, representing over 50% of the facilities in the country, proved to be effective and can be replicated to other non-target MCH facilities. 35 Interview with SUSTAIN management staff, October 2014. 36 SUSTAIN Monthly Report, August 2014. 25% Target <25% 24 SUSTAIN’s capacity-building interventions did not meet the targets to reduce rates of unnecessary CS. Provider reimbursement incentives, and mixed impressions in terms of what "the evidence says" about when CS is appropriate on a clinical basis, as well as client demand, all appear to contribute to apparent ambivalence about whether reducing CS is really important. 4.3.3 Recommendations SUSTAIN should document and disseminate the process of how professional health technical assistance based on evidence-based best practices improved the quality of perinatal care in privately-owned health facilities in Georgia. MoLHSA, should include the quality improvement activities that entail continuous education, supportive supervision, and total quality management modeled by SUSTAIN within the standards they are developing for continuous quality improvement in the regionalization of EPC. Focused discussions on implementing policies aimed at reducing unnecessary CS should be included as an explicit component of scaling up EPC within the regionalization, certification, and accreditation process. The MoLHSA does not reimburse for CS without clinical indications. Quality assurance monitoring should include processes that determine whether clinical indications for CS (or any clinical procedure) were present. Causes of unnecessary CS appear to be related to many factors. To address the non-clinical factors contributing to CS, including client perceptions and demand and conflicting factors motivating provider behavior, will require a strong commitment at the national level and a complex approach by the GoG in partnership with the professional associations and private-sector implementers. In the longer term, women's fear of pain in childbirth that was addressed by the EPC approach as well as Parents Schools, was in the right direction; however, there is still more to be done in client, family and provider education, as well as counseling and services that are provided to maternity clients and their families to encourage them to demand the minimum amount of medical intervention necessary to achieve the best outcomes for both mother and baby. Approaches should explicitly address women’s fear of pain in childbirth as well as provider and health facility management resistance to reducing CS rates. To address unnecessary CS, Georgian professionals will also need to remain engaged in the wider global OB/GYN community, such as through their relationships with the American Colleges of Obstetricians and Gynecologists (ACOG) and medical training institutions (such as was done with UCSF and Harvard) to identify how to integrate best practices into the new standards for certification and accreditation, currently under development by the MoLHSA. Financial incentives and time constraints that encourage providers to do unnecessary procedures should be closely examined. While controversial even among SUSTAIN OB/GYN experts, vaginal birth after CS should be studied as a possibility for some women where it can be safely conducted. 25 4.4 ARE THE KEY STAKEHOLDERS SATISFIED WITH THE PROJECT (WITH THE SPECIAL EMPHASIS ON MEDICAL SCHOOLS WITH REGARDS TO CURRICULUM AND EDUCATION PRACTICES AT MEDICAL SCHOOLS)? 4.4.1 Findings The ET conducted KIIs with the Dean and two faculty members of the Tbilisi State Medical University (TSMU), including the Chair of the OB/GYN Department and Head of the Clinical Skills Center of TSMU and also five fifth-year medical students that took (and passed) the OSCE exam in OB/GYN in 2014. Medical faculty, students, and professional association representatives were overwhelmingly satisfied not only with the curriculum changes but also with the capacity-building in medical student instructional methods. They are already “scaling-up” these improved teaching methods to classes in other medical disciplines in the medical school. SUSTAIN also helped develop linkages between the OB/GYN faculty and professional associations and the global MNCH professional communities such as American College of Obstetricians and Gynecologists (ACOG) and medical research and teaching institutions (UCSF, Harvard, etc). This was highly valued by medical professors and OB/GYN association officers who said that they found the improved practices they saw during the study tours they were given in US medical training and research institutions very motivating. SUSTAIN’s capacity building approach towards improving pre-service OB/GYN training was also judged to be "very high quality," especially “hands on” practice using models, including the maternity “simulator.” The medical students specifically cited very positive experiences with preparing for, and taking the OSCE examination, which is recognized internationally. Students said that they found the preparation for the examination to be “very practical.” According to them, this type of approach should be the basis for assessment in other medical disciplines studied at the medical school. The Dean of TSMU said the school has seen the value of such approach and has already planned to offer the OSCE exam in other disciplines. SUSTAIN has recently initiated assistance activities to Kutaisi Akaki Tsereteli University’s medical school. The results were not yet visible but the ET was assured that the same quality of assistance - in terms of pre-service curriculum development and provision of simulators - will be provided to Kutaisi Medical School as it was provided to TSMU. Other stakeholders’ opinions have been addressed in answers to other questions, but all stakeholders that were knowledgeable about the SUSTAIN project, including government, providers, and representatives of facilities owned and managed by private insurance companies, said they were extremely satisfied with the project. 4.4.2 Conclusions The Dean and OB/GYN professors at the TSMU said, "SUSTAIN provided technical assistance for pre-service training that was 'revolutionary' as it changed teaching methodologies, curriculum, and introduced the first ever OSCE in post-Soviet region." Linking health professionals in Georgia to global professionals, research, and training institutions and organizations, will enable them to apply new techniques and training methods to other MNCH disciplines and beyond. 26 4.4.3 Recommendations The ET recommends some additional technical assistance to TSMU to adapt the methodology learned through SUSTAIN to other medical disciplines, especially in areas of maternal and child health. To the extent possible, MoLHSA and the Ministry of Education and Science (MES), who respectively hold responsibility for in-service medical education and for pre-service education, should continue to link pre-service and in-service education with global best practices. Regionalization, accreditation, and certification policy implementation over the next few years should be developed in ways that reinforce the positive changes that have taken place in the medical and public health education sector. 4.5 QUESTION 5: HOW SUSTAINABLE ARE PROJECT ACTIVITIES? 4.5.1 Findings According to multiple respondents at many levels, SUSTAIN’s evidence-based approaches, introduced in HWG and extended and institutionalized in the current project, have permanently changed the approach to MNCH and FP. OB/GYNs enthusiastically volunteered that “We will never go back!” (to the previous way of doing things). Improved practices (AMSTL, PPH management, neonatal resuscitation) as well as EPC will continue because “we see the results.” Pre-service curricula and teaching methodologies will continue and will be applied to other disciplines. Standards of care will be incorporated into the regionalization, accreditation, and certification measures as part of improvement in health care quality. The new World Bank and European Union loan to Georgia for improved governance requires some of these changes as, according to MoLHSA officials, health care quality management is a precondition for loan approval. SUSTAIN's MNCH activities focused largely on the perinatal period. Some informants said that although not a focus of the SUSTAIN project, a similar approach with more focus on the ante￾natal period and post-neonatal period, including stronger nutrition counseling, would be appropriate to implement by the government and other donors working in MNCH programs. At the same time, continued attention is needed to expand and extend the improvements attained in the SUSTAIN project. New regulations (prescription requirements) for contraceptives will limit the sustainable impact of social marketing on contraceptive supplies. There is mixed feedback on the willingness of OB/GYNs to offer counseling on FP but village doctors are willing to counsel and provide services. However, this will need to be included in their job description to be sustained. Providers say uptake of new FP methods, especially Implanon, is increasing but will be too expensive at market prices (over $100) for use to increase. Subsidies, UHC coverage, or donated supplies will continue to be needed for use to maintain or increase current levels. Demand for implants is slowly increasing, largely due to “word of mouth. 37 ” SUSTAIN has 37 KII with OB/GYN FP service providers and trainers, October 2014. 27 ordered contraceptives sufficient for several years but affordable supply for low-income women after that is still uncertain. Contraceptives’ distribution hand over has been discussed with the National Center for Disease Control and Public Health and market forces are likely to be a factor within pharmacies. However there are remaining barriers to increased FP uptake. Among these are religious objections regarding any method of contraception and misperceptions remain about contraceptives among the general population, especially related to hormonal methods, about safety and dealing with side effects. This indicates a need for more public and community education about FP and contraception as well as improved counseling skills among health providers. Some commercial channels (Achara TV, Tabula) intend to broadcast public service announcements with FP and MNCH messages that were designed during the project communication campaigns. They will do this free of charge, as part of their social responsibility requirements. Although specific activities and approaches had to be adjusted during the lifetime of the project, the program approach based on the evidence-base and working through key partners is sound and will be part of future public and private health programs in Georgia. SUSTAIN's efforts focused on the extremely important priority areas of the perinatal period as well as preventing unwanted pregnancy. However, other areas on the Continuum of Care still require attention by MoLHSA, stakeholders and partners. 4.5.2 Conclusions The broad support to the RH and MNCH professional community at multiple levels provided by SUSTAIN guarantees that evidence-based MNCH and FP practices will continue without additional donor support. New training practices that were demonstrated through SUSTAIN's programs will be applied to other disciplines including pediatrics. With the exception of the direct supplies of contraceptives for free distribution provided from USAID through the project, SUSTAIN’s contributions to FP will be sustained and perhaps scaled-up to other areas. In the short term, SUSTAIN has ordered additional contraceptives to be distributed through the Centers for Disease Control and Public Health (CDCPH) to allow supplies to last as long as possible. Beyond facilities, additional policies and programs will be needed in Georgia to engage other MNCH practitioners (village doctors, pediatricians) to keep momentum going beyond perinatal care and FP to meet remaining needs in MNCH. After completion of SUSTAIN’s support, it will be challenging to update village doctors’ skills in RH/FP counseling and provide them with contraceptives. Without outside support, these activities will not continue. The role of village doctors in FP counseling and provision of contraceptives needs to be clarified in their job descriptions in order for them to be accountable for these activities. In the longer term, village doctors could be in a good position to increase access to long term FP methods if they were trained in IUD and implant insertion and, either they, or their clients, had access to supplies and equipment. Due to new prescription requirements and barriers to providing contraceptive counseling, additional training of pharmacists would not result in sustainable movement towards national RH and MNCH objectives. 28 4.5.3 Recommendations Extending SUSTAIN’s technical support until the regionalization process is completed will support sustainability of the capacity building activities. If USAID is able to identify additional sources of funding, a second year should be considered (see Question 1). Although SUSTAIN’s interventions focused on the perinatal period, the Promoting Maternal, Newborn and Child Health (PMNCH) Continuum of Care platform is the key to sustainable impact in MNCH. MoLHSA (with support from SUSTAIN during the extension) and other partners, should advocate with GoG to include policies that support strengthening quality services across the Continuum of Care spectrum. This would include institutionalizing FP as an essential component of quality MNCH interventions, and include strengthening linkages between perinatal and child health services and improving antenatal care services. (Supporting the entire Continuum of Care was beyond the scope of the SUSTAIN project). National discussions of quality MNCH services should be expanded beyond the perinatal referral network to include the roles (including job description changes) of village doctors and RH providers. Institutionalizing FP as part of MNCH must include commodity procurement support (including UHC policies) that will make contraceptives available and affordable to all WRA that want to use them. MoLHSA should consider training and equipping village doctors to do implant and IUD insertion to increase access to long-term and reversible FP methods in their long term strategies. Although additional pharmacists’ training is not recommended, SUSTAIN and MoLHSA should communicate with the pharmaceutical networks to find out what is likely to happen in terms of the availability of the low cost OCPs after the SUSTAIN project and USAID support has ended. Figure 6 below is an adaptation of WHO (2005) showing the Continuum of Care needed to reduce deaths and improve outcomes in MNCH. Figure 6.Adaptation of WHO (2005) Continuum of Care Interventions that were the focus of the SUSTAIN project targeted the perinatal period (just before and just after birth). This was an essential focus area as MNCH could not improve without changes during the perinatal period. Although FP is not specifically listed in the continuum, evidence has shown that FP contributes to positive outcomes at multiple points. 29 However, to sustain the impact of perinatal and FP improvements, other areas of the Continuum of Care will need attention. For example, UNICEF has wanted to strengthen child nutrition and health but has had difficulty attracting partners to advocate with the government to strengthen child health policies. As already mentioned, this is beyond the scope of the SUSTAIN project, but moving forward, the MoLHSA should strengthen programs along the continuum to sustain the positive outcomes that were achieved by SUSTAIN’s interventions. 5.0 LESSONS LEARNED Health care reforms within context of evidence-based services is highly successful in the Georgian setting and reinforced by positive clinical outcomes in both public and private health programs. Changing provider attitudes and involving them, such as through professional associations, are important steps in this process. Close professional relationships between SUSTAIN, government and private sector partners kept everyone “on the same page” and working toward the same goals (good MNCH outcomes). The inevitable positive outcomes that result from improved MNCH practices reinforce motivation for their continued practice. Pharmacies can sell contraceptives but social marketing requires counseling on FP methods and the pharmacy environment in Georgia does not provide sufficient confidentiality to do this well. It is also not well accepted by clients or pharmacists. Since there is a new prescription-only policy towards OCPs and the counseling role is now assumed to be limited to health providers, a social marketing approach for contraceptives outside of condoms is not likely to work. Engaging pharmacist networks, however, will continue to be a good idea in order to advocate for them to include affordable contraceptives in their stocks. Linkages with global medical academic and training institutions and professional organizations reinforce capacity building by projects like SUSTAIN. If these relationships can be maintained and strengthened, this bodes well for sustainable adoption of new approaches in MNCH and FP in the future 30 ANNEXES 31 ANNEX 1: EVALUATION STATEMENT OF WORK 32 SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK STATEMENT OF WORK Sustaining Family Planning and Maternal and Child Health Services Project (Sustain) Performance Evaluation I. Scope Summary Name of the Project (to be evaluated): Sustaining family Planning and Maternal and Child Health (Sustain) Project Project Number: GHS-I-00-0700002 Project Dates: 10.01.09 – 09.30.14 Project Funding: $10, 073,480 Implementing organization/s: John Snow Incorporated COR/AOR: Tamara Sirbiladze Summary of Specific Technical Requirements The contractor shall provide the following deliverables: 1. Draft evaluation design 2. In-brief with USAID management to present the detailed evaluation design and work plan 3. Conduct evaluation of Sustain project in accordance with the USAID-approved evaluation design 4. Provide draft and final evaluation reports to USAID in accordance with the reporting guidelines 5. Out-brief with USAID management to present initial findings, conclusions and recommendations The contractor shall: • review the results achieved by the project to date, • determine the extent to which the project’s activities have contributed to these results, • examine the project effects on advancement of Family Planning (FP) and Maternal and Child Health (MCH) services throughout Georgia. • identify significant changes in the target sites that may have resulted from the USG funded intervention. II. Background The USAID-funded Sustaining Family Planning and Maternal and Child Health Services Project (Sustain) in Georgia started in October 2009 and was scheduled to run for five years through September 2014. The project was designed to meet critical MCH/FP needs in Georgia while laying the foundation for long-term, sustainable family planning and maternal and neonatal health programs in the private sector, including private insurance industry plans, existing health clinics and planning health training units, while assuring continuous access to a broader range of contraceptive supplies in the private sector for the most vulnerable populations. 33 The project is focused on improving and sustaining the gains made in Georgia’s maternal, newborn, child, and reproductive health services. The project works in public and private partnership with health care providers, insurers, and policy makers to improve the quality of care in each of these areas. USAID/Sustain works toward the following four objectives: Objective 1: Promote quality maternal, newborn, and child health; reproductive health; and family planning products and services. The project uses health behavior change principles to conduct communication campaigns that encourage Georgian families to seek quality maternal, newborn, child, and reproductive health services. The campaigns reach the Georgian public through a combination of television, radio, print, and web-based media (including social media). The project also places information, education, and communication (IEC) materials at different points of service (e.g. antenatal clinics, family planning cabinets, primary health care centers) to reach specially-targeted audiences with more in-depth information. Objective 2: Ensure access to contraceptives by building strong commercial sector distribution of contraceptives and facilitating public/private partnerships with pharmacies and pharmaceutical companies. Sustain uses social marketing to launch high quality and affordable contraceptives on the commercial market in partnership with pharmacies and pharmaceutical companies. To date, the project has successfully launched three contraceptive products – two brands of oral contraceptive pills and one contraceptive implant (Implanon). More than 500 OB/GYNS have been trained to provide Implanon insertion, removal, and counseling services. The project also distributes some free contraceptives (condoms, oral contraceptives, IUDs, etc.) to areas where the population cannot afford to pay market prices. Objective 3: Strengthen the quality of maternal, newborn, and child health, reproductive health; and family planning services. The project’s focus on quality is important. Georgia implemented universal health coverage for all citizens in 2013, and now there is a renewed focus on ensuring that health services are high quality. Toward this end, the project works with the Ministry of Labor, Health and Social Affairs (MoLHSA) to develop health facility accreditation and certification programs for perinatal care and reproductive health care, respectively. As critical first steps, the project has introduced facility-based quality improvement initiatives, including spearheading a clinically-proven and cost-saving effective perinatal care program, which is now being implemented in more than half of Georgia’s maternity clinics. To date, Sustain has trained more than 500 health care providers in effective perinatal care. The project has helped standardize maternity care across Georgia by developing ten obstetric and neonatal clinical protocols, which have been approved and put into practice by the MoLHSA. 34 The project is also developing, again with the MoLHSA, a perinatal care regionalization program, which would introduce the practice of a tiered health facility system throughout the country. Pregnant women and newborns would receive birthing services and care at better equipped hospitals, with better staff capacity, and better transport systems. To improve the quality of family planning and reproductive health services, the project has trained more than 1,800 health care providers and pharmacists. The training is part of a larger technical assistance package that also includes system strengthening and health promotion. As a result, eight hundred primary health care facilities now provide comprehensive reproductive health services. The project also works to strengthen post-abortion family planning counseling and immediate method provision – an important initiative given Georgia’s relatively high abortion rate. The project has also developed an innovative parent education curriculum, which currently is offered both online (www.mshobeltaskola.ge) and in five maternity clinics across Georgia. Objective 4: Improve pre-service medical education. For decades medical education in Georgia was focused heavily on knowledge, while clinical and communication skills were considered less important. The lack of clinical and patient communication practice for students presented challenges when they transitioned to become medical practitioners. Sustain provides technical assistance to four medical universities across Georgia to introduce clinical skills practice. To date, the project has trained 50 medical faculties on curriculum development, advanced methods of teaching, and student evaluation techniques including the internationally renowned objective structured clinical examination (OSCE). Georgia’s first OSCE was successfully held for more than 250 fourth-year medical students at Tbilisi State Medical University in February 2014. III. Purpose of the Evaluation and Its Intended Use The purpose of this evaluation is to examine the project effects on advancement of FP and MCH services in target facilities throughout Georgia and determine how likely it is to sustain the quality services in these facilities after the end of the project. While this will not be an impact evaluation, it must be able to identify significant changes in the target sites that may have resulted from (are highly attributable to) the USG-funded interventions. This evaluation must provide the independent view of how the project has been implemented and what results have been achieved in the target regions. The contractor must review actual progress toward achieving key expected results and identify accomplishments, delays, external and internal challenges, and their impact on the project, and the project’s flexibility and effectiveness in adjusting to the changes in the political environment. The timeframe to be covered by the evaluation is from the start of the project in October 2009 through the time of initiating this evaluation. The results of the evaluation will be used to summarize the results of this major MHC and FP activity in Georgia, derive the prospects for sustainability and share the information within USAID’s health community across the region and beyond. The results of the study will also be 35 shared with local stakeholders (Ministry of Labor, Health, and Social Affairs, the Donor Coordination Unit under the Chancellery of the Government of Georgia under the Prime Minister of Georgia, National Center for Disease Control and Prevention, National Maternal and Child Health Council) other donors working in this area, including UNICEF and UNFPA, and the interested NGOs. Finally, evaluation results will also be used for reporting purposes to Washington-based stakeholders. IV. Evaluation Questions and Methodology The evaluation report must provide evidence-based answers. The contractor must answer the following questions in the evaluation: 1. How flexible and effective38 was the project in adjusting to the radical changes in the political environment, and did those changes affect the key outcomes? 2. What were significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted from the project interventions? 3. To what extent has the project contributed to private sector-led service delivery development (in particular, as it relates to the provision of evidence-based Effective Perinatal Services in the privately owned and operated health care facilities)? 4. Are the key stakeholders satisfied with the project (with the special emphasis on medical schools with regards to curriculum and education practices at medical schools)? 5. How sustainable39 are project activities? The finalized evaluation design shall be submitted to the TOCOR three workdays prior to the team’s arrival in-country. The evaluation design must outline in detail what methods the contractor will use to get answers for each evaluation question. The evaluation design shall include a detailed evaluation matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, a work plan, and a dissemination plan. This information together with the Mission’s comments will be discussed in detail during the in-brief meeting with USAID. The work plan shall include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. The following represents the illustrative evaluation matrix: 38 Flexible is defined as being able to adjust to environment and effective relates to accomplishing its purpose. 39 Sustainable is defined as project sites (or participant institutions) being capable of continuing services and maintaining quality of services. 36 Research Question Data Source Methodology 1. How flexible and effective40 was the project in adjusting to the radical changes in the political environment, and did those changes affect the key outcomes? Project personnel MoLHSA Key informant interviews 2. What were significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted specifically from the project interventions? Target institutions Personnel documentation Comparison of two groups of facilities – with and without USAID assistance41 Document review 3. To what extent has the project contributed to private sector-led service delivery development (in particular, as it relates to the provision of evidence-based Effective Perinatal Services in the privately owned and operated health care facilities)? MoLHSA, Private Networks in ownership of health care facilities individual pilot hospitals, primary health care clinics, pharmacy chains Key informant interviews, Document review, Focus groups with health personnel, Private network representatives, Ministry representatives. 4. Are the key stakeholders satisfied with the project (with the special emphasis on medical schools with regards to curriculum and education practices at medical schools)? Medical schools, professional associations, insurance companies, Hospital networks and pharmaceutical companies. Mini survey, Interviews 5. How sustainable are project activities? Documents or Management of health facilities - hospitals, primary health care clinics, pharmacies? Interviews, document review V. Evaluation Team The evaluation shall be conducted by a team composed by international and local experts. All evaluation team members must be familiar with USAID’s January 2011 Evaluation Policy42 . 40 Flexible is defined as being able to adjust to environment and effective relates to accomplishing its purpose. 41 The project works in 56 out of 95 facilities throughout Georgia, so the comparison groups can be selected to compare them with those facilities to which the project has provided assistance. It is expected that 20 facilities will be selected in each group with a standard deviation of 8% 42 http://www.usaid.gov/evaluations/USAIDEvaluationPolicy.pdf 37 All team members are required to provide to USAID a signed statement attesting to a lack of conflict of interest in relation to the Sustain project being evaluated. VI. Projects Documents for Review and Logistics The COR/AOR will put the contractor in contact with its implementing partner and might provide help with a small number of meetings (such as meeting with USG agencies where needed). To the extent possible, relevant reports and other project documentation will be provided by the Mission to the contractor prior to travel to Georgia. These documents are: • Project Description as is stated in the award; • Review of relevant SUSTAIN documents, including: Project Monthly and Annual Reports, Work Plan (PY1-3 and 4), Internal Mid-Term Evaluation Report; and Communications (including website, BCC spots, and pamphlets); • Initial list of in-country contacts; • PMP indicator tables; • M&E plans submitted and approved by USAID; • Implemented monitoring reports; • Other deliverables (expert report, publications) produced by partner. 38 ANNEX 2: EVALUATION MATRIX 39 Research Questions Sub Questions Data Source Methodology 1. How flexible and effective was the project in adjusting to the radical changes in the political environment and did those changes affect the key outcomes?  What changes were there in the political environment that impacted the project?  At what point during the duration of the project did these changes occur?  To what extent was the project able to adjust to radical changes in the political environment? Did those changes affect the key outcomes  How effective (as in accomplishing the purpose of the SUSTAIN project) was the project in adjusting to the radical changes in the political environment? To what extent did they affect project outcomes?  To what extent will these adaptations affect sustainability of project interventions? Project Personnel Project Documents (e.g. work plans, annual reports, M&E plans) MoLHSA interviews Project Monitoring Plans or similar documents MoLHSA interviews USAID and project briefings  Key Informant Interviews  Project data reviews  Timeline of project activities  Comparison of project activities with timeframe of political changes that impacted the project  Record reviews  Project data review  Stakeholder interviews 2. What were significant positive changes in the target sites, evidenced by the improved health outcomes that may have resulted from the project interventions?  What were the intended changes at the target sites?  What were the outcome indicators selected by the project to measure improved health outcomes?  What were the measurement methods used?  Did these measurements show improvements?  To what extent can they be attributed to the SUSTAIN project? What were the lessons learned about what worked and did not work in improving health outcomes? To what extent will the positive changes achieved be sustained after the project has ended? Target institutions personnel Facility records (registers, reports, etc) HMIS (if applicable) Project Monitoring Plan with indicators Monitoring and evaluation activities of the project Secondary data sources and surveys including GRHS, QI studies, etc.  Comparison of a sample of project target sites compared with non-project facilities  Health Facility Record review  KII and FGDs with health personnel  Interviews with health professionals familiar with health system changes before and after the 2012 change in government  Record reviews 40 3. To what extent has the project contributed to private sector-led service delivery development (in particular, as it relates to the provision of evidence based Effective Perinatal Services in the privately owned and operated health care facilities?  What were the selection criteria for privately￾owned and operated health care facilities that received capacity building, especially in Effective Perinatal Care (EPC)?  How did the project measure improvements in health services as a result of project activities?  Were all components of EPC included in the capacity building provided by the project?  How satisfied are clients in privately owned and operated facilities that received capacity building?  Were the privately owned and operated health care facilities able to implement EPC? To what extent will the privately owned and operated facilities be able to sustain EPC services after the project has ended?  Would they recommend that other private health facilities adopt EPC in their services?  To what extent can additional capacity building be extended to additional facilities after the project ends? Project Documents; MoLHSA, Private Networks that own health care facilities, individual hospitals, primary health care clinics, pharmacy chains, Project capacity building plans and measures; project M&E plans  Hospital record reviews  Health facility assessments (site visits with check lists)  KIIs and FGDs with health workers and health facility managers  KIIS with private network representatives, ministry representatives 41 4. Are the key stakeholders satisfied with the project (with special emphasis on medical schools with regards to curriculum and education practices at medical schools)  Which medical schools received capacity building from SUSTAIN?  What were the topics/skills where SUSTAIN provided training or upgraded facilities?  Which categories of health workers were targeted for improved education practices?  Which protocols were changed as result of project assistance?  Have improved curricula and education practices been institutionalized in all medical schools that received assistance?  How effective have the improved educational practices and curricula been in implementation improved of evidence-based medical practices in those who have received training?  Were students able to implement what they have learned (e.g. did they have the supplies? Medical schools, professional associations, insurance companies, hospital networks, and pharmaceutical companies. MoLHSA, Medical School faculty, document review, policy reviews  Key informant interviews  Site visits  Mini-surveys (at least 2) one for physicians trained in implants; one from a sample of nurses.  Stakeholder (including students as available) interviews 42 ANNEX 3: SUMMARY OF MINI-SURVEY’S RESULTS 43 I. Mini Survey Methodology The quantitative component of this evaluation consisted of mini-survey of health workers that were conducted at 23 participating and 15 non-participating health facilities in Tbilisi, Imereti and Samegrelo-Zemo Svaneti regions. Regions, districts within regions (Samtredia and Tskhaltubo in Imereti and Abasha and Senaki in Samegrelo-Zemo Svaneti) as well as health facilities were selected randomly, using the respective data provided by the SUSTAIN project, supplemented with information on health facilities available on Georgian government websites. Interviewers also collected information about facilities available at the corresponding districts. Specifically, within each selected district 23 health facilities, whose staff took part in the SUSTAIN-sponsored training seminars, were chosen as research sites for mini-survey. There, three to five medical professionals-participants of the training seminars were sampled using the list of names provided by the project. Although in some cases these lists turned out to be outdated so that some respondents could not be located. Matching characteristics of the participating facilities, 15 non-participating medical institutions were selected in the respective districts where another 63 health professionals were sampled. Their qualifications were similar to those of the SUSTAIN’s trainees. Overall, 140 respondents were targeted for interview, 127 interviews were completed successfully, with a response rate set at 91%. Sampling design ensures that the mini-survey’s results are representative of views of medical professionals in the respective regions falling within 5% of the respective population parameter at the 95% confidence level. No significant differences between participating and non-participating facilities in terms of methods of contraception having been introduced since 2010 or perceived barriers to introduction of new services or products were detected. But SUSTAIN facilities were more than three times as likely to provide implant services for family planning as non-SUSTAIN facilities. 1. Respondent was interviewed in treatment or comparison facility Frequency Percent Valid Percent Cumulative Percent Valid 0 Comparison 63 49.6 49.6 49.6 1 Treatment Facility 64 50.4 50.4 100.0 Total 127 100.0 100.0 2. Region where interview was conducted Frequency Percent Valid Percent Cumulative Percent Valid 1 Imereti 20 15.7 15.7 15.7 2 Samegrelo 33 26.0 26.0 41.7 3 Tbilisi 74 58.3 58.3 100.0 Total 127 100.0 100.0 3. District where interview was conducted Frequency Percent Valid Percent Cumulative Percent 44 Valid 1 Kutaisi 15 11.8 11.8 11.8 2 Samtredia 3 2.4 2.4 14.2 3 Senaki 6 4.7 4.7 18.9 4 Tbilisi 74 58.3 58.3 77.2 5 Tsalenjikha 5 3.9 3.9 81.1 6 tskaltubo 2 1.6 1.6 82.7 7 Khobi 3 2.4 2.4 85.0 8 Zugdidi 19 15.0 15.0 100.0 Total 127 100.0 100.0 4. Number of participating and non-participating facilities covered within each district teatment_bin 0 Comparison 1 Treatment Facility district_rcd district 1 Kutaisi 0 15 2 Samtredia 3 0 3 Senaki 1 5 4 Tbilisi 31 43 5 Tsalenjikha 2 3 6 Tskaltubo 2 0 7 Khobi 0 3 8 Zugdidi 12 7 5. Respondent’s gender Frequency Percent Valid Percent Cumulative Percent Valid 1 Male 10 7.9 7.9 7.9 2 Female 117 92.1 92.1 100.0 Total 127 100.0 100.0 6. Age of respondent Frequency Percent Valid Percent Cumulative Percent Valid 1 1 20-29 4 3.1 3.1 3.1 2 2 30- 39 17 13.4 13.4 16.5 45 3 3 40-49 50 39.4 39.4 55.9 4 4 50-59 42 33.1 33.1 89.0 5 5 60+ 14 11.0 11.0 100.0 Total 127 100.0 100.0 7. Educational level of respondent Frequency Percent Valid Percent Cumulative Percent Valid 1 College 22 17.3 17.3 17.3 2 Higher medical education 105 82.7 82.7 100.0 Total 127 100.0 100.0 8. Respondent’s position Frequency Percent Valid Percent Cumulative Percent Valid 1 Doctor of obstetric services (Ob/Gyn) 76 59.8 60.8 60.8 2 Doctor of RH/FP services (Ob/Gyn) 9 7.1 7.2 68.0 3 Doctor of obstetric and RH/FP services (Ob/Gyn) 8 6.3 6.4 74.4 4 Doctor of Neonatal care services (Pediatrics) 10 7.9 8.0 82.4 5 Doctor of family medicine center/polyclinic (Family medicine or internal medicine or pediatrics) 1 .8 .8 83.2 6 Nurse of obstetric care services (General practice nursing and/or midwifery) 15 11.8 12.0 95.2 7 Nurse of neonatal care services (General practice nursing) 6 4.7 4.8 100.0 Total 125 98.4 100.0 Missing System 2 1.6 Total 127 100.0 46 9. What is your overall impression of these training programs and technical assistance provided by SUSTAIN (trainees only) Frequency Valid Percent Valid 1 Very satisfied 34 44.2 2 Satisfied 37 48.1 3 Neutral 5 6.5 4 Dissatisfied 1 1.3 Total 77 100.0 10.Do you consider these training programs and technical assistance helpful in improving your MCH and FP/RH skills? (trainees only) Frequency Valid Percent Valid 0 No 1 1.3 1 Yes 76 98.7 Total 77 100.0 11.What is your opinion about the quality of capacity building or health education technical assistance provided by SUSTAIN? (trainees only) Frequency Valid Percent Valid 1 Very satisfied 34 44.2 2 Satisfied 38 49.4 3 Neutral 4 5.2 4 Dissatisfied 1 1.3 Total 77 100.0 12. Have any new methods of contraception been introduced here since 2010? Frequency Percent Valid Percent Cumulative Percent Valid 0 No 49 38.6 38.6 38.6 1 Yes 78 61.4 61.4 100.0 Total 127 100.0 100.0 47 13. What type of MNCH/FP services are provided in your institution? Frequency Percent Valid Percent Cumulative Percent Maternal Care￾SUSTAIN 57 49.5 49.5 49.5 Maternal Care￾NON-SUSTAIN 58 50.5 50.5 100.0 Newborn Care￾SUSTAIN 51 56.6 56.6 56.6 Newborn Care￾NON- SUSTAIN 39 43.3 43.3 100.0 Reproductive Health/FP￾SUSTAIN 54 48.2 48.2 48.2 Reproductive Health/FP- NON￾SUSTAIN 58 51.8 51.8 100.0 Q14. What contraceptive methods do you provide here? Frequency Percent Valid Percent Cumulative Percent Condoms￾SUSTAIN 42 54.5 54.5 54.5 Condoms-NON￾SUSTAIN 35 45.5 45.5 100.0 IUDs-SUSTAIN 53 48.6 48.6 48.6 IUDs-NON￾SUSTAIN 56 51.4 51.4 100.0 OC-SUSTAIN 49 50.0 50.0 50.0 OC- NON￾SUSTAIN 49 50.0 50.0 100.0 Subdermal Implants￾SUSTAIN 31 77.5 77.5 77.5 Subdermal Implants-NON￾SUSTAIN 9 22.5 22.5 100.0 48 15. Have you received any FP or MCHN training since 2010? Frequency Percent Valid Percent Cumulative Percent Valid 0 No 37 29.1 29.1 29.1 1 Yes 90 70.9 70.9 100.0 Total 127 100.0 100.0 16. Were there any barriers to implementation of new services/techniques/products in your facility? Frequency Percent Valid Percent Cumulative Percent Valid 0 No 102 80.3 80.3 80.3 1 Yes 25 19.7 19.7 100.0 Total 127 100.0 100.0 49 ANNEX 4: LIST OF INTERVIEWEES 50 I. List of Interviewed Stakeholders within the scope of End of Program Evaluation of “Sustaining family Planning and Maternal and Child Health” Project 1. Tamara Sirbiladze, Director of Health and Social Development, USAID Georgia 2. Nino Berdzuli, Chief of Party, Sustain project/John Snow Institute 3. Tinatin Chikovani, Dean of Faculty of Medicine Tbilisi State Medical University (TSMU) 4. Nicholas Kintraia, Vice-President of Georgian Obstetrician/Gynecologists and Perinatologists Association, Head of OB/Gyn department of Chachava Clinic 5. Tamar Antelava, Head of Obstetrics Department of Evex Medical Corporation 6. Mariam Jashi, Head of Solidarity Fund of Government of Georgia 7. Vera Baziari, Chief Specialist, Ministry of Labour, Health and Social Affairs 8. Tamar Ugulava, Health Specialist, UNICEF Georgia 9. Lela Bakradze, Country Director, UNFPA Georgia 10. Lela Sturua, Head of non-Communicable Diseases Department, National Center of Disease Control and Public Health 11. Nani Marsagishvili, Family Planning National Trainer, Reproductologist of InVitro Clinic 12. Platon Machavariani, Effective Perinatal Care National Trainer, Head of Ob/Gyn department, Patriarchate of Georgia St. Ioakime and St. Ana Medical Center 13. Ketevan Nemsadze, Effective Perinatal Care Trainer, Head of Georgian Pediatric Academy 14. Irma Manjavidze, Head of Clinical Skills Center, Tbilisi State Medical University (TSMU) 15. Tengiz Asatiani, President of Georgian Obstetricians and Gynecologists, Head of Maternity Clinic #1. II. List of Interviewed Health Providers (participating and non-participating) 16. Tinatin Gagua, David Gagua Maternity Clinic 17. Kote Bochiroshvili, Head of Zestaphoni Maternity Hospital "Elite" 18. Nana Kapanadze, Village doctor, Kveda Sakara PHC facility 19. Anjela Orjonikidze, Head of Ob/Gyn Department, Z. Tskhakaia National Intervention Center (Evex medical corporation) 20. Mzevinar Kublashvili, Pharmacist, PSP Pharmacy Kutaisi 21. Elsa Porchkhidze, Manager, PSP Pharmacy Kutaisi 22. Rusudan Kvitashvili, Pharmacist, PSP Pharmacy Kutaisi 23. Tata Shalamberidze, Manager, PSP Pharmacy Kutaisi 24. Nino Ugrekhelidze, Ob/Gyn, Ltd Leri Khonelidze Clinic 25. Shorena Sulamanidze, Ob/Gyn, Ltd Leri Khonelidze Clinic 26. Ketevan Kajaia, FP National Trainer, Kutaisi #3 Maternity Clinic 27. Gia Kebuladze, FP National Trainer , Kutaisi Bomondi Hospital 28. Ketevan Jugeli, FP National Trainer, Ob/Gyn at Nazarashvili Family Medicine Center 29. Marika Davituliani, FP National Trainer, Head of HERA (Local NGO) Kutaisi 51 30. Andro Pipia, Head of Ob/Gyn department, Ltd Geo-Hospitals - Chiatura Multi-profile Clinic 31. Nestan Abesadze, Village Doctor, Kvatskhi PHC facility, Chiatura 32. Tamar Kviriliani, Village Doctor, Sviri PHC facility, Chiatura 33. Irina Pkhakadze, Deputy Dean of Medical School, Kutaisi Akaki Tsereteli University, Medical school 34. Zhana Shamatava, Head of Ob/Gyn, LLC Senaki Maternity House 35. Lasha Adonia, Ob/Gyn at Ltd Tsalenjikha Multi-profile Clinic Evex 36. Zaira Kalandia, Village Doctor, Nakipu PHC facility, Tsalenjikha 37. Ala Gridasova, Head of Ob/Gyn department, Zugdidi St.Luka Medical Center Evex 38. Tsisia Bakhia, Ob/Gyn at Khobi Multi-profile Clinic Evex 39. Klara Kiria, Ob/Gyn at Khobi Multi-profile Clinic Evex 40. Tekla Abakelia, Pharmacist, Pharmacy Impex, Khobi 41. Inela Kukava, Village Doctor, Khamiskuri PHC facility, Khobi 42. Nana Tsitlidze, Ob/Gyn at Zugdidi – Refugee clinic, Curatio. 52 ANNEX 5: SUMMARY OF RESULTS OF FOCUS GROUP DISCUSSIONS AMONG WOMEN OF REPRODUCTIVE AGE (AGE 15 TO 49) 53 I. Focus Group Discussions Summary Mean for satisfaction with service accessibility among participants (scale 1 to 10) 7.18 Mean for satisfaction with service quality among participants (scale 1 to 10) 7.56 1. What type of MNCH or FP services are available at this facility?  Respondents report improvements in availability and quality of MNCH and FP services since 2009  They also note doctors and nurses being better trained and polite, although there are some complaints about medical staff not paying adequate attention to health issues of patients  The range of counseling services available at health facilities is reported to have had increased.  Availability of spinal anesthesia is appreciated  Single-patient units for women giving birth are available  Husbands are allowed to attend delivery  Information on contraceptives is available upon discharge  However, availability of some services is dependent on whether women have health insurance  Cases of wrong diagnosis are reported in Kutaisi  Some respondents report that doctors tend to avoid excessive use of drugs 2. What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)?  Respondents are aware of such contraceptives as condoms, pills, and IUD. They are less well-informed about implants  There are concerns about using oral contraceptives and implants among respondents due to their perceived side-effects. Side-effects among those using IUDs are also reported.  Oral contraceptives are perceived as resulting in women gaining weight to due to hormonal disfunction  Condoms are reported to be contraceptives of choice.  None of respondents is actually using implants  Opposition from church with respect to use of contraceptives is reported 3. In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________  Post-delivery services including electronic ones, admitting husbands to the delivery, early breastfeeding, skin-to skin contact are listed among recently introduced services  Women are registered with local polyclinics by MNCH facilities for child being monitored and vaccinated 4. Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? 54  Mostly, respondents receive their information from websites, friends and relatives, TV broadcasts, and MNCH and FP-related publications.  There were some complaints about doctors not being pro-active in providing information 5. For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain.  Limited availability of FP consultations/advice is reported 6. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go?  As a rule, women receive postpartum check-up at the polyclinic by place of residence  Some women do not consider follow-up check –up necessary and avoid it.  Limited number of respondents received post-delivery care  Women are provided with information on diet  Pediatricians visiting mothers with newly-born children upon their discharge from hospital is common practice 7. Are there any barriers to your use of MNCH services or FP methods?  Some respondents report no barriers to obtaining MNCH and FP services  Others mention opposition from the Church as a factor making it difficult for them to use contraceptives  Financial concerns are also a barriers for some respondents who report price of oral contraceptives to be high. 8. (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use?  Only a few respondents claim to have heard about Parents’ School  Some visited the on-line version of Parents’ School 9. Abortion  There were respondents who experienced abortions in all focus groups but one.  Post-abortion care appears to vary by locations as some respondents report to having received it, others point out that it tends to be focused on those with health issues while still others did not receive it at all. 55 II. Focus Group Discussion Notes SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 1 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Kutaisi Type of Facility Hospital ☐ Maternity ☒ PHC ☐ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? Participants has received both maternal care and family planning services. All reported that quality has been improved when compared to previous years from 2009. Doctors and nurses are more polite and they assisted the patients with everything. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? All respondents report that the diversity of the methods for FP has been increased drastically. Although, there was opinions that none of the contraception were safe for the women. Most commonly IUD is used among women. OCs are used as well, but they treat them very carefully because of side effects and they have fear to use them, not to gain weight or damage the health If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? Many respondents mentioned that they have heard about Implanon, but they are afraid, it is still new and they are apprehensive (side effects still unknown). In addition, respondents mentioned religious context: Implanon is forbidden by church, it is objected by priests as well as IUD. In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ After delivery, women are automatically registered in the corresponding nearest clinics and they receive the info about upcoming vaccination and checkups by sms, or doctor is calling and making appointment at home. 56 Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? Only one participants was using internet as the source of the additional information Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. None of the patients have received any FP consultation directly. but some of them consulted their private doctors about OC or other means to use. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? Some participants believe that mandatory 3 days to stay at hospital after delivery isn’t enough to make sure that there’s no complications to the health of mother, they think that 5 days might work. Very few respondents were checking back for follow-up checkup, although there are controlled by doctors, even fewer consider it to be important. In general, women are checked for postpartum during 40 days, doctor visit the patients or provide phone consultation about child care and other topics. Patients never checked back to the clinics where they gave birth. They are going to the local clinics for postpartum check and child care. Are there any barriers to your use of MNCH services or FP methods? If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? Only one participant knew about such school, and she was also using internet as the source of the additional information Abortion Almost all FGD participants have had abortions. Even one participant had 15 mini-abortion. 57 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 2 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Kutaisi Type of Facility Hospital ☐ Maternity ☒ PHC ☒ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? Participants have their own private doctor during the pregnancy, which offers consultation services. All respondents liked the services before, but it has been improved even more. All noted that most useful was consultation about prevention of unwanted pregnancy. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ Patients were satisfied with new methods about anesthetics, but only one responded that she prefers C-section. Doctors are trying to avoid c-section, and they offer it only in special cases. Separate rooms for delivery are applicable but none of the participants were interested to be alone. What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? All except one responded that they get information about contraception after discharge. All of them have used or are using IUD, while reporting to have had some sort of side effects from it (gain weight, hormonal imbalance). One participant altered the IUD with spermicide “candles”. If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? There are serious objections to women using Implanon from Church. All participants emphasized that this was the main barrier for them. One even responded that she would get one if her priest allowed it. 58 Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? Flyers were offered to the patient and other type of materials. Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? Are there any barriers to your use of MNCH services or FP methods? All respondents report that there are no barriers and they can get all the services If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? Few participants knew about such school, never used one. About e-version of the program, they have heard that who used it are very satisfied. Abortion 2 FGD participants have had abortions. Post abortion care they received was minimal.. Comments Respondents want more information about new methods, to get more knowledge about reproductive cycle, to get info about fertile days, when is the sex more safe to protect themselves from unwanted pregnancy. 59 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 3 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Kutaisi Type of Facility Hospital ☐ Maternity ☒ PHC ☒ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? All types of services are available, especially when women have the insurance. Now it is much easier to take children to the clinic. Some participant were unsatisfied with the quality of service, doctors sometimes don’t pay enough attention. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? None of respondents have used OC, most common practice is condoms. They consider IUD to be more widely used and safer than Implanon. But they prefer condoms after all. Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? Respondents consider the internet as one of main source to get more knowledge. Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. 60 All participants reported that separate blocks for delivery are available, and that husband is able to attend delivery. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? Only one participant got post-delivery care. Are there any barriers to your use of MNCH services or FP methods? Sometimes doctors don’t pay enough attention. If doctor is very famous, everyone want to visit her and there are very long queues, hence doctor isn’t providing enough information. If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) Women are afraid of OCP; they think that they might gain weight. They prefer IUD, but priests are against them using IUDs. Moreover, abortion and IUD became more expensive. (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? They knew, even it was noted that it was mandatory to attend such sessions. E-version was very informative for the women; you can get all sort of information, even its better then talking to doctor. Abortion One participant had abortion using the pills. Other one was consulted by doctor to get post￾abortion treatment. Comments 61 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 4 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Kutaisi Type of Facility Hospital ☐ Maternity ☒ PHC ☐ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? Participants reported that they accessed maternity and women consultation centers. The approaches and services has been changed. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ It is used to be one but now you can access different type of delivery services. Participants named antenatal training using balls, which helped women to ease the pain during the delivery. They also named anesthetics if women required one, which wasn’t applicable before. They were also satisfied that now it was possible other family member to attend the delivery process, which was very helpful to have someone on your side. Before it used to be very painful, there were no anesthetics or boosters applicable (but some responded that they are afraid and prefer local anesthetics). 4 visits is free of charge, the approaches has been changed before the delivery and after. You get sms when its required for child to get vaccination. We have all useful info about post-delivery period for mother and child health. Earlier it is used to be 9 days, not its only 3 when you have to be in a hospital/clinic. It is very helpful as its more convenient. Moreover you can take meals from outside, earlier it wasn’t possible. Infrastructure is modern, hygiene is preserved everywhere. More contraceptive options are applicable. Doctors consult you during pregnancy, they access all the risks associated and you have all the information that might be problematic during the pregnancy, which is very helpful for decision making. What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? Doctor consulted and prescribed contraception, she was using OC. Others had IUD, some were afraid of pills, as it might cause side effects. There were cases when IUD were altered with OC but their body rejected and now they don’t use any.Fallopian tube closure was also other option for participants, but sometimes doctors refused to do such operation - only in 62 cases when it was dangerous to gave birth. One reported that now she want to get pregnant and doctor recommended to cancel OCs and then check back again to plan best period for pregnancy. Only two respondents knew about Implanon, when doctor recommended the implant and the other heard it in hospital. They don’t have detailed information about that. They don’t trust, even could believe that it existed. If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? No one asked doctors, but in some cases they used books or asked other for experience. There were also some magazines available. Participants reported now there are booklets and there are some ads at clinics and it was more or less useful (like they learned about fertile days and which periods were more safe) . Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. Services improved, now doctor visits patient at home for consultation. Before only IUDs were available, now choice is very diverse. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? After delivery doctor subscribed diets, giving other recommendations as well. Both parents get consultation together, system is more sophisticated. Information is accessible through computer. Are there any barriers to your use of MNCH services or FP methods? Services are available regardless of geographic location. Before there were services but the information was very little. They mentioned religion, but they had to use contraception anyway. Also, economic conditions have effect on choosing contraception methods as OCs are expensive. If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) There are no barriers at all, if you have time and will you can always get any desired services.. 63 (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? They have heard it but never used it before, they were sure that it was available in Tbilisi. Abortion One had several abortions but after IUD they didn’t have any unwanted pregnancy. One had experience that doctor recommend abortion due to her health conditions. Participants also reported that doctor prescribed the medicines for post-abortion care. There were cases for selective abortion, when participant wanted son. Comments 64 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 5 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Kutaisi Type of Facility Hospital ☐ Maternity ☒ PHC ☒ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 4 What type of MNCH or FP services are available at this facility? All participants who visited treatment and comparison facilities responded that they have concerns in terms of the service quality. some patients were misdiagnosed and they had to go to Tbilisi to get proper treatment there. Some of them had problems with antenatal care, doctors aren’t well qualified. What type of services did you come here today to access? (Check all that apply) Patients were giving birth in different hospitals, clinics. They responded that in certain facilities the waiting time and the check-up period is very short and you can’t get sufficient treatment to know if she and the baby won’t have any complications. A patient from comparison facility was miss-diagnosed and had to check with another institution to get right follow-up procedures. She also wasn’t aware about new services like less medicines, husband attending, while patients from treatment facilities reported such experience. Comparison case reported that although the facility was rehabilitated, you need to pay additional money to get services like room cleaning. After C-section she had to take care herself after delivery. Treatment patients reported that they have received such services, like baby care after delivery and doctor asked them to come back for follow-up checkup. Antenatal services ____ Delivery Services_______ Post-Partum Care _____ Newborn Care_____ Family Planning______ How long have you been coming here for this type of services? What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? Only one patient had heard about Implanon, but she was afraid. She never heard how it works or how to use it. None of the doctors offered it to her; she was informed by her sister who is working at the drug store. 65 If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. For FP patients get treatment/consultation in various clinics. When delivering 3rd child, patients request or respond to doctor’s suggestion, to block the fallopian tubes. Only one treatment patient reported that she was getting consultation for better planning the family (she wanted to get pregnant). None the patients were offered different contraception methods, except for OC & IUD. Patients themselves reported that they consider IUD to be most convenient, if they don’t have the health problems. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? All respondents are taking babies to the nearest polyclinics, based on the registration address. After delivery they get information about upcoming vaccinations via sms. Are there any barriers to your use of MNCH services or FP methods? They reported that religion play a huge role while choosing the contraception methods, one of the patient even had IUD removed because of the that. Respondents also mentioned that because of the financial constraints when thinking of buying OC. One treatment patient report that doctor at the clinic was giving away the OC free. but in general they have negative attitude toward the pills, as some of them experienced some side effects. If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? 66 None of them have heard about parental school, very few have received any kind of materials within the maternity clinics. Do you ever go elsewhere to get these services? If yes, where do you go? Only one treatment patient reported that she was obtaining information through internet. Usually they rely on their own experience Additional Comments: 67 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 6 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Kutaisi Type of Facility Hospital ☐ Maternity ☒ PHC ☐ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? Respondents report that there even before 2009 services were available, but not as diverse as it is now. All participants were satisfied with the services, one responded that for all 3 delivery she had to have C-section because of health conditions, but in general she is satisfied with the quality of the services. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? Respondents named OC, IUD and Implanon. About Implanon they have heard form doctor, but never used it, not heard of anyone using it. One respondent said that none of them is allowed for her. Also closing of fallopian tubes was other option as well for participants. One didn’t do it as she was young, but now she has them closed and is very satisfied. If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? Respondents reported that if Implanon was available before, doctors would have offered it and they might have used it. They have very limited information about Implanon, that’s why they don’t trust it. There was case when doctor recommended IUD and not the OC. Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? 68 Books, information from maternity hospitals, internet or from others. Main source is doctor’s consultations and internet. Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. There was no OC before, no other types of contraception. IUD was more common among women. Now the comfort level is higher. The qualification and attitude of the doctors were low, even one participant reported that because of the doctors in Kutaisi she had sEPCis and was cured at Gudushauri clinic in Tbilisi. One respondent reported that she was discharged from clinic when she had temperature, without taking analysis. Also awareness level of contraception methods used to be low. Respondents also reported that consultation used to be free, but now you have to pay the fee. Respondents reported that there are new services like separate block for delivery, etc but you have to pay extra money for that. Previously they used to give lot of medication to the patients, now they are reasonable with them, although there are some cases when doctors provide more vitamins then are required. Earlier the baby was not provided with milk, but now it different, baby is provided with milk immediately. In addition, child care after delivery wasn’t organized before, now the situation has been improved. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? Doctor asked one patient to come back after 40 days, to get IUD. But she had to remove as her body rejected it. Another participant also reported that she couldn’t make until 3rd month, she get the IUD and was supposed to visit again, but couldn’t. 3rd reported that no one warned her to revisit, only was asked to re-check if she had any complications after delivery. Other one reported that she was counseled in Tbilisi, not in Kutaisi. About child care, respondents answered that earlier they had to take baby at hospital, but now the pediatrician visits them at home. Are there any barriers to your use of MNCH services or FP methods? If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) Sometimes doctors forbid the patients to take of the IUD, explaining that they are still young, to get treatment and to get pregnant. That is why in general women use home-made devices to prevent the pregnancy. There is lack of information on FP. Also, economic conditions play a big role: you need money to get the consultation and visit the doctor. That’s why some of them trying to protect themselves on their own. 69 (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? They have heard but never attended one. Abortion None of them had abortion Comments Basically the situation has improved, but still there is no habit of visiting the doctor regularly. They even reported that it has been a while since their visit to the gynecologist. 70 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 7 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Tbilisi Type of Facility Hospital ☒ Maternity ☒ PHC ☒ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 4 What type of MNCH or FP services are available at this facility? Respondents applied for different women consultation centers and maternity houses. They have reported that they are very satisfied with the conditions and with quality, although there are some facilities which provide better services, more comfort. When comparing experiences between delivery before 2009 and after, respondents emphasized that the quality has been improved, staff seems to be more qualified, as there were more practitioners before. Women went to the antenatal consultation at Women’s centers; the facilities had all the modern equipment and personnel to get proper treatment. For patients who had c-section separate block were provided, they had all kind of services for mother and for child as well. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? FGD participants were aware of OCs and IUD but haven’t heard about Implanon. Mainly, they use condoms as they are more effective when compared to IUD or “candles”. One reported that doctor recommended the IUD but she had to alter it due to her health conditions. If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? Books or facilities they visit. One reported that she get additional materials from the polyclinic she was visiting. 71 Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. Respondents reported that now the treatment is more effective, earlier the diagnosis was problematic, but now with proper consultation, treatment they are very satisfied. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? The practice is that respondents are taking their children to the corresponding polyclinics, they have their own pediatricians and, if it is needed, doctors visit them at home. After delivery doctors are providing information on how to take care of newborn (bath, change diaper etc). There was only one case when patient was asked to re-visit after two days. in other cases, if patients felt well after delivery they were just asked to check back if they was some issues afterwards. Are there any barriers to your use of MNCH services or FP methods? If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) Respondents reported no barriers, that its more preferable to use contraception rather than having the abortion. (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? Respondents have seen the commercials, TV programs or internet but never used it. Abortion 3 participants have had the abortion. they went to the doctor for follow-up check, one patient had complication and she still having the ongoing treatment. In general, women are visiting doctors for post-abortion care, some are asked by doctors but mainly they visit based on their health conditions. Comments They wish that such comfort to be sustained. Also mentioned that opening new clinics will make the quality become more better. 72 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 8 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Tbilisi Type of Facility Hospital ☐ Maternity ☒ PHC ☒ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? FGD participants reported that before no one even have dreamt about FP, but now we can get consultation, planning for delivery or child care. The conditions improved, participants were satisfied with the doctors, but they have complaints about nurses. Sometimes they don’t pay enough attention. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ One respondents had sEPCis and doctors did everything to save the child. She was warned not to get pregnant until her blood would become suitable, but she got pregnant and gave birth without any problems. Doctors were giving instructions to patient who to take care of the newborn baby. Pediatricians were teaching to feed the baby. All reported that there is a big difference when comparing before 2009 and later. What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? One respondent reported that she consulted with the doctor and he suggested the pills, but she was afraid and chose condoms. Another respondent was also offered OCs by doctor. In general, doctors were suggesting the OC, but patient prefer not to use them. One respondent reported that doctors provide information briefly, they tried to get rid of her.Very few knew about IUD, they are not using them. Doctor also suggested OC, not the IUD. Only one respondent knew about implants, none of them were provided with the info by doctors. Respondents expressed the interest about Implanon and asked FGD participant to share her knowledge about that. If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? 73 Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? One respondent answered Internet, other were reporting books, booklets, friends or doctors and nurses at consultation centers/maternity clinics. This type of information isn’t regular, they hear it from relatives, friends etc. Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. Respondents reported that separate blocks are available, husband attendance is possible during delivery. There are consultation centers where you can go for regular health checkup. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? All respondents had their own pediatricians who are checking babies after delivery. Doctors also asked all women to come back after 40 days for follow-up. They are very satisfied and are receiving all the services. Are there any barriers to your use of MNCH services or FP methods? If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) Respondents reported that due to the religious beliefs it is forbidden to use contraception but they use it, anyway. They consider that its more irresponsible not to use contraception, especially when you don’t have means to take care of the children financially. (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? Only one respondent was aware of such service, she knew that such schools are available at the Chachava clinic. Abortion Respondents reported that they were never asked to check back after abortion. If they didn’t take the initiative themselves, they weren’t contacted. Comments 74 Respondents mentioned that availability of more information would be very helpful, especially for young people. Having some sort of information center, more TV programs is recommended. Participants also emphasized that there is not enough qualified consultation services available. 75 SUSTAIN Project Evaluation SUSTAIN Project Evaluation FGD 9 Results of Focus Group Discussions among Women of Reproductive Age (age 15 to 49) MNCH Location Tbilisi Type of Facility Hospital ☐ Maternity ☒ PHC ☒ Sustain Target Facility Yes ☐ No ☐ Both ☒ Number of Participants 5 What type of MNCH or FP services are available at this facility? Most of respondents go Women’s consultation centers and they are very satisfied, especially with the Sandra Roelofs clinic. What type of services did you come here today to access? (Check all that apply) How long have you been coming here for this type of services? In the time you have coming to this facility, have any new services been introduced? If yes, please list what they were ___________ Participants reported that they were very satisfied with the improved conditions and quality of services, especially the attitude of med personal was emphasized. Women get the consultation after delivery; they taught them how to take care of themselves and newborn baby. What Family Planning methods do you know? Have you heard about subdermal implants? (Implanon)? One respondent uses “candles” recommended by friend. Doctor also offered consultation services, but she didn’t visit them. 2nd respondent uses condoms, then she altered with OCs, but had to cease the consumption due to the side effects. 3rd was using IUD for 2 years, changed it to OC and sometimes uses condoms as well by the recommendation of doctor. 4th is also uses OC, has IUD as well and getting occasional medical assessment. 5th was explained by doctor how to use calendar day and don’t use any other methods. Respondents haven’t heard about the Implanon. None was informed by their doctors at clinics or during consultation. If yes, would you be willing to try it? How do you compare this method to IUD: which one is safer in terms of protecting women’s health? Do you receive education or materials on MNCH or FP here? If yes, how? Do you receive information on MNCH or FP from other sources (e.g. radio, television, newspaper? If yes, what have you heard? From what sources? 76 Main source of the information is TV, friends and relatives. Also, the internet was mentioned as one of the sources. There is no specific site, they are using different internet websites. Have you come here before for other types of services? (see the list above) For Family Planning clients: What types of FP methods are available here? Have any additional methods or services been added since 2010? Are MNCH or FP services that you want available when you need them? If not, please explain. For MNCH facilities: After you are discharged after delivery, do you go anywhere for postpartum (make sure to get accurate Georgian word) checkup? When would you go? After discharge from delivery, will you/did you take the newborn baby anywhere for a checkup? When would you/did you go? After delivery women were consulted and asked to check back if they had any complications. Most of respondents are taking children to the nearest polyclinics, they are more or less satisfied with the quality of services, the only issue long lines and the waiting time is considerable. Are there any barriers to your use of MNCH services or FP methods? If there are barriers such as those listed above, what would make it easier for you to use these services? (probe for all answers) Respondents reported that they do not have any problems or barriers, but they know some women who have problems due to the religion beliefs. (For maternity service points only) Is there a Parents School here? If yes, what topics are included in the Parents School? If yes, did you use the online “e”program of the Parents Schools? If you know about the Parents School but didn’t use it, why? If you didn’t use it, but would like to use it what would make it easier for you to use? Respondents have heard about them but they aren’t enrolled in them. Abortion None of the respondents had any issues after abortion, they were consulted by doctors and asked to check back if there were any complication. Comments