EVALUATION OF KEY PROJECT NOVA ACTIVITIES IN NORTHERN ARMENIA INTERNAL EVALUATION REPORT September 2008 This publication was produced for review by the United Stares Agency for International Development. It was prepared by Zaruhi Mkrtchyan, Project NOVA. EVALUATION OF KEY PROJECT NOVA ACTIVITIES IN NORTHERN ARMENIA INTERNAL EVALUATION REPORT DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes i Author Zaruhi Mkrtchyan, MPH Project NOVA, Senior Research, Monitoring and Evaluation Officer; IntraHealth International, Inc. Reviewers Virginia Hight, Dr.P.H. Monitoring and Evaluation Manager; IntraHealth International, Inc. Inna Sacci, MA Project NOVA, Chief of Party; IntraHealth International, Inc. Amy Armistad, MA Program Officer; IntraHealth International, Inc. Participants Zaruhi Mkrtchyan, MPH Project NOVA, Senior Research, Monitoring and Evaluation Officer; IntraHealth International, Inc. Rebecca Kohler, MPH Project NOVA, Chief of Party (Former); IntraHealth International, Inc. Lusine Ghazaryan, MD, MS Project NOVA, Project Manager/Acting Chief of Party (Former); IntraHealth International, Inc. Haik Ghuzalyan, PhD Project NOVA, Monitoring and Evaluation Officer (Former); IntraHealth International, Inc. Data Collection and Processing Field Coordinator Artak Ordyan Clinical Data Collectors Gohar Zohrabyan, Rita Janvelyan, Ruzanna Manucharyan, Svetlana Martirosyan, Izabella Mirzoyan, Flora Gharibyan Social Data Collectors Arusyak Sevoyan, Aghavni Dilanyan, Anna Tadevosyan, Irina Muradyan,Irina Nersisyan, Nune Hovsepyan, Nune Truzyan, Lusine Arushanyan, Hrachuhi Ghazaryan, Ruzanna Martirosyan, Hasmik Hakobyan, Ani Aivazyan, Armine Ghazaryan, Laura Harutunyan MIS Officer Harutyun Shahumyan Data Entry Operator Nelly Sargsyan, Arusyak Sevoyan, Rouzanna Gevorgyan, Nune Dolyan Watercolor Painting on the Cover Manushak Mkhitaryan, 13 years; Talin Art School From Project NOVA’s “Happy Family in my Imagination” Artwork Contest Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes ii Acknowledgments Project NOVA wishes to express its gratitude to all of the people – from health administrators and healthcare providers to women interviewed during household surveys – who contributed to evaluation of the Project success in Tavush, Shirak, Gegharkunik and Kotayk marzes. Representatives of the Ministry of Health’s Mother and Child Division, as well as the heads of Marz Health Departments of Tavush, Shirak, Gegharkunik and Kotayk marzes provided their support and help during the planning and implementation of the evaluation in their respective marzes. Special words of appreciation go to the heads of polyclinics and health service providers from polyclinics, health centers, medical ambulatories and health posts, who kindly agreed to participate in the evaluation. Gratitude and appreciation are given to all 384 women who devoted their time and efforts to aid in evaluating the Project. Thorough recognition and appreciation is given to data collection and processing teams and their leaders, whose tireless efforts ensured the rigorous daily management and oversight required to implement this evaluation and guarantee the high quality of data collected. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes iii Table of Contents Acknowledgments .............................................................................................................................. ii Table of Contents .............................................................................................................................. iii List of Tables and Figures ................................................................................................................. iv Abbreviations and Acronyms .............................................................................................................. v Executive Summary .......................................................................................................................... vii Background ........................................................................................................................................ 1 Methodology ....................................................................................................................................... 7 Objectives ....................................................................................................................................... 7 Study Groups .................................................................................................................................. 8 Sampling and Sample Size............................................................................................................. 8 Data Collection ............................................................................................................................... 9 Data Entry, Management and Analysis .......................................................................................... 9 Ethical Considerations .................................................................................................................. 10 Study Limitations .......................................................................................................................... 10 Instruments ................................................................................................................................... 11 Results .............................................................................................................................................. 13 Provider Performance ................................................................................................................... 13 Clarifying Job Expectations .......................................................................................................... 13 Providing Performance Feedback ................................................................................................ 13 Improving Physical Environment, Supplies and Equipment .......................................................... 13 Improving Knowledge and Skills of Providers ............................................................................... 14 Provision of Antenatal Care ...................................................................................................... 14 Provision of Postpartum, Newborn and Infant Care ................................................................. 17 Women’s Attitudes, Practices and Experience ......................................................................... 19 Utilization of Primary Healthcare Services .................................................................................... 22 Conclusions ...................................................................................................................................... 23 Appendixes ....................................................................................................................................... 27 Appendix 1: Project NOVA Geographical Coverage by Marz and Implementation ....................... 27 Appendix 2: List of Selected Communities for Community Partnership for Health Interventions .. 31 Appendix 3: ANC Assessment Form ............................................................................................ 33 Appendix 4: PP/IC Assessment Form .......................................................................................... 35 Appendix 5: Facility Inventory Assessment Form ......................................................................... 37 Appendix 6: Client Record Review Form ...................................................................................... 39 Appendix 7: Provider Interview Form ............................................................................................ 41 Appendix 8: Client Interview ......................................................................................................... 45 Appendix 9: Consent Forms ......................................................................................................... 53 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes iv List of Tables and Figures Figure 1. Maternal Mortality Ratio in Armenia (average for 3-year period), 1997-2007 ...................... 1 Figure 2. Causes of Maternal Death in Armenia ................................................................................. 2 Figure 3. Project NOVA Geographical Coverage in Northern Armenia During 2004 - 2006 ............... 3 Figure 4. Organizational Framework of the Safe Motherhood Clinical Skills Training ........................ 4 Figure 5. Performance of Nurses in ANC Following Project Interventions ........................................ 14 Figure 6. Performance of Nurses in PP/IC Following Project Interventions ...................................... 17 Figure 7. Utilization of Health Posts for Antenatal Care .................................................................... 22 Figure 8. Health Post Utilization for Antenatal Care by Project Intervention ..................................... 22 Table 1. Childhood Mortality Rates in Armenia .................................................................................. 2 Table 2. Childhood Mortality Rates by Residence ............................................................................. 3 Table 3. Number of Rural Community Nurses Trained in SMCS ....................................................... 4 Table 4. Instruments and Target Groups ......................................................................................... 11 Table 5. Mean Number of Basic Equipment and Supply Items in Health Posts ............................... 13 Table 6. Mean Knowledge Test Scores of Nurses in Antenatal Care .............................................. 14 Table 7. Mean Health Post Nurse Performance in ANC by Marz and Type of Project Intervention . 15 Table 8. Mean Nurse Performance Score by Skills Area ................................................................. 15 Table 9. Provider Performance of Selected Actions of Antenatal Care............................................ 16 Table 10. Mean Knowledge Test Scores of Nurses in Postpartum, Newborn and Infant Care ......... 17 Table 11. Mean Performance Score of Nurses in Postpartum, Newborn and Infant Care by Marz .. 18 Table 12: Mean Performance Scores of Nurses in Postpartum, Infant and Newborn Care by Skills Area ................................................................................................................................. 18 Table 13: Provider Performance of Selected Actions of Postpartum, Newborn and Infant Care ...... 19 Table 14. Number of Women Interviewed by Marz ........................................................................... 19 Table 15. Percentage Distribution of Women by Marz and Socio-demographic Characteristics ...... 20 Table 16. Percentage of Women with Early Antenatal Care Coverage by Marz ............................... 21 Table 17. Primary Reasons for Not Seeking ANC During the 1st Trimester ...................................... 21 Table 18. Percent of Women Reported Being Informed About Pregnancy Danger Signs ................ 21 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes v Abbreviations and Acronyms ADHS Armenia Demographic and Health Survey ANC Antenatal care ARI Acute Respiratory Illnesses ASTP Armenia Social Transition Program (USAID-funded health reform project in Armenia) CEE Central and Eastern Europe CPI Client-Provider Interactions EMG Emerging Markets Group, Ltd. FD Family Doctor FM Family Medicine FP Family Planning GIS Geographic Information Systems GoA Government of Armenia GP General Practitioner HC Health Center HIS Health Information System HP Health Post IC Infant Care IH IntraHealth International, Inc. IMCI Integrated Management of Childhood Illnesses MA Medical Ambulatory, same as Ambulatory Marz Province; there are 10 marzes in Armenia, plus the capital city of Yerevan MCH Maternal and Child Health MoH Ministry of Health NGO Non-governmental Organization NIS Newly Independent States T&E Training and Equipment provision Ob/gyn Obstetrician/Gynecologist PC Polyclinic PHC Primary Health Care PMP Performance Monitoring Plan PP Postpartum PPC Postpartum Care PRIME II USAID project focused on strengthening the performance of primary care providers as they work to improve services in their communities QA Quality Assurance RH Reproductive Health SC Save the Children SMCS Safe Motherhood Clinical Skills SPSS Statistical Program for Social Sciences (statistical software package) STIs Sexually Transmitted Infections USAID United States Agency for International Development WCC Women’s Consultation Center WHO World Health Organization Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes vii Executive Summary Following independence in 1991, Armenia experienced a painful phase of economic and social transition that was accompanied with a decline in health status of the population, as a consequence of the downturn in the health and social systems of the country. Vital health indicators, such as mortality and morbidity, life expectancy and health status of the population, declined reaching alarming levels. Reproductive Health (RH) and Maternal and Child Health (MCH) care at the primary level became one of the key focus areas for the Armenian Ministry of Health (MoH). From 1990 to the present the MoH has been supported by international governments and non-governmental organizations (NGOs) to improve the health status of its population. Taking into account the still alarming situation in MCH in Armenia and in response to MoH requests for support in continuation of comprehensive MCH programs in Armenia, in October 2004, the United States Agency for International Development (USAID) awarded Emerging Markets Group, Ltd. together with IntraHealth International, Inc, and Save the Children a contract for a 5-year program to improve RH/MHC in rural areas throughout Armenia (Project NOVA). Program activities began in the four northern marzes (provinces) starting from Shirak and Tavush in 2005 and Gegharkunik and Kotayk in 2006. Service improvements concentrated on the following four technical areas: (1) Improve RH/MCH performance of providers through training and equipment provision; (2) Strengthen management and supervision of rural RH/MCH services; (3) Improve RH/MCH policy formulation; and implementation and (4) Increase consumer demand for high quality RH/MCH services through community education and mobilization. In 2006 the Project NOVA completed the first stage of its implementation in the northern marzes of Armenia. Following program closing in the North, the Project conducted an end-line assessment to compare data with baseline statistics and to evaluate overall project achievements. Key findings are summarized below: • Project NOVA’s impact on the increased use of rural primary healthcare services for antenatal care (ANC) in northern Armenian marzes was significant, leading to an average five-fold increase in the utilization of services. This increase is correlated with the extent of Project NOVA involvement. Rural health posts (HP) receiving the full scope of Project interventions – training of community nurses, provision of essential equipment and supplies, and Community Partnership for Health initiative – had the greatest increase (six-fold) in the utilization of services, whereas increase in the patient flow in those HP that received interventions limited to nurse training and provision of equipment and supplies was four-fold. • Programmatic interventions had a positive effect on the supportive supervisory visits to the rural health posts that considerably contributed to the improved performance of rural community nurses. Not only did the interventions boost the number of the supervisory visits from 0.9 visits per month to 1.4, but also increased the duration of these visits from 2.5 hours to 3.5 hours. • During its first two years of implementation, the Project trained 159 community nurses and 88 midwives in key aspects of safe motherhood. These nurses provided services in 78% of rural HP in northern Armenia. Providers’ overall knowledge in antenatal, postpartum (PP), newborn and infant care (IC) increased with a mean improvement of 20%. Although the overall increase in knowledge among community nurses was noteworthy throughout all key technical areas, advances in learning for PP and newborn care were greater (26%) than those for ANC (22%) and IC (14%). • Moreover, observations of provider’s performance showed positive changes in all aspects of care demonstrating almost two-fold improvements in both antenatal and postpartum/new￾born/infant care areas reaching the highest possible performance scores of 52% and 57% respectively. Nurses who received all Project interventions showed greater improvements in client-provider interactions and counseling due to more support through additional commu￾nity education and mobilization activities. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes viii • Sixty-five rural health posts were renovated under the terms of NOVA’s Community Partnership for Health initiative contributing to the overall improved performance of rural primary healthcare providers. • A total of 178 rural HP received basic equipment and supplies from Project NOVA. The results of the end-line assessment indicated a 50% increase in the availability of medical supplies in the rural HP. An assessment of basic equipment and supplies showed that the mean number of essential supplies and equipment across the clinics increased from 10 to at least 15 (out of possible 20 items). • Given the short programmatic intervention time (one year), the impact of Project activities on the population indicators was less significant leading to minimal changes in the population’s access to health care and information. The average number of ANC visits during pregnancy increased from 5.9 visits at baseline to 6.3 visits at end-line. This change indicated that the average number of visits increased by a non-clinically meaningful 0.4 visit per pregnancy for all four northern marzes. Neither did the Project have any impact on increasing the early coverage for ANC. However, positive attitudinal change was observed in the percent of women valuing preventive healthcare: 37% of women reported that people generally delay healthcare due to no urgent need of interventions at the baseline as compared with 24% in the end-line evaluation. • Women are still not adequately informed on the pregnancy-related danger signs. There was no difference in the percent of women who recalled being informed of the pregnancy danger signs. Although the results of the performance assessment showed an almost 40% increase in percent of HP providers informing women on danger signs of pregnancy, this pattern was not confirmed by women. Findings of this internal evaluation suggest that Project NOVA had significant positive im￾pact on the knowledge and performance of rural primary healthcare providers in key RH/MCH areas, as well as utilization of primary healthcare (PHC) services in four northern Armenian marzes. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 1 Background Following independence in 1991, Armenia experienced a painful phase of economic and social transition that was accompanied with a decline in health status of the population, as a consequence of the downturn in the health and social systems of the country. After the collapse of the Soviet system, the country inherited a centralized healthcare system but lacked sufficient funding to support its functioning. The population’s deteriorating health indicators, greatly affected by the devastating socio-economic situation, armed conflict in Nagorno-Karabakh and consequences of the 1988 earthquake, have deteriorated even further given the system inefficiencies in place. The de￾mand for quality health services on the one hand, and insufficient funding on the other, necessitated large-scale health sector reforms by the Government of Armenia (GoA). These reforms were designed to target the scarce resources towards the most vulnerable groups of the population, and to re-orient services to provide more cost-effective PHC. Improving RH/MCH services at the primary level has been a key focus area for the Armenian Ministry of Health (MoH) since the 1990s. For many of the RH/MCH indicators, Armenia is still far below average European levels. In spite of significant improvement over the past decade, survey data from 2000 and 2005 Armenia Demographic and Health Survey (ADHS) and other surveys, indicated that the health status of the population, including women and children, is poor and utilization of RH services is low. The maternal, perinatal, and child mortality rates are higher than in developed countries. From 1999 to 2001 the registered three year average maternal mortality rate was 42.3 per 100,000 live births. In 2002 – 2004 it was 24.9/100,000, and in 2005-2007* it was 25.1 (Figure 1). The maternal mortality rates in Armenia are lower than the average regional rates (34/100,000), and higher than the average rates for Central and Eastern Europe (15/100,000). Nevertheless, the maternal mortality is considerably higher than in Western Europe (8.8/100,000). Figure 1. Maternal Mortality Ratio in Armenia (average for 3-year period), 1997-2007 30.8 39.9 42.3 37.8 20.2 24.9 27.6 32.6 25.1 0 5 10 15 20 25 30 35 40 45 Source: Ministry of Health, 2008 * Data for 2007 are preliminary and not officially published by MoH. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 2 The major causes of maternal deaths within the period of 2000 - 2006 were PP hemorrhage, contributing to 31% of all maternal deaths, non-obstetrical causes (18%), hypertension (15%) and infections (7%) (Figure 2). Figure 2. Causes of Maternal Death in Armenia The perinatal mortality rate indicates the num￾ber of deaths that occurred in the period from the 28th week of gestation until the early neo￾natal period (0-6 days of life). The perinatal mortality rates in Armenia did not change radi￾cally during 2000-2005 and fluctuated between 14.6 and 16.6 per 1,000 births (MoH, 2006). This rate would be higher (~30/1,000) if new￾borns with body weight of 500-1,000g that had died before the 7th day of life had been included in the sample (MoH, 2006). Neonatal mortality (0-28 days), according to data of the Ministry of Health, was 8.5 per 1,000 live births in 2005 and 10.7 in 2006; stillbirths in 2005 were 7.9/1,000. The number of deaths among children of 0-1 year and 0-5 years age groups in 2006 were 13.9 and 15.8 per 1,000 live births respectively and decreased in 2007 to 10.7 and 12.3 respectively. Overall, perinatal mortality rates in the Europe varied from 5 to 20 per 1,000 births. Neonatal mortality ranged from 6 to 21 per 1,000 live births in the Newly Independent States (NIS), from 3 to 7 in the Central and Eastern Europe (CEE) and from 2 to 5 in Western Europe. As reported by the National Statistical Service, neonatal mortality rates in Armenia showed a slight decrease from 9.1 in 1996 - 2000 to 8.8 deaths per 1,000 live births in 2001 - 2005. As reported by MoH, 50% of the perinatal deaths occurred in the prenatal period due to congenital defects, hypoxia and asphyxia, which may be attributed to the low utilization of antenatal services and poor evaluation and diagnostic care. The other contributing cause of perinatal mortality was intrauterine infections. The ADHS (2005) reported higher rates for infant and child mortality: neonatal death rate of 17 per 1,000 live births; infant mortality (0-1 year) rate of 26/1,000; and under five mortality was 30/1,000. This leads to the conclusion that these rates could have been underestimated by official MOH repor￾ting or could have been calculated using different approaches (including or excluding the newborns with the weight less than 500g, different definitions of neonatal mortality and live births, etc.). Improvements in childhood mortality in Armenia have been observed since 1991 (Table 1). To some deg￾ree the decline was hastened by health interventions initiated by the MoH. In 1994 programs focused on management of diarrhea and acute respiratory infections, promotion of breastfeeding, implementation of Integrated Management of Childhood Illnesses, etc. However, there is a considerable difference in the child mortality rates between urban and rural populations of the country, with rural population having higher mortality rates (Table 2). Table 1. Childhood Mortality Rates in Armenia Approximate calendar period Neonatal Mortality Infant Mortality Child Mortality Under-five Mortality 1991 – 1995 17 41 7 48 1996 – 2000 20 30 7 36 2001 – 2005 17 26 4 30 Source: ADHS 2005 Rupture of uterus, 2% Hemorrhage, 31% Non￾obstetrical causes, 18% Hypertension, 15% Embolism, 9% Septic, 7% Other causes, 7% Extrauterine pregnancy, 6% Abortion, 5% Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 3 The 2005 ADHS also highlights that access to family planning ser￾vices is limited and abortions are still used as a means to control and regulate fertility, often accom￾panied by higher risks than for family planning (FP). The percen￾tage of pregnancies ending in in￾duced abortion increased from 45% in 2000 to 55% in 2005. In October 2004, the United States Agency for International Development (USAID) awarded Emerging Markets Group, Ltd. (EMG), together with IntraHealth International, Inc. (IH) and Save the Children (SC), a contract for a 5-year program to improve RH/MCH in rural areas throughout Armenia (Project NOVA). In 2006, Program activities began in the following four northern marzes (provinces): Shirak and Tavush in 2005 and Gegharkunik and Kotayk (Figure 3). Detailed maps for each target marz are available in Appendix 1. Figure 3. Project NOVA Geographical Coverage in Northern Armenia During 2004 - 2006 Table 2. Childhood Mortality Rates by Residence Residence Neonatal Mortality Infant Mortality Child Mortality Under-five Mortality Urban 18 25 2 26 Yerevan 19 24 3 26 Other urban 16 26 1 27 Rural 19 31 11 42 Source: ADHS 2005 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 4 ToT for Clinical Preceptors Module 1. Learners Orientation, Basic Nursing Module 2. Infection Prevention Self Study at Workplace Working with Facilitator and Partner Clinical Practice using Mannequins FINAL EXAM Module 8. Expanded Role of Midwifes Module 3. Antenatal Care Module 4. Intrapartum Care Module 5. Postpar￾tum/Newborn Care Module 6. Infant Care Module 7. Working with Community Area 1: Improve RH/MCH performance of providers In order to improve performance of rural healthcare providers in the four northern marzes, Project NOVA first established 12 clinical training sites. The clinical training sites were used to train nume￾rous healthcare providers from surrounding healthcare facilities in key MCH/RH technical areas, including: (1) Short updates in infection prevention, ANC, PP and IC for 300 healthcare providers from hospi￾tals and polyclinics to identify potential clinical preceptors and to improve supervisory support to community nurses at HP. (2) Safe Motherhood Clinical Skills (SMCS) training (Figure 4) – an innovative performance-based modular learning initiative for rural community nurses aimed at improving skills of PHC providers in major cli￾nical areas: counseling, in￾fection prevention, ANC, in￾trapartum care, PP and newborn care, IC, and com￾munity outreach. Written training materials included clinical guidelines, interac￾tive self-paced modules, clinical checklists and a facilitator’s guide. Self-paced interactive learning modules for each clinical component served as the centerpiece of the training program. Implementation of each module took approximately 4-6 weeks for all learners to complete. A module consisted of working through the self-paced exercises with a learning partner. After learners had successfully completed the theoretical component, they went for clinical practice at NOVA’s clinical training sites under the supervision and guidance of clinical preceptors. Clinical training included demonstrations with mannequins, role-plays and then practicing skills with actual patients* . During its first two years of implementation, the Project trained 159 community nurses and 88 midwives from the four target marzes in SMCS. This training involved providers from 78% of all rural HP in the northern marzes (Table 1). Figure 4. Organizational Framework of the Safe Motherhood Clinical Skills Training * PRIME II Armenia Final Report, September 2004. Table 3. Number of Rural Community Nurses Trained in SMCS Marz No. of Nurses Comments Shirak 46 45 Health Posts out of total 77; 58% coverage Tavush 34 33 Health Posts out of total 37; 89% coverage Gegharkunik 50 46 Health Posts out of total 52; 88% coverage Kotayk 29 28 Health Posts out of total 29; 97% coverage Total 159 78% coverage Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 5 (3) Integrated Management of Sexually Transmitted Infections (STIs) - a 5-day training course for primary healthcare providers, namely ob/gyns and dermato-venerologist * from the target marzes. A total of 50 ob/gyns and dermato-venerologists completed the training from the four target marzes. (4) Key Reproductive Health Competencies training for family physicians is a 10-day training prog￾ram in which the program trained 46 family physicians from target marzes. Area 2: Strengthen management and supervision of rural RH/MCH services Project NOVA developed a Project-wide Quality of Care Framework that served to guide and integrate quality in all its interventions. The Quality Framework defines quality in terms of five key dimensions: 1) technical competence, 2) management and supervision, 3) access, 4) responsiveness, and 5) environment. Based on this framework, the Project designed a site-level quality assurance (QA) initiative that highlights the role of internal quality teams in guiding the QA process, and introduced it at 24 rural healthcare facilities. On completion of Project activities in these facilities, there was a 16% increase in the facility self-assessed quality score in the four target marzes. In addition to the QA initiative, the Project developed and produced a Management Handbook. Using a pool of national trainers and relying on the newly produced training book, NOVA trained 78 rural managers from polyclinics and medical ambulatories. Their training included effective organizational structure, quality improvement approaches, supportive supervision methodology and its application in their routine daily work, financial management and the legislative environment. Area 3: Improve RH/MCH policy formulation and implementation Project NOVA collaborated with the MoH and with other policy organizations and entities to contribute to changes in the regulatory system leading to overall better healthcare delivery. The Project supported policy changes in the areas of improved clinical protocols, better targeted health financing and expanding the PHC providers’ role. Area 4: Increase consumer demand for high quality RH/MCH services through community education and mobilization The Community Partnership for Health (CPH) is a feature program for Project NOVA. CPH is a methodology to improve the quality and accessibility of health services with community involvement in defining, implementing, and monitoring the quality improvement process. CPH is based on building partnerships among healthcare providers, community leaders and local authorities, and involving and empowering communities in quality improvement. This is a collaborative process, which requires commitment from key members of the community and the health system. As part of this initiative, Project NOVA created local Health Action Groups to identify existing health problems at the community level and to develop an action plan and budget in order to solve these problems. For the vast majority of rural communities, problems identified can be characterized as: ∼ Poor physical conditions of HPs; ∼ Inadequate relationship between rural HPs and supervisory healthcare facilities: physicians from supervisory healthcare facilities do not visit HPs on a regular basis, there is no established partnership between the HP and supervisory facilities, and poor referral/counter-referral system; ∼ Poor general population awareness regarding RH/MCH; and ∼ Lack of information regarding free state-guaranteed PHC services. * A physician treating skin diseases and STIs. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 6 These Partnerships increased community and local government involvement in the improvement of RH/MCH in target communities. They also worked with local NGOs to build marz-level capacity in community mobilization and to ensure sustainability of the program. Sixty communities in the four target marzes were selected for engagement in these partnerships (see Appendix 2 for the list of communities). Project NOVA added community mobilization and health education activities to increase general awareness of RH/MCH issues. Within the scope of the program, these commu￾nities were mobilized to develop and implement plans to target community-level health problems. In addition, six rural communities of Shirak and Tavush marzes were provided with seed grants for renovating their community HPs. * * * In October 2006, the Project completed its interventions in the North, and launched an expanded scope of work in five health networks – Armavir, Vedi, Talin, Sisian and Vayk – one in each Southern Marz: Ararat, Armavir, Aragatsotn, Syunik and Vayots Dzor. These health networks were identified based on a set criteria including: selected key RH/MCH indicators (annual numbers of deliveries, complications during delivery, etc.); physical conditions of the healthcare facilities; num￾ber of physicians within each network; number of rural health posts; and the extent of existing involvement by local and international NGOs. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 7 Methodology Objectives On completion of its activities in the northern marzes, the Project NOVA Research, Monitoring and Evaluation team evaluated its programmatic activities. The purpose of the internal evaluation was to assess to what extent the project achieved its key objectives in the four northern marzes during the first two years of the Project implementation. Specifically the evaluation objectives were: (a) To compare the performance of PHC providers before (baseline assessment) and after (end-line assessment) Project NOVA training events, provision of basic equipment and supplies, and rehabilitation of healthcare facilities. (b) To compare the impact of the Project on population level indicators. (c) To compare the utilization rates of RH/MCH services by conducting client-flow analysis at PHC facilities at the baseline and end of the program activities and comparing with facilities that did not receive Project NOVA interventions. The overall evaluation methodology for the first and second years of Project implementation included pre-post design with a control group of communities, facilities and/or providers who had no direct exposure to Project NOVA interventions. Communities from the corresponding marz, where the Project did not have any direct interventions were included in the control group. Brief description of data collection methodology is described below: Facility assessment (physical structure, supplies, equipment) conducted using a checklist deve￾loped based on WHO recommendations for facility physical structure, availability of basic medical supplies and equipment. Data collected through observations and interviews with facility personnel and through review of facility records. Provider performance assessed through observation during antenatal, PP/newborn and IC. Taking into account that the number of real patients in the facilities (especially in rural areas) was low, scenarios were developed and used for simulating real clients. Reasons for simulating real cases were explained to providers and they were instructed to act as they would have in the real situations. Data collectors with clinical backgrounds, observed the client-provider interaction and assigned a performance score for completion for each item in the checklist. Provider interviews assessed provider perspectives on training and the job environment. The inter￾view lasted approximately 20 minutes and was conducted at the facility. Household survey conducted using a standardized questionnaire addressing women’s experience with RH/MCH. The interviews were conducted at women’s homes in a separate room to ensure a comfortable environment and confidentiality of information. The duration of interview was 20-30 minutes. Evaluators used a total of six instruments (Appendices 3-8) to assess the Project’s activities in four northern marzes. Most of these instruments were adapted from previous tools used in PRIME II globally and in Armenia, while a few were developed specifically for Project NOVA. The same instruments were used for the baseline and end-line assessments to ensure comparability of results. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 8 Study Groups Baseline assessment involved appraisal of facilities and/or providers prior to Project NOVA direct or indirect interventions in the facility and/or community. All PHC facilities in the four target marzes were involved in the assessment. End-line assessment was conducted for intervention and control groups. The intervention group includes facilities and/or providers participating in Project activities (nurse training in SMCS and pro￾vision of equipment and/or CPH). • Limited NOVA Interventions: include the group of facilities and/or providers participating in nurse training in SMCS and receiving basic medical equipment and supplies. • Full NOVA Interventions: include providers and/or facilities participating in nurse training in SMCS and provided with basic medical equipment and supplies and participating in CPH component. The control group included facilities that were not involved in any of the Project-related activities directly. Some of the facilities from the control group benefitted from Project activities indirectly through training of providers at the supervisory facility. Sampling and Sample Size Sampling for Health Facility and PHC Providers The facility sampling list used for the baseline assessment obtained from the marz health depart￾ments, included all primary healthcare facilities: Health Posts (HP), Medical Ambulatories (MA), Health Centers (HC) and polyclinics (PC) in the target marzes. Because the accuracy of these lists varied, the assessment team first cross-checked them with several local (marz) health departments. All primary level facilities were involved in the baseline data collection. Sampling for the end-line evaluation was conducted using the Project NOVA Health Information System (HIS). The Project NOVA HIS contains full list of all PHC facilities (information collected at the baseline assessment) including a full list of providers employed in these facilities. The HIS includes data on Project interventions - the data on providers and facilities were disaggregated according to Project interventions. Because there were different types and levels of interventions, it was not feasible to evaluate each type of intervention separately. Thus, the evaluation mainly focused on the nurse training in SMCS and equipment provision (Training and Equipment, or T&E) and full Project NOVA interventions, which included CPH in addition to T&E. The facility selection for end-line assessment was based on the following criteria: • HPs receiving T&E – facilities where the nurse was trained in SMCS and provided with basic medical equipment and supplies. The provider who participated in the training was involved in the assessment. If there was more than one provider who underwent training in SMCS, one provider was randomly selected. • HPs receiving full Project NOVA interventions – facilities and/or communities, where the CPH component was introduced in addition to T&E. The provider who participated in the training was involved in the assessment. If there was more than one provider who underwent training in SMCS, one provider was randomly selected. • Control group - HPs with no intervention. If there was more than one provider, health providers were randomly selected. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 9 Sampling for Household Survey The assessment team selected a sample of 96 women with a child under 12 months of age from each of the target marzes. This produced a total sample of 384 women for baseline and end-line evaluations. The sample size was based on the formula n=z2 pq/d2 , permitting estimates of propor￾tions and frequencies for the population. A probability-proportional-to-size sampling technique was used based on the number of children under one year of age served by each of the pediatric facilities. The sampling procedure was implemented in the stages described below: • Identification of all pediatric facilities where the records of children under one-year old are kept. In all marzes, these are district polyclinics’ pediatric clinics or pediatric departments in general polyclinics. • Identification of all pediatric sub-departments or districts. The number of pediatric districts generally equals the number of pediatricians in the pediatric polyclinic. Each pediatrician has a record of all children served by him or herself. • Tabulation of the number of children under one year of age in each pediatric district (number of children recorded in the district pediatrician’s journal). • Determination of the number of children and mothers to be sampled from each pediatric district, based on the overall number of children registered in the district and the total desired sample size. • Sampling of the children (for interviewing their mothers) using a simple random sampling technique, using a statistical calculator with random number generator. • Generation of a list of additional women (50%) using the same technique as described in step 5 (excluding children/women already sampled for the main list) to replace possible non￾respondents. Data Collection Based on the Project implementation workplan, the fieldwork took place in four different phases: Baseline assessment in Tavush and Shirak (November – December 2004) Baseline assessment in Gegharkunik and Kotayk (November 2005) End-line assessment in Tavush and Shirak (April 2006) End-line assessment in Gegharkunik and Kotayk (April 2007) A total of 20 data collectors participated in the four data collection studies: 6 clinical data collectors and 14 social data collectors. All data collectors were trained in a two-day session for each stage of fieldwork (baseline in Shirak/Tavush and Gegharkunik/Kotayk and end-line in Shirak/Tavush and Gegharkunik/Kotayk). The training included a review of the research procedures, content of the data collection instru￾ments, standardized scenarios for role-playing and study logistics. Procedures for data collection and interviewing techniques were reviewed in detail. Each data collector received a copy of the data collector’s manual summarizing procedures. Supervisors conducted spot checks to ensure the quality of data collection. Data Entry, Management and Analysis Collected data was entered into Project NOVA’s Microsoft Access-based HIS and then transferred into SPSS v.13 for statistical analysis. To ensure data accuracy, frequency checks were performed to find and correct data entry errors. As necessary, data from different sources were combined for further analysis. The data analysis included descriptive analysis (frequencies and percentages), with comparisons and appropriate statistical tests. For the observation checklists, the tasks performed by a provider (and performed sufficiently well) were summed (assuming equal weights). Sums were divided by the number of items in the checklist to provide an overall performance score (percent of tasks per￾formed successfully) for that provider. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 10 Ethical Considerations The assessment team obtained verbal informed consent from each study participant (both provider and client) prior to interview or observation. Prior to obtaining verbal consent participants were informed of the study objectives, the benefits and possible risks of their participation in the study, and the voluntary nature of their participation. The consent forms for providers and clients are provided in Appendix 9. Study Limitations Two aspects of the study may have influenced the results of this evaluation. In the majority of cases, the provider performance assessment was conducted using simulated scenarios, with one of the data collectors acting as patient and the healthcare provider simulating his or her usual responses. The observation process may have influenced the provider’s behavior. There is an assumption that during performance assessment, the provider acts more professional in a real-case situation as compared with the simulated scenarios. There is also a possibility that providers will perform better during simulations as compared to real-life cases, since it replicates training process for them. In most cases, simulation rather than direct observation was used and the number of actual cases was too small to test if there are any differences between provider performance with actual patients versus actors in simulated cases. Given Project NOVA’s scope of work, some healthcare providers (mainly physicians) received an unequal level of intervention (i.e. individual providers participated in different combinations of trainings). In addition, some HP supervisors underwent Project NOVA trainings, while others did not. Thus, there is a possibility of synergized performance increases due to these uncontrolled training effects. Due to the low number and wide range of different types of possibilities, the testing for that level of detail was not possible. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 11 Instruments Table 4 presents a list and brief description of the instruments used in the study. Copies of all instruments are provided in the Appendices 3 - 8. Table 4. Instruments and Target Groups Instrument Details of the instrument Target group Observation of antenatal care visit (Appendix 3) A checklist of 44 actions used to measure the performance of a PHC provider during an ordinary ANC visit Ob/Gyn Family doctors (FD) General Practitioners (GP) (only in cases where there was no Ob/Gyn or FD in a facility) Pediatricians (only in cases where there was no Ob/Gyn, FD or GP in a facility) Nurses, midwives (in HP, MA, HC) Observation of postpartum and infant care visit (Appendix 4) A checklist of 44 routine actions, intended to measure the performance of a PHC provider during an ordinary PP/IC Ob/Gyns FDs Pediatricians GPs (only in cases where there was no Ob/gyn or FD in a facility) Nurses, midwives (in HP, MA, HC) Inventory (Appendix 5) A standardized checklist reviewing the standard minimal equipment, optional equipment, written guidelines, infrastructure, and medical supplies. Facility Facility Journal Review Form (Appendix 6) A form used to assess the number of clients at PHC facilities during the period from October 2005 to September 2006. Facility Provider Questionnaire (Appendix 7) A standardized questionnaire exploring provider performance factors, with a particular focus on supervision. Nurses, midwives, physicians Household Interview (Appendix 8) A standardized questionnaire exploring women’s perception of the quality of services and their knowledge of and practices in ANC/PP/IC Women with a child under one year of age The performance of nurses in delivering ANC was conducted by observation of real or simulated skills used in routine ANC visits. The instrument contained a checklist of 44 items in the following key areas of care: client-provider interaction (CPI), consisting of 10 items; counseling, consisting of 8 items, general clinical skills, consisting of 18 items; and ANC specific skills, which included 8 items. The PP/IC instrument consisted of 44 items, including CPI (9 items), counseling (10 items), general clinical skills (8 items), PP specific skills (5 items) and IC specific skills (12 items). Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 13 Results Provider Performance Improving the performance of healthcare providers can be achieved by addressing the factors necessary for healthcare workers to perform their jobs to standard. Some evidence shows that attention to the performance factors can also improve job satisfaction and staff retention.* Provider performance can be influenced by five major factors: 1) clear job expectations, 2) clear and imme￾diate performance feedback, 3) adequate physical environment, access to proper tools, supplies and workplace; 4) motivation and incentives to perform as expected; 5) and skills and knowledge required to do the job.† Clarifying Job Expectations One aspect of Project NOVA interventions was to facilitate the clarification and revision of the role of community nurses in Armenia. The National Nursing Development Strategy was prepared by the MoH in collaboration with World Health Organization (WHO) in 2004 and the community nurse job description was approved by MoH decree on October 14, 2005. This job description was originally conceptualized and drafted by Project NOVA and then elaborated by Jinishian Memorial Foundation in partnership with MoH. It will turn into a normative act after its formal registration with the Ministry of Justice. However, since the job description makes reference to the draft Armenian Law on Healthcare, which is not formally approved by the GoA, final registration is still pending. In spite of this, the MoH has adopted the job description as an official document and Project NOVA has distri￾buted it. During SMCS trainings the job description was discussed with nurses in order to make sure that they have a clear understanding of their job requirements and scope of practice. Providing Performance Feedback Supportive supervisory visits are critical to improving quality of care. The supervisory visits were scored according to criteria defined as supportive by Project NOVA. The criteria were: the super￾visor performed administrative tasks, attended patients, worked with the nurse, inspected the envi￾ronment and instruments, solicited patient feedback on services, gave updates on procedural chan￾ges, clarified instructions, discussed difficult clinical cases, suggested a service quality improvement plan, and consulted with a nurse before making decisions. According to this definition only 64% of visits met the criteria for quality of supervision at the baseline. Whereas, at the end-line, the percentage of visits meeting the same criteria for quality of supervision increased to 75%. According to the current MoH regulation, a supervisory healthcare facility physician attached to the HP should visit that HP every month. The mean number of supervisory visits per month increased from 0.9 visits in the baseline to 1.4 in the end-line. The mean duration of these visits was 2.5 hours at the baseline and increased to 3.5 hours at end-line. Improving Physical Environment, Supplies and Equipment In order to facilitate provider performance in rural areas, the Project provided essential equipment and supplies to a total of 178 rural HPs. Project staff also worked with communities and healthcare providers to improve the capacity of HPs to leverage funds for supplies and medical equipment. End-line assessment results showed a 50% increase in the availability of medical supplies. To assess adequacy of equipment and supplies a list of 20 essential items was developed. The proportion of essential supplies and equipment at rural HPs increased from 10 at baseline to at least 16. This increase was observed for all types of HPs participating in the assessment, including facilities with no direct interventions from Project NOVA (Table 5). The increase in availability of supplies in the facilities where the Project did not have direct interventions could be partially attributed to work with HPs’ supervisory facilities (MA and PC) * Yumkella F. Retention of health workers in low-resource settings: challenges and responses. Capacity Project Technical Brief No. 1. Chapel Hill, NC, IntraHealth International, 2006. Available: http://www.capacityproject.org/images/stories/files/technical_brief_no1_retention.pdf † Learning for Performance: A Guide and Toolkit for Health Worker Training and Education Programs. Capacity Project, IntraHealth International, 2007. Table 5. Mean Number of Basic Equipment and Supply Items in Health Posts Baseline 10.1 End-line Intervention 15.9 End-line Control 15.9 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 14 which covered almost all health facilities in the marz. These supervisory facilities are responsible for providing supplies and equipment to the HPs. In addition, as part of Project NOVA’s CPH initiative, a total of 65 rural HPs were renovated with 45- 70% of in-kind and financial contributions from rural communities. Improving Knowledge and Skills of Providers An important indicator for evaluation of the Project performance was the providers’ knowledge (short-term indicator) and performance (long-term indicator) in providing PP, newborn and IC. These indicators were assessed to determine the impact of Project activities on quality of care. The Project monitored change in provider knowledge using pre-and-post tests conducted for all training participants before and after each training module.* Provider performance was measured through observation of ANC and PPC. Overall 292 HP nurses participated in the baseline perfor￾mance and skills assessment and 146 nurses were included in the end-line assessment. Although training of community nurses in safe motherhood covered a variety of training subjects, ANC, PPC and IC were selected as proxy indicators to evaluate for both knowledge and overall performance of healthcare providers before and after NOVA’s interventions. Provision of Antenatal Care Table 6 presents the mean change in knowledge among rural community nurses related to ANC. The mean change in score was 22%. Among nur￾ses trained during the first year (Shirak and Ta￾vush) changes in knowledge scores were less (8.8% and 9.5% respectively) than nurses trained during the second year. Among nurses from marzes trained during the second year the know￾ledge score in ANC increased more than 30%. Pre-test scores in the first year marzes were about 15% higher than the second year marzes. How￾ever, the differences in provider knowledge bet￾ween the marzes became smaller following training (around 10%). As described in the methods section, the performance assessment of nurses in delivering ANC was conducted through an observation of real or simulated skills used in routine ANC visits. The instrument contained a checklist of 44 items in the following key areas of care: CPI, consisting of 10 items; counseling, consisting of 8 items, general clinical skills, consisting of 18 items; and ANC￾specific skills, which included 8 items. Figure 5. Performance of Nurses in ANC Following Project Interventions (% of Highest Possible Score) Figure 5 shows that there was an increase in overall performance scores of commu￾nity nurses from 28% at baseline to 52% at end-line (p<0.01). Nurses who did not participate in the project interventions also showed an increase in performance: 28% in the baseline as compared to 36% in the end￾line evaluation (p<0.05). More detailed analy￾sis of nurses’ performance in ANC showed that between the full and limited NOVA interventions those nurses exposed to full Project interventions (i.e. in addition to nurse * The pre/post-test questions should be interpreted with caution. The pre-test questions were easier as compared with the post-test questions. In addition to evaluating the general baseline knowledge of the group and doing necessary adjustments for the module depth, the pre-tests also seek not to discourage the nurses by introducing the concepts which are totally new for them. Post-test questions targeted the content of the course and measured knowledge which the providers obtain on completion the module. Table 6. Mean Knowledge Test Scores of Nurses in Antenatal Care Marz Pre-test score Post-test score Dif. Shirak (n=46) 76.42 85.21 8.79* Tavush (n=34) 77.44 86.94 9.50* Gegharkunik (n=50) 57.72 93.42 35.70* Kotayk (n=29) 61.62 95.93 34.31* Overall (n=159) 68.28 90.02 21.74* * Dif. is significant: p < 0.01 (paired-sample t test) 28% 52% 36% 0% 10% 20% 30% 40% 50% 60% Baseline End-line Intervention End-line Control Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 15 training in SMCS and equipment provision, also participated in the CPH component) showed higher performance demonstrating 28% difference compared to baseline (Table 7). The marz-level analysis given in Table 7 also reveals that highest performance scores for ANC at the end-line assessment were observed among the nurses from Gegharkunik, Shirak and Tavush marzes ranging from 60.61 in Shirak to 52.11 in Tavush. This difference can be explained by the fact that HPs in Gegharkunik, Shirak and Tavush are located in remote geographical areas far from the MA and nurses working there have comparatively less oversight and support from supervisory facilities. For this reason, they have to perform a wider range of tasks to serve their populations. In contrast, nurses from Kotayk work in more favorable conditions: the population they serve is located close to the capital city and/or is less scattered. Thus, they have better access to higher-level facilities where they can receive care from physicians. Table 7. Mean Health Post Nurse Performance in ANC by Marz and Type of Project Intervention (% of Highest Possible Score) Marz Baseline End-line Control Total Intervention Total Training & Equipment (T&E) Full NOVA Intervention Score (n) Score (n) Dif. Score (n) Dif. Score (n) Dif. Score (n) Dif. Shirak 33.86 (108) 37.25 (18) 3.39 60.61 (39) 26.75* 54.67 (18) 20.81* 65.69 (21) 31.83* Tavush 27.84 (24) 36.69 (7) 8.85 52.05 (20) 24.21* 46.82 (10) 18.98* 57.27 (10) 29.43* Geghar￾kunik 21.74 (74) 37.12 (6) 15.38* 52.63 (32) 30.89* 55.30 (15) 33.56* 50.27 (17) 28.52* Kotayk 20.17 (32) 17.05 (2) 3.13 38.89 (18) 18.72* 35.35 (9) 15.18* 42.42 (9) 22.25* Overall 27.65 (238) 35.88 (33) 8.28* 52.11 (109) 24.46* 50.0 (52) 22.36* 55.94 (57) 28.29* *Difference is significant: p<0.01 (independent sample t test/equal variances) In three marzes (Shirak, Tavush, Gegharkunik) where nurses worked in communities that received full Project NOVA interventions, there were higher performance scores than among nurses who received only nurse training and equipment provision. However, nurses from Gegharkunik commu￾nities that received full Project interventions, received lower performance scores than nurses who received training and equipment provision only (33.56 vs. 28.52). Overall, data show significant improvements (p<0.01) in overall performance between baseline and end-line assessments among facilities where Project NOVA did not have any interventions in all four marzes (Table 7). The stratified analysis of aspects of ANC and their change following Project implementation is pre￾sented in Table 8. The increase in performance is gradual for all aspects of care, however, the grea￾test increase (almost 100%) was observed in ANC specific skills and counseling. This significant change can be attributed to the fact that prior to training, HP nurses had almost minimal knowledge and skills in ANC, and their scope of work included limited activities related to ANC. After training, their scope of work was extended and included basic ANC. Differences between full and limited Project NOVA intervention groups are also presented in Table 8. Table 8. Mean Nurse Performance Score by Skills Area (% of Highest Possible Score) Skills Area Baseline End-line Control Total Intervention Total T&E Full NOVA Intervention Score Score Dif. Score Dif. Score Dif. Score Dif. Client-Provider Interactions 35.5 37.7 2.2 58.8 23.3* 53.2 17.7* 61.8 26.3* Counseling 20.5 30.2 9.7* 45.2 24.7* 43.5 23.0* 44.9 24.4* Clinical care specific skills 30.6 41.8 11.2* 54.6 24.0* 51.7 21.1* 56.0 25.4* ANC specific skills 27.5 33.6 6.1 62.9 35.4* 58.7 31.2* 66.8 39.3* *Difference is significant; p<.01 (two sample t test) Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 16 A separate analysis to evaluate effectiveness of the training was conducted for some of the key indi￾cators which are associated with Project NOVA training objectives. Selected key focus points of the Project intervention activities and detailed analysis of selected questions stratified by types of interventions are presented in the Table 9. Table 9. Provider Performance of Selected Actions of Antenatal Care (% of HP nurses performing the skill correctly) % of providers performing the skill successfully Base￾line End-line Control Total Intervention Total T&E Full NOVA Intervention Score Score Dif. Score Dif. Score Dif. Score Dif. Collecting medical anamnesis (history) for the 1st visit 28.3 48.5 20.2 39.0 10.7 26.0 -2.3 45.6 17.3 Measuring blood pressure 88.8 97.0 8.2 99.1 10.3 98.1 9.3 100 11.2* Examining legs for edema, redness or vari￾cose veins 41.6 45.5 3.9 69.7 28.1 67.3 25.7 71.9 30.3 Palpating uterus to detect the height, mea￾suring uterine height and abdomen circum￾ference 18.6 33.3 14.7 69.7 51.1 67.3 48.7 71.9 53.3 Listening to the fetal heart rate (18+ weeks) 13.7 18.2 4.5 56.9 43.2 40.4 26.7 71.9 58.2 Informing on danger signs 19.7 30.3 10.6 57.8 38.1 57.7 38.0 57.9 38.2 Orients woman on breastfeeding (28+ weeks) 19.0 42.4 23.4 52.3 33.3 46.2 27.0 57.9 38.9 *Difference is significant; p<.01 (two sample t test) As shown in Table 9, the findings demonstrate significantly improved performance in ANC specific skills, such as listening to the fetal heart rate and/or palpating the uterus, followed by counseling (i.e. informing women of danger signs and/or educating them on breast￾feeding). The detailed analysis between different types of interventions shows that community mobilization seems to increase the impact on nurse performance partially effected by improved physical conditions of rural health posts and better relationships with supervisory facility. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 17 Provision of Postpartum, Newborn and Infant Care Table 10 presents average aggregate increases in community nurses theoretical knowledge in PP, newborn and IC following NOVA’s SMCS training – namely Module 5 and 6 – assessed individually by the knowledge evaluation questionnaires before and after the training. On average, community nurses demonstrated a 26% increase in PP and newborn care and a 14% increase in knowledge of IC. Similar to the knowledge in the areas of ANC, rural community nurses from the first-year marzes (Shirak and Tavush) showed smaller changes (12.2% and 22.2% respectively) in PP and newborn care as compared with the nurses from second-year marzes (Gegharkunik and Kotayk), who de￾monstrated more than 30% increase. In infant care knowledge, how￾ever, this pattern was different: the greatest knowledge increase was obtained by nurses from Gegharkunik (19.3%) and the lowest increase was observed in the co￾hort of nurses from Ko￾tayk (8.1%). Evaluation of change in nurses’ performance was also conducted for PP, newborn and IC. The assessment team measured provider performance using a checklist of 44 items pertaining to CPI (9 items), counseling skills (10 items), general clinical skills (8 items), PP/newborn care specific skills (5 items) and IC specific skills (12 items). Figure 6 shows increases in performance of nurses in the first and second years of the program activities. There was a significant 22% increase in the score of nurses participating in the Project NOVA interventions as compared with the baseline. Increase in the performance of nurses who did not participate in the Project NOVA direct interventions was 9% (p<0.01). Figure 6. Performance of Nurses in PP/IC Following Project Interventions (% of Highest Possible Score Detailed analysis between full and li￾mited NOVA intervention types showed that nurses undergoing nur￾se training and equipment provision showed overall an 18% performan￾ce improvement as compared to the baseline, while nurses receiving full project interventions showed a 24.9% increase in performance. As in the case of ANC, the increase was different in the end-line assess￾ment. Nurses receiving the full ran￾ge of interventions achieved the highest performance score of 60.3%. The marz-level analysis of the community nurses performance in PP/newborn and IC is presented in the Table 11. Positive changes in nurse’s performance following Project NOVA interventions were significant for all Project-supported northern marzes, but the differences varied across the marzes from 15.7 in Kotayk to 24.8 in Shirak. Table 10. Mean Knowledge Test Scores of Nurses in Postpartum, Newborn and Infant Care Marz (n) Postpartum & Newborn Care Infant Care Pre-test score Post-test score Dif. Pre-test score Post-test score Dif. Shirak (n=46) 74.60 86.8 12.2* 70.9 83.2 12.3* Tavush (n=34) 73.1 95.3 22.2* 76.5 88.8 12.3* Gegharkunik (n=50) 58.2 90.9 32.8* 72.8 92.1 19.3* Kotayk (n=29) 56.0 98.5 42.5* 87.7 95.9 8.1* Overall (n=159) 65.7 92.1 26.3* 75.8 89.5 13.8* *Difference is significant: p < 0.01 (Paired sample t test) 35% 57% 44% 0% 10% 20% 30% 40% 50% 60% Baseline End-line Intervention End-line Control Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 18 Analysis of specific skills areas that contributed to increases in performance score showed that specific PP care skills made the greatest contribution to the overall improvements in performance in the provision of PPC in general (Table 12). Table 12: Mean Performance Scores of Nurses in Postpartum, Infant and Newborn Care by Skills Area (% of Highest Possible Score) Skills Area Baseline Score Control Total Interventions Total T&E Full NOVA Intervention Score Dif. Score Dif. Score Dif. Score Dif. Client – Provider Interactions 39.2 38.1 - 1.1 57.0 17.8* 51.9 12.8* 61.6 22.4* Counseling 36.2 44.1 7.9 56.8 20.6* 54.9 18.7* 58.6 22.4* Clinical Care Specific Skills 20.0 26.6 6.6 45.6 25.6* 40.2 20.2* 50.5 30.5* Specific Infant Care Skills 42.2 52.9 10.0* 58.4 16.2* 57.4 15.1* 59.3 17.1* Specific Postpartum Care Skills 38.0 60.0 22.0* 77.1 39.1* 71.9 33.9* 81.7 43.7* * Difference is significant (p≤.01) - independent sample t test (equal variances assumed) Table 13 shows performance scores for selected questions that were the primary focus areas in the SMCS training. The table indicates performance improvements in all key areas of clinical skills and counseling practices for nurses participating in the Project events. The greatest increase in perfor￾mance was observed in providing PP clinical care and counseling on nutrition. Table 11. Mean Performance Score of Nurses in Postpartum, Newborn and Infant Care by Marz (% of Highest Possible Score) Baseline Score (n) End-line Control Total Interventions Total T&E Full NOVA Intervention Marz Score (n) Dif. Score (n) Dif. Score (n) Dif. Score (n) Dif. Shirak 37.8 (114) 45.2 (20) 7.4 62.6 (39) 24.8* 56.1 (18) 18.2* 68.3 (21) 30.5* Tavush 31.3 (38) 45.8 (6) 14.5 55.0 (20) 23.7* 54.8 (10) 23.5* 55.2 (10) 24.0* Geghar￾kunik 34.3 (77) 43.6 (6) 9.3 55.7 (33) 21.4* 54.1 (15) 19.8* 57.1 (18) 22.8* Kotayk 34.6 (32) 25.0 (2) -9.6 50.3 (18) 15.7* 46.7 (9) 12.1 53.8 (9) 19.2* Overall 35.4 (261) 43.9 (34) 8.4 57.2 (110) 21.8* 53.6 (52) 18.2* 60.3(58) 24.9* * Difference is significant (p≤.01) - independent sample t test (equal variances assumed) Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 19 Table 13: Provider Performance of Selected Actions of Postpartum, Newborn and Infant Care (% of HP nurses performing the skill correctly) Clinical Practice Baseline End-line Control Total Interventions Total T&E Full NOVA Intervention Score (n) Score (n) Dif. Score (n) Dif. Score (n) Dif. Score (n) Dif. Washes hands 5.4 (14) 6.1 (2) 0.7 21.1 (23) 15.7* 17.6 (9) 12.2* 24.1 (14) 18.7* Asks about danger sighs 45.0 (117) 67.6 (23) 22.6* 80.0 (88) 35.0* 75.0 (39) 30.0* 84.5 (49) 39.5* Measure blood pressure 23.8 (62) 26.5 (9) 2.7 55.0 (60) 31.2* 49.0 (25) 25.2* 60.3 (35) 36.5* Inspects/palpates abdomen 16.6 (43) 29.4 (10) 12.8* 57.8 (63) 41.2* 45.1 (23) 28.5* 69.0 (40) 52.4* Examines breast, asks about lactation problems 54.0 (141) 76.5 (26) 22.5* 88.1 (96) 34.1 * 84.3 (43) 30.3* 91.4 (53) 37.4* Examines vaginal discharge 18.6 (48) 44.1 (15) 25.5* 65.7 (71) 47.1* 62.7 (32) 44.1* 68.4 (39) 49.8* Discusses breast￾feeding 87.1 (217) 91.2 (31) 4.1 98.2 (108) 11.1* 98.1 (51) 11.0* 98.3 (57) 11.2* Discusses nutrition 47.8 (122) 79.4 (27) 31.6* 82.6 (90) 34.8* 74.5 (38) 26.7* 89.7 (52) 41.9* Discusses vaccination 66.7 (174) 61.8 (21) 4.9 76.4 (84) 9.7* 75.0 (39) 8.3 77.6 (45) 10.9 Discusses birth spacing and contraception 5.4 (14) 0.0 (0) 5.4 31.8 (35) 26.4* 30.8 (16) 25.4* 32.8 (19) 27.4* * Difference is significant at p<0.05 Women’s Attitudes, Practices and Experience A series of the household interviews was conducted to measure the impact of Project activities on women’s attitudes, practices and experiences in the area of RH/MCH in four northern Armenian marzes. Although it is obvious that many of the impact indicators outlined in this analysis are not directed to measure Project NOVA outcomes only, they give general implications of the Project on the health of the population of the selected marzes. In addition to collecting data on internal Project monitoring indicators, the household interviews sought to assess women’s perspectives on RH/MCH services, their access and utilization of these services and their awareness of health-related issues. Women with a child under 12 months of age were iden￾tified by district pediatric records and selected random￾ly via a probability-proportional-to-size sampling techni￾que (see Methodology section). There were some dif￾ferences * in the sampling techniques utilized at the follow-up data collection; however, there is an assump￾tion that the sampling differences had minimal influ￾ence on the sample. Table 14 presents details of the number of women interviewed during baseline and end-line assess￾ments, disaggregated by marzes. Overall 346 women participated in the baseline household survey and 403 participated in the end-line. * For the baseline assessment the sampling was done using delivery journals from the maternities, while in the end-line assessment pediatric journals from the district pediatric polyclinics were utilized. The change of sampling frame was based on simplifying the logistics of the survey implementation. Table 14. Number of Women Interviewed by Marz Marz Baseline End-line Total Shirak 97 90 187 Tavush 53 96 149 Gegharkunik 99 104 203 Kotayk 97 113 210 Overall 346 403 749 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 20 There were no significant differences in key socio-demographic characteristics: age, educational attainment, number of children, and marital status for baseline and end-line interview groups (Table 15). The mean age of women interviewed at baseline and end-line was 24 years. The average num￾ber of children in the family was 1.8 children (at baseline and end-line). With respect to the number of children, 44.9% of the baseline sample and 45.8% of the end-line sample reported having one child. Vast majority of women interviewed had completed secondary technical education (84.8%; 82.6%) with only 0.6% respondents at baseline and 0.2% at end-line had attended no school (all were from Shirak marz). The prevention of complications during pregnancy and childbirth and successful pregnancy outcome for both mother and newborn is associated with the quality of ANC, the number of visits during pregnancy and the timing of the first visit. As reported in the household interviews, the average number of ANC visits during pregnancy increased from 5.9 visits at baseline to 6.3 visits at end-line, indicating that the average number of visits increased insignificantly (p > 0.05) by 0.4 visit per pregnancy for all four northern marzes. In terms of number of ANC visits, the MOH has adopted the WHO strategy of at least four antenatal visits for all pregnant women at specific times during which interventions beneficial for the health of the mother and/or the baby can be administered* with at least 6-8 ANC visits recommended for a normal pregnancy. ANC, also known as prenatal care, is the list of interventions that a pregnant woman receives from organized healthcare services. The purpose of ANC is to prevent or identify and treat conditions that may threaten the health of the fetus/newborn and/or the mother, and to help a woman approach pregnancy and childbirth as positive experiences. To a large extent ANC can contribute greatly to this purpose and can, in particular, help provide a good start for the newborn child.† According to WHO, ANC visits should include, at a minimum, the measurement of blood pressure, testing of urine for bacteriuria and proteinuria, and blood tests to detect syphilis and severe anemia. * WHO Monitoring and Evaluation, Antenatal Care http://www.who.int/reproductive-health/global monitoring/anc.html Accessed September 2008. † WHO What is the efficacy/effectiveness of antenatal care? http://www.euro.who.int/HEN/Syntheses/antenatal/20031223 Accessed September 2008. Table 15. Percentage Distribution of Women by Marz and Socio-demographic Characteristics Baseline End-line Shirak Tavush Gegh-k Ko￾tayk Overall Shirak Ta￾vush Gegh-k Kotayk Overall Age > 20 years 11.7 15.1 9.2 10.3 11.1 10.0 8.3 3.9 8.0 7.5 20-24 years 57.4 47.2 56.1 47.4 52.6 51.1 59.4 57.3 53.6 55.4 25-35 years 27.7 34.0 31.6 39.2 33.0 37.8 32.3 35.9 37.5 35.9 36-44 years 3.2 3.8 3.1 3.1 3.2 1.1 0.0 2.9 0.9 1.2 Mean 23.4 24.4 23.8 24.6 24.0 23.5 23.7 24.5 24.3 24.0 Education Secondary 80.6 84.9 89.8 83.5 84.8 82.2 86.5 74.0 87.6 82.6 Higher 15.1 5.7 8.2 16.5 12.0 16.7 13.5 26.0 11.5 16.9 No education 2.2 0.0 0.0 0.0 0.6 1.1 0.0 0.0 0.0 0.2 Number of Children One child 54.6 43.4 41.7 39.2 44.9 52.2 46.3 43.3 42.5 45.8 Two children 27.8 35.8 41.7 44.3 37.6 33.3 40.0 36.5 41.6 38.1 ≥ 3 children 17.5 20.8 16.7 16.5 17.5 14.4 13.7 20.2 15.9 16.2 Mean 1.7 1.9 1.8 1.9 1.8 1.7 1.7 1.8 1.8 1.8 Marital status Married 96.9 100.0 99.0 98.5 86.3 100.0 97.9 99.0 99.1 99.0 Never married 1.0 0.0 0.0 0.6 12.2 0.0 1.1 0.0 0.0 0.3 Divorced/widowed 2.1 0.0 1.0 0.9 0.6 0.0 1.1 1.0 0.9 0.8 * The percentages do not sum to 100% due to rounding Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 21 The first ANC visit during the first trimester of gestation, also known as early coverage, is another indicator which shows overall utili￾zation of ANC services and population kno￾wledge of safe pregnancy. The MoH re￾commends that the first visit takes place by 12 weeks of gestation. The assessment showed that mean gestational age at the time of the first ANC visit was 15.6 weeks at the baseline which decreased to 15.2 weeks at the end-line (not significant). The Project did not have any impact on increasing the early coverage for ANC. Data indicate no change in early coverage for ANC. Slightly more than half of the women surveyed went for their first ANC visit during their first trimester (53.8% at baseline and 55% at end-line). However, the vast majority of women interviewed did receive their first ANC within their first 16 weeks of gestation (81.3% and 82.4 respectively) (Table 16). Lack of recourses for transportation and general tendency to delay care if there is no urgent health problem were mentioned as main reasons for delay￾ing ANC visits (Table 17). These find￾ings did not differ in baseline and end￾line assessments. However, there was an increase in the percent of women reporting lack of information as an ex￾planation for postponing care. Positive change was observed in the percent of women valuing preventive healthcare, (i.e. 37% of people reported that peop￾le generally delay healthcare due to no urgent need of interventions at the baseline as compared with 24% in the end-line evaluation). Women were asked if they remembered being informed by their PHC provider about pregnancy danger signs. There was no difference in the percent of women who recalled be￾ing informed of the pregnancy danger signs. Although the results of the performance assessment showed an almost 40% increase in percent of HP providers informing women of the pregnancy danger signs, this pattern was not confirmed by women. Moreover, a smaller percentage of women from Shirak and Tavush marz recalled that their provider spoke to them about the pregnancy-related danger signs in the end-line assessment, as compared with the baseline (Table 18). A significant difference was observed in the percent of women reported being advised on issues related to nutrition 65.9% at the baseline versus 80.4% in the end-line assessment (p<0.01). Table 16. Percentage of Women with Early Antenatal Care Coverage by Marz Within first 12 weeks Within first 16 weeks Marz Baseline End-line Baseline End-line Shirak 55.2 52.9 82.3 79.3 Tavush 62.3 48.9 83.1 80.8 Gegh-k 41.9 52.0 71.6 76.0 Kotayk 56.7 63.3 86.6 89.0 Overall 53.8 55.0 81.3 82.4 * The differences are not significant Table 17. Primary Reasons for Not Seeking ANC During the 1st Trimester Baseline End-line Lack of recourses for transportation 38% 31% People tend not to go if nothing goes wrong 37% 24% Lack of knowledge, information 6% 15% Traditional beliefs 4% 8% Some women do not know that they are pregnant 2% 5% Dissatisfaction with quality of care 2% 2% Other 10% 15% Total 100% 100% Table 18. Percent of Women Repor￾ted Being Informed About Pregnancy Danger Signs Marz Baseline End-line Shirak 62.9% 54.7% Tavush 83.0% 78.9% Gegharkunik 71.0% 71.3% Kotayk 64.6% 66.0% Overall 68.7% 68.0% Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 22 Utilization of Primary Healthcare Services One of the key objectives of Project NOVA is to increase utilization of PHC services for ANC, PP and IC as a means to improve MCH of the rural Armenian population. The assessment team determined the utilization of PHC services by reviewing patient records and facility journals. Patient medical records, facility journals, record/journal keeping practices and regulations differed among facilities at the baseline.* Figure 7. Utilization of Health Posts for Antenatal Care At baseline, the patient records review re￾vealed that rural PHC facilities, overall, had rather low annual patient load and a very small number of pregnant and PP women visited rural HPs or MAs for ANC/PPC. Generally, rural HP nurses identified and re￾gistered pregnant women, then referred them to the ob/gyn or FD from the super￾visory healthcare facility [MA, PC or Women Consultation Centers (WCC)]. The end-line assessment demonstrated an almost five￾fold increase in overall ANC visits to ru￾ral HPs compared to baseline (Figure 7). Further analysis showed a four-fold increase in ANC visits to HPs in the communities with training of nurses and provision of basic equipment and supplies (T&E) – from 10.4 visits to 42.5, while in communities receiving full Project NOVA interventions at the rural community level consisting of nurse training, provision of equipment and supplies, and CPH activities, the increase was almost six-fold – from 5.4 to 33.2 (Figure 8). Figure 8. Health Post Utilization for Antenatal Care by Project Intervention An insignificant increase in the number of ANC visits was observed in the com￾munities which did not have any direct interventions from the Project (control). This increase can be attributed to several factors including improved record-keeping practices after baseline assessment, training of HP supervisory facilities’ managers and overall improvement of the health system. * Project NOVA, Reproductive and Child Health Services in Armenia; Baseline Assessment of Primary Health Care Facilities in Gegharkunik and Kotayk Marzes, October, 2006. 8.2 6.7 39.2 9.9 0 5 10 15 20 25 30 35 40 45 Intervention Control Baseline End-line 6.7% 10.4% 5.4% 9 9% 42.5% 33.2% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Control Training & Equipment Full NOVA Intervention Baseline End-line Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 23 Conclusions The results of Project NOVA’s internal evaluation provided crucial highlights of its activities, including its success and limitations. It also provided valuable information for decision-making for the next years of implementation and can serve as a reference to similar projects which might be implemented in the country. In addition the results of the end-line assessment provide background data on the current situation in the healthcare facilities and rural community nurses performance in the northern Armenian marzes - Shirak, Tavush, Gegharkunik and Kotayk. The comparative analysis also present information on best achievements following Project inter￾ventions. Provider Performance Provider performance is influenced by many confounding and interrelated factors. Project NOVA key activities were multifaceted focusing on improving rural community nurses performance through simultaneously introducing supportive supervision at supervisory facilities, enhancing the physical environment of HPs, improving knowledge and skills in ANC, IC, newborn and PPC by trainings related to these topics. Job description. Although community nurses are better positioned than any other rural PHC pro￾viders to offer basic ANC and PPC, their original scope of practice for these activities was limited. Based on the MoH Order, the job description of those community nurses who completed NOVA’s training in SMCS was amended to include additional activities specifically related to ANC and PPC* . Supportive Supervision. An important initiative of the Project was increasing not only the number (from 0.9 visits per month to 1.4) but also the duration (from 2.5 hours to 3.5 hours) of supportive supervisory visits to the HPs. Supportive supervisory visits considerably contributed to the improved overall performance of rural community nurses as they were offered additional opportunities to re￾ceive technical updates, clarify specific instructions, discuss difficult cases and attend to the patients together. Facilities, Supplies and Equipment. As part of its strategy, Project NOVA provided equipment and supplies to 178 HPs in the northern marzes. As a result, mean number of basic equipment and supplies at rural HPs increased from 10 to at least 15 items out of a possible 20. In addition to this, the Project equipped HPs and the rural communities with techniques to leverage funds for equip￾ment and HP maintenance through creating better links between HPs and their supervisory health￾care facility. Through NOVA’s CPH, a total of 65 rural HPs were renovated with 45-70% in-kind and financial contribution from rural communities. This built overall community ownership of the changes thus increasing their sustainability. It also contributed to both improved provider performance and increased utilization rates. Training. Analysis of the Project’s training and evaluation data showed significant increases in rural healthcare providers’ technical knowledge and overall performance in key areas: ANC, PP/newborn and IC. On average, nurses, who participated in Project NOVA interventions demonstrated a 20% increase in knowledge and skills. Nurses, whose communities were involved in the full Project NOVA intervention showed greater increases in ANC performance compared with nurses receiving limited Project intervention at 28.3% vs. 22.36% (p<0.01) respectively. Although the overall increase in knowledge among community nurses was noteworthy throughout all key technical areas, advances in learning for PP/newborn care were greater (26%) than those for ANC (22%) and IC (14%). Aggregate performance assessment of community nurses following Pro￾ject NOVA interventions also demonstrated almost two-fold improvements in both ANC and PP/new￾born/IC areas reaching the highest possible performance scores of 52% and 57% respectively. Among all aspects of ANC and PPC provision, the highest increases in performance were observed in specific ANC and PPC and general counseling skills. Following the training, community nurses were significantly more likely to be able to palpate the uterus to detect the height, measure uterine * MoH Order No. 1373-A (27.12.2005) Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 24 height and abdomen circumference; examine vaginal discharge, discuss PP danger signs and coun￾sel women about appropriate nutrition. Although the greatest increases were observed in care-specific skills, some improvement was also observed in general nursing skills. Community nurses involved in full Project NOVA interventions achieved the greatest increases in performance of CPI and counseling. This may have been due to the fact that more one-on-one work with the healthcare providers was conducted by Project NOVA’s CPH community mobilization team by providing support to the community nurse in organization of health talks and providing patient education. In rural healthcare services in Armenia, routine PP care is integrated with IC. Generally, the first IC nurse visit in the PP period takes place within the first three days after discharge from the maternity unit. Community nurses are responsible for IC and make these visits. The findings of this research demonstrate that prior to nurse training in SMCS, few nurses were focusing on issues related to PPC, although PPC is part of the nurses scope of practice. The study findings demonstrate positive changes in the performance of rural community nurses in certain areas of ANC and PPC provision across all groups involved in the end-line assessment (control, limited Project NOVA intervention and full Project NOVA interventions). While positive changes in the groups which participated in any of the Project interventions can be attributed to direct interventions, the changes in performance of nurses who did not participate in NOVA trainings can be due to several factors, including: • Indirect exposure to the training through word of mouth, spread of information and experience sharing between the nurses who participated NOVA’s interventions and those who did not. • Indirect exposure to Project NOVA interventions through physicians from supervisory facilities who also experienced different aspects of NOVA interventions and integrated them into their routine clinical and supervision practices. For example, family physicians who underwent training in Key Reproductive Health Competencies may have provided on-the-job training and coaching of community nurses from the HPs under their supervision and ambulatory care managers who received management training could have applied newly-acquired approaches in their jobs, which would include better management of HPs under their supervision. This may have lead to better performance of nurses in some clinical and non-clinical aspects of care. • The nurses became accustomed to the data collection instruments after the baseline and potentially could perform better in the end-line assessment. Women’s Attitudes, Practices and Experience The prevention, timely identification, and management of complications during the pregnancy and PP periods depend not only on the quality and accessibility of healthcare services, but also on women’s knowledge, attitudes and practices. This includes prompt initiation of and attendance at ANC periodically throughout the pregnancy. Because Project NOVA had a relatively short time for programmatic interventions, population level changes in regard to women’s knowledge, attitudes and practices was not observed. According to women interviewed, the average number of ANC visits during pregnancy for both complicated and un-complicated cases did not change (5.9 visits at baseline vs. 6.3 visits at end-line). Early coverage for ANC is another important indicator of utilization of ANC services and population knowledge of safe pregnancy. Project NOVA measured average pregnancy term and observed no significant change in gestational age at first ANC visit 15.6 weeks at baseline to 15.2 weeks at end￾line. This fluctuation is not significant and cannot be attributed to Project interventions. Only half of women applied for ANC during their first trimester of pregnancy (53% at the baseline and 55% at the end-line). Four out of five pregnant women made their first ANC visit within 16 months. When asked, women named ‘lack of financial resources for transportation’ as their main reason for not coming for ANC. A positive shift in women’s attitude was suggested by responses to the reasons for not see￾king ANC among rural dwellers. Prior to the Project, 37% of women reported that ‘people tend not to go (for ANC) if nothing goes wrong’ while 24% gave this answer at end-line. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 25 Women are still not adequately informed by the healthcare providers on the pregnancy-related dan￾ger signs. Although the results of providers’ assessment demonstrated almost three-fold raise in percent of healthcare providers informing women on danger signs during pregnancy, women’s over￾all awareness of pregnancy danger signs was low. Utilization of primary healthcare services for antenatal care The project was able to meet its key goal to increase utilization of PHC facilities in rural areas. Moreover, Project NOVA’s impact on the increased use of rural health posts for ANC in northern Ar￾menian marzes was significant, leading to an average five-fold increase in the utilization of services. This increase is correlated with the depth and extent of Project NOVA involvement, especially at the community level. Rural HPs receiving the full scope of Project interventions – training of community nurses, provision of essential equipment and supplies, and the CPH initiative – had the greatest increase (six-fold) in the utilization of services, whereas increase in the patient flow in those HPs that received project interventions limited to nurse training and provision of equipment and supplies increased four-fold. Findings of this internal evaluation suggest that Project NOVA had a significant positive impact on the knowledge and performance of rural PHC providers in key RH/MCH areas, as well as utilization of PHC services in four northern Armenian marzes. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 27 Appendixes Appendix 1: Project NOVA Geographical Coverage by Marz and Implementation Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 28 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 29 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 30 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 31 Appendix 2: List of Selected Communities for Community Partnership for Health Interventions Year 1 Year 2 Shirak Marz Tavush Marz Gegharkunik Marz Kotayk Marz 1. Hajkadzor HP 1. Aknaghbyur HP 1. Ttou Djur HP 1. Fantan HP 2. Ani Kajaran HP 2. Enokavan HP 2. Vahan HP 2. Karenis HP 3. Musaelian HP 3. Hovk HP 3. Drakhtik HP 3. Nor Artamet HP 4. Nor Akhurian HP 4. Gosh HP 4. Astghadzor HP 4. Karashamb HP 5. Beniamin HP 5. Dovegh HP 5. Vardadzor HP 5. Aghavnadzor HP 6. Vahramaberd HP 6. Deghdzavan HP 6. Madina HP 6. Marmarik HP 7. Hatsik HP 7. Ptghavan HP 7. Hayravank HP 7. Solak HP 8. Haikavan HP 8. Debedavan HP 8. Tsovazard HP 8. Geghard HP 9. Getk HP 9. Lchkadzor HP 9. Tsaghkashen HP 9. Geghadir HP 10. Haritch HP 10. Artchis HP 10. Gegharkunik HP 10. Hatis HP 11. Saratak HP 11. Lusadzor HP* 11. Lanjaghbyur HP 12. Nor Kyank HP 12. Baghanis HP* 12. Semyonovka HP 13. Geghanist HP 13. Norashen HP 14. Lernakert HP 14. Varser HP 15. Pokr Mantash HP 15. Geghamavan HP 16. Djradzor HP 16. Tsaghkunk HP 17. Hoghmik HP 17. Shatdjrek HP 18. Vokhghi HP 18. Vanevan HP 19. Berdashen HP 19. Lchavan HP 20. Meghrashat HP 20. Aghpradzor HP 21. Nahapetavan HP* 22. Voskepar HP* 23. Jujevan HP* 24. Arevshat HP* * Communities renovated under seed grants program Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 33 Appendix 3: ANC Assessment Form CHECKLIST FOR ASSESSMENT OF ANTENATAL CARE VISIT Observer’s first name, last name: _______________________________ Team #: ___ ___ GENERAL INFORMATION Observation date: (dd/mm/yy) __/__/__ Observation start time: __:__ Facility name: ______________________________________________________ Facility type: 1. Polyclinic 2. Ambulatory/Health Center 3. FAP Facility address _____________________________________________________ ID # of the provider (the Interviewer should ensure that the number coincides with the list number foreseen for the observations). Specialization of the provider: 1. Therapeft 2. Ob/gyn 3. Pediatrician 4. Dermatologist-venerologist 5. Family Doctor 6. Nurse/midwife NOTE TO THE OBSERVER This is a young married woman of 23, first-time pregnant who comes first time to the provider and the clinic. She is approximately 7 months pregnant, has not had a prenatal care visit before. Her LMP was April 15 (supposed delivery period January 22), let the midwife determine the pregnancy period. The reason for making a late visit was her mother-in-law’s influence. By the end of counseling ask “How are the matters with me?” In other cases try to avoid giving additional information by brief answers. Got married in March 2002. Complaints • Headache • Getting plump If the observation is carried out with a real client, ask the client’s agreement to start. Do not remind the provider about steps forgotten to include. Only register steps/procedures spontaneously carried out/men￾tioned by the provider. Mark the way in which the information was collected, below. Do not remind the provider about the steps missed by her during the assessment. Register only the performed steps/protocols. If one point states two operations that are separated by “AND”, put “1” ONLY if both operations are implemented. Check one of the following available options of the data collection: 1. Information was collected through a simulated exchange and not through observation of a real case 2. Information was collected through a real-case observation Use the following guide to mark the results of your observations: 1 = Done 0 = Not done, or done unsatisfactorily 9 = Not applicable Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 34 # Item 1/0/9 1 Greets and calls woman by her name/surname and introduces him/herself if first visit 2 Washes hands with soap & water 3 Talks about the purpose of the visit and/or nature of the interventions 4 Ensures woman is in a comfortable environment 5 Asks questions and allows the woman to express herself 6 Pays attention and is interested in personal problems of the woman 7 Reviews clinic records before starting the session (if not the first visit)/makes a new record for the new client (in case of first visit) 8 For first consultation, checks about previous pregnancies: number, evolution and outcomes (only for 1st visit, for other cases write ‘9’) 9 For current pregnancy: assesses LMP (only for 1st visit, for other cases write ‘9’) 10 Correctly determines the pregnancy period 11 Asks about complains 12 In case it is possible performs medical examination (urine AND blood) (write ‘9’ if not possible to determine 13 Refers for medical examination (urine AND blood) at another facility 14 Collects woman’s medical anamnesis (only for the 1st visit) 15 Explores pulse rate 16 Explores blood pressure 17 Explores temperature 18 Gets anthropometric measurements: weight AND height (height only in case of 1st visit) 19 Examines skin and conjunctivae 20 Examines the legs for edema OR redness OR varicose veins 21 Examines mouth cavity 22a Examines thyroid (Physician) 22b Ask about thyroid (Nurse/Midwife) 23 Examines breasts 24a Examines the heart and lungs, in case of necessity refers to the relevant specialist (Physician) 24b Ask about the heart and lungs, in case of complains sends her to the relevant specialist (Nurse/Midwife) 25 Inspects abdomen for scars, pigmentation, striae 26 Palpates uterus to detect the height AND measures uterine height AND abdomen circumference 27 Performs maneuvers to detect fetal position and situation (in case of pregnancy of 28 weeks and more, in other cases write ‘9’) 28 Listens to the fetal heart rate (in case of pregnancy of 18 weeks and more, in other cases write ‘9’) 29 Verifies probable delivery date based on previous findings 30 Informs woman about the progress of pregnancy and the fetus’ health condition 31 Informs woman about her health condition and any complications 32 Informs woman on danger signs: pain, fever, bleeding and leaking of vaginal fluid 33 Orients woman for the place of delivery (hospital, contacts, transportation, etc) (in case of pregnancy of 28 weeks and more, in other cases write ‘9’) 34 Orients woman on management of common pregnancy-related afflictions 35 Orients woman on personal hygiene, nutrition, rest and general care 36 Orients woman on STI prevention, general information, risk factor 37 Orients woman on sexual life during pregnancy 38 Informs woman on positive and side effects of medicines during pregnancy 39 Informs or asks woman about iron supplementary therapy and prescribes iron and/or folic acid on as needed basis 40 Orients woman on breastfeeding, (in case of pregnancy of 28 weeks and more, in other cases write ‘9’) 41 Solicits questions to ensure client has understood 42 Schedules the next appointment according to clinic needs and woman’s convenience 43 Records all findings, assessments, diagnosis and care with client 44 Thanks client for her time Observation end time ___: ___ Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 35 Appendix 4: PP/IC Assessment Form CHECKLIST FOR ASSESSMENT OF POSTPARTUM CARE Observer’s first name, last name: _______________________________ Team #: ___ ___ GENERAL INFORMATION Observation date: (dd/mm/yy) __/__/__ Observation start time: __:__ Facility name: ______________________________________________________ Facility type: 1. Polyclinic 2. Ambulatory/Health Center 3. FAP Facility address _____________________________________________________ ID # of the provider (the Interviewer should ensure that the number coincides with the list number foreseen for the observations). Specialization of the provider: 1. Therapeft 2. Ob/gyn 3. Pediatrician 4. Dermatologist-venerologist 5. Family Doctor 6. Nurse/midwife Now let’s observe post partum care delivery. Please, perform all the actions that you usually perform during post partum care delivery. Include all usual examinations, counseling and procedures. Physical examination details can be mentioned orally without demonstration. NOTE TO THE OBSERVER Conduct this observation whenever possible through a real client-provider interaction. If there are no clients/patients at the time of the visit, conduct a simulated exchange with the following scenario: this is a young married woman of 23, first pregnancy who has gone to the nearest hospital for delivery, had a normal delivery and was discharged 2 days later. If the observation is carried out with a real client, ask the client’s agreement to start. Do not remind the provider about steps forgotten to include. Only register steps/procedures spontaneously carried out/mentioned by the provider. Mark the way in which the information was collected, below. Check the information collection method for this observation: 1. Information was collected through a simulated exchange and not through observation of a real case 2. Information was collected through a real-case observation Use the following guide to mark the results of your observations: 1 = Done 0 = Not done, or done insufficiently 9 = Not applicable Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 36 # Item 1/0/9 1 Greets and calls woman by her name or surname and introduces him/herself if first visit 2 Washes hands with soap & water 3 Ensures woman is in a comfortable environment 4 Explains purpose of the session and nature of the procedures 5 Asks questions and allows client to express herself 6 Pays attention and is interested in personal problems of the woman 7 Asks about last pregnancy and delivery: evolution, outcome, any complications (ONLY FOR 1st VISIT, FOR OTHER CASES WRITE ‘9’) 8 Asks about danger signs (bleeding, fever, excessive pain) 9 Explores pulse rate 10 Explores blood pressure 11 Explores temperature 12 Examines skin AND conjunctivae 13 Checks for legs - edema, redness and varicose veins 14 Inspects AND palpates abdomen for uterine involution 15 Examines breasts AND inquires for any lactation problem 16 Examines vaginal discharge (amount, color, smell) 17 Asks about baby’s health: feeding 18 Asks about baby’s health: sleeping 19 Asks about baby’s health: posture 20 Asks about baby’s health: skin color 21 Asks about baby’s health: breathing 22 Asks about baby’s health: fever 23 Assesses baby’s health: feeding 24 Assesses baby’s health: sleeping 25 Assesses baby’s health: posture 26 Assesses baby’s health: skin color 27 Assesses baby’s health: breathing 28 Assesses baby’s health: fever 29 Informs woman about her health condition 30 Informs woman about the baby’s health condition 31 Informs woman about potential complications of woman or baby and trains on self-assessment 32 Orients woman on breast-feeding (only for 1st visit, for other cases write ‘9’) 33 Orients woman on breast care (only for 1st visit, for other cases write ‘9’) 34 Orients woman on personal hygiene (only for 1st visit, for other cases write ‘9’) 35 Orients woman on STI prevention 36 Consults woman on sexual life 37 Consults on nutrition 38 Orients woman on hospital/clinic services (e.g. location, hours, etc), follow up visits (only for 1st visit, for other cases write ‘9’) 39 Orients woman on baby vaccination 40 Orients woman on the period between deliveries and contraception 41 Solicits questions to ensure client has understood 42 Schedules appointment/next visit according to needs and woman’s convenience 43 Records all findings, assessments, diagnosis and care with client 44 Thanks client for her time Observation end time ___: ___ Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 37 Appendix 5: Facility Inventory Assessment Form INVENTORY General Information Observer’s first name, last name______________________________ Team # __ __ Date (day/month/year)_____/_____/_____/ Start Time __:__ Facility name ________________________________________________________________ Facility type 1. Polyclinic 2. Ambulatory/Health Center 3. FAP Address of the Facility_________________________________________________________ ID # of the facility (the data collector should ensure that the number coincides with the given facility ID number list number). Inventory assessment should be implemented in all facilities. In bigger facilities offices should be considered as units. Ask the provider to show all inventory. Record only those tools /inventory that is in the facility (that is seen). Please circle only one of the options provided in the right two columns. # Item Availability Yes = 1 No = 0 A. Obligatory items 1 Physicians tape measure (flexible) 2 Thermometers 3 Stethoscopes 4 Portable sphygmomanometer w/sm., med, lg cuffs 5 Adult scale metric 6 Infant scale 7 Outpatient Surgical sets [scalpel, holders, iris, scissors/Kelly clamps] 8 Glucometer [without strips] 9 First aid kit 10 Pelvimeter 11 Obstetrical stethoscope or Doppler 12 Disinfection solution 13 Soap 14 Sterilized gloves # Item Availability Yes = 1 No = 0 15 Gauze or cotton balls 16 Injectors 17 Kitchen or stove 18 Examination gloves 19 Box for single use injectors, syringes 20 Pregnancy test Posters 21 ASTP poster on free services 22 Vaccination posters 23 Iron tablets (write quantity) Protocols 24 Nurse/midwife training modules 25 STI Syndromic management guidelines 26 MoH orders 27 UNFPA materials 28 Other health education materials B. Condition of facility Ideal condition Medium condition, operating Needs renovation 29 Electrical power 1 2 3 30 Running water 1 2 3 31 Functioning toilet (in the facility or around) 1 2 3 32 Heating system (please describe) 1 2 3 33 Windows 1 2 3 34 Floor 1 2 3 35 Shelves 1 2 3 36 Examination table 1 2 3 37 1 table and 2 chairs 1 2 3 38 Refrigerator/Freezing bag 1 2 3 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 38 C. Other Availability Yes = 1 No = 0 39 Exam light-floor based adjustable portable 40 Penlights – reusable diagnostic 41 Infant stethoscope 42 Infant sphygmomanometer 43 Spatula 44 Specula 45 Straight urinary catheter 46 Surgery threads D. Other: Drugs Write down drug quantity in small facilities. Check drug existence in PCs Availability Yes = 1 No = 0 Quantity 47 Multivitamins (write quantity) 48 Oxytocin 49 Intravenous fluids 50 Paracetamol or any other medication containing Paracetamol, e.g. Efferalgan, Panadol, Calpol 51 Negram/nalidixi acid 52 Magnesium sulfate 25% or 50% 53 Contraceptives(any) 54 Quinolons: ciprofloxacin, ofloxacin 55 Cephalosporin’s: cefixim, ceftriaxon 56 Tetracyclines: doxycycline, tetracycline 57 Macrolids: azithromycin, azatril, sumamed, erythromycin 58 Penicillins: benzatin benzyl penicillin, procaine benzyl penicillin, penicillin G, amoxicillin 59 Antifungal: clotrimazole, miconazole, fluconazole, ketoconazole 60 Antiprotozoal: metronidazole, flagyl, tinidazole, ornidazole. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 39 Appendix 6: Client Record Review Form Facility Journal Review Forms Interviewer’s first name, last name (ID)_______________________ team # __ __ GENERAL INFORMATION: Date of Visit (dd/mm/yy): _____/_____/_____ Interview start time ___:___ Name of facility ______________________________________ Type of facility 1. Health Post 2. Medical Ambulatory 3. Health Center 4. Polyclinic 5. Hospital (Delivery department/Maternity) 6. Medical Center 7. Women Consultation Center Address of Facility ______________________________________________ ID # of the facility (the Interviewer should ensure that the number coincides with the list number foreseen for other data collection instruments). ___ ___ ___ ___ ___ ___ ___ ___ 1. Number of personnel providing Reproductive Health Services NOTE: Consider all providers who deliver services at the given facility. Personnel: Men Women General practitioners Obstetrician-gynecologists Pediatricians Dermatovenerologists Family Doctors Nurses Midwives Other physicians _______________ Other personnel ________________ Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 40 II. Review of Client Records NOTE: Place an “N/A” in the cells if the records are not available and a zero “0” if there were no such services offered that month Year I II III IV V VI VII VIII IX X XI XII Total A. ANTENATAL CARE 1. Total number of women seen (from total number): 2. Number of women referred to higher level facilities because of complications B. POSTPARTUM 1. Total number of women seen (from total number): 2. Number of women referred to higher level facilities because of complication C. FAMILY PLANNING 1. (from total number): 2. Number of women referred for FP services to higher level facility D. INFANT CARE 1. Total number of children up to 1 year old seen 2. number of infants seen for immunization 3. number of infants seen for well child visit (except immunization) 4. number of infants seen for sick care Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 41 Appendix 7: Provider Interview Form Provider Interview Interviewer’s first name, last name (ID#)________________________ Team # __ __ GENERAL INFORMATION Date (dd/mm/yy) ___/___/___ Interview start time __:__ Name of the Facility _______________________________________________________ Type of the facility 1. Health Post 2. Medical Ambulatory 3. Health Center 4. Polyclinic 5. Women Consultation Center Address of the Facility ________________________________________________________ Provider ID # (Interviewer: make sure that the number corresponds to the numbers of the remaining instruments). ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ INFORMATION ABOUT PROVIDER 1.1. What are your responsibilities/position? 1. Nurse 2. Midwife 3. General Practitioner 4. Pediatrician 5. Family Physician 6. Ob/Gyn 7. Other (specify) _______________________ 1.2. How long have you worked in the health services? (WRITE NUMBER OF FULL YEARS) _________YEARS 1.3. How long have you worked in this facility? (WRITE NUMBER OF FULL YEARS) _________YEARS 1.4. How old are you? RECORD IN FULL YEARS ______ years old 1.5. Sex (DO NOT READ) 1. Male 2. Female 2. PREVIOUS TRAININGS The following questions refer to your professional education and training 2.1. When did you receive your last training in maternal or child health? __________________DATE (year) 88. Do not receive ÆSKIP TO Q. 2.6 2.2. In which area were you trained? _________________________________________ 2.3. Have you been able to use the knowledge/skills learned in the training course? 1. YesÆSKIP TO Q. 2.6 0. No 9. Do not know Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 42 2.4. Why? ______________________________________________________________ 2.5. Do you think you have the knowledge or skills necessary to perform all your responsibilities? 1. Yes 0. No 9. Do not know 2.6. Do you think you need an additional training? 1. Yes 0. No 9. Do not know 3. JOB EXPECTATIONS In this section of the questionnaire we would like to learn more about your job. 3.1. Do you have a written job description for this job? 1. Yes 0. No Æ SKIP TO Q. 3.3 9. Do not know Æ SKIP TO Q. 3.3 3.2. Please show your job description. MENTION THE RESULT. 1. The job description was shown 0. The job description was not shown 3.3. Do you know/understand what roles and tasks you have to carry out in your job? 1. Yes 0. No 9. Do not know 4. MOTIVATION/INCENTIVES In this set of questions we will ask you how you are awarded for your work. 4.1. Have you had bonuses or raises in your salary within the last 3 years? 1. Yes 0. No ÆSKIP TO Q. 4.3 9. Do not know 4.2. Was it related to your good performance at work? 1. Yes 0. No 9. Do not know 4.3. What are non-monetary incentives coming from the employer if you do your work well? MENTION ALL THAT APPLY 1. Verbal recognition 2. Written recognition 3. Uniforms 4. Free/discounted medicines 5. Equipment 6. Training courses 7. Other, please specify _______________ 8. No incentives DO NOT READ 4.4. What are non-monetary incentives which come from the client or community if you do your work well? MENTION ALL THAT APPLY 1. Verbal recognition 2. Written recognition 3. In-kind products or small gifts 4. Services in return 5. Respect in community 6. Other, please specify _______________ 7. No incentives DO NOT READ Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 43 5. ORGANIZATIONAL SUPPORT In this part of the questionnaire we would like to ask how your organization helps you to perform your job. Has your supervisor ever given you orientation towards: Yes No DK 5.1. Organizational structure of the marz health care system 1 0 9 5.2. Reporting lines of authorities 1 0 9 5.3. Organizational behavior 1 0 9 5.4. Your duties, rights and responsibilities 1 0 9 5.5. Are you able to influence the decision-making process in this facility regarding the organization of the health care service (through meetings, by voting, etc.)? 1.Yes 0. No 9. Do not know 5.6. Do you work in the same facility with your supervisor? 1. Yes IF THE RESPONDENT IS PHYSICIAN Î SKIP TO SECTION 6 0. No 9. Do not know 5.7. When was the last supervisory visit conducted to this facility? ________months ago IF THE VISIT WAS DURING THIS MONTH, NOTE 0.5 88. S/he has never visited 5.8. When the supervisor comes to supervise, what does she/he do? (READ ALL ANSWERS, CIRCLE ALL THAT APPLY) 1. Supervisor performs administrative tasks: checks registry, other papers, financial management 2. Supervisor attends patients, for ex., attends home visits, treats patients, and supervises the pregnant women's visit delays. 3. Supervisor checks environment/tools quality: for instance checks sanitarian state, cleanness 4. Supervisor solicits client feedback on services 5. Supervisor gives update on changes in procedures, clarifies instructions 6. Supervisor asks about the situations when the provider has been unable to provide health care and in case of necessity teaches how to do that 7. Supervisor suggests service quality improvement plan 8. Supervisor consults with you before making decisions 9. Other actions (specify)__________________________________________ 5.9. Has your supervisor ever made a negative remark to you in presence of a client? 1. Yes 0. No 9. Don’t know/don’t remember 5.10. How long does the visit usually take? MENTION IN MINUTES _____ minutes 5.11. How many times has your supervisor made supervisory visits to this facility in the past 2 months? _______ times 6. PERFORMANCE SELF-ASSESSMENT The following questions will reflect your opinion on different aspects of the work. How would you evaluate Very good Good Bad Very bad 6.1. Your relations with your supervisor? 1 2 3 4 6.2. Your relation with the community/your patients 1 2 3 4 6.3. The level of your professional development 1 2 3 4 6.4. Your willingness to work 1 2 3 4 6.5. Your relation with the colleagues 1 2 3 4 Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 44 6.6. If you were to assess your performance quality, how would you assess yourself with 10 score scale where 1 is the worst point and 10 is the best. 1 2 3 4 5 6 7 8 9 10 6.7. What do you think how would your supervisor assess your performance using the same 10 score scale where 1 is the worst point and 10 is the best. 1 2 3 4 5 6 7 8 9 10 THANK YOU FOR YOUR TIME! Interview end time __:__ Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 45 Appendix 8: Client Interview Client Interview Interviewer’s first name, last name (ID number)____________________________ Team # __ __ GENERAL INFORMATION Date (dd/mm/yy) ___/___/___ Interview start time __:__ Client address (village/city)________________________________________________________ Client ID INSTRUCTIONS TO INTERVIEWER READ: Good morning/afternoon. My name is _________________. I represent Project NOVA which conducts this survey together with the Ministry of Health with the aim to assess the quality of maternity health care services. Your name was randomly selected from the regional maternity home, however I would like to let you know that the inquiry is confidential, which means that your name will not be mentioned anywhere and the information provided by you will be presented only in a summarized form. You may refuse to participate in the interview or any part of it, however your participation is very important and it will help us to understand the current status of the maternal and child care in your region and provide recommendations for its improvement. The interview process will take 20 to 25 minutes. Can we start now? IN CASE OF DISAGREEMENT THANK THE RESPONDENT FOR THE TIME, FILL IN THE ‘GENERAL INFORMATION’ SECTION AND LEAVE. The questions we are going to ask you refer to your last experience with your health care provider. Please try to remember your last visit to health care facility or the health provider’s home visit. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 46 SECTION 1: INFORMATION ABOUT LAST VISIT We would like to discuss with you your last pregnancy visit and your delivery. Please be sincere and honest answering our questions. 1. Overall how many times did you visit health facility during your pregrnancy? ___________ times 99. Don’t know/Don’t remember 2. Now please remember your first visit to health care facility. What month of pregnancy did you have on your first visit to health care facility? ___________ months 99. Don’t know/Don’t remember 3. Did you see a physician or a nurse/midwife at your first ANC visit? 1. Nurse/midwife Æ Go to question 5 2. Physician 3. Both (nurse/midwife and physician) 4. What was the narrow specialization of the physician whom you saw at your first visit? 1. Therapeft 2. Ob/Gyn 3. Family Physician 4. Other (specify) _____________________ 99. Don’t know/Don’t remember 5. By which specialist were you mainly consulted during the pregnancy? MENTION ONLY ONE RESPONSE 1. Doctor/therapist 2. Doctor/Ob/Gyn 3. Doctor/Family doctor 4. Nurse/Midwife 5. Both of them 99. Don’t know/Don’t remember 6. During this pregnancy visits did the provider explain you the complications which require urgent intervention? 1. Yes 2. No Æ Go to question 8 99. Don’t know/Don’t remember 7. Please mention all complications, which were discussed with you by the health provider? ______________________________________________________________________________ ______________________________________________________________________ 8. Did the provider prescribe you iron or folic acid or poly-vitamins during this pregnancy? 1. Yes 0. No Æ Go to question 11 99. Don’t know/remember 9. Did the provider tell you about the side effects of these medications during this pregnancy? 1. Yes 0. No Æ Go to question 11 99. Don’t know/remember 10. What are the side effects? DO NOT READ THE OPTIONS, CIRCLE ALL MENTIONED RESPONSES. 1. Nausea 2. Black feces 3. Constipation 4. Other (specify) ________________________________________ 99. Don’t know/Don’t remember Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 47 11. Did the provider consult you on nutrition during this pregnancy? 1. Yes 2. No 99. Don’t know/remember 12. Overall, how would you evaluate cleanness of the facility which you visited during the pregnancy? 1. Very clean 2. Enough clean 3. Not enough clean 4. Not clean 13. Overall were you satisfied with the consultation? 1. Yes Æ Go to question 15 2. No 88. Didn’t get any consultation Æ Go to question 15 99. Don’t know/don’t remember Æ Go to question 15 14. If no, what was the reason? ____________________________________ SECTION 2: CLIENT SATISFACTION 15. How useful did you find the information given to you during your last visit? Use 1-4 scores evaluation scale where 1 is “Very useful” and 4 is “Not useful”. 1. Very useful 2. Useful 3. Slightly useful 4. Not useful 99. Don’t know/remember 16. Do you think any person other than those caring for you could see you during your exam? 1. Yes 2. No 99. Don’t know/remember 17. When meeting with the provider during your visit, do you think that other clients could hear what you said? 1. Yes 2. No 99. Don’t know/remember 18. Do you THINK the information you shared about yourself with the provider will be kept confidential? 1. Yes 2. No 99. Don’t know 19. During the last visit to the clinic, how did the provider treat you? 1. Very well 2. Well 3. Poor 4. Very poor 20. During the last visit to the clinic, how did the other staff treat you? 1. Very well 2. Well 3. Poor 4. Very poor 99. There was no other staff Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 48 21. During the last visit, did the provider give you any material to take home for reading? 1. Yes 2. No 99. Don’t know/don’t remember 22. Apart from PNC what other services are you provided in the facility? (MENTION ALL POSSIBLE RESPONSES) 1. Family Planning consultation 2. STI consultation 3. HIV/AIDS consultation 4. Screening/diagnosis of sex partners 5. Other (specify) __________________ 6. None 23. Now let’s discuss the facility which you attended. What is (are) the major reason(s) that you chose that facility? (DON’T READ THE RESPONSES, ACCEPT NO MORE THAN TWO OPTIONS) 1. Nearest to me 2. Staff provides good service 3. I like/know the staff 4. Better facilities 5. Good reputation 6. Always come here 7. Friends/relative recommend 8. Treatment charges are affordable 9. Other (specify) ______________________ 24. Overall, how do you rate the services you received at this facility? Use 1-4 scores evaluation scale where 1 is “Excellent” and 4 is “Very poor”. 1. Excellent 2. Satisfactory 3. Not satisfactory 4. Very poor 99. Don’t know 25. Give one or more major suggestion(s) that you think will improve the services at this facility. DON’T READ THE OPTIONS, CIRCLE ALL MENTIONED RESPONSES 1. Increase space 2. Improve hygiene/cleanliness 3. Improve drug supplies 4. Buy necessary equipment 5. Regularly available doctor 6. Increase number of doctors 7. Increase motivation of providers 8. Increase professional level of providers 9. Supervise providers 10. Increase working hours of the clinic 11. Community be involved in supervision/organization 12. Support to providers from supervisors and colleagues 13. Other (specify) ___________________ 26. What means of transport did you use to travel to medical facility? 1. Walking 2. Private Motor Vehicle 3. Public Bus 4. Taxi 5. Other (specify) _________________ 27. How much time (in minutes) did it take you to travel to the medical facility? (CONVERT HOURS INTO MINUTES) ___________ minutes 99. Don’t know Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 49 28. Now let’s discuss the delivery facility. Where did you deliver? 1. Nearest facility or regional maternity hospital (specify facility) ____________ 2. Marz maternity hospital (specify facility) _______________ 3. Yerevan facility (specify facility) _____________________ 4. Home 5. Other (specify) _________________ 99. Don’t know/remember 29. What is the major reason for your place of delivery choice? (MENTION ONLY ONE ANSWER) 1. Nearest to me 2. Qualified service 3. Good reputation 4. I like the staff 5. Always deliver here 6. Friends/relative recommends 7. Less expensive 8. Other (specify) ____________________ 99. Don’t know SECTION 3: FINANCIAL ACCESS As we know the medical services are always connected with some expenditures. In this section we would like to know how much you paid over all for the antenatal care, delivery and other services. Now I will read a list of medical service and I would like to ask you to specify how much did you pay for each of the services specified. Please try to remember all the costs associated with the type of service, i.e. include all payments, like buying presents, fuel, etc. INSTRUCTIONS TO DATA COLLECTORS: Write down the sum in Armenian drams. If the fee is given in other currency, recalculate translate it into Armenian drams. If the fee is provided in the form of a present, clarify the cost with the respondent and write the approximate financial equivalent. If the particular service was not used, write 99, if no money was paid, write 0. If the respondent does not remember or does not know, or does not want to respond, write 98. Service provided Cost in AMD 30. Antenatal care service with nurse/midwife (consultations, meetings with the provider) 31. Antenatal care service with the physician (consultations, meetings with the provider, etc) 32. Antenatal care: laboratory fees, tests, ultrasound, EKG 33. Delivery: midwife fee 34. Delivery: ob/gyn fee 35. Delivery: stay in the hospital 36. Delivery: services by other staff of the facility 37. Postpartum care (home visits) 38. Postpartum care (laboratory fees) 39. Postpartum care (vaccination) 40. Drugs overall (prenatal care, delivery, postpartum care) 41. Overall estimation of travel expenses for all the services? 42. Other expenses during the pregnancy and postpartum care that were not mentioned? 43. Were you requested or asked to bring a present or pay any fees by your provider or any other facility staff? 1. Yes 2. No 99. Don’t remember 44. Do you know what of the above mentioned services should have been provided to you with no charge, meaning being paid by the government? 1. Yes 2. No Æ Go to question 46 99. Don’t remember Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 50 45. Please mention the services which should be paid by the government __________________________________________________________________________ 46. Please assess your means to use medical services by 4 scores scale, where 1 is “medical services are not affordable at all” and 4 – “medical services are completely affordable” 1. Medical services are not affordable at all 2. Medical services are not generally affordable 3. Medical services are pretty much affordableÆ Go to question 48 4. Medical services are completely affordable Æ Go to question 48 47. What is the reason for services not being completely available for you and your family? ______________________________________________________________________________ ______________________________________________________________________________ __________________________________________________________________ SECTION 4: PARTNER’S INVOLVEMENT In this section we will discuss all types of assistance you received from your family during the pregnancy and delivery. 48. Who accompanied you to your first pregnancy visit to the health provider? (CIRCLE ALL POSSIBLE OPTIONS) 1. I went alone 2. My mother-in-law or other in-law 3. My partner/husband 4. My mother or other family member 5. Other person (specify) __________ 49. How did your partner/husband participate in the process of antenatal care? 1. Accompanied me at the visits to the provider 2. Helped in housework 3. Didn’t help 4. Other (specify) ______________________ 50. Did your provider give your husband/partner any instructions on how to treat you during the pregnancy? 1. Yes 2. No 51. Did your partner/husband follow the instructions of the provider on how to deal with a pregnant woman regarding rest, types of work, nutrition, other? 1. Yes 2. No 99. No information/instructions were given 52. Up to which month of pregnancy were you doing the house work? ______ month 53. Do you think a stronger inclusion of a partner/husband in the antenatal care would help pregnant women? 1. Yes 2. No Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 51 SECTION 5: SOCIO-DEMOGRAPHIC CHARACTERISTICS 54. How old are you? AGE IN FULL YEARS _______________ 55. What is the highest level of school you completed? 1. Primary 2. Unfinished secondary 3. Secondary or Vocational 4. Higher/University 5. Not attended school 56. How sufficient is your family budget to live in Armenia? 1. Significantly higher than needed 2. A little bit more than needed 3. As much as needed 4. A little bit lower than needed 5. Significantly lower than needed 57. What is your current marital status? 1. Married 2. Co-hebetating 3. Single, never married 4. Engaged 5. Divorced/separated/widowed 58. How many children do you have? __________ children 59. Pregnant women should seek antenatal care services during the first three months of pregnancy. In your opinion, what makes women delay antenatal care services later of the first three months of pregnancy? MENTION NOT MORE THAN RESPONSES 1. Lack of resources or transportation 2. Lack of knowledge, information 3. People tend not to go if nothing goes wrong 4. Some women do not know they are pregnant 5. Traditional beliefs 6. Dissatisfaction with the quality of care 7. Religious reasons 8. Other (specify) Now please tell us who was the PH care provider whom you first attended to for your antenatal care? INTERVIEWER: WRITE DOWN THE PROVIDER’S NAME. FIND OUT HER ID # WITH THE SUPERVISOR AND FILL IN THE FIRST PAGE 60a. Name of the Health care provider _________________________________________ 60b. Location of the attended facility ___________________________________________ 60c. Position of the health care provider 1. Nurse/midwife 2. Family Doctor 3. Ob/gyn 4. Therapist 60d. Code of the provider THANK YOU! Interview end time ___:___ Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 53 Appendix 9: Consent Forms Informed Consent for Observation of Health Care Provider’s Performance in Care Delivery Good morning/afternoon. My name is_________________ (I am physician) and also present are ____________________. We represent the USAID-funded Project NOVA, which conducts Reproductive Health Care survey together with the Ministry of Health. We would like to talk to you regarding your medical/nursing practices in maternal and child care, reproductive health, management of STIs, etc. We would like to state that the survey is not an examination or a test. We just want to assess what are practices of health care providers regarding the issues identified above. Please be as sincere as possible and remember that neither your name nor name of your facility will appear along with information we collect from you. We will present only the summary of the data on all facilities we assess. Your participation is voluntary and you can refuse to participate in the study or any component of it. Within the next hour we would be observing you and your client(s) during her/their visit(s). If there are no clients one of our data collectors will act as a client. Can we start now? NOTES FOR THE OBSERVER In case of refusal, fill in the general information section, thank the provider and go the next facility according to the list. READ Please, perform all the actions that you usually perform during antenatal/postnatal/FP/STI care delivery. Include all usual examinations, counseling and procedures which you do and pay no attention to our presence. Physical examination details can be mentioned orally without demonstration. NOTES FOR THE OBSERVER If there is a client present during the observation, read the appeal for the client on the back of this page. Project Evaluation in Shirak, Tavush, Gegharkunik and Kotayk Marzes 54 Informed Consent for Client Visit Observation Good morning/afternoon. My name is_________________ (I am physician) and also present are ____________________. We represent the USAID-funded Project NOVA, which conducts assessment of maternal and child health services in Armenia. We would like to be present during your visit/check-up to observe the interactions between the health care provider and yourself. This observation is completely voluntary for you and you can refuse to participate in the study or any component of it. Please remember, that this observation is anonymous, we are not asking or recording your name. Whatever we observe or hear today will be available only for the NOVA and we will not share it with anyone else. Only the summary of all data will be presented to the public. Can we start now? NOTES FOR THE OBSERVER In case of refusal, wait for the next client or do a role playing. READ Please, do not pay any attention to our presence and behave as if there is no one except you and the health provider in the room.