July 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Leni M. Silverstein, Dr. Ana West; Dr. Jaime Matias de Araujo Lobo and Dr. Paul Burns, also of IT Shows, Inc., participated in the in-country research for this evaluation. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011-2015, Angola Program Title: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011-2015 Angola, Final Performance Evaluation Sponsoring USAID Office: USAID/Angola Contract Number: AID-OOA-I-15-00023 Task Order Number: AID-654-TO-17-00001 Contractor: IT Shows, Inc. 1655 N. Fort Myer Drive Suite 625 Arlington, VA 22209 Date of Submission: July 20, 2017 Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola iii DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola iv ACKNOWLEDGEMENTS The evaluation team would like to thank all our key informant stakeholders for their openness and the time taken to answer our questions. Our gratitude extends to national level stakeholders, Luanda and Huambo provincial- and municipal-level government officials, healthcare facility doctors and technical staff, former SIAPS colleagues, MSH directors linked to this project, and USAID officials attached to the Angola Mission. A full list of all interviewees is included in Annex III. We would also like to express our gratitude for the support that the It Shows, Inc. team provided throughout the gestation of this report. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola v Table of Contents Acronyms .................................................................................................................................................. vii I. Executive Summary............................................................................................................................ 1 II. Project Background ........................................................................................................................... 5 A. Description of intervention strategy, project activities, theory of change and technical approach ... 5 Chart 1: SIAPS Theory of Change ........................................................................................................ 5 B. Socio-economic and political context ............................................................................................... 7 Chart 2: Map of Angola ......................................................................................................................... 8 C. Structure of Angola’s health systems ................................................................................................ 8 Chart 3: Angola Public Health Systems.................................................................................................. 9 D. Hospital and pharmaceutical sector ................................................................................................10 E. External funding sources .................................................................................................................11 Chart 4: SIAPS Funding by Program Area ............................................................................................12 F. Background information of supply chain initiatives ........................................................................12 III. Evaluation Methods and Limitations ...............................................................................................14 A. Evaluation questions suggested by USAID/Angola .........................................................................14 B. Data collection methods..................................................................................................................14 C. Evaluation constraints and mitigating actions .................................................................................15 IV. Achievements, Conclusions, Challenges and Recommendations ....................................................17 A. Technical approach..........................................................................................................................17 B. Governance .....................................................................................................................................19 C. Information......................................................................................................................................20 D. Human Resources:...........................................................................................................................20 E. Overarching Challenges...................................................................................................................23 F. Recommendations for Follow-on Projects ......................................................................................26 ANNEXES ................................................................................................................................................30 ANNEX I: Evaluation Scope of Work (SOW).............................................................................................30 Performance Evaluation of USAID/Angola SIAPS Program..................................................................30 Evaluation Matrix/Work Plan – December 2016 ..................................................................................34 ANNEX II: Methodology ..........................................................................................................................38 Data collection methods..........................................................................................................................38 Field visits selection process, schedule and visit formats ........................................................................38 Chart 4: Institutions Contacted and Number of Point of Contact Interviews.......................................38 Data collection.........................................................................................................................................39 Data analysis............................................................................................................................................39 Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola vi ANNEX III: Data Collection Instruments...................................................................................................41 Individual Interview Questionnaire ......................................................................................................41 Evaluation Questionnaire: Rapid Data Quality Verification for Warehouse Visit ................................. 43 Notes Template ...................................................................................................................................44 ANNEX IV: Sources of Information ..........................................................................................................45 Persons Interviewed.............................................................................................................................45 Points of Contact (POC), Number of Years with SIAPS ......................................................................46 ANNEX V: Evaluation Team Members .....................................................................................................47 ANNEX VI: Manuals and Documents Developed by SIAPS......................................................................48 ANNEX VII: Bibliography ..........................................................................................................................49 Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola vii ACRONYMS ACT Artemisinin-based combination therapy AIDS Acquired immunodeficiency syndrome CECOMA Central Procurement Agency for Medicines and Medical Supplies (Central de Compras de Medicamentos e Meios Medicos de Angola) DNME National Directorate of Medicines and Equipment DPS Provincial health office EUV End-user verification (survey) FP Family planning GF the Global Fund to Fight AIDS, Tuberculosis and Malaria GRA Government of the Republic of Angola HDI Human Development Index HF Health facility HIV/AIDS Human immunodeficiency virus/acquired immunodeficiency syndrome HR Human resources INADEC National Institute for Consumer Protection INLS National Institute against HIV/AIDS KPIs Key performance indicators LMIS Logistics Management Information System M&E Monitoring and evaluation MICs Middle-income countries MOH Ministry of Health MSH Management Sciences for Health NEML National Essential Medicines List NGO Non-governmental organization NMCP National Malaria Control Program PEPFAR President’s Emergency Plan for AIDS Relief PMI President’s Malaria Initiative PNME National Essential Medicines Program POC Point of contact PPMRN Public Performance Measurement & Reporting Network PSI Population Services International PSM Procurement and supply management RDT Rapid diagnostic test RH/FP Reproductive health/family planning RPM Rational Pharmaceutical Management SASH Strengthening Angolan Systems for Health SC Supply chain SCMS Supply chain management system SIAPS Systems for Improved Access to Pharmaceuticals and Services Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola viii SOW Scope of work SPS Strengthening Pharmaceutical Systems TA Technical assistance TB Tuberculosis UN United Nations UNAIDS Joint United Nations Programme on HIV/AIDS UNICEF United Nations Children’s Fund UNDP United Nations Development Program UNFPA United Nations Population Fund USAID U.S. Agency for International Development USG United States Government WB World Bank WHO World Health Organization Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 1 I. EXECUTIVE SUMMARY The U.S. Agency for International Development’s (USAID’s) Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program in Angola goal was to improve the availability and use of quality products for effective pharmaceutical service delivery and to enhance better health. The Management Sciences for Health (MSH)/SIAPS Program adopted a comprehensive approach aimed at improving and strengthening health systems overall through initiatives that cut across the healthcare spectrum. The SIAPS Program applied a health/pharmaceutical system￾strengthening approach that addressed governance, human resources, management, information, and service delivery, and focused on four major health and disease areas: HIV/AIDS, tuberculosis, malaria, and reproductive health. Differential emphases were weighted in these areas, depending upon government commitments and international funding. The SIAPS Program design and implementation aimed to create continuous, sustainable, systemic improvements in supply chain management through the building of institutional and individual capacity. The program designed innovative approaches and proposed policy and technical guidelines to support the unique requirements of disease programs in the Angolan context. By leveraging the vast and growing investments of the Government of the Republic of Angola (GRA) and the private sector towards advancing health outcomes while also enhancing citizen participation, USAID hoped to transform its partnership with the GRA to support the effective and sustainable accomplishment of the nation’s own development goals. The SIAPS Program underscored USAID/Angola’s recognition that the regular availability and use of quality assured essential medicines and other health products and services are key to achieving improved health outcomes for its priority health programs. Angola continues to lag in key development indicators. It remains in the lowest quintile of the United Nations Development Programme’s (UNDP’s) Human Development Index with a ranking of 148 out of 187 countries. On the World Bank “Doing Business” index it ranks 179 out of 187 countries. Social and economic issues such as high unemployment, cost of living, and land tenure are potentially volatile. The lack of a skilled workforce affects the growth of Angolan business and serves as a deterrent to foreign investment. Public health and health service delivery continue to present challenges. The country still has one of the highest rates of mortality among children under 5 years of age. The Ministry of Health (MINSA) reported (for 2011) a level of 195 deaths per 1,000 births. A lack of both access to and/or understanding of modern contraceptives contributes to high maternal mortality rates, and only six percent of women of reproductive age use modern contraceptive methods. Only about half of the population has access to safe drinking water (47 percent) and sanitation (53 percent). Because Angola lacks the regular availability of essential health products with the required quality and reliability, and to achieve the project’s overall objective (the well-being of the population), SIAPS was designed to ensure more sustainable country-owned pharmaceutical systems by addressing key challenges: 1) strengthening pharmaceutical sector governance; 2) enhancing local capacity for pharmaceutical management; 3) promoting appropriate information for pharmaceutical management decision-making; and 4) improving pharmaceutical services to achieve desired health outcomes. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 2 The overarching purpose of this end-of-project performance evaluation was to determine how the program performed relative to its objectives, as well as to provide an objective review of the progress made toward the expected results. The threefold aim of this review was: 1) to identify accomplishments and constraints of the SIAPS Program, with a focus on recommendations for future follow-on projects; 2) to assess the success of SIAPS in improving the planning, management, monitoring and delivery of essential pharmaceuticals; and 3) to support the GRA in its implementation of a comprehensive pharmaceutical management and delivery system. The target areas included the capital cities of Luanda and Huambo and provincial warehouses; entities involved included central, provincial, and municipal government officials, and three hospitals in Luanda. The evaluation data collection and analysis used a mixed method approach, primarily with qualitative data, and secondarily with quantitative data from SIAPS’ annual work plans, quarterly and annual reports, and monitoring data furnished to USAID. Using data from 36 in-person interviews, this report also analyzed the collaboration and barriers to cooperation among the GRA, USAID, United Nations agencies, bilateral donors and partners, and international and national nongovernmental organizations (NGOs). This report has four main sections, in line with the SIAPS evaluation Scope of Work (see Annex I). The first section provides background on the SIAPS Program, context, and objectives, and also reviews the political, economic and health contexts within which SIAPS operated throughout the five years of its existence. The contextual factors within which SIAPS was embedded must be taken into consideration to understand the full impact of what was accomplished and what obstacles were encountered. The second section provides an overview of the evaluation methods used to generate the data upon which the conclusions are based. Discussed are the data collection methods, data gathered, the analyses performed, and an enumeration of evaluation constraints. The fourth section describes the report’s key findings by evaluation question, and links the program’s achievements to the challenges and opportunities generated by the SIAPS Program approach. Lessons learned and gaps are also identified here. The fourth section also offers an assessment of what needs to be strengthened and supported in the future, and describes probable constraints to future follow-on projects. Finally, the fifth section presents overall priorities distilled from the MSH/SIAPS evaluation and presents considerations for the Chemonics Procurement and Supply Management (PMS) follow-up program. The main findings and conclusions underscore the considerable contributions that SIAPS and its collaborating partners made towards organizing, regulating and controlling the purchase, storage and distribution of Angola's medical supplies. After conducting an initial assessment, SIAPS identified key gaps in the Angolan public supply chain (SC) systems, which included: inadequate coordination of SC stakeholder efforts and the lack of a national coordination mechanism or platform; the lack of a national SC strategy; the lack of a national medicines regulatory system; suboptimal logistics management information systems (LMIS); poor availability and lack of a culture of consistent use of data to inform decision-making; and inadequate performance measurement and monitoring for improvement across all levels of the SC system. In addition, SIAPS noted that the country’s shortage of qualified, skilled, and Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 3 competent human resources at all levels of the supply chain; its suboptimal institutional and human resource (HR) capacity at the central, provincial and lower-level medical warehouses; and poor to non-existent forecasting and supply planning at the national, provincial and municipal￾levels, all resulted in an imbalance between demand and supply and frequent stock-outs or wastage. The widespread use of a passive (“push”) national public health distribution system for most health commodities, instead of a “pull” system based on real needs and demands of service delivery points, also affected SIAPS’ efforts. Throughout its five-year contract, SIAPS collaborated with other USAID implementing partners, NGOs and other local partners to assist the Angolan MOH to implement pharmaceutical management strengthening interventions at the central and lower levels of the healthcare delivery system. The government of Angola establishes the supply chain for public facilities1 . All projects must work within governmental constraints, and are unable to alter the government’s supply chain plan. In Angola, weaknesses in the supply chain system contribute significantly to the poor availability of medicines for public facilities at all levels. Thus, the SIAPS Program worked with the central government to 1. strengthen the management of essential medicines and health supplies so that more Angolans could have access to medicines, and 2. increase the use of quality assured, essential drugs and other health products and services. SIAPS also engaged with Central Drug Stores (part of the governmental supply chain) at the national and provincial levels to increase the stores’ capacity to serve their priority health programs by adopting widely recognized international standard operating procedures. However, much of their work, in terms of manuals, regulations and recommendations, has been stymied by the changes in leadership at the Ministry of Health, and its inactivity in both approving fundamental guidelines and operationalizing its governing structures. Nevertheless, SIAPS succeeded in organizing the Central Procurement Agency for Medicines and Medical Supplies (CECOMA) and the pharmacies in nine hospitals in Luanda. Had the GRA designated the public health sector as a higher priority, SIAPS would have made greater inroads into organizing and regulating the rational distribution of medicines in the country. Other challenges identified include divergences in medicine governance, human resource allocation, information flows, inadequate service delivery, and widespread dislocation of funds. Inconsistent government ownership and the full commitment of healthcare providers in project implementation were additional difficulties enumerated by evaluation respondents. The extremely cumbersome, opaque and paternalistic Angola bureaucracy at all levels of government constituted major obstacles to the robust implementation of the SIAPS Program. The Angola SIAPS Program was part of a USAID global initiative to improve healthcare service delivery in the developing world. The SIAPS approach was also part of a conscious effort on the part of USAID to gather comparative data in order to learn from fact- and experience-based 1 Private facilities and military hospitals have different procurement strategies and sources. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 4 work in the field. USAID/Angola will use this report to facilitate discussions internally and with the implementing partners about whether the project achieved its intended results, and what lessons can be taken for future, similar projects. The findings may also be used to inform future project design. Findings may also be used to enhance in-house organizational learning, and will provide important information about integrated program stakeholders, including the GRA and implementing partners. The findings of this evaluation will serve to enhance the ability of the new Chemonics/PSM project to provide critical health products and supplies, and to incorporate the types of adjustments that the SIAPS model made in order to adapt to the Angolan reality. Primary recommendations include: stronger advocacy by USAID, including urging the GRA Ministry of Health to approve and implement SIAPS policy documents and manuals (see Annex V), and to maintain USAID’s engagement with and advocacy of the government. Continued cooperation among implementing partners and stakeholders (a practice inaugurated by SIAPS) is encouraged as it presents opportunities for increasing both human and non-human resources and coordination. Additional training and resources are needed, both to maintain the systems and staffing that SIAPS put into place, and to strengthen the coordination and management of Angola’s pharmaceutical delivery system throughout the country, particularly in the provinces and municipalities. Additional recommendations are found at the end of the main report body. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 5 II. PROJECT BACKGROUND A. Description of intervention strategy, project activities, theory of change and technical approach The Systems for Improved Access to Pharmaceutical Services Project (SIAPS) aimed to build capacity in Angola to effectively manage all aspects of the country’s pharmaceutical systems and services. To accomplish this, SIAPS took an evidence-based, systems strengthening approach that was rooted in collaborative relationships with all partners and clients at the local, national, regional, and global levels. Ultimately, SIAPS would be considered successful if the Program could demonstrate, not only outcomes, but also evidence of its sustainability. The SIAPS theory of change is presented in Chart 1 below and shows a comprehensive set of dynamic relationships among the five health systems building blocks (governance, human resources, information, financing, and service delivery) with a medical products overlay. This framework provided technical focus and helped to identify substantive areas of concern and the appropriate corrective measures. As is depicted below, an important aspect of the approach is monitoring and evaluation of the project’s performance. By using this overall M&E plan to help guide the alignment of SIAPS’ anticipated results with strategic plans in an iterative process, the Program proposed to provide managers and decision makers with the data required to make informed decisions and monitor progress towards attainment of results. Chart 1: SIAPS Theory of Change Source: SIAPS, 2013 The SIAPS team determined that the best approach to the wellbeing of the population in Angola was to provide technical assistance in four priorities areas: Strengthen Pharmaceutical Sector Governance; Enhance Local Capacity for Pharmaceutical Management; Promote Information for Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 6 Pharmaceutical Management Decision-Making; and Improve Pharmaceutical Services to Achieve Desired Health Outcomes. Because Angola still lacks the regular availability of essential health products with the required quality, SIAPS was also designed to ensure more sustainable, country-owned pharmaceutical systems. Reflecting its theory of change, SIAPS worked with the Angolan central government to strengthen the management of essential medicines and health supplies to provide improved access for citizens to medicines. Its aim was to increase the distribution and use of quality￾assured essential drugs and other health products and services. SIAPS also focused on central drug warehouses at the national (Luanda) and provincial (Huambo) levels to increase their capacity for predicting the flow, demand and safe storage of medicines for its priority health programs: HIV/AIDS, tuberculosis, malaria, and reproductive health. In order to truly advance development outcomes in Angola, the SIAPS approach was integrated across sectors to reinforce the inter-sectoral relationships that are required for socioeconomic progress. Due to Angola’s particular country context and the need to be selective and focused, SIAPS concentrated on the interconnectedness of sustainable advances in health systems strengthening and governance. By leveraging the vast and growing investments of the Government of the Republic of Angola (GRA) and the private sector towards advancing health outcomes while also enhancing citizen participation, USAID will be able to transform its partnership with the GRA to support the effective and sustainable accomplishment of the nation’s own development goals. This strategic approach would support the GRA in pursuit of its own development goals. USAID, through SIAPS, would assist the GRA as it develops and implements policies, protocols, and programs in conjunction with non-government stakeholders. With minimal USG resources, USAID would maximize the added value that the USG can offer by leveraging its vast network of technical experts and five decades of overseas development experience. As a result, the USAID–GRA relationship would presumably mature into one that is based on facilitating and leveraging partnerships to improve the health and wellbeing of Angolans, and with broader ramifications for the USG in the larger context of bilateral affairs by adopting widely recognized international standard operating procedures. The targeted areas included those capital cities of selected provinces with provincial warehouses, as well as a number of selected municipalities. Relevant groups would include central, provincial and municipal government and authorities, as well as Ministry of Health officials, health office managers, hospital drug store staff, and others. Specific intended results included: 1. Strengthened pharmaceutical sector governance: That is, the governance capacity of all pharmaceutical sector actors, including the Ministry of Health, regulatory authorities, managers, and providers, among others. 2. Building individual, organizational and institutional capacity for pharmaceutical supply management and services; 3. Addressing the information for decision-making challenges in the pharmaceutical sector; Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 7 4. Improved pharmaceutical services to achieve desired health outcomes. The critical assumptions were that political and civil stability will generally prevail, and no major negative impacts to the country’s economic situation will occur. SIAPS interventions included technical assistance to strengthen systems in the following:  Building human resource capabilities and skills in the supply chain through pre- and in￾service capacity building.  Development and implementation of business processes and procedures in the medicine supply chain across all levels, including standard operating procedures (SOPs) and key performance indicators (KPIs).  Coordination of pharmaceutical supply chain management planning for a more rational use of resources.  Strengthening and implementation of a strong logistics management information system (LMIS) that will improve the quality of data and use of information in decision making.  Advocacy to develop and implement an effective “pull supply chain system” with the active participation of provincial warehouses and health facilities that use the commodities. B. Socio-economic and political context The Republic of Angola is located on the west coast of Sub-Saharan Africa. The country has 18 provinces, 164 municipalities and 557 communities. It gained independence from Portugal in 1975. Shortly afterward, a ruthless civil war broke out, lasting almost 30 years. The socio￾economic situation in Angola is still colored by the outcomes of the civil war, which left the country’s infrastructure in ruins, the interior heavily land-mined, and the political and economic institutions underdeveloped. While there have been large efforts to improve the country’s infrastructure, and some progress can be noted, there are still huge barriers to rebuilding the country. Only 42 percent of the population has access to potable water, only 60 percent has access to sanitation, and only about 40 percent has access to reliable electricity. Oil exports accounted for 97 percent of total exports, followed closely by diamonds and agricultural products (World Bank 2015). The worldwide decline in international crude oil and diamond prices, starting in 2008, has had a significant impact on Angola’s economy. While Angola is wealthy in natural resources, it has one of the most unequal income distributions in the world, In terms of human development (HDI 2015), Angola is ranked 149 out of 177 countries. The current population of the country is 26,433,620 as of March 2017, based on the latest United Nations estimates, equivalent to 0.35 percent of the total world population. The population density in Angola is 21 per square kilometer (55 people per square mile); 41.4 percent of the population is urban (11,047,226 persons in 2017), and the median age is 16.2 years; 65 percent of the population is under 24 years of age2 . By gender, the country is 48 2 Angola Population (2017) - Worldometers: www.worldometers.info/world-population/angola-population (accessed 3.30.17). Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 8 percent male and 52 percent female. Life expectancies are 58 years for males, and 63 years for females. Approximately 43.5 percent of the population lives on less than US$1.25 per day. The Angolan bureaucratic structure presents a difficult, complicated and personalized environment, which made it very challenging to implement the SIAPS Program. The biggest obstacle to SIAPS’ implementation was the country’s bureaucracy and rampant corruption at all levels of government. During the SIAPS Program, for example, the Minister of Health was replaced three times, and the process of creating relationships had to begin again each time. U.S. Government foreign assistance in Angola supports the Ministry of Health (MOH) to develop, strengthen, and sustain Angolan health systems, and to increase capacity within the Angolan government as well as among community and civil society organizations. USAID’s primary focus in the Angola health sector is providing integrated prevention, care and treatment for malaria and HIV/AIDS. Chart 2: Map of Angola Luanda Province: 3 site visits, 29 interviews Huambo Province: 2 site visits; 5 interviews C. Structure of Angola’s health systems Public health services in Angola are delivered at three levels: primary, secondary and tertiary, with the secondary and tertiary levels defined as specialized care. The health delivery level Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 9 corresponds to the three government levels: municipal, provincial and national (see Chart 2). The military has its own health delivery system, providing services and health commodities to its personnel and family members. In fact, the military health system is well organized and often provides a higher quality of services than the civilian health system. Increasingly, private health care facilities are available for those who can afford them. Chart 3: Angola Public Health Systems Strengthening the health systems in Angola is a fundamental need, as is indicated by the country’s poor health indicators. Maternal mortality here is one of the highest in Africa, and as of 2015 it was estimated at 460 per100, 000 live births; the under-5 mortality rate is 167 per 1,000 live births; and life expectancy is 51 years (World Health Organization [WHO] 2012). The crude birth rate in Angola has consistently decreased since 1960 from 54.4 per 1,000 persons to 45.5 per 1,000 in 2014 (WHO 2012). The major causes of death in Angola are preventable and easily treatable conditions: pneumonia, diarrhea and malaria. Another concern is the spread and containment of HIV/AIDS, which affects just over 224,000 persons, although tuberculosis (TB) and HIV co-infection has become a serious concern. HIV prevalence is low for the region, estimated at 2 percent (WHO 2012). Administrative Level Central Provincial Municipal Service Delivery Level Tertiary Care Secondary Care Primary Care Health Unit Type Central Hospital General Hospital Referral Health Center/ District Hospital Health Center Health Post Angolan Public Health Systems: Administrative and Service Delivery Levels Source: Connor 2010 Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 10 In 2015, the country was hit with a yellow fever outbreak, the first such outbreak in 30 years. The first cases were reported in December 2015 in the province of Luanda, and by May 2016 cases were reported in all 18 provinces, and confirmed in 14 provinces. As of May 2016, the MOH has reported 2,420 cases of yellow fever, of which 298 were fatal (WHO 2016). In addition, in 2015, Angola experienced a surge of malaria cases that accounted for 60 percent of hospital admissions, as well as 35 percent of child and 25 percent of maternal mortality. Following a wave of administrative reforms, the Law on Improvement of Municipal Management established in 2007the municipalities as the primary unit responsible for health service management. Each municipality has an administrator appointed by the province governor, who is in turn appointed by the president (Connor 2010). In 2008, each municipality became its own budgetary unit, and was allocated a health budget that could be spent without oversight or authorization from either the national and provincial governments. The goal was to promote and enable each health department to develop municipality strategies and operational plans appropriate to its needs and population (Connor 2010). Unfortunately, many of the municipalities lacked the infrastructure or the expertise to deliver such plans and services. The impact of the economic crisis and of decentralization has led to a decrease in health services and availability of commodities across all levels of government. D. Hospital and pharmaceutical sector Angola’s healthcare system is composed of both public and private facilities and services. According to Angolan law, the public health service is available at no cost and offers basic primary healthcare as well as specialized health services. However, Angola’s public healthcare system suffers recurrent shortages of doctors, medicines, nurses, and primary healthcare workers, as well as inadequate training and a lack of computerized (and compatible) information management systems to efficiently track historical records of patients. Access to healthcare services and to pharmaceuticals for the majority of the population is limited. The best quality health services are found in Luanda and in the other major cities of Benguela, Lobito, Lubango and Huambo. Although Angola’s National Pharmaceutical Policy has prioritized developing the local capacity for the manufacturing of basic primary pharmaceuticals, currently, the majority of medicines are imported from China, India, and Portugal. Local production is limited to Nova Angomédica, a joint venture between the Angolan Ministry of Health and a private company, Suninvest. The quality of products, prices, and services vary according to location. For example, smaller pharmacies on the outskirts of major cities tend to sell unregulated, lower cost pharmaceuticals, often from India and China, while higher quality and fully registered pharmaceuticals are more commonly available in urban centers. Counterfeit pharmaceuticals pose a major challenge for the Ministry of Health. The National Institute for Consumer Protection (INADEC) with the Ministry of Health’s National Direction for Medicines and Equipment (DNME), are responsible for enforcing registration requirements and keeping non-conforming pharmaceuticals and counterfeit products out of the market. The Central Government Medical Procurement Center (CECOMA) manages and distributes stocks for all public health institutions nationwide. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 11 The healthcare sector budget for 2016 is US$2.1 billion, 5.3 percent of the Angolan federal budget. While the local currency value of the total government budget and corresponding healthcare budget increased slightly, the real (US$) funding has declined due to currency devaluation. Government revenues started decreasing in late 2014 with the drop in global oil prices, which forced government budget reductions, estimated at 40 percent in February 2015. The severe budget cuts have also negatively impacted waste collection services, particularly in densely populated Luanda. Annual per capita health expenditures in Angola is US$355 per year, with 67 percent from the government budget for public health services and 33 percent from the private sector. The latter is largely financed by individuals, employers and private insurance. E. External funding sources A funding review showed that the weight given to each program area by SIAPS was directly related to the amount of funding allocated to malaria, HIV/AIDS, and family planning. It also became clear that much autonomy existed among the programs in terms of procurement, tracking, warehousing and distribution of drugs. This “silo effect” created a challenging environment that discouraged integration, knowledge sharing, and centralized planning. 1. Malaria: The Global Fund (GF) and the GRA have an integrated plan to buy malaria drugs; PMI and the USG have a separate program. Because of the large amounts of thefts suffered in 2009, PMI created a quasi-parallel system using a private transportation company that clears the drugs at the port of entry, and transports them directly to the provincial warehouses (not via CECOMA), based on distribution plans developed by the NMCP. Under PSM (i.e., the new USAID/Angola-funded activity that replaces SIAPS), USAID will work with the MOH to develop phased benchmarks for USAID, GF and other partners to re-integrate into CECOMA. 2. HIV/AIDS: The National Institute against HIV/AIDS (INLS) is an autonomous organization, completely separate from the other Angolan government health agencies; it also has its own warehouse to store HIV commodities. The GF provides money to UNDP to buy anti-retroviral drugs. The warehouse where the HIV medications were stored afforded less than ideal storage conditions, such as no security measures or trained staff, despite the high management fees paid by the GF. Given these conditions, the GF contracted Neofarma, a private company that hires its own workers, provides trucks, and has a secure distribution network, but this approach proved very expensive. The new Minister of Health recently decided that CECOMA should play an increased role in managing the HIV/AIDS drugs to diminish the costs, and questioned the value of a parallel warehousing system. While PEPFAR is also a major HIV donor in Angola, the evaluators did not learn about its warehousing and distribution processes and problems from evaluation informants. 3. Family planning (FP) is integrated into CECOMA, but it is a very small program. One informant explained its smaller presence by saying that the government would prefer to increase the population in the country rather than control population growth. Others said that there was neither a high demand for FP in the country nor enough trained personnel to Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 12 implement a program that involves injectable contraceptives or the insertion of intrauterine devices (IUDs). Currently, there is no GRA–approved program for family planning. Pathfinder, formerly the main source of reproductive health commodities in Angola, left the country when the program ended in December 2015. Subsequently, commodities received through the United Nations Population Fund (UNFPA) support were housed in CECOMA. Pathfinder’s distribution was concentrated mainly in Luanda; with USAID agreement, SIAPS took over the logistics of family planning commodities in Huambo. The remaining 16 provinces of the country were not served. When Pathfinder left, SIAPS also assumed responsibility for its work in Luanda. Since Pathfinder was only allocated US$ 300,000 a year to further family planning activities in Angola, family planning received the least amount of attention by SIAPS. In Angola, the annual budget for PMI is US$ 27 million; the PEPFAR budget is US$ 17 million. According to evaluation informants, both programs were very well organized and held annual planning meetings to readjust programs to the changing country context, maintained significant amounts of follow-up and supervision, and regularly checked their facilities. Chart 4: SIAPS Funding by Program Area Source: SIAPS documents and staff interviews F. Background information of supply chain initiatives In 2000, the Swedish government helped Angola develop its National Program of Essential Medicines. It created a list of essential medicines that should be available at the different levels of healthcare, from primary health centers to specialist hospitals. This function was situated in the DNME. But in 2007, WHO conducted an evaluation and identified a conflict of interest between the function of the DNME and the functions of procurement and logistics. Thus, it Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 13 recommended that procurement and logistics be separated from the DNME. So, the Angola government established Angomédica, a private/public partnership that created a national medicine deposit housed in rented warehouse space. Although the plan for CECOMA was approved in 2012, it was not implemented until 2014. This national warehouse is an operational body for storing, distributing and procuring3 medicines for malaria, TB, HIV/AIDS and FP. The quantity of the drugs available in the country at any one time is directly related to the available funding and procurement strategies for each disease. Despite the existence of defined administrative structures and the improved availability of funding for pharmaceuticals, the Angolan public supply chain management system faces notable challenges to ensure an uninterrupted availability and access to quality, essential pharmaceutical products and effective services. Gaps across all levels of the health commodity supply chain adversely impact national efforts to achieve the goals of identified health priorities, such as the elimination of malaria; achieving an AIDS–free generation by reducing HIV incidence and increasing access to care; reducing high mortality and morbidity, especially of mothers and children under 5; and ending preventable mother and child deaths4 . 3 Team interviews with CECOMA staff indicated that the organization was not charged with the procurement of medicines, and had no budget to do so. 4 Information has been incorporated from the SIAPS Final Report, Dec. 2016. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 14 III. EVALUATION METHODS AND LIMITATIONS A. Evaluation questions suggested by USAID/Angola  Technical Approach: To what extent was the technical approach used by SIAPS effective in strengthening pharmaceutical systems in Angola?  Governance: To what extent has SIAPS improved accountability, policies, norms and standards? What system changes have taken place at the national and sub-national level because of SIAPS technical assistance by program area?  Information: To what extent has SIAPS improved the collection of strategic information for decision-making and planning purposes? What key strategies, tools, and methodologies have been critical to the project’s success?  Human Resources: To what extent has the GRA used the Pharmaceutical Management tools, key operating standard procedures, quantification and planning tools, and methodologies developed or implemented by SIAPS at national and sub-national levels? What is the GRA’s perception of the value and quality of the tools and methodologies that they received? B. Data collection methods A participatory performance evaluation was designed, with a mix of data collection approaches selected to allow representative stakeholders’ participation. Semi-structured interviews, with uniquely formulated questions, were developed to facilitate the collection of qualitative data (see Annex II for a full description of methodology, and Annexes III and IV for questionnaires and instruments). The scope of work (SOW) evaluation questions required the collection and analysis of data about program operations, acquired mainly from SIAPS quarterly and yearly reports and other program-related documents furnished by USAID/Angola. The team was able to obtain some numerical data on available stocks and stock-outs of drugs at CECOMA and the Huambo provincial warehouse. In addition, interviews assessing the quality and content of interventions from the perspective of the program staff, stakeholders and other participants were secured and are a primary source of information. Qualitative data practices were used to encourage stakeholders to provide descriptive responses. Qualitative methods, primarily semi-structured interviews with stakeholders, were used to obtain information that was used to assess SIAPS Program relevance, implementation, successes, barriers and overall satisfaction. Individual, face-to-face interviews were conducted with key informants whenever possible. Open-ended questions were used and ample time was provided for the participant to comment, explain, and share experiences and attitudes. The feedback from these interviews helped to gauge participants’ satisfaction and gather recommendations for improvement. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 15 C. Evaluation constraints and mitigating actions Time in the Field: Time constraints prevented the evaluation team from visiting any municipal healthcare centers to see how they were incorporating SIAPS training procedures into the management of their pharmaceutical stocks. To mitigate the reduced time in the field, the evaluation team members divided into two groups, whenever feasible, to maximize ability to visit as many sites and interview the greatest number of stakeholders possible. Persons Interviewed: The evaluation team supplemented the number of sites and stakeholders that were originally recommended by USAID in response to comments and recommendations from those interviewed. This proved valuable, since the original list of people to be interviewed had omitted the original SIAPS Program team. Testimonies from the staff greatly enhanced the understanding of the constant barriers SIAPS employees faced in implementing the project as well as their achievements in surpassing such difficulties. Some key informants were either unavailable or refused to meet with the evaluation team. A few potential informants refused to speak with the evaluation team due to lack of prior introduction or authorization from the USAID Mission. To mitigate this constraint, the evaluation team asked for referrals from those who agreed to speak to the team. In the end, the team was able to speak with representatives from most of the principal stakeholders. Site Visit Selection: It would have been beneficial to evaluate the project in the five provinces (Luanda, Huambo, Biel, Cuna and Huila) where SIAPS worked to obtain a comparative perspective. By visiting only Luanda and Huambo, the most developed provinces where SIAPS focused the majority of its efforts, the team was unable to gauge the program’s impact in more distant provinces. In addition, we would have liked to trace and evaluate the supply chain of pharmaceutical delivery from the central warehouse (CECOMA) to, for example, the central hospital, the general hospital, referral health centers or district hospitals, and to the health centers of health posts. To address this lacuna, the team interviewed respondents who could reliably provide some notion of the efficacy of the system in its different levels of service delivery sophistication. Reliable Quantitative Data: SIAPS documents contained little quantitative data, making it difficult to establish a comparative baseline from which to measure the SIAPS Program impact on the supply chain management in Angola. The evaluation team was unable to obtain any consistent data to run an appropriate quantitative data analysis. To mitigate the lack of quantitative data, the team increased the number of individuals to be interviewed and sites to visit in order to capture reliable and robust qualitative data analysis for a broad range of individuals who, in one way or another, interacted or worked with SIAPS, and to capture a variety of programmatic perspectives. SIAPS reports provided quantitative data for the number of persons trained, which was impossible to verify since the names of participants were unavailable. Thus, no participants in Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 16 the training programs could be identified, contacted or interviewed. The team also systematically requested data from the various stakeholders who were interviewed, but was mostly unsuccessful in obtaining reliable statistical data. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 17 IV. ACHIEVEMENTS, CONCLUSIONS, CHALLENGES AND RECOMMENDATIONS Conducting an initial assessment, SIAPS identified key gaps in the Angolan public supply chain systems, which included inadequate coordination of SC (supply chain) stakeholder efforts and a lack of a national coordination mechanism or platform; the lack of a national SC strategy; the lack of a national medicines regulatory system; suboptimal logistics management information systems (LMIS); poor availability and lack of a culture of consistent use of data to inform decision-making; and inadequate performance measurement and monitoring for improvement across all levels of the supply chain system. In addition, SIAPS noted the country’s shortage of qualified, skilled and competent human resources at all levels of the supply chain, and its suboptimal institutional and human resource (HR) capacity at the central, provincial and lower level medical warehouses. Poor to non-existent forecasting and supply planning at the national, provincial and municipal levels resulted in an imbalance between demand and supply and frequent stock-outs or wastage. The widespread use of a passive (“push”) national public health distribution system for most health commodities, instead of a “pull” system based on real needs and demands of service delivery points, would also define and impact its work. Along with achievements and challenges, below are indicated the most salient constraints identified during interviews and document review that could impact the successful implementation of any future follow-on projects. Respondents had a keen awareness of a number of serious impediments that had negatively impacted the full implementation of the SIAPS project, and they expressed the belief that many impediments would likely present serious barriers in the future. The SIAPS Program adopted a four-pronged strategy to address some of these problems. A. Technical approach To what extent was the technical approach used by SIAPS effective in strengthening pharmaceutical systems in Angola? i. Goal: Strengthen pharmaceutical sector governance: the governance capacity of all pharmaceutical sector actors—including the Ministry of Health, regulatory authorities, managers, and providers, among others. ii. Achievements: The SIAPS program improved coordination among public medicine supply chain stakeholders by providing technical assistance to the National Directorate of Medicines and Equipment (DNME) to organize 20 meetings for the Logistics, Operations and Procurement Subcommittee of the Interagency Coordination Committee for Revitalization (Sub-Comissão para a Logística, Aprovisionamento e Operações [CCI/SCLAO]), in terms of agenda preparation, follow up meetings, drafting recommendations and logistics. This approach enhanced coordination and information Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 18 sharing among key local partners and enabled them to leverage resources and design complementary activities. SIAPS provided a mechanism for stakeholders, with the active participation of provincial warehouses and health facilities, to develop an advocacy plan to create and implement an effective “pull supply chain system” that use the commodities. At the national or central level, SIAPS enhanced supply chain coordination, human resource capacity building, and national quantification for HIV and AIDS, malaria and RH/FP health commodities. With SIAPS’ technical assistance, the MOH established a national team of 13 facilitators in pharmaceutical management. The team has been called upon whenever needed to build upon the technical capacity at the provincial and municipal levels. The national facilitators have trained 23 provincial technical staff in 6 provinces as trainers in pharmaceutical management. The provincial teams have trained 203 municipal level staff as trainers in pharmaceutical management, and provided continual mentoring through supportive supervisions. In 2016, SIAPS collaborated with other stakeholders to develop the first-ever National Supply Chain Strategic Plan. Unfortunately, it remains unsigned on the desk of the Minister of Health. iii. Challenges: A full technical assessment to diagnose the gaps in the system should be undertaken so that the next project can efficiently address the failures and train qualified people to carry on the work. There is no consistency in software platforms across stakeholders; a single, integrated system for the logistics, management and distribution of medical supplies would enable all participants to work together and share information. Computer information systems should be compatible. iv. Constraints: Persistent poor educational system. Across the board, the educational system is not succeeding in training the next generation of local leaders and administrators, and replicating a competent professional class. Many lack even rudimentary computer skills. Lack of trained personnel. The lack of a pool of trained professional personnel in Angola was mentioned several times by those interviewed. Lack of language proficiency/fluency in both Portuguese and English. Since the vast majority of Angolans are not proficient in English, many respondents were dismayed that many of the consultants did not speak Portuguese. They expressed the need to ensure that all consultants speak Portuguese. Furthermore, they identified a persistent need to have all relevant documents, forms, training materials and software, etc., translated into Portuguese. To build upon SIAPS achievements, greater integration and collaboration is needed among stakeholders to promote a system for the logistical training of technical staff to build capacity across the provinces; there is a clear need to continue and expand the technical training provided under SIAPS. Furthermore, it is important to hire and train local persons whenever possible to promote the sustainability of the work. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 19 B. Governance To what extent has SIAPS improved accountability, policies, norms and standards? What system changes have taken place at the national and sub-national level because of SIAPS technical assistance by program area? i. Goal: Build individual, organizational and institutional capacity for pharmaceutical supply management and services. ii. Achievements: SIAPS provided technical assistance to develop and implement business processes and procedures in medicine supply chains levels, including standard operating procedures and key performance indicators across all institutional points. At the provincial level, SIAPS created a supply chain for malaria commodities: Luanda (7 municipalities and 6 urban districts); Bié (9 municipalities); Cunene (6 municipalities); Huambo (11 municipalities) and Huila (14 municipalities). For reproductive health commodities, SIAPS worked in 11 municipalities in Huambo and provided capacity building in warehouse management for all 18 provinces. It also contributed to the preparation of the SOP manual for pharmaceutical management of HIV/AIDs commodities. SIAPS trained trainers in pharmaceutical management in 4 provinces (supported by ExxonMobil) and developed and implemented a new HIV and AIDS LMIS form. Thus, the SIAPS Program improved the pharmaceutical supply management capacity of local institutions and individuals and the ability to measure performance improvements. SIAPS built individual and institutional capacity in supply chain management through pre- and in-service training, on-the-job, supportive supervision and mentoring at the national (DNME, PNME, INLS, NMCP, CECOMA), provincial and municipal levels. iii. Challenges: The evaluation team was not able to verify the extent to which the training and use of SOPs and KPIs were in continued use at the relevant sites, except at the CECOMA warehouse. In addition, it was unclear how much of the training would be retained by staff and passed on to others, since few reinforcement trainings were offered. Continual supportive supervision and mentoring at all levels is fundamental for reinforcing such gains. iv. Constraints: Lack of infrastructure. Much of the country, with the exception of the capital, Luanda, has unreliable power supplies, inadequate health facility structures and infrastructure, and secure and well-equipped warehouses. The warehouses that the evaluation team visited lacked proper ventilation, functioning thermometers, and reliable refrigeration and electricity. Thus, medical supplies were being stored in hazardous or undependable locations, often leading to the deterioration of much-needed pharmaceutical products. Furthermore, in some of the sites where medicines were stocked, interviewees identified another constraint: the lack of reliable transport for the delivery of medications. The MOH-managed warehouses currently rely heavily on third party transportation services that are costly and unpredictable, and subject to delays or Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 20 the disappearance of pharmaceuticals. Future programs need to understand and address this bottleneck in any planned pharmaceutical delivery system. Reduced government financial resources. Due to the declining worldwide demand for petroleum, oil prices have plummeted, causing a reduction in the national health budget. C. Information To what extent has the GRA used the Pharmaceutical Management tools, key operating standard procedures, qualification and planning tools, and methodologies developed or implemented by SIAPS at national and sub-national levels? What is the GRA's perception of the value and quality of the tools and methodologies that they received? What were the challenges faced in using these tools? i. Goal: Address the information for decision-making challenges in the pharmaceutical sector. ii. Achievements: SIAPS identified challenges in the national LMIS for informed improvements and created and made innovative and proven tools broadly available. The program strengthened and implemented a strong LMIS at the national or central level, which should improve the quality of data and use of information decision-making. The program also enhanced the coordination of pharmaceutical supply chain management planning for a more rational use of resources. iii. Challenges: A number of respondents noted that the MOH was considering instituting a new management information system. If that occurs, the advances in the comfort and ease for users of the current system will be lost. The team was unable to verify either the content of the training curriculum or the continual mentoring through supportive supervisions described in SIAPS documents. iv. Constraints: Lack of a reliable communication system, both mobile devices and internet. Internet connectivity among health facilities, warehouses and those trying to keep track and manage the stock of any pharmaceutical programs is sketchy and unreliable at best. The sub-par technical infrastructure impedes the regular transference of medical information. D. Human Resources: To what extent has SIAPS improved the collection of strategic information for decision-making and planning purposes? What key strategies, tools and methodologies have been critical to the project's success? Was anything lacking that could have improved the project? i. Goal: Improve pharmaceutical services to achieve desired health outcomes. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 21 ii. Achievements: Throughout the five-year contract, SIAPS collaborated with other USAID implementing partners, NGOs, and other local partners to assist the Angolan MOH to implement pharmaceutical management strengthening interventions at the central and lower levels of the health care delivery system. The government of Angola establishes the supply chain for public facilities. All projects must work within governmental constraints and are unable to alter the supply chain plan developed by the government. In Angola, weaknesses in the supply chain system contribute significantly to poor availability of medicines for public facilities at all levels. The SIAPS Project worked with the central government to strengthen the management of essential medicines and health supplies so that more people could have access to medicines and increase the use of quality assured essential drugs and other health products and services. SIAPS also engaged with Central Drug Stores at national and provincial levels to increase the stores’ capacity to serve their priority health programs by adopting highly recognized international standard operating procedures. The SIAPS program improved the collection of strategic information by assisting 20 technical staff in the quantification of HIV and AIDS (and malaria) health commodities, in order to create a pool of technical personnel who are conversant with forecasting and supply planning methods at the national level, and to develop the terms of reference for a national quantification technical working group, in order to create a more inclusive and efficient quantification exercise at the national level. It also established the national quantification technical working group for HIV and AIDS health commodities in order to improve key stakeholders’ involvement and coordination in forecasting and supplies planning. The National Institute against HIV/AIDS (INLS) and partners conducted two annual quantification exercises of HIV/AIDS commodities; the results are being used in procurement (by the Global Fund, government, and partners, including PMI). In addition, SIAPS assisted the INLS in designing a stock-monitoring tool to inform the Country Procurement Plan for HIV/AIDS health commodities. SIAPS provided assistance to the INLS in revising its reporting and requisition forms, and to implement them at provincial and health facility levels to improve data quality and completeness. The data collected from these record-keeping procedures should help to better inform key pharmaceutical management decisions such as requisitions, stock monitoring, and national and provincial forecasting. SIAPS facilitated the review of the national patient management electronic tool to incorporate HIV/AIDS logistics information, in order to increase the relevance and completeness in reported data that could generate sufficient information for programmatic decision-making, including quantification and stock redistribution. It instituted a system to regularly monitor the stocks of HIV/AIDS products at the national level and at the nine PEPFAR focused health facilities (HFs); the results were used in distribution planning. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 22 Furthermore, SIAPS facilitated and provided TA during a review of past quantification exercises to assess tools, data and methodology and to identify challenges, and areas of improvements for future quantification during the monitoring of stock levels at CECOMA in order to inform semi-annual national distribution plans. SIAPS assisted 15 municipal focal points in RH/FP in Huambo province and collaborated with UNFPA in training the RH/FP program staff to develop and implement two long-term forecasts for health commodities. iii. Challenges: SIAPS faced challenges in collaboration with and barriers to cooperation among the GRA, USAID, UN agencies, bilateral donors and partners, and international and national NGOs. Challenges identified include divergences in medicine governance, human resource allocation, information flows, inadequate service delivery, and widespread corruption. Inconsistent government ownership and the full commitment of healthcare providers in project implementation were additional difficulties enumerated by participants. iv. Constraints: Weak coordination with stakeholders. Much of the current delivery system is divided into silos, with stakeholders addressing a single disease such as HIV or malaria. Follow-up programs should follow SIAPS’ energetic attempts to integrate health care and pharmaceutical delivery systems. Duplication of programs and efforts. Stakeholders noted that there is a need to create greater coordination among partners to avoid overlap in activities and programs. The current lack of coordination confuses partners and can create an unpleasant, competitive environment among NGOs. To improve the project, the SIAPS follow on should assist with the creation of a single, integrated system for the logistics, management and distribution of medical supplies, which would enable all participants to work together and share information. Ensure that computer information systems are compatible. Following on the SIAPS achievement, a greater integration and collaboration between stakeholders should be promoted; instead of building new databases, new programs should focus on improving the ones currently being used to meet the needs of the end user. It would also increase participation. It would have been an asset if SIAPS could have assisted in the implementation of the policies that they and others suggested, e.g., the National Strategy. It is imperative that subsequent programs build on the SIAPS work and urge the Minister of Health to pass the SIAPS policies currently sitting on his desk. In addition, there is a clear need for the annual training of pharmacy students on assessing the quality, storage, distribution, and tracking of stock, and understand how to plan for future stock outs. Instead of concentrating mainly on HIV/AIDS, USAID projects should instead adopt a holistic approach and focus on the diseases that are a contributing factor to the high infant mortality rate in Angola. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 23 E. Overarching Challenges Despite the existence of defined administrative structures and the improved availability of funding for pharmaceuticals, the Angolan public supply chain management system faces notable challenges in efforts to ensure uninterrupted availability and access to quality, essential pharmaceutical products and effective services. These challenges are in addition to the specific challenges mentioned above with the main evaluation questions, and cut across the SIAPS project as a whole. For example, gaps across all levels of the health commodity supply chain adversely affect national efforts to achieve the goals of identified health priorities, such as: the elimination of malaria; achieving an AIDS–free generation by reducing HIV incidence and increasing access to care; reducing high mortality and morbidity, especially of mothers and children under 5; and ending preventable mother and child deaths5 . SIAPS also faced challenges in collaboration with and barriers to cooperation among the GRA, USAID, UN agencies, bilateral donors and partners, and international and national NGOs. Challenges identified include divergences in medicine governance, human resource allocation, information flows, inadequate service delivery, and widespread corruption. Inconsistent government ownership and the full commitment of healthcare providers in project implementation were additional difficulties enumerated by evaluation respondents. Overall obstacles: Lack of data, a cumbersome and opaque bureaucracy, a weak education system that limits the pool of qualified employees, a lack of English language proficiency among the Angolans, and a lack of Portuguese language proficiency among the consultants. Political: Angola presents a challenging work environment; SIAPS encountered obstacles at every turn, which impeded the smooth implementation of the program:  Health systems strengthening and capacity building are not a focus of the government, as indicated in the minimal budgetary allotment to health.  The biggest obstacle to SIAPS’ implementation was the country’s bureaucracy: several documents/manuals were developed (see Annex VI) with SIAPS support, but are waiting to be approved by the MOH.  USAID had its own work plan, with priorities that often did not align with the MOH or the country’s priorities. Therefore, SIAPS had to juggle these different (and sometimes contradictory) priorities with its own goals to implement the program. Education: Many are not inspired to train for work in the health system, where remuneration is low and the conditions are challenging. Data: Data proficiency in Angola is almost non-existent over all levels in the health system, and there is little oversight to ensure data accuracy. Assessing the quality of whatever data exist in 5 Information was adapted from the SIAPS Final Report (December 2016). Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 24 Angola is very difficult; whoever “owns” the data will often not share it with other prospective partners. Data are often collected on individual paper index-like cards (when available), and input into a computer system is erratic. Computer systems that are available are not necessarily compatible, and little technical assistance is available on a consistent basis. Gender Component: Of the 36 people interviewed for this evaluation, 19 were women. Of the final SIAPS Angola staff composition of 8 people (as of December 2016), 5 were men and 3 were women. The quantitative data provided by SIAPS in its reports showed that the individuals receiving SIAPS trainings were primarily women. While a greater number of women than men were trained, it seems that this result may not have been by design, since it was the GRA that indicated which employee would be eligible for the training. While visiting the sites, however, it was clear to the evaluators that the majority of health workers are women; it is impossible to say how much this perception was influenced by circumstances. The only data found that disaggregated by gender was the number of persons trained in warehouse management (for CECOMA and regional warehouses). SIAPS provided training to 25 women in warehouse management procedures. Language: Most training manuals were in English, and the translation from English to Portuguese is very costly. This was a limiting factor; the evaluation team was told that the majority of Angolans working in the health sector do not possess sufficient English to use the manuals constructively. Decentralization: The goal of decentralization was to give the municipalities greater control over their health system, and to provide a tailored approach for their constituents. Instead, due to corruption and a lack of skill, training, and education of municipal staff, the problem has been exacerbated. In addition, health funding for the municipalities is not based on population size; all municipalities receive the same budget. Most municipalities in Angola lack the proper infrastructure and human resources to manage their own health systems. So, although they now receive funding, there is no government assistance on how to manage it. The municipalities urgently need capacity building in human resources, infrastructure, data management, and technical assistance. In addition to the challenges above, numerous challenges remain to creating a suitable supply chain system. These include: Infrastructure barriers:  Unreliable power supply.  Dilapidated buildings housing medical supplies; no funding for building maintenance; warehouses that are not properly ventilated; unreliable thermometers; and a lack of proper refrigeration.  No reliable transport or delivery methods for medications.  Lack of subsidized and working telephones.  Little or no internet connectivity to transmit information.  Few computers and/or persons who know how to use them effectively. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 25  Lack of basic supplies, such registration cards, pencils, paper, printer ink, and power cords. Political and economic barriers/obstacles:  Healthcare is not a high priority for the government.  Decentralization has led to an increase in stock-outs and a lack of basic supplies.  Funding chains from international donors may overlap or may not be renewed, and therefore are not sustainable in the long term.  Angola’s classification as a middle-income country (MIC) has created limitations on the type and quantity of foreign aid for which the country is eligible.  Diversion of funds and commodities at all levels in the delivery chain makes planning and purchasing challenging.  Little coordination and integration of donor programs.  No communication and coordination between all levels of the supply chain.  A lack of transparency at all levels of government. Personnel/professional obstacles:  Insufficient supervision, lack of refresher courses and follow up, no system to answer questions.  A lack of qualified, dedicated personnel, and a lack of retention of trained staff; qualified staff are siphoned off by the higher paying international NGOs.  Lack of sustainability, as Angola does not produce its own medicines.  Lack of funds for transportation.  The government pay scale is below the living wage, and low salaries are deterring qualified personnel from remaining in healthcare.  Staff members become discouraged and leave.  No government scholarship program exists to recruit and train a new generation of doctors, nurses, and pharmacists.  Statistics and monitoring and evaluation (M&E) professional and academic fields, and current curricula, are inadequate. SIAPS obstacles:  Insufficient SIAPS staff, given the program’s scope of work.  There were difficulties in hiring Portuguese speakers who were also proficient in English, a requirement because the managing organization (MSH) was based in the US.  Lengthy delays in gaining approval from the GRA to work in the country (MSH took 10 years to obtain a permit).  Visa delays in hiring staff.  No single computer system was agreed upon, so the sharing of data among warehouses, hospitals, pharmacies and clinics proved a challenge in gathering and analyzing data, and predicting and planning for stock-outs and budget allocations. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 26 F. Recommendations for Follow-on Projects Keeping in mind potential constraints as previously noted, below are recommended items that respondents repeatedly suggested would enhance any follow-on projects. The recommendations are prioritized according to: (1) issues that the follow-on project could reasonably address to increase potential success, and (2) issues for which USAID and the Angola Mission could reasonably support and advocate. 1. Recommendations that the follow-on project could consider in order to increase the possibility of success:  Management Develop relevant management tools from the outset of the project. Some evaluation respondents, for example, were unsure of their exact job descriptions and how their performance would be evaluated. From our sessions with the informants it was suggested that all technical personnel in future USAID/Angola projects must speak both English and Portuguese. If no local expertise is available, the project should recruit staff from Lusophone Africa, Brazil, or Portugal. Create an ongoing and consistent training program for a single group of technicians to promote sustainability and enhance group learning. Whenever possible, try to hire, support, and utilize the local personnel previously trained by SIAPS, and who have acquired experience and knowledge of the project.  Training Improve the level of education and training of employees working in the health posts, particularly those who track pharmaceutical products and other medical data. Most have never been trained to compile data, and do not understand the relevance of accuracy in their information gathering. Close follow up, repeat trainings, and investment in local staff are imperative to improving the accuracy of record keeping. Build local capacity to promote sustainability by training and hiring qualified staff in key positions. Identify, train, and maintain a greater number of technical personnel than the prior project. Continual trainings and mentoring are needed, as is the development of a financially and professionally rewarding career path for Angolans who wish to work as health professionals. Training must be done at the location of service, rather than in a reunion or meeting setting (in a hotel in Luanda, for example) with many unknown persons. In this way, trainers will be able to understand the obstacles to transmitting specific information in a particular locale, and adjust their approach accordingly. Building a support and back-up group of colleagues would also improve the quality of the work. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 27 Continue the academic course that SIAPS developed, to provide yearly training for fourth-year pharmacy students in how to track stocks of medicine. No such course is currently part of the curriculum. Future programs should devise the means to advocate for an improved and upgraded status for the occupation of f health worker. Advocate for increased professional respect and advancement in this employment category. Many respondents claimed that their profession, while vitally important, did not receive the recognition and remuneration commensurate with their responsibilities.  Communication Provide employees access to reliable electricity, telephones, computers, and internet services to allow them to optimally manage the collection, recording, and transmission of pharmaceutical stocks. One approach, implemented with success by Vertrou, is to rely on a variety of inexpensive measures to guarantee the monthly communication and monitoring of stock-outs throughout the country. Vertrou provided phone charge cards to their employees in each health center, allowing technicians to inform the central warehouse of their consumption and needs on a monthly basis via cellphone, bypassing irregular power supplies.  Collaboration Continue the regular meetings instituted by SIAPS with all key health sector actors to jointly resolve bottlenecks, duplications, and failures in the system, and encourage ongoing communication and coordination. Almost all partners interviewed, and specifically those in the areas of RH, HIV/AIDS and family planning, mentioned a desire for enhanced collaboration among partners.  Patient compliance Educate end users on the appropriate use of medication. While improving the supply chain is fundamental, many doctors in particular feel that it is also necessary to integrate patient education on taking medicines into any new initiative. For example, is the patient taking his/her meds correctly; how does the patient address side effects? Is the patient taking the complete prescription? The issue of patient compliance has been completely ignored.  Update current medical procedures Angola should have access to and adopt current technologies, to ensure that the correct medication is being prescribed to treat various diseases. Respondents mentioned that there are issues around the use of more appropriate (i.e., newer) medications. 2. Constraints that the Mission could address through consistent political support Given that both USAID and the Mission are constrained by the terms of the U.S. Government’s agreement with Angola, these recommendations will be circumscribed within the realm of Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 28 support or advocacy and financial leverage. Evaluation respondents suggested that the Mission use its support, financial leverage, and convening powers towards the following:  Advocacy The Mission should focus on continuing to strengthen its relationship with the GRA by maintaining close and cordial interactions with the MOH, in order to ensure that its programs receive essential support and promotion.  Passage and implementation of essential policies Although SIAPS, along with other stakeholders and partners, played a fundamental role in drafting several national policy documents (see Annex V), most have languished at the MOH for more than one year while awaiting approval. Had the MOH approved these policies, SIAPS could have assisted in their implementation (e.g., the National Strategy). The Mission should strongly advocate for their approval, and future programs should strive to implement such policies.  An independent CECOMA Leverage t support from the Mission to further the plan for CECOMA to become a separate organ from the MOH, so that it can assume responsibility for the centralized purchasing and distribution of commodities.  Support through funding Given the lack of in-country trained staff, as well as the expense and difficult logistics of implementing programs in Angola, the Mission should allocate adequate funding for the hiring of trained personnel when reviewing future program budgets. Many evaluation respondents underscored the fact that SIAPS lacked sufficient staff and funding to fully implement its program goals.  Ease the visa process Review and revise the process of applying for and obtaining a work visa. It is very difficult to recruit people from the outside to come to work in Angola, since the current visa process is arduous, cumbersome, uncertain, and expensive.  Promote collaboration The Mission should employ political and financial leverage to facilitate greater engagement and advocacy with other stakeholders working to improve Angola’s access to pharmaceuticals and services. For example, numerous respondents encouraged the Mission to engage in consistent collaboration with international partners to employ their collective leverage with the GRA to facilitate the implementation of programs and policies that enhance their joint program goals.  Encourage accountability with the GRA Some respondents encouraged the current U.S. Ambassador to Angola to increase engagement with and encourage accountability from the GRA. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 29  Address deficient warehouse and delivery systems Encourage the central government to support medicine deliveries to those provinces that have inadequate warehousing facilities, or that have experienced expiration and pilferage problems. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 30 ANNEXES ANNEX I: Evaluation Scope of Work (SOW) Performance Evaluation of USAID/Angola SIAPS Program A. Background The Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Activity, awarded in 2011, is a five-year activity aimed at strengthening the management of essential medicines and health supplies so that more people can access the health care they need. USAID/Angola recognizes that the regular availability and the use of quality assured essential medicines and other health products and services are key to achieving improved health outcomes for its priority health programs. The project has focused on four principle areas to achieve the project’s objectives and IRs: SIAPS Objective 1: Pharmaceutical Sector Governance Strengthened • National Directorate of Medicines and Equipment (DNME) coordination role for supply chain stakeholders improved; • Provide support to DNME to continue its coordination role in ICC/R logistics, procurement, and operations sub-commission meetings; • Provide support to DNME to validate and disseminate the national supply chain strategy; • Provide support to DNME to validate and disseminate the National Essential Medicines List and National Formulary; • Improve medicines’ policies, legislation, regulations, norms, and standards; • Provide specific training and mentorship program in HIV and AIDS pharmaceutical management in 9 PEPFAR focus HFs in Luanda, as per the new orientation of PEPFAR SIAPS Objective 2: Local capacity for pharmaceutical management enhanced • Pharmaceutical supply management capacity of local institutions and individuals enhanced for performance improvements; • Provide support to National Institute against HIV/AIDS (INLS) to implement pharmaceutical management trainings at national and provincial levels and in selected health facilities; • Provide support to CECOMA Central Procurement Agency for Medicines and Medical Supplies) to implement human resource capacity improvement plan to its warehouse staff at central and regional warehouses; • Provide support to National Directorate of Medicines and Equipment (DNME) to organize pre-service and in-service pharmaceutical management trainings. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 31 SIAPS Objective 3: Information for pharmaceutical management decision-making promoted • Challenges in the national Logistics and Management Information Systems (LMIS) identified for informed improvements; • Support Ministry of Health (MOH) to conduct a comprehensive review of LMIS to address its key bottlenecks; • Support CECOMA to improve its logistics management and information system; • Provide support to MOH DNME, NMCP, and INLS, and RH/FP program to regularly implement strategic monitoring tools such as extended End of User Verification (EUV), Procurement Planning and Monitoring Report for malaria (PPMRm), and Coordinated Procurement Planning (CPP) to inform procurement and supply chain management. SIAPS Objective 4: Pharmaceutical services to achieve desired health outcomes improved • Support National Malaria Control Program (NMCP) and selected provincial directorates to regularly review forecasting and supply planning and monitor the availability of malaria health commodities; • Support the National Institute against HIV/AIDS INLS and the national quantification technical working group to regularly review forecasting and supply planning and monitor the availability of HIV and AIDS health commodities; • Support Ministry of Health, Reproductive Health/Family Planning (MOH RH/FP) to conduct a long-term quantification exercise for Family Planning commodities and monitor their availability. B. Document review Prior to the start of in-country work, USAID/Angola will provide the evaluation team with the SIAPS related documents for review. The team will read the materials before arriving in country. This documentation includes project documents and other existing materials:  SIAPS undertook Analysis of the Angolan Public Health Supply Chain System in 2013; a review of the Rapid Assessment of the Supply Chain System for HIV/AIDS Commodities in Angola, in 2010; an Analysis of the Key Functions of the Regulatory Authority in 2012; and a Situation Analysis on Introducing Pharmaceutical Product Registration Policy in Angola.  SIAPS annual work plans and quarterly and annual reports  Monitoring data: SIAPS collects regular monitoring data and reports quarterly to USAID. A Monitoring and Evaluation Plan (with a list of the indicators tracked and annual targets) and the most recent annual report will be provided to the evaluators to review. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 32  SIAPS Gender Analysis  End Use Verification reports  Site visit reports  SIAPS Global Technical Strategies  SIAPS Global Technical Management Tools  Other technical reports C. Evaluation Questions The questions to answer, in order of priority, are as follows:  To what extent was the technical approach used by SIAPS effective in strengthening pharmaceutical systems in Angola?  Governance: To what extent has SIAPS improved accountability, policies, norms and standards? What system changes have taken place at the national and sub￾national level because of SIAPS technical assistance by program area?  Information: To what extent has SIAPS improved the collection of strategic information for decision-making and planning purposes? What key strategies, tools, and methodologies have been critical to the project’s success?  Human Resources: To what extent has the GRA used the Pharmaceutical Management tools, key operating standard procedures, quantification and planning tools, and methodologies developed or implemented by SIAPS at national and sub-national levels? What is the GRA’s perception of the value and quality of the tools and methodologies that they received? D. Evaluation Design and Methodology The evaluation data collection and analysis is expected to apply a mixed method approach, utilizing both qualitative and quantitative methods. Data should also be collected using primary and secondary sources. The following illustrative examples are provided to stimulate thinking and guide the decision-making process in the final design of the evaluation methods by the team. E. Data collection methods Quantitative Data Collection – Data collected using quantitative methods may include surveys of respondents using structured questionnaires. This may include a mini-survey that is administered face-to-face. For example, as part of the SIAPS activities, municipality officials have participated in workshops and surveys conducted in both Luanda and Huambo. These officials could potentially respond to surveys that could help inform question 1 and 2 above. Qualitative Data Collection – Qualitative data collection and analytical methods provide opportunities for capturing insights, perceptions, and opinions about the project, from those Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 33 who have directly or indirectly experienced or participated in the activities of SIAPS. For example, there are a variety of qualitative methods that the evaluation team can apply in collecting primary data to answer the key evaluation questions. These may include, but are not limited to in-depth interviews with key informants, and focus group discussions. In-depth Interviews - Using a structured questionnaire comprised primarily of open ended questions, the assessment team could conduct in-depth interviews with stakeholders who will serve as key informants. These may include appropriate key respondents from the Provinces of Luanda and Huambo, respondents from the Ministry of Health, who participated in SIAPS activities or are knowledgeable about the project, respondents from other institutions such as University Jean Piaget, etc. These interviews are expected to provide insights into the programmatic, technical, management, and operations of SIAPS over the last two years. Depending on the respondents, and the design of the data collection instrument, these interviews are also likely to yield evidence of the results that were achieved by SIAPS in pharmaceutical management and health systems strengthening. Focus Group Discussions –The evaluation team may also conduct a series of FGDs with groups of respondents that have directly participated in the SIAPS activities either as implementers or recipients of the expected benefits derived from the SIAPS activities. Each group may be comprised of 8-10 respondents. For example, individual groups may include discussions with key provincial and municipal counterparts, university partners, non￾governmental organizations and civil society organizations (NGOs/CSOs) and community members. If feasible, the groups should also be conducted with respondents at the district and provincial levels. Data collected will contribute to answering questions on the results and challenges experienced by SIAPS during its initial two years of implementation. Review of Documents – Secondary data will be collected by the evaluation team, through the review of documents that were produced on the project by its partners throughout the implementation period of SIAPS. These documents may include work plans, quarterly and annual reports, or other documents related to the technical aspects of implementation. These documents will be reviewed prior to primary data collection and may provide the team with an historical narrative of SIAPS implementation, while contributing background information for the development of primary data collection tools by the evaluation team. SIAPS Monitoring Data – Data collected by SIAPS as part of the project monitoring and evaluation system will also serve as a source of secondary data collection by the evaluation team. Based on these data, the evaluation team may be able to conduct secondary analyses of key output indicators that may contribute to the analysis of the primary data that will be collected by the team. While these methods are suggested, it is expected that the evaluation team will determine the most appropriate data collection and analytic methods that best address the evaluation questions. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011-2015, Angola 34 Evaluation Matrix/Work Plan – December 2016 IT Shows - Evaluation Matrix/Work Plan 12.27.16 Evaluation Question Sub-Question (s) Indicator/Measu re Data Sources Collection Method Sampling Method Data Analysis Methods (to be determined by evaluation team) To what extent was the technical approach used by SIAPS effective in strengthening pharmaceutical systems in Angola? 1. What were the pros and cons of using the award mechanisms? 2. Was the choice of mechanisms appropriate for the activities' intended outcomes? 3. Were the "Prime" and "Sub" partners (and their operational arrangements) 4. Did SIAPS staff feel they were given adequate guidance from USAID on what to accomplish? Increase in capacity level; Systems improved/ strengthened Project documents /document reviews; key informant interviews Online surveys - is this feasible? Key informant interviews Review of key documents identified by USAID project managers’USAI D/W and USAID Mission Staff NGOs and sub￾partners To what extent has SIAPS improved accountability, policies, norms and standards? What system 1. Is there evidence that SIAPS improved accountability? 2. What specific policies, norms or standards were implemented by SIAPS to improve accountability? Standards and policies put in place Organizations using norms and standards (nationally or Project documents /document reviews; key informant interviews Online surveys - is this feasible? Key informant interviews Review of key documents identified by USAID project managers’ USAID/W and Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011-2015, Angola 35 changes have taken place at the national and sub￾national level because of SIAPS technical assistance by program area? Were these adopted alone of in tandem with other methods? 3. How did these policies, norms and standards improve accountability? 4. What systems and standards promoted resulted in accountability? 5. What were the major challenges towards implementing the program? sub-nationally) Systems improved USAID/Mission Staff, NGOs and sub￾partners Digital Dev. Regional Advisors The To what extent has the GRA used the Pharmaceutical Management tools, key operating standard procedures, qualification and planning tools, and methodologies developed or implemented by SIAPS at national and sub-national levels? What is the GRA's perception 1. What are the tools mostly used in standard procedures? 2. How are methodologies and planning tools used nationally and/or sub￾nationally? 3. What are the most widely used standards? 4. What was not useful or inappropriate? 5. Was anything missing that could have helped fulfills the goals? Number of management tools used. Number of individuals trained under SIAPS Project documents /document reviews; key informant interviews Online surveys - is this feasible? Key informant interviews (via phone, video conferencing and face to face to interviews) Review of key documents identified by USAID project managers’ USAID/W and USAID/Mission Staff. NGOs and sub-partners Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011-2015, Angola 36 of the value and quality of the tools and methodologies that they received? What were the challenges faced in using these tools? Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011-2015, Angola 37 To what extent has SIAPS improved the collection of strategic information for decision-making and planning purposes? What key strategies, tools and methodologies have been critical to the project's success? Was anything lacking that could have improved the project? 1. How is the information and data used for planning and decision￾making? 2. How have the tools and methods promoted improved data for decision-making and information use? 1. Number of decisions taken based on PMI’s activities. 2. Quality of audit tools and filing systems. 3. Information reporting forms, e.g. indicator sheets and status reports are in place and used correctly. 4. Was the Data is consistently collected and sent to appropriate units. 5. Quality of data. 6. Quality of facilities/warehou sing. 7. Quality of inventory control systems, e.g. commodity management, dispensing and patient management Review of key documents. M&E reports; surveys; key informant surveys. Surveys. Facility/site inspection. Data quality assessments Review of key documents identified by USAID project managers, USAID/W and USAID/Mission Staff, NGOs and sub-partners Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 38 ANNEX II: Methodology Data collection methods A participatory performance evaluation was designed, with a mix of data collection approaches selected to allow representative stakeholders’ participation. Semi-structured interviews, with uniquely formulated questions, were developed to facilitate the collection of qualitative data (See Annex III for data collection instruments and Annex IV for sources of information). During the evaluation period, the team: 1. Conducted five field site visits in two provinces, Luanda and Huambo, as suggested by the USAID/Angola staff. In Luanda, the team visited CECOMA, Hospital Divina Providencia Warehouse, and Hospital Sanatorio Warehouse. In Huambo, the team visited the INLS Warehouse and Provincial (Huambo) Warehouse. 2. Conducted 36 semi-structured interviews with individuals from national-level stakeholder organizations, including USAID, GRA Departments, The Global Fund, UNDP, UNFPA, PSI, and other organizations (see Annex III, Sources of Information, for full list of key informants and organizations). 3. Sent questions via email to stakeholders not available for in-person interviews (e.g., key staff from MSH, Strengthening Angolan Systems for Health [SASH] and SIAPS). 4. Collected and analyzed secondary data from SIAPS reports (see Annex IV). 5. Reviewed secondary background sources from USAID and the GRA (see Annex IV). Field visits selection process, schedule and visit formats Site selection for field visits included CECOMA, tertiary hospitals and pharmacies, and warehouses in Huambo. The evaluation team also selected the most appropriate logistical arrangements, given the time constraints. Site visits and interviews were predominantly conducted between January 12 and 26, 2017, in Angola. Additional interviews with those unavailable to meet during this time period were conducted from February 2 to 13, 2017, via telephone. Chart 4: Institutions Contacted and Number of Point of Contact Interviews Institution Points of Contact (POCs) Interviewed CECOMA 4 DNME 3 DPS Huambo 2 Global Fund 1 HUAMBO RH 1 Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 39 HUAMBO Warehouse 2 ICAP 1 INLS 1 Luanda - Hospital 2 MSH/SIAPS Angola 7 NMCP 3 PSI 1 SASH/Força Saúde 2 UNDP 1 UNFPA 2 USAID 1 VERTROU 2 Total 36 Data collection Qualitative and quantitative data regarding CECOMA’s stocks were collected as part of the evaluation process. Extensive document reviews were conducted to obtain demographic information about the country and to understand the context in which the program was situated. The documents reviewed included SIAPS documentation, presentations, past evaluations, and available literature on supply chain management services in Angola (see Annex IV, Sources of Information). The scope of work (SOW) evaluation questions required the collection and analysis of data about program operations, acquired mainly from SIAPS quarterly and yearly reports and other program-related documents furnished by USAID/Angola. The team was able to obtain some numerical data on available stocks and stock-outs of drugs at CECOMA and the Huambo provincial warehouse. In addition, interviews assessing the quality and content of interventions from the perspective of the program staff, stakeholders and other participants were secured and are a primary source of information. Qualitative data practices were used to encourage stakeholders to provide descriptive responses. Qualitative methods, primarily semi-structured interviews with stakeholders, were used to obtain information that was used to assess SIAPS Program relevance, implementation, successes, barriers and overall satisfaction. Individual, face-to-face interviews were conducted with key informants whenever possible. Open-ended questions were used and ample time was provided for the participant to comment, explain, and share experiences and attitudes. The feedback from these interviews helped to gauge participants’ satisfaction and gather recommendations for improvement. Data analysis Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 40 The analysis began with a preliminary review of the qualitative data collected. The team read through the interview transcripts, and corrected translated and clarified responses as needed. Notes taken during interviews were assembled daily using a template (see Annex II) that captured negative and positive aspects of the program, recommendations, and overall impressions. Detailed notes were taken by at least two team members to record perceptions of the SIAPS Program. While this increased the length of each interview and/or site visit, it also provided a more in-depth and focused review of each participant’s responses. In addition, each team member was assigned the lead on a particular interview, while the remaining team members took notes. Such notes helped identify common issues and obstacles that the SIAPS Program faced. Whenever possible, all team members were present at the interviews and site visits in Luanda. For the interviews and site visits in Huambo, only two team members were designated. Content analysis was used to capture and summarize information, as indicated in Chart 5 (below), describing SIAPS achievements. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 41 ANNEX III: Data Collection Instruments Individual Interview Questionnaire Name & Title of Contact Gender Contact information Organization Type of Organization Location Program Level Role with SIAPS Length of time with SIAPS Role Focus area Date The questions to answer, in order of priority and depending on the informant, are as follows: 1. To what extent was the technical approach used by SIAPS effective in strengthening pharmaceutical systems in Angola? What were its weaknesses? a. What were the pros and cons of using the award mechanisms? b. Was the choice of mechanisms appropriate for the activities’ intended outcome? c. Where the prime and sub partners appropriate for the activities’ intended outcome? d. Did SIAPS staff feel that they were given adequate guidance from USAID on what to accomplish? What was lacking? 2. Governance: Is there evidence that SIAPS improved accountability? What system changes have taken place at the national and sub-national level because of SIAPS technical assistance by program area: PEPFAR, HIV, PMI a. To what extent has SIAPS improved accountability, policies, norms and standards? b. What system changes have taken place at the national and sub-national level because of SIAPS technical assistance by program area: PEPFAR, HIV, PMI c. What specific policies, norms or standards were implemented by SIAPS to improve accountability? d. How did these policies, norms or standards improve accountability? e. What systems and standards promoted resulted in accountability? f. What were the major challenges towards implementing the program? g. To what extent has the GRA used the pharmaceutical management tools, key operating standard procedures, qualification and planning tools and methodologies developed or implemented by SIAPS and national and subnational levels? h. To what extent is SIAPS contributing to the availability of HIV/SRH/Malaria commodities at the national and local levels? Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 42 3. What is the GRA’s perception of the value and quality of the tools and methodologies that they received? What were the challenges faced in using these tools? a. What are the tools mostly used in standard procedures? b. How are methodologies and planning tools used nationally and/or sub￾nationally? c. What are the most widely used standards? d. What was not useful or inappropriate? e. Was anything missing that could have helped fulfill the goals? 4. Information: To what extent has SIAPS improved the collection of strategic information for decision-making and planning purposes? For example, planning, stock outs, shortages, pricing, procurement? What key strategies, tools, and methodologies have been critical to the project? What was lacking? How would you improve the project? a. Did you receive any training and supervision to implement the key strategies? Was the training adequate? b. How is the information and data used for planning and decision-making? c. How have the tools and methods promoted improved data for decision-making and information use? d. Have you collected any data pertaining to SIAPS? In what format is it? Was the data relevant for fulfilling USAID requirement? Was it useful for your own purposes? e. Does the project consistently ensure that the country’s quantifiable need for HIV/SRH/Malaria commodities match the actual demand/requirement? f. How effective are medicine supply chain management (SCM) capacity building activities (both training and supervision) for health facilities? Do these activities produce the expected results, and has the scale of SIAPS’ implementation been sufficient? g. What are the current processes and constraints in the procurement of pharmaceuticals? 5. Recommendations: a. How would you evaluate the program? b. What would have made the program better? c. What would you have needed to accomplish the program goals? d. What is the most significant change that happened as a result of the program? e. In your opinion, what was missing that could have made the program more successful? f. Who were your most effective collaborators and partners? Will that interaction continue now that SIAPS is being replaced by PSM? g. What were your biggest obstacles in fulfilling your part of the program? 6. Other comments: is there anything you would like to add? 7. Who would you recommend we talk to? Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 43 Evaluation Questionnaire: Rapid Data Quality Verification for Warehouse Visit Name & Title of Contact Gender Contact information Organization Type of Organization Location Program Level Role with SIAPS Length of time with SIAPS Role Focus area Date Questions to answer, in order of priority and depending on the informant, are as follows: 1. Have they received Coartem recently? 2. When did they receive it? 3. How much they received (quantity)? 4. Could we see it? 5. Is it properly stored and organized (our observation)? 6. Count how many small white boxes (not individual cards) are left in each larger box - this will give you the total number currently in stock 7. Can we see the stock card? 8. Count the quantities of Coartem that has been sent to municipalities (conduto), starting from the date they received their latest Coartem (for the same delivery batch) 9. Quantity received minus Coartem sent should equal total number currently in stock 10. If received minus sent does NOT equal current stock, ask why? Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 44 Notes Template Name: Professional title during SIAPS Project: Contact information: Gender: Organizational affiliation during SIAPS project: Role with/during SIAPS: Background or professional training: Length of time with/during SIAPS: Focus area/area of expertise: Date of interview: Location of interview: Notes/observations: Positive aspects of SIAPS: Negative aspects of SIAPS: Recommendations: Overall impressions: Political context: Follow up: Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 45 ANNEX IV: Sources of Information Persons Interviewed Name Organization Katiza Mangueira CECOMA, Interim Director /DNME, Director Maribel Perez Diaz CECOMA, Quality Assurance Julia Simao CECOMA, Assistant Director Juliana Ferreira CECOMA, Deputy Director Maria Roberto Fernandes CECOMA, Pharmaceutical Technician Rafael Dimbu NMCP, Program Coordinator Eliza Miguel NMCP, Malaria Case Management Manzambi Ferreira NMCP, Head of Logistics Constancio Joao DNME, Head of Logistics Adelino Manaças DNME, Program Coordinator Edilson Pedro INLS, Quality Control Supervisor Frederico Juliana DPS – Huambo, Provincial Health Director Euclides Chipalavella DPS – Huambo, HIV/AIDS Supervisor Yava Ricardo The Global Fund Francisco Teles PSI, Project Manager Samson Ngonyani SASH, HIV Team Leader Julia Perri SASH, Senior Operations Manager Raquel Pacagnella VERTROU, Technical Advisor Luis Mines VERTROU, Information System Technician Celestino Teixeira ICAP, HIV & TB Technical Advisor Irene Machado UNDP, Project Manager Hirondina Cucubica UNFPA, Assistant Country Representative Fernando Fumuassuca UNFPA, Procurement & Logistics Associate Noemia Estovo Huambo Warehouse, Supervisor Armando Joao Huambo Warehouse, Assistant Supervisor Adelina Bonga DPS – Huambo, Provincial HIV/AIDS Supervisor Chiara Giusto Hospital Divina Providencia, Administrator Felizbela Bandua Hospital Divina Providencia, Licensed Pharmacist Domingas Canhanga USAID – Angola, SIAPS Activity Manager Wonder Goredema MSH, SIAPS Technical Support Lead & Interim Country Director Dinah Tijipura MSH, Senior Manager Patrick Gaparayi MSH, SIAPS Angola Project Director Veronica Mbengi MSH, SIAPS Angola Data Analyst Olinda da Costa MSH, SIAPS Angola Technical Advisor Lubaki Joao MSH, SIAPS Angola Technical Advisor Jane Sousa MSH, SIAPS Angola Technical Advisor Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 46 Points of Contact (POC), Number of Years with SIAPS Institution POC Years working with SIAPS CECOMA Katiza Mangueira 5 years Maribel Perez Diaz Did not work directly with SIAPS Julia Simão Did not work directly with SIAPS Juliana Ferreira 2 years and 6 months DNME Katiza Mangueira 5 years Constancio Joao 5 years Adelino Manaças Did not work directly with SIAPS DPS Huambo Frederico Juliana 5 years Euclides Chipalavella 5 years Global Fund Yava Ricardo 5 years Huambo - RH Adelina Bonga 2 years Huambo Warehouse Noémia Estovo 5 years Armando Joao 5 years ICAP Celestino Teixeira 1 year and 9 months INLS Edilson Pedro 5 years Luanda - Hospitals Chiara Giusto 2 years Felizbela Divina 2 years MSH/SIAPS Angola Jane Sousa 1 year and 3 months Wonder Goredema 5 years Dinah Tijipura 3 years Patrick Gaparayi 5 years Veronica Mbengi 1 year and 3 months Olinda da Costa 1 year and 3 months Lubaki Joao 3 years NMCP Rafael Dimbu 1 year Elisa Miguel Did not work directly with SIAPS Manzambi Ferreira 5 years PSI Francisco Teles 5 years SASH/Força Saúde Samson Ngonyani 5 years Julia Perri Did not work directly with SIAPS UNDP Irene Machado 5 years UNFPA Hirondina Cucubica 5 years Fernando Fumuassuca 3 years USAID Domingas Canhanga 5 years VERTROU Raquel Pacagnella 1 year and 6 months Luis Mines Did not work directly with SIAPS Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 47 ANNEX V: Evaluation Team Members Core team:  Leni M. Silverstein, Team Leader (IT Shows, Inc, Consultant)  Ana West, Health System Strengthening Specialist (IT Shows, Inc, Consultant)  Jamie Matias de Arujo Lobo, Evaluation Specialist (IT Shows, Inc, Consultant)  Paul Burns, Information Systems Specialist, (IT Shows, Inc. Consultant) Supporting team:  Eliane Mbounga, USAID/Angola, President’s Malaria Initiative  Lungi Okoko, USAID/Angola, President’s Malaria Initiative Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 48 ANNEX VI: Manuals and Documents Developed by SIAPS  Ficha de Stock (Stock Card)  Modelos de Cadernos de Encargos (Job Descriptions)  Planos de Acquisições (Procurement Plans)  Plano Estratégico para o Aprovisionamento de Medicamentos à CECOMA (Strategic Plan for the Procurement of Medicines for CECOMA)  Elaboração dos Planos Estratégicos Anuais (Development of Annual Strategic Plans)  Lista Nacional dos Medicamentos Essenciais (National List of Essential Medicines)  Formulário de Medicamentos Essenciais (Essential Medicine Forms)  Gestão da Cadeia de Abastecimento (Supply Chain Management)  Guia de Supervisão Farmacéutica das Unidades Sanitárias (Guide for the Pharmaceutical Supervision of Health Units)  Sistema de Recepção, Gestão e Expedição (Reception, Management and Expedition System)  Manual de Estrutura Organizativa (Organizational Structure Manual)  Manual Farmacéutico dos Produtos de HIV & SIDA (Pharmaceutical Guidelines for HIV & AIDS Commodities)  Manual de Gestão de Medicamentos (Guidelines for Managing Medicines)  Manual de Controle de Qualidade de Medicamentos (Manual for Quality Control of Medicines)  Manual de Procedimentos de Concursos Públicos (Manual for Public Competitions Procedures)  Manual de Procedimentos Operacionais (Manual of Standard Operating Procedures)  Manual de Procedimentos Padrão e Procedimentos de Gestão (Manual of Standard Operating and Management Procedures) Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 49 ANNEX VII: Bibliography Angola National Institute of Statistics. (2014). Angola Population and Housing Census. Retrieved from: http://www.healthdata.org/angola Connor, Catherine, Denise Averbug, and Maria Miralles. (2010). Angola Health System Assessment 2010. Bethesda, MD: Health Systems 20/20, Abt Associates Inc. Cosep Consultoria, Consaúde, and ICF International. (2011). Angola Malaria Indicator Survey. Maryland. Retrieved from: http://dhsprogram.com/pubs/pdf/MIS11/MIS11.pdf. Management Sciences for Health. (2012). MDS-3: Managing Access to Medicines and Health Technologies. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services (SIAPS). (2016). Systems for Improved Access to Pharmaceuticals and Services Program Quarterly PMP Review. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services Program Angola Work Plan: October 1, 2012 – September 30, 2013. Submitted to the U.S. Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health, Inc. Systems for Improved Access to Pharmaceuticals and Services Program Angola Work Plan: October 1, 2013 – September 30, 2014. Submitted to the U.S. Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health, Inc. Systems for Improved Access to Pharmaceuticals and Services Program Angola Work Plan: October 1, 2014 – September 30, 2015. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health, Inc. Systems for Improved Access to Pharmaceuticals and Services Program Angola PEPFAR and POP funded Work Plan: October 1, 2015 – September 22, 2016. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health, Inc. Systems for Improved Access to Pharmaceuticals and Services. (2012). Strengthening Pharmaceutical Systems Program: Activity and Product Status Report, October-December 2012. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2015). Systems for Improved Access to Pharmaceuticals and Services Program Quarterly Report: Project Year 5, Quarter 1: October-December 2015. Submitted to the US Agency for International Development by Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 50 the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2015). Systems for Improved Access to Pharmaceuticals and Services Program Quarterly Report: Project Year 5, Quarter 2: January-March 2016. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2016). Systems for Improved Access to Pharmaceuticals and Services Program Quarterly Report: Project Year 5, Quarter 3: April￾June 2016. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2016). Systems for Improved Access to Pharmaceuticals and Services Program Quarterly Report: Project Year 5, Quarter 4: July￾September 2016. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2013). Strengthening Pharmaceutical Systems Program Annual Report: Project Year 1, October 2011–September 2012. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2013). Systems for Improved Access to Pharmaceutical Services Annual Report: Project Year 2, October 2012–September 2013. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2014). Systems for Improved Access to Pharmaceutical Services Annual Report: Project Year 3, October 2013–September 2014. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2015). Systems for Improved Access to Pharmaceutical Services Annual Report: Project Year 4, October 2014–September 2015. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2016). Systems for Improved Access to Pharmaceutical Services Annual Report: Project Year 5, October 2015–September 2016. Final Performance Evaluation Report: Systems for Improved Access to Pharmaceuticals and Services (SIAPS) 2011- 2015, Angola 51 Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services. (2013). Systems for Improved Access to Pharmaceuticals and Service: Program Guidance. Arlington, VA: Management Sciences for Health. Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. (2014). Promising Practices: Procurement. Arlington, VA: Management Sciences for Health. Retrieved from: http://siapsprogram.org/wp-content/uploads/2014/07/2_Procurement-final.pdf UNDP. (2009) The Human Development Index (HDI). The Human Development Index (HDI). Retrieved from: http://hdr.undp.org/en/countries/profiles/AGO US President’s Emergency Plan for AIDS Relief (PEPFAR). (2015). Angola Country Operation Plan for Year 2014. Office of U.S. Global AIDS Coordinator and the Bureau of Public Affairs, U.S. State Department. Washington, DC. World Bank. (2015). World development indicators 2015. Washington, D.C., World Bank Publications. World Health Organization (2012). Country Statistics – Angola. World Health Organization, Geneva. World Health Organization. (2016). Yellow fever (Fact sheet). World Health Organization, Geneva. Retrieved from: http://www.who.int/mediacentre/factsheets/fs100/en/. World Health Organization. (2016). Yellow Fever – Angola. Disease Outbreak News 14 March. World Health Organization, Geneva. Retrieved from http://www.afro.who.int/en/yellow￾fever/sitreps/item/8440-sitrepyellow-fever-outbreak-in-angola-14-march-2016.html.