USAID MIDTERM PERFORMANCE EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY NOVEMBER 2018. This publication was produced for review by the United States Agency for International Development/Nigeria. It was prepared by DevTech Systems, Inc. under Task Order 16-00001. USAID/NIGERIA MIDTERM PERFORMANCE EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY Contract Number: AID-OAA-I-15-00018/620-TO-16-00001 Submitted to: Dr. Chidimma Anyanwu, Contracting Officer’s Representative, USAID/Nigeria Ms. Whitney Jensen Rodrigues, Alternate Contracting Officer’s Representative, USAID/Nigeria Prepared by: Katia Peterson - Team Lead Nelson Ochekpe - Pharmaceutical Sector Technical Expert Clement Inyagi - Health System Expert Elijah Irmiya Nep - Research Assistant Samuel N. Gyang - Senior Monitoring, Evaluation and Learning Specialist Submitted by: Traci L. Dixon, Chief of Party, Monitoring, Evaluation and Learning Activity, No. 40 Mississippi Street, Off Ivan Ikoku Way Maitama, Abuja, FCT, Nigeria. Photographer credit: USP Nigeria PQM. Training on handheld Roman spectrometer in Benin. DISCLAIMER This report is made possible by the support of the American People through the United State Agency for International Development (USAID). The contents of this report are the sole responsibility of DevTech Systems, Inc. and do not necessarily reflect the views of USAID or the United States Government. CONTENTS EXECUTIVE SUMMARY.............................................................................................................. 2 SECTION 1. INTRODUCTION .................................................................................................. 7 EVALUATION PURPOSE AND OBJECTIVES.......................................................................................................7 EVALUATION QUESTIONS.....................................................................................................................................7 SECTION 2. PQM BACKGROUND ........................................................................................... 9 PQM GLOBAL PROGRAM........................................................................................................................................9 RESULTS FRAMEWORK............................................................................................................................................9 PQM IN NIGERIA.......................................................................................................................................................10 NAFDAC ......................................................................................................................................................................11 ACTIVITIES BY IR.......................................................................................................................................................12 SECTION 3. METHODOLOGY ................................................................................................ 14 EVALUATION DESIGN............................................................................................................................................14 TYPES OF DATA........................................................................................................................................................14 ANALYSIS METHODS ..............................................................................................................................................15 SECONDARY RESEARCH DATA: DOCUMENT REVIEW............................................................................15 LIMITATIONS AND MITIGATION MEASURES................................................................................................15 SECTION 4. FINDINGS AND CONCLUSIONS..................................................................... 16 IR 1: STRENGTHEN MEDICAL PRODUCT QUALITY ASSURANCE SYSTEMS.....................................16 EVALUATION QUESTION 1.............................................................................................................................16 IR 2 INCREASING THE SUPPLY OF QUALITY-ASSURED PRIORITY MEDICINES ...............................26 EVALUATION QUESTION 2.............................................................................................................................26 EVALUATION QUESTION 3.............................................................................................................................30 IR 3: UTILIZATION OF MEDICAL PRODUCT INFORMATION FOR DECISION-MAKING INCREASED.................................................................................................................................................................32 EVALUATION QUESTION 4.............................................................................................................................32 EVALUATION QUESTION 5.............................................................................................................................33 CROSSCUTTING FINDINGS.................................................................................................................................34 SECTION 5. RECOMMENDATIONS AND FUTURE DIRECTIONS ................................... 35 OVERALL CONCLUSIONS ....................................................................................................................................35 RECOMMENDATIONS............................................................................................................................................36 RECOMMENDATIONS FOR PQM ..................................................................................................................36 RECOMMENDATIONS FOR USAID...............................................................................................................36 RECOMMENDATIONS FOR PMG-MAN AND PQM.................................................................................37 RECOMMENDATIONS FOR NAFDAC..........................................................................................................37 RECOMMENDATIONS FOR FUTURE USAID PRIORITY INVESTMENT.............................................38 ANNEX I. EVALUATION SOW............................................................................................... 39 ANNEX 2. DATA COLLECTION TOOLS: INTEVIEW GUIDES......................................... 53 ANNEX 3. KEY INFORMANT CONTACTS AND INTERVIEW SCHEDULE.................... 62 ANNEX 4. REVIEWED DOCUMENTS.................................................................................... 65 ANNEX 5. CONFLICT OF INTEREST .................................................................................... 66 ANNEX 6. EVALUATION TEAM SUMMARY ........................................................................ 71 TABLES Table 1. PQM Intermediate Objectives and Intermediate Results, 2009–2015.................... 10 Table 2. Key Informant Interviews ............................................................................................ 15 Table 3. ISO 17025: 2005 Scopes............................................................................................... 23 Table 4. Manufacturer PQM Milestones for WHO PQ ........................................................... 27 Table 5. Quantity of Medicines Supplied to Public Health Organizations, 2013–2018 ........ 28 FIGURES Figure 1. PQM Results Framework, 2016–2019 ....................................................................... 10 Figure 2. PQM’s Technical Approach........................................................................................ 11 1 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ACRONYMS API Active Pharmaceutical Ingredient CAPA Corrective Action/Preventive Action CLM Collaborative Learning Model DER Drug Evaluation and Research DHIS-2 District Health Information System DG Director General FMoH Federal Ministry of Health GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GHSC-PSM Global Health Supply Chain Procurement and Supply Management GMP Good Manufacturing Practices GoN Government of Nigeria HPLC High-Performance Liquid Chromatography HPN Office of Health, Population, and Nutrition ICT Information and Communications Technology IHP Integrated Health Program IR Intermediate Result ISO International Organization for Standardization LS laboratory services M&E monitoring and evaluation MCH maternal and child health MNCH maternal, newborn, and child health NAFDAC National Agency for Food and Drug Administration and Control NIPRD National Institute for Pharmaceutical Research and Development NMEP National Malaria Elimination Program NQAP National Quality Assurance Policy NQCL National Quality Control Laboratory NSCIP Nigerian Supply Chain Integration Project OHS Office of Health Systems PCN Pharmaceutical Council of Nigeria PMG-MAN Pharmaceutical Manufacturers Group of the Manufacturers’ Association of Nigeria PQ Prequalification (World Health Organization) PQM Promoting the Quality of Medicines Activity PV-PMS Pharmacovigilance-Post-Marketing Surveillance QA quality assurance QC quality control SWOT strengths, weaknesses, opportunities, and threats (analysis) TA Technical Assistance UNICEF United Nations International Children’s Emergency Fund USAID United States Agency for International Development USP United States Pharmacopeia WHO World Health Organization 2 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS The purpose of this midterm performance evaluation is to provide the United States Agency for International Development (USAID)/Nigeria an objective assessment of Promoting the Quality of Medicines (PQM) activity in Nigeria. Specifically, the evaluation was conducted to: ● Assess the effectiveness of PQM’s technical assistance (TA) in two areas: strengthening medicines regulatory quality assurance (QA) systems in Nigeria and building the capacity of local manufacturers to produce high-quality medicines ● Identify accomplishments and challenges in activity implementation to improve efficiency and highlight opportunities for adjustments in current technical activity ● Provide recommendations to USAID/Nigeria for potential future investments in medicines QA systems strengthening and, specifically, in strengthening the capacity of local pharmaceutical manufacturing This evaluation was designed to answer five evaluation questions: 1. To what extent has PQM’s TA been effective in strengthening QA systems for medicines in Nigeria? ● PQM has provided TA to the National Agency for Food and Drug Administration and Control (NAFDAC). To what extent has this TA improved NAFDAC’s core functions? Is TA still needed? If so, what type of TA? ● Is targeting NAFDAC the most effective strategy for improving medicines QA systems in Nigeria or are there other regulatory bodies that should receive TA? If so, which other regulatory bodies and what type of TA? ● Do NAFDAC quality control laboratories have the technical and human resource capacities to handle the medicines quality control and analysis requirements for Nigeria? If not, what type of TA is needed? What type of resources should be provided? 2. To what extent has PQM’s TA to local manufacturers improved the production capacity and quality for several priority Maternal and Child Health (MCH) and malaria medicines [chlorhexidine, amoxicillin dispersible tablets, oral rehydration salts (ORS)/zinc sulfate, oxytocin, Artemether+Lumefantrine, and Sulphadoxine/Pyrimethamine]? What milestones have been achieved? What, if any, milestones still need to be achieved for local manufacturers to produce these quality￾assured medicines? 3. What are the perceptions of local pharmaceutical manufacturers toward PQM’s TA? Have they made plans to sustain or expand local production of quality-assured priority medicines beyond the life of the project? 4. To what extent has PQM’s TA been effective in increasing the capacity of national and state regulatory agencies to utilize medical product quality information for decision-making? Is the TA still needed? 5. What type of plans have national and state health officials made to sustain the regulatory systems for medicines, including the internationally accredited laboratories, beyond the life of the project? Are there any gaps in plans? 3 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV EVALUATION DESIGN AND METHODS This midterm performance evaluation was designed as a qualitative research study utilizing primary and secondary data. Primary qualitative data were derived from Key Informant Interviews (KII). Key informants included stakeholders who received and/or are currently receiving TA. The PQM activity provided the names of stakeholders and their contact information. In order to obtain a holistic understanding of stakeholder experiences with PQM at all levels of implementation, the evaluation team invited respondents in leadership positions and staff to participate in the interviews. One USAID Implementing Partner (IP) that interacts with PQM but does not receive TA was also included. Secondary data were extracted from documents provided by USAID, PQM, and stakeholders. The combination of primary and secondary data allowed the evaluators to construct a complete picture of what PQM has accomplished (or not accomplished), the ways in which the results were accomplished, and approaches that were ineffective. Understanding the “how” component of PQM’s performance was essential to generating “multi-dimensional” results that can support evidence-informed decision-making for both USAID and PQM stakeholders. FINDINGS Overall, PQM was found to be effective at increasing the capacity of NAFDAC to regulate and control the quality medicines, and to increase manufacturers’ capacity to produce safe and efficacious medicines. Evaluation Question 1: To what extent has PQM’s TA been effective in strengthening QA systems for medicines in Nigeria? The Director General (DG) of NAFDAC was dedicated to ensuring PQM’s successes. She is a champion of laboratories and has assigned significant focus to ensuring their continued improvement and sustainability. Following PQM’s TA: ● Inspectors have the capability and confidence to carry out inspections of sterile and non-sterile facilities and manufacturing processes. The inspectors’ ability to conduct different types of inspections ensures that NAFDAC closely monitors the manufacturing processes that pose the greatest risk to public health, such as products and medicines that require sterile manufacturing processes and facilities. Prior to PQM, NAFDAC’s Drug Evaluation and Research (DER) Directorate inspectors applied a “one size fits all” approach to Good Manufacturing Practice (GMP) inspections. ● NAFDAC’s Planning, Research, and Statistics (PRS) Directorate has a more robust monitoring and evaluation (M&E) system with “M&E Champions” in each Directorate. M&E Champions provide stewardship of data to ensure its fidelity, accuracy, and timeliness. PRS staff have the capacity and confidence to track indicators at the output and outcome levels, and are able to synthesize the data to support evidence-informed decision-making. ● NAFDAC’s Registration and Regulatory Affairs (R&R) Directorate has increased capacity to critically review and evaluate medical product dossiers. It can confidently ensure that products that meet NAFDAC’s standards are eligible for registration and marketing authorization ensuring end users receive safe and efficacious medicines. ● NAFDAC’s Yaba, Agulu, and Kaduna labs achieved International Organization for Standardization (ISO) 17025 accreditation. After Yaba accreditation, NAFDAC successfully used PQM’s Collaborative Learning Model to train staff at the Agulu and Kaduna labs. Those labs subsequently achieved accreditation using the model. The laboratories’ capacity, evidenced by accreditation, is a critical component to ensuring safe, effective medicines enter and remain in circulation. 4 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ● The Nigerian Supply Chain Integration Project (NSCIP) developed the National Quality Assurance Policy (NQAP), which ensures that medicine quality is maintained throughout the entire supply chain, from the manufacturer or Marketing Authorization Holder1 (MAH) to the end user. Evaluation Questions 2 and 3: To what extent has PQM’s TA to local manufacturers improved the production capacity and quality of priority MCH and malaria medicines? What are the perceptions of local pharmaceutical manufacturers toward PQM’s TA? Have they made plans to sustain or expand local production of quality-assured priority medicines beyond the life of the project? 1. The success of PQM, and a reflection of its TA, is evident in the achievements of local manufacturers. CHI Pharmaceuticals, Juhel Nigeria Limited, and Drugfield Pharmaceuticals have developed ORS/zinc sulfate, oxytocin, and chlorhexidine digluconate gel. Each has obtained NAFDAC approval or marketing authorization for the Nigerian market. 2. Manufacturers are looking beyond Nigeria to ensure sustainability; most are already exporting or have plans to export their product to other West African countries. ● Drugfield is the first manufacturer in Africa to produce and market chlorhexidine digluconate gel, which it is exporting to Ghana for inclusion in safe delivery kits. This product is also registered and supplied to five other African countries, and the process is underway in Togo. ● Juhel is the first company in West Africa to manufacture and market oxytocin injection. ● CHI Pharmaceuticals was selected by the United Nations International Children’s Education Fund to supply ORS/zinc sulfate to four countries. 3. Four manufacturers secured ISO 9000 series certification, which suggests an organizational framework that would support World Health Organization (WHO) Prequalification (PQ). 4. CHI Pharmaceuticals is expected to have a WHO PQ assessment visit in June 2018 for zinc sulfate. If successful, they will become the first local manufacturer to obtain a WHO PQ certificate for zinc sulfate. 5. Manufacturers say uncertain market demand makes it difficult to adequately plan production volumes to meet future market demands, increasing the chances of product expiration. Evaluation Question 4: To what extent has PQM’s TA been effective in increasing the capacity of national and state regulatory agencies to utilize medical product quality information for decision￾making? Is the TA still needed? 1. Before PQM the Pharmacovigilance-Post-Marketing Surveillance (PV-PMS) Directorate did not have a systematic way to conduct surveillance or operationalize surveillance data. PQM facilitated the development of the “PMS Guidelines,” which ensure that PMS surveys are carried out systematically and reliably. Reliable surveillance data is the basis of NAFDAC’s regulatory actions against substandard and falsified medicines. 2. A 2016 PMS on MCH medicines, sponsored by PQM, found that 74 percent of sampled oxytocin and 34 percent of sampled misoprostol failed laboratory testing. Based on this information, the PV-PMS Directorate established risk-based surveillance2 of these medicines. 1 Marketing Authorization Holders are granted permission, usually by a regulatory body, to market a specific medicinal product. 2 Risk-based surveillance is an approach to monitoring medicines and medical commodities with a high likelihood of falsification or being substandard, including in geographic areas with known problems of poor storage, high black market demand, porous 5 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 3. Following PQM support, The National Malaria Elimination Program was better able to reliably determine the percentage of substandard and falsified malaria medicines available in the market and map where they were concentrated. Based on the results, NAFDAC has prioritized funds for risk￾based surveillance of medicines that pose the greatest safety risk to consumers. Evaluation Question 5: What type of plans have national and state health officials made to sustain the regulatory systems for medicines, including the internationally accredited laboratories, beyond the life of the project? Are there any gaps in plans? 1. NAFDAC does not have a formal sustainability plan for regulatory system strengthening, but some inadvertent sustainability measures in place. These include using the CLM approach to train staff in the Yaba, Agulu, and Kaduna “sister laboratories” for accreditation and using the revenue generated from testing samples for the Global Fund to Fight AIDS, Tuberculosis and Malaria and Catholic Relief Services to fund laboratory operations; and partnership with Merck for the bulk purchase of laboratory consumables. 2. The Director General has cracked down on unnecessary spending as a way to free up resources for Directorates that need more funding, such as Laboratory Services. 3. NAFDAC advocated for ongoing budget line items specifically for laboratories. In the 2017 appropriations bill, for example, there was a 50 million naira (approx. $140,000) line item for training and capacity building, and the 2018 proposed appropriations bill included a 100 million naira (approx. $280,000) line item for laboratory equipment and 50 million naira (approx. $140,000) for refurbishment of laboratory buildings. CROSSCUTTING FINDINGS ● PQM branding is inadequate. Several stakeholders, including some local manufacturers, especially outside of NAFDAC and the Federal Ministry of Health (FMoH), do not know PQM; they know only of United States Pharmacopeia. ● The evaluation team assessed if women were well represented in PQM activities. It found that women were more often overrepresented. The workforce at NAFDAC, the FMoH, and other Government of Nigeria stakeholders is predominately female, so more women were represented in PQM training events and workshops. ● Stakeholder awareness of PQM’s phase-out plan is not uniform, especially among manufacturers. Those who are aware of the plan had only a vague understanding of it. It is not clear how much, and to what degree, PQM has shared the plan versus how much stakeholders remember. ● Lack of outcome-level indicators. This makes it difficult to measure achievements and determine to what degree they are being met. CONCLUSIONS AND KEY RECOMMENDATIONS TA from PQM has been critical to the capacity building of a number of functions within Nigeria’s medicines supply chain. Notable among these is the improved regulatory capacity of NAFDAC partners at the national level, and the concomitant increase in the agency’s laboratory testing capacity. With the assistance of PQM and its commitment to improving the agency’s human resources, technical capacity, and borders, drug resistance, complex distribution and supply channels, and large populations. Additional high-risk medicines include new medicines and commodities and those with the greatest safety risk profile. 6 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV technological monitoring systems, NAFDAC is in a stronger position to oversee activities in Nigeria. PQM is the only partner working on these issues in Nigeria, particularly laboratory capacity, so improvements in this sector are largely attributable to its work. PQM has also moved the manufacturing sector forward in its efforts to produce high-quality finished pharmaceutical products through the employment of GMP. The activity is considered a leading expert in this sector, and its assistance, particularly the opening of a local office, has been critical to the advancement of manufacturing product improvement. Given its close work with leading companies in this sector, PQM can be credited with increases in the quality and volume of certain priority MCH and malaria products. The evaluation team recommends that TA continue unencumbered, with only a few changes to better assist stakeholders transition out of PQM by September 2019. Key recommendations for PQM: 1. PQM should help NAFDAC develop a formal sustainability plan that incorporates individual approaches to sustainability into one cohesive plan. 2. PQM should help NAFDAC to undertake a strengths, weaknesses opportunities, and threats (SWOT) analysis to identify assets that can help sustain PQM and threats to sustainability. 3. PQM should work collaboratively with NSCIP to provide TA for the implementation of the NQAP. Key recommendations for USAID: 1. Increase collaboration with complementary supply chain and health system strengthening programs, such as Global Health Supply Chain Procurement and Supply Management (GHSC-PSM) activity and the Integrated Health Program (IHP): ● USAID/Nigeria should request that PQM collect data from manufacturers about the number of commodities available (or that may soon be available) so that information can be shared with procurement agents in the private and public health sectors. The Mission is well positioned to share that information with state public health facilities via the GHSC-PSM and IHP. ● PQM collaboration with IHP may be a cost-effective, streamlined way to help states pool their resources to buy medicines in bulk. The outcome would be two-fold. First, smaller states will increase purchasing power and manufacturers can better plan production. Second, because the goal of IHP is to strengthen health systems at the state level, and regulatory systems are part of the overall health system, IHP and PQM can combine resources to offer a more holistic approach to health system strengthening. 2. Mandate that NAFDAC develop a formal sustainability plan or framework to sustain regulatory system strengthening activities as a precondition of a follow-on award. 3. Advocate to the FMoH to expand use of the District Health Information System 2 (DHIS-2) to other Government of Nigeria agencies and subscribe for cloud data storage. 4. Include outcome-level indicators in the PQM Activity Monitoring, Evaluation and Learning (MEL) Plan. 7 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV SECTION 1. INTRODUCTION EVALUATION PURPOSE AND OBJECTIVES The purpose of this midterm performance evaluation is to provide the United States Agency for International Development (USAID)/Nigeria an objective assessment of Promoting the Quality of Medicines (PQM) activity. Specifically, the evaluation was conducted to: ● Assess the effectiveness of PQM’s technical assistance (TA) in two areas: strengthening medicines regulatory quality assurance (QA) systems in Nigeria and building the capacity of local manufacturers to produce high-quality medicines; ● Identify accomplishments and challenges in activity implementation to improve efficiency and highlight opportunities for adjustments in current technical activity; and ● Provide recommendations to USAID/Nigeria for potential future investments in medicines QA systems strengthening and, specifically, in strengthening the capacity of local pharmaceutical manufacturing. EVALUATION QUESTIONS The evaluation was designed to answer five questions, as outlined in the scope of work and developed in collaboration with the USAID/Nigeria Office of Health, Population, and Nutrition (HPN). The questions were designed to assess PQM’s performance and identify any areas where TA may still be required to help strengthen Nigeria’s regulatory systems and manufacturer capacity. The questions also serve to provide information on what type of assistance may be required to sustain PQM activities beyond the life of the activity. Evaluation Question 1: To what extent has PQM’s TA been effective in strengthening QA systems for medicines in Nigeria? ● PQM has provided TA to the National Agency for Food and Drug Administration and Control (NAFDAC). To what extent has this TA improved NAFDAC’s core functions? Is TA still needed? If so, what type of TA? ● Is targeting NAFDAC the most effective strategy for improving medicines QA systems in Nigeria or are there other regulatory bodies that should receive TA? If so, which other regulatory bodies and what type of TA? ● Do NAFDAC quality control (QC) laboratories have the technical and human resource capacities to handle the medicines QC and analysis requirements for Nigeria? If not, what type of TA is needed? What type of resources should be provided? Evaluation Question 2: To what extent has PQM’s TA to local manufacturers improved the production capacity and quality for several priority maternal and child health (MCH) and malaria medicines [chlorhexidine, amoxicillin dispersible tablets, oral rehydration salts (ORS)/zinc sulfate, oxytocin, Artemether+Lumefantrine, and Sulphadoxine/Pyrimethamine]? What milestones have been achieved? What, if any, milestones still need to be achieved for local manufacturers to produce these quality-assured medicines? Evaluation Question 3: What are the perceptions of local pharmaceutical manufacturers towards PQM’s TA? Have they made plans to sustain or expand local production of quality-assured priority medicines beyond the life of the project? 8 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Evaluation Question 4: To what extent has PQM’s TA been effective in increasing the capacity of national and state regulatory agencies to utilize medical product quality information for decision-making? Is the TA still needed? Evaluation Question 5: What type of plans have national and state health officials made to sustain the regulatory systems for medicines, including the internationally accredited laboratories, beyond the life of the project? Are there any gaps in plans? 9 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV SECTION 2. PQM BACKGROUND This section discusses the PQM global program and USAID/Nigeria’s implementation of PQM. PQM GLOBAL PROGRAM PQM is a global program managed by USAID’s Office of Health Systems (OHS), which is located in the Bureau of Global Health. It is USAID’s primary mechanism to help assure the quality, safety, and efficacy of priority medicines, with the overall goal of ensuring the quality and safety of medical products by strengthening QA systems. PQM seeks to accomplish this by implementing activities designed to strengthen medical products QA systems, increasing the supply of quality-assured medicines, and increasing the utilization of medical product quality information for decision-making. PQM is a 10-year centrally managed cooperative agreement awarded to the United States Pharmacopeia (USP). In 2009, the activity had an initial $35 million ceiling; this was reached in 2014. A costed extension was awarded in August/September 2013 with a ceiling increase to $110 million. The increase was a direct response of unparalleled demand from USAID/Washington and country Missions for uninterrupted and increased activity TA. PQM has met its ceiling and is on track to close September 2019. In 2013, PQM commenced in Nigeria by providing TA to improve the quality of malaria medicines, with a total fiscal year obligation of $350,000. This included strengthening NAFDAC’s regulatory capacity. In 2014, the activity TA was extended to building the local manufacturing capacity of priority MCH medicines with a budget of $850,000 ($500,000 for malaria and $350,000 for MCH). Since then, PQM funding obligations have steadily increased to the current fiscal year 2018 level of $2.5 million. RESULTS FRAMEWORK PQM is a dynamic program that has evolved to meet the changing needs of USAID and in-country stakeholders. Its original 2009 results framework was defined by an overall objective to ensure the quality, safety, and efficacy of medicines of relevance to USAID health programs (Table 1, next page). This results framework had four intermediate objectives3 and four intermediate results (IRs). In 2012, OHS was established within USAID’s Bureau of Global Health, and it absorbed PQM along with other health systems strengthening programs. PQM’s costed extension coincided with the creation of OHS, presenting a unique opportunity to revise the results framework to reflect OHS’ priorities for sustainable system strengthening. The results framework was revised in 2014 but was not fully implemented until 2016. The results framework was revised again in 2016 to include only three IRs and with an objective to sustainably ensure the quality of medicines to protect public health (Figure 1, next page). This change reflected OHS’ increased focus on sustainable activities and results. A significant change was specific to using information for decision-making, with a new IR reflecting the need for regulatory authorities to put “evidence into practice.” This new IR 3 replaced intermediate objective 3 and IRs 3 and 4. 3 The term “intermediate objective” is no longer in use. It has been replaced by “development objective.” Award ID: GHS-A-00-09-00003-00 Award Date: Sept. 18, 2009 Award Type: Cooperative agreement, sole source Performance Period: 2009–2019 Ceiling: $110 million Implementer: USP USAID Office: OHS 10 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Figure 1. PQM Results Framework, 2016–2019 PQM’s approach to TA is based on six key characteristics: holistic; systems-based and sustainable; risk￾based and pragmatic; in-line with international standards; collaborative; and informed by nearly 200 years of experience. PQM’s technical approach (Figure 2, next page) has evolved based on lessons learned and OHS priorities.4 PQM IN NIGERIA Since 2013, PQM has helped Nigeria to address significant threats to public health caused by substandard, falsified, and unapproved medicines. The need for TA was urgent not only because of the threat to public health, but also because poor-quality medicines resulted in the waste of scarce resources, undermining decades of USAID health investments in the country. PQM serves a critical mandate: ensure quality, safety, and efficacy of medicines throughout Nigeria with an emphasis on USAID and Government of Nigeria (GoN) priority malaria and MCH medicines. PQM’s overall theory of change is that if local manufacturing 4 USAID’s Strategic Health Framework 2012–2016: Better Health for Development. http://pdf.usaid.gov/pdf_docs/pdacu025.pdf TABLE 1. PQM INTERMEDIATE OBJECTIVES AND INTERMEDIATE RESULTS, 2009–2015 OVERALL OBJECTIVE: TO ENSURE THE QUALITY, SAFETY, AND EFFICACY OF MEDICINES OF RELEVANCE TO USAID HEALTH PROGRAMS INTERMEDIATE OBJECTIVES INTERMEDIATE RESULTS 1.To strengthen national QA systems More developing countries have a better functioning/fully operational medicines QA system in place 2.To increase the supply of good-quality medicines of direct relevance to priority USAID health programs Increase availability of good-quality medicines of direct relevance to priority USAID health programs 3.To combat the availability of substandard and counterfeit medicines Reduced presence of substandard and counterfeit medicines in the supply chain of developing countries 4.To provide technical leadership and global advocacy regarding the importance of medicines QA Improved medicines QA tools and mechanisms, increased awareness of their importance, and increased funding for their implementation and use 11 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV capacity to produce quality medicines is increased and regulatory systems are strengthened, then Nigerians’ access to quality-assured medicines will increase. Figure 2. PQM’s Technical Approach The activities under the three IRs contribute to the production of and strengthening medicines QA systems and work in synergy to ensure the availability of quality medicines. The first IR works to strengthen medicines regulation and QA systems, the second to increase the supply of quality-assured priority medicines, and the third to utilize medical product quality information for decision-making. IR 1: Strengthen medical product QA systems. A key obstacle to promoting quality-assured medicines and combating substandard and falsified medicines in Nigeria is the lack of institutional, financial, technical, and human resource capacity in medicines regulatory systems to protect supply chains. Medicines QA depends to a large extent on the capacity of the national regulatory authority’s ability to safeguard the quality, safety, and efficacy of the medicines in the market. IR 2: Increase the supply of quality-assured priority medicines. Quality-assured medicines are not readily available; governments, development partners, health facilities, and/or patients have little choice but to use medicines that have not undergone rigorous regulatory oversight. Quality-assured, efficacious, and safe medicines are needed to improve positive health outcomes. PQM works to increase the supply of quality-assured medicines of direct relevance to priority USAID health programs, including malaria and maternal, newborn, and child health (MNCH). It provides technical support to enable manufacturers to comply with international standards for Good Manufacturing Practices (GMP) and, ultimately, to receive stringent regulatory authority approval for essential drugs, allowing the manufacturer to build stronger QA systems and satisfy medicines regulatory requirements for marketing authorization and procurement. IR 3: Increase the utilization of medical product quality information for decision-making. Poor-quality medicines pose a grave threat to patients, but the problem is largely unknown to the public. PQM uses a medicines quality monitoring post-marketing surveillance program developed with assistance from PQM and the medicine regulatory agency to establish a system to regularly examine the quality of medicines circulating in its markets. NAFDAC NAFDAC is an implementation agency within the Federal Ministry of Health’s (FMoH) Department of Food and Drug Services. Its mandate is to safeguard public health through the regulation and control of 12 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV the manufacture, importation, exportation, distribution, advertisement, sale, and use of food, drugs, cosmetics, chemicals, detergents, medical devices, and packaged water (known as “regulated products”). NAFDAC ORGANIZATION NAFDAC is led by a Director General (DG) who is a Presidential appointee and serves as long as the President is in office, a maximum of two five-year terms. A new President may request that the DG continue to serve, but this is the exception, not the rule. NAFDAC comprises 14 Directorates, each is led by a Director who reports to the DG. Six Directorates received TA from PQM: 1. Planning, Research and Statistics (PRS) Key tasks: Designing operations research to support evidence for decision-making; monitoring and evaluation (M&E) of NAFDAC’s annual work plan, and compiling data for quarterly, biannual, and annual reports; reviewing reports of foreign GMP inspections; and overseeing information and communications and technology (ICT) for agency-wide reporting. 2. Laboratory Services I & II: LS I (Drugs) and II (Food) Key tasks: Testing medicines and food through the network of National Quality Control Laboratories (NQCL), and determining the quality, safety, and efficacy of imported and domestically produced regulated products. 3. Drug, Evaluation, and Research (DER) Key tasks: Implementing Quality Monitoring Systems to support clinical trials, document review, evaluation of food and medical products, and GMP inspections. 4. Registration and Regulatory Affairs (R&R) Key tasks: Reviewing dossier5 applications for registration of products in Nigeria; issuing marketing authorization to manufacturers and importers of pharmaceutical products; and suspending, withdrawing, or canceling a registration and marketing authorization in the event of quality, safety, or efficacy issues. 5. Ports Inspection Key tasks: Monitoring and controlling regulated products imported into Nigeria, issuing certification for exportation of regulated products, and routine sampling of products for laboratory analysis. 6. Pharmacovigilance and Post-Marketing Surveillance (PV-PMS) Key tasks: Implementing the National Pharmacovigilance Policy and carrying out routine PMS of regulated products. ACTIVITIES BY IR The following section outlines the main activities carried out under each IR. These activities work in concert to achieve the two PQM/Nigeria key objectives: 1. Increase the capacity of NAFDAC to ensure the quality and control of anti-malarial and MNCH priority medicines in Nigeria 2. Increase the availability of quality assured priority medicines 5 A dossier contains information compiled by the manufacturer on a product’s composition, including active pharmaceutical ingredients (APIs), pharmacological profile, quality control testing of APIs and the finished product, and clinical trial results, including toxicology, efficacy, and safety. 13 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV PQM implemented regulatory system strengthening activities throughout Nigeria. It provided TA to NAFDAC’s drug QC laboratory (Yaba in Lagos), its zonal laboratories (Agulu in Anambra State, Malali in Kaduna state), and the National Institute for Pharmaceutical Research and Development (NIPRD) laboratories (Idu in Abuja). IR 1 ACTIVITIES IR 1 activities, listed below, seek to improve laboratory standards, attain and maintain internationally recognized certifications [e.g., International Organization for Standardization (ISO) 17025], and train PV￾PMS staff on how to undertake PMS studies (testing of medicine samples from different locations and facilities in Nigeria). Samples are screened and undergo confirmatory compendial testing in laboratories. 1. National regulatory systems strengthened: PQM provided continual TA and advocacy for the development of the National Quality Assurance Policy (NQAP) and its adoption by Nigeria’s National Council of Health at its 2016 annual meeting. 2. Development of the PMS Program Implementation Framework and Guideline: PQM provided TA to NAFDAC to strengthen the capacity of its PMS unit to undertake studies that monitor the quality of medicines in the country’s supply chain. 3. Sustained accreditation of NAFDAC central and zonal drug control laboratories in Yaba, Agulu, and Kaduna: PQM delivered TA to these laboratories, all of which have been officially accredited. In addition, the NIPRD QC laboratory in Abuja is on track to attain international accreditation; PQM also provides TA to this laboratory. IR 2 ACTIVITIES PQM’s tailored approach extends to local pharmaceutical manufacturers, assisting selected companies in improving their GMP compliance, developing dossiers for medicines, and supporting manufacturers to comply with international standards. There are two activities under IR 2: 1. Availability of quality medicines increased: PQM GMP specialists supported local manufacturers of USAID priority medicines to improve GMP compliance and develop dossiers to submit to World Health Organization (WHO) Prequalification (PQ) of Medicines Program for certification. 2. Manufacturing sites complying with GMP standards increased: PQM provides TA to Nigerian manufacturers that produce ORS, zinc sulfate tablets, chlorhexidine digluconate gel, and other MCH priority commodities. The supply of locally produced quality-assured medicines was increased through consistent TA provided to 11 local manufacturers. 6 IR 3 ACTIVITIES There is one activity under IR 3: 1. Capacity to detect poor medical products increased: PQM helps combat substandard and falsified medicines by collaborating with country medicines regulatory authorities and national health programs by establishing or strengthening PMS systems that regularly examine the quality of medicines circulating in markets. PQM supports the national regulatory authorities to assess existing medical products by selecting sites to monitor based on criteria such as epidemiology, geography, border region, and history of trafficking falsified medicines. 6 Emzor, Swiss Pharma, Phamatex, May & Baker, Juhel, Tuyil, Daily Needs, Drugfield, CHI, Dana Dabs, and Nemel. 14 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV SECTION 3. METHODOLOGY EVALUATION DESIGN This evaluation was designed as a qualitative research study utilizing primary and secondary data. Primary qualitative data was derived from key informant interviews, including stakeholders who received and/or are receiving TA. PQM provided the names of stakeholders and their contact information. In order to obtain a broader understanding of stakeholders’ experiences with PQM at all levels of implementation, the evaluation team invited respondents in leadership positions and their staff to participate in the interviews. Chemonics International, a USAID IP that does not receive TA from PQM but interacts with the activity via the Global Health Supply Chain Procurement and Supply Management (GHSC-PSM) activity, was also included. Secondary data was extracted from documents provided by USAID, PQM, and stakeholders. The combination of primary and secondary data allowed the evaluators to construct a holistic picture of what has been accomplished (or not accomplished), the ways in which the results were accomplished, and approaches that were not effective. Understanding the “how” component of PQM’s performance was essential to generating “multi-dimensional” results that can support evidence-informed decision-making for both USAID and PQM stakeholders. Due to a lack of outcome indicators in the PQM Performance Monitoring Plan, the effectiveness of TA could not be directly measured. In lieu of direct measurement, the evaluation team asked stakeholders to outline what they considered to be the most significant change in their Directorate, department, agency, or company as a result of PQM TA. TYPES OF DATA PRIMARY RESEARCH DATA: KEY INFORMANT INTERVIEWS The evaluation team collected primary data through key informant interviews. PQM/Nigeria contacted stakeholders in advance of data collection to inform them of the evaluation, and provided the evaluation team with contact names, emails, and phone numbers. With the exception of manufacturers, the evaluation team arranged interviews directly with the stakeholder contact. PQM/Nigeria scheduled interviews directly with manufacturers because they are not familiar with USAID evaluations and their premises are highly restricted to minimize the risk of intellectual property theft. In-person interviews were conducted in Abuja, Lagos, and Kaduna by a minimum of two evaluation team members. Respondents who were unavailable to attend in person were interviewed by phone or Skype. Key informant interviews (KII) captured stakeholder perceptions and experiences regarding PQM activities and how well PQM’s assistance helped them achieve mandates. The interviews provided clarification, context, and triangulation with respect to work plans, technical documents, and success stories. Interviewers scribed their notes during each interview. Following the interviews, interviewers merged notes and identified areas for follow-up. Interviews were recorded when feasible and when respondents gave permission. Recordings were used for note taking purposes only, not transcription. Table 2 (next page) shows the number of groups interviewed within each stakeholder type. Interview guides are included in Annex II, and a detailed list of respondents is in Annex III. 15 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANALYSIS METHODS Interview transcripts were thematically coded in NVivo. Key themes included the quality and effectiveness of TA, challenges (external and internal to PQM), evidence for decision-making, and sustainability. SECONDARY RESEARCH DATA: DOCUMENT REVIEW Secondary data were derived from PQM quarterly reports, work plans, technical briefs, and communication materials; Nigerian stakeholder policy documents and reports; special studies (e.g., NAFDAC gap analysis report); and the global PQM evaluation report.7 Information from these documents provided background and context, in addition to triangulation with interview responses. Annex IV contains the full list of documents. LIMITATIONS AND MITIGATION MEASURES ● Leadership and staff turnover resulted in shortened institutional memory of PQM. Some responses are skewed to reflect more recent experiences with PQM. This was mitigated by supplementing interview notes with information from document reviews. ● PQM does not have outcome-level indicators, only output-level, so objective measurement of effectiveness is difficult. This was mitigated by capturing stakeholder stories about what they perceive as the most significant change resulting from PQM TA. Recorded interviews were not transcribed; therefore, the availability of direct quotes was limited. This was mitigated by using two or more interviewers to ensure the integrity and accuracy of available quotes. ● The staff who work most directly with PQM were not always present in interviews despite requests by the evaluation team. This limited information on more detailed aspects of PQM’s TA. The evaluation team requested names/contacts of people who could provide more information and, when possible, followed up directly with those individuals. There were no security limitations carrying out the evaluation. 7 The Evaluation Team was provided access to the report and granted permission to quote the report by USAID and the Global Evaluation Team. TABLE 2. KEY INFORMANT INTERVIEWS STAKEHOLDER NUMBER OF GROUPS INTERVIEWED WITHIN EACH STAKEHOLDER TYPE Regulation and control (NAFDAC and FMoH) 9 NQCL 5 Manufacturers, including the Pharmaceutical Manufacturers Group of the Manufacturers’ Association of Nigeria (PMG￾MAN) 11 Other non-GoN agencies 1 Other non-GoN regulatory agencies 1 Supply chain stakeholders (e.g., the Nigerian Supply Chain Integration Project, or NSCIP) 3 PQM/Nigeria 2 USAID/Nigeria/HPN 3 TOTAL 35 16 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV SECTION 4. FINDINGS AND CONCLUSIONS IR 1: STRENGTHEN MEDICAL PRODUCT QUALITY ASSURANCE SYSTEMS EVALUATION QUESTION 1 To what extent has PQM’s TA been effective in strengthening QA systems for medicines in Nigeria? 1a) PQM has provided TA to NAFDAC. To what extent has this TA improved NAFDAC’s core functions? Is TA still needed? If so, what type of TA? OVERVIEW The core functions of NAFDAC—to control and regulate food and medical products—are supported by the work of its 11 Directorates. PQM supported six of these to increase NAFDAC’s overall capacity to regulate and control medicines by ensuring GMP compliance, dossier review for registration, laboratory testing of sampled products, and PMS. TA to laboratories is addressed separately in question 1c (p. 17). QUALITY AND EFFECTIVENESS OF TA Overall regulatory system strengthening. A crosscutting regulatory system strengthening outcome is the development of the NQAP. The activity provided TA to NSCIP to develop a policy to provide a common definition of safety, quality, and effectiveness of medicines and health products to organizations, companies, and the distribution sector. Moreover, standards did not exist for the ways in which medicines or health commodities were distributed and monitored through the supply chain. The result was a supply chain with varying degrees of quality, safety, and effectiveness that negatively affected the product that reached end users. The purpose of the NQAP is to align stakeholders against one definition of quality and mandate that medicine and health products are distributed in accordance to Good Distribution Practices and tested by Nigerian laboratories with a valid certificate in Good Laboratory Practices. The policy applies to the public sector, including other GoN agencies, the private sector, and civil society, including international NGOs. The NQAP also describes NAFDAC’s mandate to ensure Marketing Authorization Holders (MAHs) products adhere to all of the agency’s regulatory requirements (e.g., product specification, packaging, and labeling) through surveillance at all points of the supply chain, not only product registration. The NQAP outlines that NAFDAC ensure MAHs and their subcontractors regularly test their commodities for quality at all levels of the supply chain, including storage, distribution, and end use, to ensure they are not substandard or falsified. If any commodity does not meet regulatory requirements, NAFDAC will act to withdraw the product from circulation. The result is that NAFDAC and other government agencies and stakeholders tasked by the FMoH to implement the NQAP act cohesively to ensure there are no weak links in the supply chain so end users receive a safe and efficacious product. NAFDAC leadership. The DG has praised PQM, which she described in an interview as “a philosophy and not just a program,” and is highly committed to ensuring the success and sustainability of its activities. She also made it clear that human resources are NAFDAC’s most important asset, and that she is committed to building and maintaining the agency’s capacity in this regard. Before the NQAP it was chaos. Everyone had their own definition of quality and testing was irregular along the supply chain. The NQAP is the crowning achievement of NSCIP. — NAFDAC respondent “The PQM/Nigeria staff are VERY knowledgeable and are well aware of the challenges and context in which they work.” — DG, NAFDAC 17 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Upon assuming her role as the DG, she immediately developed and implemented new policies based on results from a NAFDAC gap analysis conducted by PQM. (that she referenced in a speech she gave when she became DG.) Based on the results, she was able to identify some important challenges that could be rectified quickly. For example, the gap analysis showed that NAFDAC staff were not consistently using their government emails to conduct business. She related her belief that using official email is essential to demonstrate the credibility and professionalism of NAFDAC and its staff, and today all staff required to use their official email for all business. Another policy change aimed to decrease the number of trained laboratory staff transferring to positions where their training was not relevant, a problem identified in the gap analysis. (This was also a problem among Regulatory and Registration staff.) Low morale due to lack of upward mobility and the absence of a professional cadre motivated some laboratory staff to seek other jobs. Previously, the Administration and Human Resource Department, not Directors, was responsible for releasing or accepting staff during transfers, which often meant competent laboratory staff moved to departments where their skills were of no use. The DG instituted a policy that Supervisors and Directors must approve transfers of laboratory staff, and that a person could not move to a position where their skills were not relevant. The DG is also addressing morale by working with Laboratory Directors to develop incentive schemes to encourage staff to stay, such as a professional laboratory cadre to ensure staff have the opportunity for advancement. GMP compliance. DER supports NAFDAC’s overall mission by ensuring locally manufactured food and medical products are produced, stored, and distributed according to GMP. It is tasked with conducting GMP inspections of local and foreign facilities. Inspection procedures depend on the type of facility and manufacturing processes (e.g., sterile or non-sterile facilities), and the ability of inspectors to carry out different types of inspections is critical to risk management. Risk management is an essential component of sustainability because it allows resources to be prioritized for inspection of medicines and health products that carry the greatest risk to end users if QA procedures are not followed. For example, medicines and health products that require sterile manufacturing processes have a greater inherent risk than non-sterile products; prior to PQM, DER used the same inspection procedures for both. PQM provided classroom and on-the-job training to teach the DER inspectors how to undertake different types of GMP inspections and how to review dossiers to ensure that manufacturers were following the appropriate GMP. Recipients of the PQM training described it as high-quality, practical, and hands-on. A supervisor reported that inspectors have the “confidence to carry out their duties” and the supervisor has confidence in their abilities. Dossier reviews for product registration. The ability of NAFDAC to critically evaluate a dossier is the foundation for initial product registration, renewal, and license revocation. Prior to 2013, DER was responsible for reviewing dossiers; after the agency was restructured in 2013, that responsibility was moved to the R&R Directorate. (Although there are some areas of overlap, the Directorate does have primary responsibility for dossier reviews.) Before PQM TA, its staff had limited capacity to undertake a thorough and high-quality dossier review, and the Directorate did not have sufficient processes or the foundational knowledge of the technical components. When a dossier was submitted, there was no systematic process for accepting it, no tools and templates for review, and no system to issue approvals (or non-approvals). The staff assigned to conduct the reviews were sometimes unqualified to review medicine dossiers, lacking the appropriate background and/or the capacity to critically evaluate a dossier. “Before PQM we approached inspections as ‘one size fits all.’ Now inspectors can inspect both sterile and non-sterile facilities.” — DER respondent 18 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV NAFDAC staff said PQM provided “high-quality” training on the preparation necessary to review a dossier and increased capacity to critically evaluate various types of medical and health commodities and GMP processes. PQM helped the R&R Directorate develop dossier review templates and tools and standardize review processes. TA resulted in consistent, high￾quality critical reviews, and the processes put in place ensured a standardized, systematic, and efficient review. An R&R Directorate respondent noted that some of the key remaining gaps (e.g., capacity of personnel to effectively review certain types of product dossiers and conduct new types of GMP inspections) are being addressed. A work plan is under development that will comprehensively address those gaps, as well as others as identified in the gap analysis report. Post-marketing surveillance. The PV-PMS Directorate is responsible for coordinating and leading regular, systematic active and reactive surveillance of medicines. PQM provided TA to the Directorate to develop standardized processes and procedures for PMS surveys, and trained staff at the national and zonal levels on sampling and screening techniques and data analysis methods. PQM supported PV-PMS on four rounds of PMS surveys (one for MCH and three for malaria). Through increased capacity, PV-PMS staff generated information that contributed to more than 700 regulatory actions and provided NAFDAC with evidence to raise public awareness about substandard and falsified medicines throughout Nigeria. The activity supported NAFDAC to convene stakeholders and MAHs to discuss PMS results and determine follow-up regulatory actions. It also assisted the Director of PV-PMS to identify “PMS Champions” as a way to ensure the sustainability and scale up of surveillance activities after PQM. PMS Champions will build the capacity of MAHs, distributors, central medical stores, and hospital pharmacy managers to develop and implement QA monitoring systems. M&E and data management. The PRS Directorate plays a crosscutting function across all NAFDAC Directorates and is central to providing data in support of evidence-based decision-making for all Directorates. It is responsible for coordinating M&E activities and undertaking operations research to inform policymaking. PRS maintains IT infrastructure and manages the database that houses PMS data. Before PQM, the Directorate had limited capacity to track anything other than activity-level indicators, and its use of data for decision-making was insufficient due to the Directorates’ limited capacity to use the data and types of data that were not useful. Moreover, the Directorates lacked “M&E Champions” to support better reporting and use of data. With PQM’s assistance, PRS identified M&E Champions in each Directorate and provided training on basic M&E practices, including on data quality procedures. PQM increased the capacity of M&E staff to collect output- and outcome-level data and synthesize the findings into communication briefs. PRS’ ability to synthesize medical product information into communication briefs, tailored to the decision-maker, is a significant step for regulatory action. CHALLENGES/THREATS TO SUSTAINING ACHIEVEMENTS It is unequivocal that the current DG is committed to increasing and maintaining NAFDAC’s human resource capacity. This was made clear by medium- and long-term goals that build directly on the foundational accomplishments of PQM. In her short tenure, she has made some small, yet critically important, changes based on results of the gap analysis; these smaller quick wins have been critical in laying the foundation for more significant changes. The bigger changes will require significant resources that must be planned and budgeted well in advance and sustained over time (e.g., financial resources to increase the number of laboratory staff and purchase/maintain modern laboratory equipment). Currently, NAFDAC “PQM TA has had a huge impact on R&R human capacity development. PQM has had a big footprint in this regard. They are diligent and focused.” — NAFDAC respondent 19 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV has significant financial obligations that, although improving under the DG, call into question if the agency can realistically fund 100 percent of PQM activities by September 2019. A formal sustainability plan can help guide planning and budgeting, but NAFDAC does not have an agency sustainability plan or a formal plan to ensure the sustainability of regulatory system improvements resulting from PQM. Although it has adopted approaches to sustainability, they are not a substitute for a formal sustainability plan. With no plan—as well as significant financial obligations—it is not possible to determine if NAFDAC can sustain improvements made under PQM for the long term. The DG said that sustainability is an ongoing process, and therefore will be ensured by incorporating PQM activities into NAFDAC quality management systems processes. However, the evaluation team is not convinced this is an adequate response to sustaining a large, multifaceted activity like PQM, especially because sustaining regulatory systems is resource-intensive and must always evolve to meet new challenges. NAFDAC will have to continually advocate to the GoN and other stakeholders for funding, but its ability to advocate may be hampered by a lack of a formal sustainability plan. NAFDAC and other stakeholders outlined a number of assumptions regarding what resources may be available in the future, such as laboratory revenue, and threats PQM may face, such as unsupportive government policies. It is important to note, however, that these are assumptions, not actual evidence of assets or threats. This suggests that there has been no formal internal review of PQM activities, even at the Directorate level, as assets and assumptions are discussed in vague terms and as a series of “what ifs?” The NAFDAC gap analysis was published almost a year ago, at the time of the evaluation, and has been a helpful analytical road map for guiding PQM activities; however, the analysis did not indicate to what degree NAFDAC had addressed the remaining gaps that fall outside of PQM’s scope. The results of this evaluation, though they may be helpful for NAFDAC’s future planning, will be insufficient to inform internal resource allocation. NAFDAC itself is in the best position to identify assets and liabilities. Moreover, the gap analysis identified important areas that must be addressed to ensure the success of PQM, but many fall well outside the activity’s scope (e.g., legal enforcement). NAFDAC is responsible for addressing those challenges and tracking progress. The DG is a Presidential appointee and serves as long as the President is in office, a maximum of two 5- year terms. A new President may ask a DG to extend their term, but this is the exception, not the rule. If a new DG is appointed, they may not continue policies that support PQM activities and may not prioritize laboratories as the current DG is. In other words, the sustainability of the achievements made with PQM support is not assured. Although DER inspectors have greatly increased their capacity, DER leadership say they need more qualified inspectors. At present, there are not enough highly-trained inspectors to keep up with current demand. The NQAP has not been implemented, and there are no guidelines on how to implement the policy. Implementation at the national and state levels is the responsibility, respectively, of the FMoH’s Department of Food and Drug Services and the Department of Pharmaceutical Services. Though the FMoH has touted the importance of the policy and its commitment to implementation, no progress has made since 2016. There is no indication that an effort is underway to move the NQAP from a road map to an operationalized policy. Without a fully implemented medicine QA policy at every stage of the supply chain, ensuring medicine quality by everyone (if at all) will The Federal Ministry of Health is fully committed to the provision of a good quality assurance system guided by the goals and strategies of the NQAP. As a sign of this commitment, the coordination and supervision of implementation of activities under the NQAP will be strengthened by establishment of relevant coordinating and strategic units, both at the federal and local levels of the ministry. — NQAP Preface 20 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV remain a fragmented and non-standardized effort. NAFDAC stakeholders were also concerned that, even when guidelines are developed, the states will not have the capacity to adequately implement the policy. National-level PRS and PV-PMS staff told the evaluation team that they had greater capacity and carried out their duties with minimal PQM support. However, they also said capacity at the state level was low and inadequate to meet increasing regulatory and control activities. The issue of low state capacity is not unique to PRS and PV￾PMS—it is an agency-wide challenge. Staff said they felt confident to train state-level staff but they did not have the time or staff availability. Simply issuing of a regulatory action does not protect consumers from substandard and falsified medicines. Consumers are protected only when the action is enforced. The ability of NAFDAC’s Investigation and Enforcement Directorate to confiscate and destroy substandard and falsified medicines is hampered by a lack of staff, capacity, and material resources. In 2016, the Directorate arrested only 23 drug hawkers. The challenges facing PRS are testament to its success. Though the District Health Information System 2 (DHIS-2) database is replete with data—it houses all PMS data, as well as data from the National District Health Information System—they are siloed within the FMoH. Furthermore, other government ministries and agencies do not have access to the DHIS-2, meaning PRS must manually extract data for requestors. PRS is struggling to meet the increased demand for and amount of data. The PV-PMS Directorate has a similar challenge: It also has a lot of data and decision-makers who need it quickly, so if laboratory testing takes too long, the process is slowed down even further. Due to limited staff, the time required for sample collection to laboratory testing and synthesis of findings is too slow for situations that require quick action. The concern is that decision-makers will make decisions without data if they are unable to obtain it in a timely manner. OPPORTUNITIES FOR NAFDAC PRS is working with mobile phone service providers to secure medicine verification data from NAFDAC’s automated Mobile Authentication System, which enables consumers to check the authenticity of a medicine by texting a verification code located on the box. Consumers then receive an automated response that confirms the authenticity of the medicine. Currently, NAFDAC does not have access to the data, so it does not know how many medicines are inauthentic or the geographic location of inauthentic medicines. PRS is working with mobile phone companies to secure these data to better generate statistics and determine if there are geographical “hot spots” that require closer monitoring. Supplementing PMS data with data from the Mobile Authentication System will greatly increase the scope of surveillance data and better support risk-based surveillance. Respondents said the capacity of NAFDAC staff at the national level is good. This is supported by the fact that these HQ staff perform their work with little to no PQM assistance. National-level staff are confident of their training skills, and using them to train state and zonal office staff is potentially the most cost￾effective means of sustaining capacity building. However, the evaluation team acknowledges the formidable challenge of making HQ staff available to deliver training with enough frequency and intensity to build capacity. “The capacity of staff at the head office has been developed and they have excellent capacity for PMS. However, this is not the case at the zonal and state office levels. Experts at the head offices [and] once in a while experts at the head office take time to train staff at the zones and states, but this is hardly adequate.” — PV-PMS respondent 21 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV CONCLUSIONS PQM TA has been successful at building NAFDAC capacity at the national level. TA is effective and should continue as planned through the remainder of implementation. The evaluation team found no evidence that any major course corrections should be made at this point. The biggest threats to the success of PQM are external to the activity and USAID, but there are some smaller recommendations that can help ensure the bigger threats are mitigated. KEY RECOMMENDATIONS Cost-effective direct PQM TA ● NAFDAC will greatly benefit from a regulatory system strengths, weaknesses, opportunities, and threats (SWOT) analysis. PQM is uniquely positioned to assist in this endeavor. A SWOT analysis is inexpensive, relatively quick, and by nature produces results that can be quickly operationalized. ● NAFDAC would benefit from activity assistance to develop a formal PQM sustainability plan that incorporates the individual approaches to sustainability, including the Collaborative Learning Model (CLM) and R&R Directorate mentorship. PQM knows what resources are required (e.g., types of training) to help NAFDAC prepare for short-, medium-, and long-term needs. The assistance will require minimal cost and produce a high-value outcome. ● A weak supply chain is a significant threat to PQM’s success. The ability of a supply chain to maintain the quality of medicines from the time they leave the manufacturer to the time they reach the end user cannot be overstated. One approach to strengthening the supply chain is through the implementation of the NQAP. PQM can have a big impact in this regard with minimal resources. First, the activity should work collaboratively with NSCIP to provide TA for the implementation. Second, as the success of NQAP is contingent on state and zonal offices implementing the policy via the guidelines, PQM should collaboratively work with NSCIP to develop guidelines and hold training workshops for state-level stakeholders. Cost-effective indirect PQM TA ● The PRS Directorate needs assistance to improve its IT operating systems. The current system is inadequate to meet PMS’ growing requirements. The evaluation team recognizes that this challenge is addressed by an organization other than PQM, but believes the activity can provide valuable insight. PQM can assist PRS to explore public-private partnerships with the for-profit sector (i.e., corporate responsibility initiatives). Nigeria is fortunate to have universities with strong IT programs, and NAFDAC could leverage partnerships these institutions to improve their IT infrastructure to connect with other departments and agencies that would benefit from access to PMS data (e.g., the Nigerian Customs Administration). The commercial sector and universities are well-versed in free open-source software that requires minimal cost for custom tailoring. ● NAFDAC state and zonal offices have significant human resource capacity challenges that must be addressed to sustain the achievements realized with PQM support. Every Directorate included in the evaluation cited low state and zonal capacity for surveillance and control activities as a significant challenge to scale-up of successful initiatives. PQM funds and resources should be shifted from the national level to the state and zonal offices. The activity does not have to provide direct TA to the states, especially because the capacity of national-level NAFDAC staff is high. Instead, PQM resources can be used to implement the CLM approach within NAFDAC, whose HQ staff can train state-level staff, who can then train staff at the zonal offices. USAID/Nigeria 22 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ● USAID/Nigeria can advocate to the FMoH to expand the subscription of DHIS-2 to other GoN agencies that would benefit from FMoH and NAFDAC data. ● If there is a follow-on award, USAID/Nigeria should give serious consideration to shifting PQM TA from the national level to the state level, where the need is now the greatest. ● The Mission should consider if and how PQM can build the capacity of NAFDAC staff at the state level via the Integrated Health Program (IHP). Because this activity’s goal is to strengthen health systems at the state level, and regulatory systems are part of the overall health system, IHP and PQM can combine resources to offer a more holistic approach to health system strengthening. 1b) Is targeting NAFDAC the most effective strategy for improving medicines QA systems in Nigeria or are there other regulatory bodies that should receive TA? If so, which other regulatory bodies and what type of TA? NAFDAC is central to all regulatory system strengthening efforts. There is no other regulatory agency with the legal mandate and broad scope to oversee medicine regulation and control. There are other regulatory bodies charged with overseeing QA, and they are currently receiving or previously received PQM TA. The regulatory body with the most relevance and greatest impact on PQM is the Pharmacy Council of Nigeria (PCN). PCN is recognized as a public regulatory agency that, by law, regulates pharmaceutical manufacturing premises, personnel, and practices. PCN carries out GMP inspections and accredits universities’ pharmacy programs. It received TA to develop a QA/QC curriculum for undergraduate pharmacy students to prepare the future workforce to undertake QA/QC activities in the local manufacturing of medicines. The curriculum is now before the National Universities Commission for consideration of adoption into the basic minimum standard for pharmacy students. Some NAFDAC respondents felt that too many non-government regulatory organizations were involved in medicine QA. The challenge is delineating the roles of regulatory bodies and establishing authority. One NAFDAC respondent described this by saying, “Everyone wants to be involved so they can show their importance. It is a type of ‘moral greed.” They continued: “Greater importance equates with greater resources. For example, [Standards Organization of Nigeria] sets their own standards regarding the acceptable range of active [drug] ingredients. However, that range is larger than what is allowed by SRAs [Stringent Regulatory Authorities]” The evaluation team independently considered if there were other regulatory agencies that have not received TA that should, but could not identify any that were relevant to ensuring quality of medicines. Conclusions There are no other regulatory agencies that should receive PQM TA. KEY RECOMMENDATION PQM should extend TA to PCN on GMP and increase the Council’s GMP capacity to the same level of NAFDAC DER inspectors. 23 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 1c) Do NAFDAC QC laboratories have the technical and human resource capacities to handle the medicines QC and analysis requirements for Nigeria? If not, what type of TA is needed? What type of resources should be provided? Three of the six NQCLs supported by PQM have obtained ISO 17025 accreditation: Yaba and Agulu labs were reaccredited in 2017, and Kaduna lab was due for reaccreditation in June 2018. This is a notable accomplishment, as no labs were accredited prior to PQM’s work in Nigeria. The Yaba lab was the first to achieve accreditation with PQM’s assistance. Using the activity’s CLM approach, Yaba laboratory trained staff in Agulu and Kaduna labs to build their capacity for accreditation. The approach was effective because Agulu and Kaduna achieved accreditation with minimum direct PQM TA. It will be used to train staff in the other four NAFDAC laboratories that are not supported by PQM. Laboratory stakeholders unanimously attributed their success to PQM’s high-quality TA and financial support. No respondent said they had a negative experience, and all had the highest praise. For example, one laboratory respondent said, “PQM TA is excellent and [it] was always available to provide assistance, either physically or on the phone.” The praise is not without merit, given the multiple achievements laboratories have made over the years. As staff build on their successes and gain confidence, they want to undertake more advanced testing. This confidence is reflected in all three labs expanding their scopes (i.e., testing methods) within a one-year period (Table 3). TABLE 3. ISO 17025: 2005 SCOPES LAB FY 2016 FY 2017 SCOPE LIST Yaba Lab 7 17 High-performance liquid chromatography (HPLC); UV-visible spectrophotometry; dissolution testing; disintegration; loss on drying; pH; Karl Fischer; uniformity of dosage; friability; hardness; volumetric titration; melting point determination; polarimetry; sterility testing; microbial limit testing; and bacterial endotoxin testing Agulu Lab 7 16 HPLC; UV-Visible spectrophotometry; dissolution testing; disintegration; loss on drying; pH; Karl Fischer; uniformity of dosage; friability; hardness; volumetric titration; melting point determination; polarimetry; sterility testing; microbial limit testing; and bacterial endotoxin testing Kaduna Lab N/A 7 HPLC, UV-visible spectrophotometry, dissolution testing, loss on drying, pH, Karl Fischer, and uniformity of dosage The only complaint about PQM, which is actually another testament to its success, is that requests for more advanced TA were declined. This was because the TA fell outside of PQM’s current scope. Laboratories have also asked for more equipment to reduce the turnaround time of testing and ensure that equipment is not overworked. Laboratory Services leadership has rightfully pointed out that as technology evolves there is a real need for updated equipment. However, PQM’s budget does not allow for the purchase of more equipment; most lab equipment is donated. “PQM has been a tremendous help especially helping to carry out medicine surveys. They helped support the testing of over 800 samples just at the Yaba lab.” — Laboratory respondent “Prior to PQM the lab could only test 50 percent of medicines according to the official monograph [when a monograph did not exist, testing was performed according to manufacturer specifications]. Now the labs can test at least 95 percent of common drugs. But with the right equipment, and equipment that is in working condition, they could test 100 percent.” — Laboratory respondent 24 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV The three accredited laboratories have passed all previous proficiency testing with PQM TA support. NAFDAC has taken over the cost of reaccreditation and proficiency testing—an outward show of financial commitment to sustainability because these processes are ongoing and expensive. PQM TA is notable for increasing human resource capacity of laboratory staff, but also improving laboratory leadership to better manage labs. Prior to PQM, there were no formal processes for accepting, logging, and processing samples, nor were there internal auditing processes for proactively preventing mistakes or processes to identify what caused a mistake. One laboratory Director stated that TA helped staff but also helped them be a better manager. The Director of Laboratory Services is highly committed to sustainability, and his commitment is evidenced by action. The Laboratory Services Directorate is the only Directorate included in this evaluation that has a formal sustainability plan (currently in a draft phase). The success of this plan is ultimately contingent on an overall regulatory system sustainability plan, but the proactive approach did not go unnoticed. It was clear to the evaluation team that significant consideration was put into the plan. Notable elements of the plan include: ● Staff receive refresher training each month. Staff sent away for training must train all other personnel, even if those personnel do not perform that particular testing method. ● Internal QC meetings are held three times a year to identify gaps in testing and proactively find solutions to those gaps before they can cause a crisis. ● The Directorate established a contract with Merck to help with the bulk purchase of consumables, which keeps costs down. ● Laboratories have proven reliable and the accreditation status has attracted patronage from Catholic Relief Services and the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) to provide testing services of medicines on their behalf. The revenue from the testing will support laboratory costs after PQM. ● NAFDAC has contracted with Nigerian Accreditation Service, a local company, to provide equipment calibration services. This is the only calibration company in West Africa with ISO 17025 certification. This contract is cost-effective and sustainable. ● NAFDAC is expecting a grant from the GoN for the purchase of critical pieces of laboratory equipment. It has stopped purchasing equipment from third-party vendors due to high costs and impropriety, and is working to identify reputable local and foreign vendors. CHALLENGES TO SUSTAINABILITY The greatest threats to laboratory sustainability are external to PQM and USAID. Laboratory leadership and PQM make clear the greatest challenges are sustained financing for accreditation/reaccreditation, “PQM TA has helped management to make decisions on where to move staff [different positions] but also what positions need to be created. The knowledge gained from PQM has helped the laboratory leadership to put systematic processes in place to troubleshoot mistakes when they happen, such as out-of￾specification investigations and root cause analysis, and also proactively prevent mistakes before they occur by regular internal audits to make sure procedures are followed.” — Laboratory respondent “The DG is very supportive of the labs. For the first time in 5–6 years, NAFDAC leadership has approved PT [Performance Testing] procurement requests when we make them. This means we get the materials on time to take the tests on time.” —Laboratory respondent 25 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV materials, and to increase/retain staff. PQM has successfully built the capacity of staff; it cannot control challenges related to financing. It is unknown if NAFDAC will be able to simultaneously fund reaccreditation of its three PQM-supported labs and the accreditation its other labs that the activity does not support. The assumption made by NAFDAC is that part of that funding will come from revenue generated by the accreditation of its other laboratories while sustaining funding for activities (e.g., fees from testing samples from development partners, including Catholic Relief Services and the GFATM). Although NAFDAC is currently able to fund reaccreditation and proficiency testing, and the purchase reference standards and reagents, a number of respondents still expressed concern about whether the agency can maintain financial support over the long term in the absence of laboratory-generated revenue. Some NAFDAC respondents did not feel that staffing levels were adequate to quickly conduct testing for the current volume of samples and would be considerably inadequate to meet future needs. For NAFDAC to take swift regulatory action on medicines, the time from sampling to results must be quick. In addition to increasing staff, laboratory leadership said they did not have a sufficient amount of equipment. NAFDAC is awaiting a grant from the GoN for laboratory equipment and is in discussions with the GFATM and WHO to secure equipment donations, but grants and donations will not cover the cost of equipment training. Maintaining and repairing both basic and sophisticated equipment remains a challenge, though not for a lack of trying. There is no vendor on the African continent that can service more sophisticated laboratory equipment, so a service company must be flown in from abroad. (Currently, the nearest company is in Egypt.) The result is that equipment does not receive preventive service and repairs to broken equipment can take weeks. This leads to overuse of equipment. The evaluation team recognizes that there are no easy solutions for these challenges. PQM has identified 10 laboratory staff who will receive training on preventative maintenance of basic equipment, but training had not commenced at the time of this evaluation. OPPORTUNITIES FOR LABORATORY SUSTAINABILITY The laboratories have a real opportunity to market themselves to local partners as reliable and cost￾effective testing centers. NGOs such as Catholic Relief Services require ongoing testing of samples which, with enough volume, could result in a continuous revenue stream. The revenue from external sample testing can help cover laboratory operational costs. NAFDAC laboratories are increasingly well-positioned to become a regional testing center for other West African countries. Similar regional approaches are under consideration in East Africa, and Nigeria may be able to learn from their experiences. CONCLUSIONS PQM TA has been highly effective in increasing laboratory capacity. No other interventions target laboratories, so the improvements in laboratory capacity as evidenced through accreditation and reaccreditation is largely attributable to activity TA. The improvement in processes, human resources capacity, and sustainability are commendable. Stakeholders are overwhelmingly positive about the quality of PQM TA and expertise; the laboratories are a true success story. The evaluation team recommends that TA continue as planned and sees no evidence that any major course correction is required. 26 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV KEY RECOMMENDATIONS There are a limited number of recommendations within the scope of PQM and USAID. The two recommendations below are high-impact, low-cost modifications. 1. PQM should offer training on how to use new equipment donated by WHO and GFATM. 2. Provide training to laboratory staff on small equipment maintenance and repair. PQM Nigeria can use their local contract staff or use the services of USP in Rockville. IR 2 INCREASING THE SUPPLY OF QUALITY-ASSURED PRIORITY MEDICINES EVALUATION QUESTION 2 To what extent has PQM’s TA to local manufacturers improved the production capacity and quality for several priority maternal and child health (MCH) and malaria medicines (chlorhexidine, amoxicillin dispersible tablets, oral rehydration salts/zinc sulfate, oxytocin, Artemether+Lumefantrine, and Sulphadoxine/Pyrimethamine)? What milestones have been achieved? What, if any, milestones still need to be achieved for local manufacturers to produce these quality-assured medicines? OVERVIEW In 2014 USAID/Nigeria selected PQM to support strengthening the capacity of Nigerian manufacturers to produce priority MCH commodities. The quality of finished pharmaceuticals products begins with the capacity of manufacturers to implement GMP. GMP ensures that quality is built into the product from sourcing and processing of raw materials to the way raw materials are processed into the product. PQM has worked with local manufacturers to build their capacity to implement GMP to support the production of zinc sulfate, magnesium sulphate, oxytocin, chlorhexidine, Sulphadoxine/Pyrimethamine, Artemether+Lumefantrine, ready-to-use therapeutic food, and amoxicillin dispersible tablets for the local Nigerian market. Eleven manufacturers were selected to receive TA based on their capabilities to manufacture MCH and malaria medicines: Emzor; Swiss Pharma; Phamatex; May & Baker; Juhel; Tuyil; Daily Needs; Drugfield; CHI; Dana Dabs; and Nemel. Activity TA supported these manufacturers to achieve ISO certification and WHO PQ8 for their selected products. MILESTONES TOWARD WHO PQ Manufacturers are at various stages of meeting their milestones toward ISO accreditation and/or WHO PQ, which differ according to their goals. Ten of 11 have continuously met milestones. Manufacturers that want to sell in Nigeria or to neighboring countries typically require only local registration. International development partners (e.g., USAID and GFATM) and developed countries require WHO PQ or stringent regulatory authority-approved medicines. Regardless of the goal, PQM tracks manufacturer progress toward their milestones. As of March 2018, the percentage of manufacturers meeting their milestones were: ● Three products by three manufacturers have met 90 percent ● One product by one manufacturer has met 75–89 percent 8 Manufacturers are not required to seek WHO PQ under PQM. If a manufacturer chooses to sell in Nigeria, WHO PQ is not necessary for product licensing. Furthermore, not all medicines have a pathway toward WHO PQ (e.g., amoxicillin dispersible tablets). 27 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ● Two products by three manufacturers have met 50–74 percent Table 4 (next page) lists the 20 PQM milestones manufactures must meet to attain WHO PQ. It is important to note that manufacturers started with PQM at different levels of capacity and facility readiness. For example, some manufacturers had to undertake extensive product reformulation while others did not. Some had to make extensive facility upgrades, while others needed minor retrofits. Some milestones require more time and resource to achieve than others. Only two manufacturers are not making adequate progress against their milestones, but this is due to recent changes in senior leadership; it is not a reflection of PQM’s work. TABLE 4. MANUFACTURER PQM MILESTONES FOR WHO PQ S/N MILESTONES S/N MILESTONES 1 Publish expression of interest 11 CAPA follow-up 2 Selection of manufacturers 12 Product development or reformulation 3 Initial & technical questionnaire 13 Product dossier preparation 4 Initial gap assessment 14 Dossier submission and acceptance 5 PQM decision to offer TA 15 WHO dossier follow-up 6 TA agreement 16 Mock audit 7 Conduct GMP inspection 17 WHO site approval 8 Bioequivalence / Biowaiver TA 18 WHO CAPA follow-up 9 GMP report completion 19 WHO dossier bioequivalence approval 10 Corrective Action Preventive Action (CAPA) planning 20 WHO final approval QUALITY AND EFFECTIVENESS OF PQM TA The success of PQM is directly evident in the achievements of local manufacturers. CHI Pharmaceuticals, Juhel Nigeria Limited, and Drugfield Pharmaceuticals have developed zinc sulfate/ORS, oxytocin, and chlorhexidine. All three have obtained NAFDAC approval or marketing authorization for the Nigerian market. Drugfield is the first manufacturer in Africa to produce and market chlorhexidine digluconate gel, an achievement that has not gone unnoticed. It is currently exporting chlorhexidine gel to Ghana for inclusion in safe delivery kits. This product is also registered and supplied to five other African countries, and the process is underway in Togo). Juhel is the first company in Africa to manufacture and market oxytocin injection. A Juhel respondent stated, “If it were not for PQM TA, we would not have been manufacturing oxytocin and magnesium sulphate.” “At the onset, when the TA was provided from the U.S., there was delay in response time and TA visits were few and far between. But once the PQM Nigeria field office was set up, TA visits from became much more frequent.” — Manufacturer respondent 28 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV PQM has been physically present with assistance through all the processes of QA and GMP. CHI Pharmaceuticals was selected by the United Nations International Children’s Education Fund (UNICEF) to supply zinc sulfate/oral rehydration salts to four countries, and four manufacturers secured ISO 9000 series certification. CHI Pharmaceuticals is expected to have a WHO PQ assessment visit in June 2018 for zinc sulfate. If successful, it will become one of the local manufacturers to obtain WHO PQ. PQM TA did increase manufacturers’ production of high-quality priority MCH commodities and malaria medicines. It is important to note that most manufacturers were already producing the same medicines they improved under PQM; any increase in quantity is due to market demand and cannot be solely attributed to improvements under PQM. Most production information is not available to the public, so the evaluation team did not request sensitive information. This makes quantifying percentages of increase in medicines due to PQM impossible to calculate through this evaluation. Table 5 shows the quantity of high-quality medicines supplied to public health organizations/agencies during implementation of PQM. TABLE 5. QUANTITY OF MEDICINES SUPPLIED TO PUBLIC HEALTH ORGANIZATIONS/AGENCIES, 2013–2018 MEDICINE NO. OF DOSES PURCHASER Zinc sulfate + ORS-co-pack 1,850,000 National Primary Healthcare Development Agency 896,472 Society for Family Health 267,107 Crown Agents 100,000 UNICEF Low-osmolarity ORS 570, 024 Society for Family Health 1,912,000 UNICEF 112,509 Crown Agents Dispersible zinc sulfate tablet 1,005,540 UNICEF 40,000 John Snow 12,150 Crown Agents 88,000 Health for Africa Chlorhexidine digluconate gel 4,174,508 tubes Supplied to the Nigerian market CHALLENGES FOR MANUFACTURERS The manufacturers stated that they face several challenges, including: ● Lack of procurement by state central medical stores. State MoH offices are ultimately responsible for procuring drugs for their public health facilities. Manufacturers do not market themselves to the states, so states are unaware of what each manufacturer produces. When states do procure medicines, it is often in a fragmented manner, which results in “siloed” purchasing (e.g., MCH medicines are procured separately from malaria medicines). Although larger states can purchase in large volumes (production is more economical at scale), smaller states have less purchasing power. 29 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ● Non-harmonization of regulatory requirements in the West African sub-region. Nigerian manufactures supported by PQM are well-positioned to capitalize on regional markets, but the opportunity is hampered by non-harmonization of regulatory requirements. This not only limits the sales and marketing of products across the West Africa sub-region, but constitutes a barrier to international procurement of pharmaceuticals. ● Uncertain market demand for products. Manufacturers are faced with uncertain market demand for products. This makes forecasting difficult, with manufacturers are unable to plan production volumes to meet market demands, which increases the chances of product expiration. ● Cost-prohibitive interest rates and difficult access to start-up capital. Access to capital is difficult; when it is available, interest rates on loans are high and cost-prohibitive. The lending interest rate in Nigeria has remained steady at 14 percent since 2016. This greatly curtails investments for facility upgrades to manufacture new products and hampers the overall growth of the manufacturing sector. More manufacturers will be required for products to be cost￾competitive. ● Uncertain political climate and inconsistencies in government policies. The uncertain political climate in Nigeria and inconsistencies in government policies pose a risk to investments and future profits. Manufacturers are concerned that policies that favored large investments—and ensured sustainability—could cease to exist with little warning. ● Instability with the naira exchange rate. APIs and consumables are purchased with U.S. dollars, and the exchange rate is unpredictable. Nigeria’s recent recession decreased the purchasing power of the naira and cut into profits, which threatened return on investment. For example, in 2014, the exchange rate was about 160.00 naira to $1.00, but it now stands at 360.00 naira to the dollar. The cost of the finished pharmaceuticals product ends up higher and makes locally produced medicines less competitive than imported medicines. CONCLUSIONS PQM TA has been effective at helping manufacturers achieve their milestones. All but two manufacturers have achieved impressive progress toward their milestones, and it is unlikely they would have done so much in such a short period without PQM’s assistance. The greatest threats to sustained production of priority MCH and malaria medicines are external to PQM and USAID, and are a concern for the future. If government policies are not supportive and consistent in increasing market demand, cost competitiveness, and access to capital, the long-term future production of these medicines faces significant risk. The evaluation team does not see an immediate risk to the success of manufacturers. It believes TA should continue as planned and sees no evidence that a course correction is needed. KEY RECOMMENDATIONS High-value PQM TA in the medium term ● PQM should help identify companies/consultants that can help manufacturers adopt a “production on demand” model to prevent products from expiring in their warehouses. This approach encourages manufacturers to stock APIs with longer expiry dates from pre-qualified suppliers. ● PQM should work with state MoH offices to help them identify products for purchase from each manufacturer and to disburse procurement funds in a timely manner. Long-term, high-value advocacy from PMG-MAN ● Regional cooperation and harmonization of regulatory practices is ongoing. Manufacturers, with the assistance of PMG-MAN and USAID/Nigeria’s Economic Growth Office, should explore the 30 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ways in which their advocacy can be intensified to support the registration of local products in other African countries and beyond. Nigeria’s membership to the Economic Community of West African States could be leveraged to a greater extent to support this effort. ● PMG-MAN man should commence a formal dialogue with The Bank of Industry to offer lower interest rates and more favorable repayment plans on loans for production of antimalarial and other MCH priority medicines. ● PMG-MAN, along with manufacturers, should advocate to government for a tax-free holiday for local manufacturers of antimalarials and other MCH priority medicines so they can be competitive with imported medicines. USAID ● USAID can leverage IHP-supported states to pool resources to buy in bulk. This would bring down the cost of medicines and also help manufacturers forecast production. EVALUATION QUESTION 3 What are the perceptions of local pharmaceutical manufacturers towards PQM's TA? Have they made plans to sustain or expand local production of quality-assured priority medicines beyond the life of the project? QUALITY AND EFFECTIVENESS OF PQM TA Manufacturers rate PQM’s TA as “excellent” and consistently referred to activity staff as “experts,” “very knowledgeable,” “helpful,” and “supportive.” The opening of the PQM/Nigeria office was a watershed moment for manufacturers, who unanimously agreeing that progress increased once the office opened. Furthermore, most respondents in the local manufacturing sector said TA from PQM/Nigeria staff was better tailored and delivered than TA from the United States (so-called “parachute TA”), resulting in a better understanding of the challenges and local context. Most manufacturers rated overall TA as “excellent” in terms of quality and responsiveness of assistance to management. Some manufacturers complained they were unable to fully utilize their increased capacity. They expressed a desire to export outside of Nigeria to ensure better capacity utilization, return on investment, and greater market share. There is a huge opportunity for sales in the West African market for medicines such as chlorhexidine gel and oxytocin injection, because no other companies in West Africa manufactures these products. Respondents often used the term “increased confidence” when they described the effectiveness and results of PQM’s assistance. Several respondents stated there was a steep learning curve in the beginning and “they didn’t know what they didn’t know.” One respondent said, “We didn’t even know what a good dossier looked like or the steps to produce one. Now we can take that knowledge and apply it to future products.” Most manufacturers expressed interest to pursue other priority medicines because they had confidence in their ability and have made important facility upgrades, such as installing air conditioning units. “There was improvement not only in the quality of oxytocin and magnesium sulphate, but also in the quality of other products. Now we have the confidence to take the knowledge we learned from PQM and apply it to other medicines.” — Manufacturer respondent “At the onset, when the TA was provided from the U.S, there was delay in response time and TA visits were few and far between. But once the PQM Nigeria field office was set up, TA visits from became much more frequent.” —Manufacturer respondent 31 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV CHALLENGES Threats to sustainability of local production of quality-assured priority medicines beyond the life of PQM include: 1. Nigerian Customs imposed taxes on APIs. These can increase the already high cost of production and have had the following effects: ● Conflict with the government’s goal to increase local production of medicines to 70 percent (along with taxes on excipients and other production accessories) ● Decreased the return on investment, especially with the high costs of power and water ● Decreased price competitiveness with imported products (Although NAFDAC’s advocacy to eliminate taxes has been unsuccessful, the agency is continuing its efforts.) 2. Imported products (parallel imports) saturate the market and “crowd out” locally manufactured products. Some manufacturers still expect to continuously procure medicines through bilateral and/or multilateral agreements and/or that PQM-supported local manufacturers will receive USAID preferential treatment for procurement. It is not clear how much of an impact, if any, this may have on the long-term production of priority medicines. OPPORTUNITIES A significant start-up cost was incurred for facility upgrades, but this was a one-time expense. However, if there is a need for reinvestment, it is unlikely to require the same level of resources and time. A major opportunity is for manufacturers to capitalize on existing facility and human resources to support the production of other medicines needed in Nigeria. CONCLUSIONS Manufacturers are very impressed with the expertise of PQM/Nigeria. It is clear to the evaluation team that the opening of the PQM office was a “game-changer” for manufacturers. The team recommends that TA continue as planned and sees no evidence for a course correction. KEY RECOMMENDATIONS Long-term, high-value advocacy from PMG-MAN 1. PMG-MAN can help ensure sustainability by assisting manufacturers to market their products to other West African countries and the rest of Africa. 2. PMG-MAN can help local manufacturers explore opportunities for acquisitions and mergers to increase their capital base in line with global best practices in the industry. This approach would be more cost-competitive and increase influence on government policies. 3. PMG-MAN should increase support to NAFDAC as it lobbies Nigerian Customs to eliminate taxes on APIs, excipients, and other production accessories. 32 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV IR 3: UTILIZATION OF MEDICAL PRODUCT INFORMATION FOR DECISION-MAKING INCREASED EVALUATION QUESTION 4 To what extent has PQM’s TA been effective in increasing the capacity of national and state regulatory agencies to utilize medical product quality information for decision-making? Is the TA still needed? Before PQM assistance, the PV-PMS Directorate could not adequately undertake active and regular surveillance of medicines. Additionally, there was no systematic way to conduct surveillance or fully apply the resultant data for decision-making. PQM facilitated the development of the “PMS Guidelines” as a way to ensure surveillance was done systematically and regularly. The Directorate is currently using these guidelines. PV-PMS staff rate PQM’s TA as “very good” and “very effective building the capacity of PV-PMS staff” to carry out surveillance in accordance with the “PMS Guidelines.” Staff also cite their increased capacity to use PMS data for decision-making. For example, one PV-PMS respondent said, “Decisions were taken based on the PMS report on maternal and child health medicines. Those decisions, such as the recommendation of cold storage for oxytocin, was a direct result of the data.” Another respondent referenced a 2016 PMS report on MCH medicines, sponsored by PQM, which found that 74 percent of sampled oxytocin and 34 percent of sample misoprostol failed laboratory testing. Armed with this information, NAFDAC confiscated 1,183 ampoules of oxytocin from circulation. Based on this experience PV-PMS, implemented risk-based surveillance of these medicines. The National Malaria Elimination Program (NMEP) had a similar experience. Based on the results of three rounds of PMS on malaria medicines, sponsored by PQM, NMEP was better able to reliably determine the percentage of substandard and falsified medicines among those sampled and where the medicines were concentrated. NMEP was heartened to see that their work had a positive impact, illustrated by the sharp decrease in poor-quality medicines. Based on the PMS, the activity has prioritized funds for risk-based surveillance. Also, NMEP stated that the results made it more confident to approach donors for funding and advocate for more resources. The confidence is timely, as GFATM will switch to a model of direct funding to governments. In addition, NMEP is seeking funds from other sources, such as the World Bank. CHALLENGES Previous sections have addressed the challenges to decision-making. At the state level, limited human resource capacity—as well as limited financial and material resources—to carry out PMS surveys present a formidable challenge to collect and analyze data that can inform decision-making. OPPORTUNITIES PV-PMS can apply experience with MCH and malaria medicine surveillance to other high-risk medicines, such as tuberculosis. NMEP suggested that it would also like to build on its knowledge and use PMS to monitor medicines that are used in conjunction with malaria treatment (e.g., ORS). CONCLUSIONS The success of PMS to increase the use of evidence for decision-making is notable. The quality of data is reliable, and decision-makers have confidence to use it. PMS is an activity success story. The evaluation team believes that TA should continue as planned and does not see a need for a course correction. 33 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV KEY RECOMMENDATIONS The key recommendation to support decision-making was outlined in Evaluation Question 1a. The evaluation team recommends that ICT infrastructure be improved to support quicker and more efficient decision-making. EVALUATION QUESTION 5 What type of plans have national and state health officials made to sustain the regulatory systems for medicines, including the internationally accredited laboratories, beyond the life of the project? Are there any gaps in plans? At the time of this evaluation, PQM had not developed a formal phase-out plan for NAFDAC because TA is ongoing. PQM will develop this plan in 2019 in consultation with NAFDAC and USAID. The plan will guide the transition of PQM activities that have not been transferred to NAFDAC. NAFDAC does not have an agency-wide regulatory system strengthening sustainability plan, although it has adopted some sustainability approaches, such as the CLM approach, to train laboratory staff and a mentorship model in which more experienced R&R Directorate staff gradually build the capacity of less experienced staff to review dossiers. NAFDAC’s tariff committee is in the process of developing a strategy to levy a “PMS tariff” on MAHs as a means to fund future surveillance surveys. PQM is currently working with the laboratories to develop a formal sustainability plan. KEY SUSTAINABILITY APPROACHES 1. Adoption of the CLM approach for skill transfer. Yaba laboratory staff employed the model to train their counterparts at Agulu and Kaduna laboratories. There was no obvious drop in technical abilities, as these laboratories attained ISO 17205 certification. Plans are underway to replicate the CLM approach at the other four NAFDAC laboratories. 2. NAFDAC did not dedicate adequate funding to laboratories. As a result, securing of reference standards, consumables, and equipment was a great concern. However, for the first time in more than five years, NAFDAC procured all proficiency testing materials on time (as stated by the Director Laboratory Services, Drugs), owing to the commitment of the new DG to upgrading the laboratories’ standards, and has taken over the cost of equipment calibration for all its labs. 3. NAFDAC advocated for ongoing budget line items, specifically for laboratories. In the 2017 appropriations bill, NAFDAC had a 50 million Naira (approx. $140,000) line item for training and capacity building. In the 2018 proposed appropriations bill, NAFDAC has a 100 million Naira (approx. $280,000) line item for laboratory equipment and 50 million Naira (approx. $140,000) for refurbishment of laboratory buildings. 4. Personal development is ongoing. This is especially true at the Yaba laboratory, where staff receive new and refresher training on a monthly basis. Periodic internal audits of laboratory processes and quarterly QC meetings are held to identify gaps in testing and appropriate solutions. Additionally, equipment calibration and maintenance are carried annually. 5. Purchase agreement with Merck. With PQM assistance, NAFDAC established a purchase agreement with Merck that enables it procure consumables at a more affordable rate. 6. Equipment training. PQM has identified 10 laboratory staff who will receive advanced training for preventative maintenance of basic equipment. CONCLUSIONS The lack of a formal agency-wide plan that will ensure the sustainability of achievements realized with PQM’s assistance is lamentable, but this shortcoming cannot be attributed to the activity. Although PQM 34 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV is helping laboratories develop a sustainability plan, they do not operate in a “vacuum,” and their sustainability depends on the overall sustainability of NAFDAC as an agency. Key sustainability approaches (e.g., CLM and partnership with Merck) are in place, but cannot substitute for a formal sustainability plan. Moreover, with the exception of CLM in the accreditation of Agulu and Kaduna labs, these approaches have not been proven to be effective in the long term. Without an overall agency sustainability plan, the evaluation team cannot determine if the approaches to sustainability can be sustained over time. Regardless, the evaluation team believes TA should continue as planned. KEY RECOMMENDATION The evaluation team is sensitive to the fact that USAID cannot mandate an autonomous government agency to have a formal agency-wide sustainability plan. If there is a follow-on award, USAID should require NAFDAC to have such a plan, not only approaches to sustainability, for regulatory system strengthening activities directly supported by PQM. This is particularly important if USAID shifts activity TA to the state level, where many regulatory system activities are carried out and resources are often inadequate. CROSSCUTTING FINDINGS 1. PQM branding is inadequate. A number of stakeholders, especially outside of NAFDAC and the FMoH, know only of USP, not PQM. This issue is very conspicuous among manufacturers. A number of manufacturers don’t understand that USP is the implementing partner of the PQM activity, which is funded by USAID. One respondent said he couldn’t speak to the quality of PQM TA but stated that USP is doing a good job. One Managing Director had never heard of USAID and only vaguely recalled PQM. This is not a problem restricted to Nigeria rather it was found in the global PQM evaluation.9 2. Representation of women in PQM activities. When the evaluation team assessed if women were well-represented in PQM activities, it found that they were more often overrepresented. The workforce at NAFDAC, the FMoH, and other GoN stakeholders is predominately female, so more women were represented in PQM training and workshops. Manufacturers stated that staff were selected for PQM training based solely on job function. 3. Stakeholder awareness of PQM’s phase-out plan is not uniform, especially among manufacturers. Some manufacturers were surprised when the evaluation team informed them PQM will close in September 2019. Stakeholders who were aware of a phase-out plan had only a vague understanding of it. It is not clear how much, and to what degree, PQM has shared the plan and how much stakeholders remember. 4. Lack of outcome-level indicators in the Activity MEL Plan. The lack of these indicators makes it difficult to measure achievements and determine to what degree they are being met. 9 The evaluation team was provided access to the report and granted permission to quote the report by USAID and the global evaluation team. 35 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV SECTION 5. RECOMMENDATIONS AND FUTURE DIRECTIONS OVERALL CONCLUSIONS The purpose of this midterm performance evaluation is to assess the effectiveness of PQM’s TA in strengthening medicines regulatory QA systems in Nigeria and increasing the capacity of local manufacturers to produce quality-assured medicines. Its findings demonstrate that PQM is largely on track to meet its objectives, and that its activities should continue as planned without any course corrections. TA from PQM has been critical to building the capacity of several functions within Nigeria’s medicines supply chain. Notable among these is the improved regulatory capacity of NAFDAC partners at the national level, and the concomitant increase in the agency’s laboratory testing capacity. With PQM assistance and its commitment to improving NAFDAC’s human resources, technical capacity, and technological monitoring systems, this national regulatory body for medicines is in a stronger position to oversee activities in Nigeria. PQM is the only partner working on these issues in Nigeria, particularly laboratory capacity, so improvements in this sector are largely attributable to its work. PQM has also moved the manufacturing sector forward in its efforts to produce high-quality finished pharmaceutical products through the employment of GMP. The activity is considered a leading expert in this sector, and their assistance, particularly the opening of a local office, has been critical to the advancement of manufacturing product improvement. Given its close work with leading companies in this sector, PQM can be credited with increases in quality and volume of certain priority MCH and malaria products. The greatest threats to the activity’s successes are outside of PQM’s and USAID’s control. Though not immediate, these threats are real and must be addressed if there is a follow-on award. The laboratories are at the greatest risk of losing gains if NAFDAC does not budget adequately, as they require significant funding. The labs are well-positioned to take on human resources capacity building with very little assistance from PQM; however, without a formal sustainability plan—and if the labs do not or cannot generate revenue—it is not clear if NAFDAC is prepared to continually purchase reference materials and supplies, budget for more staff, and fund reaccreditation for all of its laboratories, not only the four supported by PQM. NAFDAC should be commended for the steps it has taken to address regulatory systems challenges. However, new challenges will arise, and the agency will need to put resources in place to make sure regulatory systems are resilient to address them. A first step to strengthening the system is to develop an agency-wide sustainability plan for regulatory system strengthening or, at a minimum, a formal plan that incorporates all of the approaches to sustainability adopted under PQM, such as CLM. Currently, the approaches to sustainability are fragmented and siloed in each Directorate and do not have an overarching framework to ensure cohesiveness. A formal sustainability plan can help ensure approaches are complimentary, not contradictory. Moreover, the long-term success of assuring drug quality in Nigeria depends largely on government policies to support market demand, cost competitiveness, and access to capital for production. NAFDAC has an important advocacy role to ensure the GoN recognizes the importance of quality medicines. Without a sustained commitment to reinforcing medicine development at the national level, the gains made under PQM could be lost. Overall, PQM has been successful in meeting its aims and should be commended for its dedication and success. It has consistently met work plan goals and targets in its Activity MEL Plan. The activity’s success reflects its staff’s commitment and their deep bench of expertise. The staff and Chief of Party were consistently singled out as dedicated partners who are highly skilled at their jobs. The evaluation team 36 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV recommends that PQM continue with planned TA. For stakeholders who will not receive additional TA, the activity should ensure that phase-out plans are underway. RECOMMENDATIONS The body of the report highlights recommendations, big and small, that can be applied immediately to improve the functioning and long-term health of Nigeria’s medicines sector. Below, we highlight the key recommendations that are essential to the continued success of PQM and USAID’s work in the region. RECOMMENDATIONS FOR PQM 1. Perform a NAFDC SWOT analysis. PQM should help NAFDAC to undertake a SWOT analysis to identify actual assets that can support activities and mitigate threats to sustainability. 2. Improve production timing and sales channels for manufacturers. PQM should identify consultants/consulting companies that specialize in assisting manufacturers to adopt a “production on demand” model to prevent products from expiring in their warehouses. This approach encourages manufacturers to stock APIs with longer expiry dates from prequalified suppliers. 3. Strengthen QA systems. Improvements in drug quality and regulation can be sustained only if they are accompanied by improvements in complementary sectors, such as the supply chain. A strong supply chain is essential to the maintenance of quality medicines in the health sector. One way to help strengthen the supply chain is would be to implement the NQAP, with TA and monetary resources to NSCIP to develop guidelines for implementation. 4. Improve branding. The problem of inadequate branding was raised in the global PQM evaluation. The evaluation team is aware that USP/Rockville is working to address the problem at a global level, but in the interim PQM/Nigeria should address this issue directly with manufacturers. The evaluation team believes that a brief conversation should suffice until USP/Rockville implements a formal plan. RECOMMENDATIONS FOR USAID 1. Increase collaboration with complementary QA programs, such as GHSC-PSM and IHP. Currently, there is not an efficient or systematic way for public health facilities via GHSC-PSM to know what local manufacturers are producing and at what capacity. Previously, when USAID/Washington alerted GHSC-PSM about local commodities, it did so through its implementing partner, Chemonics International, from its Washington, D.C., headquarters. USAID/Nigeria should request that PQM require manufacturers to report the number of commodities available, or that may soon be available, and then give that information to the Mission in quarterly reports. USAID/Nigeria would then be ideally positioned to share that information with state public health facilities and any other relevant partners via GHSC-PSM (e.g., in a memo or website posting). This approach would allow closer collaboration without expanding the scopes of either program or placing an undue burden on the Mission. PQM collaboration with IHP may be a cost-effective and streamlined way to help states pool their resources to buy medicines in bulk. The outcome would be two-fold. First, smaller states will increase their purchasing power and manufacturers can better plan production. Second, because IHP’s goal is to strengthen health systems at the state level, and regulatory systems are part of the overall health system, it can combine resources with PQM to offer a more holistic approach to health system strengthening. 2. Advocacy for DHIS-2 expansion. USAID/Nigeria can advocate to the FMoH to expand the DHIS-2 subscription to other GoN agencies. 37 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 3. Include outcome-level indicators. The global PQM evaluation and this evaluation have illustrated the challenge of objective measurement of performance when there is a lack of outcome indicators. If USAID elects to have a follow-on award, it should include outcome-level indicators. Suggested indicators include the following; these could be revised to include specific numbers: IR 1 ● Percent or number of NAFDAC laboratories that pass proficiency testing per year10 ● Percent or number of samples tested within required turnaround time IR 2 ● Percent or number of manufacturers that have attained WHO PQ per year11 ● Percent or number of manufacturers that have secured NAFDAC marketing and license approval per year ● Percent or number of manufacturers with approved site inspection reports ● Percent or number of manufacturers that have retained their annual site inspections IR 3 (None) RECOMMENDATIONS FOR PMG-MAN AND PQM 1. Increase advocacy and access to funds. Manufacturers, with assistance from PMG-MAN and PQM, should explore the ways in which their advocacy can be intensified to support the registration of local products in other African countries and beyond. PMG-MAN should also spearhead dialogue with The Bank of Industry to offer lower interest rates and more favorable repayment plans on loans, and with government entities to establish tax-free holidays for production of antimalarial and other MCH priority medicines. USAID/Nigeria’s Economic Growth Office may also have a role in assisting advocacy. 2. Improve regional sales through marketing and mergers. Through PMG-MAN, local manufacturers have an extensive network that can help ensure sustainability by assisting them to market their products to other West African markets and beyond. Local manufacturers should explore opportunities for acquisitions and mergers to increase their capital base. 3. Intensify support for NAFDAC in lobbying Nigerian Customs on the elimination of taxes on APIs. PMG-MAN and PQM should intensify their support to NAFDAC in its efforts to lobby Nigerian Customs to eliminate taxes on APIs, excipients, and other production accessories. RECOMMENDATIONS FOR NAFDAC 1. Develop a regulatory system strengthening sustainability plan. NAFDAC should consider developing a regulatory system strengthening sustainability plan or, at a minimum, a formal plan that incorporates all the individual approaches to sustaining improvements made under PQM. NAFDAC will greatly benefit from PQM’s input into this plan. 2. Improve ICT systems. The PRS Directorate’s current IT operating system is inadequate to meet growing requirements of PMS and must be improved. Even though ICT improvements are outside the scope of PQM, the actuivity could provide valuable insight into what those systems should look like. 10 This assumes the denominator is the total number of NAFDAC labs, though not all of them participate in proficiency testing. Such an indicator will need to be specific to labs that participate in proficiency testing. Although testing is required for accreditation, it is not required annually. 11 It is important to note that relatively few manufacturers receive WHO PQ; therefore, an indicator such as this would not be meaningful. Additionally, many of the MCH priority products (e.g., chlorhexidine digluconate gel and amoxicillin dispersible tablets) do not have a WHO PQ pathway. 38 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV The commercial sector and universities could assist with open-source software and affordable customization. 3. Address NAFDAC human resource challenges at the state and zonal levels. NAFDAC state and zonal offices have significant human resource capacity challenges that must be addressed in order to sustain regulatory systems. Every Directorate included in the evaluation cited low state and zonal capacity to undertake surveillance and control activities as a significant challenge to PQM scale-up and sustainability. If there is a follow-on award, PQM resources (financial and otherwise) should be shifted from the national level to the state and zonal offices. RECOMMENDATIONS FOR FUTURE USAID PRIORITY INVESTMENT 1. Bolster health information systems. Strong health information systems are a major area of investment that have potential for significant impact on strengthening regulatory systems and health commodity procurement systems. Examples of future investments include: ● Improve and expand information systems and real-time updates from the NAFDAC HQ to frontline health workers. NAFDAC’s information systems should be expanded to the states and clinics. NAFDAC relies on frontline health workers’ reports of unsafe medicines and medicines with low efficacy, yet that information does not always flow “up” to the agency. Moreover, notice of recalls and safety concerns do not always “flow down” to frontline health workers. ● Develop an information system that connects state-level procurement offices (commercial, civil society, public health facilities) to local manufacturers. The two-way flow of information, updated in real time, between manufacturers and procurement offices would speed up procurement and help manufacturers better plan for production. This would ensure states get medicines quickly and decrease the amount of expired product. 2. Partner with USAID/Nigeria’s Economic Growth Office and other USAID Missions in West Africa to advocate for regional harmonization of regulatory requirements. Regional harmonization would make it more efficient for manufacturers to sell products in other countries, which increases revenue, helps ensure sustainability, and results in the need for greater production volume. As the need for production increases, Nigeria’s pharmaceutical sector would benefit from the creation of more jobs. 39 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANNEX I. EVALUATION SOW STATEMENT OF WORK Mid-Term Performance Evaluation of Promoting Quality of Medicine (PQM) I. SUMMARY INFORMATION Strategy/Project/Activi ty Name Promoting Quality of Medicine (PQM) Implementer U.S. Pharmacopeial Convention Cooperative Agreement/Contract # GHS-A-00-09-00003-00 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $110,000,000 (global) Life of Strategy, Project, or Activity 2013 to September 17, 2019 Active Geographic Regions Nigeria Development Objective(s) (DOs) DO 2 USAID Office Nigeria II. PURPOSE OF THE EVALUATION Instructions: Insert why the evaluation is being conducted (the purpose), who will use the results of the evaluation, and how they will use it. Explicitly link the evaluation to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders. The clearer the purpose, the more likely the evaluation will produce credible and useful findings, conclusions, and recommendations. The purpose of the evaluation should be consistent with, but not replicate, the evaluation questions (Section IV). Note: The Evaluation Purpose will often be picked up by the Contracting Officer and added to the contract that is executed. It is included first in this template for that reason. It can also come after the Background Section. The purpose of this performance evaluation is to provide the United States Agency for International Development (USAID) Nigeria with an objective assessment of PQM activities in Nigeria. Specifically, this interim performance evaluation is being conducted to: ● Assess the effectiveness of PQM’s technical assistance in (a) strengthening medicines regulatory quality assurance systems in Nigeria and (b) building the capacity of local manufacturers to produce quality medicines ● Identify accomplishments and challenges in program implementation to improve efficiency and highlight opportunities for adjustments in current technical activity; and ● Provide recommendations to USAID Nigeria for potential future investments in medicines quality assurance systems strengthening and, specifically, in strengthening the capacity of local pharmaceutical manufacturing. 40 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV III. BACKGROUND Instructions: Provide a detailed description of the context, history, goals and objectives, current status of the strategy/project/activity, and other relevant information to help the evaluation team understand the design and implementation plan. Complete the sections noted below. Sections can be consolidated. The USAID Promoting the Quality of Medicines (PQM) program is a 10-year, centrally managed cooperative agreement with a $110 million ceiling, under award number GHS–A-00-09-00003. The program awardee is the U.S. Pharmacopeial Convention (USP). USAID Nigeria selected the Promoting the Quality of Medicines (PQM) program to support the following activities ● Strengthening of the medicines regulatory authority in Nigeria - National Agency for Drug Administration and Control (NAFDAC) capacity in assuring the quality and control of anti-malarial and maternal, newborn, and child health priority medicines in Nigeria ● Support local pharmaceutical manufacturers to increase the availability of quality assured priority medicines. The goal of PQM is to ensure the quality and safety of medical products and protect public health. Achieving the PQM’s goal is dependent on achieving several intermediate results (IR) and sub-IRs. The activities under the intermediate results contribute to the strengthening medicines quality assurance systems. The three PQM result areas are: 1. Strengthen medical product quality assurance systems. A key obstacle to promoting quality￾assured medicines and combating substandard and falsified products in Nigeria is the lack of institutional, financial, technical and human resource capacity in medicines regulatory systems to protect supply chains. Medicines quality assurance depends to a large extent on the capacity of national regulatory authority’s ability to safeguard the quality, safety, and efficacy of the medicines in the market. 2. Increase the supply of quality-assured priority medicines. Quality-assured medicines are not readily available and governments, development partners, health facilities and/or patients have little choice but to use medicines that have not undergone rigorous regulatory oversight. To improve positive health outcomes, quality-assured, efficacious, and safe medicines are needed. PQM program works to increase the supply of quality-assured medicines of direct relevance to priority USAID health programs including malaria and MNCH. The program provides technical support to enable manufacturers to comply with international standards for Good Manufacturing Practices and ultimately to receive stringent regulatory authority approval for essential drugs, allowing the manufacturer to build stronger quality-assurance systems and satisfy medicines regulatory requirements for marketing authorization and procurement. 3. Increase the utilization of medical product quality information for decision-making. Poor￾quality medicines pose a grave threat to patients, but this is a largely unknown problem to the public. PQM uses medicines quality monitoring post marketing surveillance program in collaboration with the medicine regulatory agency to establish a system to regularly examine the quality of medicines circulating in its markets. 41 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV PQM has provided technical assistance to the country’s medicine regulatory authorities (NAFDAC), national quality control laboratories, and local pharmaceutical manufacturers. A. Description of the Problem, Development Hypothesis(es), and/or Theory of Change Instructions: Include details on: --The specific problem or opportunity the strategy/project/activity to be evaluated was designed to address; --The development hypothesis(es) often expressed as an if/then statement12; --The theory of change that underlies the design (including a list of the intended results and critical assumptions); -- Results Frameworks: Include here or as an annex the graphic of the Mission’s Results Framework and the Project’s Logical Framework (if applicable) highlighting the elements to be evaluated. If the evaluation is at the Activity level then include the Activity’s Logical Framework (and linkages to the project-level). In all cases, account for changes (if applicable) since the original design. Since 2013, the USAID-funded PQM program was engaged to help Nigeria overcome the challenges of falsified, unapproved by the national regulatory agency, and substandard medicines. The PQM program serves a critical mandate: ensure quality, safety, and efficacy of medicines that are of great importance to the USAID priority diseases—especially malaria and maternal and child health (MCH)—by providing distinctive services to local manufacturers and the National Agency for Food and Drugs Administration and Control (NAFDAC). This need for technical assistance was urgent because of the threat of poor-quality medicines to public health, waste of scarce resources that undermine decades of USAID health investments in the country. The theory of change holds that, if local manufacturing capacity and regulatory systems are improved, then Nigerians’ access to quality assured medicines will increase. Using approaches that are tailored to the needs of the country, PQM offers technical assistance in several areas to achieve four strategic objectives: ● Build capacity and strengthen Quality Assurance systems ● Help increase the supply of quality assured medicines; ● Combat falsified, substandard, and unapproved medicines ● Provide technical leadership. These approaches include building the capacity of NAFDAC to review and approve quality essential medicines (according to NAFDAC standards) and strengthening NAFDAC’s ability to protect the citizens from poor-quality medicines through improved manufacturing, inspection, and surveillance capabilities. The approaches listed above are achieved through hands-on training and technical assistance to improve laboratory standards, attain and maintain internationally recognized certifications (such as International Organization for Standardization (ISO) 17025), and field-based medicines quality monitoring (MQM) which 12 If the design document does not contain an implicit development hypothesis, consult with the DO Team to develop the development hypothesis. 42 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV allows laboratory staff to collect medicine samples different sites, levels, and zones in Nigeria. The collected samples are screened and subsequently undergo confirmatory compendia testing in the laboratory. PQM’s tailored approach extends to local pharmaceutical manufacturers, assisting selected companies in improving their Good Manufacturing Practice (GMP) compliance, develops the medicines dossiers, and provides the support that manufacturers need to attain satisfactory standards for stringent regulatory authority. With improved capacity of NAFDAC, internationally accredited laboratories, and improved GMP compliance by local pharmaceutical manufacturers, it is envisaged that the incidence of substandard medicines will be minimized if not eliminated. The three intermediate results of PQM’s work in synergy to deliver the program outcome. The first works to strengthen medical product quality assurance system, the second to increase the supply of quality assured priority medicines, and the third to utilize medical product quality information for decision making. It is expected that this strategy will ensure sustainable availability of quality-assured medicines to protect public health. PQM Results Framework B. Summary Strategy/Project/Activity/Intervention to Be Evaluated Instructions: Summarize the primary interventions or tasks implemented by the strategy/project/activity. Also include a summary of any substantive changes (modifications) in the evaluated strategy/project/activity and when they were effective. Describe the specific geographic areas in which the strategy/project/activity operates and/or targeted groups, as applicable. Attach maps if available. IR 1: Activities 1. National Regulatory Systems Strengthened: PQM provided continuous technical assistance and advocacy for the development of the National Quality Assurance Policy (QAP) and its adoption by Nigeria’s National Council of Health at its 2016 annual meeting 2. Development of PMS Program Implementation Framework and Guideline: PQM provided technical assistance to NAFDAC to strengthen the capacity of its Post-Marketing Surveillance (PMS) unit to monitor the quality of medicines in the supply chain in Nigeria 43 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 3. Sustained Accreditation of Central and Zonal Drug Control Laboratories, Lagos, Agulu and Kaduna: PQM has delivered varied technical assistance to NAFDAC central and zonal laboratories in Lagos, Agulu and Kaduna. These three laboratories have been officially accredited. In addition, the National Institute for Pharmaceutical Research and Development (NIPRD) quality control laboratory in Abuja is on the way to attain international accreditation; PQM also provides TA to this laboratory. IR 2: Activities 1. Availability of Quality Medicines Increased: PQM Good Manufacturing Practice specialists supported local manufacturers of USAID priority medicines to improve GMP compliance and develop dossiers to submit to the WHO Prequalification of Medicines Program for certification 2. Manufacturing Sites Complying With GMP Standards Increased: PQM provides technical assistance to Nigerian manufacturers that produce oral rehydration salts (ORS), zinc sulfate tablets, chlorhexidine digluconate gel, and other MCH priority commodities. The supply of locally produced quality-assured medicines was increased through consistent technical assistance provided to 12 local manufacturers. IR 3: Activities 3. Capacity to Detect Poor Medical Products Increased: PQM helps combat falsified and substandard medicines by collaborating with country medicines regulatory authorities and national health programs by establishing or strengthening PMS systems that regularly examine the quality of medicines circulating in markets. PQM supports the national regulatory authorities to assess existing medical products by selecting sites to monitor based on criteria such as epidemiology, geography, border region, and history of trafficking falsified medicines. Geographic area: The geographic area is nationwide coverage in Nigeria for regulatory systems such as NAFDAC. Specific cities where laboratories are now accredited are Lagos, Abuja, Agulu (Anambra State) and Kaduna. C. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan Instructions: Specify what relevant documents will be available to the evaluators. In particular, identify the existence and availability of relevant performance information sources, such as performance monitoring indicators and/or previous evaluation reports. In addition, identify any other documents or sources of information from outside of USAID that would be useful to the evaluation team (e.g., government or international data). If this section is long it may also be included in an annex. Document and Data Review Documents and data to be reviewed include: ● PQM Nigeria Monitoring and Evaluation Plan (including the project’s results framework provided above) ● PQM Nigeria performance indicators reported on the USAID PRS ● PQM Nigeria quarterly and annual reports, trip reports, financial tracking reports, Knowledge Management strategy, success stories, PQM training materials and evaluations of PQM trainings, etc. ● Current funding from USAID Nigeria ● PQM Nigeria Work plans ● Reports on activities/support to local pharmaceutical manufacturers and NAFDAC laboratories ● Reports of ISO accreditation of NAFDAC laboratories ● Medicine Quality Monitoring / Post Marketing Surveillance reports 44 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ● PQM Interim Evaluation Final Report Key Informant Interviews The evaluation team will conduct in-depth key informant and/or group interviews, at a minimum, with the following organizations/staff: ● PQM Nigeria project staff (Nigeria Office senior management and staff) ● USAID/Nigeria HPN Office Leadership ● USAID/Nigeria Activity Manager and other selected USAID program managers ● Subject matter experts, outside stakeholders, and other identified partners, including, but not limited to: ● Executive Secretary, Pharmaceutical Manufacturers Group of Manufacturers Association of Nigeria (PMG-MAN) ● Managing Directors of pharmaceutical manufacturing companies (Chi Pharmaceuticals, Daily Needs Pharmaceuticals, Nemel Pharmaceuticals, Swipha, Pharmatex, Tuyil, May & Baker, Emzor Pharmaceuticals, Drug Field, Juhel Pharmaceuticals, Neimeth, and DABS Nutritional Foods) ● NAFDAC Acting Director General and Departmental Directors, Head of Laboratory services, Head of Central and Zonal Laboratories in Lagos, Agulu, and Kaduna ● Director General, National Institute of Pharmaceutical Research and Development (NIPRD) ● Director, Food and Drugs services Department, Federal Ministry of Health ● Country Director, USAID GHSC-PSM Project ● National Chairman, Association of Industrial Pharmacists ● Deans, Faculties of Pharmaceutical Sciences in Nigeria Universities IV. EVALUATION QUESTIONS Instructions: Include 1–5 specific questions focused on key program areas and/or performance and directly linked to the purpose of the evaluation and its expected use. Sub-questions or narrative text may be included to elaborate on the main question, but not to add new areas of inquiry. NOTE: Not every aspect of a program, project, or activity needs to be, or should be, the focus of the evaluation. Rather, the evaluation should examine specific aspects of the program, project, or activity where there are questions unanswered by performance monitoring or other data. Guidelines: 1. Questions should be precise. Vague terms that can be defined or applied in a variety of ways (such as “relevance,” “effectiveness,” etc.) should be defined clearly. If any specific terminology or standards are included in the evaluation questions indicate the source or definitions. 2. Questions should be researchable. Questions should have an answer that can be obtained through the use of social science methods and tools (qualitative and quantitative) rather than relying on the evaluators’ judgments. 3. Questions should integrate gender. Questions should identify when sex-disaggregated data are expected. Where appropriate, the evaluation questions can include a separate question aimed at evaluating the gender￾specific effects of the activity or project. [See the How-To Note on Engendering Evaluation] 45 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 4. Questions should be presented in order of priority, or the priority of questions should otherwise be identified. 5. A request for recommendations is not an evaluation question. If you want the evaluators to provide recommendations, describe what aspects of the program, project, or activity you want recommendations to address in a separate paragraph or following the questions. 1. To what extent has PQM’s technical assistance been effective in strengthening quality assurance systems for medicines in Nigeria? a. PQM has provided technical assistance to NAFDAC. To what extent has this TA improved NAFDAC’s core functions and is TA still needed? If so, what type of TA? b. Is targeting NAFDAC the most effective strategy for improving medicines quality assurance systems in Nigeria or are there other regulatory bodies that should receive TA? If so, which other regulatory bodies and what type of TA? c. Do NAFDAC quality control laboratories have the technical and human resources capacities to handle the medicines quality control and analysis requirements for Nigeria? If not, what type of TA is needed or what type of resources should be provided? 2. For several priority maternal and child health and malaria medicines (chlorhexidine, amoxicillin DT, ORS/zinc sulfate, oxytocin, sulphadoxine/pyrimethamine, and arthemether/lumefantrine), to what extent has PQM’s technical assistance to local manufacturers improved the production capacity and quality for these medicines? What milestones have been achieved and what, if any, still need to be achieved for local manufacturers to produce these quality-assured medicines? 3. What are the perceptions of local pharmaceutical manufacturers towards PQM’s technical assistance? Have they made plans to sustain or expand local production of quality assured priority medicines beyond the life of the project? 4. To what extent has PQM’s technical assistance been effective in increasing the capacity of national and state regulatory agencies to utilize medical product quality information for decision-making? Is the TA still needed? 5. What type of plans have national and state health officials (FMoH, NPHCDA, etc.) made to sustain the regulatory systems for medicines, including the internationally accredited laboratories, beyond the life of the project? Are there any gaps in plans? 46 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Common Evaluation Problems Description Ways to Address Too many questions, so little time Evaluation Statements of Work often contain many complex questions, but the funding and time is often limited to two to three weeks in the field with a correspondingly short time for data analysis, drafts, reviews, and the final report. A survey of evaluators showed that the single most important constraint to doing high quality evaluation was the failure of the issuing client to allocate sufficient time and budget for the task. Statements of Work must provide a realistic and adequate budget for both time and resources based on the nature and scope of the evaluation purpose and questions. TIPS # 3: Preparing an Evaluation Statement of Work provides guidance. Inadequate baseline data Lack of or inadequate baseline data against which to measure changes in the target population is probably the most common problem faced by evaluators and one of the most serious threats to the validity of the evaluation. Baseline data by itself is not sufficient to assess attribution, but without it, the evaluator cannot measure change in any rigorous way Reconstructing baseline data can be done by using secondary data, individual recall, participatory group techniques to reconstruct history and assess changes produced by the intervention, and key informant interviews. Data from any one method must be used cautiously. Evaluators should triangulate the estimates of reported information by using multiple data sources to increase the validity of the reconstructed baseline. TIPS # 5: Rapid Appraisal defines and discusses triangulation methods. Dangerous program settings prevent access to collecting evaluation data Many of the USAID’s largest assistance programs are in countries that are unstable or racked with internal conflict. Reaching key segments of the population to collect data may be dangerous and, even if possible, citizens may be afraid to speak to outsiders. In such instances, evaluators must work with stakeholders to discuss alternative data sources and data collection methods that are reasonable and acceptable under such conditions. A special TIPS will be written on this subject. Maintaining comparison group differences Effective use of comparison groups in impact evaluations requires both stability in the task environment and careful management from beginning to end, often over a 3- to 5-year period. If the project or program is providing desirable benefits, it is difficult to prevent individuals in comparison groups from securing those benefits. In other situations, program effects in the target group may spillover to the control group selected for a comparison. This results in underestimation of program impact since the control group will appear better-off than they would have. In some cases, “spillovers” can be mapped and measured, and then taken into account during the analysis of data from the target and control groups. However, the most effective means to deal with such an issue is to control it in advance through an evaluation design that selects treatment and control groups that are unlikely to significantly interact with one another. See TIPS # 19: Impact Evaluation. Disagreements on findings, interpretation and conclusions Serious disagreements between stakeholders and the evaluation team on findings or interpretation/analysis and conclusions can threaten the credibility and usefulness of evaluations. Hold a facilitated discussion on the relationship of the data (evidence) and its analysis and interpretation to the findings, and how these formed the basis of the conclusions. The usual practice is for the report to identify those points of disagreement in a foot note or annex. 47 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV V. EVALUATION DESIGN AND METHODOLOGY Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1.To what extent has PQM’s technical assistance been effective in strengthening quality assurance systems for medicines in Nigeria? Stakeholders, project reports, results of global evaluation Key informant interviews, desk review [ 2. For several priority maternal and child health and malaria medicines (chlorhexidine, amoxicillin DT, ORS/zinc sulfate, oxytocin, ALs, and SP), to what extent has PQM’s technical assistance to local manufacturers improved the production capacity and quality for these medicines? What milestones have been achieved and what, if any, still need to be achieved for local manufacturers to produce these quality-assured medicines? Stakeholders interviews, project reports, results of post-marketing surveillance (e.g. DHIS2 data), NAFDAC reports Key informant interviews, desk review 3.What are the perceptions of local pharmaceutical manufacturers towards PQM’s technical assistance? Have they made plans to sustain or expand local production of quality assured priority medicines beyond the life of the project? Local manufacturers, PMG-MAN (Pharmaceutical Manufacturers Group of the Manufacturers’ Association of Nigeria) Interviews 4. To what extent has PQM’s technical assistance been effective in increasing the capacity of national and state regulatory agencies to utilize medical product quality information for decision making? Stakeholders, NAFDAC reports, PQM reports Key informant interviews, desk review 5. What type of plans have national and state health officials (FMoH, NPHCDA, etc.) made to sustain the regulatory systems for medicines, including the internationally accredited laboratories, beyond the life of the project? Are there any gaps in plans? Stakeholders, key government officials (FMoH, NAFDAC) Interviews Notes: (*) It is acceptable to include data sources that do not need to be collected but may be analyzed by the evaluation team. In planning for and preparing the Evaluation SOW it is a good practice to examine available data sources especially performance monitoring data. 48 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Within two weeks of the award of the contract, a draft work plan for the evaluation shall be completed by the lead evaluator and presented to the Activity Manager at the Mission. The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. 2. Evaluation Design: Within two weeks of approval of the work plan, the evaluation team must submit to the Activity Manager at the Mission an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. 3. USAID offices and relevant stakeholders are asked to take up to 5 business days to review and consolidate comments through the AOR/COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 business days. 4. In-briefing: Within 3 days of arrival in Abuja, the evaluation team will have an in-briefing with the Health, Population, and Nutrition Office of USAID Nigeria for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. 5. Mid-term updates via phone or email: The evaluation team will provide the evaluation manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 6. Final Exit Briefing: The evaluation team is expected to hold a final exit briefing to the Health, Population, and Nutrition Office prior to leaving the country to discuss the status of data collection and preliminary findings. In addition to an oral briefing and discussion, the team should present a one-page summary of key findings. This summary can be bullet points. The final briefing will be scheduled as agreed upon during the in-briefing. 7. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the Health, Population and Nutrition (HPN) Office will have 10 business days in which to review and comment on the initial draft, after which point the Activity Manager will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again the HPN will review and send comments on this final draft report within 6 business days of its submission. 8. Final Evaluation Report: The evaluation team will be asked to take no more than 5 business days to respond/incorporate the final comments from the HPN Office. The evaluation team leader will then submit the final report to the Activity Manager. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. The evaluation team should include the following roles and mix of skills: 49 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 1. Evaluation Team Lead: At least 10 years’ experience in evaluation design, data collection, analysis, and report writing, including both qualitative and quantitative experience; should have experience as an Evaluation Team Lead in a developing country context, preferably sub-Saharan Africa. 2. Pharmaceutical Sector Technical Expert: At least 10 years’ experience in the pharmaceutical sector with knowledge of quality assurance systems for medicines in developing country contexts. 3. Health System Expert: At least ten years’ experience working with the Nigerian health system, specifically on programmatic or policy issues related to the quality and production of medicines. 4. Logistics/planning: At least 5 years’ experience in planning and organizing field site visits; knowledge of Nigerian context essential. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. VII. EVALUATION SCHEDULE Table 1: PQM Mid-Term Evaluation Schedule and Estimated LOE Proposed activities Performance Period Number of Days Team Leader/Int. Evaluation Expert Pharm. Sector Tech. Expert National Health System Expert Research Assistant MEL Activities submits names and CV of evaluators to USAID February 1, 2018 USAID provides concurrence for evaluation team February 16, 2018 Review background documents, preparation workplan and evaluation design/protocols March 2 - 9, 2018 7 6 5 3 Submission of workplan and evaluation design/protocols March 9 USAID provides feedback on work plan and evaluation design/protocols March 29 International Travel April 7 1 In-Briefing /Team planning meetings with USAID April 9 1 1 1 1 Finalization of draft work plan and evaluation design/protocols April 9-12 4 4 4 4 Presentation of draft work plan and evaluation design/protocols to USAID April 13 1 1 1 1 Incorporation of USAID comments into work plan and evaluation design/protocols April 14 1 1 Data collection in Lagos April 16 - 21 7 7 7 7 Data collection in Abuja/Kaduna April 23 - 27 6 5 5 5 Data analysis and preparation of the draft evaluation report, preliminary findings/PPT April 30 – May 9 10 8 8 8 Submission of presentation to MEL; continuation of report draft May 10 Presentation of preliminary findings out-brief meeting May 11 1 1 1 1 International travel May 12 1 Continuation of draft report May 14 -21 6 3 1 1 Submission of draft to MEL May 22 1 Submission of draft evaluation to USAID May 25 50 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Proposed activities Performance Period Number of Days Team Leader/Int. Evaluation Expert Pharm. Sector Tech. Expert National Health System Expert Research Assistant USAID reviews and comments on final draft evaluation report May 28 – June 15 Team addresses USAID comments and finalizes the report June 18-29 6 4 2 2 Submission of final Report to MEL June 30 Submission of final Report to USAID July 8 TOTAL LOE 54 41 35 33 VIII. FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: ● The Evaluation SOW; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● All sources of information, properly identified and listed; and ● Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. ● Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. ● Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. 51 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV EVALUATION DISSEMINATION PLAN TEMPLATE Key Milestones Expected Dates Statement of Work November 15, 2017 Evaluation Design March 9, 2018 Draft Report May 11, 2018 Final Report June 21, 2018 Evaluation Title: Midterm Evaluation of the Promoting the Quality of Medicines Project in Nigeria Audience Goal Tool/Medium Forum Responsible Party Timing Follow-up Identify stakeholders by asking” “Who is likely to be affected by the evaluation and its results? Who is likely to be interested?” Are we simply pushing out information? Hoping to affect change? Contributing to the knowledge base? These may include reports, briefs, presentations, blog posts, meetings, facilitated discussions, videos, journal articles, press releases, graphics, emails to listservs Are there existing networks or venues through which findings should be disseminate d? Or will communicat ions be distributed directly to target audiences? Who is responsible? Is there a deadline? Did we achieve our goal? What was the result of the information￾sharing? Any observable outcomes? IX. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.13 ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology should be explained in detail and sources of information properly identified. 13 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 52 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ● Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. X. OTHER REQUIREMENTS [This section may include other requirements]. All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether Select those that are applicable and included: in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. XI. USAID CONTACTS Primary Contact Alternate Contact Name Laura McGough Emmanuel Ogwuche Title Senior Health Advisor Commodities & Logistics Program Manager USAID Office/Mission Nigeria/HPN Nigeria/HPN Email lmcgough@usaid.gov eogwuche@usaid.gov Telephone Cell Phone (optional) 0814 957 6013 0814 957 6017 53 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANNEX 2. DATA COLLECTION TOOLS: INTEVIEW GUIDES Interviewee group – Regulation and Control: NAFDAC, PCN and MoH (national level) Regulation and Control respondent types: 1. Administrative (FMoH), NAFDAC DG, Registrar Pharmaceutical Council Nigeria (PCN)) 2. NAFDAC Technical Directorates a. Registration & Regulation (R&R) b. Quality Control/Laboratory Services (QC/LS) c. Port Inspection Directorate (PID) d. Drug Evaluation and Research (DER) e. Pharmacovigilence/Post-Marketing Surveillance f. Planning, Research, and Statistics (PRS) Introduction: The United State Agency for International Development (USAID) Nigeria through the Technical Office of Health Population and Nutrition (HPN) has requested a mid-term evaluation of the Promoting the Quality of Medicines program (PQM) to assess the effectiveness of the project’s technical approach, progress to date, and to determine if it addresses the needs of clients and the objectives of key health initiatives. The period of performance is from the 2014 cost extension to the end of 2017. We are particularly interested in what changes, if any, you have seen over time (since 2014). Through this interview, we would like to ask you about your experience with and assessment of PQM’s work. We would also like to ask you about your thoughts about potential future directions for providing technical assistance to Nigerian stakeholders concerned with access to pharmaceuticals and health system strengthening, in particular in light of the focus on the goal of universal health coverage. Your participation in this evaluation is voluntary. You may refuse to answer any question in the interview or stop the interview at any time. And, of course, your answers are confidential. May we also record this interview for note-taking purposes? Do we have your permission to begin? Interviewer: Date & Location: Type of respondent (see above): Interviewee name/title: Section A: Agency background working with PQM 1. When did you start working with PQM? 2. What were/are your objectives (reasons) for seeking PQM TA? 3. What type of TA did you and are currently receiving? Section B: Effectiveness of TA To what extent has PQM TA been effective in strengthening quality assurance systems for medicines in Nigeria? 1. How effective has PQM TA help NAFDAC in achieving its goal of ensuring safe and efficacious medicines? [ALT: What do you consider effectiveness to be?] Please give specific examples of effectiveness. 1a. What milestones have been achieved? 1b. Please explain how PQM TA helped NAFDAC meet their milestones. 1c. What milestones have not been met under PQM? Why? 54 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 2. Are there other regulatory bodies that may benefit from PQM TA within the context of helping NAFDAC improve quality assurance systems in Nigeria? If so which regulatory bodies and why? Yes No 3. Are there other stakeholders, besides regulatory bodies, that can help NAFDAC ensure quality assurance systems in Nigeria? If yes, specify and why? (Prompt: ask them about PCN, SON, NGOs etc). Yes No 4. What is your perception of the capacity of NAFDAC quality control laboratories (technical and human resource) to undertake medicines quality control and analysis requirements for Nigeria? 5. What concerns do you have regarding sustainability? 6. What are your plans for sustainability beyond the life of PQM? Please be specific. At what stage are you in the sustainability plan? Please be specific 7. How has PQM TA supported the promotion of gender equity within your agency? 8. What is your perception of the quality of PQM TA received? 9. Do you feel you receive an adequate amount of TA? Can you elaborate why or why not? Yes No 10. What is your perception of how the PQM program is managed by PQM? 11. What areas do you feel that PQM can improve the quality of TA and/or management? Please specify. 12. What are your perceptions of PQM TA to improve HR capacity? 13. Have you had any challenges working with PQM? If Yes, please specify Yes No 14. How have these challenges been addressed? Please explain. 15. How have you applied the information and lessons learned from PQM TA to your agency? Please specify 16. What future plans do you have on how to apply results and lesson learned from PQM to decision￾making? Please specify 17. Do you have any recommendations on how PQM can improve in the future? Interviewee group – Quality Control: NQCL, Kaduna, and NIPRD Introduction: The United State Agency for International Development (USAID) Nigeria through the Technical Office of Health Population and Nutrition (HPN) has requested a mid-term evaluation of the Promoting the Quality of Medicines (PQM) program to assess the effectiveness of the project’s technical approach, progress to date, and to determine if it addresses the needs of clients and the objectives of key health initiatives. The period of performance is from the 2014 cost extension to the end of 2017. We are particularly interested in what changes, if any, you have seen over time (since 2014). Through this interview, we would like to ask you about your experience with and assessment of PQM’s work. We would also like to ask you about your thoughts about potential future directions for providing technical assistance to Nigerian stakeholders concerned with access to pharmaceuticals and health system strengthening, particularly in light of the focus on the goal of universal health coverage. Your participation in this evaluation is voluntary. You may refuse to answer any question in the interview or stop the interview at any time. And, of course, your answers are confidential. May we also record this interview for note-taking purposes? Do we have your permission to begin? 55 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Interviewer: Date & Location: Interviewee name/title: 1. What type of TA did you receive from PQM? Specify: What type, when and where? 2a. What is your perception of the quality of TA? 2b. Are there areas that PQM can improve? If so, specify. Yes No 2c. Is there an area they perform well? If so, please specify. Yes No 3a. What Laboratory accreditation program has PQM TA assisted your laboratory to achieve? And when? 3b. Or if you are undergoing accreditation what stage are you at? 4. What is your opinion on CLM in maintaining quality and sustainability of training? 5. What type of TA is still needed to achieve laboratory accreditation? 6a. Has your laboratory participated in proficiency testing under PQM technical assistance? When? Yes No 6b. And what was the outcome? If it was successful, after how many attempts? If it wasn’t, why? 7. Has the laboratory analyzed PMS samples? What was the outcome? Yes No 8. How effective has PQM TA been in helping you develop policies to improve your core functions? Please specify. 9a. What concerns do you have regarding sustainability? Please specify 9b. What are your plans for sustainability beyond the life of PQM? Please specify Yes No 10. How has PQM TA supported the promotion of gender equity within your agency? 11. How have you applied results and lessons learned from PQM to decision making? Please specify 12. Do you have any recommendations on how PQM can improve in the future Interviewee group – Supply Chain: NSCIP and NMEP Introduction: USAID/Nigeria Office of Health Population and Nutrition has requested a mid-term evaluation of the Promoting the Quality of Medicines program (PQM) to assess the effectiveness of the project’s technical approach, progress to date, and to determine if it addresses the needs of clients and the objectives of key health initiatives. The period of performance is from the 2014 cost extension to the end of 2017. We are particularly interested in what changes, if any, you have seen over time (since 2014). Through this interview, we would like to ask you about your experience with and assessment of PQM’s work. We would also like to ask you about your thoughts about potential future directions for providing technical 56 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV assistance to Nigerian stakeholders concerned with access to pharmaceuticals and health system strengthening, in particular in light of the focus on the goal of universal health coverage. Your participation in this evaluation is voluntary. You may refuse to answer any question in the interview or stop the interview at any time. And, of course, your answers are confidential. May we also record this interview for note-taking purposes? Do we have your permission to begin? Interviewer: Date: Interviewee name/title: Section A: Agency background working with PQM 1. When did you start working with PQM? 2. What were/are your objectives (reasons) for seeking PQM TA? 3. What type of TA did you and are currently receiving? Section B: Effectiveness of TA To what extent has PQM TA been effective in strengthening quality assurance systems for medicines in Nigeria? 1. How effective has PQM TA helped your organization in achieving its goal of integrating health disease programs supply chain management activities [ALT: What do you consider effectiveness to be?] Please give specific examples of effectiveness. 1a. What milestones have been achieved? 1b. Please explain how PQM TA helped your organization meet their milestones? 1c. What milestones have not been met under PQM? Why? 2. What is your perception of the capacity of your organization to carry out an integrated supply chain system for high quality medicines (technical and human resource)? 3. What concerns do you have regarding sustainability? 4. What are your plans for sustainability beyond the life of PQM? Please be specific. At what stage are you in the sustainability plan? Please be specific 5. How has PQM TA supported the promotion of gender equity within your organization/agency? 6. What is your perception of the quality of PQM TA received? 7. Do you feel you receive an adequate amount of TA? Can you elaborate why or why not? Yes No 8. What is your perception of how the PQM program is managed? 9. What areas do you feel that PQM can improve the quality of TA and/or management? Please specify. 10. What are your perceptions of PQM TA to improve HR capacity of your organization/agency? 11. Have you had any challenges working with PQM? If Yes, please specify. Yes No 12. How have these challenges been addressed? Please explain. 13. How have you applied the information and lessons learned from PQM TA to your organization/agency? Please specify 57 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 14. What future plans do you have on how to apply results and lesson learned from PQM to decision-making? Please specify 15. Do you have any recommendations on how PQM can improve in the future? Interviewee group – Manufacturers Introduction: The United State Agency for International Development (USAID) Nigeria through the Technical Office of Health Population and Nutrition (HPN) has requested a mid-term evaluation of the Promoting the Quality of Medicines project (PQM) to assess the effectiveness of the project’s technical approach, progress to date, and to determine if it addresses the needs of clients and the objectives of key health initiatives. The period of performance is from the 2014 cost extension to the end of 2017. Through this interview, we would like to ask you about your experience with and assessment of PQM’s work. We would also like to ask you about your thoughts about potential PQM future directions for providing technical assistance to Nigerian stakeholders concerned with access to pharmaceuticals and health system strengthening particularly in light of the focus on the goal of universal health coverage. Your participation in this evaluation is voluntary. You may refuse to answer any question in the interview or stop the interview at any time. And, of course, your answers are confidential. May we also record this interview for note-taking purposes? Do we have your permission to begin? Interviewer: Date & Location: Manufacturer: Interviewee name/title: ================== I would like to begin by asking you to confirm the nature of your relationship with PQM. From reviewing PQM documents, and speaking with PQM staff and USAID, I understand that the relationship that your organization has with PQM is with respect to work on (insert names of medicines/products)_____________. Is this correct? Have I missed anything? =================== Section A. General Questions 1. How do you define high quality manufacturing? Please give specifics 2. How and why were you selected for the PQM program? Year? (probe: PMG-MAN) 3. What is your objective(s) in obtaining TA from PQM? Section B: Type of Technical Assistance (Select all that apply FOR EACH DRUG. If it is not included here, please specify.) *Triangulate with MF Tracking Sheet 1. What type of technical assistance did you receive and/or currently receiving? a. Support to prepare Dossier for submission PCN, NAFDAC, and WHO PQ b. PMG-MAN mock assessment c. NAFDAC or WHO-PQ team mock audit d. GMP compliance/improvement e. Support to improve quality assurance systems 58 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV f. Marketing authorization (license from NAFDAC) g. Other areas (Please specify) 2. What stage are you in meeting the objectives under each TA? a. Are you progressing as planned? If not, why not? 3. How would you rate the overall quality of the assistance? In a scale of 1 to 5, (1= very poor; 2= poor; 3=fair; 4=good; 5= very good) a. Is there a specific type of TA that was better or worse? If yes, which type? 4. How would you rate the overall quantity of TA you received? In a scale of 1 to 5, (1= very poor; 2= poor; 3=fair; 4=good; 5= very good) a. Is there a specific type of TA you did not receive an adequate amount of? If yes, which type? Follow up Questions i. What is it about the technical assistance that PQM provided that makes it effective? Please provide some examples? ii. Was the assistance delivered in a timely manner that was appropriate for your situation? iii. Did the TA help improve your production capacity? If yes, how? If not, why not? iv. Did the TA help improve the quality of medicines? If yes, how? If not, why not? v. What was done well by PQM? Please provide some examples? vi. Have you had any challenges working with PQM? Prompt: technical capacity, management. vii. What areas can PQM improve? Section C: Gender and Sustainability 1. When you send PQM a list of trainees, do you take gender into account? Please explain. 2. What concerns do you have regarding sustainability? 3. What are your plans for sustainability beyond the life of PQM? Probe: specific activities, markets for medicines, return on investment (ROI), and investments (capital and HR). 4. What advice do you have to PQM for future activities to support your goals and objectives? Areas of future PQM investment? List of Drugs Likert Scale (1= very poor; 2= poor; 3=fair; 4=good; 5= very good) Type of Drug 1 2 3 4 5 Amoxicillin dispersible tablet Arthermether lumefertrine Chlorhexidine gel Magnesium Sulphate Injection Oxytocin Sulfadoxine + Pyrimethamine Ready to Use Therapeutic Food Zinc sulfate 59 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Interviewee group – PQM Staff Introduction: The United State Agency for International Development (USAID) Nigeria through the Technical Office of Health Population and Nutrition (HPN) has requested a mid-term evaluation of the Promoting the Quality of Medicines project (PQM program to assess the effectiveness of the project’s technical approach, progress to date, and to determine if it addresses the needs of clients and the objectives of key health initiatives. The period of performance is from the 2014 cost extension to the end of 2017. We are particularly interested in what changes, if any, you have seen over time (since 2014). Through this interview, we would like to ask you about your experience with and assessment of PQM’s work. We would also like to ask you about your thoughts about potential future directions for providing technical assistance to Nigerian stakeholders concerned with access to pharmaceuticals and health system strengthening, in particular in light of the focus on the goal of universal health coverage. Your participation in this evaluation is voluntary. You may refuse to answer any question in the interview or stop the interview at any time. And, of course, your answers are confidential. Do we have your permission to record this interview for note taking purposes? Do we have your permission to begin? Interviewer: Date & Location: Interviewee name/title: Focus question: How has PQM’s management structure, processes, and staffing patterns helped or hindered progress towards achieving the project’s goal and results and ensuring sustainability. Management Structure 1. Can you describe your management structure specifically in relation to managing field staff? 1a. Has it changed since the PQM/Nigeria office was started in 2016? If so, how? Workplan development 2. How does PQM/Nigeria settle on activities to be included in the workplan (or how do you decide which activities over others to include in the workplan)? Probe: role of senior management, managers, field staff, stakeholders, available resources, time. Evidence for decision-making 3. How does PQM apply results and lessons learned to support program decision-making? Please give examples. 4. Does PQM support stakeholders to apply results and lessons learned to support their decision￾making needs? If yes, please specify. Quality 5. Has the sequence of TA or the phasing of activities resulted in hypothesized outcomes? (or is the theory of change correct?) 6. Have there been any unintended consequences of TA on any stakeholder? If so, please be specific. Human Resources 7. How much of a challenge, if at all, has it been for PQM to recruit qualified staff including short-term consultants (this includes all areas of PQM)? Describe and provide examples. 8. How has PQM ensured that recipients of TA strive for gender equity? 9. In 2017 PQM/Nigeria adopted CLM, what impact has the model had on the quality and sustainability of training? 60 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV 9a. Have there been any challenges with CLM? If so, please specify? 10. To what extent does PQM still rely on short-term consultants and for what purposes? Sustainability 11. What is PQM’s definition of sustainability? Probe: HR capacity, technical resources, strategic planning with stakeholders, 12. How is PQM preparing each type of stakeholder to ensure sustainability in their agency/organization beyond the life of PQM? Please give specific examples. 13. Is there a formal sustainability plan in place? If yes, what are the specific components of that plan? If no, is there a plan to develop one? 14. What role, if any, does PQM envision for civil society in ensuring sustainability of PQM after the life of the program? Are there particular stakeholders in mind? Which ones and why? Recommendations 15. From your experience do you have any suggestions for how PQM can be more cost effective? If yes, please specify. Yes No 16. Do you have any advice to USAID/Nigeria regarding how PQM could be improved in the future? If yes, please specify. Yes No Interviewee group – USAID/Nigeria Mission Introduction: The United State Agency for International Development (USAID) Nigeria through the Technical Office of Health Population and Nutrition (HPN) has requested a mid-term evaluation of the Promoting the Quality of Medicines (PQM) program to assess the effectiveness of the project’s technical approach, progress to date, and to determine if it addresses the needs of clients and the objectives of key health initiatives. The period of performance is from the 2014 cost extension to the end of 2017. We are particularly interested in what changes, if any, you have seen over time (since 2014). Through this interview, we would like to ask you about your experience with and assessment of PQM’s work. We would also like to ask you about your thoughts regarding the potential future directions for PQM particularly in light of the focus on the goal of universal health coverage. Your participation in this evaluation is voluntary. You may refuse to answer any question in the interview or stop the interview at any time. And, of course, your answers are confidential. Do we have your permission to begin? Interviewer: Date: Interviewee name/title: Office: ================== 1. What is the effectiveness of the project’s implementation? a. Please explain why your office engages with the PQM mechanism? b. What is your definition of effectiveness in the context of PQMs work? c. How would you describe the effectiveness of PQM’s work in the context of your office’s goals and objectives? d. What issues, if any, have you had with PQM being able to meet their output targets? Please explain. Have they tried to address any of these issues? 61 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV e. What do you think PQM does particularly well? Please give an example(s). f. What areas could PQM improve? Please give an example(s). g. How well does PQM do, or not do, working with stakeholders NAFDAC, NQCL, FMoH? Please describe. h. How well does PQM work, or does not work, with other USAID implementing partners? (if relevant) 2. What is PQM’s ability to demonstrate their work has contributed to the sustainable strengthening of medicines quality assurance systems? a. PQM’s mandate is to contribute to strengthening country medicines quality assurance systems with a focus on sustainability. How well do you think PQM has been able to demonstrate that they are working toward this end in your country? b.What is PQM doing particularly well in this regard? Can you give some examples? c. What challenges does PQM have in demonstrating their contribution to sustainability of strengthening medicines quality assurance systems? Please give some examples. d. What advice do you have for PQM on how to better demonstrate the ways in which they strengthen medicines quality assurance systems in a sustainable way? Please give some examples 3. Do PQM’s technical focus areas continue to be relevant to your office and should additional technical areas be included in a new program design? a. The key technical focus areas of PQM are Regulatory System Strengthening (policies, laws, regulation; GMP/QA; dossier evaluation; inspections; Bioequivalence (BE) studies; post￾marketing surveillance, etc.), QC Laboratory strengthening (QA management systems; analytical instrumentation; support for WHO/ISO accreditation; SOP; training, etc.), and Manufacturing (GMP; chemistry and manufacturing controls, mock audits, WHO/ISO accreditation, Common Technical Document; dossier submission). Do you think that all these are still relevant areas for USAID/Nigeria to continue working in? Please explain. b. Is there an area of PQM’s work that you believe USAID/Nigeria should no longer be working in? For example, please explain c. Is there another area that USAID/Nigeria should work in, in the future and/or provide more resources to address? Please explain. 4. Overall, how satisfied are you with the assistance that PQM/Nigeria has provided? This may include all aspects of PQM (e.g. TA, program management, leadership). a. Are there any other issues related to PQM’s assistance that you would like to discuss? This may include all aspects of PQM’s assistance. b. Do you have any advice for PQM/Nigeria or USAID/Washington on how they can improve the program in the future? Can you give me examples? 62 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANNEX 3. KEY INFORMANT CONTACTS AND INTERVIEW SCHEDULE INTERVIEWEE DESIGNATION ORGANIZATION/ AGENCY LOCATION Day 1: Monday, 16th April 2018 Staff of USP/PQM Nigeria PQM Ikeja Day 2: Tuesday, 17th April 2018 Drugfield Pharmaceuticals Sango Otta Drugfield Pharmaceuticals Sango Otta Drugfield Pharmaceuticals Sango Otta Drugfield Pharmaceuticals Sango Otta Drugfield Pharmaceuticals Sango Otta Drugfield Pharmaceuticals Sango Otta May & Baker Sango Otta Swiss Pharma (SWIPHA) Dopemu, Ikeja Day 3: Wednesday, 18th April 2018 Emzor Pharmaceuticals Ajao Estate, Isolo Emzor Pharmaceuticals Ajao Estate, Isolo Emzor Pharmaceuticals Ajao Estate, Isolo Emzor Pharmaceuticals Ajao Estate, Isolo Emzor Pharmaceuticals Ajao Estate, Isolo Emzor Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo CHI Pharmaceuticals Ajao Estate, Isolo Daily Need Pharmaceuticals Oshodi, Lagos. Pharmatex Amuwo￾Odofin Lagos Pharmatex Amuwo￾Odofin Lagos 63 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV INTERVIEWEE DESIGNATION ORGANIZATION/ AGENCY LOCATION Pharmatex Amuwo￾Odofin Lagos Pharmatex Amuwo￾Odofin Lagos Pharmatex Amuwo￾Odofin Lagos Pharmatex Amuwo￾Odofin Lagos Pharmatex Amuwo￾Odofin Lagos Day 4: Thursday, 19th April 2018 NAFDAC Lagos (Port) NAFDAC Lagos National Quality Control Laboratory (NQCL) Yaba, Lagos National Quality Control Laboratory (NQCL) Lagos Day 5: Friday, 20th April 2018 NAFDAC Lagos National Quality Control Laboratory (NQCL) Yaba, Lagos Day 6: Saturday, 21st April 2018 Travel back to Abuja Day 8: Monday, 23rd April 2018 NIPRD Abuja NIPRD Abuja NIPRD Abuja NIPRD Abuja Pharmacy Council of Nigeria (PCN) Abuja Nigeria Supply Chain Integration Project Abuja NMEP Abuja Day 9: Tuesday, 24th April 2018 NAFDAC Abuja NAFDAC (PRS) Abuja NAFDAC (PRS) Abuja NAFDAC (PRS) Abuja NAFDAC Abuja NAFDAC Abuja National Quality Control Laboratory (NQCL) Agulu National association of Industrial Pharmacist (NAIP) Lagos Day 10: Wednesday, 25th April 2018 JUHEL Abuja Maternal and Child Health Unit FMOH Abuja Maternal and Child Health Unit FMOH Abuja 64 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV INTERVIEWEE DESIGNATION ORGANIZATION/ AGENCY LOCATION Maternal and Child Health Unit FMOH Abuja Maternal and Child Health Unit FMOH Abuja Maternal and Child Health Unit FMOH Abuja Maternal and Child Health Unit FMOH Abuja Maternal and Child Health Unit FMOH Abuja Maternal and Child Health Unit FMOH Abuja Day 11: Thursday, 26th April 2018 TUYIL Federal Ministry of Health (FMoH) Abuja USAID Abuja USAID Abuja USAID Abuja National Quality Control Laboratory (NQCL) Kaduna National Quality Control Laboratory (NQCL) Kaduna National Quality Control Laboratory (NQCL) Kaduna National Quality Control Laboratory (NQCL) Kaduna National Quality Control Laboratory (NQCL) Kaduna Day 12: Friday, 27th April 2018 NEMEL Day 13: Saturday, 28th April 2018 Pharmacy Council of Nigeria (PCN) Abuja Day 14: Monday, 30th April 2018 Chemonics International Nigeria Field Office Abuja Day 15: Thursday, 3rd May 2018 Pharmaceutical Manufacturers Group of the Manufacturers Association of Nigeria (PMG-MAN) Lagos 65 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANNEX 4. REVIEWED DOCUMENTS 1. PQM Documents 1.1. PQM country overview 1.2. Work plans 1.3. Annual and Quarterly report 1.4. M&E plan 1.5. Technical documents 1.5.1.Medicines quality monitoring reports 1.5.2. Surveillance/PMS reports 1.5.3.Oxytocin clinical study 1.5.4.NAFDAC gap analysis 1.5.5.Manufacturer Tracking Sheet 1.6. External communication (e.g. journal articles, presentations, media materials) 2. NAFDAC Documents 2.1. NQAP 3. USAID Documents 3.1. PQM Agreement Documents (2009 and 2013 modification) 3.2. GHSC-PSM STTA report 3.3. PQM Global Evaluation (unpublished final draft) 66 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANNEX 5. CONFLICT OF INTEREST TEAM LEAD: KATIA PETERSON 67 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV PHARMACEUTICAL SECTOR TECHNICAL EXPERT: NELSON OCHEKPE 68 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV HEALTH SYSTEMS EXPERT: CLEMENT INYAGI 69 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV RESEARCH ASSISTANT: ELIJAH IRMIYA NEP 70 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV SENIOR MONITORING, EVALUATION AND LEARNING SPECIALIST: SAMUEL N. GYANG 71 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV ANNEX 6. EVALUATION TEAM SUMMARY KATIA PETERSON: Team Lead, Promoting the Quality of Medicine Evaluation Dr. Katia Peterson is a dynamic research consultant with over 13 years’ experience implementing M&E activities, mixed-method evaluations, and research studies in low-resource settings across an array of development programs. Dr. Peterson is highly trained and practiced in the translation of M&E results and evaluation outcomes to ‘real world’ decisions. EDUCATION Ph.D., Public Health, University of Queensland, Australia, 2009. MPH, Epidemiology, The George Washington University, US, 2002. B.Sc., Biology, The George Washington University, US, 2000. GEOGRAPHIC EXPERIENCE Ethiopia, Malawi, India, Uganda, Philippines PROFESSIONAL EXPERIENCE Research Consultant, IntraHealth International, Chemonics, Ethiopia, Present. Principal investigator of a study on the optimization of human resource planning to support efficient and effective management of pharmaceutical and health commodity supply chain management (USAID’s Global Health Supply Chain Procurement and Supply Chain Management Project). Quantitative, qualitative, and Quality Improvement (QI) analysis models of primary and secondary research data. Production of the study report, including stakeholder recommendations, for dissemination to USAID, national and local GHSC stakeholders. Quantitative Research Consultant, USAID SHOPS Plus Program, Population Services International, Malawi and India, Current. Developed and prepared the qualitative research protocol for the formative research assessment of USAID/India and USAID/Malawi SHOPS Plus program. Development of the qualitative research training for in-country staff and partners. Thematic evaluation of interview and focus group data and final report development for program stakeholders and USAID. Qualitative Research Consultant, USAID/Uganda Population Study, QED LLC., Uganda, 2016. Qualitative data analysis of interview and FGD data. Development of final study report for USAID/Uganda mission. Assistance to QED Deputy and Chief of Party to translate study findings into Uganda’s CDCS (Country Development Cooperation Strategy). Assistance to CLA leadership in the integration of study findings into USAID and partner “Learning Forums”. Research Consultant, USDA School Feeding Program, QED LLC., Arlington VA, 2015. Finalization of meta-analysis and systematic reviews for USDA sponsored study of school feeding programs. Development of final study report. Co-facilitated USDA’s “School Feeding Dissemination Workshop” (Washington DC). Provided recommendations for future PMP indicators for school feeding programs. Qualitative Research, Population Services International, Washington DC, 2012-2014. Provided technical backstopping to in-country research teams for the development, execution, analysis, and publication of mixed-methods studies. Liaison between Research and Metrics and New Business Development team for proposal development. Specific responsibilities included: technical contribution to M&E and research sections of proposals; development of PMPs and targets in conjunction with PSI Regional Researchers and country M&E advisers; and finalization of PMPs and study designs for awarded projects. Assisted country offices in the integration of M&E data into PSI’s real-time “Dashboard for Decision Making” framework. Spearheaded external research partner relationships for PSI evaluation studies. 72 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Monitoring and Evaluation Adviser, IntraHealth International, Chapel Hill NC, 2010- 2012. Technical support to country teams implementing USAID’s health system strengthening Capacity project (Kenya, Uganda, and Tanzania). Backstopped Capacity Kenya, Uganda, and Tanzania team to ensure compliance against PMP indicators and M&E activities. Spearheaded the finalization of PMP indicators, targets, and indicator compendiums with IntraHealth and USAID missions. Developed study protocols, tools, and results dissemination strategies for internal program evaluations. Backstopped SFH/PSI’s Zambia PRISM project (data collection and analysis support for organizational gender audit) and Zambia CDC VCT project. Led capacity building workshops for in-country staff and partners on M&E indicator data collection studies, evaluation methodologies, report writing, and analysis software. Information, Evidence and Research Technical Officer (Contract), Western Pacific Regional Office, World Health Organization, Philippines, 2010. Provided coordination and technical leadership to Asia's EVIPnet network (Evidence-to-Policy resource) for the Western Pacific Region. Facilitated ‘policy dialogue’ events between researchers and senior Ministerial staff to identify potential evidence-based solutions to health system problems. Coordinated and trained in-country research teams in the development and publication of systematic reviews (Cochrane) to inform Ministerial policy questions. Led in country-capacity initiatives for national and provincial level policy makers to utilize systematic reviews for evidenced informed policy making. WPRO appointed representative on WHO Director General’s “Advisory Committee on Health Research. SELECT PUBLICATIONS Longfield, K., Moorsmith, R., Peterson, K., Fortin, I., Ayers, J., & Lupu, O. Qualitative Research for Social Marketing: One Organization’s Journey to Improved Consumer Insight. The Qualitative Report, 2016, 21(1), 71-86 Cameron, DB, Brown, AN, Mishra, A, Picon, M, Esper, H, Calvo, F and Peterson, K. Evidence for peacebuilding: evidence gap map. 3ie evidence gap report. International Initiative for Impact Evaluation (3ie). Feb 2015. Heard, A, Peterson, K, Modi, S, Esper, H, Calvo, F, and Brown, AN. Integrating HIV services with other health services to improve linkage to care, retention, and adherence: a scoping report. 3ie evidence gap report. International Initiative for Impact Evaluation (3ie). July 2014. Peterson. K, Vu.L, Ochako.R, Agot.K. Insights into Potential Users and Messaging for HIV Oral Self￾Test Kits in Kenya. National AIDS and STI Control Programme (NASCOP) and the Kenyan Ministry of Health Official Ministerial Report for Submission to Parliament. February 2014. Newman C, Mwanamwenge M, and Peterson, K. Report on the Society for Family Health Gender Assessment. IntraHealth International, Chapel Hill. January 2013. Report of the Performance Needs Assessment of the Kenya Health Training System. August 2011. Peterson, K. On Being Modern: Modernity, Sex and Reproductive Health among the srey kalip of Phnom Penh. Doctoral dissertation. University of Queensland Press. March 2009. Angela Durey, Peter Hill, Rachelle Arkles, Marisa Gilles, Katia Peterson, Susan Wearne, Condy Canuto, Lisa Jackson Pulver. Overseas-trained doctors in Indigenous rural health services: negotiating professional relationships across cultural domains. Australian and New Zealand Journal of Public Health. Oct 2008. Christine Korhonen, K. Peterson, Catherine Bruder, Paul Jung. Self-Reported Adverse Events Associated with Antimalarial Chemoprophylaxis in Peace Corps Volunteers. American J of Prev Med, September 2007. 73 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV NELSON AGABA OCHEKPE: Pharmaceutical Sector Technical Expert, Promoting the Quality of Medicines Evaluation Nelson Ochekpe is a trained pharmacist with 21 years of experience in medicine and pharmaceutical research, regulation, and academia, with an emphasis on developing and effectively utilizing systems to ensure quality of medicines. Mr. Ochekpe served for five years with the National Agency for Food and Drug Administration and Control (NAFDAC) first as Deputy Director and Head of Laboratory Services and subsequently as Director. Under his leadership NAFDAC established the national vaccine quality control laboratories, pesticide and pesticide residue, food microbiology, and organoleptic laboratories to improve the examination of drugs, vaccines, medical devices, and processed foods. Mr. Ochekpe also established laboratories for pharmaceutical research for the National Institute for Pharmaceutical Research and Development (NIPRD) where he was seconded. Following NAFDA, Mr. Ochekpe returned to an academic position with the University of Jos where he continues to conduct research on quality of medicines from the perspective of impurities, content, materials for formulation, drug delivery issues, stability of formulations, pharmaceutical waste, in vivo and in vitro bioequivalence, and pharmacokinetics. Mr. Ochekpe also serves as independent consultant. He has supported DfID, USAID, and Global Fund assignment across Nigeria. One assignment with DfID evaluated the quality of medicines in Nigeria. This study specifically established the level of counterfeit medicines at 17% and substandard drugs at about 25%; this study led to DfID and USAID interventions in the Yaba drug laboratory of NAFDAC in Lagos. On an assignment with WHO, Mr. Ochekpe surveyed the quality of anti-malarial drugs in six countries revealing that Ethiopia had the fewest cases of substandard medicines as a result of registering bioequivalent formulations. Following this Mr. Ochekpe wrote to NAFDAC to advocate adopting the same approach. Mr. Ochekpe continues to serve as a consultant as a QA/QC expert. He has a deep understanding of the Nigerian pharmaceutical industry and has surveyed most of the pharmaceutical industries in Nigeria. EDUCATION PhD, Pharmacy, Robert Gordon University, Aberdeen, Scotland, United Kingdom, 1988 BSc, Pharmacy, Ahmadu Bello University, Zaria, Nigeria, 1980 GEOGRAPHIC EXPERIENCE Democratic Republic of Congo, India, Nigeria, Switzerland. PROFESSIONAL EXPERIENCE Independent Consultant (selected assignments), 1996-Present Management Sciences for Health (MSH), Grant Management Solutions (GMS), TB/HIV Grant Making – PSM. Provided technical guidance on quality assurance. Abuja, Nigeria, September 2014-June 2015. Global UNIDO. Provided technical guidance on strengthening the local production of essential generic drugs in least developed and developing countries. Abuja, Nigeria, 2009-2010. WHO. Conducted survey of the quality of selected antimalarial medicines circulating in six countries of sub-Saharan Africa. Abuja, Nigeria. Using the minilab concept, he led the Nigerian team’s assessment of the quality of anti-malarial drugs. The study revealed that Ethiopia had the fewest cases of substandard medicines because it adopted a principle of registering on bioequivalent formulations and as a result the country had fewer brands of every formulated medicine MSH/GMS. Provided technical guidance on procurement supply management of HIV/AIDS supplies to the Abidjan-Lagos Corridor Organization (ALCO) under the Global Fund project. Nigeria, 2009. NAFDAC. Conducted quality assurance review of the Drug Quality Control Testing Laboratory. Yaba, Oshodi, Kaduna, Maiduguri, Calabar and Port Harcourt, Nigeria, 2006-2008. 74 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV NAFDAC/WHO/DFID. Performed study of counterfeit and substandard medicines in Nigeria; published report: A Study of Counterfeit and Substandard Medicines. Nigeria, November 2004- September 2006 WHO. Conducted study on regulatory strategy for new vaccines in Africa: Nigeria experience. Geneva, October 16-20, 2000. Professor, Department of Pharmaceutical Chemistry, University of Jos, Nigeria, 2009- Present. Deputy Vice Chancellor (Academic), University of Jos, Nigeria, Dec 2014-Present. Conduct research on quality of medicines from the perspective of impurities, content, materials for formulation, drug delivery issues, stability of formulations, pharmaceutical waste, in vivo and in vitro bioequivalence and pharmacokinetics. Dean, Faculty of Pharmaceutical Sciences, University of Jos, Nigeria, October-December 2014. Deputy Director, Office of Research and Development, University of Jos, Nigeria, 2010- 2014. Acting Head, Department of Pharmaceutical Chemistry, University of Jos, Nigeria, 2007- 2009. Deputy Dean, Faculty of Pharmaceutical Sciences, University of Jos, Nigeria, 2006-2009. Director/Head of Directorate, National Agency for Food and Drug Administration and Control (NAFDAC), Abuja, Nigeria, 1995-2000. Established the national vaccine quality control laboratory, pesticide and pesticide residue, a food microbiology and organoleptic laboratories among other things that brought improvement to analyze drugs, vaccines, medical devices, and processed foods. Established laboratories for pharmaceutical research at the National Institute for Pharmaceutical Research and Development (NIPRD). PUBLICATIONS Ndidi C. Ngwuluka, Ibironke M. Akindele, Nelson A. Ochekpe. Post Federal Ministry of Health mapping survey: Supply chain of medicines in some health facilities within Jos metropolis, Nigeria – submitted to a peer reviewed journal. Ochekpe, A. N., Ngwuluka, N. E. Agbowuro, A. A. and Obodozie, O. O. (2012); Dissolution profiles of twelve brands of sulfadoxine/pyrimethamine in Nigerian market Dissolution Technologies 19(1): 59-64. 3. Ochekpe Nelson A. and Ayodeji A. Agbowuro (2010); Impurities responsible for change in the physical characteristics of compressed Paracetamol tablets. International journal of pharmaceutical sciences 3(2): 771-776. Ngwuluka, N.C., Ochekpe, N.A., and Odumosu P.O. (2011); An assessment of pharmaceutical waste management in some Nigerian pharmaceutical industries. African Journal of Biotechnology 10(54): 11259-11264. A report of cGMP auditing of four Indian pharmaceutical industries; submitted to Christian Health Association of Nigeria Medi-Pharma, February 2011. 75 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Pharmaceutical Sector Profile: Nigeria - A UNIDO study. https://www.unido.org/sites/default/files/2011- 04/Nigeria_Pharma%20Sector%20Profile_032011_Ebook_0.pdf - published 2011. PRESENTATIONS Ngwuluka N.C., Akindele I.M., and Ochekpe N.A. Post FMOH mapping and recommendations: Assessment of supply chain of medicines for health priority diseases in some health facilities within Jos metropolis. – an oral presentation at 10th Global Health Supply Chain Summit, Accra, Ghana (15-17 November 2017). Ngwuluka N.C., Ochekpe N.A and Aruoma O.I. Functions of bioactive and intelligent natural polymers in the optimization of drug delivery – oral presentation at Annual Scientific Conference and Exposition of the Nigerian Association of Pharmacists and Pharmaceutical Scientists in the Americas, Houston, Texas, USA. (21st – 24th September 2017). Ngwuluka N, Ochekpe N. A. and Agbowuro A.A. Comparative dissolution profiling as a basic requirement for product licensing in the West African sub-region – Oral presentation at First Biennial Scientific Conference on Medicines Regulation in Africa, Johannesburg, South Africa (2-3rd December 2013). Ochekpe N. A., Agbowuro A.A. and Ngwuluka N.C. Presence of impurities in pharmaceutical dosage forms as a means of investigative pharmacovigilance – Poster presentation at First Biennial Scientific Conference on Medicines Regulation in Africa, Johannesburg, South Africa (2-3rd December 2013). Ochekpe, N. A. and Ngwuluka, N. C. Waste Management in some pharmaceutical industries in Nigeria – Poster Presentation at the 3rd Pharmaceutical Sciences World Congress, Amsterdam (April 2007). Ochekpe, N. A and Ngwuluka, N. C. Regulatory/policy provisions for pharmaceutical waste management: a study of the Nigerian Pharmaceutical Industry - Poster Presentation at the 79th Annual Conference of Pharmaceutical Society of Nigeria. (November 2006). 76 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV CLEMENT JOSEPH INYAGI: Health Systems Expert, Promoting the Quality of Medicines Evaluation Clement Inyagi has more than 16 years of experience working with the Nigeria health system—his expertise includes strategic planning, Quality Assurance/Quality Improvement (QA/QI), logistics and supply chain management, district health information systems, and management information systems. For seven years Mr. Inyagi served as a pharmacist in several state hospitals where in addition to providing pharmaceutical care services to ART and non-ART clients, he implemented drug logistics management systems to improve quality control and served on a number of committees including the M&E team to ensure the effective and efficient use of systems. For the past nine years since then, Mr. Inyagi has contributed his expertise to two USAID-funded national-level projects: the Global HIV/AIDS Initiative Nigeria (GHAIN) and Strengthening Integrated Deliver of HIV/AIDS Services (SIDHAS) projects. For three years on GHAIN, Mr. Inyagi implemented quality assurance and quality improvement processes for pharmacies to better serve communities delivering HIV testing and counseling, ART and TB services, and treatment of infection. He engaged and established partnerships between community pharmacists and proprietary vendors to strengthen pharmaceutical health systems improving the capacity and quality of pharmaceutical care. Similarly on SIDHAS, Mr. Inyagi provided technical leadership in strengthening pharmaceutical systems by conceptualizing and implementing quality assurance and quality of care in health systems at various levels. He implemented M&E plans and strategies to ensure drug and commodity management systems were successful. Mr. Inyagi successfully brought together various stakeholders resulting in systems that focusing on high quality production, distribution, and inspetion of pharmaceutical partners from the private and public sectors. EDUCATION MS, Health Management, Benue State University, Makurdi, Benue State, Nigeria B.Pharm, University of Jos, Plateau State, Nigeria PROFESSIONAL EXPERIENCE Senior Pharmacy Specialist, Howard University Global Initiative Nigeria (HUGIN)/Strengthening Integrated Delivery of HIV/AIDS Initiatives (SIDHAS), Rivers State, Nigeria, September 2016-Present. Manage and coordinate all HUGIN/SIDHAS activities at the state level including program planning, implementation, monitoring and reporting of interventions implemented. Ensure judicious use of project resources in the state and supervise HUGIN staff in Rivers State office to ensure compliance to technical, programmatic, contractual and financial requirements. Participated in the conceptualization of the Sustainable Financing Initiative project and coordinated the implementation of project activity two in Rivers State. Pharmacy Specialist, Howard University Pharmaceutical Care and Continuing Education Centre (HUPACE)/SIDHAS Rivers State, Nigeria, December 2011-September 2016. Participated in the conceptualization and provided supervision over the implementation of the community antiretroviral therapy (ART) concept, which targeted increasing ART in the communities. Provided technical leadership in strengthening pharmacy systems in particular and health systems in general, as well as developed capacity for health care providers to provide HIV/AIDS, STI and TB prevention information, treatment and management, and referral services both at health facilities and community levels. Conceptualized and implemented quality assurance/quality of care in health systems at the state, LGA, and community levels. Collaborated with other partners to implement zonal program M&E plan in line with country office plan. Developed and implemented strategies to ensure the installation of an integrated drug and commodities logistics management system at comprehensive ART sites and the LGA levels. Reviewed reports and databases from various sources monthly, analyzed the zonal level data and provided reports to the Country office and local partners. Participated in site visits, program management and evaluative reports as well as conceptualized and designed program log frames and work plans. Provided technical assistance and transferred capacity to designated state government officers in the performance of all the roles stated above. 77 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Pharmacy Specialist II/Monitoring & Evaluation Pharmacist, HUPACE/Global HIV/AIDS Initiative Nigeria (GHAIN), Osun State, Nigeria, March 2010-November 2011. Closely collaborated with the other project partner zonal offices to ensure the delivery of quality pharmacy￾related HIV/AIDS, and related services at both community and facility points of service through the facilitation of training workshops, distribution of job aid, SOPs, etc. Developed and implemented strategies for the installation of an integrated drug and commodities logistics management system at comprehensive ART sites and the LGA levels. Reviewed reports and databases from various sources monthly, analyzed the zonal level data and provided reports to the Country office and local partners. Participated in site visits, program management and other evaluative reports as well as conceptualized and designed program log frames and work plans. Provided technical leadership for the clinical pharmacy practice and logistic component of the GHAIN district level community-based operations by incorporating aspects of pharmacy best practices that were hitherto, lacking at the comprehensive sites, stand-alone PMTCT sites as well as primary healthcare centres in the community. Implemented HUPACE strategies for building pharmacist capacity to deliver quality pharmaceutical career services as it relates to integrated provision of treatment care and support for HIV/AIDS, OI, STI, Reproductive Health, TB, and Malaria in both facility and community pharmacies within the zone, through the facilitation of training sessions, development of advocacy plans and paying of advocacy visits to major stakeholders on pharmacy issues to the MOH and HMB, as well as on-site mentoring and provision of technical assistance to facility-based pharmacists, community pharmacists, pharmacy support staff and lower cadre PHC staff. Collaborated with partners to implement zonal program M&E plan in line with country office plan, by co-facilitating zonal M&E meetings as well as verification of available pharmaceutical care data. Served as the zonal and LGA level pharmacy services expert and liaison. Implemented country office strategies at the community level. Provided additional support for facility drug inventory control at GHAIN supported comprehensive sites. Implemented QA/QI processes for facility pharmacies by establishing or strengthening the medicines and therapeutic committees at GHAIN-supported comprehensive sites. Pharmacy Specialist, HUPACE/GHAIN, Anambra State, Nigeria, October 2008-March 2010. Supported facility drug inventory control at GHAIN-supported comprehensive sites, stand￾alone PMTCT sites, and LGA PHC sites. Implemented QA/QI processes for facility pharmacies by setting up and/or strengthening the already established medicines and therapeutic committees at GHAIN-supported comprehensive sites. Facilitated the effective linkage of PPMVs to community pharmacists to strengthen referral linkages in HAST communities. Provided technical leadership for the clinical pharmacy practice and logistic component of the GHAIN district-level community-based operations by incorporating aspects of pharmacy best practices that were lacking at comprehensive sites, stand-alone PMTCT sites as well as primary healthcare centres in the community. Pharmacist in-Charge/ART Focal Pharmacist, General Hospital Oju, Hospitals Management Board, Benue State, Nigeria, February 2006-October 2008. Implemented the drug logistics management system in the hospital through inventory/stock management and coordination of an effective and efficient drug revolving fund (DRF) system. Worked as the facility roll back malaria focal person for the distribution and documentation of long-lasting insecticide treated nets (LLIN) and artemisinin based antimalarial drugs. Developed and implemented a drug information unit in the Hospital, through the sourcing of medication related resource materials. Pharmacist in-Charge/ART Focal Pharmacist, General Hospital Obarike-Ito, Hospitals Management Board, Benue State, Nigeria, March 2004-February 2006. Superintendent Pharmacist, General Hospital Katsina-Ala, Hospitals Management Board, Benue State, Nigeria, November 2002-March 2004. NYSC Pharmacist, Primary Healthcare Centre, Charanchi, Katsina State, Nigeria, June 2001-May 2002. 78 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Pharmacist (Intern), State Specialist Hospital, Maiduguri, Borno State, Nigeria, November 1999-December 2000. Community Pharmacist, Nazo Pharmacy & Stores, Maidguri, Borno State, Nigeria, November 1999-June 2001. ELIJAH IRMIYA NEP: Research Assistant Elijah Nep has more than 20 years of experience related to pharmaceuticals. He began his career as a medical representative for a Nigerian pharmaceutical company before working for a state-run general hospital as a pharmacy manager. For the past 12 years, however, Dr. Nep has served as a lecturer at the University of Jos where he has not only taught undergraduate and graduate level courses in pharmceutics, he performs extensive research. He has published and presented on a number of pharmaceutical-related topics. EDUCATION PhD, Aston University, Birmingham, United Kingdom, 2010 MS, Learning & Teaching, Aston University, Birmingham, United Kingdom, 2005 BPharm, University of Jos, Jos, Nigeria, 1994 LANGUAGES Mwaghavul – Native; English – Fluent; Hausa – Fluent PROFESSIONAL EXPERIENCE Senior Lecturer, University of Jos, Jos, Nigeria, November 2005-Present. Teach undergraduate and postgraduate courses in pharmaceutics. Supervise undergraduate and postgraduate projects. Research. Pharmacy Manager, General Hospital Langtang, Nigeria, October 1999-October 2005. Screened and dispensed prescriptions. Provided drug information services. Managed and administered the pharmacy unit. Medical Representative, NGC PLC, Lagos, Nigeria, May 1995-September 2000. Marketed and promoted the company’s drug products. Prepared and delivered clinical presentations to prescribers. PUBLICATIONS Nep, E.I., Asare-Addo, K., Conway, B.R., Smith, A.M. (2018) Hydroalcoholic media effects on release of theophylline from sesamum gum matrices. Drug Development and Industrial Pharmacy 44(2): 251- 260. http://dx.doi.org/10.1080/03639045.2017.1386209. Nep, E.I., Aljeboury, M.H.M., Adebisi, A.O., Dawson, C., Walton, K., Bills, P., Conway, B.R., Smith, A.M., Asare-Addo, K. (2017) The influence of hydroalcoholic media on the performance of grewia polysaccharide in sustained release tablets. International Journal of Pharmaceutics 532: 352–364. Kemas, C.U., Nep, E.I., Ngwuluka, N.C., Ochekpe, N.A. (2017) Starch acetate xerogels: Effect of Acetylation on Physicochemical and rheological Properties. International Journal of biological Macromolecules 98: 94–102. Nep, E.I., Carnachan, S. M., Ngwuluka, N.C., Kontogiorgos, V., Morris, G. A. Sims, I. M. and Smith, A. M. (2016). Structural characterisation and rheological properties of a polysaccharide from sesame leaves (Sesamum radiatum Schumach. & Thonn.). Carbohydrate Polymers 152: 541–547 79 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Nep, E.I., Sims, I.M., Morris, G.A., Kontogiorgos, V., Smith, A.M. (2016) Evaluation of some important physicochemical properties of starch free grewia gum. Food Hydrocolloids 53: 134-140 Nep, E.I., Asare-Addo, K., Ngwuluka, N.C., Conway, B.R., Smith, A.M. (2016) Sesamum radiatum gum matrices: Compaction, swelling, erosion and drug release behaviour. British Journal of Pharmacy doi: 10.5920/bjpharm.2016.02 7. Asare-Addo, Kofi., Supuk, E., Mahdi , M.H., Adebisi, A.O., Nep, E.I., Conway, B.R., Kaialy, W., Al-Hamidi, H., Nokhodchi, A. (2016). Drug release from E chemistry hypromellose tablets using the Bio-Dis USP type III apparatus: An evaluation of the effect of systematic agitation and ionic strength. Colloids and surfaces B: Biointerfaces 143:481-489. Nep, E.I., Kemas, C.U., Ngwuluka, N.C., Ochekpe, N.A.(2016) Rheological and Structural Properties of Modified Starches from the Young Shoots of Borassus aethiopium. Food Hydrocolloids 60: 265-270 Nep, E.I., Asare-Addo, K., Ghori, M.U., Conway, B.R., Smith, A.M. (2016) Starch-free grewia gum matrices: Compaction, swelling, erosion and drug release behaviour. International Journal of Pharmaceutics 496 (2): 689-698. Asare-Addo, Kofi., Supuk, E., Mahdi , M.H., Adebisi, A.O., Nep, E.I., Conway, B.R., Kaialy, W., Al￾Hamidi, H., Nokhodchi, A. (2016). Drug release from E chemistry hypromellose tablets using the Bio￾Dis USP type III apparatus: An evaluation of the effect of systematic agitation and ionic strength. Colloids and surfaces B: Biointerfaces 143:481-489. Ngwuluka, N.C., Nep, E.I., Ochekpe, N.A., Odumosu, P.O., Olorunfemi, P.O. (2015) Eudragit E100 and Polysaccharide Polymer Blends as Matrices for Modified-Release Drug Delivery I: Physicomechanical Properties. Tropical Journal of Pharmaceutical Research 14 (12): 2155-2162 Ngwuluka, N.C., Nep, E.I., Ochekpe, N.A., Odumosu, P.O., Olorunfemi, P.O. (2015) Eudragit E100 and Polysaccharide Polymer Blends as Matrices for Modified-Release Drug Delivery II: Swelling and Release Studies. Tropical Journal of Pharmaceutical Research 14 (12): 2163-2170 Ochekpe, N.A., Kemas, U.C., Nep, E.I. (2013) Chemical Modifications and their Effects on Binding/Disintegrating Properties of Plectranthus esculentus Starch in Chloroquine Phosphate Tablets. American Journal of Pharmaceutical Technology and Research, 3(3):867-877. Ogaji, I.J., Nep, E.I., Audu-Peter, J.D. (2012). Advances in Natural Polymers as Pharmaceutical Excipients. Pharm Anal Acta 3:146. Nep, E.I. and Conway, B.R. (2012). Pre-formulation studies on grewia gum as a formulation excipient. Journal of Thermal Analysis and Calorimetry. 108 (1), 197-205. Nep, E.I. and Conway, B.R. (2011). Grewia polysaccharide as a pharmaceutical excipient in matrix tablets. J. Excipients and Food Chem. 2 (1), 3-15. Nep, E.I. and Conway, B.R. (2011). Physicochemical characterization of grewia polysaccharide gum: effect of drying methods. Carbohydrate Polymers 84(1), 446–453. PRESENTATIONS E.I Nep, K. Vasiliosis, G. Morris, A.M. Smith. Novel Polysaccharides from Grewia mollis: Characterization and rheology – Oral presentation at the 12th International hydrocolloids Conference Taipei, Taiwan (5th -9th May, 2014). Nep, E.I. (2013). Grewia gum for use as a sustainable pharmaceutical excipient. 1st UK Symposium on Hydrocolloids, University of Huddersfield (10th Sept., 2013). 80 | USAID MIDTERM EVALUATION OF THE PROMOTING THE QUALITY OF MEDICINES ACTIVITY USAID.GOV Nep, E.I. and Conway, B.R. (2010). Grewia gum matrix tablets for the release of cimetidine. AAPS Annual meeting and exposition, New Orleans- USA (13th – 18th Nov., 2010). Samuel N. Gyang: Senior Monitoring, Evaluation and Learning Specialist, Promoting the Quality of Medicines Evaluation A long-time monitoring and evaluation and data management specialist, Sam has more than 11 years of hands-on experience in designing and executing projects and evaluations related to public health systems, education, water and sanitation, youth inclusion, and social development, with multiple projects focusing on remote areas in Nigeria. As the National Consultant for Health Management Information System (HMIS), Sam served as an Advisor to Partnership for Transforming Health Systems II (PATHS2-DFID) and conducted a process evaluation of the HMIS outputs. He also developed plans for conducting a Rapid Data Quality Review (RDQR) in all the supported states in Nigeria, using standard design and methodological frameworks. Prior to this role, Sam provided technical advice to the MAPS program as a Monitoring and Evaluation Advisor where he improved the data collection and M&E systems and reviewed the existing database to ensure the timely sharing of quality data to relevant agencies and State-level officials in Nigeria. Sam is currently the Senior Monitoring, Evaluation and Learning Specialist for DevTech, where he leads the design and application of qualitative and quantitative approaches to USAID/Nigeria technical offices data gathering and analyses for assessments, evaluations, baselines, data quality assessments, and other M&E data gathering efforts. He also focuses on building capacity in Monitoring and Evaluation for USAID/Nigeria and its implementing partners while supporting USAID technical staff on PMP and MEL plan reviews and designing and conducting, among other duties.