STRENGETHENING PHARMACEUTICALS SYSTEMS LESSONS LEARNED EVALUATION REPORT 23 DECEMBER 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Ann Von Briesen, Management Systems International. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report A subsidiary of Coffey International, Ltd. Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contract No. DFD-1-00-05-00251-00, Task Order No. 2 Services Under Program and Project Offices for Results Tracking (SUPPORT) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report i PROJECT DATA SHEET 2 All financial figures are in United States Dollars (USD) Program Name Strengthening Pharmaceutical Systems (SPS)-Southern Sudan Region/Country Africa/Southern Sudan (Geographic Area 650) Program Bilateral Health Program Government of Southern Sudan Counterpart Ministry of Health (MOH), Government of Southern Sudan (GOSS) Funder United States Agency for International Development (USAID) Managing Contractor Management Sciences for Health (MSH) Agreement Information Centrally funded Field Support Mechanism. Direct Grant/Cooperative Agreement/Leader with Associate Award (LWA): GHN-A-00–07–00002–00 SPS Impact Evaluation Mission TBD SPS Midterm Evaluation Mission Sept 13-Oct 10, 2010 SPS Award Period Oct 1, 2005-Sept 30, 20121 Rational Pharmaceutical Management (RPM)-SPS Design August 2005 Key Program Dates RPM-SPS Implementation Period Oct 1, 2005 to Sept 30, 2012 Total Contract Value (2005–2010)2 : $5.8m Evaluation Provider Management Systems International (MSI) Team Members: Ann Lewis (External Evaluator/Team Leader) Evans Sagwa (External Evaluator/MSH-SPS) Dr. Martin Swaka (Health Project Management Specialist, USAID/Sudan). MTE Mission Site Visits Southern Sudan States of: Central Equatoria and Eastern Equatoria. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report ii ACKNOWLEDGEMENTS The evaluation team thanks the personnel of the Government of Southern Sudan (GOSS), Ministry of Health (MOH), the Management Sciences for Health (MSH) Strengthening Pharmaceutical Systems (SPS) team in Sudan, and the United States Agency for International Development (USAID)/Sudan. Mission personnel gave generously of their time and provided the Team with excellent commentary. We offer special thanks to the personnel of the State Ministry of Health (SMOH) in Eastern Equatoria and the Torit Hospital for their hospitality and assistance. Many members of the donor and development partner community improved our understanding of the context in which SPS is being implemented. The team thanks the staff of the Services Under Program and Project Offices for Results Tracking (SUPPORT), implemented by Management Systems International (MSI), for its assistance with strategic guidance, administrative aspects, and on-the-ground logistical assistance, especially Dr. Valerie Haugen (a planning, monitoring and evaluation expert). We commend our colleagues who participated as members of the midterm evaluation team: Dr. Martin Swaka (USAID/Sudan), Mr. Evans Sagwa (MSH Namibia), with other assistance from Dr. Bortel (GoSS MOH). The contributions of Dr. Robert Azairwe (SPS Team Leander and Senior Technical Advisor), and Dr. Steven Mawa (SPS Senior Technical Advisor for Pharmaceutical Management) were also invaluable. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report iii ACRONYMS ACT Artemisinin-based combination therapy AED Academy for Educational Development AS/AQ Artesunate/Amodiaquine CA Cooperative Agreement CSM Central Stores Management COP Chief of Party COTR Contracting Officer Technical Representative CPA Comprehensive Peace Agreement EPI Expanded Program for Immunization GBV Gender Based Violence GFATM Global Fund for AIDS, Tuberculosis and Malaria GOSS Government of Southern Sudan HMIS Health Management Information System HICD Human and Institutional Capacity Development IDF Institutional Development Framework ITN Insecticide treated nets LLIN Long lasting insecticidal nets LLITN Long lasting insecticide treated nets LTA Long Term Technical Assistance LTTA Long Term Technical Advisor M&E Monitoring and Evaluation MDG Millennium Development Goals MDTF Multi-Donor Trust Fund MOH Ministry of Health (Southern Sudan) MOU Memorandum of Understanding MSH Management Science for Health MSI Management Systems International MTE Mid-Term Evaluation NGO Non-Governmental Organization NMCP National Malaria Control Program OD Organizational Development OECD Organization for Economic Cooperation and Development PMP Program Monitoring Plan RBM Roll Back Malaria RPM+ Rational Pharmaceutical Management (plus) RTK Rapid Test Kit SCMS Supply Chain Management System SMOH State Ministry of Health SMS State Medical Stores SNAP Sudan National AIDS Program SOP Standard Operating Procedure SOW Scope of Work/Statement of Work SPS Strengthening Pharmaceutical Systems STTA Short-Term Technical Assistance SUPPORT Services Under Program and Project Offices for Results Tracking TA Technical Assistance TAP Technical Assistance Program TCN Third Country National Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report iv TOR Terms of Reference TOT Training of Trainers UN United Nations UNAIDS Joint United Nations Programme on HIV/AIDS UNDP United Nations Development Program UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG United States Government WHO World Health Organization Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report v CONTENTS PROJECT DATA SHEET ............................................................................................................. i ACKNOWLEDGEMENTS ..........................................................................................................ii ACRONYMS.................................................................................................................................iii EXECUTIVE SUMMARY .......................................................................................................... vi I. INTRODUCTION...................................................................................................................... 1 II. PROGRAM RESPONSE ......................................................................................................... 5 III. EVALUATION METHODOLOGY.................................................................................... 13 IV. FINDINGS AND CONCLUSIONS...................................................................................... 16 V. LESSONS LEARNED ............................................................................................................ 28 VI. RECOMMENDATIONS ...................................................................................................... 29 ANNEXES .................................................................................................................................... 31 Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report vi EXECUTIVE SUMMARY Background and Program Description In addition to conflict, natural disasters, and famine, policies and practices have destroyed fragile social and economic infrastructures and have severely affected Southern Sudan. With the (presumably) lowest human development indicators in the world, health and education remain almost exclusively externally funded, and it is estimated that 60 percent of health service delivery is provided by nongovernmental organizations (NGOs), including long-standing faith-based organizations (FBOs). U.S. government assistance in Sudan emphasizes building peace dividends such as sustained and comprehensive improvements in social services, economic growth, and democratic governance. Well before the signing of the CPA in 2005, the United States Agency for International Development (USAID) worked with members of the nascent government of Southern Sudan (GOSS) to address the pressing need to improve health care in the aftermath of nearly 50 years of war. GOSS officials have made tremendous progress since 2005, and attention is now focused on building sustainable basic health systems and infrastructure, and strengthening human capacity. Responding to a request from the GOSS Ministry of Health (MOH), USAID/Sudan introduced the Field Support Mechanism (FSM) of the Management Sciences for Health (MSH) Rational Pharmaceutical Management Plus (RPM+) in 2005, followed by its successor, the Strengthening Pharmaceuticals Systems (SPS) Program, in 2007. The SPS Program, a five-year, $5.8 million Leader with Associates Cooperative Agreement (CA), was awarded to MSH from June 2007 to September 30, 2012. Since 2005, USAID/Sudan has supported the GOSS MOH through this project, which has grown incrementally in response to the MOH’s identified priorities, including pharmaceutical management and the Expanded Program for Immunization (EPI). The SPS Program uses a mentoring approach with SPS long-term technical advisors (LTTAs), experts from the region, placed in the relevant departments of the MOH to work on a full-time basis with their national counterparts. SPS Annual Work Plans (AWPs) are developed cooperatively and establish activities that are in line with both SPS Intermediate Results and MOH strategic plans and targets. Implementation of SPS activities is undertaken with and through the relevant MOH institutions. A systems strengthening approach, including office and job-sharing, is employed to support MOH institutions at central, state, and local levels to effectively coordinate and manage the malaria, pharmaceutical, and EPI programs. At present, three senior LTTAs advise in these areas, and Sudanese advisors lead the Monitoring and Evaluation program and assist in pharmaceutical management. SPS technical support initially focused on development of key program policies and strategies at the central MOH GOSS level, but has progressively included support to implementation in three selected states. Purpose of the Midterm Evaluation and Methodology USAID/Sudan requested the Services Under Program and Project Offices for Results Tracking (SUPPORT) Project, implemented by Management Systems International (MSI), to field a team to undertake a midterm evaluation of the SPS Program. The focus of the evaluation was on learning lessons from program implementation to date in order to assess the effectiveness of the SPS Program capacity building model in the MOH. The fieldwork for this formative evaluation was undertaken in a collaborative manner with participation by the USAID/Sudan Health Office, the GOSS MOH counterparts, and the implementing partner, MSH. The evaluation was led by an independent, external evaluator who bears sole responsibility for conclusions drawn and recommendations made and reported on in this document. The original SPS assumption is that highly qualified long-term technical advisors from the region, embedded within the MOH and tasked with the responsibilities to be good role models and to mentor Sudanese staff, can build human capacity while establishing a solid framework of policies, procedures, practices, and direct program implementation. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report vii The evaluation was undertaken in September 2010 and was carried out predominantly in the capital city, Juba, in Central Equatoria State (CES) in Southern Sudan, with a site visit to the Torit Hospital in Eastern Equatoria State (EES). The team included Dr. Martin Swaka, Contracting Officer’s Technical Representative (COTR) of the USAID/Sudan SPS program, Mr. Evans Sagwa of MSH SPS-Namibia, and Team Leader Ms. Ann von Briesen Lewis. Dr. Bortel Ohisa from the GOSS MOH also participated in parts of the mission. The team designed the evaluation around the questions posed in the scope of work (SOW) and used mixed methods for data collection including adapted tools, modified focus group discussions, key participant interviews using semi-structured questionnaires with over 40 stakeholders, and direct observation. Most Significant Findings, Conclusions, Lessons Overall The SPS program utilizing the model of embedded long-term technical advisors has proven to be exceptionally well-suited to the resource-poor and conflict-affected context of Southern Sudan. The incremental addition of highly qualified regional experts dedicated to building and fostering Sudanese ownership has been, in the words of a senior MOH official, “indispensable in building policies, programs, and human capacity in a new government.” Program Design The program design and underlying (implicit) assumptions are appropriate to the capacity building needs of Southern Sudan. The MSH SPS Program has proven to be sufficiently nimble to respond to varied needs for technical assistance (TA) and additional tools and support. The focus at the central level and on policy development was appropriate for the early years of the program, and the MOH now has the structures in place for expansion to more than three states and eventually to the county level. The job descriptions and the perceived roles of the LTTAs require maintaining a fine balance between the advising and mentoring role and the gap-filling and doing role. Management Embedding technical advisors within MOH structures has allowed the advisors to be located at the same places where key decisions are made and to build trust; working relations with the MOH have been smooth. MSH oversight and support is timely and responsive, relieving the USAID/Sudan Mission of managerial burdens. The overall team (MOH, USAID, MSH, and SPS LTTAs) is strong and cohesive, and its members communicate and cooperate regularly. The SPS Program Team Leader plays a pivotal role as supervisor as well as senior technical advisor. Capacity Building: What Does Enhanced Capacity Look Like? Longevity: SPS counterparts, colleagues, and advisors in the SPS have remained in place; this finding is particularly noteworthy in the MOH context, where overall human capacity is only beginning to develop more broadly, with few skilled health professionals, and a high turnover of staff. Coordination: Other donor and implementing partner integration is very high; not only have partners bought into MOH plans and policies, but also evidence shows they now also seek strategic direction from the MOH. Other donors, NGO partners, and the MOH have adopted many of the tools developed by the MOH and the team and report evidence of increased output of individuals, work units, and programs. Leadership: MOH departments have been effectively brought into control of its programs, and many guidelines, policies, strategies, and implementation tools are in place and in use. MOH officials make the presentations, lead the meetings, and make the decisions, based on information gathered with assistance from SPS advisors. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report viii Training: The training and human resource development strengthens capacity and remains in innumerable and sometimes immeasurable ways. Sustainability: The SPS program has established a platform for the future scaling up of interventions, regardless of the funding source. Technical Aspects Training provided by SPS has exceeded targets in numbers trained, but has fallen short in achieving gender￾based targets. Dozens of policies, protocols, curricula, and tools were developed with SPS support, and evidence shows their use and impact. Advances in malaria technical advice and pharmaceutical advice to the NMCP and in developing pharmacy management, especially in drug regulation and supply chain management, are credited to SPS activities. Key Recommendations The following recommendations are key, based on evaluation findings and conclusions.3  USAID/Sudan should continue using the field support mechanism of the SPS program, planning annual “buy-in” at increasing levels commensurate with the projected increasing capacity-building needs of the MOH. (R1) Action: USAID/Sudan.  USAID/Sudan, the Implementing Partner (MSH), and the MOH should establish priority areas of expertise and appropriate timing for additional advisors in the coming years. Likely specialty areas are logistics and pharmacovigilance.4 (R2) Action: USAID/Sudan, MSH, MOH.  In coordination with the MOH Human Resource Development Department and other donors that are funding health advisors (WB MDTF, JICA, WHO), MSH/SPS should develop or participate in developing a multi-year and multi-pronged human resource development plan. (R3) Action: MSH, MOH.  MSH program managers and the MOH should develop a candidate profile and interview protocol that will elicit the qualities identified by respondents as crucial to success of an individual in an advisory position in the MOH: patience, humility, professional excellence, an amenable personality, a positive attitude, and a willingness to cooperate. (R4) Action: MOH.  MSH and USAID/Sudan managers should better define roles and responsibilities of LTTAs and their counterparts. These agreements (for which models are available) should provide the latitude and flexibility for LTTAs to respond to changing needs and priorities, while also providing guidance and 3 Bracketed figures correspond to the table in Annex K: Findings, Conclusions, Recommendations. 4 Pharmacovigilance is concerned with the detection, assessment and prevention of adverse reactions to drugs. (Taken from: http://apps.who.int/medicinedocs/en/d/Jh2934e/3.html) Lessons for Success  Time matters. The longevity and perseverance of the SPS advisors have won them the respect and trust of colleagues and supervisors. Multi-year contracts and incentives to keep advisors supported and satisfied pay off.  Personality matters. Anecdotally, the evaluation team was told that the agreeable and amenable nature of the SPS LTTAs was key to their success. Future recruitments, interviews, and reference checks should emphasize social skills along with adaptability and technical expertise.  Proximity matters. While several other programs have long-term advisors working with the MOH, only the SPS team is co-located in the MOH offices together with colleagues and counterparts. The day￾to-day flow of business promotes a collegial atmosphere and builds trust. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report ix protection for LTTAs from excessive gap-filling and extraneous assignments. (R5) Action: USAID and MOH.  Within a year, USAID and the GOSS MOH should make decisions regarding the placement of LTTAs at the State Ministry of Health (SMOH) offices. In the three states already active, placement might be considered sooner. (R6). Consider lessons from the Technical Assistance Program (TAP) implemented by Academy for Educational Development (AED) in planning state-level placements. Action: MOH, USAID, SPS.  Deliberate efforts must be made to examine and address the low representation of women in trainings and to remove obstacles to women’s full participation in training. In the context of the GOSS’ ambitious gender balance program, the MOH must increase gender awareness, expand pre￾service education, and apply gender consciousness to issues of training and recruitment, with the goal of encouraging more active female participation and improving awareness of male and female behavior norms as they affect health access and behaviors. Gender-awareness, and the differential impact of policies and program on gender and other equity issues, should permeate all levels of policymaking and program planning. (R9) Action: MOH, MSH. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report x President Salva Kiir, Southern Sudan, Inauguration Address (May 21, 2010): “Health: It shames us all that our citizens are still prone to endemic diseases such as malaria, bilharzias, and kalazar, as well as the scourge of HIV/AIDS. Child mortality rates in Southern Sudan are among the highest in the world (1700 maternal deaths per 100,000 deliveries and 250 child deaths per 1000). If this does not shame us, nothing else will. [ . . . ] We shall also continue with our efforts to enhance curative medicine facilities in urban and suburban areas, while stepping up the implementation of our programs on primary health care and sanitation. To this end I shall do my utmost to enlarge primary health care to cover the whole of Southern Sudan in the coming five years and endeavor to guarantee universal access to clean water within the coming ten years. To achieve our ambitious programs on health and education we must endeavor to double our expenditure in the two areas as a percentage of the GDP.” Figure 1: Pharmaceutical store for Eastern Equatoria State and Torit Hospital Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 1 I. INTRODUCTION 1.1 Background: War and the Comprehensive Peace Agreement (CPA) Sudan is at a critical stage in its history. The country has been ravaged by conflict for all but 11 of the last 50 years. The signing of the Comprehensive Peace Agreement (CPA) by the Sudan People’s Liberation Movement (SPLM) and the government of Sudan on January 9th, 2005, brought an end to Africa’s longest running conflict. The nascent government of Southern Sudan (GOSS) remains institutionally weak, and implementation of key of the CPA have been inadequate, particularly the Abyei Protocol and transparency in revenue sharing. While the CPA ushered in peace after decades of civil war, it also raised Southern Sudanese citizens’ expectations for peace dividends that have been slow to come. In January 2011, citizens of Southern Sudan will vote on whether their region should become an independent country or remain as a semi-autonomous entity within the Republic of Sudan. The choice is stark, controversial, and steeped in the volatile, often violent, course of the country’s history since independence from British-Egyptian rule in 1956. 1.2 Socio-Economic Facts Southern Sudan has a landmass of about 400,000 square miles, with an estimated population of between 7.5 and 9.7 million people. Poverty, food insecurity, and mortality levels (associated with poor basic services) are very high; the distribution of wealth remains a source of tension. Sudan ranked 150 out of 182 countries on the Human Development Index (HDI). If disaggregated data were available on the South, it is likely that the region would rank even lower. Nonetheless, as some researchers note, care should be taken in depicting rural Southern Sudan as uniformly ‘poor.’ The vast land and cattle holdings of some communities contrast with their ‘cash-poor’ Map of Sudan Map of South Sudan Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 2 status and impoverished basic services—and, therefore, health statistics.5 Sudan’s economic growth over the last 10 years has been remarkable: its gross domestic product (GDP) expanded from $10 billion in 1999 (the year oil exports started) to $53 billion in 2008, and annual per capita income rose from $506 in 2003 to $1,139 in 2007. Through the CPA, Southern Sudan has received about half of the country’s newfound oil wealth since 2005, or approximately $2 billion per year. Paradoxically, for such a self-evidently poor region, the per capita GDP for Southern Sudan—with its relatively small population—is higher than many countries in sub-Saharan Africa.6 The GOSS remains heavily dependent on oil, which accounts for 98 percent of its revenue according to the Minister of Finance. Competition for scarce resources and limited livelihood options threaten the fragile peace gains that have been realized. Bride-price inflation has fueled land and cattle disputes. Socioeconomic indicators have been unreliable due to the tendency to extrapolate from project and/or geographically specific data in the absence of systematic national data collection. The situation is changing gradually as baseline data become available through national institutions; meanwhile there is heavy reliance on United Nations (UN) data. The following estimates are often cited in literature:  An estimated 4.3 million people in Southern Sudan—about half the population—are expected to require food assistance at some point in 2010. The results of five nutrition surveys completed since December 2009 in various locations point to acute malnutrition rates of over 15 percent, which is the emergency threshold.7  Although the under-five mortality rate decreased from 250 (per 1,000 live births) in 2001 to 135 in 2006, one out of every six children in Southern Sudan die before their first birthday, and 305,000 children die before their fifth birthday every year from preventable causes. Southern Sudan has the lowest routine immunization coverage rate in the world.8  26,000 women in Sudan die in childbirth each year—an average of 71 women every day.9 In 2006, maternal mortality was an astounding 2,037 per 100,000 births.  Less than 50 percent of all children in Southern Sudan receive five years of primary school education, compared to 93 percent in Darfur.10  92 percent of women in Southern Sudan cannot read or write.11 1.3 Health Sector Overview Well before the signing of the CPA in 2005, USAID worked with members of the nascent GOSS at its base in Nairobi to establish and rank the critical investments and needs for the new government. One of the most pressing needs was to bring health care in the aftermath of nearly 50 years of war. Sudan’s health indicators were—and remain—among the worst in the world. Due to the war, there was an almost total lack of health system infrastructure, equipment, materials, and—most critically—trained human resources. The urgent needs were to provide primary health care services to the population, while at the same time creating a functioning public health care system to provide services over time. Meeting the two critical needs was, and remains, a careful balancing act. The health sector in Southern Sudan is extremely weak, and is characterized by sub-standard quality of services, limited coverage of health facilities vis-à-vis population size, unequal distribution of facilities, and in 5 Multi-Donor Evaluation of Support to Conflict Prevention and Peace-Building Activity in Southern Sudan Since 2005; August 2010. ITAD in association with Channel Research. 6 Ibid. 7 Mar-Apr 2010 Humanitarian Update, Southern Sudan Issue #2 OCHA Sudan UN 8 Office of the UN Resident and Humanitarian Coordinator for Sudan, website 9 Ibid. 10 Ibid. 11 Ibid. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 3 some remote areas, a total lack of health facilities. There is also a significant rural/urban divide with health facilities, services, and personnel more apt to want to work in urban areas. Health and education remain almost exclusively externally funded, and it is estimated that 60 percent of health service delivery is provided by NGOs, including long-standing faith-based organizations and a plethora of NGOs that became entrenched during the course of the war. The capacity of the government at the moment remains limited to efforts to better regulate and coordinate health services provision to the citizenry. Overall, reproductive health is poor due to a lack of access to skilled antenatal care providers (the proportion of births attended by skilled health staff is also among the lowest in the world), a high fertility rate (5.9 live births per woman according to the United Nations International Children’s Education Fund (UNICEF)), low rates of modern family planning methods, and a high percentage of home births (94 percent). The maternal mortality rate is one of the highest in the world (2,037 per 100,000 births according to U.S. Department of Health and Human Services data from 2006). The percentage of deliveries attended by trained birth attendants is five percent, and contraceptive prevalence rates are low (one percent in the South). Obstetric fistulas are common; however, there are no prevention programs and repair is not available in hospitals across Southern Sudan. HIV/AIDS is an emerging threat due to risky sexual behavior and Sudan’s proximity to the regional pandemic. HIV/AIDS prevalence, based on scarce epidemiological and behavioral information, is estimated by the Joint United Nations Programme on HIV/AIDS (UNAIDS) to be around 2.3 percent in the adult population. Rates of HIV infection have been estimated by Sudan National AIDS Control Program (SNAP) to be at 1.6 percent nationwide. However, for Southern Sudan, estimates vary from 1 percent to 7.2 percent, with alarming rates among certain population. According to the United Nations Population Fund (UNFPA), as of 2006, infant mortality was 102 per 1000 live births; mortality for children under five years of age was 135 per 1000. Vitamin A deficiency affects one of seven children in Sudan and DPT3 immunization among children is below 44 percent. Water and sanitation infrastructure is non-existent or marginal at best, and sanitation and hygiene practices are poor. Critical health risks include tuberculosis, malaria, pneumonia, and diarrhea. 1.4 GOSS Capacity GOSS officials have made tremendous progress since 2005. However, while ministry organizational structures exist on paper, they remain marginally functional, with only a handful of appropriately trained staff in each ministry. The civil service payrolls include staff lacking the skills required for their jobs, and little budget remains after paying civil service salaries. USAID/Sudan commissioned a study in mid-2010 to review ways in which donor agencies could improve capacity building efforts. The author found that, “capacity building efforts in Southern Sudan are currently neither strategic nor focused. With few exceptions, [ . . . ] objectives are sweeping, unspecific, detached from actual performance, impossible to measure, and thus unlikely to succeed.”12 During the war years, the health sector was localized and fragmented. Emergency response interventions were often disjointed, short-term, and inefficient. Previously, the focus was on first-level health services and disease-specific programs—typical for humanitarian action. This has overshadowed attention to building sustainable basic health systems and infrastructure, strengthening human capacity, and actively mobilizing and engaging civil society groups for decision-making around their own health. The shortage of skilled human resources is a major limiting factor to providing substantive health care. Since the CPA, as the MOH has increased in capacity, the role of NGOs has transformed. The NGO Forum has organized technical working groups along the lines of the United Nations Cluster Groups. A Commission of the Ministry of Humanitarian 12 Suzanne Bonds Hinsz “Government of South Sudan, Strategic Capacity Building Study,” July 2010. Management Systems International: Juba, S Sudan. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 4 Affairs with the Ministry of Legal Affairs registers NGOS in Southern Sudan. It is estimated that there are more than 150 registered NGOs working in the health sector today. The GOSS MOH advocates integrating the existing vertical programs into both the resource pool and the management structures of the mainstream health system. The MOH Basic Package of Health Services (BPHS) Policy is based on five principles: right to health, equity, pro-poor, community ownership and good governance. The main criteria for the choice of priority services were ones that would have the greatest potential for impact on the health of the population, that would be equally accessible to the largest possible part of the population, and that would be affordable in the short-term and sustainable in the long term. The drug-supply management system is fragile, and is principally managed by NGOs with very limited training and capacity in this area. Other than private pharmacies and unauthorized market drug tables in urban areas, there is a near absence of formal private sector medical care. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 5 II. PROGRAM RESPONSE 2.1 USAID in Sudan The USAID/Sudan Strategy 2006–2008, (currently being updated) states: “The U.S. Government is the leading international donor to Sudan, providing more than $6 billion in assistance since 2005. It is focused on forging a definitive end to conflict, human rights abuses, and genocide in Darfur, and peaceful implementation of the 2005 Comprehensive Peace Agreement (CPA). [ . . . ] USAID-Sudan’s strategy focuses on supporting the implementation of and reducing threats to the CPA. The program aims to contribute to the goals articulated in the Strategic Framework for Africa of “averting and resolving conflict” and “promoting stability, recovery, and democratic reform.”13 USAID’s strategy includes two Strategic Objectives: Strategic Objective Number No. 9: Avert and Resolve Conflict Strategic Objective No. 10: Promote Stability, Recovery, and Democratic Reform USAID provided support in Southern Sudan well before the signing of the CPA. USAID worked with members of the nascent GOSS in Nairobi to identify the most critical investments and needs for the new government. USAID/Sudan’s current strategy focuses on supporting the CPA by assisting the GOSS to provide peace dividends and address the factors that fuel conflict, in support of Strategic Objective No. 10. Tangible peace dividends, like quality health services increase Southern Sudanese citizens’ confidence in their government’s ability to provide basic services. In addition, improving the overall health of the Sudanese people enables citizens to become more productive, allowing education levels to advance, promotes economic growth, and reinforces stability and peace. The USAID Sudan Office of Health supports a range of assistance programs with significant capacity￾building elements: 1. Tuberculosis (TB) Control Assistance Program (now called TBCARE), implemented by the KNVC PartnershipWorld Health Organization (WHO) Polio Program 2. WHO Integrated Disease Surveillance Network (at national, state, and county levels) 3. Sudan Health Transformation Project/Phase 2, implemented by MSH 4. Southern Sudan HIV/AIDS Project II, implemented by Family Health International 5. Neglected Tropical Diseases, implemented by Malaria Consortium 2.2. USAID Field Support Mechanism The evaluator was asked to review the appropriateness of the Field Support Mechanism in implementing the SPS Program. The Field Support Mechanism is a USAID internal contracting mechanism available to USAID/Washington technical offices to accept field support and/or cost-contributions from missions or other operating units to co-fund project activities. This mechanism is used by the USAID/Sudan mission to obtain the technical expertise of the MSH SPS Program, and other programs. Field support refers to field transfer of funds to USAID/Washington in which the central bureau undertakes the managerial lead (e.g., CTO designation), but works in close collaboration with the missions to ensure successful implementation. It provides missions with an easy way to access state-of-the-art services and to take advantage of technical oversight and leadership that central programs offer. It is especially useful to serve as a mechanism to respond to urgent and rapidly evolving needs that are often associated with international assistance programs. 13 USAID/Sudan. (2005) Strategy Statement 2006–2008. USAID Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 6 2.3. Rational Pharmaceutical Management (RPM)+ and Strengthening Pharmaceutical Systems—Globally The Cooperative Agreement for the implementation of the Rational Pharmaceutical Management (RPM) + Program was awarded to Management Sciences for Health (MSH), a U.S.-based private non-profit organization, in fiscal year 2000. RPM+ is a Washington, DC-based support program that has been implemented globally. The program was scheduled to run for five years (with an end date of September 2005). The ceiling for the CA was subsequently raised twice, and the program was extended to September 2007. MSH has been working for over 35 years to help governments improve their pharmaceutical management systems. The global Strengthening Pharmaceutical Systems (SPS) program, a five-year Leader with Associates Cooperative Agreement, was awarded to MSH in June 2007 as a follow-on to the RPM+; the agreement is scheduled to end September 2012. This $147.5 million Cooperative Agreement builds on the successes of RPM+ while expanding the range of technical areas to include financing, pharmacovigilance, pharmaceutical care, integration of new health technologies, and increased use of the private sector, among others. MSH collaborates with a number of partner organizations, which provide specialized resources for SPS. The mandate of the SPS is to build capacity within developing countries to effectively manage pharmaceutical systems and ultimately save lives by ensuring access to quality-assured substances. The SPS Global Program focuses on four key results: 1. Improved governance in the pharmaceutical sector 2. Strengthened pharmaceutical management systems to support public health services 3. Contained emergence and spread of antimicrobial resistance 4. Expanded access to and improved use of essential medicines In order to guide the selection of appropriate activities, USAID identified specific Intermediate Results (IRs) that correspond to each result area. SPS country programs receive a reference guide to ensure that planned activities support the USAID-identified IRs. 14 14 See section 3E for more detail. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 7 2.4 RPM+ and SPS in Southern Sudan The RPM+ Program support to Southern Sudan began in October 2005, in support of the then-Secretariat of Health (see Annex 3, Program Timeline). According to USAID officer Anthony Boni, the original idea of the program was to address the immediate needs of the then-Secretariat of Health of the SPLA to establish a malaria control program in Southern Sudan. In Southern Sudan, USAID requested that SPS’s predecessor, RPM+, provide technical support for a newly-formed government that, at that point, did not have the infrastructure, human resources, or systems in place to undertake malaria-related activities like developing policy and regulatory frameworks and organizational structures, drafting organizational guidelines, and conducting training activities. Mr. Boni explains the thinking at the time: “It was an incubator approach for organizational/systems/human resource development in a setting where the process is just starting or where specific technical activities or areas of work are in incipient stages. The malaria￾specific approach was understandably accompanied by technical concern with access and supply chain management issues.” Interview, October 15, 2010 Once the formal government was set up, the Ministry of Foreign Affairs approached USAID to determine if a mechanism existed whereby USAID could quickly provide skilled technical advisors to work within the MOH and establish a National Malaria Control Program. At the time, the MOH lacked building facilities, and senior officers literally met under trees. Management Sciences for Health, a private non-profit international organization, was implementing the USAID RPM+ with relevant experience and the capacity to provide technical assistance. The approach employed was to assist with physical infrastructure and the implementing of startup activities, such as developing the policy and regulatory framework and organizational structures, drafting operational guidelines, and conducting training activities. Both the RPM+ Program and the SPS program approach has been to work within the government system rather to than create separate offices, as other donors have done.15 In the early days, this co-location was accomplished by utilizing prefabricated office structure for both advisors and local government counterparts. Advisors and counterparts sit together and jointly plan and implement activities, so that an atmosphere of trust and Sudanese ownership can develop. Actual full-time support commenced in July 2006, when a technical advisor was recruited to work within the National Malaria Control Program of the GOSS MOH. Recruitment took place in Nairobi, and SPS Senior Malaria Advisor and Team Leader Dr. Robert Azairwe reports: “They (MOH) did not want advisors who would be telling them what should be done, but ones that would sit with their national focal group persons and help the national officers do what should be done.” A committee including USAID, the MOH, and MSH selected Robert as the first LTTA; malaria was the wedge issue, which became the focal point of a number of capacity building measures in the MOH. The MOH staff like to joke about the first LTTA, Dr. Robert Azairwe, using his laptop on his lap for lack of a table. Dr. Azairwe also slept in a tent for the first six months of the program. 15 Japan International Cooperation Agency (JICA) and WHO Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 8 Figure 2: MSH SPS Evans Sagwa and Robert Azairwe In 2007, there was an immediate transition16 from RPM+ to the SPS Program when MSH was awarded the new contract. The SPS overall mandate in Southern Sudan includes: 1) Strengthening the capacity of the malaria control program to effectively plan, coordinate, and document control and prevention interventions in line with the national Roll Back Malaria (RBM) strategic plan. 2) Strengthening pharmaceutical management systems to promote the effective management of medicines and supplies for malaria. Specifically:  Technical coordination and office support, and strengthening operational capacity of the malaria control and pharmaceutical management programs at central and state levels.  Support in policy development, planning, and coordination of malaria control activities at the central and state level; scaling up cost-effective malaria control interventions; and supporting malaria M&E systems at central and state levels.  Pharmaceutical management activities, including coordination support and policy development; support to procurement and distribution of anti-malarial medicines and supplies; capacity building for the public and private sector; and support supervision, inspection, and quality assurance. The Southern Sudan SPS Program uses a mentoring approach, with SPS Technical Advisors placed in the relevant departments of MOH to work on a full-time basis with their national counterparts. Southern Sudan SPS annual work plan activities are designed to be in line with both SPS IRs and MOH strategic plans and targets. Implementation of SPS activities is undertaken with and through the relevant MOH institutions at central, state, and county levels. The systems strengthening approach of SPS aims to empower MOH institutions at central, state and local levels to effectively coordinate and manage the malaria, pharmaceutical, and EPI programs. SPS technical support initially focused on development of key program policies and strategies at the central MOH level, but has progressively included support to implementation in selected states. The National Malaria Control Program (NMCP) is responsible for implementing the five-year National Malaria Strategic Plan (NMSP). The NMSP goal is to reduce malaria deaths by 50 percent, focusing on providing prevention and treatment services that target high–risk groups, such as children under five, pregnant women, and internally displaced persons. The MOH strategy is to cover the whole population of 16 In some countries, RPM Plus and SPS programs were run concurrently until the close of RPM Plus in September 2008. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 9 Southern Sudan with Long Lasting Insecticide Treated Nets (LLITNs) in the next five years with funding from the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM). USAID was able to finance the renovation of a building to establish offices for the NMCP and a board room that is available to all MOH programs and office. This became the first shared meeting space and the center of MOH activity. According to Dr. Samson Baba, Director of External Cooperation, Ministry of Health, “Robert has transformed the National Malaria Control Program . . . and has harnessed resources from all the donors.” Since 2007, the malaria response has been a coordinated effort, producing a joint annual plan. The SPS Program has grown incrementally as needs were identified. The program timeline in Annex 3 highlights key events. Additional advisors and staff were added as follows: 2.4.1 SPS Technical Objectives and Rationale Malaria Technical Objective 1: Strengthen the capacity of the malaria control program to effectively plan, coordinate, and document malaria control and prevention interventions in line with the national “Roll Back Malaria” Strategic Plan. The key focus for malaria control support is to improve the planning and coordination of activities at the state level. Technical Objective 2: Strengthen pharmaceutical management systems to promote the effective management of medicines for malaria and related commodities SPS malaria program activities support the MOH in:  Strengthening operational capacity of the malaria control and pharmaceutical management programs at central and state levels  Strengthening planning and coordination of malaria control activities at central and state levels  Scaling up implementation of effective malaria interventions, such as insecticide treated nets and artemisinin-based combination therapies (ACTs)  Strengthening malaria monitoring and evaluation systems at national and state levels  Developing and implementing standard operating procedures for pharmaceutical management  Developing and implementing initiatives to capacitate and license private pharmaceutical facilities for the improved provision of pharmaceutical services  Improving quantification, procurement, and distribution of essential medicines  Strengthening mechanisms for national medicine use at health facilities through establishment of Drug and Therapeutics Committees. Pharmaceutical Management SPS continues to support the Directorate of Pharmaceutical Services in the implementation of the Pharmaceutical Master Plan. The major activities for implementing the above objectives are: 1. Coordinating and monitoring technical activity monitoring Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 10 2. Strengthening office management and operational capacity 3. Supporting the MOH in strengthening the planning and coordination of malaria control activities at the central and state level 4. Reviewing and strengthening implementation of the ACT based malaria treatment policy 5. Supporting the MOH in strengthening malaria M&E systems at the central and state level In the pharmaceutical sector, long procurement processes and inaccurate quantification of requirements leads to stock outs. Distribution mechanisms and storage facilities are inadequate, and regulation and quality assurance mechanisms are still weak. SPS endeavors to strengthen the quality assurance systems for medicines. SPS also supports states in strengthening pharmaceutical management systems. Additional activities include: 1. Supporting the MOH to develop a five-year pharmaceutical sector strategic plan for Southern Sudan 2. Supporting the procurement and management of ACTs and Rapid Diagnostic Tests (RDT) procured through U.S. government funds 3. Supporting drug registration, inspection and quality control testing activities 4. Developing private sector accreditation scheme to strengthen quality, access, and use of medicines in the private sector 5. Strengthening pharmaceutical management capacity at selected public and private sector sites Expanded Program on Immunization (EPI) SPS supported the MOH in development of the 2010 Plan of Action for Immunization Systems Strengthening in Southern Sudan, with a view to coordinated actions of all stakeholders for better coverage. In articulating the service delivery activities, the plan builds on the nationally adapted Reaching Every County (REC) strategy for immunization planning and implementation. The goal of the EPI is to build a national immunization system that will significantly contribute to the reduction of ill health, death, and disability from vaccine-preventable diseases as an integral part of the health sector strategic plan for Southern Sudan. EPI has the following seven objectives: 1. To ensure availability of potent, safe, and effective vaccines at all levels of the healthcare system 2. To build and sustain a cold chain system suitable for Southern Sudan 3. To improve and expand service delivery points for immunization in order to bring services to the communities, with the goal of making immunization services accessible to the entire population of Southern Sudan 4. To increase community demand for immunization services by creating household awareness on the benefits of immunization 5. To build onto the available Acute Flacid Paralysis/Polio surveillance structure a comprehensive vaccine-preventable diseases surveillance system that enables the program to monitor program performance 6. To strengthen the management capacity of the EPI in Southern Sudan 7. To initiate and implement relevant activities that will lead to the achievement of globally set goals of Polio Eradication accelerated measles control and Maternal/Neonatal tetanus elimination. 2.4.2 Where Does SPS Work? Some of the SPS activities have covered all the states: support supervision to the state level, facilitation of drug distribution from the center to all state capitals, coordination of the Malaria Indicator Survey, and EPI vaccination week in all the states. Pharmaceutical trainings have also used a regional Training of Trainers (TOT) approach with participants drawn from all the ten states. However, implementation has been focused on support in Central Equatoria, Eastern Equatoria, and Jonglei states. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 11 2.5 Other Capacity Building Programs in Health Donors and the international community have shown commitment to the reconstruction of Southern Sudan. The United Nations Development Programme (UNDP) has sent 200 operational civil servants to work within the GOSS. In anticipation of the massive development efforts needed to get Southern Sudan up and running, donors committed to the establishment of the Multi Donor Trust Fund (MDTF) administered by the World Bank. In this arrangement, the GOSS is expected to match every donor dollar with two dollars, a ratio of 1:2. However, since 2005, the implementation modalities and conditionality of the bank have impeded the disbursement of funds for development activities. So far, limited funds have been released for reconstruction. Beyond the MDTF/World Bank, other donors involved are the UNDP, UNFPA, United Nations High Commissioner for Refugees (UNHCR), United Nations Office for the Coordination of Humanitarian Affiars (UNOCHA,) UNICEF, WHO, World Food Programme (WFP), USAID, Japanese International Cooperation Agency (JICA), GFATM, UROHEALTH, and Italian Cooperation. JICA, through the Project on Human Resources Development for Health in Southern Sudan, focuses on strengthening the human resources department of the Southern Sudan Ministry of Health and the ten state ministries of health to work toward better implementation of various programs on human resource development in health care services; developing a Human Resources Information System (HRIS); and strengthening the partnership between involved development partners. Over the last three years, MSH SPS has supported the MOH to improve coordination among malaria partners at the central level. SPS is an active member of the Malaria Technical Working Group (TWG) and participates in other key coordination meetings such as the Country Coordination Mechanism (CCM) for Global Fund and the NGO health forum. To enhance sharing of experiences, SPS supports the NMCP to publish a malaria newsletter on an annual basis. SPS intends to continue to support the above-mentioned activities and support MOH to improve planning and coordination of malaria activities at the state and county levels. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 12 Strengthening Monitoring and Evaluation Capacity within the MOH In September of 2008, the MOH recruited and hired SPS technical advisor Dr. Betty Margaret, a Sudanese physician with extensive field experience with Médecins Sans Frontières (MSF) in internally-displaced persons camps and at a state hospital. She has received advanced training in Monitoring and Evaluation and now supports the Malaria Directorate M&E activities, data collection, compilation, and analysis. The team observed examples of her work in Torit Hospital. The staff there had undertaken training in record-keeping and monitoring. The data was regularly fed to the central MOH which could, for the first time, produce charts, graphs and tables for the hospital to assist them in spotting trends and highlighting shortfalls. The charts were prominently displayed, and staff demonstrated their use in decision- making. The Malaria program has nominated a person to work with Dr. Margaret for capacity building of MOH staff. She has been instrumental in the Malaria Indicator Survey, in conjunction with the Malaria Working Group and Center for Statistics, where she is the Principal Investigator. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 13 III. EVALUATION METHODOLOGY 3.1 Purpose of the Evaluation The SPS midterm evaluation Scope of Work (SOW) states: The purpose of the evaluation is to assess the effectiveness of the technical assistance approach to GOSS capacity building in the areas of pharmaceuticals management, malaria control, and Expanded Program on Immunization (EPI) used by the Strengthening Pharmaceutical Systems (SPS) Program in Southern Sudan. The secondary purpose of the evaluation is to advise USAID about the appropriateness of this capacity building approach in a continuing or follow-on program, taking into account recent USAID research on capacity building in Southern Sudan, a conflict-affected context. To address the questions above, USAID/Sudan commissioned a ‘Lesson-Learning Study’ of the SPS Program in Southern Sudan through the SUPPORT Program, implemented by MSI. This formative evaluation was collaborative and included active participation by the USAID/Sudan Health Office, the MOH and its counterparts, and the implementing partner (MSH), led by an independent external evaluator who bears sole responsibility for conclusions drawn and the recommendations made and reported on in this document. This evaluation was undertaken in September 2010 in Southern Sudan, centered in Juba with a field site visit to Eastern Equatoria State (Torit) to visit the State Ministry of Health (SMOH) and the state hospital that have been the site of a number of trainings and other program activities. The team included Dr. Martin Swaka, Activity Manager of the SPS program for USAID Sudan, Evan Swaka, Deputy Director of the MSH SPS Program in Namibia, and Team Leader Ann von Briesen Lewis, an experienced evaluator of USAID health programs and former embedded USAID long-term advisor in a government ministry. The team was joined by Dr. Bortel Ohisa, Deputy Director, Quality Assurance, MOH. More complete team biographical information is available in Annex 3. 3.2 Evaluation Methodology The SOW (see Annex 1) includes the following key questions to be addressed in the evaluation: Project Design 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered or addressed based on experience to date? 2. What are the pros and cons of the field support mechanism in the context of Southern Sudan and the SPS Program? 3. How does the SPS capacity building approach align with best practices in capacity building in conflict￾affected contexts? 4. How is the role of the advisers designed, given the need to balance urgent operational needs of the ministry with mentoring, training, and technical work? Project Impact and Sustainability 5. How satisfied are USAID and the Ministry of Health (at the national level and the three targeted state level ministries) with the project up to this point? 6. What is the project’s perceived value to other Ministry of Health partners (e.g. the Global Fund)? 7. Are the current benefits likely to be sustained after the program—why and why not? 8. In what ways has the input of the TA resulted in positive changes in the health sector for beneficiaries? Effectiveness 9. How effective has the SPS Sudan program been in addressing capacity within the Ministry for program planning, coordination, and technical knowledge in the areas of malaria control, EPI and pharmaceutical management? 10. Specifically, what policies, guidelines, and tools developed by SPS in the areas of the National Malaria Control Program, EPI and pharmaceutical management were adopted at the national and state levels? Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 14 11. How well has the program been able to meet indicators and contractual targets? Efficiency 12. How efficient is the model in maximizing time and effort to build capacity of the Ministry of Health to respond effectively to the needs of the National Malaria Control Program, EPI, and Pharmacy Management? Please describe techniques utilized by the model. 13. Which aspects of program management demonstrate or fail to demonstrate efficient management, considering financial implications? Cross-Cutting Issues 14. How well has the program mainstreamed gender and other equity issues? Please consider both direct and indirect beneficiaries. The team was challenged to develop an evaluation methodology to address the dual focuses of the program: efficacy of the model of embedded long-term advisors in the MOH, and capacity building and technical skills transfer (see Annex 4 for an expanded discussion of the methodology). The team obtained necessary information using a mix of data collection methods and by developing several tools. Primary data collection methods included semi-structured and unstructured interviews, surveys, group interviews, observation, collection of case studies, and site visits. Secondary data collection methods encompassed document review (including PowerPoint presentations), internet research, and database mining (including the quarterly and annual reports developed by the SPS team and submitted to USAID). (See Annex 6.) Following a stakeholder mapping exercise to identify key participants (see visual below), the team developed sample questions on three tiers to serve as interview guides to elicit answers to the evaluation questions. Illustrative questions are attached as Annex 4. 3.3 Stakeholder Mapping Center: Program Specific Questions, key participant interviews: TA advisors, USAID, MSH program backstops, counterparts Middle: Partners: Mid-level MOH officials, Malaria, Pharmaceutical Management and EPI colleagues, other GOSS officials, trainees Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 15 Outer: other donors, other LTTA programs, other capacity building programs. Senior GOSS officials, State level senior officials, pharmacists, clinicians, patients, communities, and families. The team developed a list of stakeholders to be interviewed, including GOSS officials at the central and state levels, USAID officers, MOH colleagues and supervisors, other donors with long-term advisors placed in ministries, cooperating NGOs, trainees, and the general public, especially health care seekers. Over 40 people were interviewed over the course of two weeks. (See Annex 5: List of Persons Interviewed). Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 16 IV. FINDINGS AND CONCLUSIONS This section outlines key findings and conclusions of the program evaluation. In sum, the team found the field support model and technical approach used by the program in pharmaceutical management, malaria control, and the expanded program on immunization to be very effective. The Program has positively impacted the control of malaria, the development of pharmaceutical management, and the strengthening of the Expanded Program on Immunization in Southern Sudan. The codes in parenthesis below correspond to the table in Annex 7: Findings, Conclusions, and Recommendations. 4.1 Program Design 1. Are there any issues with respect to program design and assumptions (documented or implied) that should be reconsidered or addressed based on experience to date? This evaluation found the SPS field support model to be very powerful, providing the bureaucratic flexibility to respond to the changing and actual needs of the MOH. Embedding technical advisors within MOH structures allowed advisors to be where the action is and key decisions are made, and working relations with the MOH have been smooth. Thus, expansion of the program has been responsive to the priority needs and requests of the MOH, with program planning taking form of collaborative Annual Work Plans development. The deployment and work of the LTTA at the MOH is described by all stakeholders as successful. The bureaucratic flexibility built-in to the SPS Global program allows for incremental growth of the program, minimizing USAID mission managerial responsibilities while responding to GOSS needs as they arise. Program managers report satisfaction in the level of MSH home office support, and MOH officials appreciate the ad hoc responsiveness of the model. (F2) The training component of the design is formulated annually during the development of the work plans and is responsive to training needs and requests. Training is short-term, in-service and program-specific, and conducted primarily by the LTTAs.17 (F3) Most SPS training is targeted to in-service training of mid- and senior-level personnel, where women are underrepresented. Special efforts must be undertaken to assure women have equal access to the benefits of working with the LTTAs and to training. (C13) The SPS mentoring model is well-suited to conflict-affected Southern Sudan, with its very limited human resources and low institutional capacity in health services. (C14) Longevity and duration: the SPS program offered LTA multi-year contracts and determined to recruit long￾term advisors from the regions who could commit to staying for a long time. All the LTTAs hired have stayed on; none has left early or departed at the end of a contract period. All respondents cited personality as the key element to success, above professional competence. (F4) Counterparts: the GOSS MOH is involved in developing the SOW and in the recruitment and selection of LTTAs. MOH Officials participate in the MSH/SPS annual performance review. There is no memorandum of understanding (MOU) or written commitment on the part of GOSS for the assignment of appropriate counterparts and definition of roles and responsibilities. Evaluation surveys indicate a divergence of role expectation and on-the-job requirements of LTTAs to be both active implementers and mentors and teachers. (F5) 17 Further data regarding meeting training targets is provided in section 4.3 “The SPS team built a literal space—the board room at the ministry—and an official ‘space’ to allow for coordination and planning to take place at the ministry” Dr. Samson Baba, MOH Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 17 Central level, expansion plan: the design decision to place advisors at the central level in the early stages to assist with policy development is credited with the development of 14 major policies, building the framework for later, iterative expansion to the states and counties. While some program elements are national in scope, PM and training activities have expanded to three states. (F6)  The program design and underlying (implicit) assumptions were well-suited for the capacity building needs of fragile, conflict-affected Southern Sudan over the life of the program. (C1)  The MSH SPS program takes different shapes in different countries (such as Kenya and Namibia) and has proven sufficiently nimble in responding to varied needs for TA and additional tools and support. With reported (and evident) excellent working relationships and high levels of mutual trust, annual work plans, targets, and priorities are negotiated with full participation of the GOSS MOH, allowing for the local ownership and responsibility crucial to success, consistent with the key elements of the Global Health Initiative. (C2)  Based on experience to-date, a long-term training plan could improve the quality and efficiency of training. Educational objectives, regular curricular updates, training of trainers (adult learning), remedial and refresher training, and incentives for career advancement could elevate the quality of training. The end point of training should be improvement in quality of patient care. Thus it is advisable to consider that training requires follow-up and supportive supervision, so as to change behavior. (C3)  In an environment of frequent personnel changes both within the GOSS and the USAID mission, the longevity and perseverance of the SPS advisors have won them the respect and trust of colleagues and supervisors. (C4) SUCCESS STORY: Increasing availability and access to essential medicines: the case of first-line antimalarials In South Sudan, malaria is the leading overall cause of morbidity and fourth leading cause of mortality in children under five years. In 2007, SPS staff uncovered a serious stock-out situation of Artesunate and Amodiaquine (AS/AQ) in the country; these first-line malaria medicines were not being included in medicine kits sent to health centers. However, there were no funds within the MOH for emergency procurement, nor were there funds for rapid distribution to service delivery points once emergency supplies of AS/AQ finally became available. SPS response SPS identified the problem through supportive supervision and highlighted it to MOH colleagues and partners, including USAID. However, most donors, including USAID, could not immediately allocate funds to plug the gap. SPS lobbied the World Bank to make the much-needed funds available for an emergency procurement through the Multi-Donor Trust Fund, resulting in about $500,000 being made available. SPS worked with the MOH procurement department and pharmaceutical directorate to ensure that roughly 554,000 treatments of AS/AQ were procured in mid 2008. Still faced with the transportation challenge, SPS staff successfully lobbied UNOCHA EP&R (Emergency Preparedness & Response) program to airlift the AS/AQ to state capitals and actively coordinated NGOs to work with state ministries of health to help deliver these products to health facilities where patients need them urgently. Additionally, SPS provided logistics support for the distribution exercise and funded direct deliveries to states accessible by road. Results Artesunate and Amodiaquine were made available to the public at the beginning of the rainy season, which corresponds with the beginning of a surge in malaria cases in Southern Sudan. USAID has since committed funds annually to ensure no gaps in supply of antimalarials occur. In 2009, USAID donated about 1.6 million treatments of AS/AQ, while in 2010, an additional 1.1 million treatments were provided. MOH has integrated AS/AQ into the new revised kits to ensure at least all health facilities are provided with some AS/AQ through the kits, bringing South Sudan in line with accepted protocols. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 18  Both MOH officials and the LTTAs would benefit from increased clarity and specificity of roles and responsibilities between and among counterparts and advisors. Regularly reviewed and explicit expectations would ease the occasional tension around the ‘advisors as trainers’ or ‘advisors as operational partners’ balance. (C5)  In the absence of central-level policies, the advisors assisted with the development of policies and procedures first to address the malaria crisis, leading naturally to pharmaceutical management, and second to addressing EPI. The focus at the central level was appropriate for the early years of the program, and the MOH now has the structures in place for expansion to state- and, eventually, county￾level capacity building. These efforts have already begun. (C6)  Based on respondents’ judgments, positive aspects outweigh negative. Planning on an annual basis is not ideal, but works in the context of a rapidly maturing organization’s county-level capacity. (C7)  The SPS approach in Sudan, while slightly different from SPS TA placement elsewhere (see Annex L: Namibia Program Description), is appropriate for the Southern Sudanese context and is aligned with current best practices in conflict-affected areas (see SPS Afghanistan). (C8)  The design of the roles of the LTTAs, based on the job descriptions, emphasizes operational expertise and filling vital roles in crucial technical areas where the MOH lacks sufficient trained personnel. The mentoring role is secondary in the design. (C9) 2. What are the pros and cons of the field support mechanism in the context of Southern Sudan and the SPS Program? USAID Washington provides central funding and oversight through the COTR Field mission and directly supervises the project through the Mission Team leader and Technical Activity Manager. An SPS country leader oversees implementation of activities, with program oversight from SPS/headquarters. Respondents report pros: the mechanism is flexible, can build gradually and incrementally, is responsive, and places minimal managerial demands on small USAID/Sudan staff. There is no lengthy process of procurement; money is remitted directly to the implementing organization (MSH) in Washington, DC. Field support mechanism is not dependent on the local mission. Cons: without a long-term Sudan strategy, and definition of capacity, SPS is open-ended and elastic depending on availability of funding. Difficult for partners to plan on future levels and programs. (F7) Managerial burden: There is substantial involvement of USAID key staff; Budget and Work plan approved by COTR; financial management by COTR/ Washington i.e. accrual reports and pipelines; monitoring of implementation; helping in resolving of challenges with MOH and other partners; and following up on waivers e.g. construction; tax exemption. 3. How does the SPS capacity building approach align with best practices in capacity building in conflict-affected contexts? Literature review of current best practices in capacity building confirm the placement of LTA embedded within government offices, sharing office space, combined with a variety of mentoring and training opportunities and approaches, including individual, pairing, small group, and just-in-time training have shown greater impact than standard approaches. Other donors have LTA in the MOH with less satisfaction and “success.” (F8) The capacity building of the SPS Program was found to be very significant; it set a platform for the future scaling up of interventions, regardless of funding source. Confirming the key assumption of the design, respondents (GOSS, TA, colleagues, peers) overwhelmingly report that the SPS advisors have played a “crucial’ and “essential” role in building the MOH over the past five years. (F1) Counterparts, colleagues, and advisors in the SPS, NMCP, PM, and EPI have remained in place. This finding is particularly noteworthy in the MOH context, where overall human capacity is low, with few skilled health Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 19 Figure 3: Sample materials produced with assistance from SPS. Photo by AvBLewis. professionals, many vacant positions, and a high turnover of staff in general. Research also showed partner integration to be high; not only have partners bought into MOH plans and policies, but some have also come to seek strategic direction from the MOH. The SPS program has also been found to be significant in the eyes of other donors, NGO partners, and MOH officers, which have adopted many of the tools developed by the MOH and the team and report increased capacity of individuals, work units, and programs. The SPS Program has effectively implemented capacity building by working through MOH structures. However, the roles and responsibilities between advisors and MOH counterparts are ambiguous. Advisors view themselves as primarily operational and seek stronger, more structured counterpart and mentoring relationships. Nonetheless, MOH departments have been effectively brought into control of the programs, and guidelines, policies, strategies, and implementation tools have been effectively put into place. The evaluation also found the TAs’ ‘leading from the rear’ approach to be well-executed, including its stepping back as appropriate to ensure MOH ownership (such as in presentations). For example, in branding, there was a deliberate attempt to establish the MOH’s lead while working within USAID guidelines. The SPS program’s work to build MOH technical capacity was also found to be high-impact: in training key staff in focal areas; in giving technical support and advice at the ministry level; in providing malaria technical advice as well as pharmaceutical technical advice to the NMCP; and in developing pharmacy management, especially in drug regulation and supply chain management. MSH has developed the conceptual framework illustrated in the graphic below to inform the capacity building aspects of the SPS program. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 20 Figure 4: Conceptual Framework for Building Capacity of Pharmaceutical Services. 18 4. How is the role of the advisers designed, given the need to balance urgent operational needs of the Ministry with mentoring, training and technical work? Reviews of the job descriptions of the five advisors, interviews, and a simple exercise of grading job responsibilities on a scale revealed a focus on technical skills and outcome measures in the job description and the personnel evaluation. The LTTAs described their perceptions of their jobs as averaging 60 percent operational and 40 percent capacity building. LTTAs report they occasionally face interpersonal challenges and dilemmas—for instance, whether to report passive resistance and lack of cooperation to the counterpart’s supervisor. (F9) 4.2 Program Impact and Sustainability 5. How satisfied is USAID and the Ministry of Health (at the national level and the three targeted state level ministries) with the project up to this point? Key informant interviews confirmed that both individually and together these highly skilled and accomplished advisors have met and exceeded the expectations of the funders and counterparts and are viewed as leaders and models in the donor, NGO and government communities. Interviews with USAID, the MOH at the central level, and two state MOHs provided strong testaments to the overwhelmingly positive impact this project has had within and beyond the MOH. Unlike other organizations that set up systems parallel to the GOSS/ MOH, duplicating efforts and sidelining the MOH staff, the SPS advisors work with the MOH as a team and are very well-integrated within the MOH. This success contributed to USAID award of the SHTP II contract to MSH to capture the synergies of sharing support, resource pools, and cooperation. (F10) 6. What is the program’s perceived value to other Ministry of Health partners (e.g. the Global Fund)? The evaluation team interviewed other MOH partners, including UNDP, WHO, WB MDTF, UNICEF, JICA, PSI, DFID, and the Malaria Consortium. Without exception, these partners reported the high value added by the SPS advisors, pointing to their “lead from the rear” approach and supportive manner with 18 From Management Sciences for Health 2007, Strengthening Pharmaceutical Systems (SPS) RFA No.: USAID M/OAA/GH/HSR-07-082, Figure 1.3, p. 27. Graphic adapted from Potter C., Brough R. 2004. Systematic capacity building: hierarchy of needs. Health Policy and Planning;19 (5):336-345. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 21 GOSS officials. They play important roles in coordination by supporting the establishment of malaria and pharmaceutical TWGs to serve as foci for discussion of issues and challenges. (F11) Other partners have adopted material and tools developed by and for SPS. UNICEF, PSF, PSI, and UNDP had funds for pharmaceutical training, but lacked materials, so used the materials developed by SPS to facilitate trainings in Malakal and Rumbek. (F12) 7. Are the current benefits likely to be sustained after the Program? Why and why not? During the four years the program has been active, MOH capacity has been increased by many measures: national strategies and policies are in place, regulatory legislation and increased professionalism have raised standards in pharmaceutical management; and the MOH has attracted other partner and donors, creating a stronger human resource base. The MOH has more trained and qualified staff, the organization is rationalized and more efficient formal and informal training and skills-transfer have taken place. (F13)  The SPS Team Leader held a key role in assisting the MOH in drafting the funding proposal for the GFMAT Round 7, resulting in an award of $73 million to the MOH. Additionally, the proposal for Round 10 is presently under review and is likely to be funded. The MOH openly credited the SPS Team Leader for leveraging these funds for the MOH. As an indicator of quality and acceptance, other donors and partners have adopted SPS materials for their own programs and training. (C10)  Key benefits of the program are likely to endure. (1) a stronger MOH working within a rational organizational structure, and (2) implementing programs under internationally accepted standards and policies. The initial focus on the central level and policy level has built a strong framework for program expansion to state and county levels. The benefits of training and human resource development remains immeasurable. (C11) 8. In what ways has the input of the TA resulted in positive changes in the health sector for beneficiaries? While direct attribution is challenging for evaluators of capacity-building programs, this report identifies improvements in 1) MOH leadership, as measured by quantity and quality of policies developed and promulgated, all directed at improving health service delivery to beneficiaries, 2) improved access to appropriate and safe pharmaceuticals through improved physical infrastructure, improved job performance by trained MOH personal, down to the dispensary level, 3) greatly expanded malaria treatment and prevention programs to the local beneficiary level, and 4) improved planning and coverage of immunizations through the EPI training programs. In the course of the evaluation, this question was asked directly to beneficiaries and officials (the response of Dr. Augustine Okwayi, Senior Medical Officer, Torit State Hospital is quoted in full on pg. 22 of the report). How has this program helped the average Sudanese citizen? Dr. Augustine Okwayi, Senior Medical Officer, Torit State Hospital, considered this question when a grandmother arrived at the hospital with a child who was near death with malaria. She had taken the child to traditional healers, but then, in desperation, decided to try the hospital for the first time. Although she had lost hope, at the hospital the child was treated successfully with modern antimalarial medicines, now regularly available. The grandmother returned to thank Dr. Okwayi, saying, ‘Doctor, we have killed many of our children because we didn’t know that modern antimalarial medicines are more effective and better than traditional remedies’ [paraphrased]. Torit, Sept 24, 2010 Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 22 4.3 Program Effectiveness 9. How effective has the SPS Sudan program been in addressing capacity within the Ministry for program planning, coordination, and technical knowledge in the areas of malaria control, EPI and pharmaceutical management? Despite the very limited size and scope of the SPS Sudan project, it has demonstrated significant impact in improving MOH capacity. The “wedge” malaria control program has been most effective and has played a key role in building systems of the MCP. For example:  Development of the Malaria newsletter,  Setting-up the malaria building and making available a multi-purpose board room that is shared by MOH programs,  Documenting malaria program activities,  Extensive training and supervision in: monitoring and evaluation, data and information management systems, programming,  Coordinating annual commemoration of the World Malaria day celebrations, and  Mentoring the Malaria Program Manager.  Accomplishments of the Pharmaceutical Management System and EPI are impressive, and documented elsewhere. (F14) Cross-Cutting Issues: Public-Private Partnerships Evaluation research found that the SPS Program has provided effective and crucial support to drug registration, inspection, and quality control testing activities. SPS has supported the MOH to develop guidelines for registration and licensing of pharmaceutical businesses and the corresponding tools for inspection of premises. A team of pharmaceutical inspectors in Central Equatoria has been trained to conduct inspection of premises. Inspection reports have been also sent to the MOH and included in the monitoring and evaluation database. Finally, SPS is also supporting the MOH to strengthen the product quality assurance system through the installation of a minilab facility at one port of entry (Kaya). MSH oversight and support was described to the evaluation team to be timely and professional, relieving the USAID Mission of managerial burdens. SPS provided physical as well as organizational space in the immediate post-conflict environment; office space and the new boardroom were crucial to MOH organization and development. Evaluation research also found the overall team (MOH, USAID, MSH, SPS LTAs) to be strong, cohesive, and characterized by regular communication and cooperation. The Team Leader was found to play a pivotal role as supervisor as well as senior technical advisor. Key factors in the success of the SPS LTA were found to be, in order of priority: personality, flexibility, sensitivity, management style, technical know-how, consistency, and longevity. 10. Specifically, what policies, guidelines and tools developed by SPS in the areas of the National Malaria Control Program, EPI and Pharmaceutical management were adopted at the National and state levels? Products, guidelines, and tools were developed by the program in close cooperation with the MOH, promoting buy-in and country ownership. An indicator of quality, these materials have been widely adopted and adapted by other NGOs in the health sector, and form a common platform for quality assurance in health services. (C12) A listing of the products, guidelines, and tools developed by the program and adopted at the state and national levels is provided elsewhere in the report. (Annex D) (F15) ‘‘Robert has transformed the National Malarial Program by harnessing the resources of all the donors. There is now a joint annual plan and effective coordination under the GOSS MOH.’’ Dr. Martin Swaka, USAID Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 23 11. How well has the program been able to meet indicators and contractual targets? According to quarterly reports, SPS has consistently met or exceeded the targets for the two key indicators: number of people trained and number of policies developed. When broken down by gender, SPS has fallen short of the targeted 28 percent female trainees, but it has exceeded the overall training number by almost 10 percent. (F16) USAID Indicators and Targets This section presents research findings that inform the evaluation of how the SPS program’s field support model is reflected in USAID indicators. In the SPS program, two standard USAID indicators are tracked: 1. The number of policies drafted with USG support 2. The number of people trained in malaria treatment or prevention with U.S. government funds For more specificity, reports against these indicators are separated by the three interrelated programs (malaria, pharmaceuticals management, and EPI) and by fiscal year, and trainees are disaggregated by gender. Annual targets are set by the USAID mission in collaboration with SPS. The evaluation found that SPS has consistently met or exceeded these targets, as detailed in the tables below. After FY07, gender disaggregated targets and totals were reported. With the minor exception of trainees in FY06, the overall training goals were met or exceeded for male trainees. However, the female trainee number fell consistently short of the lower targets: approximately 30 percent of the total trainees were female, falling short of the target of 37 percent. This shortfall is indicative of the gender imbalance among skilled health workers in Southern Sudan. The great majority of the trainees are in-service GOSS MOH employees, the majority of whom are also male. Policy drafting Number of malaria control and prevention policies drafted with U.S. government support US Fiscal Year Target Actual (FY05—RPM+) 0 1 FY06 2 3 FY07 1 3 FY08 2 7 FY09 1 1 Number of pharmaceutical management policies drafted with U.S. government support US Fiscal Year Target Actual FY07 1 3 FY08 1 3 FY09 1 1 Number of EPI policies drafted with U.S. government support US Fiscal Year Target Actual FY09 none 3 Training Number of people trained in pharmaceutical management US Fiscal Year Target Actual Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 24 FY07 50 (30 m, 20 f) 118 (92 m, 26 f) FY08 150 (100 m, 50 f) 166 (122 m, 44 f) FY09 100 (60 m, 40 f) 108 (95 m, 13 f) Number of people trained in EPI US Fiscal Year Target Actual FY09 50 (30m, 20f) 59 (52 m, 7 f) TOTAL number of trainees Target Actual 1050 (after FY08, 500 m, 300 f) 1283 (after FY06, 844m, 254 f) The proportion of male and female trainees is illustrated in this pie chart: of the total 1098 trainees, 23 percent were female, and 86 percent were male. Further charts that summarize the findings of the tables above can be found in Annex 10. In addition, the SPS project reports on the following indicators: Pharmaceutical Management Indicators  Availability of medicines  Number of distribution cycles per year  Number of inspection activities conducted per year EPI program performance indicators:  Coverage of the third dose of DPT (DPT3)  DPT1–to- DPT3 drop-out rate The evaluation research team found that the SPS program took effective measures to control malaria, to develop Southern Sudan’s pharmaceutical management, and strengthen the Expanded Program on Immunization. Specifically: Malaria  The NMCT was developed and implemented. Whereas bed net coverage was only 15 percent in 2006, it now stands at 53 percent, based on the recent MOH malaria indicator survey report.  The program led and coordinated successful proposal development for the Global Fund (Round 7; $73 million); this is pending for Round 10.  Private sector practitioners were engaged and trained in order to bring them under policy guidance.  Policy coherence and consolidation was improved and changed, especially with regards to malaria. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 25 Pharmaceutical management  Standard Operating Procedures (SOPs), tools, and guidelines for inventory management and warehousing were developed and implemented.  A draft law on pharmaceutical regulation has been tabled by the Parliament for ratification.  A draft law on the regulation of pharmaceutical personnel was created.  A plan was created for the distribution of anti-malarial pharmaceuticals and other essential medicines.  The program developed specification and selection criteria for a more effective rapid diagnosis test (and presented options to the MOH for the MOH to select).  The program designed organizational charts and job descriptions for the directorate of Pharmaceutical Services: these were approved by the ministry board and adopted by the public service to form the basis for recruitment, hiring, and deployment of staff.  The licensing of private sector pharmacy operators was developed.  The MOH now inspects private sector pharmacies using standards and checklist and is able to compile reports using common templates designed by SPS. EPI  The LTTA arrived during polio outbreak and helped the MOH establish routine immunization programs.  The program is strengthening data systems for monitoring and checking coverage at state, county, and village levels. Micro-planning is being implemented at the household level.  Mapping, training, and exercises have been carried out in three states.  The annual progress reporting to the Global Alliance for Vaccines and Immunisation (GAVI) has been improved.  Vaccination guidelines have been drafted. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 26 Figure 5: EPI Mapping Exercise. Photo by AvBLewis 4.4 Program Efficiency 12. How efficient is the model in maximizing time and effort to build capacity of the Ministry of Health to respond effectively to the needs of the National Malaria Control Program, EPI and Pharmacy Management? Please describe techniques utilized by the model. 13. Which aspects of program management demonstrate or fail to demonstrate efficient management, considering financial implications? With the modest investment of salary and support for a small team of expert technical advisors and their support office, SPS has produced over 200 major policies, guidelines, curricula, and reports. The technical advisors serving on multi-year contracts are regional experts who work embedded within the MOH, sharing offices with counterparts. They are seen by respondents as key players in the MOH and draw respect for their productivity, technical know-how, and personal leadership skills. (F17) According to the evaluation surveys, the USAID/Sudan mission exercises a high degree of involvement and participation, while the Leader with Associates format of the cooperative agreement with MSH relieves the mission of some time-consuming oversight and managerial duties. MSH Headquarters provides “appropriate” supervision and support, and the team leader serves both as senior technical advisor and team leader with significant and growing coordination and supervisory responsibilities. (F18) The decision to combine some administrative functions with the MSH SHTP II program has realized efficiencies by avoiding duplication in several support functions. (F19) The financial arrangement of the mission buy-in to the global SPS, compared to alternatives, is beyond the scope of this evaluation. The administrative costs of the additional layer of MSH oversight are balanced by the breadth and depth of expertise available through SPS and the global sharing of materials, information, and approaches. The tie-in to regional and global networks keeps Southern Sudan linked to current practices and standards and visible participants in the malaria, PM, and EPI communities. The regional experts have proven they adapt well to difficult circumstances and receive fair remuneration and benefits within USAID Third Country National (TCN) scales. (C15) Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 27 4.5 Cross- Cutting Issues 14. How well has the program mainstreamed gender and other equity issues? Please consider both direct and indirect beneficiaries? Gender  SPS Staff: The administrative assistant/financial manager and the monitoring and evaluation advisor are both Sudanese women. The other SPS employees are men. The corps of lead trainers developed by SPS in pharmaceutical management includes nine men and two women.  Mainstreaming: The GOSS Ministry of Gender, Social Welfare and Religious Affairs’ Gender Policy provides guidance to be integrated into all MOH policies and procedures.  Training: Since late 2007, SPS has set sex-disaggregated targets and collected gender data on all training programs offered. The annual targets for male and female trainees set in the Annual Work Plan are lower for female trainees. The total targeted in all three programs since 2007 is 500 male, 300 female. Although the actual numbers of trainees exceed the targets each year, the totals of female trainees fell below (and, in some cases, far below) the targets. (F19) Since this in-service technical training is offered primarily to employ civil servants, the pool of eligible trainees reflects the overall gender imbalance in the MOH where males predominate by a large margin. Sensitivity of Conflict-Affected Environment SPS has been working in Southern Sudan during periods of considerable insecurity and has security procedures and protections in place for SPS staff. Attention to overall security concerns is part of the planning process in determining state and program expansion, travel and training plans, and public health priorities that change in a rapidly changing political environment. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 28 V. LESSONS LEARNED The following observations were made by the team, but were not explicitly asked in the evaluation questions. 1. Time matters: It takes time for international advisors to adjust to the culture of new organizations (and for the organization to adjust to the presence of the advisory). The evaluation team was frequently advised by GOSS officials that long-term advisors are more effective than a series of short-term experts. Multi-year contracts and incentives to keep advisors supported and satisfied pay off. 2. Personality matters. Anecdotally, the evaluation team was told that the agreeable and amenable nature of the SPS LTTAs was key to their success and was more important than their technical expertise. A high value is placed on being a ‘team player.’ Future recruitments, interviews, and reference checks should emphasize social skills along with adaptability and technical expertise. 3. Proximity matters. Several other programs (WB MDTF, JICA) have long-term advisors working with the MOH. Only the SPS team is co-located in the MOH offices together with colleagues and counterparts. The day-to-day flow of business promotes a collegial atmosphere and builds trust. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 29 VI. RECOMMENDATIONS Based on evaluation findings and conclusions, the following recommendations are made. The figures in parentheses correspond to the table in Annex 11: Findings, Conclusions, and Recommendations. 1. USAID/Sudan should continue using the field support mechanism of the global SPS program, planning annual “buy-in” at increasing levels commensurate with the projected increasing capacity￾building needs of the MOH. (R1) Action: USAID 2. USAID/Sudan, the implementing partner (MSH) and the MOH should establish priority areas of expertise and appropriate timing for additional advisors in the coming years. Likely needs will include a logistics advisor and a pharmacovigilance advisor.19 Explore other constellations to maximize advisor exposure and access, for example, assigning a LTTA to a working team or rotating consultants among directorates. Individual pairing and counterparts can be very fruitful, but limited. (R2) Action: USAID, MSH, MOH 3. In coordination with the MOH Human Resource Development Department and other donors with Health Advisors (WB MDTF, JICA, WHO), MSH/SPS should develop a multi-year and multi￾pronged training plan. Should additional resources be available, the menu of training opportunities should include a mix of regional long-term training, strengthening existing linkages with Sudanese pre-service training institutes, development of accreditation training, TOT training and adult learning theory, and regular refresher training. (R3) Action: MSH, MOH 4. MSH Program managers and the GOSS MOH should develop a candidate profile and interview protocol that will elicit the qualities identified by respondents as crucial to success as a LTTA in the MOH: amenable personality, attitude, patience, humility, professional excellence, and cooperativeness. (R4) Action: MOH 5. MSH and USAID managers should explore developing a MOU to better define roles and responsibilities of LTTAs and their counterparts. These agreements (for which models are available) should provide the latitude and flexibility for LTTAs to respond to changing needs and priorities, while also providing guidance and protection for LTTAs from excessive gap-filling and extraneous assignments. (R5) Action: USAID 6. Within a year, decisions should be made by USAID and the GOSS regarding the placement of LTTAs at the SMOH offices. In the three regions already active, placement might be considered sooner. Considerations of security, access to resources and staff with whom to work, and living conditions (phone, electricity, internet, medical support, housing) need to guide the determination of placement, as well as strong programmatic reasons. Lead or expert trainers with responsibilities reaching out from the state level into the counties should be considered. Consider lessons from the TAP Program in planning state-level placements (R6) Action: MOH, USAID, MSH 7. MSH/SPS should review the job descriptions annually along with the personal performance reviews. Over the next five years, capacity building goals and targets should be included for both the advisors 19 Pharmacovigilance is concerned with the detection, assessment and prevention of adverse reactions to drugs. (Taken from: http://apps.who.int/medicinedocs/en/d/Jh2934e/3.html) Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 30 and the MOH counterparts to increase opportunities for skills transfer, and eventual replacement. Greater emphasis should be placed on active mentoring and training. (R7) Action: MSH 8. Future training should explore the Human and Institution Capacity Development (HICD) Model and determine if elements of this approach would improve the efficiency of training. HICD is “a series of structured and integrated processes designed to remove significant barriers to the achievement of an institution’s goals and objectives. HICD involves the systematic analysis of all the factors that affect performance, followed by specific interventions that address gaps between desired and actual institutional behaviors.” Systemic refers to the interactions between the different levels (individual, organizational, and institutional). Not only skills and organizational procedures but also issues of incentives and governance must be reviewed. (R8) Action: USAID 9. Deliberate efforts must be made to examine and address the low representation of women training and to remove obstacles to women’s full participation in training. In the context of the GOSS’ ambitious gender balance program, the MOH must increase gender awareness, expand pre-service education, and apply gender consciousness to issues of training and recruitment, with the goal of encouraging more active female participation and improving awareness of male and female behavior norms as they affect health access and behaviors. Gender-awareness, and the differential impact of policies and program on gender and other equity issues, should permeate all levels of policymaking and program planning. In addition, SPS must take a lead in demonstrating the pervasive role of gender in health delivery and assure that differing gender perspective are reflected in policies such as bed net distribution, access to health care, transportation and privacy issues, and heightened awareness of gender based violence and traditional health practices affected by gender. (R9) Action: MOH, MSH Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 31 ANNEX 1: EVALUATION SCOPE OF WORK Management Systems International (MSI) SUPPORT Project with USAID/Sudan20 Lessons Learned Evaluation of: Strengthening Pharmaceutical Systems (SPS) Project (Estimated Start Date: July 2010) 1. Program to be Evaluated Program Identification: Strengthening Pharmaceutical Systems (SPS) -Sudan Contract No. GHN-A-00–07–00002–00; Direct Grant/Cooperative Agreement/Leader with Associates (LWA) Program Funding: $3,200,000 Program Start/End Dates: 01-October 2005 to 09/30 201121 Key Agreement/Contract Modifications: Modifications: None Implementing Partners: Prime: Management Sciences for Health (MSH) Implementing Mechanism: Centrally Funded Infinite Quantity Contract USAID/Sudan Technical Office: Investing in People—Health Office COTR: Anthony Boni, aboni@usaid.gov Activity Manager: Martin Swaka, mcswaka@usaid.gov 2. Evaluation Purpose The primary purposes of the evaluation will be to assess the effectiveness of the field support model used by the project and review impact to date. Secondary objectives will be to determine whether a follow-on of the project or continuation of the field support model would be appropriate and how the model fits within the USAID Human and Institutional Capacity Development (HICD) policy. 3. Background A. Country Context Southern Sudan and the critical border areas (consisting of the northern states of Southern Kordofan and Blue Nile, plus Abyei—commonly referred to as the Three Areas) are characterized by years of underdevelopment, war, famine, drought and flood, producing a crisis of enormous proportions across the region and resulting in the devastation of economic, political and social structures. In addition to the loss of 20 MSI holds a 3-year contract to provide Mission-wide support to USAID/Sudan in program and project evaluation and designs, MIS management, translation services, logistics support, facilities management, VIP hosting, and research. An in-country team, based in Juba, provides these services, supplemented by short-term technical assistance. 21 The RPM Plus program started in October 2005: SPS, the follow on, started in June 2007 and is a five-year program. The MSH/SPS Advisors field support model started July 2006 under RPM Plus and continued with SPS. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 32 lives, opportunities and infrastructure, the war displaced families and divided communities. In consequence, the health, education and infrastructure status of the Sudanese people are among the poorest globally. After decades of civil war, Sudan’s warring parties signed a Comprehensive Peace Agreement (CPA) in January of 2005. Since that time the country has taken steps toward peace, reconciliation and good governance, although the pace has been slower than expected or desired. Despite the signing of the CPA, Sudan remains a vulnerable state. Its children, many of whom are orphans, returning refugees and ex-combatants, are particularly at risk—especially in the “hot spots” of the Three Areas. It is essential that displaced and other affected people, particularly orphans and ex-combatant youth, be safely reintegrated into their communities. In the case of the youth, affected by the many conflicts and tensions during the past 21 years, the provision of basic education and health services is critical to providing a solid foundation upon which their future success and contribution to society can be based. In many areas, education services are largely provided by GOSS and state Ministries of Education, while health services are primarily provided by NGOs. In both sectors, GOSS has received considerable support from development partners offering humanitarian relief including, NGOs, faith-based organizations (FBOs), and multilateral and bilateral aid agencies. As peace is consolidated, USAID will continue to support a responsible transition from emergency to development assistance that seeks to improve access to and quality of basic education. Education and health activities are reinforced by investment in other essential services, such as WASH (water, sanitation and hygiene) and livelihoods, in an effort to rebuild local communities, reduce tensions, and provide the much sought-after peace dividends. B. Sector Context Southern Sudan and the Three Areas face formidable health challenges in rebuilding from decades of civil war. Childhood deaths due to infectious diseases are rampant; the Infant Mortality Rate (IMR) is 102.4 deaths per 1000 live births. Vitamin A deficiency affects one of seven children in Sudan and DPT3 immunization among children is below 44 percent. Maternal mortality ratio is among the highest in the world, estimated at 2054 maternal deaths per 100,000 live births (2006 HHS). Overall, reproductive health is poor due to lack of access to skilled antenatal care providers (the proportion of births attended by skilled health staff is also among the lowest in the world), a high fertility rate of 5.9 live births per woman (UNICEF), low rates of modern family planning methods, and a high percentage of home births at 94 percent. HIV/AIDS is an emerging threat due to risky sexual behaviors and Sudan’s proximity to the regional pandemic. Water and sanitation infrastructure is non-existent or marginal at best, and sanitation and hygiene practices are poor. A wide range of ‘tropical’ diseases that are controlled elsewhere are endemic in Southern Sudan; many of these are also so-called ‘neglected diseases.’ Critical health risks in Sudan include tuberculosis, malaria, pneumonia and diarrhea. With few exceptions, population density is low, which remains an obstacle to both service provision and access to quality health care. The GOSS National Malaria Control Program (MCP) is responsible for implementing the five-year National Malaria Strategic Plan (NMSP). The NMSP goal is to reduce malaria deaths by 50 percent focusing on providing prevention and treatment services targeting high–risk groups such as children under five, pregnant women, and internally displaced persons. The MOH strategy is to cover the whole population of Southern Sudan with Long Lasting Insecticide Treated Nets (LLITNs) in the next five years with funding from the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM). USAID supports the MOH strategy through training, drug procurement, and technical assistance. The drug supply management system is fragile and principally managed by NGOs, with very limited training and capacity in this area. The community level covers only about 30 percent of the population in stable areas. Other than private pharmacies and unauthorized market drug tables in urban areas, there is a near absence of formal private sector medical care. In aspiring to transition from emergency to sustainable health service delivery, attention is needed to address the fragmentation that has resulted from years of neglect in the health sector. Emergency response interventions were often disjointed, short term, and inefficient. Previously, the focus was on first-level health Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 33 services and disease-specific programs—typical for humanitarian action. This has, understandably, overshadowed attention to building sustainable basic health systems and infrastructure, strengthening human capacity, and actively mobilizing and engaging civil society groups for decision making around their own health. Shortage of skilled human resources has been—and is—one of the major limiting factors to providing basic health care. The GOSS Ministry of Health (MOH) advocates an integration of the existing vertical programs into both the resource pool and management structures of the mainstream health system. The GOSS/MOH Basic Package of Health Services (BPHS) Policy is based on five principles: right to health, equity, pro-poor, community ownership and good governance. The main criteria for the choice of priority services were ones that would have the greatest potential for impact on the health of the population, that would be equally accessible to the largest possible part of the population, and be affordable in the short-term and sustainable in the long term. Since the signing of the CPA, the GOSS has developed nearly a dozen policies, strategies, curricula, cadre position descriptions and guidelines for implementing different sub-sectors within the health sector. Working with a range of NGOs/organizations offering health services and training throughout the country is a challenge. The GOSS has formed alliances with the NGO Forum for soliciting technical advice and regularly solicits guidance to formulate policies and programs to harmonize health services delivery as the government transitions from emergency to development. Current donors are working closely with the central GOSS/MOH to strengthen capacity in policy and strategic planning and with SMOHs to help build capacity for program implementation and management at the state level. C. Program Description Background: The Rational Pharmaceutical Management (RPM) Plus program was awarded to Management Sciences for Health (MSH) in FY 2000 and was to run for five years (end date September 2005). The ceiling for the agreement was subsequently raised twice and the program was extended to September 2008. The RPM Plus program is Washington, DC based support project, which has been implemented globally. Strengthening Pharmaceutical Systems (SPS), a five-year Leader with Associates program, was awarded to MSH in June 2007 as a follow-on to the RPM Plus; the award ends on Sept 30, 2011. The SPS Global Program focuses on four key results:  Improve governance in the pharmaceutical sector  Strengthen pharmaceutical management systems to support public health services  Contain the emergence and spread of antimicrobial resistance  Expand access to and improved use of essential medicines In order to guide the selection of appropriate activities, USAID identified specific Intermediate Results (IRs) that correspond to each Result Area. A reference guide22 is provided to SPS country programs to ensure that planned activities support the USAID-identified IRs. To help meet goals relating to the key result areas, the SPS Program is building on the experience of RPM Plus while expanding the range of technical areas to include financing, pharmaco-vigilance, pharmaceutical care, integration of new health technologies, and increased use of the private sector.23 RPM Plus and SPS in Southern Sudan: The RPM Plus program support to Southern Sudan began in October 2005 to the then Secretariat of Health. Actual full time support commenced in July 2006 when a Technical Advisor was recruited to work within the 22 See section 3E for more detail. 23 http://www.msh.org/global-presence/sub-saharan-africa/sudan.cfm#two Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 34 National Malaria Control Program of GOSS MOH. In 2007, there was an immediate transition24 from RPM Plus to the SPS program when MSH was awarded the new contract. The evaluators should consider the effort, through the transition from RPM Plus to SPS, as one project. Since 2005, USAID has supported the GOSS through SPS, which aims to significantly reduce the disease burden of malaria in Southern Sudan while increasing GOSS capacity to combat this disease. The SPS overall mandate in Southern Sudan includes: 1) Strengthen the capacity of the malaria control program to effectively plan, coordinate and document control and prevention interventions in line with the national Roll Back Malaria (RBM) strategic plan and 2) Strengthen pharmaceutical management systems to promote the effective management of medicines and supplies for malaria. Specifically:  Technical coordination and office support, strengthening operational capacity of the malaria control and pharmaceutical management programs at central and state levels.  Support in policy development, planning and coordination of malaria control activities at the central and state level; scaling up cost-effective malaria control interventions, and supporting malaria M&E systems at central and state levels.  Pharmaceutical management activities, including coordination support and policy development; support to procurement and distribution of anti-malarial medicines and supplies; capacity building for the public and private sector; and support supervision, inspection and quality assurance. The Southern Sudan SPS program uses a mentoring approach with SPS Technical Advisors placed in the relevant departments of MOH to work on a full-time basis with their national counterparts. Southern Sudan SPS work plan activities are designed to be in line with both SPS IRs and MOH strategic plans and targets. Implementation of SPS activities is undertaken with and through the relevant MOH institutions at central, state and county levels. The systems strengthening approach employed by SPS aims to empower MOH institutions at central, state and local levels to effectively coordinate and manage the malaria, pharmaceutical and EPI programs. SPS technical support initially focused on development of key program policies and strategies at the central MOH GOSS level but has progressively included support to implementation in selected states. Since the initial inception of SPS, the project has expanded to include MOH advisors in the following areas as of mid March 2010: Program/Department Placement Duration of support25 Malaria Control Program Senior Technical Advisor 3 and ½ years Malaria Control Program—Operations Finance and Administrative Manager 2 and ¾ years Department of Pharmaceutical Services Senior Program Associate 2 and ½ years Malaria Control Program M&E Officer 1 and ½ years 24 In some countries, RPM Plus and SPS programs were run concurrently until the close of RPM Plus in Sept 2008. 25 Approximate duration as of mid March 2010 Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 35 Program/Department Placement Duration of support25 Expanded Program on Immunization (EPI) Technical Advisor 3 weeks D. Linkage to USAID/Sudan Strategy and USG Foreign Assistance Framework USAID/Sudan’s strategy focuses on supporting the CPA by assisting the GOSS to provide peace dividends and address the factors that fuel conflict. Key to that process are interventions that will provide tangible peace dividends—e.g. quality health services—and increase the confidence of the population in their government’s ability to provide basic services. In addition, improving the overall health of the Sudanese people will enable citizens to become more productive, allowing education levels to advance, promote economic growth, and reinforce stability and peace. E. Project Strategic Summary (for SPS Global) Result 1: Improved Governance in the Pharmaceutical Sector IR 1.1: Improved medicines policy, regulation, quality assurance and pharmacovigilance IR 1.2: Improved decision making and strategic planning capabilities for the pharmaceutical sector IR 1.3: Ethical, transparent, accountable and efficient procurement practices implemented IR 1.4: Appropriate standards for the provision of pharmaceutical services established and maintained Result 2: Strengthen Pharmaceutical Management Systems to Support Priority Public Health Services and Interventions IR 2.1: Improved pharmaceutical management systems and effective approaches to support PHN service delivery interventions IR 2.2: Improved quality and increased quality of human resources capable of performing pharmaceutical management functions and services IR 2.3: Increased capacity of local institutions and networks to provide pharmaceutical management TA IR 2.4: Improved availability of essential medicines, diagnostics and equipment and other health supplies for USAID supported programs IR 2.5: Improved management of laboratory commodities, supplies and equipment. Result 3: Contain the Emergence and Spread of Antimicrobial Resistance (AMR) IR 3.1: Proven institutional interventions implemented to minimize the spread of AMR IR 3.2: AMR interventions designed and implemented to improve medicines use behaviors at the community level IR 3.3: Innovative approaches implemented at the global and country levels to mobilize resources and action to help contain the development of AMR Result 4: Expand Access to Essential Medicines IR 4.1: Increased medicines access programs in the private sector IR 4.2: Effective uptake and integration of proven new technologies into service delivery programs IR 4.3: More effective use of financing mechanisms, for enhanced access to medicines F. Geographic Orientation The initial project was focused at the national level, but has now begun to expand to include three states.  Central Equatoria Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 36  Eastern Equatoria  Jonglei However in the future, some activities such as support supervision and inspection of pharmaceutical services, will be implemented at a wider scale, in all ten states of Southern Sudan. MSH/SPS will also increasingly provide technical support to implementation of activities in thirteen counties where MSH is working under another project funded by USAID,( Sudan Health Information Transformation-2 Project (SHTP-2). 4. Available Information to Support the Evaluation The following information will be provided to the evaluation team in advance of its arrival in Juba. 1. Contract (including Project Description), with any modifications 2. Project work plans 3. All available performance monitoring data as of the most recent available date 4. Project results framework 5. Quarterly/ Annual Reports 6. Any useful technical reports 7. Any relevant assessments or studies 8. USAID Fragile States Framework 9. MSI Evaluation and Special Study Guide 10. USAID Evaluation Guidelines and Standards for Foreign Assistance 11. USAID Human and Institutional Capacity Development Handbook and policy documents 12. RPM project documents 5. Evaluation Focus and Questions The purpose of the evaluation is to assess the effectiveness of the technical assistance approach to GOSS capacity building in the areas of pharmaceuticals management, malaria control and Expanded Program on Immunization (EPI) used by the Strengthening Pharmaceutical Systems (SPS) Program in Southern Sudan. The secondary purpose of the evaluation is to advise USAID about the appropriateness of this capacity building approach in a continuing or follow-on program taking into account recent USAID research on capacity building in Southern Sudan, a conflict-affected context. Evaluation Questions Program Design 1. Are there any issues with respect to Program design and assumptions (documented or implied) that should be reconsidered or addressed based on experience to date? 2. What are the pros and cons of the field support mechanism in the context of Southern Sudan and the SPS Program? 5. How does the SPS capacity building approach align with best practices in capacity building in conflict￾affected contexts? 6. How is the role of the advisers designed, given the need to balance urgent operational needs of the Ministry with mentoring, training and technical work? Program Impact and Sustainability 8. How satisfied is USAID, and the Ministry of Health (at the national level and the three targeted state level ministries) with the Program up to this point? 9. What is the Program’s perceived value to other Ministry of Health partners (e.g. the Global Fund)? 10. Are the current benefits likely to be sustained after the program- why and why not? 11. In what ways has the input of the TA resulted in positive changes in the health sector for beneficiaries? Effectiveness Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 37 11. How effective has the SPS Sudan Program been in addressing capacity within the Ministry for program planning, coordination, and technical knowledge in the areas of malaria control, EPI and pharmaceutical management? 12. Specifically, what policies, guidelines and tools developed by SPS in the areas of the National Malaria Control Program, EPI and Pharmaceutical management were adopted at the National and state levels? 13 How well has the Program been able to meet indicators and contractual targets? Efficiency 14. How efficient is the model in maximizing time and effort to build capacity of the Ministry of Health to respond effectively to the needs of the National Malaria Control Program, EPI and Pharmacy Management? Please describe techniques utilized by the model. 15. Which aspects of program management demonstrate or fail to demonstrate efficient management, considering financial implications? Cross-cutting Areas 15. How well has the program mainstreamed gender and other equity issues? (Please consider both direct and indirect beneficiaries.) 6. Evaluation Methods and Procedures The Evaluator contracted through this SOW will be provided the information provided in Section 3, above, before departing for Sudan. They will be expected to be familiar with this information on arrival in Juba. A Team Planning Meeting (TPM) will be held upon arrival in Juba to agree on how team members will work together and how they will interact with the client and other stakeholders. The Team will also develop a common methodology to be applied consistently. Other products include a work plan, and a final Travel Schedule. The team will need to visit project site(s) so some transportation will be arranged prior to the team’s arrival. The team will spend most of its time in Juba, but will also visit sites in Central and Eastern Equatoria. During the TPM the team will finalize the methodology to be used and produce the evaluative instruments to be employed. The team will use the “Getting to Answers” approach detailed in Annex 2 of the MSI Evaluation and Special Study Guide to develop detailed methodological approaches to meeting the terms of this Scope of Work. USAID Evaluation Guidelines and Standards for Foreign Assistance will also be utilized in developing methodology, reviewing data collection procedures, and reporting formats for the results. We expect that in addition to basing the evaluation’s findings on interviews and review of project documents, the team will also utilize the following simple approaches:  Short literature review on projects that involve technical staff seconded to ministries in similar contexts (i.e. Sierra Leone, Liberia)  Development of an interview guide to ensure that the correct evaluation questions are being addressed the appropriate individuals and that they are being posed and recorded consistently.  Key Informant Interviews  Review quality of tools/data collected for monitoring pharmaceutical management system  Other methodology determined appropriate during the TPM Once the methodology has been finalized at the TPM it will be shared with USAID as part of the work plan approval process. 7. Team Composition and Participation Team Composition USAID/Sudan is conducting the Lessons Learned Evaluation in a collaborative manner to maximize USAID, GOSS and Implementing Partners learning opportunities. Accordingly, the team will be comprised as follows:  One External Evaluator, provided by MSI, serving as Team Leader Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 38  One representative of USAID  One representative of GOSS  One representative of the Implementing Partner Additional inputs may come from other staff from these agencies, as needed, and as coordinated by the respective team member. USAID’s representatives will likely include a person from the mission health team, the program office, USAID/East Africa or USAID/Washington. GOSS and the Implementing Partner (IP) may choose their representatives as they see fit, but persons selected should have experience with similar programs in Sudan. Given the significant contributions to the team expected from each team member, all are expected to be available to participate throughout the evaluation period. Team Member Roles and Responsibilities GOSS, USAID/Sudan, and IP team members will provide historical, contextual and programmatic background information that will inform the assessment. They will be expected to participate in the Team Planning Meeting (TPM), field visits, interviews, brainstorming on Findings, Conclusions, and Recommendations, and in the frequent reflections on evaluation learning, often occurring after a long day of interviews and traveling. These individuals participate as representatives of their respective organizations and are expected to share their learning with their home organizations so that all three key organizations are kept abreast of progress. It may well happen that the Team will ask GOSS, USAID, or IP representatives to be excluded from certain portions of interviews in order to ensure candid responses. The External Evaluator will take the lead in conducting the evaluation, in close cooperation with other team members, leading interviews, framing the analysis, facilitating group discussion and consensus, preparing for the debriefing, and drafting the evaluation report. The External Evaluator will also serve as the overall Evaluation Team Leader and will take lead responsibility for managing the team, organizing its work, and promoting quality control and delivery of a final report acceptable to USAID. Precise division of labor among the Team members will be determined at the TPM. Among the Team members, the following capacities must be brought to the team: 1. Strong skills in assessment and analysis of USAID projects, especially with health programs; 2. Extensive experience working in East Africa, Sudan, and/or similar post conflict environments; 3. Facilitation experience, experience leading participatory evaluations, or at least evaluations where evaluation teams include critical stakeholders as active participants; and 4. Experience arranging meetings, setting up travel schedules for field visits, reporting on meeting outcomes, and generally managing the logistics of the evaluation (although significant logistical assistance will be provided by the SUPPORT team in Juba). 5. Experience in implementing or evaluating the following: a. Health Sector reform b. Health Systems Management c. Health System Strengthening d. Monitoring and Evaluation and management information systems management e. Performance-Based contracts with USAID f. Decentralization g. Pharmaceutical management systems h. Malaria Control Programs (not required, but would be useful) The Team Leader will be the formal representative of the team and will arrange for updates regarding progress against the evaluation work plan to the COTR (or his/her delegate) and MSI’s Chief of Party (COP) or Assessment, Monitoring and Evaluation Specialist (AME), as determined at the TPM. 8. Activities, Logistics, and Timing Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 39 Prior to arriving in Juba, the External Evaluator will have familiarized him/herself with the background material provided, as referenced in Section 4, above. All team members should be present for the TPM and for initial briefings and discussions with USAID’s Health Office and other Mission officers, as well as IP and GOSS officials. A Work Plan and travel program for the in-country visit as well as the subsequent report-writing period will be submitted to USAID for approval during the first few days of work in Juba. The Work Plan will also include a schedule for periodic MSI and USAID progress reports and possible submissions of specific work products, as determined by the parties. The team will spend a majority of its time in Juba, but will also travel to sites in Central and Eastern Equatoria. Approximately four days prior to departure the Evaluation Team will present to USAID, Implementing Partner, and the GOSS an out-briefing, with succinct supporting documents. The Draft Evaluation Report will be submitted prior to the External Evaluator’s departure from Juba. The Mission and the IP will each submit their respective comments on the draft report within ten working days of receipt of the draft report. The Draft Final Report will be submitted to USAID fifteen working days after the Team Leader’s receipt of USAID’s and the IP’s final written comments on the draft. It is envisioned that the External Evaluator will be in Sudan the entire duration of the evaluation’s in-country component (six-day work weeks are authorized), including the TPM, a debriefing, and submission of a draft report to MSI’s COP or AME prior to departure from Sudan. In addition to travel days, additional days are provided for the External Evaluator to complete reading and processing all background information prior to departure for Sudan. Additional days are provided to finalize the report. (See graphic presentation in Section 9, below.) MSI’s field office in Juba will be responsible for travel arrangements (travel, housing in the field, etc.) for the USAID and GOSS team members. MSI will fund travel-related costs for GOSS team member(s), but not for IP or USAID team member(s).26 MSI and the Implementing Partners will jointly arrange all meetings for the team, in coordination with GOSS. The team will be provided office and meeting space, as needed, at SUPPORT’s Juba Office Compound. 9. Projected Level of Effort (LOE) and Timeline Tasks Work Days (6-day weeks in Sudan; 5 outside Sudan) Approximate dates Initial Preparation Review advance background documents, study assessment methodology and SUPPORT Project’s Evaluation and Special Study Guide, make travel preparations, and travel days to Juba. 6 In-Country Preparation TPM, methodology development and mutual training 3 Arrival on July 24 (TPM day 1), July 26 (TPM day 2), July 27 26 If the USAID representative is an Institutionally-Contracted Staff member provided by MSI, his/her travel costs will be provided by MSI separately. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 40 Tasks Work Days (6-day weeks in Sudan; 5 outside Sudan) Approximate dates In-Country Evaluation Meetings in Juba and field visit to Torit 7 July 28—August 4 Data Analysis and report drafting Analyze data, brief USAID and GOSS, draft report 5 August 5— August 10 Return travel 2 Final Report Preparation in home country Incorporate USAID feedback, complete final report, and submit to MSI office in Juba. 4 Total for Team Leader 30 10. Report Production and Format The team will present for approval by USAID a draft outline of the report during its first week in country. The report must:  Distinguish clearly between findings, conclusions (based strictly on findings) and recommendations (based clearly on the reports findings and conclusions);  Comply with all instructions of the SUPPORT Project’s “Evaluation/Special Study Quality Management Guide” and meet the specific requirements of the “Evaluation Report Review—Score Sheet,” contained therein;  Include a Table of Contents; a list or acronyms, an Executive Summary of no more than three pages; a section describing the project to be evaluated and purpose of the evaluation; a section on the methodology employed, including relevant skill sets of the evaluators;  Include any annexes the team considers useful to the reader; and  A copy of this SOW as an Annex. A formal debriefing will be provided to USAID, the IP and the GOSS, as scheduled during the TPM and recorded in the evaluation work plan. The team will present key Findings, Conclusions and Recommendations for comment from the stakeholders. The team will record all relevant feedback from the meeting and will respond to all comments in completing its draft reports. The External Evaluators need not include all suggestions in the report, but must consider such suggestions in finalizing the Draft Report. An electronic (in MS Word) version of the Draft Report will be presented to the IP and USAID in Juba with four hard copies being provided to the USAID/Sudan Mission and one hard copy to the IP prior to the departure of the Team Leader. The document will not exceed 30 pages, excluding annexes and Executive Summary. The Mission and the IP will each submit its respective comments on the draft report electronically to MSI’s COP—using the “track changes” and “comments” functions in MS WORD as much as possible. Each organization will combine internal comments, resulting in a unified set of comments from USAID and a Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 41 unified set of comments from the IP. The Mission will receive ten paper copies of the final report as well as an electronic version, once the Mission has accepted the product. 11. Deliverables  A draft work plan, ensuring that all aspects of Getting to Answers (from the TPM) are addressed  A schedule of travel and key activities  Interim progress briefings to MSI and the Mission, as determined during the TPM  Preliminary report outline  Draft Findings, Conclusions and Recommendations to MSI prior to completion of the first Draft Report  Out-briefing, with supporting documents  Draft report  Final report 12. Compliance to USAID Regulations The Evaluation Team will ensure that the evaluation is fully compliant with the terms for Project Evaluations contained in the USAID Automated Directives System (ADS) Series 203 and other relevant regulatory requirements, as may be determined by USAID. Additionally, the Team will utilize MSI’s SUPPORT Project’s “Evaluation/Special Study Quality Management Guide” as well as the USAID Evaluation Guidelines and Standards for Foreign Assistance. These guides will be presented to the Team members prior to their initial TPM. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 42 ANNEX 2: EVALUATION TEAM MEMBER RELEVANT BIOGRAPHICAL INFORMATION: Martin Swaka works currently with USAID as Senior Project Management Specialist/Health. Dr. Swaka is a medical doctor and has a graduate education background in public health and infectious diseases. Dr. Swaka has more than 14 years experience in management of public health and infectious diseases programs in complex emergencies; moreover, he has worked in various mid- and senior-level management positions with several international health organizations including Norwegian People’s AID, CARE International, and IRC. Dr. Swaka has participated in several health program evaluations and has broad experience in project planning, design, monitoring and evaluation, and supportive supervision. Dr. Swaka holds an MPH in International Health. Evans Sagwa is Deputy Senior Technical Advisor of the MSH Strengthening Pharmaceutical Systems Program in Windhoek, Namibia, where he has worked since 2007. He provides technical oversight and leadership in capacity building and strengthening systems for pharmaceutical governance, medicines regulation, pharmacovigilance, and human resources development in Namibia. He has supported increased production of pharmacy assistants and facilitated trainings in pharmaceutical management, designing and implementing interventions for promoting rational medicine use. He leads the SPS activity of establishing Namibia’s pioneer bachelor of pharmacy course at the University of Namibia. Prior to joining SPS, Mr. Sagwa was the chief pharmacist at the King Faisal Hospital in Kigali, Rwanda, where he was the embedded long￾term technical consultant for building and strengthening pharmaceutical management systems. Mr. Sagwa holds Bachelor of Pharmacy and Master of Business Administration degrees, has just completed his research thesis for a Master of Public Health degree, and has embarked on a PhD in Pharmacoepidemiology and Pharmaceutical Policy Analysis. Team Leader Ann von Briesen Lewis has 30 years of development experience in Africa, Asia, and South America. She has led or joined evaluations of USAID capacity building programs in Malawi, East Timor, Nepal, and Nigeria; designed, evaluated, and managed gender, humanitarian assistance, education and democracy programs for USAID, USAID/OFDA, and IRC in conflict-affected and transitional East Timor, Indonesia, Colombia, Haiti, Rwanda, and Nepal. Ms. Lewis was Executive Director of Fulbright Commissions in Nepal and Indonesia and served as an embedded Technical Advisor in the Development Planning Commission of the government of Indonesia. She holds an MA in International Development with additional certification in International Humanitarian Law. Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 43 ANNEX 3: TIMELINE Dates Program Events External key events 2000 MSH awarded RPM+, 3-year global program 2005 SPLM organizing meetings in Kenya Oct 2005 USAID begins RPM+ in S. Sudan-extended to 9/07 Jan 2006 Comprehensive Peace Agreement Signed 2006 Request from GOSS MOFS to USAID for assistance in health, especially malaria 2006 Dr. Baba, GOSS MOH requests RPM+ support June 2006 MSH Field Support places Dr. Robert Azairwe in the MOH August 2006 USAID MSH/RPM+ Field Support Office mandated New USAID Health Team Leader August 2006 With SPS leadership, the MOH develops first joint annual NMCP plan with all the donors New USAID Health Team Leader June 2007 MSH awarded Strengthening Pharmaceutical Systems CA for five years, 2007–2012 April 2007 NMCP building and board room completed at MOH with RPM Plus funding Late 2007 SPS recruits and hires Administrative and financial officer Patricia placed in MOH Late 2007 MOH requests SPS support in Pharmacy Department Late 2007 Dr. Mawa Arrives New USAID Health Team Leader 2007 SHTP II awarded to MSH Late 2007 GFATM Round 7 proposal funded 2008 More than one million LLINs distributed by PSI and JSI with MDTF and USAID funding 2008 Decision to expand into three states Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 44 2009 Management of 1.67 million doses of ACTs procured through USG funds. 2009–2010 More than four million LLINs distributed under round 7 malaria grant (PSI, Malaria Consortium and other sub￾recipients) and UNITAID donation (UNICEF) 2009 Supported MOH to conduct Malaria Indicator Survey 2010 Management of 1.1 million doses of ACTs procured through USG funds and 50,000 RDT kits June 2010 Pharmaceutical sector assessment May 2010 General Elections January 2010 Decision to expand to EPI Polio outbreak March 2010 EPI adviser, recruited, placed August 2010 GFATM proposal development August 2010 New Minister of Health January 9, 2011 Referendum September 2010 Midterm Evaluation September 30, 2012 End date SPS Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 45 ANNEX 4: EXPANDED METHODOLOGY SECTION A. Getting to Answers Matrix Getting to Answers: MID-TERM EVALUATION: SPS version 16/09/2010 Program or Activity: Team Members: Methods for Data Collection Evaluation Questions Type of Answer/ Evidence Needed (description; comparison; cause and effect) and notes on special requirements or sources of data Method Data Source Sampling or Selection Approach (if applicable) Data Analysis Methods (e.g. frequency distributions, trend analysis, cross￾tabulations, content analysis A. Project Design 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered or addressed based on experience to date? Description: Secondary data, including Project Contract, Work plans, Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews and discussions SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis 2. What are the pros and cons of the field support mechanism in the context of Southern Sudan and the SPS project? Description: Secondary data, including USAID guidance documents, SPS Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis 3. How does the SPS capacity building approach align with best practices in capacity building in conflict-affected contexts? Description: Secondary data, including Project Contract, SPS frameworks, Work plans, Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews and discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 46 4. How is the role of the advisers designed, given the need to balance urgent operational needs of the Ministry with mentoring, training and technical work? Description: Secondary data, including Project Contract, SPS frameworks, Work plans; Primary data- people’s input Collect documents; conduct meetings; interviews and discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis B. Project Impact and Sustainability 5. How satisfied is USAID, and the Ministry of Health ( at the national level and the three targeted state level ministries) with the project up to this point? Description: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews and discussions USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis 6. What is the project’s perceived value to other Ministry of Health partners (e.g. the Global Fund)? Description: Secondary data, including Work plans, Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews and discussions SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff and other partners Purposive, for documents and persons with the richest source of relevant information Content analysis 7. Are the current benefits likely to be sustained after the program- why and why not? Description: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis 8. In what ways has the input of the TA resulted in positive changes in the health sector for beneficiaries? Cause and Effect: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis C. Effectiveness Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 47 9. How effective has the SPS Sudan project been in addressing capacity within the Ministry for program planning, coordination, and technical knowledge in the areas of malaria control, EPI and pharmaceutical management? Description: Secondary data, including Work plans, Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff and other partners Purposive, for documents and persons with the richest source of relevant information Content analysis 10. Specifically, what policies, guidelines and tools developed by SPS in the areas of the National Malaria Control Program, EPI and Pharmaceutical management were adopted at the National and state levels? Description: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, and discussions SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis; frequency distributions 11. How well has the project been able to meet indicators and contractual targets? Description: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, and discussions SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis; frequency distributions D. Efficiency 12. How efficient is the model in maximizing time and effort to build capacity of the Ministry of Health to respond effectively to the needs of the National Malaria Control Program, EPI and Pharmacy Management? Please describe techniques utilized by the model. Cause and Effect: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff, other partners Purposive, for documents and persons with the richest source of relevant information Content analysis, frequency distributions, observations 13. Which aspects of program management demonstrate or fail to demonstrate efficient management, considering financial implications? Cause and Effect: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis, frequency distributions, observations E. Cross-cutting Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 48 14. How well has the program mainstreamed gender and other equity issues? Please consider both direct and indirect beneficiaries? Description: Secondary data, including Quarterly/ Annual Reports, relevant Technical reports, assessments or studies; Primary data- people’s input Collect documents; conduct meetings; interviews, discussions and FGDs SPS Project Team Leader and technical advisors; USAID activity manager; Ministry staff Purposive, for documents and persons with the richest source of relevant information Content analysis; frequency distributions Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 49 B. Interview Focus Areas and Data Source Emphases Focus USAID￾Activity Manager (Martin) SPS Team Leader (Robert) SPS Technical Advisors GoSS-MoH (Counterparts) (central level) GoSS-MoH (Counterparts) (state level) SHTAP Partners￾Malaria Consortium, LATH Other Donors (WHO, UNICEF, GFATM) Ultimate Beneficiaries (MoHSS staff and central and state levels) 1. Issues with respect to project design and assumptions to be reconsidered or addressed XXX XXX XXX X X 2. Pros and cons of the field support mechanism XXX XXX XXX XX XX 3. Alignment of SPS capacity building approach with best practices in capacity building XXX XXX XXX X 4. Design of role of the advisers to balance TA and operational work XXX XXX XX XX 5. Satisfaction of USAID and the Ministry of Health with SPS support XXX XX XXX XXX XX 6. Project’s perceived value to other Ministry of Health partners XXX XXX XXX XX XXX XXX XXX X 7. Sustainability of current benefits after the program XXX XXX XX XXX XX 8. How input of the TA resulted in positive changes in the health sector for beneficiaries XXX XXX XXX XXX XXX XX X XX 9. SPS effectiveness in building capacity within MoH for program planning, coordination, and technical knowledge in the areas of malaria control, EPI and pharmaceutical management XXX XXX XXX XXX XX XX XX XX 10. What policies, guidelines and tools developed by SPS have been adopted XXX XXX XXX XXX X X XX 11. How well has SPS been able to meet indicators and contractual targets XXX XXX XXX 12. Efficiency of SPS model in maximizing time and effort to build capacity of the MOH XXX XXX XX XX 13. Aspects of program management that demonstrate or fail to demonstrate efficient management XXX XXX XX X 14. Gender mainstreaming and other equity issues XXX XXX X X Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 50 C. Illustrative Interview Guide Level Illustrative Questions All 1. State purpose of SPS evaluation, use of the report, and clarify the role of the interviewee Key Participants 2. What has excited or pleased you most about the program? 3. What are your biggest concerns? 4. Which aspects of the SPS program have been most effective? 5. Why have they been effective? 6. What would you change? 7. What is needed most to improve capacity within the MOH? 8. How would you describe the role of technical advisors? What percentage of their time and effort is mentoring and what is “doing”? 9. What personal qualities are most important for a successful TA to possess? 10. How are gender issues integrated into the program, the training curriculum, and the policies? 11. What are the key benefits of placing TA within the ministry and what are some of the drawbacks? Partners 12. What personal qualities are most important for a successful TA to possess? 13. What are the key benefits of placing TA within the ministry and what are some of the drawbacks? 14. What changes have you seen since the program’s inception in 2006? 15. Was the balance of focus at the central and state level appropriate? Should expansion to the states have been sooner? 16. What did you learn at training and what do you do differently on the job now? 17. How does the SPS model compare with other models for capacity building and LTTA in terms of effectiveness and efficiency? 18. What kind of monitoring and evaluation of LTTA has been used? How is success defined? 19. How are gender issues integrated into the program, the training curriculum, and the policies? Senior Officials and Beneficiaries (direct and indirect) 20. Where does SPS fit in the landscape of health sector activities and what part does the program play? 21. Has the provision of TA helped build capacity in the MOH? Can you offer an example? 22. What are some of the drawbacks placing LT advisors in operational roles? Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 51 23. What differences have you seen in the area of __________ since the program began in 2006? 24. How satisfied are you with the Malaria Program, availability of appropriate drugs? Improvements in EPI? 25. Have you seen changes in the MOH performances? Examples? 26. What do you think the program will leave behind after it ends? 27. How has the average Sudanese citizen benefitted from the program? Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 52 ANNEX 5: LIST OF PERSONS INTERVIEWED Position Topical focus Contact info Gender USAID Charles Lerman, Ph.D. Health Team Leader, USAID/Sudan SPS project, USAID strategy clerman@usaid.gov m David Schroder USAID Program Office Field Support mechanism, Cap bldg dschroder@usaid.gov m Anthony F. Boni CTO RPM Plus and SPS Pharmaceutical Management Specialist aboni@usaid.gov m Dr. Martin Swaka Senior Program Specialist mcswaka@usaid.gov m SPS Sudan Patricia Ayite Administration and Finance Manager Roles in SPS program payite@msh.org f Mawa Steven Senior Program Associate (Pharmaceutical Advisor) Roles as LTA smawa@msh.org m William Mbabazi Technical Advisor (EPI Advisor) Roles as LTA wmbabazi@msh.org m Robert Azairwe Senior Technical Advisor /Team Leader (Malaria Advisor) Role as LTA, Evolution of SPS program razairwe@msh.org m Neni Daniel Program Associate (Pharmaceutical) Roles in SPS program ndaniel@msh.org m Betty Eyobo Lejukole Monitoring and Evaluation Officer Roles in SPS program mlejukole@msh.org f MSH—SHTP II Dr. John Pasquale Rumunu Technical Director MSH SHTP II jrumunu@msh.org m Dr. Uche Azie Chief of Party MSH SHTP II uazie@msh.org m Gabriel Daniel Country Program Manager for Ethiopia and Sudan Strengthening SPS program m Ministry of Health Government of Southern Sudan Dr. Samson Paul Baba Director General, External Relations and Coordination, MOH Overall program design, impact on MOH Samson_baba@yahoo.co.uk m Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 53 Dr. Thabo Othwonh Program Officer National Malaria Control Program, MOH (former Program Manager) o.ajameng@yahoo.com m Dr. Manyang Agoth Director General, Pharmaceutical Mngt Services, MOH Role of LTTA, mentoring and skills transfer thonbol@yahoo.com m Dr Bortel Ohisa Deputy Director for Quality Assurance, MOH Borte12004@yahoo.co.uk m Dr. John Lagu Epidemiologist, Acting Director General, Disease Prevention, MOH Impact of LTTA Jlagu2001@yahoo.com m Dr. Anthony Laku Director EPI, MOH Counterpart Alako_k@yahoo.com m Dr. Edward Bepo Director, Malaria Program edubepo@yahoo.com m Eastern Equatoria State Ministry of Health Elijo Omoro Tahir Director, Pharmaceutical Services, EES Torit Expansion to state level, training m Cecilia Akonyo Arato In Charge Pharmacy Torit Hospital Use of training f Patrick Okot State Medical Store Keeper EES Changes in PM m Bortel Ohibe Pharmacy Medical Assistant Torit Hospital m Dr. Augustine Okwahi Senior Medical Officer, SMOH Torit State Hospital Changes in Torit Okwahi@yahoo.com m Abwoch James Sam Deputy Hospital Patron, Pediatric Department, Data Collector Changes in Torit m Elizabeth Akong Matron, Torit Civil Hospital, Data collector Changes in Torit f Francis Okot Administrator, Torit Civil Hospital Changes in Torit m Dr. Lenny Loromo Director General, Health Services, Malaria activities in Torit lennyloromo@yahoo.com m Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 54 state MOH Amelia Iliha Eukario Charge Nurse pharmacy department data collector Torit Changes in monitoring and data collection f Dr. Ezbon William Apary Hospital Pharmacist Torit State Hospital Pharmaceutical management support in Eastern Equatoria state w.ezbon@yahoo.com m Other Partners Dr. Danny Ngemera UNICEF, EPI Efficacy of LTTA m Dr. Grace Murindwa Program Specialist, Liverpool Associates for Tropical Health (LATH) Other capacity building TA, external view of SPS murindwag@yahoo.com m Christian Fagueret COP, S.Sudan Technical Assistance Project AED TA in state GOSS Lessons learned cfagueret@aed.org m Gol Ayire Jal Program Officer, AED TA in state GOSS Lessons learned m Muna Shalita Global Fund Program Manager PSI Other donors mshalita@psi-sudan.org f Juliana Bol M and E Manager PSI Sudan jbol@psi-sudan.org f Dr. Jeyllani Abdullahi Technical Officer Malaria WHO Southern Sudan Role of SPS TA jeylaniabudulahi@yahoo.com m Dr. Shigeki Taniho President, TA Networking Corp. JICA JICA MOH assistance Taniho.shigeki@ta-n.com m Suzanne Bond-Hinsz Organizational Development advisor Capacity building in S.Sudan suzannebondhinsz@yahoo.com f Stephen Moore Country Director: Malaria Consortium, Southern Sudan External view of SPS program s.moore@malariaconsortium.org m David Olulong Financial Management Specialist, IMDTF LTA in MOH￾WB dojulong@gmail.com m Henry Owino Obbo Procurement Specialist LTA in MOH￾WB henryowino@yahoo.com m Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 55 ANNEX 6: RESULTS FROM KEY PARTICIPANT QUESTIONNAIRES AND ROLE OF TA SCALE A. Results from Key Participant Questionnaires In this questionnaire, the five key participants were asked to respond to the following three questions: 1. Which aspects of the SPS program have been most effective? 2. What would you change? 3. What is needed most to improve capacity within the MOH? Participants were asked to write as many or as few answers as came to mind; no subcategories were presented. Though participants were invited to rank responses, none did. Below are all the responses, clustered by subcategories but otherwise verbatim. The participatory structure of the evaluation—which included the USAID officer, an employee of the Implementing Partner from another country, a senior GOSS official, and the active participation of the Team Leader of the implementing partner—meant that all but the external evaluator were interested parties. Although relationships were very congenial and cooperative, group discussions were necessarily guarded; the anonymous and written format of the Key Participants Questionnaire allowed candid critical comments and concerns to be raised. 1. Which aspects of the SPS program have been most effective? Effectiveness with regards to MOH organizational capacity:  the bringing of MOH departments into control of the programs  the setting of a platform for scaling up interventions regardless of source of funding  the putting into place of policies, strategies, and implementation tools  that partners have bought into MOH plans and policies  that partners have come to seek strategic direction from MOH Effectiveness with regards MOH technical capacity:  the development of key guidelines, policies, tools and training key staff in focal areas  the technical support and advice given at the ministry level  the malaria technical advice  the pharmaceutical technical advice to NMCP  the pharmacy management as regards to drug regulation and supply chain management Effectiveness with regards the SPS program model of embedded LTAs:  the smooth working relations with MOH despite formal working agreements /MOH  the embedding of technical advisors within the MOH structures, so that advisors are where the action and key decisions are  the flexibility of implementation, which has been able to respond to actual needs of the MOH  how the program works through the MOH structures to implement capacity building  the placing of LTA at MOH  the response of the program to the priority needs of MOH in specific areas of the malaria control program, EPI and pharmaceutical management systems 2. What would you change? Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 56  Suggestions on role definition, counterparts involvement of some Sudanese local staff along with MSH team particularly in the area of pharmaceuticals (i.e. there should be clear-cut difference on what should be done by SPS within the Ministry and what should be done by Ministry staff). Perhaps a better definition of roles and responsibilities through e.g. a MOU so that both parties play their roles. This would ensure clear understanding; not dependent on individuals.  Suggested expansion to state level maintain the current approach but expand the TA to the states and counties if possible depending on the funding probably same approach embedded in the MOH, but with co-location at State level.  Increase field level presence e.g. consider placing people at state level with the MOH  Suggestion to increase emphasis on capacity building 3. What is needed most to improve capacity within the MOH? On roles, accountability  putting staff to accountability; each officer should have definable products to be delivered is a specific time. SPS Advisors or any other agency can then support the staff to produce the products  more collaboration specifically in technical areas between MSH and Ministry staff like has been going on in the Malaria Control Program. However, it is also of paramount importance that MSH team should feel part of the Ministry  the Ministry should clearly define the areas it wants SPS to help or assist it on. This way one can be able to measure what one has done and also check if it has been effective On resources, human and other  in terms of institutions, MOH level: need to be well-equipped with vehicles, VSATs etc.  good leadership in MOH (including appointing people on merit, technical competence, skills and experience)  recruitment of additional technical staff to support various departments at MOH  provide learning resources  improve the logistics to ease the implementation and monitoring On training  provide targeted training activities on leadership and management provide opportunities for long term professional training and regular in-service training  more training  expanding the training to the health facilities On MOH Structure  restructure the current MOH organizational structure Strengthening Pharmaceuticals Systems Lessons Learned Evaluation Report 57 B. Role of Long-term Technical Advisor The role of Long-term Technical Advisors embedded in government offices is a complex one, especially when reporting lines and evaluation responsibilities are external to office structure. The tension of taking an active role in responding to immediate demands, or standing back and guiding colleagues through a process, is a daily dynamic. A review of the job descriptions of the SPS LTAs revealed an emphasis on technical skills and know-how; however, the evaluation team was repeatedly told that the key element of success in a LTA was a ‘respectful’ and ‘patient’ personality. The MSH personnel performance review process is very participatory, including MOH supervisors, and puts considerable weight on interpersonal skills. To better capture how SPS advisors and their counterparts saw the balance between ‘mentoring’ and ‘doing,’ the evaluators developed a simple scale with which to gauge how LTAs and some of their counterparts in the MOH perceived their role. There were seven respondents—five SPS and two MOH. As perceptions and roles may change over time, respondents were asked to mark the three scales below, indicating how they perceived what percentage of their time was devoted to along a continuum from one end of the scale or the other. To capture changes over time, respondents were asked to mark their perceptions over three points in time: when they arrived (average 2007), 2010, and looking ahead to 2011. LTA Roles: The Mentoring/Operational Time Division Finding: Most LTAs and counterparts viewed a 50/50 division between time spent on mentoring and operational roles as ideal in the future. They presently reported about a 40/60 balance in their roles, changed from the earlier 30/70 split. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 58 ANNEX 7: FINDINGS, CONCLUSIONS & RECOMMENDATIONS, BY SCOPE OF WORK RESEARCH QUESTION Project Design The original SPS assumption is that highly qualified long-term technical advisors from the region, embedded within the MOH with responsibilities to model and mentor Sudanese staff, can build human capacity while establishing a solid framework of policies, procedures, practices and direct program implementation. 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered or addressed based on experience to date? Findings (F) Conclusions (C) Recommendations (R) F1. Respondents (GOSS, TA, colleagues, peers) report that the SPS advisors have played a “crucial’ and “essential” role in building the MOH over the past five years. C1. The project design and underlying (implicit) assumptions were well-suited for the capacity building needs of fragile, conflict-affected Southern Sudan over the life of the program. R1. USAID/Sudan continue active participation in the global SPS program, planning annual “buy-in” at increasing levels commensurate with the projected increasing capacity building needs of the MOH. F2. The bureaucratic flexibility built in to the SPS global program allows for incremental growth of the program, minimizing USAID Mission managerial responsibilities while responding to GOSS needs as they arise. Program managers report satisfaction in the level of MSH home office support, and MOH officials appreciate the ad hoc responsiveness of the model. C2. The MSH SPS Program takes different shapes in different countries (Kenya, Namibia) and has proven to be sufficiently nimble to respond to varied needs for TA and additional tools and support. With reported and evident excellent working relationships and high levels of mutual trust, annual work plans, targets, and priorities are negotiated with full participation of the GOSS MOH, allowing for the local ownership and responsibility crucial to success, consistent with the key elements of the Global Health Initiative. R2. USAID/Sudan, the IP (MSH) and the MOH establish priority areas of expertise and appropriate timing for additional advisors in the coming years. Likely needs include a logistics advisor and a pharmacovigilance advisor. Explore other constellations to maximize advisor exposure and access. Individual pairing and counterparts can be very fruitful, but limited. Consider assigning LTA to a team. Consider rotating LTA among departments. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 59 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered or addressed based on experience to date? F3. The training component of the design is formulated annually during the development of the work plans and is responsive to training needs and requests. Training is short-term, in-service and program-specific, and conducted primarily by the LTAs. Training could be improved. C3. Based on experience to date, a long-term training plan could improve quality and efficiency of the training. Educational objectives, regular curricular updates, training of trainers training (adult learning), remedial and refresher training, incentives for career advancement could elevate the quality of training. The end point of training should be improvement in quality of patient care. Thus it is good to consider: Training + Follow-up + Supportive supervision, so as to change behavior. R3. In coordination with the MOH Human Resource Development Department and other donors with Health Advisors (WB MDTF, JICA, WHO) MSH/SPS should develop a multi-year and multi-pronged training plan. Should additional resources be available, the menu of training opportunities should include a mix of regional long-term training, strengthening existing linkages with Sudanese pre-service training institutes, development of accreditation training, TOT training and adult learning theory, regular refresher training. Findings (F) Conclusions (C) Recommendations (R) F4. Longevity and duration: the SPS program offered LTTA multi-year contracts and determined to recruit long-term advisors from the regions who could commit to staying for a long time. All the LTA hired have stayed on; none has left early or departed at the end of a contract period. All respondents cited personality as the key element to success, above professional competence. C4. In an environment of frequent personnel changes both within the GOSS and the USAID mission, the longevity and perseverance of the SPS advisors have won them the respect and trust of colleagues and supervisors. R4. Despite their evident success to date, MSH Program managers and the GOSS MOH should develop a candidate profile, and interview protocol that will elicit the qualities identified by respondents as crucial to success as a LTA in the MOH: amenable personality, attitude, patience, humility, professional excellence, cooperativeness. F5. Counterparts. The GOSS MOH is appropriately involved in developing the SOW and recruitment and selection of LTAs. MOH Official participates in the MSH/SPS annual performance review. There is no MOU or written commitment on the part of GOSS for the assignment of appropriate counterparts and definition of roles and responsibilities. Evaluation surveys indicate a divergence of role expectation and on-the-job requirements of LTAs to be both active implementers and mentors and teachers. C5. Both the MOH officials and the LTAs would benefit from increased clarity and specificity of roles and responsibilities between and among counterparts and advisors. Regularly reviewed and explicit expectations would ease the occasionally tension around advisors as trainers/ or advisors as operational partners balance. R5. MSH and USAID managers should explore developing a MOU to better define roles and responsibilities of LTAs and their counterparts. These agreements (models are available) should provide the latitude and flexibility for LTAs to respond to changing needs and priorities, while also providing guidance and protection for LTAs from excessive gap-filling and extraneous assignments. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 60 2. What are the pros and cons of the field support mechanism in the context of Southern Sudan and the SPS Program? Findings (F) Conclusions (C) Recommendations (R) F7. USAID Washington provides central funding and oversight through the COTR Field Mission directly supervises the project through the Mission Team leader and Technical Activity Manager; and SPS Country leader oversees implementation of activities, with program oversight from SPS/ HQ Advantages: There are no lengthy processes of procurement; money is remitted directly to the implementing organization (MSH) in Washington, DC; Field support mechanism is not dependent on the local mission. Managerial burden: There is substantial involvement of USAID key staff; Budget and Work plan by COTR; Financial management by COTR/ Washington i.e. Accrual reports and pipelines; Monitoring of implementation; and helping in resolving of challenges with MoH and other partners; and following up on waivers e.g. construction; tax exemption, etc. Respondents report pros: flexible, can build gradually and incrementally, responsive, minimal managerial demands on small USAID staff. Cons: without long-term Sudan strategy, and definition of capacity, SPS is open-ended and elastic depending on availability of funding. Difficult for partners to plan on future levels and programs. C7. Based on respondents’ judgments, the pros outweigh the cons. Planning on an annual basis is not ideal, but works in the context of a rapidly maturing organization. R6. (See above). Strengthening Pharmaceuticals Systems Midterm Evaluation Report 61 3. How does the SPS capacity building approach align with best practices in capacity building in conflict-affected contexts? Findings (F) Conclusions (C) Recommendations (R) F8. Literature review of current best practices in capacity building confirm the placement of LTA embedded within government offices, sharing office space, combined with a variety of mentoring and training opportunities and approaches, including individual, pairing, small group, just-in-time training have shown greater impact than standard approaches. Other donors have LTA in the MOH with less satisfaction and “success.” C8. The SPS approach in Sudan, while slightly different from SPS TA placement elsewhere (see discussion on SPS Namibia secondment program) is appropriate for the Southern Sudanese context and aligned with current best practices in conflict-affected areas (see SPS Afghanistan) R8. MSH/SPS should review the job descriptions annually along with the personal performance reviews. Over the next five years capacity building goals and targets should be included for both the advisors and the MOH counterparts to increase opportunities for skills transfer and eventual replacement. Greater emphasis should be placed on active mentoring and training. 4. How is the role of the advisers designed, given the need to balance urgent operational needs of the ministry with mentoring, training, and technical work? Findings (F) Conclusions (C) F9. A review of the job descriptions of the five advisors, interviews, and a simple exercise of grading job responsibilities on a scale revealed a focus on technical skills and outcome measures in the job description and the personnel evaluation. The LTA described their perceptions of their jobs as averaging 60% operational and 40% capacity building. LTAs report they occasionally face interpersonal challenges and dilemmas—for instance, whether to report passive resistance and lack of cooperation to the counterpart’s supervisor. C9. The design of the roles of the LTAs based on the job descriptions emphasizes operational expertise and filling vital roles in crucial technical areas where the MOH lacks sufficient trained personnel. The mentoring role is secondary in the design. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 62 Project Impact and Sustainability 5. How satisfied is USAID, and the Ministry of Health (at the national level and the three targeted state level ministries) with the project up to this point? Findings (F) Conclusions (C) F10. Interviews with USAID, the MOH at the central level and at two state MOHs provided strong testaments to the overwhelmingly positive impact this project has had within and beyond the MOH. Unlike other organizations that set up systems parallel to the GOSS/MOH, duplicating efforts and sidelining the MOH staff, the SPS advisors work with the MOH as a team, and are very well-integrated within the MOH. SPS has done a commendable job in helping the MOH/malaria program to develop and implement policies, guidelines, and training programs. This success contributed to USAID award of the SHTP II contract to MSH to capture the synergies of sharing support, resource pools, and cooperation. Individually and together these highly skilled and accomplished advisors have met and exceeded the expectations of the funders and counterparts and are viewed as leaders and models in the donor, NGO and government communities. C10. The SPS Team leader held a key role in assisting the MOH draft the funding proposal for the GFMAT Round 7, resulting in an award of $73 million to the MOH. Additionally, the proposal for Round 10 is presently under review and likely to be funded. The MOH openly credited the SPS Team Leader for leveraging these funds for the MOH. An indicator of quality and acceptance, other donors and partners have adopted SPS materials for their programs and training. 6. What is the project’s perceived value to other Ministry of Health partners (e.g. the Global Fund)? Findings (F) Conclusions (C) F11. The evaluation team interviewed other MOH partners including UNDP, WHO, WB MDTF, UNICEF, JICA, PSI, DFID, Malaria Consortium. Without exception, these partners reported the high value added by the SPS advisors, pointing to their “lead from the rear” and support manner and approach with GOSS officials. They play important roles in coordination by supporting the establishment of malaria and pharmaceutical TWGs to serve as fora for discussion of issues and challenges. F12. Other partners have adopted material and tools developed by and for SPS. UNICEF, PSF, PSI, UNDP had funds for pharmaceutical training, but lacked materials, so used the SPS developed materials and advisors to facilitate trainings in Malakal and Rumbek. C10 (see above) Strengthening Pharmaceuticals Systems Midterm Evaluation Report 63 7. Are the current benefits likely to be sustained after the program- why and why not? Findings (F) F13. Within the four years of the program, MOH capacity has been increased by many measures: national strategies and policies are in place, regulatory legislation and increased professionalism have raised standards in pharmaceutical management; MOH has attracted other partner and donors creating a stronger human resource base. MOH has more trained and qualified staff, the organization is rationalized, and more efficient formal and informal training and skills-transfer have taken place. Project Effectiveness 9. How effective has the SPS Sudan project been in addressing capacity within the Ministry for program planning, coordination, and technical knowledge in the areas of malaria control, EPI and pharmaceutical management? Findings (F) Recommendations (R) F14. Despite the very limited size and scope of the SPS Sudan project, it has demonstrated significant impact in improving MOH capacity. The “wedge” malaria control program has been most effective and has played a key role in building systems of the MCP. Examples: Malaria newsletter, setting-up the malaria building and making available a multi-purpose board room, which is shared by MOH programs, documenting malaria program activities, extensive training and supervision, monitoring and evaluation, data and information management systems, programming, Coordinating annual commemoration of the World Malaria day celebrations, mentoring the Malaria Program Manager. Accomplishments of the PMS program and EPI are impressive, and documented elsewhere. R9. Future training: Broaden the range of training provision through SPS. With MSH assistance, develop a multi-year training plan and explore possibilities of funding long- and short-term training in the region, as well as pre- and in-service training. Examine the USAID Human and Institution Capacity Development Model and determine if elements of this approach would improve the efficiency of training. Develop a cadre of senior trainers with specialized skill in adult learning as well as technical expertise. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 64 10. Specifically, what policies, guidelines and tools developed by SPS in the areas of the National Malaria Control Program, EPI and Pharmaceutical management were adopted at the National and state levels? Findings (F) Conclusions (C) F15. A listing of the products, guidelines and tools developed by the program and adopted at the state and national levels is provided in Annex D. C12. Products, guidelines and tools were developed by the program in close cooperation with the MOH, promoting buy-in and country ownership. An indicator of quality, these materials have been widely adopted and adapted by other NGOs in the health sector and form a common platform for quality assurance in health services. 11. How well has the project been able to meet indicators and contractual targets? Findings (F) Conclusions (C) Recommendations (R) F16. According to quarterly reports, SPS has consistently met or exceeded the targets for the two key indicators: number of people trained and number of policies developed. When broken down by gender, SPS has fallen short of the targeted 28 percent female trainees, but has exceeded the overall training number by almost 10 percent. C13. Most SPS training is targeted to in￾service training of mid- and senior level personnel where women are underrepresented. Special efforts must be undertaken to assure women have equal access to the benefits of training. R10. Deliberate efforts must be made to examine and address the low representation of women training, despite the challenge of the inequity in the MOH workforce and in health service areas, and to remove obstacles to women’s full participation in training. Gender aware curriculum will address both men’s and women’s gender biases and constraints and improve gender equity and equality. 12. How efficient is the model in maximizing time and effort to build capacity of the Ministry of Health to respond effectively to the needs of the National Malaria Control Program, EPI and Pharmacy Management? Please describe techniques utilized by the model. Findings (F) Conclusions (C) F17. With the modest investment of salary and support for a small team of expert technical advisors and their support office, SPS has produced over 200 major policies, guidelines, curricula and reports. The Technical advisors, serving on multi-year contracts are regional experts who work embedded within the MOH, sharing offices with counterparts, are seen by respondents to be key players in the MOH with respect for their productivity, technical know-how and personal leadership skills. C14: The SPS Mentoring model is well suited to conflict affected Southern Sudan with very limited human resources and low institutional capacity in health services. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 65 Cross-cutting 13. Which aspects of program management demonstrate or fail to demonstrate efficient management, considering financial implications? Findings (F) Conclusions (C) F18. Evaluation surveys show that the USAID/Sudan mission exercises substantial involvement and participation, while the LWA format of the CA with MSH relieves the mission of some time-consuming oversight and managerial duties. MSH Headquarters provides “appropriate” supervision and support, and the team leader serves both as senior technical advisor and team leader with significant and growing coordination and supervisory responsibilities. F19. The decision to combine several administrative functions with the MSH SHTP II has reduced duplication and increased efficiencies. C15: The big picture financial arrangement of the mission buy-in to the global SPS, compared to alternatives is beyond the scope of this evaluation. The administrative costs of the additional layer of MSH oversight are balanced by the breadth and depth and expertise available through SPS and the global sharing of materials, information and approaches. The tie-in to regional and global networks keeps Southern Sudan linked to current practices and standards and visible participants in the Malaria, PM, and EPI communities. The regional experts have proven they adapt well to difficult circumstances and receive fair remuneration and benefits within USAID TCN scales. 14. How well has the program mainstreamed gender and other equity issues? Please consider both direct and indirect beneficiaries. Findings (F) Conclusions (C) Recommendations (R) F19. In project reports, gender-disaggregated training data was gathering after FY07, and male and female targets were established. Female participation fell below the targeted 30 percent, a reflection of the male-dominated civil service and health care cadres. The evaluation team found no evidence of gender bias or focuses in the training materials, curricula, policies. Two of the seven staff members are women, including a very senior Sudanese doctor. C16. In the context of the GOSS ambitious gender balance program, the MOH must focus attention on expanding pre-service education and training opportunities and recruitment to approach gender equity. Gender-awareness and the differential impact of policies and program on gender and other equity should permeate all levels of discussion. R11. Make gender and other equity issues an active consideration in all policy formulation and guidance. Increase gender awareness and apply gender consciousness to issues of training and recruitment, with the goal of encouraging more active female participation and improving awareness of male and female behavior norms as they affect health access and behaviors. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 66 ANNEX 8: NAMIBIA’S HUMAN CAPACITY DEVELOPMENT Introduction Like other developing countries, Namibia is experiencing severe shortages of medical, nursing, and pharmaceutical personnel, a problem that became more pronounced following the advent of HIV/AIDS. There is an enormous demand for healthcare personnel on the health system to meet the emergent needs of HIV/AIDS prevention, care and treatment programs. The pharmaceutical human capacity assessment conducted by the Ministry of Health and Social Services (MOHSS) in 2005 highlighted the short-, medium-, and long-term interventions for increasing the number and competencies of pharmaceutical human resources required to address the emerging challenges of HIV treatment scale up. Short-Term Intervention: Supporting Hiring of Staff for Critical MOHSS Pharmaceutical Service Functions In order to quickly fill critical vacant positions for supporting pharmaceutical services and ART scale up, MSH/SPS through an HR agency, recruited pharmaceutical staff into established positions within the Ministry of Health and Social Services (MOHSS). The staffing needs, job descriptions and job specifications are identified and developed by the Ministry, which supervises the entire recruitment, selection, placement and staff induction process. The seconded staffs are supervised by the MOHSS with oversight from MSH/SPS and are remunerated in compliance with the MOHSS salary structure and benefits. The Ministry’s compensation structure is deliberately applied so as to facilitate the absorption these staff into the MOHSS structure. Between June 2004 and December 2009 a total of 51 pharmaceutical staff were recruited, 57 percent (29) of whom were female and 49 percent (25) of whom were Namibians. Over 47 percent of those recruited were pharmacists assistants, 45 percent pharmacists, and 8 percent other professionals, including one medical doctor, one logistician, and two IT staff. Six senior managerial positions were filled, and nine new technical positions were created. The MOHSS has recognized the importance of these new positions and is working on a human resource restructuring exercise that will include these positions. Four (out of 13) regional pharmacist positions—Caprivi, Kavango, Kunene, and Erongo—were filled. As of February 2010, of recruited staff, 64 percent were absorbed into the MOHSS structure, 26 percent remained seconded to the MOHSS, and 10 percent had terminated their contract. About 70 percent of recruited pharmacists and 80 percent of pharmacists’ assistants were absorbed. On average, it took 10 months to absorb pharmacists’ assistants and 22 months to absorb pharmacists. Medium to Long-Term Intervention: Strengthening the National Health Training Center to Train More Pharmacist Assistants The aim of MSH/SPS’ technical assistance in supporting the NHTC was twofold: (1) to strengthen pre-service training of pharmacist assistants in order to increase the pool of qualified middle-level staff for delivery of pharmaceutical services in Namibia and (2) to facilitate the accreditation of the pharmacist’s assistant course and creation of a career path for the PA cadre in the MOHSS. Training of PAs was selected because the course was already available in country thus was cheaper and most feasible option for increasing the number of qualified personnel; the PA course takes two years to complete and thus is of shorter duration than the minimum of four years for a pharmacist course; PA positions were available in the MOHSS establishment; lower attrition rates of PAs as Strengthening Pharmaceuticals Systems Midterm Evaluation Report 67 compared to pharmacists and PAs are capable of delivering most of the routine services in the pharmacy. The intervention involved provision of equipment, infrastructural support and placement of qualified pharmacy assistant tutors to enable the increased (tripling) production of PAs. To facilitate the accreditation of PA course, MSH/SPS provided organizational and institutional development support to the NHTC including guidance on compliance with National Qualifications Authority (NQA) systems; guidance on compliance with the Namibia Qualifications Framework (NQF); accreditation procedures; career pathways and linkages for advanced tertiary training; regional recognition of PA competences; and revision of the PA curriculum to meet national and international standards Through this support, the PA enrollment at NHTC increased from eight per annum prior to 2007 to 28 in 2009. Likewise, the number of graduating PAs increased from six per annum prior to 2007 to over 18 per annum in 2010 and is expected to reach 35 per annum in 2012. Long-term Intervention: Support to the University of Namibia (UNAM) to establish a Bachelor of Pharmacy Course In 2008, MSH/SPS Namibia signed a Memorandum of Understanding (MoU) with UNAM to provide technical assistance in strengthening the pharmaceutical training programs offered by the institution, including supporting the establishment of a Bachelor of Pharmacy course. The overall aim for this agreement was to help improve the availability of competent Namibian pharmacists. This was achieved through conducting an assessment to explore the feasibility of establishing a pharmacy degree course; developing a pharmacist competency framework and curriculum for the new pharmacy course; and recruitment, placement and provision of salary and benefits for the pharmacy course coordinator and the pharmacotherapy lecturer for the first two years of the course. The first intake for this course will register to commence the program in February 2011. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 68 ANNEX 9: MAJOR SPS PRODUCTS BY FINANCIAL YEAR Major Products FY 05–06 1. Trip report for EARN meeting 2. Vector Control Needs Assessment 3. Integrated Vector Management Strategic Plan 4. Advocacy paper on “Scaling up ITN coverage in S. Sudan—options and way forward.” 5. Guidelines for distribution of ITNs through the public sector 6. Minutes of malaria Technical Working Group 7. ITN data collection tools—2007 and planned for 2008 8. ITN gap analysis 9. GF round 7 orientation workshop report—Nairobi 2007 10. Round 7 GFATM malaria proposal 11. GF round 7—CCM presentation 12. GF round 7—TRP responses 13. Guidelines for management of malaria based on the new treatment policy 14. Master plan for training health workers in 10 states 15. Training report—25 participants—Central Equatoria state 16. Training report—20 participants—Central Equatoria state 17. Training report—20 participants—Eastern Equatoria state 18. Orientation report—30 participants—Eastern Equatoria state 19. Technical presentation—Malaria Strategic Plan 20. Technical presentation—Integrated Vector Management (IVM) 21. Revised organogram for Directorate of Pharmaceutical Services (DPS) 22. Key functions and TOR for the Director General of Pharmaceutical Services 23. Summary of Pharmacy Policy presented to Parliament. 24. Three-year Pharmaceutical Master Plan (2007–2010 Short/Mid-term Work Plan) 25. Report of HIV and AIDS care, prevention and treatment workshop- Howard University 26. Technical presentations (3) at HIV and AIDS care workshop—Howard University 27. Study tour report—Tanzania Drug Management Systems 28. Rapid assessment of pharmaceutical systems—CES; EES and Jonglei State 29. NMCP office—constructed- opening function. 30. Office Manager was recruited to coordinate finances and program operations 31. Integrated operational plan for 2006/7 32. Directorate of Preventive Medicine 2007/8 health sector plans 33. NMCP technical presentation at the November 2006 EARN meeting 34. Presentation—“HMM—key considerations in implementation” 35. Presentation—USAID partners meeting 36. Terms of Reference (TORs) for a Task Force to design the HMM strategy 37. Malaria newsletter—2007 38. Africa Malaria Day (2007)—opening of NMCP offices. 39. Support supervision checklists—state; county and health facility level 40. Support supervision reports—Eastern Equatoria, Unity and Upper Nile 41. Malaria Indicator matrix 42. Malaria burden at 4 health facilities: 2003 to 2005 Major products FY 07 1. Presentation to Dr Jono on SPS Sudan program activities Strengthening Pharmaceuticals Systems Midterm Evaluation Report 69 2. Handover of vehicles—pictures/reports 3. Brief of an overview of key issues during Dr. Jono’s visit is attached 4. Training report in Ethiopia 5. Southern Sudan security guidelines 6. List of equipment procured—computers, printers etc 7. Inventory/Asset register 8. Minutes of National Health Assembly preparatory meetings 9. Minutes of malaria TWG meetings 10. Presentation at DPC workshop 11. Presentation on national ITN plan and allocation of MDTF nets 12. Presentation of an overview of the draft revised/updated TOR for malaria technical working group 13. Presentation on GFATM updates and TRP clarifications 14. Copy of 2008 Malaria Newsletter 15. Presentations 16. Lusaka EARN presentation and meeting report 17. TRP clarifications/responses 18. ITN data collection tool and updated MOH distribution plan 19. Updated LLIN national strategy 20. Implementation guide for HMM 21. Training report—18 health workers—malaria—Eastern Equatoria state 22. Training report—42 health workers—malaria—Eastern Equatoria state 23. Training report—21 health workers—malaria—Kapoeta South County, EES 24. Training report for 45 health workers–malaria—Central Equatoria state 25. Technical notes on use of AS+SP 26. SOW for malaria epidemic response teams 27. Presentation on malaria strategic plan 28. Presentation on long term ITN distribution strategy 29. Options for allocation of MDTF nets 30. Selection criteria/considerations for MDTF ITNs states 31. GFATM round 9 gap analysis tool 32. Concept paper on surveillance of malaria indicators 33. Support supervision reports Western Equatoria state and Nzara county 34. Support supervision report—Warrap state 35. TOR for M&E Officer 36. Supervision report—Lakes; Central Equatoria state; Lainya and Yei counties; Eastern Equatoria state and Magwi County and Obbo Health centre Supervision report 37. Malaria data from Warrap state 38. Presentation of malaria M&E plans made at MEASURE workshop Major Products FY 08 1. Inventory of MSH/SPS assets 2. EES supervision report/when generator was provided 3. Picture of handling USG funded ACTs and handover ceremony documentation 4. Malaria TWG minutes 5. Presentation to Senior UN Advisor 6. Trip report/Minutes of Nairobi HMM meeting 7. Malaria Newsletter for 2009 8. Presentations at ACSI meeting in Torit, EES 9. GFATM round 9 malaria proposal—assumptions; responses to clarifications, etc. 10. GFATM round 9—presentation to CCM Strengthening Pharmaceuticals Systems Midterm Evaluation Report 70 11. Trip report to Cairo meeting and Nairobi 12. Child Survival implementation guide—final copy 13. Presentations at malaria meeting for state representatives 14. Training reports—Jonglei; Western Equatoria and CES 15. Torit LLIN planning meeting—trip report 16. Guideline for malaria epidemic preparedness and response 17. Minutes of MIS planning meetings 18. MIS timeline and budgets 19. MIS questionnaires 20. MOU for MIS partner support 21. Structure and TOR for investigators 22. Briefings to MOH on implementation modalities 23. Malaria data for “sentinel” facilities 24. Data collection tool for daily availability of anti-malarials and monitoring IPTp 25. Checklist for health facility level supervision and follow up of trained health workers 26. Support supervision reports and synthesis report 27. Follow-up after training reports 28. Malaria epidemiological report 29. M&E training report 30. Presentation at GFATM M&E meeting 31. Strategic Approach for Coordinated Strengthening of Pharmaceutical Management in Southern Sudan 32. Report of WHO/UNICEF technical briefing seminar in Geneva 33. Plan of action 34. Updated work plan 35. Minutes of pharmaceutical TWG 36. Draft TOR for pharm TWG 37. Revised TOR for pharm TWG 38. Notification guidelines review comments 39. Provisional list of products from notification exercise 40. Data collection tool for the notification exercise 41. POA for the Rational Medicine Use & Capacity Building sub-committee of PTWG 42. Draft of guideline for introduction of FDCs 43. Key issues discussed at QA departmental meeting 44. Copies of printed regulatory documents—applications, certificates etc 45. Copy of the private sector training manual 46. Distribution plan & related tools for distribution of 544,000 ACT doses 47. FY08 quantification of AS/AQ and SP for USG procurement 48. Quantification of AS/AQ unmet needs 49. Pipeline summary for country 50. SOW for joint TDY with JSI/DELIVER—distribution plan 51. List of health facilities by partners 52. Delivery checklist, packing list and physical counts for 1.6 million ACT doses 53. Distribution plan for 1.6 million ACT doses 54. CMR forms submitted to UNOCHA 55. Guide for ACT distribution 56. Copies of waybills and vouchers 57. Briefs and talking points for handover ceremony 58. Notes for August Health NGO Forum 59. Distribution status report 60. Draft SOW for post-distribution M&E Strengthening Pharmaceuticals Systems Midterm Evaluation Report 71 61. PMIS printing estimates 62. Training report for 47 trainees in Torit 63. Training report 22 health workers—Juba county 64. Training report 36 health workers—CES 65. Internship report—Dr Ayuen 66. Trip report—Dr Odoy 67. Draft write-up for ACT subsidy model 68. Supervision report/generator installation report 69. SOPs for document review, physical inspection and Minilab® testing protocol. 70. Pharmaceutical Indicator Assessment tool 71. Pharmaceutical assessment tool used in health facility mapping—WBG and Lakes 72. Field trip to Eastern Equatoria state—sampling medicines for minilabs 73. Port of Entry checklist 74. Abridged tool for inspection of pharmaceutical premises 75. Concept note on Kaya minilab 76. Draft inspection report; List of facilities, key findings and actions taken also attached 77. SOW for renovation of minilab premises 78. Presentation at NGO forum—notification exercise, importation procedures and drug registration process 79. USAID Indicator Reference sheets 80. JD for EPI Technical Advisor 81. Presentation at one MSH meeting 82. Update on SPS activities to new Health Team Leader—part of COP meetings 83. Compilation of feedback of partners on essential supply list 84. Pipeline assessment tool and feedback Major products FY 09 (partial) 1. Minutes of Malaria Technical Working Group meetings 2. Joint 2010 MCP plan 3. Minutes of World Malaria Day meetings 4. SOW for review of the ACT based malaria treatment policy in Southern Sudan 5. Training report—malaria case management course for Terekeka and Juba counties 6. MIS presentation—status of MIS activities 7. Minutes of meetings with SSCSE—MIS data entry and analysis 8. SOW for data entry and analysis using SSCSE 9. SOW for data entry and analysis using Consultants and SSCSE 10. Trip/Supervision report—Malakal, Upper Nile to coordinate SHHS II activities 11. Revised ACT allocation/quantification for UNITAID/PSI—hospitals only 12. Completed CPIR form for USAID/DELIVER procurement 13. Documentation/memo to USAID on procurement of ACTs of questionable quality 14. Public sector pipelines for AS/AQ 15. Quantification of AS/AQ requirements for SHTP II 16. Training Report—pharmaceutical management; Bor, Jonglei State -: 38 participants; report also includes dissemination of guidelines and inspection of premises 17. Trip report—Eastern Equatoria State; inspection & supportive supervision. Activity was done in quarter 1. 18. Minutes of meeting with team of inspectors—to compare findings and chart a way forward 19. Inspection reports for Greater Bahr el Ghazal (4 States), Western Equatoria and Upper Nile attached 20. Minutes of meeting to kick start import verification process in South Sudan 21. Draft template of report to be used by the Import Verification Committee Strengthening Pharmaceuticals Systems Midterm Evaluation Report 72 22. Registers compiled by SPS for Proxy Marketing Authorization for Southern Sudan 23. Report—renovation works for Kaya minilab premises 24. Inspection algorithm used at Kaya port of entry 25. Training report—pharmaceutical management—Bor Jonglei: 42 participants 26. Reviewed PMIS concept note drafted by EHG 27. Minutes of EPI Technical Committee meeting 28. Minutes of EPI Interagency Coordination Committee (ICC) meeting 29. Proposed Immunization schedule for Southern Sudan 30. Concept paper on studying risk of HBV breakthrough infections in Southern Sudan 31. Trip report—Warrap SIAs mission 32. Supervision reports—19 facilities in EES 33. Health work follow-up reports 34. Draft MIS report 35. Global Fund Round 7 proposal 36. EPI supervision and monitoring guidelines 37. Presentations—USAID, EPI Cairo, NGO forum, etc. 38. Pharmaceutical and Logistics Assessment tools 39. SOW for pharmaceutical advisors 40. FDC—AS/AQ posters 41. Malaria databases 42. Malaria trend charts 43. Vaccination week chart Strengthening Pharmaceuticals Systems Midterm Evaluation Report 73 ANNEX 10: USAID INDICATOR COMPARISON CHARTS Strengthening Pharmaceuticals Systems Midterm Evaluation Report 74 ANNEX 11: SUCCESS STORIES Success Story: Establishing Structures and Roles for Pharmaceutical Directorate In 2007, staff recruited into the directorate of pharmaceutical services were not provided with job descriptions. Staffing needs and reporting structures were not clear. SPS response Working with the MOH, SPS advisors developed an organogram for the directorate, defining the reporting structures in the process. Job descriptions were written for each position on the organogram, and the Director General was advised to appoint or assign specific roles/responsibilities to individuals. Each staff member was given an assignment letter and job description. Results The directorate’s organogram was approved by the MOH Executive Board and endorsed by the Public Service Commission. Now the organogram and job descriptions form the basis for filling vacant posts in the directorate. With clear guidance, committed individuals in the directorate have taken their responsibilities with renewed commitment, and additional staff has been recruited based on the organogram. Success Story: Building Capacity of MOH Counterparts (With View to Sustainability) In 2007, the MOH did not have staff trained in pharmaceutical management, nor did they have any staff who could provide training. Training materials were also not available. SPS response SPS advisors adapted training materials used in other countries and organized trainers workshops for four states (Central Equatoria, Eastern Equatoria, Western Equatoria, Jonglei), with the participation of MOH/GOSS (central level) staff. SPS involved partners in this TOT, and shared the materials and tools with them. Results The directorate now has a core team of trainers at both central and state levels that can organize and conduct trainings independently. Trainers now have confidence in preparing and producing public presentations. Dr. Neni Daniel, the new SPS program associate and a Sudanese national, was one of those trained in the TOT and now supports the SPS program in various pharmaceutical activities, including training. Partners like PSF/UNDP, Global Fund, Malaria Consortium, and PSI have adopted and used the training materials for training at health facilities in their areas of operations; such trainings include MOH counterparts so as to provide them with opportunities for improving their training capacity. Strengthening Pharmaceuticals Systems Midterm Evaluation Report 75 ANNEX 12: REFERENCES ACTIONAID (2006). “Real Aid: Making Technical Assistance Work” Johannesburg: Action Aid. Haugen, V and Tilson, T. (2009) Midterm Evaluation of the Southern Sudan Technical Advisors Program (SSTAP). Management Systems International: Juba, Southern Sudan. LAND, T., HAUCK, V., & BASER, H. (2009). “Aid Effectiveness and the Provision of TA Personnel: Improving Practice,” Policy Management Brief, Brussels: European Center for Development Policy Management. ITAD & CHANNEL RESEARCH (2010). “Multi-Donor Evaluation of Support to Conflict Prevention and Peace-Building Activity in Southern Sudan since 2005 Draft Final Report” EU Award Procedure 2009/S 101–146143/NL (Ministry of Foreign Affairs, The Netherlands) MAWA, S., (2008). “NMCP/MSH Supervision Visit To Eastern Equatoria State,” December 3–4, 2008 MSH (2007). Strengthening Pharmaceutical Systems (SPS), RFA No.: USAID M/OAA/GH/HSR-07–082 Southern Sudan Ministry of Health (2010) Essential Drugs Distribution Plan for Ministry of Health/Multi-Donor Trust Fund. Directorate of Pharmaceuticals: Juba, Southern Sudan Southern Sudan Ministry of Health (2010) Expanded Program on Immunizations Program Monitoring. Director of EPI and Community-based Health Services: Juba, Southern Sudan. Southern Sudan Ministry of Health (2006) Southern Sudan Pharmacy Policy. Ministry of Health: Juba, Southern Sudan. Southern Sudan Ministry of Health (2008) Basic Package of Health and Nutrition Services for Southern Sudan. Ministry of Health: Juba, Southern Sudan. Suzanne Bonds Hinsz “Government of Southern Sudan, Strategic Capacity Building Study,” July 2010. Management Systems International: Juba, S Sudan. USAID/Sudan. (2005) Strategy Statement 2006–2008. USAID USAID (2010). “Human and Institutional Capacity Development Handbook,” Washington DC: United States Agency for International Development. SPS Work Plans, Quarterly/Annual Reports Referenced:  Annual Report 2008  SPS Work plan 2008  MSH SPS Q1 Jan 2008  MSH SPS Q2 March 2008  MSH SPS Q3 Jun 2008  MSH SPS Q4 Sept 2008  SPS Work plan 2009  MSH SPS Q1 Jan 2009  MSH SPS Q2 March 2009  MSH SPS Q3 Jun 2009 Strengthening Pharmaceuticals Systems Midterm Evaluation Report 76  MSH SPS Q4 Sept 2009  Annual Report 2009  SPS Work plan 2010 RPM Documents  MSH RPM Q1 report 06  MSH RPM Q2 report 2007 (and monthly report—March)  MSH RPM Q3 2007  MSH RPM Q4 2007  Annual Report 07 Technical Reports/Studies/Field Visit Reports  Draft Malaria joined plan April 2010  Private sector Pharmaceutical training report Oct 2009  Southern Sudan Malaria Strategic Plan—near final (2006–2011)  Final IVM Strategic Framework  ITN Strategy  Child Survival Guidelines Health Portfolio Documents  Health FY 09 OP and datasheet