Midterm Project Evaluation of The USAID | DELIVER PROJECT Final Report December 2008 This document was submitted by The Mitchell Group, Inc. 1816 11TH Street, NW Washington, DC 20001, USA Telephone: (202) 745-1919 Facsimile: (202) 234-1697 to The United States Agency for International Development/Ethiopia under USAID Contract 663-C-00-08-00409-00 Disclaimer: The authors’ views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the United States Government. Midterm Project Evaluation of Contract No. GPO-1-01-06-00007-00 The USAID | DELIVER PROJECT: Ethiopia Final Report CONTENTS Acknowledgements………………………………………………………… I Acronyms……………………………………………………………………. II Executive Summary………………………………………………………… III INTRODUCTION……………………………………………………………. 1 Purpose………………………………………………………………. 1 Background………………………………………………………….. 1 Methodology…………………………………………………………. 3 Limitations…………………………………………………………… 4 ORGANIZATION OF THIS REPORT……………………………………… 5 FINDINGS AND RECOMMENDATIONS…………………………………. 7 1. The Ethiopian Contraceptive Logistics System (ECLS) ………… 7 The Program Context ……………………………………………… 7 Program Challenges and Constraints…………………………..... 8 Progress in Achieving Results against Stated Objectives……… 8 Conclusions………………………………………………………..… 12 Lessons Learned……………………………………………………. 13 Recommendations………………………………………………….. 14 2. Contraceptive Security (CS) ……………………………………….... 14 The Programmatic Context………………………………………… 14 Program Challenges and Constraints…………………………….. 14 Progress in Achieving Results against Stated Objectives………. 15 Conclusions……………………………………………………..……. 18 Lessons Learned………………………………………………..…… 18 Recommendations………………………………………………..….. 18 3. Transition to the Pharmaceutical Logistics Master Plan (PLMP)… 19 The Programmatic Context…………………………………..……… 19 Program Challenges and Constraints……………………..……….. 22 Achievements To Date…………………………………………..…… 23 Future Directions…………………………………………………....... 25 Conclusions……………………………………………………………. 25 4. Project Staffing and Management……………………………………… 25 Findings………………………………………………………………… 25 Conclusions……………………………………………………………. 27 Recommendations……………………………………………………. 27 RE-CAP OF LESSONS LEARNED…………………………………………. 28 PRIORITIZED SUMMARY OF RECOMMENDATIONS……………….….. 28 ANNEXES A. Evaluation Statement of Work B. Evaluation Work Plan C. List of Persons Interviewed D. List of Documents Reviewed Acknowledgements The authors wish to acknowledge the support given to this activity by the many Federal Ministry of Health, Regional Health Bureaus, donor and cooperating agency personnel listed in Annex C. All provided invaluable insights into the programmatic and technical evolution of health commodities logistics systems in Ethiopia. In particular, thanks are due to Ms. Sophia Brewer, the USAID/Ethiopia Evaluation Coordinator, who provided careful coordination and technical guidance in scheduling of appointments, and to the USAID Mission and Health Office team for their valued inputs and attention during several lengthy Mission entry and exit briefings. Thanks also to the USAID│DELIVER PROJECT Team in Ethiopia who, in spite of being under scrutiny in this external Evaluation, were open, cooperative, and informative. Senior management spent many briefing hours with our Evaluation Team, facilitated scheduling and introduction to stakeholders, and provided efficient logistical backup for site visits to activities supported by the USAID│DELIVER PROJECT in four regions of the country. We sincerely hope that the evaluation exercise and this Report will assist the efforts of the Federal Government of Ethiopia, USAID, the USAID│DELIVER PROJECT, and other supporting partners in realizing their objectives for an effective health commodities supply system in Ethiopia. i Acronyms AIDS Acquired immunodeficiency syndrome BPR Business Process Reengineering BPS Protection of Basic Services CPR Contraceptive prevalence rate CS Contraceptive Security DfID Department for International Development (United Kingdom) ECLS Ethiopian Contraceptive Logistics System ETB Ethiopian Bir (currency) EU European Union FGOE Federal Government of Ethiopia FMOH Federal Ministry of Health (central Ministry of Health) FPTWG Family Planning Technical Working Group HCSS Health Commodities Supply System (same as PLMP) HEW Health Extension Worker HIV Human immunodeficiency virus HSEP Health Services Extension Program IST Implementation Support Team JSI John Snow, Inc. LIAT Logistics Indicators Assessment Tool LMIS Logistics Management Information System M&E Monitoring and evaluation MOH Ministry of Health OJT On-Job Training PFSA Pharmaceutical Fund and Supply Agency PLMP Pharmaceutical Logistics Master Plan PSLD Pharmaceutical Supply and Logistics Department RHB Regional Health Bureau SCMS Supply Chain Management System Project SIDA Swedish International Development Agency ii SNNP Southern Nations, Nationalities and People's Region of Ethiopia) SOP Standard operating procedure TWG Technical working group UNFPA United Nations Population Fund UNICEF United Nations Children's Fund USAID United States Agency for International Development WHO World Health Organization iii Executive Summary An Evaluation Team provided by The Mitchell Group undertook a Midterm Evaluation of the USAID │ DELIVER PROJECT Ethiopia from October 29 through November 28, 2008. The Project in Ethiopia (also referred to in the full Evaluation Report as, "DELIVER II"), is the country program for the USAID│DELIVER PROJECT, a five-year global project being implemented by John Snow, Inc. (JSI) during the period 2006-2011. The global USAID│DELIVER PROJECT is charged with: improving essential health commodity supply chains by strengthening logistics management information systems (LMIS), streamlining distribution systems, identifying financial resources for procurement and supply chain operation, and enhancing forecasting and procurement planning. USAID/Ethiopia provides approximately $3 million in annual funding to the Project in Ethiopia for policy advocacy and technical assistance in development of a health commodities logistics system, with special attention to family planning commodities. The evaluation focused on the Project's three main objectives: 1) improving the Ethiopian Contraceptive Logistics System (ECLS); 2) improving the availability of contraceptives at all levels of the public health system; and 3), assisting design and implementation of the new Pharmaceutical Logistics Master Plan (PLMP), a Federal Ministry of Health initiative that establishes a semi-autonomous agency in Ethiopia responsible for managing, financing, logistics and supply of all health commodities (including contraceptives and all other special purpose and essential drugs). The Evaluation Team was requested by USAID/Ethiopia to, • Examine the Project’s progress in meeting its stated objectives, and describe what results have been achieved during the period October 2006 to November 2008. • Identify strengths and weaknesses of the Project; describe what was done well and what should be improved in the future. • Determine the need for future investments by recognizing existing or additional Project approaches and activities that lead to improved availability of family planning and other essential health commodities. • Briefly review the efficiency and effectiveness of the Project's staffing, management structures, and processes. The Evaluation was not a rigorous scientific examination. USAID/Ethiopia requested a qualitative assessment of the usefulness and effectiveness of the Project's approaches and activities in achieving stated objectives and planned results. Formulation of findings and recommendations relied heavily upon the professional and technical judgments of the Evaluators. The Evaluation Team faced two notable limitations in gathering information and formulation of recommendations. Because of the constraints of time and travel, the team only visited sites and public health facilities in 4 of the country’s 11 regions. Also, because of a major on-going re￾organization process within the Ethiopian Government, the Evaluation Team spent limited time with federal-level officials concerned with drug logistics management and received limited input regarding their understanding of future roles of this area of assistance. iv The Evaluation Team found that during the period under review, the Project made progress in achieving its objective of improving the ECLS. The Project completed roll-out of the ECLS with training of 4,000 health workers in 11 regions of the country (bringing the total trained by the Project and its predecessor DELIVER Project to a total of 10,000 health workers in Ethiopia with training in drug logistics management). The Project additionally developed and installed a manual contraceptive Logistics Management Information System (LMIS) in the public sector. From 2006 to 2008, the Project provided assistance to warehouses and stores in various regions of the country, resulting in visible improvements in physical conditions and inventory management. Overall, the Project has laid important groundwork for introduction of the next generation health commodities logistics system under the PLMP. However, it was also found that the ECLS and LMIS are not optimally effective, due in part to the fact that these systems operate in an institutional environment that has many other management deficiencies. It was also observed that the LMIS has not had significant impact on monitoring of stock levels at clinics that operate within hospitals and at health posts. Further, the level of monitoring and supportive supervision by the Project in conjunction with Regional Health Bureaus has not been sufficient to improve ECLS practices to the extent needed, particularly at lower service delivery levels. Another concern is that significant losses have occurred in the numbers of health workers trained in ECLS, and the Project has not yet identified or implemented effective approaches to counteract this problem. The Project also made progress in the area of improving the availability of contraceptives. The Project has substantially influenced and improved policy level awareness of the need to have a constant, reliable supply of contraceptives in order to reach family planning targets. The Project has played an important role in fostering cooperation and coordination among family planning partners on financing and procurement of contraceptives. As a result the supply of contraceptives in-country has improved, noting that there were no stock-outs of any contraceptive at the central level in 2008. However, problems still exist: stock outs and over￾stocking still occur at some health centers and below, and the national mechanisms established to date to ensure long-term availability of contraceptives in the public health system are fragile and vulnerable to the changes that might occur(particularly in the areas of financing and procurement under the new PLMP). As it relates to the new PLMP, the Evaluation Team found that although the Project played a key role in conceptualizing and designing the new PLMP, the Project's assistance role in the next phase of PLMP needs to be clarified. There is need for the Project, in conjunction with USAID, to jointly plan and reach agreement with other USAID-funded partners on possible technical roles each might undertake to support implementation of the PLMP They must also thoroughly discuss and confirm these understandings with the government's implementer, PFSA. In the area of project management, the Evaluation Team found that the Project's staffing and management structures appear to be fundamentally sound. No problems could be detected by the Evaluation Team during its brief review of project operations at its central office in Addis Ababa and field offices in 4 regions. The Evaluation Team observed, however, that there may be opportunities for further staff development and team building that would benefit the professional v development of local staff, enhance Project capabilities, and additionally contribute to the pool of well-trained drug logistics management expertise in the country. There were several approaches undertaken by the Project that contributed to its achievements over the past 2 ½ years. The Project developed and maintained good working understanding at policy levels in the FMOH, and maintained good supportive working relationships with Regional Health Bureau implementers. The Project is recognized as having been instrumental in improving donor financing and procurement of contraceptives through its advocacy and coordination efforts. It has substantially contributed to improved contraceptive forecasting procedures and the use of forecasting for donor financing and procurement decision-making. The Evaluation Team also observed several approaches that could be improved in the future. The Project should consider further prioritization of hospital and health center upgrades. Stores should be upgraded based on geographic concentrations within regions. This will increase the impact of stores upgrading on performance of the ECLS and the health commodities logistics system. The Project might also consider certain innovations to improve the efficiency of scarce resources such as assisting RHBs with more systematic planning and the coordination and control of limited transport resources. The Evaluation Team has offered several recommendations. Recommendations, in order of priority, are summarized as follows: 1) The Project should continue its advocacy and coordinative roles with family planning partners, affirm its working mandate with the Federal Ministry of Health (FMOH)/Family Health Department and Regional Health Bureaus, and continue its strategic and technical assistance activities aimed at monitoring and improving the Ethiopian Contraceptive Logistics System and Contraceptive Security in the public sector. Specifically, until such time as various components of the new PLMP are functional and can absorb responsibility for contraceptive supplies in the public sector, it is recommended that the Project continue, and strengthen the extent possible, it's advocacy for contraceptive financing and procurement; technical support for hospital and health center stores (with special attention to contraceptive supplies); monitoring and supportive supervision of contraceptive logistics management at health facility levels; and contraceptive forecasting. 2) The Project should ensure that lessons learned from its experiences in developing the ECLS and CS are incorporated into any assistance the Project might provide in design, development, and implementation of the PLMP. 3) As a matter of urgency, DELIVER II and the USAID-funded Supply Chain Management System (SCMS) should, in conjunction with USAID, review their technical scopes of work related to assistance to the PFSA and speedily reach agreement between the two projects on complimentarity and avoidance of areas of potential programmatic or technical overlap. 4) At the earliest opportunity following resolution of the above issue, the Project should enter into discussions and reach agreement with PFSA on the designated areas of technical vi vii assistance in implementation of the PLMP that the Project will be expected to provide. These discussions should be convened with assistance of the FMOH and USAID, if necessary. Agreements should be formalized and in writing to the extent permitted by standard USAID project management practices. 5) Based on agreements reached between the Project and PFSA, the Project should review(and if necessary, modify) its FY 2008/09 Work Plan and proceed with an agreed upon portfolio of activities for all health commodities that might include: upgrading stores at hospital and health centers; further design and development of an automated health commodities LMIS; and training/technical assistance in forecasting methods (in addition to contraceptives), for non-HIV/AIDS essential health commodities. 6) The Project should initiate additional in-house technical training for its Logistics Officers. INTRODUCTION Purpose The Evaluation Team was requested to, • Examine progress of the USAID│DELIVER PROJECT in Ethiopia (DELIVER II)1 in meeting its stated objectives, and describe what results have been achieved during the period October 2006 to November 2008. • Identify strengths and weaknesses of DELIVER II's, and describe what was done well and what might be improved in the future. • Determine existing or additional DELIVER II approaches and activities that warrant future investments because they are likely to lead to improved availability of family planning and other essential health commodities. Background DELIVER II in Ethiopia is the country program for the USAID│DELIVER PROJECT, a five￾year global project (2006-2011) implemented by John Snow, Inc. (JSI) and its subcontractor partners.2 The global USAID│DELIVER PROJECT is charged with: improving essential health commodity supply chains by strengthening logistics management information systems, streamlining distribution systems, identifying financial resources for procurement and supply chain operation, and enhancing forecasting and procurement planning.3 USAID/Ethiopia provides approximately $3 million in annual funding to DELIVER II for policy advocacy and technical assistance in development of a health commodities logistics system, with special attention to family planning commodities. From 2001 to 2006, the DELIVER Project (predecessor to DELIVER II), undertook a range of programmatic and technical interventions in collaboration with the Federal Ministry of Health (FMOH) and other partners to, “strengthen the supply chain through support of innovative integrated system design, contraceptive security, data quality improvements, capacity building, resources and policy advocacy.” From 2003-2006, the DELIVER Project concentrated on design and implementation of the Ethiopian Contraceptive Logistics System (ECLS) for the purpose of ensuring that contraceptives are readily available in public sector health facilities. During this period, the DELIVER Project made substantial progress in designing the ECLS Logistics Management Information System (LMIS), establishing Standard Operating Procedures (SOPs), developing training curricula and manuals, transferring logistics information and inventory control skills, and building logistics management, training and supervisory capacity in the FMOH system. Up to 2006, the DELIVER Project had trained over 6,000 MOH staff in 1 In this Report, the USAID│DELIVER PROJECT in Ethiopia is also referred to as, "DELIVER II". To distinguish DELIVER II from its predecessor, logistics management activities undertaken by the implementer prior to start of DELIVER II in 2006, are referred to as the, "DELIVER Project". 2 John Snow, Inc. subcontractors: Program for Appropriate Technology in Health (PATH), Crown Agents Consultancy, Inc., Abt Associates, Inc., Fuel Logistics Group, Ltd., UPS Supply Chain Solutions, Family Health International (FHI), The Manoff Group, Inc., and 3i Infotech. 3 "USAID│DELIVER PROJECT, Task Order I Annual Report", January 2008. contraceptive logistics management and over 700 MOH supervisors had been trained in logistics data management. These early systems developments undertaken by the DELIVER Project paid good dividends in terms of significantly improving the reliable supply and availability of contraceptives, improving institutional awareness and systems development capacity in the area of logistics management, and most significantly, serving as a catalyst for policy change in the public health sector in the area of health commodities logistics management. Although availability of contraceptives in the public sector had improved during the period 2001-2006, due in large part to efforts of the DELIVER Project4 , the public sector was generally plagued with chronic crises in financing, procurement and distribution of essential health drugs and commodities. This situation coupled with policy pressures to vastly improve and expand the quality and availability of public health care (and the possibilities of improving the availability of health commodities as demonstrated by the DELIVER Project), prompted the FMOH to take action. As a consequence, in 2006 the FMOH requested UNICEF, with technical leadership provided by DELIVER II, to steer the conceptualization, design and development of the watershed "health commodities supply system." The resulting design, later renamed the Pharmaceutical Logistics Master Plan (PLMP) by the FMOH, envisions a health commodities logistics system that would provide, "A constant and uninterrupted supply of vital and essential health commodities for the end users of all public health facilities" The objective of the system would be, "…to ensure that vital and essential drugs and health commodities of approved quality will be readily available to public health sector health facilities, for use in the prevention, diagnosis, and treatment of priority health problems, in adequate quantities and at the lowest possible cost." The PLMP describes 11 specific objectives (outputs) that encompass Policy & Legislation, Governance, Coordination & Harmonization, Financing, Selection, Quantification/forecasting, LMIS, Procurement, Storage & inventory control, Distribution/transport, Rational Drug Use, Human resources management, Research & Development, and Monitoring & evaluation.5 Given DELIVER II's strategic role in design of the PLMP, and DELIVER II's long term experience in logistics systems development in Ethiopia, the FMOH and DELIVER II agreed upon selected areas where DELIVER II would take a technical lead in implementation of the PLMP. These understandings are reflected in DELIVER II's key planning document, “Project Objectives and Strategic Direction (2006-2011)”6 and have been incorporated into USAID│DELIVER PROJECT Annual Work Plans and funding authorizations. DELIVER II's stated project objectives are to: • Improve the Ethiopian Contraceptive Logistics System (ECLS). • Improve Contraceptive Security in the public sector (CS). 4 "Logistics Indicators Assessment Tool (LIAT), Final Report", 2006. 5 "Pharmaceutical Logistics Master Plan Summary", Federal Ministry of Health, Ethiopia, Revised, October 2007. 6 Revised, July 2008. 2 • Assist in design and implementation of the Pharmaceutical Logistics Master Plan (PLMP) - Take lead in design and implementation of the PLMP Logistics Management Information System (LMIS); - Provide assistance in non-HIV/AIDS forecasting7 ; and - Improve stores management at hospitals and health centers. Methodology Scope of the Evaluation This was not a rigorous scientific evaluation. USAID/Ethiopia requested a qualitative assessment of DELIVER II that required evaluative judgment regarding the usefulness and effectiveness of DELIVER II's approaches and activities in achieving stated objectives and planned results. The Statement of Work for this Evaluation is included in Annex A of this report. The team’s Evaluation Work Plan is also included as Annex B. Data Gathering Relevant reports of national and international agencies, surveys, and monitoring reports as well as an internal project self-evaluation8 were reviewed with the intention of extracting general impressions. Information was accepted as given, and no attempt was made by the Evaluation Team to further validate data and findings as reported (as for example, number of workshops held, or persons trained, etc.). Annex D lists the documents reviewed. A small, convenient sample of 19 public health facilities was visited. These included: • 3 Regional Bureaus: Addis Ababa Health Bureau, Dire Dawa Health Bureau, and Harari Health Bureau. • 4 Regional Warehouses: Addis Ababa, Dire Dawa, Harari, and SNNP. • 1 Zonal Warehouse: Gedowo Zonal Health Bureau • 2 Woreda Bureaus: Bole Woreda Health Bureau, and Wondo Genet Woreda Health Office • 9 Hospitals and Health Center Stores: Bole Woreda 17 Health Center; Addis Ababa Region; Chora Hospital; Dire Dawa Region; Sabian Health Center; Dire Dawa Region; Legahre Health Center; Dire Dawa Region; Hiwot Fana Hospital; Harari Region; Hassenge Health Center; Harari Region; Gedowo Hospital; Gedowo Zone; Mesenkela Health Center; Sidama Zone; and Awassa Health Center; Sidama Zone. During site visits, observations were made on warehouse/store conditions and commodity arrangements, recordkeeping practices (such as spot checking of stock cards, bin cards, and service registers), and Logistics Reports. Contraceptive supply management procedures were 7 At the time of the Midterm Evaluation in November 2008, DELIVER II had not yet expanded beyond contraceptive forecasting activities to all non-HIV/AIDS commodities. See further discussion of this issue in Section 3.4.4 of this Report. 8 "USAID│DELIVER PROJECT Ethiopia Mid-term Self-evaluation", November 2008. 3 assessed for downward distribution, upward and downward communications flow, and storage suitability and capacity. Stock levels and perceived availability of contraceptive supplies were discussed anecdotally with warehouse and stores managers. During discussions with Regional Health Bureau managers, extensive discussions centered on past experiences and on-going relationships with DELIVER II; the project's perceived value/impact to-date, and assistance from DELIVER II that will be needed in the future. Information gathered from stakeholders was drawn from meetings and discussions. Open-ended, non-structured questions were agreed upon by the Evaluation Team and used for stakeholder interviews. Although structured by pre-determined questions, responses of stakeholder informants were generally impressionistic, opinion-oriented, and therefore not quantitative. Annex C lists the persons interviewed. Data Synthesis The Evaluation Team held discussions at the end of each work day to compare impressions and agree upon issues that may require further probing. Each Saturday during the evaluation period was reserved for a full-day session in which Key Evaluation Questions used to determine whether the Evaluation Team was able to answer and elaborate upon issues. During those sessions, a format for anticipated verbal debriefings, as well as an outline for the final report, was developed to ensure that the Evaluators remained focused on evaluation end-points. Conclusions were reached on all elements of the Evaluation through extensive discussion and consensus. Formulation of findings and recommendations relied heavily upon the professional and technical judgments of the Evaluators. Limitations The Evaluation Team faced two notable limitations in gathering information and formulation of recommendations: 1) Because of the constraints of time and travel, the team was only able to visit public health facilities in four of the country’s eleven regions. The sample of 19 sites visited was too small to be significant. It is not possible, therefore, to generalize from observations at sites visited to the country as a whole. While the evaluation team’s impressions regarding the impact of DELIVER II’s technical assistance in the field was generally positive, several informants have suggested that contraceptive stock levels and the overall performance of the ECLS may be more problematic in some regions and Zones than in others. The possible unevenness of the project’s impact across regions could not have been investigated in more detail. 2) Much of the evaluation is concerned with the future direction of DELIVER II, and it would therefore have been desirable to get a frank reading from FMOH decision-makers on its expectations for the project in the coming two years. Unfortunately, the Evaluation Team was only able to hold brief and limited discussions with relevant FMOH officials about DELIVER II. The Evaluation Team understood that this limited access to relevant FMOH officials was the result of their full-time participation in the Government restructuring process. The 4 FMOH, along with other ministries of Government, has been engaged in a major restructuring exercise for the past year. This restructuring process, referred to as Business Process Reengineering (BPR), evidently consumes the full attention of senior FMOH officials, to the extent that donors and the FMOH's implementing partners have limited access, in real time, to the FMOH. Throughout the evaluation, the BPR was frequently mentioned by donors, implementing partners and Regional MOH managers as a constraint to conducting regular business with the FMOH. In spite of this situation, The Mitchell Group was able to arrange brief discussions with two key government officials9 who are relevant to FGOE drug logistics management. Those discussions were useful but quite generic in the information provided. The substance of the information received is incorporated into findings discussed in the PLMP section of this Report. The Evaluation Team felt that there was some reluctance on the part of both officers interviewed10 to openly discuss PFSA's future technical assistance requirements in the area of drug logistics management because this would likely have implications for future program agreements between the FMOH and implementing partners such as DELIVER II, and may also be procurement-sensitive. In consequence, the recommendations of the Evaluation Team do not have the benefit of substantial input from the FMOH regarding the future direction of DELIVER II. ORGANIZATION OF THIS REPORT The body of this Report, the FINDINGS AND RECOMMENDATIONS Section, is organized according to DELIVER II's key objectives. Section 1, covers DELIVER II's activities concerned with implementation and strengthening the Ethiopian Contraceptive Logistics System (ECLS). Section 2 covers DELIVER II's activities related to improving Contraceptive Security (CS) in Ethiopia. Section 3 is concerned with DELIVER II's activities intended to assist with design and implementation of the new Pharmaceutical Logistics Master Plan (PLMP) for all health commodities. Section 4 is cross-cutting and discusses DELIVER II's staffing and management structures. Product Selection Financing Procurement Warehousing Distribution LMIS Availability & Use at Facility Level Monitoring & Supervision Forecasting Health Commodities Logistics Cycle 9 The FMOH Head of the Pharmaceutical Supply and Logistics Department, and the Deputy Director for Procurement at the Pharmaceutical Fund and Supply Agency. 10 Each officer was interviewed separately during coffee break at a BPR Conference – the only time that could be made available to the Evaluation Team. 5 Each of the three project objectives and its applicable section in this Report is concerned with one or more elements of a health commodities logistics system. Elements of a health commodities logistics system generally include: Product selection, Financing, Procurement, Warehousing & Storage, Distribution & Transport, Availability & se, Logistics Management Information System (consumption-based data), Monitoring & Supportive Supervision, and Forecasting. These elements, or logistics sub-systems, are interrelated. Their relationships are often shown in the technical literature as a "logistics cycle,"11 as seen in the adjacent diagram. Each of the FINDINGS AND RECOMMENDATIONS sections of this Report discusses one or more logistics subsystems. Specifically, discussion of DELIVER II activities in the areas of Warehousing, LMIS and Monitoring and Supervision are covered in Section 1, ECLS. Discussion of DELIVER II activities in the areas of Financing, Procurement, Distribution, Availability and use of commodities, and Forecasting, are covered in Section 2, CS. In Section 3, only those elements with which DELIVER II might be concerned during transition to the PLMP in the coming two years are discussed. They are: Storage, Availability & Use, LMIS, Monitoring & Supportive Supervision, and Forecasting. Organization of this Report is further summarized in the following chart: Elements/ Sub-systems Section 1. (ECLS) Section 2. (CS) Section 3. (PLMP) Product Selection12 Financing X X (contraceptives only) Procurement X X (contraceptives only) Warehousing & Storage X (Regional, zonal, woreda warehouses, and hospital, health center stores) X Hospital/health center stores, all commodities) Distribution & Transportation X Availability & Use X X (contraceptives only) LMIS (design and implementation through training) X (contraceptives only) X (all health commodities) Monitoring & Supportive Supervision X (contraceptives only) X (contraceptives only) Forecasting X X (contraceptives only) 11 In one refinement of the logistics "cycle" concept, the LMIS sub-system is sometimes shown as the hub of the system; in others, distribution includes warehousing and transport. 12 Product selection is managed separately from other elements in the logistics cycle under direction of central Ministry of Health policy makers and the Ethiopian Drug Administration and Control Authority (DACA). 6 FINDINGS AND RECOMMENDATIONS 1. The Ethiopian Contraceptive Logistics System (ECLS) 1.1 The Programmatic Context Prior to 2003, the drug and health commodities logistics system in Ethiopia, including contraceptives logistics management, was not very effective. There were no standardized procedures for distribution of commodities, and warehouses were poorly designed and poorly maintained. Inventory control and stock management practices were not institutionalized and there were chronic shortages and stock-outs, as well as over-stocking and expired stock. Except for a few vertical programs such as TB & Leprosy and immunizations, there was no systematic collection of data about drug stock levels, consumption, or re-supply requirements. Lack of a reliable supply of contraceptive commodities was particularly troublesome because of the constraint this placed on the FMOH's attempts to mount an aggressive family planning program. As early as 2001, a Logistics Indicators Assessment Tool (LIAT) study had determined that there was a critical need to improve contraceptive logistics and supplies in the country. In addition, lack of contraceptive consumption data hindered the ability of the FMOH to estimate future commodities needs and maintain funding commitments from donors. As described by more than one donor partner13, prior to 2005/2006, family planning donors had little confidence in contraceptive commodities forecasting data. Some donors developed their own alternatives for projecting donations, while others remained reluctant to commit resources based on poor planning. It was within the above described context that the DELIVER Project in Ethiopia was requested by the FMOH to assist in design and implementation of an Ethiopian Contraceptive Logistics System (ECLS) for the purpose of ensuring that contraceptives are readily available to clients of public sector health facilities. The ECLS was designed in 2003 A pilot phase was completed in four regions (Amhara, Tigray, SNNP, and Oromia) and Addis Ababa in 2004 and in 2005, ECLS implementation was rolled-out to other regions, Zones and cities. By the end of 2006, 6,000 MOH staff had been trained in contraceptive logistics management, over 700 supervisors had been trained in logistics data management, and service providers were routinely collecting and reporting contraceptive consumption data. During 2003-2006, the ECLS began to function and show improvements in contraceptive availability at all levels of the system. Institutional logistics management capacity was built in the public sector, and important groundwork was laid for the FMOH to begin considering how to tackle the enormous task of developing a functional logistics management system for all drugs and health commodities. 13 A point made in discussions with UNFPA and UNICEF, both of whom have historically been involved with brokering cooperation among donors for support of the FMOH's MCH and FP programs. 7 1.2 Program Challenges and Constraints DELIVER II is faced with several challenges related to ECLS: 1) To complete roll-out of ECLS, and to continue improving the functionality and performance of the system; and, 2) To assist the FGOE in expanding beyond the ECLS to design and implementation of a logistics management system for all drugs and health commodities. A constraint to completing roll-out of the ECLS is the frequently mentioned problem of high staff turnover and attrition of health workers trained in ECLS. By one FMOH officer's estimate, as much as 40 percent of previously trained workers are no longer in the positions for which they were trained in ECLS. Some workers rotate; others leave the system altogether. Another constraint to on-going systems strengthening and improving the performance of ECLS is the fact that DELIVER II's developmental approach is to capacitate the Ministry rather than operating a system for the Ministry. By design, ECLS operates within an institutional setting (i.e., the public health system) that has itself considerable managerial and infrastructural weaknesses. The functionality and performance of any management tool will only be as good as the management environment in which it operates. At best, performance of the ECLS can be expected to be as uneven as was observed in Evaluation site visits. The condition of LMIS data and the logistics knowledge of health workers in some sites visited were impressive, and very limited in others. Assisting the FGOE in expanding the logistics system beyond contraceptives to cover all drug and health commodities presents DELIVER II with both a challenge and an opportunity. The challenge lay in designing an automated Logistics Management Information System and logistics management processes that must be introduced and work in a far more complex management environment than was required for ECLS. The new health commodities LMIS will need to be developed and implemented in an environment that involves inter-ministerial cooperation and harmonization, re-organization of logistics management authorities, and redefined roles of supporting agencies and partners. There is an opportunity for DELIVER II to build from the ECLS experience and take the technical lead in conceptualizing and designing the new health commodities supply system. 1.3 Progress in Achieving Results against Stated Objectives DELIVER II's stated objective for ECLS is: To improve the Ethiopian Contraceptive Logistics System. The intent of the objective, and the result sought is, "improvement." DELIVER II's progress in improving the ECLS is discussed below in 3 areas of the health commodities logistics cycle: Warehousing & Storage, LMIS, and Monitoring & Supportive Supervision. 1.3.1 Warehousing & Storage From October 2006 to August 2008, DELIVER II had completed 80 assessments to determine technical support needs of warehouses and/or facility stores and regional, zonal, hospital, and health center levels. Five Regional Health Bureau warehouses, 9 zonal 8 warehouses, and 1 hospital had been assisted with physical and systems improvements, and an additional 9 warehouses and stores were in process.14 Most of the warehouses (not all)15 in the Evaluation sample had been upgraded by DELIVER II. There were observable improvements in facilities where de-junking, re-shelving and ventilation work had been assisted by DELIVER II. Most of the warehouses and stores had also been assisted in developing procedures to systematically shelve and easily retrieve contraceptives and other store items. Warehouse and stores personnel that had received support from DELIVER II were proud to explain their stock control procedures and show their improved facilities. However, where DELIVER II or other logistics technical assistance had not reached, the warehouses/stores were found to be in very poor condition with no orderly arrangements of goods. Those facilities typically had very inadequate shelf space and poor ventilation. One warehouse visited had a visible infestation of termites, and no working lights or ventilation. The storekeeper did not keep regular store hours. Additional resources and technical assistance are needed for basic, low-tech physical improvements such as shelving, and more training of personnel in record keeping and storage practices. Personnel interviewed at warehouses and stores where problems existed tended to attribute their problems to shortage of trained personnel, lack of co-ordination (and supportive supervision), and shortages of resources for cleaning, stock movement, removal and disposal of expired stock, etc. Distribution of drugs and health commodities, including contraceptives, has traditionally been managed in a long supply chain that roughly aligns with the FMOH's tiered administrative structure (from national level to regional to zonal to woreda holding warehouses, then to hospitals and/or health centers, and from there to health posts who also supply community outreach Health Extension Workers). The predecessor DELIVER Project as well as DELIVER II have advocated for the FMOH to consider shortening the supply chain as a basic principle of improving the efficiency of re-ordering, re-supply, and distribution. However, the FMOH would not allow this change to be made until all commodities could be included in an improved short chain system. Under the new PLMP, the supply chain will indeed be shortened to include the national level, Hubs or Sub-hubs, direct to hospitals and health centers. In the interim period from 2006 to date, it has been necessary for DELIVER II to work with, and expended some of its field-based technical resources on upgrading regional, zonal, and woreda warehouses whose functions will eventually become obsolete in the new PLMP. Under the new PLMP, warehousing will be provided through (yet-to-be-established) "Hub/Sub-hub" warehouses, and hospital/health center stores. DELIVER II's possible future role in providing technical assistance and training in the upgrading of hospital and health center stores, is discussed in Section 3.3.1, PLMP, of this Report. 14 "USAID│DELIVER PROJECT, FY2008-09 Work Plan for Ethiopia", August 2008. 15 One store visited had been upgraded by Pathfinder; another by the Rational Pharmaceutical Management Plus (RPM Plus) Project. 9 1.3.2 The Contraceptive Logistics Management Information System (LMIS) The Evaluation Team sought to answer several specific questions regarding DELIVER II's experiences in design and implementation of the contraceptive LMIS: Are LMIS development goals being met? Yes. Development goals are being met. The LMIS (forms, recording and reporting procedures, and use of data for decision making), are accepted as standard operating procedure and are in evidence at management and service delivery levels. RHB managers and supervisors were fully conversant with the purposes, importance, and procedural requirements of health workers accurately recording and reporting contraceptive consumption data in a timely manner. To what extent is the LMIS component of ECLS now institutionalized and sustainable? The Evaluation Team defines "institutionalized" to mean that the contraceptive LMIS forms introduced by DELIVER II are generally recognized and accepted at policy, management, and service delivery levels throughout the public health system, and that LMIS procedures and protocols are routinely carried out by health workers. According to this definition (and based on the limited sample of MOH offices and sites visited in the evaluation), it is the judgment of the Evaluation Team that the contraceptive LMIS has been effectively institutionalized in the public sector. However, the system performs unevenly in an institutional environment that is plagued with general management deficiencies. In spite of this, LMIS recordkeeping and reporting procedures are being carried out routinely, and compliance is likely to continue until such time as health workers and managers receive new guidance (as is expected with introduction of the new PLMP). As with any information system, sustainability of the contraceptive LMIS introduced by DELIVER II could be further improved through information feedback (i.e., reported information is recognized, or delivery of commodities results, etc.), and monitoring and supportive supervision. How are LMIS data used at operational levels? Introduction of LMIS in family planning clinics appears to have heightened the appreciation of health workers for service data and information. In one family planning clinic visited by the Evaluation Team, the clinic nurse proudly explained the additional journals she had developed on her own to keep track of family planning service statistics and consumption data for internal management purposes. In another facility, a small woreda store, the health worker had summarized and displayed contraceptive consumption data by month, by health center and health posts to give a composite picture of performance in the woreda. Health workers at clinic level appear to generally understand the connection between recordkeeping, reporting, re-supply, and the relationship of supplies to quality of care. Does LMIS assist in monitoring of stock status, and has monitoring been institutionalized? Based on the Evaluation Team's visit to a well-run regional warehouse, consumption data generated by the LMIS was being used to assist the warehouse manager in inventory control and maintenance of stock levels. In another facility, the stores manager had converted all drugs and health commodities to the system set up to monitor contraceptives. In yet another 10 facility, the warehouse manager expressed appreciation for getting help from DELIVER II to "dejunk" his warehouse and dispose of expired stocks. However, at the clinic level where clients are seen and service providers maintain small quantities of contraceptives, monitoring of stock status is less systematic and institutionalized. As reported to the Evaluation Team, stock outs at the facility level usually do not trigger an urgent call to action because the situation can usually be solved by draw down from another nearby facility, borrowing from the private sector, or referral of the client to a retail pharmacist. Furthermore, as discussed with the Evaluation Team, one partner who supports grants for community outreach16 expressed concern that, although the overall contraceptive supply situation has improved since introduction of the ECLS, outreach workers supplied by health centers and/or posts), continue to experience shortages of some contraceptive methods from time-to-time. The Evaluation Team concluded that LMIS has not had much impact on monitoring of stock status at health posts where re-supply tends to be informal. Have there been improvements in the "culture" of recording and reporting? The Evaluation Team heard from many informants (reinforced by observations during site visits), the perception that maintenance of client records, registers and ledgers in health facilities, stores and warehouses had improved considerably since introduction of the contraceptive LMIS. Even in one marginally functional family planning clinic visited by the Evaluation Team, very little else was working properly, but their contraceptive use data appeared to be in good order! However, the Evaluation Team was left with a less favorable impression of upward reporting. Although the reasons are not well understood, there appears to be less motivation for making timely reports. Some managers interviewed by the Evaluation Team believe problems with upward reporting may be related to a disconnection between reporting and re￾supply. For example, if a report on stock levels does not result in receiving something in return (i.e. supplies or other feedback), the reporter may be less likely to make the report the next time. It is possible that this upward reporting issue has already been substantially addressed in the new PLMP. The PLMP calls for an arrangement whereby the report and the re-order from facility level are the same, and the re-order and resupply are direct, one-stop, between the hospital or health center and the regional "Hub" warehouse. 1.3.3 Monitoring & Supportive Supervision DELIVER II served as an important technical support to the FMOH Family Health Department and Regional Health Bureaus in implementation and institutionalization of the ECLS and LMIS, and in the monitoring and taking of corrective actions to improve contraceptive stock levels and Contraceptive Security. In discussions with RHB managers, it was apparent to the Evaluation Team that these managers fully understood the objectives and activities of the DELIVER II and rely on DELIVER II's technical assistance and support. Working relationships with DELIVER II's field-based Logistics Officers appeared to be cordial and productive. 16 The Packard Foundation. 11 Although RHB managers recognize the importance of regular monitoring and supportive supervision, the Evaluation Team received hints that these activities are neither as systematic nor as frequent as they should be. RHB's have extremely limited personnel and transportation resources, and tend to rely on partners such as DELIVER II to "piggyback" for site visits. It appears supervisory trips were scheduled when opportunities arose rather than RHB managers being pro-active by planning and coordinating supervisory schedules. This observation is significant because DELIVER II envisions (according to its Midterm Self￾evaluation), that it will be able to address some of the problems of attrition of workers trained in ECLS by increasing emphasis on On-Job Training (OJT), and through more intense monitoring and supportive supervision. 1.4 Conclusions 1.4.1 Strengths • DELIVER II has completed roll-out of the ECLS through training of an additional 4,000 health workers17 in 11 regions of the country. • A manual contraceptive LMIS (and a standardized training program related to implementation of the LMIS) has been institutionalized in the public sector. • DELIVER II training and introduction of ECLS practices has resulted in improvements in the "culture" for recording and reporting of logistics data has improved. • Where DELIVER II has provided training and technical assistance to warehouses and stores, improvements can be observed in physical conditions and inventory management. • Based on its experience in upgrading of warehouses and stores, DELIVER II has developed needs assessment tools and training materials for technical assistance and logistics management training of hospital and health center personnel. • Owing to the foregoing achievements and technical experience in the areas of LMIS design, training and supportive supervision for implementation of information systems, and training/technical assistance for strengthening of warehouse/stores, DELIVER II has laid important groundwork and gained technical advantages in taking the lead in these areas of systems development in the future. 1.4.2 Weaknesses • The ECLS and LMIS are not yet optimally effective due in part to the fact that these systems operate in an institutional environment that has many other management deficiencies. • The LMIS has not had significant impact on monitoring of stock levels at lower levels of the system (such as family planning clinics attached to hospitals where re-supplies are readily available from the hospital pharmacy and there is little need to monitor stock levels, or at health posts that experience chronic supply shortages and difficulty in obtaining supplies because of problems with transport). 17 6,000 health workers were trained before 2006 by the predecessor DELIVER Project, for a total of 10,000 FMOH staff trained to date. 12 • The level of monitoring and supportive supervision by DELIVER II in conjunction with RHBs has not been sufficient to improve ECLS practices to the extent needed, particularly at lower service delivery levels. • Significant losses have occurred in the numbers of health workers trained in ECLS, and DELIVER II has not yet identified/implemented effective approaches to counteract this problem. • It was necessary for DELIVER II to expend technical assistance resources on upgrading of regional, zonal, and woreda warehouses whose functions will eventually become obsolete. Although DELIVER II has laid important groundwork, has developed warehouse/stores training materials, and has technical assessment and assistance capabilities in this area, as of this writing, DELIVER II does not have a clear mandate from PFSA regarding its future assistance role in the upgrading of hospital and health center stores. 1.5 Lessons Learned 1.5.1 Approaches that worked well • As evidenced by the confidence placed in DELIVER II to be a full technical partner with UNICEF in design and implementation of the new health commodities supply system, during design and implementation phases of ECLS, DELIVER II developed and maintained a good working understanding at policy levels in the FMOH. • As reported by RHB managers, during the implementation phase of ECLS, DELIVER II developed and maintained good supportive working relationships with Regional Health Bureau implementers. 1.5.2 Approaches that might be improved in the future • The level of support provided by the project in the area of warehouse and stores upgrading covered only 5 of the 11 regions, and appeared to be far beyond the capacity of DELIVER II to provide on a national scale. DELIVER II responded by attempting to cover at least some regions and prioritizing within regions on the basis of those sites most ready to benefit. In the future, DELIVER II might also consider further prioritization geographically to concentrate regionally (with the commitment to move to additional regions as resources can be made available). In this way, the impact of stores upgrading on performance of the ECLS might be more evident. • Significant numbers of health workers trained in ECLS and LMIS are lost to the public health system annually through attrition. DELIVER II needs to develop systematic approaches to counteract the affects of attrition on performance of the ECLS. • Monitoring and supportive supervision provided by RHB managers is critical to well￾functioning of any logistics system at service delivery levels, but the capacity of RHBs to provide this type of support is limited. DELIVER II needs to intensify its efforts to assist RHBs in this area. 13 1.6 Recommendations • Over the next several years (until such time as contraceptive logistics can be absorbed into the new PLMP), DELIVER II should proceed with its stated objective of providing technical support to improve the manual ECLS. Specifically, DELIVER II should: - Continue to de-emphasize assistance in upgrading regional, zonal and woreda￾level warehouses, and increase concentration of assistance in upgrading hospital and health center stores as stated in the Project's FY 2008-09 Work Plan (to be confirmed with PFSA during 2009). - Maintain and improve to the extent possible, performance of the manual contraceptive LMIS (in tandem with any other automated LMIS design and implementation for all health commodities under PLMP). Increase concentration of assistance in supporting RHB Monitoring and Supportive Supervisory activities, giving priority to regions that can best benefit from this type of assistance. • In connection with transition to the PLMP, DELIVER II should advocate for, and incorporate to the extent possible, lessons learned from its experiences in improving ECLS in the areas of stores upgrading, LMIS and Monitoring/supportive supervision. 2. Contraceptive Security (CS)18 2.1 The Programmatic Context The demand for family planning and contraceptive commodities in Ethiopia has grown rapidly in the past decade. The Contraceptive Prevalence Rate (CPR) for all methods nearly doubled from 8.1 to 14.7 in the 5-years from 2000 to 2005. In the same 5-year period, the Total Fertility Rate (TFR) declined from 5-9 to 5.4 nationwide. Much of the fertility decline was confined to urban women, with no significant decline in rural fertility during the period.19 Providing access to family planning services, especially in rural areas, presents a major challenge to the public health care system (estimated to provide 80% of all services available). Added to this, national targets for CPR (all methods) have been set at 60% by 2010. National policy anticipates rapid declines in rural fertility in the 2006-2010 period and a faster than historical increases in CAR20 due to the planned expansion of the Health Services Extension Program (HSEP). To date, the HSEP has trained and deployed 24,000 Health Extension Workers (HEWs), with 13,000 more to be trained by 2008. Considered together, these factors point to the need for a well-financed and efficient contraceptive logistics system. 2.2 Program Challenges and Constraints At a minimum, the following contraceptive logistics sub-systems must be functional to provide any expectation that Contraceptive Security can be achieved: 18 By standard definition, Contraceptive Security exists when every person is able to choose, obtain, and use quality contraceptives and condoms whenever s/he needs them. 19 2005 Ethiopia Demographic and Health Survey, Central Statistical Authority [Ethiopia] and ORC Macro 2006, Addis Ababa, Ethiopia and Calverton, MD, USA. 20 For planning purposes, Contraceptive Acceptance Rate (CAR) is sometimes used by the FMOH in preference to CPR. 14 • Financial resources must be adequate and reliable; • Procurement mechanisms must be timely and efficient; • Distribution capacity must be adequate and reliable; • A functioning LMIS must be in-place; • Availability of contraceptives at service delivery levels must be adequate; • A relatively accurate forecasting feedback loop is needed to inform future financing and procurement decisions and actions. Two years ago, at the beginning of DELIVER II, none of these logistics sub-systems were operating efficiently and effectively. Financing and procurement of contraceptives was done separately by various donors21. Distribution of contraceptives followed a long supply chain approach involving 6-7 distribution and holding steps before the contraceptive item actually reached the client. There was chronic over and under stocking of contraceptives at all levels of the system, and contraceptives were frequently not available to clients on demand. The contraceptive LMIS introduced in 2004 was not fully operational, and contraceptive forecasting was fragmented by method and donor and did not systematically consider contraceptive consumption data. Sector-wide development of these logistics sub-systems falls within the purview of multiple FGOE implementers and is beyond the scope of supporting partners such as DELIVER II. However, some discrete improvements in each of the areas of financing, procurement, distribution and forecasting would be necessary to ensure a reliable supply of contraceptives in the system. Finding innovative approaches to improving the functioning of these sub-systems in relation to contraceptives and Contraceptive Security (approaches that were also cost-effective and within the parameters of the project's limited budget), presented DELIVER II with a major challenge. 2.3 Progress in Achieving Results against Stated Objectives DELIVER II's stated objective in CS is: "To improve Contraceptive Security." The intent of the objective, and the result sought is "improvement." DELIVER II pursues improvement by promoting better understanding of Contraceptive Security, and by helping to strengthen logistics sub-systems to the extent needed to ensure an adequate and reliable supply of contraceptive commodities. As reported by DELIVER II in its Mid-term Self-evaluation22, there is clear evidence that DELIVER II has, indeed, actively promoted understanding of Contraceptive Security concepts and supported development of an overall Contraceptive Security strategy in Ethiopia. Following are examples of related DELIVER II activities: • DELIVER II was instrumental in re-energizing the Family Planning Technical Working Group, a group comprised of the FMOH and supporting partners concerned with family planning and contraceptive logistics issues. In 2006, with support of DELIVER II, the FPTWG organized a workshop entitled "National Workshop on Contraceptive Security: 21 Basket funding for contraceptives under the Protection of Basic Services (PBS) started in May 2006. Thereafter, UNFPA became the contraceptive procurement agent for the PBS. 22The summary of achievements shown in this section of the Report are drawn from the PROJECT's "Mid-term Self-evaluation, November 2008", and spot-checked anecdotally through in the course of discussions with Evaluation informants and stakeholders. 15 Ensuring Access to Family Planning, "which was attended by over 100 participants from 9 of the 11 regions. In 2006, DELIVER II arranged for a twelve member team headed by the Amhara Regional Health Bureau Head, to join the East Africa Reproductive Health Contraceptive Security Workshop in Dar Es Salaam, Tanzania. This experience served to substantially broaden the policy perspective of the Ethiopian health team. • With launch of the Ethiopian Contraceptive Logistics System and subsequent training of thousands of staff and local officials at all levels of the health system since 2006, Conceptive Security concepts are widely understood among health care workers. DELIVER II's progress in improving CS is further discussed below in 5 areas of the health commodities logistics cycle: Financing, Procurement, Distribution & Transport, Availability & Use (as documented by the LMIS), and Forecasting. 2.3.1 Financing and Procurement DELIVER II, along with several other family planning partners, has actively facilitated coordination and cooperation of donor financing and procurement of contraceptives. As a result of advocacy and coordination efforts on the part of DELIVER II and other family planning partners, in FY 2007-2008, the Federal Government of Ethiopia allocated ETB 1 million for the procurement of contraceptives in the national budget. Commitments have also been made by several regional governments, as well as various donors, specifically the World Bank, DfID, SIDA and the EU through the decentralized Protection of Basic Services (basket funding) Program (PBS). UNFPA is the contraceptive procurement agent for the PBS. USAID annually provides US$ 6 million for contraceptives (through local NGOs to public sector sites). Some contraceptives from DKT, a private sector social marketing firm, are also donated to the public sector. 2.3.2 Distribution & Transport As also discussed in Section 1.3Warehousing and Storage, DELIVER II will eventually have been instrumental in improving the distribution sub-system for all health commodities, including contraceptives. Distribution of drugs and health commodities, including contraceptives, has traditionally been managed in a long supply chain that roughly aligns with the FMOH's tiered administrative structure. This supply chain carries from national level to regional to zonal to woreda holding warehouses, then to hospitals and/or health centers, and from there to health posts who also supply community outreach Health Extension Workers. The predecessor DELIVER Project as well as DELIVER II23 have advocated for the FMOH to consider shortening the supply chain as a basic principle of improving the efficiency of re-ordering, re-supply, and distribution. However, as stated in DELIVER II's Mid-Term Self-evaluation (See page 10), DELIVER II was not successful in convincing the FMOH that the supply chain for contraceptives should be shortened (in the interim before implementation of the new PLMP). As conceptualized in the new health commodities supply system and PLMP, the supply 23 As reported by DELIVER II management, the Project engaged in policy dialogue with the FMOH on this issue in late 2004 and early 2005. 16 chain will be shortened to include the national level, Hubs or Sub-hubs, direct to hospitals and health centers. While DELIVER II will eventually have had important technical influence over shortening the supply chain and improving the efficient distribution of all commodities, including contraceptives, the project has had no significant impact on improving the FMOH's transport capacity for distribution of contraceptives. According to an FMOH official and several RHB managers, lack of transportation within the MOH (and the FGOE, in general), continues to be a critical deficiency. The problem is systemic and beyond the scope of DELIVER II (except for possible low-cost innovation such as transport networking). 2.3.3 Availability & Use of Contraceptives In spite of many advances in Contraceptive Security in the past 2 years, a "bottom-line" question needs to be raised: Has DELIVER II contributed to increasing Contraceptive Security and availability of family planning commodities? The answer is a partial "Yes." While DELIVER II, along with other family planning partners, has been instrumental in improving the availability of contraceptives in-country, the coalition of government and donor resources, and the financing and procurement mechanisms that have been put in place are fragile and may not be self-sustaining in the long-term. Furthermore, while general availability of contraceptives has vastly improved24 at facility and community levels, over- and under- stocking and stock-outs are reported to be quite common. Poor functioning of ECLS sub-systems plays some role (such as flaws in inventory control, and lack of timely reporting and re-ordering, etc.) but the overarching problem is the almost total absence of transport capacity in the public health system. As reported by RHB managers, distribution of commodities to facility levels is somewhat haphazard, frequently reliant on vertical program vehicles or private sector partners. Although private sector partners and support agencies with more reliable distribution capacity willingly step in to supplement contraceptive shortfalls in public sector facilities, availability of the desired contraceptive at the point of service is not always certain from the client's point of view. More than one clinic nurse said that it is common for the client to be requested to purchase the commodity (for instance, an injectible), out-of-pocket from a commercial pharmacist. In this scenario, the end user client eventually receives the contraceptive, but contraceptive security in not certain. 2.3.4 Forecasting DELIVER II has a well-established technical role in the area of commodities forecasting methodology in Ethiopia. Under the predecessor DELIVER Project, assistance has been provided to the Federal Family Health Department in the area of contraceptive forecasting and resource gap analysis since 2004. Since 2006, DELIVER II has provided assistance to the FMOH in preparation of national contraceptive forecasts using a 24 The FMOH Family Planning Head confirmed that there were no contraceptive stock outs of any method in 2008 at the central level compared to prior years. 17 combination of demographic data, service statistics, and contraceptive consumption data from the ECLS/LMIS. DELIVER II has trained FMOH and RHB staff in forecasting methods, and supported a national long-range contraceptive forecasting exercise in the FMOH Family Health Department in early 2007. 2.4 Conclusions 2.4.1 Strengths • The PROJECT has substantially improved Contraceptive Security awareness and national policy environment, donor coordination, financing and procurement mechanisms. • Since inception of DELIVER II in 2006, the supply of contraceptives in-country has improved. There were no stock-outs of any contraceptive at the central level in 2008. 2.4.2 Weaknesses • Stock outs and over-stocking are still a problem at some health centers and below. • The mechanisms established to date to ensure long-term availability of contraceptives are fragile and vulnerable to changes that might occur, particularly in the areas of financing and procurement, under the new PLMP. 2.5 Lessons Learned 2.5.1 Approaches that worked well • As reported to the Evaluation Team by a large number of informants interviewed, DELIVER II is considered to have been instrumental in improving donor financing and procurement of contraceptives through its advocacy and coordination efforts. As evidenced by documents and the accounts of RHB managers, DELIVER II has substantially improved contraceptive forecasting procedures and the use of forecasting methods for donor financing and procurement decision-making. 2.5.2 Approaches that might be improved in the future • Extremely limited distribution and transport capacity throughout the public health system is a major hindrance to contraceptive logistics management and ensuring the availability of contraceptives at lower service delivery levels. According to DELIVER II senior management, the project is constrained by procurement guidelines and budget from purchasing logistics support vehicles in sufficient volume to impact this problem. In the interim until such time as distribution and transport capacity within the health system improves as a result of inputs for the PLMP, DELIVER II might attempt innovations (such as assisting RHBs with more systematic planning, coordination and control of limited transport resources), to make distribution of contraceptives more efficient. 2.6 Recommendations • Over the next several years (until such time as contraceptive logistics can be absorbed into the new PLMP), DELIVER II should proceed selectively with its stated objective of providing technical support to improve Conceptive Security. Specifically, DELIVER II should: 18 - Continue to advocate for donor financing and procurement of contraceptives, and assist with coordination in this area (such as continued support to the Family Planning Technical Working Group and development of Family Planning/Reproductive Health Networks at regional levels). - Consider undertaking an operations research/pilot effort in a defined geographical area to improve reliable distribution and transport of contraceptives down to the community outreach level, and measure the impact of these improvements (along with other improved logistics supports) on CPR in the study area. - Continue to monitor the availability, off-take and reporting of contraceptive consumption data in support of supportive supervision and contraceptive forecasting. - Continue to organize and coordinate annual contraceptive forecasting exercises in all regions. • In connection with transition to the PLMP, DELIVER II should advocate for, and incorporate to the extent possible, lessons learned from its experiences in improving Contraceptive Security in the areas of contraceptive financing, procurement, use of LMIS to monitor/manage commodities supply levels in health facilities, and RHB roles in contraceptive and other commodities forecasting. 3. Transition to the Pharmaceutical Logistics Master Plan (PLMP) 3.1 The Programmatic Context In 2005, in response to continuing, overarching drug logistics management problems in the public health sector, the Minister of Health requested UNICEF, with technical assistance from the DELIVER Project, to develop the conceptual framework and implementation plan for a comprehensive Health Commodities Supply System (HCSS). The DELIVER Project was tapped for this assignment because of its experience and the technical advantage gained through design and progress in implementing a vertical Ethiopian Contraceptive Logistics System (ECLS). This HCSS plan, renamed the Pharmaceutical Logistics Master Plan (PLMP) by the FMOH, has become the definitive blueprint for development of Ethiopia’s public sector health commodities supply system. The PLMP identifies specific objectives on policy legislation: coordination and harmonization; financing, selection, quantification, forecasting, procurement, distribution and transport; LMIS; and monitoring and supportive supervision. PLMP is intended to: • Ensure access to a constant and uninterrupted supply of vital and essential health commodities, as described by affordability, geographic accessibility, equity and quality; • Focus on capacity building at all levels and for each component activity (human resources, infrastructure, system etc); • Financing the national procurement system based on Revolving Drug Fund (RDF) principles, assuming that facilities at decentralized levels will support RDF principles where feasible; • Establish implementing organizational structures and functions based on Business Process Re-engineering (BPR) for efficiency, quality and accountability at all levels; 19 • Pool funds and ‘in kind’ contributions from various sources (e.g. Federal and local budgets, Federal loans, donor funds, etc.) to facilitate efficient execution of procurement according to approved procurement plans, the realization of significant benefits in economies of scale (competitive purchase prices), and flexibility to adjust planned procurements; and • Provide flexibility in the interim for the procurement of special program products by donor funds such as UNICEF, WHO, UNFPA, USAID etc, to ensure that already initiated and functional supply chain systems are not disrupted. 3.2 Program Challenges and Constraints Based on the essential role played by the DELIVER Project in conceptualizing and design of the PLMP, the follow-on DELIVER II (which began in October 2006), was programmed to play a key technical role in implementing the new plan. However, major changes have occurred over the past 2 years that present challenges and constraints to DELIVER II in supporting PLMP implementation. Key among these changes has been the shift of public sector health commodities logistics management out of the FMOH to a semi-autonomous agency, and the related move of many supply chain functions out of the FMOH. The approximate chronology of these changes and their implications for DELIVER II are further discussed below. In March 2007, soon after the FMOH authorized the PLMP, it also established the PLMP Implementation and Support Team (IST) to coordinate technical and financial support to implementation of the PLMP. Working closely with FMOH policy makers, DELIVER II was requested to take the lead in facilitating development of an IST/PLMP Accountability Matrix that identified lead and support roles of MOH departments, other government entities, and key supporting partners on PLMP implementation. In 2007, DELIVER II also proceeded with a variety of technical developments in cooperation with IST support partners according to roles established in the IST Accountability Matrix. For example, DELIVER II assisted WHO with the methodology used to undertake a commodities forecasting baseline for various programs and products and developed protocols to assist RHBs in assessing health facility stores for renovations in 2007. A change of major import to DELIVER II occurred in September 2007, when a proclamation issued by the Government transferred the drug management responsibilities of the FMOH Pharmaceutical Supply and Logistics Department (PSLD), to the semi-autonomous Pharmaceutical Fund Supply Agency (PFSA). PFSA would support and work in cooperation with the FMOH, but would not fall directly under the FMOH's authority as does the PSLD. PFSA was given the mandate to implement the PLMP, and became the new authority responsible for most health commodities logistics sub-systems (including financing, procurement, warehousing, distribution, LMIS, some monitoring, and forecasting; and excluding Availability/Use, some monitoring, and product selection). Since PSLD was one of DELIVER II's key technical counterpart offices in the FMOH, this transfer represented a critical challenge to DELIVER II. It meant that DELIVER II would need to establish a new mandate directly with PFSA concerning DELIVER II's future technical assistance role in implementing the PLMP. 20 Following this major shift of health commodities logistics management to PFSA, the FMOH requested IST to disband, recognizing that future technical roles of government departments as well as those of support partners would need to be revamped with the newly-designated PFSA. A complicating factor, however, was the government's on-going re-organization exercise under the BPR25. It appears that DELIVER II will not be able to sit with PFSA officials to clarify the Project's future roles in PLMP implementation until the BPR has been completed. In 2008, DELIVER II and other PLMP supporting partners have not had much Government input on coordination of activities related to PLMP implementation. Although DELIVER II continues to have access to the Minister of Health (who brought DELIVER into to the PLMP design and implementation process originally), DELIVER II, as well as all other donors and partners interviewed during this Evaluation, reports that it is not privy to on-going BPR deliberations, PLMP implementation plans, and DELIVER II future roles in assisting PLMP implementation. Throughout 2008, DELIVER II has proceeded under the assumption that its Terms of Reference in connection with assistance to the PLMP will remain as had been agreed with the FMOH at the beginning of the DELIVER II's current authorization in 2006. Specifically, that DELIVER II has continued to assist in design and implementation of the PLMP by: - Taking the lead in design and implementation of an automated health commodities LMIS; - Providing assistance in forecasting (in non-HIV/AIDS drugs and commodities); and - Improving stores management in hospitals and health centers. Since these assumptions have not yet been re-confirmed with the newly-designated counterpart PFSA, DELIVER II is currently operating with a degree of uncertainty about its future role in the PLMP transition. DELIVER II is faced with a challenge to present its technical advantages in the areas of LMIS, forecasting, and hospital/health center stores upgrading, and to re-confirm this operating mandate with PFSA. Added to this uncertainty, entry of another USAID-funded partner, one with a technical mandate similar to that of DELIVER II and an already established working mandate with PFSA in Ethiopia, the Supply Chain Management System (SCMS), raises questions about the relative advantages of DELIVER II as the technical lead in the above described areas of PLMP implementation. SCMS has a specific mandate for HIV/AIDS drug and health commodity logistics, but plans to carry out this mandate are within the broader context of support to the health commodities logistics system as a whole. It is the perception of the Evaluation Team that SCMS could provide assistance at all levels of the evolving logistics 25 Effects of Business Process Re-engineering (BPR) on the working relationships of the GFOE, donor, and supporting partners is discussed in the INTRODUCTION/Limitations Section of this Report, and is repeated, in part, as follows: "The FMOH, along with other ministries of government, has been engaged in a major restructuring exercise for the past year. This restructuring process, referred to as Business Process Reengineering (BPR), evidently consumes the full-time and attention of senior FMOH officials, to the extent that donors and the FMOH's implementing partners have limited access, in real time, to the FMOH. Throughout the evaluation, the BPR was frequently mentioned by donors, implementing partners and Regional MOH managers as a constraint to conducting regular business with the FMOH." 21 system, but might be best positioned to assist PFSA at the sector level (versus the decentralized service delivery levels) because of its link to PFSA at the federal level. SCMS has in-place a resident Technical Advisor to PFSA, and is also engaged in LMIS design work which the Evaluation Team understands is mainly focused on consolidation of commodities consumption data at the Hub/Sub-hub level and above. SCMS will also support basic logistics infrastructure development such as warehouse upgrading and building a logistics transport fleet through provision of vehicles26. 3.3 Achievements to Date DELIVER II's stated objective for the PLMP is: "Assist in design and implementation of the Pharmaceutical Logistics Master Plan.” The result sought is assistance in both design and implementation phases of the PLMP. DELIVER II has played a well-documented role as the technical lead in design of the PLMP. During the past year, in a period of uncertainty about its future assistance role in implementing the PLMP, DELIVER II has continued with design and development of the 2 logistics sub-systems with which it assumes it will be working in the future. DELIVER II's progress in design and development of PLMP is discussed below in 2 areas of the health commodities logistics cycle: Warehousing & Storage, and LMIS. In addition to assisting with the PLMP transition, DELIVER II has a continuing mandate in the area of contraceptive logistics and Conceptive Security. DELIVER II will therefore need to give continued special attention to these areas as the PLMP evolves. 3.3.1 Warehousing & Storage As previously discussed in Section 1 of this Report (See ECLS/Warehousing & Storage), the current supply chain for health commodities can involve as many as 6-7 warehouse/storage tiers (central to regional, to zonal, to woreda, to hospitals/health centers, to health posts, to HEWs and community outreach workers). Under the new PLMP design, the supply chain will be substantially shortened (PFSA central to PFSA Hubs/Sub-hubs, to hospital/health centers, and from health centers to posts). In anticipation of re-focusing and concentrating on assistance in upgrading hospital/health center stores, DELIVER II has already begun systematically assessing and prioritizing the facilities that can best benefit from physical upgrading and improvements in inventory management. Related to this, in April 2008, DELIVER II assisted the FMOH in developing and publishing a curriculum (Trainer's Guide and Participant's Course Workbook) to train hospital and health center personnel in the national health commodity supply system. 3.3.2 Automated Health Commodities LMIS In December 2007, the PSLD in the FMOH (then the DELIVER II logistics counterpart), conducted a workshop to design the integrated logistics management information system and aspects of the MOH inventory control system for health facility stores for the new pharmaceutical logistics system. System participants from all levels in the health management system were involved in the workshop in the design of an integrated logistics 26 SCMS information as derived from discussions with the SCMS Resident Advisor and staff. 22 management information system and a related inventory control system that responds to the parameters as outlined in the (PLMP) for management of commodities for all health programs. Following the PLMP Integrated LMIS Design Workshop in December 2007, the DELIVER II has made major progress on development of the new automated LMIS. DELIVER II LMIS design specialists have been assigned to fully develop a prototype, and field testing is currently being carried out in Tigray Region. The automated LMIS design was reviewed by the Evaluation Team, and it appears to be robust and user-friendly. DELIVER II's LMIS developers emphasized that systems software is locally developed, designed to be simple and user-friendly, and customized specifically to assumed PFSA requirements. The system is also intended to be "open" to other health information system interfaces. 3.4 Future Directions 3.4.1 Upgrade stores at hospitals and health centers During transition to the PLMP, the DELIVER II plans to scale down its technical support to regional, zonal and woreda warehouses and stores, and to primarily focus on hospital and health center stores. It is the intention of PFSA to have the 3-tier supply chain installed and operating nationally within the next 2 years. However, as reported to the Evaluation Team by more than one RHB manager, RHBs, zonal, and woreda level warehouses do not want to completely abandon their back-up facilities during the transition. DELIVER II plans to continue with some level of support for training in de-junking and inventory management for zonal warehouses, but will no longer cover the actual cost of de-junking or purchase and installation of shelving and related equipment. 3.4.2 Further design and develop the automated health commodities LMIS DELIVER II's work in the area of automated LMIS design should continue. The Evaluation Team noted, however, that the December 2007 design workshop was organized by the FMOH/PSLD which is no longer responsible for health commodities LMIS. DELIVER II therefore needs to confer with the newly-designated health commodities supply manager, PFSA, to ensure that current LMIS design work is compatible and consistent with PFSA's requirements. 3.4.3 Ensure availability of contraceptives at all levels of the health care system DELIVER II will need to exercise increased vigilance to ensure that contraceptive supply levels to hospitals and health centers are not adversely affected during transition to the PLMP. Shift in DELIVER II's technical assistance to focus on hospitals and health center stores provides a strategic means for the project to continue monitoring the availability of contraceptives and taking corrective action as needed. Monitoring and supportive supervisory activities are another means of monitoring contraceptive supply levels (and being able to take corrective actions as necessary) during transition to the PLMP. DELIVER II should continue its support for contraceptive supply monitoring and supportive supervision at the RHB level. Innovations intended to further 23 improve the impact of monitoring and supervision on facility supply levels and services could be introduced. One innovation might include providing feedback to facilities periodically on their performance compared to others in maintaining supply levels. Another innovation might include a system of recognition/rewards for reaching facility supply and performance targets for family planning. These types of innovations might assist RHB managers in: systematically monitoring LMIS data and contraceptive supply levels; increasing the vigilance of providers in monitoring supply levels and in making the link between the availability of supplies and the quality of care; and in monitoring contraceptive utilization patterns to inform higher level decision makers in such areas as long-range forecasting, product selection, and rationale drug use. As noted in Section 1.3.3, ECLS/Monitoring & Supportive Supervision, RHB capacity for Monitoring & Supportive Supervision appears to be limited. DELIVER II might consider piloting an approach with a cooperative RHB that would entail developing a 3-month advanced Monitoring & Supportive Supervision Master Schedule with RHB supervisors and partners. This might serve to improve the frequency, focus, and impact of monitoring and supportive supervision. 3.4.4 Improve contraceptive forecasting Functional roles specified by the PLMP/IST in 200727 called for DELIVER II to expand its activities in the area of extrapolated forecasting methods to cover all non-HIV/AIDS essential health commodities much in the same way as the Project had been approaching contraceptive forecasting up to that point. However, based on the DELIVER II design of the new health commodities supply system, aggregated health commodities forecasting and the government offices engaged in financing and procurement decision making may be very different than pre-PLMP. In the new PLMP, forecasting functions of the system are basically "built-in" because the system is designed to estimate supply levels to maintain a pre￾determined, full-supply situation at each level. Although the DELIVER II work plan for FY 2008-09 included forecasting activities that would expand beyond contraceptives to cover all non-HIV/AIDS essential health commodities, DELIVER II management reports that this work objective was dropped at the beginning of the work year (October 2008) because of the uncertainty regarding PFSA's interest in this type of support from DELIVER II. In the interim, until such time as DELIVER II's role in the area of non-HIV/AIDS essential health commodities forecasting is clarified with PFSA, it will be important for DELIVER II to continue with a full program of assistance in forecasting for contraceptives. DELIVER II's continued promotion of extrapolated forecasting methods for contraceptives may have two important benefits. Firstly, the long-range contraceptive forecasting activities supported by DELIVER II have contributed to development of managerial skills and knowledge of RHB/Family Health personnel, and thus has inherent value and application within the health care system. RHB health managers need to fine tune their skills related to drug supply management at service delivery levels and to be knowledgeable technical advisors to higher levels of the health care system on health commodities availability and use. Secondly, as with other future directions discussed in Section 3.4.3 above, DELIVER II's continued involvement in organizing long-range contraceptive forecasting exercises at the regional 27 "Support to the HCSS Master Plan – Accountability Matrix", February 2007. 24 levels presents yet another opportunity for DELIVER II to monitor contraceptive security during transition to the PLMP. 3.5 Conclusions 3.5.1 Strengths • DELIVER II has played a lead role in conceptualizing and design of the PLMP. In its capacity as a lead architect of the PLMP, DELIVER II has made a major contribution to strengthening the overall national drug supply system. • DELIVER II has taken the lead in designing and field testing an automated health commodities LMIS, and should continue to lead in providing technical assistance to PFSA in implementing the new LMIS at service delivery levels. • DELIVER II is well-positioned and should, through its established working relationships with RHBs, to facilitate implementation of new PLMP procedures and practices, (including LMIS, hospital and health center stores management, and monitoring and supportive supervision), at service delivery levels. • DELIVER II is well-positioned to continue in its strategic role of improving the availability of contraceptive commodities in public health facilities. 3.5.2 Weaknesses • There are areas of potential technical and programmatic overlap between the DELIVER II and the USAID-funded SCMS. • DELIVER II's comparative strengths are not currently being fully utilized in the implementation of the PLMP. • DELIVER II does not have a clear operating mandate with PFSA in designated areas of health commodities logistics technical assistance. DELIVER II needs to seek clarification and confirmation with PFSA on its Terms of Reference in health commodities logistics technical assistance, as a matter of urgency. • DELIVER II has discussed in its Work Plan for FY 2008/09 a proposal to, "Assist FMOH to make financial systems decisions to support PLMP goal of ‘full supply’ for Essential Health Commodities." The USAID│DELIVER PROJECT has health commodities financing capabilities, and DELIVER II has had some success28 in facilitating the financing of contraceptives. However, DELIVER II needs to concentrate on resolving the uncertainty of its future assistance role with PFSA (in the areas of stores upgrading, LMIS, and forecasting), before proposing involvement in sector-wide financing of health commodities – an area in which DELIVER II has not played a prominent role in Ethiopia the past. 4. Project Staffing and Management 4.1 Findings The Evaluation Team was requested to spend a limited 10% of its time in response to this element of the evaluation. Findings are anecdotal, observational, or derived from review of project documents and discussions with DELIVER II staff. 28 See Section 2.3.1, CS/Financing and Procurement, of this Report. 25 • What monitoring and evaluation systems are in place to assess the Project's impact and the sustainability of results? DELIVER II has a formal "Monitoring, Evaluation and Reporting Plan" dated October, 2007. That Plan specifies and provides details of the monitoring system, data collection and reporting formats, and reporting periods. The Plan was judged by the Evaluation Team to be comprehensive and implementable. In addition, DELIVER II Quartering Reports to USAID were reviewed and found to be complete and clear. This serves as an additional tool for DELIVER II in monitoring its performance and results. • Do M&E systems identify implementation problems so that corrective actions can be taken? Yes. The M&E Plan identifies key indicators intended to accurately measure the different intervention areas of DELIVER II. When events are not occurring as expected, such as "No joint supervisory visits conducted," the monitoring report will reflect that as a problem so that corrective actions can be taken. • What internal processes are in place for on-going assessment of strategic direction of the Project? DELIVER II's senior management team is all located in Addis Ababa, and meets formally and informally on a regular basis. Monthly meetings are held at project headquarters with field-based technical staff. • To what extent are strategic thinking and on-going management and mid-stream corrections supported by USAID? By JSI? The strongest evidence of collaboration on strategic thinking, on-going management and mid-stream corrections between the DELIVER II, JSI headquarters, and USAID, is that DELIVER II's key strategic document, "Project Objectives and Strategic Direction (2006- 2011)" was recently revised in July 2008. Unfortunately, as noted elsewhere in the Report, these strategic agreements related to the PLMP design and implementation portion of DELIVER II's mandate were not additionally discussed with government by either DELIVER II or USAID. • Is the decentralized project structure efficient and effective? The decentralized project structure is the only arrangement for establishing DELIVER II's presence and building sound working relationships with MOH counterparts and regional and service delivery levels. Communications and coordination of activities between the field and headquarters appears to work reasonably well. At least no problems could be detected by the Evaluation Team during its brief exposure to regional staff. • Is the central office able to adequately support its regional staff? No problems were reported or could be detected by the Evaluation Team during its brief exposure to regional staff. 26 • Does the Project's management structure allow staff performance to be reinforced or strengthened? DELIVER II's senior management described standard operating personnel practices which call for a mandatory annual personnel review. This is accomplished through a self-evaluation, 360 degree input, and verbal interview with the employee's supervisor. In the judgment of the Evaluation Team, these personnel practices appeared to be adequate. • Is additional staff development indicated? Yes. DELIVER II should give consideration to a series of in-house seminars that allow field-based staff to develop new knowledge and skills related to recent programmatic and technical developments (i.e., logistics management theory, or the new automated health commodities LMIS). This cost-effective approach, if correctively structured, might prove to be effective as a team-building and retention-enhancing management tool, and should be tried in preference to study opportunities abroad. • What are the most significant management challenges currently faced by the PROJECT? As discussed in Section 3.3, Program Challenges and Constraints, of this Report, the most significant management challenge currently faced by DELIVER II is not resource or administrative in nature. Rather, DELIVER II's major management challenge is concerned with the gap in understanding between the PFSA and itself on the technical roles PFSA would like DELIVER II to play in implementation of the PLMP. 4.2 Conclusion DELIVER II's staffing and management structure appear to be fundamentally sound. No problems could be detected by the Evaluation Team during its brief exposure to DELIVER II at its central office in Addis Ababa and its field offices in 4 regions. 4.3 Recommendation DELIVER II might consider organizing an innovative in-house staff development program for its field-based staff for the purpose of staff development and team building. 27 RE-CAP OF LESSONS LEARNED Lessons Learned are summarized below, and discussed in more detail in Sections 1.5 and 2.5, above. Approaches that worked well Approaches that might be improved in the future ECLS • DELIVER II developed and maintained a good working understanding at policy levels in the FMOH. • DELIVER II developed and maintained good supportive working relationships with Regional Health Bureau implementers. ECLS • DELIVER II might consider further prioritization of hospital and health center stores to be upgraded based on geographic concentrations within regions with the intent of making the impact of stores upgrading on performance of the ECLS and the health commodities logistics system more evident. • DELIVER II needs to develop systematic approaches to counteract the affects of attrition on performance of the ECLS • DELIVER II needs to intensify its efforts to assist RHBs CS • DELIVER II is generally recognized by stakeholders as having been instrumental in improving donor financing and procurement of contraceptives through its advocacy and coordination efforts. • DELIVER II has substantially improved contraceptive forecasting procedures and the use of forecast for donor financing and procurement decision-making. CS • DELIVER II might attempt innovations (such as assisting RHBs with more systematic planning, coordination and control of limited transport resources) to make use of limited transport resources more efficient. PRIORITIZED SUMMARY OF RECOMMENDATIONS Recommendations are also discussed in further detail at the technical sections of this Report. The Evaluation Team recommends that DELIVER II's first priority in addressing the recommendations of this Report should be given to preserving progress already made by DELIVER II in the areas of implementing the Ethiopian Contraceptive Logistics System (ECLS) and a national Contraceptives Strategy (CS). The second priority should be clarify and reach agreement with USAID, other partners, and the Federal Government of Ethiopia/Pharmaceutical Fund and Supply Agency (PFSA) on DELIVER II's mandate and portfolio in implementing the Pharmaceutical Logistics Master Plan (PLMP). A third priority is concerned with an important, but non-urgent strategy for long-term staff development. Recommendations, in order of priority, are summarized as follows: 28 29 1) DELIVER II should continue its advocacy and coordinative roles with family planning partners, affirm its working mandate with the Federal Ministry of Health (FMOH)/Family Health Department and Regional Health Bureaus, and continue its strategic and technical assistance activities aimed at monitoring and improving the ECLS and CS in the public sector. Specifically, until such time as various components of the new PLMP are functional and can absorb responsibility for contraceptive supplies in the public sector, it is recommended that the Project continue, and strengthen to the extent possible, it's advocacy for contraceptive financing and procurement; technical support for hospital and health center stores (with special attention to contraceptive supplies); monitoring and supportive supervision of contraceptive logistics management at health facility levels, and contraceptive forecasting. 2) DELIVER II should ensure, to the extent possible, that lessons learned from its experiences in developing the ECLS and CS are incorporated into any assistance DELIVER II might provide in design, development, and implementation of the PLMP. 3) As a matter of urgency, DELIVER II and the USAID-funded Supply Chain Management System (SCMS) should, in conjunction with USAID, review their technical scopes of work related to assistance to the PFSA and speedily reach agreement between the two projects on complimentarity and avoidance of areas of potential programmatic or technical overlap. 4) At the earliest opportunity following resolution of the above issue, the DELIVER II should to enter into discussions and reach agreement with PFSA on the designated areas of technical assistance in implementation of the PLMP that the DELIVER II will be expected to provide. These discussions should be convened with assistance of the FMOH and USAID, if necessary. Agreements should be formalized and in writing to the extent permitted by standard USAID project management practices. 5) Based on agreements reached between DELIVER II and PFSA, DELIVER II should review, (and if necessary, modify) its FY 2008/09 Work Plan and proceed with an agreed upon portfolio of activities for all health commodities that might, if agreed with PFSA, include: upgrading stores at hospital and health centers; further design and development of an automated health commodities LMIS; and training/technical assistance in forecasting methods (in addition to contraceptives), for non-HIV/AIDS essential health commodities. 6) DELIVER II should initiate additional in-house technical training for its Logistics Officers. Annex A Evaluation SOW Draft Statement of Work (SOW) for the Mid-Term Project Evaluation of Contract No. GPO-I-01-06-00007-00 The Deliver Project Draft– October 28, 2008 PROJECT IDENTIFICATION DATA 1. Project Title: The Deliver Project 2. Project Number: Contract No. GPO-I-01-06-00007-00 3. Project Dates: October 1, 2006 - September 30, 2011 4. Project Funding: USD $2.75 billion global project 5. Implementer: John Snow Inc. (JSI) 6. CTO: Sharmila Raj I. Identification of the Task The USAID/Ethiopia (USAID/E) Health, AIDS, Population and Nutrition (HAPN) Office requests technical assistance from the Mitchell Group (TMG), under the USAID/E Evaluation Contract Number 663-C-00-08-00409-00, to design and implement an independent external mid￾term project evaluation of The Deliver Project. The assessment will draw from and build on a self-assessment from the Project, feedback from USAID/Ethiopia, and key informant interviews with in-country partners conducted by external consultants to determine the strengths, weaknesses, and future direction of services provided by the DELIVER Project in Ethiopia. The DELIVER Project’s overall purpose is to design, develop, strengthen, and upon request, operate safe, reliable and sustainable supply systems that will provide a range of affordable, quality essential health commodities to clients in country programs. USAID/Ethiopia implements activities under this global project through the use of field support funds. As part of its overall evaluation of all HAPN projects, USAID/Ethiopia is commissioning this mid-term assessment to examine the following: • The project’s progress toward achieving results against stated objectives. • Identify strengths and weaknesses within the project’s portfolio. • Determine areas and activities that may warrant continued investment, as well as other key initiatives and approaches not covered by the project, but which would likely contribute to improving the availability of family planning and other essential health commodities. The USAID/E HAPN office requests that the evaluation be completed by November 30, 2008 in order that the findings, conclusions and recommendations can be used to enhance project implementation and inform future project direction. II. Background In Ethiopia, ensuring a reliable supply of contraceptives, condoms, and other essential health commodities to clients is critical to improving the overall health system and achieving ambitious targets set forth by the Federal Ministry of Health (FMOH). For example, the FMOH’s Health Sector Development Plan set a target of a 60 percent contraceptive prevalence rate by 2010 (the unmet meet for family planning is approximately 34 percent). USAID/Ethiopia continues to invest in commodity logistics across health technical areas including family planning, HIV/AIDS, and maternal and child health. USAID/Ethiopia provides approximately $3 million in annual funding to the USAID | DELIVER PROJECT for technical assistance in logistics, with emphasis in family planning. III: Overview of the Deliver Project The USAID | DELIVER PROJECT is a five-year $2.75 billion global project that began on October 1, 2006, and will end on September 30, 2011. The Project is implemented by John Snow, Inc. (JSI) (contract no. GPO-I-01-06-00007-00) and its subcontractor partners (Program for Appropriate Technology in Health [PATH], Crown Agents Consultancy, Inc., Abt Associates, Fuel Logistics Group Ltd., UPS Supply Chain Solutions, Family Health International [FHI], The Manoff Group, Inc., and 3i Infotech). The DELIVER Project’s overall purpose is to design, develop, strengthen, and upon request, operate safe, reliable and sustainable supply systems that will provide a range of affordable, quality essential health commodities to clients in country programs. USAID/Ethiopia implements activities under this global project through the use of field support funds. As part of its overall evaluation of all HAPN projects, USAID/Ethiopia is commissioning this mid-term assessment to examine the following: • The project’s progress toward achieving results against stated objectives. • Identify strengths and weaknesses within the project’s portfolio. • Determine areas and activities that may warrant continued investment, as well as other key initiatives and approaches not covered by the project, but which would likely contribute to improving the availability of family planning and other essential health commodities. The Project employs key short and long-term interventions in the following areas: Ethiopian Contraceptive Logistics System (ECLS): To assist the FMOH and Regional Health Bureaus (RHBs) in the design, management, and implementation of a public sector information system for contraceptives. This is a short-term intervention in the absence of an eventual transition to an integrated logistics system for all health commodities, as part of the government’s long-term Pharmaceutical Logistics Master Plan. Contraceptive Security (CS): Work closely with the FMOH and RHBs to improve the long￾term availability of contraceptives in Ethiopia. Activities include policy/advocacy work, facilitating CS stakeholder meetings, and conducting national contraceptive forecasts. Pharmaceutical Logistics Master Plan (PLMP): Support and assist in the design and implementation of the government’s Pharmaceutical Logistics Master Plan for health commodities. The DELIVER Project is the lead coordinating contractor for Master Plan activities, and also collaborates with UNICEF and additional USG partners -- the RPM+ (transitioning to SPS) and SCMS projects -- that are also working on various components of the Master Plan. Key Master Plan activities under DELIVER include designing and working with the MOH to implement the logistics management information system (LMIS), improve warehouse management at health centers and hospitals and provide forcasting assistance. DELIVER maintains an office in Addis Ababa and has additional technical staff located in regions throughout Ethiopia – with a collective total of approximately 20 professional/staff members. The Project works with diverse partners in Ethiopia, including the Government of Ethiopia (Family Health Department, PSLD), multilateral/donor agencies (i.e. UNFPA, UNICEF, Clinton Foundation), and other CAs (Pathfinder International, Crown Agents, Carter Center, SCMS, RPM+). Indicators The primary indicator in USAID/Ethiopia’s Operational Plan for “improving government logistics systems” is “stock out rates.” This indicator is used for maturing systems. Historically, USAID has struggled to define indicators for government logistics systems and this may be a constraint for the evaluation team. IV. Purpose of the Evaluation The external assessment team will be expected to develop a detailed scope of work and plan of action prior to the evaluation. Given the complexity and diversity of the DELIVER Project’s portfolio, much of the assessment should be qualitative in nature – relying on key informant interviews with a range of partners. The assessment team will have three main tasks: 1) Task 1: Review the USAID | DELIVER PROJECT’s technical and programmatic strengths, weaknesses, successes and constraints, identifying contributing factors. Based on the assessment findings, the team will present results achieved to date, document lessons learned, and make recommendations towards achieving planned results in the remaining period of project implementation (Project ends September 30, 2011). 2) Task 2: Evaluate the Project’s staffing and management structure. 3) Task 3: Identify project activities and approaches that warrant continued investment, as well as new areas and approaches not covered by the project, but which would likely contribute to improving the availability of family planning commodities and strengthening the overall public sector supply system. Illustrative questions to assist in the assessment are provided below. The assessment team is expected to refine, prioritize and finalize evaluation questions in discussion with USAID at the start of the assessment. TASK 1: Assess Progress to Date in Achieving Planned Results (Estimated level of effort – 50%) 1) What has been the USAID | DELIVER PROJECT’s progress to date in achieving planned results and performance indicators? 2) What have been the Project’s most important lessons learned to date? 3) How has DELIVER supported the Contraceptive Security strategy in Ethiopia? How has Contraceptive Security improved since DELIVER began work in Ethiopia, and to what extent are improvements attributable to project activities? 4) What contributions has DELIVER made in increasing the availability of family planning commodities in Ethiopia? 5) Are DELIVER workplan activities (short and long-term) compatible with the Pharmaceutical Logistics Master Plan (PLMP)? Are DELIVER’s comparative strengths best being leveraged in the implementation of the Pharmaceutical Logistics Master Plan? 6) Are Logistics Management Information System (LMIS) development goals being met? Are additional monitoring processes needed to ensure this important and complex project activity stays on track? If so, please suggest new monitoring systems or implementation strategies based on evaluation results. 7) To what extent has DELIVER coordinated and actively collaborated with other CAs to implement the Master Plan. 8) How is DELIVER strengthening the overall national supply system? 9) What monitoring and evaluation systems are in place to assess DELIVER’s impact, including for sustainability of project results? Do the systems identify implementation problems so that corrective action can be taken? 10) Please also identify key barriers/constraints that have moderated projects results. TASK 2: Evaluate DELIVER’s Staffing and Management (Estimated level of effort – 10%) 1) Is the project’s current decentralized staffing structure (with regional technical advisors throughout the country) effective and efficient in achieving planned results? Does the DELIVER regional staff feel adequately supported by its central office in Addis Ababa? 2) Is the DELIVER management structure able to identify and reinforce good staff performance as well as strengthen weak performance? 3) Is the current staff skill set responsive in meeting work plan demands? Are there additional skill sets that are needed? 4) What are the most significant management challenges currently faced by DELIVER (budget, staffing retention, etc.)? 5) Does the local office feel adequately supported by its headquarters office? By USAID? TASK 3: Identify Additional Approaches or Activities to Achieve Objectives (Estimated level of effort – 40%) 1) What are the key project initiatives, activities, and approaches that warrant continued investment in the future (consider each of the workplan components)? 2) Are there initiatives, activities, and approaches that should be scaled back, reformulated or eliminated altogether? 3) Are there other promising initiatives, activities, and approaches not addressed by the project that should be considered for future investment to maximize impact? 4) What are the assessment team’s expectations regarding the project’s future progress? 5) What are the contributions of DELIVER implementation of the PLMP? V. Methodology of the Evaluation The evaluation will be conducted by a team of professional and independent external consultants over a period of approximately four weeks. The methodology of data collection will include: Pre-Assessment Briefing: The assessment team will hold a preliminary meeting (even virtual) with the management team of USAID to review the scope of the mid-term assessment, agree on the key research questions, and finalize the schedule. The outcome of this meeting will be a detailed work plan for the assessment, including milestones and deliverables with due dates clearly established. In addition to formal briefing and debriefing meetings, the assessment team may contact the USAID management team as necessary to provide updates on their progress and obtain additional guidance on logistics, additional data and information sources. Document Review: USAID and DELIVER will provide the assessment team with a package of briefing materials relevant to the assessment. This documentation will include strategy/concept papers, project quarterly reports, workplans, and any management reviews that are developed and reviewed as part of the continuous monitoring of the project. Field Visits: The assessment team is expected to conduct site visits to areas in which the DELIVER Project implements substantial activities, including Oromia, Amhara, Tigray and Dire Dawa. Key Informant Interviews: The assessment team will conduct qualitative, in-depth interviews with key stakeholders and partners (a preliminary list of stakeholders and partners is attached in Annex 1, but the assessment team should add to this list as necessary). Key informants should include, but not be limited to: • USAID/Ethiopia staff, particularly the DELIVER Project management unit. • The Ministry of Health (FHD, PSLD). • USAID | DELIVER PROJECT staff (representative of both the Addis Ababa office and regional offices). • In-country partners, including UNICEF and UNFPA. • Other Cooperating Agencies, including RPM+ and SCMS. VI: Identification of Information Sources Consultants will be provided the background documents in preparation of the assignment. These will include, but not be limited to the following: • The USAID | DELIVER PROJECT self-assessment • The USAID | DELIVER PROJECT global contract • The MOH Master Plan strategy and other relevant policy documents • The MOH Master Plan matrix of implementing partners • DELIVER work plans • DELIVER quarterly reports • DELIVER management reviews and memos • DELIVER Performance Monitoring Plan • DELIVER Contraceptive Security Strategy • Other relevant technical reports/concept papers VII: Tasks to be accomplished Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort for each task. Team reviews background documents/Develops draft evaluation work plan and site visit schedule/ submits to HAPN prior to departure 3 days TPM meeting at TMG in Washington, DC 2 Days Travel 1 Day Team In briefing with USAID/Ethiopia/ Team Planning/ Team stakeholder interviews in Addis 1 Day Team Planning/ Team stakeholder interviews in Addis 4 Days Full Team Planning Meeting/Site Visits in Addis 2 Days Full Team Field Visits 7 Days Full Team Synthesis (.5 day) Core Team Draft Report (.5 day) 2 Days Core team conduct debriefings for USAID and JSI (separately) 1 Day Core team completes Draft report in-country and departs 3 days Travel 1 Day Incorporation of Mission comments info final report (TL 3; TM 2) 3 Days The evaluation is expected to be conducted in Ethiopia and completed over a four week period. Total LOE – 30 days of LOE for Team Leader and 26 days for Team Members, including travel days. A six-day work week is assumed. VIII: Team Composition and Participation USAID seeks three member assessment team composed of a Team Leader, Senior Local Team Member and Logistics Assistant. Between them, the team members should have substantial knowledge of international public health, what it takes to design and implement a national distribution system for health commodities, and the working context of Ethiopia. Specifically, team members should have between them: 1. 5-10 years of experience in supply chain management for health commodities. Particular experience in the area of family planning/reproductive health and related health commodities would be beneficial. 2. Expertise in designing and implementing a logistics management information system (LMIS). LMIS development is an integral and very complex part of the DELIVER Project’s scope of work in Ethiopia; this skill set would be a helpful determiner for the direction of future project activities in this area. 3. 5-10 years of experience in the area of research, monitoring and evaluation. USAID project management experience is preferable, as is program implementation within a developing country context. In addition, each team member should have, at minimum, the following skills and experience: 1. An understanding of the Ethiopia country context. 2. An advanced degree in Public Health, Social Sciences, Business Administration, or other relevant course of study. 3. Demonstrated skill in written and oral communication. 4. Demonstrated knowledge of USAID policies and procedures. 5. Ability to work effectively in, and communicate with, a diverse set of professionals. 1. The Team Leader will be a senior expatriate with extensive experience in supply chain management and must have excellent English language skills (both written and verbal) as s/he will have the overall responsibility for pulling together the different elements of the assessment for the final report. Particular experience in the area of family planning/reproductive health and related health commodities would be beneficial. S/he will agree to fulfill his/her responsibilities in approximately four weeks, spending up to three weeks in-country, and will play a central role in guiding the evaluation process. The consultant may hold a conference call with core team members and USAID/E representatives before and after the visit to Ethiopia, if needed. The Team Leader will: • Discuss the team Work Plan for the assignment with USAID/E and finalize the work plan based on USAID/E comments. • Define assignment roles, responsibilities, and tasks for team members. • Oversee logistics arrangements in the field. • Participate in aTMG Team Planning Meeting (TPM) should it be required. • Lead the preparation of and coordinate team member input, submitting, revising and finalizing the report. • Lead team meetings. • Coordinate and support the team on tasks and ensure that team works effectively. 2. The second team member will be a local senior consultant with expertise in supply chain management. This senior consultant should also have extensive experience in program monitoring and evaluations. S/He will be responsible for designing appropriate tools for data collection and analysis. Other Team Participants: This evaluation may include USAID/Washington staff and up to 4 GOE experts from the Regional Health Bureaus (RHBs) and/or Ministry of Health (MOH). USAID/E staff may also join the Evaluation Teamduring the site visits. JSI partner agencies may accompany the team on site visits as appropriate, but will not be present during interviews with stakeholders or beneficiaries. Evaluation Logistics: Evaluation Logistics will be provided by the local sub-contractor hired by TMG with support staff who are fluent in Amharic, with a demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision; ability to work collaboratively with a range of professional counterparts. The local sub-contractor will be responsible for logistics, coordination and administrative support. TMG’s local sub-contractor staff will assist the Team in facilitating meetings, coordinating logistics and organizing site visits. As needed, the local sub-contractor will collect and disseminate background documentation to the evaluation team. TMG will be responsible to manage and direct the efforts of local sub- contractor. Evaluators Selection Criteria for Team (Maximum 100 %) distributed as follows: Team Leader 1. Education: (25 %) An advanced degree (Master’s and above) in Public Health, Social Sciences, Business Administration, or other relevant course of study. 2. Work Experience: (35 %) – 5-10 years of experience in supply chain management and/or program evaluation for health commodities. Particular experience in the area of family planning/reproductive health and related health commodities would be beneficial. 3. Skills and Abilities: (40 %) - Demonstration of strong analytical, managerial and writing skills are very critical for the evaluation work. Exceptional leadership in coordinating, assigning the team with appropriate responsibilities, communication, and interpersonal skills are absolutely critical. In addition, the Team Leader must be able to interact effectively with a broad range of internal and external partners, including International Organizations, Host Country Government Officials, and NGO counterparts. Must be fluent in English and have proven ability to communicate clearly, concisely and effectively both orally and in writing. Must be able to produce a quality document that can give direction and facilitate implementation similar Urban Agriculture Projects in the future. Local Team Member 1. Education: (25 %) An advanced degree (Master’s and above) in Public Health, Social Sciences, Business Administration, or other relevant course of study. 2. Work Experience: (35 %) – 5-10 years of experience in supply chain management and/or program evaluation for health commodities. Particular experience in the area of family planning/reproductive health and related health commodities would be beneficial. 3. Skills and Abilities: (40 %) – Has track record for strong analytical, managerial and writing skills. Is expected to write portions of the evaluation report, assist in the development, pre￾test, and fielding of both qualitative and quantitative data collection instruments. IX. Relationships and Responsibilities Overall Guidance: The USAID/Ethiopia Health Team will provide overall direction to the assessment team. USAID Contacts: Karen Towers, M&E Advisor, Anita Gibson, Health Team Leader, and Eshete Yilma, USAID | DELIVER PROJECT Activity Manager, will be the official contacts for the assessment team. 1. USAID | DELIVER PROJECT Contact: Jeff Sanderson, Chief of Party. Responsibilities: • The Mitchell Group will be responsible for obtaining country clearances for travel for consultants. • Consultants will be responsible for coordinating and facilitating assessment-related field trips, interviews, and meetings in conjunction with the DELIVER Project contractor. • Consultants will be responsible for submitting a budget for all estimated costs incurred in carrying out this review. The proposed cost may include, but not be limited to: (1) international and in-country travel; (2) lodging; (3) M&IE; (4) in-country transportation; and (5) other office supplies and logistical support services (i.e., laptop, communication costs, etc.) if needed. X. Schedules and Logistics The TMG Sub-contractor, in collaboration with JSI, will arrange all stakeholder/partner meetings and site visits. Meeting space will be provided at USAID, but the agency cannot provide access to fax and e-mail. All associated travel and per diem costs will be covered under the contract with USAID/E. XI. Period of Performance Work is to be carried out over a period of approximately four weeks (three weeks in-country). The in-country portion of the evaluation will begin on or about (o/a) November 7, 2008, and conclude o/a November 28, 2008 (not including approx four weeks time for USAID/E (up to ten days) comment and completion of final editing of the Draft Evaluation Report by TMG (3 weeks). XII. Financial Plan A budget agreement between the USAID/Ethiopia and TMG will be reached and USAID/E will approve the evaluation activity by TMG under the USAID/Ethiopia Evaluation Program. XIII. Deliverables Prior to arrival (preferably two weeks): Team leader will develop a Draft Work Plan with evaluation methodology and field visit and interview schedule in consultation with the USAID/E CTO, USAID/E Evaluation Coordinator, TMG, and DAI. The Draft Work Plan will clearly present roles and responsibilities, a planned interview schedule, and an analysis plan of who will be responsible for writing various sections of the report. Three days after Team arrival: Team meeting and in-briefing with USAID/E. USAID/E HAPN technical staff to review and comment on evaluation methods. The Draft Work plan will be presented and discussed with Mission Staff to ensure the assessment is on track and can be met on time. After agreement, the Draft becomes a Final Work Plan. Prior to departure: Team makes presentation to USG PEPFAR staff, a separate presentation to HCRCP partners, and Team Leader submits a draft report in the format specified by the USAID/E Evaluation coordinator (See separate MS Word file for TMG Evaluation Report Guidelines) to USAID/E CTO - two hard copies and one electronic copy on CD ROM or flash drive. After departure: Team leader submits final draft report content to USAID/E within one week of receiving comments from USAID/E. The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, Conclusions, and Recommendations in English in a format specified by the USAID/E Evaluation Coordinator in consultation with TMG. (See Annex 2 for Proposed Outline of Final Assessment Report.) Upon final approval of the content by USAID/E, TMG will have the report edited and formatted within three weeks. The final report will be submitted electronically to USAID/E CTO and Contract Officer. TMG will make the results of its evaluations public on the Development Experience Clearinghouse and on its project web site unless there is a compelling reason (such as procurement sensitivities) to keep the document internal. Therefore, TMG will request USAID/E confirmation that it will be acceptable to make this document publicly available. If there are certain restrictions regarding specific parts of the report that should be removed from a public version due to procurement-sensitive information, TMG will produce a second version suitable for public availability. ANNEX 1: PROPOSED MEETING LIST The assessment team should add to this preliminary list of stakeholders and partners as necessary. USAID Meri Sinnitt Anita Gibson Eshete Yilma Fikru Bekele Kassahun Deneke Jamie Browder MOH Dr. Kebede Worku Berhanu Feyisa, Head, PSLD Tesfaye Seifu, PSLD Sr. Woinshet, FHD Alemhayu Lemma Dr. Mehdin Zewdu Dr. Hassan Mohammed, Head, Health Services Dept. UNICEF Viviane van Steirteghem Jan Debyser Able Kuiper Rory Nefdt Jurgen Hulst, Copenhagen SCMS Hany Abdallah Francis “Kofi” Aboagye-Nyame (in Arlington, VA) RPM+ Negussu Mekonnen Pathfinder Dr. Mengistu Asnake DKT Andrew Pillar Clinton Foundation Ruth Lawson Lillian Kidane UNFPA Kidane Gebrekidane Dr. Michael Tekie Addis Ababa Health Bureau Dr. Hassan Mohammed Oromia Health Bureau Dr. Kassa Ato Ababu Beshane, FP Team Leader SNNPR Regional Health Bureau (RHB) Dr. Sahle Sita, FP Team Leader Ato Bassamo Daka, FP Team Leader Other Amhara, Dire Dawa, Harari RHBs Gelila Kidane, EngenderHealth ANNEX 2: PROPOSED OUTLINE FOR ASSESSMENT REPORT TABLE OF CONTENTS ACKNOWLEDGEMENTS ACRONYMS AND ABBREVIATIONS EXECUTIVE SUMMARY INTRODUCTION Purpose Background Methodology Limitations FINDINGS Program (include section on program constraints) Technical Management CONCLUSIONS LESSONS LEARNED PRIORITIZED RECOMMENDATIONS FUTURE DIRECTIONS ANNEXES: Assessment Scope of Work Annotated List of Documents Collected and Reviewed Persons Contacted ANNEX 3: Illustrative Planning Calendar for The DELIVER Project – All dates subject to change October 28, 2008 Monday Tuesday Wed Thurs Fri Sat Sunday October 27 October 28 October 29 Team Document Review/Draft Evaluation Plan and Site Visit Schedule October 30 Team Document Review/Draft Evaluation Plan and Site Visit Schedule; Conference Call with USAID/E to discuss SOW October 31 Team Document Review/Draft Evaluation Plan and Site Visit Schedule in consultation with TMG, JSI and USAID/E November 1 November 2 November 3 TPM in DC November 4 AM: TPM in DC PM: Team Leader travels to Ethiopia November 5 Travel Day for Team Leader November 6 Team Leader arrives in Ethiopia November 7 AM: In-briefing w USAID/Ethiopia PM: Meeting with JSI November 8 Full Team Planning Meeting; Tools Development November 9 Sunday Rest November 10 Stakeholder Interviews November 11 Stakeholder Interviews HOLIDAY￾VETEREN’S DAY November 12 Depart for Dire Dawa and Harar Site Visits November 13 Full Team Sites Visits in Dire Dawa and Harar November 14 Full Team Sites Visits in Dire Dawa and Harar November 15 Return to Addis November 16 Drive to Awassa November 17 Awassa Site Visit November 18 Awassa Site Visit November 19 Awassa Site Visit/Return to Addis November 20 Follow up Stakeholder Interviews in Addis to fill in gaps November 21 Follow up Stakeholder Interviews in Addis to fill in gaps November 22 Full Team info synthesis/ stakeholder interviews November 23 November 24 Team Leader & Team Member Draft Outline of Report/Powerpoint November 25 Team Leader & Team Member Prepare slides for De-brief November 26 AM: Core Team Oral Debrief to USAID/Ethiopia PM: Core team Oral Debriefing to Stakeholders November 27 Core Team Draft report HOLIDAY￾THANKSGIVING November 28 Core Team Draft report Team Leader Returns to US November 29 November 30 December 1 December 2 Core Team Finalize and Submit Report to USAID December 3 December 4 December 5 December 6 December 7 . Annex B Evaluation Work Plan Mid-Term Project Evaluation of Contract No. GPO-1-01-06-00007-00 The USAID | DELIVER PROJECT: Ethiopia EVALUATION WORK PLAN The Mitchell Group November 10, 2008 I. INTRODUCTION USAID/Ethiopia has requested this mid-term assessment of DELIVER to examine the following: • The Project’s progress toward achieving results against stated objectives. • Identify strengths and weaknesses within the Project’s portfolio. • Determine areas and activities that may warrant continued investment, as well as other key initiatives and approaches not covered by the project, but which would likely contribute to improving the availability of family planning and other essential health commodities. II. SCOPE OF THE EVALUATION Because the DELIVER Project has documented its key objectives and achievements to date, as part of its self-evaluation, this mid-term evaluation is therefore not primarily concerned with quantitative measures of the Project’s success to date. Rather, the evaluation will qualitatively assess the extent to which the Project has met its strategic objectives and make recommendations to inform the future direction of the Project. The evaluation will examine four Project components: 1) The Ethiopian Contraceptive Logistics System (ECLS) 2) Contraceptive Security (CS) 3) Pharmaceutical Logistics Master Plan (PLMP) 4) The Project’s Staffing and Management III.METHODOLOGY The Evaluation Teamwill conduct a desk review of all relevant material, including the USAID | DELIVER PROJECT’s mid-term self evaluation, country strategy report, annual work plans, and quarterly reports. The team will also conduct key informant interviews with a range of in-country partners (Table 1, as discussed below, lists organizations interviewed), including the Ministry of Health, USAID/Ethiopia, and the USAID | DELIVER PROJECT. Additionally, the team will conduct site visits to Dire Dawa, Harar and SNNPR/Awasa. IV. THE EVALUATION FRAMEWORK A draft Evaluation Framework was presented and discussed with USAID/Ethiopia at the time of the Evaluation In-briefing on November 7, 2008, and was found by USAID/Ethiopia to be acceptable. Table 1 is a refined Evaluation Framework in the format of a matrix that shows for each of the four Project components: • An outline of subjects to be covered in the assessment and “Findings and Recommendations” sections of the report • Sources of information and informants • Assignment of Evaluation Teamresponsibilities V. EVALUATION QUESTIONS Based on the USAID/Ethiopia In-briefing and Evaluation Teamplanning sessions, evaluation questions (as first presented in the SOW) have been refined and prioritized as shown in Table 2. VI. EVALUATION SCHEDULE AND DELIVERABLES Table 3 shows the schedule of tasks and deliverables to be completed. Table 1 - Mid-Term Project Evaluation, The DELIVER Project/Ethiopia EVALUATION FRAMEWORK 11/10/08 Project Component 1- Ethiopian Contraceptive Logistics System (ECLS) Project Component 2 - Contraceptive Security (CS) Project Component 3 - Pharmaceutical Logistics Master Plan (PLMP) Project Component 4 - Project Staffing/Management Assessment & Report Outline FINDINGS A. Background B. Project Interventions C. Achievements D. Challenges/constraints CONCLUSIONS A. Improved or Additional Approaches/Activities to Achieve Objectives B. Lessons Learned RECOMMENDATIONS A. Prioritized Directions B. Future Directions Information Sources Desk Review JSI/DELIVER Central MOH Regional Bureaus Site Visits UNFPA Information Sources Desk Review Federal MOH Regional Health Bureaus PFSA Pathfinder DKT Packard Foundation UNICEF UNFPA JSI/DELIVER Information Sources Desk Review Federal MOH PFSA USAID SCMS SPS Pathfinder UNICEF DKT JSI/DELIVER Information Sources Desk Review JSI/DELIVER Assessment & Report Categories Systems Development Systems/Program Development Program Development Management Team Assignments Dr. Tsige Gebre￾Mariam Dr. Tsige/M. Howard Mildred Howard, T.L. Mildred Howard, T.L. Table 2 – Mid-Term Project Evaluation, the DELIVER Project/Ethiopia PRIORITIZED KEY QUESTIONS Ethiopian Contraceptive Logistics System (1) What progress has been made in achieving planned results and performance indicators? (2) Are LMIS development goals being met? To what extent is ECLS now institutionalized and sustainable? (3) How are LMIS data utilized at operational levels for decision-making? (4) Is there adequate monitoring and supervision at lower level facilities? (5) Have there been improvements in the “culture” for recording and reporting? (6) Are additional monitoring criteria/processes or implementation strategies needed? (7) What have been key barriers/constraints to achieving results and development goals? (8) What is the Project’s outlook for continued progress and gains in the area of ECLS in the future? (9) What approaches warrant continuation, scale up, reformulation or elimination? (10) What additional approaches should be considered? (11) What are noteworthy lessons learned? Pharmaceutical Logistics Master Plan (1) What progress has been made in achieving planned results and performance indicators? (2) How is the Project strengthening the overall national drug supply system? (3) Are short and long-term Project activities compatible with the PLMP? (4) How well is the Project able to carry out its designated role of coordinating Support Partners and Support Agencies in implementation of the Master Plan? (5) Are the Project’s comparative strengths being leveraged in the implementation of the PLMP (i.e. LMIS)? (6) What are key barriers/constraints to the Project being able to achieve results and development goals? (7) What is the Project’s overall outlook for continued progress and gains in the area of PLMP in the future? (8) What approaches warrant continuation, scale up, reformulation or elimination? (9) What additional approaches should be considered? (10)How can the Project be better supported in its lead role in coordinating the Master Plan? Contraceptive Security (1) What progress has been made in achieving planned results and performance indicators? (2) Has the Project adequately facilitated and supported an overall Contraceptive Security strategy in Ethiopia? (3) Has the Project adequately facilitated coordination among donors, partners and the MOH in the area of Contraceptive Security? (4) Has Contraceptive Security improved? Can improvements be attributed to the Project? (5) What impact has the Project had on systematic forecasting of contraceptive supply needs? (6) What impact has the Project had on donor coordination and financing of contraceptives? (7) To what extent have Project initiatives in Contraceptive Security become institutionalized? Likely to be carried over into implementation of the Master Plan? (8) What have been key barriers/constraints to achieving results and development goals? (9) What approaches warrant continuation, scale up, reformulation or elimination? Project Staffing and Management (1) What monitoring and evaluation systems are in place to assess the Project’s impact and the sustainability of results? (2) Do M&E systems identify implementation problems so that corrective actions can be taken? (3) What internal processes are in place for on￾going assessments of the strategic direction of the Project? (4) To what extent are strategic thinking and on￾going management and mid-stream corrections supported by USAID? By JSI? (5) Is the decentralized Project structure efficient and effective? (6) Is the central office able to adequately support its regional staff? (7) Does the Project’s management structure allow staff performance to be reinforced or strengthened? (8) Does the existing staff reflect the right mix of technical skills needed to achieve objectives? (9) Is additional staff development indicated? (10) What are the most significant management challenges currently faced by the Project? Table 3 – Mid-Term Project Evaluation, the DELIVER Project/Ethiopia SCHEDULE OF TASKS AND DELIVERABLES TARGET DATES TASKS AND DELIVERABLES October 29- November 4 Prior to arrival: Team leader will develop a Draft Work Plan with evaluation methodology and field visit and interview schedule in consultation with the USAID/E CTO, USAID/E Evaluation Coordinator and TMG. The Draft Work Plan will clearly present roles and responsibilities, a planned interview schedule, and an analysis plan of who will be responsible for writing various sections of the report. November 7 One day after Team arrival: Team meeting and in-briefing with USAID/E. USAID/E HAPN technical staff to review and comment on evaluation methods. The Draft Work plan will be presented and discussed with Mission Staff to ensure the assessment is on track and can be met on time. After agreement, the Draft becomes a final Evaluation Work Plan. November 10-21 Stakeholder Interviews and Site Visits: Visits to Dire Dawa and Harar, November 12- 14; Visit to Awasa, November 16-18. November 19 Informal discussions and evaluation feedback with DELIVER November 22-27 Oral presentations and report preparations November 26 November 28 November 28 Prior to departure: Team makes presentation(s) to USAID/Ethiopia, and a separate presentation to HCRCP partners. Team Leader submits a draft report in the format specified by the USAID/E Evaluation coordinator (See separate MS Word file for TMG Evaluation Report Guidelines) to USAID/E CTO - two hard copies and one electronic copy on CD ROM or flash drive. USAID/E Evaluation Coordinator de-briefs with Team Leader and provides general, verbal feedback/comments on the oral presentations and suggestions for the draft report prior to Team Leader’s departure. December 5 December 12 After departure: Team leader submits final draft report content to USAID/E within one week of receiving comments from USAID/E. The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, Conclusions, and Recommendations in English in a format specified by the USAID/E Evaluation Coordinator in consultation with TMG. (See Annex 2 or SOW for Proposed Outline of Final Assessment Report.) TMG verifies with USAID/E that the Consultant’s final draft report content is acceptable and meets all requirements of the SOW. Upon final approval of the content by USAID/E, TMG will have the report edited and formatted within three weeks. The final report will be submitted electronically to USAID/E CTO and Contract Officer. Annex C List of Persons Interviewed Annex: Names of representatives of partner institutions, health officers, health care providers, and health facilities visited Name Position Institution Jeff Sanderson Country Director DELIVER/JSI Anita Gibson Health Team Leader USAID Ethiopia James Browder HIV/AIDS Officer USAID Ethiopia Sophia Brewer Evaluation Coordinator for Health & HIV/AIDS Programs USAID Ethiopia Eshete Yilma Deputy Team Leader Health USAID Ethiopia Aster Letta Nurse, Family Planning Expert AA Health Bureau Dile Tadesse Druggist, Store Manager AA Health Bureau Warehouse Bethlehem Nega Druggist, Store Manger Bole Woreda 17 Health Center Melkam Wondimenmeh Nurse, Coordinator, HB, Clinic and HP Bole Woreda Health Bureau Gizaw Habtewold FHD Head AA Health Bureau Andrew Piller Country Director DKT Ethiopia Gelila Kidane (Dr) Country Director Engender Health Ethiopia Woyeneshet Nigatu Nurse, FP Head FMOH Berhanu Feyssa Head, PSLD MOH Wonwossen Ayele Deputy Director PFSA Michael Tekie (Dr.) Country Representative UNFPA Tilahun Gidaye Country Representative Pathfinder Sami Tewfik Edris Pharmacist, Regional Logistics Manager DELIVER/JSI Meselu Atnafe Head Nurse Chora Hospital, Dire Dawa, Hilawe Hawaz Pharmacist and Store Manager Sabian Health Center, Dire Dawa Demmelash Assefa Store Manager Dire Dawa RHB Warehouse Tsegereda kifle (Dr) Health Bureau Head Dire Dawa Region Tesfaledet Gebre-Mariam Head Nurse Legahre HC, Dire Dawa Kemal Abdi Vice Head, RHB Harai Region Abrahim Salili Pharmacist, Warehouse Head Harai Region Nejeha Abdush Nurse, FP Head, Harai RHB Harai Region Tewodros Bogale Head Nurse, Hiwot Fana Hospital Harai Region Gebtsit Gebrekidan Clinical nurse, Hassenge HC Harai Region Melaku Legesse Public Health Officer, Logistics Head, SNNP DELIVER/JSI Mohammed Feleke Druggist, Store Manger SNNP, Regional warehouse Wubeshet Mekuria Head, Health Department Gedowo Zonal Health Bureau Aneko Sorsa FP Coordinator Gedowo Zonal Health Bureau Desalegne Fanta Head Pharmacist Gedowo Hospital Bezawit Asmerom Druggist, Warehouse Manager Gedowo Zonal Health Bureau Munaye Teklu Pharmacy Technician, Store Keeper Mesenkela HC, Sidama Zone Sister Segenet Daniel MCH coordinator Mesenkela HC, Sidama Zone Teshome Deres Pharmacist, Logistic Officer, SNNP DELIVER/JSI Tesfanesh Mamo Nurse, MCH coordinator Awassa HC, Sidama Zone Binaiam Tesfaye Druggist, Store Manager Awassa HC, Sidama Zone Tenagene Kebede Nurse, Head, Communicable Diseases Prevention Wondo Genet Woreda HO Hany Abdallah Resident Advisor SCMS Mike Healy PFSA Technical Advisor SCMS Gashaw Shifera ACTS Coordinator SCMS Viviane Van Steirteghem Deputy Representative UNICEF Negussu Mekonnen Chief of Party MSH/SPS Laike Gebre-Selassie Deputy Director MSH/SPS Yemeserach Belayneh Program Coordinator, Population Program Packard Foundation Annex D List of Documents Reviewed List of Documents Reviewed "Recommendations form the JSI/Deliver Transportation/Distribution Study", November￾December 2003 "Logistics Indicators Assessment Tool (LIAT), Final Report", 2006 "Health Commodities Supply System (HCSS) Priorities", December 2006, Revised "Accountability Matrix, Support to the HSCC Master Plan", February 2007 "Pharmaceutical Logistics Master Plan (formerly Health Commodities Supply System), PLMP Summary", October 2007, Revised "The National Health Commodity Supply System, Trainers Guide for Hospitals and Health Center Personnel", April 2008 "The National Health Commodity Supply System, Participant Course Workbook, Hospitals and Health Center Personnel", April 2008 "Ethiopia: Pharmaceutical Logistics Master Plan, Integrated LMIS System Design Work, Workshop Proceedings, Addis Ababa", December 2007 "An Overview of Outstanding Policy and Procedure Issues related to the Design of the Logistics Management Information System (LMIS)", May 2008 "National Contraceptive Forecast 2006-2010", July 2006 "Contraceptive Inventory", April 2007 "Contraceptive Security Strategy", February 2008 "SCMS: Providing quality medicines for people living with and affected by HIV and AIDS" "What is the USAID│DELIVER PROJECT?" October, 2007 "USAID│DELIVER PROJECT Ethiopia Project Objectives and Strategic Direction 2006-2011", July 2008, Revised "USAID│DELIVER PROJECT Ethiopia Monitoring, Evaluation and Reporting", October 2007 " USAID│DELIVER PROJECT Task Order I Annual Report", January 2008 "USAID│DELIVER PROJECT Ethiopia, Country Quarterly Report – Routine Contractual Information" Quarters: October – December, 2007 January – March, 2007 April – June, 2007 July – September, 2007 October – December, 2008 January – March, 2008 April – June, 2008 July – September, 2008 "USAID│DELIVER PROJECT Ethiopia FY2008-09 Work Plan", August 2008 "USAID│DELIVER PROJECT Ethiopia Mid-term Self-evaluation, November 2008"