External Project Evaluation of The NetMark Plus   Malaria Social Marketing Program Final Report December 2008 External Project Evaluation of Agreement No HRN-A-00-99-00016-00 The NetMark Plus Malaria Social Marketing Program This document was submitted by The Mitchell Group, Inc. 1816 11TH Street, NW Washington, DC 20001, USA Telephone: (202) 745-1919 Facsimile: (202) 234-1697 to The United States Agency for International Development/Ethiopia under USAID Contract 663-C-00-08-00409-00 Authors: Dr. Afework Hailemariam Tekle Dr. Valery Lodewyk Gilbos Disclaimer: The authors’ views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the United States Government. i Table of Contents 1 EXECUTIVE SUMMARY ................................................................................................ 1 1. Purpose, Scope and Methods ...................................................................................... 1 2. Overview of the Project ............................................................................................... 1 3. Key Findings and Conclusions .................................................................................... 1 4. Key Lessons Learned .................................................................................................. 2 5. Key Conclusions and Recommendations .................................................................... 3 6. Impacts ........................................................................................................................ 3 7. Management ................................................................................................................ 4 8. Advocacy and Certification ......................................................................................... 4 9. Linkages ...................................................................................................................... 5 10. Capacity Building For IEC ...................................................................................... 5 11. Data And Information For Program Design And Adaptation ................................. 5 2 INTRODUCTION .............................................................................................................. 2 1. Malaria in Ethiopia ...................................................................................................... 2 2. National Malaria Response in Ethiopia ....................................................................... 2 3. Current Malaria Programs in Ethiopia ........................................................................ 3 3 USAID/ETHIOPIA’S RESPONSE TO "ROLL BACK MALARIA" ............................... 3 4 EVALUATION PURPOSE ................................................................................................ 5 5 EVALUATION METHODOLOGY .................................................................................. 5 1. Team Composition ...................................................................................................... 5 2. Time Planning ............................................................................................................. 5 3. Methodology ............................................................................................................... 5 4. Constraints ................................................................................................................... 5 6 FINDINGS.......................................................................................................................... 6 1. Background ................................................................................................................. 6 7 IMPACT ON OBJECTIVES .............................................................................................. 6 1. Objective 1: Improving Access and Demand .............................................................. 6 2. Objective 2: Targeted ITN subsidy program for pregnant women ............................. 9 3. Objective 3: Catalyzing local production of ITN products ....................................... 11 4. Impact on National Malaria Control Strategy and National ITN Strategy ............... 12 5. NM+ Input and Impact on the National ITN Strategy .............................................. 13 6. NM+ Efforts and Challenges .................................................................................... 13 7. Facilitation of registration ......................................................................................... 14 8 IMPACT ON HEALTH AND LIVES ............................................................................. 15 9 CHALLENGES AND RESPONSE ................................................................................. 18 1. Response to free net distribution ............................................................................... 18 Program Management .............................................................................................................. 21 10 STRATEGIES .................................................................................................................. 21 1. Effectiveness of NM+’s Planning and Implementation of Strategies and Interventions ..................................................................................................................... 21 11 MANAGEMENT ............................................................................................................. 22 12 PROCESSES .................................................................................................................... 23 13 OVERSIGHT.................................................................................................................... 23 Program Sustainability and Expandability 14 SUSTAINABILITY ......................................................................................................... 24 15 EXPANDABILITY .......................................................................................................... 25 16 REPLICABILITY ............................................................................................................ 26 ii 17 SPILL-OVER EFFECTS .................................................................................................. 26 18 LINKAGES ...................................................................................................................... 27 19 CAPACITY ...................................................................................................................... 28 20 EFFECTIVENESS ........................................................................................................... 29 21 SYSTEM .......................................................................................................................... 29 22 USEFULNESS ................................................................................................................. 30 23 LESSONS LEARNED ..................................................................................................... 31 24 CONCLUSIONS AND RECOMMENDATIONS ........................................................... 32 APPENDICES Appendix A: Statement of Work Appendix B: Agenda/Interviews Appendix C: NetMark Plus Ethiopia Timeline 2004-2008 Appendix D: Poll Results Discussion Appendix E: Opinion Questionnaire Responses Appendix F: Resources Reviewed List of Tables Table 1: Malaria control agencies and malaria activities in NM+ intervention areas ............... 3 Table 2: Total monthly and annual bednet sales by NM+ partners (2004-08) ......................... 8 Table 3: Utilization of ITNs by targeted vulnerable groups (2004-08) ..................................... 9 Table 4: TSP target population coverage (2004-07) ................................................................ 11 Table 5: Malaria related KAP changes (2004-07) ................................................................... 28 Table 6: USAID and NM+ opinions on program management and outcomes ........................ 14 Table 7: USAID and NM+ opinions on program management and outcomes ........................ 14 List of Figures Figure 1: Origin of ITN distributed in Amhara and Oromia Regional States. .......................... 4 Figure 2: Ethiopia Formal Partner Sales .................................................................................... 7 Figure 3: Evolution of TSP sites, sales and redemption rates (2004-08) ................................. 10 Figure 4: Malaria mortality trends under scale-up interventions (2003-07) ............................ 16 Figure 5: NM+ partner total bed net sales (2004-07) .............................................................. 17 Figure 6: Impact of Free Nets on NM+ Partner Sales (2004-2008) ........................................ 18 Figure 7: NM targeted subsidy sales (2005-07) ....................................................................... 19 Figure 8: Evolution of various types of NM partner sales ....................................................... 19 iii Acknowledgments The evaluation team would like to thank Alex McKenzie and Mikhail Dejene for organizing the evaluation contracts and the survey logistics. We are obliged to NetMark Plus staff at the regional and country levels for providing background documentation, for informing the evaluation exercise and for facilitating the field study visits. We are also indebted to the officials of the Federal Ministry of Health and various development agencies, as well as to several key informants from the private sector. Most of all, we are grateful to the many mothers and (fewer) men who willingly answered our probing questions in various Amhara and Oromia villages. The surveyors were pleased to be given the opportunity to conduct discussions with NetMark Plus and USAID at several stages throughout the exercise to provide progress briefings, explore lines of inquiry and test preliminary findings and conclusions. Special thanks are due to Rose Schneider for her careful and sustained long distance coordination. Addis Ababa - October 2008 Dr. Valery Lodewyk Gilbos Dr. Afework Hailemariam Tekle iv List of Acronyms N/A not available AAU Addis Ababa University ACT artemsinin-based combination therapy AED Academy for Educational Development AMD Africa Malaria Day ANC antenatal care ARHB Amhara Regional Health Bureau ARV anti-retroviral (drugs) ATL above the line BD Bahir Dar BTL below the line CAME Coalition Against Malaria in Ethiopia CIDA Canadian International Development Agency CM country manager CTO Cognizant Technical Officer CTO/PR Chief Technical Officer/ public relations DACA Drug Administration and Control Authority DC District of Columbia DDT Dichlorodiphenyltrichloroethane DFID Department for International Development DHS Demographic and Health Survey EAG EAI East Africa Group East Asian Industries FGA Family Guidance Association FMA full market approach FMOH Federal Ministry of Health FO field officer FY financial year GFATM Global Fund to Fight Aids, Tuberculosis and Malaria GOE Government of Ethiopia HC health centre HEC Health Education Centre HEW health extension worker HH household HIV/AIDS human immunodeficiency virus/acquired immunodeficiency syndrome HMIS health management information system HR human resources HSDP Health Sector Development Program IEC/BCC information education communication / behavior change communication IRS indoor residual spraying ITN insecticide treated net KAP knowledge attitudes practices LLIN long-lasting insecticide-treated net M&E monitoring and evaluation MC Malaria Consortium MCH Maternal and child health MDG Millennium Development Goals v MIS Malaria Indicator Survey MO marketing officer MOFED Ministry of Finance & Economic Development MOU memorandum of understanding MQ mosquito(es) NGO non-governmental organization NM NetMark NM+ NetMark Plus NMCP National Malaria Control Program NMCST National Malaria Control Support Team NOC National Oil Corporation OPD outpatient department ORHB Oromia Regional Health Bureau PEPFAR President’s Emergency Plan for Aids Relief PMI President’s Malaria Initiative PMTCT prevention of mother to child transmission PSI Population Services International QL qualitative QN quantitative RBM Roll Back Malaria RDT rapid diagnostic test RHB Regional Health Bureau SA South Africa SM social marketing SNNPR Southern Nations, Nationalities, and People's Region SOW scope of work SWOT strength, weakness, opportunities and threat T&T taxes and tariffs TA technical assistance TBS Tuberculostatics TCC The Carter Center TOR terms of reference TS targeted subsidy (voucher) TSP targeted subsidy program TT tax & tariff U5 under five years of age UNICEF United Nations Children’s Emergency Fund USAID United States Agency for International Development USAID/E USAID/Ethiopia UV ultra violet VAT value added tax V-F Vestergaard-Frandsen WHO World Health Organization WHOPES World Health Organization Pesticides Evaluation Scheme 1 1 EXECUTIVE SUMMARY 1. Purpose, Scope and Methods The purpose of this assignment was to conduct an independent external end-of-project evaluation of the NetMark Plus (NM+) Program in Ethiopia, covering the program's performance from 2004 through September 2008. The evaluation includes information collected about the program’s impact and achievements, implementation and progress, and challenges encountered. It documents lessons learned and formulates recommendations for future social marketing activities and alternate strategies in the context of the national availability of free bednets. The team developed an evaluation methodology in consultation with USAID-Ethiopia to include qualitative and quantitative methods. Standard social research tools were adapted for this evaluation (e.g., document reviews, stakeholder interviews, focus discussions, polling questionnaires, field observations, and data charts). 2. Overview of the Project The NM+ program is an eight-year; $65.4 million so-called ‘field support’ program funded by the U.S. Agency for International Development (USAID) and has been implemented by the Academy for Educational Development (AED) since 1999. Its goal is to reduce illness and death from malaria in sub-Saharan nations by creating sustainable commercial markets for insecticide-treated nets (ITN) in 10-15 African countries. The global NM+ program is managed by a Program Director located at AED headquarters in Washington, DC, and a Deputy Director based in Midrand, South Africa. The NM+ Ethiopia Office manages the NM+ country implementation program, collaborating closely with local commercial and communication partners. In Ethiopia, NM+ activities started in October 2004 and were set to run until September 2009. The program’s overall goal was to create an environment conducive for private-sector net supplies, with the following main objectives: (1) to improve access and create demand for insecticide-treated nets; 2) to establish targeted ITN subsidy programs for pregnant women; and 3) to catalyze local production of ITN products. USAID/Ethiopia provided the program with $3.8 million (malaria funds and funds from the President’s Emergency Plan for AIDS Relief [PEPFAR]), plus $300,000 from its USAID core funds. Expenses covered a household baseline survey, extensive behavior change efforts, direct partner support, voucher redemptions, a net utilization study, and technical assistance plus management costs. 3. Key Findings and Conclusions The key findings and conclusions described below summarize the main evaluation findings and recommendations. The body of the report presents a more comprehensive discussion of these findings and conclusions. 1. NM+ must be credited for being among the first in-country stakeholders to put the need for bednets and the notion of malaria-vulnerable groups on Ethiopians' mental maps. Successful advocacy also brought about the inclusion of a private-sector component in the National ITN Strategy, and the removal of taxes and tariffs from all ITNs. By using evidence-based information, education, communication/behavior change communication (IEC/BCC), the program identified and addressed users' inappropriate bednet practices. It also reduced the Drug Administration and Control Authority's (DACA) registration process time by more than 50 percent by providing DACA with testing equipment and technical 2 assistance. Because of NM+'s sponsorship and innovative open source technology, there will soon be a first Ethiopian local manufacturer of long-lasting impregnated bednets (LLINs) as well. 2. NM+ successes were achieved despite a number of enduring obstacles, which prevented NM+’s prime aim (to create a viable private market for bednets in Ethiopia) to be reached. These obstacles were: (1) the (Federal Ministry of Health) FMOH's mass-distribution of free ITNs; (2) the relatively new concept of a private sector; (3) the downward push on sale price exerted by PSI's social marketing activities and (4) DACA's insistence on in-country testing for ITNs. 3. NM+ has opportunities for greater success by improving its approach in regard to contextualizing its intervention model, realistically planning its human resources and the timescale, estimating the needs for field-level collaborators and maintaining high quality monitoring and evaluation efforts. Improving the clarity and frequency of USAID feedback in Addis Ababa and with the NM+ Washington and South Africa offices on how to cope with major threats to core goals is also needed. 4. Key Lessons Learned The evaluation report highlights the following lessons learned. 1. A formal project document would better align USAID and NM+ expectations. Given the success of NM+ model in several African countries, there was not a strong felt need for a formal project document detailing NM+ specific approach, goals or methods in Ethiopia. Instead, routine reference to NetMark Africa's Concept Note was made in yearly work plans. These formulations failed to align significantly divergent expectations between USAID Ethiopia and NM+ in Ethiopia. This was reflected in concerns expressed by USAID Ethiopia that addressed NM+’s approach (e.g., whether NM+ should loyally pursue long-term goals or seek more direct success instead). 2. The NM+ model requires proper consultation and customization. The Federal Ministry of Health (FMOH), Regional Health Bureaus (RHBs), DACA and others should have been actively consulted and the approach more customized as to why or how to bring NM+’s global model to Ethiopia with USAID/E support. This would have made it easier to avoid or work around difficult constraints to the program’s success, such as leakage control, registration hurdles, free net impacts, and health staff motivation. The NetMark Concept Note's Full Market Approach (FMA) that was used in Ethiopia did not fully appreciate pre￾conditions that were poorly met in Ethiopia (non-malaria Addis, a small middle class, an incipient private sector, few secondary industries). Tailoring the FMA to local realities was not given as an option to NM+ in the Ethiopian context. 3. A strong link with FMOH health departments at various levels is requisite. A public-private partnership based on two-way communication is needed: for NM+ to share program information or, better still, to plan together with FMOH departments, would have eased obtaining reciprocate data on system-wide impacts from them. It would also have brought woreda (district), zonal and regional authorities to feel that they jointly owned the project. In addition, FMOH staff’s wishes and opinions, especially at the targeted subsidy program (TSP) site level, should have been better solicited to secure their ongoing voluntary contributions and collaboration. 3 4. Key NM+ staff positions needed to be filled quickly and project progress meetings held frequently. With the difficulties in building support for the project in agencies, with retailers and others, and the relatively high level of resistance to new market ideas, a second, preferably local advocate/manager with solid commercial experience would have been helpful for accelerating the projects' efforts. This second advocate arrived nearly two years after the start of the project. Key positions such as the three Field Officer positions should have continued, given the 81 sites that needed supervision. NM+ was staffed with these three Field Officers for only 9 of 16 full capacity months. In addition to the above, the evaluation team assessed that project and donor staff had limited face-to-face progress meetings, resulting apparently in misalignment of expectations on both sides on the project’s outcomes. 5. Competing social marketers offload costs on donors, not on suppliers. A free market axiom is that competition helps consumers by forcing higher productivity and lower pricing on manufacturers. Social marketer competition helps customers too, but does so by taking donor funds that could be used for other worthy causes, thus bypassing those who could pay a socially marketed price. This risk was not fully appraised when the NM+ program was introduced into Ethiopia. Program planners likely used experiences from the global NM+ program model from Uganda as a reference. This was the only other African country where the program and Population Services International (PSI) coexisted successfully. However, to be successful in Uganda, both organizations respected segmentation, something that did not occur in the more difficult Ethiopian context. 5. Key Conclusions and Recommendations The evaluation report highlights the following conclusions and recommendations: 1. The evaluation team recommends that the program interventions, but not the NM+ Ethiopia project as previously defined, should receive continued USAID support. USAID should explore continued support for the following activities, some of which are being implemented in the country by other organizations and stakeholders: 1) increasing supply and access to bednets, especially for vulnerable groups, and 2) promoting malaria-related IEC/BCC to the general population. 2. Technical assistance should also be continued (for example by NM+ Global/USAID Field￾support Cooperative Agreement) to support local production and registration objectives. Strong advocacy interventions continue to be needed and could be provided by organizations currently active in-country. This would address donor purchases and conform to the World Health Organization Pesticides Evaluation Scheme (WHOPES) standards. [The recommendations below assume that free and subsidized bednets will continue to be provided in large numbers in the foreseeable future.] 6. Impacts 1. Several of NM+ program goals remain valid and still need to be implemented; justifying continued funding support for them. 2. NM+'s TSP coverage is very low while TSP costs are quite high, while other organizations (e.g., PSI, with its model and the freedom that it provides), has a more cost-effective way of achieving the same coverage. Free bednets and PSI’s marketing approach work against NM+’s efforts to increase access and supply. Future malaria program interventions should instead try and promote NM+ partners’ institutional sales. This means that NM+ should 4 continue to support its partners' (manufacturers') efforts to obtain WHOPES certification (that could enable it eventually to bid for Global Fund to Fight AIDS, Tuberculosis and Malaria [GFATM] funding). NM+ should also support efforts to obtain DACA registration of local bednets for the brands these partners can supply in-country. Technical assistance (TA) for local bednet manufacturing and quality control for regulating and their scale-up is still needed as well. IEC/BCC remains a major need at the grassroots level, and continued USAID support to NM+ for expansion of its previous excellent technical work in this area should be provided. 3. NM+ was not assessed to be the best placed to continue taking on the supply/access and TSP goals. Assessment of these program components show that other in-country organizations have more cost effective opportunities to provide these interventions. USAID should therefore explore more cost-effective options for these particular interventions. This assessment takes into account that Ethiopia’s Round 8 Application to GFATM has been recommended for funding, which should ensure that 23 million free nets are procured and distributed to replace the nets distributed in 2005-2008. Since this support might discontinue at a future point, USAID together with FMOH will eventually need to consider alternative strategies for ITN replacement. This is very likely to include commercial net outlets, requiring the continuing existence of private bednet supply channels. PSI will continue operating in Ethiopia for the foreseeable future, and DACA will grant in-country registration to at least one LLIN to bring in free nets. Aiming for a viable private market is less realistic than supplying as many outlets as possible with a choice of subsidized nets. Since blanket subsidies cost far less than managing vouchers, these might be replaced by 2 nets a pack priced at 150% the normal price. USAID PMI funds should be used to support such 'young family packs.' A formalized rural vs. urban distribution scheme could be developed to include agreements with fertilizer or gasoline distributors to carry subsidized nets further down to the country roadside, where also bednet brokers at weekly (off-road) rural markets would welcome easy access to ultra-low cost (conical) bednets. 7. Management Management lessons learned should be applied to future malaria program interventions. Ensuring that the program model, implementation plan, project documentation and local conditions are incorporated into future malaria program interventions is critical. Particular attention should be paid to ensuring adequate staffing throughout the project cycle, supported by open communications and regular face-to-face discussions with USAID/E. 8. Advocacy and Certification Further advocacy work is needed to assist the FMOH and the government to update their National Malaria Policy. Advocacy efforts, in collaboration with the FMOH and GoE directed at the GoE’s International Donors, should strongly press for allowing at least 20% of the nets purchased with donor funds earmarked for the Global Fund to Fight Aids, Tuberculosis and Malaria (GFATM) to be bought from in-country suppliers instead of from global companies, thus using institutional sales to keep the local private sector afloat. USAID ought to lead the way in this respect. Taxes and tariffs on various malaria control products could be reduced further still. More advanced tracking of social marketing (SM) nets leaked by officials (usually at the kebele level) should be defined and enforced. DACA could be encouraged to accept WHOPES certification used elsewhere as sufficient proof of quality also for Ethiopia. NM+ should try to bring FMOH and World Bank (WB) requirements into alignment: this could allow DACA to revise its requirements and to adopt WHOPES standards as its standard. 5 9. Linkages Strong links are needed both with the FMOH at various levels and between NetMark and PSI. Ownership is fostered by initial joint planning, but thereafter, close collaboration should be fostered at lower levels as well. High health staff turnover requires NM+ to provide repeat training workshops and refresher sessions; these trainings would boost morale and motivate reliable effort from these often overburdened government workers. NM+'s linkage to PSI would have benefited from a more formal agreement such as a Memorandum of Understanding (MOU) on segmentation, so as not to rely on verbal and email exchanges only. 10. Capacity Building For IEC IEC/BCC is recognized as the most urgent output needed country-wide. IEC/BCC efforts in general might consider: contrasting various malaria control costs (e.g., coil vs. spray vs. repellent vs. cure vs. bednet); having Health Extension Workers (HEWs) promote messages door-to-door following on kebele net handouts; including male household heads, often deciding against their wives using the family net; and using HIV or hospitalized patients as groups for targeted messages and access. In addition, radio drama series with health education messages from nearby counties could be dubbed and broadcast in Ethiopia. 11. Data And Information For Program Design And Adaptation More surveys and baseline information will help develop better messages and track program effects. For example, an important survey question should be why malaria is not more feared, since patients often do not get treatment even though a free cure is available locally. Similarly, if properly analyzed, yearly bed usage surveys with disaggregate data could pinpoint emergency flashpoints. Anthropological methods could provide information showing more realistic field descriptions to central planners. 1 2 2 INTRODUCTION 1. Malaria in Ethiopia Approximately 68% of the Ethiopian population (52 million people) live in areas of less than 2000 m altitude and are considered to be at risk of malaria1 . Until 2006, the disease was reported to be the leading cause of morbidity and mortality in the country, accounting for about 25-60% of the total outpatient cases, 15-40% of the total hospital admissions, and 5- 40% of total recorded deaths2 . Annually, around 4-6 million clinical malaria cases are reported across all in-country health facilities. The actual number of malaria cases in Ethiopia is estimated to be as high as 10-15 million3 . About 60-70% of the malaria cases are due to Plasmodium falciparum, while the others are due to P vivax infections4 . The diverse climatic and geographical features of the country result in unstable transmission in most areas. This transmission varies greatly between seasons and years, ranging from about three months in semi/arid and highland fringe regions to over nine months in the rest of the country. The general background immunity against malaria infection in the population is weak, allowing for occasional focal or large-scale epidemics. 2. National Malaria Response in Ethiopia The control of malaria in Ethiopia has a history of more than four decades. Pilot control projects in the 1950s were launched into a national eradication campaign in the 1960s, followed by a longer- term control strategy in the 1970s. A program involving indoor residual spraying (IRS), mass antimalarial (chloroquine) drug administration and intensive surveillance activities was introduced in the mid-1950s, initially in four pilot project areas. These anti-malarial interventions were scaled to national level from 1959 to 1972. In 1993, malaria prevention and control activities, including vector control, were decentralized and integrated into primary health care units. In 2000, the government formally organized a national workshop to adopt the Roll Back Malaria (RBM) initiative as the country's strategy to fight against malaria5 . Major scale-up of malaria prevention and control interventions took place in 2003 when the country received support from the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM). Another major change was in the country’s malaria diagnosis and treatment guidelines, with the introduction of artemisinin-based combination therapy (ACT) in July 2004. At the same time, the Federal Ministry of Health (FMOH) decided to introduce long-lasting impregnated nets6 (LLINs). Further scale-up of malaria prevention and control activities with wide distribution of rapid diagnostic tests (RDTs), ACTs, LLINs and indoor residual spraying began in the third quarter of 2005. In the period 2004 through 2007, the FMOH distributed a total of 12.5 million RDTs, 15.4 million courses of artemether-lumefantrine and 18.2 million LLINs. 1 Federal Ministry of Health: Guideline for malaria epidemic prevention and control in Ethiopia. 2nd edition. Addis Ababa, Ethiopia; 2004. 2 Federal Ministry of Health: Health and Health Related Indicators: 2005 3 Federal Ministry of Health: National Five Year strategic plan for malaria prevention and control in Ethiopia, 2006-2010. Addis Ababa, Ethiopia; 2006. 4 Deressa W, Ali A, Enqusellassie F: Self-treatment of malaria in rural communities, Butajira, southern Ethiopia. Bull World Health Organ 2003, 81:261-268. 5 Federal Ministry of Health: National Five Year strategic plan for malaria prevention and control in Ethiopia, 2001-2005. Addis Ababa, Ethiopia; 2000. 6 the insecticide in treated nets (ITNs) normally works for one year <> long lasting impregnated nets (LLINs) have an insecticide that should last till the end of a normal net life (3 to 5 years); confusingly, 'ITN' is at times used to contrast impregnated (ITNs and LLINs) to untreated nets. 3 3. Current Malaria Programs in Ethiopia Table 1 below illustrates the main malaria-related activities currently being deployed in Ethiopia by various NGOs and local associations in NM+'s intervention area. It can be seen that many of the agencies provide one or more activities in their project area/s, but none address the combined objectives that NM+ has been pursuing, which are to create and sustain a viable local commercial market for treated bednets. Figure 1 illustrates the proportions of where bednets in NM+ Ethiopia’s intervention area come from; omitting those distributed by NGOs other than PSI. In considering these proportions, it is important to keep in mind that the program’s primary goal was to create a viable market for privately sourced nets, not necessarily to dominate the Ethiopian bednet market it was trying to foster (that, by contrast, was PSI's main focus). 3 USAID/ETHIOPIA’S RESPONSE TO "ROLL BACK MALARIA" At the global level, USAID has been a Roll Back Malaria (RBM) partner since 2000. In Ethiopia, USAID became a member of the National Malaria Control Support Team (NMCST) as early as 1998. In support of the Government of Ethiopia's various malaria control efforts, the USAID/E documents state the following: insecticide-treated nets (ITNs) are one of the four key components of the RBM initiative in its campaign to reduce morbidity and mortality due to malaria. In addition to USAID’s response, RBM has publicly recognized the vital role of the commercial sector in providing a sustainable supply of ITNs to those who can afford it, and to those relying on subsidized delivery. RBM's "ITN Consensus Strategy" emphasizes the need for a coordinated intersectoral effort to simultaneously expand the commercial sector, prime the market where that sector is still absent, and provide equity to the poor, without discouraging private investment or unnecessarily distorting the market. Thus USAID supports increased access to and utilization of insecticide-treated nets through the development of a sustainable commercial ITN market. A four-year field-support Cooperative Agreement called “NM+” was awarded to the Academy for Educational Development (AED) in October 2003 with three main objectives: 1) improving access to and creating demand for insecticide-treated nets (ITNs); 2) establishing targeted ITN subsidy programs for pregnant women; and 3) catalyzing local production of ITN products. In Africa, USAID's NM+ program thereby became the commercial sector activity that was important for expanding a sustainable supply of ITNs to those who could afford to or who relied on subsidized delivery. In Ethiopia, NM+ activities were started in October 2004 and set to run until September 2009. USAID has provided the program with $3.8 million (malaria funds and funds from the President’s Emergency Plan for AIDS Relief [PEPFAR]), plus $300,000 from its USAID core funds. Table 1: Malaria control agencies and malaria activities in NM+ intervention areas Geographic Coverage Organization Activities National Al Bruk Public Leasing Company (PLC) vans for anti-malaria IEC/BCC National Catholic Relief Services (CRS) awareness creation, ITN distribution National Coalition against Malaria (CAME) policy advocacy, information & training National Malaria Consortium (MC) coordination and technical assistance National Population Services (PSI) social marketing of ITNs NM+ areas CARE awareness creation, ITN distribution 4 Geographic Coverage Organization Activities NM+ areas Carter Center (TCC) malaria prevention and control NM+ areas Clinton Foundation HIV/AIDS malaria prevention and control Oromia GOAL Ethiopia awareness creation, ITN distribution Oromia Ethiopian Malaria Control Ass. malaria prevention and control Amhara Amhara Development Ass. malaria prevention and control Amhara Anti Malaria Association malaria prevention and control [Source: Coalition against Malaria in Ethiopia – Sep 2008 Member Database / Malaria Consortium] Figure 1: Origin of ITN distributed in Amhara and Oromia Regional States. 5 4 EVALUATION PURPOSE The purpose of the current assignment is to conduct an independent, external end-of-project evaluation of the NM+ Program in Ethiopia. With less than a few months remaining for NM+, this evaluation is to collect information about the program’s impact and achievements, implementation and progress, and challenges. It will document lessons learned and formulate recommendations for future ITN social marketing activities and programs. 5 EVALUATION METHODOLOGY 1. Team Composition The evaluation team included 1) a senior expatriate Team Leader with expertise in social marketing and ITNs (28 days); 2) a local Senior Expert in malaria, social marketing, and ITNs (23 days); and 3) a local mid-level Logistics Assistant (23 days). Dr Gilbos, Team Leader, was responsible for the correct conduct of field work and the finalization of the evaluation report. Dr Afework's tasks included drafting data collection tools, as well as for gathering and analyzing those data, specifically addressing following areas: access, demand, vouchers, health and economic impacts. 2. Time Planning Appendix B presents the major team activities/interviews conducted during the field work in Ethiopia. In addition to these interviews there were team briefings, meetings, data collection, analyses, report preparation and presentation of results to USAID and stakeholders. 3. Methodology The team developed an evaluation methodology in consultation with USAID/E to include both qualitative and quantitative methods for assessing program processes and impact. Standard social research tools were used and adapted for this evaluation. These included: • Key stakeholder interviews; • Document review; • Quantitative data description / charts; • User-completed questionnaires [central stakeholders]; • Semi-structured interviews [in person][by phone]; • Focus group discussions; and • Field observations. 4. Constraints The original methodology was modified to accommodate the situation at the time of field work without compromising data quality. However, the evaluation team encountered some constraints during field data collection, which included the following: • The team was all male (including the excellent interpreter), although mostly females were interviewed; • Households were sampled purposefully, not representatively; samples were small, ad hoc and female-biased (males are known often to decide bednet use); • Organizing village focus group discussions would have been too time-consuming, though the team was lucky to attend and probe a (single) town (male) elders’ meeting; 6 • Major holidays reduced availability of respondents in the field, in Ethiopia, U.S. and South Africa; • There was limited time and familiarity with non-FMOH and non NM+ data sources to be explored; • NM+ data were still being developed (pre-post KAP survey around 2008 IE Campaign; 2004 repeat Bednet Use Survey, scheduled 2009), or some data were not readily available from NM+; and • A NM+ workshop bringing together both staff and supervisors was not geographically feasible. 6 FINDINGS 1. Background Prior to November 2004, when NM+ launched operations in Ethiopia, there was no wide access to ITNs, and they were not affordable to poor populations. Import duties accounted for 5% for nets and 10% for insecticides, along with a 15% VAT for both. The private market for nets was very small, comprised principally of donor nets diverted for commercial sale. Those nets that were available were mainly rectangular (consumers prefer conical), of poor quality, and ranged in price from 35-80 birr. At the time, bednet awareness was low (43% of rural people had never seen or heard about them), and ownership heavily skewed to the most affluent (55% of the wealthiest households owned a net, compared to just 3% in the poorest SES quintile). Awareness of pregnant women’s vulnerability to malaria was poor, as was ITN use in this group (overall, 6% of pregnant women slept under an ITN the night before)7 . Before the launch of NM+ in Ethiopia, there were no Ethiopian ITN distributors and no indigenous brands. There were no viable options for ITN production in Ethiopia, nor were any Ethiopian companies investing in local production. The FMOH's National ITNs Strategic Plan [2004-2007] prescribes free bednet distribution to target the rural and remote areas of the country with high malarial prevalence, while subsidized or priced sales are targeted to urban and semi-urban areas. The table in Appendix C, Timeline shows the key activities that NM+ used to implement the program. These include improving access /creating demand, establishing a targeted ITN subsidy program for pregnant women, and catalyzing local production of ITN products. The activities in bold provide additional information and context for program activities and key activities performed by other agencies that affected the project. Italicized text refers to program personnel management events. 7 IMPACT ON OBJECTIVES 1. Objective 1: Improving Access and Demand a. Improving Access to ITNs NM+’s social marketing program improved access and created demand by carrying out its joint role of sustained demand creation, time-limited market priming, and stimulation of the private sector. NM+ was able to create a partnership with two major in-country distributors 7 [Source: NetMark Plus 2004 Survey on Insecticide‐Treated Nets (ITNs) in Ethiopia] 7 having relatively good retail networks. East Africa Group Holdings, later renamed East Asia Industries (EAI), was the first to be included as a NM+ partner in August 2004; Petram, the second partner, started expanding access by selling subsidized pre-treated (DAWA) nets from Siam Dutch in September 2005. These program partners supplied newly opened private retailers, first in Amhara and then in Oromia, up to a maximum of 81 sites. This expansion of private retailers increased access by bringing bednets further down the system into smaller communities and rural areas than ever before in Ethiopia. The result of an MIS survey conducted in 2007 shows that in areas below 2000 m, 69% of households owned at least one net. Among children under age 5 in those areas, 44% slept under a net the night before the survey. Among pregnant women, the number was 45%. [Source: Report of Malaria Indicator Survey 2007. FMOH – 2008] Improved access and increased demand were evidenced by a NM+ 2008 Campaign Preparation Survey showing that, while due to forces beyond NM’s control, 75% of households in NM's intervention area now own a free net. In Addition, 22% of the households there had bought a commercial or subsidized net over the last four years. As above Figure 1 illustrates, the rate of PSI over NM+-purchased nets is approximately 2 to 1, where PSI outlets actually overlap with NM+'s. So a likely 7% of the priced nets found in NM+ intervention area homes must have been purchased from NM+ partners. The 75% free net ownership in these intervention areas (in 2008) also compares favorably to the 69% net ownership reported countrywide from the MIS 2007 Survey quoted above. This suggests NM+ activities in the area also had a modest demand creation effect (cf infra). Figure 2, Ethiopia Formal Partner Sales, shows the increase in access due to partner sales in the period 2005 to 2008, when NM+ partner direct sales increased from zero in 2005 to 73,277 in 2008. Sales in 2006 were also high at 75,983. Since ITNs are effective for multiple years, the access of the population to NM+ nets is cumulative. Figure 2: Ethiopia Formal Partner Sales [Source: NM+ sales reports] 8 Table 2 shows the monthly and annual numbers of bednets sales by NM+ partners. Table 2: Total monthly and annual bednet sales by NM+ partners (2004-08) NM Partners' Sales OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP TOTAL 2004‐05 220 91 1,302 2,179 1,895 3,902 8,775 10,592 44,321 73,277 2005‐06 17,610 5,995 8,460 3,280 7,379 4,060 2,447 3,612 7,812 3,921 7,400 4,007 75,983 2006‐07 4,078 4,212 1,784 5,099 8,399 2,271 1,249 4,360 1,791 3,586 4,661 4,066 45,556 2007‐08 409 1,240 9,905 1,405 1,672 668 950 805 699 1,626 2,905 1,594 23,878 Conclusion (improving access to ITNs) Before NM+ brought nets into the country, virtually no nets were available in Ethiopia. The program helped to increase access by successfully advocating to the FMOH's ITN Strategic Plan of 2003-2008 to promote market segmentation and private-sector ITN markets. It worked hard to improve access that increased sales and to create public-private partnerships for sustainable net supply; NM+ created two partnerships during the project's life time. b. Creating Demand for ITNs In order to achieve the program’s objective #2, creating demand for ITNs (including of freely distributed public-sector nets), and NM+ conducted extensive generic bednet promotion campaigns. In the first two project years, NM+ used various outlets, including radio and TV ads, road shows and public relations activities. These efforts were accompanied by East African Group (EAG), a partner recruited and supported by NM+ to launch a brand-driven radio and print campaign using matched program funding. After those first two years, the program concentrated on two major campaigns (USE and ANC) on social marketing and community mobilization. These campaigns were designed to create demand in the local populations for ITNs. The main focus of the "ANC" community mobilization campaign was to boost visits to Antenatal Clinics (ANC) and PMTCT (prevention of mother-to-child transmission) facilities, and to promote ITN use during pregnancy. This campaign provided training and support to women’s groups and other community-based organizations. It produced interactive radio programs that were aired on several radio stations to cover the population of 20 woredas (districts) of the Amhara and Oromia regions. The program’s more recent 'USE' BCC campaign endeavored to create demand and to encourage proper use of nets by “free net” beneficiaries and by voucher program beneficiaries. To create demand and promote correct handling of nets, NM+ developed messages and used them on national radio programs, billboards, posters and flyers, and conducted 24 roads shows, four of which were held in marketplaces. As part of the 2007 and 2008 Africa/World Malaria Day celebrations organized by FMOH in Meki (Oromia) and Awassa town (SNNPR), NM+ presented a road show with malaria prevention educational dramas. The program’s approach took into account that creating awareness and knowledge is a first step in creating demand and thus utilization of products such as ITNs. A comparison of the NM+ ITNs 2004/2008 Surveys and the Malaria Indicator Survey of 2007 Report shows that between 2004 and 2007, the percentage of all age respondents having heard about mosquito nets rose from an average of 0.7% (in NM+ areas) to a (nationwide) average of 26% in rural and 59% in urban areas. Because several organizations were bringing the same activities to the same geographic areas (the full CAME Database, cf. Table 1 lists 6 Amhara, 9 Oromia 9 and 10 National malaria-related initiatives), NM+ cannot be credited alone for increasing awareness and therefore demand for nets especially in urban areas; however, the program certainly contributed substantially to this increase. Table 3 below shows a significant increase in utilization of bednets reflecting an increase in demand in a relatively short time (i.e., between 2004 and 2008). These data are particularly striking since they reflect utilization by targeted vulnerable groups. Table 3 Utilization of ITNs by targeted vulnerable groups (2004-08) KAP Indicators NM+ Survey 2004 MIS Oct￾2007 NM+ Survey Urban Rural 2008 % of children under five slept under an ITN the prior night 6 79 57 67 % of pregnant women slept under an ITN the prior night 6 61 59 63 [Source: NM+ ITNs 2004 & 2008 Surveys, MIS 2007 Report] Table 3 also shows that in 2007, urban areas were more aware of and active in using ITNs than rural areas. Since NM+ road shows and other BCC activities promoting awareness and demand took place mainly in urban and suburban areas (and no other NGOs did road shows there at that time), and since those areas were not targeted for free net distribution, the tenfold increase (from 6 to 65%) in bednet use by NM+'s target groups suggests that the program’s social marketing of ITNs played a major role in increasing demand in those areas Conclusion (creating demand for ITNs) In November 2004, EAG, a NM+ supported partner unveiled the first-ever Ethiopian-owned brand of insecticide-treated net (Selam Enkilf). Ten months later, this was followed by the Siam Dutch manufactured Wobalba brand of Petram. These historic firsts8 were made possible by NM+'s insistent drive and promotion activities and helped to raise awareness about ITNs, which until then were virtually unknown in-country. For its first two campaigns, NM+ collected data in 2004 to develop messages and to clear up misconceptions around malaria and bednets, apparently to good effect; all over the central region, NM+’s road shows, dramas and malaria broadcasts were well received by audiences. Prior to these promotion activities by partners, hardly anyone worried much about malaria. NM+'s last two campaigns have now also added pre- and post-message uptake surveys, the results of which will be known soon. 2. Objective 2: Targeted ITN subsidy program for pregnant women The program worked actively to establish a Targeted Subsidy Program (TSP) for pregnant women, starting with 10 Amhara sites in April 2005. It later expanded into the Oromia region, reaching a maximum total of 81 sites in December 2006. Although initial attempts to work in SNNPR were also made, USAID decided that NM+ should concentrate its efforts in the Oromia and Amhara regions. In 2006, NM+ secured funds from USAID-PEPFAR to integrate community mobilization and social marketing of ANC and PMTCT services into its TSP. The program is currently 8   PSI's SafeNight brand was launched in Feb'04 but is not locally owned 10 still active, subsidizing pregnant women’s purchase of bednets in 32 health facilities linked to retailer sites across Amhara and Oromia. To respond to challenges in the targeted subsidy program, NM+ conducted an internal evaluation in consultation with USAID's PMI office in February 2007, to check the viability of the Targeted Subsidy Program that was severely challenged by free public-sector distribution and heavily subsidized PSI nets. At the time, there was also concern that some health workers were reluctant to give NM+ ITNs out because they believed most residents had already received free ITNs, which made NM+’s Targeted Subsidized Net program expansion even more arduous. A NM+ internal evaluation found that, out of 81 TSP sites, 65 were still operating moderately. As a result, USAID advised NM+ to close down 49 of its lower-performance sites located in minor urban areas. The staff presented the situation that their low performance was often due to free net distribution in the area and that other NM+ sites were targeted for free net distribution in the near future. This closing of 49 TSP sites then left NM+ with only 32 outlets in major urban areas that were still operational in late 2008. NM+ staff observed that, even though free nets distributed by FMOH were competing with NM+ subsidized nets, the free nets were actually not so popular because their rectangular shape was hard to hang up in a small circular farmer's hut (tukul). In part, this explained why NM+’s (18 birr voucher) subsidized nets continued to be sought by pregnant women as their key target population. In conclusion, NM+ had established a TSP but faced serious competition and was advised by USAID/E to close down many of their sites due to FMOH and other agency competition that resulted in the program’s low performance. For that reason, therefore, the remaining sites were insufficiently cost-effective. Figure 3: Evolution of TSP sites, sales and redemption rates (2004-08) [Source: NM+ Sales and Annual Reports] [Source: NM+ Sales and Annual Reports] Factors affecting the expansion of the Targeted Subsidy Program coverage: Figure 3 shows the TSP expansion and impacts on voucher and ITN sales. It demonstrates that when there were 58 NM+ TSP sites, both TSP/voucher sales and redemption rates were at their peak and had risen in parallel. Then TSP sales declined sharply while voucher redemption, though lagging, still rose somewhat before declining as well. Soon, however, both collapsed, notwithstanding NM+'s continued expansion of outlet sites. TSP sales were threatened from the start by distribution from the FMOH of free bednets and by heavily subsidized PSI nets. 11 NM+’s TSP program was also later negatively affected by a DACA-caused import 'ban' on its partners' ITNs. The partners (and all other organizations) were unable to get registered due to DACA's stringent registration requirements. At the health services level, the FMOH's high staff turnover and its demanding other public health campaigns for vertical programs (e.g., Vitamin A, polio, trachoma) sapped health staff support for NM+'s TSP intervention as well. Table 3: TSP target population coverage (2004-07) TSP 2004-05 2005-06 2006-07 2007-08 Annual target9 population 180,000 190,000 200,000 224,250 Annual redemption 3,673 20,305 39,840 6,633 % NM+ area pregnant women reached 2 11 20 3 [Source: NM+ Monthly Sales Reports, National Demographic and Health Surveys] Table 4 data show quite a positive increase in coverage, especially between 2005 and 2007. During that three-year period, the TSP intervention/program was able to expand coverage and reach growing numbers of women with newborns (later also children under 5). Table 4 shows the increase in numbers of "annual redemption" of vouchers going from 3,673 to 39,840. However, the voucher redemption numbers dropped steeply (from 39,840 to 6,633) at the time NM+ closed the 49 TSP outlets. The proportion of pregnant women (later those with children under 5) reached10 grew from 2% in 2004-05 to 20% in 2006-2007; but then dropped to 3%. Conclusion (establishing TSP subsidy programs for pregnant women) Before the TSP, virtually no one could identify malaria-vulnerable groups in-country. The TSP program was established and increased the coverage of ITN vouchers and sales to pregnant women. This demonstrated NM+’s ability to target and supply this vulnerable group. Health messages such as those about HIV were conveyed to pregnant women there, in addition to malaria and bednet talks. ANC attendance increased indirectly due to this promotion (although no hard data are available to confirm this) not just where TSP BCC promotion and sales ran but also nationally, since radio broadcasts reached beyond NM+ areas. The TSP program’s BCC promotion provided a platform on which HIV/AIDS prevention and the promotion of ANC and PMTCT also took place. One mechanism was that NM + used was voucher stubs to track and invite 25,000 women to attend radio show discussion groups for 6 weeks to promote changes in behavior regarding nets, ANC and PMTCT attendance. Also, TSP was instrumental in keeping the private-sector sales of bednets afloat when free nets came in. At the same time that NM+ had success with the TSP Program expansion, Figure 3 suggests that closer monitoring of TSP data might have forewarned NM+ earlier that further TSP expansion was unlikely to succeed in countering falling sales. In addition, shrinking (legal) net supplies were another factor in its reduced expansion. 3. Objective 3: Catalyzing local production of ITN products In May 2005 NM+ and USAID/E, in collaboration with the FMOH, provided technical assistance to local businesses to assess the feasibility of producing mosquito nets in-country. NM+ shared with local net manufacturers NM+’s broader experience in other African 9 Estimated annual urban pregnancies in the NM intervention area (not counting U5's families) 10 As of 2007 families with under fives were also given vouchers, whereby the reference population grew considerably [not adjusted here] 12 countries, a view of the global and regional net market and information on the economic and technical viability of local ITN production. In September 2005, NM+ sent a representative of FMOH and of Ethio-Japan Textile to an ITN Technology Seminar in Kenya. Further technical assistance, subcontracted by NM+, was given by Anovotec textile engineers to Ethio-Japan Textile and to the Adami Tulu Insecticide Processing Plant near Addis. As a result, a feasibility study and a business plan were produced that provided sufficient rationale for local bednet production. The feasibility study resulted in approval by the GoE of funding of local bednets for local bednet production, scheduling the roll out of local bednet production by Adami Tulu for the end of 200811. Program TA continued beyond the feasibility study; during the implementation stage: the NM South Africa office’s experts facilitated the creation of a partnership with experienced foreign firms (Siam Dutch, BAYER) to gain access to the required technology and technical know-how in LLIN production (e.g., how to best stitch nets, what insecticide treatment methods to use, etc.). This TA will help the partners to successfully reach the quality control parameters established by WHOPES. Meeting these requirements will then enable Adami Tulu be approved by DACA and begin local bednet production. Conclusion (catalyzing local production of ITN products) NM+ tried hard to catalyze the interest of private companies in local net production by providing TA and by helping to create partnerships between local companies and experienced foreign firms. However, in the end, only the Government of Ethiopia saw the advantages of doing so. Private local manufacturers had too many strong reservations about the future profitability of such an undertaking. The evaluation team feels, however, if the Adami Tulu plant can increase bednet production as planned, from 1 to 3 m nets/yr till 2020, it will be able to meet the need for replacement nets once free nets stop, as well as be able to fill demand-side gaps that form even where bednets are being given out for free. 4. Impact on National Malaria Control Strategy and National ITN Strategy Background In August 2004, the Government of Ethiopia (GoE) published a National Strategic Plan12 for the Utilization of Insecticide Treated Nets, stating: • Indoor Residual Spraying and ITNs are the two most important vector control interventions for use in Ethiopia; and all areas considered prone to infection (below 2000 m altitude) are targeted for ITN distribution. • Subsidies must be targeted at highly vulnerable groups such as rural pregnant women and young children; subsidized distribution requires a segmented approach. • The GoE supports involvement of the private sector in ITN distribution, since some population segment will always need ITN access through commercial channels. • The Government’s role in private sector supply will be to create an enabling environment, which is needed for the long-term provision of commercial sector ITNs 11 Techno‐Economic Feasibility Study on Production of Long Lasting Insecticide Treated Nets, May 2007 12 National Strategic Plan for Utilization of Insecticide Treated Nets (ITNs) – (2004 – 2007) 13 Despite the official statement on private sector ITN distribution and segmentation, in early 2005 the GoE started giving out free ITNs, first in rural areas, but later also in towns and cities. This affected private sector bednet distribution segmentation. Through the support of the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) and other donors, the GoE secured funding for enough bednets to cover 10 million households: by late 2008, 20.3 million ITNs were distributed for free throughout the malarial districts. NM + supported local and international TA to the National Malaria Control Strategy providing technical input to the Malaria Control Support team to develop and implement the Strategy. 5. NM+ Input and Impact on the National ITN Strategy According to Dr. Jimma [FMOH] and Dr Agonafir [Malaria Consortium/CAME], NM+ gave significant technical input when the ITN Strategy Plans were updated by the Malaria Control Support Team (MCST) under FMOH and UNICEF. The additions NM+ advocated for were: giving targeted subsidies to ITNs for vulnerable groups; removing taxes and tariffs on nets and on their components; supporting the creation of a commercial sector in areas where people can afford to buy bednets; and the segmented targeting and distribution of free bednets to protect the market for nets sold by private suppliers. It should be noted that these NM+ supported updates to the National ITN Strategy Plan, or the issue of moving toward the preferred Long Lasting Insecticide Treated Nets (LLINs) rather than toward ITNs, have so far not officially been published by FMOH. Conclusion As part of the Malaria Control Support Team under the FMOH, NM+ was a prime contributor of technical assistance to the National Malaria Control Strategy, second only to UNICEF. Without the staff’s advocacy efforts and representation of needs, the principles of private-sector supply and segmented bednet provision would not have been established in the National ITN Strategy. 6. NM+ Efforts and Challenges Removal of net taxes and tariffs Prior to distributing their product, the East Africa Group with NM+ support successfully advocated to the GoE to achieve removal of the customary 15% value added tax (VAT) on imported products including ITNs. This partner advocacy effort paved the way for other formal and informal partners with different ITN brands also to try and enter the net market. In August 2007, NM+ conducted a well attended Taxes and Tariffs Workshop with its malaria control partners, which was instrumental in the later removal of the 5% customs duties on ITN/LLINs in early 2008. The program’s 'seed' and 'matched' funding was an additional incentive to potential partners: 50% of the cost of imported nets (seed), and 50% of any promotion/distribution related costs (match) were funded by the program Conclusion (taxes and tariffs) All stakeholders agree on one achievement: the NM+ program in Ethiopia and its partners achieved the removal of the 15% VAT on ITNs and the 5% customs duty on ITNs. The removal of these taxes and tariffs has, in fact, benefited all net and insecticide suppliers across the board. At the same time that the program was successful, more work will need to be done on other malaria control-related products (e.g., tests, drugs, and some chemicals) that have not yet been cleared of import duties and taxes. 14 7. Facilitation of registration In March 2006, NM+ and its partners faced an unforeseen obstacle to private ITN market success. This was the repeated request by the then two-year-old Drug Administration Control Administration (DACA) for ITN suppliers to (re)register their product. At that time, the DACA process took several years to accomplish, and only one supplier (Vestergaard￾Frandsen's PermaNet, a contractor with exclusive distribution rights with PSI) had been registered by DACA. Although it was strictly the responsibility of NM+' partners themselves to address this issue, NM+ staff also responded to the DACA demands by providing TA and working with USAID and the FMOH to address the difficult DACA requirements that imposed efficacy testing for all ITNs/insecticides. Since then, NM+ has constantly provided TA and facilitated each of its partners’ attempts at DACA registration. Despite NM+’s TA and facilitation, DACA has not yet registered any of NM+’s partners. To further support registration of ITNs with DACA, NM+ has contacted all WHOPES￾recommended global manufacturers to start a registration process in Ethiopia (i.e., BASF, BAYER, SYNGENTA and TANA Netting). NM+ with TA from the NM Regional office in Johannesburg has assisted the Pathobiology Laboratory of Addis Ababa University, the only facility recognized by DACA to carry out efficacy tests in-country. In July 2008, NM+ also brought special equipment (i.e., WHO bioassay test kits, Insecticide impregnated papers, PCR reagents, UV light and Polaroid camera) and provided technical assistance to DACA’s ‘Fast Track Registration’ process. This equipment and TA should shorten DACA’s previously two- to three-year process to a nominal 15 months (i.e., 3 months for treating the product application, and 12 months – or two seasons – for field testing the product). At the time of this evaluation, no LLIN can legally be imported into Ethiopia, given DACA's insistence to meet its bio-efficacy test requirements that are to achieve 95% mosquito knock￾down with 80% mosquito kill rates in field conditions. These DACA requirements are reportedly stricter than the WHOPES specification [BASF]. The evaluation team found that other governments and a number of international donors are increasingly using the WHOPES recommendation as a core criterion to bid on large tenders. WHOPES itself insists its strict Phase 2 criteria should not function as an obstacle, but rather that companies could bring LLINs to market once these companies have full confidence in them – and that it is not necessary to wait for the 12-18 months of Phase 2 testing. However, major tenders (including USAID's) often do demand LLINs with WHOPES Phase 2 acceptance (i.e., to have solid technical dossiers and to have completed Phase 1 lab tests). So far, in the three years since DACA requirements changed, the companies that have gone through the full process include Verstergaard-Frandsen, for its PermaNet; Sumitomo for its Olyset; and BASF for its Interceptor LLIN. Conclusion (registration) Although not strictly NM+’s (but rather its partners') responsibility, NM+ provided significant TA on the registration process that continues to be complicated in Ethiopia. DACA does not recognize the WHOPES certification as a substitute for in-country bio￾efficacy tests and registration. This strict DACA regulation and its implications for slowing the ITN marketing may be an issue that is above the level of the NM+ contractor. Still, to support the strict DACA registration process, NM+ brought in bio-efficacy testing equipment in July 2008. This adequate reaction came one year after DACA's sustained strictness about registering LLINs became clear and three years after LLINs (with their implied testing requirements) became Official National Policy. The delay in this response is, in no small part, due to NM+ staff's understanding of its role as a facilitator rather than as an implementer; 15 both of its partners made considerable effort to meet with DACA to try to resolve ongoing registration problems. While NM+ was providing TA, support and equipment, it also repeatedly tried to obtain FMOH and USAID/E high-level intervention with DACA. The FMOH and USAID/E did not take an active role to request changes by DACA to its stringent registration requirements for nets. The evaluation team was told that the situation with DACA was delicate and that foreign pressure would highly displease DACA officials, for this reason, the potential for USAID action and/or multi-donor policy-level discussion would require careful consideration. During this same time, the uncertainly about DACA's certification decisions understandably kept alive NM+’s hope that things would be decided favorably at a reasonable future date: this positive hope for resolution of the situation continues within NM+. Finally, despite all the difficulty and delays, many agencies recognize that NM+ has been the only organization so far to have constructively engaged and aided DACA in its legitimate concern and in its role to protect the Ethiopian population against counterfeit chemicals and pharmaceuticals. 8 IMPACT ON HEALTH AND LIVES Effects on Lives The evaluation team did not obtain solid data of the impact on lives of bednets in general and of those brought by NM+ partners in particular. Anecdotal evidence and logical deduction, however, suggest that NM+'s efforts had positive effects not just on the survival but also on the lives of targeted beneficiaries. The evaluation team’s field visits found evidence of several ways in which NM+ contributed to improving people's lives. During interviews with people using bednets and with health workers, they stated that, when sleeping under a net, they felt safe and that they slept freely since mosquitoes could not reach them; nets were even said to protect against nightmares. This related to health worker recognition that if someone acts in a bizarre manner and has nightmares this can be a sign of cerebral malaria. People also stated that nets repel or kill fleas, spiders, bed bugs, and cockroaches, and stop flies from reaching a baby’s eyes during day naps (thus preventing trachoma). Nets also give protection against roof droppings (e.g., against rodents' or against hurtful worms that typically drop out after the rains). Although not the original purpose of nets, some people appreciate that nets keep them warm, can be used as a bed cover, floor carpet, door curtain, hairnet, shopping bag, wedding dress, harvest cover, or latrine wall, and can keep roaming goats from damaging seedlings. Effects on Vulnerable Groups' Health NM+ has improved the lives and health of the initially targeted population of pregnant women. In addition, the program expanded its definition of vulnerable groups and provided TSP nets not just to pregnant women but also to households with children under the age of five. Though not official NM+ policy, several nurses told the evaluation team that known (over the age of five) HIV/AIDS patients were not refused TSP net vouchers when they asked for them. Malaria prevalence typically rises around harvesting and school exam times in relation to the seasonal rains that precede them. NM+ was particularly active in marketing its nets at these times, when populations were more likely to be interested and able to buy nets. Logically 16 their bednets would have boosted labor productivity and reduced school drop-out rates [no quantitative data were available to support this]. Finally, bednets would have decreased the vulnerability to malaria and its consequences for those living with HIV/AIDS. NM+ has contributed to decreasing Ethiopia's malaria burden. Yet between program inputs and effects, it is not possible to establish a definitive one-to-one relationship. For one thing, there are no reliable, comprehensive and pertinent data; more importantly, there are many confounding factors to attribute cause and effect. A major factor impeding correct interpretation of malaria trends in Ethiopia in 2008 is that the last ‘epidemic year’ was in 2003/2004. Thus, the decline in malaria morbidity and mortality currently observed may be due to the natural 5-8 year cycle characteristic of malaria in Ethiopia, as the next epidemic year is not expected until 2010. In addition, nets are available not just because of NM+ but because of FMOH/GFATM free ITNs, and because of the many other NGOs (especially PSI) and organizations giving out nets in NM+'s intervention area. The evaluation team estimates that NM+'s 211,900 nets have contributed about one third of the subsidized nets found in Amhara/Oromia (sub) urban households with the others likely from PSI (cf Figure 4). These nets logically must have caused a proportionate drop in the malaria burden. The Ethiopian Demographic and Health Survey (DHS) data showed that, during the period of 2005-2006 and 2006-2007, malaria morbidity dropped countrywide from 1st to 6th rank in outpatient department (OPD) consultations13 and reported malaria cases came down from a high of 5.7m in 2004 to a low of 1.2m in 200714. Although no data show conclusively that NM+ alone definitely lowered Ethiopia's malaria burden, it is nevertheless plausible to infer that the distribution of nets of any origin has brought malaria prevalence (e.g., in Oromia (2007)15 down to around 1% [95 CI = 0.5 – 1.6%] at village level). In addition to the drop in morbidity discussed, a marked drop in malaria mortality was noted as shown in Figure 4. Figure 4: Malaria mortality trends under scale-up interventions (2003-07) 0 5 10 15 20 25 2003 2004 2005 2006 2007 Million units 0 5 10 15 20 25 30 35 40 % malaria deaths CoArtem distributed Nets distributed % malaria deaths [Source: FMOH DHS 2005, DHS 2007] 13   Federal Ministry of Health: Health and Health Related Indicators 1999 EFY(2006/07 G.C.) Addis Ababa, Ethiopia: Federal Ministry of Health, Planning and Programming Department; 2006/2007. 14   World Malaria Report 2008 (WHO) < DHS 2000, 2005, 2007 (FMOH) 15   Estifanos B Shargie et al. Malaria prevalence and mosquito net coverage in Oromia and SNNPR regions of Ethiopia/ BMC Public Health 2008, 8:321 17 Although no data were available to show that NM+ definitely contributed to lowering Ethiopia's malaria burden, it is nevertheless plausible that the massive distribution of nets of any origin has brought malaria prevalence in Oromia (2007)16 down to around 1% [95 CI = 0.5 – 1.6%] at village level. Figure 4 also demonstrates similar sized health impacts (that malaria in Ethiopia has an epidemic character can be learned from the initial rise in that graph–rendering causal links between trends and interventions always less than certain). Ethiopia now reports the highest net coverage in all sub-Saharan Africa after Togo and Sierra Leone17. A recent WHO assessment18 on the impact of LLINs and ACT scale-up using health facility data showed a marked reduction in malaria cases and deaths. The weighted average decline for malaria cases and deaths in all ages between 2001-4 and 2007 was 53% and 55% respectively, while non-malaria cases increased by 14% and non-malaria deaths declined by only 8%. [according to its numerical proportion in its area,NM+ can only be credited for a 2% contribution to this impact] Figure 4 (above) above also demonstrates similar and significant impacts on malaria mortality. That malaria in Ethiopia has an epidemic character is shown in the initial rise in 2003-2004 of the red line on the graph. Conclusion (effect on health and lives) At the same time that the program was expanding access to nets, the GoE, the RBM in￾country partners, PSI and other NGOs were also carrying out activities to increase access and create demand for ITNs. This makes singling out NM+'s unique impact on health and lives in Ethiopia very difficult. Numerically speaking, NM+ contributed just 2% to regional bednet availability. However, as discussed above, other program inputs significantly affected knowledge about malaria and its effects, as well as knowledge about the proper use and handling of nets, that was necessary to impact Ethiopia's malaria burden. Figure 5 shows total sales for two NM partner distributors in the periods 2004-05 and 2006-07. Especially noted is the sharp decline of EAG from a high of 100,000 in the early period. Petram sales also declined in the 2006-07 period. As illustrated in Figure 6 below, sales show great increases until mid 2006 when free net distribution drove down partner net sales in the second half of program duration. 16 "Malaria prevalence and mosquito net coverage in Oromia and SNNPR regions of Ethiopia" Estifanos B Shargie et al. BMC Public Health 2008, 8:321doi:10.1186/1471‐2458‐8‐321 17Data Sources: Demographic and Health Surveys; Multiple Indicator Cluster Surveys, Malaria Indicator Surveys, Country‐specific National Surveys: MACRO; UNICEF; RBM. 18 http://www.who.int/malaria/docs/ReportGFImpactMalaria.pdf Figure 5: NM+ partner total bed net sales (2004-07) [Source: NM partner monthly 18 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 Apr-Jun 05 Jul-Sep 05 Oct-Dec 05 Jan-Mar 06 Apr-Jun 06 Jul-Sep 06 Oct-Dec 06 Jan-Mar 07 Apr-Jun 07 Jul -Sept 07 launch of TSP in Amhara Launch of TSP in Oromia Massive free net distribution & leakage in major urban areas Figure 6: Impact of Free Nets on NM+ Partner Sales (2004-2008) [Source: FMOH Annual NM+ Distributors Report] 9 CHALLENGES AND RESPONSE 1. Response to free net distribution The evaluation team assessed that the 2005 shift in FMOH's ITN Policy (to actively cover all malarial areas with 2 nets per household) presented a significant challenge to the program’s initial model/approach since free nets were being distributed in several suburban areas where its voucher programs were running. At the same time, people expressed interest in voucher programs in those geographic areas that had not yet received free nets. A second challenge came with the requirement from DACA to conduct extensive local efficacy testing of LLINs regardless of their WHOPES recommendation. At that time, only Vestergaard-Frandsen's PermaNet was registered in Ethiopia and this under an exclusive distribution agreement with PSI (NM+ actually tried to acquire nets from PSI on several occasions but failed, since PSI was not allowed to provide NM+ with product). Therefore, NM+ partners continued to distribute ITNs (pre-treated nets) from Thailand and Tanzania to roll out their activities. In late 2006, one NM + partner in particular, EAG, was told by DACA it could no longer import ITNs unless its manufacturer was registered in Ethiopia. Several other partners/potential distributors were given similar notice. Finally DACA notified PermaNet as well. As a result, in late 2008 not one single company is officially registered and cleared for legal distribution of long-lasting insecticide-treated nets in Ethiopia. A third challenge to the program came when heavily subsidized nets from PSI entered the market and undermined NM+'s Targeted Subsidy Program. As long as segmentation held and PSI sold its PermaNets at 45 birr, NM+ partners were in a position to compete at least in sub/urban centers. In late 2006, however, the U.K. Department for International Development (DFID) asked PSI to bring down their net price (first to 12 birr, but later the price was set at 20 birr), in exchange for a US $1 million subsidy. In response to this challenge, NM+ again requested that PSI would respect a segmented approach; that is, to accept working in suburban or rural areas only (the one other African country where NM+ in Uganda and PSI coexisted and did so successfully, was Uganda where both SM organizations strictly respected their segmentation). When PSI tried to respond to NM+’s request on segmentation, it quickly realized there were insufficient rural outlets to distribute nets from. Also, their 19 PermaNets were in fact already targeted toward a wealthier urban public. So facing a 20 birr priced nets marketed by PSI in their operating area, NM+ took action and consulted USAID/E. Working with NM+, USAID/E decided to increase the subsidy on NM+ partner TSP nets, so as to have a (voucher redemption) price of 10 or 18 birr (depending on net type). To further respond to the challenge presented by PSI net marketing, NM+ added children under five years of age as a new category to legitimately benefit from the TSP (until then, only pregnant women had been entitled to subsidized nets). In another strategic reorientation and in reaction to their recent bednet USE survey, NM+ launched the large-scale USE communication campaign to improve retention and proper use of all types of bednet, including FMOH's freely distributed ones. Figure 7: NM+ partners ITN Sales by Outlet [Source: NM+ Annual Report and Distributors ITNs Sales Database] Institutional sales (green) varied, but commercial sales (blue) declined steadily while voucher sales (red) followed geographical expansion first, but then dropped down, believed to be related to the increased free net distribution, the spread of strongly subsidized PSI nets, and the DACA imposed increasing difficulty to legally import LLINs. Figure 8 demonstrates how partner sales decline almost as a mirror-image of the rise in free net distribution. A final challenge to NM+ was that many retailers preferred PSI's nets and scheme over their partners' nets: they stated that PSI gave them just what they wanted, and left them a 25% margin. PSI did not expect them to prove the nets were distributed to vulnerable persons, i.e. pregnant women. By contrast, with NM+ partners, retailers stated they had to take too many nets at once, and were given just a 2% margin, and that these nets were only to be given to pregnant women When this challenge became evident, it prompted NM+ to consider raising subsidies on their TSP nets to a full 100%. At that point however, none of its partners were able to import LLINs as FMOH policy then prescribed. In response to a situation of rapidly declining sales numbers, and after meeting with USAID/E, NM+ was told to close down 49 of its 81 outlets, refocusing instead on IEC/BCC, taxes and tariffs, local production and registration issues. Multiple responses to challenges Figure 7: NM targeted subsidy sales (2005- 07) [Source: NetMark PowerPoint Figure 8: Evolution of various types of NM partner sales 20 NM+ in Ethiopia responded in a number of ways to address several significant challenges to its goal and objectives. To overcome the challenges, NM+ undertook the following initiatives: 1. Insisted (in vain) with FMOH on respecting segmentation and on better leakage control. 2. Discussed segmentation with PSI, asking it to leave urban outlets to NM+. 3. Gave vouchers to mothers of children under age five, not just to pregnant women. 4. Increased the amount of subsidy to voucher nets so that women paid 10 or 18 (compared to PSI's 20 birr). 5. Considered making voucher nets 100% free (however, its partners lacked legal LLINs by then). 6. Met with PSI after the exclusive agreement with VF lapsed (2007), and bought products from them. 7. Requested DACA several times to allow importation of ITNs until manufacturers could complete the lengthy DACA registration process. 8. Met with USAID/E to request support with the DACA issue requiring higher intervention. 9. Wrote a letter to the Minister of Health explaining program challenges and requesting support. 10. Gave tools and TA to Addis Ababa University to expedite bio-efficacy testing for LLINs. 11. Started closing down (2007) the least performing voucher outlets (going from 81 to 32). 12. Refocused on IEC/BCC, taxes and tariff, local production and registration issues. Despite these many efforts, NM+ continued to have considerable difficulty in carrying out its goal and key objectives consistent with its program design. In discussions with USAID/E, there was agreement to adjust program strategies within the core mission, but there was no decision that would have enabled NM+ to drop any of the key objectives or to change its overall goal. Conclusion (Responses to challenges ) The initial surge in NM+ partner sales in Ethiopia proves that the broader NM+ Africa model might have survived a less adverse context. It actually fell to NM+ to draw attention to the threatened collapse of segmentation that FMOH itself had initially prescribed. But NM+ succeeded in convincing the GoE that free distribution of nets alone cannot fill or prevent excess private demand. Clearly the regional NM+ office in Johannesburg did not plan for the situation where NM+ Ethiopia would have to compete with free net distribution: this is despite the fact that both it and USAID/E had been well aware that such a major change was looming. It then became NM+’s role to minimize the impact of free nets on its situation in Ethiopia. In this, NM+ was arguably reactive enough but remained constrained by the slow speed of reaction from the many stakeholders it had to influence in order to effect change. On the issue of leakage, NM+’s belief, shared by other stakeholders, that segmentation would keep free nets from leaking back into commercial urban net markets appears naïve, given the low malaria fears, the poor IEC/BCC brought to address behaviors related to malaria prevention, and the issue of equity for the urban poor. Ironically, such back-leaked nets, in fact, recycled into a purely demand-side private supply, the very core goal that NM+ sought to promote. 21 Program Management 10 STRATEGIES 1. Effectiveness of NM+’s Planning and Implementation of Strategies and Interventions The broader NM+ strategy for African countries came out of a collective, mainly internal process (cf infra). Its ‘Full Market Approach’ prescribes the tracking of 29 indicators but preparatory field surveys also inform major alterations to NM+’s strategy in Ethiopia. Since the broader NM+ approach has been refined in other countries in Africa since 1999, formal partners are apparently asked for input only “where possible, when the need arises”[NM+ Ethiopia]. In judging the program’s performance in Ethiopia, the evaluators noted that their role toward 'increasing supply and access to ITNs' was less to implement by themselves, but to support implementation by its partners – for example, market priming through 'seed funding' (50% contribution to the cost of imported bednets) and brand promotion through ‘matching funds’ (50% contribution to their promotion costs). This caveat, however, does not hold for the other four major interventions (i.e., Targeted Subsidy Program (TSP); enhanced communication campaigns (IEC/BCC); creation of a supportive environment for suppliers/distributors through removal of taxes and tariffs; and incubating local bednet production). Here it is NM+’s own efforts that decide outcomes, though the last two do require active and adequate partner response. Major activities to support NM+ strategies were found by the evaluation team. They include the following: NM+ road shows were well attended, attracted people from up to a 30-minute walk away, and were thought (by staff) to have a lasting effect [pre-post KAP comparisons are being added to its IEC/BCC interventions in 2008]. A van with voucher and full cost nets was part of the caravan but usually made few sales. For partnership development and capacity building, all malaria stakeholders agreed that they appreciated NM+’s efforts at sharing information, such as suppliers to hear from FMOH about pending net procurement, NGOs to trade IEC/BCC materials and synchronize malaria messages, and future manufacturers or current testing facilities to receive valuable inputs of TA. NM+’s field partners (health staff, net retailers) appeared less content. They reported that they did not receive reports on their inputs, were promised workshops that failed to materialize, at times had difficulty contacting NM+ and were not consulted when their contribution to the target families' health was unilaterally suspended by the closure of most TSP sites. Several net retailers also seemed poorly briefed (there is a high staff turnover). When interviewed by the team, they made comments that revealed they were unaware that they could now provide bednets to under-five children as entitled recipients, or that they could continue selling private bednets independently from NM+. Many did not understand the need for vouchers, when nets could also be gotten directly from the health centers. The evaluation team also observed that as a logical result of DACA's stringency, precisely during the season with highest contraction rates and with reportedly high demand, bednets of any type (including PSI's) were out of stock in the sites the team visited, except for back￾leaked nets in Bahir Dar's open market. The evaluation team conducted a small poll of key informants (see Appendix D) to gauge the degree of agreement between donor and contractor regarding NM+'s strategies and interventions. From this poll, it appears that the contractor and USAID/E’s perspectives agree most that IEC/BCC is a critical component of the program. There is less agreement on the 22 issues on supply/ access to bednets and the TSP components. The opinions regarding NM+ efforts on the removal of taxes and tariffs and local production are also not aligned. Conclusion (strategies/interventions) Given its major success in other African countries, it was understandable that there were lower levels of apprehension when preparing its 7th African launch. Unfortunately, NM+'s strategy for Ethiopia appears to have been more of an external strategy and approach, bypassing local inputs that could have enabled it to take into account a difficult and exceptional local context. This external approach also did not take into account the need for a formal agreement on what exactly was expected of the program (i.e. was NM+ to try to bring any kinds of nets for sale everywhere, or rather take another approach such as that of PSI, of trying to sell everywhere as many of one’s own branded nets as possible). The major issue when the original strategy did not bring the expected progress, was whether NM+ should fight on using the same strategy or rather should cut its losses. If the latter approach was taken, USAID/E's expectation regarding NM+’s role in the promotion of private net supplies would have been clearer. Discussions and agreements on the need to adapt activities or even the larger strategies would have been helpful, using a more common best practice: this would have lessened the need to abandon primary goals altogether. In addition, in addressing project objectives, NM+ efforts to remove taxes and tariffs appears to look far more sustainable to USAID/E than it does to NM+, who is acutely aware that DACA registration constraints limit net supplies more than does the VAT. Assessments of the situation did not provide a different approach to be adopted. Finally, in the poll conducted by the evaluation team, it appears that NM+ did not adequately track its program indicators and thus may have felt more satisfied about its TSP and IEC/BCC efforts than USAID/E found warranted. This could be interpreted from the difference in their poll scores. 11 MANAGEMENT The NM Regional staff in South Africa found that in Ethiopia NM+ reporting on project activities and finances were detailed and timely. NM+ itself would conduct bi-weekly to monthly monitoring of its partners’ and their retailers' sales and stocks, and review the monthly marketing and 'cost-sharing' reports it received from them. NM+ partners monitored their retailers in turn, with one partner making almost daily phone calls (mainly tracking non￾bednet products' stock), while another partner did monitoring instead by quarterly visits. Looking into staffing issues, the evaluation team assessed that there were two major management issues resulting in the NM+ team in Ethiopia having not worked at optimal staffing capacity, except for nine of the 48 months of its work. The first issue was that it took nearly two years for the Marketing Officer to join the country team. The second arose because the 81 TSP sites were supervised by just a single Field Officer for nine of the 16 active months; the sites should have been tracked by at least two of the full three field officers who initially conducted the site visits (cf Timeline supra). In addition, the Marketing Officer's job description did not mention "assistance with communication activities, with local production activities and with registration issues," although it probably should have been part of this staff person’s job description. It was noted that instead, the Admin/Finance Officer's job description did list these tasks. During 23 interviews and from poll results, the evaluation team sensed that the Country Manager and Marketing Officer limited their project-related information exchanges largely to IEC/BCC tasks. It was noted also that when NM+ replaced its two remaining FOs with a single third person, it did so within a short, two-week on-the-job handover timeframe, which appears not to have been an adequate timeframe for training the single, incoming Field Officer. Conclusion (in-country management) NM+’s role was to provide TA and support to partners to carry out project activities and to that extent, depending on partner action was built into the approach. Advocating for unusual approaches and ideas with a great variety of stakeholders to act on them is a considerable challenge that calls for more than one dedicated advocate. A second advocate should have been the Marketing Officer, who normally takes a far more public role than an Admin/Finance Officer, even though it was the latter's job description that actually defined this type of assistance. The Marketing Officer arrived nearly two years after the project began; this left the Country Manager (CM) to deal with the numerous stakeholders alone. Given her workload, this could have been the reason that more than a year elapsed from contacting potential manufacturers to sending them to a Kenyan seminar on ITN manufacturing. It was another year before the NM+ partner, Adami Tulu Pesticide Processing Plant, could secure government funds for local production. Bioassay instruments, destined to shorten the DACA efficacy testing period, arrived some 20 months after their need was first recognized. Human resources were insufficient to supervise the 81 TSP sites for much of the time. Given the unusual Ethiopian context, the NM South Africa office might have proposed adjusting the Addis staffing structure and roles to cope with the workload. A more rapid recruiting and placement of staff could also have supported the project’s workload. 12 PROCESSES NM+ in Ethiopia used a process of first trying to solve problems on its own initiative. The Country Manager stated that “any problems soon enough found her whenever the NM+ model was challenged. In such cases she would try to talk, to seek acceptable solutions, and to understand/ask how the program could be of assistance. If this process did not resolve things, NM+ would ask FMOH, the Regional NM+ office in South Africa or USAID/E for advice. If major funding changes were involved, NM+ Ethiopia also consulted the NM+ HQ in Washington. NM+ Ethiopia would promptly email or phone the person in charge so that, depending on signatory's availability, important documents were signed within 24 hours. If more than a three-day lag ensued, NM+ normally followed up on it. As for NM+ field officers, they could solve most problems on the spot, but if not, they could easily phone their office in Addis. An explanation of the M&E processes as part of program management is given further below. 13 OVERSIGHT The NM+ HQ office in Washington directed the Regional Office in South Africa to travel to Addis in order to provide TA and supervision while keeping costs down. The U.S. HQ used teleconferencing, as needed, to provide support and supervision in their interactions with the NM+ office in Ethiopia. This provided a more efficient, and cost-effective alternative for oversight and support. In the early phase of the project, the Regional Supervisor from the South African office visited NM+ in Addis every 6-8 weeks, during a 'performance-based management' initiation phase. As confidence in NM+ Ethiopia’s Country Manager grew, this gradually gave way to a 3-4 month schedule of supervision visits later on. The evaluation 24 team was told that NM+’s Country Manager would have preferred the Regional Supervisor to visit at least quarterly. NM+ staff in Ethiopia felt that support from the NM+ HQ office in Washington had been very effective, both in providing TA for the local production setup and with ad hoc problem solving, as the HQ could draw on its wide and deep NM+ experience in the other African country programs. However, since NM+ is a centrally funded award, many matters have to be negotiated and agreed to with the NM+ HQ office in Washington. According to the USAID/E CTO, this made things complicated and could create, for example, frequent delays in money transfers and in agreements between Washington and Addis or between AED and USAID. Conclusion (processes and oversight) In addition to the absence of a formal project document and with the small size of the country team, NM+ HQ office inputs appeared efficient enough despite limited communication. The fact that this was a centrally funded project meant NM+ Ethiopia worked with multiple layers of program support and communications (USAID/E, NM+ Regional Office in South Africa, and NM+ HQ in U.S.). This complex structure likely opening the door to the (occasional) misalignment of efforts, and some accumulation of delays. In addition to suboptimal staffing numbers, the evaluation team’s overall assessment was that NM+ suffered from misalignment caused by too few and irregular face-to-face progress meetings. The NM+ HQ office in Washington gave input, which caused some difficulties such as delaying the direct, reactive decision-making provided by the local NM+ Ethiopia office. These NM+ management difficulties were, of course, taking place within an overall difficult Ethiopian context. Program Sustainability and Expandability 14 SUSTAINABILITY The sustainability issue for private-sector nets is complex. Disregarding for the moment the ultra-poor who appear unable to even pay 2 birr, the evaluation found that the moderately poor may be able to pay 10-20 birr and the less poor 30-60 birr for an ITN. However, the full price of an in-county LLIN ranges from 75 to 100 birr. It has been discussed also that Ethiopia's potentially biggest and most affluent as well as educated market, Addis, is not malarial – nor are most other major urban centers; these are usually located in mosquito-free highlands, this limits substantially the market and potential demand. Global viability of 'made in Ethiopia' nets would depend on rivaling Vietnamese, Thai or Chinese nets ($2-3 US each or 20-30 birr) and on securing WHOPES certification. For this reason, and because some organizations (e.g. UNICEF) take a commission on FMOH procurements, big tenders are normally taken by the leading global LLIN companies, while smaller ones regularly fall to PSI, because of its 'legal' LLIN and convenient warehouse size. When asked, NM+ partners say their viability depends on big volumes of either preferably institutional or greater voucher sales, with at least 5 birr/net margins and further ‘cost￾sharing’ (50% transport, salaries, promotion) to be provided by NM+. They consider that it is not viable to outreach and supply down into small rural towns, to bear the cost of raising awareness, or to try to promote purely commercial sales (despite NM+'s matching funds). Partners agree the bednet market needs time to grow (through IEC/BCC or free net exposure) but think it might grow also by such means as adding more regions to their intervention area. 25 The rationale for optimism regarding the long-term future of bednet sales remains poor, since even, for instance, “rich” South Africa still has 4 out of 5 people living in poverty. Partners such as Pertam and EAI therefore both stated that they would stop as soon as NM+ stopped According to additional key informant interviews and the poll done by the team, it appears a more sustainable approach would be the removal of taxes and tariffs by the GoE and, to a lesser extent, and local partners continuing to import and sell bednets to expand access. The GoE and donors can continue to support access to vulnerable groups. The FMOH and NM+ appear to agree that local production, IEC/BCC outcomes and expanded supply/access are equally sustainable (while the voucher component might prove to be unsustainable. Conclusion (sustainability) There may not be enough wealthy people at risk of malaria in Ethiopia to keep private-sector nets afloat, especially when it is not possible to enforce a clear rural-urban segmentation in the program. If vouchers were to be stopped as well, only institutional sales could help the private net sector survive until a behavior of bednet use eventually is adopted by a growing Ethiopian middle class. Sustainability for the marketing of bednets is also limited. For NM+ partners to get a chance at institutional (international) bidding, their DACA registration is needed in Ethiopia. Their WHOPES certification (as a minimum) would be needed for sales outside of Ethiopia. The process of DACA registration for nets in Ethiopia could take years: and without a guarantee that rival prices can be beaten or that profit margins will be viable. Thus GoE and donor aid may be needed to support sustained private-sector activity for the foreseeable future. 15 EXPANDABILITY Free net distribution began in early 2005 and replacement of nets is now due because of decreasing insecticidal residual effect and because of wear and tear. The easiest way to replace nets is to distribute new, free nets to everyone. Recently, a FMOH application to the GFATM Round 8 was recommended for funding. At this point, the key question becomes how much longer will these free nets come to Ethiopia? The evaluation team’s quick poll shows most stakeholders (USAID/E, NM+, MC) expected another 3 to 5 years of this, but both FMOH and UNICEF believe ITNs will be provided for free for at least another 20 years. Free ITNs cannot fully eliminate malaria; private demand will continue if only to cater to individual tastes regarding net shape, size and color. To date, distribution of large quantities of free nets have taken place soon after the goods arrived in-country, but their timing was not predictable. This irregularity has caused some entitled families to not receive free nets. Reliably identifying and tracking those in need of replacement nets is also quite a challenge in rural areas, where communications, distances and other difficulties challenge comprehensive net distribution systems. This then implies that there is an unmet demand in-country into which NM+ partners could expand in the future. Free net distribution has also overlooked the urban poor (including pregnant women and U5s) and many additional children in a household (free net numbers may not match the number of recipient family members) or adolescents (who are not willing to continue sleeping with younger siblings) as well as migrants who have left nets to those staying behind. Despite attempts to distribute to vulnerable populations, there remain gaps in supplying to all those who need protection, and difficulties in addressing taboos that affect the use of bednets in the household. Also there are still many people who say they will buy a 26 new net once the present one becomes ineffective or they just wish to buy a white oversized conical net perhaps to demonstrate their relative purchasing power. To this real and current demand (an anecdotal estimate would be one in four to five families), the evaluators judge that neither free net distribution nor even voucher schemes can respond in the absence of private supply. Thus NM+'s main goal of creating sustained commercial bednet availability can realistically count on such ongoing future demand. NM+ staff's preference in programming for the future, however, centers on increasing their efforts for more and better IEC/BCC, further work toward registration and local production, and possibly promoting the distribution of 'integrated malaria packages,' including subsidized malaria tests and drugs plus LLINs. Conclusion (future plans) There remains a continued need for demand-side net availability in cities, towns and the countryside despite the free supply of bednets. Very few retailer nets can go full-priced, so a socially marketed subsidy program with expansion to additional markets is still needed. IEC/BCC will be necessary to further promote malaria prevention and bednet use. 16 REPLICABILITY The Ethiopia program is actually an attempted application of what the broader NM+ program very successfully brought to six other African countries. However, producing a thriving private market was not possible because the right context was not found in Ethiopia. The NM+ program could be expanded into other Regional States in Ethiopia including into malarial urban sites below 2000 meters: this would include supply and access to nets, the TSP program for vulnerable groups, and IEC/BCC. The removal of the high taxes and tariffs has benefited the entire country. Conclusion (replicability) Policy, registration and taxes and tariffs issues have country-wide impacts. The local production of nets could be replicated to meet demand across more regions in Ethiopia. This production would require centralized funding and a centralized production location. NM+'s efforts in IEC/BCC can be of great benefit and can be replicated to promote the demand and proper use of bednets elsewhere in Ethiopia. 17 SPILL-OVER EFFECTS There have been spill-over effects from the program to other areas in Ethiopia. PSI was reportedly pleased to note its sales went up (by 7-10%) whenever NM+ and its partners put out generic or even branded advertising. NM+ road shows normally attracted food and other stalls. This had the spill-over effect of people conducting official or community business (e.g., microfinance) during or after the NM+ performances. Retailers who were initially reluctant to sell ITNs discovered that women often purchased other items besides nets from them, so their businesses expanded. Also, according to interviews with representatives from CAME, some NGOs (Carter Center, Malaria Consortium) have, on a modest (25%) scale, copied some of NM+’s IEC/BCC road shows in the Southern Regions, in Tigray, Afar and Somali. In addition, DACA’s new testing equipment (UV light, Polaroid camera, alpha￾imaging bio-efficacy components), provided by NM+, are now also used for non-insect molecular studies. 27 The evaluation team also noted that the broader NM+ joint investment with Tana Thailand and Bayer Environmental Science in a number of countries in Africa, created a mechanized LLIN treatment process in 2005 that was environmentally friendly, safe for workers, easy to adopt by African manufacturers and in the public domain; it became a LLIN technology freely available as 'open source'. The new process has been shared with many African and international manufacturers and Siam-Dutch used it in mid-2005 to build a $5 million factory in Thailand. Conclusion (spill-over effects) NM+'s global network created innovative open-source technology that is now freely available and can move into whatever company adopts it. Those who reproduced NM+ IEC/BCC and other promoting activities have expanded their malaria activities to reach additional populations. These organizations could be given further support and TA to reach beyond their current areas. The team noted that NM+'s was the only project that provided a social responsibility role as a model both for successful companies and for small-town sellers and this could be replicated for additional effect. Program Linkages   18 LINKAGES NM+ recognized FMOH’s policies for the role of health extension workers and other health staff who are responsible for health service activities in the home, kebele or in facility-based case management. NM+ has played a strong role in malaria prevention and control linked to the FMOH’s service delivery role, with staff training and through promoting the use of bednets and other prevention behaviors. NM+ is a full member of the FMOH’s NMCST and of its IEC/BCC subgroup. They meet monthly and quarterly to coordinate various malaria-related initiatives in-country and to discuss updates to the National Malaria Control Policy. CAME is hosted by the Malaria Consortium and currently counts 70 members, 15 of them businesses brought in by the program. NM+ is also a member of its Steering Committee and Chair of its Technical Advisory Committee. The program has established a very strong relationship with free net distributors by gearing its communication activities toward free net beneficiaries (e.g., the Umbrella and the USE campaigns). Conclusion (program activities and linkages) NM+ is widely acknowledged as an open and eagerly collaborative partner with particularly high quality IEC/BCC materials and messages to share. Retailers and health staff also appreciated NM+'s purpose and teachings. Program staff actively linked with service delivery staff at regional and health facility levels. NM+ also linked with the CAME coalition at the national level to provide expertise. As an organization, NM+ kept friendly relations with all other programs distributing bednets, including PSI. The most intense relationship NM+ undoubtedly had was with its competitor, PSI. On expected linkages when there is actually intense competition between PSI and NM+'s partners for the bednet market, USAID/E repeatedly commented that these organizations should collaborate. This overlooked the fact that this was hardly logical given that PSI staff is paid a bonus proportionate to PSI branded net sales and that true competition negatively affects NM+'s main objective. The two organizations had different and non-compatible 28 strategies. In addition, PSI's price for nets rendered the competitiveness of any commercially supplied bednets not viable. Capacity Building 19 CAPACITY NM+ provided capacity building to implement its market priming initiative by using TSP as an entry point. Since this program type was new to Ethiopia, a number of trainings were given to health personnel, to retailers and to distributors. Throughout the life of the project, NM+ trained 312 health staff (on social marketing approaches to disease control), 162 retailers, and 400 coordinators for its ANC campaign. These coordinators were “foot soldiers;” five people were selected per kebele – locally well known people who were very active and liked by their community. They were trained for 4 days on how to communicate and how to encourage pregnant mothers to visit ANC. Using voucher stubs, 25,000 of these were tracked and invited into discussion groups to listen in on a six-week interactive radio program about malaria and HIV. Because of the DACA-related lack of LLIN supplies, there was no reason for NM+ to start up training of private partners on social marketing or on program management. Table 4: Malaria related KAP changes (2004-07) KAP Indicators NM + Survey 2004 MIS Oct-2007 NM+ Survey 2008 Urban Rural % of respondents had heard of mosquito nets 0.7 59.4 26.5 % who heard of malaria 89.5 71.0 % who report mosquito bite as a cause of malaria 59.7 30.1 % who recognize fever as a symptom of malaria 64.8 39.5 % of children under five slept under an ITN the prior night 6 79.5 57.1 67 % of pregnant women slept under an ITN the prior night 6 61.3 59.1 63 [Source: NM+ ITNs 2004 & 2008 Surveys, MIS 2007 Report] The team found indirect evidence that the program has indeed increased the capacity of the public on malaria knowledge (Table 5). NM+ has influenced people's malaria-related KAP; much higher KAP scores are found for most urban areas as compared to rural areas. The evaluation team judges that this increase can be partly linked to NM+'s communication campaigns there. It is true that many other factors and influences must have affected these changes as well. Numerical comparison data on changes in malaria-related KAP will become available in 2009, when the follow-up to its 2004 survey is conducted. Conclusion (individual and community training) NM+ worked to build the capacity of health staff, with hundreds of health staff, retailers and coordinators having been trained. Health staff and retailers were quite happy to receive malaria-related training. However, due to high staff turnover and low refresher follow-up, health staff has been requesting additional training. At the community level, a significant proportion of the population remembered hearing that bednets protect against malaria or hearing ANC-related radio messages. While they remembered hearing messages, the link of malaria to mosquito bites was often unclear to the 29 many who recall first that one catches malaria from swampy areas, but who then stated that they were avoiding swampy areas already. Monitoring and Evaluation 20 EFFECTIVENESS As of the 2006-2007 NM+ work plan, USAID/E helped NM+ adjust its framework indicators to include inputs, processes, outputs and outcomes related to ITN utilization, ANC attendance and voucher redemption. Earlier work plans tracked 18 of the 29 indicators on beneficiaries, facilities, retailers and distributors suggested by the M&E Guide for NM+ TSP activities. Those data were collected on the one hand from distributors' monthly sales reports and on the other from bi-monthly monitoring visits (e.g., vouchers given out and redeemed, problems identified) during a total of 95 visits between 2005 and 2008 (roughly twice monthly, varying with Field Supervisor availability). These voucher data do not distinguish between pregnant women and mothers of children under 5, and absent are graphs tracking program coverage (% of target groups reached) or health and retailer staff knowledge regarding malaria, bednets and vouchers. Currently, some such data are available at the program office, but these data have not been compiled or analyzed and are thus difficult to use for program decisions. The 2006-2007 framework lists progress indices on improving access, creating demand and increasing voucher redemption. NM+, however, did not link its program M&E indices to routine reporting the various health delivery facilities put out. Field visits were conducted by field officers from Addis Ababa without involving local authorities at various levels, though an initial agreement with Regional Bureaus had been obtained. Conclusion (M&E effectiveness) NM+ usually monitored voucher numbers and redemption rates, plus commercially sold net numbers and stock outs. These data were later used in deciding which TSP outlets to phase out. Though it appears that demand, access/availability and affordability of ITNs have improved, the relative dearth of more pertinent data does not allow the program to address questions such as: to what extent, where, or whether benefits have extended beyond urban centers. The field monitoring reports present a general impression of the situation without providing a systematic and thorough analysis. For example, no evaluation was conducted to identify causes of redemption rates remaining at or below 65%. Of concern is the absence of systematic tracking of prices and trade margins from suppliers through to retailers. This impedes gauging the extent to which the ITN market operated efficiently to make affordable nets available to consumers. From the team’s admittedly limited number of health facility staff interviews, staff reported that field administration of the voucher program was seen somewhat as an additional workload imposed from the outside, so they often mechanically filled in forms as a favor. Collection of the NM+ M&E data/information was discussed in the interviews as an added burden, not as a support to better local decision making. It was commendable that NM+ provided the prompt addition of pre-post KAP indicators to its later campaigns, soon after attending the PEPFAR M&E workshop in 2007. 21 SYSTEM The M&E system, as described in the NM+ Evaluation 2007 annexes, lists 22 indicators, of which just three were not applicable to NM+ in Ethiopia (because the corresponding activity was not (yet) implemented). Of the remaining 19 indicators, NM+ routinely tracked eight and 30 (through repeat-surveys) intermittently evaluated six. Of the other five indicators, two are transnational and three track in-country events. All require a single yes/no answer in the Ethiopia case. Review of NM+ program documents from other programs in Africa and related documents suggests another 30 worthwhile indicators might have been added. However, the decision to not monitor them was jointly decided against by NM+ and USAID/E in 2006. It is unclear whether the NM Regional Office in South Africa for its part tracked a number of progress/impact indices as part of its supervision and support of the NM+ program in Ethiopia. Conclusion (M&E systems) It was good practice to conduct a baseline study at the outset in 2004. This study will enable the 2009 scheduled follow-up survey to present and compare data on the interventions’ effects. The majority of the NM+ broader program M&E system’s applicable indicators were tracked by NM+ Ethiopia as part of its 2006-2007 work plan. To properly evaluate performance, an additional set might have been tracked, with trends then charted so as to identify corrective turning points for program decisions at the appropriate time. NM Regional and HQ offices’ supervision could have been more detailed and demanding in requesting these. A situational analysis done even earlier, during the program design phase (best at the national level and before embarking on regional operations), would have been useful in promoting realistic expectations (e.g., through analysis with key stakeholders of existing situation, potential problems, constraints and opportunities). 22 USEFULNESS USAID/E requested that NM+ start in Amhara based on its population and malaria prevalence figures. It was later decided, because of the substantial voucher uptake there, to expand into Oromia. At the time, the voucher program had a seemingly successful start, and attracted $300,000 PEPFAR funds. These additional funds then allowed for a country-wide promotion of ANC visits, including their integrated PMTCT goals. NM+'s monitoring of partners' sales figures led them to write a letter warning that they would cut match and seed funds if Petram's performance remained low. Also, after conducting a survey, NM+ realized there was a need for IEC/BCC messages promoting the proper installation, use and care of all bednets, including free ones. USAID/E directed NM+ to close 49 outlets based on their sales figures. Finally, this evaluation shows USAID/E and NM+’ commitment to use data to boost impact. Conclusion (M&E usefulness) Despite some noted weaknesses in M&E, NM+ did in fact track the fundamentals and, thus, routinely generated enough data/information to support various program decisions, some of them quite far-reaching. These commendable outcomes confirm the importance of information gathering and analysis. 31 23 LESSONS LEARNED The absence of a formal project document has affected the alignment between the expectations of USAID/E and those of NM+. Given the success of the NM+ global model in several other African countries, there was not a strong need felt for a formal project document detailing NM+ specific approach, goals or methods in Ethiopia. Instead, routine reference to NM Africa's Concept Note was made in yearly work plans. These formulations did not align significantly divergent expectations between USAID Ethiopia and NM+ in Ethiopia. This was reflected in concerns expressed by USAID/E that addressed NM+’s approach (e.g., whether NM+ should loyally pursue long-term goals or seek more direct success instead). They stated that PSI was much more flexible; and moved quickly to a better strategy as soon as things did not work out. The NM+ model requires proper consultation and customization to serve the specific needs of Ethiopia. The FMOH, Regional Bureaus, DACA and others should have been actively consulted and the approach more customized as to why and/or how to implement NM+ in Ethiopia. This would have made it easier to avoid or address difficult constraints to the project’s success, such as leakage control, registration hurdles, free net impacts, and health staff motivation. The NM Concept Note's Full Market Approach (FMA) also took for granted pre-conditions that were poorly met in Ethiopia (non-malaria Addis, a small middle class, an incipient private sector, few secondary industries). Tailoring the FMA to local realities would have greatly benefited the implementation of NM. A strong link with the MOH at various levels is requisite for project success. A strong link between partners would benefit from productive two-way exchanges: for NM+ to share project information or, better still, to plan together with FMOH departments, would have eased obtaining reciprocate data on system-wide impacts from them. These linkages and exchanges would also have helped woreda (district), zonal and regional authorities to feel that they jointly owned the project. In addition, health staff wishes and opinions, especially at the TSP site level, would have, if been better solicited, secured their ongoing voluntary contributions and collaboration. Filling Key NM+ staff positions promptly and continuously is critical to program success: frequent progress meetings help provide for effective communications and support. With the critical need for building support for the project in agencies with retailers and others, and the relatively high level of resistance to market ideas, a second, preferably local advocate/manager with solid commercial experience would have been helpful for accelerating the projects' efforts. This second advocate arrived nearly two years after the start of the project. Other key positions such as the three Field Officer positions should have been continued, given the high number of sites (81) that needed supervision. The program was staffed with three Field Officers for only 9 of 16 full capacity months. Frequent progress meetings are helpful for program effectiveness: the evaluation team assessed that program staff and donor and GOE staff had limited face-to-face progress meetings, resulting apparently in misalignment of expectations regarding key issues. NM+, as well as key FMOH officials, apparently lacked adequate interchange regarding the feasibility of the strategies to be used for the program. NM+ trusted some official insights that in fact negatively affected the program strategies. These insights included the ungrounded rationale with which the FMOH decided that rural areas were to receive socially marketed nets for free although urbanites had subsidized nets available. In addition the FMOH decided that urban markets were to be saturated before rural distribution was to be started. Limited interchange on the 32 inherent conflict between segmentation and equity, for example, is seen in the fact that the urban poor initially were left out of the population that needed nets. A more robust monitoring and evaluation system provides data for program decisions. The program’s M&E capacity was limited early in the program. NM+'s Country Manager attended one M&E workshop, in late 2007, three years after program start up. Prior to that NM+ tracked a limited number of voucher and outlet related indicators. Thus, data/information about population coverage, economic/health related impacts, prices and margins and objectives-related performance were not fully adequate to guide program decisions and make timely adjustments. In addition to internal staff capacity to build M&E, FMOH staff input into M&E could help build a strong, pertinent M&E system. Competing social marketers offload costs on donors, not on suppliers. A free market axiom is that competition helps consumers by forcing higher productivity and lower pricing on manufacturers. Social marketer competition helps customers too, but does so by taking donor funds that could be used for other worthy causes, and bypassing those who could pay a socially marketed price. This issue would need to be addressed during program design and reviewed periodically during program implementation. This issue was not fully addressed when the NM+ program was designed for Ethiopia. Project planners likely used, as a reference, the program experience in Uganda, the only other African country where NM+’s broader African program and PSI coexisted successfully. This success, however, was based on both SM agencies formally agreeing and strictly complying with unambiguous segmentation conditions. 24 CONCLUSIONS AND RECOMMENDATIONS The evaluation team recommends that certain future program interventions, not the NM+ project as initially designed, should receive continued USAID support. The two key objectives to be pursued further are to increase supply and access to bednets (especially for vulnerable groups), and to promote malaria related IEC/BCC widely. Additional technical assistance should be provided to strengthen and expand local production and registration objectives. Strong advocacy interventions continue to be needed and should be provided in relation to donor purchases and WHOPES integration. The following sections discuss the recommendations and assume that free/subsidized nets will continue to be provided for the near future. Impact Several of NM+ Ethiopia’s objectives remain valid and still need to be implemented; justifying continued funding to that effect. Continuation of the current NM+ objectives should be supported to promote its partners' and others' institutional sales of bednets. This would include assured support to its partners' (manufacturers') efforts to obtain WHOPES certification (eventually to bid for Global Funds) and DACA registration (nationally to distribute nets). TA for local bednet manufacturing and quality control for regulating their scale-up is still needed as well. IEC/BCC remains a major need at the grassroots level, and here NM+ has excellent credentials. NM+ does not seem best placed to continue taking on the supply/access and TSP goals. NM+'s TSP coverage is still very limited while TSP costs are quite high. Assessment of these program components show that other in-country organizations have more cost effective 33 opportunities to provide these interventions. USAID should explore options for these particular interventions. In addition, since blanket subsidies cost far less than managing vouchers, USAID/E could consider this approach. The vouchers might be replaced by 2 nets a pack priced at 150% the normal/single pack price. USAID PMI funds should be used to support these 'young family packs.' Management Future malaria program interventions should be supported by early, adequate and consistent staffing to manage and implement activities. Program management meetings should be held on a regular basis to assure clear and frank communications between USAID, the implementer program staff and the GoE. Future program design take into account and adapt the NM+’s broad Africa model to the specifics of the environment of Ethiopia. This would include strong efforts at proper local consultation and customization of the Africa-wide model and the formal documentation of the design developed for Ethiopia. Advocacy Advocacy should be continued on malaria policies, certification and other issues. In the future, USAID should continue to support TA and other activities to strengthen advocacy initiatives with the FMOH and GoE. This support should be furthered to assist the FMOH and GoE to update their National Malaria Prevention and Control Strategy. In addition, advocacy and discussions should address the issue of at least 20% of the nets purchased with GFATM funds for future distribution of free nets to be purchased from in￾country suppliers. This would help use institutional sales to keep the local private sector afloat. USAID/E should take a role in leadership and support for these key issues. Advocacy to address further reduction of taxes and tariffs on various malaria control products should be provided in any future program More advanced tracking of SM nets leaked by officials (usually at the kebele level) should be enforced. Careful dialogue and advocacy with DACA could also be supported by USAID/E to help address the issue of DACA’s acceptance of WHOPES certification (used elsewhere as sufficient proof of quality) for Ethiopia. Currently, the stringency of their requirements is largely due to WB's demand for safety guarantees in exchange for WB loans to the GoE. USAID/E should support donor dialogue and efforts to try to bring FMOH and WB together in order to revise DACA's requirements and accept WHOPES certification standards. This would also need to address the issue of the $20,000 fee DACA is allowed to charge for each round of certifications. Linkages Strong links with FMOH departments at various levels are needed. Additional strong links and support to public sector health workers should be provided by USAID/E in future interventions. Ownership is fostered by initial joint planning between USAID/E and the FMOH; thereafter, close collaboration should be supported at lower levels as well. This should include promoting bednet coverage for all hospitalized patients, and very strictly so in the malaria wards. Acknowledging the FMOH's manpower issues with high staff 34 turnover rates, support should be provided for repeat training workshops and refresher sessions. This would also boost morale and motivate reliable effort from these often overburdened government workers. Capacity Building for IEC/BCC   IEC/BCC is recognized as the most urgent output needed country-wide. Future program interventions should promote IEC/BCC as its most needed output country wide. Medical anthropology should be included in the development of clearer and more convincing messages, noting that effective health communication is particularly challenging where the audience lacks the full core concept of the malaria pathogen. Data and information   Strong M&E systems support planning and monitoring of program interventions Program support to a well designed and promptly tracked M&E system should be included as a priority in future programs. The M&E system provides guidance for planning and monitoring key program achievements. More surveys and baseline information should be included in the design and implementation of future malaria programs. Yearly bed usage surveys with disaggregate data can pinpoint emergency flashpoints if promptly analyzed. Anthropological methods could be used to provide better information to present realistic field descriptions for assisting central planners to develop program interventions. APPENDICES 1 Appendix A: Draft Statement of Work (SOW) for the External Project Evaluation of Agreement No HRN-A-00-99-00016-00 The NetMark Plus Malaria Social Marketing Program Draft 1 – August 21, 2008 PROJECT IDENTIFICATION DATA 1. Project Title: NetMark Plus 2. Agreement Number: HRN-A-00-99-00016-00 3. Project Dates: October 1, 2003 - September 30, 2008 4. Project Funding: $2,600,000 5. Implementer: Academy for Educational Development (AED) 6. CTO / AM: Sonali Korde (CTO) / Richard Reithinger (AM) I. Identification of the Task The USAID/Ethiopia (USAID/E) Health, AIDS, Population and Nutrition (HAPN) Office requests technical assistance from the Mitchell Group (TMG), under the USAID/E Evaluation Contract Number 663-C-00-08-00409-00, to design and implement an independent external evaluation of The NetMark Plus Malaria Social Marketing Program (‘NetMark Plus’). This four-year $2.6 million program was awarded as a field support cooperative agreement to the Academy for Educational Development (AED) in October 2003 with the objectives of: 1) improving access and creating demand for insecticide-treated nets (ITNs); 2) establishing targeted ITN subsidy programs for pregnant women; and 3) catalyze local production of ITN products. This external end-of-project evaluation will assess the achievements, impact, and challenges for the project in addressing the above objectives. The USAID/E HAPN office requests that the evaluation be completed by October, 2008 so that the findings, conclusions and recommendations can be used in the implementation of planned follow-on activities and projects. II. Background Malaria is ranked as the leading communicable disease in Ethiopia, accounting for 30% of the overall Disability Adjusted Life Years (DALYs) lost. Almost 75% of Ethiopia’s landmass is malarious with 68% of the population at risk of contracting malaria. Malaria transmission is seasonal with two peak transmission seasons, from September to December after the heavy (meher) rains and from May to June after the short (belg) rains. Sixty and forty percent of malaria cases are caused by Plasmodium falciparum and P. vivax, respectively. Malaria is unstable in most parts of the country, erupting in periodic epidemics every seven to eight years with local annual epidemics. Such epidemics usually coincide with prolonged drought and subsequent famine. Attack rates are up to 30% and case fatality rates range from 2.2% to 5%. Over five million clinical cases of malaria are reported annually, representing 16% of all outpatient consultations, 20% of all hospital admissions and 27% of hospital deaths. However, as 36% of the population does not have access to health care services, these figures may under-represent the true burden of malaria in Ethiopia. The burden of malaria has been increasing due to a combination of large population movements, increasing large-scale 2 epidemics, mixed infections of Plasmodium vivax and P. falciparum, increasing parasite resistance to anti-malarial drugs, vector resistance to insecticides, low coverage of malaria prevention services, and general poverty19. Since 2005, primarily through support by the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM), the GoE has massively scaled-up malaria prevention and control activities in the country in line with its National Strategy for Malaria Prevention and Control [http://www.moh.gov.et/index.php?option=com_remository&Itemid=47&func=fileinfo&id= 178. Thus, approximately 20 million ITNs have been distributed, with households in malarious areas targeted to own two bednets per household [http://www.moh.gov.et/index.php?option=com_remository&Itemid=47&func=fileinfo&id= 180. Rapid diagnostic tests (RDTs) and ACTs have been rolled out to peripheral health facilities. Additionally, as part of its Health Sector Development Plan III (HSDP III) 2005 – 2010 [http://www.moh.gov.et/index.php?option=com_remository&Itemid=47&func=fileinfo&id= 192, the goal is to dramatically increase access to health care services in rural areas. A key element in achieving this goal is the Health Extension Program (HEP), which is characterized by 30,000 Health Extension Workers (HEWs) who will provide basic health care delivery at kebele level. Unlike for the other health packages, HEWs are supposed to diagnose (using RDTs) and treat malaria cases, besides being involved in more general (e.g. IEC/BCC) or seasonal (e.g. indoor residual spraying) activities. Without doubt the GoE’s scale-up in malaria case management, prevention and control activities has been a tremendous achievement and appears to have led to a reduction in malaria morbidity and mortality, but gaps have been identified following the implementation of the scale-up. With regards to ITNs, these are as follows: No long-term strategy exists to make current and future ITN coverage in the country sustainable, requiring significant financial support for ITN procurement and distribution each year; Whilst ITN coverage (i.e. defined as two ITNs per household in ‘malarious areas’) is exceeding 70%, ITN utilization (specifically by children and pregnant women) is as low as 30% in some areas; Community knowledge with regards to malaria diagnosis, treatment, prevention and control is very variable. III: Overview of the NetMark Program Summary: NetMark Plus maintains an ITN targeted subsidy program for vulnerable populations in multiple health centers in Amhara and Oromia. To date, the program has distributed more than 200,000 ITNs through a range of distribution mechanisms including targeted subsidy, commercial sector, full cost recovery and full subsidy/free distribution; contributed to the increased use of ITNs by the population at risk; prepared the ground for local production of ITNs; and facilitated the registration of ITNs in the country. This work has been supplemented by various surveys assessing end-user knowledge, attitudes and practices (KAP) with regards to malaria and ITNs. Additional background information is available at http://www.netmarkafrica.org. 19 All statistics listed above is taken from Ethiopia’s National Statistical Data and many can be found in the website links listed in the document. 3 Geographic coverage: Implementation was initiated in selected urban and rural areas of Amhara and Oromia Regional States. Currently the program is active in 82 locations in those two Regional States. Program Objectives and Expected Results: As stated above, the objectives of the NetMArk Plus program are 1) improving access and creating demand for insecticide-treated nets (ITNs); 2) establishing targeted ITN subsidy programs for pregnant women; and 3) catalyze local production of ITN products. Strategic Framework: USAID/E’s NetMark Plus program was initiated under the USAID/E Integrated Strategic Plan (ISP) for the period FY 2004 to FY 2008 under the strategic objective (SO) SO 14: Human Capacity and Social Resiliency Increased and SO 14.2 HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased. In 2007, the SO14 was incorporated into a new United States Government Foreign Assistance Framework (F-Framework) for the USAID 2007 Operation Plan. The activities under the NetMark Plus program now fit under the F-Framework Priority Objective: Investing in People, Program Area: Health; Program Element: Malaria; Program Sub Element: Insecticide-treated nets (ITNs) to prevent malaria. Monitoring and Evaluation (M&E): The NetMark Plus program reports on the following indicators mandated by USAID/E and include: • Number of ITNs distributed that were purchased or subsidized with USG support • Number of people trained in malaria treatment or prevention with USG funds Emerging Concerns: Over the past year, the following concerns have been identified by AED: • The availability of free ITNs distributed by the GoE since 2003 and particularly 2005 have impacted NetMark Plus activities and resulted in decreases in subsidized ITNs distributed through the program; • The GoE strategy to procure and distribute ITNs free-of-charge to the population at risk of malaria will not change in the short to medium-term future, making it difficult to maintain the commercial market and targeted subsidy approach as implemented under NetMark Plus. IV. Purpose of the Evaluation The purpose of this evaluation assignment is to implement an independent external end-of￾project evaluation of the NetMark Plus in Ethiopia. With less than a few months remaining of NetMark Plus, this evaluation will collect information about the program’s impact and achievements, implementation and progress, and challenges encountered. It will document lessons learned and formulate recommendations for future ITN social marketing activities and programs. The evaluation will cover NetMark Plus’s performance for 2004 to through September 2008. The management structure of NetMark Plus requires that the evaluation team work with both the Addis Ababa and Washington D.C. offices of AED NetMark Plus. While the primary focus will be on Ethiopia-based activities, the evaluation team should conduct interviews (either by conference call or in person) with senior Washington D.C. staff at the outset of the evaluation. It is envisaged that the evaluation will recommend amendments to NetMark Plus’s current strategy and/or alternate strategies in the context of 4 the national availability of free ITNs. Thus, this evaluation will be crucial for the development of any NetMark Plus follow-on activities both in Ethiopia as well as globally. The proposed end-of-project evaluation will attempt to answer the following illustrative questions (listed in order of priority for USAID/E): 1. Impact • To what extent were the program’s three objectives achieved: o In improving access and creating demand for ITNs? o In establishing targeted ITN subsidy programs for pregnant women? o In catalyzing local production of ITN products? • What was the program’s impact on the national malaria prevention and control strategy and, specifically, the national ITN strategy? • What were NetMark Plus’s efforts, challenges and solutions to adapt to the national availability of free ITNs? • What was the extent and success of the program’s technical assistance for the in– country registration process for ITNs? • In addition to any achievements related to the above three objectives, did the project improve the lives of the target population/beneficiaries in other ways? If so, how and to what extent? • How has the program changed the livelihoods and health among the target population/beneficiaries and some of the population groups at greater risk of malaria, e.g. pregnant women, children <5 years of age and/or people living with HIV? 2. Program Management • How effective has been NetMark Plus’s planning and implementation of strategies and interventions? • How effective have been the program management by in-country NetMark Plus staff? • What management processes are used by NetMark Plus to identify problems and resolve them effectively? • How efficient and effective did NetMark Plus staff based in Washintgon D.C. provide technical assistance to the field, particularly with regards to overall program strategy development and implementation? 3. Sustainability • Are NetMark Plus activities sustainable? In other words, is the community in a position to continue and replicate such activities in the future? • What are the plans for future expansion and sustainability of the program? • Can the program be replicated? • Have there been spill-over effects of the program to areas not targeted by the program? If so, to what extent and of what nature? 4. Program Linkages To Related Services • How well is the program linked to home, community and facility-based health service delivery, particularly regards to malaria case management, prevention and control? • What is the program’s relationship with other subsidized or free ITN distribution programs implemented by other in-country stakeholders? 5. Capacity Building • Has NetMark Plus built the capacity of health professionals or private sector individuals with regards to malaria prevention and control, social marketing approaches to disease control, and program management? 5 • Has the program built the capacity of the community, particularly vis-à-vis KAP relevant to malaria prevention and control? 6. Monitoring and Evaluation • How effectively has NetMark Plus monitored and evaluated program impact, quality and effectiveness? • Were the NetMark Plus monitoring and evaluation systems and efforts adequate to comprehensively track program progress and impact? • Has the data collected through monitoring and evaluation used to plan and manage the program and/or change some of its strategic approaches? 7. Lessons Learned • What are the positive and negative lessons learned from implementing NetMark Plus? • What are the recommendations for improving the implementation of NetMark Plus or similar activities in the future? The evaluation requires a core team of two experts: 1) a Senior Expatriate Team Leader and 2) a Senior Local Expert. The local TMG sub-contractor will provide all needed logistical support. Additional technical team members may be provided by local and international stakeholders, potentially including GoE staff participation. V. Methodology of the Evaluation The evaluation will be conducted by a team of professional and independent external consultants over a period of approximately four weeks (three weeks in-country). The methodology of data collection may include: key informant interviews; focus group discussions; field observations; review of technical and programmatic NetMark Plus reports and other documents. Interviewees may include the following: • USAID/E staff; • AED NetMark Plus staff; • Government of Ethiopia representatives (Federal Ministry of Health, Regional Health Bureau, Zonal [woreda] and District [kebele] Administration); • Beneficiaries; • Other in-country malaria and non-malaria partners / stakeholders. VI. Identification of Information Sources Consultants will be provided the following background documents in preparation of the assignment. • NetMark Plus Contract and all relevant modifications; • NetMark Annual Reports and Work Plans; • NetMark Plus Semi-Annual Reports (SARS); • NetMark Plus Quarterly Reports; • NetMark Plus Technical and Programmatic Reports; • NetMark M&E tools, if relevant; • USAID/E trip reports summarizing past field visits to NetMark Plus sites VII. Tasks to be accomplished 6 Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort for each task. See Attachment 2 for an illustrative evaluation planning calendar. Team reviews background documents, develops draft evaluation work plan and site visit schedule, submits to USAID/Ethiopia prior to departure. 3 days TPM meeting at TMG in Washington D.C. 2 days Travel 1 day Team in briefing with USAID/E, team planning, team stakeholder interviews in Addis 1 day Team planning, team stakeholder interviews in Addis 2 days Full team (w GoE Expert), planning meeting, site visits in Addis 2 day Full team travel to field visits and interviews 7 days Full team synthesis (1 day) Core Team Draft Report (1 day) 2 day Core team conduct debriefings for USAID/E and AED (separately) 1 day Core team completes Draft Report in-country and departs 3 days Travel 1 day Incorporation of USAID/E comments info final report (TL 3; TM 2) 3 days Total LOE – 28 days of LOE for Team Leader and 23 days for Team Member, including travel days. A six-day work week is assumed. VIII. Team Composition and Participation USAID/E seeks two consultants – a Team Leader with experience in social marketing and ITNs, and a Senior Local Team Member with experience in malaria, social marketing, and ITNs. The Team Leader will be a senior expert with experience in social marketing and ITNs. Ideally, the Team Leader will have extensive program implementation and evaluation experience. S/he will agree to fulfill his/her responsibilities in approximately four weeks, spending up to three weeks in-country, and will play a central role in guiding the evaluation process. The consultant may hold a conference call with core team members and USAID/E representatives before and after the visit to Ethiopia, if needed. The consultants will produce a draft followed by a final report for USAID/E. The Team Leader will: • Discuss the team Work Plan for the assignment with USAID/E and finalize the work plan based on USAID/E comments; • Define assignment roles, responsibilities, and tasks for team members; • Oversee logistics arrangements in the field; • Participate in a TMG Team Planning Meeting (TPM) should it be required; 7 • Lead the preparation of and coordinate team member input, submitting, revising and finalizing the report; • Lead team meetings; • oordinate and support the team on tasks and ensure that team works effectively. The Second Team Member will be a Senior Local Team Member with expertise related to malaria, social marketing and ITNs. The Local Team Member should have extensive experience in evaluating community-based vector-borne disease projects/programs in Africa. S/He will be responsible for designing appropriate tools for data collection and analysis. Other Team Participants: This evaluation may include up to 2 GoE experts from the Regional Health Bureaus (RHBs) and/or Federal Ministry of Health (FMOH). USAID/E staff may also join the evaluation team during the site visits. NetMark Plus may accompany the team on site visits as appropriate, but will not be present during interviews with stakeholders or beneficiaries. Evaluation Logistics: Evaluation logistics will be provided by the local sub-contractor hired by TMG with support staff who are fluent in Amharic, with a demonstrated ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision; ability to work collaboratively with a range of professional counterparts. The local sub-contractor will be responsible for logistics, coordination and administrative support. TMG’s local sub-contractor staff will assist the Team in facilitating meetings, coordinating logistics and organizing site visits. As needed, the local sub-contractor will collect and disseminate background documentation to the evaluation team. TMG will be responsible to manage and direct the efforts of local sub-contractor. Evaluators Selection Criteria for Team (Maximum 100 %) distributed as follows: Team Leader 1. Education: (25 %) An advanced degree (Master’s and above) in pertinent areas of public health, epidemiology, social / behavioral sciences, economics. 2. Work Experience: (35 %) Minimum of 6-10 years of progressively responsible experience in the design, implementation and evaluation of operational disease control programs, including social marketing approaches for disease prevention and control and ITNs. 3. Skills and Abilities: (40 %) Demonstration of strong analytical, managerial and writing skills. Exceptional leadership in coordinating, assigning the team with appropriate responsibilities, communication, and interpersonal skills are absolutely critical. In addition, the Team Leader must be able to interact effectively with a broad range of internal and external partners, including USAID/E, international organizations (e.g. UNICEF, WHO), host country government officials, and NGO counterparts. Must be fluent in English and have proven ability to communicate clearly, concisely and effectively both orally and in writing. Senior Local Team Member 8 1. Education: (25 %) An advanced degree (Master’s and above) in the specific areas of public health, epidemiology, entomology, social / behavioral sciences, economics. 2. Work Experience: (35 %) Minimum of 6-10 years of progressively responsible experience in the design, implementation and/or evaluation of operational vector-borne disease programs, including ITNs for the prevention and control of malaria.. 3. Skills and Abilities: (40 %) Track record for strong analytical, managerial and writing skills. Is expected to write portions of the evaluation report, assist in the development, pre￾test, and field implementation of both qualitative and quantitative data collection instruments. IX. Schedules and Logistics This is a three-week evaluation with a desired start date in-country of September 15, 2008. The TMG Sub-contractor, in collaboration with NetMark Plus, will arrange all stakeholder/partner meetings and site visits. Meeting space will be provided at USAID/E, but the agency cannot provide access to telephone, fax and e-mail. All associated travel and per diem costs will be covered under the TMG contract with USAID/E. X. Period of Performance Work is to be carried out over a period of approximately four weeks, beginning on or about (o/a) September 15, 2008 and concluding o/a October 10, 2008 (not including approximately four weeks time for USAID/E (up to ten days) comment and completion of final editing of the Draft Evaluation Report by TMG (3 weeks). XI. Financial Plan A budget agreement between the USAID/E and TMG will be reached and USAID/E will approve the evaluation activity by TMG under the USAID/E Evaluation Program. XII. Deliverables Prior to arrival (preferably two weeks): Team leader will develop a Draft Work Plan with evaluation methodology and field visit and interview schedule in consultation with the USAID/E AM, USAID/E Evaluation Coordinator, TMG, and AED NetMark Plus. The Draft Work Plan will clearly present roles and responsibilities, a planned interview schedule, and an analysis plan of who will be responsible for writing various sections of the report. Three days after Team arrival: Team meeting and in-briefing with USAID/E. USAID/E HAPN technical staff to review and comment on evaluation methods. The Draft Work Plan will be presented and discussed with USAID/E staff to ensure the assessment is on track and can be met on time. After agreement, the draft becomes a Final Work Plan. Prior to departure: Team makes presentation to USAID/E HAPN staff, a separate presentation to NetMark Plus partners, and Team Leader submits a draft report in the format specified by the USAID/E Evaluation Coordinator (See separate MS Word file for TMG Evaluation Report Guidelines) to USAID/E AM - two hard copies and one electronic copy on CD-ROM or flash drive. After departure: Team leader submits final draft report content to USAID/E within one week of receiving comments from USAID/E. The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, Conclusions, and Recommendations in English in a format specified by the USAID/E Evaluation Coordinator in consultation with TMG. 9 Upon final approval of the content by USAID/E, TMG will have the report edited and formatted within three weeks. The final report will be submitted electronically to USAID/E AM. TMG will make the results of its evaluations public on the Development Experience Clearinghouse and on its project web site unless there is a compelling reason (such as procurement sensitivities) to keep the document internal. Therefore, TMG will request USAID/E confirmation that it will be acceptable to make this document publicly available. If there are certain restrictions regarding specific parts of the report that should be removed from a public version due to procurement-sensitive information, TMG will produce a second version suitable for public availability. 10 Appendix B: Agenda / Interviews Date Organization/Location Name Title/ Status 22-Sep-08 USAID/E Kassahun Deneke Strategic Information Advisor USAID/E Sofia Brewer CTO HAPN Evaluations Coordinator USAID/E Karen Towers Former CTO HAPN Evaluations Coordinator USAID/E Richard Reithinger NetMark Plus Activity Manager NM+ Ethiopia Shoa Girma NetMark Plus Country Representative NM+ Ethiopia Mimi Merkuria Sales and Marketing Advisor NM+ Ethiopia Lemlem Bezabech Finance and Administration Officer NM+ Ethiopia Eyob Woldegebriel Targeted Subsidies Coordinator B-Nyse Bayou Belainah General Manager 23-Sep-08 NM+ Ethiopia Shoa Girma Country Manager Syngenta Seifu Rikiti Country Representative PSI Henock Gezahega Marketing Director 24-Sep-08 Adama Ms. Birtukan Gebramikhel ANC / Health Centre Adama Ms.Esete Tekle FGA of Ethiopia Adama Dr. Molla Mekonen MCH/Hospital Adama suburb Ms. Fanos Kebede farmer Adama suburb Ms. Shugute Dinku farmer 25-Sep-08 Zewai suburb Ms. Chege Bariso farmer Zewai suburb Ms. Alamitu Bekele fruit growers Zewai city Ms. Emebet Tafese Miharet Clinic / Pharmacy assistant Zewai city Mr. Haile Gebre Oromia Regional Health Bureau Zewai city Mr. Sisay Teferi Batu Health Centre Zewai city Ms. Ester Adine wood seller Zewai city Mr. Emanuel N/A Drug store manager 26-Sep-08 Traveling Eyob Woldegebriel Field Officer Dukem City Ms. Mulunesh Geleto ANC / Health Centre Dukem City Mr. Werkeshet Temetime Shoa Textile Shop 27-Sep-08 USAID/E Brad Corner USAID HAPN 29-Sep-08 Bahir Dar City Mr. Derege Tamene Regional Health Office Bahir Dar City Mr. Beza Hirot Drug store manager Bahir Dar City Ms. Alemtsehay Adam Gabriel Pharmacy Bahir Dar suburb Mr. Chekene Kasa Head Farmers' Association Bahir Dar suburb Ms. Tesfaye Adane student mother Bahir Dar City Ms. Lenssa Abdissa Universal Drug Store / Pharmacy assistant 30-Sep-08 Bure Town Ms. Guday Neberu (urban poor) Bure Town Ms. Abeba Alebachew shopkeeper Bure Town Mr. Muluken Bekele Rural drug vendor Bure Town Mr. Solomon Nigatu Bitanya Pharmacy / drug vendor 01-Oct-09 Malaria Consortium Dr. Agonafir Tekalegne CAME Coordinator 11 02-Oct-09 NM+ Ethiopia Mimi Merkuria Marketing Officer Petram Mr. FMOHamed Bedri Commercial Director Petram Mr. Hassen Abdulrahman General Director UNICEF Rory Nefdt Malaria & Emergency Health AED-South Africa Kevin Peddie Regional AED Coordinator FMOH Dr. Daddi Jimma Head Malaria and Vector-borne Diseases 03-Oct-09 BASF Volker Sthamer Representative for East-Africa DACA Dr. Solomon Tibebu Head Drug Evaluation and Registration East Asian Industries Mr. B.S. Shetty Managing Partner 05-Oct-08 NM+ Ethiopia Shoa Girma NetMark Plus Country Representative 07-Oct-08 NM+ Ethiopia Shoa Girma NetMark Plus Country Representative 14-Oct-08 USAID/E Richard Reithinger NetMark Plus Activity Manager 12 Appendix C: NetMark Plus Ethiopia Timeline 2004-2008 13 NETMARK PLUS ETHIOPIA TIMELINE 2004‐2008 2003 NetMark was prepared in Ethiopia out of the AED‐South Afr Sep 2004 VF gives PSI  exclusive agreement for 3 years to supply PermaNets to the country Sep 2004 Removal of 15% VAT on treated nets Oct 2004 Shoa Girma hired as Country Coo Oct 2004 Yodit Tekle hired as Administration/Financ Nov 2004 Berhanu Abebe hired as TSP Coo Nov 2004 Launch of NetMark Ethiopia operations Nov 2004 Launch of EAG's Selam Enkulf Jan 2005 GoE began mass free net distribution in early 2005: 100% net coverage projected for late 2008. Jan 2005 Launch of  generic radio campaign to support Petram's Selam Enkilf . Launch of EAG's  branded rad March 2005 Results of the NM household survey of Sept'04. April 2005 Launch (with Amhara Regional Health Bureau +EAG) of an ITN targeted subsidy program for 21 A April 2005 FMOH discussions with RBM/NMCST partners on how better to coordinate and segment free net dist May 2005 4 private companies + 3 civil servants of Ethio‐Japan Textile & FMOH, go on a local production TA July 2005 EAG wins a tender to supply ICRC with 50,000 Olysets Sep 2005 Launch of Petram's ITN brand, WobaLba (Siam Dutch) Oct 2005 Launch of NMs “umbrella” media campaign (radio, TV, road shows, PR activities) Nov 2005 Added  18 more Amhara sites to the ITN targeted subsidy program March 2006 DACA (formed 2 years ago) requests BAYER to re‐register their product April 2006 Tedla G/Medhin hired as a TSP Co April 2006 Launch of NM in 5 major towns of Oromia  by adding 19 TSP sites there May 2006 Tadele Shiferaw hired as June 2006 Start of problems with DACA refusing to allow EAG to bring in ITN (except for one container voucher Sep 2006 Officials attend seminar on removal of reduction of taxes and tariffs on ITNs. Sep 2006 Petram joins the TS program. Subsidy increased from 30Birr to 40Birr Aug 2006 Tinfelesh Zeleke hired as an office at Aug 2006 Zelalem Mekuria hired as a Marketin Oct 2006 Discussion with National Oil Company (NOC) to distribute LLINs Oct 2006 Send FMOH and Ethio‐Japan Production to Kenya for Technology Transfer Conference Oct 2006 Large quantities of free LLINs saturate commercial markets in urban areas: partners see low sales fig Nov 2006 PSI has downed its net price to 20 bir Nov 2006 Addition of 7 more Amhara and 16 more Oromiya TSP sites Nov 2006 Bereket Menna hired as TSP Coo Dec 2006 Adami Tulu and Ethio‐Japan finalise feasibility study and commitment letter to produce local LLIN 2007 NM PEPFAR funded communications campaigns to increase ANC use at PMTCT facilities 2007 Sales continue to plummet, both of TSP and of commercial nets Feb 2007 Meeting USAID‐E on free/subsidized net situation: NM to strengthen TSP and boost ITN awarene 2007 Petram  agrees to represent Tana as a distributor but never obtained the right to import its DAWA P April 2007 BASF receives a transitional DACA certificate allowing unlimited imports (within a limited time windo April 2007 Tedla G/Medhin quits as a TSP Consul 2007 EAG makes agreement to purchase PermaNet® from PSI; EaG completely refused by DACA 2007 Adami Tulu secures funding from government to produce nets 2007 Access to LLIN products is becoming increasingly difficult 2007 Bayer has signed an MOU with Addis Ababa University to conduct efficacy tests on KO Tab and KO Ta 2007 Letter to EAG officially communicates NM will no longer match salaries of sales personnel 14 Appendix D. POLL RESULTS DISCUSSION Table 5: USAID and NM+ responses on program management and outcomes (a) Rating on a scale 1-10 on NetMark's performance USAID RS CM MO FO How well (0 to 9) were field conditions known at the time of planning NM+'s interventions? 6.0 9.0 8.4 8.0 8.2 How effectively were NM+ interventions planned? 5.0 9.0 8.6 8.0 8.0 How effectively were NM+ interventions implemented? 3.8 6.8 8.2 6.4 8.0 What success rate (+outcomes for beneficiaries) would you give to NM+'s interventions? 3.2 5.6 8.0 7.2 6.4 [Regional Supervisor (RS ) - Country Manager – Marketing Officer – Field Officer] Interpretation of poll results In table 6, overall USAID rating are lower than the NM+ Regional Supervisor (RS). The RS generally rated NM+ in Ethiopia higher than did the NM+ staff themselves. The gap in scores between USAID, RS, and NM+ is smallest on knowledge of field conditions at the time of planning. This gap widens somewhat on planning effectiveness, widens further on opinions on execution effectiveness. The difference in ratings is most pronounced on outcome success, where USAID score (3.2) is less than half of NM+ country manager and marketing officer. Table 6: USAID and NM+ responses on program management and outcomes (b) Rating on a scale 0-9 of factors affecting management and outcomes well implemented helps beneficiaries USAID NM+ RS USAID NM+ RS Supply / access 3.0 7.7 5.0 2.0 7.3 3.0 TSP 4.0 8.3 8.0 3.0 7.7 7.0 IEC/BCC 4.0 8.7 9.0 4.0 8.3 8.0 T&T 5.0 6.3 5.0 5.0 6.3 3.0 Local production 3.0 6.7 7.0 2.0 6.3 7.0 Interpretation of poll results Table 7 shows first the ratings for the four interventions being “well implemented”. NM+ & the RS rate their IEC/BCC as best implemented. USAID rates their T&T intervention the highest, while NM+ & the RS rate T&T the lowest of its four interventions. USAID rates both the supply/access and local production interventions low compared with the other interventions, while NM+ and the RS ratings vary with local production rating higher than supply/access. Poll results of key informants show that: for the category of “helps beneficiaries”, T&T is rated as the lowest on benefit of the four interventions, according to NM+ & the RS, but is rated the highest of all interventions by USAID. IEC/BCC scores are similar : IEC/BCC is rated as the most successful of the four interventions by NM+ & the RS yet the score given by USAID is considerably lower . On the score for TSP, NM+ and the RS’s rating reflected more than twice the score given by USAID. The local production issue is rated by both NM+ 15 and USAID as the least beneficial of the interventions, though the RS rates it among the more beneficial. Lastly, NM+’s score on the value of the supply & access intervention is significantly higher than USAID & the RS scores it. The differences between USAID, NM+'s and the RS’s scores in the poll results presented in tables 6 and 7 tie in with other comments on NM+'s management and outcomes. 16 Appendix E: Opinion Questionnaire Responses Richard USAID Kevin RS/AED Shoa CM/NM Mimi MO/NM Eyob FO/NM Dr.Agonafir MC Rory UNICEF Dr.Daddi FMOH End of Project Evaluation of NetMark Plus in Ethiopia: September 2008 Opinion Questions 1 How much % of health officials would you say think it is a good idea to try and develop a commercial market for long-term net supply? at Federal level % 10 n/a 60 50 n/a 75 n/a 100 at Regional level % 10 n/a 50 n/a n/a 50 n/a >50 at Woreda level % 20 n/a 20 n/a n/a 25 n/a <50 2 How many more years does your agency/institution foresee that bed nets will go on being given out for free? YEA 5 2 5 5 n/a 5 20 20 3 How high (0 to 9) would you rate NetMark’s impact on the National ITN strategy? (0 being very insignificant and 9 having a very strong impact) 3 7 7 5 n/a 9 5 9 4 What in-country actor would you say has most influenced National ITN Policy adjustments since 2004? MCST MCST FMOH UNICEF MoH NM WHO NM NM MC UNICF PSI FMOH TCC PSI UNICEF MACEPA NM WHO USAID MC WB PSI USAID 5 Please give a score (0 to 9) to how much each of following stakeholders has benefited from the NetMark program: (0 being very insignificant benefit and 9 being very strong benefit) Urban families 5 7 7 8 n/a 5 7 9 Rural families 2 5 0 7 n/a 9 2 0 Foreign net suppliers 7 6 7 6 n/a 9 3 5 In-country net distributors 7 6 7 8 n/a 9 6 8 Local net retailers 7 6 8 8 n/a 9 6 9 Health officials 2 7 0 4 n/a 6 4 8 6 Please give a score (0 to 9) to how sustainable each of following NetMark outputs/outcomes will prove to be? (0 being very unsustainable and 9 being very strongly sustainable) INCREASING SUPPLY AND ACCESS OF ITNs 2 3 7 9 n/a 6 4 8 Mother and Child TARGETED SUBSIDY PROGRAM 4 5 0 9 n/a 5 0 0 ENHANCED COMMUNICATION CAMPAIGN 4 8 9 9 n/a 5 5 8 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 6 7 3 6 n/a 5 4 4 CATALYZING LOCAL PRODUCTION OF ITN PRODUCTS 4 8 9 6 n/a 5 7 9 7 How high would you rate the consistency of general malaria related messages over the last 5 years? (0 being very inconsistent and 9 being very strongly consistent) 4 5 8 n/a n/a 4 8 9 8 What national malaria agent was most/ what agent was least implied in this coordination? MOST implied Actor n/a NM FMOH NM n/a n/a n/a HEEC LEAST implied Actor PSI n/a n/a n/a FMOH n/a n/a n/a n/a 9 Please give a score (0 to 9) to how hard it would be to replicate each of following NetMark interventions elsewhere. (0 being not replicable at all and 9 being very strongly replicable) INCREASING SUPPLY AND ACCESS OF ITNS 5 9 9 8 n/a 4 7 9 Mother and Child TARGETED SUBSIDY PROGRAM 5 9 9 8 n/a 6 8 8 ENHANCED COMMUNICATION CAMPAIGN 5 9 9 9 n/a 7 8 9 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 5 9 9 8 n/a 7 7 9 CATALYZING LOCAL PRODUCTION OF ITNS PRODUCTS 5 9 9 6 n/a 8 5 4 10 Please give a score (0 to 9) to how effectively the program built the capacity of communities to improve their malaria prevention and control behaviour. (0 being very ineffective and 9 being very strongly effective) 3 n/a 8 7 n/a 6 3 8 Richard USAID Kevin RS/AED Shoa CM/NM Mimi MO/NM Eyob FO/NM Dr.Agonafir MC Rory UNICEF Dr.Daddi FMOH NETMARK STAFF 11 Please give a score (0 to 9) to how effectively each of following NM interventions were planned? INCREASING SUPPLY AND ACCESS OF ITNS 4 9988 TARGETED SUBSIDY PROGRAM 5 9999 ENHANCED COMMUNICATION CAMPAIGN 5 9988 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 6 9788 CATALYZING LOCAL PRODUCTION OF ITNS PRODUCTS 5 9977 12 Please give a score (0 to 9) to how well the plans for each of following interventions proved to be, when trying to execute them? INCREASING SUPPLY AND ACCESS OF ITNS 3 4967 TARGETED SUBSIDY PROGRAM 4 8977 ENHANCED COMMUNICATION CAMPAIGN 4 9978 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 5 4757 CATALYZING LOCAL PRODUCTION OF ITNS PRODUCTS 3 7836 13 How good (0 to 9) was the knowledge of field conditions at the time of planning each of following interventions? INCREASING SUPPLY AND ACCESS OF ITNS 6 9889 TARGETED SUBSIDY PROGRAM 6 9989 ENHANCED COMMUNICATION CAMPAIGN 6 9998 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 6 9778 CATALYZING LOCAL PRODUCTION OF ITNS PRODUCTS 6 9987 14 Please give a score (0 to 9) to how effectively each of following NM interventions were implemented? INCREASING SUPPLY AND ACCESS OF ITNS 3 5878 TARGETED SUBSIDY PROGRAM 4 8979 ENHANCED COMMUNICATION CAMPAIGN 4 9989 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 5 5757 CATALYZING LOCAL PRODUCTION OF ITNS PRODUCTS 3 7857 15 What overall success rate (positive outomes for beneficiaries) (0 to 9) would you allocate to each of following interventions? INCREASING SUPPLY AND ACCESS OF ITNS 2 3886 TARGETED SUBSIDY PROGRAM 3 7986 ENHANCED COMMUNICATION CAMPAIGN 4 8988 CREATING A CONDUCIVE ENVIRONMENT FOR SUPPLIERS AND DISTRIBUTORS 5 3676 17 18 Annex F . Resources Reviewed AED Carol A. Baume 2008 ETHIOPIA BEDNET UTILIZATION STUDY: : WHY SOME NETS OWNED ARE NOT USED AED 2008 Revised Description for NETMARK PLUS NetMark Ethiopia 2008 Evaluation Planning Meeting Notes NetMark Ethiopia 2008 Evaluation Planning Meeting Notes NetMark 2008 NETMARK PROJECT – OVERVIEW SUMMARY: October 1, 2007 to March 31, 2008 USAID Development Alternat 2008 External End‐of‐Project Evaluation: Urban Agriculture Program for HIV/AIDS Affected Women and Children (Ethiopia) USAID Anjum Asim Shahid Ra 2008 Mid‐Term Assessment of Social Marketing Program (2003 – 2008) NetMark Ethiopia 2007 October 2006 ‐ September 2007 Annual Report USAID Paul Smithson et al. 2007 NETMARK EVALUATION: Reviewing Achievements to date of the NetMark Programme and Future Directions AED David McGuire 2007 MEMORANDUM: Response to NetMark Evaluation NetMark Plus 2007 Semi‐Annual Report: October 1, 2006 to March 31, 2007 NetMark 2007 Work Plan Ethiopia Final 06‐07 USAID Paul Smithson et al. 2007 NetMark Evaluation 2007 Annexes NetMark 2007 NETMARK PROJECT – OVERVIEW SUMMARY: April 1, 2007 to September 30, 2007 NetMark 2007 NETMARK PROJECT – OVERVIEW SUMMARY: October 1, 2006 to March 31, 2007 SNV Pamela Opiyo et al. 2007 An Exploratory Study of Community Factors Relevant for Participatory Malaria Control on Rusinga Island, Western Kenya NetMark Ethiopia 2006 October 2005 to September 2006 Annual Report NetMark Ethiopia 2006 2006/2007 Work Plan AED 2005 NetMark: A Case Study In Sustainable Malaria Prevention Through Partnership with Business NetMark Ethiopia 2005 October 2004 to September 2005 Annual Report AED 2005 NetMark: A Case Study In Sustainable Malaria Prevention Through Partnership with Business id21 Joy Hum. Dev. Centre 2005 Hitting malaria where it hurts ‐ Household and community responses in Africa id21 Caroline Jones 2005 Buying the best ‐ Household malaria prevention in The Gambia id21 Collins Stephen et al. 2005 To buy or not to buy ‐ Communities and bed nets in rural Ghana id21 Rebecca Marsland 2005 Mosquito nets challenge tradition in Tanzania DFID John Meadley et al. 2003 REVIEW OF DFID APPROACH TO SOCIAL MARKETING id21 B Jacobs et al 2003 A marketing breakthrough: social marketing sexual health treatment kits in Uganda Trop. Med.Intl Health J. Webster, et al., 2003 A health facility based case‐control study of effectiveness of insecticide treated nets Lancet Teklehaimanot, R 2002 Will the Global Fund help roll back malaria in Africa Trop. Med.Intl Health C.A. Maxwell et al 2002 Effect of community‐wide use of insecticide‐treated nets for 3‐4 years on malarial morbidity in Tanzania’ id21 C. Curtis 2001 ‘Should the poor be made to pay for insecticide treated nets?’ Malaria Consortium K. Hanson et al. 2000 Social marketing of insecticide treated mosquito nets, Tanzania