January 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI). The authors are Veronique Praz, Alice Morton, and Ian Matondo. EVALUATION REPORT End-of-project Evaluation of the PSI Social Marketing Project in Madagascar Caption: A community-based distributor presents her stock of socially marketed products. This page is intentionally left blank. END-OF-PROJECT EVALUATION OF THE PSI SOCIAL MARKETING PROJECT IN MADAGASCAR January 31, 2013 Contract Number AID-RAN-I-00-09-00016, Task Order Number AID-687-TO-12-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report TABLE OF CONTENTS ACKNOWLEDGEMENTS iv GLOSSARY v ACRONYMS vi EXECUTIVE SUMMARY i 1. INTRODUCTION 1 2. EVALUATION DESIGN AND METHODOLOGY 5 3. QUESTION ONE: PERFORMANCE IN KEY HEALTH AREAS 11 CONCLUSIONS ON DIARRHEA PREVENTION (MCH) 13 RECOMMENDATIONS FOR DIARRHEA PREVENTION (MCH) 14 FINDINGS: MCH—DIARRHEA TREATMENT 14 CONCLUSIONS ON DIARRHEA TREATMENT (MCH) 15 RECOMMENDATIONS FOR DIARRHEA TREATMENT (MCH) 15 FINDINGS: MCH—PNEUMONIA TREATMENT 16 CONCLUSIONS ON PNEUMONIA TREATMENT (MCH) 17 RECOMMENDATIONS FOR PNEUMONIA TREATMENT (MCH) 17 FINDINGS: MCH—INTEGRATED MANAGEMENT OF CHILDHOOD ILLNESSES (IMCI) 17 CONCLUSIONS ON IMCI (MCH) 18 RECOMMENDATIONS FOR IMCI (MCH) 18 FINDINGS: MCH—UTILIZATION OF MCH RESEARCH RESULTS 18 CONCLUSIONS ON UTILIZATION OF MCH RESEARCH RESULTS 18 FINDINGS: FAMILY PLANNING AND REPRODUCTIVE HEALTH 19 CONCLUSIONS ON FP/RH 20 RECOMMENDATIONS FOR FP/RH 20 MALARIA PREVENTION AND TREATMENT FINDINGS 20 CONCLUSIONS ON MALARIA PREVENTION AND TREATMENT 21 RECOMMENDATIONS ON MALARIA PREVENTION AND TREATMENT 21 FINDINGS: HIV/STI PREVENTION 21 CONCLUSIONS ON HIV/STI PREVENTION 22 RECOMMENDATIONS FOR HIV/STI PREVENTION 23 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 4. QUESTION TWO: IMPLEMENTATION OF THE TOP RÉSEAU FRANCHISE 23 FINDINGS ON ACCESS AND USE OF SERVICE BY THOSE MOST IN NEED 25 CONCLUSIONS ON ACCESS AND USE OF SERVICE BY THOSE MOST IN NEED 26 RECOMMENDATIONS ON ACCESS AND USE OF SERVICE BY THOSE MOST IN NEED 26 FINDINGS ON THE SUSTAINABILITY OF TOP RÉSEAU WITHOUT PSI/M SUPPORT 27 CONCLUSIONS ON THE SUSTAINABILITY OF TOP RÉSEAU 27 RECOMMENDATIONS FOR THE SUSTAINABILITY OF TOP RÉSEAU 28 5. QUESTION THREE: PERFORMANCE OF SUPPLY CHAIN & DISTRIBUTION SYSTEMS 28 FINDINGS: ACHIEVEMENTS AND WEAKNESSES IN MAINTAINING THE SUPPLY CHAIN 29 CONCLUSIONS ON THE ACHIEVEMENTS AND WEAKNESSES IN MAINTAINING THE SUPPLY CHAIN 31 RECOMMENDATIONS ON THE ACHIEVEMENTS AND WEAKNESSES IN MAINTAINING THE SUPPLY CHAIN 31 FINDINGS: PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THROUGH THE PHARMACEUTICAL SUPPLY CHAIN 32 CONCLUSIONS ON PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THROUGH THE PHARMACEUTICAL SUPPLY CHAIN 33 RECOMMENDATIONS ON PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THROUGH THE PHARMACEUTICAL SUPPLY CHAIN 33 FINDINGS: PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THOUGH ITS DISTRIBUTION SYSTEM TO RURAL AREAS 34 FINDINGS: SUCCESSES AND OBSTACLES IN PSI/M’S SUPPLY CHAIN MODEL FOR COMMUNITY-BASED DISTRIBUTION 36 SUMMARY OF STOCK-OUT ISSUES IN PSI/M’S SUPPLY CHAIN SYSTEM 37 6. QUESTION FOUR: OTHER CROSS-CUTTING ISSUES 39 FINDINGS: CAPACITY BUILDING EFFORTS WITH LOCAL ORGANIZATIONS 39 CONCLUSIONS ON CAPACITY BUILDING EFFORTS WITH LOCAL ORGANIZATIONS 41 FINDINGS: HARMONIZATION WITH NATIONAL PROGRAMS 41 CONCLUSIONS ON THE HARMONIZATION WITH NATIONAL PROGRAMS 41 FINDINGS: GOM PERCEPTIONS OF PSI/M’S OUTCOMES AND FUTURE NEEDS 41 CONCLUSIONS ON GOM PERCEPTIONS OF PSI/M’S OUTCOMES AND FUTURE NEEDS 42 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report FINDINGS: LEVERAGING PARTNERSHIPS WITH LOCAL AND INGOS VIS-À-VIS DISTRIBUTION SUPPORT 42 CONCLUSIONS ON LEVERAGING PARTNERSHIPS WITH LOCAL AND INGOS VIS-À-VIS DISTRIBUTION SUPPORT 44 FINDINGS: INFLUENCE OF THE POLITICAL SITUATION ON PSI/M’S PERFORMANCE 44 CONCLUSIONS ON THE INFLUENCE OF THE POLITICAL SITUATION ON PSI/M’S PERFORMANCE 45 ANNEXES 48 ANNEX A: PSI MADAGASCAR SUB-AWARDEES 48 ANNEX B: PSI/M MAPS OF COMMERCIAL COVERAGE AND TOP RÉSEAU CLINICS 51 ANNEX C: TABLE OF RESPONDENT CATEGORIES 53 ANNEX D: EVALUATION SCOPE OF WORK 54 ANNEX E: MAP OF REGIONS AND DISTRICTS VISITED BY THE EVALUATION TEAM 60 ANNEX F: TABLE OF DATA COLLECTION RESULTS 61 ANNEX G: LIST OF REFERENCE DOCUMENTS 68 ANNEX H: DATA COLLECTION INSTRUMENTS 71 ANNEX I: PSI/M TRAINING SCHEDULE FY 2008 – FY 2012 92 ANNEX J: SUPPLY CHAIN GRAPHIC 101 ANNEX K: AGGREGATED RESULTS OF SELECTED LSAT VARIABLES 102 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report ACKNOWLEDGEMENTS The Evaluation Team would like to thank the personnel from USAID whose views helped form a more complete understanding of the context in which PSI/Madagascar (PSI/M) was designed and implemented. Special thanks are extended to: Mr. Jean Claude Randrianarisoa, Mr. Hery Zo Ramiandrisoa, Mr. Jacky Ralaiarivony, Ms. Lucie Raharimalala, Ms. Fabienne Ranjalahy, Ms. Cathy J. Bowes, Mr. David J. Isaak, Ms. Jackie Ramanitrera, Mr. Robert Kolesar, Mr. Benjamin Rakotondranisa, Ms. Vololontsoa Raharimalala, and Dr. Jocelyne Andriamiadana. Special thanks are given to Mr. Rudolf Thomas, USAID Mission Director and Acting Chief of Mission, United States Embassy, Madagascar, who kindly attended and spoke at the Evaluation Out briefing at the Mission, in Antananarivo on August 6, 2012. The Team is also very grateful to the staff of Population Services International (PSI), including Mr. Brian McKenna, Deputy Regional Director, and Daniella Fanarof, Senior Program Manager for Southern Africa and at PSI/Washington, as well as the very dynamic team at PSI/M Headquarters, led by Mr. Chuck Szymanski, Country Representative, and Monique Weiss and Henri Rabesahala, Deputy Country Directors, and including many others such as Ms. Ietje Reerink, Ms. Marie-Alix Valensi, Ms. Iarimalanto Rabary, Mr. Davy Robson, Mr. Maminirina Rakotoarisoa, Mr. Kenny Dakadisy, Dr. Mbolatiana Razafimahefa, Ms. Velonirina Andrianifahanana, Dr. Andry Nirina Rahajarison, Mr. Njakatiana Rajaonarivelo, Mr. Teddy Lala Ramaroson, Mr. Solofo Andriaherinosy, Mr. Kenny Dakadisy, Dr. Eric Ratsaravolana, and Dr. Phillipe Botralahy. We appreciate the valuable and transparent information they provided. Our thanks go also to PSI/M Inter-regional Directors in Antsiranana and Tamatave, as well as PSI/M Top Réseau Regional Coordinators at Toliara, Mahajanga, Antsirabe, Fianarantsoa, Tôlanaro, and Morondava for the valuable information and introductions they provided on PSI/M activities. We present our special thanks to all the program and project beneficiaries of PSI/M’s partner organizations, as well as their representatives and managers, and others whom participated in the evaluation. Finally, the Evaluation Team would also like to its great appreciation to the IBTCI home office Team in Vienna, Virginia, who provided technical and managerial support to the Team during field work. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report GLOSSARY DELTA A marketing planning process developed by Population Services International (PSI) FARMAD Local pharmaceutical manufacturer and retailer Fokontany The smallest administrative subdivision in Madagascar ITEM A local non-governmental organization (NGO) Malagasy Of or from Madagascar MAHEFA Stands for Malagasy Heniky ny Fahasalamana, a project implemented by John Snow, Inc. (JSI) in Madagascar NIPHAR A local pharmaceutical manufacturer SALAMA A government agency which is the Central Medical Store of Madagascar SantéNet2 A project implemented by Research Triangle Institute, International (RTI) in Madagascar SIGMA A private manufacturer of Sur’ Eau SISAL A local NGO Supply Points For the purposes of this report, Supply Points are places in the supply chain at the community level from which to distribute social marketed health products directly to community health workers (CHWs) Top Réseau A franchised network of private, clinic-based providers established by PSI/Madagascar in 2000 with support from the Bill and Melinda Gates Foundation ZETRA A local NGO USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report ACRONYMS ACT Artemisinin-based Combination Therapy AFSPS Agence Française de Santé de Produits Sanitaires AGMED Agence du Médicament de Madagascar AIDS Acquired Immunodeficiency Disease Syndrome ARI Acute Respiratory Infection BCC Behavior Change Communication CDC Community Development Committees CHW Community Health Worker CROM Conseil Régional de l’Ordre des Médecins CSB Centre de Santé de Base CSW Commercial Sex Worker CU5 Children Under Five Years Old CYP Couple Years of Protection DAMM Direction de l’Agence des Médicaments de Madagascar DRS Directeur Régional de la Santé DTK Diarrhea Treatment Kit FBO Faith Based Organization FGD Focus Group Discussion FISA Fianakaviana Sambatra FP Family Planning GFATM Global Fund to Fight HIV/AIDS, Tuberculosis, and Malaria GOLD Groupements et Organisations Locales de Developpement GOM Government of Madagascar HIV Human Immunodeficiency Virus IUD Intra Uterine Device JSI John Snow, Inc. IBTCI International Business & Technical Consultants, Inc. IMCI Integrated Management of Childhood Illnesses INGO International Non-Governmental Organization IPC Inter-Personal Communication IUD Intrauterine Device MAHEFA Malagasy Heniky ny Fahasalamana (project) KII Key Informant Interview LLIN Long Lasting Insecticide-treated Net LSAT Logistics System Assessment Tool MCH Maternal and Child Health MOH Ministry of Health MOU Memorandum of Understanding MSM Men Who Have Sex with Men MVU Mobile Video Unit NGO Non-Governmental Organization NMCP National Malaria Control Program ODDIT Organization for Development in the Diocese of Toamasina OMAPI Office Malgache de la Propriété Industrielle ONM Ordre National des Medecins ORS Oral Rehydration Salt OSIEM Organisation Sanitaire Inter-Entreprises de Madagascar USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report PAC Post-abortion care PERForM Performance Framework for Social Marketing PMI President’s Malaria Initiative PMP Performance Management Plan PPH Postpartum hemorrhage PPT Pre-packaged Therapy PSI/M Population Services International/Madagascar RCC Rolling Continuation Channel RDT Rapid Diagnostic Test RH Reproductive Health RTI Research Triangle Institute, International SALFA Health Agency for the Lutheran Mission in Madagascar SOW Scope of Work STI Sexually Transmitted Infection TRaC Tracking Results Continuously USAID United States Agency for International Development USD United States Dollar USG United States Government VCT Voluntary Counseling and Testing WHO World Health Organization USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report i EXECUTIVE SUMMARY International Business & Technical Consultants, Inc. (IBTCI) is pleased to present the end-of￾project evaluation report of the United States Agency for International Development (USAID) funded Social Marketing Project for Child, Maternal, and Reproductive Health Products and Services in Madagascar (hereafter referred as PSI/M). This project was implemented by Population Services International in Madagascar (PSI/M) under USAID Cooperative Agreement Number 687- A-00-08-00032-00 during July 14, 2008 to September 30, 2012. The data collection phase for this evaluation was conducted between May – August 2012. PURPOSE OF THE EVALUATION The purpose of this evaluation was two-fold: 1) Collect qualitative and quantitative information about PSI/M’s implementation, progress and challenges, with particular emphasis on whether PSI/M successfully achieved the expected results of the project. Additionally, share any programmatic, management and/or financial obstacles that affected project implementation to inform future projects. 2) Inform USAID/Madagascar and key stakeholders on recommended strategies, structures or actions for future projects involving supply chain and distribution systems to increase efficiency and impacts, e.g., viability of community supply chain system; priority areas for local capacity building; gaps in management structure and oversight; consumer focus in product distribution and price determination, etc. PROGRAM BACKGROUND The goal of the PSI/M project was to improve the health status of the Malagasy people, and especially women and children. The purpose of the project was to increase the use of effective health products, services and behaviors in the areas of family planning and reproductive health (FP/RH), as well as the prevention and treatment of sexually transmitted infections (STIs), Human Immunodeficiency Virus (HIV), malaria, diarrheal disease, and pneumonia. Project strategies included engagement with the private sector and evidence-based social marketing. From 2008 to 2012, PSI/M supported the promotion, communication, research, and training for health services, and the social marketing of 16 subsidized health products in wholesale commercial, pharmaceutical, and community outlets (the former based at established supply points). As well, a selection of PSI/M products was made available for Top Réseau, which is a franchised network of private, clinic￾based providers established by PSI/M in 2000 with support from the Bill and Melinda Gates Foundation.1 Per USAID’s request, PSI/M did a major overhaul of its project design for fiscal years (FY) 2011 and 2012. This was in response to the difficult political and socio-economic environment following the 2009 political crisis.2 To meet this request, PSI/M expanded its private sector support activities but the period of performance was decreased by one year. During FYs 2011—2012, PSI/M expanded activities with private sector partners, and strengthened the supply chains at the commercial, pharmaceutical, and community levels in order to absorb and distribute product inventories slated for national level distribution. PSI/M also expanded its portfolio of socially￾marketed products and related services, especially for maternal and child health (MCH) and expanded the range of services offered at the Top Réseau clinics. This evaluation it meant to be viewed in terms of these socio-political and programmatic challenges. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report ii CONCLUSIONS AND RECOMMENDATIONS OVERVIEW In general, the evaluation found PSI/M works nationally, reaching a significant number of end users through the distribution of subsidized products and the provision of integrated health care services through the Top Réseau clinics. Overall, the respondents frequently recognized and had positive perceptions of PSI/M socially-marketed products and services. Health care providers, distributors, retailers, and end users generally recognized and reported positive perceptions of the effectiveness of PSI/M branded and unbranded health products for MCH interventions. Activities to support the branded Top Réseau franchise clinics were considerably expanded since 2008 to encompass a total of 213 franchised clinics including locations in 46 Districts and 18 regions. In order to address gaps in national healthcare services, the mandate of the Top Réseau has expanded to include the whole family. Its services now include the provision of FP/RH and HIV/STI products and services to youth and other vulnerable populations. However, other findings also revealed evidence of frequent stock-outs; a lack of understanding of the importance and correct use of certain socially-marketed products. The evaluation’s overarching conclusion is that PSI/M is successful in many respects and has employed innovative methods for promotion, training, research, and distribution of health products. This has not always translated into the correct use of their products despite PSI/M’s continuing efforts to improve compliance. The following is a summary of the conclusions for the evaluation questions and their sub-questions. When relevant, the recommendations are included. Evaluation Question One – How did PSI/M perform in the areas of MCH, FP/RH, malaria, and HIV/STIs? Child Health—Diarrhea Prevention Conclusion: Knowledge transfer about the purpose of Sur’Eau (a PSI/M socially marketed safe water treatment product), when to use it, and how to use it correctly, does not guarantee that targeted audiences will adopt the new practice by purchasing and using this product correctly. The rationale for sporadic, inconsistent, and/or incorrect use of Sur’Eau is uncertain but might be related to unclear messaging and marketing; an unfavorable odor; competition from free sources of purified water; cost of the larger, 150 milliliter (ml) formulation; and/or the level of effort from community health workers (CHWs) to promote the product among other reasons not identified through this evaluation. PSI/M was cognizant of incorrect use and took appropriate steps to ameliorate the situation albeit more work remains in this regard. Recommendation:  A situational analysis should be conducted to determine the barriers to consistent and correct use of Sur’Eau and also to explore the feasibility and market suitability of a water treatment tablet. Interventions in the follow-on project should address these findings and piloted before being brought to scale. Particular attention should be placed on creating streamlined, but comprehensive behavior change communication (BCC) and inter-personal communication (IPC) messages to address: 1) the correct use of safe water treatment products for diarrhea prevention, and 2) to provide increased fluids (with treated water) and feed normally USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report iii when a child has diarrhea. Project monitoring of correct and consistent use of Sur’Eau should be assessed at baseline, mid-term, and again at the end of the next project through studies similar to the PSI/M Tracking Results Continuously (TRaC) surveys. Medium-term/Implementation  Sufficient budget should be allocated to educate consumers about the new Sur’Eau format in rural areas. CHWs will need to review the message on the proper use of Sur’Eau with beneficiaries. Short-term/Work planning  PSI/M should monitor the uptake of the new 40 ml formulation of Sur’Eau. Supervisors at supply points should train and assist the CHWs to record the number of sales of the 40 ml Sur’Eau in order to ascertain if it is successful in comparison to sales of the old larger format. Short-term/Performance Monitoring Plan (PMP) Child Health—Diarrhea Treatment Conclusion: The use of diarrhea treatment kits (DTKs), though rising, has been handicapped by a low product distribution over the life-of-the project compared to anticipated program targets. Four key messages on the correct use of DTKs have not been fully absorbed among the targeted beneficiaries interviewed in this evaluation. These messages include: 1) the need to use treated water, not boiled, to prepare oral rehydration salt (ORS); 2) sick children need additional liquids during their illness; 3) regular feedings should continue while treating for diarrhea; and 4) the full course of zinc tablets should be consumed to ensure a speedy recovery. Recommendations:  BCC campaigns should hone in on the correct sequence of steps for the proper treatment of diarrhea. Future social marketing and BCC activities for diarrhea treatment should examine previous campaigns and messages to address gaps in the four key messages outline above. The situational analysis suggested above to better understand the constraints to correct use of Sur’Eau could also include a component on the correct use of DTKs. Future programs should refine BCC and IPC messages regarding the rationale for administering the complete blister of zinc tablets to treat children with diarrhea. Medium￾term/Implementation  Communication efforts need to continue to further build awareness of the HydraZinc and Viasûr brands. BCC campaigns, IPC, and educational materials should be done in concert to relay a unified and simple message. Medium-term/Implementation Child Health— Integrated Management of Childhood Illnesses (IMCI) and Pneumonia Conclusion: PSI/M was ambitious to design BCC activities and launch new pneumonia treatment products in the last year of the project. Nonetheless, the delay in launching Pneumostop’s tablet formulation will likely prevent PSI/M from reaching its FY 2012 targets. It is not possible to determine the sustainability of this MCH intervention that was just introduced in late FY 2011. The mixed results in sales and product recognition are likely due to the rapid roll out of the pneumonia treatment activities. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report iv By the end of the project, PSI/M’s approach to the integrated management of childhood illnesses (IMCI) had not been refined as evidenced by the mixed findings on caregiver comprehension of the range of MCH messages and how to relate them to one another. This is likely due to the late introduction of IMCI into PSI/M’s work plan. Nonetheless, PSI/M clearly took some positive steps toward building partnerships to relay targeted IMCI messages to key audiences. There is great merit in the approach to integrate PSI/M MCH activities funded by USAID, with those funded by other donors. This integration helps to leverage resources and ensures child survival messages are harmonized with other USAID funded programs. Recommendation:  PSI/M should develop a case study on the launch of Pneumostop to use the experience as a lesson learned so as to mitigate future problems with product introduction. The future program should take into consideration the need to repeat the BCC campaign and social marketing activities to address any problems resulting from the long delay between the product introduction and availability on the market. Medium-term/Implementation  Training curricula for CHWs should encompass a comprehensive approach to identifying and treating childhood illnesses. As one example, future programs should improve IPC training for CHWs that promote continuing breastfeeding, giving the child treated water, and/or normal feedings while a child had diarrhea, malaria, or pneumonia. Short￾term/Design Family Planning and Reproductive Health (FP/RH) Conclusion: The Top Réseau franchise is now positioned as a network of branded clinics where all family members can seek—and find—health services and some products. Top Réseau clinics have been successful at making FP/RH services more accessible. The prevalent gender inequalities experienced by many women present a serious obstacle to securing FP and a healthy RH life. The availability of vouchers could affect women’s decision whether to commit to short or long terms methods and to use the methods properly. As with other preventive treatments, contraception could be perceived as a less necessary expense when faced with limited purchasing power. PSI/M’s approach to promote FP/RH (and MCH) with commercial sex workers (CSWs) was a successful BCC and IPC strategy and served to set the foundation for ongoing discussions. Recommendations:  The future project should balance the needs for vouchers and the design of an effective strategy for long term sustainability. Medium Term/Implementation  This project also should consider including more BCC and IPC messages for men of reproductive age about the benefits of spaced births and using condoms for FP/RH. Short-term/Implementation USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report v Malaria Conclusions: The coordinated role of multiple donors in support of the Government of Madagascar (GOM) National Malaria Control Program (NMCP) is one of the best examples of implementing partners working together in Madagascar on a public health problem. The GOM gleaned the best practices of PSI/M’s trainings on malaria prevention and treatment for children under five (CU5) using the IMCI approach. The MOH’s awareness of this approach helped it to move forward the national malaria program. Recommendation:  The follow-on project should be tasked with developing and branding new malaria products and launching communication campaigns, if they are to continue their work under the Global Fund to Fight HIV/AIDS, Tuberculosis, and Malaria (GFATM). Long-term/Implementation HIV/STIs Conclusions: The Top Réseau franchise is now positioned as a network of branded clinics where all family members can seek—and find—health services and some products. Top Réseau clinics have been successful at making FP/RH services more accessible. The prevalent gender inequalities experienced by many women present a serious obstacle to securing FP and a healthy RH life. The availability of vouchers could affect women’s decision whether to commit to short or long terms methods and to use the methods properly. As with other preventive treatments, contraception could be perceived as a less necessary expense when faced with limited purchasing power. PSI/M’s approach to promote FP/RH (and MCH) with CSWs was a successful BCC and IPC strategy and served to set the foundation for ongoing discussions. Recommendations:  The future project should balance the needs for vouchers and the design of an effective strategy for long term sustainability. Medium Term/Implementation  This project also should consider including more BCC and IPC messages for men of reproductive age about the benefits of spaced births and using condoms for FP/RH. Short-term/Implementation Evaluation Question Two–How did PSI/M perform in implementing and working with Top Réseau clinic franchise? Conclusion: In general, PSI/M collaboration with Top Réseau providers has been fruitful and has facilitated coordination and understanding among the network providers. Those interviewed were satisfied with PSI/M’s support, collaboration, and capacity building, especially in the areas of basic care (FP, STI, MCH). The management capacity of Top Réseau providers is in question. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report vi The Top Réseau network is a complementary solution to the public health care. PSI/M’s promotion campaigns have sensitized providers and increased their knowledge base on service delivery. The medical training delivered to Top Réseau providers facilitated the improvement of quality of services and the uptake of products. Recommendation on Sustainability:  The future project should assess the management capacity of Top Réseau providers and offer technical assistance and trainings based on those findings. This would require an implementing partner which possesses capability statements in human resource capacity building on operations and fiscal management. Medium-term/Implementation Specific examples of management needs include: o Development of business plan(s) for the Top Réseau clinics. o Administrative processes for paying refunds and other financial structures should be identified to streamline administrative processes. The aim would be to increase efficiencies and improve profit margins. o Curricula for Top Réseau clinicians and health service providers should include courses on performance monitoring and reporting, and financial administration.  PSI/M’s existing standard operating procedure manual should continue to be promoted for use by the Top Réseau providers. Continue to encourage good management practices, supporting those providers with a willingness to learn more, especially in the areas of financial controls; human resource development including skills and performance improvement; and administrative systems. Long-term/Design and Implementation  Any future situational analyses should include a focus on the PSI/M reporting system. If needed, the system should be revised to address issues with compliance and quality controls should be built-in to minimize spurious data entry. Oversight in the compliance with reporting should be subsumed within a broader business plan for the franchises. Medium-term/Implementation Evaluation Question Three–How did PSI/M perform in the Supply Chain/distribution systems? Conclusions: As noted in the findings from the previous evaluation questions, the project made significant inroads in promoting access to community-based products. The project accomplished a great deal in expanding community-based distribution, improving access of rural end users to the specific health products, training CHWs in the use of health products, training supply points in stock management, and engaging community stakeholders. It has also improved the distribution system by offering a community-based supply system as a complement to the GOM’s supply chain. The demand-driven PSI/M inventory control system has resolved several problems. For example, instances of over￾stocking which had resulted in products expiring before they are sold and used. Through collaboration with SantéNet2 and other local non-governmental organizations (NGOs), PSI/M ensured the rural end users got access to the products and that they have positively impacted USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report vii on the programs targeting the rural populations. PSI/M’s initiatives that fortified community stakeholder participation will likely ensure the sustainability of the rural supply chain. Yet, the supply chain experienced a number of weaknesses as well. The community-based supply points who self-selected to serve in the role of the CHW, created an operational conflict, which was disruptive to the supply-chain process. This is of particular concern because it diminished the full complement of services designed for the end users (e.g., IPC). Supervisory visits at the community level were too infrequent. Communications lapses between the PSI/M distribution team, community-based supply points and CHWs added to the stock-out problem and contributed to unnecessary interruptions in service provision to the community. Such visits should ostensibly have uncovered the inadequate pharmaceutical storage conditions which threatened the product efficacy. CHWs work part-time and are frequently unavailable which makes their products and services more difficult to access. Recommendations: To improve communications that would support an efficient and streamlined supply-chain, USAID-supported community-based programs should consider subsidizing the purchase of mobile phones and calling cards. The sustainability of any such support should be a paramount consideration. Explore alternative approaches such as providing an incentive (i.e., calling card) for the CHWs who maintain a certain threshold of sales. Other options might include creation of income-generating activities under the auspices of the SantéNet2 project implemented by Research Triangle Institute, International (RTI) and/or the Malagasy Heniky ny Fahasalamana (MAHEFA) project implemented by John Snow Inc. (JSI) under the broader rubric of their work with NGOs. SMS technology should also be used for supply points to help them place their order. Short￾term/Implementation  Future programs should consider the cost-effectiveness and sustainability of offering on-site supervision of supply points on a monthly basis. Short-term until project phase-out/Implementation  Clearly define and communicate the role of the community supply points and CHWs vis-à-vis the supply chain. Short-term/Implementation  Consider increasing the production of promotional materials to support CHWs in the delivery of their services. Evaluation Question Four – Other Crosscutting Questions What were the results of PSI’s capacity building efforts with local organizations? Conclusion: PSI/M’s capacity building was conveyed to partners through various training sessions, supervisory visits, technical/coordination meetings, and technical assistance in the manufacturing of products have resulted in a functional and reliable supply chain that ensured delivery of quality products to end users. Stock-outs were kept to a minimum and when they did occur, they were result of regulatory delays, manufacturing problems and delays with internationally procured products delivery (as were described in more detail in evaluation question three).3 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report viii To what extent has PSI/M been able to align and harmonize its work with national programs? Conclusions: PSI/M has been able to work in harmony with the public sector supply chain system through its community-based supply chain. It collaborated in a complementary manner with national programs such as the NMCP, and helped to fill the GOM’s gaps in service provision and therefore, contributed to the provision of health products in all the sectors of society. How do GOM actors with whom the project still works perceive the project and its outcomes, and how do they see their related needs in the immediate and medium term future? Conclusion: On the whole, GOM agencies perceive PSI/M as a useful partner in reaching end users of health services and products. They view it as a collaborative partner and its supply chain as complementary to the public supply chain. However, despite Agence du Médicament de Madagascar (AGMED) being a pivotal GOM agency for PSI/M, communications with them could have been better, or at least more frequent. More frequent communication would have likely minimized interruptions and delays in the introduction of new products. Was the PSI project successful in leveraging partnerships with local and international NGOs to distribute health products? Conclusions: In spite of political restrictions, PSI/M was successful in maintaining its collaboration with public institutions for the distribution of health products, especially at the regional and district levels. Thanks to PSI/M, SIGMA (a private manufacturer of Sur’ Eau) has acquired the technical skills to develop appropriate formulas and appropriate packaging for Malagasy people’s needs. SIGMA’s success has been impressive. Largely because of PSI/M’s aggressive promotion and distribution activities, nationwide demand for Sur’Eau is much larger than SIGMA’s production capacity.4 PSI/M’s partnership with SIGMA has been very efficient, complementary and instrumental in reaching outcomes in MCH. PSI/M has raised people’s awareness of the advantage of using Sur’Eau and of clean water not only in urban areas, but also in semi-urban and rural areas. PSI/M has successfully leveraged partnerships with NGOs to ensure that their products reached rural end users. It was instrumental in the achievement of the partner NGOs service delivery goals and the NGOs were very positive about the partnership. As well, PSI/M instituted a functional community supply chain through involvement of the supply points and CHWs. What was the impact of the political situation on the project’s performance? Conclusion: PSI/M seamlessly adapted to the local circumstances following the 2009 political crisis and adopted new modes of operation with no discernible interruption in its services. The project continued to make gains in access to its products and services by working in collaboration with local and international non-governmental organizations (INGOs) while at the same time, maintaining the USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report ix needed communications with the GOM. PSI/M maneuvered through those turbulent times with acumen, finesse, and flexibility.                                                              1 Top Réseau is a franchise network of clinics focused on providing youth-friendly RH and child survival services. The original objective of the Top Réseau clinics was to supply youth-friendly, high quality, and affordable RH services to young people aged 15 to 24. After the first 10 years of the Top Réseau brand, PSI/M decided to revise the branding and service delivery model to reengage clients and stay competitive as a health care services network. In August 2010, PSI/Madagascar created a new marketing plan for the Top Réseau network and decided to integrate other health areas to benefit more clients and to increase the geographic coverage of the network. 2 On March 17, 2009, after demonstrations in the capital, President Ravalomanana signed power over to the military, which in turn conferred the presidency on opposition leader Andry Rajoelina, the mayor of Antananarivo and leader of the demonstrations. Rajoelina declared himself “President of the High Transitional Authority” (HAT) and pledged to hold presidential elections by October 2010 (a pledge that he did not fulfill), following a constitutional referendum and revision of the electoral code. The United States condemned the unconstitutional and undemocratic change of power in Madagascar and considers the series of events of early 2009 that led to the installation of the de facto leadership to be a military coup d'etat. On September 17, 2011, representatives of most of Madagascar's major political factions signed a "Roadmap for Ending the Crisis in Madagascar," endorsed by the Southern African Development Community (SADC), aimed at ending the long political crisis through the formation of a more inclusive, power-sharing interim government that would prepare the country for elections. The United States considers the series of events in Madagascar in early 2009 to be a military coup d'état and, as a result, has suspended all assistance programs that directly benefit the government as well as all non-humanitarian assistance to Madagascar. http://madagascar.usaid.gov/crisis-in-madagascar 3 The supply chain management is discussed in more detail under Evaluation Question three in section three, both in terms of stock-outs and actions taken by PSI/M to improve distribution. It was not possible to provide an average for each variable in the LSAT data collected. The data for some variables was not available at certain locations or not relevant to each location. Please see Annex K for a bar graph that displays the aggregated scores of selected variables from the Logistic System Assessment Tool guided site observations that the Evaluation Team conducted at the PSI/M central warehouse in Antananarivo and at eight regional PSI/M warehouses including: Fianarantsoa, Haute Matsiatra Region; Tôlanaro, Anosy Region; Morondava, Menabe Region; Antsiranana, Diana Region; Toamasina, Atsinanana Region; Toliara, Atsimo-Andrefana Region; and Mahajanga, Boeny Region. 4 Guided interview with SIGMA Director, June 2012. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 1 1. INTRODUCTION This paper reports on the findings, conclusions, and recommendations from an end-of-project evaluation conducted by International Business & Technical Consultants, Inc. (IBTCI) contracted by the United States Agency for International Development (USAID). The project evaluated was the Social Marketing Project for Child, Maternal, and Reproductive Health Products and Services in Madagascar (hereafter referred to as PSI/M) implemented between 2008 and 2012 and funded at a level of approximately $30,450,000 United States dollars (USD). This project was implemented by Population Services International in Madagascar (PSI/M) under USAID Cooperative Agreement Number 687-A-00-08-00032-00. There were a variety of international and local sub-grantees and other sub-recipients which are described in Annex A, including their funding levels. The Evaluation Team collected data between May – August 2012 and conducted follow-up interviews and discussions with USAID/Madagascar and PSI afterwards. MALAGASY CONTEXT Since a military-supported coup in January 2009, Madagascar has been struggling with an ongoing political and socio-economic crisis that has made everyday life increasingly difficult for the Malagasy population. Prior to the crisis, 70 percent of the country was living on $1.25 USD or less per day. Inflation, decreased investment, and increasing unemployment all have compounded the country’s problems.5 As of November 2012, the government failed to hold free and fair elections. The United States Government (USG), along with other bilateral and multilateral donors, suspended support to the Government of Madagascar (GOM) and all non-humanitarian assistance until an internationally recognized, democratically elected president takes over. With this policy change, all USAID non￾humanitarian support to the public sector effectively ended. However, more recently, USAID and its implementing partners are allowed to coordinate and share information with the Ministry of Health (MOH). Diarrhea is one of the three main causes of mortality and morbidity in Malagasy children under five years of age (CU5), and an estimated 3.5 million Malagasy school days and five million work days are lost due to problems related to water, hygiene and sanitation.6 Access to potable water is a serious health constraint in urban areas and is particularly dire in rural Madagascar. An estimated 60 percent of the Malagasy population does not have access to potable water. Dirty or used (gray) water is the main source for drinking, bathing, and washing clothes. Children from the poorest 20 percent of the population are three times as likely to have diarrhea as compared to the 80 percent of the population with higher incomes. More than twice as many rural children of the same ages die of diarrheal diseases as do urban children.7 Thirty-seven percent of Malagasy caregivers who seek care for CU5 ill with pneumonia-like symptoms go to the private sector.8 Among them, almost a quarter, (24 percent) are seeking care in private health centers and another 11percent are going to pharmacies. Nonetheless, among the poorest two quintiles, less than half of caregivers seek any care to treat pneumonia even though it kills more children worldwide than AIDS, malaria, and measles combined. For the past two decades, Malagasy populations have been exposed to family planning/reproductive health (FP/RH) behavior change communication (BCC) messages about health products that can be used to limit family size or to plan for birth spacing. Malaria remains a critical concern for the overall health system, despite decreases in prevalence and mortality as reported in the 2008-2009 Madagascar DHS. Malaria is endemic in 90 percent of Madagascar and the entire population is considered to be at risk for the disease. Malaria is the USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 2 leading reported cause of overall hospital mortalities. Reported malaria cases and deaths through the national Health Management Information System have shown decreasing trends in morbidity and mortality between 2003 and 2009. Overall, hospital deaths attributed to decreased malaria rates fell from 17 percent in 2003 to six percent in 2009 according to the National Program to Control Malaria (NMCP). In 2009, malaria was responsible for an estimated four percent of all reported outpatient visits, and 14 percent of all CU5 admitted to a hospital were diagnosed with severe malaria.9 Rates of sexually transmitted infections (STIs) in Madagascar, both congenital (syphilis) and sexually transmitted, are unusually high. Although Human Immunodeficiency Virus (HIV) prevalence among the general population is reported as less than one percent, commercial sex workers (CSWs) have an HIV prevalence rate of around nine percent, and men who have sex with men (MSM) have a prevalence rate of 14.5 percent and above. It is against this epidemiological and socio-cultural backdrop that USAID formulated PSI/M’s expected results. PURPOSE OF THE EVALUATION According to the scope of work (SOW), the purpose of this evaluation was two-fold: 1) Collect qualitative and quantitative information about PSI/M’s implementation, progress and challenges, with particular emphasis on whether PSI/M successfully achieved the expected results of the project. Additionally, share any programmatic, management and/or financial obstacles that affected project implementation to inform future projects. 2) Inform USAID/Madagascar and key stakeholders on recommended strategies, structures or actions for future projects involving supply chain and distribution systems to increase efficiency and impacts, e.g., viability of community supply chain system; priority areas for local capacity building; gaps in management structure and oversight; consumer focus in product distribution and price determination, etc. PROGRAM BACKGROUND The goal of the PSI/M was to improve the health status of the Malagasy people, especially women and children. The purpose of the project was to increase the use of effective health products, services and behaviors in the areas of FP/RH, STI treatment, HIV prevention, malaria prevention and treatment, diarrheal disease prevention and treatment, and pneumonia prevention and treatment. Project strategies included engagement of the private sector and evidence-based social marketing. Between 2008 and 2012, PSI/M supported the promotion, communication, research, and training for health services, and the social marketing of 16 subsidized health products in wholesale commercial, pharmaceutical, and community outlets based at established supply points. As well, a selection of PSI/M products was made available for Top Réseau, which is a franchised network of private, clinic-based providers.10 Per USAID’s request, PSI/M did a major overhaul of its project design for fiscal years (FY) 2011 and 2012. This was in response to the difficult political and socio-economic environment following the 2009 political crisis.11 On June 6, 2011, PSI/M responded with a proposed readjustment of program activities and requested a ceiling increase of $4,950,000 from its original Cooperative Agreement ceiling of $25,500,000 in order to align project activities within the altered operational context (the budget increase was received in October 2011). During FYs 2011 and 2012, PSI/M expanded activities with private sector partners, and strengthened the supply chains at the USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 3 commercial, pharmaceutical, and community levels so as to absorb and distribute product inventories slated for national level distribution and to implement social marketing, BCC, inter￾personal communication (IPC), supply chain distribution enhancement, and other health promotion and capacity building activities. PSI/M increased training, management and technical support, and other capacity building activities with private sector partners such as the Top Réseau franchised clinics, pharmacies, and other commercial retailers, as well as community level actors such as local distributors at supply points, NGOs, faith based organizations (FBOs), and community health workers (CHW) to meet project objectives while complying with USG’s restrictions on direct collaboration with the GOM. For the Top Réseau clinics, PSI/M also expanded its portfolio of socially-marketed products and related services, especially for maternal and child health (MCH) and expanded the range of services offered. This evaluation should be viewed in terms of these socio￾political and programmatic challenges. Annex B provides maps of PSI/M’s commercial and Top Réseau coverage areas. PSI/M SOCIAL MARKETING PROJECT KEY RESULTS The evaluation questions are centered on the project’s key results which are presented in the Cooperative Agreement in lieu of project objectives. These expected results are described below: Result One—MCH Increase the availability and use of proven lifesaving interventions that address the major killers of mothers and children and improve their health and nutrition status. Target Groups  Caregivers of CU5, with an increased focus on peri-urban and rural areas  CU5 suffering from diarrheal and pneumococcal disease, with an increased focus on peri￾urban and rural areas. Result Two—FP/RH Expand access to high-quality, voluntary FP services and information, and RH care thus reducing unintended pregnancy and promoting healthy reproductive behaviors of men and women, reducing abortion, and reducing maternal and child mortality and morbidity. Target Groups  Women of reproductive age (15-49 years) , with an increase focus on rural areas  Young women (15-24 years) in urban target areas Result Three—Malaria Reduce malaria related mortality through support for implementation from President’s Malaria Initiative (PMI), related malaria control programs and malaria research activities. Target Groups  Households (limited to Long-Lasting mass distribution campaign)  Pregnant women with an increased focus on peri-urban and rural areas  CU5 and their caregivers (women 15-49) with an increased focus on peri-urban and rural areas. Result Four—HIV/AIDS USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 4 Reduce the transmission and impact of HIV/AIDS through support for prevention, care and treatment programs. Target Groups  Youth (15-24 years) with a focus on urban target areas  Female CSWs and their clients, including men at risk; and  MSM with a focus on urban target areas. PSI/M conducted periodic qualitative and quantitative studies across its four result areas—MCH, FP/RH, malaria, and HIV/STI—with the intention to produce timely and actionable evidence to develop targeted interventions. These studies focus on the building blocks of social marketing activities, often referred to as the “4 Ps”: Product, Price, Placement, and Promotion. By using and disseminating its research, PSI/M aimed to strengthen and expand the availability and use of essential health products and services throughout Madagascar, while it continuously adapted to changes in the local operating context. The Top Réseau clinic franchise was established in 2001 by PSI/M to give access to medical services to the youth, and to encourage the adoption of safer health behaviors, especially in relation to FP/RH, and HIV/STI prevention. Through Top Réseau, PSI/M established an innovative and unique network focused on clinical services—a first for PSI in Madagascar. This is a fractional franchise, meaning that not all the services offered by the enrolled private clinics are supported by PSI/M but nonetheless, PSI/M creates demand around the clinics. Unlike the Marie Stopes International (MSI) franchised clinics, Top Réseau clinics are independent and financially autonomous. In supporting the Top Réseau clinic franchise, PSI/M focused on integrating service delivery-related activities and training. PSI/M also provided on-the-job support for a basic package of health services that included three components: Integrated Management of Childhood Illnesses (IMCI), FP, and HIV/STI. Key elements in IMCI included the prevention and treatment of diarrhea, pneumonia, and malaria. FP was comprised of both short- and long-term contraceptive methods, as well as training on cervical cancer screening and basic maternal care (post abortion care counseling). HIV/STI efforts were focused on prevention and encouraging use of qualified providers for STI treatment and voluntary counseling and testing (VCT). Table 1 below lists the 16 health products that are subsidized by PSI/M, and indicates whether the products were also branded by PSI/M. Table 1:  PSI/M Subsidized Health Products Result Area   Name of Health Product PSI/M Branding Result 1 Products – MCH (Prevention and Treatment of Diarrhea and Pneumonia / Integrated Management of Childhood Diseases (IMCI)) Sur’Eau (safe water treatment) Yes Pneumostop – in syrup (for non‐severe pneumonia treatment) Yes Viasûr Diarrhea Treatment Kits (DTKs) Yes HydraZinc DTKs No Result 2 Products ‐ FP/RH Pilplan (oral contraceptives) Yes Confiance  (aka DEPO‐PROVERA) (injectable contraceptive)   Yes Rojo CycleBeads (Standard Days Method ‐ SDM) Yes USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 5 Table 1:  PSI/M Subsidized Health Products Result Area   Name of Health Product PSI/M Branding Copper T 380A (Intra Uterine Device ‐ IUD) No Implanon (contraceptive implant) No Feeling (female condom)   Result 3 Products ‐ Malaria Prevention and Treatment Super Moustiquaire (Long Lasting Insecticide‐treated Nets ‐ LLINs) Yes ACT (Artemisinin‐based Combination Therapy ‐ formerly ACTIPal) No (as of 2012) RDT (Rapid Diagnostic Test) No Result 4 Products ‐ HIV/STI Prevention and Treatment Cura 7 (Non‐ulcerative STI treatment) (The new product is awaiting the authorization to market is, and will be a branded product) Yes Genicure (Ulcerative STI treatment) Yes Protector+ (male condom) Yes Feeling (female condom) (This same product is used as a contraceptive.) Yes 2. EVALUATION DESIGN AND METHODOLOGY USAID put forth to the Evaluation Team 22 questions to answer. For each PSI/M result area, there is an overarching question and a set of related questions. Table 2 below delineates the specific questions. The presentation of the findings, conclusions, and recommendations in this report are structured around these questions. Note that not all questions will have a corresponding recommendation. Table 2:  Evaluation Questions Program Result Area Key Evaluation Questions I. How well did PSI perform in the areas of Maternal Child Health (MCH),  Family Planning (FP) and Reproductive Health (RH), Malaria, and Sexually Transmitted Diseases/ Human Immunodeficiency Virus (HIV/STI)? 1. MCH  How did PSI perform in the area of diarrhea prevention and treatment activities?  How did PSI perform in the area of pneumonia prevention and treatment activities?  How did PSI perform in the area of the integrated management of childhood illness?  How were the findings of research results in MCH used by PSI or others? 2. FP/ RH  How did PSI perform in the provision of family planning services? 3. Malaria  How did PSI perform in the provision of malaria prevention and treatment services? 4. HIV/STI  How did PSI perform in the provision of STI and HIV prevention and treatment services?  What are the major achievements of PSI’s peer outreach activities for USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 6 Table 2:  Evaluation Questions Program Result Area Key Evaluation Questions prevention of HIV/STI? II. How did PSI perform in implementing and working with Top Réseau clinic franchise? Capacity building of the Top Réseau clinics franchise  In regard to the Top Réseau clinic franchise, what were PSI’s major achievements in developing private sector services?  Did these services increase access and use for those most in need?    To what extent are the Top Réseau franchise clinics sustainable without PSI assistance? III. How did PSI perform in the supply chain/distribution systems? Supply Chain/ Distribution Systems  In maintaining the supply chain for community‐based products, what were the major achievements and weaknesses?  What contribution did the PSI project make to national coverage of health products through its supply chain/distribution system?     How about in rural areas?  To what degree has the project been successful in establishing a supply chain model for USAID‐funded Community Based Distribution programs since 2009?    What were the obstacles?    How were they overcome? IV. Other cross‐cutting questions Cross‐cutting questions  In the areas of FP and Malaria, MCH and Child Survival, HIV/AIDS/ STIs, and Supply Chain, what were the results of PSI’s capacity building efforts with local organizations?  To what extent has PSI been able to align and harmonize its work with national programs, despite constraints, in all the program areas listed above?  How do GOM actors with whom the project still works (e.g., National Institute of Statistics, National Laboratory, National Medical Distribution System, Customs and public sector facilities in urban and rural areas), perceive the project and its outcomes, and how do they see their related needs in the immediate and medium term future? (NEW).  Was the PSI project successful in leveraging partnerships with local and international NGOs to distribute health products?  What role did the PSI project’s contributions (in BCC, Products, Research, Distribution, Training, Housing…) play in the attainment of other partners’ achievements?  What was the impact of the political situation on the project’s performance on MCH/FP, HIV/STI, and Malaria, Child Survival, and capacity building? To what extent has PSI been able to adapt to new circumstances and modes of operation? EVALUATION TEAM USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 7 The Evaluation Team was comprised of international consultants from France, the United States, and Kenya, and four local consultants, namely: Veronique R. Praz, Team Leader & Social Marketing Specialist Alice Morton, Senior Evaluation Methodologist Ian Matondo, Supply Chain Specialist Ramy Razafindralambo, Local Monitoring and Evaluation Specialist Jean Clément Andriamanampisoa, Local Translator/Enumerator Honorée Razafimiandry, Local Enumerator Mahatsindry Randzato Solo, Local Logistician APPROACH This evaluation gathered primarily qualitative information and to a much lesser degree, quantitative data. Given the broad array of stakeholders who vary according to their relationship with PSI/M and their background experiences, the Evaluation Team felt it made most sense to rely on qualitative methods to best respond to the detailed list of evaluation questions. Therefore, the greater part of the data presented herein, are based on the opinions, recollections, and experiences of people either directly or indirectly involved with PSI/M. Key sources of information included representatives from USAID; PSI/M and its sub-grantees and other partners; GOM entities in the capital and at the local level; implementing partners; organizations and participants including wholesalers, retailers, and distributers; end users of PSI/M products (beneficiaries); and other project beneficiaries. USAID/Madagascar’s Program Officers and Health Officers accompanied the Evaluation Team on many of the eight regional site visits as observers. They also monitored the interpretation from French or Malagasy to local dialects and back into French. SAMPLING STRATEGY Given the complex nature of the activities being evaluated as well as the aforementioned qualitative nature of the evaluation, the evaluators adopted a purposive sampling strategy based on the identification of information-rich cases – the sources from which they could learn most about the activities. The information-rich cases that were selected for in-depth exploration represented a broad variety of approaches and activities used by PSI/M and its partners in different results areas and across project components (e.g., BCC, IPC, Top Réseau, supply chain management support, etc.). This approach also enabled the evaluators to identify and further examine emerging issues and salient factors affecting project outcomes. The purposive selection considered the following: Box 1: Key Informant Respondent Categories (n = 213)  USAID (13 KIIs)  PSI/ M staff (35 KIIs)  Other donors (4 KIIs with representatives from UNICEF, UNFPA, GFATM, and the World Bank)  PSI/M implementing partners and sub‐ awardees (25 KIIs)  GOM representatives at the central, regional and district levels (6 KIIs)    Top Réseau owners, staff, and providers (27 KIIs)  CROM Presidents (8 KIIs)  End‐users of PSI/M products (35 KIIs)  CHWs, pear educators, and their supervisors (20 KIIs)  Community based distributors and supervisors at supply points (12 KIIs)  Commercial Wholesalers (20 KIIs)  Pharmaceutical Wholesalers (8KIIs)  Retailers of PSI/M products in the supply chain (124 site observation visits) USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 8  Balance of the number of stakeholders in each respondent category;  The respondent’s level of exposure or involvement with project activities in order to create a balanced representation of the extensive range of project support products and services;  The respondent’s proximity to project sites and activities within the nine regions selected; and  Availability of the respondent. METHODS To assess the availability of PSI/M products, as well as the effectiveness of social marketing and capacity building activities, the following data collection methods were utilized: 1) qualitative data collection based on twelve structured instruments for key informant interviews (KIIs) and focus group discussions (FGDs) with target groups such as beneficiaries (the end users of PSI/M socially marketed products), most-at-risk populations including MSM, and CSWs, youth, women and men of reproductive age, and CHWs; 2) site visits to PSI/M warehouses both at central and regional levels; 3) observation visits guided by the Logistics Systems Assessment Tool (LSAT) in Antananarivo and eight regions; and 4) site observation visits to retailers (kiosks, pharmacies, hotels, boutiques, grocery stores, etc.). Most interviews 12 took place during visits to project-supported distribution supply points, service delivery clinics, and selected villages in the nine regions. Respondents for KIIs were selected from project stakeholder lists provided by USAID and PSI/M and provided a balance of perspectives. Box 1 lists each of the KII respondent categories and the respective number of KIIs for each for a total of 213 interviews. Annex C provides more specific information on the respondents. Participants for FGDs were selected to create a balance of perspectives among the following categories of stakeholders: the targeted beneficiaries of PSI/M-supported products and services in rural, peri-urban and urban areas (i.e., caregivers of CU5, men and women of reproductive age, youth, CSWs, MSM, and CHWS who are directly involved in service provision and the distribution of products. When it was possible, participants for each FGD were selected among persons with similar age, gender, residence, and other relevant characteristics. Box 2 delineates the number of FGDs for each type of respondent. SITE SELECTION CRITERIA    The selection of specific locations for site visits and respondents was based on consultations with PSI/M and the following criteria: the presumed ability of respondents to answer the evaluation questions and provide information to validate, corroborate, or refute PSI/M reports and interviews; inclusion of perspectives from rural areas; security concerns; and the availability of respondents, given that data collection was conducted while the PSI/M project was in the process of closing out. Per the SOW for this evaluation (see Annex D), the Team visited project sites in nine regions, including Antananarivo and surrounding areas. In each of the nine regions, they visited a minimum of four sites and at least one rural and one urban community for a total of 22 sites. Please refer to Annex E for a view of all PSI/M intervention sites and those visited by the Team. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 9 To allow the Team to assess a variety of perspectives on PSI/M project implementation, the Evaluation Team visited sites ranging from ones with the longest involvement with the project (with the most staff and activities) to site at the other end of the spectrum (ones that were more recently involved in project activities (with fewer staff, activities, and partners). The following characteristics were considered for selecting data collection sites and sources:  Central plateau and coastal areas;  Cities, peri-urban, and rural including communes, villages including those with poor access;  Variety of target populations such as men and women of reproductive age, caregivers of CU5, youth, CSWs, MSM, and their associations, etc.; and  Sites less commonly visited by project staff. The Evaluation Team subdivided into two sub-teams. Each sub-team collected data in Antananarivo and four regions of Madagascar that were selected in consultation with USAID during the first two weeks of field work.13 During in-country work (June 11, 2012 - August 6, 2012), the Evaluation Team employed a comprehensive approach to evaluate the effectiveness of PSI/M’s communication and supply chain strategies with a view to producing substantive, actionable recommendations to inform the design of new or follow￾on interventions. Also, the Team examined achievements towards program Results 1—4 to answer each of the evaluation questions. The Team visited urban, peri-urban, rural, and more remote sites. During these visits, the Team engaged stakeholders to “ground truth” reported behavior changes and product distribution levels, as well as spot-check financial and management data, and observed service delivery at first hand. These KII and FGD guides, as well as the Logistics System Assessment Tool (LSAT), generated qualitative and quantitative data to maximize triangulation with data from the desk review. Data were analyzed during and at the end of field work. See Annex F for a detailed listing of the regions and districts visited with a breakdown of the types of respondents and facilities visited at each site. Desk Review Before the start of in-country field work, the Team conducted a desk review of PSI/M materials and reports to understand the project’s components and strategies as well as reporting of their achievements and challenges. The documents included the PSI/M Monitoring Cooperative Agreement and modifications, PSI/M monitoring and evaluation (M&E) plans and manuals; the PSI/M Performance Monitoring Plan (PMP); PSI/M financial and administrative standard operating procedure (SOPs) and manuals; PSI/M annual reports; PSI/M research reports; and general health information about Madagascar such as the Demographic Health Survey 2008-2009 as well as other reference materials. Annex G includes the list of reference documents. During the desk review, the Team examined the project’s qualitative and quantitative data. Key Informant Interviews The list of key informants was finalized in collaboration with USAID and PSI/M headquarters and regional staff. PSI/M’s Director for Top Réseau and Health Service Delivery Director facilitated a Box 2: FGD Results by Respondent Type (n =52)  Women of Reproductive Age (12)  Men of Reproductive Age (10)  Youth (7)  CSW (6)  MSM (5)  CHW (8)  PSI/M Central‐level Staff (4) USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 10 meeting between the Evaluation Team Leader and PSI/M’s Top Réseau Regional Coordinators in Antananarivo. This meeting helped identify a core set of key informants representing all of the respondent categories. Please see Box 1 for the totals of KIIs by respondent categories and Annex F for a detailed list of data collection results in each region. Focus Group Discussions Using the snowball method, discussions with key informants generated a list of FGD participants that included a range of project stakeholders, including service providers, sales agents, community￾based distributors and end users. Where community-based distributors or CHWs were too distant from each other, they were asked to assemble at a central location to participate in FGDs (and in some cases, KIIs). Each FGD aimed to include a minimum of six and a maximum of 10 participants where possible. See Box 2 for the composition of the 52 FGDs conducted according to the type of respondent. The FGD of PSI/M staff represented different roles within the project: senior management, supply chain and distribution, BCC, and service delivery. Logistics Systems Assessment Tool LSAT Since being introduced in 2001 by the USAID/DELIVER project implemented by John Snow Incorporated (JSI), the LSAT has been a standard method for conducting evaluations of health commodity supply chains in the developing world. The LSAT is designed to facilitate a comprehensive quality assessment of the separate components that make up a logistics system. It was developed for use by ministries of health, NGOs, and other supply chain stakeholders. The LSAT was used in this evaluation to conduct nine assessments at the central and regional levels. DATA COLLECTION INSTRUMENTS After the Team conducted in the desk review of project documents, they designed 12 qualitative data collection guides tailored to specific respondent categories. These instruments are presented in Annex H. To collect information on the logistics systems, the Team borrowed JSI’s LSAT instrument. DATA ANALYSIS PLAN Qualitative data were analyzed after each KII and FGD took place. Each evening, the respective Evaluation Team (Teams A and B) and their interpreters/facilitators met to review their notes and FGD flip charts, and determine whether there were any marked differences from groups at other sites. Each sub-team concluded these meetings by arriving at an agreement on the findings of the day. Midway through the field work, the two sub-teams met to assess the implications of the qualitative data they had gathered to date. At times, this process led to some modifications in the data collection tools, note taking, and data entry. LIMITATIONS As mentioned earlier, the Team was unable to select KII or FGD participants randomly from a known universe of possible respondents. As for most categories of respondents, comprehensive lists of respondents from which a statistically random sample could be selected, did not exist, or were not readily available. Instead of a randomized selection approach, the Team implemented a purposive selection process as described above. As the results of this study are not based on an experimental design, its results cannot be generalized to make any statements about the regional populations or the stakeholder population. Attribution USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 11 cannot be assigned in the absence of a counterfactual and the inability to control confounding variables. Also, the nature of a qualitative interviews leads to certain biases, namely, interviewer and respondent biases. In particular, this study was susceptible to interviewer bias because there were six primary interviewers who might have instilled their individual interpretations of the questions. The data collection instruments were developed with the participation of all Evaluation Team members which helped to mitigate this bias. In regard to potential respondent bias, respondents might perceive the need to express positive results or withhold from sharing negative results (known as a “halo” bias). Unfortunately, the recall period for this study was four years and therefor respondents might not have completely remembered all the facts. It is possible that respondents mistakenly attributed interventions or results to PSI/M or alternatively, they might have forgotten important milestones, which were attributable to PSI/M (recall bias). The Evaluation Team was reliant on others to invite participants to join the FGDs. Despite the aim to have a balanced and unbiased group of participants, there were situations with homogenous, self￾appointed volunteers for the FGDs that might have skewed the perspective the group was meant to have represented. Lastly, several languages were at play during the interviews. The Evaluation Team Leader is a native French speaker but French is the second language of the other team members with the exception of the Supply Chain Specialist who does not speak French and was reliant on a local team member to serve as an interpreter during site visits and interviews he conducted when collecting LSAT data. Malagasy is native tongue of the majority of the respondents, which might have been an impediment to the interpretation of the interview. NOTE TO READER The reader is encouraged to read the endnotes used throughout this report. Many of these endnotes provide specific information on the data sources for the information reported and in some cases, the exact number of respondents who reported on a particular finding. Each recommendation is prioritized according to short, medium, or long term for planning purposes. The associated stage of project development is included as a suggested guide for managers. Also, note that not all of the sub-questions necessarily have a corresponding recommendation for a couple of reasons. First, there is some overlap among the 22 evaluation questions; therefore, a recommendation might be based on conclusions from more than one sub-question. As well, not all sub-questions warranted a recommendation. 3. QUESTION ONE: PERFORMANCE IN KEY HEALTH AREAS This section covers the various components covered under Evaluation Question One: “How did PSI/M perform in the area of maternal and child health (MCH); family planning (FP); malaria; and STI/HIV?” The MCH programmatic area is comprised of diarrhea and pneumonia prevention and treatment for CU5. Support for FP comes in the form of promotion and distribution of contraceptives, and service delivery through franchised clinics. PSI/M markets and distributes insecticide-treated nets (ITN), rapid test kits to prevent and diagnosis malaria, and malaria treatment for infants and CU5. PSI/M also distributes products for the prevention and treatment of STIs. Presented below are the respective findings, conclusions, and recommendations for: 1) diarrhea USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 12 prevention (MCH), 2) diarrhea treatment (MCH), 3) IMCI (MCH), 4) pneumonia treatment (MCH), 5) MCH research results, 6) FP product promotion and distribution, and FP service delivery, 7) malaria prevention and treatment, and 8) HIV/STI prevention, in this order. FINDINGS: MCH—DIARRHEA PREVENTION PSI/M activities for diarrheal diseases prevention included the social marketing and distribution of PSI/M’s Sur’Eau, a water treatment product produced by SIGMA, a local Malagasy company. Sur’Eau is one of the first socially-marketed products that PSI/M offered in Madagascar unrelated to FP. PSI/M distributes Sur’Eau through private sector wholesalers and distributors, such as CHWs, throughout Madagascar. PSI/M research results on acceptance and use of Sur’Eau has varied over the life of the project. A PSI/M Tracking Results Continuously (TRaC) study on diarrhea prevention and treatment revealed the proportion of mothers and caregivers reporting that they had used Sur’Eau in the month prior to the study had risen from 9.4 percent in the 2008 baseline to 14.9 percent in 2009. But in 2011, PSI/M reported that this proportion decreased to 5.4 percent.14PSI/M also reported that 32.3 percent of the 2011 TRaC respondents reported “ever using Sur’Eau”, which includes those who used it and abandoned it, and those who rarely used it.15 By the end of project year four, PSI/M reported sales of 6.58 million bottles of Sur’Eau, putting them on target by achieving 82 percent of the life of project target.16 Another 1.39 million bottles of Sur’Eau were sold between October 2011 and March 2012, which is close to double the average rate of sales in a six-month period in the previous four years. Based on the assumption that all of the bottles of safe water solution were used correctly, PSI/M reported that over 14.4 billion liters of drinking water were treated over the life of the project, which exceeded its target by 15 percent.17 The majority of respondents for FGDs and KIIs said using a water treatment product is important. Yet, hard data on the actual and correct use of Sur’Eau from product purchase to drinking water consumed was not available. The number of bottles of Sur’Eau sold and the number of liters of water that can be cleaned by one bottle of Sur’Eau are used as proxies for estimated product use. No empirical data—including those from the TRaC—were available to estimate the conversion of Sur’Eau units into improved drinking water and subsequently consumed as such. Sur’Eau has been available in the 150 milliliter (ml) formulation since the project start. PSI/M identified problems in the sales and use of the 150 ml format of Sur’Eau in rural areas. Indeed, findings from this evaluation confirmed that the larger bottle of Sur’Eau is more popular in urban areas compared to rural. People in rural areas usually buy food and other commodities in very small quantities because of their limited cash on hand. To boost Sur’Eau sales, PSI/M launched a new product format in a smaller 40 ml bottle targeted for community-based sales in October 2011. The product was still in the introductory stage and relatively new to users. To satisfy demonstrated urban demand, PSI/M decided to keep the original Sur’Eau 150 ml product on the market at the same subsidized price. During the fourth quarter of FY12, CHWs were trained by project staff on social marketing techniques for the new, smaller 40 ml format, and some were provided with start-up kits. However, FGD and KII respondents stated that PSI/M was having difficulties in selling the Sur’Eau 40 ml bottle because of insufficient communication and promotion at the community level. Respondents in a few of the Mahajanga communes reported that there was competition from other NGOs that distribute gallons of drinking water for free which they supposed might have caused decreased sales. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 13 In at least two of six FGDs with CHWs who had already been promoting and selling the older, large format bottle, participants cited Sur’Eau as a product that was hard to sell. The consensus was that half of the eight large format (150 ml) bottles in the start-up kit sold relatively quickly, but that the second half did not sell. CHWs were instructed to keep the remaining bottles and to continue to attempt to promote and sell them until they were expired, at which point they could trade them in for new ones at their supply points. According to one commune-level supervisor, CHWs cannot trade expired Sur’Eau large format bottles for a product that sold well, like Confiance (the Depo-Provera injectable) because this would impede forecasting and record-keeping.18 The PSI/M management in Antananarivo advised CHWs to continue promoting and selling Sur’Eau rationalizing with them it would take more sensitization and effort to increase sales. CHWs who participated in PSI/M trainings on how to promote and use the smaller format of Sur’Eau anticipated that it would be easier to sell the smaller formulation in rural areas because the smaller format is less expensive than the larger bottles. The evaluators did not find sufficient sales data to substantiate this assertion. (For a listing of the trainings offered to CHWs, see Annex I.) The importance of safe water treatment was discussed during FGDs with caretakers of CU5, the target groups for diarrheal diseases prevention and treatment products and related communications. As time permitted, this topic was also broached with non-targeted users such as men, youth, CHWs, CSWs, and MSM to ascertain the range of beneficiaries who had been exposed to BCC messages. As well, all FGDs were queried on the recognition of all PSI/M branded products. Almost all of FGD participants could identify Sur’Eau as a water treatment product, ranking it fifth out of the 15 most frequently used PSI/M-branded products. During these same FGDs with end users of PSI/M products, participants frequently mentioned Sur’Eau’s unpleasant odor as the reason why they did not consistently use the product. FGDs with outreach workers—such as CHWs, and peer educators trained in Sur’Eau promotion, use, and sales—did not raise the issue of an unpleasant odor as the reason for low usage. Rather, these respondents suggested incorrect use of Sur’Eau stemmed from BCC and IPC messages which instruct caregivers on how to use the product correctly. Noteworthy is that PSI/M has already re￾labeled the product’s instructions for use in response to their own research indicating low comprehension. At the time of this evaluation, the new labels were in effect. In September 2011, PSI/M changed the labeling of Sur’Eau as a result of its marketing planning process for the diarrhea prevention program. The instructions for use on the label were simplified because its research (TRaC) showed knowledge of the correct usage had been very low and that self￾efficacy to correctly use Sur’Eau had been decreasing. The evaluators found that PSI/M’s BCC strategies reinforced the linkages between Sur’Eau use and diarrhea prevention. Some BCC radio and television spots included messages on the use of Sur’Eau to clean water but without a reference to diarrhea prevention. Yet, there are other project-supported BCC mass media messages, which broadcasted the use of Sur’Eau in relation to PSI/M-supported diarrhea treatment kits (DTKs) (such as Viasûr, HydraZinc, and the generic brand format). Those messages explained the need to use clean water to mix the oral rehydration salts (ORS) found in the kits. CONCLUSIONS ON DIARRHEA PREVENTION (MCH) Knowledge transfer about a Sur’Eau’s purpose, when to use it, and how to use it correctly, does not guarantee that targeted audiences will adopt the new practice by purchasing and using this product USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 14 correctly. The rationale for sporadic, inconsistent, and/or incorrect use of Sur’Eau is uncertain but might be related to unclear messaging/marketing; unfavorable odor; competition from free sources of purified water; cost of the larger, 150 ml formulation; and/or level of effort from CHWs to promote the product among other reasons not identified through this evaluation. PSI/M was cognizant of incorrect use and took appropriate steps to ameliorate the situation albeit more work remains in this regard. RECOMMENDATIONS FOR DIARRHEA PREVENTION (MCH)  A situational analysis should be conducted to determine the barriers to consistent and correct use of Sur’Eau and to explore the feasibility and market suitability of a water treatment tablet. Interventions in the follow-on project should address these findings and piloted before being brought to scale. Particular attention should be placed on creating streamlined, but comprehensive BCC and IPC messages to address 1) the correct use of safe water treatment products for diarrhea prevention, and 2) to provide increased fluids (with treated water) and feed normally when a child has diarrhea. Monitoring of correct and consistent use of Sur’Eau should be assessed at baseline, mid-term and again at the end of the next project through TRaC surveys. Medium-term/Implementation  Sufficient budget should be allocated to educate consumers about the new Sur’Eau format in rural areas. CHWs will need to review the message on the proper use of Sur’Eau with the beneficiaries. Short-term/Work planning  PSI/M should monitor the uptake of the new 40 ml formulation of Sur’Eau. Supervisors at supply points should train and assist the CHWs to record the number of sales of the 40 ml Sur’Eau in order to ascertain if it is successful in comparison to sales of the old larger format. Short-term/Performance Monitoring Plan (PMP) FINDINGS: MCH—DIARRHEA TREATMENT PSI/M diarrhea treatment activities included collaboration between PSI/M and Abt Associates since 2008. Abt was the prime contractor and PSI was a sub-contractor for a USAID centrally-funded project called POUZN (Point-of-use Water Treatment and Zinc) that was dedicated to improving the water quality and adoption of zinc treatment for diarrhea. Through POUZN’s contractual vehicle and with technical assistance from Abt Associates, PSI/M developed two DTKs, each containing 10 tablets of 20 mg zinc sulfate and two sachets of the new low-osmolarity ORS. The HydraZinc DTK was launched in June 2009 and the Viasûr DTK was launched in April 2009. Once the prototype and feasibility had been completed, USAID/Madagascar was convinced this was a viable intervention and directly funded PSI/ to continue supply and promotion of DTKs. By the end of FY 2011, PSI/M had distributed 388,312 units of socially-marketed DTKs, representing 52 percent of the life of project target.19As of the middle of FY 2011, another 84,181 DTKs had been distributed through both community and pharmaceutical channels, meeting only 23 percent of the FY 2012 target. PSI/M attributes this to a delivery delay of 200,000 units of ViaSur and 100,000 units of HydraZinc which resulted in two months of stock-outs of HydraZinc at the pharmacy level in January-February of 2012. This delay was the result of an administrative oversight at APEX (the manufacturer) as their certificate of Good Manufacturing Process expired and had to be renewed prior to shipping. Despite this, FY 2012 performance increased by 40 percent compared to FY 2011, with 265,120 DTKs distributed in FY 2012 as compared to 156,676 in FY 2011. Some KIs credited this increase to a larger budget for BCC activities in 2012. The Viasûr DTK remains on the market for community-based distribution and is promoted and sold by CHWs under the USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 15 SantéNet2 project. According to PSI/M reported data, Viasûr sales increased by eight percent from the beginning of the PSI/M Social Marketing Project in FY 2009 through FY 2011.2021HydraZinc had greater name recognition among respondents than did Viasûr. Slightly more than four percent of FGD participants recognized or spontaneously mentioned the HydraZinc product, and slightly fewer stated it as the one of the most frequently used the socially-marketed products. Compared to Sur’Eau, which has been marketed for 10 years, HydraZinc is still relatively new. Moreover, the DTK launch was compromised by restrictions on advertising through government-owned media channels and diminished radio communication infrastructure. When asked how to treat diarrhea in general, most FGD participants mentioned using ORS, without adding a brand name product. Almost all interviewed caregivers volunteered information about how to prepare and use ORS. FGD participants were less clear about why and how to use the zinc tablets as part of the treatment. The majority of FGD participants that discussed how to prepare Viasûr, and ORS in general, stressed the requirement to use clean, boiled water. Most said they used boiled water rather than Sur’Eau to treat the water before preparing ORS. When asked the same questions about HydraZinc, they still talked about boiling water, and said that they could prepare the ORS at home. When asked whether they always ensure the affected child took all 10 zinc tablets, many FGD participants admitted that they “saved” some tablets to use for the next bout of diarrhea for the same child or for another child. Yet there were others who said they gave the entire course of zinc consecutively. Some FGD female caregivers were aware of the necessity to give sick children liquids in addition to the HydraZinc DTK for the proper treatment of diarrheal diseases but they did not mention additional food should be given as well. A few respondents also mentioned that HydraZinc stocks are not being sold because it is too costly. CONCLUSIONS ON DIARRHEA TREATMENT (MCH) The use of DTKs, though rising, has been handicapped by a low product distribution over the life￾of-the project compared to anticipated program targets. Four key messages on the correct use of DTKs have not been fully absorbed among the targeted beneficiaries interviewed in this evaluation. These messages include: 1) the need to use treated water, not boiled, to prepare ORS; 2) sick children need additional liquids during their illness; 3) regular feedings should continue while treating for diarrhea; and 4) the full course of zinc tablets should be consumed to ensure a speedy recovery. RECOMMENDATIONS FOR DIARRHEA TREATMENT (MCH)  BCC campaigns should hone in on the correct sequence of steps for the proper treatment of diarrhea. Future social marketing and BCC activities for diarrhea treatment should examine previous campaigns and messages to address gaps in the four key messages outline above. The situational analysis suggested above to better understand the constraints to correct use of Sur’Eau could also include a component on the correct use of DTKs. Future programs should refine BCC and IPC messages regarding the rationale for administering the complete blister of zinc tablets to treat children with diarrhea. Medium￾term/Implementation  Communication efforts need to continue to further build awareness of the HydraZinc and Viasûr brands. BCC campaigns, IPC, and educational materials should be done in concert to relay a unified and simple message. Medium-term/Implementation USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 16 FINDINGS: MCH—PNEUMONIA TREATMENT Prior to FY 2011, PSI/M had not provided services or products specifically targeting children or adult’s pneumonia. At the national level, however, the MOH made the reduction of pneumonia in children a high priority in its strategic plan. Under Modification 11 (September 30, 2011) to the Cooperative Agreement, PSI/M began activities to develop and distribute an antibiotic to meet this priority need. During FY 2012, PSI/M introduced cotrimoxazole, a highly subsidized prepackaged treatment for non-severe pneumonia at the community level. This new product was branded as Pneumostop in two formulations: as an oral suspension for children aged two to 11 months; and as a tablet format for children aged one to four years old. The brand Pneumostop was official launched in November 2011 during World Pneumonia Day; however, securing the authorization to market the drug (AMM) from the Direction de l’Agence des Médicaments de Madagascar (DAMM) for the syrup formulation took much longer than anticipated. Even though PSI/M already had syrup stocked and trained staff (partner NGOs and CHWs) it was only able to commence distribution of the syrup in June 2012. The registration process for the tablet formulation is still pending and has been a much more complex and lengthy process than usual. Further complicating the situation is there was a change in the supplier of Pneumostop tablets and as a result, all the administrative documents had to be renewed in the name of the new supplier as well as all of the various certificates and samples. Given these delays in registration, PSI/M did not risk procuring the tablets until they are certain the registration process has been completed. Outside of the launch, communication activities around Pneumostop were limited to Mobile Video Unit (MVU) sessions and special events. No future BCC activities were planned until the registration of the tablets is completed. Although potentially outside of the project’s manageable interest, PSI/M did not have a contingency plan in place for the long delay following the launch. As Pneumostop syrup had not been available until the middle of this evaluation’s data collection period, its recognition and ranking vis-à-vis other socially-marketed products by targeted beneficiaries was quite low. Only about two percent of participants and interviewees knew about the drug; it ranked eleventh out of the 15 most frequently used PSI/M products. CHWs said Pneumostop had been an excellent product and sold well, but that the last time they tried to re-stock, the product was no longer available. Neither they nor the supply point managers could offer a reason for the stock-out. Without a re-supply of Pneumostop kits, CHWs feared that caregivers might revert to home remedies to treat children with pneumonia. In contrast, health care providers and distributors generally recognized and reported positive perceptions of the effectiveness of Pneumostop. Irrespective that the tablet formulation for children between one to five years was not on the market, the overwhelming majority of respondents for KIIs—including PSI/M staff and public doctors at the commune level—said that the product was extremely effective to treat pneumonia in CU5 and they claimed demand was high.22 KII respondents from the MOH at the communal and district levels highlighted the effectiveness of Pneumostop syrup for treatment of pneumonia in children under one year. “Not enough caregivers are seeking treatment when children showed symptoms of acute respiratory infections and only 42 percent of children under five with a chest cough and rapid breathing were brought to a health center in 2008 compared to 48 percent in 2003.” National Institute of Statistics (Madagascar), Madagascar Demographic and Health Survey 2008‐2009. Calverton, United States: ICF Macro, 2010 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 17 CONCLUSIONS ON PNEUMONIA TREATMENT (MCH) PSI/M was ambitious to design BCC activities and launch new pneumonia treatment products in the last year of the project. Nonetheless, the delay in launching Pneumostop’s tablet formulation will likely prevent PSI/M from reaching its FY 2012 targets. It is not possible to determine the sustainability of this MCH intervention that was just introduced in late FY 2011. The mixed results in sales and product recognition are likely due to the rapid roll out of the pneumonia treatment activities. RECOMMENDATIONS FOR PNEUMONIA TREATMENT (MCH)  PSI/M should develop a case study on the launch of Pneumostop to use the experience as a lesson learned to mitigate future problems with product introduction. The future program should take into consideration the need to repeat the BCC campaign and social marketing activities to address any problems resulting from the long delay between the product introduction and availability on the market. Medium-term/Implementation FINDINGS: MCH—INTEGRATED MANAGEMENT OF CHILDHOOD ILLNESSES (IMCI) At the end of FY 2011, PSI/M added an IMCI component to its portfolio of activities that aimed to integrate child survival BCC messages about socially-marketed MCH products to leverage opportunities to increase health seeking behavior. PSI/M also used MCH related events and activities with partners and sub-grantee organizations to harmonize IMCI messages sent to targeted populations. In the field, MOH officials and sub-regional health providers mentioned the importance of adopting an IMCI approach to improve and harmonize child survival messages, as did some Top Réseau providers. Service providers from NGOs (e.g., Mercy Ministries) and FBOs (e.g. Health Agency for the Lutheran Mission in Madagascar (SALFA)) acknowledged the importance of harmonizing IMCI from PSI/M, outreach workers, and service providers. Caregivers of CU5, and other women who are potential end users of PSI/M products spoke about diarrhea, pneumonia, and malaria products, but did not use the term “IMCI.” This is not surprising, given that this approach had only been disseminated by the project for a year, and not many CHWs had been trained in “IMCI” or given materials to support this approach in the months before the evaluation.23 Evaluation findings uncovered gaps in the content of CHW training on how to deliver effective IMCI messages. CHWs in FGDs did not specifically refer to “IMCI” activities; however they did indicate the need for more training to increase their knowledge of predisposing factors that place children at risk for common illnesses. They also requested trainings focused on approaches to improve the behaviors of caregivers grappling with multiple illnesses. A number of CHWs also expressed that they need more training to correctly identify illnesses more quickly. Training in MCH modules is offered to health care providers at Top Réseau clinics and for other physicians. This training is done either directly, by PSI/M, or through the representatives of the Council of the Regional Order of Physicians (called the Conseil Régional de l’Ordre des Médecins in Box 3: IMCI Description IMCI includes prevention and treatment of diarrhea and severe dehydration, pneumonia and ARIs, malaria, and promotion of exclusive breast feeding. At the institutional capacity building level, the IMCI strategy includes improving health worker skills, health systems, and improving family and community practices. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 18 French - CROM) who often share information about training content with other physicians and health care providers in the private and public sectors. CONCLUSIONS ON IMCI (MCH) By the end of the project, PSI/M’s approach to IMCI had not been refined as evidenced by the mixed findings on caregiver comprehension of the range of MCH messages and how to relate them to one another. This is likely due to the late introduction of IMCI into PSI/M’s work plan. Nonetheless, PSI/M clearly took some positive steps toward building partnerships to relay targeted IMCI messages to key audiences. There is great merit in the approach to integrate PSI/M MCH activities funded by USAID, with those funded by other donors. This integration helps to leverage resources and ensures child survival messages are harmonized with other USAID funded programs. RECOMMENDATIONS FOR IMCI (MCH)  Training curricula for CHWs should encompass a comprehensive approach to identifying and treating childhood illnesses. As one example, future programs should improve IPC training for CHWs that promote continuing breastfeeding, giving the child treated water, and/or normal feedings while a child had diarrhea, malaria, or pneumonia. Short￾term/Design FINDINGS: MCH—UTILIZATION OF MCH RESEARCH RESULTS At central level, results of PSI/M’s applied research on various aspects of MCH including IMCI are shared with its partners, including those working with World Health Organization (WHO) and Global Fund to Fight HIV/AIDS, Tuberculosis, and Malaria (GFATM). MOH and other GOM representatives asserted PSI/M MCH research results helped the Ministry to assess and amend MCH policy and develop appropriate implementation strategies. PSI/M research is incorporated into improved MCH care and services, and is used to improve project strategies such as targeting the appropriate audiences for socially-marketed MCH products and BCC campaigns. PSI/M also uses its research findings to improve trainings and IPC messages for implementing partner staff, CHWs and peer educators. Results also are entered into the PSI/M’s Monitoring and Information System (MIS) to use for monitoring and reporting on specific project sub-components. The results of applied research carried out by PSI/M are used as the basis for decision-making on products, communications, and training content for MCH services offered at the community level and at Top Réseau clinics. Research results are available to all PSI staff worldwide on the PSI Intranet. As well, findings are summarized and shared with external audiences through print media, including brochures. CONCLUSIONS ON UTILIZATION OF MCH RESEARCH RESULTS One important result of sharing PSI/M’s research results is they are publically available for use by MOH officials at its discretion. Further, it becomes more likely that MCH care offered at both public and private facilities will be improved by incorporating empirically proven MCH interventions. Once political barriers are lifted, future social marketing programs will be better able to coordinate with MOH and promote better use of MCH research through the public and private sectors. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 19 FINDINGS FAMILY PLANNING AND REPRODUCTIVE HEALTH KI and FGD respondents noted that religious beliefs and pressures were a real obstacle to women attempting to access and using FP/RH products and services. There is a perceived traditional preference to have large families, at least noted by most male respondents. Key informants and female FGD participants frequently mentioned polygamy is an endemic practice. Though not officially sanctioned, it presents another barrier for women to negotiate and control their sexual and reproductive behavior, regardless if they are in some form of a stable union. PSI/M’s goal was to achieve 2,363,949 couple years of protection (CYP) through the sales of contraceptives (excluding condoms) over the life of the project.24Prior to the existence of the youth-focused Top Réseau clinics, adolescent women had difficulty procuring contraceptive products in part due to the lack of targeted services and in part due to traditional customs. Initially, the Top Réseau clinical network was created to provide youth with access to quality information, products and services on contraception. PSI/M conducted a training of trainers for youth peer educators and targeted leaders of youth associations for outreach. The outreach was focused on the promotion of FP methods to ensure the long term health of young mothers. Subsequently, USAID and PSI/M re-orientated the focus of the Top Réseau clinics to promote FP methods for all age groups including women in stable relationships. Additionally, FP counselors were trained and hired to carry out an intensive outreach program to ensure that women who had freely chosen a contraceptive method would continue to receive regular care at Top Réseau clinics (or another recommended facility—public or private) for long term methods such as intrauterine devices (IUDs) and implants; and at pharmacies to obtain short term contraceptives. PSI/M has distributed or re-branded short and long-term FP methods. Confiance—a three-month injectable contraceptive—is the most popular short-term method and PilPlan — an oral contraceptive — is the most popular short-term method. Project activities that supported RH included cervical cancer screening and post-abortion care. Top Réseau providers were trained, in long and short term FP methods, STI management and treatment, and VCT for HIV/AIDS. Some Top Réseau providers are currently promoting IUD insertion and removal which is an in-demand product and service. Data were also collected on stakeholder and end-user experiences and perceptions of the quality of project-supported FP services and products. A number of KII and FGD respondents reported that the availability of discount coupons had made it much easier to access a variety of FP methods at Top Réseau clinics. Some respondents said they could not afford to seek FP, HIV/STI, and MCH, products nor other services in cases where discount vouchers were not available. As vouchers are Box 4: PSI/M FP Achievements ‐Between July 2008 and March 2012, with six months left to end of project, the project had attained 82 percent of the targeted number of contraceptive sales; ‐During the first six months of FY 2012:  Community‐based sales of Pilplan rose by nearly 50 percent which corresponds to 42 percent of the overall annual target for CYPs;  Nearly 300,000 CYPs were achieved through the distribution and promotion of socially‐marketed contraceptives, excluding condoms;    Confiance sales increased overall by 62 percent during the same time period in FY 2011, with pharmaceutical sales tracking about the same.    Community‐based sales doubled as a percentage of all sales—from 27 percent to 54 percent in the first half of FY 2012 compared to rates from FYs 2010‐2011. PSI/M Annual Reports USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 20 for specific products and should not be used outside of the Top Réseau network, some key informants and FGD participants complained that they are not able to use the vouchers at pharmacies when they have prescriptions to obtain contraceptives which are not available at Top Réseau. This led to missed follow-up appointments for Confiance, for example, which are supposed to take place every three months; and led other women to change to less expensive methods, such as pills. Very few of these respondents reported using the cycle beads. The use of socially-marketed FP/RH products and services was also discussed with CSWs. According to them, PSI/M’s approach to CSWs as mothers facilitated its ability to introduce and promote FP/RH products and services. CSWs were most enthusiastic when discussing MCH, diarrhea, IMCI, acute respiratory infections (ARIs), and malaria issues about their children, as well as their use of modern FP/RH methods. They seemed proud of their knowledge as they shared it. CONCLUSIONS ON FP/RH The Top Réseau franchise is now positioned as a network of branded clinics where all family members can seek—and find—health services and some products. Top Réseau clinics have been successful at making FP/RH services more accessible. The prevalent gender inequalities experienced by many women present a serious obstacle to securing FP and a healthy RH life. The availability of vouchers could affect women’s decision whether to commit to short or long terms methods and to use the methods properly. As with other preventive treatments, contraception could be perceived as a less necessary expense when faced with limited purchasing power. PSI/M’s approach to promote FP/RH (and MCH) with CSWs was a successful BCC and IPC strategy and served to set the foundation for ongoing discussions. RECOMMENDATIONS FOR FP/RH  The future project should balance the needs for vouchers and the design of an effective strategy for long term sustainability. Medium Term/Implementation  This project also should consider including more BCC and IPC messages for men of reproductive age about the benefits of spaced births and using condoms for FP/RH. Short-term/Implementation MALARIA PREVENTION AND TREATMENT FINDINGS There has been a national scale-up of prevention interventions, which are largely supported by the GFATM, PMI, and other donors. Through PMI, USAID supports 1) free mass distribution of long lasting insecticide-treated nets (LLINs) every three years; 2) year-round distribution of socially￾marketed LLINs; 3) provision of rapid diagnostic tests (RDTs) at the community level, and 4) BCC activities for malaria prevention and treatment. All these activities will continue to receive support from USAID, with the exception of social marketing of LLINs. As a result, major progress has been made toward achieving high LLINs usage among all individuals and vulnerable groups. LLINs used by caregivers for CU5 increased from 58 percent in 2008-2009 to 89 percent in 2011. Among LLIN￾targeted districts, ownership increased from 72 percent for one or more LLINs in a household in 2008/2009 to 94 percent in 2011. PSI/M was able to add PMI funding to its existing Roll Back Malaria and USAID portfolio under this project. At the same time, all the other donors mentioned above were ramping up their programs, with PSI/M as primary recipient, or a sub-recipient. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 21 Notwithstanding, PSI/M’s malaria product promotion activities yielded somewhat mixed results. Beneficiaries ranked SuperMoustiquaire, a LLIN fourth out of the 15 most used products as reported by respondents. Respondents also frequently acknowledged the necessity of LLINs to prevent malaria and save lives, resulting in high demand for this product. However, respondents reported that there are frequent stock-out even though LLINs are freely available in specific zones during national malaria campaigns. During FGDs with caregivers of CU5, and KIIs with service providers and distributors, respondents reported that there is high demand for the malaria treatments such as the generic brand format of Artemisinin-based Combination Therapy (ACT). FGD participants were able to identify ACTs as the appropriate treatment for malaria, and reported the recent availability of RDTs. Unfortunately, FGD participants reported that chloroquine and quinine are still in use. CONCLUSIONS ON MALARIA PREVENTION AND TREATMENT The coordinated role of multiple donors in support of the GOM’s National Malaria Control Program is one of the best examples of implementing partners working together in Madagascar on a public health problem. PSI/M helped to move forward the national malaria program by providing additional sensitization and training on malaria prevention and treatment using the IMCI approach for CU5. RECOMMENDATIONS ON MALARIA PREVENTION AND TREATMENT  The follow-on project should be tasked with developing and branding new malaria products and launching communication campaigns, if they are to continue their work under the GFATM. Long-term/Implementation FINDINGS: HIV/STI PREVENTION PSI/M supported the wide distribution of socially-marketed female condoms such as Feeling and male condoms such as Protector+. Feeling is distributed through direct sales by IPC agents (CSW peer educators) to sex workers’ associations and individual sex workers. Unlike Protector+, Feeling is not available at the community level (through supply points and CHWs). Like Protector+, Feeling is sold by interested Top Réseau providers (especially those who see many CSW and MSM clients). Uptake of condoms at community level (i.e., through supply points and CHWs) has been challenging due to cultural barriers, stigmatization of condoms, limited interest from the community (and hence limited interest of CHWs to sell/promote them. There was poor uptake of condoms in the community-based supply chain. However in urban areas, peer educators reported the uptake of Protector + condoms was strong and uptake of Feeling condoms was relatively good through their outreach activities. All IPC agents (peer educators) are recruited, trained, and supervised by PSI/M staff. They work in urban areas serving as a referral contact for Top Réseau providers for FP, VCT and STI services. With USAID funding, different teams of IPC agents were created including youth, high-risk men, CSW, and MSM. IPC activities include promotion and sales of male and female condoms to individual target group members, and the referral and promotion of HIV/STI (and FP) services at Top Réseau centers. Peer educators trained by PSI/M were successful at reaching target audiences such as sexually active youth, CSWs, and MSM through associations and clubs such as youth groups. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 22 Through engagement with these associations and clubs, peer educators also facilitated access to service providers such as Top Réseau for these most-at-risk populations. Caregivers and MARPs reported vouchers are an effective incentive for them to access preventative consultations and products. These consultations increased their knowledge of and access to other health services such as for MCH.CSWs and MSM reported that they get tested more frequently when vouchers are available. And youth respondents indicated that they are receptive to PSI/M’s BCC and IPC messages about HIV/STI prevention and treatment. Without PEPFAR funding which ended in September 2012, access to HIV/AIDS prevention services has become limited. The decreased support to peer educators—who provide HIV/STI prevention outreach to MSM and CSWs and other vulnerable groups—resulted in uncertainty about how they will access products and services. As USAID funding for HIV activities wound down in 2012, contracts for all peer educators—with the exception of those in Morondava who work for the public sector—were concluded. Recruitment for peer educator replacements started in the early August 2012, with funding from GFATM. Feeling condoms are to be distributed to CSWs through peer educators once they are hired and trained. This translated into a complete drop-off of Feeling condoms in July and August to practically nil (down from close to 3,000 sales in June and over 4,000 in May 2012. In September, sales picked up again (1,000). During KIIs with peer educators and FGDs with CSWs and MSM, respondents voiced their concern with the dismissal of current peer educators; they were especially worried about a possible gap in receiving vouchers for HIV/STI testing. As mentioned earlier, Top Réseau providers have been trained in STI management and treatment, and VCT for HIV/AIDS. The clinics which provide VCT services are called Top Réseau Plus. All interviewees confirmed that the STI rate in Madagascar is high: about seven percent in the general population; and women represent the majority of the cases. Most interviewees feel that much remains to be done for the promotion of STI prevention and treatment; VCT for HIV/ AIDS; and post abortion care. Some women patients who participated in the FGDs reported that they requested IUDs, but were ineligible because they had existing STIs. These women were told they needed to be successfully treated for an STI before having an IUD inserted. STI treatment usually requires a repeat visits to the clinic which often becomes an economic disincentive to continue treatment. This problem is compounded by limited funding for STI treatments. Vouchers for IUD insertion are covered under a non-USG grant and are only allowed for IUD clients from a certain socio-economic level in Antananarivo. For this voucher program there is no link with STI treatment. During KIIs with five Top Réseau providers in the Mahajanga area, respondents indicated service providers do not have time for counseling on STIs and focus on HIV/AIDS prevention and on treatment only. The findings on gender inequality mentioned earlier in the discussion on FP/RH also pertain to HIV/STI prevention in terms of women’s ability to convince their partners to use condoms. CONCLUSIONS ON HIV/STI PREVENTION Peer educators supported by PSI/M have been successful at reaching MARPs such as sexually active youth, CSWs, and MSM through associations and clubs. Findings suggest this was an effective strategy to disseminate information on HIV/STI prevention and treatment. The strategy of using vouchers for discounts to subsidize HIV/STI testing and FP counseling at Top Réseau clinics brought new clients to use prevention services more frequently. Inequality between USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 23 men and women remains a deterrent to women demanding the consistent use of condoms which have direct implications to HIV/STI transmission. RECOMMENDATIONS FOR HIV/STI PREVENTION  Consider providing STI treatment vouchers for women who come to a Top Réseau clinic for an IUD but whom upon examination are found to have an STI requiring treatment prior to the insertion. The purpose would be to increase the chances the women would continue STI treatment and become eligible for an IUD once the infection has been successfully treated. Medium Term/Implementation  Future programs may want to consider allocating resources for effective messages for male youth and men of reproductive age. Short-term/Work Plan and Budget 4. QUESTION TWO: IMPLEMENTATION OF THE TOP RÉSEAU FRANCHISE The second evaluation questions asked: “How well did PSI/M perform in implementing and working with the Top Réseau clinic franchise?” The response to this question is broken down into its three composite parts: PSI/M’s major achievements in developing private sector services; access and use of services for those most in need; and the sustainability of the franchise. FINDINGS: MAJOR ACHIEVEMENTS IN DEVELOPING PRIVATE SECTOR SERVICES During the first half of 2012, PSI/M reported 23,174 people were treated for STIs by trained Top Réseau providers (76 percent of the annual target), and 8,195 people received quality counseling and testing for HIV (73.2 percent of the annual target).25 Box 5 presents highlights of 2009’s successes. Findings from the desk review and KIIs revealed that since 2001, PSI/M successfully established a Top Réseau network of 213 private clinics operational in 46 districts within 18 regions of Madagascar (the life-of-project target was 220 clinics by the end of 2012).One third of these clinics (70) are located in the Antananarivo urban area.26 Most of them are privately owned, but some belong to associations or FBOs. Overall key achievements presented herein were gleaned from 1) PSI/M Top Réseau coordinators of the eight regional sites visited; 2) 25 Top Réseau providers, and 3) PSI/M service delivery staff at the central level (FGDs). Guided interviews of 25 Top Réseau medical providers reveal that the network is a unique private sector strategy and approach employed by PSI/M that has been instrumental in achieving outcomes. For the most part, Top Réseau providers are satisfied with Box 5: Highlights of Top Réseau Results in 2009 Some Top Réseau clinics reported outstanding results between April and September 20091 :    3,746 youth clients for RH services;  1,546 youth clients for FP counseling;  298 youth clients for STI services;  >5,941 discount coupons distributed by peer educators; and  509 IUDs inserted.   Overall, 46,124 family planning clients were seen among all clinics in 2009. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 24 the support and training they received from PSI/M, and perceived their patients to be generally satisfied with their services. For the past 10 years, PSI/M has invested considerable human and financial resources for capacity building of owners and medical providers. PSI/M offered different types of trainings: initial or refresher training, and clinical training or counseling and other trainings related to clinical and professional development such as management of a health clinic, budget management, stock management. PSI/M offered the Top Réseau providers the choice of two training packages—the basic package which included IMCI, FP (short term contraceptives) and STI and the optional package which included the basic package plus long-term FP methods, VCT, post-abortion care and cervical cancer prevention. All Top Réseau providers were to have received the basic package. However, not all Top Réseau providers were eligible for the optional package because it was contingent upon the provider’s interest to integrate those services into their practice and the availability of the necessary infrastructure and/or PSI/M’s capacity to expand some of the services (e.g., VCT). (For a description of the type of trainings refer back to Annex I). All 25 providers interviewed had been trained by PSI/M in stock management and product use. Each clinic has a memorandum of understanding (MOU) with PSI/M that specifies norms and standards of service quality to be maintained by each provider. Respondents stated that they are regularly visited by PSI/M technical staff for supervision, reporting, and guidance. However the majority of respondents stated that future social marketing programs should provide more frequent guidance visits and more advanced follow-on training to increase their capacity to provide high quality services. Box 6 notes PSI/M’s key achievements with the Top Réseau franchise. KIIs with two Top Réseau medical providers said their clinics carry most PSI/M contraceptives including Protector+ condoms. For key informants in Toliara and Antsiranana, the most well-known and used PSI/M products are: Sur’Eau, Protector+, Pilplan, Confiance, Cura7, and Genicure. PSI/M’s Implanon (a product not branded by PSI/M) is the most popular product overall from the perspective of the provider, as demonstrated by a monthly insertion average of five per clinic.27 From the user perspective, the injectable is the contraceptive they demand most often. The key informant at the Top Réseau clinic in Toliara noted that PSI/M’s distribution system facilitates product availability and there were no major stocks outs; a comment echoed by other interviewees in that region. In general, Top Réseau providers and patients interviewed consider PSI/M supported products to be of good quality at affordable prices. Top Réseau clinics were able to reach a consensus on a 4,000 Ar ($1.80 USD) as a standard service charge for youth clients. However at Top Réseau clinics in Toliara and Mahajanga, some end-user respondents mentioned that these relatively lower prices are becoming out of reach for the poorest segments of the population. Patients who cannot afford Top Réseau are referred to other health care facilities offer more affordable services, some of which are Box 6: Key Achievements of the Top Réseau Clinic Franchise  Rapid growth of the franchisees nationwide due to PSI/M’s investments in human resources and financial support;  Continuous promotion of the Top Réseau brand name through TV, radio and IPC;  Increased access to integrated services and PSI/M for youth, CSWs, MSMs, and women through the expanded reach of Top Réseau clinics;  Quality of services monitored annually by PSI/M site inspections as protocol for renewing MOU with PSI/M;  Top Réseau fees for services are usually lower than those in other private facilities; and  Top Réseau medical providers benefited from PSI/M staff’s high quality training. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 25 even free of charge like the public health facilities such as the Centres de Santé de Base (CSB) and SISAL (a local NGO). Top Réseau key informants revealed that most, but not all, medical providers received training on FP and HIV/STI prevention and BCC. During these interviews, the providers requested follow-on trainings. PSI/M reported of exceeding targets for this type of training. CONCLUSIONS ON PSI/M ACHIEVEMENTS In general, PSI/M collaboration with Top Réseau providers has been fruitful and has facilitated coordination and understanding among the network providers. Those interviewed were satisfied with PSI/M’s support, collaboration, and capacity building, especially in the areas of basic care (FP, STI, and MCH). The management capacity of Top Réseau providers is in question. The Top Réseau network is a complementary solution to the public health care in urban and peri￾urban areas. PSI/M’s promotion campaigns have sensitized providers and increased their knowledge base on service delivery. The medical training delivered to Top Réseau providers facilitated the improvement of quality of services and the uptake of products. FINDINGS ON ACCESS AND USE OF SERVICE BY THOSE MOST IN NEED In order to increase use of services, PSI/M targeted vouchers to the poorer segments of the population. Top Réseau KII asserted these vouchers increased access to integrated medical services and PSI/M products, and in particular to youth, CSWs, MSMs, and women of reproductive age. Some FGD respondents confirmed that they were able to access medical services Top Réseau because the cost of services was lower than other providers.28Top Réseau providers said they saw an increased number of clients among poor neighborhoods using the clinic services after PSI/M activities that promote Top Réseau through TV, radio, and IPC. Still respondents stated that they were not able to pay the cost of some services and products—even with vouchers. PSI/M’s communication strategy demonstrated innovation, creativity and adjustment as attested by project documents, respondents’ input and direct observation. For example, PSI/M respondents reported the popularity of Protector+ condoms and project reports also indicated that there was increased visibility and access to the new Protector+ brand following its extensive promotion campaign and new packaging during FY 2010. Some of PSI/M’s other media campaigns, like the partner reduction campaign “Great Love”, or the “Gasy Band Cool” integrated media campaign (July 2008-January 2009) achieved or exceeded project PMP targets. PSI/M’s studies and annual reports suggested PSI/M BCC activities led to positive behavior change especially related Box 7: Illustrative PSI/M Communication Activities  MVU spots    Production of features broadcasted on national TV  Production of radio and television spots  Brochures and Point‐of‐Sales materials  Banners for stands/presentations  Five‐minute documentary films (for the promotion of IUDs etc.)  Community sensitization support cards, counseling cards etc.  IPC that promotes effective and consistent use of a given product  HydraZinc launch (DTK)    Radio programs: “Trust and Confidence” and “It’s my choice” (for FP communication)  Articles in women magazines “FEMINA” USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 26 to FP and malaria outcomes. PSI/M’s communication and distribution strategies also led to the increased the use of LLINs and oral contraceptives among other methods. Respondents suggested that PSI/M’s social marketing and BCC campaigns mobilized an important number of Malagasy towards healthier behavior; in particular youth aged 15-24 years and women of reproductive age. During KIIs and FGDs, respondents demonstrated that even where their health knowledge about how to use PSI/M product was not correct, they desired greater knowledge about healthy behaviors and increased access to PSI/M supported products. These assertions were supported by the expansion of the Top Réseau franchises to 213 clinics during FY 2010 and FY 2011. In addition, PSI/M considerably expanded outreach and service provision through the CHWs and Top Réseau clinics to vulnerable rural and urban populations including youth, CSWs, MSM, and caregivers of CU5. Box 7 depicts an illustrative list of PSI/M communication activities. Through innovative mass media campaigns and community-level IPC, PSI/M promoted and launched essentials products like Sur’Eau, IUDs, Pneumostop, and DTKs albeit with mixed success due to stock-outs. Notwithstanding, because of PSI/M’s significant expansion of the community-based distribution, sales of contraceptives like Confiance and Pilplan increased considerably.29 CONCLUSIONS ON ACCESS AND USE OF SERVICE BY THOSE MOST IN NEED Since 2008, PSI/M’s various approaches and implementation strategies have shown some demonstrable outcomes, namely, the distribution of health products at subsidized prices to the most disadvantaged segments of the population including those in rural areas. These successes were the result of a mix of innovative communication approaches and the optimization of existing public and private health infrastructure. PSI/M used integrated communication campaigns on prevention and/or treatment. To promote its products, PSI/M used traditional commercial marketing, and community-based distributors in rural areas (community wholesalers and CHWs). PSI/M found creative solutions to promote distribute and sell subsidized branded products. Although some end users were unable to afford medical services, findings suggest PSI/M contributed to increasing access to medical services for the most-in-need, in part because of competitive fees and also because of its sustained BCC efforts. But there is no empirical evidence to substantiate this assumption which is based on qualitative findings. RECOMMENDATIONS ON ACCESS AND USE OF SERVICE BY THOSE MOST IN NEED  Access and use of services should be empirically measured in the follow-on project. If USAID plans to access and use of services in future evaluations, baseline and end-line surveys with counterfactuals should be conducted in order to measure program impact on these two variables. Given the importance that USAID-funded projects achieve agency objectives, this recommendation is a high priority. Other quantitative and qualitative data should be collected that would help to inform areas for improved programing. The metrics used to measure access and use and the consequent budget implications should be discussed between USAID and its contractor. Short-term/PMP USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 27 FINDINGS ON THE SUSTAINABILITY OF TOP RÉSEAU WITHOUT PSI/M SUPPORT Several service providers/clinicians stated that management is not their area of expertise. As well, they complained that PSI/M FP advisors do not perform regular follow-up as they had in the past. One FGD conducted with PSI/M staff at the central level uncovered difficulties they had in supervising Top Réseau management staff as they do not always follow proper accounting and management procedures. Each cluster of clinics has a trained medical supervisor and a regional medical coordinator, who provide on-the-job coaching at least three times per year. In addition, PS/M enlisted Jhpiego, a sub-grantee to provide FP quality assurance coaching support. The vast majority of Top Réseau key informants revealed that PSI/M’s reporting system is time￾consuming and as a consequence produced data of poor quality. Waste management continues to be a problem at Top Réseau sites and, in part, is a direct result of the use of PSI/M products and supplies. While PSI/M trains CHWs on waste disposal, it is outside the scope of PSI/M to directly address problems with waste management. Top Réseau clinics purchase drugs and medical supplies from pharmaceutical wholesalers or the open market. Most Top Réseau clinics have a small pharmacy with essential medical products and supplies. DAMM will not permit PSI/M to provide Top Réseau clinics with any pharmaceutical products. All confirmed that they did not see any conflict with the two supply chains that are believed to be complementary. Some PSI/M products, such as Pneumostop and Super Moustiquaire, are not intended to be available in the Top Réseau sites. In spite of the elimination of Cura 7 from the market by the MOH, demand for this product remains high. The MOH supplies the Organisation Sanitaire Inter-Entreprises de Madagascar (OSIEM) with free condoms and testing kits for HIV detection. OSIEM are regional organized corporate clinics which provide health services to the employees of member enterprises. Many of them are Top Réseau members and as such, they can distribute socially-marketed products. Sometimes they receive free products and distribute them for free to their patients. Nonetheless, socially-marketed products cover most their needs. A few respondents indicated this led to decreased sales of Protector+ condoms. The majority of Top Réseau clinicians interviewed stated that profit margins, PSI/M promotional materials, and financial support are insufficient. Some Top Réseau providers stated that due to administrative procedures, they experiences delays in receiving refunds from PSI/M. Respondents suggested that remunerated motivations (provider vouchers) should be given to Top Réseau clinicians should be made available as they had been provided in the past(a frequent complaint). However, this is no longer permissible except in Antananarivo. In general, all interviewees noted that PSI/M’s educational materials are not available (such as posters and leaflets). For some Top Réseau clinics (like OSIEM in Mahajanga), PSI/M educational (i.e. posters and brochures) and promotional items (i.e., tee shirts) have not been available for more than four years. CONCLUSIONS ON THE SUSTAINABILITY OF TOP RÉSEAU Top Réseau clinics are privately owned and autonomous with nominal supervision by PSI/M and hence not always well managed. Respondent statements about dubious reporting, financial administration, and waste management issues indicate a broader problem of poor management capacity. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 28 In the past, promotional items had been complementary to PSI/M’s communication activities but they did not prove to have any impact on changing behaviors. Faced with budget limitations and no supporting evidence, it was appropriate for PSI/M to scale-down this intervention. Despite the distribution of free condoms to OSIEM and other organizations, there is no hard evidence to suggest that this has much bearing on overall Protector+ sales. Nonetheless, any large￾scale, free distribution of products would likely impact the sales of similar socially-marketed products. PSI/M’s total market approach tracks the potential for these distortions and is poised to mitigate them should they arise in the future. RECOMMENDATIONS FOR THE SUSTAINABILITY OF TOP RÉSEAU  The future project should assess the management capacity of Top Réseau providers and offer technical assistance and trainings based on those findings. This would require an implementing partner which possesses capability statements in human resource capacity building on operations and fiscal management. Medium-term/Implementation Specific examples of management needs include: o Development of business plan(s) for the Top Réseau clinics. o Administrative processes for paying refunds and other financial structures should be identified to streamline administrative processes. The aim would be to increase efficiencies and improve profit margins. o Curricula for Top Réseau clinicians and health service providers should include courses on performance monitoring and reporting, and financial administration.  PSI/M’s existing standard operating procedure manual should continue to be promoted for use by the Top Réseau providers. Continue to encourage good management practices, supporting those providers with a willingness to learn more, especially in the areas of financial controls; human resource development including skills and performance improvement; and administrative systems. Long-term/Design and Implementation  Any future situational analyses should include a focus on the PSI/M reporting system. If needed, the system should be revised to address issues with compliance and quality controls should be built-in to minimize spurious data entry. Oversight in the compliance with reporting should be subsumed within a broader business plan for the franchises. Medium-term/Implementation 5. QUESTION THREE: PERFORMANCE OF SUPPLY CHAIN & DISTRIBUTION SYSTEMS Evaluation Question Three answers: How did PSI/M perform in the supply chain/distribution systems? This section consolidates evaluation question three’s various sub￾questions into four categories. The findings, conclusions, and recommendations when relevant, are presented for each of these categories: 1) achievements and weaknesses in maintaining the supply chain; 2) PSI/M’s contribution to national coverage of health products through the pharmaceutical supply chain (at the wholesale and retail levels); 3) contribution to national coverage of health products though PSI’s distribution system to rural areas; (broadly speaking for commercial wholesale and retail, and specifically for rural areas); and 4) successes and obstacles in PSI/M’s supply chain USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 29 model for community-based distribution. This section concludes with a summary of the stock-out issues within the PSI/M supply chain system by incorporating the findings from evaluation questions one through three. Distributors who are based at the commune level are called “supply points” and they sell socially￾marketed products exclusively to CHWs. These supply points are usually local stores as they are already in the distribution business and CHWs can buy other goods there. Top Réseau clinics can buy pharmaceutical, socially-marketed products directly from pharmaceutical wholesalers and sell them directly to their patients. Non-pharmaceutical products can be supplied directly by PSI/M. Annex J delineates the entire supply chain from international procurement to the end-user. FINDINGS: ACHIEVEMENTS AND WEAKNESSES IN MAINTAINING THE SUPPLY CHAIN In the original Cooperative Agreement, USAID asked PSI/M to deliver products to SantéNet2 at the district level and PSI/M was not supposed to have had any direct relationship with CHWs. SantéNet2 was in charge of distributing the products to the commune level where CHWs could then supply products. In November 2011, USAID determined the distribution system was backed-up at the commune level and at that point decided to use PSI/M’s distribution systems for most of the SantéNet2 communes to ensure the CHWs received the products, and that is when PSI/M started delivering products directly to supply points at the commune level and also started training those supply points’ staff in stock management. PSI/M had to modify its entire distribution strategy in order to focus on the expansion of the community-based distribution coverage. It accepted this shift in responsibility with no additional funding. Hence, PSI/M needed to shift its commercial distribution channel from a “push” system to a “pull” system in order to free-up its distribution team to focus on community-based distribution. PSI/M staff visits normally stops at the supply point level. If PSI visits CHWs, it is to learn about stock issues or inquire about a drop in demand. PSI’s objective is to 1) help create demand from users to CHWs (through communications activities), and 2) create demand from CHWs to supply points (also called in French “PAs” for Points d’Approvisionnement) to ensure a regular flow of product supply. This process expanded to the interventions areas of USAID/Madagascar’s primary health care project (MAFEHA) which began operations in August 2011. PSI/M also supports other supply points and CHWs in rural areas outside SantéNet2 and MAHEFA zones. By the end of March 2012, PSI/M reported 934community-basedsupply points had been established (805 in Santénet2 zones, 65 in MAHEFA zones and the rest outside these two zones).30 The NGOs and the community health centers supervised and trained the CHWs, and PSI/M provided the products. PSI/M meets with the NGOs on a regular basis for planning, coordination, and information sharing, and planning. PSI/M engaged the rural community stakeholders, such as the mayors of the villages and other community leaders, advocating for the necessity of the supply point concept. As a result, most Community Development Committees (CDCs) have given their full support to the community￾based distributors. Some of the CDCs have gone as far as funding supply points’ procurement of PSI/M’s products, and allocating space within the municipality buildings for use by supply points.31 The community-based partners, supply points, CHWs, and the rural end users see the PSI/M community-based supply system as complementary to the GOM supply chain. When products are in short supply in one system, they use the system that has the stock to maintain access to the products they need. This has been particularly useful recently during the political crisis in the country. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 30 As mentioned, PSI/M switched to an inventory control “pull” system where product forecast is based on needs expressed by the community-based supply points and CHWs. This system has the advantage of being demand driven and has very little product expiry. This system helps to build the capacity of the community-based supply points and CHWs in inventory management. Additionally, in an effort to increase the accessibility of products to CHWs in remote areas, PSI/M identified and helped establish supply points there. These supply points can supply products to other supply points based in communes accessible by road (PSI/M does not have transportation means to reach communes inaccessible by 4x4). The project has also emphasized waste disposal in the implementation of the project in the rural communities. All CHWs have been trained in the disposal of the waste generated. Often the community-based supply points and CHWs run out of products because they had mismanaged their finances and did not have money for restocking the product lines. These situations put the project and the various programs at risk, as the rural end users could not find products when they visited the CHWs. The PSI/M distribution team currently conducts supervisory visits to community-based supply points once every two months, a schedule perceived as insufficient by most of the community-based distributors at supply points who were interviewed. These respondents prefer monthly visits for verification of stocks level and proper product conservation.32Some of the community-based distributors at supply points had started to offer services directly to the rural communities— instead of being a distribution point as expected and thereby substituting to the roles of CHWs. Community-based supply points were not trained to deliver BCC and IPC messages that support the proper product use nor do they have regular clients with whom they should conduct follow-up visits. Besides, the agreement signed between PSI/M and community-based supply points specifies that supply points are not allowed to distribute products directly to end users and should exclusively sell products to CHWs. Product storage conditions maintained by both community-based distributors and CHWs were quite poor and not fit for the storage of pharmaceuticals. Products were not protected from temperature fluctuations, which reduces product efficacy. Most CHWs rendered their services on a part time basis, because they cannot rely on the selling of the PSI/M products for their income. The CHWs see their profit margins as being too small for them to survive on. This often meant that when clients visited them, they were not at home. Noteworthy is that the total profit for a CHW is a function of the margin per product times the number of people she serves. There were not enough promotional materials to support the community-based supply points and CHWs in the delivery of their services. The rural communities had no exposure to advertising materials for the various products being offered by the CHWs. According to PSI/M this is done purposely in an effort to avoid distribution of products by supply points directly to end users. During all FGDs with CHWs, respondents voiced concern that communications had failed between the PSI/M distribution team, the community-based supply points, and the CHWs. Respondents believed this lack of communication resulted in stock-outs of various products at the community level until the next regular delivery date. They suggested mobile phones would help alleviate this problem. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 31 The majority of key informants mentioned that deficient sensitization of rural communities on condom use results in low sales of Protector + in all rural areas, which in turn leaves a gap in the prevention strategies for STIs in these communities. Stock-outs of products from the supply chain at the central level of products like Confiance, Sur’Eau, Pneumostop, and Super Moustiquaire, affected the programs negatively at the service delivery level. As reported by most community-based supply points interviewed, some of the stock-outs lasted longer than three months and were due to a variety of reasons, including product registration, manufacturing problems, mosquito nets being largely used for mass campaigns and not social marketing settings. However, community-based supply points sometimes mismanaged their finances, and failed to restock product in a timely manner. CONCLUSIONS ON THE ACHIEVEMENTS AND WEAKNESSES IN MAINTAINING THE SUPPLY CHAIN As noted in the findings from the previous evaluation questions, the project made significant inroads in promoting access to community-based products. The project accomplished a great deal in expanding community-based distribution, improving access of rural end users to the specific health products, training CHWs in the use of health products, training supply points in stock management, and engaging community stakeholders. It has also improved the distribution system by offering a community-based supply system as a complement to the GOM’s supply chain. The demand-driven PSI/M inventory control system has resolved several problems. For example, instances of over￾stocking which had resulted in products expiring before they are sold and used. Through collaboration with SantéNet2 and other NGOs, PSI/M ensured the rural end users got access to the products and that they have positively impacted on the programs targeting the rural populations. PSI/M’s initiatives that fortified community stakeholder participation will likely ensure the sustainability of the rural supply chain. Yet the supply chain experienced a number of weaknesses as well. The community-based supply points who self-selected to serve in the role of the CHW created an operational conflict which was disruptive to the supply-chain process. This is of particular concern because it diminished the full complement of services designed for the end –users (e.g., IPC). Supervisory visits at the community level were too infrequent. Communications lapses between the PSI/M distribution team, community-based supply points and CHWs added to the stock-out problem and contributed to unnecessary interruptions in service provision to the community. Such visits should ostensibly have uncovered the inadequate pharmaceutical storage conditions which threatened the product efficacy. CHWs work part-time and are frequently unavailable which makes their products and services more difficult to access. RECOMMENDATIONS ON THE ACHIEVEMENTS AND WEAKNESSES IN MAINTAINING THE SUPPLY CHAIN  To improve communications that would support an efficient and streamlined supply￾chain, USAID-supported community-based programs should consider subsidizing calling cards for mobile phones. (The purchase of the phones should be a personal expense.) The sustainability of any such support should be a paramount consideration. Explore alternative approaches such as providing an incentive (i.e., calling card) for the USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 32 CHWs who maintain a certain threshold of sales. Other options might include creation of income-generating activities under the auspices of SantéNet2 and/or MAHEFA under the broader rubric of their work with NGOs. SMS technology should also be used for supply points to help them place their order. Short-term/Implementation  Future programs should consider the cost-effectiveness and sustainability of offering on-site supervision of supply points on a monthly basis. Short-term until project phase-out/Implementation  Clearly define and communicate the role of the community supply points and CHWs vis-à-vis the supply chain. Short-term/Implementation  Consider increasing the production of promotional materials to support CHWs in the delivery of their services. FINDINGS: PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THROUGH THE PHARMACEUTICAL SUPPLY CHAIN Wholesale Pharmaceutical Level Pharmaceutical wholesalers, NIPHAR, and CROM presidents,33 mentioned PSI/M has only one private manufacturing partner (NIPHAR) for repackaging and supply to 13 registered pharmaceutical wholesalers, which in turn supply 2,200 retail pharmacies and depots nationwide. National regulations only authorize licensed manufacturers to distribute pharmaceuticals. The majority of respondents believed that there are enough wholesalers to cover the entire country to access retail pharmacies. PSI/M delivers three products (female condoms, IUDs, and implants) directly to Top Réseau clinics. Box 8: PSI/M’s Support to the Wholesale Pharmaceutical Sector OBSTACLES • Actors’ profit margins in absolute terms is relatively low, because price to the consumer must remain affordable; • NIPHAR is the only entity to invoice wholesalers and retailers; • PSI/M has no direct control over pharmaceutical distribution because it does not have a license for direct distribution to pharmacies; • PSI/M budget for operational distribution was insufficient; and • Most pharmaceutical wholesalers are based in Antananarivo. STRENGTHS • PSI/M staff has a solid partnership with NIPHAR and pharmaceutical wholesalers across the country; • PSI/M products are less expensive than similar pharmaceutical products; • Interviewed pharmaceutical wholesalers found the quality and prices of PSI/M products satisfactory. • All wholesalers felt that PSI/M had built their capacity through the courses they received in product use, and advantages; • PSI/M staff visited the wholesalers on a regular basis, at least twice a month, for technical discussions on the various product lines; and • Most pharmaceutical wholesalers interviewed did not have any major stock‐outs in the last 12 months because they had set their minimum stock levels high to prevent supply disruption, linked to the political crisis. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 33 Some towns like Morondava, Toliara, and Tôlanaro have no pharmaceutical wholesalers, and retail pharmacies have to order their products from Antananarivo wholesalers. PSI/M and the private sector distributors decided not to have wholesale operations in these towns because the volumes that would be involved were rather small and it would not be viable to set up a wholesale operation in these towns. The products usually come from Antananarivo by road, resulting in higher prices for the end-user. Also key informants from the pharmaceutical supply chain noted that one of the reasons for temporary stock outs were long distances and poor roads. However some key informants noted that the road network was improving and they anticipated this would result in faster deliveries. Box 8 delineates the main strengths and obstacles in PSI/M’s support to the wholesale pharmaceutical sector. Pharmaceutical Retail Level In most pharmacies visited, all PSI/M products were available and selling well, except for Protector+ and Cycle Beads-Rojo. However, in tourist centers, ports and mining towns, Protector+ sold well. Pharmacies carry all seven PSI/M pharmaceutical products. In towns that did not have any wholesalers, the pharmacies ordered their stocks directly from NIPHAR and, at times, from pharmaceutical wholesalers in Antananarivo. In general, key informants at supply points did not have stocks of expired PSI/M products. Respondents stated that PSI/M training and promotion advisors visited them regularly, usually twice a month. All pharmacies had participated in a PSI/M training on product use in the last year. Overall, respondents from retail pharmacies found PSI/M’s technical assistance satisfactory and PSI/M products affordable and of good quality. The evaluators found that the main obstacles to PSI/M support to the pharmaceutical retail level supply chain included frequent stock-outs. At most of the pharmacies the Team visited, respondents stated that they had experienced a stock-out of Pneumostop that lasted between four to six months. This was caused by delays in deliveries by the international manufacturer, due to regulatory requirements in Madagascar. CONCLUSIONS ON PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THROUGH THE PHARMACEUTICAL SUPPLY CHAIN PSI/M established strong working relationships with NIPHAR and the wholesalers across the country. These partners had favorable perceptions of the quality and value of PSI/M’s products. One of the main obstacles to national coverage of the pharmaceutical supply chain is there is one manufacturer which is in a position to monopolize the market since it is the only entity with a license to distribute pharmaceutical products. Another significant constraint is the limited number of wholesalers outside of the capital which drives up costs in some districts. These increased costs for transportation might make some products unaffordable for the poorer segments of the population who do not live near a wholesaler. RECOMMENDATIONS ON PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THROUGH THE PHARMACEUTICAL SUPPLY CHAIN  The follow-on project should explore ways to keep costs of products down for those pharmacies without access to local wholesalers. One such avenue could be to obtain a license to distribute pharmaceutical products. This would help increase access to the products by the end users. Medium-term/Implementation USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 34 FINDINGS: PSI/M’S CONTRIBUTION TO NATIONAL COVERAGE OF HEALTH PRODUCTS THOUGH ITS DISTRIBUTION SYSTEM TO RURAL AREAS Commercial Wholesale Level The evaluators conducted interviews with 20 commercial wholesalers located in Antananarivo and in the eight regions visited. All of these respondents interviewed found the prices affordable and the quality of products satisfactory. The respondents indicated PSI/M distributes mainly two products (Protector+ and the 150 ml Sur’Eau bottle), and occasionally, the Super Moustiquaire through the commercial supply chain level. This distribution system encompasses 286 commercial wholesalers nationwide and 14,000 retailers across the country. For the past 10 years or more, PSI/M has helped SIGMA, the Malagasy company that produces Sur’Eau, build its technical capacity by developing proper formula and packaging materials and marketing. SIGMA’s CEO stated that he thinks SIGMA needs both working capital and investments in order to increase production capacity. PSI/M does not make advanced payment on orders but makes payment a few weeks after delivery. All 20 wholesalers interviewed attended a PSI/M training course on product use and conservation, and found the course helpful in increasing their knowledge. The respondents had differing opinions on the frequency of PSI/M visits. Most mentioned that PSI/M distribution teams visit them on a regular basis, from once to twice month. Some respondents mentioned that they needed more support to deal with stock-outs. Box 9 provides a summary of the main strengths of PSI/M’s support to the wholesalers. The main obstacles to PSI/M support to the commercial wholesale supply chain system included periodic stock￾outs of certain products. Most interviewees experienced periodic stock-outs of the 150 ml format of Sur’Eau. These stock-outs were a result of production stoppages because of a problem with the supply of bottle tops for the 150 ml format. Stock-out periods varied from one to a few months and PSI/M managers had to wait for deliveries.34 At times there were transportation problems causing delays in deliveries from PSI/M at the central level. Also, temporary stock-outs were caused by delivery delays due mostly to bad roads. There were no long term stock-outs of any PSI/M product lines. The agreements between PSI/M and the wholesalers are periodically negotiated and the wholesalers were satisfied with the terms offered by PSI/M. As mentioned above, key informants mentioned that there were cases where, large amounts of free condoms were distributed and this created unfair competition for commercial sales of Protector+. PSI/M reported there had been a marked drop in those sales figures in the second half of FY 2011. Fortunately, due to increased promotion campaigns in the commercial supply chain, Protector+ sales rebounded, from 214, 206 units sold in January 2012, to 761,570 units sold in March 2012 (256 percent increase).35 Conclusions for Commercial Wholesale Level The consensus among the wholesalers is PSI/M delivers good quality products at affordable prices. As well, they find the PSI/M trainings to be relevant and of high caliber. The causes for the stock￾Box 9: Main strengths of PSI/M’s support to the commercial wholesale supply chain system 1. High quality trainings in product use and conservation; 2. Affordable prices of products; and   3. Satisfactory quality of products. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 35 outs at the wholesale level were outside the manageable interest of PSI/M. However, PSI/M’s current payment system contributes to these problems. Recommendations for Commercial Wholesale Level  Consider the feasibility of payment advances when placing an order with SIGMA. The purpose would be to facilitate its purchase of manufacturing inputs and thereby avert production stoppages. Short-term/Implementation Commercial Retail Level The Team’s visits to 124 retailers (stores, kiosks, boutiques, groceries stores, motels, hotels, etc.) in the nine regions under consideration confirmed that PSI/M developed a wholesale distribution network that supplies retailers with two MCH products (including Sur’Eau 150 ml bottle and Super Moustiquaire (LLIN) and one HIV/STI prevention product (Protector+ condom). Direct observation during the evaluation confirmed PSI/M products are widely distributed at the retail level. These products were found in remote semi-urban settings and sometimes in rural shops36. The main obstacles to PSI/M support to the commercial retail supply chain system included price variations and stock-outs. Most of these 124 retailers price the PSI/M’s products as recommended by PSI/M. However some price variations still existed depending upon the establishment. Also, all of the respondents interviewed mentioned that they had experienced stock-outs of 150 ml Sur’Eau bottles as a result of the problem with the supply of bottle tops as mentioned above. This caused a bottle neck (gridlock) between the wholesalers and retailers of 150 ml Sur’Eau bottles. All of the commercial retail level respondents also complained that promotional materials were generally not available. Conclusions for Commercial Retail Level PSI/M could have performed better in terms of mitigating supply chain issues that might have avoided shortages at the commercial retail level. Better monitoring might have helped SIGMA identify alternative sources for the raw materials needed for production. Recommendations for Commercial Retail Level  Build SIGMA’s capacity to prevent stocks-outs at the wholesale level. The follow-on project should work with SIGMA to examine the causes and solutions to manufacturing challenges. Capacity building efforts should include a regular meeting schedule with the manufacturer to work on forecasting. This could include an assessment and plan for alternative payment terms to facilitate SIGMA’s purchase of raw materials and inputs to help avert production shortages. Medium-term/Implementation  PMPs of the follow-on project should track production of manufacturers at the wholesale level to mitigate problems with shortages at the retail level. The PMP should also track compliance at the retailer level. Short-term/Implementation Rural Coverage of Supply Chain Through innovative mass media campaigns, community level IPC messages, and strong partnership with local and international NGOs (INGO) and private entities, PSI/M generated recognition and the understanding of the importance of essential products like Sur’Eau, and DTKs.37 Through a significant expansion of the community-based distribution via other USAID-funded projects USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 36 (SantéNet2 and MAHEFA), PSI/M considerably increased the sale and consumption of short-term contraceptives Confiance and Pilplan. 38 During interviews with local and international NGOs, as well as community-based supply points and CHWs,39 respondents perceived PSI/M’s support to community-based distribution has been efficient because they have not faced long term stock-outs of health products in rural areas. These partners have different priorities and PSI/M follows and respects the needs and aims of each. For example, SantéNet2 prioritizes the supply of oral contraceptives and injectables, and other products such as Pneumostop, ACTs, and the Viasûr DTK (Viasûr being the DTK dedicated to community￾based distribution and HydraZinc being the DTK dedicated to pharmaceutical distribution). In collaboration with INGOs such as Medical Care Development International (MCDI), and recently with the MAHEFA project, PSI/M supports other supply points and CHWs in rural areas outside SantéNet2 zones. Since 2008, PSI/M was not able to manage the recurring stock-outs of essential products like Sur’Eau, Pneumostop kits, and the Viasûr DTK. However, to alleviate this problem at the community level, quarterly needs assessments were initiated by PSI/M. These assessments are based on a minimum stock threshold for resupply and were submitted to PSI/M’s distribution team in collaboration with local NGOs on the ground and CHWs. In addition, the Procurement and Stock Management Committee (Gestion des Approvisionnementset des Stocks) was established.40 Conclusions of Rural Coverage of Supply Chain Since 2008, PSI/M’s various approaches and implementation strategies have shown some demonstrable outcomes. PSI/M found creative solutions to promote, distribute, and sell subsidized branded products. The Gestion des Approvisionnementset des Stocks should be in a good position to minimize recurring stock-outs and facilitate the regular supply of the products. FINDINGS: SUCCESSES AND OBSTACLES IN PSI/M’S SUPPLY CHAIN MODEL FOR COMMUNITY-BASED DISTRIBUTION The nine LSAT assessments and KIIs41 and one FGD with the PSI/M distribution team at central level revealed PSI/M had three distinct supply chains systems to supply 16 products: pharmaceutical, commercial, and community-based distribution channels. PSI International and USAID are wholly responsible for international procurement. PSI/M meets regularly with USAID to conduct supply and procurement planning sessions using the Pipeline software which is specific for FP products. Bed nets (LLIN) for mass campaigns are procured with PMI and GFATM funds. Bed nets for social marketing purposes are procured by GFATM and distributed by PSI/M in a timely manner. PSI/M provides PSI/International with its forecasted needs and transmits data for procurement purposes. Annex K provides a bar graph that displays the aggregated scores of selected variables from the LSAT guided site observations that the Evaluation Team conducted at the PSI/M central warehouse in Antananarivo and at eight regional PSI/M warehouses including: Fianarantsoa, Haute Matsiatra Region; Tôlanaro, Anosy Region; Morondava, Menabe Region; Antsiranana, Diana Region; Toamasina, Atsinanana Region; Toliara, Atsimo-Andrefana Region; and Mahajanga, Boeny Region. The regional assessments provided qualitative information on PSI’s key central-level supply chain functions. As well, these regional assessments revealed a number of functions devolved from the central level to the regional such as inter-regional management and field coordination.42 The key USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 37 functions are still retained by the central level, however. The end result is one coordinated supply chain system between the central and regional levels, rather than two distinct systems. Over the project life span PSI/M has set up 93443 supply points in rural settings, 213 of which were established in the FY 2012 reporting period, 65 in the MAHEFA areas, and 148 in the SantéNet2 areas.44PSI/M sells directly to community-based supply points who supply each between six to 20 CHWs, who usually live in remote rural locations. CHWs make their way to the supply points to collect the products. Supply points are generally well stocked with all the relevant products that are sold at retail level by the various CHWs. For distribution purposes, the CHWs are divided into two categories, those that deal with childhood diseases and those that deal with FP/RH issues. PSI/M’s distribution teams restock supply points at least once every two months. Although the main roads are generally good, the side roads are not easily accessible, even with PSI/M’s four wheel drive vehicles used to make the deliveries to supply points. As per the Team’s 12 KIIs with community￾based supply points, these deliveries follow schedules that are compiled at the regional levels. The products are subsidized to make them affordable to the community-based supply points, CHWs and the rural end users,45 who usually find the products to be effective and of good quality. PSI has trained the community-based distributors and partner NGOs on stock management. Besides the formal training courses, PSI/M conducts supervisory visits once every two months and uses the on-the-job coaching to build capacity in stock management of the community-based supply points. The community-based supply points all know when to place an order for product replenishment. Some of them were proactive, and went to regional PSI/M offices on their own accord to purchase products instead of waiting for deliveries.46 Conclusions on the successes and obstacles in PSI/M’s supply chain model for community-based distribution Although not envisioned as part of the original Cooperative Agreement with USAID, PSI/M managed to seamlessly implement a community-based distribution system. While there were obstacles uncovered in other points of the supply chain which indirectly impinged supply to the community-based distribution system, the system itself had no major obstacles to report on. SUMMARY OF STOCK-OUT ISSUES IN PSI/M’S SUPPLY CHAIN SYSTEM Stock-outs in the supply chain were the result of a number of reasons, some of which were outside the control of the implementer. Herein are a few worth noting. At the international level of the supply chain, PSI/International and USAID are involved in the procurement and delivery of products to PSI/M. Products are generally delivered on time, however, there were exceptions when delays were caused by local registration requirements that had not been addressed to the satisfaction of the MOH. These delays caused stock-outs in the system as was the case with HydraZinc. The launch of some products had also been delayed because of registration delays at the agency for the registration of medicines. A case in point was Pneumostop. Demand had already been generated in the marketplace and this delay caused stock-outs of the product. As noted before, PSI/M uses a local company, SIGMA, to manufacture and package Sur’Eau. Production of this product has been interrupted because of the supplier did not deliver supplies of the raw materials for the bottle tops needed to package the product. This delay caused stock-outs of Sur’Eau at PSI/M-supported supply points and throughout the system. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 38 At the lower end of the supply chain, a number of local conditions cause stock-outs. Community-based supply points and CHWs said they had experienced stock-outs of certain products because of financial mismanagement. Funds for restocking of products were sometimes used for other purposes resulting in under stocking. Accessibility of certain CHWs for re-supply proved to be a challenge causing stock-outs of the products at that level. Poor communication channels between PSI/M and the community-based supply points also causes temporary stock-outs. In some cases, community-based supply points had to wait two months for the next visit by PSI/M before receiving stock. In some towns there were no pharmaceutical wholesalers, resulting in the pharmacies having to place orders from wholesalers in Antananarivo. These orders at times come by road and, because of poor conditions and the distances involved, there are delays in product delivery causing temporary stock-outs at the pharmacies. At times supplies are transported by air at a high cost, which is passed on to the end users. Conclusions on stock-out issues in PSI/M’s supply chain system PSI/M implemented a comprehensive supply and distribution system with multiple chains and supply points which were supported by a robust and fully functional information system. However, there were numerous shortcomings in the stocking system, detailed in Box 10. Suboptimal inventory control procedures caused delays in the pipeline which, in turn, caused transient stock-outs from time to time. Recommendations on stock-out issues in PSI/M’s supply chain system  Inventory control procedures at both the central and regional levels need to be scrutinized and improved. Short-term/Implementation  The future project should explore options with SIGMA to secure at least two prequalified suppliers of caps. This would help to address operational issues. Short to Medium-term/Implementation  Examine ways to incrementally improve logistics through utilization of bi-annual LSAT assessments to track improvement. The supply chain actors have experienced the LSAT being applied to their operations during the evaluation and are likely amenable to future LSATs. Short to Medium-term/PMP  Establish a commodity security plan for branded products at all levels of assistance to the supply chain. This potentially small investment could go a long way to ensure that products are available on a medium- to long-term basis, and should be done with the participation of all donors and partners. Medium-term/Work plan  Explore an incentive-based system for the community-based supply points and CHWs. This system could introduce a bonus scheme that would reward the highest performers in the system with cash or gifts such as bicycles and/or raincoats. This Box 10:  Summary of Stock‐out Issues  Delays caused by local registration requirements;    Production delays of Sur’Eau due to of a lack of supplies;    Financial mismanagement by Community‐based distributers and CHWs;  Insufficient communication between PSI/M and the community‐based stakeholders; and    Shortage of local pharmaceutical wholesalers. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 39 recommendation pertains to SantéNet2 and MAHEFA, not a social-marketing project. Short￾term/Design  Automate the regional logistics system and link it with the head office so the system will be unified. An automated system would allow for faster and more accurate product scanning at the community-based distribution level by using handheld devices, and would allow this information to be downloaded directly into the Logistics Management Information System. Also to be considered is SMS technology. This intervention should be part of a larger business management training strategy for the supply points. Long￾term/Design 6. QUESTION FOUR: OTHER CROSS-CUTTING ISSUES This question is focused on five distinct areas that cross-cut the PSI/M project and reflect its ability to work within the national context. These questions also respond to PSI/M’s ability to collaborate with other partners in the broader development context in Madagascar. PSI/M contributions to the achievements of other partners are covered under the fourth sub-section which discusses leveraging partnerships with local and INGOs. More specifically, the five areas covered within are: 1) Capacity building efforts with local organizations 2) Harmonization with national programs 3) GOM perceptions of PSI/M’s outcomes and future needs 4) Leveraging partnerships with local and INGOs vis-à-vis distribution support 5) Influence of the political situation on PSI/M’s performance FINDINGS: CAPACITY BUILDING EFFORTS WITH LOCAL ORGANIZATIONS PSI/M implemented a comprehensive and inclusive capacity building program which took its shape primarily in trainings and on-the-job coaching. The program covered all the integral actors from the manufacturer down to the providers. Once again, we refer the reader to Annex I for specific details on the trainings. Following are brief summaries which provide the gist of the capacity building efforts according to the type of stakeholder: Commercial Manufacturer PSI/M built the manufacturing capacity for SIGMA, the local supplier of Sur’Eau, by providing the formulation of the product and logistics of the distribution of the product. Although SIGMA continues to have raw material sourcing problems on the international market, there has been a marked improvement in the availability of the raw materials (bottle tops) in the last few months. PSI/M owns the Sur’Eau brand name but SIGMA owns the formula developed with PSI/M’s technical assistance in 2004. Not only did PSI/M reinforce SIGMA’s Research and Development technical capacity, but PSI/M provided logistical support for deliveries of Sur’Eau both at commercial and community levels. PSI/M has been SIGMA’s largest client of Sur’Eau. In 2004, SIGMA produced 450,000 Sur’Eau bottles per year. In 2012, SIGMA produced three million Sur’Eau bottles per year. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 40 Commercial Wholesalers PSI/M has trained all wholesalers (286) countrywide in stock management, which has improved their businesses as attested by all the wholesalers interviewed. PSI/M has also provided promotional materials to them when they were in supply. Pharmaceutical Manufacturer PSI/M held regular meetings with NIPHAR to coordinate production and distribution issues. NIPHAR’s capacity was also augmented through the provision of pharmaceutical products and packaging materials. Pharmaceutical Wholesalers PSI/M trained 13 wholesalers countrywide in product use and stock management. PSI/M technical advisors visited these sites on a regular basis, at least once a month to discuss product use and stock management. Pharmacies and Depots PSI/M technical and sales staff made regular visits, and advised on product use and stock management. Retailers PSI/M staff routinely visited retailers encouraging them to order products from wholesalers on time and distributed promotional materials. Community-based Distributors PSI/M conducts supervisory visits to all community-based supply points (934) countrywide at least once every two months. All community-based supply points have been trained in stock management and they are given refresher training during the sales/supervisory visits. They received three trainings in total from PSI/M—in 2010 and 2011 in the SantéNet zones, and a training in 2012 in MAHEFA zones. NGOs PSI/M has also trained a number of NGOs in product use and stock management. NGOs held monthly consultations with SantéNet2 and MAHEFA to discuss forecasting issues. (NB: PSI/M’s partnership with JSI and Research Triangle Institute, International (RTI) are discussed in more detail in the following sections.) CHWs CHWs have been trained in product use and stock management by the NGOs they work under and some were directly trained by PSI/M. A total of 19,355 CHWs were trained over the life of the project—80 percent of whom were trained under the auspices of SantéNet2 (62 percent) and MAHEFA (18 percent). Top Réseau Providers All Top Réseau providers have received at least the basic training package (e.g., IMCI, short term FP methods, and STI). All 213 Top Réseau clinics have had at least one staff member trained in product use and stock management. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 41 CONCLUSIONS ON CAPACITY BUILDING EFFORTS WITH LOCAL ORGANIZATIONS PSI/M’s capacity building was conveyed to partners through various training sessions, supervisory visits, technical/coordination meetings, and technical assistance in the manufacturing of products have resulted in a functional and reliable supply chain that ensured delivery of quality products to end users. Stock-outs were kept to a minimum and when they did occur, they were result of regulatory delays, manufacturing problems and delays with internationally procured products delivery (as were described in more detail in evaluation question three).47 FINDINGS: HARMONIZATION WITH NATIONAL PROGRAMS The two distribution systems—one implemented by PSI/M and the other by the public sector— were widely seen by all stakeholders to be a complementary system and not a parallel one, in part because they provided alternative sources (and, at times, pricing structures) for the health products. SALAMA is a quasi-public entity established in 1996 by GOM for the central procurement and distribution of medicine to the public, NGOs, and FBOs. SALAMA is also the main sub-recipient of the GFATM for the financial management of sub-recipients for the National Strategy Application (NSA) project. PSI is the Principal Recipient for Rounds RCC4 and Seven (malaria) and Eight (HIV/AIDS). For the two malaria grants, PSI works very closely with the government’s NMCP. For the HIV grant, the Comite National de Lutte Contre le VIH/SIDA (GOM) is also a Principal Recipient and PSI/M coordinates and supports them extensively in HIV coordination and implementation. PSI/M’s community distribution to the private sector is complementary to SALAMA’s community distribution to the public sector which has become increasingly underfunded and less functional in recent years. While there are no direct links between the two distribution systems, PSI/M coordinates with SALAMA. PSI/M has also been able to address the gaps in coverage of the National FP Program through its Top Réseau clinic supply chain and pharmaceutical supply chain which distributed products to the end users nationwide. PSI/M’s community distribution supply chain worked to prevent HIV/AIDS/STI prevention through the distribution of Protector+, in concert with the national program. With the addition of IMCI to project objectives in FY 2011, PSI/M added the distribution of LLINs, DTKs, and pneumonia treatment kits to the existing community-level supply chain, and extended those supply chains to new partners further along the distribution network. CONCLUSIONS ON THE HARMONIZATION WITH NATIONAL PROGRAMS PSI/M has been able to work in harmony with the public sector supply chain system through its community-based supply chain. It collaborated in a complementary manner with national programs such as the NMCP, and helped to fill the GOM’s gaps in service provision and therefore, contributed to the provision of health products in all the sectors of society. FINDINGS: GOM PERCEPTIONS OF PSI/M’S OUTCOMES AND FUTURE NEEDS All interviewed respondents based in regional and community health centers perceived project￾supported supply chains to be complementary and sustainable.48 Key informants from the MOH USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 42 shared positive opinions about the PSI/M community-based supply chain and considered PSI/M a valued and collaborative partner. Many did suggest that more supervision of CHWs is needed. In the medium term, MOH officials would like to see measures put into place which would prevent resistance to the antibiotics being dispensed by CHWs. Agence du Médicament de Madagascar (AGMED) is the GOM authority for the registration of all health products. It works with Agence Française de Santé de Produits Sanitaires (AFSPS), the WHO and Club Sida. AGMED has an evaluation committee that supervises quality control, inspection of products for conformity and the provision of authorization for product commercialization (usually for five years). AGMED provides quality inspection services and collaborates efficiently with PSI/M for pre-product distribution. PSI/M complies with AGMED’s requirements involving the committee’s validation of all PSI/M’s BCC and promotion messages before distribution in mass, mid or IPC media. AGMED representatives thought PSI/M should have met with them more often in order to prevent delays in product registration. In the immediate future, they wanted to get an understanding of the broader scope of PSI/M distribution activities. In the medium term, they would like to know who is dispensing PSI/M’s products so as to ensure they are in compliance with national norms. Some efforts were made by PSI/M to build local capacity through on-the job-training of key district level public collaborators (e.g. CSBs).49 The Office Malgache de la Propriété Industrielle (OMAPI) is pleased with its partnership with PSI/M and consider it as one of its major clients. OMAPI’s confidence in PSI/M is demonstrated by it having registered a number of brand names with them. In the immediate and medium future, OMAPI officials would like to see more collaborative interaction with PSI/M, such as they did when they had joint media publicity campaigns to promote socially-marketed products together with OMAPI activities. CONCLUSIONS ON GOM PERCEPTIONS OF PSI/M’S OUTCOMES AND FUTURE NEEDS On the whole, GOM agencies perceive PSI/M as a useful partner in reaching end users of health services and products. They view it as a collaborative partner and its supply chain as complementary to the public supply chain. However despite AGMED being a pivotal GOM agency for PSI/M, communications with them could have been better, or at least more frequent. More frequent communication would have likely minimized interruptions and delays in the introduction of new products. FINDINGS: LEVERAGING PARTNERSHIPS WITH LOCAL AND INGOS VIS-À-VIS DISTRIBUTION SUPPORT Interviews with local and INGOs, as well as community-based distributors, and CHWs, indicate that community distribution with PSI/M has been efficient because they have not faced long lasting stock outs of health products in rural areas. Partners have different priorities that PSI/M follows and respects. PSI/M has worked closely with both local and INGOs and firms (RTI/SantéNet2, and more recently with JSI/MAHEFA starting in June 2012)50 to distribute their products to the rural end users. Through its NGO partners, PSI/M realigned its activities under the project, providing communications campaigns and materials, and trainings as well as product distribution for over USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 43 4,000 CHWs, and supply points. SantéNet2 has 805 community-based distributors, MAHEFA has 65 community-based supply points, and local NGOs have 64 community-based supply points. Countrywide there are currently 15,400 CHWs, and this likely increased in number by September 2012, as the MAHEFA project became fully operational. PSI/M distributes its products through these implementing partners (not all of which are NGOs) to get to the rural end users. PSI/M also had regular meetings with them to coordinate the distribution activities. Through the provision and distribution of its products, PSI/M has been instrumental in contributing to the achievement of NGO partner goals to provide services at the community level. Although there have been occasional stock-outs of various products, it was only for short periods of time, and interruptions to the service delivery were kept to a minimum.51 In general CDCs were appreciative of the roles CHWs were playing in the rural communities and perceived them to be key stakeholders to achieve the GOM’s goal to providing health-for-all in the rural communities. PSI/M’s partnership with RTI helped to facilitate the latter’s implementation of SantéNet2, USAID’s community-based project. RTI has four regional offices and manages 16 NGOs, all of which received PSI/M training on stock management and product utilization (e.g., Action Santé Organization Secours -ASOS, Association ZETRA, etc.). For community-based distributors, PSI/M supplied essential products like ACT and contraceptives via supply points in rural areas. For the Santénet2 project, priorities in supplies were placed on oral contraceptives and injectables and other products like Pneumostop, Viasûr, and other malaria and diarrhea treatment products. PSI/M worked in 833 communes, 800 of which were located in Santénet2 zones. In the past four years, PSI/M supplied nine health products (branded and generic)52 at supply points in Santénet2 zones. For Santénet2, PSI/M forecasted, procured and distributed these nine health products to communities via supply points and community-based distributors in rural areas. Santénet2 and PSI/M have complementary distribution systems of health products. PSI/M supplies subsidized products directly to community-based distributors at supply points, while Santénet2- supported NGOs provide technical assistance to CHWs who purchase PSI/M products at supply points. RTI estimates that Santénet2 is serving around 120,000 women in rural Madagascar and estimates on average each CHW attends 20 rural women per month.53 While the actual partnership between PSI/M and JSI started in June 2012, PSI/M had been supplying products to CHWs working in many of the MAHEFA zones in 2011.The MAHEFA project is fairly new (less than a year old) and PSI/M has been quite responsive by increasing the number of supply points, retailers and community outlets in MAHEFA’s nine regions. As with Santénet2, MAHEFA NGOs receive commodities from PSI/M. JSI is in charge of the planning, coordination and forecasting of MAHEFA’s activities, while PSI/M is in charge of warehousing, distribution and training of community-based distributors. At the communal level, MAHEFA supports the CHWs’ purchase of PSI/M products at supply points. As with Santénet2, MAHEFA interfaces with PSI/Mat three levels:  PSI/M identifies needs in MAHEFA zones in order to define activities to be conducted with local NGOs;  PSI distributes health products via supply points to CHWs;  Local NGOs provide technical assistance to CHWs who purchase PSI/M health products at supply points in MAHEFA zones. Despite the good relations with implementing partners, there was evidence of a gap in PSI/M’s ability to coordinate its activities among project partners. NGO partners on PSI/M indicated that USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 44 stock-outs adversely affected the performance of their activities which are dependent on the availability of health products for PSI/M. PSI/M’s partnership with Fianakaviana Sambatra (FISA), a local NGO promoting FP in Madagascar for the past 45 years, has been effective for the provision of HIV/STI related treatment products for youth in particular. FISA’s peer educators found the PSI/M training satisfactory and they also commented on occasionally receiving free male condoms from PSI/M. CONCLUSIONS ON LEVERAGING PARTNERSHIPS WITH LOCAL AND INGOS VIS-À-VIS DISTRIBUTION SUPPORT In spite of political restrictions, PSI/M was successful in maintaining its collaboration with public institutions for the distribution of health products, especially at the regional and district levels. Thanks to PSI/M, SIGMA has acquired the technical skills to develop appropriate formulas and appropriate packaging for Malagasy people’s needs. SIGMA’s success has been impressive. Largely because of PSI/M’s aggressive promotion and distribution activities, nationwide demand for Sur’Eau is much larger than SIGMA’s production capacity.54 PSI/M’s partnership with SIGMA has been very efficient, complementary and instrumental in reaching outcomes in MCH. PSI/M has raised people’s awareness of the advantage of using Sur’Eau and of clean water not only in urban, semi-urban and rural areas. PSI/M has successfully leveraged partnerships with NGOs to ensure that their products reached rural end users. It was instrumental in the achievement of the partner NGOs service delivery goals and the NGOs were very positive about the partnership. As well, PSI/M instituted a sustainable community supply chain through involvement of the supply points and CHWs. FINDINGS: INFLUENCE OF THE POLITICAL SITUATION ON PSI/M’S PERFORMANCE As mentioned in the Introduction to this report, following the political crisis of 2009, USAID and PSI/M realigned its activities, and considerably expanded the implementation of major programs using communication and distribution strategies to meet most of its social marketing targets in MCH, FP/RH, malaria, HIV/STI, supply chain and cross-cutting processes and systems. These activities continued at urban Top Réseau facilities and added peri-urban clinics, and extended outreach into to rural areas with a package of products and services designed for residents in remote communities. To meet this new mandate, an additional $4,950,000 was obligated the budget realigned in 2009 to adjust for the additional effort to work at the community level .The aim was to extend social marketing of proven products as a complement to those already available at public health facilities. At existing urban Top Réseau facilities, products and services continued and some even increased. These same services and products, however, were made available at newly branded Top Réseau clinics in peri-urban locations around those sites where Top Réseau was already present. The second—and most radical change in the project during the post-crisis—was the extension of quality, socially-marketed products and services to rural communities, where the majority of Malagasy live. The addition of community-based programming significantly realigned and changed the nature and content of the project’s interventions, and added others. Normally, these changes to the project’s critical assumptions and underlying logic would have been reflected in changes to both the Office of Health, Population and Nutrition (HPN) and Mission PMPs, and had there been one, the project’s Results Framework. Instead, changes were made to the USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 45 project’s logical framework in 2011 and 2012, providing revised activity and output indicators, and adding them for new project components and sub-components. These were then used in monitoring performance in MCH, malaria, IMCI, HIV/STIs, in the Top Réseau social franchise to a lesser degree, and related changes to and extensions of, the supply chain. CONCLUSIONS ON THE INFLUENCE OF THE POLITICAL SITUATION ON PSI/M’S PERFORMANCE PSI/M seamlessly adapted to the local circumstances following the 2009 political crisis and adopted new modes of operation with no discernible interruption in its services. The project continued to make gains in access to its products and services by working in collaboration with local and INGOs while at the same time, maintaining the needed communications with the GOM. PSI/M maneuvered through those turbulent times with acumen, finesse and flexibility.                                                              ENDNOTES 5 UNDP, Human Development Index 2010, New York, New York: UNDP 6 World Bank (2011) World Development Indicators database - databank.worldbank.org 7 National Institute of Statistics (Madagascar), ICF Macro. Madagascar Demographic and Health Survey 2008-2009. Calverton, United States: ICF Macro, 2010 8 World Bank, Africa's Future, Africa's Challenge, Early Childhood Care and Development in Sub-Saharan Africa. Editors Marito Garcia, Alan Pence, and Judith L. Evans, © 2008 The International Bank for Reconstruction and Development / The World Bank. National Institute of Statistics (Madagascar), ICF Macro. Madagascar Demographic and Health Survey 2008-2009. Calverton, United States: ICF Macro, 2010. 9 World Bank, Africa's Future, Africa's Challenge, Early Childhood Care and Development in Sub-Saharan Africa. Editors Marito Garcia, Alan Pence, and Judith L. Evans, © 2008 The International Bank for Reconstruction and Development / The World Bank. National Institute of Statistics (Madagascar), ICF Macro. Madagascar Demographic and Health Survey 2008-2009. Calverton, United States: ICF Macro, 2010 10 Top Réseau is a franchise network of clinics focused on providing youth-friendly RH and child survival services. The original objective of the Top Réseau clinics was to supply youth-friendly, high quality, and affordable RH services to young people aged 15 to 24. After the first 10 years of the Top Réseau brand, PSI/M decided to revise the branding and service delivery model to reengage clients and stay competitive as a health care services network. In August 2010, PSI/Madagascar created a new marketing plan for the Top Réseau network and decided to integrate other health areas to benefit more clients and to increase the geographic coverage of the network. 11 On March 17, 2009, after demonstrations in the capital, President Ravalomanana signed power over to the military, which in turn conferred the presidency on opposition leader Andry Rajoelina, the mayor of Antananarivo and leader of the demonstrations. Rajoelina declared himself “President of the High Transitional Authority” (HAT) and pledged to hold presidential elections by October 2010 (a pledge that he did not fulfill), following a constitutional referendum and revision of the electoral code. The United States condemned the unconstitutional and undemocratic change of power in Madagascar and considers the series of events of early 2009 that led to the installation of the de facto leadership to be a military coup d'etat. On September 17, 2011, representatives of most of Madagascar's major political factions signed a "Roadmap for Ending the Crisis in Madagascar," endorsed by the Southern African Development Community (SADC), aimed at ending the long political crisis through the formation of a more inclusive, power-sharing interim government that would prepare the country for elections. The United States considers the series of events in Madagascar in early 2009 to be a military coup d'etat and, as a result, has suspended all assistance programs that directly benefit the government as well as all non-humanitarian assistance to Madagascar. http://madagascar.usaid.gov/crisis-in-madagascar 12 During in-country work, the team conducted guided interviews with 13 USAID personnel, 27 Top Réseau medical practitioners, six GOM officials, eight CROM Presidents, 35 PSI/M senior management, monitoring and evaluation (M&E), financial administration, and technical staff, both at central and regional levels, 25 representatives of sub￾awardees and other implementing partners, 35 end-users, 20 CHWs, eight pharmaceutical wholesalers, 12 community based distributors and supervisors at supply points, and 20 commercial wholesalers. The team also conducted site observation visits at 124 commercial retailers. The team conducted nine site observations guided by the LSAT, one at USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 46                                                                                                                                                                                                 PSI/M Central level and eight at PSI/M regional level, in order to examine supply chain management and support activities. 13The team collected data in Antananarivo, Diego, Antsirabe, Toliara, Fianarantsoa, Mahajanga, Tôlanaro, Morondava, and Tamatave. 14(n = 50) Other results included: % of people who used 1 cap to treat (10-15l) bucket of water = 81.5% and % of people who waited more than 30 minutes before using the treated water = 67.5%. 15 USAID PSI Social Marketing Project Semester Report, October 2011 -March 2012 Cooperative Agreement # 687-A￾00-08-00032-00, April 30, 2012, p. 2 16 PSI Social Marketing Project, USAID Fiscal Year 2011 Annual Report Cooperative Agreement # 687-A-00-08-00032- 00, November 30, 2011, p. 2 17PSI Social Marketing Project, USAID Fiscal Year 2011 Annual Report Cooperative Agreement # 687-A-00-08-00032- 00, November 30, 2011, p. 2 and an update from PSI/M senior management in January 2012. 18 Some CHWs who participated in FGDs said they would like to be able to trade in expired Sur’Eau for a comparably priced product that sold well, but this issue was not resolved during FGDs, though it was raised in interviews and observation visits by the Supply Chain Specialist (please see the discussion under Evaluation Question 3). 19 PSI Social Marketing Project, USAID Fiscal Year 2011 Annual Report Cooperative Agreement # 687-A-00-08-00032- 00, November 30, 2011, p. 2 20 PSI Social Marketing Project FY 2012 Semi Annual Report 21 Please refer to the Table in Annex E BCC Data Tables from KIIs and FGDs. 22 KIIs with PSI/M staff and with Medecins Chefs at the communal level doctors who are clinic administrators 23 A qualification to this observation is that project-introduced terms and phrases like “MSM” have been adopted by PEs, and MSM associations, whose members say they prefer MSM to “pede” or “homosexuel,” both of which they feel are pejorative, if not always by Top Réseau Coordinators and other PSI staff. 24 PSI Social Marketing Project FY 2012 Semi Annual report 25 PSI Annual Report, October 1,2008-September 30,2009 26 All Top Réseau clinics are located in urban and semi-urban areas: 70 in Antananarivo, 25 in Antsirabe, 21 in Fiarantsoa, 12 in Mahajanga, 11 in Tôlanaro, 9 in Morondava, 27 in Tamatave, 28 in Northern Region including six in Sava and 22 in Diana 27 Average calculated from 27 guided interviews of Top Réseau providers 28 Corresponds with project collected data (Social Marketing Project Annual reports for FY 2010 and FY 2011, Social Marketing Project First Quarter Report FY 2012) 29 See PSI’s annual reports, FY 2009, 2010, 2012,and first semester 2012 30 PSI USAID Semester report October 2011-March 2012 31 Findings from 12 guided interviews of community based distributors 32 Findings from guided interviews of 12 community based distributors and 20 FGD with CHWs. 33 These KIIs include eight pharmaceutical wholesalers and PSI/M partners like NIPHAR, and eight CROM presidents 34 Findings from guided interview of SIGMA key executives in June 2012 and confirmed by guided interviews of commercial wholesalers in Diego, Toliara, Mahajanga and Tamatave in June and July 2012. 35 Please see sales figures for Protector+, mentioned on page 26. 36 Protector + and Sur Eau were found in shops located in small villages in each rural visited 37Actipal and Pneumostop were not available for four to six months at a time over the past year alone. (PSI staff comments).Actipal is not funded by USAID but PSI/M distributes Actipal with the support of other donors. 38 See PSI annual reports FY 2010, 2011, and 1st FY2012 semester Actipal and Pneumostop were not available for 4-10 months at a time over the past year alone. 39 12 community based distributors and 20 CHWs were interviewed during the team’s field visits in rural areas 40 See PSI/M Semester October 2011-March 2012, page 23; the Gestion des Approvisionnementset des Stocks Committee (GAS) was established in 2012 to help face recurrent stockouts. 41 Guided interviews of 25 Social Marketing Project sub-awrdees and implementing partners, 12 community based distributors at supply points, eight pharmaceutical wholesalers and 20 commercial wholesalers, and 25 Top Réseau providers 42 The team visited two PSI/M interregional offices (in Diego and Tamatave) and six Regional Coordinators (Toliara, Mahajanga, Antsirabe, Fiarantsoa, Tôlanaro, and Moronda) 43 See page 34, PSI’s USAID Semester Report October 2011-March 2012 44 See page 34, PSI’s USAID Semester Report October 2011-March 2012 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 47                                                                                                                                                                                                 45 Findings from eight FGDs with CHWs and KIIs with 35 end users 46 Findings from guided interviews with 12 community based distributors. 47 The supply chain management is discussed in more detail in section three, both in terms of stock-outs and actions taken by PSI to improve distribution. It was not possible to provide an average with the LSAT data collected. 48 ‘Sustainability’ is discussed in more detail in Section Five: Conclusions. It is likely that the respondents here considered that the process was sustainable, now that participants were trained and understood ordering and quality control. Subsidizing prices for health products that are otherwise not already distributed on this scale is, obviously, not sustainable. There was no discussion of cost recovery measures in the project, although there was already considerable concern that the economic situation was placing even the heavily subsidized projects out of reach of the target population (for example, youth). 49 Information collected from Team A guided interviews with Direction Régionale de la Santé (DRS) and CSB medical practitioners in Mahajanga and Tamatave 50 Information obtained from the synthesis of KIIs conducted in Antananarivo, June 20-22, 2012. 51 While some of the stock outs have lasted for months, the positive perception of the products and their availability is striking in the FGDs and KIIs. Either this was viewed as ‘normal’ in the local context, or respondents were somewhat isolated from those stock outs (they didn’t need those products during a stock out, or the stock outs didn’t occur where they were). 52 These nine products include: Actipal (zaza and zazakely), Confiance, Viasûr, Pilplan, Pneumostop, Protector Plus, Sur’eau, LLIN, and RDT. 53 Information from the guided interview with RTI Country Director, June 2012 54 Guided interview with SIGMA Director, June 2012 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 48 ANNEXES ANNEX A: PSI MADAGASCAR SUB-AWARDEES PSI’s USG Sub‐ Awardee FY 2008‐ 2012 Description VS : VOAHARY SALAMA PSI/M signed an agreement with Voahary Salama (VS) in February 2009. VS provides training and support to community health workers (CHWs) in four communes of Moramanga: Sabotsy Anjiro, Ambatovola, Amboasary Gara and Andaingo. PSI/M provides support to social marketing activities including training of CHWs, product supply and resupply, and interpersonal communication activities. Trained CHWs conduct educational sessions among mothers and caregivers of children under five on malaria prevention and treatment, nutrition, pneumonia and family planning. The role of VS is to disseminate and share lessons learnt, and to help build capacity of local partner organizations and CHW networks. Activities in FY 2011 included more than 4,500 group sessions and more than 8,000 home visits on Family Planning and Integrated Management of Childhood Illnesses (IMCI). GOLD: GROUPEMENTS ET ORGANISATIONS LOCALES DE DEVELOPPEME NT PSI/M signed an agreement with GOLD in September 2010 to continue activities initiated under the POUZN program (Point of Use Safe Water and Zinc), which ended in November 2010. Activities continued for five months and focused on IMCI in five communes in Betafo District (Vakinankaratra region). GOLD is the only health NGO working in this area. Due to the late arrival of the Pneumonia treatment product (to be launched in November 2011) and delays with the validation of the CHW training curriculum on malaria, CHWs focused on diarrhea prevention and treatment. More than 80 CHWs received training and products for diarrhea prevention and case management. A no cost extension was recently signed for five additional months of project implementation (November 2011-March 2012). Main focus will be the introduction and promotion of the smaller Sur’Eau bottle for community based distribution. MCDI: MEDICAL CARE DEVELOPMENT INTERNATIONA L PSI/M signed an agreement with MCDI in 2009 for the start of the POUZN program (Point of Use Safe Water and Zinc), which had as one of its objectives to increase use of Oral Rehydration Salt (ORS) and zinc for treatment of diarrhea at community level. To this end the project trained over 1,000 CHWs and supplied them with OSR/zinc treatment kits. In an effort to integrate management of childhood illnesses across intervention areas, MCDI supported CHWs also received training in malaria treatment with funding from the Global Fund Round 7. MCDI areas of intervention under the USAID funded partnership are concentrated in eight districts in the South West, with 97 out of 105 communes covered. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 49 PSI’s USG Sub‐ Awardee FY 2008‐ 2012 Description MCDI no cost extension While the agreement was initially set to end in September 2010, a no cost extension was proposed and approved in October 2011. The no cost extension is for five months (November 2011-March 2012). The main reason for the no cost extension is to introduce the use of a new algorithm focusing on integrated case management of children under five with malaria, diarrhea or pneumonia. The algorithm will be piloted for use by MCDI trained CHWs. To date, PSI/M has supported the training of more than 770 CHWs in IMCI. SALFA: SAMPAN'ASA LOTERANA MOMBY NY FAHASALAMAN A In the second semester of FY 2011, PSI/M established a sub-award agreement with the Health Department of the Lutheran Church (SALFA), and identified five remote communes where community-based activities were implemented. PSI/M trained SALFA trainers/supervisors in family planning, and reinforced their management, monitoring and evaluation capacity. These trainers/supervisors in turn identified, trained and supported 65 CHWs for family planning provision including standard day method, condom use and oral contraceptive pills. After five months of activities, the 65 CHWs had reportedly recruited more than 500 new family planning users. In FY12, PSI/M will reinforce the partnership with SALFA in order to respond to rural women’s demand for modern contraceptives. New CHWs in the five remote communes will be recruited and trained to reach a total of 155 trained for community-based distribution of products by end of project. SALFA trainers/supervisors will be trained in injectable contraception provision and will subsequently conduct trainings of 80 CHWs on this method. WCS: WILDLIFE CONSERVATION SOCIETY PSI/M signed an agreement with WCS in 2008. At the end of FY 2011, project activities covered 16 communes in four districts (Maroantsetra, Mandritsara, Andapa and Befandriana Nord). The project has trained 132 CHWs in socially marketed products, including 44 who have been trained in injectable contraception provision. In each locality, the Social Development Committee provides support and supervises training and monitoring of CHW activities. WCS – supply chain activities In addition to collaboration on training and community health activities, PSI/M also works with WCS on improving the supply chain from the district level down to the community level. To date, PSI/M has established 16 supply points at community level, where social marketed products are available to CHWs for resupply. PSI/M and WCS have also collaborated in communication activities including local radio programs and the celebration of 'World xxx Days' for several health-related themes. ODDIT : ORGANISME DE DEVELOPPEME NT DU DIOCESE DE TOAMASINA PSI/M subcontracted ODDIT in 2009 for support with distribution, stocking and logistics related to the mass net distribution campaign. In 2010 they were subcontracted for the same tasks along with social mobilization (pre, during and post campaign), and local coordination. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 50 PSI’s USG Sub‐ Awardee FY 2008‐ 2012 Description MRC : MALAGASY RAITRA CONSULTING PSI/M contracted MRC and MC2S in 2010 to conduct trainings in preparation for the mass net distribution campaign. Nearly 41,000 community agents were trained in 71 districts. In addition the organizations trained more than 1,000 local heads of health centers in micro-planning for the campaign. MC2S : MALAGASY CONSULTING SOLUTION ET SERVICE See MRC above. SAGE: SERVICE D'APPUI À LA GESTION DE L'ENVIRONNEM ENT PSI/M subcontracted SAGE in 2010 for support with distribution, stocking, logistics, social mobilization and coordination related to the mass net distribution campaign. CONSORTIUM SAF/ECREA PSI/M subcontracted SAF/ECREA in 2010 for the same tasks as ODDIT, as describe above (See ODDIT). CLR: CONSORTIUM LOVA/Ressources Vertes PSI/M subcontracted CLR in 2010 for the same tasks as SAF/ECREA and SAGE i.e., for support with distribution, stocking, logistics, social mobilization and coordination related to the mass net distribution campaign in the South part of Madagascar. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 51 ANNEX B: PSI/M MAPS OF COMMERCIAL COVERAGE AND TOP RÉSEAU CLINICS USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 52 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 53 ANNEX C: TABLE OF RESPONDENT CATEGORIES Respondent categories USAID and other Donors 17 KIIs with USAID personnel and representatives of other donors: 13 - USAID; 1 - World Bank; 1 - UNFPA; 1 - GFATM; 1 - UNICEF. GOM 6 KIIs with GOM representatives at the central, regional, and district levels. PSI/M 35 KIIs with PSI/M staff including senior management, monitoring and evaluation, financial administration, and technical staff, both at central and regional levels. PSI/M Implementing Partners 25 KIIs with Sub-Awardees and other Implementing Partners including, but not limited to the following. Sub-Awardees: 1 - MCDI; 1 - SALFA; 1 - ODDIT; 2 - RTI (COP and DCOP for SantéNet2 in Antananarivo); 1 - JSI (COP for MAHEFA Project in Antananarivo) Other Implementing Partners: 8 - CROM representatives; 1 - ZETRA (SantéNet2 NGO); 1 - ASOS (SantéNet2 NGO). KIIs with Service Providers  27 - Top Reséau Staff (e.g., owners, managers, doctors and other staff)  20 - CHWs Non-PSI/M partners 25 KIIs with representatives of private firms and NGOs that are not partnered with the Social Marketing Program Supply Chain Actors  40 KIIs with actors involved in the supply chain of PSI/M products: 8 - pharmaceutical wholesalers; 12 - community based distributors and supervisors at supply points; and 20 - commercial wholesalers.  124 site visits with retailers including pharmacies, kiosks, hotels, motels, grocery stores, and boutiques. Beneficiaries 35 KIIs with end users of PSI/M products including:  Caregivers of children under 5 years old  Women of reproductive age  Men of reproductive age  Youth  CSWs  MSM FGD Results by Respondent Type  Women of reproductive age (12)  Men of reproductive age (10)  Youth (7)  CSW (6)  MSM (5)  CHW (8)  PSI/M Central-level Staff (4) LSAT The Team conducted nine site observations guided by the LSAT, one at PSI/M Central level and eight at PSI/M regional level, in order to examine supply chain management and support activities. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 54 ANNEX D: EVALUATION SCOPE OF WORK SECTION C – PROGRAM DESCRIPTION C.1 PURPOSE USAID/Madagascar is seeking an external Team of evaluators to conduct an end-of- project evaluation for the PSI Social Marketing Project. The purpose of this evaluation is two-fold: 1) The evaluators will collect both qualitative and quantitative information about PSI’s implementation, progress and challenges, with particular emphasis on whether PSI successfully achieved the objectives of the project. Additionally, the Team will also share any programmatic, management and/or financial obstacles that affected project implementation to inform future projects. 2) The evaluation results will inform USAID/Madagascar and key stakeholders of recommended strategies, structures or actions for future projects involving supply chain and distribution systems to increase efficiency and impacts, e.g., viability of community supply chain system; priority areas for local capacity building; gaps in management structure and oversight; consumer focus in product distribution and price determination, etc. Evaluation questions should correspond to the major areas of focus for the project, including: maternal and child health; family planning; malaria; STI/HIV; supply chain. There are several cross￾cutting themes, such as behavior change communication and the development of the Top Réseau franchise of health clinics that should also be evaluated. The primary evaluation question is: To what extent did PSI achieve the stated results outlined in the Cooperative Agreement 687-A-00-08-00032-00? More specific illustrative evaluation questions are as follows and evaluation results should include the key achievements in each area, strengths and facilitating factors, challenges and barriers and level of sustainability. Original questions: 1. How did PSI perform in the area of maternal and child health; family planning; malaria; and STI/HIV? Result 1: Maternal and Child Health  How did PSI perform in the area of diarrhea prevention and treatment activities?  How did PSI perform in the area of pneumonia prevention and treatment activities?  How did PSI perform in the area of the integrated management of childhood illness?  How were the findings of research results in maternal and child health used by PSI or others? Result 2: Family Planning  How did PSI perform in the provision of family planning services? Result 3: Malaria  How did PSI perform in the provision of malaria prevention and treatment services? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 55 Result 4: STI/HIV  How did PSI perform in the provision of STI and HIV prevention and treatment services?  What are the major achievements of PSI’s peer outreach activities for prevention of STI/HIV? 2. How did PSI perform in implementing and working with Top Réseau clinic franchise?  In regards to the Top Réseau clinic franchise, what were PSI’s major achievements in developing private sector services?  Did these services increase access and use for those most in need?  To what extent are the Top Réseau franchise clinics sustainable without PSI assistance? 3. How did PSI perform in the Supply Chain/distribution systems?  In maintaining the supply chain for community-based products, what were the major achievements and weaknesses?  What contribution did the PSI project make to national coverage of health products through its supply chain/distribution system? How about in rural areas?  To what degree has the project been successful in establishing a supply chain model for USAID-funded Community Based Distribution programs since 2009? What were the obstacles? How were they overcome? 4. Other Possible Cross-cutting questions  In the areas of family planning and malaria, what were the results of PSI’s capacity building efforts for local organizations?  To what extent has PSI been able to align and harmonize its work with national programs (especially malaria), despite constraints?  What was PSI’s performance in terms of organizational capacity building?  Was the PSI project successful in leveraging partnerships with local and other international NGOs to distribute the health products? What role did the PSI project’s contributions (BCC, Products, Research, Distribution, Training, Housing…) play in the attainment of other partner’s achievements?  What was the impact of the political situation on the project’s performance on MCH, HIV, malaria, and FP? To what extend has PSI been able to adapt to new circumstances and modes of operation? Revised set of questions: Program Result Area Key Evaluation Questions I. How well did PSI perform in the areas of Maternal Child Health (MCH),  Family Planning (FP) and Reproductive Health (RH), Malaria, and Sexually Transmitted Diseases/ Human Immunodeficiency Virus (STI/HIV)? 1. MCH  How did PSI perform in the area of diarrhea prevention and treatment activities?  How did PSI perform in the area of pneumonia prevention and treatment activities?  How did PSI perform in the area of the integrated management of childhood illness?  How were the findings of research results in MCH used by PSI or others? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 56 2. FP/ RH  How did PSI perform in the provision of family planning services? 3. Malaria  How did PSI perform in the provision of malaria prevention and treatment services? 4. STI/HIV  How did PSI perform in the provision of STI and HIV prevention and treatment services?  What are the major achievements of PSI’s peer outreach activities for prevention of STI/HIV? II. How did PSI perform in implementing and working with Top Réseau clinic franchise? Capacity building of the Top Réseau clinic franchise  In regard to the Top Réseau clinic franchise, what were PSI’s major achievements in developing private sector services?  Did these services increase access and use for those most in need?  To what extent are the Top Réseau franchise clinics sustainable without PSI assistance? III. How did PSI perform in the Supply Chain/distribution systems? Supply Chain/ Distribu‐ tion Systems  In maintaining the supply chain for community-based products, what were the major achievements and weaknesses?  What contribution did the PSI project make to national coverage of health products through its supply chain/distribution system?  How about in rural areas?  To what degree has the project been successful in establishing a supply chain model for USAID-funded community-based distribution programs since 2009?  What were the obstacles?  How were they overcome? IV. Other Possible Cross‐cutting questions Cross‐ cutting questions  In the areas of FP and Malaria, MCH and Child Survival, HIV/AIDS/ STIs, and Supply Chain, what were the results of PSI’s capacity building efforts with local organizations?  To what extent has PSI been able to align and harmonize its work with national programs, despite constraints, in all the program areas listed above?  How do GOM actors with whom the project still works (e.g., National Institute of Statistics, National Laboratory, National Medical Distribution System, Customs and public sector facilities in urban and rural areas), perceive the project and its outcomes, and how do they see their related needs in the immediate and medium term future? (NEW).  Was the PSI project successful in leveraging partnerships with local and international NGOs to distribute health products? What role did the PSI project’s contributions (in BCC, Products, Research, Distribution, Training, Housing…) play in the attainment of other partners’ achievements?  What was the impact of the political situation on the project’s performance on MCH/FP, HIV/STI, and Malaria, Child Survival, and capacity building? To what extent has PSI been able to adapt to new circumstances and modes of operation? C.2 ACTIVITIES This activity will be a performance evaluation in which USAID/Madagascar encourages the use of mixed-method approach for the evaluation. It may include (1) desk review of documents and available data, (2) key informant interviews (KIIs), (3) site visits and focus group interviews, and (4) mini-survey. USAID/Madagascar expects to discuss the methodology with the Evaluation Team that is selected. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 57 Data collection methods should be appropriate to answer the evaluation questions referenced above. Methodologies should adhere to the highest quality standards, considering issues of relevance, reliability, and ethics. Data analysis and evaluation results must be disaggregated by urban/rural and by gender as appropriate. It is strongly encouraged to gather data from as many provinces of Madagascar as appropriate to the evaluation questions and activities’ locations. A list of suggested methodologies is proposed below: 1. Desk review of documents and available data: The Evaluation Team will be given access to relevant published and unpublished project documents for review and analysis. The Team will review all available materials prior to conducting any field work. 2. Key informant interviews: The Evaluation Team will interview selected respondents according to stakeholder list that will be provided by USAID/Madagascar. This step will likely involve travel to places in remote places depending on the final field observation plan agreed upon by the Evaluation Team. 3. Site visits and focus group interviews: The Evaluation Team will conduct site visits for direct observations and interviews with selected focus groups (end users, community health volunteers, etc.). This step will likely involve travel to places in remote places depending on the final field observation plan agreed upon by the Evaluation Team. 4. Mini survey: As needed, the Evaluation Team may conduct mini-survey of selected stakeholders. Samples for the mini-survey should be representative of PSI beneficiaries at the province level. 5. Other methods as developed by consultants in collaboration with USAID C.3. TEAM COMPOSITION The Evaluation Team must be qualified as experts in both quantitative and qualitative evaluation methodologies, so that its recommendations can be viewed as authoritative and influential. Composition of the Team should include 2-3 persons with:  Significant knowledge of PSI/M’s and associated structures utilizing both private sector and community-based channels for widespread supply chain and distribution;  Demonstrated ability to implement a large, field-based program performance evaluation and submit a quality report within specified time-frame;  Expertise in quantitative and qualitative evaluations;  At least two persons with excellent spoken and written skills in French and English. In addition, the Team should include Malagasy staff (logisticians, IT staff, and data collection assistants), all of whom the contractor will recruit locally. Finally, two to three USAID staff will participate as observers of the evaluation process and the field implementation. C.4 REPORTS AND DELIVERABLES a. Reports and Briefings USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 58 1. In briefing in Madagascar (DATE TBD): The Evaluation Team will have two meetings – one with key USAID and a second with PSI staff - to present the Team, the draft evaluation plan and timeline, and to address any issues. The meeting with PSI staff will be conducted in French. No field work may take place before the in-briefing meeting with USAID and PSI. 2. Detailed evaluation plan and Preliminary Summary Report: Three weeks after the award of this Task Order, a detailed evaluation plan (outlining the evaluation design, methodology, and data analysis plan and data collection schedule) will be submitted to USAID/Madagascar for comment and approval. This plan should include concise explanations for the methodologies chosen and how they best respond to the evaluation questions. This plan will be submitted to USAID in English, and once approved, the contractor will have one week to finalize the plan both in English and in French. Also, a summary report outlining the results of a comprehensive review of PSI project documents, (e.g., proposal, baseline study, annual reports, monitoring surveys, etc.), will be submitted to USAID for comment. This report will be in English. 3. Out briefing in Madagascar (DATE TBD): Field work should be completed by 07/20/2012 in order to ensure access to implementing partner activities. During the week after completion of the field work and data collection, the Evaluation Team will have between three to five days in-country to prepare a briefing outlining preliminary findings, conclusions and recommendations to key USAID staff; and to make a formal oral debriefing including a slide presentation, followed by discussions to USAID. The presentation will be delivered in English (complete with slides) and will be provided in an open forum to be determined in collaboration with the USAID manager of this evaluation. Minutes of this meeting will be prepared by the contractor and attached as an annex to the draft evaluation plan. 4. Draft Evaluation Report: Two weeks after the formal oral debriefing, the contractor will submit a near-final draft report to USAID/M for review and feedback. This report will be in English and will not exceed 40 pages (not including annexes and attachments). USAID has 10 working days to review the Draft report and provide a single set or written comments on the Draft Report. 5. Final Evaluation Report: After reception of the written comments from USAID, the contractor will have one week to finalize this report both in English and in French. The evaluation report should describe the research design and methodology, present findings and conclusions related to the key evaluation questions, detail any lessons learned and offer recommendations for future projects. This report will not exceed 40 pages including a 3-page executive summary and annexes as needed to clearly illustrate or highlight key points. The contractor will also submit to USAID (with the final report) all instruments and tools used in implementation of this evaluation and all data collected on appropriate storage devices, e.g., flash drive, compact discs, interview records, etc. USAID/Madagascar’s additional comments, if any, on the Final Evaluation Report will be sent to the contractor within 14 working days of its receipt by USAID/Madagascar. Once the final evaluation report is complete, the contractor will submit four hard copies (two in English and two in French) and electronic versions of these reports to USAID/Madagascar. The contractor will also send an electronic copy to the Development Experience Clearinghouse for archival. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 59 C.5 PROGRAM MANAGEMENT The USAID technical office responsible for the Task Order is the Program Development and Assessment Office (PDA) at USAID/Madagascar. The contractor will report directly to USAID/Madagascar through the COR within PDA. Reviewers will be provided with the following documents in preparation for the assignment:  Background Information : Country Context and Project Description (Annex A) ;  PSI Intervention Zones (Annex B);  PSI annual progress reports for FY 2009, FY 2010, and FY 2011 (Annex C);  Other relevant documentation as necessary or upon request.    USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 60 ANNEX E: MAP OF REGIONS AND DISTRICTS VISITED BY THE EVALUATION TEAM USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 61 ANNEX F: TABLE OF DATA COLLECTION RESULTS Region Districts Commune (village) - Indicates if site is urban (U), peri￾urban, (PU), or rural (R) Key Informant Interviews (KIIs) at Service Delivery Clinics KIIs and site visits at Distribution/ Supply Points - Indicates type: Community Based Distributor (CBD), Commercial, Retail, or Pharmacy KIIs with PSI staff, USAID personnel, GOM representatives, implementing partners and other stakeholders FGD - Indicates total number for each respondent category55 LSAT Team A Diana Region, Northern Madagascar Monday June 25-Saturday June 30 Ambilobe District; and Antsiranana II District; Ambanja District Urban (U) areas in Ambilobe District and Antsiranana; Rural (R) areas of Ambilobe District (135 kms from Antsiranana); Peri-urban (PU) areas of Ramena Commune and Joffreville Commune in Antsiranana II District; and Antsiranana City (U- the capital of Diana Region, formerly called Diego-Suarez prior to 1975, commonly called Diego by French speakers). Total = 2 Top Réseau KIIs: Dr. Radilahy Basile at Top Réseau Cabinet Medical, Anoronala Commune/ Antsiranana II District; and Dr. Marc Richard at Espace Medical Prestataire du Top Réseau, Antsiranana Total = 1 Supply Point (SP) site visit in Ambilobe Commune (R) Total = 12 KIIs: 5 with PSI/M staff at Antsiranana office (U); 1 with Community Health Worker (CHW) in Antsiranana (U); 2 with CHWs in Ambilobe Commune (R); 4 with PSI/M Implementing Partners Total = 6 FGDs 1 with Women (15-45 years old) 3 with Youth 1 with men (15-45 years old) 1 with women with children 1 LSAT conducted at PSI/M in Antsiranana USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 62 Atsimo￾Andrefana Region, South Western Madagascar Monday July 2 - Saturday July 7 Toliara II District (U); and Ampanihy District (R). Toliara (U - also known as Toliary, formerly called Tuléar, and is the capital of Atsimo￾Andrefana Region); and Ankiliabo Commune (R) in Ampanihy District. Total = 2 Top Réseau KIIs in Toliara (U) 1 at Dispensaire Fraternite (U) 1 at Centre Medical de Bethesda (U) Total =1 SP KII and site visit at Ankiliabo Commune (R) Total = 19 KIIs: 17 KIIs in Toliara (U and PU areas): -5 with PSI/M staff -2 with GOM staff -5 with PSI/M partners -5 with Non PSI/M partners 2 KIIs in Ankiliabo Commune (R): -1 with GOM staff -1 with NGO partner (ASOS) Total = 7 FGDs 2 FGDs in Ankiliabo Commune (R): -1 with female CHWs -1 with women with children 5 FGDs in Toliara (U): -1 with MSM -2 with Commercial Sex Workers (CSWs) -1 with Youth (musicians) -1 with mobile men (pousse-pousses or rickshaw drivers) 1 LSAT at Toliara USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 63 Antananarivo, the Capital of Madagascar and the Analamanga Region Team A made two trips – during June 11-24, 2012 and during July Antananarivo (U and PU areas) Antananarivo (U and PU areas) Total = 1 Top Réseau visited in Antananarivo (during June 11-24) Top Réseau OSIEM in Ampasika [No Supply Chain KIIs were conducted by Team A – the Team was preparing the Evaluation Methodology and Plan] Total = 5 Supply Chain KIIs and site visits in July: 1 SP (PU); 1 PSI/M pharmaceutical wholesaler; 2 PSI/M commercial wholesalers; 1 CBD at SP. Total = 34 KIIs: 10 KIIs with USAID personnel, including: Director of Program Office, Senior Acquisition Specialist, PMI Program Management Assistant, Senior Health Administrator /OPHN, Senior HPN Program Manager, Program Officer, Program Specialist, M&E Assistant, HPN Financial Analyst. 13 KIIs with PSI/M staff at central office. 8 KIIs with PSI/M partners: Mr Volcan Cakir, Chief of Party (COP) and Dr. Nirina Ranaivoson, Deputy COP for RTI International/ SantéNet2; Dr. Penny Dawson, COP for JSI/ MAHEFA Project; Yvette Rakotobe, Pharmacy Inspector from AGMED; Christian B Rabefaniraka, Deputy General Manager for R&D, Production, and Quality at SIGMA; Dr. Rabakomahefa Voahirana, Programs Coordinator for FISA; 3 KIIs with GOM staff: [No FGDs conducted by Team A – the Team was preparing the Evaluation Methodology and Plan] Total = 6 FGDs 3 of which in PU areas: 4 FGDs with PSI staff at Central Office (Distribution, Communication, Service Delivery, and Financial Administration teams) 1 FGD with Youth at the University of Antananarivo 1 with Women (Mahitsy Commune in ASOS intervention area). 1 LSAT at PSI/M Central in Antananarivo USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 64 Boeny Region, in North Western Madagascar and Betsiboka Region July 16 - 21, 2012 Mahajanga II District (called Majunga District in French) Boeny Region; Maevatanana District in Betsiboka Region; Marovoay District in Boeny Region. Mahajanga I (U - also called Majunga City and is the capital of the Boeny region); Mahavoky Commune, Maevatanana District; Mahatsinjo Commune in Maevatanana District; La Corniche Commune in Mahajanga II District; Station Agricole Commune Total = 4 KIIs with Top Réseau staff and 4 site visits: Top Réseau in Mahavoky Commune; Top Réseau in La Corniche Commune; Top Réseau in Station Agricole Commune; and Top Réseau in Mahatsinjo Commune. Total = 5 Supply Chain KIIs and site visits: 1 KII with Mr. Philippe, CBD SP in Belobaka, Majunga II District (PU); 1 KII with Mr. Jo Marovoay At SP in Akazombora Commune (R); 2 KIIs with PSI/M Commercial wholesalers; 1 KII with PSI/M Pharmaceutical wholesaler Total = 18 KIIs: 4 KIIs with PSI/M staff 3 KIIs with PSI/M partners; 2 KIIs with Non SM Project partners in Antanambao village; 3 KIIs with GOM staff; 2 KIIs with rural men at Anbondromany village; 2 KIIs with CHWs in Bealoy and Akazomborona Communes in Marovoay District; 2 KIIs with MSI staff - Dr. Maurin Mananantitra, Outreach Doctor Boeny and Yvonne Randrindra, Outreach Coordinator Boeny-Betsiboka. Total: 11 FGDs conducted in U, PU, and R areas￾including 5 FGDs conducted in Mahajanga I (U): 1 FGD with youth in Maison du quartier, Mahajanga (U); 1 FGD with MSM; 1 FGD with women of reproductive age; 1 FGD with CSWs; 1 FGD with Youth. Including 6 FGDs in the rural communes (Bealoi and Akazomborona): 2 FGDs with men of reproductive age; 2 FGDs with women of reproductive age in Bealoi; 1 FGD with male CHWs; 1 FGD with women in Akazombora Commune (R) USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 65 Team B Antananarivo, the Capital of Madagascar and the Analamanga Region July 2012 Antananarivo (U and PU areas) Besardy Total = 4 KIIs and site visits at Top Réseau Clinics Total = 27 KIIs: 10 with PSI/M Staff; 5 with CROM Officials/ Members; 2 with MCDI Managers (Sub-awardee); 1 with Director of PNLS; 2 with AGMED representatives; 1 with SIGMA representatives; 1 with World Bank (WB) representative; 3 with USAID Officials 2 with USG Officials Total = 4 FGDs: 1 FGD with CSWs that are also Mothers of CU5 (8 participants) 1 with men of reproductive age; 1 FGD with CSWs, La Rotonde Besarety; 1 with MSM in Andraisoro near Be Camion (7 participants). Diana Region June 25-30 Ambilobe District ( R and U areas); Antsiranana / Diego (U - capital of Diana); Antsiranana (U - capital of Diana, formerly called Diego); Ambilobe District (U); Ramena (PU); Joffreville (PU); and Ambilobe (R – about 135 - 145 km from Antsiranana). Total = 2 KIIs with Top Réseau staff: Dr. from Top Réseau Cabinet Medical in Anoronala Commune in Diego; Dr. Marc Richard, Service Provider at Espace Medical Prestataire du Top Réseau in Antsiranana. Total = 11 semi￾structured interviews with Supply Chain informants Total = 44 SPs visited: 1 Supply Point in Ambilobe Commune; 7 pharmacies; 2 wholesalers; 1 CHW at a SP And site visits at 44 commercial retailers Total = 14 KIIs in Antsiranana (U): 5 PSI staff 3 CHWs 4 PSI Partners In Ambilobe: 2 CHWs Total = 6 FGDs: 1 with Women (15-45 years old); 3 with Youth; 1 with Men 15- 45 years old); 1 with Women with children under 5 yrs old (CU5) Vakinankaratra Region in Central Madagascar Monday June 25 - Saturday June 30 Areas surrounding the regional capital, Antsirabe, Antsirabe II District; Tanambao Ambary (R), Mandoto Antsirabe (U); Mandoto / Mangarano Commune (R); Morafeno Village / Anivosaha Fokontany; Ambalamarina Fokontany/ Total = 7 KIIs with Top Réseau staff: 5 KIIs at the Top Réseau Coordination Office in Antsirabe (these interviews included supply Total = 5 Supply Chin KIIs and sites visited including: 2 KIIs and site visits to CBDs 2 KII and site visits at retail pharmacies; 1 KII and site visit at Commercial pharmacy: Dr. Harinavalona Total = 12 KIIs: 4 with GOM - Dr Melimanantsoav, (GOM- Chief at CSB II Tanambao Ambary -R); Dr. William Rumhadison (GOM- Directeur Régional de la Santé - DRS); Dr. Yves (GOM - Regional HIV/AIDS Total = 1 FGD 1 FGD with GOLD (a local NGO) CHWs (2 males & 7 females) in Tanambao Ambary (R), Mandoto District 1 LSAT in Antsirabe USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 66 District Mangarano Commune (R); Belazao/ Anboniavaratra (R) Tanabao Ambary (R) in Mandato District - about 97 km from town) chain questions as well): Dr. Pati Razafindramekoa; Dr. Dina Randriambololona; Dr. Andriamahatana Ridasoa; Ms. Andriamihaja Santa Noar’jim; and Dr Ginette (also a CROM President). 2 KIIs at Top Réseau clinic in Mandoto (R - about 90 km from town): Dr Voahangy Hantanairina, Chief Doctor; and Pauline Hananairina, Health Aid. Mihaja, Pharmacist, and Ms. Raharimanana Zoly, Manager at Retail Pharmacy Vakinanankaratra; Ms. Ranoromalala Verohanitra, Commercial Wholesaler in Centre Ankaratra; Mr. Raymond Rasolomanga, CBD in Antsoatany; Mr. Randriamanjato Richard, CBD at Mangarano; Ms. Raholiarisoa Lalao (CBD at Yvonne Village, Amboniavaratra Fokontany, Belazao Commune). Director); Dr Ginette, CROM President at Antsirabe (U); 2 Animateurs de District from Association Kilonga (PACT & SIVE) at Antsirabe (U); Clarisse Ravololonirina, (pediatric CHW at Mangarano Commune); Marie Claire Ravololonirina (CHW for Mothers of CU5 at Morafeno Village/ Anivosaha Fokontany/ Mangarano Commune); Celestin Rakotondrandria (CHW and Chief of Ambalamarina Fokontany/ Mangarano Commune). 4 with PSI/M Partners Haute Matsiatra Region in South Central Madagascar Fianarantsoa Commune (U - the capital of Haute Matsiatra Region); Ambalavao District Vohitsaoka Commune (R), Iaritsena Commune (R), Tsienimparih Commune (R), and Ivoamba Commune (R) in Ambalavao District Total = 1 Top Réseau KII and site visit: KII with Dr. Razanajohary Total = 13 Supply Chain KIIs and site visits: 1 with a pharmaceutical wholesaler; 5 with commercial wholesalers; 5 with commercial retailers; 2 with CBDs Total = 14 KIIs: 1 with a peer educator (PE) Coordinator; 4 with GOM staff; 6 with PSI/M Partners; 1 with Non Partner Doctor; and 2 with a man and a woman of reproductive age Total = 3 FGDs: 1 FGD with Ny Tanintsika CHWs (8 M & 5 F) from 7 to 100 km from Iaritsena/ Ambalavao; 1 FGD with ODEFI CHWs (3 M & 3 F) from 1 km to 17 km from Vohitsaoka Ambalavao 1 LSAT in Fianarantsoa USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 67 Anosy Region on the Southeast coast of Madagascar Tôlanaro District; Tôlanaro, (u￾formerly called Fort Dauphin and also called Tolagnaro Commune, is the capital of the Anosy Region and of the Tôlanaro District) Total = 2 KIIs at the Top Réseau Coordination Office: 1 with the Coordinating Doctor and 1 with a Medical Supervisor Total = 13 KIIs: 1 with a peer educator (PE) Coordinator; 2 with GOM staff; 4 with PSI/M Partners; 2 with a man and a woman of reproductive age; 3 with CHWs (1 M and 2 F); 1 with a male PE Total = 2 FGDs at Tôlanaro/ Ft. Dauphin: 1 FGD with MSM 1 FGD CSW that are also Mothers of CU5 1 LSAT in Tôlanaro Menabe Region in Western Madagascar Morondava (U) 1 urban and 1 rural Commune in Morondava [Accessibility was limited due to terrain, water courses]. Total = 2 KIIs at the Top Réseau clinics: 1 with the Coordinating Doctor and 1 with a Medical Supervisor Total = 7 KIIs with Supply Chain informants and 7 site visits to SPs: 2 with pharmaceutical retailers; 4 with commercial retailers; and 1 with commercial wholesaler Total = 10 KIIs: 1 with PE Coordinator; 2 with GOM staff; 4 with PSI/M Partners; 2 with a man and a woman of reproductive age; 1 CHW for nutrition Total = 6 FGDs: 2 with male and female care givers of CU5; 2 with members of Women’s Associations; 1 with boatmen – mobile men; 1 with FP Counselors - Municipality 1 LSAT in Morondava USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 68 ANNEX G: LIST OF REFERENCE DOCUMENTS  USAID-PSI Madagascar Cooperative Agreement No. 687-A-00-08-00032-00 July 14, 2008  Mr. Andriaherinosy- PSI Madagascar Termes de Reference (TDR)- Directeur de Distribution & Promotion October 2006  Rajaonarivelo Njakatiana- PSI Madagascar Termes De Reference- Directeur des Ressources Humaines, Administratif December 2009  Termes de Reference Mami January 10, 2012  PSI Madagascar- Job Descriptions 2009  PSI Madagascar- Termes De Reference- Directeur Administratif et Financier November 2006  PSI Madagascar- Termes De Reference- Directeur de Prestation de Services de Santé September 13, 2011  PSI Madagascar Administrative and Financial Standard Operating Procedures and Manuals (22 documents and 34 Excel spreadsheets)  Association Malgache de Marketing Social (AMMS) PSI Madagascar- Bon De Reception No. 009084 April 10, 2012  PSI Madagascar- Bon De Sortie Et De Retour February 23, 2012  AMMS- PSI Madagascar Bordereaud D’ Expedition No. 009050 February 23, 2012  PSI Madagascar- Confirmation of Receipt June 7, 2012  Fiche de Stock April 23, 2008  Récapitulation des mouvements de stock August 5, 2003  PSI Madagascar- Suivi Température et Humidité October 22, 2009  Liste CRFS January 16, 2011  PSI Madagascar- FY 2009 Annual report (performance report) November 2009  PSI Madagascar- FY 2010 Annual report (performance report) November 30, 2010  PSI Madagascar- FY 2011 Annual report (performance report) November 2011  Pretoria Christine M. Byrne- Audit final FY 2011 of USAID/Madagascar’s Family Planning and Reproductive Health Activities No. 4687-11-012-P September 22, 2011  Cabinet Mazars Fivoarana -Audit report FY 2009 PSI Madagascar- AMMS Financial Statements July 2010  Cabinet Mazars Fivoarana - Audit report FY 2010 of Social Marketing Program for child, Maternal, Reproductive Health product and Services No. 687-A-00-00032-00 July 2011  Mission Portfolio Review (as of end of FY 2011) May 24, 2012  PSI/M Monitoring and Evaluation (M&E) Plan for the Implementation of the Evidence Based Social Marketing for Child, Maternal, and Reproductive Health in Madagascar” Cooperative Agreement No. 687‐08‐A‐006, Update: December 2011  Jules Allen- PSI Madagascar organizational chart March 24, 2009 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 69  Social Marketing Research Series (SMRS)- Madagascar (2008): Family Planning TRaC May 2009  PSI Madagascar- Focus for Marketing Planning 2011  PSI Madagascar- TRaC Summary Report 2011  PSI Madagascar- MAP Summary Report 2011  USAID Annex A- Background Information March 15, 2012  USAID Annex B- PSI Intervention Zones March 13, 2012  PSI Madagascar- The Social Marketing Program for Child, Maternal, and Reproductive Health Products and Services in Madagascar FY 2009 Annual Report Oct 2008- Sept. 2009 November 200  PSI Madagascar- Evidence-based Social Marketing for Child, Maternal & Reproductive Health in Madagascar FY 2010 Annual Report Oct. 2009- Sept. 2010 November 30, 2010  PSI Madagascar- FY 2011 Annual Report  USAID- Analysis of the Operational Policies Related to Financing and Procuring Contraceptives in Madagascar February 2010  US Global Malaria Coordinator -Presidents Malaria Initiative-- Malaria Operational Plan 2011  PSI Madagascar- All Flowcharts March 10, 2011  USAID PSI Madagascar Modification of Assistance September 30, 2011  PSI/M Monitoring and Evaluation (M&E) Handbook, Second Draft- SvL December 2010  PSI Madagascar -USAID Semester Report October 2011- March 2012 No. 687-A-00-08- 00032-00 April 30, 2012  USAID Program Level- Data Quality Assessment February 10, 2012  USAID Data Quality Assessment- Number of People trained on Family Planning and Reproductive Health (FP/RH) using USG funds December 14, 2011  Madagascar Demographic and Health Survey 2008-2009. Calverton, United States: ICF Macro, 2010 Desk review for supply chain assessment:  PSI/M Cooperative Agreement No. 687-A-00-08-00032-00  PSI/FY 2009 PSI/M semi-annual and Annual report (performance report)  PSI/FY 2010 PSI/M semi-annual and Annual report (performance report)  PSI/FY 2011 PSI/M semi-annual and Annual report (performance report  PSI/FY 2012, PSI/M semi-annual report (performance report)  Auditors’ reports (last three years)  USAID’s FY 2011 Inspector General Report  Mapping Survey (MAP) Phase 2 Condom Study Follow-up  Outlet Survey 3-Nov/Dec 2011  PSI Mystery Client Survey (2010)  PSI FoQus On Concept Development-Sur’Eau  Procurement reports (HIV/AIDS Test Kits/AAPD-07-05) USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 70  Procurement reports (anti-retroviral for HIV/AIDS program (AAPD 07-01)  Measuring Access and Performance (MAP) on PILPLAN and CONFIANCE nationwide  Top Réseau related reports (Study Among Top Réseau Providers of June-July 2010)  Stock management reports  Administrative /stock management forms  List of PSI products and respective price lists  PSI pricing structure per product  Lists of distributors, wholesalers, retailers, per product, per region  Sales statistics per product  PSI/M Performance Management Plan (PMP) • PSI/M Annual work plans • PSI/M logical frameworks • PSI/M Performance Monitoring Plan (M&E) • PSI/Annual implementation plans • Data on Couple of Years of Protection (CYP) in PSI supported program per product • The Pouzin project report (model for access and use of diarrhea treatment) • Qualitative Evaluation of Youth Campaign “it’s your choice” • Tracking Results Continuously (TRaC) Surveys among sexually active youth between 15 and 24 years old • Tracking Results Continuously (TRaC) Surveys among women aged 15 and 49 years old • Tracking Results Continuously (TRaC) Pneumonia Surveys • Tracking Results Continuously (TRaC) Diarrhea Surveys • Tracking Results Continuously (TRaC) Malaria Prevention and treatment integrated in Malaria Indicator Survey (MIS) • Measuring Access and Performance (MAP) on Pilplan • Measuring Access and Performance (MAP) on Confiance • Madagascar Demographic and Health Survey 2008-2009. Calverton, United States: ICF Macro, 2010 • Madagascar Ministry of Health (MOH) Country Strategy • USAID/OPH Country Strategy    USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 71 ANNEX H: DATA COLLECTION INSTRUMENTS Evaluation du Projet Marketing Social de PSI/Madagascar Sous financement de l’USAID Guide d’entretien sur le système de Suivi et d’Evaluation avec Le Staff de Suivi et Evaluation de PSI (ou de Partenaires) Cet entretien a pour objectif de :  Vérifier la cohérence des indicateurs dans le cadre logique du projet de PSI par rapport aux objectifs et résultats attendus  Vérifier dans quelles mesures l’organisation et les outils spécifiés dans le plan et le manuel de suivi-évaluation (SE) sont effectivement opérationnels à tous les niveaux ;  Apprécier l’efficacité et la fiabilité du système décrit dans le plan de SE pour alimenter les indicateurs prioritaires du projet ;  Vérifier si les résultats des recherches sont effectivement utilisés pour informer la prise de décision en vue d’améliorer les interventions en marketing social. Date de l’entretien : Nom de l’interviewe: Email : Masculin / Féminin : Fonction/titre : Département : Nom de l’Organisation (si non PSI) : 1. Décrivez brièvement vos responsabilités au sein de votre département : 2. Depuis combien de temps travaillez-vous avec PSI (ou organisation partenaire)? A. Mise en place du système de suivi et évaluation et capacités techniques existantes en Suivi-Evaluation 1. Avez-vous (ou votre organisation) déjà été impliqué dans la mise en place du système de suivi￾évaluation d’un projet ? Si oui, décrivez de quelle manière. 2. Avez-vous reçu des formations ou de séances de renforcements de capacités dans la mise en œuvre de votre système de SE ? Si oui, décrivez les contenus :  Compréhension du cadre logique et indicateurs /__/  Utilisation des outils de collecte de routine /__/  Conduite des enquêtes quantitatives /__/  Conduite des enquêtes qualitatives /__/  Utilisation des outils de traitement de données /gestion de base de données /__/  Remplissage de canevas de rapport /__/  Contrôle des données /__/  Utilisation des résultats de recherche /__/ USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 72  Autres : ______________________ 3. Quelles sont les avantages de ces formations dans la mise en œuvre de votre système de Suivi￾Evaluation ? 4. Quelles sont les lacunes de capacités en SE dans votre département ou votre organisation, y compris celles du personnel de terrain ? 5. Les personnes impliquées dans les activités de Suivi Evaluation, ont-elles des responsabilités de SE bien définies dans leurs termes de référence? Oui /__/ Non /__/ Si non, citez les postes en question qui n’ont pas de responsabilités définies dans les TdR : B. Maitrise des indicateurs et du fonctionnement du système de Suivi-Evaluation 1. Connaissez-vous les indicateurs qui mesurent les objectifs du projet de Marketing Social de PSI dans votre domaine ? Lesquels ? 2. Quels sont les indicateurs qui posent des problèmes de collecte, et/ou de traitement et/ou de reporting ? Pourquoi ? 3. Est-ce que les outils de collecte des données sont actuellement standardisés pour tous les indicateurs et pour tous les acteurs impliqués ? Oui /__/ Non /__/ Si non, énumérez les indicateurs concernés : 4. Le circuit d’envoi des données collectées est-il clair pour vous ? Si oui, décrivez les responsables qui les envoient et les reçoivent à tous les niveaux : 5. Est-ce que vous utilisez le MIS ? Si oui, quels sont les avantages de ce système ? 6. Quels sont les faiblesses et les contraintes du MIS selon vos expériences ? 7. Avez-vous un budget alloué aux activités de suivi-évaluation ? Si oui, décrivez les affectations de ce budget dans les activités ? 8. Quelles activités du suivi-évaluation ne sont pas couvertes par un financement ? C. Efficacité du système de contrôle de Données et qualité des données 1. Décrivez le système de contrôle de données existant auprès de votre organisation ? 2. Quels sont les avantages de ce contrôle de données ? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 73 3. Quelles sont les faiblesses de la qualité des données les plus fréquentes ? 4. Quelles sont vos recommandations pour améliorer le fonctionnement du système de SE, si nécessaire ? D. Efficacité de l’approche « Recherche Action » de PSI 1. Pouvez-vous donner des exemples qui illustrent la recherche-action dans le marketing social grâce aux différentes enquêtes menées ? 2. Partenaire : avez-vous accès aux résultats des enquêtes de recherche qualitative et Les partenaires de terrain qui utilisent ces résultats envoie-t-il des feedbacks ? Si oui, lesquels ? 3. Quelles sont vos recommandations pour améliorer la « recherche-action » si besoin ? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 74 Interview Guide for GOM Date of interview: Location of interview: First Name, Last Name of interviewer: First Name, Last Name of interviewee: Tel/Mobile of interviewee: Email of interviewee: Male……..Female…….. Title/Position: Department: Purpose of the interview: to collect specific information on PSI/M (Social Marketing Project). 1. Describe your specific position within this Government department: 2. What do you do in relation to PSI/M? 3. For how long have you been partnering with PSI/M? 4. From your perspective, what is working well with this project? 5. What is not working well? 6. What are their main constraints? 7. How well do they address these constraints, in your opinion? SOUND EVIDENCE OF EFFECTIVENESS 8. Do you think PSI/M communication campaign is effective? 9. Do you think PSI/M distribution/sales activities are effective? 10. a. Do you have strong evidence that PSI/M communication/marketing interventions target rural and underserved areas as well as high risk population? PARTNERSHIP/COLLABORATION 11. Does PSI/M have strong partners? If yes, give some concrete examples LONG-TERM SUSTAINABLITY/TRANFERRING SKILLS/CAPACITY BUILDING: 12. PSI/M’s technical assistance increase the capacity of local organizations to carry-on many of the key health sector interventions? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 75 13. Do you think that PSI/M has contributed to the achievement of PSI/M project goal : Improve the health status of the Malagasy people especially women and children? 14. Do you have specific recommendations for improvements/a follow-on program? 15. Do you have specific data/information to share? Yes…….1 No…..2 If yes, specify data/information/title of document? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 76 Interview Guide for Partners Date of interview: Location of interview: First Name, Last Name of interviewer: First Name, Last Name of interviewee: Male……..Female…… Organization: Department: Title/Position: Purpose of the interview: To collect specific information on your organization’s partnership with PSI/Madagascar- Social Marketing Program. 1. Describe your specific role within your department: 2. What do you do in relation to PSI/M (Social Marketing Project)? 3. Describe your organizations’ role in the partnership? 4. From your perspective, what is working well with this partnership? 5. What is not working well? 6. What are the main constraints in the partnership? 7. Did PSI/M address these constraints? 8. How effective has your partnership with PSI/M? 9. What was PSI/M’s performance in terms of organizational capacity building? 10. Was the PSI/M Project successful in leveraging partnership with local and other international NGOs to distribute the health products? 11. What role did the PSI’s project’s contributions (BCC, products, research, distribution, training etc.) play in the attainment of other partners’ achievements? 12. Do you have suggestions on how the partnership can be strengthened in follow-on activities? LONG-TERM SUSTAINABLITY/TRANFERRING SKILLS/CAPACITY BUILDING 13. Do you think that PSI/ M is transferring skills to your organization? 14. Did PSI’s technical assistance increase the capacity of local organizations to carry-out many of the key health sector interventions? 15. Do you have specific data/information to share? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 77 Interview Guide for key informants outside the social marketing program Date of interview: Location of interview: First Name, Last Name of interviewer: First Name, Last Name of interviewee: Tel/Mobile of interviewee: Email of interviewee: Male……..Female…… Organization: Department: Title/Position: Purpose of the interview: To collect specific information on your organization and your views, if any about USAID funded social marketing project being implemented by PSI. 1. Describe your specific role within your organization: 2. Describe the main activities of your organization. 3. Does your organization do capacity building? If yes, please describe in what way? 4. Would your organization be interested in a potential partnership with PSI/M (Social Marketing Project)? a. If yes or no, why? 5. If your organization were to have a relationship with PSI/M’s social marketing project, what would its content be? 6. If your organization was to partner with PSI/M’s social marketing project, could you describe the specific role to be played by your organization in this partnership? 7. If your organization were to partner with PSI/M’s social marketing project, could you describe the specific role to be played by PSI in this partnership? 8. If your organization were to partner with PSI/M’s social marketing project, could you describe the specific role to be played by other actors (if any) in this partnership? 9. From your perspective, how might such partnership help you meet the organizational objectives of your organization? 10. Any comments or suggestions for sustainability? 11. Do you have specific data/information to share? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 78 Interview Guide for PSI Administrative and Financial Staff Date of interview: Location of interview: First Name, Last Name of interviewer: First Name, Last Name of interviewee: Tel/Mobile: Email: Male……..Female…….. Title/Position: Department: Purpose of the interview: to collect specific information on USAID funded PSI/Madagascar Social Marketing Project (PSI/M) 1. Describe your specific position within this department: 2. What do you do in relation to PSI/ M? 3. For how long have you been at PSI/M? 4. From your perspective, what is working well with this project? 5. What is not working well? 6. What are your main constraints? 7. How well does PSI/M address these constraints, in your opinion? A. SOUND EVIDENCE OF EFFECTIVENESS 8. Do you think PSI/M communication campaign is effective? 9. Do you think PSI/M’s marketing/distribution/sales activities are effective? 10. Do you have strong evidence that PSI/M’s communication/marketing interventions target rural and underserved areas as well as high risk population? B. PARTNERSHIP/COLLABORATION 11. Does PSI/M have strong partners? If yes, give some concrete examples USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 79 C. LONG-TERM SUSTAINABLITY/TRANFERRING SKILLS/CAPACITY BUILDING: 12. Do you think that PSI/M is transferring skills to local organizations? 13. Does PSI/M’s technical assistance increase the capacity of local organizations to carry-on many of the key health sector interventions? 14. Do you think that PSI/M has contributed to the achievement of project goal “Improve the health status of the Malagasy people especially women and children? 15. Do you have specific recommendations for improvements/a follow-on program? 16. Do you have specific data/information to share? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 80 PSI Social Marketing Program Madagascar Interview Guide for PSI Technical staff Date of interview: Location of interview: First Name, Last Name of interviewer: First Name, Last Name of interviewee: Tel/Mobile: Email: Male……..Female…….. Title/Position: Department: Purpose of the interview: to collect specific information on USAID funded PSI/Madagascar Social Marketing Project (PSI/M) 1. Describe your specific position within this department: 2. What do you do in relation to PSI/ M? 3. For how long have you been at PSI/M? 4. From your perspective, what is working well with this project? 5. What is not working well? 6. What are your main constraints? 7. How well does PSI address these constraints, in your opinion? D. SOUND EVIDENCE OF EFFECTIVENESS b. Do you think PSI communication campaign is effective? 8. Do you think PSI’s marketing/distribution/sales activities are effective? 9. Do you have strong evidence that PSI’s communication/marketing interventions target rural and underserved areas as well as high risk population? E. PARTNERSHIP/COLLABORATION 10. Does PSI/M have strong partners?, If yes, give some concrete examples USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 81 F. LONG-TERM SUSTAINABLITY/TRANFERRING SKILLS/CAPACITY BUILDING 11. Do you think that PSI/M is transferring skills to local organizations? G. CAPACITY BUILDING/TRAINING 12. Does PSI/M’s technical assistance increase the capacity of local organizations to carry-on many of the key health sector interventions? 13. Do you think that PSI/M has contributed to the achievement of project goal “Improve the health status of the Malagasy people especially women and children? 14. Do you have specific recommendations for improvements/a follow-on program? 15. Do you have specific data/information to share? Yes…….1 No…..2 Specify data/information/title of document USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 82 PSI/Social Marketing Program Madagascar Interview Guide for End users of PSI products Date of interview: Location of interview: First Name, Last Name of Interviewer: First Name, Last Name of Interviewee: Tel: Mobile of Interviewee: Female…. Male….. Organization/Commerce: Title/ Position: Purpose of the interview: to collect specific information on PSI/M products that you use and the services you get from Community Health Workers (CHWs) 1. Describe the activities of your organization (or commerce). 2. Describe your specific role within your organization (or commerce). 3. Do you use the services of a CHW? If so, provide the name of that CHW. 4. Are you happy with the services you get from the CHW? 5. Do you use PSI products? If yes, which products do you use? 6. If you do, at what price do you buy PSI’s products? 7. Do you find PSI’s products affordable? 8. Do you find PSI’s products everywhere, and at all time? 9. Do you ever visit your CHW, and you are told that the products are not available? 10. Do you have anything to say about PSI Products or CHW’s service? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 83 PSI/Social Marketing Program MADAGASCAR Guide d’Entretien pour les prestataires de « TOP RESEAU » Date de l’interview: Lieu de l’interview: Prénom, Nom de l’Enquêteur: Prénom, Nom de l’interviewé: Homme……..Femme…… Organisation: Département: Titre/Fonction: Objectif de l’interview: Collecter des informations spécifiques sur les prestations de services de votre organisation et votre partenariat avec le Programme de Marketing Social de PSI/Madagascar. 1. Combien d’employés travaillent dans votre organisation ? 2. Quelles sont leurs fonctions respectives ? 3. Pourriez-vous nous communiquer l’organigramme de votre organisation ? 4. Quels sont les types de soins offerts par ce Top Réseau ? Planning Familial  pour les mères /__/  pour les adolescents /__/ Suivi de grossesse /__/ IST/VIH/SIDA  Dépistage des IST /__/  Test et conseil sur le VIH/SIDA /__/  Distribution de Protector Plus /__/ Gestion intégrée des traitements de maladies infantiles /__/ Distribution de Sur’Eau /__/ Via’Sur /__/ HydraZinc /__/ Paludisme  Distribution de supermoustiquaire /__/  Distribution de ACTIPAL /__/  Test Rapide de Diagnostique /__/ USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 84 AUTRES 5. Quels sont les produits contraceptifs vendus par votre organisation ? Produits Prix unitaire en Ariary Prix de la consultation ou insertion En rupture Oui/non Préservatifs feeling (pour femmes) Contraceptives orales Confiance- injectables Implants  implanon  zarin DIU 6. Si une patiente avait des hémorragies suites à une fausse couche ou un avortement, quels sont les types de soins offerts par ce Top Réseau? A.VOLET COMMUNICATION : CONSEILLER EN PLANNING FAMILIAL 7. Quelles sont vos activités de communication et de conseil pour les groups cibles ? 8. Ces activités de communications sont-elles efficaces ? Expliquer pourquoi. 9. Avez-vous des fiches de suivi des clients ? Sont-elles remplies régulièrement? 10. A votre avis, quels sont les volets les plus performants de votre Top Réseau? 11. Quels sont les défis que vous avez rencontrés? 12. Quels sont les types de collaboration entre les conseillers en PF et votre Top Réseau? 13. Quel pourcentage de votre clientèle est amené au Top Réseau par les conseillers en PF? 14. Avez-vous d’autres commentaires? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 85 PSI/Social Marketing Program Madagascar Interview Guide of USAID Officers Date of interview: Location of interview: First Name, Last Name of interviewee: Tel/Mobile: Email: Male……..Female…….. Title/Position: Department: Purpose of the interview: to collect specific information on PSI/Magadascar, Social Marketing Project Management and technical activities 1. Describe your specific position within USAID 2. What do you do in relation to PSI/M project? 3. For how long have you been following PSI/M activities? 4. From your perspective, what is working well with this project? 5. What is not working well? 6. What are PSI’s major strengths? 7. What are PSI’s major weaknesses? 8. What are PSI’s major constraints? 9. How well do they address these constraints? 10. How responsive are they to communication from your office? 11. How do they manage their contract, grants and sub grants? 12. What are the implications of multi-donor funding for USAID’s management and financing? 13. Do you think that PSI/M has contributed to the achievement of USAID’s Country Strategy? 14. Any suggestions for post-project sustainability? 15. Any questions you may have on PSI/M? 16. Do you have any specific data/information you could share? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 86 PSI/Social Marketing Program Madagascar Interview Guide for Community Distributers or Community Health workers Date of interview: Location of interview: First Name, Last Name of Interviewer: First Name, Last Name of Interviewee: Tel/ Mobile of Interviewee: Female…. Male….. Organization/Commerce: Occupation: Purpose of the interview: to collect specific information on your role and function in relation to PSI products and their provision to the public (community), which you service. 1. Describe your present occupation. 2. Do you sell PSI products? If yes, which ones? 3. If you sell PSI products, at what retailed price? 4. Do you follow the pricing structure that is recommended by PSI? If yes, what is your profit margin on each product? 5. If you do not follow the pricing structure of PSI, tell us why not? 6. In your opinion, do your clients/customers find the products affordable and acceptable? 7. In your opinion, do people in your community find PSI products everywhere? at all times? 8. Do you experience stock outs? If so how often? 9. How quickly does PSI address these problems of stock-outs? 10. Do you have products that expire or are damaged whilst in your possession? If so what do you do with them? 11. How to you dispose of the waste that is generated from the use of the products? Were you taught how to dispose of the waste? * 12. Did you get training in stock management from PSI? What other training was given to you?* 13. Is the storage condition of the products satisfactory? 14. Do you keep records of your stock and do you share these with PSI/ NGOs? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 87 15. Do you get any promotional materials from PSI or other NGOs? 16. Do you get regular visits from PSI / NGO sales staff? If yes, do you find it helpful? 17. Do you have any suggestions as to how the partnership can be improved? 18. Questions for community Health Workers USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 88 PSI/Social Marketing Program Madagascar Interview Guide for Commercial Retailers Date of interview: Location of interview: First Name, Last Name of Interviewer: First Name, Last Name of Interviewee: Female…. Male….. : Occupation: Purpose of the interview: to collect specific information on your role and function, in relation to the PSI/M products, and their provision to the public which you service. 1. Are you selling PSI products? If yes, can you list these products? 2. If you carry PSI products, at what price do you sell them? 3. If you do not carry PSI/M products, would you like to have some? If yes, which ones? 4. Could you tell us, at what price do you purchase PSI/M products from commercial wholesalers? 5. Could you tell us, how much profit margin do you make on each PSI/M product that you sell? 6. Do you experience stock outs from not finding the products at the wholesaler? If so, how often? 7. Do you have products that expired or get damaged whilst still in your stock? If so what do you do with them? 8. How are PSI/M products selling? 9. Do you ever get visited by PSI/M sales staff? 10. Do you get promotional materials from PSI? 11. Do you have any comments you might want to make on PSI/M products and services? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 89 PSI/Social Marketing Program Madagascar Interview Guide for Pharmaceutical Retailers Date of interview: Location of interview: First Name, Last Name of interviewer First Name, Last Name of interviewee Tel/ Mobile of Interviewee: Email of interviewee: Female…. Male….. Name of Pharmacy: Title/Position: Purpose of the interview: to collect specific information on your role and function in relation to PSI/M products, and their provision to the public which you service. 1. Are PSI/M products available in your pharmacy? If yes can you list the products you have available? 2. At what retail price to you sell each PSI/M product? 3. Do you experience stock outs from not finding the products at the wholesaler? If so, how often? 4. Do you have products that expire or get damaged whilst still in your stock? If so what do you do with them? 5. How are PSI/M products selling? 6. Do you ever get visited by PSI/M sales and advisory staff? 7. Do you get promotional materials from PSI? 8. Do you get training on the use and/conservation of PSI/M products? 9. Do you have any comments you might want to make on PSI/M products and services? USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 90 10. CANEVAS DE RAPPORT D'INTERVIEWS ET DE FOCUS GROUP EVALUATION DU PROJET DE MARKETING SOCIAL DE PSI FGD /___/ INTERVIEW /___/ REGION DISTRICT COMMUNE LOCALITE/VILLAGE VILLE/RURAL (VILLE=1, RURAL=2) NOM DE L'INTERVIEWE FONCTION/TITRE NOM DE L'ORGANISATION DATE DE L'INTERVIEW NOM DE L'ENQUETEUR SECTION 1: ACTIVITES DE COMMUNICATION A REPONDRE PAR TOUT INTERVIEWEE Entourer les codes qui s'appliquent aux réponses données N ° QUESTIONS CODES 1 Quelles sont les ACTIVITES de communication réalisées par PSI/M les PLUS CONNUES dans la zone ? 1. SPOT RADIOS 2. SPOT TV 3. POSTERS/DEPLIANTS 4. CINEMOBILE PSI/M (MVU) 5. EVENEMENT 6. SENSIBILISATION PAR AC/PE/CPF 7. SENSIBILISATION PAR TOP RESEAU 99. AUTRE préciser . . . . . . . . . . . . . . . . . . . USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 91 2 Quels sont les MESSAGES de communication réalisés par PSI/M les PLUS CONNUS dans la zone ? 1. TOKY SY ANTOKA 2. HARENA NY TARANAKA 3. IST/VIH 4. SUR'EAU 5. PLANNING FAMILIAL 99. AUTRE préciser . . . . . . . . . . . . . . . . . . . USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 92 ANNEX I: PSI/M TRAINING SCHEDULE FY 2008 – FY 2012 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Top Réseau providers Sexually transmitted infections (STI), Family Planning (FP) and Social Franchising 2 Initial training 90% 10/2008 9 Top Réseau providers Private Clinic Management 2 Initial training 80% 05/2009 8 Top Réseau providers Quality assurance, quality of service on FP and STI 2 Refresher training 90% 05/2009 26 Top Réseau providers Quality assurance on Reproductive Health (RH) and STI 2 Refresher training 83% 06/2009 10 Top Réseau providers Quality of services 2 Refresher training 100% 06/2009 9 Top Réseau providers FP – STI 2 Refresher training 100% 06/2009 5 Top Réseau providers FP – STI 2 Refresher training 100% 06/2009 57 Top Réseau providers Quality assurance, quality of service on FP and STI 2 Refresher training 93% 06/2009 13 Top Réseau providers RH and STI 2 Refresher training 100% 06/2009 6 Community Health Workers (CHWs) STI, Malaria, FP, Diarrhea, Standard Day Methods 5 Initial training 100% 09/2008 21 CHWs STI, Malaria, Diarrhea 3 Initial training 100% 09/2008 64 CHWs STI, Malaria, Diarrhea 3 Initial training 100% 10/2008 30 CHWs STI, Malaria, Diarrhea 3 Refresher training 95% 10/2008 61 CHWs Standard Day Methods 2 Initial training 100% 11/2008 67 CHWs STI, Malaria, FP, Diarrhea, Standard Day Methods 5 Initial training 100% 11/2008 22 CHWs STI, Malaria, Diarrhea 3 Initial training 100% 11/2008 23 CHWs Standard Day Methods 3 Initial training 100% 11/2008 96 CHWs Standard Day Methods 3 Initial training 100% 11/2008 30 CHWs STI, Malaria, FP, Diarrhea, Standard Day Methods 5 Initial training 100% 01/2009 63 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 93 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants CHWs Diarrhea case management (Viasûr) 2 Initial training 100% 05/2009 23 CHWs Malaria case management (ACT) 2 Initial training 100% 05/2009 27 CHWs Diarrhea case management (Viasûr) 2 Initial training 100% 06/2009 27 CHWs Malaria case management (ACT) 2 Initial training 100% 06/2009 54 CHWs Malaria case management (ACT) 2 Initial training 100% 07/2009 54 CHWs Diarrhea case management (Viasûr) 2 Initial training 100% 07/2009 53 CHWs FP 3 Initial training 100% 07/2009 53 CHWs STI, Malaria, Diarrhea 3 Initial training 100% 11/2009 27 CHWs FP 3 Initial training 100% 02/2010 20 CHWs Diarrhea case management (Viasûr) 2 Initial training 100% 02/2010 25 CHWs FP 3 Initial training 100% 02/2010 25 CHWs Diarrhea Treatment 2 Initial training 100% 02/2010 130 CHWs FP 3 Initial training 100% 03/2010 20 CHWs Diarrhea, Malaria 3 Initial training 100% 03/2010 20 CHWs STI, Malaria, Diarrhea 3 Initial training 100% 03/2010 25 CHWs FP 3 Initial training 100% 03/2010 20 CHWs FP 3 Initial training 100% 04/2010 69 CHWs FP 3 Initial training 100% 05/2010 103 CHWs FP 3 Initial training 100% 06/2010 74 CHWs FP 3 Initial training 100% 07/2010 50 CHWs FP 3 Initial training 100% 08/2010 176 CHWs FP 3 Initial training 100% 09/2010 161 Health private care providers STI, Malaria, FP, Diarrhea 1 Initial training 100% 10/2008 4 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 11/2008 9 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 12/2008 7 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 04/2009 26 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 05/2009 50 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 06/2009 26 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 07/2009 56 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 94 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 08/2009 13 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 11/2009 5 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 12/2009 70 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 01/2010 21 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 02/2010 29 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 03/2010 27 Health private care providers STI, Malaria, FP, Diarrhea 1 Initial training 100% 03/2010 23 Health private care providers STI, Malaria, FP, Diarrhea 1 Refresher training 100% 03/2010 104 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 04/2010 23 Health private care providers STI, Malaria, FP, Diarrhea 1 Refresher training 100% 04/2010 57 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 05/2010 14 Health private care providers STI, Malaria, FP, Diarrhea 1 Refresher training 100% 05/2010 92 Health private care providers STI, Malaria, FP, Diarrhea 1 Refresher training 100% 06/2010 93 Health private care providers STI, Malaria, FP, Diarrhea 1 Initial training 100% 07/2010 26 Health private care providers STI, Malaria, FP, Diarrhea 1 Refresher training 100% 07/2010 46 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 08/2010 3 Sales persons pharmaceutical outlets STI, Malaria, FP, Diarrhea 1 Initial training 100% 09/2010 17 Health private care providers STI, Malaria, FP, Diarrhea 1 Refresher training 100% 09/2010 55 Top Réseau providers STI, FP, and youth RH 2 Refresher training 100% 10/2009 6 Top Réseau providers Voluntary, Counseling and Testing (VCT) 2 Refresher training 100% FY10 16 Top Réseau providers VCT 2 Refresher training 100% FY10 7 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 6 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 24 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 10 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 6 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 14 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 18 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 6 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 95 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% FY10 12 Health private care providers FP 2 Refresher training 100% FY10 77 Peer Educators STI/HIV prevention 2 Initial training 100% FY10 32 Health private care providers FP 2 Refresher training 100% FY10 51 Peer Educators STI/HIV prevention 2 Initial training 100% FY10 6 Health private care providers and paramedics FP 2 Refresher training 100% FY10 443 Health private care providers Malaria case management 2 Refresher training 100% FY10 88 Health private care providers FP 2 Refresher training 100% FY10 131 CHWs Pneumonia protection 3 Initial training 100% 08/2011 51 Sales persons pharmaceutical outlets Pneumonia and PTK 1 Initial training 100% 02/2011 23 Sales persons pharmaceutical outlets Pneumonia and PTK 1 Initial training 100% 03/2011 5 Sales persons pharmaceutical outlets Pneumonia and PTK 1 Initial training 100% 07/2011 9 Sales persons pharmaceutical outlets STI, Malaria, Diarrhea and Pneumonia 1 Initial training 100% 09/2011 131 Health private care providers Malaria, Diarrhea and Pneumonia 1 Refresher training 100% 03/2011 43 Health private care providers (Sage femme) Malaria, Diarrhea and Pneumonia 1 Refresher training 100% 12/2010 248 Health private care providers and paramedics Malaria, Diarrhea and Pneumonia 1 Refresher training 100% 06/2011 336 Top Réseau providers Integrated Management of Childhood Illnesses (IMCI) 3 Refresher training 100% 08/2011 60 Top Réseau providers IMCI 3 Refresher training 100% 09/2011 108 CHWs Diarrhea, Malaria 2 Initial training 100% 10/2010 93 CHWs Diarrhea, Malaria 2 Initial training 100% 11/2010 67 CHWs Diarrhea, Malaria 2 Initial training 100% 12/2010 63 CHWs Diarrhea, Malaria 2 Initial training 100% 02/2011 22 CHWs Diarrhea, Malaria 2 Initial training 100% 03/2011 16 CHWs Diarrhea, Malaria 2 Initial training 100% 08/2011 112 Sales persons pharmaceutical outlets STI, Malaria, Diarrhea 1 Initial training 100% 11/2010 6 Sales persons pharmaceutical outlets STI, Malaria, Diarrhea 1 Initial training 100% 02/2011 23 Sales persons pharmaceutical outlets STI, Malaria, Diarrhea 1 Initial training 100% 03/2011 11 Sales persons pharmaceutical outlets STI, Malaria, Diarrhea 1 Initial training 100% 07/2011 70 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 96 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Sales persons pharmaceutical outlets STI, Malaria, Diarrhea and Pneumonia 1 Initial training 100% 08/2011 50 Sales persons pharmaceutical outlets Diarrhea 1 Initial training 100% 05/2011 11 Sales persons pharmaceutical outlets Diarrhea and FP 1 Refresher training 100% 06/2011 35 Health private care providers Diarrhea 1 Refresher training 100% 11/2010 170 Health private care providers Diarrhea 1 Refresher training 100% 02/2011 23 Health private care providers and paramedics Malaria, Diarrhea and FP 1 Refresher training 100% 06/2011 50 Health private care providers and paramedics Malaria, Diarrhea and FP 1 Refresher training 100% 07/2011 75 Health private care providers and paramedics Malaria, Diarrhea and FP 1 Refresher training 100% 08/2011 63 Health private care providers and paramedics Malaria, Diarrhea and FP 1 Refresher training 100% 09/2011 835 CHWs Diarrhea, Malaria 2 Initial training 100% 12/2010 53 CHWs Diarrhea, Malaria 2 Initial training 100% 03/2011 17 CHWs Diarrhea, Malaria 2 Initial training 100% 04/2011 15 Sales persons pharmaceutical outlets Malaria 1 Refresher training 100% 06/2011 117 CHWs FP 3 Initial training 100% 10/2010 68 CHWs FP 3 Initial training 100% 11/2010 77 CHWs FP 3 Initial training 100% 12/2010 42 CHWs FP 3 Initial training 100% 02/2011 22 CHWs FP 3 Initial training 100% 03/2011 8 CHWs FP 3 Initial training 100% 04/2011 50 CHWs FP 3 Initial training 100% 05/2011 15 CHWs FP 3 Initial training 100% 07/2011 114 CHWs FP 3 Initial training 100% 09/2011 50 CHWs Depo-Provera injection 4 Initial training 100% 11/2010 12 CHWs Depo-Provera injection 4 Initial training 100% 04/2011 20 CHWs Depo-Provera injection 4 Initial training 100% 06/2011 16 CHWs Depo-Provera injection 4 Initial training 100% 07/2011 12 CHWs Depo-Provera injection 4 Initial training 100% 08/2011 55 Sales persons pharmaceutical outlets FP 1 Initial training 100% 02/2011 23 Sales persons pharmaceutical outlets FP 1 Initial training 100% 03/2011 11 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 97 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Sales persons pharmaceutical outlets FP 1 Initial training 100% 05/2011 11 Sales persons pharmaceutical outlets FP 1 Refresher training 100% 06/2011 152 Sales persons pharmaceutical outlets FP 1 Initial training 100% 07/2011 70 Sales persons pharmaceutical outlets FP 1 Initial training 100% 08/2011 50 Sales persons pharmaceutical outlets FP 1 Initial training 100% 09/2011 131 FP Health Counselors FP 1 Initial training 100% 11/2010 1 FP Health Counselors FP 1 Initial training 100% 02/2011 2 FP Health Counselors FP 1 Initial training 100% 03/2011 4 FP Health Counselors FP 1 Initial training 100% 04/2011 10 FP Health Counselors FP 1 Initial training 100% 06/2011 10 FP Health Counselors FP 1 Initial training 100% 07/2011 25 FP Health Counselors FP 1 Refresher training 100% 08/2011 51 FP Health Counselors FP 1 Refresher training 100% 09/2011 64 Health private care providers and paramedics FP 1 Refresher training 100% 10/2010 148 Health private care providers and paramedics FP 1 Refresher training 100% 11/2010 76 Health private care providers and paramedics FP 1 Refresher training 100% 12/2010 19 Health private care providers FP 3 Initial training 100% 01/2011 13 Health private care providers and paramedics FP 1 Refresher training 100% 03/2011 50 Health private care providers and paramedics FP 1 Refresher training 100% 05/2011 110 Health private care providers and paramedics FP 1 Refresher training 100% 06/2011 224 Health private care providers and paramedics FP 1 Refresher training 100% 07/2011 84 Health private care providers and paramedics FP 1 Refresher training 100% 08/2011 63 Health private care providers and paramedics FP 1 Refresher training 100% 09/2011 38 Top Réseau providers Standard day methods (Cyclebeads product) 1 Initial training 100% 09/2011 46 Top Réseau providers FP Long term methods 1 Initial training 100% 10/2010 1 Top Réseau providers FP Long term methods 1 Initial training 100% 11/2010 3 Top Réseau providers FP Long term methods 1 Initial training 100% 04/2011 26 Top Réseau providers FP Long term methods 1 Initial training 100% 05/2011 5 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 98 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Top Réseau providers FP Long term methods 1 Initial training 100% 08/2011 5 Top Réseau providers RH services and FP products 2 Refresher training 100% 10/2010 34 Top Réseau providers RH services and FP products 2 Refresher training 100% 11/2010 14 Top Réseau providers RH services and FP products 2 Refresher training 100% 12/2010 45 Top Réseau providers RH services and FP products 2 Refresher training 100% 01/2011 9 Top Réseau providers RH services and FP products 2 Refresher training 100% 05/2011 5 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 11/2010 10 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 03/2011 36 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 05/2011 3 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 07/2011 8 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 08/2011 41 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 09/2011 118 Top Réseau providers Cervical Cancer Detection 3 Initial training 67% 09/2011 39 Top Réseau providers Post-abortion care & postpartum hemorrhage (PAC/PPH) Counseling 1 Initial training 89% 09/2011 24 Top Réseau providers PAC/PPH Counseling 1 Initial training 79% 09/2011 22 Top Réseau providers PAC/PPH Counseling 1 Initial training 69% 09/2011 9 CHWs Pneumonia, Diarrhea 2 Initial training 100% 11/2011 52 CHWs Pneumonia, Diarrhea 2 Initial training 100% 12/2011 1170 CHWs Pneumonia, Diarrhea 2 Initial training 100% 01/2012 382 CHWs Pneumonia, Diarrhea 2 Initial training 100% 02/2012 1495 CHWs Pneumonia & pre-packaged therapy (PPT) 2 Initial training 100% 03/2012 16 CHWs Pneumonia and PPT 2 Initial training 100% 06/2012 16 Health private care providers Pneumonia, Diarrhea 4 Refresher training 100% 10/2011 57 Health private care providers Pneumonia, Diarrhea 4 Refresher training 100% 02/2012 27 Health private care providers Pneumonia, Diarrhea 4 Refresher training 100% 03/2012 20 Top Réseau providers IMCI 4 Refresher training 100% 04/2012 21 Top Réseau providers IMCI 4 Refresher training 100% 05/2012 5 Health private care providers Pneumonia, Diarrhea 1 Refresher training 100% 06/2012 63 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 99 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Health private care providers Pneumonia, Diarrhea 1 Refresher training 100% 09/2012 1 Sales persons pharmaceutical outlets Diarrhea 1 Refresher training 100% 02/2012 36 Sales persons pharmaceutical outlets Diarrhea 1 Refresher training 100% 06/2012 15 Sales persons pharmaceutical outlets Diarrhea 1 Refresher training 100% 07/2012 287 Sales persons pharmaceutical outlets Diarrhea 1 Refresher training 100% 09/2012 73 CHWs FP 3 Initial training 100% 10/2011 21 CHWs FP 3 Initial training 100% 12/2011 50 CHWs FP 3 Initial training 100% 02/2012 125 CHWs FP 3 Initial training 100% 03/2012 187 CHWs Depo-Provera injection 4 Initial training 100% 11/2011 7 CHWs Depo-Provera injection 4 Initial training 100% 01/2012 12 CHWs Depo-Provera injection 4 Initial training 100% 02/2012 60 CHWs Depo-Provera injection 4 Initial training 100% 03/2012 54 CHWs Depo-Provera injection 4 Initial training 100% 04/2012 38 CHWs Depo-Provera injection 4 Initial training 100% 05/2012 26 CHWs Standard day methods (Cyclebeads product) 3 Initial training 100% 02/2012 90 Youth peer educators STI, FP, RH training and IPC 3 Initial training 100% 08/2012 28 FP Health Counselors FP 5 Initial training 100% 11/2011 9 FP Health Counselors FP 5 Initial training 100% 12/2011 9 FP Health Counselors FP 5 Initial training 100% 01/2012 8 FP Health Counselors FP 5 Initial training 100% 02/2012 51 FP Health Counselors FP 5 Initial training 100% 03/2012 22 FP Health Counselors FP 5 Initial training 100% 05/2012 106 FP Health Counselors FP 5 Initial training 100% 06/2012 8 FP Health Counselors FP 5 Initial training 100% 07/2012 13 FP Health Counselors FP 5 Initial training 100% 08/2012 6 Sales persons pharmaceutical outlets FP 1 Initial training 100% 12/2011 1 Sales persons pharmaceutical outlets FP 1 Initial training 100% 02/2012 18 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 100 Training Participant ANNEX G: PSI/M TRAININGS Duratio n of Training (# days) Refresher or continuation of trainings percent of total participant type targeted (i.e., 100percent of all Top Réseau doctors) Month/ Year Training Total number of training participants Sales persons pharmaceutical outlets FP 1 Initial training 100% 03/2012 106 Sales persons pharmaceutical outlets FP 1 Initial training 100% 05/2012 28 Sales persons pharmaceutical outlets FP 1 Initial training 100% 06/2012 13 Sales persons pharmaceutical outlets FP 1 Initial training 100% 07/2012 289 Sales persons pharmaceutical outlets FP 1 Initial training 100% 09/2012 47 Health private care providers FP 1 Refresher training 100% 12/2011 13 Health private care providers FP 1 Refresher training 100% 03/2012 125 Health private care providers FP 1 Refresher training 100% 06/2012 25 Health private care providers FP 1 Refresher training 100% 08/2012 57 Top Réseau providers RH services and FP products 1 Initial training 100% 02/2012 8 Top Réseau providers RH services and FP products 1 Initial training 100% 03/2012 37 Top Réseau providers RH services and FP products 1 Initial training 100% 04/2012 7 Top Réseau providers RH services and FP products 1 Initial training 100% 05/2012 3 Top Réseau providers Standard day methods (Cyclebeads product) 1 Initial training 100% 12/2011 11 Top Réseau providers Standard day methods (Cyclebeads product) 1 Initial training 100% 02/2012 20 Top Réseau providers Standard day methods (Cyclebeads product) 1 Initial training 100% 03/2012 48 Top Réseau providers Cervical Cancer Detection 3 Initial training 100% 03/2012 33 Top Réseau providers PAC/PPH Counseling 3 Initial training 100% 04/2012 9 Top Réseau providers PAC/PPH Counseling 3 Initial training 100% 05/2012 16 Top Réseau providers PAC/PPH Counseling 3 Initial training 100% 06/2012 11 Top Réseau providers FP Long term methods 1 Refresher training 100% 12/2011 52 Top Réseau providers FP Long term methods 1 Refresher training 100% 03/2012 32 Top Réseau providers FP Long term methods 1 Refresher training 100% 04/2012 20 Top Réseau providers FP Long term methods 1 Refresher training 100% 05/2012 16 Top Réseau providers FP Long term methods 1 Refresher training 100% 06/2012 11 USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report    101 ANNEX J: SUPPLY CHAIN GRAPHIC USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 102 ANNEX K: AGGREGATED RESULTS OF SELECTED LSAT VARIABLES Presented here is a bar graph that displays the aggregated scores of selected variables from the Logistic System Assessment Tool guided site observations that the Evaluation Team conducted at the PSI/M central warehouse in Antananarivo and at eight regional PSI/M warehouses including: Fianarantsoa, Haute Matsiatra Region; Tôlanaro, Anosy Region; Morondava, Menabe Region; Antsiranana, Diana Region; Toamasina, Atsinanana Region; Toliara, Atsimo-Andrefana Region; and Mahajanga, Boeny Region. Please note that there is no score for “Organizational Support for Logistics System” in Mahajanga because data on this variable was not available at this site. Please note that LMIS stands for Logistics Management Information System. USAID/Madagascar: End-of-project Evaluation of the PSI Social Marketing Project – Evaluation Report 103 0 20 40 60 80 100 120 Antsirabe, Vakinankaratra Region Fianarantsoa, Haute Matsiatra Region Tôlanaro, Anosy Region Morondava, Menabe Region Antsiranana, Diana Region Toamasina, Atsinanana Region Toliara, Atsimo‐ Andrefana Region