USAID/HAITI: PROMARK PROJECT MIDTERM EVALUATION JANUARY 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Tim A. Clary through the Global Health Technical Assistance Project. USAID/HAITI: PROMARK PROJECT MIDTERM EVALUATION DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 11-01-436) is available in printed and online versions. Online documents can be located in the GH Tech website library at http:resources.ghtechproject.net. Documents are also made available through the Development Experience Clearing House (http://dec.usaid.gov). Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. USAID/HAITI: PROMARK MIDTERM EVALUATION i ACKNOWLEDGMENTS The author would like to acknowledge the assistance of the USAID/Haiti Health and Education team, particularly Wenser Estimé and Stephane Morriseau, the staff of the PROMARK Project, and GH Tech. ii USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION iii CONTENTS ACRONYMS ............................................................................................................ v EXECUTIVE SUMMARY ..................................................................................... vii Summary of Recommendations For Promark ....................................................................... vii BACKGROUND ..................................................................................................... 1 PROMARK’s Parameters.............................................................................................................. 1 Social Marketing: Basic Concepts ............................................................................................... 1 Previous social marketing programs in Haiti ............................................................................ 2 Country background and implications for PROMARK .......................................................... 2 METHODOLOGIES ............................................................................................... 5 PROMARK: ACHIEVEMENTS AND CHALLENGES ........................................ 9 Appropriateness of technical areas and approaches .............................................................. 9 Effectiveness of Activities and Products ..................................................................................10 Demand Creation .........................................................................................................................11 National Impact .............................................................................................................................11 Leadership, Management, and Sustainability Project ............................................................13 USAID’s Coordinating Role.......................................................................................................13 SUMMARY OF MAJOR FINDINGS ................................................................... 15 RECOMMENDATIONS ...................................................................................... 17 APPENDICES APPENDIX A. SCOPE OF WORK .................................................................... 21 APPENDIX B. PERSONS CONTACTED ........................................................ 27 APPENDIX C. PROMARK’S PERFORMANCE MONITORING PLAN ........ 29 APPENDIX D. REFERENCES ............................................................................ 37 TABLES Table 1. Socially Marketed Products and Year of Introduction to Haiti .................... 2 Table 2. Summary of Strengths and Challenges Observed During Site Visits .......... 7 iv USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION v ACRONYMS AIDS Acquired immune deficiency syndrome BCC Behavior change communication CA/POZ Christian Aid Zerosida CS Child survival CSW Commercial sex worker FC Field coordinator FoQUS Framework for Qualitative Research in Social Marketing FOSREF Foundation pour la Santé Reproductrice et l’Éducation Familiale FP Family planning HIV Human immunodeficiency syndrome IPC Interpersonal communication LDP Leadership Development Program LMS Leadership, Management, and Sustainability Project M&E Monitoring and evaluation MSPP Ministère de la Santé Publique et de la Population NGO Non-governmental organization OC Oral contraceptive ORS Oral rehydration salts PEPFAR President’s Emergency Fund for AIDS Relief PMP Performance management plan PSI Population Services International RH Reproductive health TBD To be determined TRaC Tracking Results Continuously USAID U.S. Agency for International Development vi USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION vii EXECUTIVE SUMMARY The PROMARK Project, a contract funded by the U.S. Agency for International Development (USAID) in Haiti and implemented by Population Services International (PSI), the prime contractor, started in 2009. Its primary goal is to improve the health of the Haitian people by using social marketing to advance healthy behaviors through behavior change communication (BCC), health product promotion, and sales strategies. Though the project works nationwide, it focuses on rural and remote areas and underserved populations. Branded and generic messages are used to promote healthy behaviors toward human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS), family planning (FP), and child survival (CS). In conjunction with the branded messages, PROMARK offers six products through social marketing. They include male condoms, female condoms, oral contraceptives (OCs), injectable Depo-Provera, a water treatment product, and oral rehydration salts (ORS). PSI has two partners (or subcontractors), Foundation pour la Santé Reproductrice et l’Éducation Familiale (FOSREF) and Christian Aid/Zerosida (CA/POZ). PSI is the leading organization for the socially marketed products and promotional activities, and FOSREF and CA/POZ focus on sensitizing the Haitian population through mass and interpersonal communication (IPC). Even though there were some early problems in implementing PROMARK, nothing could have prepared the project, or any other stakeholder, for the events of 2010. On January 12, a 7.0 magnitude earthquake struck Haiti, killing more than 200,000 people and leaving more than 1.5 million homeless. In October 2010, a cholera epidemic quickly spread to all ten departments, killing more than 3,000 people. Political unrest, centered on the December 2010 presidential election, also hampered aid efforts. These events, combined with the already-low ranking of Haiti in the human development index, continue to challenge all partners. Thus the midterm evaluation of PROMARK had to be accomplished in a difficult context. Rather than examining and evaluating PROMARK’s activities and results solely on intended targets, the evaluation team decided to look at systemic issues to produce a more useful review, assessment, and recommendations. SUMMARY OF RECOMMENDATIONS FOR PROMARK PROMARK’s achievements are detailed in the report. In brief, recommendations for strengthening the project are the following: Reexamine and Define Coverage Responsibilities Previous social marketing projects focused on urban areas and left rural areas uncovered, a problem PROMARK seeks to remedy. PROMARK needs to better define how far it should penetrate rural and remote areas. Strengthen Sales and Distribution Networks and the Supply Chain PROMARK needs not only to reexamine its geographic coverage, but also to strengthen its methods for ensuring that products reach the target populations. Stock-outs are common, and because of an expanding sales network, sales agents visit fewer points of sale. The system must be brought in line with the project’s goal, objectives, and budget. Revitalize and Control the Brands PROMARK’s branded products differ primarily in price, rather than quality. PROMARK needs a strategy to address brand slippage of its six products and control prices, which drive purchase. viii USAID/HAITI: PROMARK MIDTERM EVALUATION Use Optimal Communication Channels Because of poor infrastructure or ineffective media, some means of communicating PROMARK’s messages have proved more effective and sustainable than others. PROMARK should be prepared to use only channels that prove successful. Streamline and Strengthen Recording, Reporting, Monitoring, and Evaluation The format for reviewing PROMARK’s results still needs improving. Monthly reporting should focus only on quantitative results, using a shared database. Targets should be adjusted when needed, the project’s monitoring and evaluation (M&E) plan should be strengthened, and sufficient funding for M&E should be ensured. Focus on “big-picture” operational research Although PROMARK undertakes a number of research studies, it tends to examine only some brands or target populations. PROMARK needs to ask questions on general operations. Understand Contracting All major stakeholders (USAID, PSI, FOSREF, and CA/POZ) need to understand that PROMARK is a contract with subcontracts. Therefore, stakeholders must thoroughly understand contracting, adhere to contractual rules, and meet expectations. Cultivate Relationships and Build Capacity PROMARK has maintained and cultivated relationships well. However, its subcontractors still need substantial capacity building at the local (primarily) and central levels. Further, although PROMARK should be commended for its work to obtain greater Ministère de la Santé Publique et de la Population (MSPP) buy-in for social marketing, particularly at the local level, the project must still win the ministry’s complete recognition of social marketing’s value. The following report will provide the background of PROMARK, briefly cover basic social marketing concepts and previous social marketing programs in Haiti, and place the project in the context of Haiti’s epidemiological and political situation and recent events. Then it will present the methodology and summarize observations made during field visits. The evaluation will discuss the appropriateness of technical areas and current approaches; the effectiveness of activities and products; the demand created by the program; its impact at the national level; its collaboration with other U.S. government health projects; and MSPP support, coordination, and expectations. The evaluation will conclude with findings and recommendations for strengthening PROMARK. USAID/HAITI: PROMARK MIDTERM EVALUATION 1 BACKGROUND PROMARK’S PARAMETERS The PROMARK Project was awarded as a cost-plus, fixed-fee contract issued originally as a task order under the AIDSTAR indefinite quantity contract on April 15, 2009. It is a five-year project, with three base years and two option years and a total three-year budget ceiling of $13.5 million. The lead organization, PSI, works with two subcontractors, FOSREF and CA/POZ, which have three-year contracts valued at $797,243 and $1,045,243, respectively. Project activities began in May 2009 at a kick-off meeting, and full implementation began in June 2009. The primary goal of PROMARK is to reinforce social marketing as a viable strategy for improving the health of the Haitian people by promoting healthy behaviors through BCC, health￾product promotion, and sales strategies. Activities include campaigns for both branded and generic products. PROMARK focuses on HIV/AIDS, FP, and CS. Corresponding socially marketed products include the following: HIV: male condoms (Panté) and female condoms (Reyalité) FP: injectable Depo-Provera (Confiance) and OCs (Pilplan) CS: ORS (Sel Lavi) and a water treatment product (Dlo Lavi) PROMARK’s activities focus on expanding access to health products and information nationwide in rural and remote communities outside of Port-au-Prince. PROMARK works through the two subcontractors to implement community-based training and information, education, and communication with PSI’s supervision and technical guidance. PROMARK covers Haiti’s departments in the following way: FOSREF: North, Northwest, Northeast, Artibonite, Grande Anse, and the metropolitan area and south side of the West department CA/POZ: South, Southeast, Nippes, and the north side of the West department PROMARK focuses on reaching the following target populations: HIV: Commercial sex workers (CSWs), people living with HIV/AIDS, and youth (particularly as clients of CSWs) FP: All women of reproductive age, aged 15–49 and their partners CS: Caretakers responsible for children under five years of age SOCIAL MARKETING: BASIC CONCEPTS Social marketing is the methodical use of marketing techniques, along with other strategies and approaches, to achieve specific behavioral changes. Social marketers might have challenging, long-term behavior change goals, such as encouraging condom use and planned pregnancies and using water treatment systems. Social marketing, like commercial marketing, relies on standard market research to determine target populations, products and services, pricing, distribution channels and locations, and ways to communicate products’ benefits. Within social marketing, the ―total market approach‖ developed in response to the growing realization that neither donor nor host country governments, such as Haiti’s, had sufficient resources to develop markets for social goals. The total market approach selectively targets the poorest and most in need with free or subsidized goods and services through the public sector, donors, and nongovernmental organizations (NGOs) and opens the market to the commercial 2 USAID/HAITI: PROMARK MIDTERM EVALUATION sector, which introduces low-cost, mass-produced products to a segmented, targeted market. If done correctly, the approach can grow the market by reaching low-income (often rural) users through sales points, the informal sector, and community-based distribution. The total market approach underlies many of PROMARK’s activities. PREVIOUS SOCIAL MARKETING PROGRAMS IN HAITI USAID has supported social marketing programs in Haiti for more than 20 years. In 1989, the first male condom was introduced through social marketing, and since then USAID has promoted other health products (see table 1) through social marketing. Targeted health messages and mass media advertising that promote health products and services will result in positive behavior change and better health outcomes. Thus PROMARK is the continuation and culmination of two decades of experience and a valuable source of best practices and lessons learned—as long as they are accessed, utilized, and kept current. In short, there should be few surprises while implementing social marketing programs in Haiti. Table 1. Socially Marketed Products and Year of Introduction to Haiti Socially Marketed Product Date of Introduction in Haiti Male condom 1989 Female condom 1996 FP injectable 1996 OCs 1996 ORS 1998 Insecticide-treated nets 2005 Micronutrients 2006 Water treatment systems 2006 COUNTRY BACKGROUND AND IMPLICATIONS FOR PROMARK Health conditions in Haiti remain poor. Health indicators, although continuing to improve slowly, reveal a struggling health care system. Figures on the three health areas PROMARK addresses attest to the need for further development assistance. The total fertility rate in 2008 was 3.5,1 with 37.5% of women aged 15–49 having an unmet need for a FP method.2 The infant mortality rate in 2010 was approximately 77.26 per 1,000 live births,3 and the under-five mortality rate in 2006 was 117 per 1,000 live births.4 Though HIV/AIDS prevalence in Haiti appears to have plateaued, it remains stubbornly high. In 2009, there were approximately 120,000 people living with HIV, resulting in an estimated 1.9% prevalence in adults aged 15–49.5 All of the figures are the worst in the western hemisphere. The dire health statistics and ongoing political, social, and economic turmoil, which has affected Haiti for several decades, present challenges for PROMARK. Underdeveloped Haitian government institutions and weak governance have resulted in periodic crises, including, most 1 World Bank, World Development Indicators, 2008. 2 Demographic Health Survey, Haiti, 2005–2006. 3 CIA World Factbook, https://www.cia.gov/library/publications/the-world-factbook/fields/2091.html. 4 http://www.who.int/whosis/mort/profiles/mort_amro_hti_haiti.pdf. 5 http://www.unaids.org/en/regionscountries/countries/haiti/. USAID/HAITI: PROMARK MIDTERM EVALUATION 3 recently, the December 2010 presidential election. An uncertain outcome and perceived electoral corruption led to protests and a deadlock in political and development assistance. Project activities must be viewed in the context of the 2010 earthquake and cholera epidemic. On January 12, 2010, a 7.0 magnitude earthquake struck Haiti, killing more than 200,000 people. Like many other projects, PROMARK ceased and only slowly resumed. Indeed, a number of months passed before PROMARK could estimate how many of its sales points survived the earthquake. Cholera, which struck the country in October 2010 also diverted attention, time, and resources from PROMARK’s core mission. Though many stakeholders commended PROMARK for its responsiveness to the outbreak, the project was not designed to respond to emergencies. 4 USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION 5 METHODOLOGIES The methodology for conducting the PROMARK midterm evaluation consisted of a document review, stakeholder meetings, and site visits. The documents reviewed included the following: PROMARK’s master contract (PSI and USAID) and subcontracts Quarterly progress reports First annual report Research reports Sales and distribution plan Quarterly sales-point audit PROMARK’s M&E plan PROMARK’s annual report for President’s Emergency Fund for AIDS Relief (PEPFAR) Stakeholder meetings were conducted with USAID, PSI, FOSREF, CA/POZ, local-level MSPP representatives (in Arbitonite and Nippes), the Management Sciences for Health/Leadership, Management, and Sustainability (LMS) Project, and product retailers and wholesalers. Site visits were conducted in the Arbitonite (St. Marc) and Nippes (Miragoâne) departments on January 11 and January 13, 2011, respectively. During the site visits, meetings were held with the PSI field coordinator (FC), staff of FOSFREF and CA/POZ, MSPP representatives, wholesalers, and retailers. FIELD VISIT OBSERVATIONS Site visits were made to Arbitonite (in the north) and Nippes (in the south), which were chosen to observe the full range of PROMARK activities and meet other stakeholders. PSI Field Coordinators Both FCs were well aware of their responsibilities and believed that working in the field offices of their local partners had helped program implementation. However, they found that their partners’ lack of capacity had caused problems. Although activities were well coordinated, internally, through the submission of quarterly work plans and, externally, through the leadership of the MSPP, in many instances, activities were not carried out because of administrative difficulties between partner field offices and headquarters or the partners’ lack of technical knowledge and/or materials. The FCs noted difficulties with most of the branded products. In both departments, there was very low demand for Reyalité, and in Nippes, there was low demand for Panté because of supply of free condoms and perceptions of quality. Dlo Lavi and Sel Lavi were also in low demand because of distribution of free ORS by other organizations, stock-outs, or poor promotion. Both FCs noted that stock-outs continued to be a problem with all products, especially Dlo Lavi. Partner Organizations’ Staff FOSREF and CA/POZ are to be commended for highly dedicated staff, who face significant challenges at the field-office level. Specifically, there is an urgent need for technical and managerial capacity building. For example, staff in Arbitonite seemed unsure when or what type of activities had been implemented. In Nippes, there was a reluctance to conduct FP trainings because of insufficient training, few materials, and little enthusiasm for spending the necessary time and money to travel to more rural and remote areas. 6 USAID/HAITI: PROMARK MIDTERM EVALUATION In both departments, however, the most substantial issue was project administration, especially the slow or late flow of funding from headquarters to the field offices. In Arbitonite, activities stopped in August 2010 because of lack of funds, and in Nippes, implementation was affected most significantly by the slow flow of funds. As the lead organization, PSI needs to take responsibility for speeding the flow of funds. Neither site had an operations manual for implementing the project. A manual should be developed immediately. MSPP (Department Level) MSPP staff in Arbitonite and Nippes were very supportive of PROMARK and praised its willingness to coordinate activities with MSPP. Because of the close coordination and comprehensive work plans developed at the department level, MSPP felt there was little duplication of activities and good rapport between them and PROMARK. MSPP particularly cited PROMARK’s quick responsiveness to the cholera outbreak, and officials from both departments believed they were receiving sufficient information about the project and its results. Wholesalers The two wholesalers, Le Galien Pharmacie in Nippes and Depopharm in Arbitonite, sold all six branded products. Both noted that Confiance was the best seller and Reyalité, the worst. Although FCs and sales agents often visited, periodic stock-outs of some products still occurred. Both wholesalers were grateful for promotional materials and products supplied by the FCs and sales agents and felt they had enough technical information to allow them to advise customers on all of the products. Both believed that price most influenced consumers’ purchase of branded products. Le Galien Pharmacie, a youth-friendly sales point, said that the training and follow-up visit by PSI’s FC was of great assistance. Although both wholesalers confronted product expiration, for Depopharm the problem was worse, perhaps because, unlike Le Galien Pharmacie, it stocked only PROMARK’s branded products. Retailers The retail sites visited were quite varied. Evaluators traveled to a well-established pharmacy in an upscale location in Arbitonite and a street kiosk and a roadside vendor in Nippes. In Arbitonite, the pharmacy sold all six products, but in Nippes one site carried only Panté, and the other offered Dlo Lavi, Sel Lavi, Panté, and Pilplan, though the owner admitted that he was not selling Pilplan officially. The retailer in Arbitonite repeatedly criticized the new policy on expired products; the owners said they had switched from the wholesale to the retail business because of expiration issues. With large stocks of Panté set to expire, they had resorted to giving them away as gifts with other purchases. Because customers viewed Panté as a low-quality product, its sales were low and supplies expired. Sales of Dlo Lavi were also low because, they believed, of competition from Aquatabs. In contrast, because of high sales, ―Confiance‖ and ―Sel Lavi‖ were increasingly used to mean any injectable or ORS, respectively. All retailers had periodic stock-outs, although they had means of mitigating them. The Arbitonite retailer drove to Port-au-Prince to buy stock; one of the Nippes retailers contacted the sales agent, and the other bought from other retail sales points. All remarked that price differentiated the socially marketed products from others available in Haiti. USAID/HAITI: PROMARK MIDTERM EVALUATION 7 Table 2. Summary of Strengths and Challenges Observed During Site Visits Strengths Challenges 1) PSI FCs well established in their roles 2) Strong brand recognition 3) MSSP cooperative and taking responsibility for coordination 4) Good commitment from local networks 5) Wholesalers and retailers generally supportive of products 1) Continuing stock-outs 2) Products differentiated mainly by price, not brand quality 3) Expired-product policy affecting wholesaler support 4) Lack of administrative (primarily funds flow) and technical support for subcontractor networks 5) Ability to reach more remote areas limited 8 USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION 9 PROMARK: ACHIEVEMENTS AND CHALLENGES The following section forms the core of the evaluation and is based on the following six questions from the scope of work for the consultancy: Has PROMARK appropriately and effectively provided socially marketed products (ORS, condoms, and other contraceptives) at affordable prices to help Haitians change their behavior and lead healthier, happier lives? Does the project increase the targeted vulnerable population’s opportunity, ability, and motivation to use socially marketed products? Do the program indicators match the social marketing goals? Identify possible gaps. Are the technical areas and current approaches appropriate now and for follow-on programming? What are the gaps, if any? Provide recommendations to address any gaps. PROMARK works across Haiti. Has the intervention had an impact on the public at the county level? Are project interventions adequate? Have PROMARK activities been effective in completing other U.S. government health promotion and BCC activities or projects? Does PROMARK reach the government of Haiti/Ministry of Health expectations by integrating the strategic planning of the Department of Health Promotion? The questions should aid in examination of the appropriateness of technical areas and current approaches; effectiveness of activities and products; demand creation; national impact; collaboration with other U.S. government projects and partners; and MSPP support, coordination, and expectations. APPROPRIATENESS OF TECHNICAL AREAS AND APPROACHES As shown in the background section, the statistics on HIV/AIDS, FP, and CS are dire. Approximately 37% of the PROMARK budget is devoted to HIV/AIDS, 33% to FP, and 30% to CS. There is little evidence of an integrated approach to activities, even though PROMARK has been tasked with forming one. Network partners may be reluctant to discuss technical areas with which they are less familiar (FP), and headquarters may need to engage in more strategic planning. The overall approach takes advantage of each organization’s strengths: PSI focuses on branded and generic promotions, materials development, and supervision, while the subcontractors’ networks conduct local-level trainings, mass sensitization, and IPC. During the site visits, staff consistently commented that the project’s structure was effective in delivering the needed services and knowledge to the target populations. It eliminates duplication; preexisting networks can access populations easily because of their local knowledge. In practice, the outcomes have been less than ideal. For reasons mentioned in the section on field-visit observations, local-level activities have been delayed, ignored, or halted. For example, in Arbitonite, FOSREF undertook no activities in five months because of lack of funds. Further, as noted in many of PROMARK’s quarterly reports, this was not a single occurrence. Activities designed for the specific brands have proceeded, but those for generics have lagged. Technical areas such as FP have been most affected. PROMARK’s ―catch-up‖ plan might remedy the situation but might not sufficiently reach target populations in the rural and remote areas. The project may need to spend considerable additional effort on strengthening the capacity of the subcontractors or on developing new networks. 10 USAID/HAITI: PROMARK MIDTERM EVALUATION Whether PROMARK is expanding geographically in the most (cost-) effective manner is also questionable. Reaching underserved populations in rural and remote areas, where there have been little social marketing and few services, seems appropriate. However, there has been little analysis of the marginal costs of accessing geographic zones. Now would seem the ideal time to examine marginal costs, given the upcoming release of the new USAID/Haiti health strategy, which will include a revised geographic scope. Finally, all immediate stakeholders (PSI, FOSREF, CA/POZ, and USAID) must be aware of their contractual responsibilities and of how funding is handled. They must complete activities in a timely manner or provide convincing reasons for delays. Because they will be reimbursed after implementation, they must have sufficient funds. They must also understand that USAID will have substantial involvement in the management of the contract. In turn, USAID management must take responsibility for ensuring that their actions and decisions, or lack thereof, do not affect the timely implementation of the contract. They should not make any changes to the program description without amending the contract, providing additional funding, and receiving full agreement from the contractor. The events of 2010 and subsequent substantial increases in workloads may have caused delays in approval or production of materials and slowed implementation. EFFECTIVENESS OF ACTIVITIES AND PRODUCTS Judging by the sales of branded products, PROMARK has not substantially expanded the market. PROMARK’s audit does not show overall growth in sales. Although the target populations appreciated information about generic and branded products, their behaviors have not changed significantly. Persistent stock-outs of many products, the earthquake, and the need to revitalize the branded products may have affected sales. PROMARK must be given credit for recognizing and addressing the issue of low sales early in its implementation. Through formal research and informal market surveys and audits, PROMARK is finding better ways to reach its target audience. It has plans to improve a number of its branded products and better understand its target audience, but it must make objective decisions about a number of its products. For example, youth vulnerable to HIV infection see Panté as a low-cost, low-quality product. PROMARK must decide whether to rebrand the whole product, create sub-brands, or start a whole new line. Likewise, despite introduction over 14 years ago, sales of Reyalité remain woefully low. Respondents say it is difficult to use, relatively expensive, and limited in its market appeal. PROMARK needs to determine whether to continue the product. Finally, widespread introduction of Aquatabs will force PROMARK to decide whether to keep Dlo Lavi (a liquid), move to tablets for water treatment, or combine the two. The same objective examination must be made of the messages and channels through which PROMARK communicates. For example, despite the resources put into promoting the FP hotline, the number of calls it receives each month remains quite low. Although insufficient infrastructure may be partly to blame, PROMARK must better organize the hotline. Because survey results show that most of the target population receives and remembers messages about branded and generic products via radio, PROMARK should reconsider using the networks, particularly if local-level activities continue to be delayed for administrative reasons. Finally, PROMARK must ensure that its promotions of branded and generic products are synchronized. Work plans must balance brand promotion and health promotion. Without promoting healthy behavior, all brands will be undermined. USAID/HAITI: PROMARK MIDTERM EVALUATION 11 DEMAND CREATION PROMARK’s success in creating demand for its products was difficult to assess because of the events of 2010. The 2010 quarterly audit report showed that a significant percentage of the target populations had seen or heard a PROMARK-generated social marketing message during the previous three months (primarily on the radio), but a review of sales records did not reveal that the messages had resulted in greater demand for the branded products. In most cases, the demand for nonbranded products was even lower. According to PROMARK’s first annual report, the project had exceeded its target for opening new sales points by 125% by the end of September 2010. It had reached more than 490,000 people through mass sensitization activities and more than 250,000 people through promotional activities, 274 of which focused on branded products. Yet sales for almost all of the products were significantly below their original targets. Sales of Panté and Reyalité only reached 55% and 82%, respectively, of their targets, although the target for Reyalité (232,500) was substantially lower than that for Panté (7,875,000). Sales for Confiance reached 47% of the target, Sel Lavi 70%, and Dlo Lavi 67%. Only Pilplan (99%) nearly reached its target, but several PROMARK reports mentioned special circumstances that may have unduly influenced Pilplan’s sales. 6 Certainly, management of the supply chain influenced some of the lower than expected results. Dlo Lavi, for example, is manufactured in the United States and has a relatively short lifespan. Thus it can only be ordered in small quantities and requires a long lead time. Similarly, because the packaging for Reyalité and Pilplan is made outside Haiti, the products can be delayed. Given that international development should never create demand for unavailable products and services, the supply chain needs immediate attention. This is particularly troublesome because the lead contractor, PSI, has been in the country for many years. The January 2010 earthquake affected activities, vendors, and thus demand for products. All partners noted that, for a number of months in early 2010, activities either slowed substantially or halted completely. Approximately 30% of sales points were affected, as the Haitian people became preoccupied with rebuilding their lives. The cholera outbreak in late 2010 also influenced sales, particularly of the water treatment product and ORS. Distribution of free ORS in a number of departments unsurprisingly caused ORS sales to drop. The already-low quantities of Dlo Lavi and its limited free distribution after the earthquake and during the cholera epidemic negatively affected sales. Considering the competition from the more available Aquatabs, Dlo Lavi sales were reasonable. In short, there has been insufficient demand creation through either brand activities or generic messages. Sales plateaued, and in some cases (notably, Panté, Reyalité, and Dlo Lavi), there is real risk of brand slippage. PROMARK must reconsider the activities, messages, and channels it uses to promote branded and generic products and rethink some of the brands. Ironically, demand continues to be high for the two FP products with which the subcontractors are least comfortable. This begs the question of whether the activities are having any real influence on the target populations. NATIONAL IMPACT Is part of PROMARK’s goal to have a national impact? If so, can the impact be measured? The first question is slightly easier to answer than the second. Because PROMARK is designed to operate nationwide and influence behaviors by expanding accessibility to health products and 6 PSI/Haiti Special Report on Management of Sales: June 1, 2010. 12 USAID/HAITI: PROMARK MIDTERM EVALUATION information, it should have some level of national impact. Though PROMARK must reexamine how many of its resources it should devote to reaching each geographic zone, its activities are sufficiently large in scale to have some effect nationwide. In the future, PROMARK anticipates measuring its impact through disability-adjusted life years or another mechanism. Specific attribution at the national level will, as always, be challenging to obtain. The second question is more difficult, and its answer lies in examining the project’s performance monitoring plan (see appendix C), its M&E system, and the research it conducts to measure results. An examination of the PMP in the September 2010 annual report reveals some issues. Some of the indicators seem unnecessary or duplicative (such as measuring the percentage of most-at￾risk populations reached and breaking down the results into the subgroups). Others (such as the percentage of child caregivers interviewed in social marketing target sites who correctly treated the last episodic case of childhood diarrhea with ORS) may not be worth measuring because they show insignificant change. Given PROMARK’s goal, there should be consistent disaggregation of urban, rural, and remote areas. Finally, none of the indicators truly measure national impact. In view of its scale, the project should not be held accountable for any major impact at the country level. In reviewing the PMP and the results, the immediate questions are whether PROMARK adequately set its original targets and, given the events of 2010, why the targets were not recalibrated. Because numerous sales points were destroyed, activities curtailed, resources redirected, and a catch-up plan developed, PROMARK and USAID should have adjusted targets collaboratively. It serves no purpose for the project to retain its original targets if factors outside of its control cause it not to deliver intended results. Also problematic are targets yet to be determined (TBD), deciding which targets are cumulative, and correcting any lack of correspondence between indictors. (For example, compare the target for ―Number of people receiving information about FP through IPC+‖ and ―Percentage of women and men interviewed in social marketing target areas who have seen or heard a social marketing FP and reproductive health (RH) message.‖) In order to measure the results, there needs to be a sufficient M&E system. Although the field and central levels insisted there was one, only slightly more than $8,000 (approximately 0.4%) of the subcontractors’ approximately $1.8 million budget and only 4.6% of the lead contractor’s budget were dedicated to M&E. Even though meetings and site visits did not uncover any major issues with the M&E system, the lead contractor, PSI, and subcontractors, FOSREF and CA/POZ, should provide USAID an M&E-specific work plan and budget. After all, in a standard M&E plan, one of the usual annexes is a work plan and budget. Results also need to be presented in a manner technical and nontechnical audiences can understand, in part, to justify additional funding. While reviewing the quarterly reports, evaluators often had difficulty understanding which time periods were covered, whether they were comparable, and, thus, whether the project was on track to achieve results. Clarity seems to be improving. Finally, PROMARK may not be asking all of the research questions it should. Although the project is very good about asking brand- and method-specific questions through its Tracking Results Continuously (TRaC) and FoQUS methodologies, some important questions remain unanswered, including: What are the marginal costs in terms of remoteness of reaching different geographic zones? USAID/HAITI: PROMARK MIDTERM EVALUATION 13 Is the rapidly expanding sales network effective in increasing product sales and reaching the target population? Because PROMARK tracks the number of sales points per product, deriving the average number of product sales per point should be easy. What has been the return on investment for each branded product? In other words, how much money has been dedicated to promoting each product and what have the expenditures led to in terms of sales? LEADERSHIP, MANAGEMENT, AND SUSTAINABILITY PROJECT The LMS Project, which initiated activities at approximately the same time, addresses several areas that overlap with PROMARK, specifically, HIV-community activities, FP support to the MSPP, capacity building through the Leadership Development Program (LDP), and community management of FP commodities provided by USAID. With similar but distinct approaches, there have been opportunities for PROMARK and LMS to collaborate. Primary among their collaborative efforts is the joint work-plan analysis, or pipeline exercise, which the MSPP leads every six months. It allows all USAID-funded health partners (primarily, LMS, PROMARK, and USAID’s Community Health and AIDS Mitigation Project to examine each other’s two-year work plans to eliminate duplication of activities and look for opportunities to synchronize their work. After the January 2010 earthquake, all partners united to develop a single logistics and transport system for supplies and develop educational materials. There remain additional opportunities for collaboration. The three partners should consider developing a comprehensive work plan that would not only allow them to see duplication, but also obligate them to better coordinate activities. LMS, through its LDP initiative, has trained central-level FOSREF staff in ensuring timely implementation of activities. LMS intends to roll out the training to the department level. If PROMARK worked closely with LMS, they could make sure FOSREF staff obtained the same skills. And if PROMARK distributed socially marketed products through the CA/POZ networks, results could be shared with LMS to determine whether FOSREF, which is already distributing free commodities, could distribute socially marketed products. USAID’S COORDINATING ROLE Partners at the central and local levels said that USAID had helped coordinate projects because of MSPP’s limited capacity. By strengthening the MSPP through a health systems approach, USAID’s leadership should decrease. However, because USAID is the major supplier of free and socially marketed products, it will still have to convene partners to discuss technical issues, provide feedback, make decisions, and supply all products in a timely manner. 14 USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION 15 SUMMARY OF MAJOR FINDINGS Social marketing helps provide health products to segments of the Haitian population and should be incorporated into USAID’s continued strategy of supporting health in Haiti. Product sales are not increasing despite the growing penetration of sales points. The value of all brands must be examined to determine whether they need to be rebranded, revitalized, or dropped, along with corresponding messages and media. Stock-outs continue to be an issue. The distribution system needs to be rationalized because stock-outs are probably affecting sales. The project needs a balanced approach toward its branded and generic messages. Issues of coverage (rural and remote) need to be addressed better. Management of the project has been very responsive but needs to be more proactive. Activities have not always proceeded as planned, not only because of the earthquake and cholera outbreak, but also because of delays by the lead contractor, subcontractor, and USAID. Subcontractors need more capacity building at the local and central levels. All project partners must better understand contracting requirements Though still not fully appreciated by the MSPP, social marketing’s value is more widely understood, particularly at the local level. Collaboration is good but could be strengthened at the central level. In practice, M&E appears adequate, but its documentation needs refinement. 16 USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION 17 RECOMMENDATIONS 1) PROMARK, in conjunction with USAID, must better establish its geographic coverage. PROMARK should not change its primary focus on reaching rural and remote areas but should use specific criteria to determine in which rural and remote areas it should implement activities. PROMARK must ask two questions: What changes might be forthcoming in USAID/Haiti’s new health strategy and geographic focus? What are the marginal costs for overcoming geographic barriers? Both subcontractors’ networks noted that accessing some of the more remote areas consumes disproportionate resources. They also found finding volunteers or agents to go to remote areas difficult. PROMARK may need to base its sales and distribution networks on wholesalers and strengthen its capacity to perform geographic analysis. Given that PROMARK has already initiated a global positioning system for its sales points, using geographical information systems for data analysis should not be too difficult. 2) Even though social marketing has been in Haiti more than 20 years, the supply chain and logistics for socially marketed products remain inadequate to ensure consistent stocks at sales points. Not all of the stock-outs can be attributed to the earthquake or cholera epidemic. There are systemic problems, including external sourcing of products and packaging; overexpansion of sales networks; product expiration and wholesalers’ inability to manage standard quantities; sales agents’ having too many sales points to visit regularly; poor distribution that does not take advantage of wholesalers and NGO networks; and insufficient information and coordination with other stakeholders, including those who provide free products through the public sector and higher cost products through the private sector. Addressing these issues is particularly urgent because promoting unavailable products and services can be detrimental to health. Additionally, unless PROMARK can ensure proper disposal of expired products, there may be environmental repercussions. PROMARK should, first and foremost, rationalize its sales and distribution network by developing a hierarchy in which each level is responsible for a limited number of subordinate distribution points and there is no duplication of sales or distribution points. PROMARK must examine its geographic coverage and number and types of sales points to see if it has an optimal system for reaching the target populations and geographic areas. PROMARK also must accept expired products or develop a plan for proper disposal. Finally, if needed, PROMARK must get technical assistance, internally or from its partners, to strengthen its supply chain, particularly to choose sources for products and packaging. 3) Currently, the socially marketed products are primarily differentiated by price. Although this is reasonable for market segmentation, it is not sustainable given the free products and competitive products available through the private sector. PROMARK must consider quality, ensure products are priced correctly, control prices, revitalize or rebrand products as needed, and continue surveying the target populations to guarantee that project messaging translates into product sales. As noted in PROMARK’s quarterly audit report, vendors do not always charge the recommended prices. Although PROMARK must dispose of products before introducing new ones in price-marked packaging, it cannot delay much longer, especially because of the sometimes-long lead times necessary for products to reach sales points. Spot checks by field staff and quarterly audits should assure reasonable price control. 18 USAID/HAITI: PROMARK MIDTERM EVALUATION If evaluation does not demonstrate a product’s value, PROMARK and USAID must decide if they should rebrand or replace it with another one, such as Aquatabs. All products need periodic examination to determine if they are worth keeping. 4) Although its messages about branded and generic products are generally thought appropriate, some activities merit reexamination. For example, PROMARK has noted that blitz activities do not sustain sales. Poor infrastructure limits the FP hotline, and the youth￾friendly sales points have had only moderate success. Because reaching youth, through sales points, clinics, or the hotline, is one of PROMARK’s primary objectives, PROMARK should work harder to make sure staff of sales points are sufficiently trained and visited. PROMARK also needs to set a deadline for resolving the hotline’s infrastructure issues, assign oversight to another partner, or drop the activity. In order to sustain behavior change, especially of difficult-to-reach groups, repeated interventions and messaging are necessary. PROMARK might consider shifting to a more geographically focused and continuous presence in target areas through mobile providers of information and products. To support branded products, there must be sufficient messaging through mass media, sensitization activities, or IPC. PROMARK needs to ensure that its local partners have sufficient capacity and educators have adequate demonstration tools and materials. 5) PROMARK’s implementation of monitoring and evaluation is satisfactory, but M&E documentation and structure are not particularly impressive. Specifically, some of the indicators in the PMP are somewhat duplicative, and others yield values too insignificant to warrant including information on their measurement. Almost all of the targets need to be reset, especially given the events of 2010. PROMARK must also ensure that the local partners (FOSREF and CA/POZ) report only quantitative results monthly, share a database to facilitate reporting, and adequately budget for M&E. All partners should consider developing M&E work plans and budgets; doing so should be required. Despite delays in a number of its research activities, PROMARK has made some progress through TRaC and FoQUS. But they concentrate on product-specific questions, not the ―big picture.‖ There should be analysis of return on investment in the six branded products, so PROMARK can decide whether to drop, further segment, or revitalize brands. Simple examination of the connection between opening additional sales points and product sales could lead to further research into optimal arrangements for sales and distribution. 6) The role of the subcontractors and their local networks in accessing the target populations is crucial. The MSPP supports the subcontractors’ role, so the ability of FOSREF and CA/POZ to implement their portions of the PROMARK Project must be assured. Unfortunately, lack of funding has resulted in halting some activities, and motivation is low for volunteers because they do not receive incentives. Similarly, a lack of standardized and documented operational procedures and trainers’ not having enough materials, supplies, and instructional tools have slowed implementation. Finally, PROMARK’s goal is undermined by the reluctance of some agents and volunteers to carry out training activities in FP because it is new to them. USAID/HAITI: PROMARK MIDTERM EVALUATION 19 PROMARK has two options: First, PSI, as the lead partner, can build the capacity of its local partners sufficiently to ensure their success. The capacity building should focus primarily on strengthening the two organizations’ management at the central and department levels and their abilities to provide technical education on FP. Operations manuals should be developed, training on USAID contracting should be provided, and lessons on financial management should be offered. PROMARK must help CA/POZ examine the subcontract and project budgets to see if there are funds to purchase volunteer incentives. For both organizations, PROMARK must develop a capacity-building plan. Second, PROMARK can explore developing new networks to serve as partners in geographic areas where FOSREF and CA/POZ are consistently unable to effectively implement activities and thus achieve results. However, this option would be costly, time consuming, and run against the MSPP. Nevertheless, all partners must recognize that PROMARK is a contract and that results ensure even payment. 7) Although the evaluators commend PROMARK’s senior management for taking immediate steps to resolve issues brought to its attention, it must be more proactive. Given more than 20 years of social marketing and management experience in Haiti, there are few excuses for lethargy or delays in activities, at least in normal, noncatastrophic times. PROMARK was not designed to be a relief project, thus its organizational structure should not be held too accountable for struggling to handle emergency requests. Although there was proper coordination at the local level, PROMARK should coordinate its activities with partners under the leadership of the MSPP. PROMARK depends on USAID for oversight, guidance, decisions, and timely organization of activities. More of PROMARK’s senior management should obtain mentoring from PSI headquarters, if necessary, on contract management at the USAID, prime-contractor, and subcontractor levels. Even though PROMARK is PSI’s first contract, the differences between agreements and acquisitions must be mastered thoroughly and quickly. PROMARK should undertake biannual strategic reviews of its activities, focusing on the overall vision of the project rather than on field-driven demands. Although the project wants to respond to all beneficiaries, only at the central level can efforts can be charted, planned, and coordinated sufficiently for cohesive action. Finally, PROMARK must better engage the private sector and integrate its three technical areas (HIV, FP, and CS), perhaps through a more general health systems– strengthening approach at the department level, within the NGO networks, or through stronger dual marketing of products (e.g., promoting condoms as an HIV-prevention and FP method). Engagement of the private sector can continue through the Total Market Action and Innovation Forum, although it has had limited success because the private sector is reluctant to share data. First and foremost, PROMARK must share its data on publicly and socially marketed products with its private sector partners. 20 USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION 21 APPENDIX A. SCOPE OF WORK Global Health Technical Assistance Project GH Tech Contract No. GHS-I-00-05-00005-00 SCOPE OF WORK I. TITLE Activity: USAID/Haiti: PROMARK Project Evaluation Contract: Global Health Technical Assistance Project (GH Tech), Task Order No. 01 II. PERFORMANCE PERIOD o/a January 1, 2010–February 28, 2010 III. FUNDING SOURCE Mission IV. PURPOSE A short-term consultant is required to conduct a mid-term evaluation of PSI/PROMARK’s design, implementation, and mid-term achievements in order to develop lessons learned that can be integrated into the current program and future design programs. V. BACKGROUND PSI/Haiti was set up in 1989 aiming at increasing access to high quality, affordable condoms, and at delivering HIV/AIDS prevention messages to low-income populations across the country. For more than 15 years, PSI/Haiti has been working to increase HIV/AIDS awareness, to reduce high risk behaviors, and to provide easy access to affordable health services and products. Through its social marketing approach, PSI/Haiti delivers its products to Haitians through a network of traditional outlets (pharmacies, health centers, etc.) and nontraditional outlets (kiosks, markets, street vendors, community-based distribution, etc.). Since 1989, PSI/Haiti has sold over 100 million male condoms and over 300,000 female condoms. With support of USAID, PSI/Haiti introduced in 1996, Pilplan (an oral contraceptive) and Confiance (a 3-month injectable contraceptive). PSI/Haiti works with pharmaceutical distributors and retailers to make these products available to women of reproductive age across the country. (From 1996 to 2008, PSI/Haiti has sold nearly 1,600,000 units of Pilplan and nearly 1,900,000 units of Confiance, leading to 532,906 Couples Years of Protection (CYPs). In April 2009, PSI/Haiti obtained funds for a project financed by USAID in order to conduct a project of social marketing in Haiti, PROMARK. This project has three parts or components: 1) HIV/AIDS, 2) Family Planning and Reproductive Health, and 3) Child Survival. Several indicators regularly have to be monitored for each field or component of the project. This document describes how these indicators will be obtained, namely: their source, the methodology, the 22 USAID/HAITI: PROMARK MIDTERM EVALUATION data collection procedures, their periodicity, etc. Two local partners FOSREF and POZ/Christian Aid are associated with PSI/Haiti, for the execution of this project. VI. SCOPE OF WORK The consultant will evaluate the PSI/PROMARK project and focus on the effectiveness and outcomes of USAID’s social marketing assistance. The overall objectives of the evaluation are to: 1. Determine success of meeting proposed benchmark indicators for PSI/PROMARK social marketing strategy and whether interventions were sufficient to reach the desired outcomes according to the workplan and targets 2. Evaluate the current level of collaboration with other U.S. government programs and non￾USG development partners and make suggestions for improving synergies 3. Provide recommendations and improvements for the follow-on program. When conducting the evaluation, the consultant should consider the following illustrative questions: The methodology used should consider addressing the following questions as well as others in order to meet USAID expectations: Have PROMARK activities been appropriate and effective in contributing to enable Haitian populations to change their behavior and lead healthier, happier lives by providing socially marketed products (ORS, condoms and other contraceptives) at an affordable price to the targeted population? Does the project generate increased ―demand‖ in social marketing product in the target vulnerable population by increasing their opportunity, ability, and motivation to use socially marketed products? Check the program indicators and how they match the social marketing goals and identify possible gaps. Are the technical areas and current approaches appropriate now and for follow-on programming? What are the gaps, if any? Provide recommendations to address gaps. The PROMARK project works currently in all of Haiti. Has this intervention increased public impact at the county level? Are project interventions adequate for improving? Have PROMARK activities been effective by completing other U.S. Government Health Promotion and/or Behavior Change Communication activities or projects? Does PROMARK reach the Government of Haiti/MOH expectations by integrating the strategic planning of the Department of Health Promotion? VII. METHODOLOGY Relevant data will be gathered from multiple sources including site visits to the counties, interviews with the Ministry of Health representative, PSI/PROMARK representative, local counterparts, other international donors and implementers, and review of project’s program and financial documents and reports. The evaluation team will conduct field visits to access project performance using qualitative and quantitative data collection methods. USAID/HAITI: PROMARK MIDTERM EVALUATION 23 VIII. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT One consultant will be needed for a period of 3 weeks in-country. A six-day work week is authorized while in-country. Required qualifications include: Advanced degree (Masters or above) in Public Health or in a field related to an area of expertise required for evaluations (e.g., participatory research, monitoring and evaluation); Minimum of ten years experience in the monitoring and evaluation of development activities; Demonstrated experience with and understanding of monitoring and evaluation requirements of HIV/AIDS prevention, family planning, or child survival programs; Extensive experience in assessing the impact of development projects. S/he will be knowledgeable of USAID performance indicators, data quality assessment procedures (DQA), and performance management plans (PMP); Excellent English writing and communication skills and French fluency or sufficient skills to be able to conduct interviews in French; Experience interacting with developing country government, international organizations, other bilateral donor and civil society representatives, and senior-level government officials; Ability to work with diverse international teams and excellent interpersonal skills. Illustrative LOE Table Task LOE Document Review 2 Travel to and from Haiti 2 Meeting with USAID/Haiti 1 Information Collection and Site Visits 7 Data Analysis and Report Writing 6 Debrief with USAID/Haiti 1 Incorporating USAID comments in draft report 2 Revising/Finalizing report for submission to USAID 3 Total 24 days IX. LOGISTICS GH Tech will arrange logistics for the consultant’s travel to Haiti. USAID/Haiti will provide basic logistics (clearances in liaison with the GOH and USAID partners, lodging recommendations) and some administrative support for the team. GH Tech will arrange and pay for lodging and workspace and equipment as needed. X. DELIVERABLES AND PRODUCTS Debrief with USAID/Haiti prior to departure. Draft Evaluation Report: The draft report will analyze the replicability, feasibility, scale up and sustainability of existing and proposed program models. It will also highlight achievements with reference to established plans and objectives and discussion of challenges 24 USAID/HAITI: PROMARK MIDTERM EVALUATION and recommendations, and promising practices country-specific report. The draft report will be given to the Mission prior to the consultant’s departure. Final Evaluation Report: After the consultant departs, USAID/Haiti will have 10 working days to review the report and provide one set of written comments. The consultant will then submit the final draft to the Mission one week after receiving comments from USAID/Haiti. GH Tech will provide the edited and formatted final document approximately 30 days after USAID provides final approval of the content. USAID/Haiti requests both an electronic version of the final report (Microsoft Word 2003 format) and 5 hard copies of the report. The report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) (http://dec.usaid.gov) and the GH Tech project web site www.ghtechproject.com). XI. RELATIONSHIPS AND RESPONSIBILITIES The consultant will work under the general guidance of IHSI and USAID/Haiti. GH Tech will conduct and manage the consultant and will undertake the following specific responsibilities throughout the assignment: Recruit and hire the consultant Make logistical arrangements for the consultant, including travel and transportation, country travel clearance, lodging, and communications Respond to all points included in the SOW, including the submission of final audit report USAID/ Haiti will provide overall technical leadership and direction for the consultant team throughout the assignment and will undertake the following specific roles and responsibilities: Prior to in-country work: Consultant Conflict of Interest: To avoid conflicts of interest (COI) or the appearance of a COI, review previous employers listed on the curricula vitae for proposed consultants and provide additional information regarding any potential COI Background Documents: Identify and prioritize background materials for the consultant and provide them to GH Tech as early as possible prior to team work Key Informant and Site Visit Preparations: Provide a list of key informants, site visit locations, and suggested length of field visits for use in planning for in-country travel and accurate estimation of country travel line items costs (i.e., number of in-country travel days required to reach each destination and number of days allocated for interviews at each site) Lodging and Travel: Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation) During in-country work: USAID/Haiti will undertake the following while the team is in country: Mission Point of Contact: Ensure constant availability of the Mission Point of Contact person to provide technical leadership and direction for the consultant’s work Meeting Space: Provide guidance on the team’s/consultant’s selection of a meeting space for interviews and/or focus group discussions (i.e., USAID space, if available, or other known office/hotel meeting space) Meeting Arrangements: While consultants typically will arrange meetings for contacts outside the Mission, support the consultant/team in coordinating meetings with stakeholders USAID/HAITI: PROMARK MIDTERM EVALUATION 25 Formal and Official Meetings: Arrange key appointments with national and local government officials and accompany the team/consultant on these introductory interviews (especially important in high-level meetings) Other Meetings: If appropriate, assist in identifying and helping to set up meetings with local professionals relevant to the assignment Facilitate Contacts with Partners: Introduce the team to local government officials and other stakeholders and, where applicable and appropriate, prepare and send out an introduction letter for team’s/consultant’s arrival and/or anticipated meetings Following in-country work: USAID/Haiti will undertake the following once the in-country work is completed: Timely reviews: Provide timely review of draft/final draft reports and approval of the deliverables XII. MISSION AND/OR WASHINGTON CONTACT PEOPLE/PERSON Stephane Morisseau, DDS, MPH Public Health Specialist USAID/Haiti 509-2229-8273 509-2229-6273 509-3701-6660 (cell) smorisseau@usaid.gov 26 USAID/HAITI: PROMARK MIDTERM EVALUATION USAID/HAITI: PROMARK MIDTERM EVALUATION 27 APPENDIX B. PERSONS CONTACTED HAITI U.S. Agency for International Development Wenser Estimé, Community Health Advisor Stephane Morriseau, M&E Advisor Christian Aid/POZ Samantha Brangeon, Program Officer, HIV/Accountable Governance Joseph Meance, POZ Field Coordinator for Nippes Jocelyn Numa, Program Officer Prospery Raymond, Country Manager Depopharm (St. Marc) Vernet St. Il, Sales representative FOSREF Cam-suze Berthomieux, Community Agent (Artibonite) Gladys Clonvil, Community Agent (Artibonite) Moise Fritz, Executive Director Dunbar Lesly, Director of Programs Patrick Schüt, Finance Director Le Galien Pharamcie (Miragoâne) Martin Fouquet, Jr., Owner Management Sciences for Health/LMS Project Dr. Antoine Ndiaye, Chief of Party Ministère de la Santé Publique et de la Population Dr. Rony D’Haiti, Assistant Director for Artibonite Dr. Petit-Frere Kesnel, MSPP Nippes Pharmacie du Peuple (St. Marc) Jacque Dorilas, Owner Justine Dorilas, Owner Piyay Kleren (Miragoâne) Maritte Sintil, Owner Population Services International Emmanuella Augustin, Communications Manager Léonie Desroses, Field Coordinator for Artibonite Anick Dupuy, Deputy Director Pierre Louis Yves J. Gerard, Head of Program Samuel E. Jean, Research, M&E Director Caliste Bird John, Finance Director Myriam Leandre Joseph, MCH Director Steve Laguerre, HIV and AIDS Director Carla López, Technical Advisor for Health Communications 28 USAID/HAITI: PROMARK MIDTERM EVALUATION Alison Malmqvist, Executive Director Myldrine Nelson, Field Coordinator for Nippes Stephanie Paul, Executive Assistant Matacha Riviere, Administration Unnamed Roadside Retailer (Miragoâne) Madam Tinel, Owner UNITED STATES The QED Group, LLC Taylor Napier-Runnels, Program Manager, GH Tech Project USAID/HAITI: PROMARK MIDTERM EVALUATION 29 APPENDIX C. PROMARK’S PERFORMANCE MONITORING PLAN HIV/AIDS Indicator Number HIV INDICATORS Disaggregated by Baseline Baseline data Year one targets (Apr. 09– Sept. 09) Year two targets (Oct. 09–Sept. 10) Year three targets (Oct. 10–Sept. 11) Year four targets (Oct. 11–Apr. 12) Total for 3 years (Apr. 09–Apr. 12) Number of individuals reached through community outreach that promotes HIV/AIDS prevention through behavior change beyond abstinence or being faithful Gender 2008 18,991 2000 138,000 144,500 72,250 356,750 Number of individuals trained to promote HIV/AIDS prevention through behavior change beyond abstinence or being faithful Gender 2008 80 30 705 538 0 1,273 Percentage of general population in social marketing target sites who correctly identifies ways to prevent HIV Age & gender Youth TRaC 2008 79% 81% 83% 85% 87% 87% Percentage of targeted high risk populations in social marketing target sites who correctly identifies ways to prevent HIV Age & gender Youth TRaC 2008 79% 81% 83% 85% 87% 87% 30 USAID/HAITI: PROMARK MIDTERM EVALUATION Percentage of youth (aged fifteen to 24 years) in social marketing target sites who correctly identifies ways to prevent HIV Age & gender TBD TBD TBD TBD TBD TBD Percentage of commercial sex workers (CSW) in social marketing target site who correctly identifies ways to prevent HIV Age & gender TBD TBD TBD TBD TBD TBD Percentage of men having sex with men (MSM) in social marketing target sites who correctly identifies ways to prevent HIV Age & gender TBD TBD TBD TBD TBD TBD Percentage of general population in social marketing target sites reporting the use of a condom the last time they had sex with a nonmarital, noncohabitating partner (Principal Indicator) Age & gender Youth TRaC 2008 69% 72.0% 74% 75.0% 76.5% 76.50% Number of male condoms sold None 2008 4,693,802 2,500,000 5,375,000 5,912,500 3,300,000 17,087,500 Number of female condoms sold None 2008 140,934 72,500 160,000 176,250 100,000 508,750 USAID/HAITI: PROMARK MIDTERM EVALUATION 31 FAMILY PLANNING Indicator Number FAMILY PLANNING INDICATORS Disaggregated by Baseline Baseline data Year one targets (Apr. 09– Sept. 09) Year two targets (Oct. 09–Sept. 10) Year three targets (Oct. 10–Sept. 11) Year four targets (Oct. 11–Apr. 12) Total for 3 years (Apr. 09–Apr. 12) Number of OC units sold None 2008 359,355 197,500 425,000 467,250 250,000 1,339,750 Number of injectable contraceptive units sold None 2008 266,910 150,000 333,750 378,750 215,000 1,077,500 Number of people receiving information about FP through IPC sex 2008 602 300 160,500 170,000 84,750 415,550 Number of women’s support groups put in place to help women use FP method correctly sex N/A N/A 0 56 56 0 112 32 USAID/HAITI: PROMARK MIDTERM EVALUATION 3.1.7. 2.F Couple year protection (CYP) provided through contraceptive sales None 2008 107,633 50,667 111,771 125,838 70,417 358,692 Percentage of women and men interviewed in social marketing target areas who have seen or heard a social marketing FP and RH message Sex TRaC PF 2007 32% 35% 38.00% 41% 44.00% 44% Number of women ages 15 to 49 interviewed in social marketing target areas who report usage of branded FP product one year after initiation None TBD TBD TBD TBD TBD TBD TBD Percentage of sexually active young women ages 15 to 24 interviewed in social marketing target areas who report consistent use of socially marketed FP products None TBD TBD TBD TBD TBD TBD TBD USAID/HAITI: PROMARK MIDTERM EVALUATION 33 CHILD SURVIVAL Indicator Number CHILD SURVIVAL INDICATORS Disaggregated by Baseline Baseline data Year one targets (Apr. 09– Sept. 09) Year two targets (Oct. 09–Sept. 10) Year three targets (Oct. 10–Sept. 11) Year four targets (Oct. 11–Apr. 12) Total for 3 years (Apr. 0–Apr. 12) Number of branded ORS product units sold None 2008 737,635 400,000 860,000 946,000 532,700 2,738,700 Number of branded clean water product units sold None 2008 92,774 50,000 107,500 118,250 66,500 342,250 Number of people receiving information about safe water through IPC Gender 2008 897 500 100,000 130,000 50,000 280,500 Number of women support groups put in place to help support safe water (the use of ORS and Dlo Lavi) to treat and prevent diarrhea None N/A N/A 0 56 56 0 112 34 USAID/HAITI: PROMARK MIDTERM EVALUATION Percentage of child caregivers interviewed in social marketing targets sites who correctly treated the last episodic case of childhood diarrhea with ORS None TRaC SRO 2007 31% 33% 35% 37% 39% 39% Percentage of new sales points for socially marketed oral rehydration salt products outside major urban centers None N/A N/A >50% >50% >50% >50% >50% Percentage of people interviewed in social marketing target areas who correctly identified at least two ways to treat contaminated water sex TBD TBD TBD TBD TBD TBD TBD Percentage of people interviewed in social marketing target areas who have heard a social marketing child health/CS message sex TRaC Water 2009 35% 38% 41% 44% 47% 47% USAID/HAITI: PROMARK MIDTERM EVALUATION 35 MARKETING Indicator Number CHILD SURVIVAL INDICATORS Disaggregated by Baseline Baseline data Year one targets (Apr. 09– Sept. 09) Year two targets (Oct. 09–Sept. 10) Year three targets (Oct. 10–Sept. 11) Year four targets (Oct. 11–Apr. 12) Total for 3 years (Apr. 09–Apr. 12) Number of new sales points delivering MS products Area N/A N/A 239 956 956 478 2,629 Percentage of sales points for socially marketed products with stock-outs for the reporting period Area N/A N/A <10% <10% <10% <10% <10% Number of sales points where the current price of the product is respected for the reported period Area N/A N/A >90% >90% >90% >90% >90% Number of special events conducted to promote MS product Area 2008 8 3 12 12 6 33 36 USAID/HAITI: PROMARK MIDTERM EVALUATION Percentage of sales outlets that are outside major urban areas Area 2008 40% 43% 46% 49% 52% 52% Percentage of brand specific marketing and promotion expenditures that is covered by product sales revenue (HIV, FP, water products) None N/A N/A >50% >50% >50% >50% >50% USAID/HAITI: PROMARK MIDTERM EVALUATION 37 APPENDIX D. REFERENCES Population Services International, Haiti. Annual Report (June 2009–September 2010). Population Services International, Haiti. Audit trimestriel des points de vente de PSI/Haïti: Round1 2009. Population Services International, Haiti. Étude TraC PF jeunes filles 15–24 ans: Rapport de supervision de terrain. 2011. Population Services International, Haiti. Évaluation du réseau de distribution de PSI/Haïti après le passage du seisme du 12 janvier 2010. 2010. Population Services International, Haiti. PSI/Haiti Special Report on Management of Sales. 2010. Population Services International, Haiti. USAID-PEPFAR Annual Report,Reporting Indicators. 2009. (Narrative) Population Services International, Haiti. Haiti ARP. Population Services International, Haiti. E-BULLETIN PSI-HAITI. August 2010. Population Services International, Haiti. E-BULLETIN PSI-HAITI. March. Population Services International, Haiti. PSI Haiti Cholera Response. 2010. Population Services International, Haiti. PSI-HAITI Distribution Plan. 2009. Population Services International, Haiti. Quarterly Report (April–June 2010). 2010. Population Services International, Haiti. Quarterly Report (April–September 2009). 2010. Population Services International, Haiti. Quarterly Report (January–March 2010). 2010. Population Services International, Haiti. Quarterly Report (October–December 2009). 2009. Population Services International, Haiti. Recensement général des Points de Vente des produits de PSI/Haïti. 2009. Population Services International, Research Division. Monitoring and Evaluation Plan (PROMARK). 2010. For more information, please visit: http://resources.ghtechproject.net Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com