USAID/HAITI: SANTE POUR LE DEVELOPPEMENT ET LA STABILITE D’HAÏTI (SDSH) PROJECT EVALUATION NOVEMBER 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Gerard Bowers, Rachelle Cassagnol, and Williams Ojo through the Global Health Technical Assistance Project. USAID/HAITI: SANTE POUR LE DEVELOPPEMENT ET LA STABILITE D’HAÏTI (SDSH) PROJECT EVALUATION DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 11-01-584) is available in printed or online versions. Online documents can be located in the GH Tech website library at http://resources.ghtechproject.net. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. USAID/HAITI: SDSH PROJECT EVALUATION i CONTENTS ACRONYM LIST ............................................................................................................. III EXECUTIVE SUMMARY ............................................................................................... VII The Project ............................................................................................................................................ vii Evaluation Purpose and Methodology ............................................................................................. vii Key Findings and Recommendations ................................................................................................ vii Project Monitoring .............................................................................................................................. viii Project Implementation........................................................................................................................ ix Capacity Building .................................................................................................................................... x Lessons Learned .................................................................................................................................... xi Next Steps .............................................................................................................................................. xi I. INTRODUCTION ..................................................................................................... 1 Summary Project Description ............................................................................................................. 1 Purpose of the Evaluation ..................................................................................................................... 1 Methodology ............................................................................................................................................ 2 2. FINDINGS AND RECOMMENDATIONS ............................................................. 3 Project Monitoring ................................................................................................................................. 3 Project Implementation......................................................................................................................... 7 Capacity Building ................................................................................................................................. 27 Lessons Learned .................................................................................................................................. 33 Needs Assessment .............................................................................................................................. 35 ANNEXES ANNEX A: SCOPE OF WORK ..................................................................................... 39 ANNEX B: EVALUATION DESIGN AND METHODOLOGY .................................. 47 ANNEX C: LIST OF PERSONS INTERVIEWED ........................................................ 55 ANNEX D: LIST OF DOCUMENTS REVIEWED ....................................................... 59 ii USAID/HAITI: SDSH PROJECT EVALUATION FIGURES Figure 1. Map of Haiti Showing Spatial Distribution of SDSH-Supported Sites .................................................................................................. VIII Figure 2. Ideal National HMIS Integrated Data Flow .................................................... 5 TABLES Table 1. Proposed Schedule ........................................................................................... 43 Table 2. SDSH Evaluation Design and Methodology ................................................... 50 USAID/HAITI: SDSH PROJECT EVALUATION iii ACRONYM LIST AEADMA Association d’Entr’Aide des Dame-Mariens AIDS Acquired immunodeficiency syndrome BCC Behavior change communication CA Cooperating agency CBO Community-based organization CDC Centers for Disease Control and Prevention CDS Centres pour le Développement et la Santé CHW Community health worker CIDA Canadian International Development Agency CM Community mobilization CONASIS Comité National du Système d'Information Sanitaire (National Committee for the Health Information System) COP Chief of party COR Contracting Officer Representative COTR Contracting Officer Technical Representative DQA Data quality assessment DRI Direct Relief International DSF Direction Santé de la Famille (Division of Family Health, MOH) FA Financial advisor FBO Faith-based organization FONDEP Fondation pour le Développement de Petit Trou de Nippes FOSREF Fondation pour la Santé Reproductive et l’Education Familiale FP Family planning GH Tech Global Health Technical Assistance Project GHI Global Health Initiative GOH Government of Haiti GUC Grants-Under-Contract HCF Health care financing HIV Human immunodeficiency virus HMIS Health Management Information System HR Human resources HS 2004 Haiti Health Systems 2004 HS 2007 Haiti Health Systems 2007 iv USAID/HAITI: SDSH PROJECT EVALUATION HSS Health systems strengthening ICC International Child Care JHU-CCP Johns Hopkins University’s Center for Communications Programs JHPIEGO Johns Hopkins Program for International Education in Gynecology and Obstetrics LAPM Long-acting and permanent methods (of family planning) MCH Maternal and child health M&E Monitoring and evaluation MOH Ministry of Health MOU Memorandum of understanding MSH Management Sciences for Heath MSPP Ministère de la Santé Publique et de la Population (Ministry of Public Health and Population) MSSF Monthly Statistics Summary Form MYAP Multi-Year Assistance Program NGO Non-governmental organization PADESS Projet d’Appui au Développement du Système de Santé PBF Performance-based financing PDI Plan Départemental Intégré (Integrated Departmental Plan) PEPFAR President’s Emergency Plan for AIDS Relief PLWHA Persons living with HIV and AIDS PMP Performance monitoring plan PMTCT Prevention of mother-to-child transmission (of HIV/AIDS) POI Plan Organisationnel Intégré (Integrated Operational Plan) PPP Public-private partnership PPS Point de prestation de services (service delivery point) PWW Pure Water for the World RH Reproductive health RNDI Réseau national de distribution des intrants (national health sector supplies distribution) SADA Service and Development Agency SDMA Service Delivery and Management Assessment SDSH Santé pour le Développement et la Stabilité d’Haïti SOG Soins obstétricaux gratuits (free obstetrical care) SOW Scope of work TA Technical advisor TB Tuberculosis USAID/HAITI: SDSH PROJECT EVALUATION v TBA Traditional birth attendant UCS Unité Communale de Santé (Communal Health Unit) UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund UPE Unité de Planification et d'Evaluation (Planning and Evaluation Unit, MOH) USAID U.S. Agency for International Development WHO World Health Organization ZC Zone ciblée (targeted area) vi USAID/HAITI: SDSH PROJECT EVALUATION USAID/HAITI: SDSH PROJECT EVALUATION vii EXECUTIVE SUMMARY THE PROJECT The Santé pour le Développement et la Stabilité d’Haïti (SDSH) Project is the U.S. Agency for International Development/Haiti (USAID/Haiti) flagship health sector activity. Awarded to Management Sciences for Health (MSH) in August 2007 for an initial period of three years (2007–2010) at a cost of $42.5 million, the contract was amended four times between 2007 and 2009 to increase its cost to $81.4 million, and to extend the activity to September 30, 2012. The SDSH Project builds on two predecessor projects managed by MSH—Haiti Health Systems 2004 (HS 2004) and Health Systems 2007 (HS 2007)—and continues the earlier activities’ support for the health delivery programs of (currently) 28 non-governmental organizations (NGOs). The most important new component of the project is the inclusion of a focus on 31 target zones or zones ciblées (ZC)—areas identified by the Ministry of Health (MOH) as seriously underserved by MOH resources. Health services available through those networks are expected to reach approximately 50% of the population by the end of the project in 2012. MSH manages the SDSH project with two subcontractors: The Johns Hopkins Program for International Education in Gynecology and Obstetrics (JHPIEGO) and the Johns Hopkins University’s Center for Communications Programs (JHU-CCP). EVALUATION PURPOSE AND METHODOLOGY In October/November 2011, USAID/Haiti sponsored an evaluation conducted by a consultant team recruited by the Global Health Technical Assistance Project (GH Tech). The key purposes of the evaluation were to (1) assess the performance of the SDSH project and, drawing on the findings of that assessment, (2) propose, for USAID’s consideration, recommendations regarding the shape and objectives of a follow-on project. The project evaluation was carried out in four departments (West, Artibonite, Central Plateau, and Nippes), where the team observed service delivery programs conducted by seven NGOs and two MOH ZCs. The team analyzed quantitative and qualitative data collected at NGO and MOH service sites; interviewed SDSH, NGO, MOH, and other donor personnel in Port-au-Prince and at the four MOH health departments (departments sanitaires); and visited one grantee undertaking work further to the project’s Grants-Under-Contract (GUC) initiative. (See Figure 1 for a map of Haiti with information on SDSH-supported sites.) KEY FINDINGS AND RECOMMENDATIONS The evaluation team collected, analyzed, and arranged its findings in general conformance to categories suggested by USAID/Haiti, namely project monitoring, project implementation, and capacity building. viii USAID/HAITI: SDSH PROJECT EVALUATION Figure 1. Map of Haiti Showing Spatial Distribution of SDSH-supported Sites Note: SDSH currently supports service delivery activities in 78 health facilities (NGO_PPS = service delivery points, operated by 28 NGOs; ZCs_PPS = 79 MSPP public sector sites in 31 target zones) PROJECT MONITORING Key Findings: Although the evaluation team did not undertake an assessment of the validity of data reported in the project’s performance monitoring plan (PMP), an informal review of those data indicate that the project is meeting or surpassing most of its performance objectives. The SDSH project has developed and utilized a robust performance monitoring system that effectively tracks the performance of the project’s participating NGOs and MOH ZCs. That monitoring system generally exists in parallel, however, to other MOH (and President’s Emergency Plan for AIDS Relief [PEPFAR]) data collection systems in use at the MOH department level. No meaningful efforts are currently under way to harmonize the collection, distribution, or use of those separate data flows. The disconnect between these systems is at least partly a consequence of (1) MOH delays in extending its Health Management Information System (HMIS)—itself developed with assistance from SDSH and the two predecessor projects—down to the MOH department level; (2) the absence of a weak data culture at the department level, where little priority is given to data-related training, coordination among persons responsible for data collection and reporting, or ensuring the validity of collected data; and (3) USAID/HAITI: SDSH PROJECT EVALUATION ix inadequate efforts by SDSH (technical and financial) advisors at the department level to promote better integration of the SDSH data system into the other systems. Key Recommendations:  The SDSH project should seek to align its data collection and reporting system with other systems in use at the MOH departmental level by promoting closer interaction between the project’s on-site SDSH advisors (i.e., the technical and financial advisors) and MOH department counterparts.  SDSH should encourage the MOH to provide additional training in data management and performance-based financing (PBF) procedures to MOH department-level staff. PROJECT IMPLEMENTATION Key Findings:  SDSH got off to a slow start because of USAID-mandated changes in the contract’s scope of work and was beset with delays and disruptions throughout its four-year implementation period (e.g., temporary deactivation of subcontractors; high turnover of key staff, including three chiefs-of-party; the hurricanes of 2008 and subsequent political instability; the January 2010 earthquake and subsequent cholera epidemic). The project nonetheless remained generally on track and continued to achieve most of its performance targets. The project’s inability to retain key personnel, however, prompts some concern over its ability to maintain a stable project management and oversight capability.  SDSH has developed several management and oversight tools that it applies with consistent vigor. These include a PMP and the creation of specific performance targets for each participating NGO and ZC; close monitoring of partners’ performance by SDSH department-level personnel (technical and financial advisors) and SDSH headquarters personnel; annual reviews (service delivery and management assessments, or SDMA) of project partners; and annual meetings of the partners, to name a few. Although several NGO managers noted their preference for less “rigidity” in the manner in which SDSH assigns and enforces project targets, the rigor with which SDSH applies its tools and enforces its targets contributes to the project’s effectiveness.  SDSH has successfully expanded the scope of the predecessor projects (HS 2004 and HS 2007) by incorporating the MOH ZCs into its health service delivery program.  The project’s success in expanding service delivery at the MOH department level is not matched by its more lackluster performance in improving the executive capacity of the MOH at the central level, where SDSH efforts to help develop a HMIS and financial management tools have yet to produce durable results.  The project’s success in expanding service delivery is due in good measure to its adoption and application of a sound service delivery model with several interlocking elements (community health workers (CHWs), trained technical and support personnel at service delivery points and at the MOH departments, engaged SDSH advisors at the department offices, attentive SDSH headquarters staff, and successful application of the PBF approach to project contracting). The model, however, is not currently sustainable from either a financial or a management perspective, as participating NGOs and the MOH are financially dependent on project resources and the MOH lacks any meaningful capacity to implement or monitor a PBF system.  Long-term sustainability of the service delivery program will require (1) development of program management skills at MOH central and departmental levels and (2) continued donor and MOH provision of the financial resources needed to implement the service delivery program. x USAID/HAITI: SDSH PROJECT EVALUATION  The project succeeded in responding—to the extent it could within the constraints of its contract— to the cholera epidemic.  Some elements of the project, although generally successful and useful in their own right, are not contributing substantively to the attainment of the project’s primary objective: the expansion of integrated health services to underserved populations.  USAID/Haiti has not adequately explored closer linkages between the SDSH project and the Mission’s PL-480 Title II program. Key Recommendations:  USAID should consider encouraging bidders for the follow-on project to propose a Haitian national for the chief-of-party position in order to (1) better reflect Haitian ownership of the activity. and (2) improve the stability of a key position that has been subject to high turnover over the past four years.  The follow-on project should include provision for a comprehensive effort to develop and install professional-level capacity at both the central and departmental levels of the MOH to effectively install capacity to design, execute, monitor, and support (though not necessarily fund) performance￾based contracts.  The design of the follow-on project should call for a lean activity (in terms of its various components) focused on the extension of basic health services, plus the capacity-building efforts required to support that expansion. Extraneous elements that do not directly support that objective should not be included in the project.  USAID should expand and formalize NGO-cooperating agency (CA) arrangements by encouraging its food assistance partners in Haiti to reach out to NGOs that participate in the SDSH project, and to include their plans for collaboration in the CAs’ multi-year assistance plans. CAPACITY BUILDING Key Findings:  Although capacity building of local NGOs was not a central aspect of SDSH, the project helped strengthen the capacity of local organizations, mainly through mentoring and technical assistance for service delivery. SDSH is overly involved, however, in the routine oversight of NGO service delivery points, a practice that weakens the leadership of NGOs and hinders their path toward graduation from donor reliance.  SDSH participation on various MOH technical committees contributed to the development of norms, standards, and procedures for selected service delivery activities. However, the project’s participation did not result in significant transfer of knowledge to reinforce the capacity of MOH partners at the central level.  MOH departmental directorates have only a limited role in the management of performance-based contracts signed between NGOs and SDSH in the department.  SDSH training and support for several thousand CHWs and traditional birth attendants(TBAs), plus the project’s role in establishing mothers’ clubs, are helping communities improve access to basic health care services. However, the project has not realized its goal of establishing local health task forces at the community level. USAID/HAITI: SDSH PROJECT EVALUATION xi Key Recommendations:  SDSH should utilize its annual Service Delivery and Management Assessment exercise to analyze the level of organizational capacity of each participating NGO.  The follow-on project should include provision for (1) continuation of on-site technical and financial advisors at MOH departmental levels, and MOH central-level access to advisors who would assist in the development of improved MOH executive capacity; (2) a focus on the development of local health task forces; and (3) efforts to increase access by youth to health services.  The follow-on project should help selected local NGOs develop innovative ways to increase funding, including the exploration of prospects for community health insurance to increase users’ financial access to health services. LESSONS LEARNED Two general lessons emerge from the SDSH project evaluation: 1. The design and implementation of health assistance projects in Haiti needs to take place over a long planning horizon—beginning several decades ago, and continuing at least 10 years into the future. 2. The SDSH project has succeeded in its primary goal to expand the availability of essential health services, but it has not played a strategic role in helping to ensure that such services are and remain financially and managerially sustainable. A more strategic orientation would position the project to begin the transfer of responsibility for project management and oversight from the contractor to the Government of Haiti (GOH)/MOH. Other, more specific lessons include the following:  A project focused almost exclusively on the achievement of performance targets runs the risk of giving inadequate attention to measures that track quality of care, ethical issues in the delivery of health services, and progress toward the organizational strengthening of partners.  Similarly, a project heavily invested in its own performance monitoring system can inadvertently contribute to the emergence of separate, parallel reporting systems that compete for the time and attention of scarce data management personnel.  Haiti lacks a functioning patient referral system, and there are no patient tracking systems in place that can be used to follow up on patients who are nominally referred to other health facilities.  No long-term plan is in place to effectively respond to the lack of long-acting and permanent methods (LAPM) of family planning in rural Haiti. NEXT STEPS Short-term adjustments that SDSH can adopt in the remaining 10 months of the project include the following steps:  Undertake an assessment, as part of the project’s SDMA exercise, of (1) NGO and ZC capacity to implement a more methodical response to the cholera epidemic; and (2) track the progress of participating NGOs toward a higher level of institutional readiness for “graduation”  Finalize the MOH HMIS  Begin the process of aligning data reporting systems at the MOH department level  Ensure that SDSH documents best practices applied during the project and reports these practices to USAID/Haiti xii USAID/HAITI: SDSH PROJECT EVALUATION  Consult with prominent NGOs to develop human resource (HR) policies designed to minimize staff resentments and turnover  Re-examine the use of SDSH patient registers at health facilities For the longer term, USAID should consider incorporating the following factors into the design of the follow-on to the SDSH project:  Retention of key components of the current project (the service delivery model; performance-based contracting; NGO partnerships; and so forth)  Elimination of extraneous activities (the Strategic Partners and Grants-Under-Contract initiatives; marginally productive efforts at the MOH central level)  Transition of project management responsibilities from the contractor to the MOH, with an end-of￾project goal of at least two departments capable of negotiating and managing performance-based contracts with NGOs USAID/HAITI: SDSH PROJECT EVALUATION 1 I. INTRODUCTION SUMMARY PROJECT DESCRIPTION The Santé pour le Développement et la Stabilité d’Haïti (SDSH) Project is the U.S. Agency for International Development/Haiti (USAID/Haiti) flagship health sector activity. Awarded to Management Sciences for Health (MSH) in August 2007 for an initial period of three years (2007–2010) at a cost of $42.5 million, the contract was amended in 2009 to extend the project to September 30, 2012, and to increase project funding to $81.4 million. The SDSH Project builds on two predecessor projects managed by MSH: Haiti Health Systems 2004 (HS 2004) and Health Systems 2007 (HS 2007). All three projects were designed to increase access to and use of a package of integrated basic health services that covers maternal and child health (MCH), nutrition, family planning (FP), HIV and AIDS, and tuberculosis (TB). All three projects stressed close cooperation and collaboration with Haiti’s Ministry of Health (MOH), the Ministère de la Santé Publique et de la Population (Ministry of Public Health and Population, or MSPP). With some minor changes, all three projects supported essentially the same group of (currently) 28 non-governmental organizations (NGOs)—two were dropped for various reasons in 2008 and 2009; one joined in 2006; and the most recent addition joined in 2010. Finally, the SDSH Project continues the HS 2007 Project’s reliance on performance-based financing (PBF) as a basis for MSH’s relationships with participating NGOs—and more recently as an assistance mechanism to support MOH target zones, or zones ciblées (ZC). This is not to say, however, that the SDSH project does not include changes, some of which are significant. The most prominent changes include (1) a heightened focus on measures to increase community involvement in project implementation; (2) the inclusion, at the MOH’s request, of the ZCs in the project; (3) a shift in the focus of capacity-building efforts from the MOH central level to the departmental level; and (4) a more robust application of the PBF model as a basis of SDSH relationships with NGOs and the MOH. Notwithstanding those changes, the SDSH project serves as a relatively seamless transition from the predecessor activities to the current one. The project is currently supporting service delivery activities in 81 health facilities (points de prestation de services, or PPS), operated by the 28 participating NGOs, and through 79 MSPP public sector sites in 31 ZCs. The project’s primary beneficiaries are (1) children and youth under 25 years of age; (2) women; and (3) groups with special concerns, including persons living with HIV and AIDS (PLWHA) and TB. SDSH currently covers 42% of the Haitian population, or 4,254,166 people. Other new components of the project include a Grants-Under-Contract (GUC) program that was slow getting under way for a variety of reasons (discussed in detail below), and a Strategic Partnership initiative designed to leverage health-related contributions from other partners, especially in the private sector. MSH manages the SDSH project with two subcontractors: The Johns Hopkins Program for International Education in Gynecology and Obstetrics (JHPIEGO) and the Johns Hopkins University’s Center for Communications Programs (JHU-CCP). PURPOSE OF THE EVALUATION In several important respects, USAID/Haiti views the SDSH project as an antecedent to a more direct agency relationship with selected, institutionally capable NGOs, and most especially with the MOH. The Mission’s interest is driven in part by the Global Health Initiative (GHI) and USAID Forward, both of which call for increased reliance on and ownership by host-country partners. Looking to the future, and 2 USAID/HAITI: SDSH PROJECT EVALUATION in the wake of a successful democratic transition in Haiti, USAID/Haiti is embarking on a broad-based effort to help ensure the success and durability of Haitian government institutions and is planning to develop the needed assistance modalities to do so. The two-fold objective of this evaluation, then, is to provide USAID/Haiti with (1) an assessment of the project’s performance in laying down a firm groundwork for the next phase of USAID assistance in the Haitian health sector (i.e., the project’s relative success in preparing NGOs and the MOH to assume a more mature and direct relationship with USAID under the strategic direction of the Government of Haiti (GOH)); and in a more prosaic sense, (2) an assessment of the project’s performance in meeting its contractual requirements. METHODOLOGY The project team, consisting of two expatriate and one Haitian consultant and three Haitian data collectors, familiarized themselves with the project by reviewing project reports prepared by SDSH, interviewing SDSH staff, and travelling extensively in Haiti to visit NGO and GOH offices and service delivery sites, where team members interviewed NGO and government program managers and service providers and reviewed project data. The team also attempted to gather qualitative data via discussions with patients, health care providers, and community health workers. (Annex B presents the team’s evaluation design and methodology.) Further, to supplement guidance provided to the evaluation team by USAID/Haiti program managers, the evaluation team attempted to focus its data-gathering and reporting efforts on information that is not readily available to the Mission in SDSH project documents. USAID/HAITI: SDSH PROJECT EVALUATION 3 II. FINDINGS AND RECOMMENDATIONS PROJECT MONITORING The evaluation attempted to assess the extent to which the SDSH project achieved the expected results specified in the project’s performance monitoring plan (PMP), and the extent to which the SDSH project sought to ensure the validity and reliability of performance indicators via the application of the project’s PBF model. In addition, the evaluation team was asked to determine whether the health ministry’s Health Management Information System (HMIS) was being used to monitor SDSH project performance. This aspect of the evaluation considered the status of the HMIS as a mechanism that all levels of the MOH could use to generate, compile, analyze, communicate, and use health data in decision-making and policy planning. Finding 1 Despite several disruptions faced by the project, SDSH affirms and project data indicate that the project met or surpassed virtually all of its targets. A notable exception was the TB targets, a shortfall that was due at least in part to an abrupt withdrawal of Global Fund support. The project’s confidence in making this claim rests on its faith in a rigorously implemented performance monitoring system, which does, in the evaluation team’s judgment, produce generally accurate and reliable information. Discussion Prior to 2008, SDSH used five indicators to determine NGO performance in meeting its contractual targets. The NGOs, however, knew the indicators that SDSH was using to track its performance could theoretically focus on attaining “incentive” targets. In 2008, SDSH established 18 performance indicators; since that time, it has randomly selected one incentive indicator each year for each NGO. This indicator is not known by the NGO, driving it to achieve all of their performance targets in order to ensure successful achievement of the unknown incentive target. SDSH, meanwhile, reinforces its partners’ motivation by carefully scrutinizing each partner’s monthly performance report. If SDSH reviewers discover any slippage in partners’ performance against the indicators, project personnel quickly contact the partner (by email, letter, phone call, etc.) to discuss the issue, and examine the steps needed to correct the problem. The project’s on-site technical and financial advisors also follow up with the partner to ensure that appropriate corrective measures have been taken. As designed and implemented by SDSH, this performance-monitoring system produces timely, high￾quality data that accurately track partners’ progress—or lack thereof. To be sure, the system has a small number of vulnerabilities. These include (1) health care provider “data entry fatigue,” a possible result of providers’ obligation to painstakingly enter data for each patient into cumbersome SDSH registers; (2) errors associated with the transcription of data from one register to another; (3) the lack of any regular assessment of the quality of data submitted by the PPSs; and (4) software artifacts in the Microsoft Excel data sheets used by the project. Taken together, however, any errors that these factors might introduce into the overall performance monitoring system would be relatively minor. SDSH project staff explained the project’s shortfall in meeting its TB performance targets by noting that when they had elaborated their TB objectives and activities in 2007; moreover, the MSPP National Tuberculosis Program had substantial support from the Global Fund for TB interventions in Haiti. They were not overly concerned that only 4% of the SDSH budget was planned for TB activities because many of the needed inputs (e.g., drugs, trainings, equipment) were provided via the National 4 USAID/HAITI: SDSH PROJECT EVALUATION Tuberculosis Program. Unfortunately, Global Fund support for these activities ended abruptly in 2010 and to date neither SDSH nor other parties have been able to fill the funding gap. SDSH advises that it will continue to advocate through ongoing technical discussions with MSPP counterparts for greater collaboration and interaction between the MSPP HIV and TB programs. Finding 2 Discussions with NGO and MOH PPS managers indicate that they have been motivated by the project to develop innovative ways to meet their contractual targets, and extend their reach to target populations (e.g., use of community “rally posts,” fielding mobile teams of physicians and nurses to remote areas, coordinating health information messages with traditional healers, etc.). However, none of the 18 indicators used to monitor NGO performance provide any gauge of the NGOs’ progress toward improved organizational/institutional growth or their readiness to graduate from the SDSH project. Finding 3 The SDSH project has made little progress in strengthening overall data management procedures within the MOH. Only in March 2011 did MOH set up a technical committee on HMIS to review and harmonize the Monthly Statistics Summary Form (MSSFs) produced by the PPS). The HMIS is not yet functioning ministry-wide, and SDSH has not attempted any meaningful integration of the project’s data management system into the broader MOH data management system. Discussion At the central level of the MOH the Unité de Planification et d'Evaluation (Planning and Evaluation Unit, or UPE) is responsible for compiling health system data. UPE collects data from MOH, NGO, and private PPS through MOH reporting channels according to the sequence (PPS  Commune (where a Communal Health Unit is functioning)  Department  Central Level. Although data are reported “upward” to the UPE, there is little feedback downward from the UPE to the PPS, a shortcoming that the UPE attributes to a lack of staff and resources. Factors that might alter this process and facilitate departmental use of data for management purposes remain unlikely in the absence of a national strategic information plan or policy or a single HMIS that could serve all elements of the MOH. Lacking such a unified data management system, the MOH departments continue to rely on parallel and uncoordinated data collection and reporting systems, including the upward-oriented MOH system discussed above, the SDSH project reporting system, and (in selected departments) the President’s Emergency Plan for AIDS Relief (PEPFAR) reporting system. There does not appear to be any intention on the part of the MOH to consolidate or harmonize these disparate systems into one system that could provide management￾relevant information to all users. Recommendation Since the MOH has a standardized data flow system in place from PPS via the commune to MOH departments, and from there to the UPE, SDSH should align its project data reporting system and data flow procedures with that existing system. Such integration would help strengthen the overall MOH health data reporting system, as shown in Figure 2. USAID/HAITI: SDSH PROJECT EVALUATION 5 Figure 2. Ideal National HMIS Integrated Data Flow HMIS Data Flow and Reporting MOH/MSPP(SDSH embedded) Department(SDSH TA embedded) Health Facility (Zone Cibles & NGO PPS) Unite Communal de Santé / NGO office Routine data collection of Harmonized MSSF Coordinate PPS data, Supervision and Feedback Data Collation, Supervision, Feedback and DQA Data Consolidation, Planning, DQA, Feedback ,Donor Reports, Recommendation Prior to the end of the project, SDSH staff should convene a forum of department- and commune-level data producers and users to explore next steps in harmonizing the various data systems in use at those levels. Finding 4 There is a strong culture of data use at the MOH central level, demonstrated by UPE’s interaction with other agencies of government, and that unit’s publication of quarterly and annual reports for GOH and public use (although data are frequently incomplete). That culture is much weaker at the MOH department level, where the team found little interest in data use for strategic planning, for aggregation of data into one health information system, or for staff interaction on data use. At the MOH department level, the SDSH technical advisor is often the only person involved in the collection, monitoring, and reporting of SDSH data. The advisor attempts little mentoring and/or inclusion of MOH staff in those efforts, further contributing to the perseverance of separate, nonintegrated data systems at that level. Discussion Most of the MOH and NGO PPSs visited by the evaluation team had designated data management personnel on their staff. All were able to produce a data management manual distributed by SDSH, including an indicators definition guide for reference. Moreover, most of these personnel had good knowledge of the indicators and knew how to transcribe them from client service registers into project reports. Most indicated that they generally met their monthly submission deadlines for SDSH reports, but were usually late in submitting their MOH monthly reports. Once those reports reached the MOH department offices, SDSH data—processed most frequently by the SDSH technical advisor (TA)—were reviewed and dispatched quickly to SDSH headquarters. MOH data, however, were processed in a more desultory manner and without critical review by data managers or more senior department staff. Some MOH department offices have a departmental 6 USAID/HAITI: SDSH PROJECT EVALUATION epidemiologist, statistician, and surveillance officer on staff, but such personnel do not generally coordinate their work with each other or with the SDSH technical advisor. Few of the MOH personnel responsible for monitoring and evaluation (M&E) tasks received training in that function, nor have they attended regularly scheduled data review meetings. Among the more significant consequences of the MOH departments’ lack of attentiveness to the management utility of data is the non-use of those data to (1) monitor drug and commodity supplies to help prevent stock-outs, and (2) track patients who are referred from a PPS to another health facility. Recommendation SDSH technical and financial advisors should try to more fully engage MOH staff in the processing and oversight of SDSH project data to build the capacity of MOH personnel in data management and to promote closer alignment of SDSH and MOH data systems. Recommendation The MOH should, with assistance from the follow-on project, explore ways to more effectively utilize MOH data to monitor medical supplies at PPSs and track referral patients. Finding 5 Service providers’ entry of data into the SDSH client data registers is excessively time-consuming, and could limit health care workers’ time and ability to interact with their patients. Moreover, each register is compiled on an annual basis, requiring service delivery personnel to manually transcribe all of the data from prior-year registers into new annual registers. Thus, by the fourth year of the project, service data for a single patient may have been transcribed four times into as many registers. Recommendation Managers of the follow-on project should explore the feasibility of more user-friendly, less time￾consuming technologies for data entry and retrieval, including the use of electronic tablets if the price for such tools declines in the near future. Finding 6 The MOH departments’ capacity to manage their ZC programs is not as well developed as the capacity of the NGOs, since to a significant extent the MOH/ZC programs are still managed directly by SDSH. Discussion The successful application of a PBF system requires leadership, capacity to plan and optimize resource allocation, and a functioning health care delivery system. Those management skills, tools, and infrastructure are not yet adequately developed at the MOH department level. Until the MOH departments develop greater managerial strength, it might be appropriate to measure performance in the ZCs against a less demanding standard than might be applied to NGO programs. Recommendation As noted in this report, the follow-on project should provide for long-term capacity-building at the MOH department level in the management and implementation of a PBF contracting system. USAID/HAITI: SDSH PROJECT EVALUATION 7 PROJECT IMPLEMENTATION The scope of work (SOW) that USAID/Haiti prepared for this evaluation called for, among other things, an examination of factors that positively or negatively affected SDSH’s ability to deliver the services required to achieve expected results. USAID encouraged the evaluation team to look beyond specific topics noted in the SOW, but also asked the team to assess the role of the project’s specific characteristics and their role in supporting or impeding the achievement of project goals. These factors included project staffing and structure; project management; USAID management; and the partnership model employed by the project, with a special focus on the project’s application of the PBF model to relations with its project partners. Finding 7 The project did not have a smooth start and experienced a number of disruptions during implementation. These disruptions included the following:  Delayed project launch: In March 2007, USAID decided to eliminate several activities that had been called for in the request for proposal and that had been included in MSH’s successful proposal. Steps to reconfigure the project’s content and decrease its funding delayed execution of a USAID-MSH contract for approximately six months, to August 2007.  The 2008 hurricanes: The devastating hurricane season of 2008 pummeled Haiti with four huge storms in the space of 30 days, killing hundreds and perhaps thousands of people. The SDSH project, like most of the USAID projects then under way, worked with disaster relief personnel to address population needs after the storms, as directed by USAID. While not in any way comparable to the event’s tragedy for the people of Haiti, the storms and the donor response also affected the capacity of the SDSH project to continue effective project implementation. Specifically, the influx of donor agencies and NGOs working on hurricane relief produced a bidding contest for Haitian staff, and several SDSH personnel migrated to the better-paying jobs in those organizations.  Political instability: The post-hurricane civil unrest and food riots complicated the project’s ability to maintain service delivery in high-strife areas of the country. (For instance, project personnel reportedly wore bulletproof vests as they carried out project monitoring.) Instability also impeded MSH’s ability to recruit a replacement for their original chief of party, who had moved back to the United States upon receiving a promotion within MSH.  COP turnover: The project has had three chiefs of party in four years and, at the time this report was being written, was searching for a fourth. Those changes inevitably contribute to disruptions in project orientation, internal and external communications, and “drift” in project implementation.  January 2010 earthquake: The January 2010 earthquake was the most profoundly significant event in Haiti’s recent history. Again, as in the instance of the hurricanes of 2008, SDSH personnel were enlisted into USAID’s broad-based response to the crisis. And again—but on a much larger scale— international NGOs and donor organizations were able to recruit away SDSH staff to an extent that is still affecting the project’s personnel structure. Both SDSH personnel and key staff members of the project’s NGO partner organizations and central and department MOH personnel were also drawn away by the relief organizations. Moreover, the higher salary scales of these organizations continue to complicate recruitment of replacement staff at all levels of the SDSH project. The “cluster groups” hosted by various United Nations agencies reportedly serve as job fairs for Haitians seeking to increase their income.  The cholera epidemic: The cholera epidemic that emerged after the earthquake poses a serious long-term threat to public health in Haiti. The SDSH project, although lacking contractual authority to address cholera, continues to support the international effort to control the disease via the project’s mandate to work on the prevention and management of diarrheal disease. 8 USAID/HAITI: SDSH PROJECT EVALUATION  Contraceptive stock-outs: Parliament’s unexpected rejection on a GOH decision to procure contraceptive commodities led to major stock-outs of FP supplies at service delivery points, and setbacks in the project’s ability to meet its and reproductive health (RH) goals.  Temporary loss of subcontractor participation in the project: USAID’s decision in 2009 to extend the project through August 2012 (later revised to September 30, 2012) called for the termination of MSH’s subcontracts with JHPIEGO, JHU-CCP, and the NGO Fondation pour la Santé Reproductive et l’Education Familiale (FOSREF). The agency’s decision was apparently driven in part by funding constraints, but also by an effort to bring the project into closer compliance with the GHI, as the project’s reliance on several U.S.-based partners seemed to fly in the face of GHI’s call for greater reliance on host-country institutions. MSH argued for a continuation of its subcontracts with JHU￾CCP and JHPIEGO, but the USAID-MSH negotiation process took 10 months to complete, during which time the professional staff of the two organizations were effectively immobilized. The project’s communications director was not able to wait for resolution of the issue and moved on to another job, and her position in the SDSH project remains vacant. The departure of FOSREF from the project (as a subcontractor) negatively impacted the ability of the project to successfully implement efforts to expand and intensify the involvement of youth. If that subcontractual role had been continued, FOSREF would have provided technical assistance and guidance to the other 27 NGOs to strengthen and/or install youth-focused activities at those other NGOs (it did so with seven of the NGOs before the subcontract was terminated). Despite that loss, the evaluation team observed that most of the NGOs’ field programs include youth-focused elements, even if these are not as extensive as they might have been if the FOSREF subcontract had been continued.  Change of government: A new GOH administration was installed on October 17. Although the incoming administration will in all likelihood be supportive of the project, the change will result in a large-scale turnover of management, professional, and technical personal at all levels (central and departmental) of the government. At the very least, it is likely that many of the project’s counterparts, and many of the staff who have been trained in project tools (the HMIS, financial accounting programs, logistic systems, etc.) will move on—requiring extensive efforts to train their replacements. A way to summarize these distractions would be to recall the two assumptions included in the original project design, which noted that achievement of project goals would be contingent on (1) governmental stability and (2) an absence of major natural disasters. The ability of the project to continue project implementation in keeping with its design, and to achieve most of its planned goals despite these and other setbacks, reflects positively on the commitment and resilience of the project and its staff. Project Staffing and Structure SDSH project personnel include 50 staff at SDSH headquarters in Port-au-Prince and approximately 20 staff assigned to the 10 departments, with one technical advisor and one financial advisor in each department (the former position is currently vacant in West Department, and is being covered on a part-time basis by a member of the headquarters staff). MSH is the project’s prime contractor, and is supported by subcontractors JHPIEGO, which focuses on maternal health and family planning/reproductive health, and by JHU-CCP, which focuses on communications/behavior change communications and efforts to increase community involvement. Many of the 25 technical staff at project headquarters support and backstop the separate health service “silos” (e.g., child health, maternal health, HIV/AIDS prevention and treatment, family planning/reproductive health, nutrition, TB), the elements of which comprise the essential health package promoted by the project. A particular challenge for project management is to ensure that the managers of these nominally separate foci adopt and promote an integrated approach to service delivery, notwithstanding the risk of tunnel vision. USAID/HAITI: SDSH PROJECT EVALUATION 9 Finding 8 Despite this vertical structure at headquarters, evaluation team observations indicate that the project has been successful in promoting an integrated approach via cross-training of personnel at headquarters and in the field. As discussed above, however, the project has not succeeded in developing a practically useful indicator to measure “integration”—an indicator that has eluded health care providers virtually everywhere. Finding 9 Departures of key personnel and the slow recruitment of their replacements have affected project performance on the margin, but do not appear to have substantively impeded the attainment of project goals. That said, the recent departure of several high-level managers calls into question MSH’s ability to maintain continuity in project management. Discussion During the evaluation team’s visit, the project’s third chief of party in four years resigned, as did the project’s senior financial manager. Those departures came three months after the project’s deputy chief resigned. Meanwhile, the project has been functioning without a communications director and a formally approved technical director (both designated as “key positions” in the MSH contract) for several months. Six technical advisors in as many departments quit the project for higher-paying jobs with other NGOs in the wake of the hurricanes and the earthquake. The continuing absence of a technical adviser for the West Department had been previously mentioned. The reasons for these departures and/or turnover are varied. The chief of party issue was a mixture of the routine (a promotion) and the problematic (internal MSH management issues), with recruitment efforts for previous chiefs of party complicated by unstable conditions in Haiti. The former deputy chief of the project and the chief financial officer left SDSH for senior positions in the MOH and Prime Minister’s office. The continuing absence of a communications chief has the potential to reduce the project’s contractual requirement to produce at least “one success story per month” (according to an SDSH project manager), but the project has been able to comply with that obligation thus far. The absence of a technical adviser in the West Department is producing some friction with West Department managers, who informed the evaluation team that the part-time arrangement discussed previously was not serving the office’s needs well. The managers want a full-time TA on the team (an acknowledgment of the importance that departmental leaders attach to the role of SDSH personnel attached to their offices). The abrupt departure in mid-October of the project’s chief financial officer may be especially problematic, given the project’s complex and labor-intensive obligations to simultaneously manage dozens of PBF contracts and be accountable to USAID for the use of project funds. Recommendation Given the difficulty that MSH has faced in recruiting and retaining expatriate chiefs of party, and in view of anecdotal reports that other key staff have left because they felt that any further upward mobility in the project was unlikely, the evaluation team suggests that USAID encourage prospective contractors to broaden the recruitment pool for the chief of party position to include Haitian nationals. Some USAID Missions have been reluctant to encourage the assignment of host-country nationals to COP positions, usually out of concerns that host-country nationals can be subjected to extraordinary pressures from colleagues and associates, that they might encounter conflicts of interest, and that they might be inaccessible to the U.S. justice system if they misuse project resources. These are real concerns; but so is the capacity of the SDSH project and its successor activity to function smoothly without the disruptions and discontinuities of personnel losses. A cautious approach to this 10 USAID/HAITI: SDSH PROJECT EVALUATION recommendation might be to apply it to the successor project and not to the current activity, primarily to avoid potential awkwardness with the GOH in the event that MSH—which has extraordinarily deep connections with the MOH—should not compete successfully for the follow-on activity. [“Today, our streamlined Economic Growth bureau is taking steps to improve the way we work on strengthening enabling environments. First—because we know that economic reforms require sustained high-level commitment from governments—we will focus our core economic growth assistance in countries that demonstrate real commitment to private sector led growth. Second, we’re going to accelerate our shift from a model where we provide technical assistance to governments through foreign consultants to one that employs local expertise, responds to the requests of partner governments and pays out based on measurable impact.” Rajiv Shah speaking at the Public-Private Partnership Forum, Oct. 20, 2011] Recommendation USAID should closely monitor the project’s efforts to replace the departing chief financial officer, and determine whether that person’s departure is negatively affecting the project’s financial monitoring and reporting responsibilities. Despite this flux in the project’s personnel structure, the overall project has consistently met its key goals and targets—most particularly in the all-important service delivery component of the program. Some of the reasons behind this sustained high performance are suggested later in this document. Finding 10 Reductions in the amount of funding that MSH expected to receive for the project required the reduction or elimination of some planned activities. Discussion Initial (FY 2007) funding for the SDSH project amounted to $42.5 million. Subsequent amendments executed between 2007 and 2009 increased the project’s funding to $51.4 million. A fourth amendment in 2009 added $30 million to the activity (new total: $81.4 million) and extended the project to the end of August 2012. A later no-cost amendment extended the project to September 30, 2012. SDSH personnel informed the evaluation team that the final $30 million funding increase fell short of their expectations and did not provide the funding needed to maintain the pace of expenditures that the project had established over the 2007–2009 period. As reported by SDSH managers, the $51.4 million that USAID provided to the project between 2007 and 2009 enabled the project to sustain an annual burn rate of $17 million. The final $30 million provided to the project, however, was intended to cover the final two years of project activity, and was adequate to sustain an annual expenditure rate of $15 million—or a reduction of $2 million/year in the project’s burn rate. According to MSH, this reduction in expected funding compelled SDSH to reduce the project budgets of virtually all of the participating NGOs, requiring many of them to make their own painful adjustments, including the release of some staff members. Several NGOs expressed resentment at those cuts, claiming that MSH refused in most cases to reduce project targets established for the NGOs’ programs despite the reduction in funds needed to achieve those targets. SDSH further informed the evaluation team that plans to create community task forces composed of elected officials, business leaders, religious leaders, school teachers, and others were shelved at least in part because funds were not available for that initiative. An equally important factor in that decision, however, appears to have been delays attendant to the 10-month USAID-MSH negotiation around the continued participation of JHPIEGO, JHU-CCP, and FOSREF in the project. By the time that negotiation USAID/HAITI: SDSH PROJECT EVALUATION 11 process was completed, little time was left to undertake a meaningful effort in support of the task force objective. Lacking the time and resources to address this issue, the project relies on proxies (e.g., mothers clubs, youth sensitized to health issues, and PLWHA groups sponsored by PEPFAR) to play that role. Finding 11 The essential structure of the project has remained fairly consistent over the past four years, despite the disruptions noted above and the variable priorities of USAID managers. Discussion The project has had three Cognizant Officer Technical Representatives (COTRs), (now called Contracting Officer’s Representatives [CORs] over the life of the activity. SDSH managers report that each COTR brought a particular set of perceptions and priorities to the project, and encouraged SDSH managers to pay more attention to that priority. One COTR, for example, wanted more emphasis on FP; another wanted to focus on HIV/AIDS, and yet another retreated from the FP component in the wake of a (subsequently reversed) GOH decision to procure FP commodities. The Mission’s current perception (as reported by SDSH staff) seems to be more holistic: that the SDSH project has many of the characteristics of a health systems strengthening (HSS) activity. Indeed, the project does include many of the elements of a classic HSS program, such as service delivery, human resource development, health care financing, health planning, governance, and development of a HMIS. However, as the project was not designed to be a HSS activity, several of those discrete elements lack the intensity and focus that would normally be found in a designed-from-scratch HSS. Perhaps the most notable weakness when viewed through the HSS lens is the project’s relatively light treatment of health care financing within the MOH. Finding 12 However various aspects of the project are defined, it is of course a health services delivery project, with some ancillary activities designed to promote better governance and management (and ultimately, better service delivery), and some companion initiatives (the Strategic Partners program and the GUC activity) that were appended to the project for reasons more indirectly related to service delivery. Discussion Approximately 80% of project funds support service delivery. The 25 technical staff at the SDSH office in Port-au-Prince, financial oversight personnel at that office and in the departments, and the technical advisors in the departments all clearly understand that the project’s essential purpose is to deliver an integrated package of health services. SDSH project managers have structured staff tasks to address that purpose. The participating NGOs and MOH departments look to SDSH as an essential—in many respects, the essential—partner in health service delivery. The NGOs and health departments, for example, depend on the project to support the recruitment, training, and operational costs (including salaries) of most of the NGOs’ community health workers and virtually all of the CHWs who work in MOH ZCs, as well as the training and salaries of many of the doctors, nurses, lab technicians, social workers, and department￾level administrators who support service delivery at PPSs managed by both the NGOs and the departments. Finally, one of the primary roles of the SDSH technical advisors at departmental levels is to work with the NGOs and departments to ensure that their action plans successfully target resources on service delivery, and to closely monitor partners’ implementation of those programs. The performance-based 12 USAID/HAITI: SDSH PROJECT EVALUATION contracting model, applied first with the NGOs and more recently in support of the ZCs, was specifically designed to incentivize the achievement of service delivery targets. Project Management SDSH staff work collaboratively with partner NGOs and with department-level counterparts to develop customized strategies, budgets, and targets for each organization/ZC program. Once targets are established for each NGO and ZC, however, SDSH (referred to by most partners as “MSH”) is a stern taskmaster in its pursuit of the partners’ compliance with those targets (MSH would say, in its “pursuit of excellence…”). Finding 13 Over the course of the last two projects and the current SDSH activity, MSH has developed several management and oversight tools which it applies with consistent rigor in its relations with its partners. Some of the NGOs welcome that rigor; others are candid in noting their preference for more “give” and flexibility on the part of the project. Discussion NGOs: Each year SDSH negotiates—some NGOs might say “imposes”—new performance targets with participating NGOs. With those targets in hand, the NGOs develop action plans and budgets that reflect those targets and submit copies to SDSH headquarters, with copies to the SDSH technical advisor and the health departments. The technical advisor, often in concert with department-level staff, reviews the draft action plans and forwards the draft plan to SDSH headquarters, along with his/her comments and recommendations. SDSH headquarters staff (including technical, financial, and management personnel), review the action plans and exchange email questions and clarifications with the NGOs, if needed, until both parties are satisfied with the plan and budget. SDSH then executes annual subcontracts with the NGOs. Each NGO action plan, and the subcontract itself, includes a discussion of all other assistance relationships maintained by the NGO, thereby enabling SDSH to ensure that project assistance complements rather than duplicates other support available to the NGO. The departments also try to eliminate duplication of effort and promote complementarity of efforts by hosting annual, quarterly, and monthly meetings of NGOs and donors active in the department. MSH closely monitors each NGO’s performance in meeting its contractual requirements, and especially its service delivery targets. For routine oversight, SDSH relies on NGO monthly reports, which the NGOs send to both SDSH and the department to track NGO performance and to flag any slippage in attainment of service delivery targets. Such slippage, should it occur, prompts a telephone call and a formal letter from SDSH—with copies to the department, the SDSH technical advisor, and the service site(s)—calling the NGO’s attention to the performance shortfall and inviting it to discuss remedial measures. SDSH similarly monitors the NGOs’ much more extensive quarterly reports (components include separate reports on community organization, training, supervision, procurement, finances, and service statistics). Those quarterly reports, which are the basis for quarterly disbursement of funds from SDSH to the NGOs, are consolidated at the MOH department office and forwarded to SDSH from that office. Once SDSH approves the budget request, it sends funds directly to a departmental bank account established for the participating NGO. SDSH’s department-level finance advisor focuses especially, but not solely, on the department’s management of project funds. In addition to this “remote” oversight, SDSH also conducts a service delivery and management assessment (SDMA) at least once per year, during which time SDSH technical, financial, and management teams (often accompanied by a USAID representative) undertake in-depth reviews of the NGOs’ USAID/HAITI: SDSH PROJECT EVALUATION 13 performance, capacity, strategy, and procedures. At the time this report was written, SDSH planned to field four SDMA teams (15 people in all) at the end October 2011, during which it planned to visit approximately 60% of participating NGOs over a six- to eight-week exercise. Finally, SDSH convenes meetings twice each year of all of the project’s participating NGOs. The meetings provide an opportunity for the NGOs to identify and discuss problems they have encountered, hold one-on-one meetings with SDSH managers, and share experiences with other NGO leaders and managers. SDSH also uses the meetings to underscore the project’s strategic objectives and to provide additional training to NGO managers via workshops, small group sessions, and the like. Finding 14 Discussions with a selection of NGOs indicate considerable variation in their satisfaction with SDSH management procedures, especially with regard to MSH’s firmness in its demand for NGO compliance with the project’s reporting schedules and service targets. Discussion Delays in the submission of monthly and/or quarterly reports can result in a budgetary penalty that, while not substantial, is often perceived as unreasonable by some NGOs. To be sure, other NGOs welcome such a disciplined approach, pointing out that it promotes professionalism in transactional relationships to an extent not commonly observed in Haiti. Several NGO managers pointed out, for example, that SDSH is probably the only donor or donor-supported effort in Haiti that imposes such discipline on its partners. The most oft-cited complaint of NGO managers interviewed was the project’s (i.e., MSH’s) “rigidity” in its insistence on the attainment of project targets, even when circumstances—the earthquake, the cholera epidemic, product stock-outs, and so forth—made attainment of those targets impossible. Others pointed to the project’s imposition of increased year-over-year targets without any increase in financial or other resources needed to achieve those elevated targets (“If they want more work,” one manager said, “they should give us more funds for more staff to do the job, and more transportation to get them around.”) In general, the thrust of such remarks was a call for MSH to be more open to dialogue around the issue of targets, and to retreat from a position characterized by one NGO director as “take it or leave it.” Other changes called for by NGO managers included the following:  Salary adjustments: The low salary (the equivalent of $150/month) of community health workers was by far the most oft-cited concern of NGO managers. SDSH has not, according to these managers, factored in cost-of-living adjustments to these or other staff salaries in several years, and salaries are lower than those offered by other NGOs. Several managers reported that they have lost key staff to higher-paying organizations.  Prompt arrival of reimbursement payments: One NGO manager, who described the quarterly payments as being “chronically late,” noted that “when we submit a late report, we get penalized.”  Less paperwork: Each year, SDSH distributes ledgers to the NGOs and requires manual entry of service data into those ledgers. Since the ledgers are annual documents, NGO staff—usually nurses responsible for patient care—must painstakingly transcribe the immediate past year’s data into the next year’s ledgers. Over a period of four years, the same data would be transcribed three times. The evaluation team observed one set of such ledgers that was approximately 10 inches high.  More cars: NGO managers seem to be under the impression that MSH has some discretion in this area, and believe that SDSH has chosen to ignore the NGOs’ critical need for transport. 14 USAID/HAITI: SDSH PROJECT EVALUATION  Inclusion of an indirect cost rate in the subcontracts: This issue, raised by one NGO that had other contracts with USAID, pointed to the absence of NGO flexibility in managing a 100% direct-cost contract with no overhead or fringe. USAID Contract Office personnel reportedly told him to “work it out with MSH,” which will not negotiate on the issue.  Less reliance on population-based performance targets: Some NGO managers noted that population data in Haiti are badly out of date and do not take into account the significant population shifts after the earthquake. Others called the team’s attention to a cultural curiosity in Haiti, whereby people not uncommonly seek health care at PPSs fairly distant from their home community. (Whether this is done to protect their confidentiality, or to visit other areas, or for other reasons, is not known.)  English-language subcontracts: The subcontracts that MSH executes with partner NGOs are in English, with no French or Creole translation provided. Some NGO managers said they were signing contracts and committing themselves to tasks that they did not fully understand. The evaluation team is not proposing this as a formal recommendation, but does urge MSH to explore the possibility, with the USAID/Haiti Contract Office, of providing NGOs with courtesy copies of subcontracts in French and/or Creole, even if those translations would not be the legally binding version of the subcontract. Notwithstanding the concerns cited above, virtually all of the NGO managers interviewed were positive in their overall assessment of the management procedures employed by the SDSH project and the critical role that the project plays in helping NGOs provide essential health services. “MSH is like a parent,” said one manager. “A sometimes stern parent, but a good one.” Also, MSH had not worked with the MOH to support activities in the ministry’s underserved, underfunded ZCs under either of the two predecessor projects. Prior to the end of HS 2007, however, the MOH informed MSH that any subsequent project should include efforts to deliver health services to the ZCs. Finding 15 Despite MSH’s lack of prior experience with the ZCs, the SDSH project responded well to the MOH request that the project work in those areas. Discussion Soon after the launch of the SDSH project, the MOH presented MSH with a list of the specific ZCs that the MOH wanted SDSH to support under the project. MSH complied, and subsequently worked successfully with MOH leadership at the central and departmental levels to expand health services in those underserved areas and to help prepare the MOH to apply the PBF model in all ZCs. In October 2011, the final four departments agreed to participate in the PBF model; at the time of writing, SDSH managers expected that PBF contracts will be in place in all 10 departments by April 2012. SDSH oversight of the departments’ ZC programs is similar to the monitoring it provides to the NGO programs. The key differences are that (1) ZC action plans are developed at the department level by MOH staff responsible for the ZC program (e.g., medical directors, nurses, program managers input from the SDSH technical advisor); (2) ZC contracts are signed by the MOH department director and by MSH representing SDSH, with separate contracts signed for each ZC; (3) both SDSH—represented by the technical advisor—and department-level technical personnel monitor ZC performance (the Statistical Service at the ZC sends monthly performance data to SDSH for each PPS in the ZC), and in the case of “slippage,” perform joint supervisory visits to project sites; and (4) many of those ZC contracts are not yet performance-based agreements. Department-level managers interviewed by the evaluation team noted that service delivery performance in ZCs improved when they transitioned to performance-based contracts. USAID/HAITI: SDSH PROJECT EVALUATION 15 All of the department-level managers interviewed by the evaluation team were positive in their assessment of the project’s performance in support of ZC programs. Many stressed the point that those in chronically underserved areas would lack any meaningful level of health services were it not for the SDSH project. One mildly critical MOH official encountered by the evaluation team—a senior administrative official in West Department—agreed that the project was playing an indispensable role in support of the department’s programs and characterized MSH as a “member of the family.” He noted his preference, however, that SDSH project funds be channeled through the department; and he objected to SDSH’s recruitment of some staff without departmental vetting or approval. The departmental director in another department described MSH as overly “rigid,” but not in the sense expressed by some NGO managers. His objection related to SDSH’s refusal to increase project (i.e., ZC) budgets in the face of health emergencies, with specific reference to the cholera outbreak. The Canadian International Development Agency (CIDA)-supported Projet d’Appui au Développement du Système de Santé (PADESS) project, he said, provided an immediate infusion of funds for the recruitment of 20 additional nurses, while SDSH stood firm on its project budget. Perhaps the most useful recommendation offered by a department director was a request that any future project authorize the USAID contractor to identify candidate ZCs with department chiefs, rather than (or perhaps in addition to) the central MOH office. Moving that discussion to the department level would be consistent with Government of Haiti and MOH plans to move purposely on the government’s decentralization initiative. The evaluation team is not offering a recommendation on this point, as action in this area is wholly dependent on Haitian government decisions, rather than those of USAID and/or SDSH (though USAID and/or MSH could engage in policy dialogue to encourage such a decision). Finding 16 The enthusiasm with which MOH departmental leaders have embraced the PBF model does not imply their readiness to successfully manage that approach. The development of MOH (central and departmental) capacity to successfully implement the PBF model will require a long-term program of technical collaboration with the ministry at both central and departmental levels. Discussion MOH departmental leaders and project managers consulted by the evaluation team expressed favorable views regarding the PBF contracting model. These managers cited, for example, the model’s use of rewards and penalties to promote the attainment of project targets and the close oversight exercised by on-site technical and financial advisors and SDSH staff in Port-au-Prince. MOH staff were generally aware, moreover, that the leadership of the MOH (and the president of the republic) have strongly endorsed PBF as a management approach that will be adopted and broadly implemented by the Government of Haiti in the future. MOH departmental staff further acknowledged, however, that the managerial depth of their offices was very thin and that they were not prepared technically to assume substantive responsibility for management of SDSH-supported activities. As a practical matter, they said, the two on-site SDSH advisors effectively managed SDSH activities in the departments. Indeed, MOH department staff invariably invited the two SDSH advisors to the evaluation team’s interviews, and deferred to those advisors in addressing most of the team’s questions regarding project performance and oversight. Recommendation The follow-on project should include provision for a comprehensive effort to develop and install professional-level capacity at the MOH departmental level to design, negotiate, monitor, and implement a PBF system. (See below, under “Project Performance,” for a similar recommendation regarding the MOH central level.) 16 USAID/HAITI: SDSH PROJECT EVALUATION Finding 17 SDSH staff—the technical and financial advisors—were observed to be well-integrated into department￾level management teams. They were highly respected by MOH colleagues, by the staff of NGOs active in the various departments, and by MOH personnel responsible for the ZCs. Their success in developing such relationships has contributed to the achievement of key project objectives at the department level. Discussion The project’s success in integrating SDSH advisers into departmental teams has had a dual effect: First, it has helped ensure the operational success of the project itself, such as by establishing an on-site capacity to help develop action plans (both NGO and ZC), and to provide close and consistent follow-up with NGOs and ZCs to ensure compliance with project targets. Second, SDSH’s on-site advisers have played a major role in supporting the SDSH project’s governance objectives at the department level by promoting the departments’ bottom-up planning process. It is likely that some departments would be thorough and consistent in their efforts to convene all organizations active in the health sector at the department level, and to work with them to plan an all￾department strategy—represented in the Plan Departmental Integral (Integrated Department Plan). It is equally likely, however, that this process would be less thorough and less productive without the encouragement and support of the on-site SDSH personnel. As a consequence of that support, the MOH at the department level has been able to put down an important “marker” that it is responsible for technical and managerial leadership in the health sector, and that nongovernmental partners have a high degree of accountability to the MOH for their plans and programs in the departments. Recommendation The follow-on project will continue to require close collaboration with MOH counterparts at the department level. USAID should include provision for department-level advisors in the contract for the follow-on project. Finding 18 The ability of the SDSH project team (technical and financial advisors) to blend so well into the MOH management team at the departmental level contrasts with the project’s “separateness” at the SDSH headquarters level. Discussion The SDSH office complex is located a considerable distance from the MOH compound in Port-au-Prince (especially when measured in travel time, rather than miles). In addition to this physical separation, USAID-mandated and pervasive “branding” (e.g., vehicles, documents, office equipment, reports) tends to undermine the perception of MOH ownership in the project. Such extensive branding is consistent with USAID policy and regulations—Automated Directives System Chapter 303, specifically (and is probably required by statute)—but seems to fly in the face of GHI and USAID Forward directives calling for tighter association of USAID projects with host-country partners. Recommendation USAID/Haiti should consult with USAID/Washington to determine if any relaxation of USAID branding regulations might be possible, given the directives set forth in the GHI and USAID Forward initiatives. As will be noted below, key elements of a follow-on project proposed by the evaluation team will call for close collaboration between the USAID contractor and MOH partners at the central and departmental levels. Symbolic and geographic distance between the parties can interfere with that collaboration. USAID/HAITI: SDSH PROJECT EVALUATION 17 Project Performance The SDSH project prepares a variety of reports (e.g., annual and semiannual project reports, annual PMPs, milestone reports, ad hoc reports requested by USAID). As discussed with USAID/Haiti, this report will not repeat the information in those documents, which are already available to USAID project managers. Moreover, the evaluation was not asked to validate the accuracy of performance data shown in those reports. Rather, this section of the evaluation assesses the project’s performance from a broader perspective and attempts to examine aspects of project performance not addressed in SDSH project reports. Finding 19 The SDSH project has performed very well in the most critical assignment: to extend the availability of integrated health services in Haiti. Discussion As noted in SDSH project reports, the activity is broadly successful across almost all of its targets, albeit with some slippage in TB, maternal health, and FP. A particularly compelling statistic encountered by the evaluation team says a great deal about the project’s impact: 85% of children in ZCs supported by SDSH have received their full course of immunizations vs. 47% of children fully immunized in the rest of the country.1 Although this is only one indicator among the several dozen in the project’s PMP, the immunization rate is a useful proxy for the overall project’s reach and impact—in this case, into areas (ZCs) that are the least-served parts of the country in terms of the population’s access to basic health care. In the judgment of the evaluation team, this success is due to the project’s adoption and implementation of a service delivery model that has a number of key, interlocking components. These include:  Community health workers: CHWs recruited, trained, and supported by the project deliver health services at the doorstep, monitor family health, motivate and organize (mostly women and children) at hundreds of gathering posts where they receive essential services and can be referred to PPSs for more extensive care. These workers, selected by leaders from their own communities, form the “spine” of the SDSH-supported program. SDSH resources enable the 28 participating NGOs to greatly augment their existing cadre of CHWs, and fields scores of new workers for work in ZCs, where the MOH had previously fielded only a scant number of such workers. These agents are supported by and work seamlessly with other trained personnel.  Trained PPS personnel: SDSH upgraded the technical skills of health care providers (in maternal health, child health, FP, HIV/AIDS prevention and management, nutrition, TB detection and treatment), managers, and support personnel at NGO and ZC health posts, and ensured that the training provided to these staff complemented the training provided to the community health workers. Both sets of health providers are supervised by senior staff.  Trained personnel at the NGOs and MOH departmental offices: SDSH trained senior technical and managerial staff to monitor performance of the service providers, identify problem areas, and provide supportive supervision for health care providers. These staff were in turn motivated by MOH departmental leadership.  MOH departmental leadership: The MOH has bought into the concept of a much more robust, community-based approach to health care delivery, and has embraced a more active role for itself in providing coordinating and technical leadership for all health-related activities at the department level. In performing these tasks, MOH departmental leaders have looked for support from on-site                                                              1 Source: MOH Division of Family Health. 18 USAID/HAITI: SDSH PROJECT EVALUATION SDSH technical and financial advisors, who provide essential oversight of NGO and ZC performance; they have also sought to maintain liaison with SDSH headquarters and serve as advocates and advisors on the application of a PBF model for participating NGOs and ZCs. All these functions are supported by SDSH project management in Port-au-Prince, which provides close and consistent monitoring, technical and financial support, and training. Finding 20 As a cautionary note, it needs to be pointed out that neither the overall project nor any of its components are sustainable, in the sense that the activities have any realistic likelihood of continuing in the absence of donor support. Neither the NGOs nor the MOH has, nor is likely to have, adequate resources in the foreseeable future to support and sustain any of the elements of the service delivery program. Indeed, many of the training costs of the project are more recurring than one might prefer, as staff turnover—mostly at technical and professional levels—has continued to bedevil the project since the arrival, post-earthquake, of several better-paying NGOs. Recommendation The follow-on project should retain all elements of the service delivery model, including the funding required to ensure their successful operation. Finding 21 Project efforts to improve MOH executive capacity at the central level have had mixed success. Discussion MSH has been engaged with the MOH at the central level since its HS 2004 project. Although that long involvement has generated considerable MOH familiarity with and respect for MSH, it has yielded relatively few operational products. The current project did provide valuable assistance in the development of new norms, standards, and protocols for RH, and played a contributory role in the development of a national strategy for prevention of mother-to-child transmission of HIV/AIDS (PMTCT). But years of project work on a new financial management manual and efforts to develop and implement a national HMIS have yet to produce operational tools for the MOH. The financial management manual and related tools, for example, are not yet in use at the departmental level. One department director explained that they had the financial software—a version of QuickBooks— provided to them by SDSH, and that department staff had been trained in its use. However, that software conflicted with another, PEPFAR-related financial management program which the U.S. Centers for Disease Control and Prevention (CDC) insisted that the department use. Similarly, the HMIS is in place at the central level, but is not being utilized to gather, analyze, and package data from the departments. Moreover, the modality of work in these areas—work is focused in technical committees composed of MOH counterparts and other donors—does not appear to have resulted in meaningful transfer of skills and capacity to the MOH. Looking beyond those still-underutilized tools, the MOH is still poorly equipped to address the overarching and unresolved challenge of health care financing. The ministry lacks a clear framework for health care financing, and has not indicated keen interest in addressing such critical issues as costing of service delivery, cost recovery, gap analysis, and resource allocation. Almost as pressing is the disconnect between the health ministry’s (indeed, the GOH’s) explicit commitment to PBF and the lack of technical and managerial skills (not to mention the financial resources) needed to manage a PBF approach to health care. USAID/HAITI: SDSH PROJECT EVALUATION 19 On a more optimistic note, the new leadership at the MOH has signaled its intention to work constructively with donors on a wide range of strategic, policy, and procedural reforms, starting with an invitation to donors to help the ministry develop a new strategic plan for the health sector. As one of the ministry’s key partners, USAID will be expected to participate in those efforts over the years ahead. Recommendation Other donors are currently working with the MOH at the central level on health care financing (HCF), and will probably (as reported by CIDA-PADESS) continue to do so. In the interest of keeping its assistance efforts lean and focused, USAID should defer to those donors in the broad HCF arena and instead target its future assistance efforts on PBF, which has the highest-level political support and is critically important to the long-term success and durability of the health services model. USAID engagement in PBF will give the Mission entrée to the other HCF-related conversations at the center, but the Agency’s prior history and marginal impact with HCF in Haiti suggest that a major USAID investment in this area would distract the agency from an alternative effort (installing PBF capacity at the MOH) that would have greater likelihood of long-term success. Finding 22 Project data indicate that SDSH exceeded or met targets for the majority of indicators in all groups, with the exception of TB and maternal health. SDSH reports note, moreover, that eight of the project’s FP targets were surpassed, including the percentage of users using long-term methods. A member of the MOH senior leadership informed the evaluation team, however, that “family planning is the one significant ‘weakness’ of the SDSH project.” Discussion SDSH actions to intensify its community-based FP efforts, and particularly its efforts to inform people about long-term methods, were reportedly set back by delays in reactivating the JHPIEGO subcontract. That subcontract was executed at the end of 2010, and three experienced RH advisors joined the SDSH team early in 2011. Those advisors should enable the project to address critical training needs to increase the number of providers trained to dispense long-term FP methods. Even under the best of circumstances, however, this augmented training capacity will encounter a basic structural obstacle in Haiti, namely, the scarcity of obstetricians/gynecologists in rural parts of the country. Recommendation The RFP for the follow-on project should include instructions that prospective contractors describe steps they would take to address the critical lack of medical personnel in rural areas, and how they would overcome this scarcity to expand the availability of long-acting and permanent FP methods (LAPM) in Haiti. Finding 23 The project responded well to the post-earthquake cholera epidemic. Several NGOs visited by the evaluation team noted, for example, that SDSH personnel provided their service delivery staff with the first and only training they had ever received in the care of patients suffering from the disease. SDSH demonstrated administrative creativity, moreover, in responding to conditions that technically fell outside of their contract by intensifying its work under the “prevention and management of diarrheal diseases” element in the SDSH project budget. Lacking a clear contractual mandate to address cholera, however, SDSH is limited in its ability to respond aggressively to an epidemic that has become, in the words of Paul Farmer, “the most serious in the world proportionate to the size of the ravaged 20 USAID/HAITI: SDSH PROJECT EVALUATION Caribbean nation. Close to 500,000 Haitians out of a population of more than 9.5 million have been sickened by the disease and more than 6,500 have died in just one year's time” (Reuters, October 20). Recommendation To the extent allowable within the terms of the USAID-MSH contract, USAID should direct MSH to conduct an assessment of its partners’ capacity to broaden and intensify their cholera prevention and treatment efforts. The assessment could be incorporated into SDSH’s next and presumably final annual SDMA exercise in 2012. (The 2011 SDMA began on or about November 1 and continued into December.) USAID should draw on that assessment to include more explicit provisions regarding cholera prevention and treatment in the follow-on contract to the SDSH project. Any prospects for an enhanced NGO response to the epidemic should be coordinated closely with the MOH and other donors involved in the cholera prevention and treatment campaign. Recommendation The evaluation team notes that donors have not established consensus around the most effective response to cholera on a long-term strategic level, such as through broader use of vaccines or more attention to water and sanitation systems. USAID/Haiti should closely monitor this discussion and be prepared to tailor the cholera-related elements in the follow-on project to most effectively address the epidemic. Finding 24 Some elements of the project, though generally successful and useful in their own right, are not contributing substantively to attainment of the project’s primary objective: the expansion of integrated health services to under-served populations. These marginally useful elements could be dropped from the project without negatively affecting the primary objective. Discussion Strategic Partnerships The initial impetus for the Strategic Partnership component of the project was a requirement in the USAID-MSH contract that MSH provide a cost-sharing contribution amounting to 20% of USAID’s initial contribution, which amounted to a $6 million cash liability for MSH. Following MSH’s request for a closer review of that provision by the USAID Contract Office, USAID determined that the $6 million obligation could be satisfied if MSH were able to leverage that amount from other donors, such as the private sector. With guidance from the USAID Global Development Alliance, MSH embarked on an ambitious effort to solicit funds from the business sector, starting with the banking industry. MSH’s initial if modest success was an agreement with UNIBANK, which provided $5,000 toward the cost of a presentation by local and expat Haitian performance artists at the Rex Theater in Port-au￾Prince. More successful, and far more relevant to the SDSH project’s objectives, was a partnership with Pure Water for the World (PWW), supported by Rotary International and the Rotary Club of Petionville. That activity is introducing potable water in 450 schools, 50 health centers, and 150 to 200 homes each year, with a value to date of approximately $500,000 in in-kind contributions. Over the past three years, MSH has developed several other partnerships, including the CARIS Foundation, which developed a simple pediatric tool to test for HIV in infants; KOMBIT Santé, which helped convert shipping containers to clinics; and others. All of the partnerships are formalized by memoranda of understanding (MOUs) between MSH and its various partners. By far the most significant partnership as measured by the in-kind value of its contribution was MSH’s partnership with Direct Relief International (DRI). DRI had worked in Haiti after the 2008 hurricanes to USAID/HAITI: SDSH PROJECT EVALUATION 21 pre-position supplies needed in case of future disasters. Introduced to SDSH by KOMBIT, DRI needed a local organization after the 2010 earthquake that could identify sites in need of DRI relief supplies, assist with clearance of those supplies through Haitian customs, and help with distribution of the supplies. A year later, during the cholera epidemic, SDSH played a similar role for DRI when DRI delivered IV fluids, electrolyte solution, and oral rehydration salts for the treatment of cholera victims. That relationship with DRI—the value of which included $4 million (in kind) for cholera supplies and $54 million (in kind) for earthquake relief—enabled MSH to easily surpass its $6 obligation. (SDSH reports that the total amount leveraged from all sources amounted to $72.5 million as of January 2011.) MSH deserves considerable credit for having played such a supportive role in the delivery and distribution of essential supplies for earthquake and cholera victims. The partnership with PWW, moreover, is a useful adjunct to the project’s efforts to prevent diarrheal diseases, including cholera. But with the exception of the PPW initiative, few if any of the partnerships will have a lasting impact in Haiti and they do not impact directly on the project’s key objective. They are not, in short, “strategic,” nor do they fit into any long-term fundraising or service-delivery strategy on the part of MSH. As an aside, the team notes that MSH might not have been able to satisfy its contractual obligation were it not for two crises, the likes of which one hopes will not occur again. The evaluation team does not want to suggest that such partnerships are not valuable. They are, and USAID/Washington attaches considerable importance to Mission efforts worldwide to seek and execute public-private partnership (PPP) arrangements with private sector partners. But placing that burden within an already labor-intensive project (SDSH), essentially as a means to satisfy a financial obligation, poses a significant distraction to a project management team that should be focused on its primary goals. Finally, SDSH personnel responsible for the Strategic Partnership activity informed the evaluation team that they deliberately avoided the solicitation, receipt, or management of cash contributions from their partners in order to avoid complicating the project’s bookkeeping and their accountability to USAID. The evaluation team notes the inconsistency between that concern and SDSH managers’ parallel concerns that USAID budget cuts were negatively affecting the project. Recommendation USAID should continue its efforts to engage private sector partners, but not within the framework of a follow-on to the SDSH project. Recommendation PPPs are a positive goal in their own right, and should not be undertaken by USAID contractors as a means to satisfy cost-sharing or leveraging requirements. Recommendation If USAID decides to retain a PPP component within the follow-on project, the project contractor should be authorized and encouraged to seek cash contributions as well as in-kind donations and to reinvest any additional funds in the project. The evaluation team notes that SDSH is currently managing and distributing funds to and through dozens of department-level bank accounts, and is confident that adequate controls and oversight procedures can be put in place to monitor additional resources. Grants-Under-Contract As stated in SDSH reports, the purpose of the GUC program is to “establish coalitions or partnerships with community-based and faith-based (CBO/FBO) organizations to support decentralization of health care with a focus on SDSH priority services. The GUC initiative coincides with USAID’s new strategy for Haiti that aims to strengthen local capacity and enable a more inclusive civil society.” 22 USAID/HAITI: SDSH PROJECT EVALUATION This GUC activity got off to a slow start, delayed at least in part by the need for MSH to secure USAID authorization to execute grants (MSH had existing authority to execute its performance-based contracts with NGOs and MOH department directors, but had no authority to execute grants with local entities). MSH executed its initial eight grants in April and May of this year, following a March 2011 workshop at the SDSH office in Port-au-Prince, during which selected proposals and work plans were discussed and finalized. The eight existing grants provide support for the recipient organizations and (through six) of them funds for subgrants to 35 smaller community groups and associations. The cost of this initial round of grants was approximately $232,000. SDSH plans to extend the GUC program to an additional 12 communes and 24 umbrella organizations before the end of the project in September 2012. The evaluation team visited only one umbrella organization funded under the GUC program—the Fondation pour le Développement de Petit Trou de Nippes (FONDEP) in Nippes—so the team’s findings are tentative at best. Based on that very limited experience, however, the team questions whether the time and, to a lesser extent, the modest amount of funding invested in the program is necessary to the achievement of project objectives. The team’s concerns were as follows: Grantee ScreeningTtransparency in the Selection Process FONDEP was the sole successful applicant from among the 12 organizations that submitted applications and the required paperwork to the Department Sanitaire in Nippes (invitations were sent to some 50 organizations). The screening process was conducted by department personnel, who applied selection criteria provided by SDSH with little participation by the SDSH technical or financial advisors, although the financial advisor did work with FONDEP on its financial reporting responsibilities after the grant was executed. Although MSH’s deference to the department was in keeping with efforts to encourage a more activist role by the department in project management, the transparency of the process remains somewhat murky. When pressed by the evaluation team to explain the screening process, department staff indicated that the most important criterion among the SDSH-provided criteria was applicants’ ability to provide a copy of a letter from a GOH organization (a ministry, a town hall, etc) that authorized the applicant to function as a community service organization. Not clear, however, was the extent to which the screening process considered the practical capacity of the applicants to provide meaningful services to the public. Financial Management The SDSH grant to FONDEP amounted to the equivalent of $50,000 for six months, which FONDEP was to distribute to six other CBOs to support the latter’s projects. These included efforts to increase community awareness regarding HIV/AIDS, cholera, STDs, sanitation, malaria prevention, personal hygiene, and disaster preparedness; water treatment at schools; leadership training; and the construction of hygienic display tables for fish vendors. The six-month $50,000 grant to FONDEP was a 16-fold increase in FONDEP’s annual budget of $6,000 (FONDEP estimate), calling into question the organization’s capacity to manage such a dramatic increase in its budgetary flow-through. More important, however, are considerations regarding the net added value that such small programs will have on the SDSH project’s essential task, which is to extend basic health services to the poorest and most vulnerable elements of the population. Time SDSH project managers note that implementation of the GUC initiative was a “long and labor-intensive process that demanded several trips to the field and many meetings and discussions with community organizations that had limited or no experience with donor-funded projects. Because this was a new initiative with untested partners, the USAID review and approval process took several months” (SDSH Semi-Annual Progress Report, October 1, 2010–March 30, 2011). Given the modest net increase that the GUC program will generate on availability of basic health services, the evaluation team questions whether the opportunity cost of the GUC initiative outweighs its benefits. USAID/HAITI: SDSH PROJECT EVALUATION 23 The evaluation team acknowledges that the incorporation of the GUC and Strategic Partners activities into the SDSH project helps to address some of USAID’s cross-cutting priorities in Haiti. These include, in the case of the GUC program, efforts to strengthen civil society and to make elected officials more accountable to the citizenry. The Strategic Partners activity reflects USAID policy that encourages the creation of PPPs. The evaluation team believes, however, that every project in the Mission’s portfolio need not incorporate agency initiatives that might create distractions from the project’s primary goals. From an external observer’s vantage point, it appears that MSH may have attempted to fashion the SDSH project to address such USAID priorities without factoring in the potentially negative effects that such actions might have on attainment of the project’s essential goal, which is to expand the availability of basic health services to under-served populations in the country. Recommendation The design of the follow-on project should call for a lean activity (in terms of its various components) focused on the extension of basic health services plus the capacity-building efforts required to support that expansion. Extraneous elements that do not directly support that objective should not be included in the project. USAID Management Finding 25 SDSH managers noted their generally high level of satisfaction with the oversight, backstopping, and support they receive from USAID/Haiti. Discussion Among the few concerns raised by SDSH project staff, the most significant related to the protracted process needed to reactivate the JHU-CCP and JHPIEGO subcontracts after USAID dropped the subcontractors’ participation in the project. The evaluation team assumes that USAID’s decision represented both an attempt to economize on project costs, and to bring the SDSH contract into closer compliance with the agency’s GHI and USAID Forward principles, which call for greater reliance on host-country partners and, by implication, less reliance on the usual assortment of U.S.-based contractors. In the judgment of the evaluation team, that decision—if it was indeed based more on GHI and USAID Forward considerations and less on financial considerations—was well-intentioned but ill￾advised. Both of the USAID initiatives cited are calls to action by agency leadership, but they are not— again, in the view of the evaluation team—directives that USAID Missions immediately amend all of their contracts to bring them into compliance with the initiatives. Such compliance can best be assured through a gradual and orderly process carried out as older projects end and new ones begin. Other concerns include a two-month delay in receipt of USAID approval for a senior-level SDSH appointment (technical director), and the high turnover rate of USAID CTOs. The former issue has since been resolved, and was at least partially due not to USAID inaction, but rather to an MSH freeze on personnel actions at the time. The second issue is a minor one: it adds some occasional burdens on the project, mostly in terms of the need to ensure that each new manager is fully briefed on the project, and the project’s need to respond to the sometimes dissimilar priorities and operating styles of each new CTO. From a more general perspective, the evaluation team observed that USAID/Haiti assigns a high value to the monitoring and evaluation (M&E) function, and seeks to apply rigorous M&E standards across all elements of the Mission’s assistance portfolio. During the evaluation team’s visit, that rigor was evident in the case of the SDSH project by the Mission’s practice of reviewing performance targets set forth in SDSH’s annual PMP. That review—which apparently includes an examination of each one of the nearly 90 performance indicators in the PMP—falls somewhere on a continuum between close and supportive 24 USAID/HAITI: SDSH PROJECT EVALUATION oversight on one side of the bar, to micromanagement and substitution of USAID technical judgment for SDSH technical judgment at the other end. The evaluation team did not participate in that review, so it has no informed observations to make on the process, except to suggest that the Mission should retain a high degree of self-awareness as it undertakes those reviews. At the very least, Mission decisions to change project-proposed targets should be based on empirical evidence, and should be the product of give-and-take discussions with SDSH managers. Finding 26 USAID/Haiti may be missing an opportunity to increase the impact of its Title II food assistance and its health programs by not ensuring closer integration of the two activities. Discussion The evaluation team found only one instance of close cooperation between an SDSH-supported NGO (MEDISHARE) and a Title II implementing organization (World Vision). The two organizations in the Central Plateau Department plan and conduct joint MCH feeding programs at project gathering points, where MEDISHARE service delivery personnel provide health services and World Vision provides food rations to the mothers and children being treated by MEDISHARE health workers. That collaborative effort is the result of the initiative of the managers of the two organizations, which is to say that it is an ad hoc arrangement. Recommendation USAID should expand and formalize NGO cooperating agency (CA) arrangements by encouraging its food assistance partners in Haiti to reach out to the NGOs that participate in the SDSH project and to include their plans for collaboration in the CAs’ Multi-Year Assistance Plans (MYAPs). USAID/Haiti could also include a requirement that CAs include such proposals in their applications for Title II resources, and announce that preference would be given to applications that include such collaboration. Partnership Model Grantee selection: As noted earlier, the SDSH project has not significantly altered the array of NGOs that are participating in the project. Indeed, with just a few changes, the current “membership” in the project is essentially unchanged since 2004. In many important respects this is a key to the project’s success. By now, the participating NGOs are for the most part highly experienced in applying the performance￾based contracting approach; they have come to understand (if not always agree with) the project’s rigorous oversight procedures, and they are fully committed to the community-based model that is key to the project’s effectiveness. Finally, they have evidently become full and supportive participants in the SDSH project’s efforts to ensure the primacy and leadership role of the MOH health department in all aspects of health service delivery at the department level. On a less positive note, it is not evident that many of the participating NGOs are poised for graduation from the project, as discussed in greater detail later in this report. Finding 27 Many of the participating NGOs have become highly dependent on the project; termination or reduction of project support would produce significant pull-backs in their service delivery programs. Discussion Most of the participating NGOs have fallen into an “aid trap,” in the sense that they used SDSH project resources to expand personnel recruitment and field operations that are largely funded by those USAID/HAITI: SDSH PROJECT EVALUATION 25 resources. All of the NGOs consulted by the evaluation team explained that any reduction in project support would lead to major cutbacks in their service delivery programs, as alternative support for those services is not available. To be sure, different NGOs depend on SDSH to varying degrees, and the cutbacks the NGOs describe also vary from organization to organization. SAVE, for example, plans to end its support for the four PPS and related activities that SDSH supports in Central Plateau, largely because the duration of SAVE’s support for those programs far exceeds its 10- to 15-year policy (the four PPS have been in SAVE’s network for 25 years). The Service and Development Agency (SADA), on the other hand, depends on SDSH for fully 70% of its program costs and would have to shut down most of its program in the absence of SDSH assistance. On the surface, FOSREF would appear to be relatively immune from a major reduction in support, as only 6% of its overall budget is provided by SDSH. But that predictable, year-to-year 6% takes on added weight in an institution that depends on other donors who closely tie their assistance to specific, often short-term projects. In short, and as noted earlier, the NGO component of the SDSH project is not poised for long-term sustainability in the foreseeable future. Any reduction in project assistance will lead to a significant dismantling of the NGOs’ community-based delivery programs. Similarly, the MOH is not prepared to assume increased responsibility for costs associated with the ZC component of the program. Senior leaders at the MOH acknowledge that the GOH has stated its intention to increase allocation of resources to the health sector. They add, however, that such additional allocations will continue to depend on the availability of donor assistance. Recommendation The follow-on project should include adequate funding to fully support the key elements of the current SDSH project plus the costs needed to expand service delivery to additional parts of the country, consistent with USAID strategy (e.g., to intensify coverage in the “corridors”). Performance-based Financing The Center for Global Development Working Group on Payment for Performance defines PBF as “the transfer of money or material goods from a funder or other supporter to a recipient conditional on the recipient taking a measurable action or achieving a predetermined performance target. The goal of PBF is to promote hard work, innovation, and improvements along specified dimensions.” (“Impact of performance-based financing in Haiti.” Unpublished report prepared for MSH by Schneider Institutes for Health Policy, Heller School, Brandeis University, and the Center for Health Services, MSH. June 2, 2011). Finding 28 Application of the PBF approach is having a significant and positive impact on the achievement of SDSH project objectives Discussion PBF is not a new concept in Haiti; MSH has been applying the approach to its programs since 1999, and evaluations of the model in 2001 and 2009 demonstrated that NGOs enrolled in the PBF scheme performed better than other, nonparticipating NGOs. With this prior success in mind, MSH scaled up its PBF initiative in its HS 2004 and HS 2007 projects, and carried it over to the SDSH activity. The most significant change incorporated into the current project was an MSH decision to select one indicator for payment in each performance category (e.g., HIV/AIDS, maternal health) randomly, out of all the indicators in that category, and to not disclose that indicator to the NGO. The NGOs do not, therefore, know which services will be incentivized, thus minimizing the possibility that they would focus efforts on the incentivized services and ignore the non-incentivized services. 26 USAID/HAITI: SDSH PROJECT EVALUATION The evaluation team was able to review a print copy of the Brandeis University/MSH evaluation cited above, but on the grounds that the team would not distribute the document, as it is currently embargoed pending publication in a professional journal. In sum, and at the risk of oversimplifying its findings, that evaluation echoed the findings of the 2001 and 2009 assessments, which is to say that PBF had in fact improved both coverage and quality of health services delivered, with such improvements particularly associated with the utilization of services for children under 1 year of age and for pregnant women. Finding 29 The SDSH project’s application of PBF in Haiti is more technical and labor-intensive than conventional “textbook” applications of the PBF model. Discussion In a fundamental sense, PBF is meant to encourage contractors to innovate and to experiment with ways to maximize the reach and impact of their programs. Importantly, it accepts the prospect that some participants will fail to achieve their targets and will not be rewarded for their efforts. SDSH project personnel, however, leave little to chance in applying the PBF model. They provide extensive technical assistance to NGO and ZC staff responsible for data collection and reporting, they closely monitor monthly and quarterly reports for any indication of slippage in meeting project targets, and they work with project partners to quickly address the causes of such slippage. In short, the project is managed in a way that does not look to the partners as innovators but rather as partners that will not be allowed to fail. SDSH managers acknowledge that their approach to PBF is more on the order of PBF-Plus; but they also point out that given the high stakes involved—matters of life and death—they cannot afford, ethically, to apply a more conventional laissez-faire approach to the model. SDSH personnel also acknowledge that their own performance as managers of the overall project is judged by USAID in terms of their ability to consistently achieve project targets. Finding 30 The SDSH project has helped to create a large number of practitioners and advocates of the PBF model at the project’s participating NGOs and, perhaps more important, at both the departmental and central levels of the MOH. Those skills are spread thinly, however, and managers rely heavily on SDSH backstopping, monitoring, and follow-up. Discussion MOH leadership in all 10 MOH departmental offices has embraced the PBF model, and most of the MOH departmental offices are currently utilizing it. The GOH, meanwhile, represented especially by the President of the republic, has anointed PBF as the contracting mode-of-choice for the entire GOH. In an important respect, then, the SDSH project has helped create a cadre of GOH managers who will play key roles in the broader roll-out of PBF in Haiti. That said, the labor-intensive PBF model currently employed by the SDSH project is not, as a practical matter, replicable at MOH departmental or central levels. Recommendation Again, and as proposed elsewhere in this paper, USAID is urged to include provision in the follow-on project for long-term technical assistance at MOH central and departmental levels to effectively install MOH capacity to design, execute, monitor, and support (though not necessarily fund) performance￾based contracts. USAID/HAITI: SDSH PROJECT EVALUATION 27 The longer-term objective of such assistance, as discussed later in this document, will be to help the MOH create an independent capacity to contract out to various providers for the delivery of essential health services. CAPACITY BUILDING One of the objectives of this evaluation was to assess the extent to which SDSH builds the capacity of local NGOs and MOH (departmental and central levels) to deliver health care services as well as build the capacity of communities, especially women and youth, to become active partners in working to improve the health system. USAID asked the evaluation team to assess the readiness of NGOs and MOH to implement service delivery programs and the extent to which SDSH prepared local NGOs for successful graduation and the MOH was prepared to assume an eventual role as a direct recipient of USAID assistance. The SDSH project’s main goal is to increase access to and use of an integrated package of basic health services that includes MCH care, FP, HIV/AIDS, and TB. SDSH achieves its objective through its support to public and private nonprofit sectors. SDSH has two pillars:  To provide health care service delivery through the public and the private nonprofit sectors  To strengthen the capacity of the MOH to carry out its executive management and oversight function at the central and departmental levels Capacity Building for Local NGOs Finding 31 Although capacity building of local NGOs was not a central aspect of SDSH, the project helped strengthen capacity of local organizations, mainly through mentoring and technical assistance for service delivery. SDSH encouraged participating NGOs to share their experience and supported cross￾fertilization between NGOs to reinforce their capacity during their annual meeting. Discussion All NGOs the team visited—with the exception of MEDISHARE, which joined SDSH in 2010—indicated that they have received training in financial management and accounting under the SDSH predecessor project HS 2007. The evaluation team attributes the focus on mentoring and technical assistance for service delivery rather than activities to strengthen institutional development, in part, to the fact that SDSH has been working with the same pool of NGOs as its predecessor projects HS 2004 and HS 2007, whose main objectives were capacity building and organizational development of local NGOs. During its predecessor projects, the support has taken a variety of forms and utilized a range of assistance mechanisms. For instance, MSH used a bottom-up organizational approach with the training of members of the organizations to provide them with skills and knowledge to effectively manage the organization and deliver quality services and a top-down organizational approach by reinforcing the organizational structure to improve planning and management. Therefore, the service delivery organizations had already in place the basic financial management and M&E systems to successfully manage performance￾based contracts under the SDSH Project. The project has a strong focus on expanding service delivery. From January 2008 to September 2011, more than 5,000 people had been trained, with more than 90% of them receiving training in health care service delivery. That said, few training sessions were offered on management, health information systems, or communication during that period. 28 USAID/HAITI: SDSH PROJECT EVALUATION SDSH classifies participating NGOs based on their performance meeting the targets in the NGOs’ subcontracts with SDSH. The three classification categories are as follows:  High performance: Achieved more than 90% of the targets set for service delivery  Medium performance: Achieved between 80% and 90% of targets  Low performance: Achieved less than 80% of targets During the reporting period from October 2010 to March 2011, the low-performing NGOs included SAVE, FOSREF, and Clinique la Fanmi; medium-performing NGOs were International Child Care (ICC)/Grace, and SADA; and high-performing NGOs included Centres pour le Développement et la Santé, Lucelia Bontemps, Pierre Payen, the Association d’Entr’Aide des Dame Mariens, and Claire Heureuse. This classification does not take into account any managerial capacity of the NGOs. It is important to note that organizations like SAVE that are classified as low performing are also participating in a $87 million program for Haitian earthquake response and recovery. Also, FOSREF, a local organization that only receives 6% of its budget from SDSH, has a total of 900 employees, and was classified as a low-performing organization. Recommendation Data from the SDSH SDMA exercise that (which, at the time of writing, was planned for November 2011, and presumably 2012) should be carefully analyzed to determine the level of organizational capacity of each participating NGO. Finding 32 As reported above, most participating NGOs are strongly dependent on external funding and the activities funded by SDSH are not financially sustainable. Recommendation The follow-on project should help selected local NGOs look for innovative ways to increase funding, such as how to plan and organize fundraising activities, as well as explore the possibility of developing community health insurance to increase users’ financial access to health services. Finding 33 SDSH is overly involved in the routine management of service delivery points. Not enough opportunity is given to participating NGOs to manage their own service delivery points, which weakens the leadership of NGOs and hinders their path toward graduation to engage in bilateral, direct relationships with USAID. Discussion MSH closely monitors each NGO and each service delivery point’s performance in meeting its service delivery targets. If a service delivery point seems to be falling short of reaching its objective, SDSH sends a formal letter to the PPS and the NGOs, with a copy to the SDSH technical advisor. Recommendation The routine management of service delivery points should be left in the hands of participating NGOs, which have been nurtured by SDSH and its predecessor projects for more than a decade. USAID/HAITI: SDSH PROJECT EVALUATION 29 Capacity Building for MSPP at the Central and Departmental Levels Finding 34 SDSH provided support to MOH to strengthen its executive functions at both the national and departmental levels. The project contributed to the development of management tools to improve coordination and service delivery. SDSH participated on various technical committees and contributed to the development of norms, standards, and procedures. However, there was no significant transfer of knowledge to reinforce the capacity of government entities. Discussion The project supported the MSPP’s initiative in creating the National Committee for the Health Information System (CONASIS), and has participated regularly in the different work sessions of this committee. SDSH also contributed to the design of important aspects of the HMIS such as the list of program monitoring and evaluation indicators, and data collection and reporting tools. However, those efforts have not contributed to the development of a fully functional national HMIS. Moreover, the SDSH project’s reliance on its own, parallel health data reporting system represents a missed opportunity to incorporate that system into a consolidated HMIS. To strengthen the MSPP’s financial management and accounting system, the project worked to develop a new financial management manual and accounting system. SDSH organized training sessions and workshops for staff from the central and departmental directorates. The new financial management and accounting system has been initiated in the departments. MOH departmental financial and accounting staff were trained in QuickBooks. However, the manual is not yet in use at the departmental level, and not all departments are using the software, because it conflicts with the Centers for Disease Control and Prevention software used to manage the PEPFAR program. At the departmental level, there was no real reinforcement of MOH managerial capacity to sustainably maintain the delivery of SDSH services in the absence of SDSH. Project activities in the ZCs, for example, were managed by SDSH technical and financial advisors with little involvement of departmental staff. Similarly, SDSH oversaw the work of the NGOs without substantive participation of the MOH either at departmental or central levels. In consequence, even as the project is in its final year, the MOH still lacks the technical and managerial skills, both at the central and departmental levels, to adequately manage a PBF approach to health service delivery. On a more positive note, SDSH has worked successfully with the MOH Division of Family Health (DSF) to review maternal health norms, including the integration of PMTCT at the institutional and community level. The project contributed to the development of the national strategy for PMTCT and assisted the Division of Family Health (DSF) in the design of the national Cervical Cancer Prevention Program. SDSH also helped the UPE—the planning and evaluation unit within the MOH—to develop a supervision manual, and assisted UPE in the development of its Plan Organisationnel Integré. Finding 35 SDSH supported the MSPP’s efforts to decentralize planning and implementation responsibilities to the departmental level. Support for MSPP capacity building at the departmental level was provided mainly through on-site SDSH technical and financial advisors. Discussion SDSH technical and financial advisors have been appointed to each department. SDSH departmental advisors work closely with department staff to support SDSH-funded initiatives, especially in the ZCs. MSPP staff work with SDSH staff to analyze project results, conduct supervision, prepare and manage budgets, and develop strategies to improve performance and facilitate the liaison between SDSH and MSPP departmental level for the ZCs. 30 USAID/HAITI: SDSH PROJECT EVALUATION The SDSH departmental advisors also took part in other department-wide activities. The departmental advisors participated actively along with MSPP in the development of integrated departmental plans, which include activities funded by SDSH, the MSPP and other donors. SDSH contributed to the development of departmental supervisory plans and joined the Table Sectorielle, which is a coordination mechanism at the departmental level. To strengthen the ministry’s financial and accounting management systems, SDSH project financial advisors conducted an assessment of the financial and accounting system in the various departmental directorates. Based on the results of the assessment, departmental staff were trained on financial management and on finance software (QuickBooks) that was subsequently installed. The software is in use in the departments; one department director explained, however, that they received the financial software from SDSH but they are using another financial software program provided by the CDC. Finding 36 There is little integration of SDSH project management procedures at the departmental levels. This lack of integration of monitoring and management procedures at the departmental directorates (i.e., of the activities in ZCs) represents a missed opportunity to strengthen management capacity of the health department. The MOH has a limited role in the management of the performance-based contracts signed between NGOs and SDSH. Discussion The main task of the SDSH departmental advisors is to work with departmental staff to support SDSH￾funded initiatives. In all the departments visited, the activities in the ZCs were coordinated by the SDSH advisors with various level of integration of the departmental staff. The administrator of one department pointed out that MSPP staff was not involved in SDSH personnel management of the ZCs. The SDSH advisor recruits new employees without consulting MSPP staff to determine the category of employees needed. This situation causes frustration from the MSPP side because they feel sometimes that their needs are different. Also, PPS employees who are paid by SDSH do not always integrate their annual leave plans with PPS directors or other PPS staff. As discussed earlier in this report, the SDSH central office uses the integrated departmental plans, previous performance, and catchment area population to develop the terms of reference and targets for NGOs. The terms of reference are sent to NGOs. Upon the NGO request, the targets might be discussed and lowered. The NGOs prepare an action plan that is discussed with SDSH, and upon agreement a copy is sent to the SDSH departmental advisors. The MOH receives a copy of that contract, but is not really involved in the negotiation process. Some NGOs send performance reports to the MOH and to SDSH. Others send their report directly to the SDSH central office with a copy to the SDSH departmental advisors. The fact that the NGOs negotiate their contracts directly with the SDSH central office undermines the leadership role of the MOH and represents a missed opportunity to strengthen the capacity of the MOH toward the long-term goal, which is to engage the Government of Haiti in performance-based contracting with private sector partners. Recommendation The follow-on project should make provision for the continuation of technical and financial advisors at the departmental levels of the MOH—where those advisors would seek to more fully engage MOH staff in the management of ZC activities supported by the project. The new project should also provide technical assistance at the MOH central level, with a special focus on efforts to improve financial systems and procedures needed to support the health ministry’s adoption of the performance-based contracting model. USAID/HAITI: SDSH PROJECT EVALUATION 31 Capacity Building for Local Communities Finding 37 A key component of SDSH is the community network made of community health workers. Several thousand community health workers are being paid by SDSH through local NGOs or ZCs. This network of CHWs helps improve access to basic health care services to communities. Discussion The use of health services is limited by geographical access. To increase coverage, SDSH, through its public and private sector partners, expanded its integrated community-based service delivery and prevention activities. Several thousand of CHWs were trained and are being paid under the project. CHWs are chosen by community members to provide basic health and medical care to their communities. Most of them are lay members of the community and have had no formal training in health prior to their selection. They are selected by community leaders for their commitment to their community and their leadership. CHWs are trained to provide basic health education and essential prevention services such as growth monitoring, immunization, FP services during gathering posts and home visits, and referrals to health service delivery points. In addition to these tasks, the CHWs help various community clubs and support groups such as mothers clubs, satisfied FP users, and HIV support groups. They also support the training of traditional birth attendants (TBAs). The evaluation team interviewed 11 CHWs working at five PPSs in three different departments. All of them indicated that they have received training to do their job. Most believed that the SDSH project has improved the quality of their work because they have more material available and training to perform their duties. All believed that the service that they have delivered to the population was very good. The CHWs demanded better working conditions, including more competitive salaries. In response to the cholera outbreak, which claimed the lives of thousands across Haiti, the SDSH project through its partners were able to rapidly mobilize its community health workers who already had training in hygiene and sanitation; SDSH trained them in cholera prevention and treatment to reach out to communities. Finding 38 Another key activity of SDSH is the training of and provision of support for TBAs. Several thousand of these attendants have been trained in order to improve maternal health. Discussion According to the 2005-06 Haiti Demographic and Health Survey (Cayemittes et al., 2007) approximately 76% of women give birth at home and 65% of deliveries are assisted by TBAs. The maternal mortality ratio for Haiti is 630 per 100,000 live births (Cayemittes et al., 2007). SDSH supported the MSPP’s short-term strategy to improve pregnancy and delivery outcomes by strengthening community-based maternal health interventions through the training and support to the TBAs. The project’s long-term strategy is to strengthen and expand institution-based services. TBAs are generally women who are respected by their communities. Most are illiterate and have acquired their competencies by working with other older attendants. The TBA works independently and receives some compensation for services to assist women during delivery and immediate postpartum. Through its Relance des Matrones strategy, SDSH helped adapt MSPP norms to officially recognize the TBA as a community health agent to promote safe maternity within their communities. As noted, several thousand TBAs have been trained and certified through the SDSH project. They received home delivery kits and were expected to improve home service delivery and identify at-risk delivery with timely referral for obstetrical care. Some health care centers provided incentives to the 32 USAID/HAITI: SDSH PROJECT EVALUATION TBAs for the referral. Those incentives consisted of transportation fees and a cash compensation varying from $13 to $19. Those incentives were covered mainly through the World Health Organization–United Nations Population Fund (WHO-UNFPA) funded project Soins Obstetricaux Gratuits. Finding 39 SDSH supported community activities through various peer education programs like mothers’ clubs. Discussion As part of its behavior change/community mobilization (BCC/CM) strategy, SDSH helped develop guidelines for training of mothers’ clubs. All the PPSs visited by the team supported mothers clubs. The CHWs trained mothers on the most common health issues, such as maternal and child care, HIV/AIDS, STDs, and hygiene/sanitation. SADA, one of the participating NGOs, has more than 5,000 mothers who have been trained and “graduated.” Those mothers are called “Maman consequent” and created a network of community volunteers. This volunteer network of mothers was mobilized during the cholera epidemic and was trained to prevent the disease in their neighborhoods. SADA is currently looking for funds for water purification and this network of volunteers will help. ICC/Grace indicated that 100 to 150 mothers were trained by the CHWs last year. At ICC/Grace Hospital, the mothers become volunteers (collaborateurs volontaires) upon graduation and support the CHWs in organizing gathering health posts to provide essential health services to the community. Recommendation The follow-on project should maintain and expand the community network that can be considered as the backbone of success of this project. Finding 40 Through FOSREF, SDSH developed the youth component with a vision that goes further than sexual and reproductive health service delivery. A total of 10 sites have been qualified as youth-friendly, where 242 peer educators provided quality sexual and reproductive health services in 2010. Discussion Using an adapted SDMA protocol and pathfinder guide for assessing youth-friendly clinics and health institutions, FOSREF identified PPSs that were ready to offer youth-friendly sexual and reproductive health services. The goal was to provide quality services in sexual and reproductive health, including counseling, FP methods, and HIV testing and referrals for the continuum of care available throughout the SDSH HIV program. Unfortunately, the departure of FOSREF as a project subcontractor limited the ability of the SDSH project to successfully implement efforts to expand and intensify the involvement of youth. Recommendation The follow-on project should consider expanding the training of youth peer educators and intensify the involvement of youth in health care service delivery. Finding 41 The project has not reached deeply into local communities by the creation of local health task forces. No local health task force committees are currently in place. USAID/HAITI: SDSH PROJECT EVALUATION 33 Discussion An as-yet unrealized aim of the SDSH project is to educate and mobilize communities, via local health task forces, to create more demand for services. The local health task forces would bring elective officials, community members, and leaders as well as health care service providers together. Recommendation The follow-on project should retain focus on establishing local health task forces as a component of the health service delivery model discussed earlier in this paper. Quality of Services As indicated earlier, the service delivery component of the project was a success; most of the targets set for service delivery were met. To gain a better understanding of quality of care issues, the team interviewed 21 clients, including 6 FP clients, 4 women in prenatal clinics, and 4 parents accompanying their child to the clinic. Of the interviewees, 2 were men and 19 were women. All clients reported satisfaction with the care they received on the day they were interviewed. They indicated that the reason behind that success was the competence of health care providers. Around 40% of clients interviewed described an improvement in the quality of care they received over the past four years. Some patients complained about the long waiting time at the clinic, which varies from five to eight hours. At one PPS, clients were being told they had to have an HIV test in order to access health services. The team interviewed 13 health care providers, including six auxiliary nurses, five nurses, and two physicians working at seven PPSs. All health care providers had pre- and in-service training for health service delivery. Of the 13, 8 had participated in at least one training session on health care service delivery in the past 12 months. All of them described as “good” or “very good” the quality of care they provided to the patients and linked that success to better training and availability of materials. Of the 13 health care providers interviewed, 5 reported that they would like to have a better salary. LESSONS LEARNED Two key lessons emerged from this evaluation. The first is a general lesson that relates to donor and partner work in the Haitian health sector; the second relates to the SDSH project itself. 1. Work in this sector must be planned and executed in the full knowledge that success, or the beginnings of success, will take a long time, and that progress will suffer unexpected setbacks along the way. USAID and other donors have been heavily involved in the Haitian health sector for decades. That involvement, however, was neither consistent in its focus (i.e., sometimes focusing on the public sector, more frequently on the NGO sector), nor reliable in its consistency (e.g., suspensions of assistance during political instability or civil unrest to express dissatisfaction with electoral procedures). More recently, assistance suffered the disruptions attendant to hurricanes, the earthquake, and the cholera epidemic. In brief, nothing seems to happen in a linear, uninterrupted manner in Haiti, and donors should not expect that it will in the future. In this context, the SDSH project is but a step in a long process. 2. When viewed from that longer perspective, two things can be said about the current SDSH effort. The first is that it is succeeding in realizing its primary objective: to expand the availability of essential health services to underserved elements of the population. The second, however, is that the SDSH project is not playing a particularly strategic role in the health sector, meaning that it is not an activity that has the potential to compress or foreshorten a timeframe that does not appear to have any end-point in sight. The project is applying some effective tools to ensure that health services are expanded, monitored, and well managed, and includes some valuable capacity-building 34 USAID/HAITI: SDSH PROJECT EVALUATION components. But taken together, they do not create the necessary conditions for a gradual transition of responsibility for health care delivery to the GOH. Specifically, the service delivery model being implemented by SDSH and its predecessor projects is not sustainable in the absence of donor support. Notwithstanding the project’s capacity-building objectives, SDSH has created a significant degree of dependency on the part of participating NGOs and GOH partners who would be forced to dismantle large segments of their community-based delivery programs in the absence of project support. That dependency is not limited to partners’ reliance on SDSH for financial, equipment, training, and operational support, but also for managerial oversight, and in particular, for SDSH stewardship of the PBF model used to support service delivery activities. The next section on needs assessment offers suggestions on ways to improve the strategic impact of a follow-on project. Other lessons:  The data collection and management system utilized by SDSH is focused almost entirely on performance targets, and does not provide meaningful insight into NGOs’ quality of care, ethical practices, fundraising initiatives, or institutional maturity. The PMP does include some measures (FE.1-FE.7) that touch on managerial improvements in MOH departments; but does not include similar indices that might capture managerial growth at the NGOs—an explicit objective of the project. In the final months of the project, SDSH staff should draw on their years of experience with the PMP and institutional strengthening, and attempt to develop meaningful measures of their partners’ institutional capacity.  The GOH health system is utilizing multiple, uncoordinated data collection systems. At the department level, for example, it is not uncommon to encounter the SDSH data collection and monitoring system, a PEPFAR system, and the health ministry’s regular service statistics system. No one appears to be responsible for efforts to consolidate the data generated by those systems into one data set, or to package the data in ways useful to program administrators. The follow-on project should work with USAID and the CDC to help harmonize these disparate systems.  A functioning patient referral system does not exist. To be sure, CHWs refer patients to PPSs, and PPSs refer patients to more specialized facilities. But once the referred patients leave sight of the referring health worker, they are effectively lost to any follow-up. Health workers rarely follow up with the referees and receiving institutions do not provide patient feedback to referring institutions or health workers. In practice, most persons in need of more complicated or extensive services understand that the PPS is not equipped to deal with their problem, and they self-refer to higher￾level health facilities.  CHWs associated with the project are, as reported by virtually every person consulted on the evaluation, angry. Actual turnover within this personnel cadre has been relatively low (most of the turnover is in the professional and technical categories), but the mood of the CHWs is a long-term risk to the success of the community-based model. Their discontent is not always related to pay, though they are keenly aware of the better pay offered by various NGOs; they also reportedly resent the harsh working conditions and transportation difficulties they are facing, especially in the ZCs. Approaching this issue head-on is tricky, as it could raise unrealistic expectations that could further complicate project management. During the final months of the project, however, SDSH should undertake some discrete studies of CHW attitudes and needs, such as through focus groups, and be prepared to offer some practical, affordable recommendations to improve the current situation. USAID/HAITI: SDSH PROJECT EVALUATION 35  A more effective strategy is needed to expand the availability of LAPM FP services in rural parts of the country. Training targets established the PMP are generally being met; but those numbers are silent with regard to the geographic distribution of service delivery personnel being trained. NEEDS ASSESSMENT This section of the paper proposes changes that should be considered for implementation in the short term—i.e., over the remaining life of the SDSH project—and for the longer term, i.e., changes that USAID could incorporate into its planning for a follow-on project. 1. Short Term:  As part of its just-beginning SDMA exercise, SDSH personnel should assess the readiness of its NGO and ZC partners to implement a more methodical response to the cholera epidemic, and should provide the results of that assessment to USAID/Haiti. The USAID Mission could review those findings as it considers whether to include an explicit cholera prevention and treatment component in the follow-on project.  Also as part of its SDMA review, SDSH should undertake institutional assessments of the participating NGOs and share with USAID/Haiti SDSH recommendations regarding the respective readiness, or lack thereof, of the NGOs to function successfully as partners to USAID and/or other donor agencies.  SDSH should use its participation on the HMIS technical working (at the MOH) to help finalize the structure of the MOH HMIS. SDSH technical advisors should concurrently seek to prepare MOH department-level staff to effectively utilize the HMIS by promoting closer integration of SDSH project data and data collected/reported from other sources.  SDSH should undertake informal discussions with the leaders of prominent NGOs in Haiti, with a view toward establishing understandings regarding their respective human resources policies, including salary scales. Given the nature of SDSH’s partnership arrangements with the MOH, it is understood that SDSH salary scales must remain within GOH-approved brackets. The other NGOs might be urged, however, to factor in the overall human resources environment in Haiti in their recruitment activities and to take steps to avoid unnecessary disruptions to SDSH activities in their recruitment practices. 2. Long Term: The follow-on project to SDSH should be structured to (1) build on the achievements of the existing activity, (2) eliminate extraneous activities in order to maintain a lean and focused activity, and (3) begin a long-term transition process from a donor-managed health service delivery program to a program managed by Haitian institutions. The GOH, represented by the Ministry of Health, should be prepared to play the leading managerial and technical role in the program that emerges from that transition process. Specific elements of a follow-on project should include the following: Build on the foundation: 1. PBF appears to have gained political traction in Haiti, but it is still being embraced more for its conceptual appeal and less so as an operational tool of the government. The experience gained under the SDSH project can be extremely valuable in helping the MOH incorporate PBF procedures into practice. As recommended elsewhere in this paper, the follow-on project should include a major technical assistance component designed to develop the professional capacity of MOH central and departmental managers to successfully implement the PBF model. 2. NGOs will continue to serve as key elements in the country’s health delivery network. Their many years of work in the country and their by-now extensive experience with PBF will continue to make 36 USAID/HAITI: SDSH PROJECT EVALUATION them natural partners of the MOH, especially as the departments step up their own role as overall coordinators of health care agencies. The follow-on project will need to maintain the participation of the most of the NGOs included under the SDSH project, and will have to attract additional NGO partners as well. An approach to the management of the NGOs’ participation is discussed below. 3. The role of the health departments as managers and overseers of department-level health services will continue to grow in the future, but their ability to successfully undertake that role is still constrained by a lack of professional depth and lack of management experience. The follow-on project should be designed to collaborate with and complement the department-level efforts of other donors, including CIDA (which is considering a follow-on to the PADESS project), and a UNICEF-supported activity at the department level. The SDSH project’s placement of technical and financial advisors at the department level has shown itself to be a successful mechanism to promote good project performance by NGO and MOH partners, reinforce the leadership role of the department, and ensure responsible use of project resources. Such presence should be retained in the follow-on project. 4. Most important, the community-based model developed incrementally under the HS 2004, HS 2007, and SDSH projects should be retained and expanded under the follow-on project. Eliminate marginally useful activities: 1. Neither the Strategic Partnerships nor the GUC initiative is contributing substantively to the attainment of current or future project goals, and should be dropped from a follow-on activity. 2. Other donors will continue to support the development of an HMIS and the financial management manual/system, neither of which will proceed much beyond their current status unless the MOH acts far more definitively in support of those two tools. Begin the transition to host-country management and ownership: The SDSH project has, as noted above, created the foundation for future success. It has developed a successful model for health services delivery; helped create a culture of cooperative relations between the NGO and governmental sectors, while enhancing GOH primacy in that relationship; and has helped prepare the first generation of MOH personnel having hands-on experience with PBF and the contracts supported by that approach. The simple continuation of the current project, however, would only freeze that model in place and would not generate any further movement toward real country ownership of the country’s health delivery model/system. The follow-on project should be designed to begin the transition process from management of large segments of the health delivery system by a U.S. contractor to management of that system by the MOH. The ultimate objective of the project would be to install, on a pilot basis, the institutional capacity at the MOH and a selected number of MOH health departments, to contract out to NGOs for the delivery of basic health services. (See the World Bank publication Performance-Based Contracting for Health Services in Developing Countries, A Toolkit, 2008: http://siteresources.worldbank.org/INTHSD/Resources/topics/415176- 1216235459918/ContractingEbook.pdf.) A USAID request for proposals for the new activity should invite prospective bidders to describe how they would support that transition process. The essential elements of that process would include: 1. Development of a schedule for transitioning the skills and implementation responsibilities for performance-based contracting from the U.S. contractor to the MOH, and specifically to a select number of health departments. During the initial years of the activity, the U.S. contractor would manage the service delivery program much as it is managed under the SDSH project. But over the five-year life of the project, the contractor would gradually shift those responsibilities to no less than USAID/HAITI: SDSH PROJECT EVALUATION 37 two health departments and would create an oversight capacity for performance based financing at the central level of the MOH. 2. Inclusion of an extensive and rigorous training regime for MOH central- and department-level managers. The training would include workshops, study tours for MOH leaders to observe successful examples of governmental contracting-out programs in other countries, and on-site tutorials/follow-up for staff from both levels. The MOH might decide to establish a formal certification program for trained and qualified contract managers. 3. An end-of-project goal of no less than three performance-based contracts executed by as many MOH departments and participating NGOs. USAID should not expect that the MOH will have the financial resources to fund these initial contracts. The U.S. contractor would therefore be expected to fund the pilot contracts and to exercise fiduciary oversight for contract costs, even as the MOH departments take on responsibility for development of performance targets, negotiation of action plans, execution of the contracts, and oversight/monitoring/follow-up with the participating NGOs. As noted earlier, it will be important to maintain a long-term perspective for health assistance activities in Haiti. Thus the dual objectives of the follow-on project will be to (1) continue and expand the community health model currently in place and (2) begin the process of transitioning management and ownership of that model to the Government of Haiti. A complete transition would likely require at least two project cycles, or up to 10 years. 38 USAID/HAITI: SDSH PROJECT EVALUATION USAID/HAITI: SDSH PROJECT EVALUATION 39 ANNEX A: SCOPE OF WORK Global Health Technical Assistance Project GH Tech Contract No. GHS-I-00-07-0006-00 SCOPE OF WORK (9-19-11) I. TITLE Activity: USAID/Haiti: SDSH Project Evaluation Contract: Global Health Technical Assistance Project (GH Tech), Task Order No. 01 II. PERFORMANCE PERIOD End of September – end of November 2011 III. FUNDING SOURCE USAID/Haiti. IV. PURPOSE The purpose of this Scope of Work is to conduct a formative evaluation of the Santé pour le Développement et la Stabilité d’Haïti (SDSH) project implemented by Management Sciences for Health (MSH). This evaluation will focus on project implementation with the aim to improve project performance during the remaining months of the project. In addition, it will help guide future programming. The primary stakeholders of this evaluation include USAID as whole, team members of other USAID health projects, the Ministry of Health (MOH), and MSH. V. BACKGROUND Approximately 40% of Haitians have no access to basic primary health care. Health indicators reveal that Haiti’s health system is weak. Under-five mortality is about 12.5%, which is the highest in the Western Hemisphere. Haiti also has the highest maternal mortality rate in the region with about 523 deaths per 100,000 live births. Haiti’s poor health profile is a result of the lack of access to quality health care services and to potable water, as well as chronic food insecurity. Access to quality health care is challenging because of poor health infrastructure and the disruption of services due to insecurity, violence, and disasters, such as the earthquake of January 12, 2010. This combination of factors has led to the lack of essential equipment and supplies, inadequately trained staff, and the poor management of health facilities, especially in the public sector. Although communities and civil society, mainly NGOs, are 40 USAID/HAITI: SDSH PROJECT EVALUATION engaged in addressing health issues, health systems need to be strengthened both in the public and nonpublic health care delivery sectors. The SDSH project was awarded by USAID on August 7, 2007 to Management Sciences for Health (MSH) for a three-year time period and was extended to September 30, 2012. This project targets about 50% of the Haitian population in all 10 departments. Its main goal is to increase target beneficiaries’ access to and use of an integrated package of basic health services that includes:  Maternal and child health care  Nutrition  Family planning  Prevention and control of infectious diseases, including tuberculosis and HIV & AIDS. The SDSH project supports both public and private (nonprofit sector) health care delivery. Project activities are implemented by 26 local nongovernmental organizations (NGOs) in 72 health facilities and by the Ministry of Health (Ministère de la Santé Publique et de la Population [MSPP] or MOH) in 31 targeted geographic areas known as zones ciblées (ZCs), which include 75 health facilities. The project aims to strengthen the ability of both MSPP and its NGO partners to deliver health care services. In terms of performance, the Annual Progress Report: October 1, 2009 – September 30, 2010, indicates that “despite the January 12 earthquake, the SDSH Project achieved or surpassed almost all of its project indicators. Overall, during this project year, results show significant improvements.” Nevertheless, tuberculosis-related indicators were the lowest, raising some concerns about the implementation of tuberculosis-related activities. SDSH performance indicators provide quantitative data on various health care services, but are limited in informing about the quality of such services. In the SDSH project, MSH uses performance-based financing in supporting its NGO partners. It negotiates with them to definite subcontract indicators and performance targets. Partners report monitoring data on a regular basis and participate in periodic evaluation of the services provided. Reported outcomes are validated by an independent group before final payments are made by MSH. Several assumptions and critical success factors were specified while designing the SDSH project. Two are particularly relevant in the current context: 1. The Haitian government and the MSPP leadership remains the same and relatively stable. The new Haitian government expected for May 2011 will challenge this assumption. The impact of this change, especially potential changes at the level of the MSPP, should be examined for its possible effect on the SDSH project. 2. No major social emergencies or natural disasters occur in Haiti that force a shift of project programming or resources. The earthquake of January 12, 2010 changed the dynamics of health-related development assistance by dramatically increasing the number of international development actors involved in supporting the delivery of health care services. As a result, the existing network of partners was disrupted, sidelining many people and organizations with intimate knowledge of the Haitian context, culture, approaches, key local players, and communication and decision-making processes. The response of SDHS to these changing dynamics among health-related partners should be studied, as well as the resulting changes, if any, in its role and place within the current network of health-related development actors. VI. EVALUATION QUESTIONS In order to produce information that is useful for its stakeholders, this evaluation will answer the following questions: USAID/HAITI: SDSH PROJECT EVALUATION 41 1. To what extent did SDSH achieve expected results, as specified in the project’s Performance Monitoring Plan? (Focus: Project effectiveness) 2. What factors positively or negatively affected SDSH’s ability to deliver the services required to achieve expected results? While answering this question, the evaluation should examine how SDSH addressed these factors to optimize project performance. (Focus: Project process/Implementation). Factors to take into consideration may include, but are not limited to:  Project structure  Project management (e.g. level of participation of partners in project processes such as strategic planning, budgeting, implementation, monitoring and evaluation)  Project staffing  USAID management  Partnership model  Selection process of sub-grantees  Performance-based financing  Funding mechanism, i.e. channeling funds through the Ministry of Health 3. To what extent did SDSH build the capacity of local NGOs and of the Ministry of Health (departmental and central levels) to deliver health care services, as well as the capacity of communities, especially women and youth, to become active partners in working to improve the health system? (Focus: Capacity-Building) 4. To what extent did MSH’s management of sub-grantees under SDSH promote the optimal use of project resources? (e.g. minimizing duplication and/or redundancy of effort, addressing gaps and preventing the use of multiple funding sources for the same activity) (Focus: Project efficiency) 5. While implementing its activities, what did SDSH learn that could optimize the effectiveness of the current project, as well as that of similar future projects? While answering this question, the evaluation should identify and discuss SDSH approaches and activities that should be continued or replicated and approaches and activities that should be changed. (Focus: Lessons learned) 6. What issues and gaps still need to be addressed in order to improve the access, use and quality of integrated health services? (Focus: Needs assessment) VII. SUGGESTED METHODOLOGY The evaluation team leader will prepare an evaluation plan during the Team Planning Meeting for USAID’s review. This methodology will specify the research design, as well as methods and procedures for sampling, data collection and data analysis. Efforts should be made to use multiple data collection methods and data sources to allow for triangulation of data and cross-validation of results. In particular, the team should use the following methodological approaches: 1. Review of documents: All team members should review relevant background documents in preparation for the assignment. Documents will be provided to GH Tech by USAID/Haiti well in advance of the assignment. 2. Team planning meeting: An initial two-day team planning meeting will be held to agree on the process and methodology needed to achieve the objectives and desired outcomes of the assignment. This time will be used to clarify team roles and responsibilities, deliverables, development of tools and approach to the evaluation, and refinement of agenda. In the TPM the team will:  Share background, experience, and expectations for the assignment; 42 USAID/HAITI: SDSH PROJECT EVALUATION  Formulate a common understanding of the assignment, clarifying team members’ roles and responsibilities;  Agree on the objectives and desired outcomes of the assignment;  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion;  Develop data collection methods, instruments, tools and guidelines, and methodology and Develop an assessment timeline and strategy for achieving deliverables;  Develop a draft report outline for Mission review and approval. VIII. EVALUATION TEAM COMPOSITION The evaluation team will be composed of three full-time consultants (a team leader, an assistant team leader, and a field coordinator) and three local, part-time, data collectors. Required qualifications for the full-time consultants include:  Advanced degree (Master’s or above) or equivalent in Public Health or in a field related to an area of expertise required for evaluations (e.g. quantitative and qualitative research, monitoring & evaluation);  Minimum of five years’ experience in the monitoring and evaluation of development activities;  Demonstrated experience with and understanding of monitoring and evaluation of HIV/AIDS prevention, family planning or child survival programs;  Excellent oral and written communication skills in English, as well as fluency in French for at least one of the consultants to be able to develop instruments and conduct interviews in French;  Experience interacting with developing country governments, international organizations, other bilateral donors, civil society representatives, and senior level government officials;  Ability to work with diverse international teams and excellent interpersonal skills. In particular, the Team Leader will have demonstrated experience in monitoring and evaluating public health projects, preferably in one of the above-mentioned areas. He/she will be responsible for planning the evaluation, coordinating the implementation of the evaluation, assigning evaluation responsibilities and tasks, and authoring the report, in particular findings, conclusions and recommendations. The Assistant Team Leader and Field Coordinator will have combined expertise that will best complement the Team Leader’s profile to ensure that all the areas of expertise required for the evaluation are effectively covered. One of the three full-time consultants should be local, mainly for two reasons: a) because of local consultants’ knowledge of the Haitian context and culture, and b) to further build local capacity in evaluation. The Team Leader will have significant knowledge in one or more of the following: a) Reproductive, maternal and child health, b) infectious diseases (HIV/AIDS and tuberculosis) and c) institutional capacity-building. Finally, the three data collectors will assist the consultants with data collection activities such as conducting interviews. They should be fluent in French and Creole and have demonstrated experience in conducting interviews, preferably within the area of public health. USAID/HAITI: SDSH PROJECT EVALUATION 43 IX. SCHEDULE AND LOGISTICS It is estimated that the evaluation team will require a total of 7 weeks to plan and implement the evaluation and to write the report. Evaluation planning should take 2 weeks, while data collection and analysis should be completed in about three weeks. Two additional weeks will be allowed to draft and finalize the evaluation report. All team members are authorized to work six days a week while in country. GH Tech will arrange logistics for international consultant’s travel to Haiti. USAID/Haiti will provide basic logistics (clearances in liaison with the GOH and USAID partners, lodging recommendations, etc.) and some administrative support for the team. GH Tech will arrange and pay for lodging and workspace and equipment as needed. The Evaluation Team’s primary points of contact within USAID/Haiti will be Stéphane Morisseau, Webert José and Réginalde Massé. Table 1. Proposed Schedule Task Number of working days Document review/prep work 5 International consultants travel to Haiti 1 Team planning meeting, including meeting with relevant USAID staff; preparation of evaluation and logistics plan, to be submitted to USAID for review/approval; hiring of data collectors and driver(s) 4 Data collection and analysis, including relevant site visits 20 Briefing on key findings with USAID/Haiti 1 First draft report development and submission to USAID/Haiti 6 International consultants depart from Haiti 1 USAID/Haiti reviews first draft, provides comments to GH Tech/team leader (within ten business days) - Finalize report based on USAID/Haiti comments, completed remotely 3/2 TOTAL 41/40 X. DELIVERABLES The team leader will submit the following deliverables to USAID/Haiti: 1. An evaluation plan that will include the overall evaluation design, methods for sampling, data collection and analysis, and data collection instruments. It will also include a logistics plan. The evaluation plan will be prepared during the TPM and approved by USAID/Haiti prior to implementation. 2. Summary key evaluation findings to be presented during a debriefing to USAID/Haiti Mission staff. 3. A draft report to be submitted to the USAID/Haiti Mission for review and feedback one week after the end of data collection and analysis, and prior to the team’s departure from Haiti. The draft report will include an analysis of the replicability, feasibility, and sustainability of existing and proposed program models. It will also discuss opportunities to scale up project activities. 44 USAID/HAITI: SDSH PROJECT EVALUATION USAID/Haiti will review the draft report and provide comments to GH Tech and the Team Leader within 10 working days of receiving the draft report. 4. The Team Leader will submit the final report within 3 working days of receiving feedback from USAID/Haiti. The final report should integrate USAID/Haiti’s comments, and contain an executive summary, evaluation context, brief project description including approach, objectives and activities, evaluation methodology, evaluation findings. Based on evaluation findings, the consultant will present results achieved to date, draw conclusions and document lessons learned, and make recommendations towards achieving planned results in the remaining period of project implementation, as well as in similar future projects. Details about writing an evaluation report is available in the USAID publication Performance Monitoring and Evaluation TIPS: Constructing an Evaluation Report available at the following website: http://www.usaid.gov/policy/evalweb/documents/TIPS-ConstructinganEvaluationReport.pdf Once USAID/Haiti signs off on the final report, GH Tech will have the report professionally edited and formatted. GH Tech will require up to 30 business days to complete editing and formatting. Once available, USAID/Haiti requests both an electronic version of the final report (Microsoft Word 2003 format) and 5 hard copies of the report. The report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) (http://dec.usaid.gov) and the GH Tech project web site (www.ghtechproject.com). XI. RELATIONSHIPS AND RESPONSIBILITIES The Evaluation Team will work under the general guidance of the Office of Health, USAID/Haiti with Stéphane Morrisseau and Réginalde Massé as main points of contact. GH Tech will undertake the following:  Recruit and hire consultants.  Make logistical arrangements for the consultant, including travel and transportation, country travel clearance, lodging, and communications.  Provide feedback to all deliverables included in the SOW. USAID/Haiti will provide overall technical leadership and direction for the evaluation team throughout the assignment and will be responsible for the following: Prior to in-country work  Consultant Conflict of Interest. To avoid conflicts of interest (COI) or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding any potential COI.  Background Documents. Identify and organize background materials for the Evaluation Team and provide them to GH Tech as early as possible before the team begins work.  Key Informant and Site Visit Preparations. Provide a list of key informants and site visit locations.  Lodging and Travel. Taking into consideration safety and security issues, provide recommendations for lodging and means of transportation for in-country travel (e.g. car rental companies). USAID/HAITI: SDSH PROJECT EVALUATION 45 During in-country work USAID/Haiti will undertake the following while the team is in country:  Mission Point of Contact. Ensure availability of the Mission point of contact person to provide technical leadership and direction for the Evaluation Team’s work.  Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space).  Meeting Arrangements. While consultants typically will arrange meetings for contacts outside the Mission, support the team in coordinating meetings with stakeholders.  Formal and Official Meetings. Arrange key appointments with national and local government officials and accompany the team/consultant on these introductory interviews (especially important in high￾level meetings).  Other Meetings. If appropriate, assist in identifying and helping to set up meetings with local professionals relevant to the assignment.  Facilitate Contacts with Partners. Introduce the team to local government officials and other stakeholders, and where applicable and appropriate, prepare and send out a letter to introduce the Evaluation Team Leader. Following in-country work USAID/Haiti will undertake the following once the in-country work is completed:  Timely reviews: Review and provide timely feedback and approval for draft and final reports. XII. USAID/HAITI CONTACT PERSON Stephane Morisseau DDS, MPH Public Health Specialist USAID/Haiti XIII. REFERENCES Relevant project documents to be provided by the Mission in advance of the assignment. 46 USAID/HAITI: SDSH PROJECT EVALUATION USAID/HAITI: SDSH PROJECT EVALUATION 47 ANNEX B: EVALUATION DESIGN AND METHODOLOGY SANTE POUR LE DEVELOPPEMENT ET LA STABILITE D’HAÏTI (SDSH) PROJECT An evaluation team (team leader, deputy team leader, field coordinator, and three data collectors) recruited by the GH Tech Project met with USAID representatives at the U.S. Embassy on October 10, 2011. At that meeting, the evaluation team and agency staff reviewed, and clarified, as needed, the scope of work for the evaluation; examined and refined a draft outline for an eventual evaluation report (Attachment A); and discussed USAID expectations for the assignment. Team members subsequently (October 11–13) met with staff of the SDSH project to discuss the objectives of the evaluation; to further familiarize themselves with the project; to identify key themes, issues, successes, and challenges that might not have been fully discussed in project documents reviewed by the team before beginning the assessment in Haiti; and to develop a methodology for the assignment. That methodology is summarized below: Document review The team requested project documents beyond those that had been provided earlier to the team, including internal evaluations, subcontractor scopes of work, reporting formats and guides for SDSH assessments of participating NGOs and ZCs, complete lists of service points, and internal assessments of NGO performance. Interviews The team conducted individual and group interviews of key project staff to gain a more complete understanding of the project’s achievements and challenges, with special attention to the project team’s goals for the remaining year of project activities. The team also worked with SDSH staff to develop a fieldwork plan and schedule designed to capture information needed to prepare a report that addressed USAID requirements. Fieldwork The team will spend the second, third, and part of the fourth week of the five-week assignment conducting interviews and observing project operations in four departments, including visits in those areas to at least seven NGOs2 and their service networks, the Directions Departementale and Directions Sanitaire in the departments, project operations in three ZCs in Gonaives and Nippes, and one CBO participating in the recently-launched GUC initiative. Fieldwork will include visits to “corridor” sites in St. Marc and Cul-de-Sac. The team will seek to maximize its opportunity to visit service delivery points and to meet with health care delivery personnel, including clinicians and community health workers, and with clients and patients at service delivery points. Interview guides have been prepared to facilitate the collection of qualitative data from health system clients and community health workers, and quantitative data from persons who generate and process project data at service delivery sites and administrative offices. Upon return to Port-au-Prince, the team will meet with staff of the MOH and the MOH Planning Unit, with project partner organizations, and with SDSH staff to discuss and validate key findings of the fieldwork phase of the evaluation.                                                              2 NGOs were selected by the evaluation team to include a cross-section of NGOs identified in SDSH documents as high, medium, and low performers, with such characterization determined by the NGO’s relative success in achieving its contracted targets. 48 USAID/HAITI: SDSH PROJECT EVALUATION Information gathering during the fieldwork phase of the assignment will be guided by the questions noted in, but not limited to, those shown in Attachment “A.” The research design by which those questions will be explored is shown in Attachment “B.” Jerry Bowers Rachelle Cassagnol Williams Ojo USAID/HAITI: SDSH PROJECT EVALUATION 49 DRAFT REPORT OUTLINE Acknowledgments Acronyms and Abbreviations I. Introduction  Description of the Project  The Evaluation Purpose and Methodology II. Findings and Recommendations A. Project Monitoring B. Project Implementation  Project Staffing & Structure  Project Management  Project Performance  USAID Management  Partnership Model C. Capacity Building D. Lessons Learned E. Needs Assessment Annexes  SOW  Evaluation Design and Methodology  List of Persons Interviewed  List of Documents Reviewed 50 USAID/HAITI: SDSH PROJECT EVALUATION Table 2. SDSH Evaluation Design and Methodology Evaluation question Type of analysis conducted Data sources and methods used Type and size of sample Limitations 1. Project Monitoring: How effective/robust is the project’s performance monitoring system? i.) As a tool that the project uses to monitor its own performance ii.) As a tool to help manage/support performance based agreements between the project and participating NGOs and the MOH Validation and review of data quality and report content of the SDSH project based on expert opinion Project reports, annual planning, reports, performance monitoring plan, and results sheets SDSH project staff (all) in PAP; interview with Health delivery, financing and Performance monitoring team Possible travel limitations due to weather, road conditions; availability of key informants Assessment of effectiveness and quality of performance monitoring system Interviews; service delivery, NGO and dept., verification of supervisory reports, HIS data, SOP reports, project training records Verification of service statistics, reports to SDSH project, HIS guide manual, data quality assessment at department and NGO Verification of health systems strengthening and quality of performance based finance approach Use of simple qualitative/ quantitative tool guide to validate effectiveness of health systems and services Interview; NGO departmental and MOH staff Assessment of effectiveness of HMIS and financial information system and use of data for budget and planning NGO, departmental, MOH, and CBO staff in four departments, including four ZCs USAID/HAITI: SDSH PROJECT EVALUATION 51 Evaluation question Type of analysis conducted Data sources and methods used Type and size of sample Limitations 2. Project Implementation: What factors positively or negatively affected SDSH’s ability to deliver the services required to achieve expected results? While answering this question, the evaluation should examine how SDSH addressed these factors to optimize project performance. Factors to take into consideration may include: i.) Project structure ii.) Project management (e.g., level of participation of partners in project processes such as strategic planning, budgeting, implementation, monitoring and evaluation) iii.) Project staffing iv.) USAID management v.) Partnership model, including an examination of: -The project’s PBF model -The selection process of subgrantees -The project’s funding mechanism in support of the MOH Assessment of project policies and procedures. Assessment of relevance of project structure to project objectives. Project reports; key informant interviews; MOH, NGO, and CBO service statistics SDSH project staff (all) in PAP; SDSH financial and technical advisors in four departments, including four ZCs Possible travel limitations due to weather, road conditions; availability of key informants Assessment of partner satisfaction with the quality and effectiveness of collaboration with the SDSH project NGO, departmental, MOH, and CBO staff in four departments, including four ZCs Assessment of the depth, substance, and impact of SDSH collaboration with NGOs, MOH, and grantees Representatives of other organizations, NGO, and corporate donors that provide support to project-related activities Assessment of the extent to which SDSH application of the PBF model has stimulated enhanced performance by SDSH partners and grantees Grantee service statistics, reports to SDSH project, SDSH records 52 USAID/HAITI: SDSH PROJECT EVALUATION Evaluation question Type of analysis conducted Data sources and methods used Type and size of sample Limitations 3. Capacity Building: To what extent did SDSH build the capacity of local NGOs and of the MOH (departmental and central levels) to deliver health care services, as well as the capacity of communities, especially women and youth, to become active partners in working to improve the health system? To prepare local NGOs for successful “graduation” and an eventual role as direct recipients of USAID assistance? To prepare the MOH for a closer (i.e., bilateral) relationship with USAID? Assessment of improvement in the skills, capacity, and readiness of NGO and MOH staff to implement service delivery programs Key informant interviews; longitudinal review of grantee reports; project training records; qualitative interviews with clinic patients at service delivery points Project records; interviews w/leadership of MOH Dept. of Family Health and Planning Unit; MOH officials in four departments; client interviews and focus group discussions in four departments Possible travel limitations due to weather, road conditions; availability of key informants Assessment of the role of the SDSH project in improving the capacity of CBOs to play a substantive role in the planning for and execution of health service activities Examination of SDSH GUC procedures and NGO selection criteria; interview with selected CBOs Project records; interviews with CBO representatives in two departments Assessment of the role and impact of SDSH efforts to improve the capacity of local communities (e.g., community health committees) to identify and communicate health priorities to NGO and MOH service providers, and especially those priorities that affect women and youth. Interviews with community health workers; focus group discussion with village health committees USAID/HAITI: SDSH PROJECT EVALUATION 53 Evaluation question Type of analysis conducted Data sources and methods used Type and size of sample Limitations 4. Project Efficiency: To what extent did MSH’s management of subgrantees under SDSH promote the optimal use of project resources? (e.g., minimizing duplication and/or redundancy of effort, addressing gaps and preventing the use of multiple funding sources for the same activity) Identify extent to which project resources supplement/ complement other resources available to grantees and partners; assess extent to which SDSH staff closely monitor such assistance flows Key informant interviews; review of other-donor reports; discussion with NGO, ZC staff Nine NGOs, two CBO, at least three ZCs in four departments Inclement weather and/or road conditions that might impede travel    54 USAID/HAITI: SDSH PROJECT EVALUATION USAID/HAITI: SDSH PROJECT EVALUATION 55 ANNEX C: LIST OF PERSONS INTERVIEWED Name Institution Title SDSH project Bernateau Desmangles MSH/SDSH Performance Management Unit Team Leader Georges Dubuche MSH/SDSH Senior Technical Advisor/ Strategic Partnership Gina Rolles MSH/SDSH Assistante Exécutive à la Direction Herve Rakoto Razafimbahiny MSH/SDSH COP Kathy M. Kantengwa MSH/SDSH Acting COP Marie Florence Placide MSH/SDH Performance Management Unit Deputy Team Leader Martha Telfort MSH/SDSH Training Manager Serge Conille MSH/SDSH HIV/AIDS Unit Director Uder Antoine MSH/SDSH Deputy Chief of Party – Management Systems Strengthening Polone Daniel MSH/SDSH Departmental Technical Advisor (Artibonite) Jean Joseph Kesnel Benoit MSH/SDSH Departmental Technical Advisor (Centre) Calerbe Saint Louis MSH/SDSH Departmental Technical Advisor (Nippes) Francois Laviolette MSH/SDSH Departmental Financial Advisor (Nippes) Lucito Jeannis JHPIEGO Representant-pays Elsie Lauredan JHUCCP NGOs Balte Jacques Centre de Santé de Marmont Coordonateur de Terrain et Statisticien Ronald Chalestin Centre de Santé de Pierre Payen Directeur Virlande Dorcelus Centre de Santé de Pierre Payen Infirmière technique/Coordinatrice Projet SDSH Jesulene Mercy Centre de Santé de Pierre Payen Archiviste Jean Julien Daniel Beauge Centre de Santé de Pierre Payen Statisticien Mario Coty FONDEP Directeur général Rene Coty FONDEP Directeur adjoint Richardson Camilien FONDEP Responsable de Suivi Fritz Moise FOSREF Executive Director Marc Phillipe Depestr FOSREF Data Manager Marie Marthe Merisier Pierre Grace Children’s Hospital Responsable du Programme de Santé 56 USAID/HAITI: SDSH PROJECT EVALUATION Name Institution Title Louis Communautaire Jocelyne Arnoux Grace Children’s Hospital Responsable de Soins Infirmiers Gueldie Dupré Grace Children’s Hospital Opératrice de Saisie de Données Monge Germain Grace Children’s Hospital Disease Reporting Officer HIV program Jacsmine Tinasse Bien Aime ICC Data Manager Frédéric Vilmé ICC/Grace Children’s Hospital Directeur Médical Sœur Maria Aparecida Scatolin Lucelia Bontemps Responsable Gestion de Projet Sœur Marie Rose Jean Lucelia Bontemps Coordonatrice du Centre Sœur Gloria Herrera Sixco Lucelia Bontemps Infirmière/Responsable du Programme HIV/`TB Marie Caroline Etienne Lucelia Bontemps Accountant Dusson St. Jean Lucelia Bontemps Responsable des Archives Marie Chery MEDISHARE Directrice Serge Pintro MEDISHARE Conseiller Technique Wisline Celestin MEDISHARE Coordonatrice Santé Communautaire Faiderme Casseus MEDISHARE Médecin/Programme Manager Robert Nicolas SADA Executive Director Yolaine Remy SADA Responsable Activité Santé Mario Rinchere SADA Accountant Brucely Simon SADA Responsable Logistique Williome Jean Joseph SADA Responsable Section Education Jiyane Pierre SADA Responsable Programme Cabaret Marie Francoise Fleurimond SADA Infirmière en Charge de Formation Marius Alcinvil SADA Responsable Monitoring and Evaluation Marie Tania Emile SADA Centre de Santé de Pont Matheux Médecin de Programme Jocelyne Paulerin SADA Centre de Santé de Pont Matheux Auxilliaire Responsable Jean Feguens Regis SADA Centre de Santé de Pont Matheux Statisticien SDSH Yvon Labissiere Save the Children Health/Nutrition Manager MOH Brignol Boulin Direction Sanitaire de l’Ouest Coordonateur Technique Jean Richard Perard Direction Sanitaire de l’Ouest Administrateur Elizabeth Jean Noel Nelson Direction Sanitaire de l’Ouest Epidémiologiste USAID/HAITI: SDSH PROJECT EVALUATION 57 Name Institution Title Jacques Valguy Direction Sanitaire de l’Ouest Informaticien Mercédès Philogène Direction Sanitaire de L’Artibonite Responsable Santé Infantile Monelle Lormeus Direction Sanitaire de l’Artibonite Infirmière Santé Communautaire Papayoute Lucfentz Direction Sanitaire de l’Artibonite Comptable en Chef Maxene Michel Direction Sanitaire de l’Artibonite Comptable en Charge des Zones Ciblées Djefco Joseph Direction Sanitaire de l’Artibonite Data Manager SDSH Marcel Chatelier Centre de Sante Intégral de Raboteau Directeur Ronald Jean Pierre Centre de Sante Intégral de Raboteau Officier de Surveillance Ronald Saint Jean Centre de Santé de K-Soley Directeur Anejean Clerzeus Centre de Santé de K-Soley Responsable Statistiques Nadege Jacques Direction Sanitaire du Centre Epidemiologiste Départemental Anne Thérèse Touissaint Direction Sanitaire du Centre Responsable Programme VIH/SIDA Jacques Laroche Direction Sanitaire des Nippes Director Maurice Fils Mainville DSF PEPFAR Consultant Antoine Alceus UPE Directeur Gabriel Thimothé MSPP Directeur Général USAID Clint Cavanaugh USAID Senior HIV/AIDS Advisor Charlotte Eddis USAID Health Pillar Coordinator Elsie Michel Salnave USAID Health Systems Strengthening Advisor Harry François USAID Mission Monitoring and Evaluation Officer Joyce Kim USAID Mission Evaluation Officer Karen Welch USAID Health Office Chief Katherine Reyniers USAID Program Officer – PCPS Health Office Backstop Kovia Gratzon-Erskine USAID Senior MCH Advisor Olbeg Desinor USAID Pediatric Aids and OVC Advisor Reginalde Masse USAID Family Planning and Reproductive Health Advisor Stephane Morisseau USAID Public Health Advisor OTHER Agathe Pellerin PADESS Coordonatrice Michel Brun UNFPA Reproductive Health Advisor 58 USAID/HAITI: SDSH PROJECT EVALUATION USAID/HAITI: SDSH PROJECT EVALUATION 59 ANNEX D: LIST OF DOCUMENTS REVIEWED Alliance for Health Policy and Systems Research. Neglected Health Systems Research: Governance and Accountability. Geneva: WHO, 2008. Alva, S., Kleinau. E., Pomeroy, A., and Rowan, K. Measuring the Impact of Health Systems Strengthening: Literature Review. Washington DC: USAID, 2010. “Appraising the State of Performance Appraisal.” Health Manpower Management, Vol. 22, Issue 6 (1996), pp. 9–13. Association of Haitian Physicians Abroad (AMHE). Helping Haïti Rebuild: The U.S. Government’s Strategy. AMHE, September 2011. Auxila, P. and Antoine, U. MSH’s Experience in Haiti with Performance-Based Initiatives. Haiti: MSH, May 2007. Boykin, D.B. What Performance-Based Contracting Really Means for Procurement Goals. IV. STG, Inc., Reston, VA: 2005. Canadian International Development Agency. Projet d’Appui au Developpement du System de Santé en Haiti: Document de Projet. Haiti: CIDA, 2006. Cayemittes, M., Placide, M.F., Mariko, S., Barrère, B., and Sévère, B. A. Enquête Mortalité, Morbidité et Utilisation des Services Haïti 2005–2006. Ministère de la Santé Publique et de la Population, Institut Haïtien de l’Enfance et Macro International Inc. Calverton, MD: ICF Macro, 2007. Center for Development Information and Evaluation. Performance Monitoring and Evaluation TIPS. Washington, DC: USAID, 1996. Eichler, R., Auxila, P., and Pollack, J. Promoting Preventive Health Care: Paying for Performance in Haiti. World Bank, 2001. Loevinsohn, B. Performance-Based Contracting for Health Services in Developing Countries: A Toolkit. World Bank, 2008. Management Sciences for Health. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Semi Annual Progress Report. Haïti: MSH, May 2008. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Semi Annual Progress Report. Haïti: MSH, May 2009. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Semi Annual Progress Report. Haïti: MSH, May 2010. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Semi Annual Progress Report. Haïti: MSH, May 2011. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Annual Progress Report. Haïti: MSH, October 2008. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Annual Progress Report. Haïti: MSH, October 2009. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Annual Progress Report. Haïti: MSH, October 2010. 60 USAID/HAITI: SDSH PROJECT EVALUATION ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Monitoring and Evaluation Plan. Haïti: MSH, February 2008. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Monitoring and Evaluation Plan. Haïti: MSH, February 2009. (Revised) ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Monitoring and Evaluation Plan. Haiti: MSH, November 2010. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Milestones Plan. Haïti: MSH, October 2007. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Milestones Plan. Haïti: MSH, February 2009. ———. Santé pour le Développement et la Stabilité d’Haïti / Pwojè Djan: Milestones Plan. Haïti: MSH, October 2009. ———. Santé pour le Développement et la Stabilité d’Haïti / Poljé Djan: Milestones Plan. Haïti: MSH, November 2010. ———. Haïti Health Systems 2004 Project: Service Delivery and Management Assessment Protocol. Haïti: MSH, (Undated) ———. Guide des donnes du rapport mensuel et des inidicateur de resultants. Haïti: MSH, February 2011. ———. Rebuilding Haiti: People, Partnerships, Pride. Investing in Partnerships for Health Impact: A Model for other Sectors. MSH, Undated. Newell, E. Performance-Based Contracting. NextGov. June 2008. Partners in Health. Program Management Guide. PIH, 2011. Shen, Y. “Selection Incentives in a Performance-Based Contracting System.” Health Services Research, Vol. 38, Issue 2 (2003). Sheppard. Wu, Z., Dilley, K., Cros, M. Impact of Performance-based Financing in Haiti: Draft Final Report. Schneider Institutes for Health Policy, Heller School, Brandeis University and Center for Health Services. Boston: MSH, June 2011. U.S. Government. Haiti Post-Earthquake Strategies towards Renewal Economic Opportunity. Washington DC, January 2011. For more information, please visit http://resources.ghtechproject.net Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com