MID-TERM ASSESSMENT of the EMERGENCY MEDICAL ASSISTANCE PROGRAM (EMAP) A program of USAID/WB&G implemented by CARE International Contract: 294-A-00-01-00130-00 8/11/2003 Prepared by: Eric Sarriot, MD, PhD; ORC Macro Carl Harris, Consultant i Table of Contents Acknowledgements......................................................................................................................... ii List of Acronyms ...........................................................................................................................iii I. Executive Summary............................................................................................................ 1 EMAP as a Response to Emergency Public Health Needs in the West Bank and Gaza ........................................................................................................................ 2 Sentinel Surveillance System (SSS) ............................................................................. 2 Procurement and Distribution of Essential Equipment, Supplies, and Pharmaceuticals...................................................................................................... 2 Emergency Medical Education Development (EMED) ............................................... 3 EMAP’s Sub-grants Component................................................................................... 3 Management of EMAP ................................................................................................. 4 Key Recommendations ................................................................................................. 4 II. Background Statement........................................................................................................ 7 A. Historical Context of the Emergency Medical Assistance Program (EMAP)................................................................................................ 7 B. Mid-term Assessment of EMAP......................................................................... 9 III. Findings............................................................................................................................. 10 A. Appropriateness of the Program As An Emergency Response Tool................ 10 B. Effectiveness of the Implementing Partner in Four Components of the Program................................................................................................... 11 C. Program Management by the Implementing Partner........................................ 28 IV. Prioritized Recommendations........................................................................................... 29 V. Annexes............................................................................................................................. 31 A: Preliminary/Tentative In-country Assessment Schedule B: Assessment Team Bios C: Field Guide/Questionnaires D: Interviews Conducted E: Sites Visited F: List of Key Documents Available to Assessment Team EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 ii Acknowledgements We would like to thank the many people who made our work easier, arranged for our travel and accommodations, guided us through the intricacies of the Palestinian reality, and received us graciously. We thank USAID/West Bank and Gaza for doing much more than expected in preparing our visit. Secondly, we would like to thank CARE and the entire EMAP team for going out of their way to make our task easier and accompanying us throughout the West Bank and Gaza. We also thank all who contributed time and valuable information, from MOH cadres, NGOs, clinics, sub￾grantees, and international organizations personnel. We want to give special thanks to Nadir Abu Al-Hawa for taking us to our destinations safely, with aplomb and friendship. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 iii List of Acronyms ANERA American Near East Refugee Aid ECHO European Commission Humanitarian Aid Office EMAP Emergency Medical Assistance Program EU European Union FAO Food and Agriculture Organization GIMS General Inventory Management System HSBR Health Sector Bi-weekly Report IR Intermediary Result JHU Johns Hopkins University MOH Ministry of Health MOU Memorandum of Understanding NGO Nongovernmental Organizations PA Palestinian Authority PFS PHS Patients Friends Society Palestinian Health System PRCS Palestine Red Crescent Society RFA Request for Applications SO Strategic Objective SSS Sentinel Surveillance System TOS Training of Students TOT Training of Trainers USAID U.S. Agency for International Development USAID/WB&G USAID assistance to West Bank and Gaza WB West Bank WB&G West Bank and Gaza WHO World Health Organization EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 1 I. Executive Summary This is the report of the assessment of the Emergency Medical Assistance Program (EMAP), which was conducted from July 6 to July 21, 2003 in the West Bank and Gaza. EMAP is implemented by CARE International and sub-contracting partners American Near East Refugee Aid (ANERA), Johns Hopkins University Bloomberg School of Public Health (JHU), and Al Quds University. The aim of the program is to “sustain and strengthen the healthcare system in the West Bank and Gaza in order to maintain the well-being of Palestinians who are threatened by the violence and economic and social dislocation associated with the Israeli￾Palestinian conflict.” It is designed, under the U.S. Agency for International Development’s assistance to West Bank and Gaza (USAID/WB&G) Strategic Objective (SO) 7—“Improved and Sustained Performance of the Health System”—, as an emergency response to the degradation in the health situation in the West Bank and Gaza following the second Intifada, which began in September 2000. It is an essential part of USAID/WB&G’s health portfolio, along with the Maram project and other health sector interventions.1 EMAP’s aims are described in the following four Intermediary Results (IRs)2 : • IR 1: Establish a sentinel surveillance system to monitor health indicators • IR 2: Provide emergency medical equipment and supplies to health facilities • IR 3: Provide training programs in trauma management and emergency medicine • IR 4: Provide temporary financial assistance to Palestinian Nongovernmental Organizations (NGOs) offering rehabilitation services. Since September 2001, EMAP has grown considerably in response to increasing needs in the West Bank and Gaza (WB&G). Its total budget is now $13 million, with half of it dedicated to emergency procurement of medical supplies and essential medicines. It operates in all 16 governorates of the West Bank and Gaza. The purpose of this exercise was to provide USAID/WB&G with an independent assessment of EMAP, and specifically to determine the following: 1. The overall appropriateness of the EMAP program’s design and implementation effectiveness in meeting crucial emergency and basic needs of Palestinian health services 2. The effectiveness of program operations in their four components: sentinel surveillance system (IR1), procurement and distribution (IR2), sustainable educational programs in trauma management and emergency medicine (IR3), and sub-grants (IR4) 3. The effectiveness of CARE’s management systems as they relate to EMAP, to sub￾contractors, sub-grantees, and its main stakeholders. 1 Other USAID/WB&G health sector support includes: Community psycho-social support program, essential vaccines, support to HealthInforum, child health research, GIS, food assistance, Comprehensive Food Security and Nutrition Assessment, field epidemiological training, and university linkages. 2 Two additional IRs were added at a later date: the Rapid Nutritional Assessment and the Food Security and Nutrition Assessment. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 2 Two evaluators, using a qualitative approach based on document review, interviews, and observation visits, conducted the assessment. Key findings can be summarized as the following: EMAP as a Response to Emergency Public Health Needs in the West Bank and Gaza EMAP is a valuable and appropriate emergency response to the ongoing crisis in the West Bank and Gaza, which seriously affects the movement of persons requiring medical care. This crisis forces people to seek health services closer to home. It also affects the ability of smaller health facilities to procure adequate quantities of medicines in response to the larger demand on their services. This demand was generated by both lack of access and a suffering economy. EMAP helped fill the health services gap created by this emergency situation in some critical areas of need. EMAP succeeded in delivering essential drugs, disposables and medical equipment to the WB&G during times of crisis. EMAP vehicles and staff became known to the 60 health facilities and 15 NGO sub-grant recipients and, more importantly to their clients, as a supportive provider of emergency assistance. It has been one of the most significant mechanisms through which the international community, through varying degrees of coordination, has helped avert a full-fledged public health catastrophe during the past few years. As such, EMAP can be assessed as a needed, strong, and effective emergency response to health care problems caused or made more acute by the closures and military incursions in the WB&G. Sentinel Surveillance System (SSS) The Sentinel Surveillance System (SSS) is an excellent and critical initiative of EMAP. As a component of intervention, it meets critical information objectives of the program, but remains somewhat short of providing all the information it was initially expected to deliver. The SSS provides valuable information to all stakeholders of health in the WB&G, through collaboration between Al Quds University, JHU, ANERA, Maram and CARE. During the past year, it has provided a basis for advocacy and has suggested action steps to answer critical health needs in the Palestinian population. The assessment identified areas for improvement of the SSS such as the definition and selection of indicators and the sampling methodology which can best answer questions about sub-areas within governorates. Procurement and Distribution of Essential Equipment, Supplies, and Pharmaceuticals The procurement and distribution systems are professional and efficient operations. The project has succeeded in its goal to deliver medicines and medical equipment to areas in the WB&G where the general population has limited access to medical services due to conflict, closures, curfews, and checkpoints. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 3 Moreover, distribution of supplies and pharmaceuticals can be shown to have been selective, needs-based, and collaborative, particularly at the 60 NGO facilities monitored actively by EMAP’s field agents. EMAP has been sensitive to the continued expression of needs by the Ministry of Health (MOH), and procured pharmaceuticals have increasingly focused on MOH needs, to the extent that procurement for the MOH central stores has now reached about half of all supplies and pharmaceuticals purchased by EMAP. EMAP has been an appropriate and effective mechanism to meet the emergency needs of an at-risk population. Emergency Medical Education Development (EMED) The Emergency Medical Education Development (EMED) component is a quality program developed with high standards of excellence in terms of technical content and educational approach. However, its design does not fit that of an emergency response, and its implementation is still in its early stages. It provides a strong and necessary basis for possible further efforts to strengthen emergency care in the WB&G. The EMED component faced a slow start, but has advanced through considerable involvement and buy-in from local partners in Gaza and East Jerusalem. It appears to be ready to start also in the West Bank. Using a Training of Trainers (TOT) approach, which develops local capacity, it has used an interactive and dynamic adult learning methodology to deliver a high-quality technical content. EMAP will need to be extended beyond its initial timetable if it is to complete the training schedule, which now appears to be on track.3 EMED can be looked upon as preparing the ground for a possible long-term effort to institutionalize best clinical emergency practices and coordination of care between different levels through supervision and quality improvement approaches. EMAP’s Sub-grants Component Sub-grants have successfully supported critical rehabilitation services, and in particular outreach services, to the neglected disabled Palestinian population. This component has been implemented through difficult conditions with high standards for selection of grantees based on capacity, effectiveness, response to needs, and through active support to the grantees, including on-project monitoring and control procedures. Most grantees have been delivering services above their stated targets. Overall, the sub-grants component’s performance has exceeded expectations in terms of supporting rehabilitation services to disabled populations. While large unmet needs for rehabilitation remain in the disabled population of the WB&G, EMAP—through its local grantees—is one of the significant players addressing this area. People will continue to live with disabilities during and after the emergency situation, and will continue to be a particularly vulnerable group. Through its effective involvement in this area, EMAP 3 Course organizers indicate that the training schedule can be completed through an extension until March 2004. Given the objective of training 540 students, this is at least conditional on no further delay in TOT and Training of Students (TOS) implementation, particularly in the West Bank. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 4 faces a range of opportunities for increasing coverage, quality, and coordination of services to disabled persons. Sub-grants have also been awarded to support health care, particularly emergency health care services. Sub-granting to support health care services in the WB&G takes place on a limited scale. It has allowed the expansion of the availability of emergency services globally (through the Palestine Red Crescent Society (PRCS)), and the delivery of some essential services in a limited number of “hot spots.” In the absence of clear program targets, it is difficult to assess whether EMAP is meeting its objectives in this area. On a case-by-case basis, EMAP’s health care sub-grants respect sound standards of quality and appropriateness. As an emergency response, they unfortunately also carry with them a number of questions about sustainability and the quality of care delivered. Management of EMAP EMAP started and was implemented in difficult conditions, with a limited staff. It was able to respond to these difficulties in collaboration with USAID/WB&G by expanding its human resources and activities. Through the recruitment of field officers qualified in public health, it has developed an important ability to monitor activities, assess needs, provide support to sub￾grantees and other clients, and to operate in a decentralized organization. This is of critical importance since freedom of movement continues to be limited. The EMAP team is a dynamic, functional, and motivated organizational structure, able to deliver the range of services required by the program, and to expand its activities if warranted. Key Recommendations Based on our findings, the assessment team makes the following recommendations: 1. In the coming months to year, EMAP needs to maintain a capacity to respond rapidly to new threats and emergencies, due to the unpredictability of the situation, the difficulty of circulation and access to essential goods and services in the WB&G, and the economic situation of the Palestinian population. 2. EMAP, or a possible extension of EMAP, also needs to support, with method and care, the transition to a more autonomous and functional Palestinian health system. 3. In the coming months, the SSS needs to be improved and redesigned to address the following methodological issues (At a minimum, however, the system should be maintained until this redesign is completed.): • Selection and definition of indicators. This should be done through a general workshop involving all key stakeholders to serve as the first step toward integrating the SSS within the MOH Health Information System. • The sampling methodology needs to be reexamined to provide better information on local situations affected by recurrent crises. • Data collection should be on a monthly to quarterly basis, depending on the situation. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 5 4. In terms of procurement and distribution: • The manual materials management system in place at EMAP-supported health facility and sub-warehouse locations should be maintained. Sophisticated materials management software is not appropriate and will not be needed, nor will the cost be justified for these facilities, even in the long term. The MS Excel-based, computerized materials tracking system used at EMAP headquarters/central warehouse is most appropriate and easily maintained. If/when assisted health facilities and sub-warehouses have continuous electricity and a computer, they can also use Excel and e-mail their inventory movements to EMAP headquarters. • MOH Medical Stores assessments and subsequent funding for upgrading materials and distribution management should be coordinated and collaborated with other donors working with MOH and health services providers in the WB&G. • Medicines, disposables, and medical equipment should be sourced locally, when possible, to encourage local industry and economic recovery. International tendering could be a medium- and long-term detriment to improved health services by denying or curtailing local initiative. 5. The current EMAP should be extended to allow completion of the current EMED training plans in Gaza, East Jerusalem, and West Bank. 6. If follow-on activities are developed after EMED, they should focus on institutionalizing best clinical practices through supervision and quality improvement approaches and coordination of care between different levels of the health system (e.g., pre-hospital to emergency room; emergency room to intensive care and other hospital services; referral and counter-referral). 7. In the coming months to year, EMAP should continue to work with its current rehabilitation sub-grantees and possibly expand to support outreach services to increase capacity building efforts, particularly for the integration of disabled persons in society and the networking/coordination of disability services’ providers. This would effectively prepare the grounds for a long-term strategic effort to address disability in the WB&G beyond the current emergency situation. 8. In terms of health care delivery, EMAP’s ability to provide resources rapidly in a pointed manner should be maintained in the immediate future, while the economy and access to care issues are still unresolved, and in case new acute emergencies arise. 9. EMAP should consider implementing an exit strategy before the end of the sub-grant relationship, helping sub-grantees to develop business plans, access additional resources (technical and financial), identify cost-savings, or refocus services. 10. Continued access problems should be considered the norm, and EMAP should continue to operate in a decentralized manner. There needs to be, however, a regular clarification of roles and pro-active identification of communication and coordination gaps, which are inherent to the decentralized, multi-organizational, and multi-functional structure of the program management. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 6 11. In the long run, EMAP’s interventions can be a spearhead to addressing a range of public health issues, from rehabilitation to quality of emergency and basic health care, rational drug use, coordination of care, and environmental health issues. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 7 II. Background Statement This is an assessment of the Emergency Medical Assistance Program (EMAP) implemented by CARE and partners, under USAID/WB&G funding. This assessment needs to be understood in the context of USAID’s assistance to the West Bank and Gaza in the evolving situation brought on by the second Intifada, which began in September 2000. EMAP is the main response mechanism of USAID to the health emergencies faced by the Palestinian population. This section presents a brief overview of its context and development. A. Historical Context of the Emergency Medical Assistance Program (EMAP) 1. Context USAID/WB&G’s program pursues the objective of peace and stability in the Middle East, in particular by assisting the Palestinian Authority (PA) in addressing the public health situation in the WB&G. The PA’s second 5-year program, “National Strategic Health Plan for Palestine (1999-2003)” was adopted in May 1999 in response to the high importance placed on health services needs by the Palestinian population. USAID/WB&G first 5-year Strategic Plan (for the period of FY 1996 through FY 2000) addressed five SOs, from Economic Development, Water Resources, Governance, Health (SO7) to Community Services (SO8). This was followed by the submission of a 36-month Transition Plan in December 2000, which maintained the same overall objective and SOs, but adapted IRs to the increasing instability and insecurity demonstrated by the outbreak of the second Intifada, which began in September 2000. The Transition Plan has been in effect since January 2001. In this evolving context, USAID/WB&G first awarded CARE $250,000 in November 2000 to meet emergency needs of the Palestinian population (through procurement of essential equipment and emergency supplies, and financial support to rehabilitation NGOs and electrification of MOH clinics). This first award was followed, in April 2001, by the Rapid Emergency Response Program (RERP), a first cooperative agreement with CARE for $1.77 million, expanding emergency medical assistance from USAID/WB&G to include first aid and emergency care training for some 2,400 Palestinian doctors and nurses in the WB&G, in addition to the continuation of procurement and NGO support activities. This program came to an end on June 30, 2002. Following a competitive bidding process, EMAP was initiated in September 2001 as it was becoming increasingly difficult for the Palestinian health care system to meet growing emergency and health care needs in the context of the on-going conflict. The second Intifada, which began in September 2000, reached a new level of intensity in the Spring of 2002 when military action intensified significantly and the curfew and closure regime became increasingly stringent. The Palestinian economy has largely collapsed and suffers from growing unemployment and poverty, with a continued worsening of the health and humanitarian situation in the WB&G. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 8 Through a series of amendments in response to the changing situation (including a doubling of the Cooperative Agreement from $5 to $10 million in July 2002), the total level of support has increased to more than $13 million. EMAP covers all 16 governorates in the WB&G. 2. EMAP The purpose of the EMAP program is to “sustain and strengthen the healthcare system in the West Bank and Gaza, in order to maintain the well-being of Palestinians who are threatened by the violence and economic and social dislocation associated with the Israeli-Palestinian conflict.” Its aims4 are the following: a. Establish a sentinel surveillance system to monitor health indicators. b. Provide emergency medical equipment and supplies to health facilities. c. Provide training programs in trauma management and emergency medicine. d. Provide temporary financial assistance to Palestinian NGOs offering rehabilitation services. This aim was later expanded to NGOs offering critical and emergency health services in areas of need. The main sub-contractors to CARE in the implementation of the program are American Near East Refugee Aid (ANERA); Johns Hopkins University Bloomberg School of Public Health (JHU), for the training and nutritional assessment components; and Al Quds University. The expansion of the EMAP program has not only aimed at expanding the procurement and distribution of products and equipment to hospitals, clinics, and other health providers in the WB&G, but also at improving the mechanisms and procedures in place under EMAP, to ensure a need-based systematic and quick response to health care facilities. The EMAP program has been shifting its focus from “emergency procurement of emergency care supplies and equipment” to “emergency procurement of basic medical supplies and essential medicines.” This shift explains most of the doubling of the size of the grant. In February 2003, USAID/WB&G requested an independent third-party midterm assessment of the EMAP program. This assessment was conducted in July 2003 and is reported in this document. At the time of this assessment, implementation of the “Road Map” has started and led to some improvement in the closures and violence known to the region. Circulation both to and within the WB&G remains, however, seriously constrained by military control (checkpoints), while the economic situation in the WB&G remains in severe crisis.5 4 A West Bank and Gaza-wide nutritional assessment of women of childbearing age and children ages 6–59 months was added in December 2001 as a fifth program activity in response to an MOH request. (Nutritional assessment of the West Bank & Gaza Strip (PN-ACR-711) ; http://www.dec.org/pdf_docs/PNACR711.pdf) A sixth activity was added in February 2003: a (follow-on) Nutrition Assessment as part of the FAO- sponsored Food Security Assessment and Nutritional Surveillance (FSANS). 5 World Bank. 2003. Twenty-seven months—Intifada, Closures and Palestinian Economic Crisis. An Assessment. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 9 B. Mid-term Assessment of EMAP 1. Purpose and Scope of Work of the Assessment The purpose of this evaluation was to provide USAID/WB&G with an independent mid-term assessment of the EMAP Program, based on qualitative investigation methods and addressing the following: a. The overall appropriateness of the EMAP program’s design and implementation effectiveness in meeting crucial emergency and basic needs of Palestinian health services b. The effectiveness of program operations in their four components: sentinel surveillance system (IR1), procurement and distribution (IR2), sustainable education programs in trauma management and emergency medicine (IR3), and sub-grants (IR4) c. CARE’s management systems and capacity as they relate to the EMAP program, to sub￾contractors, sub-grantees, and its main stakeholders. 2. Methodology The assessment was conducted from July 6 to July 21 2003 by a two-member team. (The planned (tentative) in-country assessment schedule is available in Annex A. Annex B provides short bios of the assessment team.) An assessment field guide was developed prior to the in-country assessment and revised based on discussions with USAID (Annex C); it was used flexibly as a resource to conduct semi￾structured interviews. The assessment team collected and analyzed information gathered from the program and other sources. Interviews were conducted with USAID and EMAP program staff, as well as MOH cadres, program sub-contractors, sub-grantees, and other stakeholders (e.g., international organizations; see Annex D). Interviews were combined with observation visits to NGO clinics and sub-grantee organizations. A list of sites visited is provided in Annex E. The assessment team wrote its findings and recommendations based on extensive notes taken during the in-country assessment, and after proper debriefing of USAID/WBG and EMAP staff. These findings are presented in the following section. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 10 III. Findings Findings are presented according to the three tasks set for the assessment team: a. Appropriateness of EMAP as an emergency response tool to the health situation of the WB&G, and its complementarities with the intervention of other agencies operating in the WB&G b. Effectiveness of CARE and its partners in the implementation of four components of EMAP (SSS (IR.1), procurement (IR.2), emergency medical training (EMED; IR.3), and sub-grants (IR.4)) c. Program management In section B, the general objectives set for each component are restated, with a brief overview of EMAP’s reported achievements. The assessment team’s observations are then presented, highlighting both areas of strengths and weaknesses for EMAP as a program, and for the implementing agency. Finally, recommendations are provided and prioritized in Section VI. A. Appropriateness of the Program As An Emergency Response Tool EMAP is a valuable and “noticed” emergency response to the continuing crisis in the WB&G. It was designed to overcome the access restrictions imposed on the WB&G, after the onset of the second Intifada, which began in September 2000, brought curfews, closures, and increased numbers of checkpoints. This situation seriously affected the movement of persons requiring medical care and basic life needs. It led to the destruction and/or interruptions of infrastructure such as water supply and electricity to homes, businesses, and public services, including health facilities. This emergency situation forced people to seek health services closer to home and affected the ability of these typically smaller and formerly less-used health facilities to procure adequate quantities of medicines to respond to the larger demand on their services. EMAP helped fill the health services gap created by this emergency situation in critical areas of need. EMAP vehicles and staff became known to the 60 health facilities and 15 NGO sub-grant recipients (and, more importantly, to their clients), as the provider of emergency assistance. EMAP accomplished this by employing dedicated, knowledgeable field staff and a security officer who was able to build a communications network to facilitate checkpoint crossings and passage through conflict areas. The World Bank, European Union (EU), World Health Organization (WHO), and MOH officials all stated, during interviews, that EMAP was a major player in addressing the health needs of Palestinians affected by access issues during the emergency. All made the point that, in spite of the new hope brought about by the current “road map” discussions, the emergency was not yet over. According to one WHO official, EMAP was a significant player in averting “the health catastrophe that did not happen.” It must be remembered that EMAP is an emergency response. Its procurement and delivery component was created and functions as a stopgap measure to deliver essential medicines, EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 11 disposables, and medical equipment in response to the situation in the WB&G. This component has functioned successfully. Part of EMAP’s response to the critical health situation faced by the Palestinian population however, is also addressing long-term health development needs, made more acute by, but not limited to the emergency situation. Through its sub-grant mechanism, EMAP has funded disabilities rehabilitation as part of its emergency response. Rehabilitation services are traditionally under-funded, and EMAP efforts have been welcomed by its partner clinics. This activity has uncovered a large unmet need in the community, not only for disabilities caused by conflict, but also victims of accidents and the much larger problem of congenital disabilities such as cerebral palsy, which is the leading cause of disabilities in the WB&G.6 The implication for a larger, regular program to study the cultural context of disability, and then to design an assistance package, based on current efforts, is compelling. EMAP’s support for emergency training and capacity building for NGOs also offers grounds for review and eventual inclusion into more long-term health development portfolios. Finally, EMAP includes a number of mechanisms through which information can be gathered: field officers, sub-grantees, NGO clinics, and the SSS itself. Reports made by clinic staff and sub-grantees to the assessment team about a rise in waterborne illnesses, for example, support the SSS evidence that lack of access to safe water is a growing issue in some parts of the WB&G. EMAP can play a role in providing up-to-date information and stimulating programmatic responses, even when other stakeholders can be mandated to carry out the response. EMAP can be assessed as a needed, strong, and effective emergency response to health care problems caused or made more acute by the closures and military incursions in the WB&G. EMAP is also doing more than just responding to emergencies, and includes forward-looking activities, which are discussed in the following section. B. Effectiveness of the Implementing Partner in Four Components of the Program 1. Sentinel Surveillance System The Sentinel Surveillance System (SSS) is an excellent and critical initiative of EMAP. As a component of intervention, it meets critical information objectives of the program, but falls short of providing all the information it was initially expected to deliver. The SSS provides valuable information to all health stakeholders in the WB&G, through collaboration between the Maram Project, Al Quds University, JHU, ANERA, and CARE. 6 White, T. 2002. Disability in Northern and Middle Gaza. A report of field screening and registration; 11 May-26 August 2002. European Commission Humanitarian Aid Office (ECHO); Movimniento por la Paz, el Desarme y la Libertad. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 12 During the past year, it has provided a basis for advocacy and orientation of programs through the difficult times faced by the Palestinian population. CARE and its partners are aware of limitations in both implementation and full usefulness of the tool, and steps can be recommended to improve both. The SSS is still at this point an essential tool for providing valuable information for responding to an evolving emergency situation. Objectives EMAP’s objective with regards to the SSS falls under IR1: Develop a sentinel surveillance system capable of detecting early changes in the health status of vulnerable Palestinian communities. Its expected impact was stated as follows: There will be a reliable stream of information on the accessibility and affordability of health care, food, and water for the civilian population of the WB&G. This information will document the impact of the closure and curfew policy on the population. It will enhance the current MOH health information system’s database and fill current gaps in public health information. This will provide the MOH and the donor community with a sound basis to act quickly and appropriately. The SSS was designed to collect bi-weekly information from all governorates, and to report on questions of food security, water availability, infectious diseases, and access to health services. On a bi-weekly basis, this would correspond to 26 rounds of data collection and 26 reports from May 2002 to June 2003. At the outset, the SSS was to focus on a limited number of clinics serving as sentinel sites. As a response to the closures and curfews in WB&G, which prevented patients from accessing clinics, it was redesigned to survey households. At each round it systematically surveyed 10 households in one rural and one urban cluster in each of the 16 governorates. By selecting new clusters within a governorate at each new round of data collection, the SSS was designed to provide information about the entire governorate through cumulated data. It also proposed to offer time￾sensitive information. Achievements Data was collected on a bi-weekly basis by Al Quds University through its network of students. Twenty-four rounds of bi-weekly data collection have been completed from May 17, 2002 to June 2003.7 Bi-weekly reporting has, however, never been achieved. But reports have been produced fairly consistently on a monthly basis, first integrated with MARAM’s project facility monitoring data, then independently for the last report. The 12th Health Sector Bi-weekly Report (HSBR) was 7 CARE International. Progress Report April to June 2003. Emergency Medical Assistance Program. Contract No 294-A-00-01-00130-00. (DRAFT). EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 13 released on May 24, 2003,8 posted on WHO’s Health Inforum Web site,9 and distributed by EMAP to USAID and its partners. The HSBR reports on access to food, water, and health services in the 16 governorates but does not report on infectious diseases indicators. Each HSBR provides information on the two latest rounds of data collection, as well as cumulative information on all clusters sampled since the beginning. Assessment Team Observations Observations can be made on the overall usefulness of the SSS (its effectiveness in providing information), and on strengths and weaknesses from design (sampling and selection of indicators) to implementation (data collection, analysis, and reporting). Effectiveness of the SSS in Providing Information to Health Stakeholders in the WB&G The SSS, through the HSBR, is recognized by partners and health stakeholders as a unique and most needed source of information. It has increased awareness about the situation of the population during the second Intifada, which began in September 2000; has allowed advocacy; and may have guided further research and studies. All stakeholders interviewed emphasized the unique importance of the SSS, as no equivalent source of information has been available. EMAP is right to stress in its last report that the SSS has become a “vital tool for decision makers working in the West Bank and Gaza,”10 There is some evidence of how HSBR information has been used for action and decisions. Informants interviewed during this assessment stated the following: 1. Data from the HSBR has been used for advocacy by many agencies and many channels.11 2. The donor community coordinated a convoy to Southern Hebron to respond to a lack of access to immunization services. The HSBR also stimulated the decision to implement the soon-to-be-published immunization assessment. 3. EMAP, Maram, and partners decided to include drugs for chronic illnesses in the medical kits in response to problems of access to care highlighted in the HSBR. These stakeholders generally feel that the SSS should, in the future, be integrated within the MOH’s information system.12 A number of informants, in and out of EMAP, questioned the respective value of household compared to sentinel site surveillance. 8 Health Sector Bi-weekly Report Number 12. 2003. Johns Hopkins University, Al Quds University, Maram Project, CARE International/ANERA. 9 Health Inforum News. 2003. Volume 2, No.30, 01. 10 CARE International. Progress Report April to June 2003. 11 HSBR data are referenced not only in Health Inforum, but also in other publications (e.g., World Bank. 2003. Twenty-seven months— Intifada, Closures and Palestinian Economic Crisis. An Assessment. 12 No MOH document, reporting on its own HIS information. could be made available to the evaluators. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 14 Sampling Methodology EMAP and partners appropriately chose to switch from a clinic-based surveillance system to a household survey, since lack of mobility and access was a central feature of the emergency situation faced by the Palestinian population. The basic sampling approach is sound and is briefly described above. Specific limitations/opportunities for improvement can be identified in the sampling approach with regards to the geographic and time-sensitive nature of the information desired. 1. Given the sample size in each governorate (10 urban plus 10 rural households), inferences about the actual level of an indicator can only be made for the entire territory, not at the governorate level. 2. Very small sample surveys (even sample sizes of 20 households or less) can be used to compare different areas against a threshold (for example, identifying areas with high difficulties in accessing services or water compared to areas with a low level of difficulty).13 This could be useful to classify areas within a governorate or to compare governorates (e.g., high and low need for intervention). Under the current approach, however: o Classification of different areas within governorates is not possible, because each round of data collection surveys only one urban and one rural cluster. o Classification of the different governorates between high and low level of difficulty or threat on any indicator is not possible either. This is because all data points within a governorate come from two clusters. The level of an indicator measured in these two clusters cannot be validly extrapolated to the entire governorate.14 3. This also means that time-trends cannot be described at the governorate level from month to month, since each round refers to a different cluster within a given governorate. Al Quds has resampled 20 to 30 percent of the clusters. These data still need to be exploited. Selection of Indicators We have already commented on the usefulness of the indicators reported in the HSBR to EMAP and all stakeholders of health in the WB&G. The HSBR has not reported on the infectious diseases indicators included in the SSS questionnaire, because the information collected was deemed unreliable. This may be due to difficulties in standardizing the administration of questions when surveyors are not trained together. Given the inability to revise the questions/indicators and re-train the surveyors to ensure consistency, EMAP took appropriate measures in not reporting on these indicators. 13 For details or an overview of the Lot Quality Assurance Sample (LQAS) method, see Valadez, J., W, Weiss, C. Leburg, and R. Davis. 2003. Assessing Community Health Programmes using LQAS baseline surveys and regular monitoring. A Trainer's Guide and User's Manual. www.talkuk.org. For an overview of the concepts, see Sarriot, E. G., P. Winch, W. Weiss, and J. Wagman. 1999. Methodology and sampling issues for Knowledge Practice and Coverage (KPC) surveys. Johns Hopkins University, School of Public Health, Department of International Health. http://www.childsurvival.com/kpc2000/Method.doc 14 EMAP is working on aggregating data from different clusters (different rounds of data collection) to produce this information. In a fluid and rapidly changing environment, it is not clear what the aggregated data collected during a year will actually indicate. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 15 Indicators relating to access to care, water disruptions, and food security have been reported and used. There is now a question as to what indicators should be added or withdrawn if the activity continues. Data Collection Al Quds has used its network of students to gather data fairly consistently. Difficulties have emerged in sending the questionnaires from collection sites to Al Quds operational laboratory in Jerusalem. Difficulties in circulating within the WB&G result in difficulties in training/supervising surveyors. This may have affected the ability of EMAP to take corrective measures on questions that proved unreliable. Data Analysis and Reporting Data analysis and reporting suffered from a number of difficulties, which translate to the relatively low number of HSBRs published as compared to initial expectations. Most of these difficulties relate to coordination mechanisms. 2. Procurement and Distribution of Basic Medical Supplies, Equipment, and Essential Medicines The current EMAP procurement and distribution systems are professional and efficient operations. The project has succeeded in its goal to deliver medicines and medical equipment to areas in the WB&G where the general population has limited access to medical services due to conflict, closures, curfews, and checkpoints. Objectives The objectives of the emergency procurement agreement fall under IR2: Emergency medical equipment, medical supplies, and pharmaceuticals are provided to local and regional health facilities to meet increased demands for emergency services. As stated in the EMAP proposal document, “the purpose of this activity is to provide targeted, essential medical equipment, medical supplies, and pharmaceuticals to primary healthcare clinics, rehabilitation service providers, and referral hospitals in the WB&G, to improve their capacity to supply emergency medical, rehabilitation, and trauma services. The provision of diagnostic and treatment equipment will be complemented by appropriate orientation and training.” The cost estimate for this activity is $5.9 million. EMAP uses two procurement mechanisms to assist health services in the WB&G: a. Direct procurement of essential drugs, disposables, and medical equipment, all of which are goods EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 16 b. Procurement through its sub-grants, including the emergency medical training (EMED) project, which may be a combination of goods and services. Direct procurement includes all EMAP pharmaceutical and medical equipment purchases made for the MOH, and procurement on behalf of the NGO health facilities under the EMAP project umbrella, including the range of medical kits supplied. Procurement through sub-grants is for goods or services requested by the sub-grantee, or by the nature of the sub-grant agreement itself and already negotiated. Achievements: The Numbers The EMAP logistics/procurement system was expanded in April 2002. According to the latest EMAP progress report, the following had been accomplished by end June 2003: • $5,358,089—value of commodities ordered • $5,321,020—value of commodity throughput • 103 individual contracts • 40 companies contracted • 550 individual shipments • 400 separate deliveries • 75 locations delivered to • 260 pharmaceutical kits delivered • 120 medical disposable kits delivered • 6,500 entries in materials management/logistics tracking system • 17 EMAP staff directly or partially involved in the procurement and logistics process • 60 primary health clinics included in the project • 480 villages served • 15 sub-grants awarded and serviced (see Sub-grants Component Section) • Training equipment (e.g., manikins) and emergency medical equipment have been procured for 13 MOH and NGO hospitals (see EMED Component Section) A USAID procurement-tracking document indicates the following EMAP procurement expenditures by recipient: MOH/WB&G $2,616,400 EMAP NGOs $2,547,000 Sub-Grantees $ 316,600 Total $5,480,000 EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 17 EMAP medical kits currently comprise 23 essential drugs, and EMAP disposable kits comprise 22 basic medical disposables. They fill the needs of 60 participating NGO health facilities for one month. Total distribution is divided to deliver 60 percent to the West Bank and 40 percent to Gaza. According to the World Bank health finance requirements working document, “West Bank/Gaza: planned budget/financing for 2003 expenditures” total donor funding, excluding USAID contributions, for non-salary and benefits costs is estimated at $20,792,000 against total requirements of $26,208,000. These figures indicate that the EMAP contribution to health services in the WB&G is significant, contributing about 10 percent of total MOH non-salary and benefits requirements to date, in addition to its NGO support. EMAP 6 procurement will increase to nearly 17 percent EMAP’s contribution to MOH’s non-salary and benefits portion of its budget. Assessment Team Observations Procurement and Logistics Management Direct procurement is done by the CARE WB&G procurement department, working closely with and on behalf of EMAP, MOH, and NGO staff, following procedures developed by CARE International for its offices worldwide. Procurement through the EMAP sub-grants allows for the purchase of goods and services by the sub-grantee as per the various grants awarded by the EMAP project. Some grants include line items for salaries, running costs, training, and other services; and other grants are for the purchase of medicine, medical supplies and equipment, and combinations of these. All materials purchases of more than $5000 are subject to the same rules as CARE direct procurement, including tendering, bid analysis, bid selection by committee, and so on. Sub-grantees are also required to use this purchase procedure, even when the $5000 threshold is not met, but may purchase on their own. In almost all cases the sub-grantees ask for EMAP assistance and vetting of these self purchases. All sub-grantees interviewed expressed satisfaction with the procurement procedures, and all stated that it has enhanced their own organization’s management effectiveness and has been a training exercise for them in best business practices. All medical supplies purchased by EMAP are received and inventoried at EMAP’s warehouse in East Jerusalem. Contents are checked and approved by the EMAP pharmacist and stored under the supervision of the warehouse manager. Medical supplies allocated to NGO clinics are distributed and tracked by the EMAP logistics staff. Supplies destined for MOH clinics are shipped to the main MOH warehouses and included within the MOH system. They are then distributed to MOH hospitals and clinics by MOH or with assistance from the International Committee of the Red Cross (ICRC). A review of EMAP procurement and logistics records and interviews with procurement and logistics head office and field staff revealed the following: • The current procurement and distribution system put in place by EMAP is appropriate and is meeting the objectives of the program. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 18 • The procedures meet international standards and USAID requirements. • Local procurement is a key to the rapid response needs of assisted health facilities. • Procurement is done by accepted tender mechanisms through pre-approved vendors. • Tender committees have technically qualified members included to assure the tender specifications are accurate and understandable to international standards. • Bid committees have technical and project personnel present. • Bid analysis is a joint endeavor of EMAP and MOH. • Technical personnel for drugs, disposables, and medical equipment assist in the bid analysis to ensure the most competitive selection. • The standard CARE procurement procedures manual is followed for all procurement actions. • Procedures have been developed for EMAP logistics management and are followed scrupulously. • The materials management and distribution systems are simple, efficient, and appropriate, given the constraints of access and quantities of materials provided. Telephone communication and e-mail is vital and consistent. Tracking and management forms for inventory, receiving, and dispatching are manual at health facility and sub-warehouse level, and the EMAP tracking and recording system is based on Microsoft Excel and is a modification of CARE’s GIMS, used world-wide by CARE International for materials management. Records were found to be up to date, accurate, and efficient. Logistics staffs are well trained and professional in their approach to their work. Distribution Through Challenges Delivery of procured goods was accomplished in spite of closures, roadblocks, checkpoints, and curfews. EMAP was able to accomplish this through a combination of quality staff at its headquarters and in the field, and by employing a security officer able to maintain the relations necessary to expedite travel and access to troubled areas a majority of the time. It is laudable that of the 260 medical kits and 120 disposable kits delivered to 75 separate sites, only one was opened for inspection at a checkpoint. EMAP’s Response to the MOH Needs EMAP has responded qualitatively well to emergency pharmaceutical needs of the MOH, and has increased the ratio of procured goods between MOH and NGOs (procurement for the MOH now represents about 50 percent of all medical supplies and pharmaceuticals procured by EMAP). The MOH, however, still has important recurrent (non-emergency) needs for supplies and pharmaceuticals.15 15 MOH drug list needed for the West Bank published on 22 May 2003, according to the central drug stores in Ramallah; source: HealthInforum. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 19 Since its inception, EMAP’s emergency medical supplies contribution to the MOH and health facilities in the WB&G has been approximately 10 percent of the total needs.16 This figure becomes much more significant when the context of the EMAP’s mandate to supply drugs and supplies in areas of difficult access is considered. MOH officials mentioned that EMAP’s procurement cycle was too long, and then stated that it was much faster than any of the other donors to the MOH central stores. A further procurement of $1,750,000 is in the pipeline for this coming year and will increase the EMAP contribution to MOH non-salary and benefits budget needs to nearly 17 percent. EMAP consulted extensively with MOH to establish its list of most needed medical supplies, to be included in the pre-packed kits for delivery to NGOs and additional supplies forwarded to MOH central stores. MOH stores staff confirmed that local procurement helped speed up the procurement process, and EMAP’s contribution of essential emergency medicines and medical supplies were a welcome addition during a time of shortages. All locally procured drugs were tested before purchase and met ISO standards. EMAP’s direct support to the MOH has increased regularly, and EMAP is also preparing to fund an MOH medical stores assessment. This is a positive direction that will need to be coordinated with other efforts of the MOH and donor community currently reported under way by informants in the donor community.17 EMAP’s Response to NGO Needs EMAP has met or exceeded its goals of meeting the emergency needs of assisted NGO primary health clinics. EMAP ensured its 60 participating NGO health facilities and 15 sub-grantees in the WB&G a steady supply of emergency drugs, medical supplies, equipment, and health emergency support. Participating NGOs were chosen through a clinic survey conducted from May to July 2002. Initial difficulties included access, suspicion of CARE’s motives for the survey, and inexperience of new staff. A clinic survey tool was developed with six basic criteria for health facility consideration: patient load; remote or urban; current pharmaceuticals in stock; impact of closures; access to resources; and clinic observation. A map was developed showing areas of closure, ensuring that at least one clinic was chosen within each closure area. One hundred clinics were evaluated and 60 were chosen after USAID vetting. The chosen clinics reflect a balanced geographic coverage. Four field officers were employed to assist and monitor EMAP distribution activities and coordinate between EMAP and the NGOs. Several of the initially chosen NGOs declined to be included in the project for various reasons, leaving areas with lesser coverage in the northern West Bank. All beneficiary facilities understood and accepted the support provided by USAID through CARE. 16 Approximation based on World Bank and USAID figures provided in the Achievement Section. 17 The assessment team has learned from the EC Health Task Manager that ECHO is funding Pharmacists Without Borders to assess the MOH medical stores. The World Bank, the EU, Department for International Development (DfID), and other donors are completing a health sector review, and the report will be available by the end of September. RAND Corp is also reportedly conducting an MOH review to inform a reform process. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 20 Staff at the EMAP partner NGO health facilities and the sub-grantees are happy with the medicines and medical supplies provided by the project. It has enhanced their ability to provide services to their client populations. They use EMAP drugs to bridge gaps in supply; more importantly—as observed in visits to facilities—the supplied drugs are economically accessible, provided for free or at low cost to their populations during this time of economic hardship. 3. Emergency Training and Trauma Management/Emergency Medical Education Development (EMED) The Emergency Training and Trauma Management component (now named the Emergency Medical Education Development (EMED) course) is a quality program developed with high standards of excellence in terms of technical content and educational approach. However, its design does not fit that of an emergency response and its implementation is in fact still in its early stages. What it does is provide a strong and necessary basis for possibly further efforts to strengthen emergency care in the WB&G. EMED faced a slow start, but has advanced through considerable involvement and buy-in from local partners in the WB&G and East Jerusalem. Using a Training-of-Trainers (TOT) approach, which develops local capacity, it has used an interactive and dynamic adult learning methodology to deliver a technical content of high quality. EMAP will need to be extended beyond its initial timetable if it is to complete the training schedule, which now appears to be on track. Expectations about changed practices by students are expected to remain modest, without systemic changes to individual training. Beyond this, a possible extension of this investment is probably better approached through a long-term effort to institutionalize best clinical emergency practices and coordination of care between different sectors through supervision and quality improvement approaches. Objectives The purpose of this component is to provide technical assistance to the various levels of the emergency healthcare system and to introduce and institutionalize first responder, emergency medical and trauma training as part of the current medical education system by developing both the facilities and personnel needed to conduct ongoing educational programs. Its objectives fall within IR3: Technical assistance for the development of sustainable educational programs in trauma management and emergency medicine. JHU is the implementing partner of this component of EMAP. Specifically, it proposes to train 48 trainers through TOT courses. These trainers are to then train 540 doctors and nurses through Training-of-Students (TOS) courses conducted in 6 training centers in the WB&G. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 21 Achievements18 A training needs assessment was conducted from May to July 2002, but the onset of training activities has been delayed for different reasons. Training started in 2003. A memorandum of understanding was signed in March 2003 between the Deputy Minister of Health and CARE. Training equipment (e.g., manikins) and emergency medical equipment were procured for 13 MOH and NGO hospitals. Scientific committees were formed in East Jerusalem (Makassed Hospital) and Gaza (MOH, Physicians’ and Nurses’ Syndicates, and a private partner, Target Medical Services, contracted to manage all logistical issues). These scientific committees have overseen curriculum development (adapted from JHU material), course content, and schedules. Trainers have been identified from 14 centers: El Makassed (East Jerusalem); 6 centers in Gaza; and 7 centers (including 5 MOH centers) in the West Bank. Three training centers have already been retained, one in East Jerusalem (El Makassed), two in Gaza, and two more are to be identified in the West Bank (Nablus and Hebron). Course implementation to date is as follows: • East Jerusalem, El Makassed Hospital: TOT in January 2003 (12 trainers trained); TOS in May 2003 (20 students trained); 18 more TOS courses are planned on a bi-monthly basis. • Gaza: TOT in June 2003 (20 senior nurses and physicians trained); 7 TOS are planned bi￾monthly starting in August 2003. • In the West Bank, geopolitical constraints and coordination problems have led to the postponement of the first TOT course until August 2003. A one-day information workshop was held in June 2003 with candidates for the TOT course identified by the MOH. Assessment Team Observations Implementation of this component has been slow for various reasons, but appears to be now on track for the coming months. • The EMED component of EMAP has met numerous delays in its implementation. Some of these delays have been due to the political and international situation. Another factor was the initial focus on developing the SSS and conducting the rapid nutritional assessment (RNA) by an EMAP team, which taxed its resources. • The choice of a TOT approach is a positive step for sustainability, but it probably also contributed to the slow start of the training. 18 (i) CARE International. Progress Report January to March 2003. Emergency Medical Assistance Program. Contract No 294-A-00-01-00130- 00. (ii) CARE International. Progress Report April to June 2003. Emergency Medical Assistance Program. Contract No 294-A-00-01-00130-00. (DRAFT). EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 22 • Having a number of rotating JHU consultants to conduct the TOTs facilitates the difficult task of getting clinicians away from their practice for the period of the training. • Implementation of the EMED training is now starting following the worst of the emergency situation. It is probably best thought of as a long-term clinical emergency care capacity building effort for the Palestinian Health System (PHS) than a response to urgent needs. • Completion of all planned TOS courses depends on an extension of EMAP beyond September 2003. Course organizers indicate that the training schedule can be completed with an extension to March 2004. Given the objective of training 540 students, this is at least conditional on no further delay in TOT and TOS implementation, particularly in the West Bank. EMAP and JHU are to be commended for the quality of the training, in terms of content, methodology, and involvement of the local partners. • Participants in the first TOTs and the members of the scientific committees we interviewed were very positive, if not enthusiastic, about the quality of the training. A review of the curriculum, and interviews conducted reveal that emphasis has been placed not only on technical content but also on the quality of the training methodology. Adult learning techniques, using case management scenarios and skills stations, combining interactive sessions and lectures, provide valuable resources to the trainers. Diffusion of teaching skills inside and outside of the particular content area of EMED is likely to benefit the future students. • JHU/EMAP has been relentless in navigating the Palestinian institutional landscape to maximize buy-in from key stakeholders. The benefit of this approach, in terms of motivation, is apparent particularly with the partners in Gaza. Even in the West Bank, where coordination has been the most challenging, regular contacts between EMAP staff and the MOH have identified communication problems, and will likely pay off in terms of buy-in and sustainability. Things seem to be on track for a first TOT in August 2003. The hospital director in Jenin, for example, was fully aware of the scheduled training. • The involvement of 3 Palestinian Red Crescent Society (PRCS) trainers in the scheduled TOT in the West Bank is a positive initiative, as coordination between pre-hospital and hospital-based emergency care providers is a need recognized by all sides. Impact Evaluation of the Training Poses Specific Challenges. • Evaluation of the trainers: JHU trainers have supported and monitored the first TOS conducted by the local trainers. • Evaluation of the students: the trainers validate the completion of skills stations by the students during the TOS. The TOS exam, which will be used to assess knowledge, has yet to be developed for further trainings. It must be understood that the EMED training component is a necessary but not sufficient step to improving emergency care in actual work settings. All local stakeholders make a natural linkage between the EMED training and the necessity to develop standards or guidelines of care. The EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 23 TOT approach thus opens opportunities that can be built upon. EMAP contributes to developing the emergency care capacity in Palestinian health structures through an effective and reasoned approach. 4. Sub-grants’ Component EMAP’s sub-grants component has successfully supported rehabilitation services, and in particular outreach services, to the neglected disabled Palestinian population. It has been implemented with high standards for selection of grantees based on capacity, effectiveness, and response to needs, and through active support to the grantees, including project monitoring and control procedures. Most grantees have delivered services above their stated targets. While large unmet needs for rehabilitation remain in the disabled population of the WB&G, EMAP is one of the significant players in this area through its local grantees. The sub-grants component has performed above expectations in terms of supporting rehabilitation services to disabled populations. EMAP has expanded its mission to support health care, particularly emergency health care services. Sub-granting to support health care services in the WB&G takes place on a limited scale. It has expanded the availability of emergency services globally (through PRCS) and the delivery of some essential services in a limited number of “hot spots,” particularly around Nablus and Jenin. Beyond this, in the absence of clear program targets, it is difficult to assess whether EMAP is meeting its objectives in this area. On a case-by-case basis, EMAP’s health care sub-grants respect sound standards of quality and appropriateness, but lead to a number of questions about both sustainability and quality of care. These questions are strongly related to the emergency nature of the intervention, but will be felt even after a normalization of the situation in the WB&G. Objectives Objectives of the sub-grant document fall under IR4: Provide temporary financial assistance to Palestinian NGOs currently offering institutional and/or community-based rehabilitation services to physically impaired Palestinians. In terms of rehabilitation services, the expected impact of this document is described in the original proposal documents, both in terms of specific deliverables and broad long-term accomplishments. In terms of immediate deliverables, objectives are identified as follows: • Approximately 10 rehabilitation and/or emergency care service providers will be funded to maintain ongoing services. • Approximately 1,000 additional people with disabilities will be provided with rehabilitation services from sub-recipients of EMAP grants. The quality of these persons’ lives will be EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 24 greatly enhanced. (The latest amendment to EMAP raised this target to 2,700 people with disabilities.) In terms of broader impact, this component’s objectives are the following: improving quality of service provision; increasing the variety of services, especially those relating to community￾centered and rights-based approaches offered by service providers; make the rehabilitation sector of the WB&G more responsive to the felt needs of people with disabilities as new approaches (community-centered and rights-based) are piloted; increasing the number of people with disabilities reintegrated into their communities; and building linkages between primary healthcare, outreach, and CBR organizations and referral centers. As the situation evolved, EMAP broadened its mandate to use the sub-granting mechanism to support health care—primarily emergency health care—delivery in areas affected by the closure. Achievements EMAP has quantitatively more than met its immediate objectives through 15 currently active sub-grants. • Two Requests for Applications (RFAs) were released in December 2001and in December 2002, for both rehabilitation and health care services. • EMAP has made clear efforts to distribute sub-grants through the West Bank and Gaza, with a 60/40 distribution of efforts between the two. In the West Bank, 10 grants have been awarded for $772,932; in Gaza, 5 grants have been awarded in the north, center, and southern sections of the Strip for an amount of $531,864. • The planned 60/40 distribution of effort between Rehabilitation and Health Care services has also been respected with Rehabilitation services receiving $760,798 and Health Care services receiving $519,180. According to CARE’s monitoring data, most sub-grantees are well on their way to reaching their service targets and more often than not have exceeded those targets. Because service targets are project-specific, and each grantee may have more than one type of target (e.g., disabled patients, families, adult patients, victims of Intifada, in-service clients, outreach clients), aggregated data for the sub-grant component are difficult to obtain. A few indicative numbers can be provided:19 EMAP grantees have provided rehabilitation services to 5,187 clients20 (target: 3,159), and emergency/basic health services to 4,575 clients (target 3,550).21 In addition, the extension of EMS services by PRCS Ramallah in WB&G, accounts for a portion of the 9,279 emergencies treated in 8 centers that have expanded their services from 12 to 24 hours/day. As of the beginning of July, 63 percent of the total funds awarded ($1,292,058) have been reimbursed to grantees; it is expected that most grantees will claim reimbursements of all authorized amounts by the end of the grant period (August 31st) with some possibly in need of a no-cost extension. 19 Internal EMAP sub-grant component implementation schedule, revised July 2, 2003 20 As data is compiled from monthly reports, and one person may receive services over more than one month, the number of “clients” for services may be greater than the number of physical persons served. 21 Idem. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 25 A request for application for an audit has been released, and the audit of all sub-grantees will be conducted in the coming month. Beyond these quantitative records of achievements, more specific observations are developed in the next section. Assessment Team Observations General Implementation There has been an obvious and effective effort at spreading the sub-grants throughout the governorates and to be strategic about their selection. A limited number of sub-grantees have an impact on all of the WB&G through their interventions or as a specialized referral center (e.g., PRCS Ramallah for WB&G, PFS-Abu Raya for the West Bank). EMAP has invested considerable energies in selecting 15 proposals for USAID funding (out of the 60 or so received) according to rational standards. EMAP has been appropriately selective in proposing organizations to USAID for funding. In most of the organizations visited by the assessment team, EMAP funds have served one or both22 of the following purposes: • Supporting a gap in capacity created by the situation23 • Increasing—at the margin24—the capability of an organization to expand services for unmet needs in the community.25 This is a positive strategic choice, since these organizations will at some point have to stop relying on emergency funds to cover recurrent costs (this is discussed further). A very selective award process requiring pre-existing organizational capabilities for awarding a grant has advantages and disadvantages. It avoids putting resources into organizations that may be unsustainable, but it also carries the risk that geographic areas of need for services may be ignored, simply because no prior capacity exists to meet them or to apply for a sub-grant. Both EMAP staff and other informants (e.g., World Bank) recognize this fact, and the possibility that some enclosed areas continue to be without basic services. It is, however, difficult to propose a response to this weakness without more precise information and without increasing the concerns about sustainability presented previously. 22 E.g., PFS-Abu Raya, services to patients with spinal cord injuries. 23 E.g., Little Hand Society’s progressive efforts during the past 10 years to increase its financial viability were set back by the Intifada and closures. 24 “At the margin” means that the sub-grantees had a relatively strong prior experience and some basis of financial viability to fall back on after the period of the sub-grant; it does not mean that the financial contribution channeled by EMAP was unsubstantial. It has been quite substantial given the economic situation of these organizations under the closures, incursions, and curfews. 25 E.g., PRCS Ramallah’s extension of Health Posts services from 12 to 24 hours/day; addition of outreach services for a number of rehabilitation service providers. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 26 Monitoring and Support of Grantees by EMAP It is apparent from sites visited and an examination of EMAP documentation that field visits to sub-grantees by EMAP field officers take place regularly, and that monitoring of activities, expenditures, and procedures is rigorous. All the sub-grantees visited by the evaluation team (see Annex E) appeared to pay a high level of attention to financial and activity monitoring, and respect for CARE (USAID) procedures and documentation of business operations. In some cases this attention to detail was quite meticulous and impressive. More than one informant interviewed made side remarks to the evaluators such as, “I’ve learned more through six months with CARE, than through 10 years working in a bank.” Many sub-grantees find the effort needed for documenting, monitoring, and following procedures to be intense; but most recognized the effort as valuable, and sometimes have applied the lessons learned to other interventions. The support provided by EMAP through its field officers, and the capacity building this support provides, are recognized and strongly appreciated. In fact, the support and monitoring provided by the EMAP team to the sub-grantees serves, at least informally, a capacity building function, primarily on management, procedures, and activity monitoring. This capacity building started with proposal writing. EMAP effectively had to respond to concept papers and help potential grantees develop acceptable proposals. Balancing Emergency and Sustainability Six out of 15 sub-grants have been awarded for a total period of less than 6 months. The others cover 12 to 14 months. Important services to the population have been delivered leading to an increase in demand for services. Most organizations have been working at expanding services covered by the grant in a very difficult context, and in spite of some awareness of the unsustainability of this situation, they have not had the time or ability to develop follow-on plans. This is an issue in most of the cases observed, but particularly for the shorter grants. This is possibly one weakness of this component, or at least a critical issue inherent to the emergency situation addressed by the sub-grants. While violence has receded in recent months, access and mobility remain challenging, and the economic situation is largely unchanged. The end of the grants is likely to abruptly stop the delivery of key services and the development of the institutional capacity of some of the grantees, while demand will have been created. In some cases (e.g., Attil Charitable Society, PFS-Yabad), EMAP’s funds have provided much more than marginal or gap-bridging funding to the organizations, and questions about sustainability are going to be critical after the end of the grant. But in both of these examples, monitoring reports show an important increase in services delivered (generally exceeding targets), indicating that the sub-grants have met a true need in the community, and that the organizations have been assessed appropriately as able to deliver the proposed services. The EMAP team is aware of the sustainability concerns raised in such situations, but the team supported the sub-grantees’ applications based on the emergency situation faced by the population. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 27 The selectivity of EMAP in proposing sub-grantees for award—a strength mentioned previously—limits the extent of this problem but cannot avoid it. Impact Objectives The achievement of the broader impact objectives of the sub-grant component cannot be properly evaluated through this assessment. Specific Observations on Rehabilitation Sub-grants The efforts of EMAP to support services to disabled persons through the sub-grant mechanism needs to be put in perspective with the overall burden of disability in the WB&G. 4. An ECHO-funded study26 in the northern and middle areas of Gaza found 5,649 disabled persons out of a population of 357,865. More than one in five disabled persons suffered from multiple disabilities, and almost 9 percent of households contained at least one disabled person. Although this study does not cover all of the WB&G, it provides an idea of the scale of needs. 5. There are few international agencies supporting rehabilitation services to persons with disability on any scale. EMAP has acted as a substantial player in allowing the delivery of services, including home-based services, to disabled persons. While it does not cover all of the Palestinian territory, its effort is spread throughout the WB&G. The identification of grantees with existing capacity has paid off in the relatively rapid implementation of activities. Specific Observation on the Health Care Sub-grants The number of health care grants is limited (five). This raises questions about the efficient level at which this investment is made. Building and maintaining the quality of care delivered by some sub-grantees will be a challenge in the future. Informants interviewed through the assessment exercise (e.g., in Maram, WHO) recognize, however, the ongoing need for the type of emergency response that EMAP has provided to ensure the continuity of emergency and basic services in enclosed areas. 26 White, T. 2002. Disability in Northern and Middle Gaza. A report of field screening and registration; 11 May-26 August 2002. European Commission Humanitarian Aid Office (ECHO); Movimniento por la Paz, el Desarme y la Libertad. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 28 C. Program Management by the Implementing Partner EMAP started and was implemented in difficult conditions, with a limited staff. In collaboration with USAID/WB&G, it was able to be responsive to these difficulties by expanding its human resources and activities. Through the recruitment of field officers qualified in public health, it has developed an important ability to monitor activities, assess needs, provide support to sub￾grantees and other clients, and operate in a decentralized organization. This is of critical importance since freedom of movement continues to be limited. The EMAP team is a dynamic, functional, and motivated organizational structure, well able to deliver the range of services required by the program and to expand its activities if warranted. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 29 IV. Recommendations The main determinant for the prioritization of recommendations from the assessment team will be the evolution of the military-political and economic situations. The faster the situation evolves toward freedom of circulation within the WB&G and a job-creating economy, the more the program should focus on the developmental elements of its approach. Assuming, even in the best of scenarios, it will take some time for this evolution to be felt, the priority for the coming months to year should be to maintain and strengthen the essential emergency response mechanisms, while transitioning to development of other elements of intervention. Based on our findings, the assessment team makes the following recommendations: • In the coming months to year, EMAP needs to maintain a capacity to respond rapidly to new threats and emergencies, due to the unpredictability of the situation, the difficulty of circulation and access to essential goods and services in the WB&G, and the economic situation of the Palestinian population. • EMAP, or a possible extension of EMAP, also needs to support, with method and care, the transition to a more autonomous and functional Palestinian health system. • In the coming months, the SSS needs to be improved and redesigned to address the following methodological issues (At a minimum, however, the system should be maintained until this redesign is completed.): 1. Selection and definition of indicators. This should be done through a general workshop involving all key stakeholders to serve as the first step toward integrating the SSS within the MOH Health Information System. 2. The sampling methodology needs to be reexamined to provide better information on local situations affected by recurrent crises. 3. Data collection should be on a monthly to quarterly basis, depending on the situation. • In terms of procurement and distribution: 1. The manual materials management system in place at EMAP-supported health facility and sub-warehouse locations should be maintained. Sophisticated materials management software is not appropriate and will not be needed, nor will the cost be justified for these facilities, even in the long term. The MS Excel￾based, computerized materials tracking system used at EMAP headquarters/central warehouse is most appropriate and easily maintained. If/when assisted health facilities and sub-warehouses have continuous electricity and a computer, they can also use Excel and e-mail their inventory movements to EMAP headquarters. 2. MOH Medical Stores assessments and subsequent funding for upgrading materials and distribution management should be coordinated and collaborated with other donors working with MOH and health services providers in the WB&G. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 30 3. Medicines, disposables, and medical equipment should be sourced locally, when possible, to encourage local industry and economic recovery. International tendering could be a medium- and long-term detriment to improved health services by denying or curtailing local initiative. • The current EMAP should be extended to allow completion of the current EMED training plans in Gaza, East Jerusalem, and West Bank. • If follow-on activities are developed after EMED, they should focus on institutionalizing best clinical practices through supervision and quality improvement approaches and coordination of care between different levels of the health system (e.g., pre-hospital to emergency room; emergency room to intensive care and other hospital services; referral and counter-referral). • In the coming months to year, EMAP should continue to work with its current rehabilitation sub-grantees and possibly expand to support outreach services to increase capacity building efforts, particularly for the integration of disabled persons in society and the networking/coordination of disability services’ providers. This would effectively prepare the grounds for a long-term strategic effort to address disability in the WB&G beyond the current emergency situation. • In terms of health care delivery, EMAP’s ability to provide resources rapidly in a pointed manner should be maintained in the immediate future, while the economy and access to care issues are still unresolved, and in case new acute emergencies arise. • EMAP should consider implementing an exit strategy before the end of the sub-grant relationship, helping sub-grantees to develop business plans, access additional resources (technical and financial), identify cost-savings, or refocus services. • Continued access problems should be considered the norm, and EMAP should continue to operate in a decentralized manner. There needs to be, however, a regular clarification of roles and pro-active identification of communication and coordination gaps, which are inherent to the decentralized, multi-organizational, and multi-functional structure of the program management. • In the long run, EMAP’s interventions can be a spearhead to addressing a range of public health issues, from rehabilitation to quality of emergency and basic health care, rational drug use, coordination of care, and environmental health issues. EMAP—MTA EMAP-MTA: 294-A-00-01-00130-00 31 V. Annexes 6. A- Preliminary/tentative in-country assessment schedule 7. B- Assessment team bios 8. C- Field guide/questionnaires 9. D- Interviews conducted 10. E- Sites visited 11. F- List of key documents available to assessment team Annex A Annex A A-1 Assessment Team EMAP MID-TERM ASSESSMENT PRELIMINARY WORK SCHEDULE Prior to Arrival ; June 20–27, 2003 USAID sends Documentation to Evaluation Team ; June 24, 2003 Meeting with Michael Van Rooyen, JHU Meeting with Gregg Greenough, JHU Venue: JHU, Baltimore Arrival In Country ; June, July 6, 2003 Arrival in Tel Aviv Check-in in David Intercontinental In-Country Assessment Mon, July 7, 2003 Introductory Meetings at USAID/WB&G ; 8:30–10:00am Briefing by Dr. Suzy Srouji, Senior Health Advisor, and Laila Amara, Program Assistant Venue: USAID, Conf. Rm A ; 10:00–10:30am Courtesy Meeting with Larry Garber, Mission Director Venue: USAID, Mission Director Office ; 10:30–11:30am Meeting with Erin McKee and Reine Joubran, Contracting Office Venue: USAID, Conf. Rm A ; 11:30am–3:00pm Lunch & Review of Files ; 3:00pm Drive to Jerusalem & Check-in to American Colony Hotel in taxi Annex A Annex A A-2 Assessment Team Tues, July 8, 2003 Meeting with CARE International/EMAP ; Whole day Meeting with Ian Willis, Project Manager and EMAP staff present Visit CARE warehouse [Depending on the situation, departure for Gaza] Wed, July 9, 2003 Gaza City and North Gaza ; 6:30am Departure for Gaza ; 10:30am–12:00pm Meeting with MOH, Gaza Meeting with Mr. Walid Shakoura, MOH Director for International Relations; Dr. Ziyad Suliman Shaa’th, MOH General Director for Pharmacy; Ms Rohia Soleiman, MOH USAID Liaison. Venue: MOH, Gaza City Sub-Grantee Component 12:30–1:00pm Drive to Jabalia Rehabilitation Center, North Gaza 1:00–3:00pm Visit Jabalia Center & outreach program 3:00–3:30pm Return to Gaza City 4:00–5:00pm EMED Component Meeting with Mr. Nazem D. Al Sarraj, Target Managing Director; Dr. Sobhi I. Skaik, EMED Physician Course Chairman & Head of Scientific Committee; Mr. Hatem Kraizem, EMED Nurse Course Chairman Venue: Medical Syndicate, Gaza City Check into Adera Hotel Tel: 08/28.38.300 Thurs, July 10, 2003 Gaza City and South Gaza 7:00am Departure for Rafah 9:00–10:30am Site Visit to El Wafa Rehabilitation Center 12:00–13:30pm Site Visit to Maghazi Rehabilitation Center Annex A Annex A A-3 Assessment Team 14:00–15:00pm Visit to 1 or 2 NGOs clinics receiving EMAP kit support, on the way back to Gaza City 15:00–16:00pm -- Visit to ANERA warehouse -- Discussion with EMAP Gaza Representative 16:30pm Departure for Jerusalem Friday, July 11, 2003 Jerusalem: EMAP and Interested Parties [Can possibly be divided between ES and CH] ; 9:00–10:30am Meeting with Dr. Umaiyeh Khammash, Chief of Party, Maram Project; Ellen Coates, Maram M&E Coordinator, & Dr. Rand Salman Venue: Maram Office ; 11:00–12:00pm Meeting with Tom Neu, ANERA Representative Venue: ANERA Office ; 12:30–14:00pm Meeting with WHO and Health Inforum Venue: WHO Office Remainder of Afternoon Free Saturday, July 12, 2003 Free Sun, July 13, 2003 Meetings and site visits in Ramallah ; 8:00am Departure for Ramallah ; 10:00–11:00am Meeting with MOH—West Bank Meeting with Dr. Munzer Sharif, MOH Deputy Minister of Health, and possibly, Dr. Abu-Moghli, Director for International Cooperation Venue: MOH 11:00am–12:30pm Site Visit to Sub-grantee K. Abu Raya Center ; 12:30–1:30pm Meeting with Directors of Central Stores Meeting with Nizar Mazarah, Director for Med. Eqpmt, & with Tamer [Family Name?], Deputy Director for Pharmaceuticals Annex A Annex A A-4 Assessment Team [Agreement with parties on meeting during the morning; timing is flexible] 2:30–4:00pm Meeting with PRCS Headquarters 4:00–5:00pm Visit CARE—Ramallah office and Discussion with Deputy Project Manager Venue: CARE Ramallah Office 5:00pm Return to Jerusalem [If schedule is too busy, non-MOH site visits or meetings should be rescheduled] Mon, July 14, 2003 Meetings in Jerusalem ; 10:00am-12:30pm Meeting with Veerle Coignez, CTO Venue: American Colony 3:30pm Site Visit to Al-Makassed Hospital [To Be Confirmed through CARE] Tuesday, July 15, 2003 Site Visits to Jenin ; Morning Meeting with Abu Ghali, MOH Hospital Director, Jenin [Agreement with party on meeting during the morning; timing is flexible in function of schedule of the day] -- Visit to Sub-grantee PFS-Yabad -- Visit to NGO clinics benefiting from kit support [Exact timing to be determined] Wednesday, July 16, 2003 Meetings in Jerusalem/Ramallah Morning EMAP [EMED Component/SSS Component] ; 3:00–4:00pm Meeting with Prof. Shahin, Center for Development in Primary Health Care, Al-Quds University (Sub-contractor for data gathering/analysis for RNA and SSS) Venue: Al Quds, Ramallah Annex A Annex A A-5 Assessment Team Thursday, July 17, 2003 Site Visits to Nablus -- Visit to Little Hand Society -- PRCS Nablus Friday, July 18, 2003 Open for writing and additional information gathering Saturday-Sunday, July 19-20 Weekend Monday, July 21 Out-briefing to USAID WB&G ; 9:0010:00am Out-briefing to HHA Venue: Conference Room A ; 10:00–11:00am Out-briefing to Larry Garber, Mission Director Venue: Conference Room Upstairs Afternoon Return to Jerusalem Tuesday, July 22 Out-briefing to CARE International Wednesday, July 23 Possible hand-over Draft Evaluation Report Thursday, July 24 Departure to the United States Finalization June 30, 2003 Finalization of draft Report, to be submitted at the latest 5 working days after departure from Tel Aviv August 15, 2003 Submission of Final Report, to be submitted at the latest 5 working days after USAID feedback on Report Annex B Annex B B-1 Assessment Team Assessment Team Lead Evaluator: Eric Sarriot, MD, Ph.D., MPH, DTM&H Dr. Eric Sarriot is an experienced qualitative researcher and evaluator, with firsthand management and organizational development expertise in primary health care, as well as practical clinical experience in emergency care, and work in the Arab world. Dr. Sarriot holds a Ph.D. in Public Health, where his work has focused on using qualitative methods to improve the evaluation of sustainability. He has worked for four years in community health, as researcher and as project manager, in Mauritania. This work involved evaluation and strategic planning with the central, regional, and district levels of the MOH, partnership development and capacity building. He currently works as the capacity development specialist of the Child Survival Technical Support project (CSTS). As such he has led in the use of the Institutional Strength Assessment (ISA) methodology for NGOs currently implementing child survival projects under USAID’s Child Survival and Health Grants Program (CSHGP). In addition to health program management and evaluation expertise, Dr. Sarriot lectures on sustainability planning, design, and evaluation at the Johns Hopkins University Bloomberg School of Public Health, and has written and lectured on quality in supervision systems. He has written the methodological sampling guide to the Knowledge, Practice and Coverage (KPC) survey, a key resource used by NGOs for the baseline and final assessment of child survival grants. Dr. Sarriot also holds an MD from the University of Paris, where he has worked as an emergency care and emergency transport clinician. Second Evaluator: Carl Harris Carl Harris has an extensive experience of management and logistics systems in complex emergencies or transition situations, built over 25 years of work in the Mid-East, eastern, central and southern Africa. His management and assessment experience includes planning, budgeting, grants management and contracts, program development, operations and training. He has worked in health delivery and essential drugs programs. He is experienced in logistics, procurement and assessment, including through work with USAID and UNICEF. He has been an emergency and health logistics advisor for UNICEF, a health delivery manager for the International Rescue Committee, and an emergency operations and food aid analyst for USAID. Annex C Annex C C-1 Questionnaires EMAP - Mid-Term Assessment Field Guide 7/05/2003 Introduction to the Field Guide Questionnaires: The following questions will be used as a field guide to a qualitative inquiry, as opposed to a systematic quantitative data collection tool. They will help collect appropriate information, both quantitative and qualitative, sometime from a single respondent, and sometimes from a series of respondents. Some questions will be answered by already existing documentation that has been or will be collected throughout the assessment process. In this case these questions will not be repeated. Other questions will require the opinion of different respondents in order to be answered satisfactorily. Finally, others will be lead-ins into other questions that will emerge during the assessment process. The Field Guide consists of five questionnaires, corresponding to the three tasks described in the SOW: 1- appropriateness of EMAP as a response to the health situation in the Palestinian Territories (Questionnaire 1); 2- effectiveness of CARE in implementing the four components (Questionnaires 2.1. to 2.4.); 3- management of EMAP by the Implementing Partner (Questionnaire 3). Finally, three sources of information will be tapped into to answer the questions and are summarized at the end of each questionnaire: (1) respondents to interviews; (2) observation notes from visits conducted by the evaluators; (3) documents gathered throughout the assessment. An indicative list of the primary sources of information is provided at the end of each questionnaire. Annex C Annex C C-2 Questionnaires QUESTIONNAIRE 1 Overall appropriateness of EMAP as emergency response program, given the situation in the Palestinian Territories. 1. Are The Five Components Of The EMAP Program Most Appropriate For Emergency Response Within The West Bank And Gaza? How Does EMAP Fits Within The General Question Of Emergency Care In WBG? (Strengths, Limitations, Opportunities For The Future) a. What is the response capacity of the Health System, considering both MOH and NGO capabilities, to the health needs of the Palestinian population? b. What are, intervention by intervention (procurement, health information, training, sub￾grant activities), the specific objectives of EMAP in response to these needs? c. What are the salient unmet urgent needs of the Palestinian Health System (geographic, intervention type, scale) in the foreseeable future? [* The following questions will be addressed through task 2.] 2. Level Of Effort Involved In The Different Components Of The Program In Relation To The Expected Benefits, And To The Needs Recognized By Key Stakeholders In The West Bank And Gaza. d. What are the resources (human, financial, time, physical) dedicated to each EMAP intervention?* e. What is the impact of each EMAP area of intervention?* [*Findings from all the above questions will provide the basis for answering the last two questions:] 3. Efficiency And Complementarity Of EMAP Efforts Vis-A-Vis Other Programs And Stakeholder Interventions. Does The Program Actually Fill Gaps Left By The Palestinian Health Care System Or By The International Ngo And Donor Support Activities? 4. Should Other Types Of Health Emergency Response Activities Be Considered Under This Type Of Funding Mechanism? SOURCES OF INFORMATION: ™ Interviews 1- USAID WB&G, Palestinian MOH, EMAP, WHO 2- Local partners (Maram, PCRS...) ™ Specific Visits/Observation N/A; general observation ™ Documentation - Health services (HIS) - Surveys - Needs assessments - EMAP program monitoring and progress reports Annex C Annex C C-3 Questionnaires QUESTIONNAIRE 2.1. Procurement Component (IR2) Implementation of the procurement component in an effective and efficient way, given the program components, the budget, and the external context. 5. Which USAID procurement regulations are applicable to EMAP? 6. Which department is responsible for orders generation, schedule and quality control? 7. What procedures manuals/guidelines have been developed for EMAP? 8. Which department/s is/are responsible for budget formulation and financial oversight, and what is the frequency of internal/external project audits? 9. What are EMAP’s sources of supply and what is the degree of standardization? 10. Materials requests; how are they generated for the following and what is the lead time? - Essential drugs - Chronic disease drugs and other medications - Medical supplies - Medical equipment - Medical equipment maintenance and repair supplies 11. Materials management and warehousing; give a brief explanation or example of: - Inventory records, Receiving, Dispatching, Dead stock, Backorders, Stock rotation, Expired stock, Losses - Security procedures 12. MIS: what software and hardware is used, what are the institutional links and who shares the information? 13. What is the distribution network and give examples of materials request forms, dispatch forms, and receipt forms? - Explain distribution security procedures. 14. Transport, local: who distributes the materials? 15. Transport, project: ownership, licensing and logbooks? 16. Give examples of customs and shipping forms and procedures and clearing agent contracts. 17. Are the right products always in the right place at the right time at the right price? Explain. Annex C Annex C C-4 Questionnaires Health Facilities 18. Do drugs reach your health facility in a timely manner and in the proper quantities? • What could be done to improve deliveries to your health facility? • Who is responsible for ordering medicine and equipment? • How often are supplies delivered? • If your health facility runs out of specific drugs, how long does it take to re-supply? • Has the EMAP intervention been useful to your health facility? • Are the medical/pharmaceutical kits adequate for your needs? • Is your health facility able to offer better services with the assistance of EMAP than it did in the past? • Which supplies would you like to see added/subtracted from the standard list? • Are health facilities near yours, but not part of EMAP, able to service their catchments adequately? • Are drugs and equipment available in local pharmacies of the same quality as those supplied to you? SOURCES OF INFORMATION: ™ Interviews USAID/WB&G, CARE/EMAP, Local partners and health facilities/hospitals staff members and doctors. Served community members ™ Specific Visits/Observation Health facilities, warehouses, CARE offices ™ Documentation Project reports, needs assessment, surveys, EMAP program monitoring and progress reports Annex C Annex C C-5 Questionnaires QUESTIONNAIRE 2.2. Sub-grant Component (IR4) Implementation of the sub-grant component in an effective and efficient way, given the program components, the budget, and the external context. 19. What Are The Objectives Of Sub-grant Component In Terms Of: a. Services to beneficiaries (rehabilitation; health care)? b. Awareness / social behavioral change vis-a-vis disabilities and insertion of Persons Living With Disabilities (PLWDs)? c. Capacity building of local NGO sub-grantees? d. Geographic coverage? 20. In Terms Of The Implementation Of Sub-grant Component, Describe: e. The extent of needs assessments conducted? What is known about population needs outside of Sub-grantees’ areas of intervention? f. The RFA, selection criteria, and process for awarding grants. g. The amounts disbursed or pending per type of intervention (rehabilitation; health care), and per zone of intervention (WB/G; governorates, special areas if any). h. Type and extent of technical assistance provided to sub-grantees. 21. How Are Activities And Results Being Monitored And Evaluated? 22. In Terms Of The Services Provided By Sub-Grantees: i. How are needs assessments conducted by sub-grantees? What consultations take place to determine the appropriateness of services offered? j. What are the objectives of each sub-grantee? k. What quality standards are used by sub-grantees? How is “client satisfaction” addressed? l. How do populations learn about and access services? 23. In Terms Of Results Obtained By Sub-grantees And The Sub-grant Component: m. What have sub-grantees achieved in terms of type and number of beneficiaries (rehabilitation/health care – children/adults – gender)? n. What is the geographic coverage of services provided by the sub-grants Annex C Annex C C-6 Questionnaires o. What capacity—if any—has been developed in sub-grantees (physical-structural; human; technical; organizational)? p. What effective coordination takes place among sub-grantees? 24. Describe Challenges And Opportunities That Explain The Performance Of The Sub-Grant Component Of Emap. [* Findings from these questions will provide the basis for answering the last two questions:] 25. Is The Focus On Rehabilitation Appropriate, And Are The Selection Criteria Appropriate Or Should They Be Changed? 26. Is Geographic Coverage Appropriate? SOURCES OF INFORMATION: ™ Interviews 1- EMAP; Sub-grantees 2- MOH; other partners ™ Specific Visits/Observation EMAP & Sub-grantee visits ™ Documentation - RFA to sub-grantees - Sub-grants monitoring reports - EMAP progress reports Annex C Annex C C-7 Questionnaires QUESTIONNAIRE 2.3. Sentinel Surveillance System (IR1) Implementation of the Sentinel Surveillance System (SSS) component in an effective and efficient way, given the program components, the budget, and the external context. 27. What Are The Objectives Of The SSS? q. What indicators are collected on infectious diseases, nutrition, access to water, and use of services? r. How do these indicators complement or overlap data from other existing information systems? s. What information was being looked for through surveillance posts? What information is being collected through household (HH) surveys? 28. In Terms Of Implementation Of The SSS: t. Please clarify the sampling plan for the bi-weekly HH surveys (stratification, clustering, sampling frame)? u. In how many governorates is the SSS now in-place? v. How is data collection, entry, and analysis taking place? • Who collects the data? • Who enters the data? • Who conducts the analysis? • What are the strengths/constraints at each step? w. What is the effective turnaround time for collecting, processing and disseminating the information? x. Did the nutrition survey confirm the SSS data on food consumption? 29. In Terms Of Usefulness And Sustainability Of The SSS: y. Who accesses the information from the SSS? z. How is the information provided by the SSS being used? (Who has used the information and to make what decision?) aa. How is stakeholder involvement taking place? In particular what capacity has been built locally? • To collect, • To analyze and • To disseminate the information? Annex C Annex C C-8 Questionnaires bb. What essential health information needs are not being met through the SSS? cc. What are the foreseeable recurrent costs for the SSS to be maintained over time? dd. What commitments do exist for the maintenance of the SSS after August 2003? 30. Describe Challenges And Opportunities That Explain The Performance Of The SSS. SOURCES OF INFORMATION: ™ Interviews 1- EMAP SSS; JHU; Al-Quds; USAID; MOH; WHO 2- MARAM ™ Specific visits/observation EMAP team; Al-Quds ™ Documentation - Health sector bi-weekly reports - EMAP progress reports - MOH, WHO and partner reports Annex C Annex C C-9 Questionnaires QUESTIONNAIRE 2.4. Emergency Medical Education and Development (IR3) Implementation of the sustainable Educational Programs in Trauma Management and Emergency Medicine (EMED) component in an effective and efficient way, given the program components, the budget, and the external context. 31. What Are The Objectives Of EMED? ee. What are the targets: • For TOT activities (number of trainers trained and number of training sessions to be conducted by these trainers)? • For TOS activities? • By training center, referral center and secondary referral center, in WB and Gaza? ff. How was the needs assessment for EMED conducted? gg. How does EMED fit with other emergency care needs, in particular first responders and transporters? How do EMED activities complement other CARE/PRCS activities? 32. In Terms Of Accomplishments Of The EMED Component, How Are EMED Activities Being Realized In Both WB And Gaza? hh. Planning and securing necessary agreements; ii. Curriculum development; jj. Procurement of training equipment; kk. Selection/recruitment of trainers and trainees; ll. Achievements from TOT activities; mm. Achievements from TOS activities; nn. How frequently will trained trainers have the opportunity to actually implement a training? oo. Number of referral centers with trained trainers and trained staff. 33. How Is The EMED component being monitored? pp. Have knowledge acquisition and retention already been assessed? What were the findings? qq. Has clinical practice post-training been assessed? How has it/will it be assessed (indicators and methods)? Annex C Annex C C-10 Questionnaires 34. What Is The Prospect For The Sustainability Of The EMED Component? In Particular, rr. What local institutional linkages have been built through JHU/EMED in WBG? ss. Describe the certification program and its level of advancement. tt. What are the recurring costs and what are the prospects for covering them? uu. How is EMED affecting quality of emergency care in WBG? 35. Describe Challenges And Opportunities That Explain The Performance Of The SSS vv. What has worked ‘well’ so far? What were the key challenges in launching activities and how have they been addressed? What are the remaining constraints? • Length of training vs. effectiveness? • Considering the breadth of training issues, what areas of performance have been improved? And which need further strengthening? ww. How has the ‘pool of consultants’ for EMED worked? SOURCES OF INFORMATION: ™ Interviews 1- EMAP/JHU; MOH; Target 2- USAID; WHO ™ Specific visits/observation Al Makassed; Gaza scientific committee ™ Documentation - EMED needs assessment - EMAP progress report - EMED training curricula - Partner reports Annex C Annex C C-11 Questionnaires QUESTIONNAIRE 3. Strengths and Weaknesses of the CARE/WB&G Team in terms of Project Management. What are the comparative advantages, if any, of CARE? 36. Explain Care’s Management Systems And Capacity, In Relation To The Administration Of Grant activities, internal systems, and relationships with partners, contractors and grantees. • What is the organizational structure, lines of command and communication, and responsibilities of staff? • What is the geographic distribution of CARE staff and systems? • Is their an updated management plan and how often is it reviewed? 37. What are the linkages and distribution of responsibilities with partners, sub-contractors and sub-grantees? • Are the linkages well integrated and coherent. 38. Describe the capacity of staff in the CARE/EMAP program, their professional development and role distribution within the EMAP team; • What staff training has been developed and completed? 39. Describe CARE’s monitoring and evaluation methodology, its implementation, and its effectiveness/use as a management tool for improving performance. • Is the M&E unit integrated with the MIS? 40. Describe EMAP coordination with the main stakeholders, in particular the Palestinian MOH, NGOs, and sub-contractors. 41. Explain CARE’s management of uncertainties, change and constraints in the political context. 42. What professional/management guidance does CARE give to sub-grantees? Does it include ensuring knowledge of USAID regulations? 43. How supportive or constraining have been the sub-contracting procedures regarding implementation of the EMED and the SSS? • Who are the key CARE staff for the implementation of these activities? • SOURCES OF INFORMATION: ™ Interviews USAID/WB&G, CARE, SUB￾CONTRACTORS, MOH ™ Specific visits/observation EMAP, CARE offices ™ Documentation - EMAP progress report - Surveys - CARE reports - Partner reports EMAP program monitoring reports Annex D Annex D D-1 Interviews List of People Interviewed USAID Larry Garber Mission Director Veerle Coignez EMAP CTO Suzy Srouji Senior Health Advisor Laila Amara Program Assistant Reine Joubran Erin McKee Procurement Officer Contracting Officer Melanie Mason Humanitarian Assistance Coordinator EMAP and Implementing Partners Ian Willis EMAP/CARE; Project Manager Ghassan Shakhshir EMAP/CARE; Deputy Project Manager; Ramallah Field Rep. Oscar Pinheiro EMAP/CARE; Procurement and Grants Officer Ramadan Assi EMAP/CARE; Logistician Sawson Batato EMAP/CARE; Pharmacist Irene Siniora EMAP/CARE; Administrative Assistant Nader Abu Al-Hawa EMAP/CARE; Driver Badour Dandies. EMAP/ANERA; Monitoring Officer Maisoun Filfil-Sharif. EMAP/ANERA, Gaza; Field Representative Eman Shublaq EMAP/ANERA, Gaza; Field Officer Mustafa al-Ghusain EMAP/ANERA, Gaza; Warehouse Manager Gregg Greenhough EMAP/JHU; Senior Health Advisor Derek Ehrhardt EMAP/JHU; Health advisor Michael VanRooyen JHU; Director, Center for International Emergency, Disaster & Refugee. Majeda S. Nabhan CARE Human Resource and Administration Manager Thomas Neu ANERA; Middle East Representative Mahammad Shahin Al Quds University – Center for development in Primary Health Care; Director Annex D Annex D D-2 Interviews MOH Fatih Abumoghli MOH Ramallah Health System Development Project; Director for International Relations Project Coordinator Maged Abu-Ramadan MOH Gaza Rawan Tarifi MOH Ramallah; Project Management Sanna Farha MOH Ramallah; Quality Improvement Project Nizar Masalmeh Palestinian National Authority; MOH Medical and Support Stores; Director Abu Ghali MOH Hospital, Jenin; Director Majed Abu-Ramadan Director General International Cooperation-MOH-Gaza Mohammad az-zmeili Stores Director-MOH-Gaza Rohia Soleiman MOH Gaza; USAID Liaison Partners, Sub-grantees and NGO Clinics Nazem D. Al Sarraj Target Medical Services, Gaza; Managing Director & Head of Scientific Committee Sobhi I. Skaik Physician Syndicate, Gaza; EMED Physician and Course Chairman Hatem Kraizem Nurses’ Syndicate, Gaza; EMED Nurse Course Chairman Fadia A. Masri Little Hands Society, Nablus; Chairman of the board & Social Affairs Department Wa’el Qi’dar PRCS Ramallah; Director Rabah Jabr PRCS Ramallah; Rehabilitation Coordinator Silvana Khoury PRCS Ramallah; Planning Unit Khalidah Al-Saifi PRCS Ramallah; International Cooperation Coordinator PRCS Ramallah; EMS Coordinator Ola Diaf Patients Friend Society; K. Abu Raya Rehabilitation Center, Ramallah; Project Coordinator George N. Abo Patients Friend Society; K. Abu Raya Rehabilitation Center, Ramallah; Administrative Director Mohamed Hamdan Hussein Mansour Wael Fteihah Jabalia Center and Outreach Program; Project Coordinator Mohammed Abusamra Al-Asriah Clinic, Jabalia Camp; Physician Akram al-Satari Mazen Farrah El Wafa Rehabilitation Center; Project Director Administrative Assistant Annex D Annex D D-3 Interviews Ali Mansour Raghida Jebril Maghazi Rehabilitation Center; Project Director Project Coordinator Maher Quare'i Jourit Al Lout Clinic, Khan Younis; Physician Abdel Fatah Banour Tareq Qandeel, Mss. Somia Al Salah Ben Clinic, Maghazi refugee camp; Physician Pharmacist Other Health Stakeholders Ellen Coates Maram/URC; M&E Coordinator Rand S-jarallah Maram/ANERA; Senior Health Advisor Herve Razafimbahiny Maram/Pal-Tech; Public Health Director Hassna Dajani Maram/IBM Global Services Ricardo Sole Arques WHO; Health Coordinator Yousef Muhaisen Health Inforum; Project Manager Bart Witteveen ECHO Mirca Barbolini EU – European Commission; Health Task Manager Sima Kana’an The World Bank, WB&G; Deputy Head of Office Hisham Labadi The World Bank, WB&G; Consultant Annex E Annex E E-1 Sites Visited Sites Visited 1- Jerusalem EMAP project CARE warehouse 2- Gaza Jabalia Rehabilitation Center, North Gaza El Wafa Rehabilitation Center, Rafah Maghazi Rehabilitation Center ANERA warehouse, Gaza City Jourit Al Lout Clinic, Khan Younis Al Salah Ben Clinic, Maghazi refugee camp Al-Asriah Clinic, Jabalia Camp 3- West Bank K. Abu Raya Center MOH Central Stores PRCS Headquarters Al-Makassed Hospital Yabad MOH Hospital Jenin Little Hand Society PRCS Nablus (mobile unit), including accompanying one home visit Annex F Annex F F-1 Key Documents Key Documents Reviewed General Documentation on EMAP − Implementation Reports (6) − Copy of MOU with MOH − EMAP Implementation Plan (Yr 2) − EMAP Logframe (Yr 2) − “Briefers”: All components; Sub-grantees; Program amendments − Emergency Development Medical Needs Assessment Report − EMED Curriculum Specific EMAP Documents for Sub-grant Management: − CARE ASSET/INVENTORY SELECTION CRITERIA FORM − CARE ASSET/INVENTORY CONTROL FORM − CARE SALARY CONTROL FORM − REPORTING CHECKLIST − CARE SUBGRANT PROCUREMENT − PROGRESS REPORT INSTRUCTIONS − EMAP REQUEST FOR REIMBURSEMENT FORM − Sub-grantees implementation monitoring (Excell) Specific EMAP Documents for SSS: − HSBR #11, #12 − EMAP and MARAM projects – Household level questionnaire EMAP Manuals, forms and information and monitoring sheets for procurement, logistics and materials management actions. Manuals: 1. Care G&WB Procurement Manual 2. EMAP Inventory Manual (ChapInventory) 3. EMAP Logistics Manual (GIMS WBG Instructions)0 Annex F Annex F F-2 Key Documents Information and Monitoring: 1. Summary of budget monitoring sheet 2. Implementation schedule 3. Progress report template 4. Sub-grants payment checklist 5. Up to date expenses per NGO or MOH 6. Report: Clinic Selection for USAID pharmaceutical and disposable distribution Forms: 1. $5000 sub-grant procurement process 2. inventory ledger 3. Loss adjustment report 4. Reimbursement form 5. Shipping index annex A 6. Stack card-WBG pharm annex F 7. Warehouse supply requisition 8. Waybill annex A 9. Asset registration form\ 10. CARE salary control form 11. Inventory ledger pharmaceuticals 12. Selection criteria form 13. Stack cards annex A 14. Inventory registration forms 15. Inventory asset register 16. Transfer of inventory items 17. Disposal of inventory items 18. Physical count 19. EMAP clinic selection form