Assessment of the Maram Project A program of USAID/WB&G, Strategic Objective 7, implemented by IBM Business Consulting Services Contract No: 294-C-00-01-00110-00 May 3, 2004 ` Prepared by: Eric G. Sarriot, MD, PhD ORC Macro, Lead Evaluator Jonathon S. Rose, MD, MPH MCH Consultant Nassim Nour, MBA Management Consultant This publication was made possible by support from USAID under the terms of Contract No: 294-C-00-04-00203-00. The opinions expressed are those of the authors, and do not necessarily reflect the views of USAID. Acknowledgements The evaluation team would like to thank all who have contributed time, documents and information through this assessment. We would like to particularly acknowledge all those who showed flexibility as our schedule changed drastically the day after our arrival in country. The Maram team has been most flexible, hospitable and willing to share its experience. The Gaza team and its partners, including the MOH, deserve a particular mention for a high level of flexibility, which allowed us to make the most of two very intense days. We would not have done half as much as we did without the help of Hassna Dajani; and we are grateful for the kind and patient assistance of Hazem and Iiad. Acronyms and abbreviations ACNM American College of Nurse Midwives AED Academy for Educational Development ALSO Advanced Life Support in Obstetrics ANERA American Near East Refugee Aid BCC Behavior Change Communications CB/H-IMCI Community Based/Household Integrated Management of Childhood Illnesses CBS Capacity Building and Sustainability CDC Centers for Disease Control CDPHC Center for Development in Primary Health Care CEC Bir Zeit University Continuing Education Center COO Chief Operating Officer COP Chief of Party DCOP Deputy Chief of Party EMAP Emergency Medical Assistance Project EMS Emergency Medical Services GIS Geographic Information System HDIP Health, Development Information & Policy Institute HSBR Health Sector Bi-weekly Report IBM IBM Business Consulting Services IDA Iron Deficiency Anemia INTRAH IntraHealth International, Inc. (formerly Program for International Training in Health) IMCI Integrated Management of Childhood Illnesses IR Intermediate Result LO Local organization MCH Mother and Child Health M&E Monitoring and Evaluation MH Maternity Homes MOE Ministry of Education MOH Ministry of Health MOU Memorandum of Understanding MTA Mid-Term Assessment NGO Non-Governmental Organization OCAT Organizational Capacity Assessment Tool ORC Opinion Research Corporation PAL-TECH Planning and Learning Technologies PHC Primary Health Care PI&T Performance Improvement and Training PIMS Project Implementation Monitoring System PwC PricewaterhouseCoopers RFA Request for Application RFP Request for Proposal RH Reproductive Health RME Research, Monitoring and Evaluation SFO Sa'adi Farrage Orfaly (now SOD) SO Strategic Objective SOD Sa’adi Orfaly Daher (formerly SFO) SSDS Social Sectors Development Strategies, Inc. TIPS Trial of improved practices UNFPA United Nations Fund for Population Assistance UNRWA United Nations Relief and Works Agency URC University Research Co., LLC USAID United States Agency for International Development VAD Vitamin A Deficiency WHO World Health Organization INTRODUCTION Background The United States Agency for International Development’s Mission to the West Bank and Gaza’s (USAID/WBG) program pursues the objective of peace and stability in the region, in particular by assisting the Palestinian Authority (PA) in addressing the public health situation in the West Bank and Gaza. The PA’s five-year program “National Strategic Health Plan for Palestine (1999- 2003)” and the Mission’s subsequent Transition Plan, created in reaction to the outbreak of the Al-Aqsa Intifada in September 2000, have both responded the Palestinians’ self-identified need for improved heath services through a number of emergency and non-emergency delivery projects. As USAID’s flagship maternal and child health program, managed by the Office of Health and Humanitarian Assistance (HHA) of USAID/WBG and launched in June 2001, the Maram project’s main focus was to improve the lives and health of Palestinian families, particularly women and children, through a series of primary, reproductive and nutritional health activities intended to strengthen both service delivery within health facilities and healthy behaviors in the home in a sustainable way. Growing out of the mission’s Pilot Health Project (PHP) initiated in 1999, and initially positioned as an integral part of the larger USAID-financed “Community Services Project”, Maram was created to support the strategic objective of “Healthier Palestinian Families.” In 2002, PricewaterhouseCoopers LLP (PwC), now IBM, won a three-year contract worth a total of $27.7 million to implement Maram. From the beginning, the specific goals of the Maram project were to make measurable improvements in women’s and children’s health status, maternal and neonatal mortality, modern contraception prevalence and longer birth intervals, and improved reproductive health generally. Activities to support these goals included upgrading and equipping relevant health care facilities, training for quality assurance in maternal and child health care and nutrition, behavior change and communication (BCC), data collection and surveys, making grants to local NGOs, and institutional capacity building including Health Management Information Systems (HMIS). Together, these activities translated as a series of interventions in both public and private health systems within specified WBG target areas under the mission’s Strategic Objective 7. At the beginning of military incursions in December 2001, and the resulting disruptions in health care delivery, the Maram Project team, in close association with the Ministry of Health (MoH), shifted its priorities from long-term sector development to short-term emergency activities. This translated into an emergency work plan for the period of May 1-October 31, 2002. These priorities demanded new interventions, which translated into 10 new critical activities, including identifying particularly high-risk communities, providing primary health care services closer to home, procuring immediately essential equipment and commodities, promoting safe practices through media campaigns and community events, and establishing a Maram website. In the post-emergency period, the Maram team has been relying on a new work plan for October 2002-March 2004, which combines on-going emergency support with selected longer-term activities from the original project. Concurrent to adopting the latest work plan, the Maram team submitted the Performance Information Management Systems (PIMS), approved by USAID in December 2002, which serves as the contract’s performance monitoring plan. USAID/WBG tasked ORC Macro with an independent mid-term assessment of the Maram project. This document presents the methodology of assessment and the findings of the assessment team. Methodology This assessment has generally followed a qualitative approach, focusing on processes rather than impact. No attempt has been made to assess the impact activities conducted by Maram have had. It is too soon to do so. Instead, the emphasis has been on examining process and structure, and to assess whether or not there is the potential for significant impact, and in some cases, what that impact might be. The overall evaluation methodology has been essentially qualitative and, accordingly, has used iterative steps to formulate questions, investigate, analyze, and refine questions. Quantitative methods have been used on a limited scale to complement the qualitative data. Financial and commodity management have also been closely examined. Essential methodological tools and steps used are outlined below and are in addition to specific references to methodology made throughout the report: The evaluation team convened in Washington, D.C. during a first week (Feb. 26–March 03, 2004) to review briefing materials, to interview US-based Maram staff, to develop an assessment field guide, and to draft an email survey that was later abandoned at the request of USAID. The evaluators reviewed the briefing material related to the Maram program prior to the interviews and in-country assessment and also collected and annotated additional documents and materials throughout the evaluation. The team also visited or interviewed by phone all the Washington, D.C.-based prime (IBM) and sub-contracting (AED, INTRAH, URC, Pal-Tech, SSDS) Maram partners. Particular attention was paid to the issues of coordination and communications during these interviews. The evaluation questions were developed by building on the RFQ brief, the Maram program information, the briefing material, and interviews with Maram HQ staff. The questionnaire was then sent to USAID WBG SO7 team for review prior to arrival in country. The resulting questionnaire (“Field Guide”), which included many open-ended questions, was used to guide data collection through interviews and additional document reviews. The assessment team arrived in-country on March 21, one day before the assassination of the Hamas leader in Gaza after which a new phase of increased closures restricted the liberty of movement of the evaluation team. Thanks to flexibility on all sides, as travel restrictions permitted, informant interviews and group techniques were combined with field site visits. Key informants (e.g. Maram staff, international organizations, community and beneficiary informants) were visited based on a list developed in consultation with USAID and Maram. During the in-country assessment process, the standard desk review of the program initiated in the US was built upon thorough review of additional documentation as well as interviews from USAID and program staff to MOH staff, subcontractors and sub-grantees, and other stakeholders and key informants. Some of them, particularly high-level managers, provided information that is most appropriately gathered through one-on-one interviews, for which every effort was made to conduct them in person in all four locations (Washington, Tel Aviv, WBG field offices). When that was not possible, phone interviews were conducted. Two additional participatory assessment activities were conducted with all Maram team leaders and senior staff. Participants were asked to identify things that went well in the project, and things that did not. They were also asked to draw an organizational chart of the project. Program evaluations are inherently inaccurate and can not elicit all details. When a qualitative design is the main methodology, relying on the perceptions of key informants, differences of opinion about details and subjective matters are assured. DESIGN AND PERFORMANCE MEASUREMENT PLANNING Overall design To discuss the design of Maram requires examining some of its initial assumptions, its later shift to emergency response, and finally considering how it emerged from this process into its current format. Finally, the assessment team was asked to examine three questions: the level of coordination between Maram and other stakeholders, its influence on the Palestinian health sector, and how it has or not responded to recommendations made at the end of the Pilot Health Project (PHP). Design issues can also be addressed through the lens of the structure of the Maram project, including its numerous sub-contractors, a prime agency known for management expertise rather than technical know-how, and the grants under contract (GUC) mechanism. The three phases of Maram Initial design From the beginning, the specific goals of the Maram project were to make measurable improvements in women’s and children’s health status, maternal and neonatal mortality, modern contraception prevalence and longer birth intervals, and improved reproductive health generally. Activities to support these goals included upgrading and equipping relevant health care facilities, training for quality assurance in maternal and child health care and nutrition, behavior change and communication (BCC), data collection and surveys, making grants to local NGOs, and institutional capacity building including Health Management Information Systems (HMIS). At face value, there was a reasonable rationale for a reproductive health approach integrated with mother and child health interventions. Certainly, needs existed in terms of HMIS development given the relative youth of the Palestinian MOH, while an already dynamic NGO sector could only benefit from capacity and quality approaches. Supporting such interventions through a grants program also would allow stimulating innovation and reaching broadly within Palestinian society. Palestinian authorities sought and still seek to balance meeting health needs of a developing country (with a demographic transition still in process), with a first world culture in terms of expectations, high medicalisation and qualification of its human resources, and a health transition well underway (with an increasing burden of chronic illnesses). Given this and given the ambition of Maram as the leading health investment in West Bank and Gaza, a certain amount of compromise between all these needs was going to be essential. This justified some of the efforts in equipping facilities, as a way to be responsive and supportive to the MOH’s multiple priorities. This inherent complexity of the program made a high level of technical coordination between all Maram’s interventions vital to its success. This coordination needed to be in at least three directions: ÿ externally with the multiple stakeholders of the Palestinian health sector (MOH, NGOs, UNWRA, private sector); ÿ with USAID/WBG/SO7 Team who had developed a contract vehicle to be actively and prominently involved in improving health in West Bank and Gaza; ÿ internally, with the large number of sub-contractors and technical teams that needed to be brought together toward a common vision and integrated strategy. Maram succeeded most in being involved and involving Palestinian stakeholders, primarily by leveraging the high visibility and recognition of its national leadership team (this is discussed further). Unfortunately, it was limited in the two other directions. While, to a large extent, there is mutual recognition and appreciation between the Maram and USAID teams in the field, Maram’s responsiveness to USAID’s requests, at least to the extent desired or expected from the Mission was limited. Finally, the internal coordination and cohesiveness of different technical teams was a challenge for a non-technical prime contractor, even though it provided technically recognized national leaders. It is also important to recognize that Maram, in some way, tried to play ‘catch up’ with the context in which it was implemented even before its onset. The RFP for the Improved Villages and Community Health Services (IVCHS)” project—which became Maram—was developed prior to the Al Aqsa Intifada, which drastically changed the reality on the ground. By the time Maram had started staffing up and presented a first work plan to USAID/WBG (December 2001), the situation had sufficiently degenerated that the project was asked to shift its strategy toward an emergency response to the military incursions, closures, and subsequent disruptions. In the next section, we briefly address how this shift to emergency response affected the program. Shift to emergency The emergency phase of Maram was assessed in two reports already.1 This section only examines important points for the overall understanding of Maram’s operations and accomplishments. 1 Keys MacManus, JD, MPH; Jonathan Haddad, BA, MA. Inter-phase review of the Maram project as part of SO7. November 4–20, 2002. December 10, 2002. Linda H. Archer, PhD. The USAID Maram Project: Review of the Six￾Month Emergency Workplan. December 2002. Maram as a USAID tool of response to the Palestinian Health Sector was requested to shift its activities to emergency response at the end of 2001. The need for this shift was perceived by USAID and Maram’s senior leadership, if not its entire staff. A six-month emergency work plan was accepted in May 2002. Opinions vary about the appropriateness of the total change of direction that was then mandated. The assessment team cannot look back in time and get a perfectly objective picture of the pressures and choices faced by USAID and Maram. Everyone agrees that the situation was getting desperate, that normal operations were disrupted and all agencies had to find balance between long lasting development goals and emergency responsiveness. Some informants however feel that that it was excessive for Maram to abandon totally its development objectives for a response to the emergency. It is however clear that there has been a high opportunity cost of having totally shifted activities to emergency. Maram, as a heavy multi-partner structure, seems to never have recovered from the level of flexibility that was demanded of it. Additionally, while Maram’s response to the emergency was appreciated by the MOH and partners, its emergency response was slow. Some interventions initiated under the ‘emergency phase’ were actually being completed just before the visit of the assessment team, almost 18 months after the official end of this phase. By the time the dust settled, Maram had given up on some of its interventions and could not reconcile some of its initial strategies carried out by different subcontractors. Injury prevention and adolescent health were officially dropped. Family planning was also dropped for all practical purposes, if not officially. Unfortunately, Quality Assurance disappeared from the approaches to be developed by Maram. The MTA team could never form a precise picture of why exactly this was. USAID deplores that this component of the project was never implemented. URC, the agency responsible for its implementation, feels equally disappointed that it lost the opportunity to work in its true area of expertise, instead of being “confined” to the M&E task. Intrah, which leads the PI&T work, had collaborated with URC in developing a model, which combined the experience of the two groups, but recognizes that after the emergency phase roles were redefined and collaboration never started again. Current phase: October 2002 – March 2004 After the emergency phase, the project did not return to the IRs that was accepted prior to the start of that phase. The USAID Mission reports that this issue was discussed in the second year strategic planning workshop in the fall of 2002. Because of the limited time remaining under the base period it was felt by both USAID and by many people in Maram that the project had to limit its focus, as it would not be able to address all the originally approved components and see achievable results. FINDINGS AND LESSONS LEARNED BY COMPONENT OF INTERVENTION Introduction to the findings Context and overview It is impossible to properly assess the accomplishments of Maram and the way it has conducted its efforts, without due consideration of the context in which it has been implemented. As stated previously, Maram was initially conceived of at a time of very high hopes and expectations for the future of the Palestinian-Israeli situation. All this changed with the beginning of the new Al Aqsa Intifada. The theme of irrationality has struck the assessment team as pertinent to the context for Maram: ÿ First of all, the general context of occupation, ‘curfews’, checkpoints, and closures is highly irrational, when considering that the responsibility for the welfare of the Palestinian population still rests with a Palestinian Authority who has no direct control over the conditions in which health care is delivered. ÿ This context has evolved constantly, from month-to-month and day-to-day, at times for the best, more often than not toward more violence and more restriction of movement from the Palestinian population and the project staff. While these first elements are external, some elements of irrationality also relate to Maram’s internal structure: ÿ The designed structure of Maram was original but also untested. The lead agency had limited technical expertise and no real presence on which to build in the West Bank and Gaza. ÿ Maram went through repeated changes in direction, being asked to go in and then out of emergency response mode, leaving it with less than 18 months continuous work on its development plans in the current phase. ÿ Finally, as a major contract, Maram is one of the main mechanisms for USAID/WBG involvement in the West Bank and Gaza. For this reason, it has operated with close involvement by USAID, with the caveat that most USAID/WBG staff has limited if not no access at all to the field. This presents an additional difficulty. In spite of all this, Maram is seen by many partners as a ‘heavy weight’ in the health sector, led by a “dedicated team” of national cadres. It has been a recognized channel for USAID/WBG responsiveness through both ‘hardware’ (equipping facilities, maternity homes; responding to emergencies), and ‘software,’ in the form of training, capacity building, research and advocacy. Most of its achievements will be seen in terms of capacity building at one level or another, and also in seizing opportunities notably in advancing a national micro-nutrients policy and in focusing some attention on the maternal and neonatal health through its birth-cycle focus. Some of the main weaknesses found in Maram point to an over-emphasis on opportunities instead of strategy, and an insufficient integration of its management and activities. Overall, Maram has been an expensive and expansive project, but only operated with some modicum of consistency for less than 18 months since the end of the emergency phase. Maternity Homes (MH) As the case has been made previously, Maram focused on different interventions, sometimes integrated and more often not. In terms of technical interventions, it has had a clear emphasis on nutrition and maternal/neonatal health, but not a Child Health strategy per se. Its most integrated technical strategy has formed around the Maternity Homes (MH), which are discussed here. Three Maternity Homes have been fostered by Maram through contracts, all in the last four months. A fourth, in Maythaloun has not yet opened. The concept was developed in response to the fact that some pregnant women are not able to get to hospital in time due to checkpoint delays and outright prevention of passage. The MHs are intended to provide antenatal, postnatal and primary delivery care to women with normal (no-risk) pregnancies. If a risk for complications is detected in the antenatal period then the women is diverted to the backup hospital with whom the MH has a contractual and financial arrangement to provide services to the women. If the woman is experiencing a complication and is not able to get to the hospital, due for example to closures, the MH is not capable of caring for the women. At this time, there appears to be no solution to such a situation. An inspection of equipment, pharmaceuticals, common procedures, training, staffing, availability, and backup procedures, and an inquiry of the number of infant and maternal deaths in each clinic were performed by the evaluation team. In general, all three appear to be capable of providing the antenatal, postnatal and primary delivery care services they promise. To date there have been no maternal or neonatal deaths. The number of deliveries in each home has varied (see Table 1). Table 1: Active Maternity Homes (MH) supported by Maram Maternity Home Month Opened Number of Births Since Opening Birzeit (West Bank) December 2003 4 Aqraba (West Bank) January 2004 60 Balsam (Gaza) January 2004 150 The Aqraba and Birzeit MHs are in medical facilities that have been around for some time, although they both lack an operating theater. The Al-Nada MH (Balsam) is in a hospital that has also existed for some time and has an operating theater. It remains to be seen if women continue to prefer to deliver in a hospital where there is an operating theater to handle complications, but given the current preference for hospitals it should be tracked as an aid to assessing the validity of the MH concept. Selection The selection criteria used by Maram to choose where to place a maternity home do not always take into account a definition of the most vulnerable women and children living in high risk villages (see list below). In fact, some of the criteria, however worthy they may be, seem to contradict this. Isolation from services, for example, may be found to be consistent with a lack of staff availability in the community. Maram seems to have opted instead for a compromise between serving the communities with the greatest need (as prescribed in the contract), and implementing this innovation where feasibility may be high. Additionally, if a country is in the process of building a government, as is the case in Palestine, then efforts must be made to establish the capability of the MOH to provide meaningful guidance and even approval of some health related matters. This is attempted by Maram through the licensing process by the MOH. In conflict areas, however, governments are often inherently unstable. Reliance on approval by any Ministry of Health opens the risk, however remote, of political motivation supplanting need-based criteria. Given the fact that the MHs are independent centers the need for such approval would seem negotiable, especially in a conflict area where there is increased unpredictability. Nevertheless, geographic location should take into account a lack of accessibility to a hospital; but in practice it does not, or not always. That is, a MH should be placed where it is often difficult to obtain antenatal and primary delivery services. Surely a women living in the catchment area of the Balsam Hospital, which has had a fully functioning obstetrics and gynecology department for some time, should be able to get to the hospital in which the MH is located. In practice, the choice to foster the Balsam MH is particularly not consistent with the criteria of addressing the needs of a pregnant women isolated from hospital services. Maternity Home Selection Criteria Used by Maram: ÿ Selection based on the geographical area. ÿ Catchment area in terms of population size. ÿ Accessibility to the services by beneficiaries as well as providers. ÿ Community commitment to participate and sustain. ÿ Organizational commitment. ÿ The availability of home deliveries in the area. ÿ Staff availability in the community. ÿ Endorsement by the Ministry of Health. ÿ To build on the available services and not to compete. Assuring Quality of Care Maram is planning to assure the quality of care at the MHs by development and adoption of protocols, medical records and licensing of the MH. In addition to linkage to a backup hospital, participation in internationally recognized training programs (i.e. A.L.S.O.), conducting best practices workshops and coordinating with other NGOs. To date, most of this is still in the planning or very early implementation stage. The assessment team feels strongly that ensuring best practices in the long term will depend on an active and lasting effort focused on promoting antenatal care, ensuring child birth safety and quality of care to the client. The MH concept could become a vehicle through which to address important issues of quality of care. This point has been stressed by many informants both in and out of Maram, including UNFPA. In the long-term there will continue to be a need to improve the quality of antenatal care and childbirth, and the MHs could be a useful tool to advance this. While they do not solve the problem of access to a hospital when a checkpoint blocks access by women in the West Bank, if they serve to increase the frequency and—more importantly the continuity—of antenatal care, they may serve a valuable purpose in early detection and referral of complicated pregnancies. At least in their catchment area, this would limit the negotiation of checkpoint crossings to women most at need of access to a hospital. Effectiveness The MTA team feels that the MH model is valid as an approach to improving quality of antenatal care and safe delivery of normal pregnancies. Because of the built-in linkages with backup hospitals, it has potential for advancing referral systems and promoting quality at a system level. While effectiveness and sustainability still have to be demonstrated, promoting quality of care for mothers and newborns is ample justification for this approach. Whether the MHs respond to the challenge of access to appropriate obstetrics services for complicated deliveries is an unanswered question. Finally, local organizations supporting the MHs have been somewhat reluctant to mobilize communities for increasing antenatal care, partly because of the uncertainty about Maram’s support beyond April 2004. Maram has not established a health information system for the MHs and therefore actual effectiveness of its investment will not be measured by Maram. Sustainability Unfortunately, all three MHs lack sufficient plans for sustainability. Given the cost of each delivery, the projected number of deliveries in one year’s time, and the amount charged per delivery, sustainability will not occur based on user fees alone. All three benefits from the other medical services they provide, from a health insurance system for the Al-Nada MH (Balsam), and possibly from subsidies by the supporting NGOs in Aqraba and Birzeit. Indeed, the Balsam MH was opened with an eye toward attracting more patients to the hospital in general, and the MH will be sustained in large part by the hospital. The MH in Aqraba will follow a similar model but incorporates some user fees, while the Birzeit MH will be dependent upon user fees and fundraising. In terms of a sustainability plan for the Maternity Homes, Maram presented the evaluation team with a Power Point presentation used to draw the attention of the Birzeit Women’s Charitable Society to this issue. Conceptually at least, the presentation identifies relevant factors of sustainability, such as quality of care and the ability to attract a clientele, projection figures about the number of deliveries that can be expected, from which a cost recovery plan could be derived. For example, the Power Point Presentation establishes that the population served by Birzeit Maternity Home is about 32,000, and that 26.1 births / 1,000 pop. can be expected annually. From this, it contends that the maximum number of deliveries that can be expected for Birzeit is 835/year. Cost implications suggest that the Charitable Society may have to consider subsidizing Maternity Home services over the long run. Maram also plans to train and advise its partners on fundraising and proposal writing. The three Maternity Homes visited by the assessment team are supported by organizations that have a proven capacity to bring financial resources from different sources. As of yet, however, there is no clearly articulated and available sustainability plan for any of the maternities. Scalability In the case of the MHs, scaling up means increasing the number of MHs. Inherent in the question of scaling up is whether or not the MH is effective and sustainable and therefore should be scaled up. Certainly it is too early to tell whether this is the case. Simply having the financial ability to increase the numbers of MHs should not be the only criteria upon which such a decision is made. Summary of findings and lessons learned: ÿ A satisfactory solution to women being prevented from delivering at a hospital may not yet have been found, and may never be found as long as military occupation is in effect. ÿ Maternity Homes can be defended under two rationales: one as a response to closures and poor accessibility; the other as a way to improve quality of childbirth and antenatal care. They incompletely answer the concerns of the former rationale, but may be a worthy contribution to the latter. ÿ Maram has not established a health information system for the MHs and therefore actual effectiveness of its investment will not be measured by Maram. In general, it is too early to assess the effectiveness of the Maternity Homes at this point so there is still time to establish a health information system or to link impact data from MHs to a management information system. ÿ Selection criteria do not appear to take into account a definition of the most vulnerable women and children living in high-risk villages. ÿ Geographic location does not always take into account a lack of accessibility to a hospital. ÿ Sustainability of the Maternity Homes is still uncertain. ÿ It is too early to tell if the Maternity Homes are effective enough to warrant scaling up. Nutrition The need to address maternal and child nutrition is the most consistent theme in the project from the original proposal to the present. Indeed, it is the only specific topic to have been preserved through all three iterations of the project’s objective framework. The micro-nutrients approach is understood as very systematic, from research, involvement of policy makers, to policy changes. Maram has become involved in nutrition and micro-nutriments through a range of activities discussed below. In particular, it has brought in recognized experts who have brought both technical state of the art knowledge and high visibility to Maram’s efforts. This technical expertise has also been used at least in one case to support an effort involving the private sector (Maram provided technical guidance, through an expert in nutrition, to ANERA as it worked independently with a private company (Sinnokrot) to produce fortified biscuits.) School Nutrition The Ministry of Education’s school based micronutrient supplementation project has been supported by Maram. The initiative initially targets 26,000 school children in all grades and is expected to be scaled up by the Ministry to reach 1.2 million school children in the 2004-2005 school year. The program focuses on providing a once weekly vitamin supplement tablet containing vitamins A, B-2, B-12, and C, along with folic acid, iron, and zinc all of which are inline with best practices for child survival programs and reflective of micronutrient deficiencies identified in a study conducted by Johns Hopkins University for the EMAP project. Teachers, school administrators, and trainers are receiving training and education in support of the program. In addition, there is a monitoring program, and children and their families are receiving focused, age appropriate nutritional health messages. Maram has been providing support for the BCC effort by producing and providing training and training guides, and brochures targeting children’s behavior. Conference in Amman Jordan: Nutrition Summit The January 2004 Conference, “Nutrition Programs: Challenges & Opportunities”, held in Amman, Jordan is perhaps the most widely mentioned event concerning Maram and nutrition. Maram hosted and facilitated the attendance to this regional conference on nutrition and evidence-based approaches to nutrition policy and program design, implementation and evaluation. In short, Maram conducted a rather large-scale capacity building event around the issue of nutrition. The key objectives of the conference were to facilitate exposure to the current status of and challenges to the Palestinian nutrition sector, and approaches to and lessons learned from state-of￾the-art nutrition policies and programs from the regional and international arenas. It was also intended to promote support for the Palestinian national nutrition strategy, and facilitate development of recommendations that would support implementation of that strategy, (which was developed in response to internationally endorsed research indicating acute and chronic malnutrition of rising proportions in the Palestinian population). Of particular emphasis during the presentations and plenary and working group discussions were issues related to adequate micronutrient intake throughout the life cycle, including the importance of appropriate breastfeeding, supplementation and fortification program challenges and successes, overweight and underweight, and the critical contributions that evidence-based policies make to successful nutrition programs at the service delivery, community and household levels. Maram reports the following four key outcomes of the conference: 1. The Palestinian MOH’s formal adoption of the national nutrition strategy; 2. Unspecified donor agency representatives’ commitment to include nutrition in their health agendas; 3. Acceptance of and commitment to implementation of the recommendations developed by the conference working groups on micronutrient adequacy, breastfeeding, and overweight/underweight; 4. Participants’ commitment to develop or strengthen growth monitoring services, provider training programs promoting adherence to national and international standards and protocols for nutrition service delivery and counseling, school-based nutrition education programs, and a Palestinian nutrition surveillance program to monitor nutrition status and program effectiveness. A full report on the conference was not available at the time of the assessment. The adoption of the national nutrition strategy by the MOH may actually have been achieved prior to the conference.2 MOH informants did however present the Amman conference as a highlight of Maram, and attributed the adoption of the national nutrition strategy to the project’s efforts. Nutrition Research Maram has conducted or been a part of two nutritional research projects: the Prevalence of Iron Deficiency Anemia Amongst School Children study and the Vitamin A Deficiency Study. Trials of Improved Practices in Palestine (TIPS) TIPs research tests current nutritional practices that are acceptable and feasible for families in their homes before they are recommended in programs. These tested practices are then used to develop strategies and communication messages for behavior change. TIPS is a reasonably innovative approach to promoting and studying behavior change processes, which has gained increasing interest and recognition in the health promotion field in recent years. TIPS is supposed to be a research activity and a report should have been available. No report of the TIPS research seems to be available. A PowerPoint presentation was provided to the evaluation team but does not include data or data analysis so there is no way to evaluate the claims made in the conclusions presented in the Power Point presentation. Maram claims in the Power Point presentation that the TIPS research in Palestine demonstrated that program recommendations regarding improving iron intake and its absorption are feasible; and that women can change their dietary behavior regarding increasing iron intake and absorption when given attention and support. Offering options, discussing side effects, counseling and advice were found to be key tools in changing behavior. Nutrition Protocols In response to the Johns Hopkins study that highlighted the effect closures are having on the nutritional status of women and children and the associated diseases, and the Ministry of Health’s recommendation of priorities, Maram initiated the development of nutrition protocols for service providers. These protocols, which remain in the review process at the time of the evaluation and have therefore not been evaluated. They address the following: nutritional status assessment of women and children, diagnosis, treatment, and counseling. They include issues such as breastfeeding, growth monitoring, vitamin A, and other micronutrients. Procurement Maram also procured nearly half a million dollars worth of multi-vitamins and iron supplements, which were distributed through its network of facilities as a one-time activity. 2 The MTA team did not ask to see the official letter from the MOH and did not see any written report of the adoption. By itself this procurement could be deemed useful and consistent with reasonable nutritional goals, but it is not clear why it was only a one-time event, and as such if the goals are being served in a manner that is sustainable. It seems to be another opportunity seized by Maram to be responsive to authentic needs, in an effort however dissociated from any clearly articulated strategy. Summary of findings and lessons learned: ÿ The Ministry of Education’s school based micronutrient supplementation project has been supported by Maram. ÿ Maram conducted a rather large-scale capacity building event around the issue of nutrition— the nutrition conference in Amman Jordan—that has helped facilitate advancement in meeting the micronutrient needs of children in Palestine. ÿ Research conducted or supported by Maram has provided baseline data by which to measure the impact of future interventions in the area of nutritional status of women and children in Palestine. ÿ The micro-nutrients approach is understood as very systematic, from research, involvement of policy makers, to policy changes. ÿ Maram has led or been involved in nutritional research that is useful to the country (e.g. Vit. A deficiency study). ÿ There is no way to evaluate the TIPS research. The nutrition protocols are still under review. Institutional Capacity Building The purpose of activities implemented by the Capacity Building and Sustainability (CB&S) team was to support capacity building in financial and general management for the Maram-supported MOH and NGO clinics, Maternity Homes and NGOs serving the health sector, with a focus on Maram grantees and local subcontractors. The CB&S team did not have specific funds outside of grants funds. It initially worked with the Grants team to provide pre-selection workshops, financial reporting workshops, and troubleshooting workshops to applicants and grantees (see grants section: two workshops were held: the first in Ramallah, with seven organizations participating, and the second in Gaza, with five organizations participating.) As with many Maram activities, this suffered because of the delays in the grants component. Maram staff carries an appropriate level of qualification for this activity and are knowledgeable about the challenges of organizational development, its tools and methods. In the last quarter of 2003, the Center for Continuing Education (CCE) at Birzeit University was contracted for the development of curricula and training in general management and financial management, and to conduct training in strategic planning for health sector NGOs. The CB&S team has played an active role in supervising and monitoring the development of the tools. With CCE, the team has tried to develop a systematic approach to institutional capacity building, based on: ÿ Initial organizational needs assessment; ÿ Financial Planning and Management training; ÿ Strategic Planning training and coaching to a limited number of organizations; ÿ Fund Raising and Proposal Writing training. The information available to the assessment team on these different steps is analyzed as follows: Needs Assessment An organizational needs assessment is conducted with participating organizations, using the Organizational Capacity Assessment Tool (OCAT). IBM played a role in identifying the tool, which is commonly used by NGOs in the health sector and suggested as a valuable resource in a USAID publication3 . The OCAT was used to assess the training and developmental needs of the following Maram grantees, contractors (including MH)4 : ÿ Center for Development in Primary Health Care (CDPHC), West Bank. ÿ Al-Nada maternity home, Gaza ÿ Al-Karmel Cultural Association, Gaza ÿ Al-Bait Al-Sa’eed, Gaza ÿ Al Lod Charitable Society, Nablus ÿ The Young Artists Forum, Ramallah ÿ Al-Kassaba Theater, Ramallah ÿ Kalandia Camp Women Cooperative, Kalandia ÿ Payalara, Ramallah ÿ Aqraba society ÿ Juzoor ÿ Skaka women society ÿ Center for sustainable development and community health ÿ Center for Continuing Education (CCE), Bir Zeit ** ÿ Bir Zeit Zoman society ÿ Physicians (Syndicate) ÿ PFS clinic Jenin (backup hospital for Maithaloun MH) ÿ Culture and Free Thought Association ÿ Sanabel Theater Group, Abu Tor + **: CCE is the only organization that may have used the OCAT as a self-assessment +: withdrew from work with Maram after requirement to sign Certificate of Anti-terrorism The report from Birzeit University on the needs assessment is not strong in methodological discussions as in analyses. It identifies the areas of need, but has inadequate discussions of bias or limitations in the data collection. The report actually underestimates the tool that was used, and generally seems rapidly finalized. The tool did not serve an organizational development purpose to the grantees or contractors that the assessment team met. They did not seem to retain much from the OCAT exercise. Maram’s 3 USAID Center for Development Information and Evaluation’s TIP N.15 “Measuring Institutional Capacity” http://www.dec.org/evals.cfm#1 4 Needs Assessment Report, Center for Continuing Education, Birzeit University. March 1, 2004 CB&S team has an appropriate analysis of the reason for this. These reasons are inherent to using the tool as a rapid external needs assessment tool, rather than an internal organizational assessment. It is unfortunate that this is not discussed in the CCE report, as this might indicate a perfunctory use of the OCAT tool, rather than the kind of critical experience building, which the CB&S team seemed to be sensitive to in its discussion of it. In spite of this limitation, the use of the OCAT is a positive approach in order to be systematic in approaching partner organizations. Management and Financial Management training As most activities in this component, the management and financial management training activities had really just started at the time of the assessment. Birzeit University also developed manuals for trainers and trainees in both training. Grantees and subcontractors started being trained in 2004, and the CB&S team expects that more than the target of 60 trainees in the West Bank and 60 in Gaza will be trained by April 2004. Some grantees and subcontractors met by the assessment team had benefited from the first series of training. They reportedly appreciated the quality of the training, and thought it would benefit their organization. But the timing of the training, in the midst of a rushed granting or contracting process was often not perceived as very appropriate. Strategic planning Maram used the services of the Drucker Foundation to train its trainers in strategic planning. Birzeit University translated and adapted the training guide in Arabic. At the time of assessment, five trainers have been trained, and training started in February in both Gaza and West Bank. Three organizations in Gaza and seven in the West Bank are to benefit from individualized strategic planning consulting. These include all four organizations implementing Maternity Homes, and a number of BCC grantees and contractors. This activity has however not yet started. The assessment team heard both positive and negative comments about the training. Generally, organizations slated to benefit from the consulting services as a follow-on to the training were more positive than others, probably because they were expecting to see a more practical application in situ of the strategic planning principles. The organizations supporting the Maternity Home concepts were generally concerned that this activity, as positive as it may be, was ill-timed considering the short duration of their contract with Maram. The concept of individual organizational coaching is however one that appears very promising to the assessment team, although it is too early to assess its benefit. Overall institutional capacity building effort In addition to these efforts, institutional capacity building has also taken place through the close interaction between Maram CB&S and Grants teams and grantees. Grantees report the high level of attention they have received from Maram, and frequently report on the value of the support they have received. While some point to the difficulty of complying with the large number of forms and requirements conveyed by Maram, many acknowledge a value added to their organization in terms of planning, management and organization. This stems particularly from the close interactions in the field, possibly much more than from the training workshops, which have just started. Additional capacity building is provided by Maram in a range of technical areas, particularly through the BCC, RME, and PI&T teams. This is discussed in the relevant sections. From discussions with grantees, in particular BCC grantees, and with Maternity Home associations, it seems that it is particularly the combination of technical support, including through grants or contract negotiation, management training, and availability of Maram staff for ongoing support, which carries the greatest potential for institutional development. Although USAID already supports capacity building of local NGO through other projects (e.g. Tamkeen), the value added by Maram is the focus on organizations addressing health, and the integration between technical and organizational development. This sort of benefit requires time to be established and cannot be properly assessed at this point, except through the observation that organizations working with the PI&T or BCC teams for example seem to have gained some focus and are learning by doing as activities are being implemented. The assessment team concludes that an active institutional capacity building component is of great value to Maram’s long-term objectives for the Palestinian health sector. Two things will be needed to bring this investment to fruition: ÿ field support to partner organizations, continuing not to rely on training alone, but also on coaching or mentoring organizations, ÿ a systematic monitoring of organizational performance and organizational change, which is still insufficient. The planned strategic planning activities could provide a basis to promote organizational assessment as a development tool (and not simply an external requirement), and to define organizational indicators that Maram can track with its partners. Only as this develops can sustainability be legitimately considered to be a part of the Maram institutional strategy. (For the moment, the Maternity Homes for example have no sustainability plan. A concept has been presented – see section on Maternity Homes.) Summary of findings and lessons learned: ÿ As many components, the CB&S team has effectively started institutional capacity building activities quite late in the life of Maram, in part due to delays in the Grants component. ÿ The CB&S team is composed of competent staff, and also relies extensively on Birzeit University’s Center for Continuing Education (CCE), which is headed by an experienced management professional. ÿ Maram has developed a sound approach to institutional capacity building on first examination, but its effectiveness cannot be assessed yet. ÿ The reliance on systematic needs assessment, training, individualized organizational support and mentoring, combined with the collaboration on technical issues also appears sound. ÿ Institutional capacity building efforts of Maram have involved to a greater or lesser extent all project teams, from CB&S, Grants, as well as from the technical teams. ÿ If this approach is maintained and properly monitored, it could prove of great significance in advancing the sustainability of Maram’s investments in partner organizations. Behavior Change Communication (BCC) The overall objective of Maram in Behavior Change Communication (BCC) is to “develop and disseminate effective and persuasive messages to vulnerable groups and those who influence them in West Bank and Gaza; using a variety of media, voices, and channels to promote Maram's agreed upon Best Practices.” Its secondary objective is “to increase the capacity of these groups to develop and deliver effective communication for public health behavior change.” To a large extent, the case can be made that Maram is being successful at achieving these objectives, although it cannot claim to be aiming or achieving public health impact for the moment, beyond the positive ‘success stories’ highlighted in its reports. Maram has supported the development of a large number of quality BCC productions, and built capacity in Palestinian local organizations to deliver these productions to the public in West Bank and in Gaza. Reach of Maram’s BCC efforts Table 2, below, shows the number of grantees and geographic focus of their efforts through three main delivery strategies: ÿ “Pump up the volume” activities focus on multi-media efforts; ÿ “Creative arts” activities encourage a diversified cultural expression of health themes, for example through art exhibits; ÿ As its name indicates “Child-to-Child” activities involve children in health communication through peer approaches. As Table 2 indicates, grantees are usually working in some of the more marginalized communities of WBG. Table 2: Number of Maram BCC sub-grants by approach and geographic focus West Bank Gaza Pump up the volume 2 grants 9 marginalized communities in West Bank; + 15 hard to reach villages in Ramallah district and 2 refugee camps 2 grants Khan Younis; Nussirat; Gaza city; and Jabalia; 10 deprived communities in Rafah Creative arts 2 grants 18 communities in Nablus area; 3 areas Hebron, Jenin, Ramallah & surrounding villages 0 grants Child-to-Child 1 grant 20 villages and 2 refugee camps near Ramallah 2 grants 4 communities in the middle zone of Gaza Strip; and 8 communities in South Gaza In addition to work with grantees, Maram has supported a number of media productions (most of which have not yet been broadcast) through four subcontracts. Some of the productions are for TV & Radio, but also counseling cards, take home leaflets, posters, audio program, brochures, and an interactive theatre show. In Gaza, the Hayat campaign addressed the birth cycle and nutrition and involved 152 organizations in the dissemination of a series of three newsprints. Research conducted after the outset of some of these efforts found that radio spots have high recall (listeners remember the messages appropriately) but low reach (only about 12% of the target population listens to the radio medium). This has led the BCC team to orient its efforts toward the use of more interpersonal channels of communication. Plans to use audio and video cassettes to overcome the low reach of broadcast media are being developed. Unfortunately, this might also reflect a poor sequencing of activities, starting with baseline assessments, and definition of strategic steps. Maram may also have felt victim to its own desire to achieve deliverables within a short timeframe once activities got under way. According to the Grants Team monitoring reports, the total reach of BCC grantees for the period from Oct. 2002 to Feb. 2004 is almost 116,000 information contacts. It seems that the cumulative target of 200,000 contacts of the nine sub-grants is likely to be achieved and surpassed by the end of April. Capacity building for health communication in local organizations The most important work of Maram in BCC appears to be in building the capacity of grantees (and to a lesser extent contractors) in health communication. There was disagreement between the grants committee and the BCC team on the need to expand the number of grantees. Different opinions can be heard within Maram and within its BCC team. Some feel that the trade off between having more impact and building more capacity—notably through increasing the number of grantees—has been counter-productive in terms of impact; while others emphasize that the purpose privileged capacity building from the onset. Grantees express appreciation for the training and coaching they have received. One grantee stated willingly: “this is the first donor with a clear agenda and a strong strategy.” Another, referring particularly to technical capacity to design effective communication campaigns and discussing the implications of the possible end of Maram stated: “In the future we would miss the capacity building.” Maram has been able to offer training in the use of the BEHAVE framework to some grantees through its own trainers (the BCC team includes three trainers). Child to child is a new communication approach to many organizations, and Maram is to be commended for identifying some organizations that could advance this experience, particularly in Gaza. The grantees met by the assessment team generally conveyed a sense of enthusiasm and genuine motivation, along with aspirations for ongoing support until the time their organizations are stable enough to stand on their own. One grantee, while expressing the need for more assistance in localities around Rafah stated: “The shooting does not slow the program.” The communication tools developed by Maram’s partners and observed by the assessment team appeared to be of good quality. Perhaps more importantly, the development process leading to their production and delivery includes essential steps to ensure the quality of the productions: ÿ The informational content of the messages is based on sound standards and recent science. BCC team members refer to WHO, UNICEF, Arab Resource Center technical publications, in addition to references made available by AED, which provides the leadership and backstop of Maram BCC activities. ÿ Maram works with its local partners by using ‘creative briefs,’ which provide guidance to the production. Creativity is definitely demonstrated in the range of productions developed by grantees. ÿ Pre-testing of productions is systematically conducted. Maram is aware of and tries to remedy methodological limitations in the conduct of pre-tests (e.g. testing message comprehension with insufficient focus on emotive response, for example). Monitoring and Evaluation Maram and its grantees have monitored extensively the delivery of messages and productions by audience reached. The monitoring and evaluation approach to BCC interventions is however lacking in baseline assessments, identification of information or behavioral targets, and evaluation of impact. But some steps have been taken in that direction and need to be acknowledged. Success stories are reported pointing to the value and effectiveness of messages. Grantees report cooperation with Maram’s RME team in the development of indicators specific to their activities. Grantees report anecdotal evidence of shifts in attitudes or even contemplation of new behaviors. Most are however quite reasonably aware that the short period of implementation of activities is unlikely to be sufficient to sustainably change actual behaviors. One grantee commented, “we need five years.” BCC team members point to the difficulty in building capacity and implementing measurably effective BCC interventions within the same 12 months. Finally, some grantees spontaneously acknowledge the lack of data about household and family behaviors surrounding the children. One grantee—Bait el Said—was however able to show an evaluation report including both success stories and the results of a health survey including 96 women. Even if the assessment team did not review the technical quality of this evaluation report, it is an indicator of a genuine interest in bringing evidence into BCC intervention design. One TIPS (Trial of Improved Practices) research was conducted on Iron intake. No research report was available for the assessment team review, apart from a Powerpoint presentation discussed in the Nutrition Section. Coordination The Grants Section provides more discussion of coordination issues with Maram grantees. Some points specific to the BCC grantees’ activities can be presented here: ÿ Grantees do not always know who their Maram point of contact is, even if they express appreciation to all Maram personnel they interact with. ÿ Grantees report not having been provided with the tools and materials in a timely manner, and also being pressed for time, given the presumed end of Maram. One stated: “Training should be provided at the beginning of the project, not at the end.” ÿ Delays are identified at many levels, including at USAID’s, where budget questions have been sometimes on hold for nearly three months, even as the project is pressing to complete its own targets. At other times, grantees meet longer than expected delays in obtaining answers from Maram itself. ÿ Some grantees feel that Maram is weak in the area of coordination with the media. This is compensated in some cases by working with local partners more closely tied to TV or radio media. For example Pyalara—one of Maram’s grantees in West Bank and Gaza—has helped provide access to Bait el Said in Gaza. ÿ Although some examples of coordination between grantees, such as the one above, can be found there seems to be a lack of overall coordination. In some cases grantees have been working in each other’s zones of intervention without being aware of the one another’s existence and activities. ÿ Finally, in spite of increased coordination within Maram, notably through filling the Public Health Director position, BCC activities have seldom been coordinated with other Maram activities. The work going on around Maternity Homes is potentially an exception. Intervention focus Given that each grantee selected its own health topic and strategy, within broad guidelines provided by Maram, there has not been a clear overall intervention focus of Maram’s BCC activities. Nutrition (particularly Iron Deficiency) and the Birth Cycle themes appear to be the most frequently addressed. In the last quarter of 2003 for example, Maram reports that the following audience numbers have been reached for these specific health topics (Table 3): Table 3: Audience of BCC grantees’ activities during the last quarter of 2003 West Bank Gaza Anemia 7,764 9,271 Birth Cycle 448 64 Water safety 415 199 Diarrhea 4,405 565 Breastfeeding 1,776 1,563 ARI 967 839 Being spread too thin is a problem recognized by some in the BCC team. Some acknowledge (and sometimes regret) that public health impact has not been the aim of the BCC team. Sustainability: Maram’s approach to BCC carries some valuable elements of sustainability, but also some threats to sustainability. Both are reviewed briefly below. Both Maram staff and grantees identify some elements that can potentially outlast the project itself: ÿ Equipment and productions will remain beyond the life of the project. ÿ The radio and TV productions have barely started to be broadcast, and are a resource which can be used over time if there is a will to do so, for example through the use of cassettes or if broadcasters choose to appropriate the productions. ÿ Grantees (more than contractors) report some measure of organizational change affecting both technical and management capabilities. ÿ In fact some of the skills and know-how transferred to the grantees may be ‘marketable’ in order to sustain their organizations and their health communication efforts In terms of threats to sustainability, some observations can be made: ÿ The time frame of intervention has been too short to demonstrate behavior change, much less to demonstrate sustained healthy behaviors. Additionally the programmatic focus on any one single health issue has probably been too limited to expect much of a mass effect in the individual campaigns. ÿ Most grantees feel that a pull-out from Maram at this stage would be too early and would jeopardize the development benefits achieved to date. They generally describe their organizations as being involved in a process of change, which needs to be strengthened before they can continue on their own. ÿ Unfortunately, Maram – possibly pressed for time – tends to strongly monetize all components of its communication efforts (for example, not only are productions supported financially at market prices, but distribution and broadcast is also paid for by Maram with little or no “participation” from the collaborating local organizations). This is not a Maram￾specific problem, but seems to be a widespread “donor bias” well-established in Palestine. It points however to a question of approach and ownership: if organizations produce and broadcast for Maram, their efforts will continue to have to be subsidized. If Maram supports communication efforts of organizations and communities, some agreement and ‘cost-sharing’ of the effort should be found. This would increase sustainability. Summary of findings and lessons learned: ÿ The BCC team operates through nine sub-grants and a smaller number of contracts for media and theater production. ÿ Its efforts are spread thin largely in vulnerable areas of the West Bank and Gaza, but also around Ramallah. ÿ The opportunity for integrated programming (e.g. combining community outreach with health worker training in the same target areas) became lost at some point in the life of Maram. ÿ Communication efforts tend to be spread thin as well over a range of health topics. The birth cycle and nutrition (micronutrients, breastfeeding) are a frequent issue being addressed, but health issues are not addressed exclusively or systematically in any of the sites of intervention reached by grantees. ÿ In fact, the work of the BCC team can more appropriately be described as capacity building in health communication through sub-grants and contracts rather than a social and behavior change strategy. This focus is open for debate even within the Maram team, but the assessment team feels that Maram has overall been effective in this approach. ÿ As a consequence, however, BCC activities cannot be considered to be strategically designed for health impact on a large scale, at least in the immediate future (and with notable exceptions, such as the Hayat campaign). Maram has however built a capacity of local partners in health communication and social mobilization. This capacity could be available in the future to achieve greater impact. ÿ The work of the grantees and the productions of contractors are well informed technically, rely on sound and sometime innovative communication approaches and production/development processes, produce high quality health education productions, and are building the capacity of local organizations to carry on health communication. ÿ Activities have suffered from delays, with consequences in terms of coordination of activities (late delivery of material to groups conducting outreach). ÿ There is evidence of coordination with the RME team to assist in developing evaluation plans with at least some of the grantees. But there is no systematic definition and measure of population-based knowledge, attitude or behavior targets. ÿ Coordination between grantees has taken place but is not systematic and has failed at times. ÿ Productions designed for TV and radio have yet to be fully transferred to more appropriate media, such as audio and video cassettes, which will reach a greater audience. ÿ The focus on capacity building carries with it valuable elements of sustainability, but is constrained by a tendency to monetize every contribution to Maram’s activities, instead of finding a more balanced and participatory collaboration between the project and its local partners. Local partners are aware that this will need to be corrected in the future. Grants Under Contract (GUC) Evolution of the grants activities The original design in the contract identified the management and development of GUCs as a “critical part of implementing and managing IVCHS or Maram” and assigned $8.3 millions as an initial budget in the contract signed in June 19, 2001. This budget was later revised in September of 2002 to become $6,420,440 and was drastically reduced to $1,907,305 in the latest consolidated budget submitted to USAID on February 2004. Delay in launching the Grants Program: Maram signed the first two grants around March of 2003. Fourteen more grants were signed by the beginning of 2004. One grant, awarded to the Islamic University, was completed and another grant (Sanabel Theater) was terminated because of grantee refusal to sign the Executive Order on Terrorist Financing; leaving the number of active grants at 14. Several other potential grantees also refused to sign the Executive Order. The Grants Manual was approved by USAID in August of 2002; 15 months after the start of the program. Soon afterwards, the grants program started developing RFAs and the program was launched. Three initial general reasons were identified by the evaluation team as the causes for the delay in launching and developing the Grants Program: 1. The delay in replacing the Grants Manager (July 2002); 2. The delay in producing the Grants Manual; 3. The emergency phase during the Israeli incursions into the West Bank between March and August of 2002. Once the grants manual was produced, the main obstacle that faced and delayed the grants program, however, was the refusal of some local potential partners to sign the Executive Order against Terrorist Financing. Palestinian NGOs found the Order intimidating and the definition of “terrorism” inappropriate. Apart from awarding “sub-contracts” when it could justify that it was the more appropriate mechanism (in order to obtain case-by-case approval from USAID), Maram did not explore other alternatives such as issuing new RFAs or exploring other potential grantees that might have been willing to sign. Level of coordination: The feeling from grantees was that Maram wanted to maximize the geographic coverage and diversify approaches rather than increase impact. During a group meeting of seven grantees in the West Bank, grantees expressed dissatisfaction with the level of mutual coordination and collaboration that was built into the program. While Maram staff always encouraged coordination, grantees felt that it was left up to them to coordinate, and consequently felt that they were often re-inventing the wheel. There were times that, one BCC grantee would provide the same message at locations where other grantees worked without proper coordination. Other times there was good level of coordination, which proved very rewarding. An example of this is the mobile theatre such as Al Kassaba distributing posters produced by the Young Artists Forum during their performance for maximum impact. Grantees in Gaza also reported cooperation and coordination on media access. Ambiguity in the grants program design The design emphasized the flexibility of the grants program and the need for it to be driven by “community needs, interests, and priorities.” Grantees could be providers of reproductive health, family planning or nutrition services, but could also be providers of non-health services such as marketing, research, training and even providers of equipment. Under Automated Directives System (ADS), IBM Consulting could award Grants, Simplified Format Grants and Fixed Obligations Grants. The increasing recourse to sub-contracts after September 2003 in order to involve local organizations in Maram-supported activities served well the purpose of getting activities underway. The inclination that all grants should have been awarded as contracts is not justified, despite the fact that the design allowed for different interpretations. The evaluation team believes that the original idea of having a large grants program under the Maram contract was very appropriate. The grants program empowered local organizations to use their creativity and grassroots connections to expand the horizons of Maram and maximize its outreach. The inherent structure in the design, however, also allowed for the use of grants for the implementation of “Maram’s activities.” The MTA team agrees that in this case, sub-contracting was more appropriate. The structure that maintained the grants program was not clear to some grantees and was not always conducive to good reporting and capacity building. Despite the fact that per USAID regulations a grantor should not be heavily involved with grantees after the grant has been awarded, Maram maintained substantial involvement with most aspects of the grantees’ work. Capacity building given to grantees was appreciated and thought to be the best ever received. This was applicable to the Management and Finance training but not to Strategic Planning (see Institutional Capacity Building Section). Procedures The Grants Manual provides a good reference and guide to the process of developing and managing grants programs. It is a comprehensive manual with details on each and every aspect of grants management. The evaluation team found that the Grants Management unit adhered to a high level to most of the procedures as laid out in the manual, and exceeded it by imposing more procedures. The Grants Manual clearly states in its introduction that one of its purposes is to “provide local organizations with an overview of the grants-making system.” As it was not a requirement for grantees to master a certain level of English, not having the manual translated into Arabic presented another difficulty in trying to understand the grants procedures. More procedures were exercised, however, on grantees which were not in the manual and do not conform to standard grant procedures. Those procedures mainly relate to requesting original invoices, original payment vouchers and timesheets. All those need to be handed over to Maram before any payment is made in reimbursement of expenses incurred by the grantee. In addition, the grantees were paid based on “milestones” achieved, which is a procedure usually used for contracts and Fixed Obligation Grants rather than grants. While handing over original invoices for VAT reimbursement is justified, using original invoices, original payment vouchers and time sheets as a pre-condition for payment defeats the whole purpose of grants. Once found or developed to have a sound financial management and internal control system, grantees should be reimbursed based on quarterly or monthly financial reports as per USAID standard procedures. The contractor can monitor and audit grantees as needed but without over-burdening local NGOs with such tedious procedures. Such procedures strip the grantees from original basic audit trails such as payment vouchers. While maintaining timesheets is a standard USAID requirement, these timesheets should not be a pre-requisite for payments, but rather part of the capacity building of small NGOs. All grantees interviewed reported that getting reimbursed from Maram takes from 21 to 45 days after submission of all completed paper work. While the principle of “reimbursement” as opposed to “advance” is understandable, most of the grantees do not have the capital to pay three months worth of expenses and then wait another month or so to get reimbursed. This issue is also closely linked to the design of the RFA’s and the inclusion of “milestones” upfront. The Grants Manager felt such milestones were necessary in order to expedite the granting process, which had already suffered many delays. Achieving milestones were also used as pre-condition for reimbursement, as also indicated in the Grant Manual 4.6 – Tranche Payments. Achieving milestones is not an appropriate pre-condition to reimburse grantees, especially in such volatile and unpredictable environment. Database: No database was created to develop and maintain grant files as specified in the original contract. The IT officer at Maram informed the evaluation team that Jafa.net is supposed to deliver a general Maram database by the 14th of April. Sub-contracts: Maram’s sub-contractors contracted out 24 third-tier sub-contracts to local organizations for a total of $1.3 millions. Six BCC sub-contracts were never grants while most of the rest were grants in process that were switched to contracts. Each Maram partner organization contracted out in their own name and used their own contracting procedures and formats. Each team leader is responsible for their own contracts and is accountable to his/her home office in terms of procedures, compliance and management of budget. Summary of findings and lessons learned: ÿ The inclusion of a grants program under the Maram contract was a good idea that empowered local NGOs and provided creativity and inclusiveness in implementation. ÿ There were delays in starting the grants program and developing the grants manual. ÿ Excessive procedures were imposed by the grants management unit on grantees. This is mainly related to the collection of original invoices, payment vouchers and timesheets as pre￾condition to payments. ÿ The Grants Manual is good and comprehensive, except for the pre-determined milestones and the absence of an Arabic version. ÿ Use of a “reimbursement” method for paying grantees along with the delays in payments (one month average) presented a difficult cash flow challenge to grantees. ÿ No grants management database was developed as required in the original contract. Research Research has been the main focus of the RME team. Unfortunately, Maram did not always consume the results of the studies it has conducted to guide its programming. This may be due to the fact that the studies, most of which are baseline studies, were completed so late in the project. It is easy to justify the need for the studies to be conducted; it is not as easy to justify Maram conducting the studies. Baseline studies should be conducted prior to the beginning or in the first phase of a project if they are to be useful to the project, and should be a part of the original proposal. The need for the studies was known to all parties prior to the beginning of the project, and indeed prior to the RFP being issued. This is not to suggest that the studies are of no value. Indeed, they are, but not to Maram given its timeline and structure. The research has a range of complexity (see Table 4), and for the most part is consistent with the quality found in many developing countries. It does carry some ‘leaps of faith’ in terms of some of the methodology and associations presented in the final report, but contain essentially useful findings. This was a missed opportunity by Maram. Given resources available, efforts should have been made to bring in well-qualified researchers who could have worked with the research team to increase their knowledge of appropriate methodology and analysis and especially report writing with adequate descriptions of the analysis, and appropriateness of conclusions made. Research conducted by Maram or the portion of research that Maram conducted were subjected to an institutional review board process at Al-Quds University and consent was obtained of human subjects. Table 4: Research Summary Table Title Research Performed By Research Completed Survey of Women and child health and health services in the West Bank and Gaza strip Maram, USAID, MOH and UNRWA June 2003 Facility Audit § § BCC Radio Message Coverage Survey § § Prevalence of Iron Deficiency Anemia Amongst School Children Maram, MOH MOEHE5 December 2003 Bi-Weekly Report EMAP; Maram; Al Quds University; Alpha International Gaza Health Services Research December 2003 Vitamin A Deficiency Study Maram MOH Not Yet Completed Immunization Coverage And Service Delivery Assessment Maram Islamic U. Birzeit U. 2003 (no month provided) * As determined by the evaluation team. § Reports requested verbally and in writing multiple times but not provided to the evaluation team. Records of the facility audits are available in Maram offices, but no final report. Three studies that Maram has been involved in deserve special mention as they have been presented by multiple informants across the country as having importance: ÿ The Bi-Weekly Reports possessed reasonable methodology and were utilized to some degree during the Emergency Phase to help guide the response by Maram and perhaps other NGOs. Maram contributed facility-level data to these reports. ÿ Prevalence of Iron Deficiency Anemia Amongst School Children has helped the Palestinian health care community to focus on anemia as a potential public health problem. 5 This study is part of a feasibility study of micronutrient supplementation in the schools initiated by the MOE with the MOH. Maram provided TA for the activity, pricing and formulation of supplement, logistics, health education material, training material, and a consultant for problem-solving. At the request of the MOH, Maram facilitated the collection and analysis of the hemoglobin data. ÿ Finally, the Maternal and Child Health and Health Services Survey was an extensive baseline survey designed to update available data, provide further detail on the status of key health indicators, and establish specific health indicator baselines. The RME team has assisted other teams in research and design. For example, the team assisted the BCC team in developing the “nutrition country profile”, and has collected data for the indicators provided by the BCC team. In another case, baseline information for the BCC team was gathered after many activities were completed. For example, hundreds of thousands of dollars were spent to obtain radio spots to convey health messages. It was discovered in the survey that fewer than 12% of the people of Palestine listen to the radio enough to make these spots effective. Had a strategic framework been followed and an evaluation system been in place, it would have been almost inconceivable to have not conducted the survey prior to deciding to rely on radio communication. Capacity Building Conducted by Maram in the Area of Research The Maram RME team has provided capacity building to its research partners in the areas of data collection, data entry, and analysis. Recipients have included professionals and graduate students from Islamic University Community Service and Continuing Education Center, Gaza Health Services Research Center, and the Bir Zeit University Continuing Education Center involved in supporting the research studies. Summary of findings and lessons learned: ÿ Maram did not always consume the results of the studies it has conducted to guide its programming. This may be due to the fact that the studies, most of which are baseline studies, were completed so late in the project. ÿ Baseline studies occurred too late to be useful to the project. ÿ Research conducted by Maram contains essentially useful findings. It has a range of complexity and for the most part is consistent with the quality found in many developing countries with ‘leaps of faith’ in terms of some of the methodology and associations presented in the final report. ÿ Maram missed an opportunity to increase the capacity of the research team and leave a lasting positive effect in the Palestinian health sector ability to conduct research at a higher level than it has in the past. Performance Improvement and Training A set of activities is best addressed through the general heading of the Maram team, which has spearheaded these efforts. Training and performance improvement were considered from the onset of Maram as a way to improve quality of care. Because of a deficit in standards of care throughout the West Bank and Gaza, the development of protocols has taken a prominent role in the activities of the project. Both protocols and training are for this reason considered in this section. Finally, the development of medical records is of a different nature but proceeds from a similar concern for standardization and improvement of care standards. Protocols The Palestinian health sector has practitioners who have been trained in nearly a hundred different countries, and speak different languages with varying degrees of proficiency in English or Arabic. This leaves the standard of care in the country at an unpredictable and uneven level. One way to address this problem is to have nationally recognized and accepted protocols that clearly state the steps necessary for specific diagnosis and treatment required to be consistent with best practices standard of care. The original RFP issued by USAID requested that the project engage in updating or developing standards and protocols to guide secondary level outpatient and facility care of high-risk and complicated pregnancies, management of routine and complicated labor and delivery, post￾partum care, well newborn care, and care of newborns requiring special care (including low birth weight and premature infants). Once reviewed (and updated or developed, as indicated), protocols and standards were to be widely disseminated and promulgated, including among relevant NGOs and other private sector providers. Maram appears to have developed the protocols, and is currently in the process of obtaining the approval of the MOH. Once that is achieved, it plans to begin dissemination. Nevertheless, Maram did not engage in close enough collaboration or coordination with the UN to avoid a duplication of some efforts and reinvention of some protocols that already exist. The Seven Protocols Developed Include: 1. Antenatal Care 2. Normal Childbirth (Safe Delivery Practices) 3. Postnatal Care 4. Care for Newborn 5. Infection Prevention and Control 6. Nutrition 7. Community Based Pre-hospital EMS Maram adopted a four step standardized review process for the development of the protocols that included: 1. Internal review (Maram, local consultants and selected partners); 2. International review; 3. Review from a local consultant; 4. MOH (for approval and adoption). It is interesting to note that, as a part of the process described above, meetings were held in which some stakeholders report being presented with a fait accompli and not a dialogue about the protocols. To date, only the Community Based Pre-hospital EMS protocol has been adopted by the MOH. Discussions between the MOH and the evaluation team indicate a reluctance to adopt any others, at least from some of its cadres. In addition to the protocols listed above that were developed by Maram, some arrived already complete, and have been adopted by Maram (i.e. ‘purchased off the shelf’). These are: ÿ Advanced Life Support in Obstetrics (A.L.S.O.) ÿ Neonatal Resuscitation ÿ Pre-hospital EMS ÿ And some non-obstetric primary care protocols (e.g. pediatrics, internal medicine) The protocols were reviewed by one physician on the evaluation team. This review was not exhaustive and should not be considered the review of the protocols for purposes of implementation. The protocols appear to adhere to best practice standards. Certainly the topics covered address the most critical areas. Some protocols are being supported by the development of a curriculum to teach trainers of trainers as well as practitioners who will not train others. It is hoped that if this is combined with follow-up and evaluation, the efficacy of the protocols will be maintained (see Training Section). There is currently no system in place to assure that the protocols themselves will be periodically reviewed to keep them up to date with best practices. (This is an important point and comment, could you please provide recommendations on how to insure the periodic review). Surprisingly, the protocols include the use of some equipment (e.g. automatic defibrillator) that informants report are not actually available or not widely available in Palestine. The evaluation team suggests that a possible effect of this will be to place a practitioner willing to accept the protocol as the standard of care in the position of not actually being able to carry out that standard of care because the equipment is not available. This will ultimately result in a failure of the practitioner to adopt the standard and render the exercise useless. Maram has presented a contrasting perspective to this issue. Maram contends that placing equipment and pharmaceuticals not available or not widely available in Palestine into nationally recognized protocols will force the medical community to seek them out and to continue to strive for the ideal level of care. Maram further contends that the actual training around the protocols will focus on what is currently available, with follow-up training addressing the use of the equipment as they become available. This assertion is uncertain at best if continued follow-up training can not be assured; and leaves open the question of how best to determine which centers to conduct which training in. It is the contention of the evaluation team that Maram was not able to demonstrate sufficient internal and external communication and coordination to provide this assurance. In brief, the equipment should be widely available before training begins. Summary of findings and lessons learned: ÿ Maram developed seven protocols, six of which are still in review. ÿ The protocols appear to adhere to best practice standards. Certainly the topics covered address the most critical areas. ÿ It may have been unnecessary for Maram to develop some protocols. ÿ The protocols include the use of some equipment that is not actually available or not widely available in Palestine. ÿ It is the contention of the evaluation team that Maram has was not able to demonstrate sufficient internal and external communication and coordination to assure follow-up training and equipment procurement after dissemination of the protocols. Therefore the protocols should not be disseminated prior to assuring that they match available equipment available at the centers where the protocols will be provided. ÿ There is currently no system in place to assure that the protocols themselves will be periodically reviewed to keep them up-to-date with best practices. Direct Service Provision Training Training of Practitioners To enhance the effectiveness of the protocols that Maram has been developing and has adopted it has engaged in the training of practitioners; primarily through grants and subcontracts. The training appears to have begun in the summer of 2003 and has included locations in both Gaza and the West Bank. No formal assessment of training needs was performed by Maram, nor has a system been developed to assess the effectiveness of the training. Training topics have included: 1. Advanced Life Support in Obstetrics (ALSO) 2. Neonatal Resuscitation 3. Pre-hospital Emergency Preparedness Training 4. Nutrition Protocols Recipients have included physicians, nurses, paramedics, and community health workers. There have been four different levels of training provided: validated instructors, service providers, instructor candidates, and management. Organizations that have benefited from the training include Maram, MOH, UNRWA, private practitioners, and NGOs. There have been a total of 182 persons trained. The conceptual framework upon which the training is based, and indeed the approach taken by Maram in the area of performance improvement, was initially an amalgamation of quality assurance and quality improvement entitled, “Improving Health Care Framework,” developed jointly by URC and Intrah. Ultimately Intrah reverted back to its own Performance Improvement model once URC found itself dedicated to the RME component. Advanced Life Support in Obstetrics (ALSO) is a well-validated training program developed in the United States to address the issue of advanced life support in obstetrics. Maram has invested heavily in the use of this program. It has sent Palestinians to the US to obtain training and attend a trainer of trainers course, trained providers and trainers in the West Bank and Gaza, and had one of its staff become a national representative. Maram hopes to institutionalize this program by making it a part of curriculums and a requirement for licensure. It has also been working with a subcontractor, Juzoor Foundation, to expand training and the foundation has been seeking ways to continue ALSO training on its own after Maram ends. Selection criteria for attending the TOT course in the US was: ÿ Practice delivery ÿ Work in primary care ÿ Midwives ÿ Female doctors ÿ Ability to be trainers ÿ Availability ÿ Public health degree Table 5: Practitioner Training Number Trained in Training WB Gaza Total ALSO 60 34 94 Neonatal Resuscitation 37 20 57 Emergency Preparedness 18 9 27 Nutrition 0 4 4 Total 115 67 182 The quality of the training was not evaluated directly by the evaluation team; nor was the curriculum. The ALSO curriculum is well known and validated. The pre-hospital emergency protocols have been adapted from the University of North Carolina. Training of Non-Practitioners Training of non-practitioners has focused primarily on pre-hospital emergency medical services (Pre-EMS). This is in response to the need to increase the ability of the Palestinian community to respond to emergencies even in the event of a closure that prevents transport to a medical facility. To accomplish this in Gaza, Maram contracted with and assisted the Islamic University in the development of pre-hospital EMS curriculum for laypersons, nurses, and paramedics. Maram also contracted with the Physician’s Syndicate to provide this training to physicians. The Physician’s Syndicate reports that Maram taught them how to develop a curriculum and how to teach more effectively through an emphasis on student participation, which is more consistent with the way in which adults learn. In the West Bank, Maram contracted with the Center for the Development of Primary Health Care to conduct the training. A total of 69 persons received this training (Table 6). The quality of the training was not evaluated directly by the evaluation team. Equipment was seen and appears to be consistent with best practice standards. The curriculum was examined by two physicians on the evaluation team, though not exhaustively. It appeared to be consistent with best practices. Table 6: Non-Practitioner Training Number Trained in Training WB Gaza Total Pre Hospital EMS 3 18 21 Pre Hospital EMS TOT 15 33 48 Total 18 51 69 Summary of findings and lessons learned: ÿ No formal assessment of training needs was performed by Maram; nor has a system been developed to assess the effectiveness of the training. ÿ Maram has invested heavily in the use of the ALSO training program as a means of assuring the quality of emergency obstetric care and capacity of the Palestinian health sector to provide this care. ÿ There have been a total of 182 practitioners and 69 non-practitioners who have received Maram sponsored training. ÿ A curriculum for the training of non-practitioners in per-hospital EMS has been developed, and should be quite helpful. Health Management Information System / Medical Records Maram’s contribution to the health information system of the Palestinian health sector has been the development medical records focused on maternal and child health. The records are: 1. Antenatal Record 2. Labor and Delivery Record 3. Labor and Delivery Partograph 4. Labor, Delivery, and Early Postnatal Record 5. Post-Natal Counseling Card 6. Maternity Home Log Form The records have been evaluated by one physician on the evaluation team and found to be consistent with best practice standards and a highlight in Maram’s accomplishments. Indeed, if actually implemented they should contribute greatly to the facilitation of accurate patient status interpretation as well as knowledge transfer between practitioners which is often the weakest link and can lead to devastating results. The records have been approved by the MOH, but there is a delay in implementing them. Different reasons for the delay are given depending on whom the informant is, so it is unclear what the actual reason is. Summary of findings and lessons learned: ÿ Maram’s contribution to the health information system of the Palestinian health sector has been the development of six medical records focused on maternal and child health. ÿ The records are consistent with best practice standards and a highlight in Maram’s accomplishments. ÿ There is a delay in implementing the records that is unclear. Commodity Procurement Beneficiary facilities In addition to specific requests from the MOH, such as equipping two hospitals with kitchens, procuring for the Maternity Homes, and for the BCC and PI&T activities, procurement specifically targeted a growing number of facilities. The Procurement Team developed a facility audit tool. In 2002, the audit had been completed in 23 MoH-supported clinics and in an additional ten clinics in areas of need. From the initial list of facilities supported during the PHP, a first list of 47 “Maram clinics” was identified and included in the initial facility audit. An additional 30 facilities were added during the emergency phase, with needs assessments conducted on a more or less ad hoc basis, sometime using the initial audit tool under the supervision of the senior health advisor. Maram has adapted the list of facilities it supports on an ongoing basis: including new clinics based on the MOH request, notably in Gaza, or dropping facilities which UNFPA planned to support. Support to five PRCS facilities was also cancelled after PRCS wrote to the project to cancel its cooperation with USAID projects. At the end of 2002, the list of facilities was revised to include clinics participating in the Maternity Home activities (three satellite clinics around each MH). Ultimately, Maram planned to support 84 clinics representing a catchment area “of close to 1.5 million Palestinians, representing just under forty percent of the estimated population in Palestine for 2003, including approximately 257,000 children under 5 (38% of the estimated total of children under 5 in 2003) and 314,000 women of reproductive age (39% of the estimated total of women of reproductive age in 2003).”6 Except for the satellite facilities of the MHs, procurement is not coordinated with other training, capacity building or support activities from other Maram teams. The procurement team reports having visited each facility at least twice: one for the needs assessment and once for delivery of equipment. Follow-up visits appear to have been occasional or still in the planning stages. Even Maram leadership considers this activity as a supportive component at the intersection between high demand from a hyper-medicalised health sector, actual needs, MOH and partner requests and the availability of resources. Maram has effectively expanded on the support provided during the PHP, and spread its efforts throughout the territories, effectively targeting areas affected by closures, and being responsive to both MOH and NGO needs. The initial objective of reaching 60% of West Bank and Gaza is not considered realistic by Maram, due to changes in strategy and priorities, over-ambitious initial expectations, the general delays which have affected many Maram activities, and the general implementation difficulties which were not foreseen when Maram was first designed. The most up-to-date map of facilities supported by Maram provided to the evaluation team by the procurement team, lists 79 facilities located in North Gaza, Northern West Bank and a limited number in Bethlehem and Hebron governorates. If the coverage figures for 84 facilities provided above are correct, it can be estimated that with 79 facilities Maram indirectly reaches a catchment area representing 38% of the Palestinian population through the support of clinics. Even this can only be considered an indirect impact area for Maram (which also reaches other zones through its other components.) Procurement process The process of procurement is professional and appropriate for the health sector needs of West Bank and Gaza. Except in the case of the Maternity Homes, however, there is little integration between procurement / equipment and other Maram activities. There is no evidence of systematic monitoring and supervision of the facilities. 6 Maram Semi-Annual Report. Oct.1, 2002 – March 31, 2003. The main items purchased during the emergency phase were: three Oxygen generators, Vitamins and iron supplements, two hospital kitchens and equipment for 29 additional clinics. Total funds spent on procurement that resulted from the emergency phase totaled approximately $1.8 millions. Some of what was purchased during the emergency phase is still “in process.” While the total procurement budget is under IBM, the procurement function is handled primarily by American Near East Refugee Assistance (ANERA) as a sub-contractor. A Team Leader for procurement is in place, assisted by a Procurement Specialist, a Procurement Logistics Coordinator in the West Bank, another Coordinator in Gaza and a Warehouse Manager in the West Bank. Other staff in ANERA assists on a part time basis. Three main committees are in place to manage the procurement process at Maram: 1. Request Review Committee: mainly reviews and approve/disapprove in-kind requests from local organizations and MoH. This Committee was especially active during the emergency phase. 2. Procurement Committee: Reviews Procurement Action Requests from Maram Team Leaders, reviews suppliers’ evaluations prepared by the Procurement Specialist, recommends awards to ANERA and ensures compliance with USAID regulations 3. Bid Opening Committee: Opens bids and documents prices and specifications and signs off on minutes for transparency purposes. While the first two committees are part of the procurement procedures as set in the Procurement Manual, the third committee was established after September of 2002 with an initiative from ANERA. Currently, ANERA keeps a well organized and thorough computerized system managed by the Procurement Specialist who was only hired one month before the departure of the previous COO. A Procurement Logistics Coordinator was also hired in May 2003 to follow up on all logistical aspects of procurement including delivery, installation, training, checking of specs and participating in facility audits. As a Biomedical engineer, the Procurement Logistics Coordinator proved to be of good added value to the procurement function of the program. Going through some of the procurement filing system and interviewing the Procurement Team, the evaluation team has evidence to believe that the Maram program has done a good job in meeting the growing procurement demands of the program while retaining full accountability and USAID compliance procedures. All equipment procured had warranty years for service and parts were of US source and origin, and the process of award was fair and competitive. Summary of findings and lessons learned: ÿ Through procurement, Maram now supports 79 facilities including Maternity Homes and their satellite clinics. ÿ This component of activities is generally not linked programmatically to other Maram strategies, except for Maternity Homes. It appears to be a supportive or complementary activity of Maram’s strategy for the Palestinian health sector, but probably one that has won the project appreciation and recognition by MOH and NGO partners. ÿ There has been an unjustified delay in the production of procurement manual. ÿ The program is to be commended on the establishment of the Bid Opening Committee. ÿ The procurement function had been underestimated in terms of staffing. The two additional staff recruited after September of 2002 is indispensable in terms of assuring maximum accountability and follow up. ÿ Procured goods during the emergency phase took a long time to become available to the intended beneficiaries. Management General Management Structural management issues Having a non-health organization managing a health program and leading a group of recognized health sub-contractors is both a risk and an asset. It is a risk because of the lack of understanding of the underlying issues and needs related to health in management. It could also be an asset as it provides a neutral party who is capable of taking care of all the managerial aspects of the program as well as handling the complex USAID compliance requirements. The USAID contract imposed no specific structure on Maram and left it to the prime and subcontractors to propose the best organizational structure for the program. The large number of subcontractors has been seen by some staff and stakeholders as a strength in terms of the wealth of expertise each sub-contractor brought to Maram. To be effectively a strength, however, it would have required a better arbitration and synergy between subcontractors with overlapping competencies (as with the initial development of a common QA/PI model by Intrah and URC). Unity of Command One of the main problems faced by Maram has been the absence of an effective centralized management of contracts, personnel, and administration. The design enabled sub-contractors to hire their own staff under their own home office contractual regulations. This has created inequity amongst the local staff in terms of benefits and employment regulations. The simple matter of public holidays and vacations that the staff could take was not consistent among the local staff. Recruitment of expatriate staff Maram faced difficulty in replacing senior expatriate staff members especially the COO and Medical Director positions. Two COOs were replaced in less than two years of the program lifetime. The third COO was brought in after six months from the departure of her predecessor. The timing of the replacement, during some of the most turbulent time in the West Bank and Gaza, made it difficult to find qualified candidates willing to come and work there.