REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) Submitted by: LTG Associates, Inc. and TvT Associates, a division of Social & Scientific Systems, Inc. (SSS) Submitted to: The United States Agency for International Development Under Contract No. HRN-I-00-99-00002-00, Task Order Number 817 March 2002 Report of the Evaluation of the Community Health Partnership Project (CHAPS), was prepared under the auspices of the U.S. Agency for International Development (USAID) under the terms of the Monitoring, Evaluation, and Design Support (MEDS) project, Contract No. HRN-I-00-99-00002-00, Task Order No. 817. The opinions expressed herein are those of the authors and do not necessarily reflect the views of LTG Associates, TvT/SSS, or USAID. Information about this and other MEDS publications may be obtained from: Monitoring, Evaluation, and Design Support (MEDS) Project 1101 Vermont Avenue, N.W., Suite 900 Washington, DC 20005 Phone: (202) 898-0980 Fax: (202) 898-9397 jklement@medsproject.com www.medsproject.com ACRONYMS AND FOREIGN TERMS ADB African Development Bank ADF African Development Fund AICPA American Institute of Certified Public Accountants AIDS Acquired Immune Deficiency Syndrome AIMI Africa Integrated Malaria Initiative (USAID/W) ARI Acute Respiratory Infections BASICS Basic Support for Institutionalizing Child Survival BF Breast Feeding BHMI Bakili Muluzi Health Initiative BIMI Blantyre Integrated Malaria Initiative CA Cooperating Agency CACC Community AIDS Coordination Committee CBD Community-Based Distribution/Distributor CBDA Community-Based Distribution Agent CBO Community-Based Organization CDC Centers for Disease Control and Prevention CDD Control of Diarrheal Diseases CHAM Christian Health Association of Malawi CHAPS Community Health Partnerships Project CHV Community Health Volunteer CoAg Cooperative Agreement CPA Certified Public Accountant CPR Contraceptive Prevalence Rate CSP Country Strategic Plan C-SWAp Cross-Sector Wide Approach DACC District AIDS Coordination Committee DEHO District Environmental Health Office DFID Department for International Development (UK) DHO District Health Office/Officer DHS Demographic and Health Surveys DHMT District Health Management Team DRF Drug Revolving Fund EBF Exclusive Breast Feeding EC European Commission EDHMT Expanded District Health Management Team EDP Essential Drugs Program EHO Environmental Health Office EHP Essential Health Package EPI Expanded Program on Immunization EPMU Expanded Project Management Unit EU European Union FAO Financial Aid Office FHI Family Health International FIS Financial Information System FP Family Planning FSN Foreign Service National (USAID) GDP Gross Domestic Product GOM Government of Malawi GTZ German Technical Assistance Agency HC Health Center HIS Health Information System HIV Human Immunodeficiency Virus HMIS Health Management Information System HPN Health, Population and Nutrition HQ Headquarters HSA Health Surveillance Assistants IASP International Affairs Strategic Plan IEC Information, Education and Communication IEF International Eye Foundation IFMIS Integrated Financial Management Information System IMCI Integrated Management of Childhood Illnesses IMF International Monetary Fund IPN Impregnated Nets IR Intermediate Results IS Information Systems ITN Insecticide Treated Nets IV Intravenous JICA Japanese International Cooperation Agency KAP Knowledge, Attitude and Practice LOP Life of Project MAC Mission Accounting and Control System M&E Monitoring and Evaluation MCH Maternal and Child Health MDHS Malawi Demographic and Health Survey MIM Malawian Institute of Management MIS Management Information System MK Malawi Kwacha (currency) MMR Maternal Mortality Rate MMP Mission Performance Plan MOHP Ministry of Health and Population MTCT Mother-to-Child-Transmission MTE Mid-Term Evaluation MTEF Medium Term Expenditure Framework NAC National AIDS Commission NGO Non-Governmental Organization NHP National Health Plan NORAD Norwegian Organization for International Development NSO National Statistical Office ORS Oral Rehydration Salts ORT Oral Rehydration Therapy PACD Project Activity Completion Date PHC Primary Health Care PHICS Promoting Health Interventions for Child Survival Project PHR Partnership for Health Reform (Abt Associates) PLWHA People Living with HIV/AIDS PMU Project Management Unit PSI Population Services International PVO Private Voluntary Organization QA Quality Assurance QIT Quality Improvement Team R4 Results Review and Resource Request RFA Request for Applications RFP Request for Proposals RH Reproductive Health RHO Regional Health Office RMS Regional Medical Stores RN Registered Nurse RP Results Package SCF/UK Save the Children Fund/United Kingdom SCF/US Save the Children Federation/United States SIDA Swedish International Development Agency SO Strategic Objective SOW Scope of Work STAFH Support to AIDS and Family Health Project STI Sexually Transmitted Infections SWAp Sector-Wide Approach TA Technical Assistance TB Tuberculosis TBA Traditional Birth Attendant TEO Tetracycline Eye Ointment TL Team Leader UNAIDS United Nations AIDS Program UNDP United Nations Development Program UNFPA United Nations Fund for Population UNICEF United Nations Children's Fund URC University Research Cooperation US United States of America USAID United States Agency for International Development VHC Village Health Committee VCT Voluntary Counseling and Testing WB World Bank WFP World Food Program WHO World Health Organization W&S Water and Sanitation ZC Zonal Coordinator CONTENTS ACRONYMS AND FOREIGN TERMS EXECUTIVE SUMMARY ................................................................................................. i I. Introduction............ .........................................................................................................1 II. Background ..................... ..............................................................................................3 A. Overview of the Health Sector in Malawi .........................................................3 B. Donor Assistance to the Health Sector in Support of MOHP Policies, Goals and Objectives....................................................................................................5 C. USAID’s Priorities and Strategies for Assistance to the Health Sector.............5 D. Overview of the CHAPS Approach...................................................................6 III. Purpose and Methodology for the Evaluation ..............................................................9 IV. Findings of the Evaluation..........................................................................................13 A. The CHAPS Approach.....................................................................................13 Partnership Coordination, Monitoring and Support...................................13 B. Contribution of CHAPS to USAID Strategic Objective No. 3: Reducing Fertility, the Risk of HIV/AIDS, and Improving Child Health .......................16 C. Building and Sustaining the District-PVO Partnerships ..................................16 Establishing the Partnerships .....................................................................16 Initial Needs Assessment ...........................................................................17 PVO Strategies for District Strengthening.................................................17 Mechanisms for Ongoing Documentation and Learning...........................21 D. Strengthening District Health Systems and Capacity ......................................23 District Health Office Level: Strengthening Organizational Systems, skills and Coordination .........................................................................23 Health Center Level: Capacity Building, Improving Equipment and Services.................................................................................................30 Community Level: Improving Access and Quality of Services, Increasing Community Involvement in Health Promotion.....................................34 E. Financial Management of CHAPS ...................................................................41 Establish and Build Effective Models of Public/private Partnerships.......42 Strengthening the Public/private Partnership at the District and Central Level Within the MOHP.......................................................................42 Enhance DHMT Capacity Through the Transfer of Service Delivery and Management/Financial Skills Between International NGOs and the MOPH...................................................................................................43 Additional Ways of Strengthening the Public/private Partnership at the District and Central Level Within the MOHP ......................................44 Findings Related to Improvement of the Management and Quality of District Level Health Services and Health Promotion Capabilities of Communities and Family Caregivers....................................................44 Contribution of the CHAPS Activities to the Achievement of Results of the Overall HPN Strategic Objective (HPN SO) of USAID/Malawi: Increased Adoption of Measures that Reduce Fertility and Risk of HIV Transmission, Including Child Health Practices...................................44 Support Provided by the Central Level MOHP .........................................45 The Unintended and Secondary Results ....................................................45 Determine the Effectiveness of Monitoring, Evaluation and Performance Monitoring Plans to Guide Program Direction and Monitor/Evaluate Project Activities...................................................................................45 Additional Monitoring, Evaluation and Studies to Determine More Effectively Results and Best Practices to Guide Future District Health Service Improvement; and HPN SO/Mission Programming Activities and National/Regional Approaches to Public/private Partnerships......46 Future Cost Effective Programming Measures of Mission HPN SO Activities Related to CHAPS and Africa Regional Concepts for Public/private Partnership Development ..............................................46 Appraisal of Program and Financial Management Practices by CHAPS Implementing Partners, Cooperating Agencies and of Key Operational Units......................................................................................................46 Sustainability of the CHAPS Approach.....................................................48 V. Future Strategies and Models for Public/private Partnerships.....................................49 A. Experience Based on CHAPS Model...............................................................49 Discussion of Implementation Model 4: District Level Partnership Coordinating Unit .................................................................................50 VI. Lessons Learned from CHAPS...................................................................................53 VII. Recommendations .....................................................................................................55 ANNEXES A. Statement of Work .......................................................................................................65 B. Evaluation Team ..........................................................................................................75 C. Itinerary for the Evaluation of CHAPS........................................................................79 D. Persons Consulted........................................................................................................87 E. Documents Consulted ................................................................................................107 F. Matrix of Alternative Models for Future Public/private Partnerships.......................115 i EXECUTIVE SUMMARY The Community Health Partnership Project (CHAPS) is a component of USAID's assistance to the health sector in Malawi. In 1995, CHAPS was proposed as a new model of assistance to improve health care services in target districts through public/private partnerships for health. CHAPS is a US$15 million, five-year initiative, designed to utilize Private Voluntary Organizations (PVO) experienced in the provision of Primary Health Care (PHC), as partner organizations with the Ministry of Health and Population (MOHP), to extend key health services and enhance institutional capacity. The CHAPS project activities support Strategic Objective (SO) No. 3 “Increased Adoption of Measures that Reduce Fertility and Risk of HIV Transmission and Improve Child Health Practices” by increasing the provision of quality health care services to Malawians and building the capacity of the District Health Management Team (DHMT) to deliver health care. The CHAPS project was signed in 1995 and implementation of activities began in 1998, with a one-year extension through September 2002. Cooperative Agreements (CoAg) were awarded to five PVOs (Africare in Mzimba, International Eye Foundation in Chikwawa, Project HOPE in Mulanje, Save the Children/UK (SCF/UK) in Salima, and Save the Children/US (SCF/US) in Mangochi) to support PVO-DHMT partnerships. Under the CHAPS project, support for technical assistance for Quality Assurance (QA) was given to University Research Corporation (URC). In addition, a social marketing Cooperative Agreement was awarded to Population Services International (PSI) for the social marketing of Thanzi Oral Rehydration Solution (ORS) and Exclusive Breastfeeding (EBF). Building on an established partnership with the Centers for Disease Control (CDC), support was given from CHAPS and USAID/Washington for the Blantyre Integrated Malaria Initiative (BIMI). An evaluation was requested to assist USAID/Malawi and the MOHP to assess and document the progress made under the CHAPS project. A four-member team conducted the evaluation in February-March 2002. The evaluation methodology included interviews with the CHAPS Team Leader, USAID/Malawi Mission Director, Health, Population and Nutrition (HPN) team, and evaluation and accounting staff; meetings with MOHP; meetings with other donors to the health sector; meetings with Private Voluntary Organizations (PVO) Country Directors and staff; field visits to the five CHAPS districts and meetings with PVO Project Managers and CHAPS Team, District Health Office (DHO) and DHMT, Health Center (HC) and outreach health workers, community members and village health committees, among others. Key Findings The evaluation found that CHAPS has in many ways been an effective partnership model for strengthening district health services, particularly at the sub-district level. Some of the initial problems encountered were related to limited communication and REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) ii misunderstanding among all partners (MOHP central level, DHO/DHMT, and the PVO/Non-Government Organizations (NGOs)) and therefore could be minimized with more joint planning for the project, fuller understanding of the meaning of the partnership, and greater financial transparency. Despite these problems, in addition to constraints beyond the control of the project such as severe staff shortages and frequent transfers, the DHMT and PVOs have learned together and developed supportive and productive partnerships in each of the five districts. Given the time and effort invested to date by all partners, it would be unfortunate to lose the constructive relationships developed in the CHAPS districts, including the experience of the five CHAPS PVOs/NGOs. The following summarizes the key findings of the evaluation. Major Achievements Several key findings of the evaluation relate to the nature of partnerships. It was found that the CHAPS model of public/private partnership has provided significant support for district strengthening at the DHO, health center and community level. The model has demonstrated that PVO/NGOs can effectively collaborate as partners with government and district health staff. There is clear evidence in CHAPS that the organizational strategies of all five PVOs are evolving from a focus on direct implementation to that of organizational strengthening. Through CHAPS the PVOs have strengthened their skills in the organization mode of operation and their district-strengthening expertise should be built on and expanded in the future. The model had enough flexibility so that the project could be designed and the resources used to address the particular needs of each district. Therefore the CHAPS model was able to respond to many of the district needs for implementing the decentralization of health services. All five CHAPS PVO/NGOs have contributed to strengthening district health systems at the DHO, HC and community levels. In addition to increasing access to health services, the partnership has developed mechanisms for communities to play a greater role in health and development. Thus the district capacity-building expertise developed by the PVO/NGOs should be exploited and expanded in future district strengthening efforts in Malawi. ƒ The approaches and tools developed through the district/PVO collaboration appear to be grounded, practical and adapted to district level activity. The CHAPS capacity-building approach focuses on on-the-job training and therefore can decrease district staff absences. Because the five PVOs focused on strengthening existing structures and staff, much of the system strengthening has good prospects of being sustained. ƒ An important contribution is that CHAPS PVOs succeeded in greatly increasing attention and support for HIV/AIDS activities at district and sub￾district level. EXECUTIVE SUMMARY iii ƒ The model has facilitated better coordination of all health-related activities at the district level, demonstrating the potential of contributing to formation of a form of district Sector-Wide Approaches (SWAps), including a multi-sectoral and cross-sector approach, e.g. water and sanitation, food security, etc. This was particularly true in districts where PVOs drew on other programs and resources, thus enhancing the impact of CHAPS. Major Constraints ƒ The evaluation identified several constraints and challenges to the implementation and functioning of the CHAPS partnership model. In fact, it was found that efforts to recognize and address the challenges of partnerships were limited throughout the project. If more recognition of the complexities of partnerships, some of the early delays and problems might have been avoided. ƒ USAID as the donor organization was responsible for overall accountability, monitoring and coordination of CHAPS. USAID’s involvement included: approval of annual work plans; approval of key personnel; approval of monitoring and evaluation plans; and, monitoring progress toward the achievement of program objectives during the course of the Project. Although these responsibilities were fulfilled, it was found that the partnerships would have benefited from additional support, including more frequent field visits to the districts, which would have further facilitated and nurtured CHAPS. ƒ At the central level the MOPH has had very little direct involvement and provided limited support to CHAPS. This contributes to MOHP’s limited understanding of CHAPS achievements and contributions to district health strengthening and decentralization. In fact, the channels for communication from central MOHP to districts and from districts to central level are not always clear, not just related to CHAPS but also more broadly in relationship to many aspects of decentralization. Thus there is a need for two-way communication with the center informing districts about new initiatives for decentralization and districts informing the center about experiences, problems and realities of decentralization at the district level. In addition to CHAPS, many other international organizations are supporting the MOHP’s decentralization and district health strengthening projects. However, there is no forum for coordination or a mechanism for systematically sharing experiences at the central or district level. The evaluation found that CHAPS’ PVOs/NGOs were working towards a systems approach and in collaboration with government at the district level. However, some people interviewed in MOHP, USAID and other donor organizations expressed very traditional--and in the case of CHAPS--outdated views of NGOs, for example, making statements such as "...all NGOs work in isolation from government and create parallel projects." REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) iv ƒ Staff shortages, turnover, and dual responsibilities have significantly diluted the institution strengthening efforts of CHAPS. District staff are also involved in multiple training programs supported by the MOPH, donors and PVOs, but no strategy exists for prioritizing and coordinating training. ƒ In CHAPS districts, overall coordination of district health activities has improved but requires further strengtheningin order to include all organizations working in the district (in addition to CHAPS). District Health Officers stated that they needed clarification regarding DHO authority for regulating and coordinating all health-related activities taking place in the district, including CHAPS, Christian Health Association of Malawi (CHAMS),and other NGOs working in health, etc. Contribution of CHAPS to Strategic Objective No. 3 Although district performance indicators vary from one district to another, one common indicator is Contraceptive Prevalence Rate (CPR) and therefore is useful for comparative purposes. In all districts there has been a significant increase in CPRs and the increased CPR serves as an example of the impact of CHAPS. The evaluation identified improvement in system and capacity building at all levels, district, health center, and community that contribute to this accomplishment. Collecting information on all of the intermediate results was difficult. Lessons Learned from CHAPS ƒ CHAPS' PVOs/NGOs demonstrated that NGOs are able to move beyond the more traditional roles related to implementation and work successfully on organizational/institutional strengthening. ƒ Building partnerships is never easy. Successful partnerships require close monitoring, guidance, support and nurturing. All partners need to be fully informed and involved from the beginning in planning and implementation. Full financial transparency is another requirement in a partnership. In addition to technical competency, interpersonal; communication; and, facilitation skills are essential for partnerships. ƒ Mentoring and on-the-job training is an effective method of capacity building. In some respects working side-by-side is more effective than formal training. Working together at the district level on a day-to-day basis is more effective than technical assistance from above and/or the central level. ƒ Technical assistance needs to be adapted and appropriate to the local situation as demonstrated by the limited impact of the QA Technical Assistance (TA) provided to CHAPS districts. EXECUTIVE SUMMARY v ƒ Several innovative approaches implemented during CHAPS provide important lessons as demonstrated by the following examples. 1) Health worker purchase of equipment, for example bicycles by Health Surveillance Assistants (HSAs), helped to create a sense of ownership and maintenance of the equipment that contributed to improved job performance. 2) The creation of zonal coordinators (ZCs) within a district, facilitated supervision and support to health workers and in turn, enhanced health worker performance. 3) Creation of core groups of DHO staff helped address the problem of staff absences and shortages. 4) Strengthening of fleet management greatly facilitated the DHMT’s ability to coordinate and provide services. ƒ Cross-sector activities such as water and sanitation and food security increased the effectiveness of the project. Recommendations The following key recommendations are based on the findings of the evaluation: ƒ To build on and expand the district capacity-building expertise developed through CHAPS in future district strengthening efforts in Malawi. ƒ To involve and utilize the specific district capacity-building experience of the five CHAPS NGOs/PVOs in future district strengthening efforts. ƒ To maintain a district focused and based approach in future district strengthening efforts. ƒ To continue program flexibility in order to adapt to district needs in future district strengthening efforts. Flexibility in the CHAPS program was found to make a positive contribution to program achievements. ƒ To recognize the complexity and needs of partnerships as demonstrated in CHAPS and provide more support for nurturing and strengthening such partnerships in the future. Therefore, mechanisms need to be in place to ensure stronger support and coordination to project partners. ƒ To identify weaknesses and problems early in the partnership. Based on the experience with CHAPS, the effectiveness and impact of the partnerships could have been greater if there had been mechanism for providing districts with more on-going support related to technical and management issues, as well as to PVO District relationships. ƒ To build on partnership strengthening lessons learned from CHAPS; for example, sharing of space for CHAPS/PVO/DHO facilitated the partnerships. ƒ To develop a mechanism for coordinating all district health related activities, including those provided by the MOHP, CHAMS, and all NGOs working in REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) vi the district, to ensure that all efforts support DHO priorities and fit within the annual district health plans. An initial step would be to clarify DHO authority to coordinate activities of all NGOs working in the district. ƒ To increase central level support to district partnerships, enabling the central level to provide strategic guidance to districts while at the same time listening and learning from districts’ experiences and constraints; thus to establish a two-way communication from central MOHP to district level and from district to central level. Such a two-way communication system would help inform districts about new initiatives for decentralization and for the districts to inform the center about experiences, problems and realities of decentralization at the district level. (The above refers to general support for the partnerships and not to the two specific Health Management Information Systems (HMIS) and Integrated Financial Management Informtion Systems (IFMIS) initiatives.) ƒ To provide support for building on and strengthening PVO technical capability, although this was not part of the CHAPS project. Many PVO staff will remain in the district and therefore this would further strengthen district resources. ƒ To examine the impact of structural constraints encountered in CHAPS, including acute staff shortages; frequent staff turn-over; dual (clinical and public health) responsibilities of DHO staff; and, frequent staff absences including those due to training events and meetings. These constraints could best be addressed first through policy decisions based on dialogue between the MOHP and DHMTs. ƒ To establish forums and mechanisms for sharing experiences and lessons learned, in addition to the Quarterly Meetings. These mechanisms should facilitate sharing within and between districts and between districts and the central level. Examples of sharing within disticts included regular monthly and quarterly coordination meetings supported partnership functioning and strengthened service delivery in one district. In another district, an annual retreat for all CHAPS PVO and DHMT staff was found to be an effective mechanism. ƒ To build on the experience of the CHAPS quarterly meetings, which served as the primary forum for sharing experiences. These quarterly meetings were viewed positively but it would be beneficial to involve DHMTs and PVOs in each district in planning the agenda and to hold them on a regular basis (for example, no quarterly meeting has been held in almost a year), which was not held because of USAID’s staff shortage. ƒ To develop a mechanism for discussion and utilization of the findings of evaluation and research reports. For example, more systematic and in-depth EXECUTIVE SUMMARY vii feedback and discussion could have facilitated implementation of the findings of the CHAPS Mid-Term Evaluations. ƒ To develop a forum or mechanism for coordinating and sharing experiences among donors and government on strengthening district approaches and decentralization. It was found that many organizations provide assistance for strengthening district health services (e.g. USAID, German Technical Assistance Agency (GTZ), the Netherlands, European Union (EU), Department for International Development (UK) (DFID), United Nations Children’s Fund (UNICEF). Although there is a monthly meeting of the donor subgroup on health, it does not currently serve as a mechanism for coordinating and sharing experiences learned from the various approaches to district health strengthening. ƒ To continue and provide increased support to the DHMT for HIV/AIDS activities, particularly given the limited involvement in HIV/AIDS found on the part of District Health Offices. ƒ To continue support for transportation and logistics, continuing and building on the positive experience and outcome under CHAPS of TA for fleet management.. ƒ At the Health Center (HC) level, to focus on organizational/management skills of HC staff, in addition to the current support for improving services, supervision, training, and technical program areas. Special attention is recommended for the HC level since in terms of cost-effectiveness, prospects for sustainability of capacity-building efforts at HC and community levels may be greater than at DHO level given the relatively lower turn-over among core HC staff, and particularly among HSAs and community activities (as confirmed by the Mid-Term Evaluations). ƒ To address constraints identified in QA and QA TA provided through CHAPS, including the finding that QA needs to be practical and adapted to the local context. ƒ To conduct a review of training provided during CHAPS to determine quality of training and impact of training on staff performance. In addition, to develop a mechanism for regular assessment and follow-up of training in future district strengthening programs, in addition to establishing a mechanism to monitor the benefit of training on community health. ƒ To develop a strategy for district level training provided by all organizations and MOHP. Such a strategy needs to prioritize training and address the evaluation finding that the same people are invited for multiple training sessions which contributes to DHO staff absences and unavailability. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) viii ƒ To conduct an assessment of formal training versus mentoring which would build on the findings of the positive impact of the presence and availability of staff in the district and day-to-day contact. ƒ To develop mechanisms to build on and share the best practices which were found to be innovative and suited to local realities and constraints, for example: core groups; zonal offices; bicycle ownership; revolving funds for bednets; and, fleet management system. ƒ To review experience with bicycle ambulances which were found in some areas to be inappropriate to local practices, in addition to encountering problems with maintenance. ƒ To continue support for and assessment of efforts to improve supervision, including the content and communication and interaction skills of those providing training. ƒ To document the experience of zonal offices to strengthen supervision, to determine the benefits and challenges of this approach for use in other districts. ƒ To improve capacity building for data management by DHMT, e.g. to include provision of required computer equipment and method for backing-up important data on a regular basis, and having PVO staff knowledgeable in computer systems, in addition to PVO staff with technical expertise in HMIS, finance and logistics. ƒ To continue support for construction of health facilities (e.g. out-reach shelters, disposal of medical waste, etc.) in future district programs ƒ To give increased attention to strengthening drug management at central district store and pharmacy to include improved storage methods; inventory control, forecasting techniques, distribution systems and computerized management of stocks, orders, etc. ƒ To document and share the lessons learned from the innovative strategy to train merchants in basic malaria diagnosis and treatment and sale of fansidar. ƒ To address the problem of HC staff shortages and of limited time HC staff have for outreach activities and the impact of delegating more responsibilities to HSAs as the only mechanism that has been used to deal with this problem during CHAPS. ƒ To document the lessons learned from activities to strengthen VHCs and CHVs, given the challenges and importance associated with strengthening these groups. EXECUTIVE SUMMARY ix ƒ To continue support for Drug Revolving Funds (DRF), based on the experience with CHAPS. ƒ To continue support to Traditional Birth Attendants (TBAs) and build on their contribution to improved delivery and referral services. Although the trend is to encourage delivery in health facilities, such services will not be available to much of the population in Malawi for some time and therefore it is important to continue support and capacity building of TBAs. ƒ To continue and expand efforts to improve referral services, e.g. provision of radios to health centers, ambulances, support to TBAs, etc. ƒ To document lessons learned and continue and increase support for cross￾sector activities, such as water and sanitation, and food security. ƒ To document experience with provision of water and sanitation (W&S) pumps, latrines and other supplies and disseminate between CHAPS districts and with other districts. ƒ To give greater attention to transferring accounting and financial skills to DHO staff either through full-time staff who have this expertise or through outside consultants who could provide occasional TA. ƒ To intertwine and build in sustainable analysis support and financial management structures and systems into the underlying business and operational processes, rather than having them “attached.” ƒ From a financial perspective, to update the organizational structure of the district to best support national and district strategic initiatives; to ensure a workable retention package to preclude unnecessary staff turnover; and, to apply IFMIS and HIS resources while mentoring DHMT members. ƒ To increase CHAPS involvement and coordination of improved quality of financial staff and in automation of the accounting system (e.g. IFMIS). ƒ To increase focus on the financial structures and internal transaction collection and reporting schemes of the district in future district programs. ƒ To include in future, financial management functions related to monitoring. ƒ Financial monitoring efforts at the central-district should be better linked with a risk analysis of the key processes and mission-critical functions. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) x ƒ For the financial management structure to better support the stated mission, strategy, organizational structure, key processes, staff duties, services, and desired planned versus actual outcomes of the district. ƒ To address the constraints found in the current CHAPS in future program design. ƒ To have one integrated financial management system. ƒ To review the organizational structure of the district prior to implementation of any aggressive interventions for improving the financial management systems and processes. ƒ To review the mix and skill-set of staff in the financial management areas to insure it aligns with the demand. ƒ To better identify problems, provide measures of costs and benefits, more frequently provide financial evaluation of alternatives, and better monitor and evaluate operational results for future financial management practice. ƒ To require future financial management function to provide an accurate and timely analytical capacity to forecast financial performance. Recommendations for Future Studies and Evaluations ƒ In evaluation of the future CHAPS or district health strengthening program, involve national representation on the evaluation team. It would also be beneficial to include representation from other district health strengthening projects, e.g. the Netherlands supported Lilongwe District Health Project. ƒ There are many lessons learned from CHAPS but limited mechanisms for documenting, recording, and sharing/disseminating these lessons and best practices. It is recommended that such mechanisms be developed. ƒ A study of the many approaches being supported in Malawi for strengthening district health services (in addition to and including CHAPS) is recommended and a forum established for comparing and assessing the effectiveness of these approaches. ƒ Studies of the quality and utility of training provided under CHAPS and in future programs are recommended. Recommendation for Structure of Future District Health Strengthening Project Based on the findings of the review of CHAPS, the evaluation team considered a number of strategies for a future district public/private partnership. The preferred alternative for a future district health strengthening partnership is the district level Partnership EXECUTIVE SUMMARY xi Coordinating Unit (PCU). The PCU would be located at the district level, in one of the future project’s focal districts. This would ensure on-going day-to-day mentoring and capacity building with all partners. Although the PCU would be located in one district, support would be provided to all focal districts. The PCU would be located within a district-based organization, such as a PVO/NGO or consortium composed of PVOs. This model could be formed by a consortium of interested PVOs having the capability to put together a team of technical advisors with extensive district experience and strong interactive and communication skills identified as essential for partnership building. The proposed PRC would consist of a small team of technical experts (three to four) including cross-cultural staff with Malawian equivalent counterparts. Members of the coordination unit should have strong links and/or access to senior members of the MOHP to facilitate communication and coordination with headquarters (HQ) and the on-going evolution of the decentralization process. The purpose of the PCU would include administrative oversight and management of funding; collaboration between partners to facilitate decentralization; building on lessons learned and tested strategies from CHAPS, strengthen existing relationships, and facilitate expansion into neighboring focal districts. The PCU would report to USAID and also strengthen coordination with MOPH HQ. The PCU would be pragmatic and grounded; flexible; inclusive, for example, including cross-sector approaches such as water and food security; and, build on lessons learned from CHAPS. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) xii 1 I. INTRODUCTION The Health Services in Malawi face particularly difficult challenges in addressing the country’s health problems. Malawi is described as being one of the poorest countries in Africa with a per capita income of approximately US$170. Sixty-five percent of its estimated population of 10,640,000 (1999) is poor, and the level of poverty is greater among the 86 percent of those living in rural areas. Given this situation, it is not surprising that Malawian health indicators are also poor. Life expectancy from birth is 39 years. The infant mortality rate is 104 per 1,000 live births and the under-five mortality rate is 189/1000 according to the Demographic and Health Surveys (DHS) 2000. The infant and child mortality rates reflect an improvement from the reported statistics of 234 and 134 respectively in the 1992 and 2000 DHS, but the maternal mortality ratio increased from 620 to 1,120 maternal deaths per 100,000 live births. The major causes of illness episodes and premature deaths include pneumonia, diarrhea, TB, malaria, anemia, nutritional deficiencies and AIDS, which are primarily preventable and/or curable. Despite the Government’s commitment to reducing poverty and providing basic services, the health services remain under-funded and under-staffed and thus unable to adequately address the overwhelming needs, especially of the more vulnerable population in rural areas. In addition, access to basic services such as safe water, adequate sanitation, food and education is limited. USAID and other bilateral, multilateral, and non-governmental organizations provide assistance to GOM/MOHP to enhance efforts to increase resources and improve health services. As part of its package of support to the health sector, USAID funded the CHAPS Project, a joint public/private collaboration between Government PVOs, which was designed to address the above problems by improving health services at the district level. USAID has requested this evaluation to assess CHAPS’ progress and contribution to the goals of the MOHP and USAID/Malawi. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 2 3 II. BACKGROUND A. OVERVIEW OF THE HEALTH SECTOR IN MALAWI The MOHP has overall responsibility for the formal health services in Malawi and provides 60% of services and the Christian Health Association of Malawi (CHAM) provides 37%. In recent years, the GOM has contributed toward payment of local salaries for CHAM staff. Private Practitioners comprise a small but increasing percentage of services. Many people are reported to consult the formal and informal health sector, primarily traditional healers and traditional birth attendants (TBA). A major constraint in the health sector is the limited human resources. Malawi has a severe shortage of health personnel. In addition to a shortage of trained health professionals, paramedicals and personnel at all levels, the problem is compounded by the increasing number of deaths from HIV/AIDS and transfer from the public to the private sector and abroad. In most health facilities, many staff positions are unfilled, especially in Malawi’s 27 districts where the vacancy rates are often over 50 percent. Therefore, human resource development is a GOM priority. In addition to expanding enrollment in medical and nursing training institutions, emphasis is being given to train managerial, accounting and audit staff. The National Health Accounts (June 2001), based on 1999 data, provides the most recent and comprehensive information on health care financing in Malawi. A summary of the key points follows. Total per capita health expenditure in 1998-99 was approximately US$12.40. In 1999, the health sector accounted for 7% of gross domestic product (GDP): The GOM contributed 25% of the financing resources in the health sector; donors (including non-budgetary expenditure) accounted for 30%; private financing sources accounted for 45% with 19% from employers’ contribution and 26% from household out-of-pocket expenditure. The main health problems identified in recent MOHP policy and strategy documents (e.g. Malawi National Health Plan 1999-2004; To the Year 2020: A Vision for the Health Sector in Malawi; Human Resource Development Plan) include: ƒ High child mortality and morbidity; ƒ High maternal mortality and morbidity; ƒ High HIV seroprevalence and death rate due to HIV/AIDS and HIV-related diseases such as TB; and ƒ High morbidity and mortality in the general population due to other infectious diseases, including tuberculosis (TB). REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 4 The main health service delivery challenges for 1999-2004 are stated as: limited access to high-quality and effective care; weak inter-sectoral linkages; weak policy and regulatory infrastructure. The national health goals and objectives are related to the national development goal of poverty alleviation and the national policy of decentralizing management authority to District Assemblies. The health sector policy goal to improve the health status of all Malawians is to be done through five supporting sector policies: 1. An Essential Health Package (EHP) which is being designed to address the major causes of mortality and morbidity among the general population, focusing particularly on medical conditions and service gaps that disproportionately affect the poor. These include the following conditions: malaria, acute respiratory infections (ARI) and immunizable diseases, acute diarrheal diseases, nutritional conditions, maternal pregnancy conditions, sexually transmitted infections (STIs) including HIV/AIDS, TB, eye and ear infections, injuries, and schistosomiasis; 2. The Bakili Muluzi Health Initiative (BMHI) which focuses on prevention of child mortality at the community level in rural areas; 3. Introduction of SWAp which is a framework for implementing the National Health Plan (NHP) developed by GOM and its development partners, and has two primary objectives: to develop a coherent sectoral policy and program as the basis for concerted action and to develop a donor co-ordination mechanism to promote joint implementation; 4. Decentralization of health care management to District Assemblies which will assume decision-making at the district level and a process of developing district implementation plans for the health sector; and 5. Introduction or strengthening of cost recovery/user fees. The medium-term objectives developed for attaining the national health goal include: 1. Expand the range and quality of health services focused on maternal health and children under the age of five years; 2. Improve the general health status of the population by strengthening, expanding and integrating relevant health services; 3. Increase access to health care facilities and basic health services; 4. Increase, retain and improve the quality of trained human resources, and distribute them efficiently and equitably; 5. Provide better-quality health care in all health facilities; II. BACKGROUND 5 6. Improved efficiency and equity in resource allocation; 7. Strengthen collaboration and partnership between the health sector, communities, other sectors, and private providers (allopathic and traditional); and 8. Increase overall resources in the health sector and allocate them efficiently and equitably. B. DONOR ASSISTANCE TO THE HEALTH SECTOR IN SUPPORT OF MOHP POLICIES, GOALS AND OBJECTIVES External aid financed 85% of Malawi’s development budget between 1994-1999, with an average of $400 million as the annual total donor assistance. The bilateral organizations, providing assistance include USAID, the United Kingdom (UK), Japan, Germany, Denmark, Norway, Sweden, Canada and the Netherlands. The multilateral donors are the World Bank (WB), EU, International Monetary Fund (IMF), and the United Nations Organizations: United Nations Development Program (UNDP); UNICEF; United Nations Fund for Populations (UNFPA); Financial Aid Office (FAO); World Health Organization (WHO); World Food Program (WFP); United Nations AIDS Program (UNAIDS); African Development Bank (ADB) and African Development Fund (ADF). In addition, many non-governmental organizations provide assistance to Malawi. There is reported to be a close working relationship among the donor community with regularly scheduled meetings among donors and mission chiefs to discuss policy issues and coordinate across sectors. Many of the donor organizations provide assistance to the health sector and a current priority is improving district health. In addition to USAID, several donors (e.g. Germany, the Netherlands, EU, DFID for safe motherhood, and UNICEF for PHC) provide assistance focused on various programs for strengthening district health services. Although a health sector donor group meets monthly, there is no specific forum for sharing experiences and lessons learned from these different approaches for strengthening district health. C. USAID’S PRIORITIES AND STRATEGIES FOR ASSISTANCE TO THE HEALTH SECTOR The USAID/Malawi Strategic Plan 2001-2005 is described as being consistent with the U. S. Government’s interests in Malawi which include promotion of: democracy and good governance; broad based economic growth and agricultural development; human capacity building through education and training; health benefits for all; and environmental protection and disaster preparedness. USAID’s assistance to the health sector relates to improving basic social services and contributes to the International Affairs Strategic Plan (IASP) goal of improving the global environment, achieving a sustainable world population and protecting human health. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 6 USAID’s Strategic Objective No. 3, “Increased Adoption of Measures that Reduce Fertility and Risk of HIV/AIDS Transmission including Improved Child Health Practices” addresses the key health problems of Malawi’s National Health Plan for 1999 to 2004 (listed above). In particular: 1) lowering the risk of HIV/AIDS because of the epidemic’s tremendous impact on human resources and productivity; 2) reducing fertility and population growth, which are essential for attaining board based economic growth; and, 3) lowering infant and child mortality rates. USAID’s earlier assistance to the MOHP focused on institutional strengthening of the central offices of the MOHP as part of the “Promoting Health Interventions for Child Survival” (PHICS) project. PHICS was assessed as not being successful in reaching the target rural population. Therefore, in 1995 a new model for health assistance was proposed, the CHAPS project, which is a $15 million, five-year initiative to improve health care services in target districts through public/private sector partnerships for health. CHAPS was designed to utilize PVO’s experience in the provision of PHC, as partner organizations with the MOHP, to extend key health services and enhance institutional capacity. The CHAPS project activities support the HPN Strategic Objective by increasing the provision of quality health care services to Malawians and building the capacity of the DHMT to deliver health care. D. OVERVIEW OF THE CHAPS APPROACH The CHAPS initiative was designed to support the Government’s policy of decentralization. Until 1994, Malawi’s health care system was more centralized and characterized as being top-town management and decision-making, with greater emphasis on urban facilities and less support for district and community services. Although USAID’s earlier support for PHC PHICS was intended to strengthen district health services, it followed GOM priorities and structure and thus was less successful in reaching rural communities. CHAPS was viewed as a pilot effort to facilitate decentralization and improve services for the poor by testing and evaluating the effectiveness of partnerships between PVO/NGOs and the District Health Offices (DHO)/MOHP, in particular as a method for improving the delivery of maternal and child health (MCH) and reproductive health (RH) services and reducing the risk of HIV/AIDS. In support of this objective, USAID awarded CoAgs to five PVOs to support the PVO￾DHO partnerships. Under this partnership framework, the PVOs were to collaborate with DHMTs to assess constraints, prioritize health problems, identify systemic weaknesses in the delivery of services, develop and test alternative health financing approaches, and implement appropriate interventions. An important emphasis of the partnership was that this was to be accomplished within the current government system; that is, government structures would retain and exercise their normal authorities. CHAPS interventions included both advocacy and delivery of specific services with the objective to improve practices which prevent illness and promote better care of ill children and their mothers, and actions to reinforce the health system at the level of II. BACKGROUND 7 delivery, including strengthening health facilities and structures facilitating patient￾provider contact, referral systems, and the availability of drugs and supplies. Water supply and sanitation were identified as high priorities among women consulted and therefore were included among CHAPS interventions. In addition to the five district partnerships, CHAPS supported additional interventions for malaria and diarrheal diseases, as these are major causes of infant and child mortality in Malawi. A separate grant was given for promotion of oral rehydration therapy (ORT) and social marketing of a local oral rehydration solution (ORS), and to develop a program to research and develop messages for promoting EBF. This program was based in Blantyre and focused on the Southern Region. Support was also provided from USAID/Washington and CHAPS for the BIMI, which was implemented in Blantyre District in collaboration with the CDC. As originally planned, CHAPS was to have input from all levels of the MOHP, with Headquarters/Central level setting policy and providing direction, review of plans, and support to the district partnerships. At the inception of CHAPS, it was envisioned that the Regional Health Office (RHO) would be involved in planning and oversight, but the Regional structure was discontinued and not replaced. At the district level, the PVO and DHMT were expected to work in partnership to develop the CHAPS proposal and annual work plans. The USAID/Malawi HPN team was responsible for overall accountability, monitoring and coordination of CHAPS. USAID’s involvement in CHAPS included: approval of annual work plans; designation of key positions and approval of key personnel; and, approval of monitoring and evaluation plans, monitoring progress toward the achievement of program objectives during the course of the Project. The project was signed in 1995 but implementation was delayed for two years until MOHP and USAID reached an agreement on the terms of the project. In April 1997, a Request for Applications (RFA) was released and the following five PVO/NGOs were awarded agreements: Africare for Mzimba District, International Eye Foundation (IEF) for Chikwawa District, Project Health Opportunities for People Everywhere (HOPE) for Mulanje District, SCF/UK for Salima District, and SCF/US for Mangochi District. In 2001, proposals for a one-year extension of the five projects were prepared by the district partners and approved by USAID. The extension period is through September 2002. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 8 9 III. PURPOSE AND METHODOLOGY FOR THE EVALUATION Purpose and Objectives The purpose of this evaluation is to assist USAID/Malawi and the MOHP to assess and document the progress made under the CHAPS project. The Scope of Work (SOW) states that the evaluation will provide the basis for deciding on ways of further strengthening the district public/private partnership and central level support. In addition, the evaluation will also be useful in determining additional performance monitoring, evaluation and studies that would ameliorate the results and best practices for possible replication of similar activities. The objectives of the evaluation include the following: 1. Assess and document the progress made under the CHAPS model of public/private partnerships in supporting the DHMTs; 1.1. Establish and build effective models of public/private partnerships; 1.2. Enhance DHMT capacity through the transfer of service delivery and management/financial skills between international NGOs and the MOPH; 1.3. Improve the management and quality of district level health services and health promotion capabilities of communities and family caregivers; 1.4. Determine the contribution of the CHAPS activities to the achievement of results of the overall HPN SO of USAID/Malawi: increased adoption of measures that reduce fertility and risk of HIV transmission, including child health practices; 1.5.Establish support provided by the central level MOHP; 1.6.Establish the unintended/secondary results; 2. Determine the effectiveness of monitoring, evaluation and performance monitoring plans to guide program direction and monitor/evaluate project activities; 3. Recommend additional monitoring, evaluation and studies to determine more effectively results and best practices to guide future district health service improvement; and HPN SO/Mission programming activities and national/regional approaches to public/private; 4. Recommend ways of strengthening the public/private partnership at the district and central level within the MOHP; REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 10 5. Recommend future cost effective programming measures of Mission HPN SO activities related to CHAPS and Africa regional concepts for public/private partnership development; 6. Appraise program and financial management practices by CHAPS implementing partners, cooperating agencies and key operational units; and 7. Determine the sustainability of the CHAPS approach. Methodology To meet these objectives, the MEDS Project organized the evaluation and recruited four consultants including a PVO specialist, financial analyst/certified public accountant (CPA), institutional and organizational specialist, and health services specialist. The evaluation was conducted between 20 February and 25 March 2002. The evaluation team first met in Washington, D.C. for two days for orientation, planning of the evaluation, and review of background documentation. Extensive documentation from USAID, MOHP, donor organizations, PVOs/NGOs, and other groups was collected and reviewed. In Malawi, the evaluation began with briefings at USAID with the Mission Director and Assistant Director, members of the HPN team, CHAPS Team Leader, and also meetings with financial/accounting and evaluation/monitoring staff. Meetings and interviews were held with key MOHP officials, technical advisors to MOHP, and representatives and officials of the donor group for health including bilateral, multilateral, and non￾governmental/private voluntary organizations. Country representatives and accounting staff were met at the five PVO/CHAPS headquarters in Lilongwe and Blantyre. Field visits were made to the five CHAPS districts: Mzimba, Salima, Mangochi, Chikwawa, and Mulanje. In each district, meetings were held with District Commissioners, other members of the DHMT, district financial and accounting staff, PVO, and CHAPS staff members. Visits were made to the district hospitals and health centers where meetings were held with health center staff and outreach staff, in addition to visits to villages to meet with members of the community and village health committees and TBAs, and to observe CHAPS supported activities and facilities such as water and sanitation programs and food security activities. In an effort to make the evaluation as participatory as possible, a preliminary meeting for stakeholders including CHAPS/PVO project managers and DHOs from the five districts, the CHAPS Team Leader and members of the Evaluation Team was held at USAID during the first week of the evaluation. A follow-up dissemination seminar with stakeholders to discuss the evaluation findings was not organized, although a meeting to discuss the final evaluation report with the CHAPS Team Leader was held after the departure of the Evaluation Team. A formal debriefing with USAID Mission staff was III. EVALUATION METHODOLOGY 11 not organized, but a preliminary briefing with the USAID Mission Director was held and some other members of the HPN section joined this meeting. At the request of USAID/Malawi, the time period of the evaluation was extended in order for all team members to visit the five CHAPS districts. Even with the extension, time was limited and therefore the CHAPS team leader asked that the evaluation team not review the CHAPS supported PSI activities and BIMI. TA for QA was discussed in each district, although a separate review of QA was to be conducted by URC in April 2002. Logistic and administrative assistance was provided by an in-country consultant, which greatly facilitated the evaluation. 12 13 IV. FINDINGS OF THE EVALUATION A. The CHAPS Approach Partnership Coordination, Monitoring and Support USAID and MOHP Support for the Partnerships CHAPS, described as a pilot project, confronted two major challenges. First, it involved working with district health staff to define ways to provide basic health services in a sustainable way in the context of decentralization. Second, it involved the establishment of PVO-District partnerships as a mechanism for district strengthening. Given the pilot nature of the project and the complexity of partnerships, it required considerable guidance, coordination and support not only from PVO management, but also from USAID and the central MOHP. Therefore this project would have benefited from more support and guidance. Role of USAID As noted above, USAID’s HPN team is responsible for overall accountability, monitoring and coordination of CHAPS. Responsibility for management of the CHAPS partners was assigned to a senior Malawian health officer within the USAID Mission, the CHAPS team leader (TL) and included: approval of annual work plans; approval of key personnel; and, approval of monitoring and evaluation plans, as well as monitoring progress toward the achievement of program objectives during the course of the Project. Although these functions were carried out, there were also challenges. The time required to provide both administrative and district-level field support to the five partnerships was greater than expected and was an unrealistic responsibility for only one person, particularly when the HPN team and CHAPS TL had numerous other responsibilities in addition to CHAPS in USAID’s health portfolio. Quarterly CHAPS meetings are the primary support activity organized by the CHAPS TL and all partners find them to be useful. However, many feel that the format followed in the meetings was somewhat repetitive and that they did not provide a forum for in-depth discussion of innovative practices or problems. When there were serious problems, related either to partner-relationships or other technical issues, the CHAPS TL assisted the PVO-District teams as much as possible. While all CHAPS partners appreciate the support they received from the CHAPS TL, they all state that their contact with him was relatively limited. They regret that during the life of the project he was able to make only a few visits to the districts in order to observe their accomplishments, constraints and lessons learned. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 14 The evaluation team found that the effectiveness and impact of the partnerships could have been much greater if there had been a mechanism for providing the districts with more on-going support related both to technical and management issues, as well as to PVO-District relationships. Such support would have helped to identify weaknesses or problems early on and to identify alternative strategies or solutions in a timely fashion. As it is, some of the weaknesses in CHAPS programs were not addressed in a timely fashion. For example, in three of the districts, from the outset, there was insufficient focus on information/management information system (MIS) strengthening. If this situation had been recognized and been given sufficient attention earlier, it could have been addressed in a more timely fashion through discussions with the DHMT and identification of additional technical resources. In all CHAPS districts the PVOs commissioned MTEs to assess both accomplishments and constraints to program implementation. The MTEs were an excellent opportunity for both the PVO-DHO partners and the CHAPS TL to review CHAPS progress and define priorities for strengthening project implementation. CHAPS partners expressed disappointment that the opportunities to discuss the MTEs were limited to brief presentations at the Quarterly Meetings. Partners said that more comprehensive discussion and feedback would have facilitated implementation of MTE findings and recommendations. In any future district strengthening efforts, it would be important to ensure stronger coordination and support to project partners. Given the constraints encountered by USAID in providing support to the project, in future multiple-district projects it would be advisable to have a relatively small “project coordinating unit” which could provide more frequent and on-going technical as well as administrative support to district partnerships. Role of the MOHP CHAPS was originally planned to involve all levels of the MOHP. The central level was to set policy, provide direction, review plans and support the district partnerships. At the time that CHAPS was launched in l998, there were regional health offices in the country that provided some support to the districts. It was expected that the RHOs would also be involved in planning and oversight of CHAPS. However, in the year 2000, the regional health offices were dismantled. Consequently, current MOHP support to districts comes exclusively from the central level. Senior MOHP officers said that a new zonal system has recently been developed, but oversight is from the Directors and Controllers of the six MOHP divisions who are already fully committed at the central level and agreed that they have little time at present for supporting the districts. According to DHMT in all districts the support which they currently receive from the MOHP is very limited for all programs, not just for CHAPS. The District Health Officers feel that they currently do not receive the policy, technical and moral support they need to effectively implement health programs at the local level. There is a general feeling in the DHOs that “the MOHP is far away” and that “They really don’t understand our context and constraints.” IV. FINDINGS OF THE EVALUATION 15 In the context of decentralization, the DHOs are required to deal with multiple constraints associated with inadequate human, material and financial resources. At the same time, they have not yet been provided with all of the necessary systems and tools required to operate at the district level (for example, financial software and systems, user-friendly methodology for annual district planning exercises). In addition, the support that was received from the Central level was often not in a form that best addressed district-level needs and realities. Specifically related to CHAPS, a representative from the central MOHP participated in many of the CHAPS quarterly meetings but beyond that the central level has not been substantively involved in observing or supporting the work of the CHAPS partnerships, nor in learning from the district experiences. The success of any future district-strengthening efforts will depend in part on greater involvement of central MOH level staff that provides strategic guidance to districts while at the same time listening and learning from districts’ experiences and constraints. PVO Staffing and Management of CHAPS In each district, staffing patterns differ in terms of number of CHAPS staff and the types of positions created by each PVO. The PVO CHAPS staff generally appears to be well qualified for their tasks working with district health staff. The majority are former MOH personnel, often with many years of experience working within the MOH system and with a strong commitment to district strengthening. Others have backgrounds in community development, water and sanitation and food security, which are very relevant to community level activities and whose skills complement those of MOH staff. For program managers, although clinical background and/or knowledge was helpful, it was not found to be necessary, but skills in management, organizational development and facilitation of team work were very important. In a few cases, stronger interpersonal and communication skills would have helped facilitate the partnership. As noted above, in most respects, PVO CHAPS staff had adequate experience and expertise for the district-strengthening model, particularly related to health facility and community level interventions. However, in some cases, particularly related to MIS and financial management, the PVOs did not create positions, acquire expertise and/or involve staff in district strengthening in those areas (e.g., financial management). In the case of MIS, the need for greater emphasis and expertise in this area was not addressed until late in the project in two of the districts. PVO staff turnover had an impact in some districts. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 16 B. CONTRIBUTIONS OF CHAPS TO USAID’S STRATEGIC OBJECTIVE NO. 3: INCREASED ADOPTION OF MEASURES THAT REDUCE FERTILITY AND THE RISK OF HIV/AIDS TRANSMISSION, AND IMPROVING CHILD HEALTH PRACTICES Each of the CHAPS district programs has a set of performance objectives related to SO 3, although given the specificity of each of the five projects their specific objectives vary making it difficult to compare progress among the districts. One of the common performance objectives across all five programs is contraceptive prevalence rate (CPR). The table below summarizes both baseline and most recent (late 2001) CPR data from the five sites. This data suggests that there have been significant increases in the CPR in all five CHAPS districts. This data (collected from each district during the evaluation) gives an indication of the positive impact that the CHAPS partnerships have had on district programs. CPR in CHAPS districts in 1998 and 2001* District 1998 baseline 2001 Salima 18% 33% Chikwawa 14% 26.8% Mangochi 12.9% 33.5% Mzimba 23.4% 37.9% Mulanje 22.3% 28% * The CPR for 2001 is calculated as cumulative point prevalence. C. BUILDING AND SUSTAINING THE DISTRICT-PVO PARTNERSHIPS Development of genuine PVO-District partnerships represents a significant challenge. In spite of difficulties encountered in most districts at the beginning of the project, at present it was found that there is good collaboration between the District and PVO staff in all five districts. Through the evaluation, various factors both within and outside of the partnerships were identified which have contributed either to strengthening or weakening those relationships. These factors are discussed below. Based on this analysis, a number of lessons are suggested for development of future district-strengthening partnerships. Establishing the Partnerships In most districts, development of the partnerships had to be formed quickly in order to respond to the RFA time frame. Subsequently, in almost all districts problems arose due to different understandings of the purpose and respective roles of the DHO and PVO in the partnership. In addition, below the District Health Office level, limited efforts were made to comprehensively explain the partnership to district health staff. In future partnerships, more effort should be invested at the outset to ensure that DHMT leadership IV. FINDINGS OF THE EVALUATION 17 fully understand the proposed partnership model, and that they are asked to make a choice whether or not to be involved. District staff at other levels should be fully briefed on the partnership concept and modalities to ensure a common understanding and clear expectations on the part of all DHO stakeholders. Development of CHAPS Proposals Development of the CHAPS proposal in each district was a critical step in defining the nature of the partnerships, in prioritizing district needs and in fostering district commitment to the project. In spite of the relatively short time frame allotted to the PVOs to develop their CHAPS proposal, according to DHO staff, in all cases DHMT members were involved with PVO representatives in the development process. Such involvement was critical in ensuring that proposals addressed priority needs from the DHO perspective and in engendering a sense of ownership of the project. Unfortunately, due to considerable staff turnover four years into the project, few DHMT staff in the district at the time the proposal was developed are still there today. Although PVOs brief new staff, it is difficult to have the same understanding as when DHMT are part of the initial process. Therefore, in some cases present DHO staff do not have a clear idea of how the partnership was initiated and how the proposal was developed. Rapid turnover of DHO staff, and particularly of the DHO, has constituted a significant constraint to project implementation. For example, in no case was the current DHO in the district in l997 when the proposal was discussed and in several districts, there have been four or more DHOs since CHAPS began. Each time new district staff has arrived, PVO staff have briefed them on the partnership. Similarly, in three of the districts there has been considerable turnover in the PVO program managers since 1998. Initial Needs Assessments In the first year of implementation of the CHAPS grants, in almost all districts needs assessments and/or baseline studies were carried out at both health facility and community levels. However, in no instance were organizational assessments conducted at the DHO level. Such analyses would have been useful both in strengthening the DHO’s sense of commitment to the partnership and in providing guidance regarding PVO staff recruitment needs in light of district capacity-building needs. The absence of such assessments may explain, for example, the fact that several of the PVOs did not initially perceive the need for a full-time MIS specialist to strengthen this area, which was weak in all DHOs at the outset. PVO Strategies for District Strengthening The original RFA identified core activities to be supported and/or developed within the districts related both to technical program areas and to human and institutional capacity building. The prioritized technical areas were: household management of childhood illnesses; MCH Nutrition; RH; W&S; HIV/AIDS; and community involvement in REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 18 delivery of health services and health promotion. In subsequent sections of this report, strategies and activities carried out to address these areas are discussed related to activities at the DHO, health center and community levels. RFA priorities for human and institutional capacity building were: development of DHO management skills to include planning, training, supervision, monitoring and evaluation of programs at all levels; improvements in MIS for decision-making; and expansion and strengthening of health center and outreach programs at the community level. The mechanisms used by the PVOs to strengthen district human and institutional capacity are discussed below. A series of complementary and similar strategies and activities were used by all five PVOs to address the human resource/institutional capacity building objectives of CHAPS, namely: 1) operations research and other studies; 2) formal training; 3) mentoring; 4) development of tools and approaches; 5) capital investments; 6) provision of means of transportation, equipment, other tools; and, 7) short-term TA. Such activities were carried out at multiple levels (DHO, health center and community levels). While historically the PVOs/NGOs have operated quite independently in district health programs, in the CHAPS project the modus operandi of the PVOs was to work as partners and facilitators rather than as direct implementers. In almost, but not all cases, these seven categories of activities were developed and implemented through close collaboration between DHO and PVO staff. In two of the districts (Mangochi and Chikwawa), PVO/DHO communication and collaboration has been greatly facilitated by the fact that PVO and DHO offices are located in the same building. Unfortunately, while in all CHAPS districts considerable resources were invested in capacity-building through the several strategies listed above, the impact of those efforts has been greatly diminished by a series of structural constraints related to the district’s ability to absorb and sustain the results of that support. These constraints include: acute staff shortages; frequent staff turnover; the dual (clinical and public health) responsibilities of DHO staff; and, frequent staff absences due to training events and meetings. Operations Research and Other Studies The purpose of the operations research and other studies was to provide information to CHAPS partners, which could be used to inform development of district plans and strategies. Topics addressed in such studies include, for example: fleet management (Chikwawa); community involvement in health promotion (Salima); and exclusive breastfeeding (Mangochi and Mulanje). While CHAPS partners report that the results of these studies were used to develop program strategies, two constraints were identified. In some cases, outside consultants took responsibility for carrying out the studies and district staff was not substantively involved. In other cases, although active involvement of the DHMT was planned, ultimately they were not actively involved in the studies due to other commitments. Therefore, it would be beneficial to include a budget line item for operations research and IV. FINDINGS OF THE EVALUATION 19 small studies in order for support to be available if a need is identified during project implementation. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 20 Formal Training An important and resource-intensive component in all five CHAPS districts was formal training organized for District Health Office personnel, health center staff and community organizations/actors. In most cases, training events were organized by the CHAPS partners and took place within the district, while in some cases people attended workshops elsewhere in Malawi, particularly at the Malawian Institute of Management (MIM), and in a few cases, in neighboring countries. Training activities with DHO staff dealt either with management skills or technical program areas (IMCI, FP/RH, Malaria, W &S, etc.). The majority of the formal training activities for health center staff addressed either basic management or technical topics, while at community level; they dealt with establishment or reinforcement of community skills and activities (DRF, TBAs, VHCs, latrine construction, etc.) District staff at all levels said they appreciate the formal training sessions. However, several constraints were identified which are associated with the formal training events: the combination of frequent training events and staff shortages, particularly at the DHO level, often leaves the district severely understaffed; in many of the training events teaching methods consist primarily of lectures which are known to have a limited impact on participant learning, as contrasted with more active learning methods; and, in most cases systematic follow-up of trainees after training is not carried out to assess the impact of training on their performance in the work place. Staff Absences Often the DHMT does not participate in joint DHO/PVO activities as planned, due to training events sponsored either by the central MOHP or other donors, which pull them out of the district. This situation clearly illustrates the limited capacity of the district to absorb capacity-building inputs. Both central MOHP officials and PVO staff discussed the problems associated with training activities organized by multiple donors which contribute to frequent absences of staff from HCs and even more so from the DHO. Related to this point, there is a need to rationalize training by maintaining a roster of training needed and received by each staff member. However, the financial motivation associated with formal training events will undoubtedly persist. A long-term solution to the problem of frequent absences due to training can only be solved through a policy decision based on dialogue between MOHP staff and the DHMTs. It is somewhat difficult to determine the extent to which CHAPS’ investments in formal training have lead to improvements in performance on the job. Factors which influence the impact of the training conducted include: the quality of training; whether or not there is a mechanism for follow-up and reinforcement of trainees on the job; and whether those trained are still working in the district. Quality of Training CHAPS-sponsored training activities often draw on national curricula, whereas in other cases DHO/PVO staff works together to develop training designs. Most of the PVO staff IV. FINDINGS OF THE EVALUATION 21 have had considerable experience in adult education/active learning methods of training and their involvement in designing training activities seems to have contributed to some improvements in training methods used, and in turn, on the impact of training. In this regard, the quality of training done in Mulanje and Mangochi is noteworthy. In these districts training plans and reports show that there is a greater use of participatory, adult education methods and very detailed training plans are developed which can easily be used again in the future. Impact of Training Follow-up of training events should be systematic. According to DHO/PVO staff, during supervision visits follow-up of trainees was completed, although this is difficult to verify. In Mulanje such follow-up appears to be done systematically with use of “training supervision check-lists” which are developed during the planning of all training events and are used for follow-up of trainees. Mentoring: On-the-Job Training In many health programs and projects capacity building is equated with formal “training workshops.” While CHAPS supports such training events, the evaluation strongly feels that the presence of PVO staff in the district and the day-to-day contact, collaboration and mentoring of district staff has undoubtedly contributed more to capacity building than the occasional workshops. One of the DHOs stated, “It is good having them close to us. We can call on them whenever we have a problem or are trying to figure out how to plan or implement an activity.” Another example of the impact of mentoring was described by a newly appointed District AIDS Coordinator who said he was in daily contact with PVO staff who had taught him what he knew about AIDS and provided excellent background and educational materials. Such mentoring took place in a variety of joint PVO/district staff activities related to planning, implementing, training, supervising, monitoring and reporting at the DHO, health center and community levels. Development of Tools and Approaches A product of the close and frequent collaboration between district and PVO staff has been the development of numerous approaches and tools for strengthening district programs, which are both innovative and suited to local realities and constraints. Examples of such approaches/tools include development of: “core groups” in Mzimba; “zonal offices” in Mangochi; “bicycle ownership” for HSAs in Mulanje; revolving funds for sale of bed nets in Salima; and, a fleet management system in Chikwawa. Capital Investments Depending on district-specific needs, capital investments in CHAPS vary considerably from one district to another. Examples include: construction of an office building to house DHO and SCF/US staff in Mangochi; construction of three Voluntary Counseling REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 22 and Training (VCT) centers in Mulanje; and construction of HC staff housing in Salima. Incinerators for medical waste were constructed at several HCs in Mangochi. Materials and supplies were given to communities for participatory construction of outreach centers and under-five shelters which are also used for maternal health, prenatal check-ups and family planning counseling, and for construction of TBA birthing centers, latrines and placenta pits/incinerators. This support is very much appreciated by the districts, especially given the fact that district budgets are very limited and usually do not include resources for such expenditures. Equipment, Tools and Means of Transport Based on district-defined needs, CHAPS has also provided resources for the purchase of priority items such as: vehicles; short-wave radios in HCs; ambulances; bicycles for HSAs; bicycle ambulances and laundry equipment for the hospital. Short-Term Technical Assistance In each of the partnerships, short-term TA was used from time to time to address specific issues and problems. The TA was of variable quality and usefulness. For example, in Salima TA was provided by MIM to conduct a training needs-assessment, which, unfortunately, was poorly done and was of little use to the district. Conversely, in Chikwawa, a very comprehensive analysis of fleet management was done by a Zimbabwean consulting group, which led to concrete recommendations for improvements. Maintenance of the Partnerships Sustaining open, productive partnerships of any sort is a challenging undertaking. In most CHAPS districts limited explicit attention has been given to “partnership maintenance.” As noted above, the CHAPS quarterly meeting organized by USAID is the primary forum for people from different districts to share accomplishments and constraints. CHAPS partners appreciate the quarterly meetings but it was not sufficient to solve all partnership-related problems and therefore additional mechanisms are required to nurture the individual partnerships. In Mangochi periodic “partnership retreats” were instituted of which a priority purpose was to assess partnership dynamics and solve partnership problems. These retreats proved to be very useful in terms of partnership strengthening. In the last year, similar retreats have been organized by CHAPS Salima and Mulanje. Thus such periodic workshops or retreats in each district would allow partners to analyze strengths and weaknesses of the partnership and formulate their own recommendations for strengthening partnership dynamics. Mechanisms for Ongoing Documentation and Learning In any new program such as CHAPS, it is desirable to have mechanisms for documenting and synthesizing lessons learned based both on the “successes” and the “weaknesses/ challenges” encountered during the implementation of the program. In the evaluation, IV. FINDINGS OF THE EVALUATION 23 the Evaluation Team sought to identify such “mechanisms for program learning” both within each of the partnerships and between the five partnerships. Mechanisms for Learning within Each Partnership Information activities to inform decision-making and program learning should include the collection of quantitative data, but also more qualitative documentation related to strategies used, successes and constraints/weaknesses. In all districts certain quantitative indicators were tracked in order to document outputs/activities and progress toward performance objectives. Reports of training and other workshop events were regularly drawn up, as were the observations/conclusions from supervision visits, particularly in Mzimba and Mangochi. However, as noted above, mechanisms were not in place at the district level to ensure more qualitative documentation, analysis and the sharing of lessons learned based on partnership involvement in the implementation of CHAPS. One of the few examples of systematic documentation for learning was in Mangochi where an intern studied the partnership and wrote a valuable case study documenting the steps, challenges and lessons learned regarding partnership development and maintenance. Unfortunately, the case study was not widely shared amongst CHAPS stakeholders in Mangochi, nor with other CHAPS districts. The current evaluation identified important lessons learned based both on innovative and effective approaches/tools developed in the districts, and on the constraints and weaknesses encountered in project implementation. Unfortunately, those lessons were not adequately recorded and disseminated in order to facilitate program learning. For example, it would be useful to have short descriptions; including steps followed and lessons learned, of the “core groups” developed in Mzimba, the zonal offices in Mangochi and bicycle ownership by HSAs in Mulanje. This documentation has not been done nor has the learning been consolidated to facilitate sharing within each district, with other districts and with the central MOH. The MTEs in each of the districts are also excellent opportunities for learning among program implementers and collaborators. However, it is not clear that the lessons/recommendations formulated during the evaluations have systematically been used in all districts to strengthen project implementation. Mechanisms for Learning Between the Five Partnerships In the five districts health program activities are similar as are partnership strategies. For this reason, there is considerable potential for sharing and learning between the districts. The main mechanism for sharing between districts was the system of quarterly meetings organized by the CHAPS TL. All district teams feel that these meetings are an occasion to learn what other districts are doing. However, in most cases these meetings do not allow for in-depth presentation and analysis of specific strategies and activities to enable others to implement similar approaches. As noted in other sections, all PVO and DHMT staff noted with regret that there has been no quarterly meeting during the last year. In a few cases exchange visits between district teams took place, which allowed for inter- REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 24 district learning related to specific program components. Other mechanisms have not been developed to promote documentation and sharing of lessons across districts. Therefore it was found to be important to develop various mechanisms for documentation, analysis, discussion and dissemination of program successes, constraints and lessons learned across districts in order to maximize program learning. D. STRENGTHENING DISTRICT HEALTH SYSTEMS AND CAPACITY In all five district partnerships, strategies were developed to strengthen district systems and human resource capacity at multiple levels; namely, DHO, health center and community levels. A primary aim of CHAPS is to provide resources at the decentralized level. The evaluation team concluded that the project is cost-effective in channeling resources directly to the districts. Based both on project documentation and interviews conducted with district and PVO staff during the evaluation, key findings related to accomplishments, constraints and lessons are reported here for strategies carried out at each of these levels. District Health Office Level: Strengthening Organizational Systems, Skills and Coordination The District Health Office is composed of ten to fifteen people including the DHMT and five to ten program coordinators who are responsible for various technical programs, as well as MIS and QA. The District Health Office, and more specifically the DHMT, is responsible for planning, coordination and monitoring of all health programs in the district. The efforts of the PVOs have been directed at strengthening the systems and capacity of the DHMT and larger DHO related to these functions. Efforts by the PVOs to strengthen the District Health Office encountered major constraints which include: DHO staff turnover (e.g., in several districts there have been four or more DHOs since CHAPS began); acute staff shortages (for example, in Chikwawa 75% of all staff positions are filled whereas in Salima district only 43% are filled); DHO staff are responsible for providing clinical services in the hospital and managing the district; and frequent absence of DHO staff from the district due to their involvement in activities (including training) organized by multiple donors and the MOHP. The cumulative effect of these constraints is that the time and absorptive capacity of DHO staff to be involved in on-the-job activities to strengthen district systems and capacity is greatly diminished. In spite of these constraints, there have been improvements in all five districts in their systems and staff capacity to plan, coordinate and monitor district programs. There has also been considerable turnover in PVO project management although this appears to have been less disruptive than changes in DHO staff due to continuity provided by other PVO project and country office staff. IV. FINDINGS OF THE EVALUATION 25 Planning and Coordination In all CHAPS districts considerable effort has been made to strengthen mechanisms for planning and program coordination. These efforts have met with variable success. In the context of decentralization, since 2001 the districts have been responsible for developing annual district health plans. Limited guidance has been provided by the central MOHP for this critical annual task and in addition, districts do not have a clear idea of the budgetary allocations available for each district. In all districts PVO partners have provided inputs and support to the planning process and in some cases made suggestions for the planning methodology used. In all districts, on the basis of the initial jointly prepared project proposals, most CHAPS-supported activities are being included in the district plans. As noted above, a major constraint to district planning and coordination is the fact that HMIS systems are weak in three of the districts. In Mangochi and Mzimba the HMIS systems are reasonably strong and HMIS data serve as a basis for planning. During the evaluation, the DHOs were involved in preparing their annual plans for the second time, for 2002-2003. The Evaluation Team was not able to assess the quality of the plans being developed but clearly this is an area where additional support is required both from the central level and at the local level. In all five sites, regular district level CHAPS review and planning meetings are scheduled to take place quarterly and monthly. In all districts the PVOs have played an important role in trying to ensure that such review and planning meetings are regularly held. However, the regularity of these meetings varies considerably from one district to another and a constraint encountered everywhere is the frequent absence of the DHMT staff, especially the District Health Officer. Mangochi District was found to have a regular system of databased review and planning with monthly coordination meetings of the “Project Management Unit” (PMU) composed of the DHMT, the SCF/US manager and coordinator taking place on a regular basis. Quarterly Expanded Project Management Unit (EPMU) meetings are also regularly held which are attended by the DHMT Program managers, seven SCF/US staff, five MOH ZCs and In-Charges from all HCs. These quarterly meetings are an excellent mechanism: for the DHMT to collect and discuss data-based reports from the HCs and zones regarding HC and outreach activities and constraints; quarterly presentations to the EPMU by HC and zonal staff of their data/results provides strong motivation for good performance; to enable District Health Office staff to understand community and HC needs and priorities; and to strengthen communication and coordination both between HCs, and between HC and District Health Office staff. This system in Mangochi goes a long way toward establishing effective mechanisms for on going databased monitoring and decision-making. It would be very beneficial for Mangochi to comprehensively document/describe their monthly and quarterly review and planning processes for the benefit of other districts. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 26 In Chikwawa, monthly and quarterly coordination meetings take place regularly, though the weakness in their information system constitutes a serious constraint to data-driven review and planning. In the other districts the quarterly and monthly planning and coordination meetings are not always held as scheduled which constitutes a serious constraint to overall program effectiveness. Information Systems and Monitoring In all five districts, MIS skills at DHO and HC levels were very weak at the outset of CHAPS. In most cases, data collection from the HC and community levels was irregular and of poor quality; there was very limited analysis and use of data for planning, DHMT staff did not have access to computers and most were not computer literate. All of the PVOs have focused to some extent on strengthening District Health Office computer hardware/software, MIS systems, skills and use of information in planning and monitoring. Last year the central MOH has introduced a new HMIS system, which includes 65 indicators to be tracked by all districts. MOH-coordinated training on the new system began late in 2001 In all districts some progress has been made in strengthening the information system, skills and utilization; however, the accomplishments in this area vary greatly from one district to another. In some cases, information activities within the PVO have been quite strong, such as in Africare in Mzimba, although until recently, much less attention was given to strengthening DHO information systems and skills. In Mangochi, from the outset the focus of Management Information System (MIS) activities was on strengthening district systems and skills. In the other districts attention to MIS strengthening has increased over time. In Mangochi good progress has been made in comprehensively strengthening district information activities for decision-making. With a SCF/US MIS advisor in place since l998 to guide development of this critical component, key dimensions of the strategy to strengthen information collection, analysis and use in decision-making include: provision of computer hardware/software in joint DHO-SCF/US office; training of Expanded District Health Management Team (EDHMT), HC and zonal staff in data collection and use; close MIS support/follow-up of HCs by zonal officers; very high monthly levels of HIS reporting from HCs, (approximately 90%) which is reported to be the highest in the country; graphic presentation of HC and zonal data at quarterly EPMU meetings used as a basis for data-driven decision making; quarterly written/graphic feedback to HCs on results from all HCs in the zone. According to the SCF/US MIS advisor, although great progress has been made, considerably more on-the-job support and practice will be required in order to make the DHO staff autonomous particularly in terms of their knowledge of basic computer software and the new HMIS program. As mentioned above, as part of efforts to strengthen district information systems, there is a need in all districts to systematize collection of qualitative information, which describes and analyzes program implementation strategies and activities including accomplishments, constraints and lessons learned. For example, the various facets of the IV. FINDINGS OF THE EVALUATION 27 MIS-strengthening strategy in Mangochi should be comprehensively documented and shared with other CHAPS and non-CHAPS districts. Supervision When the project proposals were drawn up in the five districts, the need to increase the frequency and quality of supervision was clearly articulated by DHO staff. Based on this need all PVO partners have invested considerable resources and effort to improve supervision through: improved scheduling/coordination of supervision visits; provision of transport; training in formative supervision; revision/development of supervision tools; and provision of feed-back to HC staff based on supervision visits. The traditional approach to supervision tends to focus on inspection and fault finding. In all districts the PVOs have encouraged DHO staff to work together to modify the approach used in supervision to focus on dialogue and problem solving with health center staff. All of the national, vertical programs have their own checklists. In most CHAPS districts, PVO and DHO staff has jointly developed an integrated checklist, which incorporates parameters related to a number of technical programs. At the same time attempts are being made to promote joint supervision visits in which various program managers travel together to minimize logistical costs. In all CHAPS districts efforts are being made to ensure that at the end of supervisory visits HC staff are informed of the main observations and recommendations and it appears that this practice is becoming increasingly common. In Mzimba, after each supervisory visit a written summary of the conclusions and recommendations is sent to HC staff. In all districts supervision visits by DHO teams have become more frequent due to the organizational and logistical support provided by CHAPS. However, in spite of the advanced planning and provision of additional logistical resources, some constraints still exist related to the lack of availability of transport from the DHO level and the fact that DHO staff are not available because of their dual responsibilities for clinical services in the district hospital, in addition to involvement in activities supported by the multiple donors-partners working in each district. The shortage of DHO staff sometimes means that either the supervision visit is cancelled or the PVO counterpart carries out the supervision alone. While the PVOs are all committed to joint DHO-PVO supervision, sometimes they do conduct these visits alone in order to ensure that periodic supervision takes place. In Mzimba this latter problem has been largely overcome through the creation of the core groups whose members share responsibility for the technical DHO programs. For example, in the event that the program manager for RH, is not available for the supervision visit, another member of the core group can assume that responsibility. In all districts, DHO staff has primary responsibility for HC supervision and to some extent for community level follow-up as well. The supervision scheme adopted in Mangochi, in which the five ZCs play a critical role in supervision at those levels, appeared to be more cost-effective than the DHO-focused approach. In Mangochi the ZCs are involved in supervising the 33 MOH and CHAM health centers in the five zones. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 28 Each ZC is equipped with a motorbike, which enables them to visit each HC at least once a month, at considerably less cost than supervision visits originating at the DHO level. Given the limited and diminishing resources available to the districts, the key role played by the zonal offices in supervision, with shorter distances to cover and using relatively less expensive means of transport, the prospects for sustainability of supervision should be better than in other districts. Financial Resource Management In the context of decentralization, the ability of DHO staff to effectively manage both MOHP and donor resources will be of critical importance. A prerequisite to the strengthening of district accounting and financial procedures is the definition of policies and procedures in this area by the MOHP. In the CHAPS programs in the five districts efforts to strengthen district financial management were given limited attention. In most cases, PVO staff with financial and accounting skills managed the CHAPS resources. They were not explicitly involved in trying to strengthen the skills of DHO staff in accounting and financial management of overall district resources. Efforts to strengthen district financial planning and management were strong in Chikwawa. The project would have been strengthened if PVOs had been able to give greater attention to transferring accounting and financial skills to District Health Office staff either through full-time staff who have this expertise or through outside consultants who could provide occasional TA. Drug Management About 35% of all MOHP resources provided to the districts are invested in drugs. In the decentralization of responsibility to the district level, DHOs are progressively being expected to take more responsibility for managing and distributing drugs and supplies from the national level central stores and pharmacy. Within the CHAPS districts, limited efforts were made to improve the management of drugs and supplies at the DHO level. In some of the health management training provided for DHO and hospital staff, drug management was addressed. However, it seems that methodical efforts were not made to work with the staff of the district stores and pharmacy in order to improve their systems and skills. The program would have benefited from more attention to strengthening drug management at the central district store and pharmacy to include improved storage methods, inventory control, forecasting techniques, distribution systems and computerized management of stocks, orders, etc. In all districts the main support provided by CHAPS to improve access to drugs was the establishment and reinforcement of community DRFs, which are discussed below. IV. FINDINGS OF THE EVALUATION 29 Development of Training Skills An important function of DHO staff is to plan and conduct training activities for various target groups. One of the needs identified in the original RFA was to strengthen the training capabilities of district staff. The impact of district level training is greatly dependent on the quality of the training provided, which in turn is largely dependent on the pedagogical approach used and on the facilitation skills of the trainers. Most DHO staff are accustomed to using directive, lecture-based teaching methods, which are known to be less effective in promoting adult learning. In all five districts, for CHAPS-supported training activities the DHMT staff have been involved in workshop planning and facilitation. It seems that through joint MT-PVO preparation of training plans, there has been some transfer of skills related to: preparing training budgets; developing training lesson plans which involve the use of more active learning methods; and facilitating training sessions. However, more could have been done in this regard, including a course for master trainers, especially to strengthen facilitation skills which are required not only for training events but also for all team/working group activities. Transport/Logistics In all five CHAPS districts considerable attention and resources have been invested in improving transport and logistics systems particularly for supervision at various levels, distribution of drugs and supplies and emergency patient referrals. This support is perceived by all District Health Office staff as being very beneficial. In the current resource-constrained environment, developing transport/logistical systems which can be sustained with lower recurrent costs is a critical challenge which all of the PVOs have attempted to address in providing support for this component. At the District Health office, CHAPS strategies have focused on increasing the number of operational vehicles and on fleet management. At the health center level the project has focused on providing less sophisticated and less expensive means of transport for HC staff and zonal officers (motorcycles) and for HSAs (bicycles). In all districts one or more vehicles have been purchased for the District Health Office with CHAPS resources. In several districts studies were conducted to analyze the status of the vehicles, fleet management and to recommend strategies for transport policies, improving maintenance, drivers’ competencies, transportation plans etc. Particularly in Mangochi, Salima and Chikwawa considerable improvements were observed in the capacity of the DHMT staff to optimally manage their vehicle fleet. In all districts it is reported that CHAPS logistical support has contributed to the increased frequency of supervision. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 30 Strengthening Technical Programs In each District Health Office there are a number of officers/program managers who are responsible for different technical and skill areas which can include: HIV/AIDS; RH; Malaria; Expanded Program on Immunization (EPI); IMCI; Breastfeeding; Control of Diarrheal Diseases (CDD); QA; HMIS; and W&S. Almost all of the program managers have clinical backgrounds, with the exception of those who trained in environmental health and the majority has at least some expertise in planning, management and coordination of field activities. On the PVO side, there is a staff person who operates as a counterpart to each of the program managers. The support provided by the PVO counterparts aims to address these areas, as well as ensuring that the DHMT staff’s technical knowledge is up-to-date and they are aware of pertinent MOHP policies and program guidelines. CHAPS’ support to these technical programs in all five districts has consisted of research activities and studies, formal training, day-to-day collaboration and mentoring. This has been accomplished through joint program planning, training implementation, supervision and other activities at HC and community levels as well as monitoring and reporting on activities carried out. All program managers interviewed stated that the CHAPS￾supported activities they have participated in have been very beneficial to them, particularly the formal training and the day-to-day support they receive through working closely with their PVO counterparts. However there is no existing mechanism to assess the impact of the various CHAPS-sponsored training activities in which they have participated at district, national and international levels. As regards the value of the daily mentoring that occurs, the following quotes from DHMT staff suggest how they perceive this support: “They are like my teachers”; “Some donors just send us money. But it’s better having the PVO staff close by to help us plan and implement”; and, “Just because someone appoints you doesn’t mean you know how to manage a program. We are working hand-in-hand.” Support for HIV/AIDS Programs The evaluation team was struck by what seemed to be a low level of concern and involvement on the part of the DHMT staff and other district officials in HIV/AIDS activities. In all districts the PVOs have played an active role in advocating for increased involvement of both district staff and community members in HIV/AIDS strategies. The PVOs have also provided technical and financial support for the development of HIV/AIDS activities including: formal training at various levels related to counseling, breastfeeding and PMCT; establishment of VCT centers; establishment of District AIDS Coordinating Committees (DACC) and Community AIDS Coordinating Committees (CACC) which are part of the national HIV/AIDS strategy. IV. FINDINGS OF THE EVALUATION 31 Health Center Level: Capacity Building, Improving Equipment and Services As a basis for developing HC-strengthening strategies, early in the project (l998) very useful health facility assessments were carried out in all five CHAPS districts with assistance from the QA project. The assessments looked primarily at: supervision; logistics and drug supply; information management; referral systems; and client satisfaction. The findings of these assessments served as a basis for development of district HC-strengthening strategies. CHAPS strategies to strengthen HCs have focused on improving: 1) services provided at the HC level; 2) outreach services/clinics conducted to some extent by clinical HC staff but primarily by the HSAs; and, the referral system. While core HC staff are supposed to be involved in supporting other community-based activities (DRFs, TBAs, Community Based Distributors (CBDs), etc.), in reality district staff involved in service provision at the HC level are very rarely involved in these activities due to their other multiple responsibilities. It is important to point out that CHAPS activities equally support programs and staff at MOH and CHAM health facilities, which are all under the responsibility of the DHO. In terms of cost-effectiveness, the prospects for sustainability of capacity-building efforts at the HC and community levels may be greater than at the DHO level given the relatively lower turn-over among core HC staff, and particularly among HSAs and community actors. This observation was made in several of the PVO Mid-Term Evaluations and was repeated in many interviews at district level. Capacity-Building: Training, Supervision and QA District health staff at the HC level include both the HC-focused staff, consisting of one or more clinically trained persons and other support staff who are involved in service delivery at that level, and the HSAs (usually between six and twelve per HC), most of whom are based in catchment zone communities and who work primarily with outreach clinics and other community-based activities. In most cases capacity-building activities for the two sets of HC staff have differed and they are discussed separately here. Strengthening Health Center-Based Staff A major contribution of CHAPS in all districts has been the extensive formal training/refresher training that it has supported. CHAPS efforts to strengthen facility￾based HC staff aim to improve the range and quality of services provided, as well as the integration of services. Capacity-building activities with HC staff have consisted primarily of formal training and follow-up supervision. These activities have focused more on the technical program areas and to a lesser extent on the organizational and management skills of HC staff. In the five districts numerous formal training activities have been carried out. Given the time constraints of this evaluation and the brief time available in each district, it was not REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 32 possible to have a comprehensive understanding of the quality of the training carried out, nor the extent to which trainees have used the skills acquired through training. It is possible, however, to provide an overview of the types of training carried out and some feedback on them. Formal training for HC In-charges and other staff members have included workshops on various technical topics, e.g., RH/FP; EPI; HIV/AIDS; Malaria; ARI; Exclusive Breast Feeding (EBF) and complementary feeding; IMCI; counseling; and, management of STIs. Based on reports from HC staff and DHMT program managers, these training activities have increased the technical knowledge and skills of HC staff and contributed to bringing their skills in line with MOH guidelines for the various programs. In all districts, CHAPS-supported training sessions have to some extent addressed HC organization and management issues including: planning, teamwork, leadership, communication skills; conflict resolution; HIS; and drug and pharmacy management. However, these dimensions of HC capacity building need to be given more attention in the future. The district in which more attention has been given to strengthening HC organization and management is Mangochi. All HC in-charges and ZCs, attended a one-month MIM course on basic management and teamwork. HC In-charges, ZCs and PVO staff all report improvements in the ability of these field staff to manage both human and other resources since the training. Other mechanisms and activities which have strengthened HC management include: monthly meetings of HC staff with ZCs; ZC support to HCs for completion and discussion of monthly MIS data (approximately 90% monthly returns from HCs); and quarterly work plans developed with each HC/HSA team based on HC and community data. All of the PVOs have promoted the idea of more integrated programs to include integrated supervision and service delivery, some with greater success than others. In some of the CHAPS districts IMCI training was supported starting in 2000 and district supervision has focused on follow-up of HC staff trained in the integrated approach. Discussions with a number of HC staff revealed that they are interested and committed to the new approach even though it is more time consuming. They also reported that mothers favor the new approach because it helps decrease the number of visits they need to make to the HCs. In all districts attempts have been made to improve the quality and frequency of HCs. In all districts there have been improvements in supervision although in most districts regular supervision does not always take place, given the constraints discussed above. Quality Assurance QA activities with Quality Improvement Teams (QIT) have been carried out in all districts as one facet of CHAPS efforts to improve services provided at the facility level. URC provided TA for QA activities, which were followed up by the district and PVO staff. As with other capacity-building and training activities supported by CHAPS, the IV. FINDINGS OF THE EVALUATION 33 QA activities have suffered greatly due to the departures of many of the district level QA coaches. Without their leadership and follow-up it has been difficult for many of the facility level teams to continue with problem-solving activities using the QA framework and tools. In general, DHMT staff appreciate the QA approach and activities and feel that they are contributing to improved quality of HC management and service delivery. However, constraints with the QA activities were also identified by some PVO and district health staff: to many health workers the QA concept is not clear, or too complicated; others stated that the concept is foreign to the local culture; some view it as a burden rather than a tool to improve the quality of services; and some QA teams are unwilling to meet without receiving allowances. In spite of these constraints, there are positive examples of how the QA problem-solving approach has been used to solve problems regarding: non￾compliance with malaria treatment; patient-waiting time at health facilities; infection prevention; and strengthening HC teamwork. In order to increase the impact of QA activities, it would be beneficial to identify ways to simplify the overall approach used with the QITs. Referral System Prior to CHAPS, in most cases direct communication between HCs and the district hospital did not exist and transporting emergency cases was often very time consuming. In all CHAPS districts efforts were made to improve the referral system from HCs to the district hospital. This involved installing short-wave radios in all HCs and training HC staff on their use. This system is currently working well and HC staff report decreases in the waiting and transfer time for the district ambulance to transport emergency cases to the district hospital. They are very satisfied with the system which facilitates communication not only related to emergencies but also for more routine communication regarding drug supply needs, etc. Another innovation developed in all districts that was intended to facilitate emergency evacuations is the bicycle ambulance. In most cases these bicycles, with a flat platform attached to them, were given to communities to use especially for emergency deliveries. Unfortunately, the success of the bicycles has been mixed due to certain socio-cultural practices, which discourage pregnant women and others from being seen transported in them. Other assessments have documented similar findings about the inappropriateness of transporting pregnant women by bicycle ambulances (e.g. SCF/UK, P. Poore, etc.). It was reported that the bicycle ambulance was imposed on communities and many do not consider it an acceptable mode of transport for pregnant women. Strengthening the HSAs The HSAs, at the lowest level of the district health staff pyramid, provide a critical link between HCs and communities. Each HSA is responsible for between four and six communities/villages and resides in one of them. The majority of the HSAs are men and most were originally trained as environmental health agents and were primarily REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 34 responsible for water and sanitation activities with a focus on involving community committees/groups in these activities. In the context of CHAPS, their training and responsibilities have been expanded to include family planning (FP), HIV/AIDS, DRF, CBDAs and TBAs. It is reported that the national HSA training curriculum is broadening the scope of HSA responsibilities, while trying to ensure that they are not over-extended. CHAPS-supported activities to strengthen the work of HSAs has mainly involved: formal training sessions; improved supervision of their work; and improved transport, i.e. bicycles. Training Activities with HSAs In most of the districts a significant number of new HSAs have been recruited since CHAPS began and the project has supported their basic training. In addition, in all CHAPS districts formal training activities have been organized for both the “old” and “new” HSAs on various topical areas, primarily: RH/FP; DRFs; TBAs; CBDAs; supervision; HIS; HIV/AIDS prevention and home-based care; latrine and dome slab construction. Supervision of the HSAs is ensured by the senior HSAs at each HC and occasionally by the district supervision teams. The involvement of HSAs in data collection and use remains a weak area in most cases. In the five districts, most HSAs are doing a good job of periodically collecting data at the community level; however, there is less evidence that they are using the data to identify problems and for action planning. Given the central role played by HSAs in all community level activities, all districts should continue to strengthen supervision and coordination of HSAs’ work, as well as on￾going skill building. More attention needs to be given to strengthening HSAs’ skills in participatory methods that promote the empowerment of community groups and actors rather than dependency on the HSAs. Bicycles for HSAs Providing a means of transportation can increase HSA involvement with the communities in their area of responsibility and in many districts bicycles have been either loaned or given to them. In past experiences, however, HSAs have not maintained these bicycles, which they would one day turn over to the HC. In order to increase the HSA’s sense of ownership and responsibility for the bicycles, in both Mulanje and Mangochi it was decided to purchase and sell bicycles to the HSAs at a subsidized rate. After several years, it is reported that all of the HSA-owned bicycles are in working order because of the good maintenance and care provided by their owners. Bicycle ownership is a Best Practice that should be systematically documented and shared with other districts in Malawi. IV. FINDINGS OF THE EVALUATION 35 Community Level: Improving Access and Quality of Services, Increasing Community Involvement in Health Promotion The third level at which CHAPS aims to strengthen district health programs is the community level where the objective is to improve access and quality of outreach clinics and services, and to increase community involvement in health promotion. In the context of decentralization in Malawi, and the anticipation of increased involvement of communities in local development, communities will be expected to play an increasing role both in articulating their own needs related to health development and in taking responsibility for community-based health promotion activities. A major challenge will be to ensure development of strong community organizations, which can effectively play these roles. The experience of the five partner PVOs with community level work has been very relevant to CHAPS efforts to strengthen community services, structures and actors. Improving Access and Quality of Outreach Services In all CHAPS districts considerable emphasis has been put on strengthening basic services, primarily for children under five and women, through outreach strategies and clinics. Prior to CHAPS, although it was expected that HC staff would carry out outreach activities in their catchments areas, their ability to do so was limited by a series of factors, e.g., lack of transport, shortage of staff and limited staff availability due to their multiple HC responsibilities. In addition, many HC staff was not adequately trained in some areas including RH/FP, and at the village level there was a lack of basic infrastructure wherein RH/FP activities could be carried out. CHAPS strategies in all districts have attempted to address several of these constraints through: construction of simple shelters where both under five and RH/FP activities can be conducted; capacity-building of HC staff in RH/FP; support for transport/logistics; and, increased involvement of the HSAs in outreach activities. Although staff shortages and the limited time HC staff have for outreach activities are among the greatest constraints to improving district health services, these problems are not within CHAPS purview. However, CHAPS has tried to address the issue in a limited way by delegating more responsibility for outreach activities to the HSAs. Increased Involvement of the HSAs in Outreach Activities With CHAPS support, the skills of the HSAs have been considerably strengthened through training (broadened HSAs range of topics/skills), improved supervision of HSAs and provision of bicycles to HSAs. A frequently mentioned constraint associated with the HSA's work, however, is the tendency to assign them an unreasonable number of tasks and responsibilities. Construction of “Under-Five Shelters” In several districts, including Mulanje, Chikwawa and Salima, CHAPS has provided support (building materials) for construction of simple shelters by community members. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 36 In the past, outreach activities addressed only under-five children. The shelters provide a private place where basic services can also be provided to women (RH/FP, antenatal consultation visits). For example, in Salima the number of outreach MCH clinics has increased from 12 in 1998 to 64 in 2000. In support of these activities, the number of FP providers has also greatly increased from 25 to 95 in the same time frame. Increasing Community Involvement in Provision of Basic Services and Health Promotion In all CHAPS districts various strategies have been developed with DHO program managers to initiate and strengthen the involvement of community organizations and actors in health promotion activities primarily related to: HIV/AIDS; DRFs; TBAs; CBDAs; malaria; and, W&S. These efforts have focused on training, supervising and supporting both village-based organizations, namely Village Health Committees (VHCs), and community members/volunteers, including Community Health Volunteers (CHVs) and TBAs. In all of these activities, HSAs have primary responsibility for providing on￾going support to community groups and individuals involved in providing services and promoting improved health practices. HSAs are usually responsible for four to eight villages which allows them to supervise activities in all villages at least once a month, and often more frequently. Strengthening Village Health Committees and Community Health Volunteers Experience in many countries has shown that community involvement depends both on existence of community organizations/structures and leadership, which is committed to promoting community health development. The effectiveness of community structures in health promotion greatly depends on two key dimensions: first, their organizational capacity to analyze problems, plan activities and evaluate the implementation of those activities; and second, and their basic technical knowledge of specific health issues/topics. In all CHAPS partnerships significant human and financial investments have been made to strengthen both these dimensions of VHCs’ and CHVs’ knowledge and skills. CHAPS-supported efforts have focused on strengthening systems of organizing, training and supervising VHCs and CHVs. For example, in Mangochi district, which has more than 800 villages, more than 700 VHCs have received either initial or refresher training in CHAPS. In the evaluation it was not possible to determine the degree of effectiveness of the numerous VHCs trained in all five districts. Given the importance and challenges associated with strengthening VHCs and CHVs, it would be very useful to document and synthesize lessons learned working with both VHCs and CHVs, both from CHAPS districts and from other districts in the country. Unfortunately, in the context of CHAPS mechanisms for documenting district program components such as this were not developed. IV. FINDINGS OF THE EVALUATION 37 Strengthening Drug Revolving Funds (DRF) An important activity developed in all CHAPS districts involves the establishment and/or reactivation of DRFs. Especially for communities that are far from HCs, where drugs are distributed free of charge, community members are very grateful for the DRFs, which provide improved access to basic drugs. During CHAPS the number of DRFs increased in all districts. The DRFs usually include Fansidar, iron folate, paracetemol, aspirin, ORS and sometimes tetracycline eye ointment (TEO). Training of DRF volunteers deals mainly with case management and record keeping. Other activities in support of DRFs include: establishment of a buffer stock of drug supplies at the HC level which are separate from HC stock; development of a check-list for HSA supervision of DRFs; and, depositing DRF funds at the closest post office. The majority of DRFs appear to be operating quite well and have learned to order drugs from the HCs well before their supplies are depleted. The main problems encountered with the DRFs are: mismanagement of DRF funds, including use of the funds for other purposes; inadequate record keeping; and shortages of drugs. Drug shortages at the DRF level are usually due to drug stock-outs at the HC level, which in turn are due either to poor management of buffer stocks by HC staff, to drug shortages at the district or national pharmacy, or to lack of transport between the district and the HCs. Several important lessons emerge from the CHAPS work with DRFs. First, DRFs will not be viable unless systems and follow-up are provided at the central MOHP level to ensure that drugs are consistently available. Second, regular supervision of the DRFs, by the HSAs, is essential in order to identify problems early on and correct them as soon as possible. DRF program managers and HSA coordinators at the DHO level need to ensure that the DRFs are regularly supervised by the HSAs. Third, involving a number of community members in DRF training increases transparency and oversight of all DRF activities. In Mzimba there has been considerable investment both in establishing and revitalizing DRF’s and several innovative activities have been developed: a DRF newsletter is periodically disseminated both in the local language (Tumbuka) and English; the DRF manual has been translated into Tumbuka; diagnostic and treatment sheets have been developed for use by DRF dispensers; and a very good DRF training module has been developed. Strengthening TBAs In rural areas in Malawi almost half (47%) of all women deliver at the community level (DHS 2000). While the maternal mortality rate is very high in Malawi and the MOHP and health workers in CHAPS districts would like all women to deliver in health facilities with trained birth attendants, various factors contribute to the fact that many women do not deliver in the formal facilities, e.g., the lack of transport and/or distance to the closest health facility and insufficient time to reach health facilities after labor begins. In various REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 38 community and health facility studies supported by CHAPS, QA, and others, another factor identified which often discourages women from delivering in the health facilities is the negative attitudes of health workers. While this problem/constraint has existed for many years, it persists as one factor, which contributes to community deliveries and the use of TBAs. In all CHAPS districts the DHO-PVO partners agreed that while women should be encouraged to deliver in HCs, it is unrealistic to anticipate that all of them will and that CHAPS should attempt to improve community deliveries by improving TBA practices. For this reason, in all five districts TBA training and supervision has been an important activity under the responsibility of the DHO program officer for RH/TBAs. The HIV/AIDS epidemic provides further justification for improving TBA practices (e.g. gloves, cleanliness, etc.). The training that has been conducted has been based on the national TBA curriculum and the number of trained TBAs has increased considerably in all districts. TBA kits have been distributed to newly trained TBAs as well as to some who were trained at an early point in time. A concern in this regard is how the TBAs will replenish their kits, e.g. gloves, razor blades, etc. At present they get these supplies from the HCs, depending on availability. This issue needs to be further examined by the districts in order to identify a sustainable strategy to ensure that TBAs are not using the same razor blade several times, or delivering without sterile gloves. In all districts the HSAs are expected to do the follow-up supervision of the TBAs and many have been specifically trained for this task in the context of CHAPS. According to all reports, HSA supervision of TBAs has increased, although in the evaluation it was not possible to precisely determine the extent to which regular follow-up visits take place, nor the quality of the supervision. Strengthening Community Based Distribution Agents Efforts to improve access to FP services at both HC and community levels have contributed to the considerable increases in CPR in all CHAPS districts. In addition to expanding and improving HC FP services, in all CHAPS districts there have been considerable efforts to improve access to contraceptives by increasing the number of CBDAs promoting FP and distributing FP methods (pills, spermicides and condoms) at the community level. Distribution of condoms by the CBDAs serves a double purpose related to both FP and HIV/AIDS prevention. While women overwhelmingly prefer Depo-Provera as a contraceptive method, it can only be obtained at HCs. Given this situation, women very much appreciate this community-based service and due to easier access many use oral contraceptives though it is not their method of preference. The number of oral contraceptives being distributed by the CBDAs is considerable. In Mzimba, for example, almost half of all oral contraceptives distributed in the district are given out by CBDAs. In all districts new community volunteers have been trained as CBDAs and many others have been refreshed. The HSAs are responsible for replenishing CBDA supplies from HC stocks. Other activities carried out in various districts in support of CBDAs activities IV. FINDINGS OF THE EVALUATION 39 include: meetings with VHCs to explain the role of the CBDAs to ensure their support; quarterly meetings with FP core providers to discuss both HC and CBDA FP activities; increased and improved supervision of CBDAs by HSAs, and in some cases by HC staff. Promotion of both FP and HIV/AIDS prevention is also carried out through CHAPS supported literacy classes for women in Chikwawa. Constraints encountered with CBDA activities include: inadequate supervision of CBDAs due to increased numbers, inadequate transport for HC staff, inaccessibility of some areas during the rainy season and HSA’s lack of commitment to FP activities which are not yet included in their performance index; inadequate reporting of CBDA activities to HCs; and occasional shortages of CBDA supplies when HSAs fail to re-supply them or when there are stock-outs at the HC level, which appears to occur relatively infrequently. As with activities at other levels of the health system, periodic supervision of the CBDAs is essential if these activities are to be maintained, if problems are to be identified and dealt with in a timely fashion, and if proper records are to be kept for reporting and planning purposes. Also, it is important that the MOHP incorporate HSA responsibility for FP promotion into their scope of work to increase their commitment to these activities. At the present time, it was found that CBDA supervision is more frequent, as is the reporting on CBDA activities in Mangochi district. This is undoubtedly due to the zonal officers who provide close follow-up and support to the HSAs, the primary CBDA supervisors, and who also conduct some community level visits themselves, which HC staff are generally unable to do. In all districts on-going efforts need to be made to ensure regular, formative supervision of CBDAs by the HSAs and core family planning staff. In order to further strengthen these activities it would be beneficial to document the lessons learned working with CBDAs in all five districts based on both successes and constraints encountered. Developing Community HIV/AIDS Strategies The evaluation team was struck by the low level of concern expressed by district staff regarding the HIV/AIDS epidemic in the country. This was found in all sectors in the district, not just health. Both PVO staff commitment to this problem and CHAPS resources have provided considerable impetus for the development of HIV/AIDS activities both within the district health system, in programs in other sectors (particularly education) and with other district partners through the District AIDS Coordinating Committees. In the first phase of CHAPS all district partnerships were involved to some extent in HIV/AIDS strategies related to both prevention and care. In light of the quickly evolving epidemic in the country, in the extension phase of the project, the districts were asked to give particular focus to this component. CHAPS-supported HIV/AIDS efforts are directed at several levels. At community level, activities have included health education and training of VHC, CHV, CBDAs, TBAs and youth clubs related to: orphan care, support groups for home-based care, prevention of Mother-to-Child Transmission (MTCT); prevention of HIV/AIDS; condom distribution; and, establishment of REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 40 Community AIDS Coordinating Committees (CACC). In addition, community volunteers have been trained in counseling and home-based care to form community-counseling groups to provide support to persons living with HIV/AIDS (PLWHA) and their families. All districts are developing VCT services. At the HC level, training for health workers has been conducted on: HIV/AIDS prevention; MTCT, including advice on breastfeeding for HIV positive women; infection prevention (in collaboration with the QA project). At the district level CHAPS partners have provided support to the DACC in terms of encouraging involvement of inter-sectoral and multiple donor partners. As noted above, CHAPS is supporting several HIV/AIDS activities are being carried out at various levels in all five districts, which is in contrast to the limited attention districts officials were found to focus on HIV/AIDS. Therefore, it is of critical importance that a mechanism be developed whereby the lessons learned from these experiences with HIV/AIDS can be documented and widely disseminated in order to accelerate program learning and ensure that future investments are made in the most cost-effective ways. Health Education/Information, Education and Communication Activities In all districts health education/IEC activities are carried out at the community level which involve the HSAs and/or community volunteers, namely the VHCs, CHVs, TBAs, CBDAs and DRF committees. Most of these IEC activities consist of traditional, one￾way message dissemination in the form of health talks, sometimes accompanied by pictures/flipcharts. The effectiveness of these approaches is probably limited in so far as they tend to be boring for people, involve telling people what to do and do not require active participation and critical analysis by community members. In several of the districts more participatory, innovative approaches to health education/IEC have been used involving: training local community groups in “drama for development”; training local bands to develop and present songs on different health topics; and the use of songs with community groups. In several districts (e.g. Mulanje, Chikwawa) health topics, including malaria, FP, HIV/AIDS, have been integrated into adult literacy materials. Thus, increased use of such methods, especially songs and stories which are both part of Malawian communication traditions, are enjoyable, require active participation by community members, could stimulate community members to identify their own strategies for dealing with various health-related problems rather than expecting them to adopt pre-set solutions and can be used by community groups on their own. Malaria In all CHAPS districts, through the DHO Malaria Program Coordinator, some efforts were made to improve malaria prevention and treatment activities at the HC and community levels. At the HC level, staff were refreshed in order to improve case management of malaria and according to DHO and PVO staff, their knowledge was reinforced during supervision visits. Malaria case management is also being reinforced IV. FINDINGS OF THE EVALUATION 41 in the IMCI training, which has begun in all districts. At the community level prevention and access to malaria drugs for early treatment are promoted through: bed net revolving funds and re-dipping activities; health education by HSAs and CHVs; as well as training of DRF distributors in malaria treatment and referral and availability of fansidar in DRFs. It would be useful to do a follow-up study to assess the adequacy of treatment advice being given by the DRF distributors. The most extensive impregnated bed net strategy was developed in Salima, which reports indicate is the district with the highest present coverage of ITNs in CHAPS districts. According to the DHO Malaria Program Coordinator, community members are very motivated to use ITN. There are several constraints, however, to optimal ITN usage: most families have only one bed net and it is usually the parents, rather than the children who use it; the market bed net price has gone up considerably and at the same time sales have fallen due to the very limited purchasing power of most households. In Mulanje, in order to improve treatment advice being given out by shopkeepers who sell fansidar, an innovative strategy has involved training these merchants in basic malaria diagnosis and treatment. It would be valuable to document the lessons learned related to these preventive and curative activities and disseminating them widely,, e.g. to other districts, organizations and MOHP. Water and Sanitation District heath programs and projects invariably include health education activities related to promote behavior change related to hygiene, sanitation, diarrheal disease and skin diseases but few programs address the underlying factors associated with the lack of clean water and latrines. In all five CHAPS districts, considerable resources have been invested in W&S activities aiming to increase access to clean water and to waste disposal facilities (e.g. pit latrines.) Many district health staff expressed their satisfaction with CHAPS W&S activities that have addressed these basic communities needs while at the same time being important preventive measures. All activities in this area were coordinated by DHO Environmental Health staff and in most cases implemented in collaboration with local Ministry of Water Development staff. In Chikwawa the W&S component is managed through a sub-grant to Concern Universal which is specialized in community W&S. More specifically, and depending on the topographic/hydraulic conditions in each district, CHAPS W&S activities to increase access to safe water consisted of: bore hole drilling and pump installation; well and spring protection; training of village volunteers in maintenance of protected water sources. In most cases the CHAPS PVOs are not the only organization supporting W&S activities in the district. In Mulanje, for example, since the mid-l990’s access to safe drinking REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 42 water has increased from 27% to 54%. Project HOPE has made a contribution to these increases in CHAPS and in the earlier child survival project. In all CHAPS districts great efforts have been made to increase access to pit latrines through: provision of sanplat casts, cement and training of local artisans to make dome sanplats; health education and promotion with VHCs to promote latrine construction, cleanliness and use. Access to latrines has considerably increased in all districts, for example, more than 600 sanplats have been installed in Mulanje. In Chikwawa more than 1000 latrines were built between 1998 and 2000, increasing latrine coverage from 25% to 36% of all households. Unfortunately, the severe floods in 2001 washed away many of the latrines and coverage fell back to 28.5%. While it is difficult to confirm, DHO staff feel that improvements in W&S infrastructure in the context of CHAPS has helped to prevent and control the yearly cholera outbreaks, which affect all five districts. For example in Losi village in Chikwawa, community members said that earlier the village had several cholera cases every year, but in the last two years since the installation of the water system, there have been no cases of cholera. During the cholera epidemics, all partner PVOs provided emergency assistance to the district in terms of intravenous (IV) fluids, other supplies and transport. According to the Assistant Environmental Health Officer in Salima, due to great investment of efforts to train and supervise water committees most have now assumed a sense of ownership of their water source and have assumed responsibility for maintenance of their pumps. Lessons learned regarding both the W&S pumps and latrines and software should be documented and disseminated between CHAPS districts, and with other districts in the country. Increased Food Security Similar to water and sanitation, access to nutritious foods is one of the underlying pillars of health. However, in most health programs nutrition-related interventions are limited to nutrition education. In two CHAPS districts activities to address the severe food security situation were undertaken. In Salima CHAPS provided families with young children with seeds for Vitamin A-rich pumpkins, carrots and okra and fruit trees (paw-paw, mango, guava and cashew). In Chikwawa, nutritional support for HIV/AIDS patients has been provided to them and their families in the form of both ground nut and vegetable seeds, and small animals, namely guinea fowl and rabbits for home production E. FINANCIAL MANAGEMENT OF CHAPS The financial management review addressed a broad group of needs and expectations designed to assist CHAPS stakeholders in the achievement of three sets of objectives: 1) the objectives specifically noted in the scope of work; 2) objectives to help guide stakeholders when considering suitable strategies for future program design; and, 3) to IV. FINDINGS OF THE EVALUATION 43 promote the implicit objectives of basic financial management. These include effectiveness and efficiency in operations; reliability of internal and external financial reporting; compliance with internal and external rules and regulations; safeguarding resources; and, contributing to an operational environment where limited resources can be employed yielding the greatest benefit to the people of Malawi. The following progress was made under the CHAPS model of public private partnerships in supporting the DHMTs. Establish and Build Effective Models of Public/Private Partnerships The original design of the CHAPS program was based on a given funding structure rather than a coordinated need-based scheme to affect concomitant change. Sustainable economic analysis support and financial management structures and systems should be intertwined and “built into” the underlying business and operational processes, not “attached.” Progress in this regard was found to be weak during CHAPS. However, CHAPS was a new project and therefore the lessons learned from this experience should be considered in future program design. Strengthening the Public/Private Partnership at the District and Central Level within the MOHP Lessons to be learned from CHAPS that were not adequately addressed include: ƒ The need for complementary skills and management styles including the need for strategic synergy between the two partnership organizations. At times, conflicting management styles among stakeholders created problems in some partnerships. ƒ The need for compatible operational and accounting systems. Although the PVO was accountable for the funds, parallel accounting systems or a better understanding of the existing system would have enhanced the partnership. Instead in some cases, the CHAPS effort increased some of the work and interfered with partnership transparency and coordination. Insufficient Levels of Financial Transparency The PVO controlled all of the money which influenced collaboration with the DHMT. ƒ Inadequate consideration of the mixing of organizational cultures, procedures and policies (e.g. per diems, accounting systems, reporting requirements, etc.). ƒ A failure to promptly manage dysfunctional working relationships. ƒ Inadequate attention paid to the realities of financial sustainability. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 44 ƒ Failure to adequately focus and prioritize given available resources and time frame. ƒ Unrealistic expectations. ƒ Project teams that lacked financial expertise. ƒ Inadequate communication. ƒ Failure to put effective metrics in place to insure that results are adequate. Enhance DHMT Capacity Through the Transfer of Service Delivery and Management/ Nutrition and Financial Skills Between International NGOs and the MOPH The actual transfer of financial management skills and knowledge at the central district level was limited under CHAPS. This is demonstrated by the following examples. The PVOs accomplished 100% of the financial accounting for the funds of the partnership. Given this one-sided division of roles, transferring financial management skills related to partnership management was precluded. For example in one case, the PVO accountant had never met the DHMT accountant. Regular planned meetings of the partnerships financial line-staff did not occur. Financial management job descriptions were in need of updating and revision. Specifically, the desired financial outcomes vis-a-vis clinical interventions that need to be accomplished and the duties and responsibilities that need to be discharged in order to accomplish the intended financial results were lacking. Performance evaluations related to the financial positions within the partnership were not done and it was found that these would improve the quality of health services and facilitate the health status of target populations. With the exception of one district, the CHAPS program did not attempt to automate the ledger systems within the central district. In the one district where this attempt was made, the possibility for automating and employing an improved ledger system was adequately demonstrated. This was documented in a July 2000 consultancy report where the financial information and communication systems and some key constraints and challenges to that process were noted. The situation has not changed materially, except in some observed cases. The ledger maintenance and information system is still manual and remains inadequate. The 2000 report did not discuss the effort by the Ministry of Finance (through the Treasury) to automate the ledger and reporting process through a system called Integrated Financial Management Information System (IFMIS). The system, with some computers partially funded through World Bank funds, is currently being installed in a number of ministries, including the MOHP and will form the backbone of financial information across sectors. IV. FINDINGS OF THE EVALUATION 45 Additional Ways of Strengthening the Public/Private Partnership at the District and Central Level Within the MOHP Interviews about central district level training with district accounting and managerial staff indicated three opportunities for improvement. First, due to staffing shortages, when people were pulled away for training it presented operational continuity problems. Therefore, future training should be highly prioritized and sequenced. Second, when training occurred at the central district level, there was subjective concern that the “wrong” person was occasionally sent or post training follow-up support was not appropriate. For example, on two occasions management training was provided to improve drug and inventory management. Yet, when interviewed, the pharmacy technician/manager who operates the central district facility could not recall any direct or indirect training. In one case, a clinical officer had been sent. In another district, the pharmacy manager was appropriately trained under an effective and well thought-out curriculum by extremely competent trainers. The individual utilized the training tools and techniques until his computer failed a year ago. Since that time, the operational value of the training has almost stopped. Third, there appeared to be a reasonable quantity and mix of all training subjects (including those outside of financial management) but concern exists over the priorities and coordination of training initiatives. With so few staff and such limited funds, programs should be guided by a clear and prioritized strategic training plan that fits with the operational and administrative priorities (including financial management) and follow-up resources of the partnership. Findings Related to Improvement of the Management and Quality of District Level Health Services and Health Promotion Capabilities of Communities and Family Caregivers From the financial management perspective, progress was minimal. Initial district-level priority activities from the financial perspective should be twofold: First, update the organizational structure of the district to best support national and district strategic initiatives. A financial management capacity that serves a cumbersome organizational structure is by definition, inefficient. Second, insure that a workable retention package is in place to preclude unnecessary staff turnover. Staff turnover carries an extremely high financial cost. This would be followed by concurrent work to apply IFMIS and HIS resources while mentoring the DHMT staff. Throughout this effort a concurrent rollout among peer districts is suggested. These measures should be started as soon as possible in the project rather than waiting until the PVO grant is ended. Contribution of the CHAPS Activities to the Achievement of Results of the Overall HPN SO Strategic Objective (HPN SO) of USAID/Malawi: Increased Adoption of Measures that Reduce Fertility and risk of HIV Transmission, Including Child Health Practices In addition to the discussion in section B about the Contribution of CHAPS to SO3, the findings of the financial review are: a) indicators were not adequately and consistently REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 46 tracked; b) there were no related or supporting/complementary financial indicators; and, c) accountability was too moldable. Support Provided by the Central Level MOHP Material support of an important foundational nature by the MOHP was observed in two areas. First, the quality of the financial staff. The MOHP implemented a University-level hiring scheme to place trained staff in these critical positions. Second, the manual accounting system is in need of automation. The MOHP has initiated a program to establish an integrated financial management information system (IFMIS) and has coordinated this effort with the Ministry of Finance. The implementation of the IFMIS coupled with the placing of more qualified University trained staff was viewed as a positive MOHP initiative. From a financial management perspective, the background work by the MOHP to plan, organize and begin implementation of these two initiatives during CHAPS is commendable. However, involvement and coordination by the CHAPS stakeholders in these areas was limited. The Unintended and Secondary Results An unexpected strategic result of the CHAPS model was the cross-sector work of some of the CHAPS players. A district level Cross-Sector Wide Approach (C-SWAp) was observed in one southern district where the PVO partner pooled resources and leveraged development efforts in the education sector with development efforts in the health sector. This was viewed as directly confronting poverty on the front lines. Pragmatic cross￾sector work at the district level should be encouraged and built into program design. In addition, if improved resource efficiency, effectiveness and coordination is the desired result of financial management capacity building, then resource coordination should be built into the project at the beginning and not after the “financial management capacity project” is complete. Determine the Effectiveness of Monitoring, Evaluation and Performance Monitoring Plans to Guide Program Direction and Monitor/Evaluate Project Activities An evaluation of monitoring was defined as a mechanism to assess the performance of the financial management control system. The scope and frequency of monitoring normally depends on the assessment of risks and the effectiveness of the monitoring procedures. In terms of district level financial monitoring, the CHAPS projects did not focus on the financial structures and internal transaction, collection and reporting schemes of the District. As a result, discreet CHAPS successes or interventions are not available. IV. FINDINGS OF THE EVALUATION 47 Additional Monitoring, Evaluation and Studies to Determine More Effectively Results and Best Practices to Guide Future District Health Service Improvement; and HPN SO/Mission Programming Activities and National/Regional Approaches to Public/Private Partnerships Financial management functions related to monitoring should include: ƒ A more comprehensive focus of integrating financial and non-financial monitoring. For example, insuring the monitoring and reporting of inventory levels and any resultant drug shortages are protected by a sufficient cash reserve to correct sudden changes in demand. ƒ A more cohesive approach to integrating strategic initiatives such as decentralization with monitoring systems. For example, before decentralization of finances can occur, each district should have clear objectives to the financial management department that represent “readiness criteria” for sustainable and responsible fiscal management and control by district staff. Such criteria should be regularly monitored. ƒ Financial monitoring efforts at the central-district should be better linked with a risk analysis of the key processes and mission-critical functions. Risk analysis should be used as a basis for prioritizing the effort. ƒ A methodical and documented focus on improving information quality. Data quality may be defined as improvements in: a) Data validity or appropriateness – does the data adequately represent performance? b) Reliability – is the collection process stable? c) Timeliness – are data collected frequently and are they current? d) Integrity – insuring data are free of manipulation. Future Cost Effective Programming Measures of Mission HPN SO Activities Related to CHAPS and Africa Regional Concepts for Public/Private Partnership Development Future program design should “listen to the numbers”. Only 25% of the sector is supported by the GOM; 65% of the population lives in poverty. This means donor and GOM resource coordination is required and development efforts must be capable of working not just intra-sector but cross-sector. This means projects such as CHAPS and SO’s must look beyond the health sector. Appraisal of Program and Financial Management Practices by CHAPS Implementing Partners, Cooperating Agencies and Key Operational Units An appraisal of the financial management practices revealed that the financial management structure needs to better support the stated mission, strategy, organizational REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 48 structure, key processes, staff duties, services, and the desired planned versus actual outcomes of the district. The CHAPS effort did not demonstrate this to an adequate degree. Given the new nature of decentralization, and only the very early emergence of coordinated district level budgets through the Medium Term Expenditure Framework (MTEF) process as well as other planning initiatives, it is too early to expect the financial management system to properly “support” the district. Therefore many of the findings based on the CHAPS experience are constraints that need to be addressed in future programs. These include: ƒ The CHAPS model provides for an excessive diversity of financial management structures. Dual or parallel accounting systems existed in every district with the resultat there there are five different accounting systems. One common integrated system shared by all districts would be preferable. ƒ The financial management staff and technical resources do not always serve the current organizational structures. A review of available district organizational charts revealed a complex and “old” organization chart with numerous new coordinator positions being “tacked” on to the organagram. This makes support by a financial management system challenging. The danger is that over time, an organizational structure can “evolve” that is as complicated as the system it is trying to oversee. Pto implementation of any aggressive interventions for improving the financial management systems and processes, a review of the organizational structure of a district should occur. ƒ Effective financial management must support key processes within the district. Periodic interventions by some CHAPS pilot sites to improve selected aspects of these processes did affect some positive change, particularly in the area of fleet management in some districts and DRF management at the community level. To sustain this and other efforts from a financial point of view, the CHAPS model revealed a need to better integrate operational processes with financial information systems. In this way, the financial function can support operations by assisting in areas that are crucial to mission success. For example, the processes that control staff, drugs and transport represent “mission-critical” issues for the district. As such, the “financial management capacity” by CHAPS stakeholders must move beyond score￾keeping and bookkeeping and become involved in process measurement and monitoring. Equally, the underlying Financial Information System (FIS) must be prepared to handle such data and analysis. ƒ Staff and their skill-set and duties, particularly within the financial management function, need to be better built around the demands of the organizational structure, information needs and key processes, not the reverse. With one exception, the CHAPS districts are essentially still operating with the same staffing mix in the financial management area as was defined before the positive changes in national policy. Eventually, there should be a review of this mix and skill-set to insure that it aligns with the demand. IV. FINDINGS OF THE EVALUATION 49 ƒ The financial management function needs to more directly and regularly support operations. The CHAPS model did not demonstrate this. The financial management practice should better identify problems, provide measures of costs and benefits, more frequently provide financial evaluation of alternatives, and better monitor and evaluate operational results. ƒ The financial management function is not currently providing an accurate and timely analytic capacity to forecast financial performance. Finally, internal control checks reveal the strengths and weaknesses of the financial and management oversight and operational structure. While this assessment was not an audit, the same principles of internal control can be applied to an assessment and used as a framework for analysis. The CHAPS partnership model was found to be positively influencing the internal control structure of the districts involved. Areas of improvement where particular progress was made were in the control environment, communication and selected control activities such as discreet activity (work plan) planning. This improvement, however, was not consistent across districts nor approached from the partnership’s perspective in a methodical fashion. Intuitively, however, PVO and district managers were seeking and noting successes in internal control by frequently remarking, “management culture has changed”, “new systems are in place”, “we are preventing shortages”, and more. The CHAPS partnership districts, therefore, were implementing an informal structure of internal control but did not know it. Sustainability of the CHAPS Approach from a Financial Perspective Any intervention that might require a variable cost to maintain the activity is currently at risk. Cash levels are extremely low, debt is being incurred and the flow of funds towards the end of this year vis-à-vis approved budgets is wavering. As such, the CHAPS model is currently not sustainable without external support. 49 V. FUTURE STRATEGIES AND MODELS FOR PUBLIC/PRIVATE PARTNERSHIPS A. EXPERIENCE BASED ON CHAPS MODEL As discussed above, the Evaluation Team found that despite problems encountered early in the implementation of the project, CHAPS has in many ways been an effective partnership model for strengthening district health services, particularly at the sub-district level, i.e. HC and community levels. Some of the initial problems encountered were related to limited communication and misunderstanding among the partners (MOHP Headquarters, DHO/DHMT, and the PVOs) and therefore could be minimized with more joint planning for the project, fuller understanding of the meaning of the partnership, and greater financial transparency. Despite these initial problems, in addition to constraints such as severe staff shortages and frequent transfers, the DHMT and PVOs have learned together and developed supportive and productive partnerships in each of the five districts. Given the time and effort invested to date by all partners, it would be unfortunate to lose the constructive relationships developed in the CHAPS districts. However, even with these CHAPS accomplishments, there are some on-going challenges that suggest the need for considering revising the model. Unfortunately, it was not possible to visit districts implementing other district health strengthening approaches such as the German Technical Assistance Agency (GTZ)-supported program in Machinga, the Netherlands-assisted project in Lilongwe District (Malawi Health, Population and Nutrition Program), and the EU Health Sector Reform and Decentralization supported activities focused in the Southern Districts. It would be useful to conduct a study of these approaches for comparison with CHAPS and use this information for better informing and developing the future public/private partnership model. Based on the findings of the evaluation of CHAPS, the evaluation team considered a number of alternative strategies to build on CHAPS accomplishments and lessons learned. The following options and alternative models were considered. (A matrix of these alternative models is in Annex E.) ƒ Implementation Model 1: A continuation of the basic CHAPS model of PVO/DHMT partnerships with direct oversight from USAID/Malawi. The main advantage of this model would be building on the CHAPS experience and what has worked. The primary disadvantage of this model would be the administrative burden it places upon USAID and the limitation of support to the district partnerships. ƒ Implementation Model 2: Technical Assistance at MOHP Headquarters with Contractor Oversight to the PVO/DHMT district partnerships. The main REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 50 advantage of this model is that it would transfer administrative oversight from USAID to the Contractor and address problems related to this arrangement under CHAPS I. The disadvantage is that it would not be effective for providing the type of support needed by the district partnerships and for strengthening district health services. ƒ Implementation Model 3: Contractor located at the district level, giving TA to PVOs working in partnership with DHMT. This model would also have the advantage of transferring administrative oversight from USAID to the Contractor. The disadvantage is that it would be difficult to find a Contractor with appropriate district experience and orientation. ƒ Implementation Model 4: District level Partnership Coordinating Unit (PCU) located in a district with peer level TA. The advantage of this model would be transfer of administrative oversight to the Partnership Coordinating Unit, which could be formed by a consortium of interested PVOs having the capability to put together a team of technical advisors with extensive district experience and strong interactive and communication skills identified as essential for partnership building. This model would also build on the CHAPS experience. A disadvantage could be that it finding the correct people might be difficult. Discussion of Implementation Model 4: District Level Partnership Coordinating Unit ƒ Based on the findings of the review of CHAPS, the Evaluation Team’s priority recommended structure for a future district health strengthening partnership is the 4th model outlined above, the District Level Partnership Coordinating Unit (PCU). The PCU would be located at the district level, in one of the focal districts of future project focal districts. This would ensure on-going day-to-day mentoring and capacity building with all partners. Although the Partnership Coordination Unit would be located in one district, support would be provided to all project districts. The PCU would be located within a district-based organization, such as a PVO/NGO or consortium composed of PVOs. ƒ The purpose of the PCU would include administrative oversight and management of funding; collaboration between partners to facilitate decentralization; building on lessons learned and tested strategies from CHAPS; strengthen existing relationships; and, facilitate expansion into neighboring focal districts. The PCU would report to USAID and also strengthen coordination with MOPH HQ. ƒ The PCU would be characterized by the following: pragmatic and grounded; flexible; inclusive, for example, including cross-sector approaches such as water and food security; and, build on lessons learned from CHAPS. V. FUTURE STRATEGIES AND MODELS FOR PUBLIC/PRIVATE PARTNERSHIPS 51 ƒ The functions of the PCU would include: ƒ Coordination of decentralization activities at district level; ƒ Coordination within and among partners; ƒ Strengthen coordination of all activities and organizations in the district; ƒ Building on findings of the evaluation of CHAPS, capacity of DHMT, in addition to health facility and community level; ƒ PVO capacity building to strengthen technical competence, mentoring and partnership; ƒ Two-way communication between central MOHP units and districts to keep informed of new decentralization plans; to facilitate districts to respond to and implement national level initiatives, e.g. HMIS, IFMIS, etc.; ƒ Facilitate communication from district to central level about district experiences with new initiatives, problems encountered, and realities at district level; ƒ Address transfer issue in DHMT and PVO/partner organization: explore possibility of requesting two-year appointments without transfer for DHO and key positions in partner organizations/PVOs. ƒ Develop format for regular meetings to facilitate sharing, problem solving, etc.; and ƒ Establish mechanisms for on-going documentation and learning. ƒ The proposed Partnership Coordination Unit would consist of a small team of technical experts (three to four) including cross-cultural staff with Malawian equivalent counterparts. Members of the coordination unit should have strong links and/or access to senior members of the MOHP to facilitate communication and coordination with HQ and the on-going evolution of the decentralization process. The technical expertise needed would include the following: o Financial expertise; o HMIS expertise; o Quality Assurance expertise; REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 52 o Leadership; management; organizational development; learning coordination to facilitate documentation and sharing of lessons learned; o Community development; social mobilization; o Community Health including community nursing/reproductive health; o Learning coordination to facilitate documentation and sharing of lessons learned; o Extensive district experience; and o Strong interpersonal communication skills required, in addition to technical expertise. 53 VI. LESSONS LEARNED FROM CHAPS ƒ CHAPS' PVO/NGOs demonstrated that Egos are able to move beyond the more traditional roles related to implementation and work successfully on organizational/institutional strengthening. ƒ Building partnerships is never easy; successful partnerships require close monitoring, guidance, support and nurturing. o All partners need to be fully informed and involved from the beginning in planning and implementation. o Full financial transparency is required in a partnership. o In addition to technical competency, interpersonal communication and facilitation skills are essential for partnerships. ƒ Mentoring and on-the-job training is an effective method of capacity building. In some respects working side-by-side is more effective than formal training. ƒ Working together at the district level, on a day-to-day basis is more effective than technical assistance from above and/or the central level. ƒ Technical assistance needs to be adapted and appropriate to the local situation as demonstrated by the limited impact of the QA Technical Assistance (TA) provided to CHAPS districts. ƒ Several innovative approaches implemented during CHAPS provide important lessons as demonstrated by the following examples: o Health Worker purchase of equipment, for example bicycles by HSAs, helped to create a sense of ownership and maintenance of the equipment that contributed to improved job performance. o The creation of ZCs within a district, facilitated supervision and support to health workers and in turn, enhanced health worker performance. o Creation of core groups of DHMT staff helped address the problem of staff absences and shortages. ƒ Strengthening of fleet management greatly facilitated the DHMT’s ability to coordinate and provide services. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 54 ƒ Cross-sector activities such as Water and Sanitation and food security increased the effectiveness of the project. 55 VII. RECOMMENDATIONS Based on the findings of the evaluation, the following is recommended: ƒ To build on and expand the district capacity-building expertise developed through CHAPS in future district strengthening efforts in Malawi. ƒ To involve and utilize the district capacity-building experience of the five CHAPS NGO/PVOs in future district strengthening efforts. ƒ To maintain a district focused and based approach in future district strengthening efforts. ƒ To continue program flexibility in order to adapt to district needs in future district-strengthening efforts. Flexibility in the CHAPS program was found to make a positive contribution to program achievements. ƒ To recognize the complexity and needs of partnerships as demonstrated in CHAPS and provide more support for nurturing and strengthening such partnerships in the future. Therefore, mechanisms need to be in place to ensure stronger support and coordination to project partners. ƒ To recognize the need that all partners be well informed from the beginning of the project; therefore, to learn from CHAPS’ early experience and invest more effort at the outset to ensure DHMT leadership fully understands the proposed partnership model and is involved; to brief district staff at all levels on the partnership concept and modalities to ensure a common understanding and clear expectation on part of all DHO stakeholders. ƒ To identify weaknesses and problems early in the partnership. Based on the experience with CHAPS, the effectiveness and impact of the partnerships could have been greater if there had been mechanisms for providing districts with more on-going support related to technical and management issues, as well as to PVO District relationships. ƒ To build on partnership strengthening lessons learned from CHAPS; for example, sharing of space for CHAPS/PVO/DHO facilitated the partnerships. ƒ To develop a mechanism for coordinating all district health related activities, including those provided by the MOHP, CHAM, and all NGOs working in the district, to ensure that all efforts support DHO priorities and fit within the annual district health plans. An initial step would be to clarify DHO authority to coordinate activities of all NGOs working in the district. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIPS PROJECT (CHAPS) 56 ƒ To conduct organizational assessments at the DHO level, which would be useful in strengthening the DHO’s sense of commitment to partnership and in providing guidance, regarding PVO staff recruitment needs in light of district capacity building needs. ƒ To increase central level support to district partnerships, enabling the central level to provide strategic guidance to districts while at the same time listening and learning from districts’ experiences and constraints; thus to establish a two-way communication from central MOHP to district level and from district to central level. Such a two-way communication system would help inform districts about new initiatives for decentralization and for the districts to inform the center about experiences, problems and realities of decentralization at the district level. (The above refers to general support for the partnerships and not to the two specific HMIS and IFMIS initiatives.) ƒ To increase visits from the central level (e.g. MOPH and USAID) to the districts which would facilitate a constructive two-way exchange and joint problem solving. ƒ To provide support for building on and strengthening PVO technical capability, although this was not part of the CHAPS project. Many PVO staff will remain in the district and therefore this would further strengthen district resources. ƒ To examine the impact of structural constraints encountered in CHAPS, including acute staff shortages; frequent staff turn-over; dual (clinical and public health) responsibilities of DHO staff; and, frequent staff absences including those due to training events and meetings. These constraints could best be addressed first through policy decisions based on dialogue between the MOHP and DHMTs. ƒ To establish forums and mechanisms for sharing experiences and lessons learned. These mechanisms should facilitate sharing within and between districts and between districts and the central level. ƒ To build on CHAPS experience to share experiences within districts through regular review and retreats of PVO/DHMT partners. For example, it was found that in one district, holding regular monthly and quarterly coordination meetings supported partnership functioning and strengthened service delivery. In another district, an annual retreat for all CHAPS PVO and DHMT staff was found to be an effective mechanism. ƒ To increase cross visits between districts to enhance sharing and learning. ƒ To build on the experience of the CHAPS Quarterly Meetings, which served as the primary forum for sharing experiences. These Quarterly Meetings were viewed positively but it would be beneficial to involve DHMTs and PVOs in each VII. RECOMMENDATIONS 57 district in planning the agenda and to hold them on a regular basis (for example, no quarterly meeting has been held in almost a year), which was not held because of USAID’s staff shortage. ƒ To develop a mechanism for documenting and synthesizing lessons generated both on the successes, weaknesses and challenges to enhance program learning. ƒ To ensure that activities to inform decision makers and learning should include not only the collection of quantitative data, but also more qualitative documentation related to strategies used, successes and constraints/weaknesses. ƒ To develop a mechanism for discussion and utilization of the findings of evaluation and research reports. For example, more systematic and in-depth feedback and discussion could have facilitated implementation of the findings of the CHAPS MTE. ƒ To include a budget line item for operational research and small studies that are identified as needed during project implementation. Such studies would be beneficial to future district strengthening projects. ƒ To actively involve DHMT in planning operational research and other studies. This would address the constraint found in CHAPS when outside consultants took responsibility for conducting studies and district staff were not substantively involved. ƒ To develop a forum or mechanism for coordinating and sharing experiences among donors and government on strengthening district approaches and decentralization. It was found that many organizations provide assistance for strengthening district health services (e.g. USAID, Germany, the Netherlands, EU, DFID, UNICEF). Although there is a monthly meeting of the donor subgroup on health, no mechanism currently exists for coordinating and sharing the lessons from the different approaches to district health strengthening. ƒ To develop a mechanism to document lessons learned from the HIV/AIDS experiences and activities that are being supported by CHAPS in the five districts; and, to disseminate these in order to accelerate program learning and ensure future investments are made in the most cost-effective ways. ƒ To continue and provide increased support to the DHMT for HIV/AIDS activities, particularly given the limited involvement in HIV/AIDS found on the part of DHMT staff. ƒ To continue support for transportation and logistics. ƒ To continue and build on the positive experience and outcome under CHAPS of TA for fleet management. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIPS PROJECT (CHAPS) 58 ƒ At the Health Center level, to focus on organizational/management skills of HC staff, in addition to the current support for improving services, supervision, training, and technical program areas. Special attention is recommended for the HC level since in terms of cost-effectiveness, prospects for sustainability of capacity-building efforts at HC and community levels may be greater than at DHO level given the relatively lower turn-over among core HC staff, and particularly among HSAs and community activities. ƒ To address constraints identified in QA and QA TA provided through CHAPS, including the finding that QA needs to be practical and adapted to the local context. ƒ To conduct a review of training provided during CHAPS to determine quality of training and impact of training on staff performance. In addition, to develop a mechanism for regular assessment and follow-up of training in future district strengthening programs, in addition to establishing a mechanism to monitor the benefit of training on community health. ƒ To assess the integrated approach to training, e.g. training for the IMCI. ƒ To develop a strategy for district level training provided by all organizations and MOHP. Such a strategy needs to prioritize training and address the evaluation finding that the same people are invited for multiple training sessions, which contributes to DHO staff absences and unavailability. ƒ To develop mechanisms for joint preparation of training programs and development of training lesson plans that involves use of more active learning methods, which will facilitate training. In addition, to develop courses for master trainers to strengthen facilitation skills required for training events and also team working and group activities. ƒ To conduct an assessment of formal training versus mentoring which would build on the findings of the positive impact of the presence and availability of staff in the district and day-to-day contact. ƒ To develop mechanisms to build on and share the best practices which were found to be innovative and suited to local realities and constraints. For example: core groups; zonal offices; bicycle ownership; revolving funds for bed nets; and, fleet management system. ƒ To review experience with bicycle ambulances which were found in some areas to be inappropriate to local practices, in addition to encountering problems with maintenance. VII. RECOMMENDATIONS 59 ƒ To continue support for and assessment of efforts to improve supervision, including the content and communication and interaction skills of those providing training. ƒ To document the experience of zonal offices to strengthen supervision, in order to determine the benefits and challenges of this approach for use in other districts. ƒ To develop a coherent process for project monitoring which is an important tool for the DHMT and requires a close working relationship between the DHMT and PVO/NGO, for example in preparing quarterly reports. ƒ To define and present project indicators, including improved mechanisms for data collection, analysis and utilization for monitoring and planning. ƒ To improve capacity building for data management by DHMT, e.g. to include provision of required computer equipment and method for backing-up important data on a regular basis, as well as having PVO staff knowledgeable in computer systems, in addition to PVO staff with technical expertise in HMIS, finance and logistics. ƒ To continue support for construction of health facilities (e.g. out-reach shelters, disposal of medical waste, etc.) in future district programs ƒ To give increased attention to strengthening drug management at central district store and pharmacy to include improved storage methods; inventory control, forecasting techniques; distribution systems; and, computerized management of stocks, orders, etc. ƒ To document and share the lessons learned from the innovative strategy to train merchants in basic malaria diagnosis and treatment and sale of fansidar. ƒ To conduct a follow-up study to assess the adequacy of treatment advice for malaria being given by DFR distributors. ƒ To address the problem of HC staff shortages and of limited time HC staff have for outreach activities and the impact of delegating more responsibilities to HSAs as the only mechanism that has been used to deal with this problem during CHAPS. ƒ To document the lessons learned from activities to strengthen VHCs and CHVs, given the challenges and importance associated with strengthening these groups. ƒ To give increased attention in future district capacity building programs to strengthening drug management at the central district store and pharmacy to include improved storage methods, inventory control, forecasting techniques, distribution systems and computerized management of stocks, orders, etc. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIPS PROJECT (CHAPS) 60 ƒ To discuss with central level MOHP the need to ensure that drugs are consistently supplied in order for DRF programs to be viable; to address pricing situation of DRF; to ensure that DRF program managers and HSA coordinators at DHO level ensure that DRFs are regularly supervised by HSAs; to ensure that HSAs provide regular supervision of DRFs in order to identify problems early and correct them as soon as possible; and, to involve community members in DRF training which increases transparency and oversight of all DRF activities. ƒ To conduct a review of the effectiveness of HAS supervisory role in DRFs. ƒ To evaluate if DRF treatment provided to patients is adequate to meet clinical needs. ƒ To provide periodic supervision at all levels, including CBDA, for maintaining activities, identify problems and keeping proper records and improved reporting and planning. ƒ To continue support to TBAs and build on their contribution to improved delivery and referral services. Although the trend is to encourage delivery in health facilities, such services will not be available to much of the population in Malawi for some time and therefore it is important to continue support and capacity building of TBAs. ƒ To continue and expand efforts to improve referral services, e.g. provision of radios to health centers, ambulances, support to TBAs, etc. ƒ To provide training on radio maintenance to health center staff. ƒ To increase the focus on use of innovative approaches to health education and IEC methods, e.g. for songs and stories; training of local groups in drama for development; and, integration of health topics into adult literacy training, etc. ƒ To document and disseminate lessons learned related to preventive and curative activities. ƒ To document lessons learned and continue and increase support for cross-sector activities, such as water and sanitation, and food security. ƒ To document experience with provision of Water and Sanitation pumps, latrines and supplies and disseminate between CHAPS districts and with other districts. ƒ To give greater attention to transferring accounting and financial skills to DHO staff, either through full-time staff who have this expertise, or, through outside consultants who could provide occasional TA. VII. RECOMMENDATIONS 61 ƒ To intertwine and build in sustainable analysis support and financial management structures and systems into the underlying business and operational processes, rather than having them “attached.” ƒ To conduct performance evaluations related to the financial positions within the partnerships. ƒ From a financial perspective, to update the organizational structure of the district to best support national and district strategic initiatives; to ensure a workable retention package to preclude unnecessary staff turnover; and, to apply IFMIS and HIS resources while mentoring DHMT members. ƒ To increase CHAPS involvement and coordination of improved quality of financial staff and in automation of the accounting system. ƒ To increase focus on the financial structures and internal transaction collection and reporting schemes of the district in future district programs. ƒ To include in future financial management functions related to monitoring: o A more comprehensive focus of integrating financial and non-financial monitoring; o A more cohesive approach to integrating strategic initiatives such as decentralization with monitoring systems; o Financial monitoring efforts at the central-district should be better linked with a risk analysis of the key processes and mission-critical functions; and o A methodical and documented focus on improving information quality. ƒ To address the constraints found in the current CHAPS in future program design. ƒ For the financial management structure to better support the stated mission, strategy, organizational structure, key processes, staff duties, services, and desired planned versus actual outcomes of the district: o To have one integrated financial management system; o To review the organizational structure of the district prior to implementation of any aggressive interventions for improving the financial management systems and processes; REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIPS PROJECT (CHAPS) 62 o To move beyond recording numbers and bookkeeping and become involved in process measurement and monitoring in future “financial management capacity” by stakeholders; o To review the mix and skill-set of staff in the financial management areas to insure it aligns with the demand; o To better identify problems, provide measures of costs and benefits, more frequently provide financial evaluation of alternatives, and better monitor and evaluate operational results for future financial management practice; and o To require future financial management function to provide an accurate and timely analytical capacity to forecast financial performance. ƒ To apply principles of internal control to an assessment and use as a framework for analysis in future programs. Recommendations for Future Studies and Evaluations ƒ In evaluation of the future CHAPS or district health-strengthening program, involve national representation on the evaluation team. It would also be beneficial to include representation from other district health strengthening projects, e.g. the Netherlands supported Lilongwe District Health Project. ƒ There are many lessons learned from CHAPS but limited mechanisms for documenting, recording, and sharing/disseminating these lessons and best practices. It is recommended that such mechanisms be developed. ƒ A study of the many approaches being supported in Malawi for strengthening district health services (in addition to and including CHAPS) is recommended and a forum established for comparing and assessing the effectiveness of these approaches. ƒ Studies of the quality and utility of training provided under CHAPS and in future programs are recommended. ANNEXES A: Statement of Work .......................................................................................................65 B: Evaluation Team ..........................................................................................................75 C: Itinerary for the Evaluation of CHAPS........................................................................79 D: Persons Consulted........................................................................................................87 E: Documents Consulted ................................................................................................107 F: Matrix of Alternative Models for Future Public/private Partnerships.......................115 ANNEX A STATEMENT OF WORK REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 66 STATEMENT OF WORK COMMUNITY HEALTH PARTNERSHIPS PROJECT (CHAPS) EVALUATION 1. Purpose: The purpose of this Statement of Work (SOW) is to assist USAID/Malawi and the Ministry of Health and Population (MOHP) to assess and document the progress made under the Community Health Partnerships (CHAPS) project. This evaluation will provide the basis for deciding on ways of further strengthening the district public/private partnership and central level support. The evaluation will also be used to determine additional performance monitoring, evaluation and studies that would ameliorate the results and best practices for possible replication of similar activities. The Evaluation methods and activities, utilized under this SOW, should be designed and implemented in a manner to enhance collaboration and dialogue among CHAPS stakeholders involved in the development of district level health services. 2. Background CHAPS is a $15 million, seven-year initiative to improve health care services in target districts through public/private sector partnerships for health. The project was signed on September 30, 1995 and the implementation of activities started in April 1998. The delay in implementation of the CHAPS activities was due to the Ministry of Health and Population's (MOHP) desire to consult with the districts on the role and responsibilities within the partnership with the Private Voluntary Organizations (PVOs). It is a pilot effort geared toward testing and evaluating the effectiveness of partnerships between PVOs/NGOs and MOHP District Health Management Teams (DHMT) as a means for improving the delivery of maternal and child health (MCH) and reproductive health (RH) services. To this end, cooperative agreements (CoAgs) were awarded to five PVOs to support PVO-DHMT partnerships and a Social Marketing CoAg was awarded to Population Services International (PSI) for the social marketing of Thanzi Oral Rehydration Solution (ORS) and Exclusive Breastfeeding (EBF). Under the partnership framework, CoAg recipients collaborate with DHMT to assess constraints, prioritize health problems, identify systemic weaknesses in the delivery of services, develop and test alternative health financing approaches, and implement appropriate interventions. This is to be accomplished within the current government system where government structures will retain and exercise their normal authorities. CHAPS interventions include both advocacy and delivery of specific services aimed at improving practices that prevent illness and promote better care of ill children and their mothers, and actions to reinforce the health system at the level of delivery. One approach for child survival services is known as integrated management of childhood illness (IMCI). This approach structures the patient-provider contact around the signs and symptoms of the child and emphasizes recognition of the most common life-threatening ANNEXES 67 illnesses of childhood, namely, malaria, diarrheal disease, acute respiratory infections, and malnutrition. Improved patient outcomes also depend upon caretakers recognizing danger signs and bringing their children in promptly for treatment, as well as on the availability of appropriate drugs and supplies. Water supply and sanitation were noted as the highest priority among women who were consulted during the design of CHAPS, and development interventions such as income generating activities may also be included among the specific district proposals. CHAPS employs a unique program approach by utilizing NGOs experienced in the provision of Primary Health Care (PHC), as partner organizations with the MOHP, to extend key health services and enhance institutional capacity. Such partnerships should optimize the transfer of worldwide best practice skills of international NGOs, for the management and delivery of key primary health care services directly to district level operational units of the public health system. The project activities support SO3 in increasing the provision of quality health care services to Malawians and building the capacity of DHMTs to deliver health care. The findings and recommendations of this evaluation are critical in determining future activities to be implemented under the Strategic Objective Agreement (SOAG). 3. Objectives The evaluation must take into consideration the current political/social environment of Malawi, the HIV/AIDS epidemic, the shortage of human resources, especially the critical shortage of health care providers, decentralization of the government, and the current actions by the MOHP and donor community to create a Sector Wide Approach (SWAp). The contractor shall: 3.1 assess and document the progress made under the CHAPS model of public/private partnerships in supporting the DHMTs to: 3.1.1 establish and build effective models of public/private partnerships; 3.1.2 enhance DHMT capacity through the transfer of service delivery and management/financial skills between international NGOs and the MOPH; 3.1.3 improve the management and quality of district level health services and health promotion capabilities of communities and family caregivers; 3.1.4 determine the contribution of the CHAPS activities to the achievement of results of the overall HPN SO Strategic Objective (HPN SO) of USAID/Malawi: Increased adoption of measures that reduce fertility and risk of HIV transmission, including child health practices; 3.1.5 establish support provided by the central level MOHP; and 3.1.6 establish the unintended/secondary results. 3.2 determine the effectiveness of monitoring, evaluation and performance monitoring plans to guide program direction and monitor/evaluate project activities; REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 68 3.3 recommend additional monitoring, evaluation and studies to determine more effectively results and best practices to guide future district health service improvement; and HPN SO/Mission programming activities and national/regional approaches to public/private partnerships; 3.4 recommend ways of strengthening the public/private partnership at the district and central level within the MOHP; 3.5 recommend future cost effective programming measures of Mission HPN SO activities related to CHAPS and Africa regional concepts for public/private partnership development; 3.6 appraise program and financial management practices by CHAPS implementing partners, cooperating agencies and key operational units; and 3.7 determine the sustainability of the CHAPS approach. 4. Evaluation Methodology 4.1 Guidance and Supervision by USAID/Malawi: Through field visits, review of progress reports, and organization of quarterly CHAPS Team meetings, the USAID Mission has provided key inputs to the ongoing supervision, monitoring and evaluation (S/M/E) of CHAPS activities in close collaboration with MOHP and PVO counterparts. These S/M/E activities have been provided primarily through the oversight of the CHAPS Team Leader and the Monitoring and Evaluation Specialist in Program Development and Analysis (PDA) office. 4.2 Meetings: The COP should arrange briefing meetings at various times during the period of the evaluation with relevant officials at USAID/Malawi, the MOHP and other key stakeholders at the beginning of the evaluation to report on the approach, duration and any pertinent issues. Meetings should also be arranged with stakeholders as required during the period of the evaluation to brief on progress and any noteworthy encounters. Lastly, a debriefing meeting with all stakeholders at the end of the evaluation should be arranged. 4.3 Focus of the Analysis: CHAPS provides an important opportunity to evaluate the effectiveness of the public/private partnership model in strengthening the capacity of district health teams to improve the quality of health services available to communities and clients in rural Malawi. As outlined in the objectives section above, the key areas of focus of the Evaluation include: • Assessing the effectiveness of the CHAPS model of public/private partnerships (as implemented by the six major NGOs/PVOs: Africare in Mzimba District, International Eye Foundation in Chikwawa District, Project HOPE in Mulanje and Phalombe District, Save the Children/UK in Salima District, Save the Children/US in Mangochi, and PSI in social marketing activities for ORT and EBF in supporting the DHMTs and the Ministry of Health and Population; ANNEXES 69 • Determining the effectiveness of the packages of service delivery (MCH, RH, HIV/AIDS, CS and water and sanitation) and capacity building (technical/facility/commodity improvements, training, IEC, management and financial) interventions utilized by the partnerships in improving the quality of health services and the health status of target populations; • Evaluating the effectiveness of monitoring, evaluation and performance indicators to guide program direction and monitor/evaluate project activities; and • Appraising program and financial management practices by CHAPS implementing partners, cooperating agencies and key operational units. The Evaluation Team will also need to review recent activities of other donors and international agencies in order to integrate their most recent program activities, findings and best practices into further CHAPS initiatives. 4.4 Review of Relevant Project Implementation Documents: CHAPS has a series of useful documents, which outline the evolution of planning, implementation and evaluation of program activities since 1995. The team and “expert panel members” will review these documents on an as needed basis. Key documents include: • USAID/Malawi Project Paper, September 1995; • USAID/GOM Project Grant Agreement (September 1995) and Amendments (September 1996, July 1998, September 2000); • Project Grant Agreements and Quarterly Reports (Population Services International, Africare, International Eye Foundation, Save the Children/US, Save the Children/UK, Project HOPE); • Cooperating Agency Agreements and Reports (Engender Health, JHPEIGO, Partnership for Health Reform (PHR), URC/QA, and others); • MAARD Documents for Services, Training and Commodities; • Proceedings Reports of Quarterly Meetings of Partners (January 1998 through December 2000); and • USAID Malawi Results Review and Resource Request (R Reports and performance monitoring plans. 4.5 Evaluations of the CHAPS Project by the Partnerships: Evaluations were done by each of the major PVO/NGO and District Health Teams. These Evaluations by the partners are some of the most relevant evaluation materials available for use by the Evaluation Core Team. Evaluations currently exist for: • Africare/Mzimba (March 2000); • International Eye Foundation/Chikwawa (September 2000); • Project HOPE/Mulanje and Phalombe (November 2000); • Save the Children UK/Salima (November 1999); and REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 70 • Save the Children US/Mangochi (April 2000). Although these Evaluations provide interesting and useful information, the Core Evaluation Team will need to give considerable thought to harmonizing the differences in evaluation methodologies/procedures and confirming the findings. 4.5.1 Evaluation Team 4.5.2 Core Team: Chief of Party/Institutional and Organizational Specialist, Health Services Specialist, PVO Specialist, and Economist/Financial Specialist; 4.5.2 District and National Expert Panel Members: Expert panel members will be invited by USAID and MOHP to facilitate the collection and analysis of data, the formulation of recommendations, and participate as needed to accelerate the work of the team. These panel members will include key stakeholders. 4.6 Activities of the Evaluation Team 4.6.1 The Core Team principal activities will include the final design of the evaluation in collaboration with the CHAPS Team Leader and the Monitoring and Evaluation Specialist, necessary information/data gathering, appropriate analysis and formation of recommendations and the completion of the Final Report. The key person responsible for the final design, implementation, analysis and writing/production of the Final Evaluation Report is the Chief of Party (COP) of the Evaluation Team. In collaboration with the CHAPS Team Leader and under the supervision of the Monitoring and Evaluation Specialist, the COP of the Evaluation Team will guide and assure the completion of the final design and implementation of the Evaluation and provide an appropriate written report of the CHAPS evaluation to USAID Malawi prior to October 30, 2001. An outline of the position description and qualifications of the COP and members of the Core Evaluation Team are outlined in the Team Composition Section. 4.6.2 Stakeholder Dialogues and Reviews: In addition to the utilization of relevant project implementation documents and the Evaluations by the partners, considerable attention will be required to obtain the impressions, observations, findings and recommendations of the large array of stakeholders who participate and/or are affected by CHAPS activities. Key staff, clients and groups that need to be involved include: • Central level MOHP officials; • District Health Management Teams; • Selected community leaders and clients; • PVO project country and regional; • Country office personnel; • Community members; and • USAID staff and other major donors and international organizations. ANNEXES 71 4.6.3 Visits to all CHAPS Districts: Visits should be scheduled for all CHAPS districts which include: Mzimba, Chikwawa, Mulanje and Phalombe, Salima, Mangochi, and selected PSI national social marketing sites. 4.6.4 The Evaluation Team should visit one or two neighboring non-CHAPS districts for “control” comparisons. The CHAPS Team can recommend such districts. 4.6.5 Periodic Meetings with CHAPS Team Leader and the Monitoring and Evaluation Specialist: The COP of the Core Team will be responsible for scheduling and conducting periodic meetings with the CHAPS Team Leader and Monitoring and Evaluation Specialist. At a minimum, the meetings will include: an initial organizational/introductory meeting; an outline and explanation of the final design of the Evaluation; a mid-evaluation review to outline progress and implementation problems; and an Evaluation summary of the data, draft recommendations/report and an exit review. The Final Report will be proceeded by a review of the draft and recommended revisions by the CHAPS Team Leader and the Monitoring and Evaluation Specialist. 5. Team Composition 5.1 Chief of Party/ Institutional and Organizational Specialist: The Chief of Party will provide team leadership, plan and coordinate activities, review improvement to management and organization of health care delivery systems in the CHAPS districts. Review roles and responsibilities of various stakeholders in implementing CHAPS activities, arrange for periodic meetings to brief USAID personnel on progress of the evaluation, consolidate reports from other evaluation team members, ensure that a draft report has been left behind on departure and provide the Mission with a final report within four weeks of his/her departure. The COP should have an advanced degree in health or a related field with a minimum of ten years of experience in management and evaluation of district level health programs, experience in leading teams of experts in health activities, experience in international health specifically dealing with institutional and capacity building at district level and should have worked extensively with PVOs that implement health programs. 5.2 Health Services Specialist: The Health Services Specialist will review health care delivery systems including Quality Assurance activities; analyze the appropriateness of these interventions under CHAPS and customer satisfaction; document commitment of stakeholders to project implementation, point out deficiencies requiring attention in the remaining period; and review the performance Monitoring Plans used by the CHAPS partners. This Specialist should have a master’s degree in social science or health related field with at least five years experience in community health and a working experience with PVOs. 5.3 PVO Specialist: The PVO Specialist will review the roles and responsibilities of the DHMT/PVO partners; the relationship between the DHMT and the PVO; PVO REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 72 responsiveness to the needs of the DHMT; and recommend any modifications needed to make the partnership work better. This Specialist should have a master’s degree in public health or related field with a minimum of five years experience working with PVOs in Africa, preferably in Malawi or Southern Africa. 5.4 Economist/Financial Analyst: To review cost effectiveness of interventions under CHAPS and sustainability of the CHAPS model, examine the flow of resources from the PVO to the DHMT and beneficiaries and the adequacy of resources for the proposed intervention taking into account the population and other factors in a district. The Analyst should have a master's degree in development economics or finance, with a minimum of five years experience in conducting economic and financial assessment in health programs. The individual should be familiar with USAID financial rules and regulations. 6. Logistics A six-day workweek is authorized. The contractor should arrange own travel, office space, computers, photocopying, appointments with stakeholders, and any other necessary equipment. 7. Reporting Format: The report should include an Executive Summary, concisely summarizing critical elements of the main report; an Introduction which describes the purpose/objectives of the evaluation and the main questions being answered by the evaluation; a Background of the project; Findings of the evaluation; Conclusions based on the evaluators interpretation of findings; Recommendations which spell out actions required to correct any anomalies observed during the evaluation; Lessons learned for the way forward; a memorandum of how the results of the evaluation would be incorporated into future programming of the HPN SO activities; and Issues that may require resolutions. Other pieces of information relevant to the evaluation but not necessarily part of the report should be included in the annexes. Delivery of reports: Six copies of the draft report should be left with the USAID/Malawi Monitoring and Evaluation Specialist prior to the departure of the COP. Mission requires twenty bound copies of the final report for distribution to stakeholders. The reports should be received four weeks from the departure date of the COP. The contractor should also provide an electronic copy on a 31/2" diskette in Microsoft Word. 8. Timing Period of the Evaluation: The Evaluation is scheduled to take place from 1 October 2001 through 10 November 2001. ANNEXES 73 9. Level of Effort and Estimated Budget 9.1 Level of Effort (LOE) Team Member LOE in Weeks Chief of Party/Institutional /Organizational Specialist 6 Health Services Specialist 4 PVO Specialist 4 Economist/Financial Analyst 3 Total LOE 17 REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 74 ANNEX B EVALUATION TEAM MEMBERS REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 76 ANNEXES 77 Evaluation Team Members Judi Aubel Team Member: PVO Specialist C/o SPC, Private Mail Bag Suva, Fiji jatao@is.com.fj Bradford Else Team Member: Financial Analyst/CPA Craig Wood Golf Course Road HCR-1 Box 48-b Lake Placid, NW 12946 teo@northnet.org Judith Justice Team Leader: Institutional and Organizational Specialist 85 Fairlawn Drive Berkeley, CA 94708 justice@itsa.ucsf.edu Alain LeFevre Team Member: Health Services Specialist 4841 Avenue Carlton Montréal (P.Q.) H3W 1G7 Canada saninter@sympatico.ca REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 78 ANNEX C ITINERARY FOR THE EVALUATION OF CHAPS REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 80 ANNEXES 81 Itinerary For The Evaluation of CHAPS 20 February: Members of Evaluation Team travel to Washington, D. C. 21 February: Meeting and orientation of Evaluation Team at MEDS Team meeting facilitation by Jim Carney Conference call with Mexon Nyirongo Meeting with Warren Robinson, Team Member for STAPH Evaluation 22 February: Meeting of Evaluation Team Meeting with Judy Moore, SCF/US (Saving Newborn Lives in Mangochi) International Medical Clinic (for malaria prophylactic) Aubel and Justice depart Washington/travel to Malawi 23 February: Else and Lafevre depart Washington/travel to Malawi Aubel and Justice in transit at Heathrow Airport 24 February: Aubel and Justice arrive Lilongwe, Malawi 25 February: Meeting with Wilma Roscoe to plan logistics for evaluation Meeting with Mexon Nyirongo, Acting HPN Team Leader And CHAPS Team Leader Else and Lafevre arrive Lilongwe Meeting with Peter Connell, Team Leader, STAPH Evaluation 26 February: Meeting with USAID Mission Director and HPN Team Meeting with Mexon Nyirongo Meeting with Michael O'Carroll, Technical Advisor to MOHP Meeting with US Peace Corps Associate Director/Health Coordinator Meeting with Health Coordinator, Christian Medical Association of Malawi (CHAM) 27 February: Meeting with Principal Secretary, MOHP Meeting with Mexon Nyirongo, USAID Meeting with Director of Africare Meeting with SCF/UK Programme Director and Health Advisor Meeting with USAID Evaluation and Monitoring Specialist Meeting with Peter Connell, STAPH Evaluation Meeting of Evaluation Team 28 February: Meeting with CHAPS Stakeholders: District Health Officers and PVO Project Coordinators from Mzimba, Salima, Mangochi, Mulanje, Chakwawa) & CHAPS Team Leader and Members of Evaluation Team Meeting with Country Representative, SCF/US REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 82 1 March: Meeting at USAID with Mexon Nyirongo and Al Smith (Assistant Mission Director and Program Director) Meeting with Director, National AIDS Commission Meeting with Japanese International Cooperation Agency (JICA) Project Formulation Advisor (Tomoko Harada) and Aid Coordinator (Evans Kachale) Meeting with First Secretary, Norwegian Embassy/Norwegian Agency for International Development (NORAD) (Jan H. Olsen) Meeting with Director, International Eye Foundation Meeting with Project Coordinator, GTZ Meeting with Health Economist, Health Sector Reform, EU/British Council Meeting with Khoti Gausi, Resident Technical Advisors, PHR/ABT Ass. 2 March: Review background documents and reports Meeting of Evaluation Team 3 March: Meeting of Evaluation Team Review Documents Prepare for field trip to five districts 4 March: Meeting with Mexon Nyirongo and Team (Wendel's Guest House) Travel from Lilongwe to Mzimba Overnight at Raiply Guest House, Mzimba 5 March: Meeting with Africare CHAPS Coordinator and Staff, Mzimba Meetings with DHO and DHTM, Mzimba Meeting with District Accountant Meeting with Accountant, Tovwirane 6 March: Meeting with Africare Staff, Mzimba Field visit to two Health Centers (Kameteka Health Center) Meeting with community members, TBA, HSA; visit to community, outreach shelter Travel from Mzimba to Salima Dinner meeting with SCF/UK Health Advisor, DHO, Environmental Health Officer, and CCO. Salima DHMT Overnight at Livingstonia Beach Hotel 7 March: Meeting with SCF/UK Health Advisor and CHAPS Staff, Salima Meeting with District Accountant, DHMT Salima Meeting with District Commissioner, Salima Travel from Salima to Mangochi Dinner meeting with SCF/UK Senior Health Advisor and CHAPS ANNEXES 83 Coordinator, Mangochi Overnight at Club Makakola 8 March: Meeting with Deputy Director, SCF/US, Mangochi Meeting with DHO, DHMT, and SCF/US CHAPS Staff Meeting with District Commissioner, Mangochi Field visit to Namwera Health Center; meeting with HC Staff, Zone Coordinators, HSAs, community groups Meeting with DHO, Mangochi Dinner meeting with SCF/US Health Advisor 9 March: Field visits to Chilipa Health Center and Nikisi Village Health Committee Meeting with Senior Health Advisor and CHAPS Coordinator, SCF/US 10 March: Review Documents (Mangochi) Begin draft of reports of field visits Travel from Mangochi to Blantyre Meeting of Evaluation Team, Blantyre (Mount Soche Hotel) 11 March: Travel to Chikwawa District Meeting with Chikwawa CHAPS Coordinator and Staff Meeting with Chikwawa DHO and DHMT Field Visit to Ngabu Rural Hospital Meeting with Clinical Officer and Nursing Matron Field Visit to Khokhwa Outreach Shelter Meeting with Community Based Distributors (11) at Ngabu Rural Hospital Field Visit to Losi Village and Water and Sanitation Project Meeting with Concern Universal Staff Dinner meeting with Country Director of Project HOPE, Blantyre 12 March: Travel to Mulanje District Meeting with Mulanje CHAPS Coordinator and CHAPS Staff, Project HOPE Meeting with Mulanje DHO Accountant Meeting with Mulanje DHO and DHMT Dinner meeting with Country Director, International Eye Foundation, Blantyre 13 March: Else, Justice, Lafevre depart Blantyre/travel to Lilongwe Aubel to Chikwawa for meetings with Chikwawa CHAPS Staff and Field Visits in Chikwawa District; Kakoma Health Center, Manjoro Village; meeting with 10 members of DRF Committee Meeting with WHO Family Health and Population Program Officer (Theresa Mwale), Lilongwe Meeting with HMIS, MOHP (Chris Moyo and Chet Chaulagi), Lilongwe REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 84 Meeting with UNICEF Country Representative and Health Officers, Lilongwe 14 March: Meeting with Chief Technical Advisor, MOHP (W. O. O. Sangala) Meeting with Planning Advisor, MOHP (Matt Robinson) Meeting with Planning Officer, MOHP (Lucy Horer) Meeting with Deputy of Clinical Services, MOHP (Davis Mtotha) Meeting with Controller of Clinical Services, MOHP (Rex Mpanzanje) Aubel to Mulanje for meetings with Mulanje CHAPS Staff and Field Visits in Mulanje District 15 March: Meeting with Director of Planning, MOHP (Ann Phoya) Meeting with Development Officer, Reproductive Health Unit, MOHP (Trish Araru, former CHAPS Coordinator, SCF/UK Salima) Meeting with Technical Advisor, Health Sector Reform, EU/British Council (Carol Barker) Meeting with Mexon Nyirongo, USAID (Wendel's Guest House) Aubel to Salima District for follow-up meetings with SCF/UK CHAPS staff and return to Lilongwe 16 March: Meeting of Evaluation Team Begin drafting report 17 March: Meeting of Evaluation Team Preparation for meeting with Mexon Nyirongo Preparation for meeting with USAID Mission Director Meeting with Mexon Nyirongo (at Wendel’s Guest House) 18 March: Meeting of Evaluation Team Meeting with USAID Mission Director and HPN Team 19 March: Else and Lafevre depart Lilongwe/travel to U.S. Meeting with Aubel and Justice to review Aubel's follow-up district visits Meeting with Health & Population Adviser, DFID (Jean-Marion Aitkin) 20 March: Draft report Follow-up with CHAPS PVOs and DHMTs by phone 21 March: Meeting with Representative, UNFPA (Charlotte Gardiner) Meeting with UNAIDS Country Programme Adviser and Programme Officer (Agnes Makonda-Ridley and Monica Djupvik) Meeting with Health and Population Adviser, DFID Meeting with Malawi Health, Population and Nutrition Programme, (Netherlands), Lilongwe District Health Office (Francine van dam Boorne) ANNEXES 85 22 March: Meeting with Public Health Specialist, World Bank (Sheila Butta) Meeting with Malawi Health, Population and Nutrition Programme, Lilongwe District Health Office (Bart van der Plotz, Programme Coordinator) Meeting with Controller Preventive Services, MOHP (Habib Samanje) Meeting with Advisor, HMIS, MOHP Meeting with USAID HIV/AIDS Team Leader (Elise Jensen) Meeting with Technical Advisor, EU/British Council (Carol Barker) 23 March: Meeting with Wilma Roscoe (to finalize logistics) Meeting with Aubel and Justice Draft report and prepare for departure 24 March: Meeting with Mexon Nyirongo at USAID Justice departs Malawi/travel to U. S. Aubel remains in Malawi for QA/URC review REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 86 ANNEX D PERSONS CONSULTED REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 88 Persons Consulted Judith Moore Senior Technical Advisor Saving Newborn Lives Initiative SCF/US Washington, D.C. Tel: 202-293-4167 Warren C. Robinson Economic and Demographic Consultant Team Member STAPH Evaluation Economic Research Associates Washington, D.C. Tel: 202-965-6610 Peter Connell Consultant, Team Leader for STAPH Evaluation Development Business Associates Singapore Wilma Roscoe Consultant Lilongwe Tel: 950-087, 772-351 Mexon Nyirongo Acting HPN Team Leader & CHAPS Team Leader USAID/Malawi Lilongwe Tel: 772-455 (864-990, 862-559) Roger Yochelson Mission Director USAID/Malawi Lilongwe Tel: 772-455 Dwight Smith Program Director & Assistant Mission Director USAID/Malawi, Lilongwe Tel: 772-455 ANNEXES 89 Linda S, Andrews Reproductive Health Specialist USAID/Malawi Elise Jensen HIV/AIDS Team Leader USAID/Malawi Robert Katuya Financial Analyst USAID/Malawi Alan McKenna Controller USAID/Malawi Kalinede Chindebvu Monitoring and Evaluation Specialist USAID/Malawi Alick Mtika Program Associate, CHAPS USAID/Malawi Alfred Chirwa HIV/AIDS Specialist USAID/Malawi Roseth Bongwe Secretary for HPN Section USAID/Malawi Ethel Mpagaja Secretary for HPN Section USAID/Malawi Richard B. Pendame Principal Secretary Ministry of Health Tel: 789543, 831636 Bisewi Mwele Executive Director National AIDS Commission Lilongwe Tel: 827530 REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 90 W. O. O. Sangala Chief Technical Advisor MOHP Tel: 789400/195; 822-262 Rex Mpanzanje Controller of Clinical Services MOHP Davis Mtotha Deputy, Clinical Services MOHP Habib Samanje Controller of Preventive Services MOHP Tel: 842-971 Chris Moyo Community Health Services & HMIS MOHP Ann Phoya Director of Planning MOHP Tel: 789-823 Lucy Horer Planning Officer MOHP Cell: 930-200 Mr. Maseya Director of Finance MOHP, Lilongwe John Qongwane Chief Accountant MOPH/STAPH, Lilongwe Matt Robinson Advisor, Planning Division MOHP (Overseas Development Institute/UK) Cell: 946-980 ANNEXES 91 Michael O’Carroll Technical Advisor Ministry of Health and Population Lilongwe Tel: 788654, 839872 Chet Chaulagai Advisor PME-HMIS/MOHP Netherlands Assistance (MHPN) Lilongwe Tel: 915-119 Edith F. Mkawa Associate Peace Corps Director/Health UC Peace Corps Lilongwe Tel: 751055, 756456 Joseph A. Theu Health Coordinator Christian Medical Association (CHAM) Lilongwe Tel: 775404, 838047, 677671, 940859 Tomoko Harada Project Formulation Advisor JICA Lilongwe Tel: 771644/554/954/234 Evans Kachale Aid Coordinator JICA Lilongwe Jan H. Olsson First Secretary Royal Norwegian Embassy/NORAD Lilongwe Tel: 774211; Cell 828203 Dieter Horneber Project Coordinator Physical Assets Management in the Health Sector German Technical Assistance (GTZ) Lilongwe Tel: 788340; cell 828491 REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 92 Fabrice Sergant Health Economist, Health Sector Reform British Council/European Union Lilongwe Tel: 789095 Carol Barker Technical Adviser Health Sector Reform The British Council/EU Tel: 789-093; cell 828206 Khoti Gausi Resident Technical Advisor PHRplus, Abt Lilongwe Tel: 826003 Theresa G. Mwale Family Health and Population Programme Officer World Health Organization Lilongwe Tel: 772-755/450 Catherine Mbengue Representative UNICEF Lilongwe Tel: 770770 Elizabeth Hughes Program Officer/Health UNICEF, Lilongwe Jean Nyondo Program Assistant/Health UNICEF, Lilongwe Trish Araru Development Officer Sexual Reproductive Health Programme Reproductive Health Unit, MOHP Lilongwe Tel: 751552; cell 844-524 ANNEXES 93 Charlotte N. Gardiner Representative UNFPA-Malawi Tel: 771-444/474 Jean-Marion Aitkin Health and Population Adviser British High Commission (DFID) Tel: 772400/939844 Cell: 840-137 Home: 794-504 Francine van den Borne Malawi Health, Population and Nutrition Programme Lilongwe District Health Office (Netherlands Assistance) Tel: 775561 (fvdborne@malawi.net) Bart Vander Plastse Programme Coordinator Malawi Health, Nutrition and Population Programme (Netherlands Assistance) Lilongwe District Health Office Tel: 833-153; cell 930-200 Sheila Dutta Public Health Specialist Africa Region World Bank, Washington Monica Djupvik Programme Officer UNAIDS, Malawi Agnew Makonda-Ridley Country Programme Adviser UNAIDS-Malawi Mark Vander Vort Country Representative Africare Lilongwe Tel: 750701, 753193, 755067 REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 94 Lizzie Funekile Nkosi Programme Director Save the Children/UK Lilongwe Tel: 771433/414/426 Buwa Dragudi Health Manager, Advisor Save the Children/UK Salima Tel: 262275, 919819 Justin Opoku Field Office Director-Malawi Save the Children/US Lilongwe Tel: 753888/919; 755020 Cell: 831710 Shankar Sunuwar Finance Manager SCF/US, Lilongwe David W. Patterson Senior Health Adviser Save the Children/US Mangochi Tel: (265) 594622/625/652 Cell: (265) 839065 Dorothy E. Namate Country Director-Malawi Project HOPE Blantyre Tel: 636282/704 Christine Witte Country Director International Eye Foundation-Malawi Blantyre Tel: 265-624-448; cell 827-115 ANNEXES 95 Mzimba District Cuthbert Nyirenba CHAPS Program Manager Africare Mzimba Tel: 342271; Tel/fax: 333475 Edwin Libanba Distsrict Health Officer Mzimba Tel: 342222 Mary Malata Community Health Coordinator Africare, Mzimba Albert Bwinger Community Health Coordinator Africare, Mzimba Chaplain Katumbi HMIS Officer Africare, Mzimba Mack Mathews Records Clerk Africare, Mzimba Raban Kalua Assistant Statistician Africare, Mzimba Sibongile Chibwe Administrative Assistant/Accounting Africare, Mzimba Paul Chidothi Driver Africare, Mzimba Dyclean Chirwa Driver Africare, Mzimba Hyghten Mungoni Community Mobilizer/IGA Africare, Mzimba REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 96 Kondwani Zimba Office Worker Africare, Mzimba Andrew Ziwa Office Worker Africare, Mzimba Vincent Kaunda HIV/AIDS Program Coordinator MOH/DHMT Christopher Finje Deptuy AIDS Coordinator Mzimba District Health Office Sifo Nyasulu Hospital Director DHMT, Mzimba Matthew Mataka Hospital Administrator Mzimba District Hospital Makegreat Milusa Enrolled Nurse Midwife Mzimba District Health Office Ketwin Kontowe Environmental Health Services Mzimba District Health Office Fletcher Ndhlovu Water and Sanitation Officer Mzimba District Health Office Moffat Sakala Clinical Officer Mzimba District Health Office T. L. Mwamsamali Central Pharmacy Manager Mzimba DHO ANNEXES 97 A. S. Nyajuly Transport Manager Mzimba DHO K.C. Kayanza Assistant Accountant Mzimba DHO Mt. T. Keromko Accountant Clerk Mzimba DHO Senior Staff Nurse Mzimba DHO Kameteka Health Center Staff Rebecca Magona, Nurse Mehalas Kalipi, HSA Kasoka Kabayodo, HSA Kwnjulu, HSA TBA and Community Members Mzimba Mkhalipi Kamteteka HSA Health Center, Mzimba Phiri Manyamula Health Assistant Health Center, Mzimba Mwakaunja Balirenge Health Assistant Health Center, Mzimba Helen Munthali Coordinator Tovwirane Center Mzimba Salima District Brigitte Mura District Commissioner Salima REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 98 David Chinyanya District Planning Officer Salima Ngoma Kondwani District Health Officer Salima Mr. Mitschi District Environmental Health Officer Salima Mr. Kasira CCO Salima Elisabeth Chaima TBA Program Manager DHMT Salima Ahmed Sadi District Accountant MOHP, Salima Buwa Dragudi Health Manager, Advisor Save the Children/UK Salima Tel: 262275, 919819 Alice Gandiwa (Ganiula) Community Health Program Promoter SCF/UK, Salima M. Grace Secretary SCF/UK, Salima M. Chisangwa Project Coordinator SCF/UK, Salima Gabriel Banda Microcopist, Malaria Coordinator SCF/UK, Salima ANNEXES 99 Patrick Nayupe HIV/AIDS Coordinator SCF/UK, Salima James Kabuli Project Administrator SCF/UK, Salima George Alundika HIV/AIDS Program Assistant SCF/UK, Salima REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 100 Mangochi District Jack Mguluwe District Commissioner Mangochi Thomas McCormack Deputy Field Office Director SCF/US, Mangochi David W. Patterson Senior Health Advisor SCF/US, Mangochi Jimmy Gama CHAPS Coordinator SCF/US, Mangochi Jessie Tambwali CHAPS Trainer (TBAs, STI, IMCI, VHCs) SCF/US, Mangochi Maggie Malunga CHAPS Trainer (FP, IMCI, QA) SCF/US, Mangochi Hilda Matope Trainer (BF/Infant Feeding, PMCTC) SCF/US, Mangochi Francis Khmymgwa CHAPS Zone Coordinator SCF/US, Mangochi Allison Zakaliya CHAPS Data Monitoring and Evaluation (M&E) Officer SCF/US, Mangochi Alison Zacharia MIS Officer SCF/US, Mangochi George Chithope-Mwale District Health Officer Mangochi ANNEXES 101 Matthew Mataka Hospital Administrator/Manager Principal Health Services Ast. MOHP, Mangochi Jameson Chausa District Environmental Health Officer MOHP, Mangochi Kennedy Ghambi Environmental Health Officer (EHO) MOHP/DHMT, Mangochi Taibu Mbalaka Accountant MOHP, Mangochi Edward Pavindi Assistant Accountant MOHP, Mangochi E.M. Banda Accountant Assistant MOHP, Mangochi A.S. Likotima Accountant Assistant MOHP, Mangochi Pharmacy Manageer MOHP, Mangochi Transport Manager MOHP, Mangochi M. Mary District Accounting Assistant Mangochi Central District Mwate Chintu Resident Advisor Linages Project Mangochi Mamwera Health Center Staff A. Kalemanjiro, Zonal Coordinator C. Condive, In-Charge REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 102 Nurses HSA Chalipa Health Center and Zonal Office Staff In-Charge, Nurses, Assistant Zonal Coordinator, HAS Mulanje Nikisi Village Chief, Headman, and Health Committee Members (10) Mulanje Chikwawa District Tommy Mthepheya CHAPS Program Manager International Eye Foundation Chikwawa Tel: 423283/274 Cell: 950194 Dr. Christine Witte IEF Country Director Frank Chola CHAPS System Advisor/Administrator IEF, Chickwawa Elton Chiumia CHAPS Reproductive Health Coordinator IEF, Chickwawa Lovemore Mvula IEF Food Security Project Manager George Mekiseni CHAPS Quality Assurance/Training Advisor IEF, Chickwawa Paul Chibisa Assistant Reproductive Health Coordinator IEF, Chickwawa Coplands Navicha CHAPS Community Health Supervisor IEF, Chickwawa ANNEXES 103 Byson Chidzalo CHAPS Community Health Supervisor IEF, Chickwawa Andrew Umali CHAPS Project Accountant IEF, Chickwawa Linda Khunga Secretary IEF, Chickwawa James Longwe CHAPS WES Project Manager Concern Universal Chickwawa Noor Aide District Health Officer Chikwawa Tel: 423266/264/223 Paul Chunga District Environmental Health Officer MOHP, Chickwawa Hilda Gausi Acting District Nursing Officer MOHP, Chickwawa Mercy Kamphinda District Reproductive Health Coordinator MOHP, Chickwawa Timeo Nyirenda Accountant MOHP, Chickwawa Shadreck Mulenga District Quality Assurance Coordinator MOHP, Chickwawa Peter Ngondo Environmental Health Officer Ngabu Rural Hospital Chickwawa REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 104 Mr. Nkango Clinical Officer Ngabu Rural Hospital Chickwawa Mrs. Mkwira Nursing Matron Ngabu Rural Hospital Chickwawa Community Based Distribution Agents (11) and HSA Ngabu Rural Hospital Chickwawa Concern Universal Staff Losi Village Water System, Chickwawa Losi Village Chief, Headman and Village Health Committee Chickwawa Mulanje District Timothy Kachyle Program Manager, CHAPS Project HOPE Mulanje Tel: 465247/417/415 Cell: 861705 Gertude HIV/AIDS Coordinator Project HOPE Elias Chimulambe Field Trainer/Water and Sanitation Coordinator Project HOPE Emma Nkhoma Field Trainer Project HOPE Naomi Chimpeni HIS Coordinator Project HOPE ANNEXES 105 James Namame Project Accountant Project HOPE, Mulanje Dr. Phire District Health Officer Mulanje Medson M. Semba Chief Clinical Officer/Assistant DHO Mulanje District Hospital Mulanje Tel: 465211/295/236 Home: 465285 Mr. Theu District Environmental Health Officer Mulanje Mr. Elyia Matron Mulanje District Hospital Mr. Madatua Accountant Mulanje District Hospital Dennis Chiombeza HIV/AIDS Coordinator DHMT, Mulanje Robson Kayira Assistant Environmental Health Officer DHMT, Mulanje Mr. Chimaliro Pharmacy Manager MOHP, Mulanje B. Makonde Hospital Administrator MOHP, Mulanje Alex Mklija Transport Officer MOHP, Mulanje REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 106 Arthur Madanitsa Accountant MOHP, Mulanje Mr. Masinkho Accountant MOHP, Mulanje Dr. Sizoo Medical Director Mulanje Mission Hospital ANNEX E DOCUMENTS CONSULTED REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 108 Documents Consulted Aubel, J. Reorienting Program Evaluation to Promote Organizational Learning. Participatory Stakeholder Evaluation. 1/1/02. Christian Medical Association of Malawi (CHAM). Strategic Plan: Christian Health Association of Malawi. 10/1/00. Africare. Africare: 2000 Annual Report. Washington: Africare/US HQ. 12/31/00. Africare. Communication and Behaviour: Principles and Practices. Africare/Malawi. 2/1/02. Africare/DHMT Mzimba. CHAPS/Mzimba 15 Month Extension Proposal. Africare/Mzimba. 3/1/01. Africare. Africare/Mzimba CHAPS Proposal Application (without annexes). AFricare/Malawi. 6/1/97. Africare. Quarterly Reports (CHAPS Mzimba). Africare. Budget and Actual Expenses as of 12/31/01. Africare. Quarter Report: CHAPS Mzimba. 12/31/01. Africare. Africare Staff Listing. 2/28/02. Africare. Africare Project Plan/Proposal. 6/1/97. Africare. Africare Annual Reports 1999 & 2000. Africare. Africare 15 Month Extension Proposal. Africare. Africare Brochure. AICPA/Auditing Standards Board. SAS-AICPA #55: Consideration of the Internal Control Structure in a FS Audit. 2002. Bossert, T. J., Joel C. Beauvais. Decentralization of Health Systems in Ghana, Zambia, Uganda and the Philippines: a comparative analysis of decision space. Health Policy and Planning 17(1): 14-31, March 2002. Chikaonda, Matthews A. P. Report on the Round Table Conference on HIV/AIDS in Malawi. Held in Lilongwe 24-25 March 2000. Ministry of Finance and Economic Planning. ANNEXES 109 Connell, Peter, Beatriz Ayala-Ostrom, Chisale Mhango, Warren Robinson. Building the Foundations for Improved Reproductive Health Care in Malawi. Assessment of USAID's Amended Support to AIDS and Family Health Project (STAFH) Project. Submitted to USAID by LTG & TvT Associates. December 2001. Department of Local Government/Office of the President and Cabinet. Decentralisation Process in Malawi. Medium Term Implementation Plan 2001-2004. Lilongwe, July 2001. Elias, Collin. Exit Notes, CHAPS Mzimba. Africare. 2/28/02. European Commission (EC). Performance Indicators. GOM/National Economic Council. Economic Report 2001. 2001 Goodwin, J. Save the Children/UK. Health Planning Consultancy. 11/26/01. Heaton, Annie. Joint Public Private Initiatives: Meeting Children's Right to Health? London: SCF/UK. 5/1/01. Huggett, T. Africare/Mzimba: CHAPS Mid-Term Evaluation. 3/2/00. International Eye Foundation. International Eye Foundation (IEF) Quarterly Reports (CHAPS Chickwawa). JHPIEGO. National Infection Prevention Standards. 2/1/02. JHPIEGO. Malawi National Reproductive Health Service Delivery. 10/1/99. JHPIEGO. Systems User Manual. 2/1/02. JHPIEGO. Reproductive Health/FP Learning Package for Malawi Preservice Training. 5/1/01. Kadzamira, Esme Chipo, et al. The Impact of HIV/AIDS on Primary and Secondary Schooling in Malawi: Developing a Comprehensive Strategic Response. Executive Summary. Centre for Educational Research and Training (CERT), September 2001. Kaponda, Chrissie, et al. Mid-Term Evaluation. CHAPS-Joint Programme for Project HOPE and Mulanje District Hospital. 5/1/00. Management Institute of Malawi. Quality Assurance Component. Participant Materials. 1/4/98. Measure/DHS. Nutrition Among Young Children and Mothers. 8/1/01. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 110 MOHP/Strategic Planning Unit. Malawi National HIV/AIDS Strategic Framework 2000- 2004. 10/1/99. MOHP/Strategic Planning Unit. National HIV/AIDS Strategic Plan: Agenda for Action, Annex 1. 10/1/99. MOHP/Planning Department. Malawi National Health Accounts (NHA): A Broader Perspective of the Malawian Health Sector. June 2001. MOHP/Planning Department. Donor Activities Mapped Against the National Health Plan. Preliminary Draft. As of 7 September 1999. MOHP/EHP Working Group/UNICEF. Malawi Essential Health Package (EHP): Contents and Preliminary Costing. First Draft. 12/1/01. MOHP. Summary of Discussion at the EHP Round Table Meeting. 2/22/02. MOHP. Malawi National Health Plan 1999-2004. 11/2/99. MOHP. Malawi Health Population and Nutrition Program (Lilongwe District). Semi￾annual Report #6. July-December 2001. Netherlands Assistance and MOHP. MOHP. To the Year 2020. A Vision for the Health Sector in Malawi. 5/1/99. MOHP/Office of Vice President/National AIDS Commission/National TB Programme. The Comprehensive HIV/AIDS Management Strategy for Malawi. Submission to the Global Fund to Fight HIV/AIDS, Malaria and TB: Five Year Programme, 2002-2226. MOHP/HMIS. Health Passport: Woman Health Profile; General Health Profile; Child Health Profile. 2002. MOHP/EC Health Sector Reform and Decentralization Project. MOHP.EC/British Council. 5/1/01 MSI. District Health Office Management Audit. 4/24/98. Nancollas, Saras. Development of an Effective and Sustainable Health Transport Management System for the Ministry of Health and Population of Malawi: Report of a follow-up consultancy. MOHP and Health Sector Reform and Decentralization Project/European Commission and British Council. February 2001. National Statistics Office. 1998 Malawi Population and Housing Census. 12/31/00. National Statistics Office & Measure/DHS. Malawi Demographic and Health Survey 2000. Preliminary Report. National Statistics Office (Zomba) and Measure DHS+. March 2001. ANNEXES 111 National Statistics Office & Measure/DHS. Malawi Demographic and Health Survey 2000. National Statistics Office (Zomba) and Measure DHS+ (ORC Macro, Calverton, Maryland). August2001. PHR/Abt. Malawi Financial Management Consultancy. 7/22/00. PHR/Abt. PHR Technical Report No. 46: Hospital Autonomy in Malawi. 9/1/00. Poore, Peter. Placing the Save the Children Health Programmes in West Africa and Southern Africa in the Context of the Global Sector Strategy (GHSS 1998). 5/1/01. Project HOPE. Project HOPE Quarterly Reports (CHAPS Mongochi). PSI. PSI Quarterly Reports and 2001 Annual Report Research Triangle Institute. Refocusing PHICS. 3/1/94. Robinson, Matt. Loose Minute: Southern Regional Planning and Budgeting Report. MOHP/Planning Department. March 2002. Rubardt, Marcie. Final Evaluation STAFH. Mid-Term Evaluation CHAPS Project. 9/30/99. Save the Children/UK. The Bitterest Pill of All: Collapse of Africa's Health System. SCF/UK. Mid-Term Evaluation: Salima Community Health Partnership. 11/1/99. SCF/UK. Community Managed Bicycle Ambulances. SCF/UK. Revised Salima CHAPS Proposal. 6/4/97. SCF/UK. 2001 Annual Report. Save the Children/UK. SCF/UK. Quarterly Reports (CHAPS). Salima. SCF/UK/MOHP CHAPS Salima: Where Are We Now? Review of the Last Three Years. Workshop Report. 7/06/01. SCF/US. Annual Reports: 1998 and 1999. SCF/US. Quarterly Reports: CHAPS Mangochi. SCF/US. Mid-Term Evaluation: CHAPS Mangochi District. REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 112 Stanecki. Karen A. The AIDS Pandemic in the 21st Century. The Demographic Impact in Developing Countries. (Paper prepared for presentation at XIIIth International AIDS Conference, Durban, South Africa, July 2000.) U. S. Census Bureau. Thunhurst, Colin. Support to the District Implementation Planning Process. MOHP and Health Sector Reform and Decentralization Project/European Commission and British Council. November 2000. Thunhurst, Colin. Review of Health Ministry Planning Processes. 11/01. UNAIDS. AIDS Epidemic Update: December 2001. Geneva: UNAIDS/WHO. 2001. UNAIDS. Measuring the Level of Effort in the National and International Response to HIV/AIDS: The AIDS Programme Effort Index (API). Summary Report. UNAIDS, USAID, and the POLICY Project. UNAIDS. AIDS Program Effort Index for Malawi: 2000 Assessment. UNDP. Common Country Assessment of Malawi. 2001 Report. United Nations System in Malawi. UNDP. United Nations Development Assistance Framework. Malawi 2002-2006. United Nations Development System. 2001. UNICEF Country Programme Recommendations. 9/02/01 UNICEF/WHO/GOM. Baseline Survey on Key Community Child Care Practices in Selected Districts of Malawi. 12/01. USAID/Malawi. Performance Monitoring Plan for Country Strategic Plan (CSP) 2001-5: Strategic Objective #8: "Behaviors Adopted that Reduce Fertility, Risk of HIV/AIDS and Improve Child Health."6/6/01. USAID/Malawi. Country Strategic Plan. 2/3/01. USAID/Malawi. CHAPS: Amendment No. 2, Project No. 612-0246. 9/1/97. USAID/Malawi. Scope of Work for Evaluation of CHAPS: Final Version 2/21/02. USAID/Malawi. Strategic Grant Agreement between the USA and the GOM. 5/31/01. USAID/Malawi. Country Strategic Plan: Condensed Version. 2/2/01. USAID/Malawi. Results Review: FY 2002. 4/6/00. USAID/Malawi. Collaborative Agreement (CoAg) PSI. 7/27/98. ANNEXES 113 USAID/Malawi. CHAPS Program Description for RFA. USAID/Malawi. Original RFA for CHAPS. 4/18/97. USAID/Malawi. CHAPS Quarterly Report for Mzimba, Mangochi, Salima, Mulanje, Chickwawa): 1998-2001. USAID/Malawi. R4 1998, 1999, 2000. USAID/Malawi. Report of CHAPS Team Quarterly Meeting Held at Sun’N’Surf Holiday Resort, Mangochi, 30 June 1998. Waltensperger, Karen. Saving Newborne Lives: Malawi. SCF/UK. 6/1/01. Wansi. CHAPS-Africare: Annual Report 1999. 12/31/99. WHO Malawi Country Programme Pamphlet. 10/01. WHO/GOM. Tools for Assessing the Qperationality of District Health Systems: Health Facility Questionnaire. WHO/GOM. Tools for Assessing the Qperationality of District Health Systems: District Questionnaire. World Bank. Multi-Country AIDS Program (MAP). Monitoring and Evaluation (M&E) Operational Manual. 9th Revised Draft, 24 February 2002. ___ Cost of Reproductive Health. 2/1/01 REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 114 ANNEX F MATRIX OF ALTERNATIVE MODELS FOR FUTURE PUBLIC/PRIVATE PARTNERSHIPS 116 Matrix of Alternative Future Coordination Strategies/Models for Public/private Partnerships (Left column lists criteria. Top row are alternative coordination strategies. ) Implementation Model 1 Implementation Model 2 Implementation Model 3 Implementation Model 4 Status Quo Continued PVO/DHMT Partnerships with Continued (Direct) USAID Oversight Top-Down TA at MOHP HQ With Contractor Oversight District Level Contractor giving TA to PVOs that work at District level District Level PCU actually at a District with Peer level TA Ease of Administrative Oversight __ + ++ +++ Ability to foster positive partnerships __ __ __ ++ +++ Ability to strengthen public private partnerships + __ __ ++ +++ Ability to strengthen relationships between central and district level + __ __ + +++ Ability to directly address issues related to poverty alleviation ++ __ __ __ +++ Ability to respond to decentralization initiatives and procedures + -- -- +++ Transparency between GOM and USAID + __ __ __ __ ++ Ability to coordinate at district level with other donor initiatives + __ __ __ __ ++ Transparency between GOM and USAID + __ __ __ __ ++ ANNEXES 117 Ability of assistance to adjust to changes in reform velocity/demand + __ __ __ ++ Monitoring Effectiveness __ + ++ +++ Contribution toward Decentralization + __ + +++ Contribution toward moving the Districts toward a District-level SWAp + __ + +++ Evaluation Accountability __ __ __ ++ +++ Ability to document best practices + __ __ ++ +++ Ability to Implement Financial management Improvements __ + ++ +++ Ability to Implement QA Improvements __ + ++ +++ Ability to retain District level field staff + ++ ++ ++ Ability to Contribute and tangibly demonstrate achievement of Short Term SO initiatives improvements + __ + ++ Ability to aid and sustain any technical transfer of knowledge + __ + +++ Ability to remain pragmatic + __ + +++ REPORT OF THE EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIP PROJECT (CHAPS) 118 Control of Indirect and Overhead Costs + __ __ __ ++ Ability to build on past experience ++ __ ++ +++ Ability to facilitate communication + __ __ __ + Overall Sustainability + __ __ __ + ++ Ability to foster a common vision between donors and GOM _ __ __ __ __ + Ability to integrate accounting, and other resource management systems between GOM and donors + __ __ __ +++ Ability respond to emergency situations and needs, e.g. cholera or conflict ++ __ __ __ + ++ 119