PROJECT HOPE Community Health Partnerships (CHAPS) Project Cooperative Agreement No. 690-A-00-98-00093-00 Final Evaluation Report Project Location: Mulanje and Phalombe Districts, Malawi Project Duration: November 1997 – March 2003 Submitted to: U.S. Agency for International Development Lilongwe 3, Malawi Submitted by: Project HOPE – The People-to-People Health Foundation, Inc. Millwood, Virginia 22646 Tel: (540) 837-2100 Fax: (540) 837-1813 March 2003 Prepared by External Evaluators: Marcie Rubardt, RN, MPH Anna Kathryn Webb, PhD Contact persons: Bettina Schwethelm, PhD, MPH Dorothy Namate, PhD, RNM Director, Maternal & Child Health Programs Country Director Project HOPE/U.S. Project HOPE/Malawi FINAL REPORT FINAL EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIPS (CHAPS) PROJECT MULANJE AND PHALOMBE DISTRICTS, MALAWI USAID GRANT NO. 690-A-00-98-00093-00 NOVEMBER 1997 – MARCH 2003 PROJECT HOPE MALAWI BLANTYRE, MALAWI PROJECT HOPE, THE PEOPLE-TO-PEOPLE FOUNDATION, INC. MILLWOOD, VIRGINIA 22646 by Marcie Rubardt, RN, MPH Anna Kathryn Webb, Ph.D. San Antonio, Texas March 14, 2003 Final Report ii March 14, 2003 The views expressed in this document are entirely those of the authors and should not be attributed to Project HOPE. Final Report iii March 14, 2003 Acknowledgements We are most grateful for the guidance of Petra Reyes, Regional Director for Africa, Project HOPE Center; Dorothy Namate, Country Director, Project HOPE Malawi; Timothy Kachule, Project Manager, Community Health Partnerships (CHAPS) Project, Mulanje and Phalombe Districts; and Bettina Schwethelm, Director, Maternal and Child Health Programs, Project HOPE Center. We also wish to express our gratitude for the generous assistance of Sue Adams, Assistant Director for Africa, Leslie MacLean, Assistant Regional Director-Finance, Africa and Haiti; and Linda Furcho, Administrative Assistant of Project HOPE Center; and Shalote Chipamaunga, Technical Support Manager, and Clifford Matunga, Finance and Administration Manager, of Project HOPE Malawi. We offer our deepest appreciation to CHAPS Project staff who were both good teachers and good hosts: Delia Chikuse, Elias Chimulambe, Christina "Chrissie" Chiwalo, Gertrude Kachitsa, Emma Nkhoma, Jimmy Phiri, and Malocho Tchuwa, Community Health Educators; James Namame, Accountant/Administrator; Naomi Chimpeni, Health Management Information System Officer; Ellah Kumwembe and Angela Luchapa, Office Assistants; and Tobias Chingwala and Mr. Dewu, Drivers. Very special recognition is reserved for the evaluation team Researchers, Hilda Mtawah, Charles Mulimi, and Betty Pwetekani, whose hard work and good humor were abundant throughout. We are deeply indebted to the many individuals who participated in the evaluation from Mulanje and Phalombe Districts—District Health Management Teams, Mulanje District Hospital staff, Mulanje Mission Hospital staff, Holy Family Hospital staff, health center staff, Mulanje District Youth Officer, District AIDS Coordinating Committee, trained community volunteers, and community members and leadership. Finally, we wish to thank Mexon Nyirongo, Population, Health, and Nutrition Team Leader, and Cheri Kamin, United States Agency for International Development (USAID), for their fruitful discussion of the project. Final Report iv March 14, 2003 Currency Equivalent Monetary Unit = Malawi Kwacha (K) US$1.00 = K89.6127 (February 7, 2003) US$1.00 = K75.4563 (July 17, 2002) Acronyms and Abbreviations AIDS acquired immunodeficiency syndrome ARI acute respiratory infections CBDA community-based distribution agent CHAM Christian Hospital Association of Malawi CHAPS Community Health and Partnerships project DACC District AIDS Coordinating Committee DHO District Health Office/Officer DHMT District Health Management Team DRF drug revolving fund DYO District Youth Office/Officer FP family planning GIK gift in kind GTZ Gesellschaft fur Technische Zusammerarbeit (German Technical Assistance) HBC home-based care HIV human immunodeficiency virus HOPE Health Opportunity for People Everywhere HSA health surveillance assistant HMIS health management information system IEC information, education, and communication IMCI integrated management of childhood illnesses MACRO Malawi AIDS Counseling and Resource Organization MASAF Malawi Social Action Fund MBORP Malawi Breastfeeding Operations Research Project MCH maternal and child health MOHP Ministry of Health and Population MTCT mother to child transmission of HIV/AIDS NAC National AIDS Commission NGO nongovernmental organization OXFAM Oxford Committee for Famine Relief PHICS Promoting Health Interventions for Child Survival project PMTCT prevention of mother to child transmission of HIV/AIDS RH reproductive health STAFH Support to AIDS and Family Health Project STI sexually transmitted infection TBA traditional birth attendant UNDP United Nations Development Programme UNICEF United Nations Children's Education Fund USAID United States Agency for International Development VCT Voluntary Counseling and Testing Final Report v March 14, 2003 Table of Contents Page Acknowledgements.............................................................................................................. iii Acronyms and Abbreviations............................................................................................... iv Executive Summary ........................................................................................................... viii Introduction............................................................................................................................1 I. Country Context............................................................................................................1 A. Poverty .......................................................................................................................2 B. Social Indicators.........................................................................................................2 C. Food Security Crisis...................................................................................................3 D. Health Sector..............................................................................................................3 II. Evaluation Methodology...............................................................................................4 A. Selection Criteria for Site Visits ................................................................................4 B. Evaluation Team ........................................................................................................4 C. Community Evaluation ..............................................................................................4 D. Health Services Evaluation ........................................................................................5 E. Partnership Evaluation ...............................................................................................6 F. Project HOPE Evaluation...........................................................................................6 G. Assessment of Project Outcome ................................................................................7 III. History of the CHAPS Project ..................................................................................7 A. Background ................................................................................................................7 B. CHAPS Project ..........................................................................................................8 C. One-Year Cost Extension...........................................................................................9 D. No Cost Extension ...................................................................................................10 IV. Community Participation in Health Care................................................................11 A. Accomplishments.....................................................................................................11 B. Capacity Building ....................................................................................................12 C. Community Perspectives..........................................................................................16 D. Project Objectives and Observed Behavior Change ................................................18 E. Lessons Learned and Recommendations .................................................................20 V. Health Services.............................................................................................................24 A. Accomplishments.....................................................................................................24 B. Training....................................................................................................................24 C. Quality Improvement ...............................................................................................27 D. Health Service Practices...........................................................................................27 VI. Health System Strengthening...................................................................................30 A. Accomplishments.....................................................................................................30 B. Supervision...............................................................................................................31 C. Staffing.....................................................................................................................32 D. Logistics: Drugs, Supplies, and Equipment .............................................................33 E. Communications and Referral System.....................................................................34 F. Work Planning .........................................................................................................35 Final Report vi March 14, 2003 G. Health Management Information System ................................................................36 H. Financial Management.............................................................................................36 I. District Coordination of Health Activities ...............................................................37 VII. DHMT and Project HOPE Partnership ....................................................................38 VIII. Major Factors that Affected the Achievement of Project Objectives ......................41 A. Factors not generally subject to Government Control .............................................41 B. Factors generally subject to Government Control ...................................................41 C. Factors generally subject to Project HOPE Control.................................................43 IX. Implementation Delays ............................................................................................44 X. Monitoring and Evaluation ..........................................................................................44 A. Project HOPE Center ...............................................................................................44 B. Project HOPE Malawi..............................................................................................44 C. CHAPS Project ........................................................................................................44 XI. Provision of Supplies and Equipment......................................................................46 XII. Studies included in the Project.................................................................................46 XIII. Project Management Performance ...........................................................................47 A. Project HOPE Center ...............................................................................................47 B. Project HOPE Malawi..............................................................................................47 C. CHAPS Project ........................................................................................................48 D. District Health Management Teams.........................................................................49 XIV. Implementation Performance...............................................................................49 A. Project HOPE...........................................................................................................49 B. District Health Management Teams.........................................................................50 C. Health Services.........................................................................................................50 D. Trained Community Volunteers...............................................................................50 XV. Project Design ..........................................................................................................51 XVI. Sustainability........................................................................................................51 XVII. Assessment of Outcome.......................................................................................53 XVIII. Summary Conclusions, Lessons Learned, and Recommendations......................53 Bibliography.........................................................................................................................55 Annex A: Scope of Work.....................................................................................................60 Annex B: Sites Visited, Persons Met, and Groups Interviewed ..........................................64 Addendum: Evaluation Instruments Final Report vii March 14, 2003 Tables Page Table 1: CHAPS Project Timetable 65 Table 2: Number of Health Center Staff trained by Intervention and Type of Training 24 Table 3: CHAPS Project Staffing, March 1998 – March 2003 66 Table 4: CHAPS Project Staff Turnover, March 1998 – March 2003 66 Table 5: Summary of Assessments 67 Table 6: CHAPS Project Progress toward Targets and Objectives 68 Note to the Reader: The evaluation instruments that were used in the conduct of the final evaluation of the Community Health Partnerships (CHAPS) project are presented in the separate document, Evaluation Instruments. Evaluation Instruments 1. Interview Guide for Group Meetings with Community Members 2. Interview Guide for Group Meetings with Trained Community Volunteers 3. Interview Guide for Youth Groups 4. Interview Guide for Extended DHMT 5. DHMT Individual Survey Questionnaire 6. Guide for Health Center Visits 7. Interview Guide for Project HOPE Final Report viii March 14, 2003 Executive Summary Goal and objectives. The goal of the Community Health Partnerships (CHAPS) project was to improve health care services in the district of Mulanje through a partnership between Project HOPE and all levels of the health infrastructure in Mulanje district, which was divided administratively into two districts, Mulanje and Phalombe, in November 2000. The partnership would be the vehicle for (a) improving maternal child health (MCH) services and reproductive health (RH) delivery, (b) building human and institutional capacity in a sustainable manner, and (c) assisting USAID in achieving its Strategic Objective 3 in Malawi, "increased adoption of measures that reduce fertility and risk of HIV transmission and improve child health practices," to be supported by three key intermediate results: knowledge and use of good reproductive health practices improved, community participation in reproductive health care improved, and the range and quality of reproductive health services for women, men and youth expanded. The project objectives were to improve the health status of the population of Mulanje district, with special emphasis on MCH and RH program activities and to improve the management capacity of the District Health Management Team (DHMT) and other health personnel to support priority health interventions within the district in a sustainable manner. Interventions. The project had six priority health intervention areas: malaria, control of diarrheal diseases (CDD), acute respiratory infections (ARI), HIV/AIDS, sexually transmitted infections (STIs), family planning, and nutrition. Voluntary counseling and testing (VCT) and the prevention of mother to child transmission (PMTCT) of HIV/AIDS were introduced in the latter part of the project. In addition, the project agreed to focus on management skills, information, education and communication (IEC) programs, health management information systems, and improved access at the district level. Project cost, financing, and duration. The project cost was US$3.7 million, which was co-financed by USAID, Project HOPE, and community contributions. The project implementation period was November 1997 to March 2003 and included a one-year cost extension and a nine-month no cost extension. Accomplishments. Over the life of the project, 181 training activities were conducted, with a total of 4,084 participants: community volunteers (38 percent), health care providers (46 percent), youth (5 percent), and others, including shop owners and literacy teachers (11 percent). Community volunteer included home-based care groups, community-based distribution agents for family planning, water point committees, drug revolving funds, drama groups, youth groups, and community health committees. Health care providers from the three district hospitals and 27 health centers included medical directors, program coordinators, medical assistants, matrons and nurses, laboratory technicians, pharmacists, VCT counselors, data entry staff, ward attendants, and cleaners. Final Report ix March 14, 2003 Training was very high quality and had a significant, positive impact on health district capacity to deliver services and on building the knowledge base in communities that is necessary for behavior change. In addition to training, the project conducted five health education campaigns and held two youth conferences and two drama/music competitions The project also financed four Under Five outreach clinics that brought health education closer to communities and increased access to Under Five services; it funded the construction of four district health staff houses to shelter and help to attract and retain health personnel. For VCT, the project remodeled a counseling room and laboratory. Water and sanitation was improved in 210 communities through the protection of water sources and latrine construction; communities contributed their time and local materials. Gift in kind (GIK) supplies and equipment valued at US$1.5 million were distributed to hospitals and health centers, which would not otherwise have been available to them. Project HOPE Malawi provided local supplies and equipment in the amount of US$177,280, for example, volunteer kits for the different volunteer groups. As part of health district strengthening, the project improved supervision through the development of supervision checklists and, to a limited extent, funded district vehicle fuel and maintenance costs for supervision, which improved supervision performance through increased supervisory visits. The project contributed bicycles to district revolving fund bicycle programs for health surveillance assistants (HSAs) to facilitate their supervision of trained community volunteers. District communication and referral systems were enhanced by the installation and/or repair of health center radios, which allowed health centers to have direct, rapid access to the districts, which was necessary for the referral of critical patients, among others. The project provided technical assistance and training for the Ministry of Health and Population's (MOHP) new health management information system (HMIS) and training and software for a computerized pharmacy system in Mulanje district. Annual meetings to prepare work plans and quarterly meetings to review plans and discuss CHAPS project progress helped to build district planning capacity. The project's major accomplishment was that remote, dispersed, and largely inaccessible communities gained access to high quality services, and community participation in health care increased significantly due to the work of trained volunteer groups and health education activities. The project reached an estimated 75 percent of the 898 communities that comprise Mulanje and Phalombe districts. Another substantial accomplishment was that Project HOPE earned the communities' trust and established an outstanding reputation. Both trained volunteers and community members were deeply appreciative of Project HOPE's presence and work in their communities. Winning community trust owed in large part to district health staff whose prior work and experience in target communities facilitated Project HOPE's entry. Outcome. The project's outcome was satisfactory, based on overall implementation performance, management performance, monitoring and evaluation, and sustainability. Final Report x March 14, 2003 The major factor affecting the achievement of project objectives was the high rate of district staff turnover. In the public health sector, 40 percent of posts are vacant, and the ability to provide health care at all levels is exacerbated by staff turnover. All district health staff received training, but by the end of the third year of implementation, no trained staff remained at the district level, including district health officers and program coordinators. At the health center level, the turnover rate was less, and the major part of health center staff who received training continue in their posts. The high turnover rate of district health staff severely limited the partnership's effectiveness because of the constant need to orient new staff to the project and gain their support for it. Factors that had a positive effect on project outcome were USAID and Project HOPE commitment, the correct identification of needed health interventions, viable technical interventions, effective training, key sector policies, adequate resources, and appropriate incentives for community participation, which included training, supervision, and the provision of supplies and equipment. The technical interventions were feasible with the exception of PMTCT of HIV/AIDS and VCT because sector policies were not put in place in a timely manner, which had a negative impact on the implementation of these interventions. The project had adequate resources to implement planned activities. The highly satisfactory performance of Project HOPE CHAPS project staff, which had low turnover, contributed substantially to the satisfactory project outcome. Apart from high MOHP staff turnover, factors that had a negative effect on project outcome were less than hoped-for supervision and follow-up, an inadequate monitoring and evaluation system, moderate management problems, the perceived lack of financial transparency on the part of the DHMTs, insufficient attention to partnership development, and the lack of MOHP commitment to not transfer district staff. Supervision and follow￾up were limited to an extent by the fact that district health personnel training was intensive, and some staff did not have time to implement training knowledge and skills or supervise. Inadequate monitoring and evaluation was the result of problems experienced in the implementation of the health ministry's new HMIS and Project HOPE's own monitoring difficulties. Both DHMT and Project HOPE management experienced moderate problems that limited their effectiveness. The perceived lack of financial transparency on the part of the DHMTs created misgivings about Project HOPE financial management that would have been allayed by a more user-friendly process of budget preparation based on project activities. Partnership development was not given adequate attention in part because technical assistance was not provided for this component. Sustainability. The sustainability of the CHAPS Project is likely, albeit in changed form. In March 2003, USAID is expected to announce the award for Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, which would continue to support the five (sic) District Health Offices currently receiving USAID assistance under CHAPS, in addition to partnerships in three new districts. Phalombe district was not included in the new initiative; thus, the sustainability of the project there is uncertain. The sustainability of trained volunteer activities is likely to the extent that the replenishment of Final Report xi March 14, 2003 needed supplies is forthcoming, HSA supervision continues, and there are incentives for local participation. The sustainability of community involvement in health care is likely with respect to malaria prevention and treatment, control of diarrheal disease, HIV/AIDS and home based-care, family planning, and STIs because communities have a sound knowledge base, key behavior changes reportedly occurred, and perceived benefits would provide the motivation to continue to support positive behavior change. The sustainability of project achievements by the Mulanje and Phalombe DHMTs is uncertain due primarily to the high staff turnover rate, the absence of systematic orientation of new staff to CHAPS interventions or systems, and the lack of financial resources to purchase, for example, volunteer supplies and infection prevention supplies, or to provide for vehicle operations and maintenance, which are essential for supervision and follow-up. Conclusions. USAID piloted the CHAPS partnership approach to strengthening DHMT, which was innovative and untested. The CHAPS project in Mulanje and Phalombe districts demonstrated that such an approach can be successful under certain conditions, namely, USAID, MOHP, and PVO commitment; the correct assessment of needed health interventions, the technical viability of interventions; adequate staffing and low staff turnover; key sector policies are in place; effective training, supervision, and follow-up; adequate resources; effective management; financial transparency, adequate monitoring and evaluation system; appropriate incentives for community participation; and a satisfactory partnership arrangement. In sum, the CHAPS project succeeded when these conditions were substantially met and was less successful when they were not. Lessons learned. In a pilot project that aims to establish a new kind of working relationship between partners who are inexperienced in partnerships, which encompasses not only building the partnership but also strengthening the capacity of one partner, in addition to implementing a wide range of activities among a variety of different target groups across a large geographical area, a considerable investment of resources (human, financial, technical) is needed during the design, planning, and early implementation phases in order to ensure that the implementation of activities and partnership development can proceed as smoothly as possible and to take corrective measures in a timely fashion in the event of problems. In fact, the CHAPS project investment in the project's technical (health interventions) component was significant. Investment in the partnership component was not as substantial as it should have been in part due to the novelty of the approach. The new approach notwithstanding, the principals involved could have made greater efforts to provide technical assistance on partnership development, especially in the areas of roles and responsibilities and financial management. Recommendations. In future partnership-based projects, partnership development, as well as institutional capacity building, should receive attention equal to that of technical components, even if it would mean a reduction in the scope and number of health interventions and related activities, because it is the capacity building and strengthening through partnership development that will sustain project achievements and benefits and not the other way around. Final Report 1 March 14, 2003 Introduction 1. This report is the final evaluation of the Community Health and Partnerships (CHAPS) Project, which was implemented by Project HOPE Malawi with a matching grant from the United States Agency for International Development (USAID). The stated purpose of the evaluation was to “assess the achievement of the CHAPS project toward under each of the three key intermediate results under USAID Strategic Objective 3, the impact of the project interventions and the effectiveness of the NGO-public sector partnership model” (Annex A). The broad objective of the evaluation was to assess project achievement of each major objective and output in accordance with USAID Malawi key intermediate results indicators, with a specific focus on the priority areas for interventions: service delivery, building of human and institutional capacity in a sustainable manner, and operational sustainability of the partnership model.1 2. Preparation for the evaluation began in early January 2003, and the formal evaluation was conducted in Malawi during the period, January 30 to February 18, 2003. The report is based on the review of documents, files, and statistical data, including the preliminary results of the end-of-project household survey; key informant interviews with Ministry of Health and Population (MOHP) District Health Management Teams (DHMTs) and staff for Mulanje and Phalombe Districts, the Mulanje District Youth Office, the Mulanje District AIDS Coordinating Committee (DACC); interviews with Project HOPE Center, Project HOPE Malawi, and CHAPS Project staff; group meetings with beneficiary communities and trained community volunteers; site visits to Mulanje and Phalombe district hospitals and health centers; and a debriefing with USAID on February 18, 2003 (Annex B). 3. Section I of the report describes the country context. Section II presents the evaluation methodology. Section III gives the history of the CHAPS project. Section IV presents the evaluation results for community participation in health care. Section V discusses evaluation results for health services. Section VI analyzes health system strengthening. Section VII reports on the partnership between DHMT and Project HOPE. Section VIII delineates major factors that affected the achievement of project objectives. Section IX identifies implementation delays. Section X assesses project monitoring and evaluation. Section XI concerns the provision of project supplies and equipment. Section XII assesses studies included in the project. Section XIII evaluates project management performance. Section XIV appraises implementation performance. Section XV addresses project design. Section XVI assesses project sustainability. Section XVII provides a summary assessment of project outcome. The final section presents summary conclusions, lessons learned, and recommendations. I. Country Context 1 Project HOPE. Scope of Work, Community Health and Partnerships (CHAPS) Final Evaluation, Project HOPE/Malawi, January 13, 2002, p. 2 Final Report 2 March 14, 2003 A. Poverty 4. Malawi is one of the poorest countries in the world, ranked 151 of 162 countries in the 2001 UNDP Human Development Index.2 In Malawi, 65.3 percent of the population— over six million people—lives in poverty, i.e., these individuals are unable to meet their basic needs. Poverty is worse in rural areas (66.5 percent) than in urban areas (54.9 percent). Nearly ninety percent of Malawi’s poor live in rural areas. The Southern region is the poorest of the three regions (68.1 percent), followed by the Central region (62.8 percent) and the Northern region (62.5 percent).3 In 1999, per capita income was US$180. B. Social Indicators 5. In 2001, the country population was 10.5 million, with a population growth rate of 1.5 percent. Life expectancy at birth was 37.1 years.4 At 189 deaths per 1,000 live births, Malawi has one of the highest childhood mortality rates in the world, although this number has declined by almost 20 percent since the early 1990s.5 The maternal mortality rate more than doubled between 1992 and 2000 from 620 to 1,200 per 1,00,000. An estimated 43 percent of the population does not have access to safe water supplies.6 6. Malawi is rated as one of the ten countries worldwide most affected by AIDS. It is estimated that 800,000 Malawians are now living with HIV. Approximately 15 percent of adults aged 15-49 are infected, translating into a 50 percent lifetime risk of contracting the virus. As a result, AIDS is now the leading cause of death in adults. More than 80,000 adults die annually—double what the number would be without AIDS. This figure is projected to rise to over 100,000 adult deaths annually by 2005. The National AIDS Commission estimates that there are approximately 70,000 new cases of AIDS cases each year. It is estimated that over 300,000 Malawian children have lost parents to AIDS—a number that is projected to increase to over one-half million by 2005.7 USAID's 1998 Children on the Brink study estimated that 36 percent of Malawian children under the age of 15 have lost one or both parents.8 2 United Nations Development Programme (UNDP). Human Development Report 2001, Making New Technologies Work for Human Development, Oxford University Press, New York, 2001 3 World Bank. The state of Malawi's poor: The incidence, depth, and severity of poverty, PMS policy brief No. 2 (revised), November 2000, p. 1 4 Population, Health and Nutrition Information Project (PHNPC). Malawi, PHNIP Country Health Statistical Report, September 2002, p. 1 5 USAID Regional Center for Southern African (RCSA). Request for Applications (RFA) Number 690-03- 003, Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, Issuance date: October 24, 2002, p. 9 6 Government of Malawi. Malawi, Demographic and Health Survey 2000, National Statistical Office, Zomba, Malawi, and ORC Macro, Calverton, Maryland, August 2001 7 USAID Regional Center for Southern African (RCSA). Request for Applications (RFA) Number 690-03- 001, Reducing High Risk Behaviors Among Malawians, Issuance date: October 24, 2002, p. 8 8 Ibid. Final Report 3 March 14, 2003 7. HIV/AIDS is also having a devastating effect on health services in Malawi. In 1996, the annual cost of treating persons with HIV/AIDS was estimated at almost US$2 million, or 7 percent of the MOHP 1995–1996 budget. In the public sector health system, over 70 percent of medical ward beds are occupied by patients with HIV/AIDS-related conditions.9 In addition, absenteeism, illness, and death from AIDS have exacerbated shortages of health care personnel. In the health and education sectors, annual personnel death rates are six times higher than they would have been without AIDS.10 C. Food Security Crisis 8. The Government of Malawi declared a national disaster due to actual and anticipated food shortages on February 27, 2002. The U.N. food agencies estimated that up to 3.2 million people (about 30 percent of the population) will be affected by food shortages by the end of the 2003.11 Adverse weather contributed to crop failures and food shortages; other contributing factors were lapses in the government's early warning systems, distortions in domestic markets, and mismanagement of food reserves.12 There is a direct linkage between household food security and HIV/AIDS, including a loss of able￾bodied labor within the household, loss of remittances from a working family member, adoption of orphans by the elderly, child-headed households, increased expenditures on health care and funerals, among others.13 As survivors of HIV/AIDS victims, women, children, and the elderly are particularly affected due to their relative lack of opportunity for wage labor as compared with able-bodied men.14 D. Health Sector 9. The MOHP is the dominant provider of health care in Malawi, accounting for 60 percent of services delivered. Facilities managed by member institutions of the Christian Health Association of Malawi (CHAM) account for 30 percent of health services delivery in Malawi. CHAM facilities are not private sector institutions in the conventional sense, as the MOHP governs policy at these facilities and is their major source of funds. The Ministry of Finance transfers to CHAM resources that cover the base salaries of all local CHAM personnel.15 At present, 40 percent of posts in MOHP are vacant, and severe shortages of health personnel can be observed at every health facility in the country. There are many causes, including the impact of the AIDS epidemic; unattractive compensation, 9 Ibid., p. 8 10 Source: www.usaid.gov/pop_health/aids/Countries/africa/malawi.html 11 U.S. Agency for International Development. Situation Report, Southern Africa Complex Drought Fact Sheet #6 (FY 2002), Southern Africa Complex Food Security Crisis, June 14, 2002 12 International Monetary Fund. Food Policy and IMF- and World Bank-supported Programs, A Factsheet, July 2002 13 Food and Agriculture Organization (FAO) and World Food Program (WFP), United Nations. Special Report, FAO/WFP Crop and Food Supply Assessment Mission to Malawi, Rome, 29 May 2002, p. 14 14 Ibid. 15 USAID Regional Center for Southern African (RCSA). RFA, Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, op. cit., p. 10 Final Report 4 March 14, 2003 benefits and incentives; shortages of qualified secondary school graduates; past shortages of appropriately qualified tutors; and inadequate funding for health training schools. Many improvement initiatives are in hand or planned, but lead times will be long, and it is unlikely that staff shortages will be resolved before 2006.16 II. Evaluation Methodology A. Selection Criteria for Site Visits 10. The selection criteria for health facilities site visits were: (a) all three hospitals in Mulanje and Phalombe districts, (b) health centers in both districts previously assessed as "strong," and (c) health centers in both districts previously assessed as "weak." The selection criteria for communities were: (a) communities would be located in a selected health center catchment area, (b) the presence of trained community volunteers, and (c) accessibility in view of rainy season road conditions. The selection criteria for trained community volunteer groups were: (a) type of volunteer group and (b) location. B. Evaluation Team 11. The expanded evaluation team consisted of the two consultants, the CHAPS project manager and staff, CHAPS project district coordinators for Mulanje and Phalombe districts, DHMT program coordinators, and the Project HOPE Malawi technical support manager. On January 30-31, 2003, the expanded team met to review and approve the evaluation design and instruments and determine the evaluation schedule of community and health facility visits. Following the site visits, the expanded team met on February 14, 2003 to discuss preliminary findings, identify lessons learned, and develop recommendations. On February 17, 2003, the expanded team presented preliminary findings, lessons learned, and recommendations to the Mulanje and Phalombe District Health Management Team members, other district health staff, and the Project HOPE Country Director. The consultants and the CHAPS Project Manager held a debriefing with USAID on February 18, 2003. 12. The evaluation was conducted by a core team that was divided into two groups. One consultant and two researchers formed the community/Project HOPE evaluation team. One consultant and one researcher comprised the health services/partnership evaluation team. C. Community Evaluation 13. The community evaluation covered four communities (3 in Mulanje and 1 in Phalombe) and seven types of trained community volunteer groups, including 3 16 See USAID Regional Center for Southern African (RCSA). RFA, Reducing High Risk Behaviors Among Malawians, op cit., p. 7 Final Report 5 March 14, 2003 community-based distribution agent (CBDA) groups, 3 home-based care (HBC) groups, 2 drug revolving fund (DRF) groups, 2 Water Point Committees, 2 youth groups, 2 traditional birth attendants (TBAs), and one drama group.17 Separate interview guides were used for trained community volunteers, community members, and youth groups. The interview guide for trained community volunteers addressed the activities of each group and training, supervision, accomplishments, challenges, health services, health education, observed behavior change in relation to the six interventions, and lessons learned. The interview guide for community members focused on the themes of health problems, health services, health education, trained community volunteers, knowledge and practice with regard to the six CHAPS Project interventions, and observed behavior change. The interview guide for youth groups focused on the themes of challenges, health problems facing youth, and aspirations for the future, training, activities, observed behavior change, supervision, reporting requirements, materials and supplies, and accomplishments. All three interview guides were modified after the first day of interviews. (See the Evaluation Instruments document for the community interview guides.) 14. The CHAPS project informed community members and trained volunteer groups in advance of the evaluation team's visit. At each meeting, the evaluation team was introduced, the purpose of the visit was stated, permission for note taking was sought, and how the meeting would be conducted was explained, i.e., one of the researchers would ask questions in Chichewa, and the other researcher would translate the responses for the evaluator who took notes. At the conclusion of the meeting, participants were invited to ask questions. The CHAPS Project provided soft drinks and cookies to participants. Group interviews lasted on average 60 minutes. Participants were generally enthusiastic and responsive. The responses from each group meeting were recorded and reviewed by the two researchers for accuracy and completeness. Due to communication mishaps and poor road conditions, site visits to a few communities and community groups were cancelled. D. Health Services Evaluation 15. The evaluation of health services utilized group discussion and key informant interviews to assess the achievement of project objectives. Group discussions were held with approximately 45 health center staff in Mulanje and Phalombe districts. Particular emphasis was given to those responsible for health services (in-charges), nurses, health surveillance assistants (HSAs), and a limited number of clients. Discussions also took place with hospital-based voluntary counseling and testing (VCT) counselors, pharmacy assistants, and lab technicians. In addition, eight members of the two DHMTs completed a survey questionnaire to complement the group discussions on the project. (See the Evaluation Instruments document for the interview guide for the extended DHMT and the DHMT individual survey questionnaire.) 17 Due to time constraints, distance, and rainy season road conditions, the evaluation team was not able to interview literacy teachers, shop owners, religious leaders, midwives, or traditional initiators. Final Report 6 March 14, 2003 16. Three hospitals that participated in the project were visited: Mulanje District Hospital, Holy Family Hospital (Phalombe), and Mulanje Mission Hospital. The three hospitals housed the only VCT sites in the project area. The hospital visits focused on the VCT service provision and interventions for prevention of mother to child transmission of HIV (PMTCT), in addition to reviewing general service provision. Seven health centers were also visited, including four in Mulanje and three in Phalombe. Four were government health centers, two operated under CHAM, and one belonged to an agricultural estate. The health center visits focused on general service provision and management issues. An effort was also made to sample stronger and weaker health centers, although this was determined subjectively by project/DHMT staff. All visits included observations and the review of service registers in addition to discussions with pertinent health center staff. (See the Evaluation Instruments document for the guide to health center visits.) 17. The CHAPS Project conducted a baseline and end-of-project health facilities assessment (HFA), which covered both service delivery and service management topics. Therefore, the evaluation did not perform direct observation of provider/client interactions but focused instead on analyzing with hospital and health center staff why interventions and systems did or did not work. As of March 14, 2003, the end-of-project assessment final results were not yet available. When available, the HFA results should be considered in conjunction with the evaluation service delivery findings. E. Partnership Evaluation 18. The partnership was evaluated on the basis of group discussions with the two the District Health Management Teams (DHMTs), the two District Health Officers, the Project HOPE CHAPS project manager, and responses to the DHMT survey questionnaire. Both sides of the partnership were queried on how the collaboration worked to achieve improved management and service delivery function. Specific characteristics of the partnership were discussed: decision making, problem solving, communication between the partners, resource management and allocation, and general strengths and challenges to working in partnership. The implementation process for the system strengthening activities such as training, supervision, logistics, and coordination was also discussed. F. Project HOPE Evaluation 19. Project HOPE headquarters and Malawi staff were interviewed, using a thematic interview guide, which covered an assessment of the project overall, the achievement of key intermediate results, project sustainability, project design, implementation delays, management performance, implementation performance, monitoring and evaluation, and appraisal of the partnership. In addition, CHAPS project technical staff were interviewed with respect to training, supervision, reporting; finance and administrative staff were queried on financial management and reporting. (See the Evaluation Instruments document for the interview guide for Project HOPE.) Final Report 7 March 14, 2003 G. Assessment of Project Outcome 20. The project outcome was assessed according to the achievement of key intermediate results, project sustainability, implementation performance, management performance, and project monitoring and evaluation. Assessment ratings were defined for each category, based on document review, statistical data, interview data, and site visit findings. III. History of the CHAPS Project A. Background 21. In September 1995, USAID signed the five-year, US$15 million Community Health Partnerships (CHAPS) project as a successor to the Promoting Health Interventions for Child Survival (PHICS) project. CHAPS transferred the focus of PHICS project activities from MOHP's central level to the district level. Partnerships between international private voluntary organizations (PVOs)/nongovernmental organizations (NGOs) and MOHP District Health Offices were formed in three districts as a means of delivering maternal and child health services. The PVOs/NGOs were contracted to help build the capacity of their partner District Health Management Teams (DHMTs) to assess constraints, prioritize health problems, identify systemic weaknesses in service delivery, develop and test alternative health financing approaches, and implement appropriate interventions.18 22. Initial resistance from MOHP's district-level staff resulted in CHAPS getting off to a slow start. In September 1997, the project was amended to combine its first two phases, increase district coverage to five, provide support for quality assurance to all districts, extend the implementation allowed to each partner PVO/NGO, and extend the project life to September 2002.19 23. Rationale. The rationale for the partnership model was based on the need to strengthen health services at the district level in combination with interventions to improve access to information and basic services at the community level. Private voluntary organizations (PVOs) traditionally worked at the community level and were relatively strong in training and community health interventions, while the district-level MOHP maintained health services structures but was constrained by weak management capacity. 18 USAID Regional Center for Southern African (RCSA). Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, op. cit., p. 10 19 Ibid., p. 10. The PVOs/NGOs were Africare in Mzimba District, International Eye Foundation in Chikwawa, Project HOPE in Mulanje, Save the Children/UK in Salima, and Save the Children/US in Mangochi. A sixth agreement with University Research Corporation provided support to all five districts for quality assurance activities. Final Report 8 March 14, 2003 Thus, the CHAPS partnership of PVO and health districts was designed to address both access and demand for services at the community level and the strengthening of services at the facility level. B. CHAPS Project 24. Goal and objectives. The goal of the CHAPS project was to improve health care services in the district of Mulanje through a partnership between Project HOPE and all levels of the health infrastructure in Mulanje district, involving also other private sector and NGO health providers.20 The partnership would be the vehicle for (a) improving maternal child health (MCH) services and reproductive health (RH) delivery, (b) building human and institutional capacity in a sustainable manner, and (c) assisting USAID in achieving its Strategic Objective 3 in Malawi:21 “Increased adoption of measures that reduce fertility and risk of HIV transmission and improve child health practices.” USAID Strategic Objective 3 was supported by the following key intermediate results: • Knowledge and use of good reproductive health practices improved, • Community participation in reproductive health care improved, and • Range and quality of reproductive health services for women, men and youth expanded. 25. The project objectives were to improve the health status of the population of Mulanje district, with special emphasis on MCH and RH program activities and to improve the management capacity of the District Health Management Team (DHMT) and other health personnel to support priority health interventions within the district in a sustainable manner.22 The main health problems in Mulanje district were malaria, HIV/AIDS, and malnutrition, followed by diarrhea with occasional cholera epidemics, and acute respiratory infections. Low literacy levels, high fertility rates, and weak health service delivery systems exacerbated these problems. 26. Due to its size and population, in November 2000, Mulanje district was divided administratively into two districts, Mulanje and Phalombe. 27. Interventions and activities. During the initial planning process, Project Hope worked with the DHMT to identify the priority interventions and activities for the project. Six intervention areas were selected: malaria, control of diarrheal diseases (CDD), acute respiratory infections (ARI), HIV/AIDS, sexually transmitted infections (STIs), family planning, and nutrition. In addition, the project agreed to focus on management skills, information, education and communication (IEC) programs, health management information systems, and improved access at the district level. 20 Project HOPE. Technical Proposal: Project HOPE's Community Health Partnership (CHAPS) with the District of Mulanje, RFA No. 690-97-002, November 1997, p. 5 21 Ibid., p. 5 22 Ibid., p. 11 Final Report 9 March 14, 2003 28. Activities at the district level included strengthening the management skills of district staff to assist them to provide better support and supervision to the health facilities, in order for the latter to provide better curative and preventive services, as well as community outreach activities. 29. Activities at the health facility level included: (a) improving the quality of care through needs-based training and supervision; (b) improving health center management; (c) increasing access to specific MCH and RH services by establishing new family planning and STI clinics, and outreach to special target groups; (d) increasing available resources for MCH and RH activities (e.g., IEC approaches, drug supply, equipment); and (e) increasing and improving community outreach activities. 30. Activities at the community level included: (a) increasing access to services by training community-based distribution agents (CBDAs), village health committees, and traditional birth attendants (TBAs) and improving their supervision; (b) establishing drug revolving funds (DRFs); (c) strengthening community management skills through training water committees, DRFs, and community leaders in Training for Transformation;23 (d) promoting literacy skills with health education messages; and (e) increasing community input at health centers. 31. Target areas and populations. The total population for Mulanje and Phalombe districts was estimated at more than 650,000 persons. Mulanje district is administratively divided into 6 traditional authority (TA) areas with 510 communities, and Phalombe district is divided into two traditional authority areas with 388 communities. 32. Duration. The CHAPS project proposal was approved on November 11, 1997; negotiations took place during the period, November 5, 1997 to January 31, 1998; the grant agreement was signed on January 31, 1998; and project start-up activities began in March 1998 (Table 1). The implementation period was from March 1998 to June 30, 2001. 33. Project cost and financing. The total, planned project cost was US$2,888,952. USAID funded 57 percent of project cost ($1,658,952). Project HOPE funded 43 percent of project cost ($1,230,000), which was distributed between cash (15 percent) and gift in kind (GIK) (85 percent). As of the planned closing date of June 30, 2001, the amount of unspent project monies was US$493,541. C. One-Year Cost Extension 34. In February 2001, Project HOPE requested a one-year cost extension in view of the fact that not all of the project objectives would be accomplished by June 30, 2001, given 23 Training for Transformation (TfT) is a participatory approach to community development that encourages community empowerment and is based on the belief espoused by Paolo Freire (1970) that communities themselves are powerful resources for problem-solving. Final Report 10 March 14, 2003 the number of interventions being implemented by the project, the limited human and financial resources, and counterpart staff turnover. In addition to addressing some training gaps, there was an urgent need to strengthen emerging management and technical skills through intensive follow-up and supervision, as well as help the district(s) to develop its own system for needs-based refresher trainings. Also, the increased impact of the HIV/AIDS epidemic on all aspects of life, as well as on the health care systems, necessitated an increased focus on the HIV/AIDS problem through education, voluntary counseling and testing (VCT), home-based care, and attention to mother to child transmission (MTCT) of HIV/AIDS.24 In particular, the project sought to introduce within the districts activities that focused on prevention of mother to child transmission (PMTCT) of HIV/AIDS, and increased access to voluntary counseling and testing (VCT) for HIV. 35. Duration. The one-year cost extension was prepared during the period July 14, 2000 to April 19, 2001; the extension was approved on June 11, 2001; negotiations took place during the period June 11, 2001 to July 24, 2001; the extension was signed on July 24, 2001; and start up activities began on August 7, 2001. The implementation period was from July 2001 to June 30, 2002. 36. Project cost. The total planned cost of the one-year cost extension was US$839,834, including carry-over funding of $493,541. The total project cost increased to US$3,729,786. D. No Cost Extension 37. In May 2002, Project HOPE requested a nine-month no cost extension in part due to the MOHP delays in approval of the prevention of mother to child transmission (PMTCT) of HIV/AIDS and voluntary counseling and testing (VCT) protocols and the unavailability of the rapid HIV test kits. The project needed additional time to complete planned PMTCT and VCT activities and to consolidate activities carried out during the previous four years, with increased emphasis on strengthening district capacity to manage an effective health program.25 38. Duration. The no cost extension was prepared during the period May 7-15, 2002; approved and signed on August 1, 2002; and start-up began immediately thereafter. The no cost extension implementation period was nine months, from June 30, 2002 to March 31, 2003. The project is scheduled to close on March 31, 2003. 39. Project cost. The total planned cost of the no cost extension was US$659,959, the amount carried forward from the grant budget remaining as of June 30, 2002. 24 Project HOPE. Community Health Partnership (CHAPS) Project, Mulanje and Phalombe Districts, Request for Extension of Grant No. 690-A-00-98-00093-00, February 2001, p. 1 25 Project HOPE. Community Health Partnership (CHAPS) Project, Mulanje and Phalombe Districts, Request for No-Cost Extension of Grant No. 690-A-00-98-00093-00, May 2002, pp. 1-2 Final Report 11 March 14, 2003 IV. Community Participation in Health Care 40. The project aimed to increase community participation in health care and improve access to and quality of health services. The objective was substantially achieved, based on evaluation findings in relation to community capacity building, community perspectives on health and health care, and observed behavior change as reported by community members and trained volunteers. This section describes accomplishments, discusses findings, assesses specific project objectives, identifies lessons learned, and makes recommendations. A. Accomplishments 41. The project built capacity at the community level through training and the provision of supplies and equipment to community volunteer groups. Over the life of the project, 1,559 persons (38 percent of the total number of training participants) and 190 youth (5 percent of total) participated in community volunteer training activities.26 Specifically, the project trained 360 persons in water well and spring protection, 120 Water Point Committee members, 270 persons in sanitary platform (sanplat) casting and latrine construction, 240 community health volunteers, 80 youth in HIV/AIDS prevention and leadership, 30 youth in Youth Friendly Health Services, 30 members of drug revolving funds (DRFs), 150 leaders and traditional initiators in HIV/AIDS prevention education, 40 community health volunteers on the prevention of mother to child transmission (PMTCT) of HIV/AIDS, 259 community-based distribution agents (CBDAs) in family planning, 100 persons in home-based care (HBC) of AIDS patients, 140 persons in drama for development on malaria, acute respiratory infections (ARI), control of diarrheal diseases (CDD), HIV/AIDS, and family planning, in addition to 75 religious leaders on PMTCT, 95 shop owners on malaria treatment and management, 45 literacy teachers, and 183 traditional birth attendants (TBAs). 42. In addition to training, the project conducted five health education campaigns, held two youth conferences and two drama/music competitions, and co-financed with communities, the construction of 8,000 pit latrines, 8,750 sanitary platforms (sanplats), and 87 protected water sources in a total of 210 communities. The project reached an estimated 75 percent of the 898 communities that comprise Mulanje and Phalombe districts. The project also built three Under Five outreach clinics in Mulanje, supported the completion of an outreach clinic in Mulanje that had been started by another donor, and built one Under Five outreach clinic in Phalombe. The outreach clinics brought health education closer to communities and increased access to Under Five services. 43. A further substantial accomplishment was that Project HOPE earned the communities' trust and established an outstanding reputation. Both trained volunteers and 26 The number of persons participating in training is not necessarily the same as the number of persons trained because some persons may have attended more than one training event. Final Report 12 March 14, 2003 community members were deeply appreciative of Project HOPE's presence and work in their communities. Winning community trust owed in large part to district health staff whose prior work and experience in target communities facilitated Project HOPE's entry. B. Capacity Building 44. Home-based care (HBC) groups. HBC groups were trained to care for and counsel AIDS patients in their communities because such care was needed and to help ameliorate the social stigma attached to persons with AIDS. Two HBC groups, totaling 33 individuals from 6 communities, were interviewed. Each group attended to 7 patients on average. Most HBCs visited their patients several times a week. In the HBCs' absence, family members cared for patients; HBCs taught patient care to family members. HBCs provided their patients with health and nutrition advice, spiritual guidance, counseling against suicide, bathing assistance, purchasing and preparing food, dressing sores, and cleaning the sick person's house and surroundings. They also provided medicines, when available, such as aspirin, Panadol, iron tablets, and oral rehydration salts (ORS), vitamins, as well as other supplies such as soap and matches. To protect themselves, HBCs reported that they used soap for washing and plastic gloves or bags when dressing open sores. Most HBCs did not have gloves and were told to use plastic bags instead, but these were in very short supply in the communities. HBCs stated that they had received supplies once; the project ordered a second round of HBC supplies in February 2003. 45. HBC training was well-liked. HBCs felt the need for additional training in the form of refresher courses because of the need to be more conversant with their work, know how to protect themselves, and the need for more counseling skills, among others. HBC training instructed the volunteers to undertake income-generating activities in order to support patient care; however, training did not include how to do this. One HBC group, which had received a large cash donation, had a garden and sold vegetables to buy patient supplies and gave produce to patients. Owing to the donation, the HBC group is the only one that is likely to sustain its activities over the long-term. Other HBCs reported doing piecework to earn monies to purchase food and supplies for their patients, and some took up collections among themselves to do the same. 46. Health surveillance assistants (HSAs) supervised HBCs monthly—sometimes weekly—which the HBCs found very helpful because they were encouraged and helped to improve their work. The HBCs would like additional supervision in order to learn new skills and develop their work further; they would also like for supervisors to visit their patients with them. 47. The HBCs reported one of their greatest accomplishments was that people now have love for the sick, which did not exist in the past. Another accomplishment was through the use of referral form letters, HBCs assisted their patients in getting admitted to the hospital, being discharged from the hospital into HBC care, and obtaining hospital transport for the deceased back to the community for burial. Final Report 13 March 14, 2003 48. HBCs were motivated by seeing the situation of a sick person and wanting to help, counseling about HIV/AIDS, which saves lives, and the fact that they are recognized by society, hospitals and health centers, and Project HOPE. 49. Water Point Committees. Water Point Committees were trained in good practices that promote water safety, the dangers of unsafe water, the protection and maintenance of protected water sources, latrine construction, and community coordination, among others. Two Water Point Committees (totaling 25 persons from 2 communities) were interviewed. The committees were responsible for the operations and maintenance of protected water sources (springs and water tanks), which were constructed by community members with technical assistance and funding from the project and community contributions. Both committees also helped to build latrines, which were financed by the project and community contributions. The committees reported that their training was very good, although the training period was too short, and stated the need for additional training. Water Point Committees were advised to establish a fund for maintenance and repair costs; however, neither committee had been able to collect adequate funds for that purpose, which is a problem faced by water committees in many countries. The committees were supervised by HSAs and observed that without supervision, they would feel that their work is useless. 50. The Water Point Committees' greatest accomplishments were the construction of the water tanks and latrines and community members' practice of good hygiene at home. Water Point Committee members were motivated by wanting to help so that diseases can be reduced and because people really wanted to drink good water. 51. Drama group. One drama group (9 persons from 4 communities) was interviewed, which performs four times a month in different communities, with an estimated attendance of about 300-500 persons per performance, in addition to holding performances at Mulanje District Hospital. The group distributed condoms at every performance. It also assisted the elderly and orphans. The drama group reported that the topics of most interest to audiences were HIV/AIDS and STIs. The drama group was trained in drama for development and, specifically, HIV/AIDS, STIs, malaria, acute respiratory infections (ARIs), and family planning. It wished to emphasize the "goodness of training," and stated the need for additional training as a motivation. The drama group supported its activities through income earned from occasional piecework and received modest assistance from the hospital, e.g., loan of the loudspeaker. The group reported that HSAs were present at every performance to provide feedback, and the District Youth Officer (DYO) supervised the group on a monthly basis. HSA and DYO supervision was much appreciated. 52. The drama group's motivation derived from the fact that many people commend their work, and they have seen people's behavior change, e.g., hand washing. Their greatest accomplishments were they were happy because they had saved lives, and they walked long distances to spread their messages. Final Report 14 March 14, 2003 53. Drug revolving funds (DRFs). Two drug revolving fund (DRF) groups (4 persons from 2 communities) were interviewed. DRFs were established to provide communities with access to low-cost drugs. The project provided DRFs with an initial drug supply, and drugs were to be priced for sale at slightly above cost in order to generate funds for continual resupply. DRF members were trained to dispense certain drugs and medications (e.g., Fansidar, aspirin, ORS), give instructions to the sick on the dosage of drugs dispensed, make follow-up visits, record any drugs sold or damaged in a record book, and report on the progress of the DRF. DRF members also inspected latrines for proper use and maintenance, helped to maintain protected water sources, and reported health problems to HSAs, among others. Based on their responses to interview questions, DRF members were very knowledgeable about their work and committed to it. The DRFs liked their training but said it was too short to absorb all of the content. HSAs supervised the DRFs on a monthly basis and reviewed drug stocks, counted monies received for purchases, and discussed problems. DRFs reported that HSA supervision was very helpful. 54. DRFs have a serious problem with drug replenishment, which jeopardizes the sustainability of providing communities with low-cost drugs on a revolving fund basis. HSAs collect DRF monies and give them to the hospital. The hospital sends the money to the government's general treasury. Next, the hospital would reclaim the money in order to resupply the DRFs, but this system has not worked because it is time-consuming, and the bureaucratic burden is too great. As a consequence, DRFs could not replenish their drug supplies. The CHAPS project encouraged DRFs to purchase drugs from local suppliers, but the purchase price of drugs was high, which meant that the sales price was also high, and the revolving funds were depleted. In February 2003, the project was negotiating with Pharmanova, a pharmaceutical supplier in Blantyre, to resupply DRF drugs at a low cost; however, the means of resupply had not been worked out because Pharmanova reportedly agreed to bring the needed drugs to Mulanje, but it did not want to deal with the DRFs on an individual basis. 55. Community-based distribution agents (CBDAs). Three groups of CBDAs were interviewed (16 persons from 7 communities). Each CBDA had on average 20 women clients and 15 men and youth clients. The most frequently distributed methods were birth control pills (Lofemino and Ovulet) to women and condoms to men and youth. The CBDAs reported that they liked training but needed refresher training to update their knowledge and skills and as a motivation. One group of CBDAs was supervised by an HSA would could not visit them in their communities as is usually done because he did not have a bicycle; the CBDAs went to the health center for supervision. The second CBDA group was supervised by the HSA on a weekly basis. An HSA supervised the third group on a monthly basis and sometimes more frequently. HSA supervision was said to be very helpful in terms of assistance with report-writing, problem-solving, and motivation. CBDA supplies were replenished on a monthly basis, and no problems with resupply were reported; however, condom supplies were often exhausted prior to the monthly replenishment. Final Report 15 March 14, 2003 56. CBDA accomplishments were that people now have access to family planning, which was not previously available, and women and children are healthier and happier as a result of family planning. CBDAs were motivated by the fact that they want to develop people's lives through family planning, which will also help to develop their communities. A further motivation was that hospitals provide written responses to CBDA referral letters for clients who seek hospital services such as Depo Provera injections. The CBDAs said that the hospital's response to their letters made them feel very proud. 57. Traditional birth attendants (TBAs). Two traditional birth attendants were interviewed who attended 7-9 and 6 births per month, respectively. TBAs were trained in how to care for pregnant women, identification of risk factors, management of common diseases associated with pregnancy, management of a woman in labor, post partum care, management of common early childhood diseases, and basic information about primary health care and health education. They were also trained in the importance and benefits of exclusive breastfeeding and were expected to impart this information to mothers attending their facilities for antenatal and delivery services. TBAs were given basic kits and one bicycle and one bicycle ambulance each to assist with the referral of emergency cases. The TBAs reported that their training was very good because they gained new knowledge and skills, which helped them with client care, especially women who experienced problem pregnancies and deliveries, and they would like to have additional training. The district community health nurse and HSAs supervised the TBAs who found supervision to be useful and motivating. TBA training made the communities in which the TBAs worked a safer place for mothers in terms of delivery. 58. Youth clubs. Two youth clubs were interviewed (totaling 28 in- and out-of-school youth). The youth reported that the most important health problems facing youth are they do not know about the developmental changes taking place in their bodies, girls who seek treatment for STIs at the hospital are laughed at by hospital staff who do not treat their problems confidentially, youth are not given a chance to explain their health problems at the hospital, and teenage pregnancy. In addition, one youth observed that some youth do not attend school due to lack of funds, which means they lack knowledge, which is a health hazard. 59. The clubs' activities included drama and song performances, campaigns against drinking and smoking hemp, community outreach, condom distribution, advocating against child abuse and other issues, sports activities, peer education, and building homes for the elderly. The topics reported to be of most interest to youth were HIV/AIDS, STIs, how to prevent teenage pregnancy, and adolescent growth and development. The approach that worked best in reaching youth with health education messages was drama and sports. The District Youth Officer (DYO) supervised the youth groups. One group was also supervised by a female, youth community-based distribution agent (CBDA) who was especially well-liked because she was very helpful, and the youth felt comfortable with her. They expressed the need for additional supervision in order to improve their work, Final Report 16 March 14, 2003 learn new ideas, and through the more constant presence of an adult, allay parents' fears about letting their children go to the youth center because many parents were dubious about the nature of youth club activities. 60. Both youth groups had participated in a three-day CHAPS Project training event on Youth Friendly Health Services and Youth Peer Educators, which was held January 22-24, 2003. The goal of the training was to strengthen the delivery of comprehensive and high quality youth friendly health services.27 The youth liked the training, particularly the information provided on adolescent growth and development. However, they stated that the duration of training—three days—was too short, the lessons were too long and complicated, and they were told that they would be trained as counselors but had not learned counseling skills, which was an apparent misunderstanding. During training, the youth clubs were promised stationery, a television/VCR, and videotapes, which were delivered in late February. 61. The youth were enthusiastic about youth friendly health services, but they did not know what to do next, although each youth group had prepared an action plan during training. The last minute nature of this activity relative to the scheduled closing of the project in March 2003 and the fact that the training itself and participants were organized without adequate notice and preparation calls into question whether undertaking the training was advisable. Training raised the youth's expectations. In order to meet their expectations, adequate follow-up, supervision, and the actual provision of health friendly services, whether at local hospitals or the District Youth Center building, where one of the clubs was housed, will be needed. C. Community Perspectives 62. Four groups of community members (totaling 58 persons) were interviewed with respect to community health problems, health services, health education, trained community volunteers, and knowledge of the six project interventions. 63. Health problems. Communities reported that their most important health problems were cholera, HIV/AIDS, and diarrhea; other problems were skin rashes, eye infections, malaria, and acute respiratory infections (ARIs). Communities noted various improvements in community health over the past few years. For example, in one community, the creation of a drug revolving fund, which introduced and supplied oral rehydration salts (ORS) was deemed a significant improvement because community members frequently suffered from diarrhea and needed ORS. In another community, 20 sanplat latrines were constructed, which improved sanitation, although the number of sanplats was not sufficient to cover all households. One community reported that the health surveillance assistant procured medicines whenever there is an outbreak. All four 27 Alice Bvumbwe. Cross Border HIV/AIDS Program, Youth Friendly Health Services Training for Health Workers and Youth Peer Educators (draft), Mulanje View Motel, January 22-24, 2003, p. 2 Final Report 17 March 14, 2003 communities stated that HBCs care for AIDS patients for whom little or no care was provided in the past. 64. Health services. In general, community access to health facilities (health centers and hospitals) was limited due to distance, lack of transport, and lack of funds for transport and to pay for services and drugs at private facilities. Holy Family Hospital in Phalombe was singled out for special mention because people were treated nicely, and the staff was cheerful. All four communities stated that the main problem with health services concerned drugs—the lack of drugs and insufficient drug supplies at public health facilities, the rising cost of drugs at private pay facilities, and the perception that the wrong drugs were prescribed. Community members preferred to go to private facilities because of the availability of drugs. There is a significant, unexplained discrepancy between communities reporting drug scarcities at health facilities on the one hand, and health facilities reporting adequate drug supplies on the other hand. 65. Health education. All four communities had participated in health education activities, whether in the form of health education campaigns, drama performances, Under Five clinics, or HSA and community volunteer health talks. Communities viewed health education activities as very important and useful because they learned how to detect and prevent diseases and felt more secure about what to do when faced with health problems. Drama performances were deemed the most effective; community members said they discussed the health messages delivered at performances for a long time afterward. With respect to information, education, and communications (IEC) materials, family planning and HIV/AIDS prevention posters were liked the best because the messages were clear and intelligible. With the exception of one community where there were health posters in the tearooms, all other posters were seen at health centers or hospitals. 66. Trained community volunteers. Four communities had home-based care groups, Water Point Committees, and youth clubs. Two of the four communities also had TBAs, CBDAs, and drama groups. One community had volunteer growth monitors, a drug revolving fund, and women's literacy volunteers. Communities greatly appreciated the work of trained volunteers and believed their work helped to improve community health. The overwhelming laudatory remarks made about trained community volunteers contrasted sharply with the criticism and lack of understanding the volunteers themselves said they often received. 67. Knowledge of the six project interventions. Community groups were asked what they had learned about the six project intervention topics, malaria, diarrhea, HIV/AIDS and home-based care, sexually transmitted infections (STIs), nutrition, and acute respiratory infections (ARIs), and whether they had put their knowledge to use. Generally speaking, communities knew the signs, symptoms, prevention, and prescribed treatment for malaria, diarrhea, and STIs and used this knowledge. They were also knowledgeable about HIV/AIDS transmission and prevention, but in the context of the group interviews, how this knowledge was used was not discussed due in part to sensitivities and in part to the Final Report 18 March 14, 2003 interviewers' lack of probing. All four communities knew about and spoke highly of HBC activities. The knowledge and practice of good breastfeeding was uneven across the four communities. Some community members had learned that exclusive breastfeeding for the first 6 months is important, but when asked about practice, some individuals reported exclusive breastfeeding for 4 months, 6 months, and 3 to 4 years. Similarly, some community members could describe weaning foods but not when or how weaning foods should be introduced. One community had not heard of weaning foods and asked, "If you wean the child quickly, what will they be given to eat?" Some mothers introduced weaning foods, and others did not due to lack of money to purchase foods. Three community groups knew that rapid and shallow breathing were signs of acute respiratory infections and that children experiencing these symptoms should be kept warm, but only one community reported that children with ARI symptoms should be taken for medical treatment. D. Project Objectives and Observed Behavior Change 68. This section assesses the achievement of specific project objectives in relation to observed behavior change. The six adult trained volunteer groups and the four community groups were asked if they had observed any change in people's behavior over the past few years in general and with respect to the six intervention areas. The youth groups interviewed were asked if they have observed any change in youth behavior in general. The behavior changes that received the most responses are described below and, where possible, compared to the initial household survey baseline and end-of-project household survey data.28 69. Malaria. The objective to increase community knowledge with respect to timely care seeking, acceptance of Fansidar for the treatment of malaria, and compliance with the full treatment was substantially met, based on the fact that all trained volunteer and community groups reported that people use Fansidar for the treatment of malaria, comply with the full treatment, and seek timely treatment. 70. Diarrheal diseases. The objective to increase access to safe water and sanitation in the targeted communities was substantially met through latrine construction and water source protection. The objective to improve the home management of diarrhea was substantially met, primarily through the knowledge and use of ORS, based on the fact that all adult groups reported the use of oral rehydration salts (ORS) as a treatment for diarrhea and that a person with diarrhea was taken to a health facility if there was no improvement in his/her condition. More behavior change was reported for diarrheal diseases than any topic, e.g., hand washing, clean homes and surroundings, use of protected water sources, use of latrines, improved hygiene, boiling drinking water, and washing food before eating. This 28 For the initial baseline data, see Irene Benech. Community Baseline Survey, Community Health Partnership Programme, Mulanje and Phalombe Districts, June-July 1998. At the time of final report preparation, the end-of-project household survey final report was not available; however, the tabulated questionnaire data was available, which was used here. Final Report 19 March 14, 2003 qualitative data supports the end-of-project household survey results relating to the control of diarrheal diseases, which found, among others, that access to pit latrines increased from the baseline of 79.9 percent to 98.4 percent; access to safe water increased from 59.3 percent to 79.8 percent (although this increase is not necessarily attributable to the project); and the correct preparation of ORS increased from 3.5 percent to 56.5 percent (Table 6). 71. HIV/AIDS and home-based care. The determination of whether there was a decrease in the spread of AIDS by improving appropriate/safe sexual behavior in the population was beyond the purview of the evaluation. However, all trained volunteers and community groups except TBAs reported the use of condoms as a preventive measure against HIV infection. All adult groups reported a reduction in casual sex and promiscuity. This data complements the end-of-project household survey results in relation to behavior change to prevent getting or transmitting AIDS. The baseline survey found that 66.3 percent of men had changed their behavior since hearing of AIDS and how it is transmitted, compared to 96.3 percent of men in the end-of-project survey. The baseline survey reported that 85.4 percent of women had changed their behavior since hearing of AIDS and how it is transmitted, compared to 95 percent in the end-of-project survey. 72. With respect to home-based care, five groups reported the acceptance of AIDS patients by families and communities. Increased acceptance of AIDS patients in communities was one of the desired effects at the end of the project. The comparison of baseline and end-of-project survey results showed that the number of respondents who had heard of home-based care increased from 21.8 percent to 38.3 percent, home-based care provided to survey respondent households increased from 3.3 percent to 16.4 percent, and respondents' thinking that it would be useful for a family or community member to be trained in home-based care increased from 85.7 percent to 92.7 percent. 73. Sexually transmitted infections (STIs). The qualitative interview data suggested that early care seeking behavior for STIS had increased. Five trained volunteer and community groups reported a decrease in STIs due to a decrease in casual sex, embarrassment at having to take one's partner for testing, and condom use. While not comparable, this data complements certain baseline and end-of-project survey findings. In the baseline survey, 69.2 percent of female respondents and 74.1 percent of male respondents reported they had informed their partners that they suffered from signs or symptoms suggesting an STI, compared to 76.8 percent of female and 68.5 percent of male respondents in the end-of-project survey. In the baseline survey, 37.2 percent of female respondents and 48.6 percent of male respondents reported they had taken some action to prevent infecting their partners, compared to 69.4 percent of female and 85 percent of male respondents reported in the end-of-project survey. 74. Nutrition. The objective to increase knowledge and improve the nutrition of young children and pregnant/lactating women was negligibly achieved. While some knowledge was reported regarding the nutritional needs of young children and pregnant/lactating women, this knowledge was not reported to have been put into practice. Five trained Final Report 20 March 14, 2003 volunteer and community groups reported that mothers knew they should breastfeed exclusively for six months, but many women do not because they believe they lack adequate nutrition (para. 104). 75. Acute respiratory infections (ARIs). The objective to increase knowledge of the danger signs of ARIs or appropriate care seeking behavior of caretakers of children with ARIs was negligibly achieved. Five trained volunteer and community groups reported having little or no knowledge of ARIs. This finding confirms to some extent the end-of￾project survey result that showed a small increase from 50.8 percent to 58.5 percent of caretakers who know the danger signs for ARI (Table 6). In addition, it is noted that the CHAPS project implemented ARI health education activities in areas that were not visited by the evaluation team.29 76. Youth. The project did not define an objective for youth per se, but it did stipulate several activities for youth. One of these was training youth leaders. It was not possible to determine whether observed behavior change in youth could be attributed to training. However, youth groups reported a decrease in teenage pregnancy, a decrease in beer drinking and hemp smoking, and the use of condoms. 77. The reported behavior changes were attributed to trained volunteer activities, Under Five Clinic health education, drama group health education messages, health education campaigns, health surveillance assistants (HSAs), advertisements, and having seen people suffer and die from AIDS. All of these helped to increase community participation in health care and improve community access to knowledge and services. The quality of health services also improved due to the high quality work of the trained community volunteers. E. Lessons Learned and Recommendations 78. Design. (a) The design of community level activities was generally good with the exception of targeting. Although targeting was based on health data, e.g., geographical areas that experienced cholera outbreaks were targeted for water and sanitation interventions, targeted communities were few in number and widely scattered across the catchment area of both districts, which diluted the impact of interventions. In the future, a smaller, more concentrated geographical area should be considered for community level interventions. For example, the Support to AIDS and Family Health (STAFH) Project targeted agricultural estate communities with all 29 The target communities for ARI interventions were in the Mlbiza area and Mloza area in Mulanje and Chitekesa area in Phalombe. Final Report 21 March 14, 2003 interventions, creating a critical mass of health activities, which had a significant impact on each community (b) The design of drug revolving funds (DRFs) was not effective because of the problem of resupply. This problem was not unique to the CHAPS project; it was also reported for DRFs supported by German Technical Assistance (GTZ) in Mulanje.30 DRFs will not be sustainable until the resupply problem is solved. Mulanje and Phalombe districts and Project HOPE should lobby MOHP to review policy with a view toward finding an efficient procurement procedure for DRF drug supply. (c) The design of home-based care (HBC) was good; however, training did not include how to undertake income-generating activities that would allow HBCs to sustain their activities over time. Prior to future HBC training, a needs assessment should be conducted to determine what kinds of income-generating activities that would support HBC activities might be feasible; the results should be incorporated into training. (d) The design of CBDA activities was good and has passed the test of time. However, CBDAs, as is the case with all community volunteers, spend a substantial amount of time doing volunteer work and, for the most part, do not participate in much￾needed income-generating activities. Project HOPE should take into account the expressed need of community volunteers for income-generating activities and explore how assistance might best be offered. For example, community volunteers should be apprised of the Project HOPE Village Health Banking project as a possible means of learning how to undertake income-generating activities. (e) The design of youth group activities was good, with the exception of the Youth Friendly Health Services initiative. The project was only able to reach a small number of the many youth clubs in Mulanje and Phalombe districts. In addition, the fact that youth grow up and leave youth clubs was not sufficiently taken into account. The District HIV/AIDS plan reportedly targeted youth as a priority group but did not provide resources for them. Youth training should be conducted regularly. The District HIV/AIDS plan should make the allocation of resources to youth a priority. 30 CHAPS Project. Community Health Partnership, Mulanje and Phalombe Districts, 4th Quarterly Report, October-December 1999 and Annual Report, 1999, February 2000, p. 3 Final Report 22 March 14, 2003 (f) Sanplat casting, latrine construction, and protection of water sources helped to reduce diarrheal diseases, and the participation of community members in these activities introduced skills and knowledge that will remain in the community. The protection of shallow well water sources was not an adequate technical solution because buckets used to draw contaminated water sources, which the monthly use of chlorine could not control. Conflicts reportedly arose when one community was provided with protected water and the neighboring community was not. The project should have used hand pumps for water well operation because they offer better protection of water sources. Before construction, there should be a meeting of neighboring communities to reach agreement on sharing water resources. 79. Implementation. (g) Some health problems can be reduced or eliminated through the work of trained volunteers who deliver community-based health services; whereas in the past, the only access to health services was at health facilities, which are very distant from many communities. Trained volunteers perceived themselves to be part of the district health care and delivery system, which was a source of great pride. It was not clear whether hospital and health center staff shared the same perception. Health districts should continue to provide training, supervision, and follow-up to community volunteers and, additionally, recognize that community volunteers are an important component of district health care and delivery systems. (h) All trained volunteer groups were knowledgeable about their activities and performed very well, despite the resupply problems that several groups experienced. Project HOPE should make every effort to assist Mulanje and Phalombe districts in the resupply of volunteer groups. For example, Project HOPE could provide the districts with a list of supplies for each group and a schedule for resupply. (i) The most effective means of health education was drama, and drama groups were willing to continue their activities without funding. More health education should be delivered through drama groups who should continue to be trained, and trained groups should receive additional training. (k) Once people were sensitized to the use of condoms, the response was overwhelming; however, supplies were inadequate to meet the demand. Final Report 23 March 14, 2003 Mulanje and Phalombe districts should create more condom distribution points. 80. Training. (l) Training provided community volunteers with the skills and knowledge to deliver community-based health services. Training was high quality and well-liked. However, the majority of community volunteers, including youth groups, reported that training time was too short, and trainees were not able to absorb all of the lessons. Also, one-time training was not enough and refresher courses are needed. Community volunteer training curricula should be reviewed to determine whether adjustments should be made in content and duration. Community volunteers should receive regularly scheduled refresher courses in order to acquire new knowledge and learn new skills and as an incentive to continue working. (m) Youth are highly enthusiastic and willing to learn. In order to maintain their enthusiasm and interest, careful attention should be given to ensuring that adequate follow-up, supervision, and the implementation of planned activities for youth take place in a timely manner. 81. Supervision. (n) All community volunteers, including youth groups, viewed supervision as very important because it helped in the preparation of work plans, problem-solving, and decision-making, in addition to boosting morale and providing volunteers with the motivation to continue working. HSA supervision was very effective and more effective than the health center/hospital-to-community supervision. The excellent supervision provided by HSAs should be recognized, HSAs should continue regular supervision, and other key actors should do more community supervision. 82. Coordination. (o) The coordination of activities among key players in Mulanje and Phalombe districts, including nongovernmental organizations (NGOs), was weak in that community volunteer groups and community members were at a loss to explain why, for example, one NGO would provide volunteer badges, and Project HOPE did not, why one NGO would provide hand pumps and another would not, and why two NGOs worked with the same target group. In addition, the coordination of key players is essential to the sustainability of health activities at the community level. Final Report 24 March 14, 2003 There should be more frequent coordination between government and NGOs who are permanent in the area to avoid creating confusion in communities and the duplication of resources, as well as working together to make the delivery of community level health services sustainable. V. Health Services 83. The project aimed to improve district health services through extensive training of service delivery staff, provision of key equipment and supplies that were lacking from the MOHP, and a small amount of infrastructure improvement such as the improvement of counseling and lab facilities for VCT. All of the planned activities for health service strengthening were completed, and many of them had a positive impact on services. A. Accomplishments 84. A significant proportion of project activities targeted health centers. All 27 government and mission health centers and the three hospitals in both districts, including 9 agricultural estate clinics, were targeted with activities to strengthen health services. These included both technical and management training, provision of key equipment and supplies, and orientation towards supportive supervision as contrasted to the traditional disciplinary model of supervision. Health services interventions were assessed with respect to the their impact on health service staff knowledge and practice and in terms of health service management. B. Training 85. In order to improve access to quality health services throughout both districts, the project trained staff from all health centers in all six interventions (Table 2). This involved a large amount of training; MOHP provides training curricula and specifies the number of training days required for most of the project interventions. Table 2: Number of Health Center Staff trained by Intervention and Type of Training Interventions Initial Training Refresher Training General HSA training 100 210 Family Planning 9 full FP / 68 core FP 68 Exclusive breastfeeding/ Prevention of Mother to Child Transmission (PMTCT) (Integrated baby friendly) 151 Control of diarrheal disease 64 69 Integrated management of childhood illness (IMCI) 43 facilitators / 32 health workers Final Report 25 March 14, 2003 Interventions Initial Training Refresher Training Sexually transmitted illness (STI)– Syndromic Management 61 15 Support staff - AIDS 100 Health center workers – exclusive breastfeeding 80 Health center workers – PMTCT 54 Health center workers – home-based care 60 Health center workers – VCT/infant feeding 50 Midwives on clean delivery 60 Infection Prevention 80 health workers / 60 support staff Full VCT counselors 24 86. Training had several positive results. Health staff expressed more confidence in their skills, particularly for infection prevention, and management of malaria and STIs. Even more significantly, for interventions such as infection prevention, exclusive breastfeeding, and PMTCT, interventions that had little or no emphasis at the health center level in the past were now covered in health education and considered important by the staff. 87. For some interventions such as breastfeeding and infection prevention, several staff including support staff at each health center were trained. When this was the case, implementation changes were more likely to be implemented. The development of a “critical mass” of interested staff as well as the inclusion of support staff who tend to have more time for implementation probably contributed. Another reinforcement for training was the simultaneous provision of equipment and supplies for implementation. This was particularly evident for infection prevention where buckets for instruments and dirty dressings were provided, incinerators constructed (5), and chlorine was supplied. 88. The training strategy had two major constraints. First, there was too much training to effectively manage and assure its impact. Trainings were not integrated across topic areas (with the exception of IMCI), and often targeted the same small number of health center providers, using the same small number of trainers at the district level. As a result, the time invested in training significantly detracted from the other responsibilities of both trainers, who were responsible for district health management, and providers who were often the only patient care providers at their health centers. Similarly, the emphasis on training limited the time and energy that health center and district level staff had to implement and follow-up on changes introduced in training. While in many cases Project Hope and district MOHP staff jointly undertook follow-up supervision after training, it was often inadequate to assure the ongoing implementation of changes. Follow-up was particularly limited when training was conducted toward the end of the project due to the lack of time to ensure adequate implementation (e.g., youth friendly services, VCT counseling training) Finally, due to limitations in Project Hope’s financial systems, it Final Report 26 March 14, 2003 seemed to be difficult to ensure that the funding and organization for each training were in place soon enough to allow for careful selection of participants and lead time for them to attend. As a result, the most appropriate people were not necessarily able to attend the training. 89. Staff turnover limited the impact of training. At one point every health center may have had a family planning provider or someone trained in STI management, but at the time of the evaluation, that was no longer the case. For example, when health center providers were transferred, the ability to provide comprehensive family planning services or syndromic management of STIs went with them. When more than one staff person or cadre was trained (where appropriate), the intervention was more likely to be maintained because the competency was less affected by transfers, and they could reinforce each other in its implementation. Some health centers have a practice of having training participants brief others on their training. In some cases this spreads the impact of that training at the health center level, particularly since staff are in a position to cover so often for each other. However, the lack of training allowance for the other staff inhibits their involvement in implementation because they feel they should have an allowance as an incentive to implement the new approach. 90. One manager made the observation that with newer interventions (such as IMCI and PMTCT) trainers, like others, may not have had direct experience with implementation since it is new and they are no longer working clinically. This limits their ability to directly address the implementation challenges and constraints that will be faced. Giving trainers the opportunity to experiment in a practical setting with implementation, even if it is part of training, could be helpful. 91. Lessons learned. The following lessons were learned with respect to health services training: (a) Training a critical mass of staff, including both junior and senior staff, led to improved implementation of training content. (e.g. VCT/MTCT and infection prevention) (b) Senior health center staff (nurses and medical assistants) spent too much time in training relative to implementation. (c) Due to overlap (lack of integration) and prescribed duration, some training may not have maximized the time spent, particularly for refreshers. (d) With new interventions and experienced training staff, trainers may not have direct experience with the interventions they are teaching. This compromises their ability to identify the real challenges to implementation. Final Report 27 March 14, 2003 92. Recommendations. The following recommendations are made with respect to health services training: (a) Train larger numbers of health center staff including both support staff (where appropriate), who are less likely to be transferred, and senior staff who are decision-makers. Support staff who will be particularly involved with implementation (i.e. ground laborers for infection prevention) should be directly trained rather than depending on briefing from their superiors. Such training also provides recognition and motivation. (b) Make an effort to cover training on different topics in one integrated training session, particular for refresher training. This would need to be based on careful needs assessments looking at differing needs and desired behaviors according to level of function. (c) Link trainees’ debriefing of other health center staff with training follow-up supervision in order to facilitate broader learning and implementation. (d) Encourage training staff to work beside their trainees to experience the constraints in implementing new interventions. C. Quality Improvement 93. The CHAPS project made a significant investment in a participatory quality improvement process early in the project. Technical assistance was provided to all five of the CHAPS districts to develop and train all health center staff in working as a team on a problem solving approach to quality improvement. The process had eight steps taking the team through a process of identifying, prioritizing, defining, and collecting information on problems with quality. They were then expected to identify and implement solutions to address these problems. 94. While a few staff who had been through the process acknowledged they appreciated learning the problem solving approach, no health center was currently doing anything consciously for quality improvement, and everyone who had been involved indicated it had been too complicated and difficult to practice. In addition, several health centers had identified problems completely outside their control so had become frustrated and dropped the effort. 95. Recommendation. Quality assurance is an important process to cultivate at the level of the health facility, but to be effective, it must be tailored to local needs and resources. D. Health Service Practices Final Report 28 March 14, 2003 96. The baseline and end-of-project health facilities assessments (HFAs) examined diagnosis and treatment practices with clients. The end-of-project HFA results were not available during the evaluation; however, the health services evaluation used observation and a limited number of questions to assess a few indicator practices. 97. In general, as indicated above, training made a difference in the knowledge and awareness of some of the key interventions such as infection prevention, VCT/PMTCT, family planning, and management of STIs. Training made a significant difference. With respect to infection prevention, most people knew what they were supposed to be doing and were making an effort to do so because the level of importance had been raised. In some cases, cleaners were assigned tasks, and those interviewed appreciated knowing how to protect themselves. Everyone appreciated the availability of chlorine and equipment to practice what they had learned. However, actual implementation was quite varied among the facilities visited. While there were two of the hospitals and a couple of the health centers where practices seemed to be completely implemented, there were still some facilities, including the district Hospital, where waste was not being burned, even when there was a new incinerator, and where needles were being mixed with other equipment rather than separated in a disposal box or separate plate for sterilization. 98. Integrated Management of Childhood Illnesses (IMCI). With respect to IMCI, while most providers indicated they had really appreciated the training, their responses to several questions did not indicate they were practicing some key elements. First, few indicated they follow the five questions for classification when they first see the client, although this may be difficult to assess completely since experienced providers probably automatically do some kind of screening. Second, only two of the providers interviewed indicated they would treat a six-week-old baby with fever differently than an older child. This was a relatively easy way to check whether they referred to the protocols and whether they had internalized some of the differential diagnosis concepts. Most could produce the protocols, but they did not appear to be used regularly. These observations, when combined with the preliminary HFA results, indicate substantial reinforcement is needed to achieve the differential diagnosis approach for IMCI illnesses. 99. It was difficult to assess practice for syndromic management of STIs because the cases are infrequent. However, providers indicated familiarity with the treatment regimens for different symptoms, and they seemed to be appropriately using the national reporting sheet, which includes the treatment regimens as part of the reporting system. 100. Voluntary Testing and Counseling (VCT) and Prevention of Mother to Child Transmission (PMTCT) of HIV/AIDS. Significant effort went into strengthening the three hospitals in integrated exclusive breastfeeding, PMTCT, and VCT interventions. In addition to training, this included assuring the availability of testing through provision of testing reagents and equipment, providing the necessary renovations for private counseling rooms and lab facilities, and provision of some basic equipment for infant feeding demonstrations. All three hospitals had separate counseling rooms, counselors who had Final Report 29 March 14, 2003 been trained for six weeks in counseling, and functioning labs with technicians who were clear on the testing protocols and able to provide results within a short time, although this depended somewhat on the work load. 101. During the nine months this service had been functioning, a total of 1,046 clients were counseled, with 995 accepting testing. Of this number, 965 got their results. The majority of the defaulters were at the Mulanje District Hospital, where the work load for the lab sometimes means there can be a delay in getting results. Slightly less than half of persons tested were HIV positive. 102. The VCT counselors were largely nurses, usually drawn from the maternity and delivery services because of the efforts to integrate testing with PMTCT. They were well trained and felt comfortable with their role. They indicated they were able to talk with clients about their diagnosis, and felt competent in addressing the health, social and even spiritual aspects. In one case, the counselors themselves were even able to do the first tests. However, particularly in centers where only nurses were trained, it was difficult for them to find the time to counsel clients in addition to their other duties. This worked better in one hospital where lay counselors had also been trained, but were supported by the nurses when needed. 103. With respect to the integrated package of PMTCT / breastfeeding / testing, staff reflected a lot of enthusiasm for the interventions and the integration of the three made intuitive sense. Promotion of VCT is being included as part of antenatal health education, some practices such as vaginal swabbing have been incorporated during deliveries, and everyone reports immediate breastfeeding. No HIV positive mother has yet opted for replacement feeding, probably due to a combination of the stigma associated and the lack of affordability. 104. Even at the community level, survey results indicate that mothers’ awareness that they should only start providing supplemental fluids or food after the child is six months old went from 6.6 percent to 46.8 percent (Table 6). 105. In contrast to the above findings, all but one of the providers interviewed at the health center level still do not feel comfortable talking to their patients about HIV. They will refer clients to the hospital for consultation and even testing, but they do not tell the clients the real reason for the referral. 106. Lessons Learned: (a) IMCI is an approach that requires intensive supervision and follow-up. (b) Nurses are too busy and few to be available as needed for VCT counseling. (c) The integrated approach to BFHI/MTCT and VCT improved staff enthusiasm and confidence. 107. Recommendations: Final Report 30 March 14, 2003 (a) Infection prevention is a great priority for a project such as this, and there is still work to be done. (b) Other cadres of workers in the hospital should be trained to support VCT/MTCT counseling. VI. Health System Strengthening 108. The CHAPS model introduced interventions in health system strengthening to complement the six technical interventions. These included training in management components such as logistics management and health information management, establishment of regular work planning and monitoring, and emphasis on regular and supportive supervision. While Project Hope Malawi had previously been involved with extensive program implementation particularly at the community and health center levels, the focus on management and system strengthening was new. Similarly, the districts had not previously partnered with NGOs to strengthen their management systems. As a result, the new relationship presented challenges both in scope of influence and establishment of trust. A. Accomplishments 109. Significant accomplishments in health system strengthening included: • Supervision was strengthened through the development and revision of a supervisory checklist, planning and provision of resources for regular supervision visits, e.g. fuel for vehicles, incorporation of follow-up supervision in training plans, and provision of bicycles for HSAs to purchase on a subsidized basis, which greatly assisted HSA supervisory activities at the community level. • One Mulanje DHMT staff member and staff from the other four participating CHAPS districts attended a one-week financial management course for non￾financial managers at Malawi Institute of Management (MIM), which was organized and funded by USAID. • A computerized management system for the central pharmacy in Mulanje was installed and people were trained in its use. However, at the time of the evaluation, the computer was not functioning and there was a new pharmacist who had not been trained. • 4 health provider staff houses were built to help with staff recruitment and retention • DHMT staff were trained in computer use for monitoring and reporting • 30 staff were trained in the new HMIS system • Establishment of quarterly work planning and annual review meetings involving review of HMIS data, monitoring progress, and development of work plans. 110. As part of project management strengthening, one member of the DHMT, a District Health Officer (DHO) was sent to Arusha for six weeks of training in health services Final Report 31 March 14, 2003 management at the beginning of the project. This was a big investment with the intention of strengthening the management skills of senior district staff to maximize the benefit of the training throughout the life of the project. Whether or not the training was worthwhile, it was an investment that had little pay-off because the recipient left the district shortly after his return. Subsequent investments in management strengthening were specific to particular aspects of management and are covered in the sections below. 111. Lessons Learned. (a) Concentration of training in senior management at both the district and health center level made the loss greater when these staff were transferred. (b) Management training was concentrated at the beginning of the project such that the benefit was lost with transferred staff. 112. Recommendation. The recommendation is made to explore ways to complement more formal management training with an “on the job” approach of regular seminars addressing one management system at a time. B. Supervision 113. The project aimed to improve supervision through the development of supervision checklists and the support of supervision visits in the districts in order to strengthen supervision in general as well as to monitor progress on the six CHAPS interventions. The HFA data indicated and health center staff corroborated that supervision visits increased in both duration and frequency since the beginning of the project, although this may be somewhat confounded in the HFA because people may not have considered the supervision for training follow-up as “supervision” in their responses. Regular contact between health center staff and their district supervisors, even in the absence of actual visits, was also facilitated by the availability of the radios. 114. Recognizing that large team supervision of the health centers was resource intensive and unrealistic, the goal of the supervision checklist was to enable a few staff to supervise all sectors and refer problems to the appropriate Program Coordinator. The supervision checklist was developed by Program Coordinators and members of the DHMT. The first one was too long, so it was revised. There is currently a checklist for quarterly supervision, and a more comprehensive one meant to be used annually. Persons interviewed stated that the list was too long and complicated. However, they also raised the alternative perspective that perhaps inadequate time is allocated for each supervision visit. At least in Mulanje district, the supervision plans have now been adjusted to allow for longer supervision visits at each health center so perhaps the supervision check lists will now be more helpful. Competing priorities resulting in the lack of availability of district staff for supervision continue to be a significant constraint, whether or not the transport is subsidized. Final Report 32 March 14, 2003 115. Supervision and re-supply, when supplies were available, of community volunteers by HSAs was also emphasized by the project. HSAs participated as co-facilitators in the training of their volunteers, and were provided specific orientation on supervision elements they should check when providing follow-up. While only found in one health center during the evaluation, where volunteers had subsequently been hired by health centers as support staff, the link between the health center and the community for both problem solving and re-supply was stronger. Conversely, in CHAM health centers, the link between the health center and the community tended to be weaker because the HSAs are government employees and do not answer to the Health Center In Charge. Rather, they answer to an arbitrarily assigned health assistant either from the district or a neighboring government health center. This person also serves as their source of supplies. 116. HSAs were also assured access to bicycles, either through the immunization program or through provision of subsidized CHAPS bicycles that they could purchase. As a result, all but one HSA encountered during the evaluation had access to a bicycle. The HSA at Mimosa Health Center (which had experienced a fire) did not have a bicycle. In addition, those that were purchased by the HSAs themselves, even if they are already four years old, were still functional. While not measured directly, this is in positive contrast to many HAS bicycles that quickly break down and are not maintained because HSAs have no vested interest in their function. 117. Lessons Learned: (a) The emphasis on supervision led to increased supervision in both frequency and duration, at the health center level, for general supervision as well as specific follow-up to training. (b) HSA supervision of community activities was effective and appreciated. Bicycle ownership provided reliable HSA transport, which made an important difference in community supervision and follow-up. C. Staffing 118. Staff turnover was probably the largest constraint that the project faced. The construction of four staff houses and sending people for training were an attempt to offset this problem, but the reality was that much of the benefit of the project was diluted or lost through loss of staff who had been trained or involved in project implementation. This was particularly the case for management training, which occurred at the beginning of the project. It was noted that some of the Program Coordinators had gone for further studies or moved for personal reasons, so not all attrition was under the control of the MOHP. The staffing shortage was exacerbated by the national shortage of nurses, as well as the obligation of management staff to also provide clinical services. While the human resource challenge is acute at all levels of the MOHP, several suggestions are included in the recommendations below to mitigate its impact. Final Report 33 March 14, 2003 119. Lesson Learned. Staff turnover was a huge constraint and needs to be considered in the project design and implementation plan. 120. Recommendations: (a) Advocate with central MOHP to maintain consistent staff for at least one year at the beginning of the project to establish momentum and direction in the district. (b) For training that needs to occur multiple times in order to cover everyone, it should be spread throughout the project duration so that newly arriving staff due to turnover can also have access to the project training.. (c) Even though there is often no overlap between outgoing and incoming DHMT and Program Coordinators, formal turnover – through written notes and/or return visits should be feasible and encouraged. D. Logistics: Drugs, Supplies, and Equipment 121. The project provided key supplies that were otherwise scarce or not available in the districts. These included chlorine and equipment for infection prevention, VCT reagents, volunteer supplies for the home-based care volunteers and TBAs, medicines for the drug revolving funds, and some basic medicines that were available through the gift in kind (GIK) match. Additional equipment and support for the VCT program was provided during the one-year cost extension, specifically, refrigerators and microscopes for two laboratories and the remodeling of a counseling room, in addition to extensive training. While some GIK medicines provided in the early years of the project were useful, the second GIK shipment in particular had limited use, according to District Health Officers who reported receiving excessive amounts of Keri lotion, and some drugs such as specialized anesthesia that were only appropriate for regional referral hospitals. The provision and management of GIK was an issue that warranted more attention (para. 186). 122. During the last six months of the project, most of the essential drugs (STI drugs, Depo Provera, contraceptive pills, oral rehydration solution, Cotrimoxazole, and condoms) were available at all levels in the district. This is attributable to increased emphasis on monitoring stock, increased availability of drugs in Central Medical Store (CMS), and increased supervision. Shortages of Depo Provera during the past month were due to a shortage at CMS, which was beyond the control of the project. Otherwise, there were occasional shortages of one or more of the STI drugs. This seemed to be due to a combination of late reporting, and inefficient lines of reporting and distribution through the STI officer at the district level. 123. The project aimed to establish a computerized monitoring system for the district pharmacy in Mulanje. Software was installed on the pharmacy computer,31 and staff were trained in its use. Unfortunately, the computer was damaged by water and, at the time of the evaluation, had not worked for more than one year. This problem was not addressed, and, as a consequence, there was no positive impact from these interventions on the 31 The pharmacy computer was donated by the Japan International Cooperation Agency (JICA) in 1998. Final Report 34 March 14, 2003 pharmacy’s logistics management. At the time of the evaluation, the pharmacist was overloaded and some STI drugs were not being restocked in a timely way. 124. While the project included computer training as part of management training for district management staff, it did not budget for the provision of computers. As a result, Program Coordinators felt they have not had the access they needed to computers, and the computers provided were leftover from other projects. This was particularly a problem in Phalombe district, where such infrastructure support for a second district was not budgeted. 125. Lesson Learned. The provision of equipment, which complemented training (i.e. infection prevention supplies, volunteer supplies) improved implementation. 126. Recommendations: (a) The provision and management of GIK should be reviewed. (b) District program and administrative managers need computer access and skills. (c) Work with the pharmacists and STI coordinators to problem-solve the problem of STI drug stock outs at the health center level. E. Communications and Referral System 127. The project assisted radio communications in both districts, providing or repairing four radios in Mulanje and two in Phalombe. All Mulanje health centers now have direct communication with the district. In Phalombe, the remaining health centers were to receive radios from the Safe Motherhood program. Radio communication significantly improved referral time, particularly when coupled with a clear prioritization of women with maternity complications. There was not, however, any intervention on triage or relaying radio messages to health providers at the district hospital. As a result, there was no mechanism to prioritize acutely ill children for ambulance use, and without a system for notifying providers that pregnancy complications were in transit it is likely that the avoidable delays that occur upon arrival to the hospital were not affected by the interventions. 128. The project provided some money for vehicle repairs and made the CHAPS vehicle available to the district when needed. The CHAPS vehicle was primarily used for supervision, which permitted greater use of district vehicles for patient transport. The project did not get very involved with fleet management. When the project was first designed this was not a significant problem and the district had enough vehicles. As time went on this became more of a problem, but the project did not reprioritize the interventions. 129. The project purchased 43 bicycle ambulances—one for each trained TBA. Bicycle ambulances were meant to facilitate TBA patient referrals to the nearest health center. Final Report 35 March 14, 2003 Insufficient information was available to determine the effectiveness of bicycle ambulances; they appeared to have worked well in some areas but not in others. 130. Lesson Learned. Even though the project supported vehicle maintenance, the lack of involvement in the overall management systems for the fleet led to limited impact on the availability of vehicles for transport. 131. Recommendations: (a) Fleet management is an important part of district management and should be addressed. (b) Set up a system for transport officers to confer with providers on radio referrals when the priority of the referral is unclear. F. Work Planning 132. The project established quarterly meetings at the district level, which reviewed data, prepared quarterly work plans, and discussed progress against those plans. DHMT staff recognized and appreciated the CHAPS support for both these and the annual meetings, indicating they allowed people both to focus their activities and recognize their accomplishments. They also indicated that planning was a skill that would be likely to be maintained after the end of the project. 133. While these meetings seem to have worked pretty well at the Mulanje district level, they neither met the unique needs of Phalombe, nor did they particularly allow for health center participation. The transition process for Phalombe to become independent was prolonged, with Mulanje responsible for many of the management functions over a period of time. As such, it was a challenge to clarify how Phalombe should be treated during the work planning and budgeting process. The Phalombe DHMT did not feel equally recognized and felt the work plans tended to more reflect the direction of Mulanje. There was a tendency to select common priorities for both districts and to determine the training schedules based on Mulanje’s needs. 134. Health centers, too, felt left out of the planning process. They felt they were told what CHAPS activities they would be doing and when, without being involved in the plan. They felt particularly left out of determining the selection of targeted communities in their catchment areas. Moving even further down, HSAs also develop monthly work plans. However, while they turn them into their supervisors at the district level, the plans are rarely recognized or discussed at any level other than among the HSAs themselves. 135. Lesson Learned. The integration of work planning and data review in quarterly meetings strengthened project activities and ownership. 136. Recommendations: Final Report 36 March 14, 2003 (a) Make an effort to involve health center staff (through opportunities for feedback and activity selection) in work planning aspects that affect work in their catchment areas. (b) Encourage monthly health center meetings as a management tool for reviewing progress based on data and planning work. G. Health Management Information System 137. Particularly during the last two years, the project invested in the implementation of the new MOHP health management information system (HMIS). The project funded several people from each health facility to attend national HMIS training and also funded significant supervision follow-up for each health center. The evaluation found that most of the health center in charges recognized how HMIS reports could help them monitor their activities and progress, and they no longer had to wait for feedback from the district. 138. There are however, still a lot of problems with the system. Persons interviewed indicated that reporting is still difficult and that they have not had adequate, supervised time to practice using the system. It is also, at least currently, very time consuming— taking between one and four days each month to complete all the calculations. Supervisors reported that completion of the HMIS supervision checklist took four hours, which made it difficult for them to adequately supervise all the health centers in the district. Finally, it is unfortunate that the national system seems to have been implemented before the bugs were worked out. As a result, a significant amount of data and time were unnecessarily lost. 139. The CHAPS project did not budget for an HMIS staff position. While making a valiant effort to provide support to the district as well as to maintain basic project monitoring information, the current HMIS officer divided her time between the CHAPS and the Cross-Border AIDS projects, which resulted in insufficient time devoted to the CHAPS project. If the HMIS had been given a high priority at the beginning of the project, two HMIS positions could have been justified—one for technical support to the district and one for CHAPS project monitoring and evaluation. 140. Lesson Learned. The introduction of a new HMIS was tremendously complicated. 141. Recommendation. Look for ways to strengthen the generation and use of HMIS data by focusing on smaller sections of the system at one time and providing adequate staff time for support. H. Financial Management 142. Other than direct financial management of the CHAPS budget, the project did not work with the two districts on strengthening internal financial management since it had not been prioritized as a problem. Project financial management is discussed under the section on partnership. Final Report 37 March 14, 2003 I. District Coordination of Health Activities 143. The CHAPS project aimed to strengthen the district’s role in the coordination of health activities throughout the district. The project supported bi-annual meetings of the District Health Board (DHB), which was composed of Project Hope and representatives of other NGOs working in health in the district, district MOHP representatives, and representatives from other health-related activities such as AIDS. The DHB met once in 1998, when the CHAPS project was starting up, twice each year during 1999-2002, and planned to meet in 2003, when the results of the final evaluation are available. 144. The DHB provided a forum for all organizations working in health to share their activities and target areas with the aim of coordinating activities to maximize the resources available for health in the district. These meetings alerted people to potential areas of overlap and/or collaboration with the expectation that they would then meet together to coordinate their activities. They also recognized the estate health services as part of overall district health services. 145. Two NGO representatives who served on the DHB reported that coordination of activities was important, and the DHB forum was valuable. However, some persons expressed the concern that Project Hope was too prominent in maintaining this board and that the board was too big to be effective. The role of the District Health Officer (DHO) would ideally be to take responsibility for the board as a means to help him coordinate the health activities in his district. However, without this commitment, Project Hope at times was “propping” up the board. In addition, the large number of participants made it difficult to convene meetings or to discuss activities in adequate depth. The Project HOPE CHAPS project manager indicated that over the life of the project, the effectiveness of the DHB varied widely, depending on the level of interest and commitment of the particular DHO in office at the time. People may not have really used this forum as an opportunity to identify the needs for collaboration even though it was there. Collaboration was also more difficult because some NGOs were working with local government delineations as target areas while others, such as CHAPS and the DHO, were working with health center catchment areas as target areas. 146. The project engaged in a variety of strategies to activate the Mulanje District AIDS Coordinating Committee (DACC), but none seemed to have worked. Specifically, the project helped the Mulanje DACC to develop Terms of Reference, but at the time of the evaluation, internal issues immobilized the DACC. The Phalombe DACC was reported to be working well and coordinating HIV/AIDS activities in the district. The project also provided training to Mulanje and Phalombe DACC Technical Sub-committees on Youth, Home-based Care, and High Risk and the Sub-Committee on HIV/AIDS Orphan Care. 147. Lesson Learned. The District Health Board is potentially an effective forum for coordinating district health activities, but it needs to be owned and managed by the DHO. Final Report 38 March 14, 2003 148. Recommendation. Maintain coordination of health and AIDS activities as a priority, continuing to explore ways to achieve it through focusing on actions and assuring participation by the people who will likely contribute the most. VII. DHMT and Project HOPE Partnership 149. The stated objective of the PVO-DHMT partnership was to "develop a collaborative working relationship with all partners and to assure that district structures can sustain program planning, management, and service delivery activities after the end of CHAPS."32 To achieve this objective, three major outputs were planned for the PVO￾DHMT partnership: (a) bi-annual meetings of the District Health Board (DHB), (b) quarterly meetings between DHMT and Project HOPE to review progress and objectives, and (c) training of DHMT in management. 150. According to persons interviewed and DHMT survey questionnaire respondents, the partnership was partially achieved. People appreciated the emphasis on work planning and the regular review meetings to review progress. However, there were gaps in management training, such as access to and use of computers, financial management, and fleet management that were still not adequately addressed. The more subjective elements of mutual trust and confidence were severely hampered by Project Hope’s centralized financial management, which made it difficult for the Program Manager to have the information he needed to be transparent with his partners. 151. The project design provided that Project Hope and the Mulanje District Health Office to work together in partnership to improve the health status, access to services and quality of services in the district. With the division of Mulanje district into two, the Phalombe District Health Office also became a partner. Two Memoranda of Understanding (MOU) were signed over the course of the project. The first, signed in June 2000, set forth principles and regulations for payment of subsistence allowances, which was a subject that generated considerable controversy.33 The second, signed in June 2002, covered the roles of Project HOPE and the DHMT, as well as training allowances and allowances for Project HOPE employees.34 As with any partnership, the partners all brought their respective strengths and challenges to the partnership. 32 Project HOPE, Technical Proposal, op. cit., p. 12 33 See Memorandum of Understanding, Allowances Schedule – Community Health Partnership Programme, Mulanje and Phalombe Districts, June 2000. The signatories were the Project HOPE CHAPS project manager, the District CHAPS Coordinator, and the District Health Officer for both Mulanje and Phalombe districts. 34 See Memorandum of Understanding on Operational Procedures – Community Health Partnership Programme, Mulanje and Phalombe Districts, June 2002. The signatories were the Mulanje DHO, the Phalombe DHO, the District CHAPS Coordinator, and the Project HOPE CHAPS project manager. Final Report 39 March 14, 2003 152. Work plans and priorities were developed during annual and quarterly meetings with participants from both districts and Project Hope. These could be changed as different needs or constraints arose. During these meetings, Project Hope told people how much money was available for the year, providing the basis for deciding how many training and other activities could be planned. The basis of the total budget amount was not clear, i.e., whether is reflected the CHAPS project technical proposal planned budget or otherwise. 153. In general, both sides worked hard to make the partnership function. Everyone contributed significantly to accomplishing the training, to planning and implementing district work plans, and to regular communication and coordination of activities. The CHAPS district coordinators were helpful since it gave the project a point person in each district through whom all the activities could be coordinated. As a result of their involvement, it was clear these people felt more ownership for the project than others on the DHMT. Informal relationships between district and Project Hope staff were also clearly very positive. 154. On the other hand, the DHMTs' perceived lack of financial transparency on the part of Project HOPE was an obstacle that the partnership was not able to successfully overcome. The DHMTs were not privy to the overall CHAPS project budget and, therefore, were not confident that the amount of funds available for CHAPS activities was the correct amount. The Project HOPE CHAPS project manager did not have up-to-date budget or complete budget and expenditure information, which limited his effectiveness in discussions about same, hence, the perceived lack of transparency, which to a large extent affected the overall roles and responsibilities of the different partners (para 202). Because they were not part of the financial decisions, the districts tended to see Project Hope as the donor and themselves as the implementers. They saw themselves as requesting Project Hope’s assistance when problems arose, and were not clear how the decision was made about whether assistance would be provided. They tended to feel as though Project Hope drove both the agenda and the schedule of activities. As a result, this partnership did not have the level of ownership and enthusiasm seen in other districts. In spite of these concerns, MOHP staff did acknowledge that most of the activities were useful and relevant, and they felt that Project Hope was genuinely trying to assist them with meeting the wide variety of needs they faced. 155. Project HOPE CHAPS project training staff and their DHMT counterparts jointly prepared and signed off on training activity budgets, which worked well. 156. Project HOPE's experience with the Malawi Social Action Fund (MASAF) in the project to remodel the Nutrition Rehabilitation Center in Mulanje illustrates the good will that can be generated by financial transparency. The Project HOPE CHAPS project manager was a member of the remodeling project management committee, which was responsible for project finances, among other activities. The project manager and another committee member were signatories on the project bank account. Joint signatures, coupled with committee oversight, were an effective, transparent mechanism for financial Final Report 40 March 14, 2003 management and accountability. In the future, Project HOPE may wish to consider the use of such a mechanism. 157. District staff turnover had a negative effect on the partnership. The Mulanje District Health Officer (DHO) changed five times over the life of the project, the program coordinators for the six project interventions also changed, and the district CHAPS coordinator changed six times. There was no hand-over of the project from one DHO to the next. In February 2003, the Project HOPE CHAPS project manager was the only person participating in the final evaluation process who had worked on developing the project proposal. A few district staff had worked in their respective districts over the past five years, but they had not worked on the project proposal and were not in management positions at that time. Personnel turnover mitigated the districts' sense of project ownership because current district staff did not know which choices or priorities to attribute to Project Hope, or which had truly come through a participatory process that involved MOHP staff. 158. Another significant factor that affected the role of all three partners was the slow process in establishing Phalombe as an independent district. People working in Phalombe felt like Mulanje got more of the project benefits, that they determined the project priorities and activities, and that Phalombe was supposed to follow. They described themselves as “the little sister.” According to Project Hope, this was because the Phalombe district had not been planned for during the original proposal development, and the process of separation was so prolonged that it was unclear when or how they should be treated separately. It was only during the past two years that the district had its own office and pharmacy. In some ways, the Phalombe situation was particularly unfortunate because CHAPS could have provided the opportunity to truly strengthen Phalombe district from the beginning. Being a smaller district with a relatively (in hindsight) more stable staff, the project could have really made a difference in the establishment of strong management systems in the district. 159. The lack of a Project HOPE CHAPS project coordinator for during the latter part of the project was a limiting factor for the partnership (para 194). When the Project HOPE CHAPS project manager was not available, coordination and decision-making were difficult for the districts. 160. Working in separate offices made communication and coordination more difficult. For Phalombe it was particularly difficult since they were so far away and had to make extra effort to communicate, submit requests for support, or get to meetings. The fact that they were also less established as a district made this even more challenging. For both, it meant that communication tended to occur more when there were problems that needed solving rather than for the less formal, day-to-day operational coordination. 161. On the part of the MOHP, they acknowledged that the extremely high turnover rate and the general over-commitment of the district staff meant they sometimes did not Final Report 41 March 14, 2003 coordinate things as well as needed either. With staff responsible for covering clinical services in addition to management duties and other program obligations, CHAPS activities sometimes fell between the cracks. 162. Lessons Learned. (a) Financial transparency is essential for the establishment of trust and ownership. (b) Coordinators who are relatively available and devoted to the CHAPS project are necessary to facilitate coordination and communication. (c) The original project should have been redesigned to take into account the management inputs needed by Phalombe as a new district, including situation of a HOPE staff person in Phalombe. 163. Recommendations. (a) Project financial reports on the full project including budget allocations and expenditures against budget should be available and shared regularly between partners. (b) Project management should consider renegotiating project design as significant project circumstances change. (c) Make an effort to situate the PVO coordinator in the hospital to facilitate communication. VIII. Major Factors that Affected the Achievement of Project Objectives A. Factors not generally subject to Government Control 164. During the period 1999-2002, adverse weather (drought and floods) contributed to crop failure that resulted in food shortages, which had a negative impact on the nutritional status of people in communities. Illnesses related to malnutrition increased, and many people reportedly got themselves admitted to hospitals in order to be fed. Poor nutrition and hunger had a partial, negative effect on the anticipated impact of health education activities because communities did not have resources to provide for themselves and were focused on hunger; they wanted food and were not interested in health talks. 165. One of the major government opposition figures is from Mulanje, and his presence created some political turbulence, which had a partial, negative effect on the achievement of project objectives. People's focuses were elsewhere than on project activities, and the NGO community as a whole was suspected to be engaged in political activities with the result that people in communities thought that they should not be associated with NGOs. B. Factors generally subject to Government Control 166. The liberalization of trade policies under the structural adjustment of the economy resulted, among others, in the import of pharmaceuticals, which were previously under Final Report 42 March 14, 2003 exclusive government control and identified as government commodities. Inadequate control of imported pharmaceuticals opened the door to their theft and mismanagement, which had a partial, negative effect on community health because of reported instances of drug theft at some health centers. This situation ran counter to what community members were taught through health education activities, i.e., persons were encouraged to seek care at health centers, but some health centers could not provide adequate care due to the lack of drugs. 167. MOHP is responsible for staff transfers, and the district health staff turnover rate was very high for Mulanje district in particular, which had a substantial, negative effect on the achievement of project objectives because of lack of continuity, loss of institutional memory, lack of a sense of project ownership, and the need for the continual orientation of new staff to the project. By the end of the project's third year, all the staff that had been trained up to that point were no longer working in Mulanje district. The private voluntary organizations (PVOs) in all five CHAPS districts made a joint request to MOHP to not change district health staff; MOHP did not respond to this request. 168. The government approved the National Decentralization Policy in October 1998, whose ongoing implementation coincided with the life of the project. The decentralization of procedures included district management of resources (monthly allocations). Mulanje and Phalombe districts did not receive their full resource allocations, and allocations received were used at times to pay vendors instead of supporting project activities, which had a partial, negative impact on the latter. Some supervisory visits to communities did not take place because district vehicles were out of operation due to lack of maintenance and fuel; reduced supervision had a partial, negative effect on the achievement of project objectives. In addition, under decentralization, District Assemblies are responsible for local government, but they do not as yet have sufficient technical capacity to meet their new responsibilities, and neither do District Health Officers (DHOs). 169. At the time of the one-year cost extension (2000-2001), the government's approach to HIV/AIDS changed significantly. In particular, voluntary counseling and testing (VCT) and prevention of mother to child transmission (PMTCT) of HIV/AIDS were strongly emphasized and promoted. VCT and PMTCT of HIV/AIDS were incorporated into the project. It was anticipated that government protocols for both would be forthcoming. VCT protocols were finalized in mid-2002, but were not made available to CHAPS partners for use because of outstanding issues, e.g., who would do counseling, where would counseling take place, and who would do the actual testing. In January 2003, draft protocols were disseminated. As of February 2003, no protocols existed for PMTCT. The lack of protocols had a substantial, negative effect on the achievement of project objectives with respect to VCT and PMTCT. 170. A large number of PVOs/NGOs work on health and health-related activities in Mulanje and Phalombe districts, including Project HOPE, GTZ, Oxfam, Salvation Army, World Vision International, Red Cross, Banja la Mtsogolo, and others. Many Final Report 43 March 14, 2003 PVOs/NGOs provided training to DHMT staff, and due to extensive training activities, DHMT staff did not have time to implement what they had learned in training, which had a partial, negative effect on the achievement of project objectives. C. Factors generally subject to Project HOPE Control 171. The outstanding reputation that Project HOPE enjoys in the target communities and elsewhere, and the fact that its work and presence are highly valued had a substantial, positive effect on the achievement of project objectives. In particular, the professionalism and competency of the Project HOPE staff who worked on the CHAPS Project had a substantial, positive effect on the achievement of project objectives. 172. HOPE Center technical support for the CHAPS Project was limited during the latter part of the project, which had a partial, negative effect on the achievement of project objectives. Project HOPE Malawi technical supervision and backstopping did not occur to the extent programmed, i.e., 20 percent of the Country Director's time over the life of the project, which had a partial, negative effect on the achievement of project objectives, especially in view of the fact that the CHAPS project manager had significant responsibilities, in addition to the CHAPS Project (para. 193). The project would have benefited from greater technical support in the areas of partnership building, monitoring and evaluation, project management, and financial management. In the project manager's absence, additional, local technical supervision and backstopping should have been provided. 173. Changes in the management of project monies had a positive impact on the achievement of project objectives. Until 2001, HOPE Center transferred grant funds to a HOPE Malawi bank account held in U.S. dollars in Blantyre, where the country office is located. Funds from the dollar account were transferred to a bank account held in Malawian Kwacha that was also located in Blantyre. Project expenses were paid out of the bank account in Kwacha. Delays experienced in the transfer of funds from HOPE Center to the dollar account to the Kwacha account in Blantyre disrupted planned project activities. In 2001, a bank account held in Kwacha was opened in Mulanje, which permitted more rapid access to funds for project activities, obviating their postponement. 174. Funding delays related to the timeliness and adequacy of monthly financial reporting had a partial, negative effect on the achievement of project objectives because some planned training activities had to be postponed from one month to the next due to the temporary lack of funds. Project HOPE should address the issue of funding delays at the earliest opportunity and, if need be, strengthen financial management capacity. 175. Some problems with the timely provision and management of gift in kind (GIK) supplies had a partial, negative effect on the achievement of project objectives. Project HOPE should aim toward more efficient and effective GIK provision and management. Final Report 44 March 14, 2003 IX. Implementation Delays 176. Implementation delays caused by funding delays had a partial, negative effect on project implementation because some planned activities had to be postponed to a later date, which created a snowball effect. 177. Government delays in issuing voluntary counseling and testing (VCT) and prevention of mother to child transmission (PMTCT) of HIV/AIDS protocols and guidelines significantly delayed the implementation of project activities in these two areas. 178. The continuation of the project under the nine-month no cost extension was interrupted for a period of more than six weeks because the no cost proposal was apparently misplaced by USAID, which delayed the signature approval of the USAID Regional Center for Southern Africa (RCSA). X. Monitoring and Evaluation A. Project HOPE Center 179. Project HOPE Center provided technical monitoring through annual project site visits and email and telephone communications by the Maternal and Child Health Program Director and the Regional Director for Africa. The MCH Program Director's portfolio covers all Project HOPE MCH-related projects. The MCH Program Director technical monitoring of CHAPS carried through the year 2000, and was followed by limited monitoring on the part of a technical specialist in reproductive health. The Regional Director for Africa changed three times over the life of the project, which made consistent monitoring difficult. On reflection, Project HOPE Center should have devoted additional human resources to technical monitoring. B. Project HOPE Malawi 180. Closer technical monitoring and backstopping on the part of Project HOPE Malawi were indicated. Specifically, there was a need to provide management support to the CHAPS project in the form of management training for the CHAPS project manager and to give technical assistance in the areas of financial management and project monitoring. Project HOPE hired a technical support manager in July 2002; however, due to the relative newness of this position, the CHAPS project did not fully profit from the technical support manager's assistance. C. CHAPS Project 181. The original, three-year CHAPS project aimed to achieve the three major objectives by means of 67 outputs (products and services), which would be measured by 56 effects Final Report 45 March 14, 2003 (changes by project end), covering multiple target groups, including district and health center staff, HBCs, DRFs, Water Point Committees, CBDAs, drama groups, youth clubs, literacy teachers, traditional initiators, religious leaders, community leaders, community health committees, midwives, and TBAs. The one-year cost extension had 118 activities and 92 key performance indicators. Infection prevention, PMTCT of HIV/AIDS, and VCT were added to the project under the one-year cost extension, which resulted in the inclusion of new target groups such as the District AIDS Coordinating Committee (DACC) and new activities. The nine-month no cost extension had 61 activities and 74 key performance indicators. 182. The number of activities and indicators was excessive and presented a significant obstacle to efficient and effective project monitoring. In hindsight, Project HOPE Center and Project HOPE Malawi, as part of technical monitoring, should have reviewed the number of activities and indicators with a view toward defining a coherent, manageable set that would satisfy monitoring requirements. 183. The achievement of project outputs was to be assessed quarterly in an abbreviated fashion and annually in a process involving all participating agencies.35 Reports were to include a brief discussion of project objectives during the reporting period, significant project developments during the reporting period, a description of problems encountered with proposed solutions, and for the annual report, a statement of projected accomplishments for the coming year.36 Past, current, and budget year information presented in table form by project input costs was also to be included.37 184. Quarterly and annual reports were produced over the life of the project (see Bibliography), in addition to training and supervision reports and monthly, quarterly, and annual work plans. Quarterly and annual reports adequately covered activities planned for the quarter, accomplishments, and planned activities for the coming quarter. Quarterly reports noted some implementation problems, e.g., PVO/NGO coordination difficulties, but did not provide clear definition of actions required to overcome such problems or adequately report on project progress. Quarterly reports did not compare planned versus actual outputs or, where possible, progress toward achieving end-of-project key performance indicators. Annual reports did not present past, current, and budget year information. In October 2002, USAID provided a quarterly reporting format that was to be used by all five CHAPS PVOs for the submission of quarterly reports, which included, among others, a budget of expenditure, quarterly tracking of statistics, and a table format for performance data covering 13 key indicators. PVOs were the data source for 5 35 Project HOPE. Technical Proposal, op. cit., p. 22 36 Project HOPE. Community Health Partnership (CHAPS) Project, Mulanje and Phalombe Districts, Request for Extension of Grant, op. cit., p. 29. Also see Project HOPE. Community Health Partnership (CHAPS) Project, Mulanje and Phalombe Districts, Request for No-Cost Extension of Grant, op. cit., p. 19. 37 Ibid. Final Report 46 March 14, 2003 indicators.38 This reporting format was not adopted. Quarterly reports during the latter half of the project were submitted late, and their quality was uneven. 185. The CHAPS project monitoring system was weak, which is not uncommon in the PVO community and elsewhere. The project's health information system (HIS) officer worked half time on the CHAPS project, and the major part of the officer's time was spent on the new MOHP health management information system (HMIS) and maintenance of a parallel system at the CHAPS project office, which involved getting reports from all health facilities, data entry, and report production. HMIS data collection and analysis did not appear to cover CHAPS project monitoring or management information requirements. CHAPS project management would have benefited from a management information system that was capable of producing information that would assist in timely decision￾making, as well as reporting. XI. Provision of Supplies and Equipment 186. The provision of supplies and equipment—largely gift in kind (GIK)—was satisfactory. The project provided much-needed supplies and equipment, which district health facilities would not otherwise have had. Over the life of the project, Project HOPE provided US$1.5 million in supplies and equipment. Project HOPE Malawi provided local supplies and equipment in the amount of US$177,280, for example, volunteer kits for the different volunteer groups. As noted, some problems were experienced with GIK. In hindsight, Project HOPE could have better managed GIK to benefit the CHAPS project. XII. Studies included in the Project 187. Based on their timeliness and the manner in which they proceeded, the initial community baseline survey and initial health facilities assessment were satisfactory.39 The results of the baseline survey were used to determine priority activities and set benchmarks for training activities. The mid-term evaluation was not as useful as it might have been because it did not effectively address the priority areas necessary to direct project activities for what was at the time the third and final year of the project.40 The impact of provider counseling on breastfeeding practices, undertaken with support from the LINKAGES project, helped in the design of PMTCT interventions. 38 The indicators were: (1) number of community drug revolving funds, (2) number of community AIDS committees, (3) number of CBDAs, (4) number of functional VCT centers, and (5) number of facilities with functional Quality Assurance or Infection Prevention committees. 39 See (1) Irene Benech. Community Baseline Survey, Community Health partnership Programme, Mulanje and Phalombe Districts, June-July 1998; (2) Irene Benech. Quality Assurance Assessment, Community Health Partnership, Mulanje / Phalombe District, April-June 1998. 40 See Chrissie P.N. Kaponda, Ph.D. Community Health Partnership Programme, Mid-Term Evaluation, A Joint Programme for Project HOPE Malawi and Mulanje District Hospital sponsored by USAID, April-May 2000. Final Report 47 March 14, 2003 188. Some research activities planned under the one-year cost extension and nine-month no cost extension were not conducted, e.g., assessment of barriers to partner referral, assessment of quality and compliance with shopkeepers' advice, impact of health education campaigns (drama and music competitions, health education), importance of participating in youth clubs to risk-reduction practices, development and testing through operations research of novel supervision approaches that can be sustained by the district (e.g., peer supervision). The planned customer surveys to assess the quality of and compliance with shopkeepers' advice on malaria prevention and treatment were not conducted because this subject was adequately covered by the mid-term evaluation. Other planned studies did not take place because there simply was not enough time, given other project activities. In the future, Project HOPE may wish to disseminate reports and studies more widely to benefit all interested parties, especially health care providers. XIII. Project Management Performance A. Project HOPE Center 189. Project HOPE Center management of the CHAPS project was satisfactory, with some shortcomings. As noted, the Regional Director for Africa changed three times over the course of the project, which affected the continuity of attention to CHAPS. HOPE Center correctly identified problem areas in implementation such as weak technical monitoring and hired a technical support manager, but it was not always able to get appropriate actions taken. In May 2002, HOPE Center provided training to Project HOPE Malawi finance and administration staff in the QuickBooks 2002 financial system, which feeds into the financial reporting system. In July 2002, HOPE Center provided training to Project HOPE Malawi staff in the areas of basic information on reporting policies and procedures and finance. In February 2003, HOPE Center provided one week of in-service education on GIK to Project HOPE Malawi. Project HOPE Malawi, including the CHAPS project, would have benefited from more staff capacity building in project management, financial management, monitoring and evaluation (management information system), and GIK management. 190. Communications between HOPE Center, Project HOPE Malawi, and the CHAPS project could have been more timely and generous in terms of fuller discussions of CHAPS project progress toward the achievement of objectives, the identification of problems, and agreed remedies. Project HOPE should promote broad-based, open communications throughout the organization. Project HOPE may wish to consider the adoption of a more formal supervisory system, which would assist management at all levels. B. Project HOPE Malawi Final Report 48 March 14, 2003 191. Country office management of the CHAPS project was satisfactory, although management's role in advocacy could have been stronger. In view of the broad project scope and excessive number of activities and target groups relative to the project implementation period and number of staff, the country director could have advocated for scaling back the project to create a better fit between project objectives and available resources (human and technical). In addition, management could have been more pro￾active with respect to the need for a project coordinator and management training for the Project HOPE CHAPS project manager. Following the planned departure of the project manager at the end of the second year, the project coordinator was promoted to project manager. The coordinator position was not filled because, at the time, the project had one year remaining, recruitment takes three months on average, and a one-year position would not be attractive. Nonetheless, a project coordinator was needed to coordinate day-to-day CHAPS activities with the Mulanje and Phalombe district CHAPS coordinators and to act on behalf of the project manager in the latter's absence. To some extent, the role of project coordinator was filled on an ad hoc basis by one of the community health educators. In retrospect, a qualified staff person should have formally filled the project coordinator position, with necessary adjustments made in staff responsibilities. 192. The project manager did not have the benefit of management training but instead learned on-the-job. In hindsight, the project manager should have received management training, which complementing his experience and technical skills and knowledge, would have enhanced management effectiveness and efficiency. In the future, Project HOPE may wish to examine the opportunity costs and cost-benefit of management training. C. CHAPS Project 193. CHAPS project management was satisfactory, despite some shortcomings. The CHAPS project had two managers over the life of the project. During the course of the project, several activities were added to the manager's portfolio, in addition to CHAPS: management of the Project HOPE Cross-Border AIDS project,41 the subgrant for the Malawi Breastfeeding Operations Research Project (MBORP),42 and the LINKAGES subgrant;43 project management committee and oversight activities for the Malawi Social Action Fund (MASAF) grant to remodel the Nutrition Rehabilitation Center in Mulanje; and preparation of and preliminary activities for the Saving Newborn Lives initiative. An 41 Management of the Cross-Border AIDS project proved particularly burdensome because the project coordinator changed three times. 42 In 1998, Population Services International (PSI) Malawi selected Mulanje and Mangochi as two CHAPS districts to pilot an exclusive breastfeeding project, inviting the participation of Project HOPE and Save the Children in their respective districts. The agreement between PSI and Project HOPE was signed in April 1999. Project HOPE activities included the development of data collection tools for focus group discussions and in-depth interviews, training data collectors, conducting research, and attending MBORP meetings, among others. The MBORP concluded on or about October 2001. 43 Activities under the Academy for Educational Development (AED) LINKAGES project included formative research on PMTCT, VCT, and infant feeding, which involved data collection through in-depth interviews, focus groups discussions, and household observations. Final Report 49 March 14, 2003 estimated 60 percent of the manager's time was spent on CHAPS project management and 40 percent on other activities, including attending a total of 10 weeks of training and conducting an estimated 20 weeks of training over a four-year period. 194. The extent to which Project HOPE was prevailed upon by donors to add on extra￾CHAPS project activities, or whether the project manager had decision-making authority over add-ons, was not clear. However, the result of add-ons was an overwhelming workload. Moreover, the value of add-ons did not appear to outweigh their costs in time and human resources. At the least, the full costs of add-ons should have been analyzed, followed by renegotiated agreements, where indicated, with the sponsoring parties. D. District Health Management Teams 195. District management performance was satisfactory within the structural constraints noted elsewhere. Had staffing been more stable, many of the management strengthening interventions would have had more impact. XIV. Implementation Performance A. Project HOPE 196. Project HOPE's implementation performance was highly satisfactory. With the exception of a small number of research activities and the constraints imposed on PMTCT and VCT activities due to lack of government protocols and guidelines, all planned activities were carried out. Between 1997 and February 2003, the project conducted 181 planned training activities, which had a total of 4,084 participants. Planned construction of district health staff houses and Under Five outreach clinics was completed, and facilities are in use. GIK supplies and equipment were distributed to hospitals and health centers; trained community volunteer supply and equipment kits were distributed to volunteers. 197. Project implementation performance by Project HOPE CHAPS project staff was highly satisfactory. In February 2003, staff numbered 19, including the project manager, accountant/administrator,44 eight community health educators, one health information system officer (HIS),45 two office assistants, two drivers, and security guards (Table 3). In contrast to district health staff, Project HOPE staff turnover was low, which was a tremendous benefit to the project (Table 4). Staff performed well consistently, despite a heavy work load, the lack of a computerized project monitoring system, and the significant constraint posed by high turnover rates of district health staff who were both counterparts and training participants. The community health educators provided high quality training, 44 For a period of time, the accountant/administrator was responsible for the CHAPS Project and the Cervical Cancer Project and was "cost-shared" accordingly. 45 The HIS officer divided her time between the CHAPS Project and the Cross-Border AIDS Project. Final Report 50 March 14, 2003 and their services were frequently sought (but not provided) outside the CHAPS project. All staff contributed to the success of the project. 198. Project HOPE gained operational efficiencies in the area of funds management, which facilitated implementation performance. The opening of a bank account held in Kwacha in Mulanje in 2001 provided immediate, on-site access to funds for project activities, which facilitated their timely implementation. In the early part of the project, trainers were responsible for the administration of training funds, e.g., payment of trainee allowances, which took time and attention away from training activities. In the latter part of the project, the project accountant/administrator was responsible for training fund administration, which worked well. 199. As previously noted, the DHMTs' perception of lack of transparency in the financial management of the project was linked to the fact that the project manager did not have up-to-date budget or complete budget and expenditure information. The main source of this problem was the difficulty in preparing a project activity budget because Project HOPE's financial system does not easily permit the translation of budget data used in financial reporting into project budgets based on activities, e.g., training, which is essential for project management and different from financial reporting to donors. Project HOPE is encouraged to increase its efforts in the development of user-friendly project activity budgets. B. District Health Management Teams 200. The implementation performance of the DHMT was deemed satisfactory by those queried, despite shortcomings that stemmed from both project design and MOH structural issues. Everyone felt a sincere effort had been made to cooperate, to achieve the project objectives, and to improve services. However, given the excessive amount of project training, district management staff were not always available when needed. In addition, with competing priorities, they acknowledged that project activities were not always prioritized. C. Health Services 201. Health services implementation was generally satisfactory and improved over the life of the project. Particularly interventions like infection prevention were generally taken seriously. Interventions for VCT and PMTCT were entirely new with the project and appreciated by hospital staff. Again, the primary constraint was staff turnover, limiting the impact of training. D. Trained Community Volunteers 202. The performance of trained community volunteers in project implementation was highly satisfactory. Volunteers were responsible, took their work seriously, and devoted Final Report 51 March 14, 2003 countless hours to improving the health of their communities. The project would not have succeeded in reaching its objectives without the trained community volunteers. XV. Project Design 203. The project design was not appropriate for achieving the project objectives because the scope and number of activities and target groups was too broad relative to the planned three-year implementation period and the number of staff. Too much emphasis was placed on training, and too little, on supervision and follow-up. There is no question that training was needed or that the high quality training provided did not have a substantial impact on the delivery of health care and services and on community health. Rather, the issue is the amount of training and the large number of groups trained. 204. The flaw in project design may be attributed in part to the fact that the CHAPS project, as it was first approved in 1997, was designed as a five-year project. Project HOPE and other participating PVOs may not have adapted their original proposals to the changed three-year timeframe, and thus, activities that were planned for a five-year period were compressed into a three-year period. USAID reported that all five participating PVOs had sought cost and no cost extensions. In hindsight, steps should have been taken to adjust the project's dimension during the initial three-year period and at the time of the cost and no cost extensions. 205. The criteria for judging the achievement of project objectives were adequately quantified in the technical proposal. The criteria were not adequately quantified in the one-year cost extension and the nine-month no cost extension because activities/outputs were not sufficiently distinguished from effects/impacts. XVI. Sustainability 206. The sustainability of the CHAPS Project is likely, albeit in changed form. In March 2003, USAID is expected to announce the award for Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, which would continue to support the five (sic) District Health Offices currently receiving USAID assistance under CHAPS, in addition to partnerships in three new districts.46 Phalombe district was not included in the new initiative; thus, the sustainability of the project there is uncertain. 207. The sustainability of trained volunteer activities is likely to the extent that the replenishment of needed supplies is forthcoming, HSA supervision continues, and there are 46 See USAID Regional Center for Southern African (RCSA). RFA, Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, op. cit. The intermediate results will be: improved prevention and management of childhood illnesses, increased use of malaria prevention practices, strengthened central hospital management systems, and hospital autonomy implemented. Final Report 52 March 14, 2003 incentives for local participation. At the time of the evaluation, there was no plan in place to ensure resupply for volunteer activities at the conclusion of project financing. If initial volunteer training activities do not continue, new volunteers will not be added to expand coverage or cover attrition. If refresher training, which is a tremendous incentive for participation, does not continue, it is possible that volunteer morale would diminish over time, leading to a decrease in or cessation of activities. At the time of the evaluation, Oxfam and the Norwegian Agency for Development Cooperation (NORAD) had shown interest in home-based care groups and Youth Friendly Health Services, respectively. 208. The sustainability of community involvement in health care is likely with respect to malaria prevention and treatment, control of diarrheal disease, HIV/AIDS and home based￾care, family planning, and STIs because communities have a sound knowledge base, key behavior changes occurred, and perceived benefits would provide the motivation to continue to support positive behavior change. The evaluation's findings based on qualitative data indicate that the knowledge base for ARIs and nutrition was inadequate to support behavior change; however, the end-of-project survey data may show otherwise and would be a better indicator due to the larger sample size and more detailed questions. 209. The sustainability of project achievements by the Mulanje and Phalombe DHMTs is uncertain due primarily to the high staff turnover rate, which is the most serious threat to sustainability, the absence of systematic orientation of new staff to CHAPS interventions or systems, and the lack of financial resources to purchase, for example, volunteer supplies and infection prevention supplies, or to provide for vehicle operations and maintenance, which are essential for supervision and follow-up. However, both DHMTs were generally positive about the potential for sustaining changes in the DHMT planning process and changes in provider practices. The sustainability of project achievements at the health center level is likely because health centers experience less staff turnover than at the district level; therefore, the knowledge and skills gained from project training will remain. 210. During the life of the project, the three voluntary counseling and testing (VCT) centers—two in Mulanje and one in Phalombe—were dependent on the CHAPS Project for VCT supplies. The project created a demand for VCT, and the sustainability of the Mulanje VCT centers appears likely because the centers will have access to supplies through the UMOYO Networks47 or Family Health International, according to USAID.48 The sustainability of the Phalombe VCT center is uncertain because Phalombe district was not included in the new proposal, although USAID observed that it is in the interest of MOHP to continue Phalombe VCT center activities and that the new USAID HIV/AIDS initiative was aware of Phalombe district's VCT needs.49 47 Under the Washington-based NGO Networks program, Umoyo Networks was initiated in 1999 to foster and support networks that enhance the scale and quality of family planning, reproductive health, child survival, and HIV/AIDS/STI programs. 48 Personal communication, USAID debriefing on CHAPS Project final evaluation, February 18, 2003 49 Ibid. Final Report 53 March 14, 2003 XVII. Assessment of Outcome 211. The project's outcome was satisfactory. The project achieved most of its major objectives, with some shortcomings, based on overall implementation performance, management performance, monitoring and evaluation, and sustainability (Table 5). The project worked to achieve difficult objectives in partnership with the two DHMTs and community groups—unlike in other projects where project staff have direct control over inputs, processes, and outcomes. For example, the project had no control over the high rate of district health staff turnover, which was a substantial constraint on implementation. The project's major accomplishments are that the majority of district health staff were trained in the six intervention areas, and the project reached remote, dispersed, and largely inaccessible communities, where access to quality health care improved, and community participation in health care increased. XVIII. Summary Conclusions, Lessons Learned, and Recommendations 212. Conclusions. USAID piloted the CHAPS partnership approach to strengthening DHMT, which was innovative and untested. The CHAPS project in Mulanje and Phalombe districts demonstrated that such an approach can be successful under certain conditions, namely, USAID, MOHP, and PVO commitment; the correct assessment of needed health interventions, the technical viability of interventions; adequate staffing and low staff turnover; key sector policies are in place; effective training, supervision, and follow-up; adequate resources; effective management; financial transparency; adequate monitoring and evaluation system; appropriate incentives for community participation; and a satisfactory partnership arrangement. The CHAPS project succeeded when these conditions were substantially met and was less successful when they were not. 213. Lessons learned. In a pilot project that aims to establish a new kind of working relationship between partners who are inexperienced in partnerships, which encompasses not only building the partnership but also strengthening the capacity of one partner, in addition to implementing a wide range of activities among a variety of different target groups across a large geographical area, a considerable investment of resources (human, financial, technical) is needed during the design, planning, and early implementation phases in order to ensure that the implementation of activities and partnership development can proceed as smoothly as possible and to take corrective measures in a timely fashion in the event of problems. In fact, the CHAPS project investment in the project's technical (health interventions) component was significant. Investment in the partnership component and institutional capacity building was not as substantial as it should have been in part due to the novelty of the approach. The new approach notwithstanding, the principals involved could have made greater efforts to provide technical assistance on Final Report 54 March 14, 2003 partnership development, especially in the areas of roles and responsibilities and financial management. 214. Recommendations. In future partnership-based projects, partnership development, as well as institutional capacity building, should receive attention equal to that of technical components, even if it would mean a reduction in the scope and number of health interventions and related activities, because it is the capacity building and strengthening through partnership development that will sustain project achievements and benefits and not the other way around. Final Report 55 March 14, 2003 Bibliography Academy for Educational Development. Internal Assessment for Integrated Baby Friendly Hospital Initiative and Prevention of Mother to Child Transmission of HIV in Health Care and Community Services, Linkages Project, Malawi, January, 2003 Aga Khan University, Aga Khan Foundation, Aga Khan Health Services, and University Research Corporation Assessing the Quality of Management, Primary Health Care Management Advancement Programme, Module 7, User's Guide, n.d. Benech, Irene Community Baseline Survey, Community Health partnership Programme, Mulanje and Phalombe Districts, June-July 1998 Quality Assurance Assessment, Community Health Partnership, Mulanje/Phalombe District, April-June 1998 Bvumbwe, Alice Cross Border HIV/AIDS Program, Youth Friendly Health Services Training for Health Workers and Youth Peer Educators (draft), Mulanje View Motel, January 22-24, 2003 Family Health International “Evaluating voluntary HIV Counseling and Testing Programs” in Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries, pp. 81-90 HIV/AIDS Prevention and Care Department: “Reducing Mother to Child Transmission – A Strategic Framework”, Funded by USAID under the IMPACT project HIV/AIDS Prevention and Care Department: “Voluntary Counseling and Testing – A Strategic Framework”, Funded by USAID under the IMPACT project Food and Agriculture Organization (FAO) and World Food Programme (WFP), United Nations Special Report, FAO/WFP Crop and Food Supply Assessment Mission to Malawi, 29 May 2002 Government of Malawi Malawi, Demographic and Health Survey 2000, National Statistical Office, Zomba, Malawi, and ORC Macro, Calverton, Maryland, August 2001 Final Report 56 March 14, 2003 International Monetary Fund Food Policy and IMF- and World Bank-supported Programs, A Factsheet, July 2002 Kaponda, Chrissie P.N. Mid-Term Evaluation, Community Health Partnerships Programme, A Joint Programme for Project HOPE-Malawi and Mulanje District Hospital sponsored by USAID, Lilongwe, April-May 2000 Matinga, Priscilla Upasani “HIV and Infant Feeding Findings and Recommendations from the Formative Research for Mulanje and Phalombe districts in Malawi”, Project Hope/Malawi in collaboration with Linkages Project, Academy for Educational Development/ Zambia Regional Office, April 2002 Monitoring, Evaluation, and Design Support (MEDS) Project Report of the Evaluation of the Community Health Partnerships Project (CHAPS), Submitted to The United States Agency for International Development under Contract No. HRN-I-00-99-00002-00, Task Order Number 817, LTG Associates, Inc. and TvT Associates, March 2002 Mulanje District, Ministry of Health and Population Health Information System (HIS) reports for January–May 2000 Phalombe District, Ministry of Health and Population Health Management Information System (HMIS) reports for January–March 2002 Population, Health and Nutrition Information Project (PHNPC) Malawi, PHNIP Country Health Statistical Report, September 2002 Project HOPE Baby Friendly Hospital Initiative Project Baseline Survey, June 2001 CHAPS Cost Extension Training Activities Schedule (work plan and budget), n.d. CHAPS Team Quarterly Meeting, Mangochi, June 30, 1998 CHAPS Team Quarterly Meeting, Salima, September 28-29, 1998 CHAPS Team Quarterly Meeting, Mzuzu, January 1999 CHAPS Training Activities, Year 1 (31/07/98), 1 Service Delivery (annual work plan and budget) Final Report 57 March 14, 2003 CHAPS Training Activities, Year 2 (last up-date 22.06.99), 1 Service Delivery (annual work plan and budget) CHAPS Training Activities, Year 3, Service Delivery (annual work plan and budget) CHAPS Work Plan for No Cost Extension, September 2002 to March 2003 (revised on October 9, 2002) (work plan and budget) Community Health Partnerships, Mulanje District, February-March 1998, 1st Quarterly Report Community Health Partnerships, Mulanje District, April-June 1998, 2nd Quarterly Report Community Health Partnerships, Mulanje District, July-September 1998, 3rd Quarterly Report Community Health Partnerships, Mulanje and Phalombe Districts, 4th Quarterly Report, October-December 1998 and Annual Report 1998 Community Health Partnerships, Mulanje and Phalombe Districts, 1st Quarterly Report: January-March 1999 Community Health Partnerships, Mulanje and Phalombe Districts, 2nd Quarterly Report: April-June 1999 Community Health Partnerships, Mulanje and Phalombe Districts, 3rd Quarterly Report: July-September 1999 Community Health Partnerships, Mulanje and Phalombe Districts, 4th Quarterly Report: October-December 1999 and Annual Report 1999 Community Health Partnerships (CHAPS) Project, Mulanje and Phalombe Districts, 1st Quarterly Report, January 2000 to March 2000 Community Health Partnerships (CHAPS) Project, Mulanje and Phalombe Districts, 3rd Quarterly Report, July to September 2000 Community Health Partnerships (CHAPS) Project, Mulanje and Phalombe Districts, 4th Quarterly Report, October to December 2000 Community Health Partnerships (CHAPS) Project, Mulanje and Phalombe Districts, Quarterly Report, January to March 2001 Final Report 58 March 14, 2003 Community Health Partnerships (CHAPS) Project, District Health Management Team, Quarterly Report Number 14, Quarter 3 (July to September 2001) Community Health Partnerships (CHAPS) Project, Mulanje District Health Management Team and Project HOPE, Quarterly Report Number 15, Quarter 4 (October to December 2001) and Annual Report (Jan-Dec. 2001) Community Health Partnerships (CHAPS) Project, Mulanje District Health Management Team and Project HOPE, Quarterly Report Number 16, Quarter 1 (January to March 2002) Community Health Partnerships (CHAPS) Project, Mulanje District Health Management Team and Project HOPE, Quarterly Report Number 17, Quarter 2 (April to June 2002) Community Health Partnerships (CHAPS) Mulanje District Health Management Team and Project HOPE, Quarterly Report Number 18, Quarter 3 (July to September 2002) Community Health Partnerships (CHAPS) Mulanje District Health Management Team and Project HOPE, Quarterly Report Number 19, Quarter 4 (October to December 2002) Community Health Partnerships (CHAPS) Project, Mulanje and Phalombe Districts, Request for Extension of Grant No. 690-A-00-98-00093-00, Submitted February 2001 HMIS supervision checklist during start up phase Report of Quarterly HMIS Performance Review Meeting, May 2002 Scope of Work, Community Health and Partnerships (CHAPS) Final Evaluation, Project HOPE/Malawi, January 13, 2002 Technical Proposal: Project HOPE's Community Health Partnerships (CHAPS) with the District of Mulanje, RFA No. 690-97-002, November 1997 __________, District CHAPS Coordinator, and District Health Officer for both Mulanje and Phalombe Districts Memorandum of Understanding, Allowances Schedule – Community Health Partnership Programme, Mulanje and Phalombe Districts, June 2000 Final Report 59 March 14, 2003 __________, District CHAPS Coordinator, the Mulanje District Health Officer, and the Phalombe District Health Officer. Memorandum of Understanding on Operational Procedures – Community Health Partnership Programme, Mulanje and Phalombe Districts, June 2002 United Nations Development Programme (UNDP) Human Development Report 2001, Making New Technologies Work for Human Development, Oxford University Press, New York, 2001 United States Agency for International Development (USAID) Southern Africa – Complex Drought Fact Sheet #6 (FY 2002), Bureau for Democracy, Conflict, and Humanitarian Assistance (DCHA), Office of U.S. Foreign Disaster Assistance (OFDA), Southern Africa Complex Food Security Crisis, 14 June 2002 USAID Bureau for Humanitarian Response/Office of Private and Voluntary Cooperation PVO Child Survival Grants Program Guidelines for Mid-Term and Final Evaluations, December 2001 USAID Regional Center for Southern African (RCSA) Request for Applications (RFA) Number 690-03-003, Reducing Child Morbidity and Strengthening Health Care Systems in Malawi, Issuance date: October 24, 2002 Request for Applications (RFA) Number 690-03-001, Reducing High Risk Behaviors Among Malawians, Issuance date: October 24, 2002 World Bank Better Health Outcomes from Limited Resources, Focusing on Priority Services in Malawi, Africa Region Human Development Working Papers Series, 24132, April 2002 The state of Malawi’s poor: The incidence, depth, and severity of poverty, PMS policy brief No. 2 (revised), November 2000 Final Report 60 March 14, 2003 Annex A: Scope of Work Program: Community Health and Partnerships (CHAPS) Position: Final Evaluation Consultant Duration: Approximately January - February, 2003 Reports to: Petra Reyes, Sc.D., MPH, Regional Director, Africa Technical: Bettina Schwethelm, Ph.D., MPH, Director, Maternal & Child Health Administrative: Sue Adams, Assistant Regional Director, Africa--Operations Background Under the USAID local mission initiative and funding, Project HOPE in partnership with Mulanje District Health Management Team (DHMT) has been implementing a Community Health Partnerships (CHAPS) program beginning in March 1998. The initial phase covered the period from March 1998 to March 2001. The main goal of the program in this initial three-year phase was to improve health care services and community awareness in the district through strengthening the district’s Maternal and Child Health (MCH) and Reproductive Health service delivery, and building human and institutional capacity in a sustainable manner. The program implementation model was one of a partnership model between an MOHP DHMT and a Non –Governmental Organization (NGO). A one-year cost extension was awarded to cover the period March 2001 to June 30, 2002. In responding to the USAID Strategic Objective 3, “Increased adoption of measures that reduce fertility and risk of HIV transmission”, the cost-extension phase continued to consolidate the main goal of the initial three years, while placing emphasis on HIV/AIDS activities. Specifically, the cost-extension year sought to introduce within the districts, activities that focused on prevention of mother to child transmission (PMTCT) of HIV, and increased access to voluntary counseling and testing (VCT) for HIV. USAID-Malawi also granted a no-cost extension to the one-year cost-extension from July 2002 to March 2003. Delay in approval of the PMTCT and VCT protocols, and unavailability of the rapid HIV test kits resulted in the program’s need for additional time to complete implementation of the planned activities under PMTCT and VCT. During the course of the initial three-year phase, the Mulanje district was sub-divided into two districts, Mulanje and Phalombe in November 2000. Project HOPE however has continued to work with these two districts under the Mulanje DHMT as Phalombe does not yet have a full DHMT. Purpose of the Evaluation The purpose of the evaluation is to assess achievement of the CHAPS program under each of the 3 Key Intermediate Results (KIR) under the Strategic Objective 3 of USAID, the impact of the program interventions and the effectiveness of the NGO-public sector partnership model in the Malawian context. The main focus of the evaluation will be to document amongst other things, key accomplishments, constraints and lessons learned from the NGO-public sector partnership model. The evaluation will also benefit Project HOPE, Mulanje and Phalombe DHMT in having a broader Final Report 61 March 14, 2003 understanding of their role in program implementation for any future program of this nature; and discover the effect of the partnership model on community and the DHMT. Broad Objective of the Evaluation: The consultant will lead a core evaluation team for this final evaluation to assess project achievement of each major objective in accordance with USAID-Malawi KIR indicators, with specific focus on the priority areas for interventions which are: service delivery, knowledge and practice changes in providers and the target population, and operational sustainability of the partnership model. Specific objectives: 1. Assess achievement of the program objectives. 2. Assess the quality and appropriateness of the service delivery and education efforts in the major intervention areas. 3. Assess quality and appropriateness of the PMTCT and VCT services initiated in the two pilot sites. 4. Determine whether USAID KIR indicators were attained. 5. Determine lessons learned from the NGO-public sector partnership model. Illustrative Evaluation Guidelines A. Program Implementation 1. Generally, based on the program proposal document, assess achievement of each major objective and output. • Document the progress that has been made toward the objectives and output. • Identify major successes • Identify constraints and unanticipated effects. • Assess effectiveness of the partnership model. • Make recommendations for the sustainability of activities by the DHMT. 2. Specifically, assess attainment of the objectives of each of the following priority areas for intervention and other relevant evaluation areas. In all the areas, provide specific recommendations for future: a) Service delivery • Review specific accomplishments on each of the targeted interventions. • Identify shortfalls/challenges in the approaches that were used • Determine recommendations for the future b) Capacity building. • Review the effectiveness of the capacity building strategies that were used. • Review the extent to which Technical Assistance strengthened capacity building and contributed towards achievement of program objectives. c) Partnership model. • Examine how the partnership model was operationalized in the district. • Identify problems and benefits associated with this model. • Determine whether the partnership worked at addressing the identified problems and at strengthening the district. d) Sustainability. • Review the human, institutional, financial and operational elements that relate to sustainability; indicators used to measure progress; and achievements to date. Final Report 62 March 14, 2003 • Assess prospects for post-grant sustainability including the financial aspects for continuing activities by the DHMT, addressing provider training needs, and community participation. e) Based on government thinking of decentralization: • Assess the design and implementation of the program in respect of the MOHP concept of decentralization. • Determine how the program activities will be sustainable in the current context. f) Drawing on lessons learnt. • Identify lessons learnt from program implementation activities since its inception. • Identify key players with whom the lessons learnt should be shared to promote the sustainability of the activities that were initiated by the program. B. Program Management 1. Assess program’s management practices and capacity including structure & quality of management: • Assess staffing levels and availability of appropriate technical and management skills to oversee program activity. • Find out whether program personnel conducted periodic review of performance data and self-evaluation and used data to make decisions to scale up implementation of activities, for example. • Determine clarity of individual roles and responsibilities based on job descriptions and other duties. Find out whether job descriptions are available and known by staff. • Ascertain the presence and use of documented program administration procedures including record keeping of quarterly and annual reports, financial reports, and evaluation and survey reports. • Find out what motivated staff to work on the program. 2. Assess capacity of the organization to monitor and evaluate program performance and impact. Is there evidence and use of? • Appropriate performance objectives & use of indicators? • Developed baseline assessment and plans for re-assessment? • Appropriate use of baseline information in program implementation? • Systems to consolidate, analyze, and interpret data • Appropriate use of OR and QI for performance improvement? 3. Financial Management: • Find out whether adequate management and monitoring systems are in place to report on and control the use of funds. Please describe. • Determine how budgets are made • Determine how funds are monitored • Assess the stock control system to determine how commodities are procured and used. • Input/involvement of the MOHP/DHMT in the financial decision-making (priorities, per diems) 4. Partnership • Determine the type and quality of relationships with stakeholders. Definition of roles and responsibilities, frequency and content of meetings (joint planning, point progress review, joint monitoring and supervision). Final Report 63 March 14, 2003 • Find out what support the project obtained from partners For each area above, provide recommendations to improve the overall program management. Background Documents. In order to gain understanding of the program, the consultant will be provided with a variety of documents. The documents will include but not limited to, program proposal, baseline and other study reports, quarterly reports, annual reports, training materials developed, consultancy reports and other relevant documents. Evaluation Methodology. The evaluation process will be participatory and complemented by a variety of program documents. The core evaluation team will be made up of the Program Manager, the CHAPS District Coordinator, DHMT program managers, the Technical Support Manager and two consultants. The consultants will design and develop the evaluation tools. The process will involve: 1. Meeting as a team to review the evaluation design and evaluation tools, determination the evaluation schedule and list of interviews to be conducted. 2. Proposed key informants: • DHMT and Project HOPE program implementing teams. • Health Centre staff in Mulanje and Phalombe Districts. • HSAs, CBDAs, TBAs, Village Health Committees. • Representatives of stakeholders of the District Health Board. • Community groups for Home-based care and Anti-AIDS club. • Community members in the project catchment area and other partners • Donors 3. Meet as an expanded team to analyze results and develop recommendations. Deliverables: 1. Presentation of preliminary findings to Project HOPE/Mulanje staff and the Project HOPE/Malawi Country Director. 2. Present results to the expanded DHMT and other relevant key stakeholders. 3. Debriefing with USAID-Malawi. 4. Draft final evaluation report provided to HOPE Centre and Project HOPE-Malawi for comments within ten days of departure from Malawi. 5. A final evaluation report of acceptable quality in accordance with this Scope of Work should be submitted no later than 10 days after feedback on the draft evaluation report. This report should be accompanied by a computer version in Microsoft Word software. Time Frame 2 weeks in country. Specific time frame to be developed together with consultants. Approved by: ________________________________________ Date:_____________ Bettina Schwethelm, Director, MCH Approved by: ________________________________________ Date: _____________ Petra Reyes, Regional Director, Africa Approved by: ________________________________________ Date: _____________ Deborah Carl, Vice President, Human Resources and Administrative Support Services Final Report 64 March 14, 2003 Annex B: Sites Visited, Persons Met, and Groups Interviewed MULANJE DISTRICT PHALOMBE DISTRICT Expanded DHMT Expanded DHMT Mulanje Mission Hospital medical director VCT counselors nurses lab technician data entry staff Holy Family Hospital matron clinical officer in charge VCT counselors accountant administrator Mulanje District Hospital matron VCT counselors lab technician CHAPS coordinator Migowi Health Center/Catchment Area health center in-charge HSAs ward attendants cleaner Bondo Health Center/Catchment Area health center in-charge HSAs home-based care group community group CBDA group (Mimosa Health Center) Phalombe Health Center/Catchment Area nurses pharmacist HSA traditional birth attendant Mpala Health Center/Catchment Area health center in-charge nurse HSA traditional birth attendant water point committee drama group youth group (Boma) Mwanga Health Center/Catchment Area health center in-charge medical assistant lab technician ward attendants cleaner HSA CBDA group community group Namulenga Health Center/Catchment Area health center in-charge HSA CBDA group home-based care group youth group community group Lujeri Health Center/Catchment Area health center in-charge nurse drug revolving fund group water point committee community group Others: District AIDS Coordinating Committee District Youth Officer Oxfam World Vision International Final Report 65 March 14, 2003 Table 1: CHAPS Project Timetable Steps in Project cycle Date Planned Date Actual/Latest Estimate 1. Original 11/1/97 – 9/30/00 11/1/97 – 6/30/01* 1.1. Identification - RFA 4/21/97 1.2 Preparation 4/21/97 – 6/4/97 1.3 Approval (preauthorization) 11/10/97 1.4 Negotiations 11/5/97 – 1/31/98 1.5 Signing 1/31/98 1.6 Start-up (baseline survey) 3/98 1.7 Midterm evaluation 7/00 1.8 Closing of original n/a 2. One-year cost extension 7/1/01 – 6/30/02 2.1 Preparation 7/14/00 – 4/19/01 2.2 Approval 6/11/01 2.3 Negotiations 6/11/01 – 7/24/01 2.4 Signing 7/24/01 2.5 Start-up 8/7/01 2.6 Closing of one-year cost extension 6/30/02 3. No cost extension (nine months) 6/30/02 – 3/31/03 3.1 Preparation 5/7/02 – 5/15/02 3.2 Approval 8/1/02 3.3 Negotiations n/a 3.4 Signing 8/1/02 3.5 Start-up n/a 3.6 Final evaluation 1/27/03 – 2/21/03 3.7 Closing of one-year no cost extension 3/31/03 Source: Project HOPE Center *Modif.#1 – Increased end date to 6/30/01 with incremental release of obligated funds. Final Report 66 March 14, 2003 Table 3: CHAPS Project Staffing, March 1998-March 2003 Classification 1998 1999 2000 2001 2002 2003 Professional Management Females 1 1 Males 2 2 2 2 2 2 Technical Females 1 1 1 7 6 6 Males 2 2 2 2 2 2 Subtotal 6 6 5 11 10 10 Support Females 2 2 2 2 2 2 Males 6 6 6 6 7 7 Subtotal 8 8 8 8 11 11 Total 14 14 13 19 19 19 Source: CHAPS Project Table 4: CHAPS Project Staff Turnover, March 1998-March 2003 Year Total Staff No. of staff who left “Turnover Rate” Reason 1998 14 1 7% Deceased 1999 14 2 21% Deceased, terminated 2000 13 2 15% End of contract Resigned following husband 2001 19 1 5.8% Further studies 2002 19 2 14% Resigned for greener pastures Dismissed 2003 19 Source: CHAPS Project Final Report 67 March 14, 2003 Table 5: Summary of Assessments RATING/ CATEGORY Highly Satisfactory Satisfactory Unsatisfactory 1. Assessment of Project Overall x Substantial Partial Negligible 2. Achievement of Key Intermediate Results (KIR) x 2.1 Knowledge and use of good reproductive health practices improved x 2.2. Community participation in reproductive health care improved x 2.3 Range and quality of reproductive health services for women, men, and youth expanded x Likely Unlikely Uncertain 3. Project Sustainability x Highly Satisfactory Satisfactory Unsatisfactory 4. Implementation Performance 4.1 Project HOPE x 4.2 DHMTs x 4.3 Health Services x 4.4 Communities x Highly Satisfactory Satisfactory Unsatisfactory 5. Management Performance x 5.1 Project HOPE Center x 5.2 Project HOPE Malawi x 5.3 CHAPS Project x 5.4 DHMTs x Highly Satisfactory Satisfactory Unsatisfactory 6. Project Monitoring and Evaluation x 6.1 Project HOPE headquarters x 6.2 Project HOPE/Malawi x 6.3 CHAPS Project (Project HOPE) x 6.4 DHMTs x Table prepared by Anna Kathryn Webb Final Report 68 March 14, 2003 Table 6: CHAPS Project Progress toward Targets and Objectives • Improve health status of the populations with special emphasis on MCH and RH • Improve MCH and RH service delivery • Improve the management capacity of the DHMT and other health personnel to support priority health interventions within the district in a sustainable manner Intervention Indicators Baseline Final (a) Activities Target Complete (b) COMMUNITY LEVEL % of households who know high fever as a danger sign 58.1% Functioning DRF with SP 10 15 % of caretakers who gave Fansidar 44.5% 20.3% (SP) 51.8% (any) Train shop owners 350 170 Malaria Prevention and Treatment % of households who know bed nets can prevent malaria 5.6% 49% CBDAs trained and delivering services 45 new 133 (81 new) Family Planning % of women/partners who report using family planning 20.1% 33.2% Increased under five shelters 4 4 Nutrition % of mothers who breast fed within 1 hr. of delivery 28.7% 36% Community members trained in Soya cultivation 280 Final Report 69 March 14, 2003 Intervention Indicators Baseline Final (a) Activities Target Complete (b) % of mothers of children under 6 mos. who are exclusively breast feeding Know child should receive fluids or food 6 mos. or later % of families planting soya 6.6% 7.2% 46.8% 20.9% VHVs trained in EBF 80 80 Increase access to pit latrines (report use) 79.9% 58.8% sanplat 98.4% access – 81.3% sanplats Increase access to safe water 59.3% 79.8% (less shal. well access) ORS correctly prepared 3.5% 56.5% % of mothers who give same or more food 82.6% Diarrhea % of mothers who give same or more breast milk 66.9% 80.4% VHC members trained in latrine construction CHC members trained in well protection and maintenance Water committees 450 260 460 360 32 (4 members trained per cmte. ARI % of caretakers knowing danger signs for ARI 50.8% 58.5% VHVs trained 300 148 HIV/AIDS # of clients at VCT centers 0 (9mos.) 1046 counseled 995 tested HBCs trained HBC groups (several pts./HBC volunteer) Initiators trained Aids club patrons trained 100 40 164 350 19 105 Final Report 70 March 14, 2003 Intervention Indicators Baseline Final (a) Activities Target Complete (b) Improved knowledge of nutrition practices for HIV % of men/women who know symptoms of STI 52.3% W 71.2%M 77.7%W 92.4% M % of people reporting condom use with last irregular partner 59% % of people who know infants can get HIV though delivery or breastfeeding 4.6% Youth groups Drama groups Youth leader training Youth friendly services training Religious leaders in PMTCT VHVs in PMTCT 60 20 75 40 60 10 groups 2 groups 65 30 65 40 OTHER % of children above 12 mos without measles vaccine % of women with ANC % of women delivering at HC or hospital 41.6% 82.6% 60.5% ? 85% 54.2% Trained TBAs/Refresher Established bicycle ambulances Provided radio communication 60 4 43/120 4 6 HEALTH CENTER LEVEL Trained HWs in malaria 133 125 Trained HWS in FP 125 (new full/core) 77 refr. 77 (9 full, 68 core) 68 refr. % of HWs convinced of advantage of EBF Trained HWs in EBF/MTCT 123new/ 77 refr. 151 Health Center Technical Capacity Trained HWs in CDD 64 69 Final Report 71 March 14, 2003 Intervention Indicators Baseline Final (a) Activities Target Complete (b) HWs/facilitators trained in IMCI 43/32 HWs trained/refreshed in STI syndromic 79/40 61/15 Support staff trained in AIDS HWs trained in PMTCT HWs trained in EBF HWs in HBC HWs trained in VCT/infant feeding 100 50 49 58 80 60 50 MW refresher training/clean delivery 100 72 VCT counselors trained Lab facilities renovated/equipped Counseling room renovated 2 0 24 2 1 HWs/support staff trained in infection prevention 100 98 QA training HSA training 100 initial 210 refr. Incinerators constructed 5 5 Renovation of infant feeding sites 2 0 Health Center Management Capacity VCT centers established Facilities with effective infection prevention committees Drugs available during 90% of working days Drug stock monitoring system in place (new HMIS) Community volunteers receiving regular supervision 0 3 1 Some stockouts of STI drugs and Depo during 6 mos. Not bad Youth friendly corners established 2 0 Final Report 72 March 14, 2003 Intervention Indicators Baseline Final (a) Activities Target Complete (b) DISTRICT LEVEL Training in HMIS 20 Management training 43 Supervisory checklist developed x Computerized central pharmacy system x but not working Houses built for staff 4 4 Computer training District Management Capacity -75% of annual work plans completed -Monthly HMIS reports received within 2 wks. -Written supervisory tools in use -Quarterly HMIS review/planning meetings -80% if facilities demonstrate improved infection control practices problems Sometimes x probably Partnership Annual DHMT retreats x DHB meetings every 6 mos. DACC training DACC meetings 10 some 60 no (a) The source of "Final" indicators data was preliminary end-of-project survey results (prior to submission of draft survey report). (b) The source of "Complete" activities data was the CHAPS project manager. Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 1 FINAL REPORT EVALUATION INSTRUMENTS FINAL EVALUATION OF THE COMMUNITY HEALTH PARTNERSHIPS (CHAPS) PROJECT MULANJE AND PHALOMBE DISTRICTS, MALAWI USAID GRANT NO. 690-A-00-98-00093-00 NOVEMBER 1997 – MARCH 2003 PROJECT HOPE MALAWI BLANTYRE, MALAWI PROJECT HOPE, THE PEOPLE-TO-PEOPLE FOUNDATION, INC. MILLWOOD, VIRGINIA 22646 by Marcie Rubardt, RN, MPH Anna Kathryn Webb, Ph.D. San Antonio, Texas March 14, 2003 Contents 1. Interview Guide for Group Meetings with Community Members.....................................2 2. Interview Guide for Group Meetings with Trained Community Volunteers.....................4 3. Interview Guide for Youth Groups ....................................................................................8 4. Interview Guide for Extended DHMT ...............................................................................9 5. DHMT Individual Survey Questionnaire.........................................................................11 6. Guide for Health Center Visits.........................................................................................15 7. Interview Guide for Project HOPE ..................................................................................19 Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 2 1. Interview Guide for Group Meetings with Community Members 1. Health Problems 1.1 What are the most important health problems in your community? 1.2. What kinds of things have been done in the past to improve the health of the community? 1.3 Is there anything that needs to be done to improve the health of the community? 2. Health Services Health Center (give name) 2.1 When was the last time you used the health services at the Health Center? What was your experience? If you have not used the health services, why not? 2.2 Do you like the services provided by the Health Center? Why or why not? 2.3 Have health services at the Health Center changed in the last two years? If yes, how? Hospital (give name) 2.4 When was the last time you used the services at the Hospital? What was your experience? If you have not used the hospital services, why not? 2.5 Do you like the services provided at the Hospital? Why or why not? 2.6 Have health services at the Hospital improved in the last two years? If yes, how? 2.7 Do you prefer to go to the Health Center, the Hospital, or a private health practitioner/private facility? Why or why not? 3. Health Education Meetings 3.1 Have health education meetings been held in your community? Did you attend any of the meetings? What was discussed? Did you find it helpful? If yes, how? Campaigns Have health education campaigns been held in your community? Did you attend? What was discussed? Did you find it helpful? If yes, how? 3.9 Have you seen any health education materials such as posters in your community or at the Health Center or Hospital? (a) Which of the health education materials did you like best and why? 3.10 Do you have any questions about health? If yes, what are they? 4. Trained Community Volunteers 4.1 Are there persons in your community who do volunteer work to improve community health such as Home-Based Care committees, Traditional Birth Attendants, CBDAs, Water Point Committees, youth groups (Anti-AIDS clubs) and, Drug Revolving Fund (DRF) members? 4.2 What do Home-based Care committees do? What do traditional birth attendants do? What do Water Point Committees do? What do youth groups (Anti-AIDS clubs) do? What do Drug Revolving Fund (DRF) members do? 4.3 Do you think that the volunteer's work has helped to improve community health? Why or why not? 4.4 Do you participate in community health activities? (probe) 5. Knowledge of Malaria Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 3 5.1 What have you learned about malaria? (e.g., transmission of malaria, signs and symptoms, care of a child with malaria, what things would prompt mothers to take child with malaria to the hospital, home care of a person with malaria, problems that can arise if a person is not treated for malaria) 5.2 Do you practice what you know? Why or why not? 6. Knowledge of Diarrhea 6.1 What have you learned about diarrhea? (e.g., factors that predispose people to developing diarrhea, waterborne diseases, sources of water, water treatment, basic sanitation facilities and use, management of children under 5 with diarrhea, recognizing dehydrated children who lost body fluids due to severe diarrhea) 6.2 Do you practice what you know? Why or why not? 7. Knowledge of Acute Respiratory Infections (ARI) 7.1 What have you learned about acute respiratory infections? (e.g., most common respiratory problems, recognition of children with ARI, dangers of common chest problems, problems in children with ARI that require parents to take them to health facilities, most effective ways of managing children with ARI) 7.2 Do you practice what you know? Why or why not? 8. Knowledge of HIV/AIDS & Home-based Care 8.1 What have you learned about HIV/AIDS and home-based care? (e.g., how HIV is contracted, behaviors that put persons at risk of contracting HIV/AIDS, how to provide home-based care for person with HIV) 8.2 Do you practice what you know? Why or why not? 9. Knowledge of Sexually Transmitted Diseases (STDs) 9.1 What have you learned about sexually transmitted diseases? (e.g., signs and symptoms, of STDs, where persons should go for treatment of STDs, when persons should seek treatment, behaviors that put persons at risk of contracting STDs) 9.2 Do you practice what you know? Why or why not? 10. Knowledge of Nutrition 10.1 What have you learned about nutrition? (e.g., breastfeeding, introduction of weaning foods, frequency of feeding children under age 5, cultivation and consumption of soya, diversification of protein foods in diet) 10.2 Do you practice what you know? Why or why not? 11. Observed Behavior Change 11.1 Have you seen any changes in the behavior of community members in the past two years? If yes, what are the behaviors that have changed and why? What activities contributed to behavior change? Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 4 2. Interview Guide for Group Meetings with Trained Community Volunteers I. Volunteer Groups A. Drug Revolving Funds (DRFs) B. Home-Based Care (HCB) groups C. Traditional Birth Attendants (TBAs) D. Community-based Distribution Agents (CBDAs) E. Water Point Committees F. Youth Groups G. Drama Group I. Traditional Birth Attendants A. How many births do you attend per month? B. What is your relationship with the Health Center? C. Where do you go to get supplies? D. For what danger signs would you send a woman to the Health Center? E. What do you do to make sure the mother and baby don't get infected? F. When do you see women after birth (post partum care?) II. Drug Revolving Funds A. What does the dispenser do? (e.g., keep drugs, dispense drugs, give instructions to the sick on the dosage of the drugs dispensed, make follow-up visits where possible to ensure that people are taking their drugs), record any drugs sold or damaged in the record book, report to the village meetings on the progress of the DRF) B. What does the Village Health Committee do? (e.g., ensure that the program is running well, receive and discuss reports from dispensers on the progress of the DRF, keep money from the sale of drugs by way of a treasurer, inform the people on the progress of their DRF through village meetings, summon DRF meetings, order drugs before the current stocks are finished, assist the dispenser to recover money if some patients got drugs on credit, supervise the running of the DRF) C. How many drug purchases are made each month? D. What types of drugs are ordered? Is the supply always sufficient? Where are they stored? Can we see the Drug Record? E. How is the price for drugs set? Is the price for drugs equal to the purchase price? Is there a profit? Can we see the record for money received? F. When does the dispenser give out Cotrimoxazole? (children under 5, respiration faster than 50) How much do they give? G. When does the dispenser give out SP? How much do they give for an adult? For a child? III. Home-based Care Committees A. How many persons are presently being cared for? B. What kind of care do you provide? How often? C. What medicines do you give, if any? D. When and how do you get supplies? E. What do you do to protect yourselves? F. Are there support groups? IV. Community-based Distribution Agents Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 5 A. How many clients do you see every month? Women? Men? Youth? B. Do you keep records? May we see your register? C. When and how do you get supplies? D. Do you use a hormonal checklist? May we see it? How do you use it? E. What side effects do you see in your women clients and what do you tell them? F. How do you explain condoms to clients? G. What methods do you distribute the most? V. Water Point Committees A. What are the activities of your committee? B. How many latrines were constructed in your community? Of the latrines constructed, how many are in use and properly maintained? C. How many water sources were protected? Of the protected water sources, how many are properly maintained? D. Dangers signs of using unsafe water and water associated diseases E. Good practices that promote water safety F. Describe a protected water point and materials required in well or spring protection VI. Drama Groups A. How many performances do you have each month? B. How many people attend the performances? C. What topics are of most interest to the persons attending the performance? D. Have the health education messages you provided been effective? Why or why not? E. How are your activities supported? F. How do you develop the health messages? With whom? How often do the messages change? 1. Training 1.1 What did you like about the training you received? 1.2 What didn't you like about the training? 1.3 Could the training have been improved? Yes or no? 1.4 Do you feel the need for additional training? Why or why not? 2. Supervision 2.1 How often were you supervised in the last three years? By whom? HAS? Health Center? Project HOPE? 2.2 Was your supervisor helpful? Why or why not? 2.3 What kinds of things did you discuss? 2.4 Could the supervision have been improved? If yes, how? 2.5 Do you feel the need for additional supervision? Why or why not? 3. Accomplishments 3.1 What do you consider to be your greatest accomplishments as a volunteer and why? 3.2 What motivates you to do volunteer work? 4. Challenges 4.1 What are the greatest challenges that you encounter in your work as a volunteer and why? (e.g., getting supplies) Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 6 4.2 How can these challenges be met? 5. Health Services 5.1 Which health services (health center, hospital, private facility/practitioner) do community members use most and why? 5.2 Have health services improved in the last several years? If yes, how? 5.3 How might health services be improved? Do you think there are areas in the health services that need to be improved? If yes, which ones? 5.4 What is your relationship to the health services? (e.g, supervision, reporting, getting restocked?) 6. Health Education 6.1 Have health education meetings and/or campaigns been conducted in your community? If yes, where? 6.2 What was discussed (topics)? (e.g., breastfeeding, HIV/AIDS, home based care of persons with HIV, STDs, malaria, acute respiratory infections, diarrhea, water and sanitation, family planning, Safe Motherhood, nutrition, other) 6.3 Were the health education activities effective? Why or why not? 7. Observed Behavior Change Since you started work as a community volunteer, have you seen any changes in the behavior of community members with respect to the following topics: 7.1 Malaria (a) If behavior changed, how did it change? (e.g., persons seek timely treatment, persons comply with full malaria treatment, functioning DRFs provide SP, community and health worker acceptance of SP as malaria treatment) (b) What activities contributed to behavior change? 7.2 Diarrhea (a) If behavior changed, how did it change (e.g., persons use oral rehydration therapy, persons seek timely care from health providers, mothers give the same or more breast milk or liquids and/or food to infant/child with diarrhea) (b) What activities contributed to behavior change? 7.3 Acute Respiratory Infections (ARI) (a) If behavior changed, how did it change? (e.g., persons who take care of children with ARI seek appropriate care, persons identify danger signs of ARI) (b) What activities contributed to behavior change? 7.4 HIV/AIDS and Home-based Care (a) If behavior changed, how did it change? (e.g., condom use, home-based care, acceptance of AIDS patients) (b) What activities contributed to behavior change? Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 7 7.5 Sexually Transmitted Diseases (STDs) (a) If behavior changed, how did it change? (e.g., persons seek timely care from health providers) (b) What activities contributed to behavior change? 7.6 Nutrition (a) If behavior changed, how did it change? (e.g., mothers practice exclusive breastfeeding for 6 months, mothers practice appropriate nutritional feeding practices in weaning and care of recovering/sick child, mothers practice good nutrition for themselves, families eat soya foodstuffs, families diversify protein foods in diet) (b) What activities contributed to behavior change? 8. Volunteer Work at the Conclusion of Project HOPE 8.1 Project HOPE activities will end in March 2003. Will you continue your volunteer work after the project ends? If yes, how? NOTE: This question was not asked after the first 2-3 times. 9. Lessons Learned 9.1 What lessons have you learned as a volunteer that you would like to share with others? Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 8 3. Interview Guide for Youth Groups 1. Challenges, Health Problems Facing Youth, and Aspirations for the Future 1.1 What are the most important challenges facing youth today? 1.2 What is the most important health problem facing youth? 1.3 What are your aspirations for the future? 2. Training 2.1 Did you like the training you received? Why or why not? 2.2 Could the training have been improved? Yes or no? 2.3 Do you feel the need for additional training? Why or why not? 3. Activities 3.1 What kinds of activities does your club undertake? How often? 3.2 How do you reach the youth? What approach to reaching other youth works best? What doesn't work? 3.3. Who are the youth reached? (age, gender, in school, out-of-school) 3.4 How many youth have you reached? How many are in school? Out of school? 3.5 What topics are of most interest to youth? (e.g., basic information on HIV/AIDS and STIs, preventing teenage pregnancy) 3.6 What approach works best in providing education/educational messages to youth on these topics? What approach has not worked? 3.7 Have the education/educational messages you have provided been effective? Why or why not? 4. Observed Behavior Change 4.1 Since you joined the Anti-AIDS club, have you observed any change in behavior of youth? If yes, what changes have you observed? 4.2 Which youth have changed their behavior? (e.g., girls, boys, both girls and boys, Anti-AIDS club members, youth who are not members, and so forth) 5. Supervision 5.1 How often are you supervised by the school patron? By Project Hope? 5.2 Was your supervisor helpful? Why or why not? 5.3 What kinds of things did you discuss? 5.4 Could the supervision have been improved? If yes, how? 5.6 Do you feel the need for additional supervision? Why or why not? 6. Reporting requirements 6.1 Do you prepare reports on the club's activities? If yes, what are the topics in the report? How often do you prepare reports? 7. Materials and supplies 7.1 What kinds of materials and supplies do you use in your activities? 7.2 Is there a need for additional materials and/or supplies? If yes, what kinds? 8. Accomplishments 8.1 What do you consider to be your greatest accomplishments as a youth volunteer and why? 8.2 What do you like the best? (motivation) 8.3 What message would you like to give to other youth? To adults 9. Other Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 9 4. Interview Guide for Extended DHMT CHAPS Project Goal and Objectives: The goal of the CHAPS project was "to improve health care services in the District of Mulanje through a partnership between Project HOPE and all levels of the health infrastructure in the Mulanje District [the Phalombe District was subsequently included], involving also other private sector and NGO health providers. This partnership will be the vehicle for (1) improving Maternal Child Health (MCH) services and Reproductive Health (RH) service delivery; (2) building human and institutional capacity in a sustainable manner; and (3) assisting USAID in achieving its Strategic Objective No. 3 in Malawi." Strategic Objective No. 3 is "behaviors adopted that reduce fertility and risk of HIV/AIDS and improve child health." The overall objectives were "to (1) improve health status of the population of the Mulanje District, with special emphasis on MCH and RH program activities and (2) improve the management capacity of the DHMT and other health personnel to support priority health interventions within the district in a sustainable manner." 1. Assessment of Project Design 1.1 Was the design of the project realistic and appropriate for achieving the project objectives? Why or why not? What are ways the design has been adjusted over time? 1.2 Are the criteria for judging the achievement of project objectives clear? If yes, how did you get your understanding of the objectives and criteria? 1.3 Were there implementation delays? If yes, what were the major reasons for the delays? For example, unrealistic implementation schedule; unexpected technical difficulties; changes in project scope; quality of management; delays in selecting staff; delays in receiving gifts in kind or other commodities 1.4 How did any of the following factors affect the achievement of project objectives? Why? - Please explain your answers. (S: substantially, P: partially, N: not at all) o Factors not generally subject to government control, such as changes in world markets and prices, natural disasters, and war and civil disturbance. o Factors generally subject to government control – central and district, such as economic policies and conditions, central and district health policies administrative procedures, including factors subject to MOHP central-level and DHMT control. o Factors generally subject to Project HOPE control, such as management effectiveness, staffing, cost changes, implementation delays, use of technical assistance, adequacy of monitoring and evaluation, and client/beneficiary participation. 2.0 Assessment of Partnership The stated objective of the PVO-DHMT partnership was to "develop a collaborative working relationship with all partners and to assure that district structures can sustain program planning, management, and service delivery activities after the end of CHAPS." To achieve this objective, three major outputs were planned for the PVO-DHMT partnership: (1) bi-annual meetings of the District Health Board (DHB), (2) quarterly meetings between DHMT and Project HOPE to review progress and objectives, and (3) training of DHMT in management. Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 10 2.1 How was the partnership set up? What were the roles and responsibilities of DHO? Of Project Hope? • How were decisions made about program choices? About allocating funds and budget priorities? About training? • How did communication occur between Project Hope and the DHO – for day to day issues? For bigger decisions? For problem solving? 2.2 What were the benefits of working in partnership? 2.3 What were the biggest challenges of working in partnership? How were these challenges resolved? 2.4 How could the partnership be improved? 2.5 Overall, was the objective of the PVO-DHMT partnership (a) substantially achieved, (b) partially achieved, or (c) negligibly achieved and why? 3.0 Project Implementation and Management Overall, most of this has been covered with your individual surveys, (so it is really important that we get them). However, as a supplement, we would just like to give you the opportunity to further explain how these aspects worked. 3.1 Overall, how did management of the project work? (Highly satisfactory, satisfactory, poorly) Why? (How might it have been without CHAPS?) 3.2 Overall, how did procurement and logistics management for the project work? (Highly satisfactory, satisfactory, poorly) Why? (How might it have been without CHAPS?) 3.3 Overall, how did project training work? (Highly satisfactory, satisfactory, poorly) Why? (How might it have been without CHAPS?) 3.4 Overall, how did supervision and monitoring in the project work? (Highly satisfactory, satisfactory, poorly) Why? (How might it have been without CHAPS?) 3.5 Overall, how did the HIS work : information collection, reporting, and use for decision making work? (Highly satisfactory, satisfactory, poorly) Why? (How might it have been without CHAPS?) 3.6 How did the CHAPS partnership coordinate with other health activities and organizations in the District? (How might it have been without CHAPS?) 4.0 Future Plans 4.1 During that past few months and up to now, what are the main CHAPS inputs in the District? 4.2 What are the biggest gaps in services and support that are foreseen with the finishing of CHAPS? Which of these gaps would be the highest priority and why? 4.3 What discussions have taken place to plan for these gaps? What alternatives are being pursued. 5.0 Lessons Learned – Recommendations for Elsewhere Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 11 5. DHMT Individual Survey Questionnaire Position:___________________________ District: Mulanje_______ (note – these will only be seen by evaluators) Phalombe_________ CHAPS Project Goal and Objectives: The goal of the CHAPS project was "to improve health care services in the District of Mulanje through a partnership between Project HOPE and all levels of the health infrastructure in the Mulanje District [the Phalombe District was subsequently included], involving also other private sector and NGO health providers. This partnership will be the vehicle for (1) improving Maternal Child Health (MCH) services and Reproductive Health (RH) service delivery; (2) building human and institutional capacity in a sustainable manner; and (3) assisting USAID in achieving its Strategic Objective No. 3 in Malawi." Strategic Objective No. 3 is "behaviors adopted that reduce fertility and risk of HIV/AIDS and improve child health." The overall objectives were "to (1) improve health status of the population of the Mulanje District, with special emphasis on MCH and RH program activities and (2) improve the management capacity of the DHMT and other health personnel to support priority health interventions within the district in a sustainable manner." 1. Overall Evaluation of the Project 1.1 How would you evaluate the project overall? Why? (a) highly satisfactory - project achieved or exceeded its major objectives, without major shortcomings. (b) Satisfactory - Project achieved most of its major objectives, with only a few shortcomings. (c) Unsatisfactory - Project failed to achieve most of its major objectives. 1.2 What do you think the project really tried to do? 1.3 What did it contribute to the community? 2. Achievement of USAID Performance Indicators (Key Intermediate Results-KIR) The project aimed to achieve had three Key Intermediate Results. How would you assess the achievement of each of the performance indicators? Why? KIR 3.2. Knowledge and use of good RH/MCH practices improved (a.) Substantially achieved WHY? (b.) Partially achieved (c.) Negligible KIR 3.3 Community participation in RH/MCH health care improved (a.) Substantially achieved WHY? (b.) Partially achieved (c.) Negligible KIR 3.4 Expanded range and quality of RH/MCH services for women, men and youth. (a.) Substantially achieved WHY? (b.)Partially achieved (c.) Negligible Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 12 3. Training 8.1 How would you assess the training carried out by the project? Why? Training Topic HS S U Reason Infection Prevention VCT Home-based Care MTCT/Infant Feeding STIs Malaria CDD ARI IMCI Nutrition FP Safe Motherhood Overall Communication and counseling Health Information System (HIS) DHMT Management Training Youth Initiators HS = highly satisfactory S = satisfactory U = unsatisfactory 4. Management and System Strengthening 4.1 How would you assess the project's activities for strengthening these management functions? Why? VS = very satisfactory S = satisfactory P = poor Overall Project performance/area System Strengthening What was done VS S P Reason personnel management e.g. Build houses, career ladder, etc. Coordination with all health providers Supervision and supervisory skills How often have you been out to do supervision during the past 6 months?____________ Program Planning and Development of Work Plans quality monitoring & assessment Development of new technical guidelines and policies – including follow￾up and enforcement (MTCT, VCT, pharmacy management) HMIS- Data available for Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 13 Overall Project performance/area System Strengthening What was done VS S P Reason timely and useful reports Use of Data for program planning and management Availability and use of computers Monitoring and Evaluation System Logistics management at central level – Supplies and transport Central Pharmacy management HC contribution to adequate logistics (ordering, storage) Fleet management – Transport District Communication Systems Budget management and accounting Institution of cost recovery systems DRF, private ward, etc. Overall Project Management Performance 4.2 How would you assess overall project management performance and why? District Health Office (a.) Highly satisfactory WHY? (b.) Satisfactory (c.) Unsatisfactory Project Hope (d.) Highly satisfactory WHY? (e.) Satisfactory (f.) Unsatisfactory 5. How would you assess the overall contribution (or not) of these different players to the successful implementation of CHAPS in your District? (mark one box – highly satisfactory / satisfactory / unsatisfactory and explain ) HS = highly satisfactory S = satisfactory U = unsatisfactory Level HS S U Reason Central MOHP Extended DHMT DHO Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 14 Level HS S U Reason District Health Board DACC Health Center Community Volunteers Project HOPE Other Partners USAID 6. Studies How would you assess usefulness of the studies included in the project? Why? Study Topic Very useful Somewhat useful Not useful Reason Health Facility Assessment – Baseline Household KPC Baseline Survey Formative Research on MTCT, and infant feeding “Exploring Social-Cultural Barriers to Behavior Change among the Traditional Initiators on Practices that Promote HIV/AIDS Transmission,” by Evelyn B. Chilemba, June 2002 Mid-term Evaluation by Chrissie P.N. Kaponda, April￾May 2000 Report of the evaluation of the Community Health Partnership Project, 3/02 (USAID overall CHAPS eval.) 7. Project Sustainability How would you assess the likelihood that project achievements can be sustained at the conclusion funding? Why? Sustainability of Achievements Likely Not likely Don’t know Reason Changes in DHMT management Changes in DHMT planning Changes in DHMT training Changes in DHMT service delivery activities Changes in provider practices Changes in community behavior in health Changes in community participation in health care 8. Lessons Learned Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 15 6. Guide for Health Center Visits OBSERVATIONS Item Project Activity E S P Reason Clinic Hygiene (infection control protocols) Waste Disposal/ Incinerator (chlorine, kerosene, plastic buckets, soap and gloves available) Latrine (clean and used) Water VCT supplies (hospitals) Counseling rooms (if appl) Baby Friendly? (maternities) Patient Flow (# and organization) E = excellent, S = satisfactory, P = poor STAFFING Position Actual Intended Clinical Officer Medical Assistant Nurse HSA Auxiliary staff (who) STAFF INTERVIEWS 1. What do you see as your greatest strengths in providing health services in this area? (What are you proudest of?) Why? 2. What are your greatest challenges? (What do you worry about when you think about work?) Why? 3. What assistance have your received from CHAPS? (over the past 5 years) 4. Did this assistance make any difference in your services? How? 5. Did this assistance make any difference in how you feel about your job? How? 6. What are ways you see people’s behavior having changed over the past several years? Their appreciation of your services? Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 16 The overall objectives of CHAPS were "to (1) improve health status of the population of the Mulanje District, with special emphasis on MCH and RH program activities and (2) improve the management capacity of the DHMT and other health personnel to support priority health interventions within the district in a sustainable manner."50 1. Overall Evaluation of the Project 1.1 How would you evaluate the project overall? Why? (a) highly satisfactory - project achieved or exceeded its major objectives, without major shortcomings. (b) Satisfactory - Project achieved most of its major objectives, with only a few shortcomings. (c) Unsatisfactory - Project failed to achieve most of its major objectives. 1.2 What do you think the project really tried to do? 1.3 What did it contribute to the community? 1.4 2. Achievement of USAID Performance Indicators (Key Intermediate Results-KIR) The project aimed to achieve had three Key Intermediate Results: KIR 3.2. Knowledge and use of good RH/MCH practices improved KIR 3.3 Community participation in RH/MCH improved KIR 3.4 Range and quality of RH/MCH services for women, men and youth expanded 2.1 How would you assess the achievement of each of the performance indicators? Why? Ratings: Substantially achieved Partially achieved Negligible MANAGEMENT 3.0 What are the greatest challenges to effective management in this health center? What, if anything, has the project done to address these? TOPIC ACTIVITIES (underline those specific to CHAPS) EFFECTIVE? HS / S / U REASON Supervision to Community (what approach and training? How often done for each community in past 6 mos., overall relationship with community workers, level of comfort with HBC, #HBC kits distributed, # youth clubs supervised) Supervision from District (How often done in past 6 mos, what discussed.) Maintaining supplies and (CHAPS transport for drugs and supervision, how does supply/order system work) Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 17 TOPIC ACTIVITIES (underline those specific to CHAPS) EFFECTIVE? HS / S / U REASON medicines Maintaining staffing (turnover, work load, recruitment, work load – particularly HSAs) Work Planning (how developed, monitored, progress against plan) Budgeting (hospitals) (how allocated, monitored) Cost Recovery (hospitals, CHAM) (advantages vs. constraints, how managed) HIS Data collection and USE, # meetings held to discuss data) Referral system (community – HC – hospital - Bicycle ambulance, communication, transport, examples of recent referrals) For these management and support systems, what are CHAPS CURRENT inputs and support? What, if any, discussion has there been about what will happen when CHAPS finishes? TECHNICAL STRENGTHS TRAINING (indicate number of staff trained for each cadre) Topic CO/MA Nurse HSA W/in last year? Malaria EBF/infant feeding VCT/MTCT IMCI STI management Management Cervical cancer Quality Assurance Other What QI activities have taken place in the past year? How were they identified/selected? What were the results? CASE MANAGEMENT Disease Protocol Available? Used? Knowledge of management Comments Sick child 2 mos. – 5 years (IMCI protocol incl. feeding, EPI, distinguish fever from ARI) Sick child under 2 IMCI protocol, EBF, EPI, Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 18 Disease Protocol Available? Used? Knowledge of management Comments mos. MTCT, difference between babies and older STI Syndromic management, partner notification, % supervised Malaria (adult) Education, SP, VCT Lab/counseling, infection control Confidentiality, counseling approach, referral practices, partner notification, education and outreach strategies, % with TB test, % counseled on feeding Delay for results, % of pts. Getting results MTCT Infant feeding, FP, ART, C￾sections, cooking demo. Integration with ANC, % of ANC pts. Seeking VCT, PARTNERSHIP MOHP/HOPE How did you see the relative roles and responsibilities of HOPE and the MOHP? Did this division of responsibilities work well? Why or why not? MEDICINE AVAILABILITY Medicine Stock out in last 6 mos. Y/N Why? Why not? STI drugs (Gentamycin, Doxicycline, Erythromycin, Benz. Pen., Metronidazole) Depo / pills Condoms Cotrimoxazole Fansidar ORS HBC kits # distributed PATIENT EXIT INTERVIEW (age of pt.: _________) 1. What brought you to the clinic today? 2. What did the provider do when s/he “DO” for the consultation? Exam etc. 3. What did the provider tell you? Diagnosis, explanation, instructions, danger signs, etc. 4. What, if any, medicine were you given ? 5. What, if any, suggestions might you have to make these services better? Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 19 7. Interview Guide for Project HOPE CHAPS Project Goal and Objectives: The goal of the CHAPS project was "to improve health care services in the District of Mulanje through a partnership between Project HOPE and all levels of the health infrastructure in the Mulanje District [the Phalombe District was subsequently included], involving also other private sector and NGO health providers. This partnership will be the vehicle for (1) improving Maternal Child Health (MCH) services and Reproductive Health (RH) service delivery; (2) building human and institutional capacity in a sustainable manner; and (3) assisting USAID in achieving its Strategic Objective No. 3 in Malawi."51 Strategic Objective No. 3 is "behaviors adopted that reduce fertility and risk of HIV/AIDS and improve child health." The overall objectives were "to (1) improve health status of the population of the Mulanje District, with special emphasis on MCH and RH program activities and (2) improve the management capacity of the DHMT and other health personnel to support priority health interventions within the district in a sustainable manner."52 1. Overall Evaluation of the Project 1.1 How would you evaluate the project overall? (a) highly satisfactory, (b) satisfactory, or (c) unsatisfactory? Why? 1.2 What do you think the project really tried to do? 1.3 What did it contribute to the community? Ratings: Highly satisfactory = Project achieved or exceeded its major objectives, without major shortcomings. Satisfactory = Project achieved most of its major objectives, with only a few shortcomings. Unsatisfactory = Project failed to achieve most of its major objectives. 2. Achievement of USAID Performance Indicators (Key Intermediate Results-KIR) The project aimed to achieve had three Key Intermediate Results: KIR 3.2 Knowledge and use of good reproductive health practices improved KIR 3.3 Community participation in health care improved KIR 3.4 Range and quality of reproductive health services for women, men and youth expanded 2.1 How would you assess the achievement of each of the performance indicators? Why? Ratings: Substantially achieved Partially achieved Negligible 51 Project HOPE. Technical Proposal, op. cit., p. 5 52 Ibid., p. 11 Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 20 3. Assessment of major factors that affected the achievement of project objectives and the degree to which they did so (substantially, partially, or negligibly). Ratings: Substantially affected the achievement of project objectives Partially affected the achievement of project objectives Negligibly affected the achievement of project objectives Did any of the following factors affect the achievement of project objectives? Factors divided into 3.1 Factors not generally subject to government control, such as changes in world markets and prices, natural disasters, and war and civil disturbance. 3.2 Factors generally subject to government control, such as macroeconomic policies and conditions, sector policies, and administrative procedures, including factors subject to MOHP central-level and DHMT control. 3.3 Factors generally subject to Project HOPE Center control 3.4 Factors generally subject to Project HOPE/Malawi control, such as management effectiveness, staffing, cost changes, implementation delays, use of technical assistance, adequacy of monitoring and evaluation, and client/beneficiary participation. 4. Assessment of Project Design 4.1 Was the design of the project appropriate for achieving the project objectives? Why or why not? 4.2 Were the criteria for judging the achievement of project objectives adequately quantified in the Technical Proposal? 5. Implementation Delays 5.1 Were there implementation delays? If yes, what were the major reasons for the delays? For example, unrealistic implementation schedule; unexpected technical difficulties; changes in project scope; quality of management; delays in selecting staff; delays in receiving gifts in kind or other commodities; inefficient procurement or disbursement procedures; security problems; natural disasters; and others. 6. Project Management Performance 6.1 How would you assess project management performance and why? Project HOPE Center Project HOPE Malawi USAID DHMT CHAPS Project Other Partners Ratings: Highly satisfactory = Project management performed its job with minor or no problems. Satisfactory = Despite some shortcomings, project management did its job with only moderate problems. Unsatisfactory = There were major deficiencies in project management resulting in Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 21 significant delays in project implementation and/or achievement or project objectives. 7. Provision of Supplies 7.1 How would you assess the provision of supplies (GIK and other)? Why? Project HOPE Center Project HOPE Malawi CHAPS Project Ratings: Highly satisfactory = Procurement was carried out in a timely and efficient manner, and commodities (supplies and equipment) procured met the needs of the project. Satisfactory = Some difficulties were experienced in managing procurement, e.g., moderate delays. Unsatisfactory = Major difficulties were experienced in carrying out procurement, leading to excessive implementation delays and procurement of commodities that did not meet the needs of the project. 8. Training 8.1 How would you assess the training carried out by the project? Why? Ratings: Highly satisfactory = All training proceeded in a timely and satisfactory manner. Satisfactory = There were some problems in making timely and/or suitable arrangements for training, or in carrying out training, but final training results met planned targets. Unsatisfactory = Some or all training experienced major problems, and there was uncertainty whether the final training results would meet planned targets. Infection Prevention Voluntary Counseling and Testing (VCT) Home-based Care Infant Feeding Sexually Transmitted Infections (STIs) Malaria Control of Diarrheal Diseases Acute Respiratory Infections (ARI) Integrated Management of Childhood Illnesses (IMCI) Nutrition Family Planning (FP) Safe Motherhood HIV/AIDS Health Information System (HIS) DHMT Management Training Youth Initiators Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 22 9. Studies 9.1 How would you assess usefulness of the studies included in the project? Why? Ratings: Highly satisfactory = All project studies proceeded in a timely and satisfactory manner. Satisfactory = Some or all project studies experienced some problems (preparation and/or agreement on Terms of Reference, staffing, conduct of field worked and/or timeliness of other required actions). Unsatisfactory = Some or all studies experienced major problems and may not meet expectations. Health Facility Assessment (baseline) Household KPC Baseline Survey Customer Surveys Impact of provider counseling on breastfeeding practices assessed with support from Linkages Collection of baseline information on MTCT, VCT and infant feeding Assessment of barriers to partner referral Assessment of quality and compliance with shopkeepers' advice (re malaria) Impact of health education campaigns (drama and music competitions, health education), Importance of participating in youth clubs to risk-reduction practices Development and testing through OR of novel supervision approaches that can be sustained by the district (e.g., peer supervision) KAPB Survey at end of project Similar survey conducted with a representative sample of health providers Health Facility Assessment at end of project "Exploring Social-Cultural Barriers to Behavior Change among the Traditional Initiators on Practices that Promote HIV/AIDS Transmission," by Evelyn B. Chilemba, June 2002 Final Quality Assurance Assessment Mid-term Evaluation by Chrissie P.N. Kaponda, April-May 2000 Report of the evaluation of the Community Health Partnerships Project by LTG Associates, Inc. and TvT Associates, March 2002 10. Project Monitoring/Supervision and Evaluation 10.1 How would you assess project monitoring/supervision and why? The evaluation of project monitoring and supervision performance should take into account: (a) the quantity and quality of monitoring/supervision, (b) whether project progress was adequately reported, (c) early and precise identification of implementation problems, (d) clear definition of actions required to overcome implementation problems, (e) ability to get appropriate actions taken, (f) the adequacy and timing of monitoring and supervision visits (e.g., actual versus recommended intervals between visits), and (g) how successfully Project HOPE Center, Project HOPE Malawi, CHAPS Project, DHMT, USAID, and other participants worked together. 10.2 What was the frequency of monitoring/supervision? 10.3 Was the Health Information System (HIS) adequate to meet the information requirements at different levels? Why or why not? 10.4 Were the project's reporting requirements met in a timely and satisfactory manner? 11. Implementation Performance Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 23 11.1 How would you assess the project's implementation performance? Why? Elaborate on the reasons for the evaluation rating such as the quantity and quality of management and staff, the performance of partners, the level and timeliness of Project HOPE's contribution and USAID's contribution to the project, and the adequacy of project monitoring and evaluation, among others. MOHP DHMT DHO District Health Board Hospital Health Center Community Volunteers Project HOPE Other Partners USAID 12. Partnership between DHMT and Private Voluntary Organizations (PVOs) The stated objective of the PVO-DHMT partnership was to "develop a collaborative working relationship with all partners and to assure that district structures can sustain program planning, management, and service delivery activities after the end of CHAPS."53 To achieve this objective, three major outputs were planned for the PVO-DHMT partnership: (1) bi-annual meetings of the District Health Board (DHB), (2) quarterly meetings between DHMT and Project HOPE to review progress and objectives, and (3) training of DHMT in management. 12.1 Was the objective of the PVO-DHMT partnership (a) substantially achieved, (b) partially achieved, or (c) negligibly achieved and why? 12.2 Did the District Health Board meet bi-annually during the life of the project? Why or why not? 12.3 Did the DHMT and Project HOPE meet on a quarterly basis during the life of the project? Why or why not? 12.4 Was all of the planned management training for DHMT conducted? Why or why not? 12.5 How would you assess DHMT management training? Why? personnel management cost reduction/recovery resource management planning services coordination communication & counseling data utilization computer inventory quality monitoring & assessment 53 Project HOPE, Technical Proposal, op. cit., p. 12g Evaluation Instruments March 14, 2003 CHAPS Project Final Evaluation 24 finance & accounting Health Information policy enforcement supervision & supervisory skills computerized HIS quality awareness computer 12.6 What was the biggest challenge of working in partnership? 12.7 What are the benefits of the partnership? 12.8 Did the partnership experience any problems? If yes, what were the problems? Were the problems resolved? 12.9 How could the partnership be improved? 12.10 What should not be attempted again? 12.11 Could the partnership be replicated across the across the health sector as a whole? Why or why not? 13. Project Sustainability 13.1 How would you assess the likelihood that project achievements can be sustained at the conclusion funding? Why? Changes in DHMT management Changes in DHMT planning Changes in DHMT training Changes in DHMT service delivery activities Changes in provider practices Changes in community behavior in health Changes in community participation in health care 14. Lessons Learned