September 28, 2012 This Report was produced at the request of the United States Agency for International Development/Regional Development Mission Asia (USAID/RDMA). Evaluation of the Malaria Control in Cambodia Project Final Report Contract No. GHS-I-00-07-00010-00 Evaluation of the Malaria Control in Cambodia Project Final Report ACKNOWLEDGEMENTS The Evaluation Team received outstanding support from both the USAID/RDMA Office and the staff of CAP-Malaria, the follow-on project to the Malaria Control in Cambodia Project (MCC). Dr. Kheang Soy Ty, Chief of Party of both MCC and CapMalaria, gave generously of his time, and the time of his staff, to interpret the work of MCC and that support was greatly appreciated. Special thanks go to Ms. Lina Kharn who facilitated a very informative visit to the Project area and accompanied the Team for the entire two weeks of interviews with informants. We also wish to express our appreciation for our driver, Mr. Kao Vuty, who traversed many tricky bridges and road hazards. We also are grateful to staff of the National Malaria Control Program (NMCP) and partner agencies who so generously gave their time to provide us with insights about the MCC Project. Finally, we thank and salute the several Village Malaria Workers (VMWs) in the Project areas who are the real heroes of the work being done in their communities. They are the “lynchpins” that connect the national, provincial and operational district levels of the NMCP with the people. Neil Brenden and Nigoon Jitthai Independent Consultants Disclaimer: The contents of the following report are the responsibility of the authors and represent our best efforts to complete the Statement of Work given. The opinions expressed herein are those of the authors only. CONTENTS ACKNOWLEDGEMENTS ACRONYMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 I. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.1. Purpose of the Evaluation and Summary of the Statement of Work 1.2. Timeline and Methodology 1.3. Limitations 1.4. Evaluation Team 2. CONTEXT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1. Malaria in Cambodia 2.2. Diminishing Efficacy of Artemisinin-based Combinaiton Therapy 2.3. U.S. Government Malaria Initiatives 2.4. The Malaria Control Structure of Cambodia 2.5. The Malaria Control in Cambodia Project (MCC) 3. KEY FINDINGS FROM AN ANALYSIS OF PROJECT DESIGN AND IMPLEMENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1. Prevention 3.2. Community Mobilization 3.3. Diagnosis and Treatment 3.4. Capacity Building 3.5. Project Management 3.6. Monitoring and Evaluation 3.7. Private Sector 3.8. Operation Research 4. GENDER CONSIDERATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5. CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6. RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 i ii ANNEXES Annex I: MCC Components, Outputs, Outcomes and Achievements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Annex II: Analysis of MCC Intervention Models . . . . . . . . . . . . . . . . .41 1. Engagement of VMWs 2. LLIN lending scheme for MMPs 3. Involving taxi drivers as health nediators 4. Public-private partnership 5. Day-3 positive surveillance and mapping of cases Annex III: Recommended New Intervention Models . . . . . . . . . . . . . 48 1. Telephone hotline/consultation 2. Participatory LLIN lending scheme 3. More meaningful involvement of volunteers 4. Friendly public health service 5. Addressing malaria among MMP within a safe mobility context 6. Vector control measures Annex IV: Statement of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 1. Statement of need and audience for the evaluation 2. Evaluation type, purpose and key questions 3. Evaluation methodology 4. Gender consideration 5. Deliverables Annex V: Evaluation Design and Methods . . . . . . . . . . . . . . . . . . . . . . .53 1. Evaluation design 2. Evaluation methods Annex VI: Data Collection Instruments . . . . . . . . . . . . . . . . . . . . . . . . 56 1. Interview guide for health service providers 2. Interview guide for community members/patients 3. Interview guide for health service providers Annex VII: Sources of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 1. List of persons interviewed 2. Documents reviewed Annex VIII: Disclosure of Any Conflicts of Interests . . . . . . . . . . . . . . . 67 iii Acronyms ACT Artemisinin‐based Combination Therapy AOP Annual Operational Plan BCC Behavior Change Communication CNM Cambodia National Centre for Parasitology, Entomology and Malaria Control DOT Directly Observed Therapy EDAT Early Diagnosis and Appropriate Treatment GMP Global Malaria Programme of the World Health Organization, IEC Information, Education and Communication ITN Insecticide‐Treated Net KHR Cambodian Riel LLIN Long-Lasting Insecticidal-treated Nets LMI Lower Mekong Initiative M&E Monitoring and Evaluation MCC Malaria Control in Cambodia Project MMP Mobile and Migrant Populations NGO Non-governmental Organization NMCP National Malaria Control Program OD Operational Districts PfD Partners for Development PMI President’s Malaria Initiative PPM Private‐Public Mix QA Quality Assurance QC Quality Control RDT Rapid Diagnostic Test URC University Research Co. LLC USAID/RDMA United States Agency for International Development/Regional Development Mission Asia US $ United States Dollar VHV Village Health Volunteer VMW Village Malaria Worker WHO World Health Organization Page 1 EXECUTIVE SUMMARY BACKGROUND Malaria continues to be a serious public health threat in Cambodia, exacerbated by the development of resistance to artemisinin combination therapy (ACT) by P. falciparum. It is estimated that about 15% of the population are at risk with about 500,000 persons living in endemic areas of high malaria transmission. The National Malaria Control Program (NMCP) provides leadership to the malaria programs in Cambodia in close collaboration with both local and international partners. The Malaria Control in Cambodia Project (MCC) was a project of USAID/Regional Development Mission Asia managed by University Research Co. LLC (URC), with assistance from a subcontractor, Partners for Development (PfD). This 4 year Project had a budget of US $3,550,000. Designed to enhance the NMCP activities, the MCC provided extensive technical assistance in addition to financial and in-kind support to the public health system to improve diagnosis and treatment. In addition, MCC educated the communities about malaria prevention, control and appropriate health seeking behaviors in 5 Operational Districts in four border provinces adjacent to Thailand: Banteay Meanchey, Battambang, Oddar Meanchey and Pailin. EVALUATION PURPOSE AND KEY QUESTIONS The main purpose of this evaluation is to assess the performance of the MCC Project and its impact. The evaluation is intended to provide insights and important feedback to each of the partners and stakeholders that should assist them to understand both the strengths and areas where technical, administrative and management efforts could be improved. It will also provide information for future malaria control programs and projects so they can learn from the MCC experience. The following four general evaluation questions provided the framework for the evaluation: 1. What did the project achieve against the expected results and outcomes as stated in the contract and what is the evidence of project coverage, effectiveness, efficiency, impact and sustainability? 2. Where did the implementation fall short and what were key constraints? 3. What major lessons learned and good practices emerged from the project? 4. What are potential considerations for follow-on programs with similar objectives? METHODOLOGY AND TIMELINE The evaluation was conducted by a team of two international consultants between July and September 2012, about 8 months after the Project was completed. The Team conducted a desk review of relevant materials and documents to form the evaluation design, methods, and tools. Two weeks of field visits to the Project sites in all four provinces were conducted and the qualitative data obtained from the field visits were analyzed and triangulated with the quantitative data derived from desk reviews. Two debriefings were conducted among the implementers, partners, and the donor both in Cambodia and in Thailand for validation of the findings and obtaining additional inputs. A draft report was shared with staff of both the donor and the implementing agency and their comments were incorporated into the final report. LIMITATIONS The following limitations of this evaluation were identified: 1) limited institutional memory due to staff turnover and because the follow-on project, CAP-Malaria, was already being implemented in the same areas so staff may have found it difficult to distinguish between the two projects; 2) the quality of the Project’s performance data was poor, implementation targets were not routinely established, and there Page 2 were discrepancies and inconsistencies in performance records; and 3) the lack of an evaluation team member from Cambodia who could provide valuable local knowledge and awareness of the context in which MCC functioned. KEY FINDINGS 1. What did the project achieve against the expected results and outcomes as stated in the contract? The task order for the Project specified the tasks with which the Project was to be engaged, but did not set targets by which to measure accomplishments. The tasks and subtasks were: 1. Diagnosis – Upgrade microscopy and rapid diagnostic tests; 2. Treatment – Train staff on appropriate treatment protocols and drug management; 3. Behavior change communication (BCC) and information, education and communication (IEC) for malaria – Support malaria advocacy, prevention and control; and 4. Capacity building through collaboration – Strengthen in-country capacity and foster collaboration. Achievements: 1.Prevention  MCC had an extensive IEC/BCC program that was supported by the design and production of many communication materials.  The village health volunteers (VHVs) reached individuals and small groups by conducting formal education sessions while the village malaria workers (VMWs) contacted families while providing rapid diagnostic tests and occasional home visits.  MCC conducted community-wide campaigns through participating in World Malaria Week and mass media.  MCC promoted the use of bed nets through their BCC messages and the distribution of ITNs/LLINs. 2. Capacity Building  VMWs provided quality diagnostic and treatment services in all of the villages in the target area.  The VHVs focused on health education through structured training events augmented by supportive visits.  Thirty-five taxi drivers participated in the Project by providing basic information about malaria prevention, diagnosis and treatment to MMPs while traveling by a taxi.  School health malaria curricula were developed and teachers were trained. 3. Diagnosis and Treatment  Presumptive treatment was eliminated in the public health system.  The number of severe cases was reduced in the project area facilities and, concurrently, the percentage of severe case admitted to the facilities increased.  Compliance with the national treatment guidelines was high. 4. Public Sector  MCC contributed to the improved capacity of public sector staff and volunteers for management, laboratory skills, research and patient care at the provincial, OD and village levels 5. Management  The Project staff developed collaborative relations with colleague agencies and the donor and were experienced in the policies and procedures of the NMCP so they understood their role in it.  Clear lines of authority and responsibility allowed for staff being utilized flexibly according to the needs of the program.  The Project adapted to the needs of the NMCP in the target areas. Page 3  MCC advocated for more responsive policies at both the national and local levels  M&E Plans and Annual Reports were prepared regularly. 6. Monitoring and Evaluation  Data collection was coordinated with local health facilities.  Baseline and endline surveys were conducted. 7. Private Sector  The PPM Pilot Project was implemented in Pailin Province  A referral system was designed to assure appropriate treatment. 8. Operations Research  MCC collaborated with the NMCP on the surveillance of drug-resistant malaria.  The quality of RDT used in communities and in health facilities was assessed in two studies.  Multi-drug resistant malaria cases were located and tested. Evidence of Project coverage:  For years 1-3 there were 290,344 persons in the five targeted ODs. In year 4 two additional ODs were added, increasing the population of the target area to 548,442, More than 125,000 persons in the Project area received malaria prevention and treatment education from the VMWs and village health volunteers (VHVs) each year of the Project. Evidence of effectiveness:  The VMWs increasingly were accepted by the community for the diagnosis and treatment of their malaria symptoms. In 2009, when the VMWs were first fully functioning in the Project area, their contribution to the treatment of malaria cases in the government programs accounted for only 11% (850/7,718) of all treated cases. Their contribution was brought up to almost 30% (4,593/15,569) in 2010 and to over 57% (5,564/9,659) in 2011.  Correct management of complicated malaria cases increased from 69% to 98% in the facilities supported by MCC. Evidence of efficiency:  MCC was a technical assistance, gap-filling project and it is difficult to calculate meaningful costs per unit of service because the interventions were incorporated into the public malaria control programs and budgets in the target ODs.  A report compiled for MCC by a consultant estimated the following annual direct activity costs: o US$ 313 for support of one VMW; o US$ 10 for support of each employer loaning bed nets (excluding the cost of nets); o US$ 423 for enrollment, follow-up and mapping of each potential drug resistant case. Evidence of impact:  Incidence of confirmed cases of malaria in the targeted ODs decreased from 21.6 to 2.5 per 1,000 population while reported malaria mortality was reduced from 4.7 to nearly 0 per 100,000 population in the Project ODs. Evidence of sustainability: As a technical assistance project supporting the NMCP, sustainability issues cannot be addressed without considering the entire public sector program in the target areas. However, individual components of the Project can be assessed for sustainability.  The taxi driver initiative could be continued as part of the public program at little cost.  The VMWs are currently considered an extension of the health centers and this component could easily be sustained.  The VHVs could be sustained as part of the health education strategy. Page 4  The bed-net lending scheme could be sustained but funding would need to be assured.  The private sector is an essential part of a comprehensive malaria control program but the nature of the incentives would need to determined for each situation. 2. Where did the implementation fall short and what were key constraints?  Shortages of critical supplies for the work of the VHWs continued throughout the life of the Project. Although the supply chain was the direct responsibility of the NMCP, MCC’s contract notes that it should ensure uninterrupted supplies of good-quality anti-malarial drugs and diagnostic tests.  A consultant report accurately described the MCC as a “gap filling” project1, which meant that it reacted to the expressed needs of the NMCP in the target ODs. This function was activated during the development of the annual operational plans, when MCC responded to specific requests for technical assistance and commodity procurements. Thus, a constraint for determining if implementation was satisfactory was that there were no standards by which to determine if its work was above or below expectations. 3. What major lessons learned and good practices emerged from the project?  Reaching the groups at highest risk for malaria infection increases the impact of a project.  With relatively modest inputs and support, the capacity of local people who are carefully selected can be enhanced to provide essential services to their communities.  MCC consistently conducted formative assessments to learn about the needs and aspirations of each target group before designing their prevention interventions.  A project or program should only invest in collecting data that is going to be of value to the management and/or for accountability to the donor.  The private sector needs to be engaged in malaria control efforts if such initiatives are to succeed. The type of involvement and the incentives need to be worked out in each situation.  Capacity building is an essential component of every project.  Training should be followed by periodic refresher courses and regular supervision at which newly acquired skills are reinforced. 4. What are potential considerations for follow-on programs with similar objectives?  The VMW approach whereby villagers are selected through a community process and trained by the health centers to provide quality health education, diagnosis with rapid tests and treatment with appropriate drugs proved to be an effective mode of health service delivery.  Public sector services can be strengthened by NGOs if they function within a public system of prevention, diagnosis and treatment with clearly defined roles and responsibilities.  Existing resources and structures should be utilized to the extent possible. For example, the Net-Lending Scheme used the existing relationship between laborers and owners to increase the availability and use of ITNs/LLINs. 1 Development Business Associates, Assessment of the Malaaria Control in Cambodia Project – Report to University Research Co LLC ( April 2011), 1. Page 5 KEY RECOMENDATIONS The recommendations at the end of this Final Report, highlighted below, are intended to be general guidelines for the development of malaria control interventions in the future and are not intended for a specific project.  Program design is to be evidence-based;  Supervision and support of frontline workers is a high priority;  Evaluation methods are shaped by the interventions;  Targeting of those at highest risk increases impact;  Monitor the changing situation;  Use data at all levels;  Be innovative in designing interventions;  Disaggregate data by significant groups;  Develop clear objectives and a robust M&E framework;  Implement a targeted BCC;  Implement strategic advocacy and community mobilization;  Donor harmonization is needed;  Health volunteers working in the community should be coordinated in relation to the needs of the people rather than the requirements of the donors.  Controlling resistant strains needs to be a priority. Page 6 I. INTRODUCTION 1. Purpose of the Evaluation and Summary of the Statement of Work This is an end-of-project, performance evaluation. It was designed to assess performance and impact of the Malaria Control in Cambodia (MCC) Project that was implemented by the University Research Co. LLC (URC) and its partners between October 2007 and November 2011. The following four general evaluation questions provided the framework for the evaluation: 1. What did the project achieve against the expected results and outcomes as stated in the contract and what is the evidence of project coverage, effectiveness, efficiency, impact and sustainability? 2. Where did the implementation fall short and what were key constraints? 3. What major lessons learned and good practices emerged from the project? 4. What are potential considerations for follow-on programs with similar objectives? In addition, the following specific evaluation questions were addressed: 1. How many beneficiaries were reached and what were the differences the project has made to the beneficiaries, as perceived by beneficiaries and other project participants? 2. Is there evidence of improved accuracy of malaria diagnosis, adherence to treatment guidelines, and detection of treatment failures? 3. Is there evidence of impact from this project’s activities, including reductions in malaria morbidity or mortality, or hospitalizations? 4. Is there evidence of potential replicability of the project elsewhere? 5. What were the major factors which influenced the achievement or non-achievement of sustainability of the project? To what extent did the benefits of the project continue after our funding ceased? Were there any activities undertaken to assure sustainability? 6. Has there been evidence of improved knowledge or practices of community members on malaria? 7. Has management for malaria control at district and provincial health systems been strengthened i.e. improved quality of services, increased capacity of health care providers, etc.? Please refer to Annex IV for a more complete description of the statement of work. 2. Timeline and Methodology This evaluation was commissioned by the United States Agency for International Development/Regional Development Mission Asia (USAID/RDMA) and was conducted over a two month period from July 28 to September 28, 2012. The overall evaluation schedule can be found in Annex V. The Evaluation Team employed both quantitative and qualitative methods for this evaluation. The quantitative methods were used mainly to analyze the secondary data related to the Project’s implementation. The determination of key constraints and challenges contributing to the Project’s shortfalls was qualitative in nature, and the data were obtained through primary data collection during the field visits to the Project’s sites. The main tasks for this evaluation are as described below: a) Desk review of relevant materials: The Team first obtained available secondary data from the Project’s workplans, reports, and special studies to gain a better understanding on the MCC’s Page 7 goals, objectives, and interventions as well as to analyze preliminary findings on implementation results. The Team also reviewed additional relevant documents. b) Development of the evaluation design and methods: Following the initial desk review, the Team developed the evaluation design and field data collection instruments for various types of informants (see Annex VI), plans for field visits and the selection of informants in consultation with USAID/RDMA. c) Field data collection: In addition to the field visits to the Project’s sites in Cambodia, the Team also conducted face-to-face and telephone interviews with key regional stakeholders based in Thailand and in other countries, both prior to and following the field visits. The field visits to Cambodia involved interviews with relevant government and non-government informants at both central and peripheral levels, as well as the targeted community members. d) Data analysis and verification: The quantitative data extracted from Project’s reports were analyzed in excel spreadsheets and qualitative data obtained from the field visits were analyzed using the grounded theory and triangulated with the quantitative data. e) Debriefing: An Aide Memoire was prepared and was the basis for a debriefing session conducted for staff from URC and representatives from government counterparts and USAID/Cambodia on August 9, 2012 in Phnom Penh. Another debriefing was held on August 14, 2012 in Bangkok with staff of USAID/RDMA, with a telephone link to USAID/Cambodia. f) Report writing: Inputs from the debriefing sessions were incorporated into a draft report and submitted it to USAID/RDMA for sharing with relevant reviewers. Additional comments from these reviewers were then incorporated into this Final Report. 3. Limitations The following limitations of this evaluation have been identified: a) There was the potential of a recall bias and limited institutional memory due to staff turnover since this evaluation was conducted about eight months after the Project ended; b) The follow-on project, CAP-Malaria, was already being implemented in the same areas and with similar interventions as the MCC, making it likely that the informants did not always distinguish between the characteristics of the two projects; c) The quality of the Project’s performance data made available to the Evaluation Team was poor. Specific implementation targets were not routinely established, and there were discrepancies and inconsistencies in performance records obtained from different reports; d) The Evaluation Team included two consultants with limited familiarity with the history of the Project. While this had the advantage of objectivity, it required that information be gathered from Project reports, which were inherently biased, and from local informants, usually with the assistance of an interpreter, in addition to other primary and secondary sources noted above. The original design of the team included one staff person from the Cambodia National Centre for Parasitology, Entomology and Malaria Control (CNM) and possibly another team member from the World Health Organization (WHO) in Cambodia. The addition of these members to the Evaluation Team would have provided valuable local knowledge and awareness of the context in which MCC functioned. This deficiency was addressed by conducting data triangulation for the analysis as well as validation of findings with partners. 4. The Evaluation Team The Evaluation Team consisted of two international consultants with considerable experience in program management and evaluations in South-East Asia. Collectively they brought to the task work experience in Cambodia and expertise in primary health care, migration health, project design and Page 8 management, behavior change communication (BCC), monitoring and evaluation (M&E), capacity building, and community development in addition to malaria control. Page 9 2. CONTEXT 1. Malaria in Cambodia Malaria continues to be a serious public health threat in Cambodia. It is estimated that about 15% of the population, 2.1 million people, are at risk with about 500,000 persons living in areas of high malaria transmission being the most vulnerable2. With the breakdown of basic public health services in the 1970s, malaria prevention and curative services were mainly available, if at all, through the informal, private sector. Since the political situation stabilized in the 1990s the National Malaria Control Program (NMCP) has made great strides and the key indicators of its impact on communities (morbidity, severe cases, and mortality), were greatly reduced during the following decade. The number of malaria cases reported and treated in public facilities decreased from 170,387 cases in1997 to 58,887 cases in 2008, a reduction of 65%3. An unknown number of cases continue to be treated in the private sector but indicators regarding those treatment episodes are not available. 2. Diminishing Efficacy of Artemisinin-based Combination Therapy There is evidence of a reduction in the effectiveness of the current first-line drugs, Artemisinin-based Combination Therapy (ACT), in treating the most lethal form of malaria, P. falciparum. The MCC Project area in Western Cambodia was the location of the initial development of resistance more than four decades ago to Chloroquine, which was the internationally accepted first-line drug for both the prevention and treatment of malaria until the1960s/1970s. Many of the behaviors that contributed to the development of resistance to Chloroquine, the marketing of expired and counterfeit drugs and non￾compliance with malaria treatment regimens, continue to some degree in the MCC Project area. The reduced efficacy of ACT became evident in 2005-2006 and in 2007 WHO convened a consultation in Phnom Penh on the containment of multi-drug resistant malaria strains. This was followed in 2008 by a number of consultations on the problem of drug resistance that led to the support from the Bill and Melinda Gates Foundation for a two-year effort known as the “Containment Project” and was focused on the MCC Project area. This chronology of events is noted because the MCC Project began in 2007 at a time when there was a high level of concern about the global implications of a possible repetition of resistant strains developing and spreading to the rest of the region and the world. The stakes remain high in the effort to control malaria in Cambodia and prevent the spread of resistant strains of P. falciparum. Failure would likely have catastrophic consequences globally, especially in Africa. Efforts to control and prevent the spread of parasite strains resistant to the current first-line drugs in 2 World Health Organization. Malaria in the Greater Mekong Subregion: Regional and Country Profiles. Geneva: Global Malaria Programme, 2010. 3 National Centre for Parasitology, Entomology and Malaria Control Program. Annual Progress Report 2009. Phnom Penh: Kingdom of Cambodia, 2009. . Page 10 Cambodia require an extraordinary commitment on the part of the Royal Government of Cambodia and the international community. 3. U.S. Government Malaria Initiatives The U.S. Government has supported malaria control programs bilaterally since the 1950s. However, global malaria received greater attention in 2005 with the launching of the President’s Malaria Initiative (PMI), a five-year plan to expand U.S. malaria efforts with a focus on 15 African nations. PMI, is administered by USAID, with technical support from the U.S Centers for Disease Control and Prevention. The program was expanded for 2009-2013 and included 4 more African countries plus the Greater Mekong Sub-region – Myanmar, Cambodia, Yunnan Province of China, Lao PDR, Thailand and Vietnam. U.S. funding for global malaria programs was further increased in 2009 with the Global Health Initiative and in that same year the Lower Mekong Initiative (LMI) was launched to deal with regional challenges, especially cross-border risks in Cambodia, Lao PDR, Thailand and Vietnam with Myanmar joining LMI earlier this year. The CAP-Malaria Project, which focuses on cross-border malaria control in Cambodia, Thailand and Myanmar, is considered to be an initiative of the LMI. 4. The Malaria Control Structure of Cambodia The Ministry of Health has designated the CNM, as the lead agency for developing and implementing the national malaria control strategy in Cambodia. The Strategic Master Plan for the NMCP, 2006-2010 was operative when the MCC began in October 2007. It called for four service packages: 1) Prevention, 2) Early Diagnosis and Appropriate Treatment (EDAT), 3) Research, and 4) Surveillance and Management. While the NMCP is vertical and centralized, the interventions are administered at the provincial and operational district (OD) levels. Each Province, for example, has a Provincial Malaria Supervisor and some ODs also have a District Malaria Supervisor. 5. The Malaria Control in Cambodia Project (MCC) URC, an international non-governmental organization (NGO) was the prime contractor for MCC and they signed a subcontract with another international NGO, Partners for Development (PfD). The Project began in October 2007 as a three-year project but was subsequently extended for 14 months through November 2011. The budget for the 50 month Project was US $3,550.000. The contract between USAID/RDMA and MCC states that it “is intended to enhance national malaria control program (NMCP) activities by providing technical assistance and support to improve the diagnosis and treatment of malaria, and educate the community about malaria prevention, control and appropriate health seeking behaviors”. In addition, the contract committed MCC to improving clinical facilities to higher standards, ensuring uninterrupted supplies of good quality anti-malarial drugs and diagnostic tests, implementing operational research on anti￾malarial resistance and supporting the dissemination of information, education and communication (IEC) materials on rational treatment of malaria. Key activities in the targeted ODs are specified as:  Provision of pre/in-service training in laboratory diagnosis of malaria and implementation of quality assurance (QA) and quality control (QC) in routine diagnostic practice; Figure 1. Malaria Control in Cambodia (MCC) Project Sites Page 11  Development of effective strategies for the clinical management of malaria cases with emphasis on compliance to rational treatment with ACT;  Development and dissemination of IEC messages with emphasis on appropriate health seeking behaviors to prevent the development and spread of anti-malarial drug resistance; and  Collaboration with other malaria control implementation partners in country and across the border, including possible sub-grants to NGOs, and potential public/private partnership on the aforementioned activities. As a result of consultations with the NMCP, MCC focused on 5 of the 11 ODs in 4 Western provinces: Banteay Meanchey, Battambang, Oddar Meanchey and Pailin. The 5 ODs in the Project area have a population of 290,344 served by 5 referral hospitals, 5 former district hospitals, 33 health centers, 3 health posts and more than 500 village malaria workers (VMWs). Two ODs , Maung Russey and Thmar Pouk, were added in the last year of the Project, increasing the target population to 548,442 but activities in those two new ODs are not covered by this report. Page 12 1 3. Key findings from an Analysis of Project Design and Implementation The tasks and sub-tasks specified in the contract do not establish measurable objectives and the annual workplans indicate the types of activities to be undertaken, but not how many or for how many persons. The following analysis, therefore, describes the components of implementation but has no basis for determining if the levels of achievement were in accordance with predetermined standards or targets. 3.1. Prevention MCC implemented an IEC/BCC strategy with several initiatives designed to improve community awareness about malaria prevention and control as well as to support the NMCP in increasing access to and utilization of insecticide-treated bed nets/long-lasting insecticide-treated bed nets (ITNs/LLINs). 3.1.1. Achievements i) MCC had an extensive IEC/BCC program that was supported by the design and production of many communication materials. Output: Low-literacy print materials were relevant and effective. MCC produced and/or repaired approximately 20 billboards each year and installed them at strategic locations. Over 5,000 posters in various designs for different key messages were displayed mainly at the volunteers’ houses and community gathering places. About 15,000 leaflets were distributed to the targeted beneficiaries, while stickers were posted on the windows of the 35 taxies participating in the Project. Several other promotional items such as t-shirts and caps were used by individuals participating in the MCC such as taxi drivers and volunteers to identify them with the Project. Over 1,400 flipcharts were developed and provided to the volunteers and staff at public health facilities. Information spots and drama were produced for playing in taxis to reach mobile migrant and migrant populations (MMPs) in addition to spots for television and radio. ii) The village health volunteers (VHVs) reached individuals and small groups by conducting formal education sessions while the village malaria workers (VMWs) contacted families while providing rapid diagnostic tests and occasional home visits. Output: The Project had approximately 125,000 face-to-face malaria education encounters each year. Approximately 55% of the participants in these sessions were reported to be women, but observations by the Evaluation Team during the site visits revealed that there were few men participating. Page 13 iii) MCC conducted community-wide campaigns through participating in World Malaria Week and mass media. Output: Approximately 31,000 individuals per year were reached through the Malaria Week Campaigns which were jointly organized annually with the government malaria programs, NGOs and stakeholders. Malaria information and ITNs/LLINs were distributed, bed nets were re-impregnated, RDTs for malaria diagnosis were provided to febrile patients, and malariacases were referred for treatment. Mass media campaigns were conducted every year but irregularly depending on the gaps in funds from the NMCP to support the programs. iv) MCC promoted the use of bed nets through their BCC messages and the distribution of ITNs/LLINs. Output: Protection for the whole family by sleeping under ITNs/LLINs was a major prevention message from the VHVs and VMWs. More than 200,000 ITNs/LLNs were distributed to targeted community members and over 24,000 bed nets were (re)impregnanted during the Malaria Week Campagns. In addition, 2,956 employers who collectively engaged about 110,000 workers received over 40,000 ITNs/LLINs from MCC to lend to their workers during their time of employment. The percentage of surveyed households that own an ITN/LLIN almost doubled, from 36% in 2008 to 61% in 2011. 3.1.2. Prevention Outcomes A comparison of the baseline and endline survey results showed that the percentage of surveyed households that have “appropriate malaria knowledge4” increased from 87% in 2008 to 97% in 2011 and the percentage of children aged below five who slept in an ITN/LLIN last night increased from 31% in 2008 to 44% in 2011 (Figure 2). 4 The MCC defined “appropriate malaria knowledge” as the knowledge that malaria is caused by mosquito bites and can be prevented by sleeping under a bed net. Figure 2. Percentages of Malaria Knowledge, Bed Net Coverage, and Bed Net Use among the Targeted Populations, 2008 and 2011 Sources: Adapted from MCC Project Baseline & Endline Surveys, 2008 and 2011 Page 14 3.1.3. Shortfalls and Key Constraints The IEC/BCC interventions were not focused on reducing risk: MCC was successful in increasing knowledge about malaria but the Project implemented IEC focused on large-scale dissemination of the information that people should know rather than BCC designed to reduce the risk of infection. Some historical misunderstandings and myths were still commonly found among the targeted communities. The MMPs and local community members visited by the Team knew that malaria is transmitted through mosquito bites but only one community member said that it is an infectious disease passed from person￾to-person through the mosquito’s saliva. Most believed that it was because the mosquitos lived in dirty environments like the toilet or bite dirty animals like dogs and cats and brought the disease to people. Many still believed that malaria was caused by drinking unclean water and the poor environmental sanitation around their houses. The IEC materials produced and the discussion sessions observed paid little or no attention to prevention of further transmission. There were limited links between the various prevention interventions that could have made the activities more comprehensive and more effective. 3.1.4. Lessons Learned Reaching the groups at highest risk for malaria infection increases the impact of a project: Government counterparts, VMWs, and community members all agreed that a huge challenge remains in reaching MMPs and they were usually found only after they came forward to seek healthcare. The interviewed MMP themselves also commented on their marginalization. Since they do not belong to the local community, they were not invited to the health education sessions regardless of the distance from their residences/workplaces to the village center and despite the fact that more than half of the MMPs (56%) stayed in the village after work and many of them (41%) repeatedly migrated to work in the targeted areas. One of the key messages derived from the assessments of risk was to focus on MMPs as a vulnerable group. However, the emphasis on the risk population group seemed to produce an adverse outcome. The Evaluation Team found that participants who were local community members often detached themselves from the risk of malaria. Some of them mentioned that “People who live here for a long time already have immunity, so we won’t get malaria,” while most of them said that “It’s no longer a problem for us. It’s a problem of migrants who work in the farms.” This indicated that they linked malaria infection to a specific population group rather than the risk behaviors and this could increase their vulnerability as they continued to work in the farms/forests in the endemic areas. 3.1.5. Good Practice i) MCC consistently conducted formative assessments to learn about the needs and aspirations of each target group before designing their prevention interventions. ii) Community members were engaged as health educators promoting malaria prevention as well as diagnosis and treatment. 3.1.6. Potential considerations for follow-on programs with similar objectives i) Groups at highest risk should be identified early and given priority attention: The MMP were identified as a high risk group and innovative interventions such as the engagement of taxi drivers and the LLIN lending scheme were implemented to reduce their vulnerability to malaria. ii) Existing resources and structures should be utilized to the extent possible: For example, the Net-Lending Scheme used the existing relationship between laborers and owners to increase the availability and use of ITNs/LLINs. Page 15 3.2. Community Mobilization Volunteers were a major resource for disseminating malaria information. Since they are from the local communities they can be effective in delivering credible health education messages in a culturally appropriate style. The VMWs not only provided information but also could diagnose malaria with the RDT and make available the appropriate treatment. 3.2.1. Achievements i) VMWs provided quality diagnostic and treatment services in all of the villages in the target area. Output: The VMWs served 264 targeted villages with an average of slightly more than 2 VMWs per village. They functioned as community-based staff of the public health facilities and were engaged in the full range of malaria control from prevention, diagnosis, and treatment to referrals. The training records of MCC indicated that there were 532 classes for VMWs at which they received formal training in malaria prevention, including bed net utilization, screening with RDT and treatment of positive cases. In addition, monthly supportive supervisory visits were organized at the health centers by NMCP staff from the province or OD plus Project staff. ii) The VHVs focused on health education through structured training events augmented by supportive visits. Output: For malaria control the VHVs focused on prevention and referrals to VMWs or public health facilities for cases suspected of requiring treatment iii) Thirty-five taxi drivers participated in the Project by providing basic information about malaria prevention, diagnosis and treatment to MMPs while traveling by a taxi. Output: The need for their function was identified through focus groups of MMPs convened by the Project. Interviews with selected taxi drivers by the Evaluation Team indicated that their incentives included the increased self-esteem from being listened to and protecting others from malaria infection. In addition, they could charge taxi fares when called to take patients to the health facilities and the Project compensated them for income lost while attending the Project’s quarterly meetings. They were well informed about malaria and able to correct misunderstandings and myths about the causes of malaria and provided information on where to go for diagnosis and treatment. MCC’s training records indicated 117 taxi driver training events. A “mystery shopper” survey of the performance of the taxi drivers found that 20% of the drivers spoke without prompting about malaria, 16% played malaria educational messages and 61% handed out malaria pamphlets to the shoppers. iv) School health malaria curricula were developed and teachers were trained. Output: As a part of the National Strategic Plan, MCC worked with the School Health Department of the Ministry of Education and other NGOs to provide technical inputs for updating standardized educational modules related to causes, modes of transmission, prevention methods, diagnosis, and treatment of malaria. School teachers were trained to more effectively reach primary school children and their Page 16 families with critical information about malaria prevention. 3.2.2. Community Mobilization Outcome Properly selected, trained and supported, the members of the community proved to be an invaluable asset to the NMCP and the MCC. An independent assessment in 2011 noted that MCC’s greatest success may be in community mobilization. MCC proudly described itself as a “community-based response” in light of the significant contributions to malaria control by these local volunteers. 3.2.3. Shortfalls and Key Constraints i) There were limited links between the various prevention interventions that could have made the activities more effective. For example, the taxi drivers could have been linked to the farm owners and the VMWs so that they could help promote the bed net lending scheme among MMPs. Likewise, they could also help promote the radio program, “I Care about My Health.” ii) The VMWs and VHVs were seen by the community as an extension of the health facilities in the OD and province. Therefore, local support was not seen as indicated or appropriate. Some support from the community and some accountability from the VMWs and VHVs to the community would enhance sustainability. 3.2.4. Lessons Learned Community members can participate in malaria control programs not only as passive recipients of services but as valuable resources for malaria prevention, diagnosis and treatment. 3.2.5. Good Practice i) Community-based interventions provide a level of access that can not be matched by medical facilities. ii) VHVs and VMWs were carefully selected through a community process and then provided with extensive training, regular supervision and support.. 3.2.6. Potential considerations for follow-on programs with similar objectives Promoting engagement of community members: MCC demonstrated that local volunteers can make a significant contribution to health education. In addition, the VMWs brought the provision of diagnosic and treatment services to the community level, greatly increasing access to these critical services. 3.3. Diagnosis and Treatment To contribute to the NMCP’s EDAT strategy, MCC strengthened malaria diagnostic and case management skills at health facility and community levels and increased access to and utilization of appropriate services in the targeted ODs. Page 17 Table 1. Compliance with the Treatment Regimens According to the National Guidelines in the Target Areas, 2008 – 2011 Total Battambang OD Sampov Loun OD Pailin OD Samrong OD O'Chrov OD % uncomplicated cases treated with correct regimen 2008 96.7% (4,916/5,085) 97.8% (1,370/1,401) 90.9% (726/799) 93.1% (527/566) 98.8% (2,120/2,146) 100% (173/173) 2009 95.8% (4,878/5,094) 94.3% (750/795) 94.4% (391/414) 90.8% (726/800) 98.0% (2,900/2,959) 88.1% (111/126) 2010 96.4% (2,480/2,573) 90.5% (201/222) 97.2% (211/217) 89.0% (372/418) 98.8% (1,652/1,672) 100.0% (44/44) 2011 99.9% (1,606/1,608) 100% (257/257) 100% (212/212) 99.1% (233/235) 100% (883/883) 100% (21/21) % uncomplicated P.f cases treated with first line ACT 2008 96.7% (4,916/5,085) 97.8% (1,370/1,401) 90.9% (726/799) 93.1% (527/566) 98.8% (2,120/2,146) 100% (173/173) 2009 96.8% (2,171/2,242) 97.9% (417/426) 95.5% (214/224) 95.8% (346/361) 97.0% (1,179/1,216) 100% (15/15) 2010 79.0% (532/673) 100% (73/73) 93.6% (44/47) 100% (93/93) 96.3% (438/455) 100% (5/5) 2011 99.4% (317/319) 100% (88/88) 100% (53/53) 96% (48/50) 100% (126/126) 100% (2/2) % confirmed P.v cases treated as per national guidelines* 2008 1,544 168 247 171 701 257 2009 96.5% (2,510/2,602) 90.3% (326/361) 94.1% (112/119) 96.1% (343/357) 98.7% (1,658/1,680) 83.5% (71/85) 2010 96.7% (2,310/2,389) 84.9% (118/139) 98.1% (202/206) 88.7% (377/425) 99.7% (1567/1572) 97.9% (46/47) 2011 N/A N/A N/A N/A N/A N/A % confirmed severe P.f cases admitted 2008 46.1% (374/811) 85.3% (93/109) 45.7% (32/70) 41.4% (96/232) 50.2% (151/301) 2.0% (2/99) 2009 86.4% (719/832) 94.9% (56/59) 95.3% (61/64) 72.7% (48/66) 86.1% (550/639) 100% (4/4) 2010 92.5% (272/294) 100% (20/20) 70.0% (7/10) 100% (6/6) 92.5% (236/255) 100% (3/3) 2011 91.7% (121/132) 81.5% (22/27) 69.2% (9/13) 66.7% (2/3) 98.9% (87/88) 100% (1/1) % severe malaria cases admitted receiving correct regimen 2008 96.4% (345/358) 100% (50/50) 100% (100/100) 87.1% (61/70) 100% (99/99) 89.7% (35/39) 2009 78.6% (575/732) 82.1% (46/56) 87.5% (56/64) 79.6% (39/49) 77.2% (429/556) 71.4% (5/7) 2010 93.0% (238/256) 100% (18/18) 100% (7/7) 66.7% (4/6) 93.2% (207/222) 100% (3/3) 2011 97.5% (118/121) 100% (22/22) 88.9% (8/9) 100% (2/2) 97.7% (85/87) 100% (1/1) Sources: Adapted from MCC Project Annual Reports, 2008-2011 3.3.1. Achievements i) Presumptive treatment was eliminated in the public health system. Output: Both the VMWs and the medical staff in the facilities were trained and Page 18 supervised in the use of diagnositic tests as an essential part of EDAT. The percentage of malaria patients who received treatment at the out￾patient department based only on clinical diagnosis alone was reduced from 10% in 2008 to 4% in 2009 and to zero percent in 2010 and 2011. ii):The number of severe cases was reduced in the project area facilities and, concurrently, the percentage of severe case admitted to the facilities increased. Output: All clinical staff in the facilities and the communities received training in EDAT and educated the communities about its importance. Regular monitoring and supervision of the staff in the health facilities gave priority to establishing a definitive diagnosis by either RDT or microscopy before initiating treatment. All 28 targeted health facilities reported improvements in diagnositic procedures, either with the RDT or microscopy or both, and the use of ACT according to the national guidelines. iii) : Compliance with the national treatment guidelines was high. Output: Clinical staff in the facilities and in the community were well trained in the application of the national guidelines: Over 96% (13,880/14,360) of the uncomplicated cases and almost 87% (1,276/1,467) of severe cases were treated in accordance with the national standards (Table 1) 3.3.2. Diagnosis and Treatment Outcome The implementation of EDAT, the use of RDT by the VMWs, the referral protocol to health facilities with qualified staff for high risk patients, the virtual elimination of presumptive treatment and the high rate of treatment according to national standards of both complicated and uncomplicated cases contributed to a reduction of malaria in the Project area. Figure 3. Percentage of Confirmed Severe P. falciparum Malaria Cases Admitted and the Number of Confirmed Severe Cases at the Targeted Public Health Facilities, 2008 - 2011 Sources: Adapted from MCC Project Annual Reports, 2008-2011 3.3.3. Shortfalls and Key Constraints i) Knowledge about malaria did not result in seeking early diagnosis and treatment: The level of knowledge among targeted community members of the danger of substandard/counterfeit drugs was 46.1% 86.4% 92.5% 91.7% 811 832 294 132 0 100 200 300 400 500 600 700 800 900 40 50 60 70 80 90 100 2008 2009 2010 2011 Number of Severe Cases Percentage Page 19 increased from 43% to 75%. However, the Project reported that the proportion of surveyed respondents who sought treatment within 48 hours of the onset of fever, the standard promoted by the Project, decreased from 63% in 2008 to 50% in 2011 (Figure 4). Although those who chose self￾treatment as the first choice dropped from 27% to 22%, those seeking treatment still went to the private sector more than the VMWs or health facilities (54% vs. 24%). Figure 4. Malaria Related Treatment Knowledge and Health Seeking Behaviors among the Targeted Community Members, 2008 and 2011 Sources: Adapted from MCC Project Baseline & Endline Surveys, 2008 and 2011 ii) Logistics management: One of the requirements for malaria control is having access to secure, reliable and sufficient supplies of the right diagnostic tools and medicines. The first line ACT has been made available from the NMCP since 2009 at all 28 public health facilities in the targeted areas. However, stock-outs were an ongoing problem, prompting MCC to procure the essential supplies when they were depleted . Figure 5. Percentages of Malaria Cases Confirmed by Rapid Diagnostic Tests and Microscopes at the Targeted Public Health Facilities, 2008 – 2011 Sources: Adapted from MCC Project Annual Reports, 2008 – 2011 iii) Reduced use of microscopic tests: The simplicity and low cost of the RDT tended to reduce the use of microscopy in the health facilities. The cases confirmed by microscopy gradually decreased from 60-73% in the beginning of the Project to only about 43% at the Project end (Figure 5). Page 20 3.3.4. Lessons Learned In general, RDTs have acceptable levels of specificity and sensitivity so that VHWs can diagnose suspected malaria cases at the village level with confidence and initiate treatment in accordance with national standards. 3.3.5. Good Practice VMWs and medical staff in the local health facilities collaborate and their complementary roles strengthen service delivery. 3.3.6. Potential consideration for follow-on programs with similar objectives Public sector services can be strengthened by NGOs if they function within a public system of prevention, diagnosis and treatment with clearly defined roles and responsibilities. 3.4. Capacity Building There was a clear expectation in the contract that training and capacity building would be a major part of the contribution by the MCC. Specifcally it noted that MCC was to improve the diagnosis and treatment of malaria and educate the community about appropriate health seeking behaviors. In addition, MCC was to provide training of laboratory technicians to develop QA/QC in diagnostic practices. 3.4.1. Achievements i) MCC contributed to the improved capacity of public sector staff and volunteers for management, laboratory skills, research and patient care at the provincial, OD and village levels Output: 2,763 training episodes (1 person participating in 1 training sessions) were conducted for health professionals in the area served by the Project. Both the VHVs and VMWs received extensive training from MCC as indicated in Table 3. But the training was not a “one off” intervention but rather one of a set designed to provide both support and incentives for these volunteers. Meetings were held regularly at which the basics of malaria education were reviewed and the VMWs skills in RDTs and treatment were strengthened. The travel allowance and other incentives provided for attendance at these meetings served as a source of support for these volunteers. In addition, there were regular meetings among service providers at the OD level as well as monthly monitoring visits. One function of the supervisory visits was to maintain the skills acquired in the training. Gradually the leadership of these supportive supervisory visits was turned over to the OD or Provincial Malaria Supervisors. Page 21 Table 2. MCC Supported Training/Meetings by Position and Sex of Participants, 2008-2011 Participants Topic Year I Year II Year III Year IV Total* M F M F M F M F M F Total VMWs 1 55 18 149 239 26 45 0 0 230 302 532 VHVs 2 14 25 170 240 38 161 0 0 222 426 648 Physicians, Nurses, Medical Assistants 3 21 6 30 8 25 11 0 0 76 25 101 Laboratory Technicians 4 13 4 89 19 76 13 66 11 244 47 291 Provincial Health/ OD Staff 5 21 22 61 42 20 12 0 0 102 76 178 Teachers 6 0 0 0 0 0 0 233 82 233 82 315 Health Center Staff 7 21 15 72 13 0 0 0 0 93 28 121 Taxi Drivers 2 0 0 0 0 0 0 81 2 115 2 117 Private sector 7 0 0 0 0 67 20 159 84 226 104 330 Researchers 8 0 0 40 0 36 6 22 26 98 32 130 Total* 145 90 611 561 322 268 561 205 1,639 1,124 2,763 Source: MCC Training Records, 2008-2011 Notes: * Number of participants in training events and meetings. 1) Malaria education, RDT, treatment and refresher training. 2) Malaria education 3) Case management, severe case management and nursing care 4) Basic laboratory skills, supervision, and quality control 5) Logistics, planning, and project cycle management 6) School health, and malaria control 7) Training of trainers on IEC/BCC 8) Coordination 3.4.2. Capacity Building Outcome VMWs increased their proportion of cases treated in the public sector from 12.3% (851/6,868) in 2009 to 57.6% (5,564/9,659) in 2011 (Figure 6). Figure 6. Comparison of the Numbers of Uncomplicated Malaria Patients Treated by VMWs and at Public Health Facilities in the Targeted Areas, 2009 – 2011 Source: Adapted from MCC Project Final Report, December 2011 Note: VMWs were trained in 2008 and started providing RDT in 2009. Page 22 3.4.3. Shortfalls and Key Constraints i) Training opportunities were not adequately marketed: Several of the OD staff visited by the Team were not aware of any management training for OD staff. ii) Training was not provided continuously for some groups: The training matrix indicates that there were no training events for the VHVs and VMWs during the final year of the Project. iii) MCC did not measure performance “on the job” before and after the training: They limited their evaluation of training to knowledge on the topic before and after the training events, with predictable positive changes. 3.4.4. Lessons Learned i) Capacity building is an essential component of every project. ii) Capacity building is an on-going process with multiple methods and phases. 3.4.5. Good Practice Training should be followed by periodic refresher courses and regular supervision at which newly acquired skills are reinforced. 3.4.6. Potential considerations for follow-on programs with similar objectives i) Capacity building opportunities should be provided to the key staff in all levels of the organization, including staff of the implementing agencies. ii) Formal training should be followed-up by refresher courses, regular supervision and monitoring. 3.5. Project Management The Project management systems of MCC were effective in utilizing their resources to make significant contributions to the NMCP. 3.5.1. Achievements i) The Project staff developed collaborative relations with colleague agencies and the donor. They were experienced in the policies and procedures of the NMCP so they understood their role in it. Page 23 Output: From interviews with staff of the NMCP, it seemed there had been a positive, collaborative relationship with MCC in planning and strategy development, training, gap-filling procurements and the adaptation of some of the innovative approaches to other provinces. Similarly, constructive relations were reported with WHO, especially in the planning and implementation of the Containment Project, the Pasteur Institute of Cambodia for testing of drugs and mapping of resistant strains and Equal Access and Population Services International in the production of a radio call-in program. Because MCC’s mandate was to strengthen the NMCP at all levels it was necessary to have good, collaborative relationships with government offices responsible for malaria at the national as well as at the provincial and OD levels and this seemed to have been achieved. The Government staff with whom the Evaluation Team spoke had only positive comments about the MCC staff. They included references to the fact that the MCC staff knew the procedures and policies of the Government and, therefore, knew how to work within the system. It also was noted the MCC had “good people on the ground.” So the prerequisites of good project management systems, able people who know how to work within the system and strong collaborative relationships, were met. ii) Clear lines of authority and responsibility allowed for staff being utilized flexibly according to the needs of the program. Output: Accountability for the management of MCC was the responsibility of the Chief of Party, with assistance from the Technical Manager. Work assignments were managed through weekly staff meetings and supervisory meetings as needed. Program priorities were developed through the Annual Operational Plans (AOPs) in the targeted provinces. The staffing of MCC was thin, with only 5 full-time staff in the provinces (3 in Battambang and 2 in the Banteay Meanachey Provincial Offices) and 9 at the Project headquarters in Phnom Penh during most of the four-year Project, in addition to 3 drivers. Four of the 9 Project staff were seconded by the subcontractor, PfD. Although the Evaluation Team did not review travel records, it was understood from interviews in the Project areas that staff based in Phnom Penh were frequently in the Project areas for training events, technical assistance, supervision, and other purposes. This staff utilization pattern enabled MCC to assign the technical staff where and when they were most needed and the provincial staff were primarily engaged in coordination and back-stopping. Management systems were appropriately established and implemented, with the exception of M&E (see Section 3.6 below). iii) The Project adapted to the needs of the NMCP in the target areas. Output: MCC exhibited the advantages of a “process” rather than a “blueprint” approach with flexibility in responding to unmet needs. The AOPs were developed at the provincial level and through that process gaps in the public programs were identified and MCC was asked to support programs in the Project areas that were not adequately funded through the government channels. iv) MCC advocated for more responsive policies at both the national and local levels. Output: MCC participated actively at the national level including collaboration with the Page 24 CNM in the planning for additional initiatives and in the Project areas with the implementation partners. This included contributions to the design of the “Containment Project” in 2010-2011. In addition, MCC staff participated in national planning for the work of the NMCP and implementing partners under the Global Fund to fight AIDS, Tuberculosis and Malaria to enhance complementarity of efforts. At the local level, the Directors of the health facilities and Malaria Supervisors in the Project areas were engaged in the joint AOP development in which the MCC’s activities were integrated into the government’s annual malaria workplans at the provincial level and down to village level. The Project also supported local public health administrators to be more involved in the routine monitoring and supervision of public health staff and volunteers. 3.5.2. Management Outcome Constructive relationships were developed with the NMCP at both the national and local levels; allocation of staff time were flexible according to the needs of the program; and project resources were used to fill the gaps in the OD malaria control programs. MCC resources were leveraged significantly and malaria control interventions were strengthened. 3.5.3. Shortfalls and Key Constraints i) The reactive design of the Project presented challenges: MCC has been described as a “technical assistance, gap-filling” project. As such it was dependent on the identification of needs of the NMCP in the 5 Project ODs as they were revealed during the development of the AOPs. Thus, the Project was more “reactive” than “proactive”, which may have contributed to a lack of targets since the Project could not predict what would be required from year to year. However, on the positive side, this ad hoc style of responding to the needs of the provincial malaria programs enabled the leveraging of Project resources and promoted collective efforts to control malaria. ii) Few professional development opportunities for Project staff were available: MCC provided staff of the health centers, the ODs and the provincial malaria programs with impressive training but there were few opportunities for professional development of the MCC staff during the life of the Project. iii) Objectives fluctuated during the life of the Project: The task order for the MCC Project specified tasks and did not require “time-limited, measurable” objectives, other than to note that the Project was in support of the NMCP. A review of the Project reports revealed that the objectives of the Project shifted over time both in number and content without explanation or evidence of approval from the donor. iv) Some reporting requirements were not met: The management of MCC did not fulfill all the requirements of the task order or their own operational plans. Neither the required Final Task Order Report highlighting major successes and shortfalls with reference to established objectives and indicators nor the Gender Report noted in the workplan for year 4 were developed. 3.5.4. Lessons Learned The “process” rather than a “blueprint” model of the project management was able to flexibly adapt to unanticipated needs. However, there should be a framework within which one functions in order to get the most “bang for the buck.” For example, there might be an agreed-upon list of procurements although the quantities might not be known at the beginning of the Project. Page 25 3.5.5. Good Practice MCC effectively used formative research on the target populations as part of the intervention design process. This enabled them to identify the need for ITNs by workers in the endemic areas which led to the net-lending intitiative and the need for better information by the MMPs which led to the use of taxi drivers as health mediators. 3.5.6. Potential considerations for follow-on programs with similar objectives This type of “gap-filling” model should be utilized only in semi-emergency situations where issues of sustainability are secondary to having an impact on the current threat. This was an appropriate model in the face of the uncertainty about resistant strains and the need to respond quickly and effectively to the possibility of multi-drug resistant strains thriving and spreading. 3.6. Monitoring and Evaluation The monitoring and evaluation system was very active in developing indicators and collecting data, but there is little evidence that this information was used for the management of the Project. 3.6.1. Achievements i) M&E Plans and Annual Reports were prepared regularly. Output: Extensive data were collected throughout the Project and used to prepared status reports. The MCC Annual Performance Reports for FY 2008 and 2009 contain partially completed M&E reports that cover that year only. The FY 2010 Annual Report of the Project has comprehensive report on 46 indicators with a 2008 baseline. ii) Data collection was coordinated with local health facilities. Output: Program data were collected systematically and in collaboration with the NMCP. While the clinical data could be utilized for both QC and as a basis for monitoring treatment outcomes, the epidemiological data (i.e. morbidity and mortality) reflected the impacts of the combined interventions on the epidemic. iii) Baseline and endline surveys were conducted. Output: The household data was collected both pre- and post-intervention. The survey data were valuable in designing interventions, especially for the high risk MMPs. MCC designed and implemented a baseline survey among the targeted households to assess the knowledge and practices related to malaria prevention and care and treatment during the first year of implementation. The same survey was repeated in the last year of the Project to determine changes of the situation at both output (knowledge) and outcome (behaviors) levels. 3.6.2. Monitoring and Evaluation Outcome Extensive data were collected to measure performance as defined chosen indicators. The utilization of this data for management purposes is not apparent. Page 26 3.6.3. Shortfalls and Key Constraints i) Contractual specifications contributed to M&E deficiencies: The absence of specific targets and objectives contributed to a less than optimal M&E system. Since the Project was developed with a very flexible style that allowed for big and small changes throughout the Project’s life, several shifts and/or additional interventions were made over time both in terms of quantity and content to fill the NMCP’s gaps. While this approach allowed the Project to respond to the emerging needs, it posed a big challenge to developing a strong M&E system. This evaluation found that the MCC did not have in place the M&E framework that could draw a causal relationship between the services delivered and the changes in risk behaviors and patterns of malaria treatment found in the targeted areas. In addition to the lack of clear targets, reasons for implementation problems and/or modifications of the plans were not documented. Some routine implementation data were inappropriately recorded and reported (e.g. double-counting and mix of different tasks into one indicator). These led to some missed opportunities to assess the Project’s contributions and to demonstrate strong evidence for replicable intervention models. ii) Some data collection tools and/or methodologies were inappropriate: Baseline and endline surveys did not include some important indicators, and therefore, the data obtained from the surveys were less useful. For example, the survey did not capture the outcome of LLIN use among adult populations who were the high risk group. Several evaluation questions such as level of exposure to the Project’s interventions and their effectiveness as well as population mobility (since MMPs were the key targeted populations) were not included. The surveys, as well as the assessments on PPM, did not use appropriate sampling frames that focused on the target populations. 3.6.4. Lessons Learned A project or program should only invest in collecting data that is going to be of value to the management and/or for accountability to the donor. 3.6.5. Good Practice The indicators selected were good measures of important variables in the implementation of the Project. 3.6.6. Potential consideration for follow-on programs with similar objectives Appropriate data should be collected and inform the implementation design as well as to modify/adjust the interventions as needed. 3.7. Private Sector The private sector is a resource that must be considered in any malaria control initiative. So the question is not “if” they are to be involved in the interventions but “how”. MCC did not formally engage the private sector until the final year of implementation when they established a pilot project on the Public Private Mix (PPM). The special circumstances were that the Government had declared that in the Containment Zone 1, treatment was only to be made available at the public facilities or the VMWs. Page 27 3.7.1. Achievements i) The PPM Pilot Project was implemented in Pailin Province: Output: Over 100 registered private health providers/drug retailers were mobilized in Pailin as participants in the PPM during the final year of the Project. Quarterly workshops were held for private providers in Pailin Province in addition to monthly visits by OD health staff. MCC supported regular meetings/workshops of private health sectors and provided information on the revised NMCP guidelines for malaria diagnosis, treatment, and related policies. The Project also worked with the NMCP to provide appreciation and/or accreditation to those who were considered good collaborators, i.e. provided appropriate diagnosis and referred the patients to the nearby government’s treatment services. They also jointly conducted monthly visits to follow-up and provide additional support to the private providers as needed. Private providers interviewed reported gaining benefits from learning more about updated malaria information, understanding of the new treatment protocols, and preventing drug resistance. ii) A referral system was designed to assure appropriate treatment. Output: The private sector in Containment Zone 1 (designated by the containment strategy as an area with highly resistant strains and/or treatment failure) were required to refer all cases to the public sector for treatment. VHVs were trained to advise community members to receive diagnosis and treatment from the nearby health facilities. VMWs treated uncomplicated malaria cases but were required to refer complicated and/or severe cases; i.e. children aged below five years and pregnant women, to the nearby health facilities MCC supported the NMCP in their referral policy. 3.7.2. Private Sector Outcome The PPM strategy increased the compliance of the private sectors to Government policies in relation to malaria treatment. About 160 malaria patients were referred to public health facilities by the private providers in the pilot project. 3.7.3. Shortfalls and Key Constraints i) The PPM pilot project did not have adequate time to show significant results. ii) Data collected from baseline and follow-up surveys are of questionable value when they were conducted in different locations with different populations. 3.7.4. Lessons Learned Powerful incentives must be made available to the private sector to shape their behavior. 3.7.5. Good Practice The pilot project did not have adequate time to demonstrate good practices. Page 28 3.7.6. Potential considerations for follow-on programs with similar objectives i) The private sector needs to be engaged in malaria control efforts if such initiatives are to succeed. The type of involvement and the incentives need to be worked out in each situation. ii) While enforcement of government policies can be effective, positive incentives are preferable but there is no “one size fits all” solution to how they are to be engaged. 3.8. Operations Research MCC engaged in operation research both independently and in collaboration with the NMCP and the “Containment Project.” 3.8.1. Achievements i) MCC collaborated with the NMCP on the surveillance of drug-resistant malaria. Output:MCC collaborated with the Containment Project in conducting community-based surveillance of drug-resistant malaria. VMWs were responsible for diagnosis after the patients completed the first-line treatment regimen. Modern technologies were employed as the Project used the Global Positioning System to map locations of drug-resistant patients in close collaboration with the NMCP and other NGOs. Mobile phones were used to inform the NMCP when Day-3 positive cases were found.16 VMWs took malaria smears, filed referral slips and provided directly observed therapy (DOT) in this pilot intervention. ii) The quality of RDT used in communities and in health facilities was assessed in two studies. Output: MCC compared RDT and microscopy use in the Anlong Veng Health Center. One hundred malaria suspected patients were tested by RDT and blood smears were read by independent laboratory staff. This comparative study found that CareStart RDT sensitivity to P. falciparum malaria was 85.8% and to non-P. falciparum was 63.6% but the specificity was 100% for both types of parasites. RDTs collected at Battambang and Samrong ODs were determined to be of good quality at each level in the period of the study. A second study reviewed the quality of a random sample of RDT tests from Battambang and Samrong ODs. iii) Multi-drug resistant malaria cases were located and tested. Output: MCC successfully tracked 48 of 71 patients with drug resistant P. falciparum parasites in Battambang and Anlong Veng Hospitals for day 28 testing. Of the total malaria cases enrolled in the study, 13 were classified as late treatment failure with ACT after 28 days. 3.8.2. Operations Research Outcome MCC contributed to refining the methodologies for operations research and added to the accumulated information about testing methodologies and multi-drug resistance. Page 29 3.8.3. Shortfalls and Key Constraints There was no feedback to the community on the surveillance results as well as no report of whether or how the research findings were used to improve the Project implementation and/or design. 3.8.4. Lessons Learned The capacity of volunteers can be developed to engage in operations research. 3.8.5 Good Practice i) MCC was responsive to the need for operations research that emerged during implementation. ii) Some research activities were coordinated through the NMCP. 3.8.6. Potential considerations for follow-on programs with similar objectives Malaria control programs engaged in operations research should consider feedback mechanisms to the community and make good use of the findings. Page 30 4. GENDER CONSIDERATIONS Gender roles are major risk factors that influence vulnerability to malaria. All malaria projects should disaggregate data by gender in order for managers to be able to design interventions to meet the needs of all persons, irrespective of age or gender, for protection from malaria infection and to monitor its impact on all groups in the target communities. MCC did not routinely disaggregate data by gender and none of the indicators in their M&E data included gender. However, a few of their reports did identify the sex of the populations being considered. The gender ratios in the communities being served appear to be nearly equal according to a very small sample of 8 villages in Pailin and another 8 villages in Chakrey. In the Pailin villages females were 48.5% of the population and in Chakrey 48% of the residents were female. These ratios are likely to be representative of the villages in the target areas. Vulnerability to malaria is not influenced by sex, but it is determined by gender roles. As the primary wage-earner for their families, men in villages served by MCC work in forest areas more than women and this is a major determinant of risk. In one surveillance, the study could recruit only 29% (16/56) female but 71% (40/56) males. However, the ratios for Day-3 positive were much closer with 54% (14/26) males and 46% (12/26) females; indicating a sex-specific Day-3 positive rate of 35% (14/40) among males and 75% (12/16) among females. The exception to malaria vulnerability being gender neutral is that pregnant women are at greater risk of severe malaria in endemic areas and their rate of infection is greater because of decreased immunity. Malaria also puts them at higher risk of miscarriage, stillbirth, premature delivery and low birth weights. Women with dual HIV and malaria infection are at heightened risk of severe anemia and adverse pregnancy outcomes5. For these reasons, MCC developed IEC material specifically for pregnant women, including 1,200 cotton bags and 8,000 flyers with messages like, “when pregnant women sleep under an ITN regularly, mother and baby will be healthy.” Adolescent girls are at special risk and are especially vulnerable when pregnant. In a training report, summarized in Section 3.4. Capacity Building, approximately 60% of the training opportunities were provided to men. This is likely a reflection of the sex of those who had positions of leadership in the provincial and OD malaria programs rather than a selection bias. This report is on training attendance so many individuals are counted several times. Of special note is the preponderance of female participants in the following groups being trained by MCC: VMWs (56%) and VHVs (65%). These two groups received 42% of all training activities. Men received the majority of the training in all other areas, with the highest proportions being among taxi drivers (98%), laboratory technicians (83%), medical staff (75%) and teachers (74%). During the two weeks of field visits the Evaluation Team observed five health education sessions led by VMWs. It was estimated that about 90% of the adults attending these sessions were women. In response to the Team’s queries, the women noted that the men were working. When elicited responses about the use of mosquito nets, a common response was that all of their families used nets every night. They also assured that their husbands had a net when he traveled to the forest areas and encouraged them to wear long-sleeved shirts at night. These observations and comments indicated that, like many other aspects of family life, it is the wives who take responsibility for malaria prevention. 5 World Health Organization. Gender, Health and Malaria. Geneva: Department of Gender, Women and Health, June 2007. Page 31 5. Conclusions The MCC was designed to strengthen the resource-poor national malaria program. It demonstrated great flexibility in responding to identified gaps in the public programs and contributed to dramatic reductions in the incidence of malaria in the Project areas. MCC creatively identified new interventions, or optimized the potential of existing models, to fill gaps and address the unmet needs of the target populations. By filling gaps in the public programs, MCC leveraged their resources and contributions to malaria control with a modest budget. One of its strengths was the strong, collaborative relationships with the staff of the public malaria program at all levels. Following is a description of the key findings based on an analysis by the Evaluation Team of information provided by USAID/RDMA, the MCC Project and the informants listed in Annex VI. 5.1. Community-based programming: Although the initial project design focused mostly on clinical aspects, it gradually moved towards a more community focus with more emphasis on village-based services. This shift from a facility-based to a community-based program was, in fact, a key to the success of the MCC. Villagers were empowered to take more responsibility for malaria control for themselves and their families. In addition, many also serve as VMWs, who are key linkage from the NMCP to the community for the prevention, diagnosis and treatment of malaria in the Project areas. 5.2. Community empowerment: Community members participated in the MCC Project not as passive recipients of services but as valuable resources for malaria prevention, diagnosis and treatment. With relatively modest inputs and support, the capacity of carefully selected local people was enhanced to provide essential services to their communities. In addition, these villagers received ongoing education and supportive supervision, monitoring, educational materials, a durable supply of test kits and drugs, and some monetary incentives. 5.3. Collaboration: MCC established a good relationship with the NMCP and were commended for knowing how to work within that Government systems. There was a complementarity between the needs of the NMCP initiatives in the target areas and what the Project could offer that made for a good fit. By working within and as a part of the public program in the 5 target ODs, the Project was able to achieve significant leveraging of their technical and financial resources. 5.4. The private sector: The private sector is an essential part of a comprehensive malaria control program. Leaving them out or ignoring them is not an option for they are the major provider of malaria and other health diagnosis and treatment in Cambodia and most other countries. The PPM pilot initiative in Pailin Province was a small but potentially important beginning in learning how an NGO working as part of a public program can provide incentives for the positive involvement of the private sector. There was limited success in providing positive and negative incentives which encouraged their collaboration. 5.5. Formative research: MCC interventions generally were informed by formative research on the populations to be addressed. This was a distinct advantage when they were searching for ways to meet the needs of the MMPs, with whom they were not familiar but were one of groups at highest risk for malaria because of their presence in the endemic areas with or without appropriate malaria knowledge and prevention. As a result of an assessment the innovative Page 32 options of engaging taxi drivers as health mediators and the employers of the workers as distributors of bed nets were developed. 5.6. Monitoring and Evaluation: Appropriate indicators were developed early in the Project and they were prominently displayed in the Annual Reports. Data were collected both independently and together with the public programs. But there was little evidence that the data was used to track progress in implementation or for other management purposes. 5.7. Continuous support: The VMWs required a consistent support system and they received it from MCC, including ongoing training, supervision and encouragement, materials for their education role, RDTs for their diagnostic role and appropriate drugs for their treatment role plus some incentives. As an extension of the health centers into the community, the VMWs also participated in the support provided to the facilities. 5.9. The challenge of attribution: The Project was embedded in the NMCP interventions in the target ODs and changes could not be attributed to the MCC interventions alone. Rather, one can only say with confidence that MCC contributed to those measures of impact but the extent of the contribution to reduced morbidity and mortality rates could not be determined with confidence because of the design of the Project. Page 33 6. RECOMMENDATIONS These recommendations are intended to guide the development of malaria control interventions in the future based on findings from evaluation of the MCC. Details on specific analysis and recommendations for replication of the MCC intervention models as well as new models can be found in Annexes 1I and III. 6.1. Intervention design is to be evidence-based: It is important to ensure that the evidence used for intervention design is derived from studies with appropriate methodologies and tools. It is also very important to appropriately document and/or produce unbiased study reports containing both positive and negative findings for future reference, including evaluations. 6.2. Supervision and support of frontline workers must be a high priority: Routine monitoring data should be regularly analyzed and fed back to relevant staff for reinforcement of the current interventions or for making timely changes and adjustments. Simultaneously, it should be channeled to staff at management level for timely support to the frontline workers. 6.3. Evaluation methods are shaped by the interventions: Evaluation questions and sampling should be specific to the interventions to enable drawing a causal relationship between the interventions and the findings. The sample size should be large enough to allow for disaggregation of data by types/sub-types of beneficiaries and include questions on exposures to the project’s interventions. 6.4. Targeting of those at highest risk increases impact: Greater impact can be achieved by reaching more individuals and groups at highest risk for malaria. While the risk groups will vary by location, such as the farm workers, migrants and other mobile populations in the MCC Project, it is important to identify them early in the design of a malaria project. 6.5. Monitor the changing situation: Malaria risk is highly dependent on both the physical environment and population movements in and out of the endemic zones. An ongoing local situation analysis should be included in future projects design as found in some areas that malaria has been losing its seasonal-disease characteristic. 6.6. Use data at all levels: Both monitoring and evaluation data should be analyzed and used at the local level since malaria is a geographically unique disease due to the different environmental and human factors. In addition, implementing a strong M&E system that can ensure the usefulness of the results requires a certain level of investment, and therefore, maximizing the use of the data will make a project more cost-effective. 6.7 Be innovative in designing interventions: Additional research is needed to develop more comprehensive prevention strategies. Surely the distribution of ITNs/LLINs, including hammock nets, is a priority prevention intervention. But, mosquitoes do not wait until bedtime to begin their feasts and additional protection is needed from early biting in addition to wearing long-sleeved shirts. 6.8 Disaggregate data by significant groups: Data should be disaggregated by sex and age and other characteristics of the target population that may assist in assuring that the interventions are reaching the intended populations. Page 34 6.9 Develop clear objectives and a robust M&E framework: Future programming should be based on a clear overall intervention logic model and a strong M&E framework that help facilitate reasonable tracking and better understanding on what achievements can be attributed to the interventions. Considering the situation in Cambodia, and other countries in the region, a foreseeable challenge for this is that projects may continue to be asked to fill the gaps. A clear strategy for gaps filling should be developed rather than being too flexible so that the program is fragmented. In so doing, the projects may have to prioritize the interventions according to where the greatest impacts can be achieved with the resources available. 6.10 Implement a targeted BCC: MCC focused on general IEC about malaria and missed the opportunity to reduce risk behaviors through targeted BCC activities. Following are some recommendations for targeted BCC strategies: a) Messages to correct misunderstandings and myths on malaria causes (e.g. sanitation, drinking unclean water) and treatments (e.g. traditional healing and avoid certain foods) should be designed and implemented. b) Group discussion and edutainment sessions should be conducted by VMWs rather than repeating the same lectures to community members. c) Risk behaviors and the environment rather than population categories should be the focus. d) Vector control strategies should be considered. Where mosquitoes are known to be more of the outdoor early biters6, the use of long sleeve clothing and scarf’s should be promoted and other locally available means should be considered, e.g. mosquito racquet and early bathing to reduce the body order that attracts mosquitoes, etc. e) A comprehensive strategy should link BCC both within the same components and across the continuum from prevention to diagnosis and treatment. f) Treatment adherence issues should be addressed. As the course of malaria treatment is very short and straightforward, improving treatment literacy among the patients is feasible and will improve treatment adherence. g) Two-way referrals should be established between VMWs and the health facilities to improve loss-to-follow-up. h) Employers of MMPs should be encouraged to be more involved than just the nets lending scheme. In addition to promoting and support their workers to access public health services, they should also be involved in case management and follow-up. 6.11. Implement strategic advocacy and community mobilization: a) Increased involvement of VHVs/VMWs and other health mediators: Involving VHVs/VMWs and employers of MMPs and other recommendations related to BCC require efforts in advocacy and community mobilization. A key is to ensure the linkages of the whole continuum of the services within and outside of the health facilities. b) Ensuring quality of diagnosis: While universal access to RDT is widely implemented, microscopic tests remains the gold standard for malaria diagnosis and should not be 6 Hewitt, Sean. Indoor Residual Spraying for Vector Control in Malaria Transmission Foci in the Artemisinin Resistance Containment Zone of Western Cambodia. Phnom Penh: Assessment Report for the MCC Project., August 2009. Page 35 displaced by RDTs7. Improving laboratory skills also contributes to broader health system strengthening. c) Establishing and/or strengthening a patient-friendly public health service: To ensure access and accessibility to public health facilities, establishing a more patient-friendly service milieu is crucial. d) Expanding DOT and follow-up test for malaria cases: DOT and follow-up test should be expanded beyond among P. falciparum cases in a limited number of drug resistant monitoring sites for the Day-3 surveillance to all high endemic areas, not only to address the drug resistant issue but also to contribute to the national agenda for elimination of malaria. e) Enhancing capacity for treatment of P. vivax infection: Since the current malaria infections in all MCC’s targeted districts8, and many other areas in Cambodia, are predominantly P. vivax, enhancing capacity of relevant health service providers and advocacy for appropriate treatment are urgently needed. 6.12. Donor harmonization is needed: There is a continuing need for donor harmonization of benefits and incentives to local staff. Some of the staff of the NMCP expressed their disappointment, even disgruntlement, that the incentives provided by MCC were less than those provided by other donors. 6.13 The multiplicity of health volunteers is inefficient: The number of health volunteers, reported to be up to 7 per village, reflects the requirements of the various donors and government departments rather than a rational design for utilizing village volunteers. Donors should review their mutual interests in utilizing health volunteers and support the development of a comprehensive policy for their assignments. 6.14. Controlling resistant strains needs to be a priority: It is unclear if the current level of effort on containment and/or elimination of resistant parasite strains in the Project areas of the MCC, now expanded in the CAP-Malaria Project, is adequate for accomplishing that agenda. USAID, in collaboration with WHO, the NMCP, and other donors, should conduct a formal review of the situation and ensure that the strategies and resource commitments are adequate for the extraordinary threat posed by the spread of resistant strains. 7 World Health Organization. Universal Access to Malaria Diagnostic Testing: An Operational Manual. Geneva: Global Malaria Programme, 2011. 8 Between 50-76% of all reported malaria cases in the 5 targeted ODs in 2010 and between 54-74% in 2011. Page 37 Annex I: MCC Components, Outputs, Outcomes and Achievements, Components Outputs Outcomes Achievements 1. Prevention 20 billboards per year, 5,000 posters, 15,000 leaflets, 1,400 flipcharts “Appropriate knowledge” increased from 87% in 2008 to 97% in 2011. Children sleeping under nets increased from 31% in 2008 to 44% in 2011. Percentage of families owning bed nets increased from 36% in 2008 to 61% in 2011. MCC had an extensive IEC/BCC program that was supported by the design and production of many communication materials. 125,000 educational encounters per year. The village health volunteers (VHVs) reached individuals and small groups by conducting formal education sessions while the village malaria workers (VMWs) contacted families while providing rapid diagnostic tests and occasional home visits. Approximately 31,000 individuals per year were reached through the Malaria Week Campaigns which were jointly organized annually with the government malaria programs, NGOs and stakeholders. Malaria information and ITNs/LLINs were distributed, bed nets were re-impregnated, RDTs for malaria diagnosis were provided to febrile patients, and malaria cases were referred for treatment. MCC conducted community￾wide campaigns through participating in World Malaria Week and mass media. 2.Community Mobilization The VMWs served 264 targeted villages. They functioned as community-based staff of the public health facilities and were engaged in the full range of malaria control. Community members became a valuable asset to MCC. MCC was seen as a “community￾based response because of the contributions of the local volunteers. VMWs provided quality diagnostic and treatment services in all of the villages in the target area. Page 38 Components Outputs Outcomes Achievements 315 teachers from 116 schools were trained to more effectively reach primary school children and their families with critical information about malaria prevention. School health malaria curricula were developed and teachers were trained. 3. Diagnosis and Treatment Both the VMWs and the medical staff in the facilities were trained and supervised in the use of diagnostic tests as an essential part of EDAT. The implementation of EDAT, the use of RDT by the VMWs, the referral protocol to health facilities with qualified staff for high risk patients, the virtual elimination of presumptive treatment and the high rate of treatment according to national standards of both complicated and uncomplicated cases contributed to a reduction of malaria in the Project area. Presumptive treatment was eliminated in the public health system All clinical staff in the facilities and the communities received training in EDAT and educated the villagers about its importance. The number of severe cases was reduced in the project area facilities and, concurrently, the percentage of severe cases admitted to the facilities increased. Clinical staff in the facilities and in the community was well training in the application of the national treatment guidelines. Compliance with the national treatment standards was high 4. Capacity Building 2,763 training opportunities used by local volunteer and staff of the NMCP in the target area. VMWs provided an increasing % diagnosis and treatment in the public sector MCC contributed to the improved capacity of public sector staff and volunteers for management, laboratory skills, research and patient care at the provincial, OD and village levels. Page 39 Components Outputs Outcomes Achievements 5. Project Management MCC staff knew the policies and procedures of NMCP. MCC staff utilized flexibly according to the needs of the local programs. MCC resources were leveraged. Malaria control interventions strengthened. The Project staff developed collaborative relations with colleague agencies and the donor. They were experienced in the policies and procedures of the NMCP so they understood their role in it. This staff utilization pattern enabled MCC to assign the technical staff where and when they were most needed and the provincial MCC staff were primarily engaged in coordination and back-stopping. Clear lines of authority and responsibility allowed for staff being utilized flexibly according to the needs of the program. The AOPs were developed at the provincial level and through that process gaps in the public programs were identified. The Project adapted to the needs of the NMCP in the target areas. MCC participated actively at the national level including collaboration with the CNM in the planning for additional initiatives and in the Project areas with the implementation partners. This included contributions to the design of the Containment Project in 2010-2011. In addition, MCC staff participated in national planning for the work of the NMCP and implementing partners under the Global Fund to fight AIDS, Tuberculosis and Malaria to enhance complementarily of efforts. MCC advocated for more responsive policies at both the national and local levels Page 40 Components Outputs Outcomes Achievements 6. Monitoring & Evaluation Extensive data were collected throughout the Project and used to prepared status reports. Regularly. The MCC’s M&E Plan listed 3 impact, 22 outcome and 23 output indicators Data were collected to measure performance against indicators. M&E Plans and Annual Reports were prepared regularly. Program data were collected systematically and in collaboration with the NMCP. While the clinical data could be utilized for both QC and as a basis for monitoring treatment outcomes, the epidemiological data (i.e. morbidity and mortality) reflected the impacts of the combined interventions on the epidemic. . Data collection was coordinated with local health facilities. MCC designed and implemented a baseline survey among the targeted households to assess the knowledge and practices related to malaria prevention and care and treatment during the first year of implementation. The same survey was repeated in the last year of the Project to determine changes of the situation at both output (Knowledge) and outcome (behaviors) levels. Baseline and end line surveys were conducted. 7. Private Sector Over 100 private providers mobilized in Pailin Province. PPM increased compliance with the NMCP referral policy The PPM Pilot Project was implemented in Pailin Province 8.Operations Research NMCP referral system was strengthened. MCC contributed to refining methodologies for operations research. A referral system was developed to assure proper treatment of severe cases RDT tests found to meet standards The quality of RDT used in communities and in health facilities was assessed in two studies. 48 of 71 patients with drug resistant Pf tracked for testing on day 28. Multi-drug resistant malaria cases were located and tested. Page 41 ANNEX II: ANALYSIS OF MCC INTERVENTION MODELS The development and replication of intervention models is one of the expected intermediate results of the USAID/RDMA’s strategy for health programs and identifying candidates was one of the tasks for this evaluation. MCC developed and/or (re)vitalized a number of intervention models during the 50 months of implementation. Below are summaries of the key intervention models, their strengths and weaknesses, and areas for consideration when replicating and/or modifying the models for future designs of similar projects. 1. Engagement of VMWs 1.1. Strengths: a) In addition to their role as a health educator or health information disseminator, the VMWs also were trained and equipped with the essential tools to provide EDAT to community members. This allowed them to function as the focal point in their village for the whole continuum of malaria services from prevention to diagnosis, treatment and referral. b) In addition to the continuum of malaria services, the VMWs also were involved in monitoring the malaria situation in their communities by engaging in community surveillance activities. c) Since access to public health facilities was still low due to the constraints on accessibility, VMWs were critical points of entry to services at the grass roots level. The increase in confirmed cases and elimination of presumptive treatment in the public health system was largely due to the VMWs in the last two years of the Project. The combination of the EDAT at the community level and reduced malaria treatment in the private sector because of the enhanced access to quality services locally is an important strategy for reducing drug resistance. d) The MCC employed a strategy to recruit married couples as the VMWs. This was not only to achieve a better gender balance but also to ensure that at least one of them would be available to provide services. e) Many of the human and financial resources to establish and maintain the VMW system were supported by MCC, but training of VMWs was conducted by the government health staff, usually the Provincial Malaria Supervisor and his/her team members. Responsible health facilities and OD staff conducted joint monitoring visits with MCC staff to provide additional support. The VMWs thus functioned as staff of the health center and were integrated into the government system. This created ownership of the VMW system among the government staff and enhanced the capacity of the government to develop and manage the volunteer system. 1.2. Weaknesses: a) While the Project’s strategy to recruit married couples as the VMWs had some benefits, the downside of this strategy was that the VMW services were focused at one delivery point in each targeted village since there were only two VMWs per village on average. The points of services were far from many community members, especially those who were at high risk for malaria because they lived at the outskirts of the villages closer to the farms/forests. b) The level of VMW turn-over was reported to be very low, but there was great variability in their competencies. The health education sessions observed ranged from 15-45 minutes, Page 42 depending on the competencies of the VMWs. The older VMWs tended to be more confident and more skillful in facilitating some discussions during the sessions while the younger ones tended to read scripts from the flipchart. This was also reflected in comments by the participants attending the sessions led by the younger VMWs who noted that they needed to attend the sessions several times to understand the contents. 1.3. Replicability: Volunteer systems are commonly implemented for health and other community based interventions, in rural areas and in other resource-constrained settings but they also have a place in urban areas and countries with more advanced economies. Although MCC neither systematically documented what made the VMWs system work nor conducted an evaluation of its effectiveness, replicability of the VMWs system to other sites seems quite feasible. Key points of concerns for replication are to ensure: 1) a sufficient investment to have an adequate number of volunteers to be able to produce tangible results; 2) technical and management capacities of the implementers to build the VMWs’ capacity and to manage the system; and 3) a dependable supply of RDTs and drugs to sustain the system. The trained local volunteer is a very cost-effective model and is recommended for situations where malaria morbidity and mortality, or other serious health issues, are high and availability of and access to services at public health facilities are inadequate. 1.4. Sustainability: The investment in human capacity building in a systematic way can be costly since it requires a long-term commitment and on-going support. The end-of-project assessment1, 2 conducted in April 2011 that was commissioned by the MCC estimated that the Project had to spend about US $ $313 to support each VMW per year. This cost only included direct and staff costs and excluded the contractor’s administrative and other costs borne by the NMCP (e.g. RDTs and medicines), and was consistent with an average of 2 VMWs for each of the 230 targeted villages. The lack of both a sound evaluation system and an exit strategy left the sustainability issue of the MCC VMW system unclear. But this became a mute point when the CAP-Malaria Project was implemented in the same areas and support for the VMWs continued. In the MCC Project the VMWs were seen as an extension of the health centers and their work was regarded as a part of the free services provided by the Government. Meaningful community support for the VMWs seemed to be discouraged in the interest of highlighting the contribution of the public services, . Sustainability would be enhanced if there was some formal accountability to the local community served and they contributed in some ways to the support of the VMW. 2. LLIN Lending Scheme for MMPs 2.1. Strengths: 1 Development Business Associates.,Assessment of Malaria Control in Cambodia Project (/Singapore, April 2011) 2. 2 Note that the assessment was conducted in March 2011, thus, the data used for the costing analysis did not include the actual costs for the entire Project. Also since the MCC did not have plan to conduct costing analysis, the data used for the analysis was based on the best estimation from the MCC staff. Page 43 In accordance with the National Strategic Plan, the use of insecticide-treated mosquito nets is a high priority for the prevention of malaria transmission. An assessment by the MCC found that about half of the MMPs carried bed nets from their homes but they were mostly untreated nets. The Lending Scheme helped to familiarize the targeted beneficiaries with the ITNs/LLINs and encouraged the targeted beneficiaries to translate malaria prevention knowledge regarding the use of LLINS into practices. 2.2. Weaknesses: Strong management and monitoring systems are necessary for any intervention model and these were inadequate. This evaluation found that many MMPs carried the lending nets with them when they moved to work with new employers or when they returned to their homes instead of leaving the nets for their peers. This was also found in the qualitative assessment of the scheme conducted by the Project in 20113. As a result, some employers had no nets and could not provide them to the new workers while some had numbers of unused nets. Some informants complained that the lending nets were of low quality, e.g. too big holes, too short, and too small. Some of them did not take good care of the nets, resulting in shorten life of the nets while some said the nets received were too dirty. Despite the knowledge on the scheme, some of them preferred to purchase larger and better quality nets from the market although they were regular untreated nets and quite costly4. 2.3. Replicability: In principle, the scheme was technically sound and there was no doubt of its replicability. The MCC’s evaluation conducted in 20115 indicated good understanding and cooperation from the employers since the scheme provided many benefits to them at no cost, e.g. free LLINs, less sick workers, and more satisfied and loyal workers as they believed that the employers took good care of them. However, it is essential to conduct a thorough assessment or evaluation to identify a clear plan for management and monitoring of the scheme prior to replicating or scaling-up the scheme. The analysis should include a plan for sustaining the scheme. 2.4. Sustainability: Since the majority of the lending nets were supplied by the NMCP, Containment Project, and other donors, the cost for implementing this scheme was modest, i.e. US $ $10/employer/year6,7. If the cost for LLINs were to be covered, approximately US $ 5/MMP/year would be required, assuming that none of them had LLIN and the nets were taken away. If the nets were well managed and maintained; only additional US $ $5/MMP/the life of the LLINs would be required but this situation tended to be rare as the nets were often used in a harsh conditions. Since the Project estimated 20 MMPs/employer, the maximum of approximately additional US $ $286,100 would be required for a project with similar size 3 University Research Co. LLC. Assessment on LLIN Loaning Scheme for Mobile and Migrant Population and Family Members along the Cambodia-Thai Border. Phnom Penh: MCC Project, February 2011. 4 Approximately KHR 85,000 or US $ 21.25. 5 University Research Co. LLC. Assessment on LLIN Loaning Scheme for Mobile and Migrant Population and Family Members along the Cambodia-Thai Border. Phnom Penh: MCC Project, February 2011. 6 Development Business Associates. Assessment of Malaria Control in Cambodia Project (Singapore: MCC Project, April 2011), 2. 7 Note that the assessment was conducted in March 2011, thus, the data used for the costing analysis did not include the actual costs for the entire Project. Page 44 of population coverage and length of the project life8. Although the estimated cost for establishing the LLIN lending scheme was not too high, the nets have to be replaced regularly. Without a co-payment from employers or MMPs, as well as capacity building and supervision for employers to appropriately manage the scheme by themselves in the future, sustainability of this model is questionable. 3. Involving Taxi Drivers as Health Mediators 3.1. Strengths: a) MMPs often live in isolation and work in remote areas so MCC made efforts to identify them before they were infected by malaria. Effective communication channels for BCC other than conventional printed materials, media, and community campaigns were developed.. b) Since MMPs in this context are economic migrants, many have low socioeconomic status, many could not read the billboards and other printed materials, and many could not be reached by the radio programs delivered by the Project for both lack of a radio and weak/no signal. Using human media like the taxi drivers proved to be a more effective means to reach such marginalized populations, especially since verbal communication was not a barrier for internal migrants in Cambodia. c) The cost for this intervention was minimal. Besides providing materials (e.g. leaflets, stickers, and audio spots/drama) that could also be used with other types of interventions and basic malaria trainings and refreshers to the drivers, no other major cost was required. 3.2. Weaknesses: a) Although they were generally visible as more than half of them (17/31) still had stickers on their vehicles during the mystery client survey9 conducted by the MCC in 2011. Interviews with selected taxi drivers for this evaluation revealed that they were not very active as comparing to VMWs, probably because of their lower level of commitment and empowerment. b) Both the survey and the interviews found that they selectively talked to the new passengers but not those they felt that they have already met; resulting in only 20% (6/31) of the drivers talked to passengers about malaria according to the survey finding although the Project assessed that almost half of the passengers (24,146/53,237; 45%) were MMPs10. c) During the interviews, they also expressed the challenge that they could only provide the information but not means for prevention such as mosquito nets and repellents. They also showed some frustration that the LLINs were only available in the villages but many of the MMPs lived outside of the villages. In addition, the survey reported that only 16% (5/31) played malaria audio spots, while 61% (19/31) distributed leaflets to the mystery passengers and had finished all the materials; resulting in only 39% (12/31) still had leaflets in the cars at the time of the survey. 8 The MCC Annual Report for 2011 indicated that 2,861 employers cumulatively participated in the scheme, therefore, 2,861 employers x 20 MMPs = 57,220 MMPs x US $ 5 = US $ 286,100. 9 Kheang Soy Ty. Malaria Control in Cambodia: Annual Progress Report, October 1, 2010 – September 30, 2011. Phnom Penh: MCC Project, October 2011. 10 University Research Co. LLC, Malaria Control in Cambodia (MCC Project), 2007-2011- power point presentation for briefing of EOP external evaluators. Phnom Penh: MCC Project, 25 July 2012. Page 45 3.3. Replicability: a) The use of secondary peer educators or health mediators have been well recognized as an effective strategy to reach the targeted beneficiaries when the use of peer educators is not feasible or has a potential for compromised effectiveness11, i.e. MMPs themselves in this case. b) The model could be modified for the other types of health mediators such as for long distance buses (and/or trains for the case of Myanmar) as the MCC had started working with towards the end of the Project. 3.4. Sustainability: Since the role of taxi drivers was limited to information dissemination, sustainability of the intervention seemed promising as this requires modest resources and support. The taxi drivers, or other types of health mediators, only need to be updated on situation/information as required and be stimulated from time to time. 4. Public-Private Partnership 4.1. Strengths: a) This intervention is a type of hitting the nail on its head approach since use of substandard/counterfeit drugs, non-compliance with the standard regimen and monotherapy by patients seen by the private sector contributes to the problem of drug resistance b) Strong law enforcement from the government to ban malaria treatment in private sectors allowed the Project to minimize investment in this intervention. It consisted mainly of semi￾annual meetings to inform the private sectors on updated policies and national guidelines for diagnosis and treatment. c) Since many of the public health staff are also private health providers, and in fact many of them are also farm owners who hire MMPs, effectiveness of the interventions for malaria control could be maximized through the networks of the same group of targeted individuals in different capacities. 4.2. Weaknesses: a) Both the NMCP and MCC indicated the use of a “carrot and stick” strategy for the PPM - supervision and appreciation plus law enforcement and training. However, this evaluation found no evidence on the impact of “carrots” or other non-financial positive incentives. In fact, the drug outlet survey conducted by the MCC in 200912 revealed that 54-62% of drug outlets in Containment Zone 1 and II sold Artemisinin monotherapy; meaning that many of them already stopped selling the regimen before the Project’s PPM intervention was launched13. One-third of them reported no demand for the treatment due to rapid decrease in number of malaria and suspected cases, 40-46% reported stop selling monotherapy because the drugs were not available, while 88% had no plan to buy more malaria drugs. The 11 Lowe, David and Caroline Francis. Protecting People on the Move: Applying lessons learned in Asia to improve HIV/AIDS interventions for mobile people. Bangkok: Family Health International, 2006. 12 Kheang, Soy Ty and Katherine Herz. Public Private Mix: Malaria Control in Cambodia. Phnom Penh: MCC Project, August 2009. 13 Specific data for Pailin and/or overall Project’s areas were not available. Page 46 proportion of drug outlets with Artemisinin monotherapy at the end line survey was unknown but reported that over 91% last treated malaria more than a year ago. All the private providers and the public health staff interviewed for this evaluation expressed the opinion that the main reasons the private sector stopped treating malaria were because of the MOH’s banning, resulting in unavailability of drugs, and no demand for the services. The evaluation team could easily obtain from a pharmacy at one of the sites visited the unexpired Sulfadoxine+Pyrimethamine; not a recommended treatment according to the national guidelines. b) Since many of the public health staff are also private health providers, the PPM concept itself could pose a conflict of interest. c) The 2011 report indicated that 65% (104/160) of the suspected cases referred actually reached the government services including VMWs, of whom 63% (65/104) were confirmed malaria cases. This could indicate that 35 out of the 56 missing cases were actually malaria patients receiving treatment outside of the authorized facilities and represented a potential for exacerbating drug resistance. 4.3. Replicability: a) Replicability of the PPM as designed and implemented does not seem to be promising. The appreciation certificates provided by the Project (with the MOH) were not strong incentive and some of the private providers visited already had misplaced them. Although some of the private providers visited reported receiving US $ $6 per each successful referral case, this was supported by other projects and some simply referred the patients to public health facilities without records since the number of cases was too small (0-2 cases/months) to make the incentive attractive. b) While all of the private providers interviewed expressed their appreciation for the knowledge on drug resistance and updated policies received from the Project, social responsibility did not seem to be their major concern. The private providers and government staff interviewed also added that it was also because Pailin is a small town and they had no confidence if the PPM would be as successful in other larger cities without law enforcement in place. Yet, the private sector needs to be an active participant if malaria control programs are to be effective. 4.4. Sustainability: Although the replicability of the PPM was questionable, it might have more potential for sustainability for a combination of factors: prolonged law enforcement will continue to make drugs less available for the private sector; a norm for not selling malaria drugs may develop; and the number of malaria cases will continue to drop due to further deforestation and urbanization and continued prevention and treatment interventions. 5. Day-3 Positive Surveillance and Mapping of the Cases 5.1. Strengths: a) This intervention maximized the use of local capacity and resources by involving VMWs in the routine community-based surveillance. b) The intervention demonstrated the use of widely available information technology systems on surveillance and rapid responses. The mobile phone’s short message system (SMS) was used to report Day-3 positive cases and allowed the CNM to conduct timely focal residual Page 47 straying for control of vectors that might carry drug resistant parasites. The global positioning system (GPS) was used to locate Day-3 positive cases. 5.2 Weaknesses: a) According to the same end-of-project assessment conducted in 2011 mentioned earlier, an estimate of US $ $423/case/year was spent to enroll, follow-up, and map confirmed malaria cases for drug resistance. Considering the seriousness of the drug resistant problem, this cost might be considered modest or moderate. b) It was unclear how the results were used other than to call for a timely focal residual spraying which does not need the GPS. The cost-effectiveness of this activity, therefore, is in doubt. 5.3 Replicability: There is no doubt about replicability of this intervention but rather its purposes and usefulness. 5.4 Sustainability: Similar to any geographical information systems, a big effort will be required to sustain the system. To be of value the data must be collected systematically, regular updates and analysis must be made available, and monitoring needs to be done routinely. Page 48 ANNEX III: RECOMMENDED NEW INTERVENTION MODELS This evaluation identified some potential new models for piloting as follows. 1. Telephone Hotline/Consultation: The radio call-in show reported over 200 missed calls. This indicated a large volume of mobile phone use in the targeted areas which has the potential for developing telephone hotline/consultations. 2. Participatory LLIN Lending Scheme: It should not be assumed that employers and MMPs are not able to pay for their health. Instead of providing free ITN/LLINs to employers, copayment from the employers for the cost of the bed nets should be piloted. To avoid resistance after the nets have been distributed for free for a long time, the pilot should start with a small copayment and gradually increase the user’s share. Alternatively or simultaneously, interventions on the MMPs side should also be tested. For example, the MMPs may be requested to pay a certain amount of deposit to be refunded once they return the lending net in good condition. This will not only create their ownership to take good care of the lending nets but will also add the value on the nets in their perceptions as well as to help sustaining the intervention. This, however, will require an assessment of their willingness and ability to pay, efforts on advocacy and community mobilization, and strong management and monitoring. 3. More Meaningful Involvement of Volunteers and Employers of MMPs: Expanding the role of VHVs/VMWs and involving employers of MMPs in the case management and follow-up would bring in additional resources at critical points of the prevention, diagnosis, treatment continuum. 4. Friendly Public Health Service: Although a universal problem with civil servants, efforts to promote a more patient-friendly atmosphere in the public facilities would potentially remove a significant barrier to accessing critical health services. 5. Addressing Malaria among MMPs within a Safe Mobility Context: The information on population migration mapped by the MCC and other projects could be utilized to develop a pilot intervention that addresses malaria issues at source, transit, destination, and returned communities of MMPs. The pilot could perhaps start with trans-district within the same province as the household survey conducted by the MCC found that more than one-thirds of the surveyed households stayed overnight or worked in other districts in their provinces. Since not all the projects can be implemented in all key communities of the migration process, a franchising service in close collaboration with other partners might be required for piloting an intervention across provinces within Cambodia, or trans-national within the Mekong region. The networked services could help improving the tracking of cases. 6. Vector Control Measures: With only the strategy for ITN/LLIN use in place, people in the high risk areas where outdoor early biting is prevalent are exposed to a significant risk of malaria. Follow-on projects should identify evidence of effectiveness of other means for vector control and/or for reducing vulnerability to mosquito bites such as the use of repellents. Page 49 ANNEX IV: STATEMENT OF WORK 1. Statement of Need and Audience for the Evaluation The USAID/Regional Development Mission Asia (RDMA), USAID/Cambodia, The US President’s Malaria Initiative (PMI), the Royal Government of Cambodia and the National Center for Parasitology, Entomology and Malaria Control (CNM) want to learn from experiences of the MCC Project. These groups are particularly interested in the MCC Project’s effectiveness to provide preventive and case management (intended to promote effective use of insecticide treated materials, i.e., bed nets and hammocks) and treatment through accurate diagnosis, directly observed treatment and follow up. The MCC Project has produced numerous reports and studies based on Project experience, but these must be evaluated independently to assess their relevance to potential new target areas. 2. Evaluation Type, Purpose and Key Questions This evaluation will be a performance evaluation as defined in the USAID Evaluation Policy (see Appendix 1). The main purpose is to assess the project performance and its impact. The evaluation will provide insights and important feedback to each of the partners and stakeholders that should assist them to understand both the strengths and areas where technical, administrative and management efforts could be improved. It will also provide evidence and learning for adapting future programs and projects, including improving USAID/RDMA designs, strategies and policies. General evaluation questions: 1. What did the project achieve (i.e. what were the actual results) relative to the expected results and outcomes as stated in the contract? The evaluation team should assess evidence of project coverage, effectiveness of the interventions, efficiency of program delivery, impact and sustainability. 2. Where did implementation fall short of achieving expected results and what factors including gaps or shortcomings constrained project performance? 3. Over the course of implementing the project, what major lessons learned and best practices emerged? 4. What are potential areas and considerations for a follow-on program with similar objectives? Specific evaluation questions: 8. How many beneficiaries were reached and what were the differences the project has made to the beneficiaries, as perceived by beneficiaries and other project participants? 9. Is there evidence of improved accuracy of malaria diagnosis, adherence to treatment guidelines, and detection of treatment failures? 10. Is there evidence of impact from this project’s activities, including reductions in malaria morbidity or mortality, or hospitalizations? 11. Is there evidence of potential replicability of the project elsewhere? 12. What were the major factors which influenced the achievement or non-achievement of sustainability of the project? To what extent did the benefits of the project continue after our funding ceased? Were there any activities undertaken to assure sustainability? 13. Has there been evidence of improved knowledge or practices of community members on malaria? Page 50 14. Has management for malaria control at district and provincial health systems been strengthened i.e. improved quality of services, increased capacity of health care providers, etc.? The evaluation team will make a particular effort to document lessons learned that are of broader relevance to Cambodia and other areas of the Greater Mekong Sub region, exhibiting artemisinin￾resistance. In most cases, this will build on existing URC and other professional publications, building from documented successes and difficulties and considering how likely they might be to apply in other contexts, specifically across Southeast Asia countries. 3. Evaluation Methodology To carry out the Evaluation, the Evaluation Team will need to consider the following. 3.1. Team composition This SOW is for two international consultants: 1) Case Management Specialist and Evaluation Team Leader and, 2) Community Behavior Change Communication Specialist. Both will have expertise in conducting evaluations. The Evaluation Team members will be required to provide a written disclosure of any conflicts of interest, to include but not limited to personal, financial, material, or professional. 3.2. Evaluation process STEP 1: Desk Review, Evaluation Methods and Work Plan • Conduct a desk review of key project and related documents including the Performance Management Plan (PMP), Results Framework, baseline assessments, gender disaggregated data, periodic progress reports, and other documents in which project activities and findings are discussed. • Itineraries to visit project sites will be developed in conjunction with USAID/RDMA. Before this phase is complete, the team should have developed any data collection instruments necessary for the methods being used, such as interview protocols, sampling plans, direct observation checklists, survey questions, etc. STEP 2: Field Review • The Evaluation Team will discuss the MCC Project with key informants as set out in the evaluation methods and work plan. The selection of key informants should be done in a way to reduce selection bias and ensure a broad range of perspectives. • The Evaluation Team will use at least one (but not limited to one) other data collection and analysis method to be proposed. Some examples could include doing onsite surveys or interviews with program beneficiaries at random as they visit project sites; analyzing secondary data such as national level or institutional level demographic or administrative data that is already being collected; professionally moderated focus groups with program stakeholders; or other approaches as proposed by the Team and agreed to by USAID/RDMA STEP 3: Data Analysis During field work, and for up to 3 days after the completion of the Field Review, the Evaluation Team will perform sufficient data analysis enabling the compilation of preliminary findings to be presented to Page 51 USAID/Cambodia and RDMA. Data will need to be disaggregated by gender where appropriate and as required by USAID Evaluation Policy as well as by other relevant factors e.g. region, sites, age, health status, income level, etc. STEP 4: Debriefing Sessions Within approximately three days after the completion of the Field Review and data analysis, the Evaluation Team will hold de-briefing sessions for USAID/Cambodia and RDMA. The team will present an Aide Memoire, and seek feedback from the project implementing partner, as well as relevant USAID/RDMA staff including members of USAID Monitoring & Evaluation Working Group, on their preliminary findings. The team will make a formal presentation open to all RDMA staff. STEP 5: Report Writing and Finalization A Draft Report on the findings of the evaluation will be prepared by the Evaluation Team. USAID/Cambodia and RDMA will review and also share the draft report with the implementing partner and other stakeholders as USAID determines appropriate and provide comments within two weeks. The Evaluation Team will prepare a Revised Draft Report after receiving comments from USAID/Cambodia and RDMA. The Evaluation Team will incorporate changes and comments, as the Team sees fit. Where there is strong disagreement in views between the Evaluation Team and USAID, these should be noted in an appendix of the Revised Draft Report, known as a Statement of Differences. USAID/RDMA will review the revised draft report and provide comments within 5 working days. The Evaluation Team will consider all comments in preparing the Final Report, which will be submitted within 5 working days of receiving final comments from RDMA. The Final Report should meet the criteria outlined in USAID’s Evaluation Policy and will be uploaded by USAID/RDMA/Program Development Office to the public area of the Development Experience Clearinghouse within three months of being finalized. 4. Gender Consideration The Evaluation Team should address relevant gender concerns within the MCC project area. Recommendations should outline the most significant gender issues that needed to be considered during activity implementation. Describe how both women and men involved were affected by the context or the work undertaken. The desk review should include a specific gender analysis. All data in the Evaluation should be disaggregated by gender. The final report should include a section on gender analysis. 5. Deliverables The Case Management Specialist/Team Leader will be responsible for coordinating the drafting of deliverables, consolidating and submitting the final report. The Community Behavior Change Specialist will be responsible for contributing to all deliverables and drafting sections of all documents and presentations based on his/her expertise and the tasks assigned by the Team Leader. The required deliverables as a joint output for the Evaluation Team include: a. Final Evaluation Report will be submitted within 5 working days of receiving comments from USAID/RDMA. The final report will include: - Executive Summary of the Evaluation, no more than 3 – 5 pages, that summarizes key points related to the project, evaluation purpose and design, and findings. - A main report; Page 52 - At minimum the following appendices: Evaluation SOW, Final evaluation design and work plan, any statements of differences, all data collection tools, and identify sources of information. b. Any data and records collected by the Evaluation Team will be provided to USAID/RDMA in an organized and readable format. c. The main findings will be presented in PowerPoint format and a briefing session on the final Evaluation and findings at USAID/Cambodia and RDMA. Page 53 ANNEX V: EVALUATION DESIGN AND METHODS 1. Evaluation Design The evaluation will be consistent with USAID’s Evaluation Policy (January, 2011). Scope of Work for this evaluation is consistent with what the Evaluation Policy defines as a ‘performance evaluation’. This type of evaluation focuses on descriptive and normative questions: what a particular project has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to project design, management, and operational decision making. The evaluation team will use a mixed method evaluation approach, triangulating quantitative and qualitative data analysis. The quantitative data will provide insights on outputs and potential outcomes and possibly impacts of activities as the qualitative data will do on effectiveness of processes employed to achieve the project’s objectives. Reiterative approaches will be used to triangulate and integrate data from different sources in the development of findings and recommendations. The evaluation team cannot assure the five data quality standards set out in USAID’s policy for the secondary quantitative and qualitative data already collected by the contractor and its implementing partners. However, qualitative data collected by the evaluation team will meet the five data quality standards and in analyzing secondary data already collected, the data will be assessed for its quality. Findings on the secondary data quality will be addressed in the report and caution will be taken in the development of findings and recommendations. Relevant data from other sources will also be reviewed and utilized to support a particular finding as required. Key contextual issues that the evaluation will need to consider include:  Cambodia’s National Strategic Plan for Malaria 2006-2010 and 2011-2025: in which the MCC and the follow-on project, CAP-Malaria, aim to contribute to.  Other donor funding: Since there are many key players in the MCC targeted areas with much larger level of funding and the MCC also collaborated and/or received some in-kind contribution from other projects, this evaluation will need to consider the extent of the MCC’s actual achievements.  Operational environment: Not only malaria but also other health programs in Cambodia work in a difficult operational environment due to a variety of factors; dysfunctional government service systems, weak community systems, and generally low level of human and organizational capacity. The evaluation will also determine how the contractor worked within this context and its effectiveness in capacity building and advocacy that are key objectives of the project.  Gender: Since gender role can have an impact on health beliefs, practices and outcomes, and this is also one of the key issues specified in the PMI, the evaluation will take this issue into consideration for data collection and analysis. Page 54 2. Evaluation Methods 2.1. Methodology and Timeline This evaluation is commissioned by the USAID/RDM-A and will be conducted over a two months period from July 28 - September 28, 2012, according to the overall evaluation schedule listed in the table below. Evaluation Timeline (July 28 - September 28, 2012) Tasks Timeline Review of relevant materials and documents June 28 - July 20 Meeting with relevant staff at USAID/RDM-A, interviews with relevant informants in Thailand, and preparation for field work in Cambodia July 23 - 24 Field work in Cambodia (including briefing and debriefing with USAID Mission, URC and its implementing partners in Phnom Penh) July 25 - August 9 Debriefing with USAID/RDM-A August 10 USAID/RDM-A Mission-wide Presentation August 14 Submission of the draft report September 15 Submission of the final report September 28 The Evaluation Team will employ both quantitative and qualitative methods to this evaluation. The quantitative method is mainly to analyze the secondary data related to the Project’s implementation, particularly in terms of its process and outputs (relevance and efficiency), outcomes (effectiveness), and impacts. Determination of key constraints contributing to the Project’s shortfalls is qualitative in nature, and the data will be obtained through a primary data collection during the field visits to the Project’s sites. In addition to the individual interviews and/or group discussions, some other qualitative methods such as free-listing, ranking, and others will be used to obtain relevant information as deemed appropriate. Besides the Project’s implementation, this evaluation will also include other social and environmental factors that might pose significant affects to the Project’s results, especially in terms of potentials for replication of innovative interventions in future programs as well as for sustainability. The evaluation team will undertake the following key tasks for this evaluation. However, since the evaluation will be conducted based on a reiterative process, these tasks will not necessarily be conducted in a sequence as listed below and some tasks will be naturally split into several steps and/or repeated as deemed appropriate. 2.1.1. Desk review of relevant materials: The team will first obtain available secondary data from the Project’s work plans, reports, and special studies to gain better understanding on the MCC’s goal, objectives, and interventions as well as to analyze preliminary findings on the Project’s achievements. The team will also review additional relevant documents such as the Strategic Master Plan for National Malaria Control Program and the Project’s Midterm Review and Lessons Learned Assessment Reports as well as those considered to have influences on the MCC such as those related to the US President’s Malaria Initiative (PMI), the Global Fund to Fight AIDS, Tuberculosis and Malaria, and the Containment Project funded by the Bill & Melinda Gates Foundation. As needed, additional available materials will be acquired during the field visits as well as after returning from the field. The rapid data analysis will be conducted in the field to allow for adjustments of further data collection and analysis. 2.1.2. Development of evaluation design and methods and briefing: Following the initial desk review, the team will develop detail evaluation design, field data collection instruments for various types of informants (see Annexes V and VI), plans for field visits and informants selection, in close consultation with the focal persons at USAID/RDM-A. The team will then brief relevant staff of Page 55 the USAID/RDM-A and Cambodia on the preliminary results of the desk review and evaluation plan to obtain initial inputs for finalization of the plan. 2.1.3. Field data collection: In addition to the field visits to the MCC Project’s sites in Cambodia, the Team will also conduct face-to-face and telephone interviews with key regional stakeholders based in Thailand, both prior to and following the field visits. The actual field visits to Cambodia will involve interviews with relevant government and non-government informants at both central and peripheral levels, and the targeted community members. In addition to the individual interviews and/or group discussions, some other qualitative methods such as free-listing, ranking, and others will be used to obtain relevant information as deemed appropriate. The field visits will also include spot checks and observations of selected activities, both at communities and health facilities, conducted by the Project’s team who continue to implement activities under the followed-on CAP￾Malaria Project. At a glance, the evaluation team will visit selected areas in all 5 targeted ODs of the 4 targeted provinces and interview relevant staff from URC and its partners, public health administrative offices, public health facilities, school health department, as well as those in the community such as volunteers, taxi drivers, private health providers/drug stores, community members including migrant workers, and malaria patients and/or their family members. 2.1.4. Data analysis and verification: The quantitative data extracted from several Projects’ reports will be reanalyzed in the Excel spreadsheet while the qualitative data collected from relevant informants during the field visits will be analyzed using the grounded theory. Triangulation of the quantitative and qualitative data will also be conducted for further data analysis and interpretation. This was to ensure the accuracy and reliability of the data and to the most possible extent the linkage between the Project’s performance (factors contributing to its achievements and constraints) and the outcomes and impacts on malaria epidemic. To ensure completeness and comprehensiveness of the data, a reiterative process will be applied during the field qualitative data collection. 2.1.5. Debriefing: An Aide Memoire will be prepared and shared with relevant partners and donors at two debriefing sessions. An in-country debriefing will be conducted among staffs from the contractor and representatives from its implementing partners and the government counterparts to present the preliminary findings, validate the findings, and to obtain additional inputs. Another debriefing with the USAID/RDM-A and Cambodia will also be conducted for the similar purpose. 2.1.6. Report writing: Inputs from the two debriefing sessions will be incorporated into further analysis of the data. The evaluation team will then prepare a draft evaluation report based on the criteria set forth in the Scope of Work and according to the USIAD’s format. The report will be emphasized on demonstrating the quantitative and qualitative evidence on which constructive recommendations will be made. 2.1.7. Report submission and finalization: The evaluation team will submit a draft report to USAID/RDM-A for sharing with relevant reviewers. USAID/RDM-A will be responsible for coordinating feedback from relevant reviewers and forward comments to the evaluation team within 5 working days. Up on receiving USAID/RDM-A’s feedback, the evaluation team will finalize the report and submit the final report to USAID/RDM-A within 5 working days. Page 56 ANNEX VI: DATA COLLECTION INSTRUMENTS Three types of interview/discussion guides were developed for different types of informants including: 1) all types of health service providers (e.g. public health care providers, VMWs, taxi drivers); 2) community members (e.g. local residents and MMPs); and 3) partners and/or stakeholders (e.g. implementing partners, health administrators, private sectors). 1. Interview Guide for Health and Other Services Providers Informant: ______________________ Affiliation: ____________________ 1) How long have you been on this position? 2) What services you provided to the beneficiaries/patients under the MCC Project? Probe: How? Caseload / workload? 3) Please describe your referral system. Probe: Any challenges on referral? How about networking and feedback mechanisms (both vertical & horizontal)? 4) What kind of training(s) did you receive from the Project? Probe: How often? What were their qualities (Qualified trainers, well-prepared, etc.)? Are they relevant to your roles & responsibilities? Why / why not? How did you apply the knowledge/ skills in your work? 5) Any activities that were usually implemented as planned? Probe: What were key factors contributing to such achievements? 6) Any activities that were usually not implemented as planned? Probe: What were the key obstacles? Experiences of supplies stock raptures? 7) Any difficulties you faced in providing the services to the community members or patients? Probe: Environmental & contextual barriers? How did you overcome the challenges? What kind of support/ supervision/ mentoring you received? From whom? Were they helpful? What kind of advocacy/actions undertaken to address the challenges? Efforts for community mobilization and/or empowerment? 8) What positive changes the MMC made to the community? Probe: Whether / to what extent the changes have been sustained? How? 9) How about the health service system? What were changes and how they have been sustained? 10) What services did the community members/patients requested that you were not providing? Do you provide such services now? If not, why not? 11) How did you/your team design the implementation model (or BCC strategies)? Probe: Who involved in the design? How the targets were set? Whether/how previous experiences and data/information were used in the design? Probe: What was your involvement in the Project M&E? 12) How well do you think the implementation model/BCC strategies fit to the local context? What fit/ what did not fit? Why? 13) In your opinion, were the services effectively implemented? Why/ why not? Probe: How well the services met the needs of each of the community member groups? [Forest goers, MMPs, women, etc.] Who were the 2nd peer educators? Whether/ how did you work with them? How effective they were? Page 57 14) Any other governments/NGOs providing malaria services in your targeted areas? Probe: Who are they? What did they do? Any duplication of the services? Whether/how did you collaborate/coordinate with them? Any added value from the collaboration/coordination? 15) What were some outstanding innovations and/or good practices from the MCC project in your opinion? Why you considered them as good practices? 16) In your opinion, what would be the best ways to improve effectiveness of malaria prevention and control in your community? 17) The interviewer to add questions reflecting comments from others/ related issues. ********End of the interview******* Page 58 2. Interview Guide for Community Members and Patients Informant: ______________________ Location: _____________________ Part I: Personal and Migration Data 1) Sex 2) Age 3) Marital status 4) What do you do for living? Probe/observe Living conditions & associated risks to malaria infection. If forest goer and/or, live in farm/ plantation/ forest? 5) Where are you from? Probe If from other towns, have you ever visited your home town since your move here? If so, how many times? When was the last time? Migration routes, means, reasons, malaria risks along migration routes 6) How long have you been living in this town? Part II: Health Condition and Health Service Access 1) Have you ever been sick in the past 12 months (other than malaria)? Probe What type(s) of illnesses? When? How did you handle it? 2) Tell me any difficulties you encountered in accessing public health services. Probe Distance, money, provider’s attitudes, etc. 3) What kind of improvements would help you to easier access the health service? Part III: Malaria Related Knowledge, Awareness and Practices 1) Tell me what you know about malaria. What’s it liked? Probe What causes malaria? How malaria is transmitted? How can it be prevented? Can it be treated and cured? How? Can it be recurred / relapsed? Who are at higher risk for getting malaria? 2) How did you learn about malaria? 3) Have you ever participated in any malaria prevention activities in your current living/working area? Probe If yes  What contents? When? Where? By whom? If participated more than once, probe about the last time. How much you could understand the contents? What were obstacles? 4) Did you get enough information (or enough good information) to protect yourself and/or your family from malaria? If no, what kind of information do you need? Probe If you wanted to find out more about malaria, where would you look for information? (newspapers, TV, NGOs, health care providers, etc.) 5) Do you know or have heard of any government or NGOs that provide malaria information, prevention and treatment services to people in your community? Probe Which ones? How would you get in touch with them? Do you think you would get in touch with them? Why/ why not? 6) Do you usually sleep under a bed net? Probe If yes  How often? What type of net? How about last night? How did you get the net? How many nets you have for how many members? How do you share the nets? If possible, could you show me the net, and how you use it? Probe If know about malaria transmission & prevention but don’t always prevent, what are the obstacles for not preventing mosquito bites? Page 59 Probe If do not know about malaria transmission & prevention but usually sleep under a bed net, what’s your purpose of using a bet net? What does it mean to you? Probe If forest goers/farm workers  Do you know the hammock net? Have you ever seen it? Do you have it? Whether/how often you use it? Why/ why not? Whether you believe that it can prevent malaria? Why/ why not? If possible, please show me the hammock net and how you use it. 7) Have you ever got malaria before you moved to this village? Probe If yes  When? Where? How many times? How did you know it was malaria (last time if got many times)? How did you handle it? Probe Treatment including self-medication, mosquito net use. Any support received? From? How? How long had you been on medication? If didn’t complete medication, why couldn’t you? What were the difficulties? (Socio-environmental factors, side effects, etc.) Who /what helped you to overcome such difficulties? (VMWs, taxi drivers, health care providers, employers, etc) How  Counselling, home visit, transportation support, etc. Anything in particular that you needed to do while on medication? Who told you that? How well you could follow? What was the treatment result? Who told you? Tell me your understanding about it. Anything you do to avoid getting malaria again? 8) Have you ever got malaria since you moved to this village? Probe If yes  When? How many times? How did you know that it was malaria (last time)? How did you handle it (last time)? (Treatment including self-medication, mosquito net use.) Any support received? From who? How? Anything in particular that you needed to do while on medication (last time)? Who told you that? How well you could follow? How long had you been on medication (last time)? If didn’t complete medication, why couldn’t you? What were the difficulties? (Socio-environmental factors, side effects, etc.) Who /what helped you to overcome such difficulties? (VMWs, taxi drivers, health care providers, employers, etc) How  Counselling, home visit, transportation support, etc. What was your treatment result? Who told you that? Tell me your understanding about it. Anything you do to avoid getting malaria again? None 9) The interviewer to add questions reflecting comments from previous informants on misunderstanding/ myths AND key factors contributing to preventive/ risk behaviors that haven’t been raised in this interview. 10) If you could share with me any ideas on how to support people in your community to ensure the malaria prevention and treatment success, that will be very much appreciated. ********End of the interview******* Page 60 3. Interview Guide for Other Partners/Stakeholders Informant: ______________________ Location: _____________________ 1) How would you describe your/your organization’s role in malaria control in Cambodia? 2) What was your/your organization’s role in the MCC Project? Probe: How did you become involved? How were you/your organization involved in the MCC Projects? 3) How would you describe the actual relationship? Was it what you expected? 4) Any capacity building opportunities received? 5) What was your understanding of the purpose of MCC? 6) What were the areas of greatest strength and success of the MCC Project? Probe In what ways did MCC succeed? To what extent that it achieved that purpose in your opinion? What made you feel that way? 7) What were the most important contributions of MCC? 8) What were the areas where the achievements were most dissatisfactory? 9) Did the MCC Project strengthen the public sector health work? What areas? How? Are these sustainable? 10) How the MCC collaboration was done? How would you rate MCC on collaboration? 11) What is your understanding on the project’s “community-based initiative” concept/model? Probe Examples of community-based initiative the project had implemented. Were the initiative(s) technically sound? Were they effectively implemented? 12) What were they key elements of the project that should be prioritized for sustaining if the funding ceased or reduced in the future? How to? 13) What were the most important lessons learned and good practices? 14) In your opinion, what would be the best way(s) to improve effectiveness of malaria prevention and related services in your community/country? 15) If a similar project is to be designed, what changes would you recommend? Probe What should be replicated/modified for future programming? What should be avoided? ********End of the interview******* Page 61 ANNEX VII: SOURCES OF INFORMATION 1. List of Informants Interviewed Date Name of Informant Position & Affiliation July 23 & August 31 Dr. Wayne Stinson Regional Malaria Supervisor, USAID/RDM-A July 24 Dr. Aye Aye Thwin Director, Office of Public Health, USAID/RDM-A Ms. Ravipa Vannakit Strategic Information Specialist, Office of Public Health, USAID/RDM-A Ms. Netima Lohpattananon Secretary, Office of Public Health, USAID/RDM-A Ms. Marisa Sanguankwumdee Project Management Assistant, Office of Public Health, USAID/RDM-A Ms. Thitima Klasnimi COR of MCC Project, Chief, Program Support Unit, Office of Public Health, USAID/RDM-A Dr. Chantha Chak Infectious Disease Team Leader Office of Public Health, USAID Cambodia Mr. James Hopkins Chief of Party, GMS-RID Project, Kenan Institute - Asia Dr. Asawin Likhitsup Technical Consultant, Medicines Quality Control, US Pharmacopeia July 25 Dr. Kheang Soy Ty Regional Director/Chief of Party, URC Cambodia Dr. Ngnon Sokmar Country Manager, URC Cambodia Dr. Chy Say Malaria Technical Officer/Research Coordinator, URC Cambodia Mr. Ek Sovan PPM/Migration Coordinator, URC Cambodia Ms. Kharn Lina IEC/BCC Specialist, URC Cambodia Dr. Yung Kunsothearith Chief of Technical Bureau, School Health Department Dr. Chavyroth Vice Chief of Technical Bureau, School Health Department July 26 Dr. Song Ngak Technical Director and Head of Operation, FHI Cambodia Dr. Doung Socheat Former CNM Director Dr. Tho Sochantha CNM Vice Director Dr. Abdur Rashid Medical Officer, Malaria, Other Vector Borne and Parasitic Diseases, WHO Cambodia Dr. Steven Bjorge Malaria Team Leader, WHO Cambodia Dr. Koy Borey Executive Director, Media One Mr. Graham Gardner Consultant, Equal Access July 28 Dr. Ouk Vithea Battambang Provincial Coordinator, URC Cambodia Mr. Chan Sophorn District Coordinator, Battambang, URC Cambodia Mr. Mean Phav District Coordinator, Battambang, URC Cambodia July 30 Dr. Chan Davoeung Provincial Malaria Supervisor, Battambang Mr. Seng Saven Malaria Officer, Battambang Mr. Preung Saroa Malaria Laboratory Officer, Battambang 3 taxi drivers Taxi Driver, Battambang-Sampav Loun Route 4 taxi drivers Battambang-Samlot Route, Battambang Mr. Ses Sarim Chief of Tasanh Health Center, Samlot, Battambang Mr. Pov Pheng Vice Chief of Tasanh Health Center, Samlot, Battambang Mother of a malaria patient Samlot, Battambang VMW O’ treng village, Samlot, Battambang VMW O’ Nanong village, Samlot, Battambang July 31 M.A Um Bunthy Malaria Supervisor, Sampov Loun OD, Battambang VMW Damnak Beng Village, Sampov Loun, Battambang Mr. Ky Sokha Chief, Damnak Beng Village, Sampov Loun, Battambang Page 62 Date Name of Informant Position & Affiliation Dr. Sio Lean Seang Director, Battambang OD, Battambang Dr. Tat Veyvath Malaria Supervisor, Battambang OD, Battambang Mr. Phuong Mony Deputy Chief, School Health, Provincial Education Department, Battambang Mr. Ly Senary Director, FM 92.70 MHz, Battambang Ms. Ly Thany General Manager, Director, FM 92.70 MHz, Battambang Mr. Chhorn Vannak Master of Ceremony, , FM 92.70 MHz, Battambang August 1 Mr. Seng Ran Director, Provincial Health Department, Pailin Mr. Doun Chen Provincial Malaria Supervisor, Pailin Mr. Tith Pharmacist, Provincial Health Department, Pailin Wife of a malaria patient Pailin Provincial Hospital Mr. Uth Sophal Field Officer, Malaria Consortium, Pailin Mr. Hak Pan Chief of O’Chra Health Center, Pailin Ms. Kieng Linkear Midwife, O’Chra Health Center, Pailin Mr. In La Public Health Staff, Krachap Health Center, Pailin VMW Prey Mongkul, Pailin August 2 Private health provider Spong, Pailin Private health provider Spong, Pailin Private health provider Phsar, Pailin Farm Owner/Net Lender Phsar,, Pailin VMW Phsar, Pailin Private health provider Phsar, Pailin Mr. Heang Chantha Pailin Prov. Coordinator, CAP-Malaria August 3 Mr. Kaing Kim Eng Chief of Malai Health Center, Bantauy Meanchay Mother of a patient Malai, Bantauy Meanchay Ms. An Bunthoeun Nurse, Malai Health Center, Bantauy Meanchay Dr. Leang Lenin Poipet OD Director, Bantauy Meanchay Mr. Sen Sam An Bantauy Meanchay Provincial Coordinator, URC Cambodia August 4 Mr. Horng Nom Trapaing Prasat Health Center, Oddor Meanchay VMW Trapaing Prasat, Oder Meanchay VMW Trapaing Prasat, Oder Meanchay Mr. Chea Saroun Chief, Anlong Veng Health Center, Oder Meanchay Mr. Neang Nuch Secondary Nurse, Anlong Veng Health Center, Oder Meanchay VMW Cheung Phnom Village, Oder Meanchay VMW Anlong Veng, Oder Meanchay August 6 Mr. Pich Morn Malaria Officer Mr. Horng Nom Health Center Chief Mr. Pich Sokha Anlong Veng Health Center, Oder Meanchay Yun Sarom Malaria Supervisor of Anlong Veng health center August 7 Dr. Im Saron Country Director, PfD Cambodia Mr. Jack Marrkand Executive Director, PfD USA August 10 Dr. John MacArthur US-CDC Atlanta (Former USAID/RDM-A’s Infectious Disease Team Leader) Page 63 2. Documents Reviewed The list below includes the most informative documents and those containing data/information quoted and/or referred to in this report. Anderson, Tim J.C. et al. “High heritability of malaria parasite clearance rate indicates a genetic basis for artemisinin resistance in Western Cambodia.” Journal of Infectious Diseases (2010): 1326-1330. Bunkea, Tol. “Malaria Situation in Cambodia.” Presentation at the National Conference on Malaria. Siem Reap, Cambodia, 28-29 April 2011. Cox, Johathan. Evaluation of community- and health facility-based systems for the surveillance of cases of day-3 positive Plasmodium falciparum in Cambodia. London: Malaria Consortium, October 2011. Development Business Associates. Assessment of the Malaria Control in Cambodia Project. Singapore: April 2011. Fairhurst, Rick M. et al. “Artemisinin-Resistant Malaria: Research Challenges, Opportunities and Public Health Implications.” The American Journal of Tropical Medicine and Hygiene (2012): 231-241. Guerra, Carlos A. et al. “The Limits and Intensity of Plasmodium falciparum Transmission: Implications for Malaria Control and Elimination Worldwide.” Plos (2008): 0300-0311. Guyant, Philippe. Malaria, Mobile and Migrants Population in the Context of Drug Resistance in Cambodia. Phnom Penh: MCC Project, undated. Hewitt, Sean. Review of USAID-funded Malaria Control in Cambodia Project. Phnom Penh: MCC Project, July 2009. Hewitt, Sean. Indoor Residual Spraying for Vector Control in Malaria Transmission Foci in the Artemisinin Resistance Containment Zone of Western Cambodia. Phnom Penh: Assessment Report for the MCC Project., August 2009. Kheang, Soy Ty et al. Cambodian Village Malaria Workers Prove Critical for Increasing Access to Early Malaria Diagnosis and Prompted Treatment in Remote Communities. Phnom Penh: MCC Project, undated. Kheang Soy Ty. Malaria Control in Cambodia: Annual Progress Report, October 1, 2007 – September 30, 2008. Phnom Penh: MCC Project, November 2008. Kheang Soy Ty. Malaria Control in Cambodia: Annual Progress Report, October 1, 2008 – September 30, 2009. Phnom Penh: MCC Project, January 2010. Kheang Soy Ty. Malaria Control in Cambodia: Annual Progress Report, October 1, 2009 – September 30, 2010. Phnom Penh: MCC Project, October 2010. Kheang Soy Ty. Malaria Control in Cambodia: Annual Progress Report, October 1, 2010 – September 30, 2011. Phnom Penh: MCC Project, October 2011. Kheang, Soy Ty. Overall Malaria Situation and Emergence of Anti-Malarial Resistance in Cambodia – power point presentation. Phnom Penh: MCC Project, undated. Page 64 Kheang, Soy Ty, Oikkonen, Aida and Duong, Socheat. Cambodian Village Malaria Workers Prove Critical for Increasing Access to Early Malaria Diagnosis and Prompted Treatment in Remote Communities. Phnom Penh: MCC Project, undated. Kheang, Soy Ty and Oikkonen, Aida. “Increasing Access to Early Malaria Diagnosis and Prompted Treatment in Remote Cambodian Villages.” American Journal of Public Health (2011): e6-e8. Kheang, Soy Ty and Katherine Herz. Public Private Mix: Malaria Control in Cambodia. Phnom Penh: MCC Project, August 2009. Kharn, Lina. Report: Radio Call-In Show Evaluation. Phnom Penh: MCC Project, August 2009. Lowe, David and Caroline Francis. Protecting People on the Move: Applying lessons learned in Asia to improve HIV/AIDS interventions for mobile people. Bangkok: Family Health International, 2006. Malaria Consortium. “Summary from the Technical Workshop on Cross-Border IEC/Behaviour Change Communication Strategies to Contain Artemisinin Resistant Malaria.” Review of meeting in Siem Reap, Cambodia, August 2009. Malaria Control in Cambodia Project. Drug Outlet End Line Survey in Pailin in November 2011. Phnom Penh: undated. Malaria Control in Cambodia Project. Hospitalized and Community-based (D-3) Surveillance and Landowner￾Supervised LLIN Distribution for MMP along the Cambodian-Thai Border - power point presentation. Phnom Penh: undated. Malaria Control in Cambodia Project, Results of Mapping of Drug Resistance among In-patient (severe) and Community –based Malaria Cases – power point presentation. Phnom Penh: undated. Malaria Control in Cambodia Project. Statement of Work – Project Year 4. Phnom Penh: undated. Ministry of Health. Strategic Master Plan for National Malaria Control Program: 2006-2010. Phnom Penh: Kingdom of Cambodia, June 2005. National Centre for Parasitology, Entomology and Malaria Control Program. Annual Progress Report 2009. Phnom Penh: Kingdom of Cambodia, 2009. Phyo, Aung Pyae, et al. “Emergence of artemisinin-resistant malaria on the western border of Thailand: a longitudinal study.” The Lancet. (2012): 1960-1966. Power, Matthew. “Slipping Through the Net,” Harper’s Magazine (April 2012): 39-46. QED Group. External Evaluation of the President’s Malaria Initiative. Washington D.C.: USAID, 2011. Song, J. et al. “Rapid and effective malaria control in Cambodia through mass administration of artemisinin-piperaquine.” Malaria Journal (2010): 9 57-67. University Research Co. LLC. Assessment on LLIN Loaning Scheme for Mobile and Migrant Population and Family Members along the Cambodia-Thai Border. Phnom Penh: MCC Project, February 2011. Page 65 University Research Co. LLC. Beyond Monitoring Drug Resistance in Cambodia: An Innovative Strategy to Actively Control the Spread of Existing Drug Resistant/Tolerant Strains-Preliminary Results. Phnom Penh: MCC Project, April 2010. University Research Co. LLC. Malaria Control in Cambodia: Building a Community Response. Phnom Penh: MCC Project. August, 2010. University Research Co. LLC. Malaria Control in Cambodia - Final Report. Phnom Penh: MCC Project, December 2011. University Research Co. LLC. Malaria Prevention and Control Activities in Cambodia Project M&E Plan. Phnom Penh: MCC Project, March 2008. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: Advocacy to Improve the Supply Chain for Malaria. Phnom Penh: MCC Project, December 2011. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: Community Mobilization for Malaria Prevention, Diagnosis and Treatment. Phnom Penh: MCC Project, April 2011. University Research Co. LLC, Technical Brief. Malaria Control in Cambodia: Improving Access to Low-cost, Accessible Diagnostic Services. Phnom Penh: MCC Project, April 2011. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: Lending scheme for Long-lasting Insecticide Nets to Mobile and Migrant Workers. Phnom Penh: MCC Project, April 2011. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: Private Sector Engagement. Phnom Penh: MCC Project, April 2011. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: Surveillance and Mapping of Malaria Drug Resistance at the Facility and Community Levels. Phnom Penh: MCC Project, April 2011. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: Strengthening Laboratories. Phnom Penh: MCC Project, June 2011. University Research Co. LCC, Technical Brief. Malaria Control in Cambodia: The Role of Supportive Technical Supervision. Phnom Penh: MCC Project, June, 2011. University Research Co. LLC, Malaria Control in Cambodia (MCC Project), 2007-2011- power point presentation for briefing of EOP external evaluators. Phnom Penh: MCC Project, 25 July 2012 University Research Co. LLC,/Partners for Development. Mobile and Migrant Workers in Malaria Endemic Areas along the Cambodia-Thai Border. Phnom Penh: MCC Project, undated. University Research Co. LLC. Performance Management Plan – Malaria Prevention and Control Activities in Cambodia. Phnom Penh: MCC Project, March 2008. USAID. Contract No. GHS-I-00-07-00010-00. University Research Co. LLC and USAID. Phnom Penh: MCC Project, September 2007. Page 66 USAID. Evaluation: Learning from Experience: USAID Evaluation Policy. Washington D.C.: Evaluation Policy Task Team, January 2011. USAID. Evaluation Policy. Answers to Frequently Asked Questions (FAQs, Issue 1). Washington D.C.: Bureau for Policy, Planning and Learning, 25 March 2011. Wongsrocjhanala, Chansuda, and Steven R. Mestrilnick. “Declining Artesunate-Mefloquine Efficacy against Falciparum Malaria on the Cambodia-Thailand Border.” Emerging Infectious Diseases (2008): 14(5) 716-719. World Health Organization. Gender, Health and Malaria. Geneva: Department of Gender, Women and Health, June 2007. World Health Organizaion. Guidelines for the Treatment of Malaria – Second Edition. Geneva: Global Malaria Programme, 2010. World Health Organization. Progress on the Containment of Artemisinin Tolerant Malaria Parasites in South￾East Asia (ARCE) Initiative. Version 1. Geneva: Global Malaria Programme, April 2010. World Health Organization. Universal Access to Malaria Diagnostic Testing: An Operational Manual. Geneva: Global Malaria Programme, 2011. World Health Organization. Malaria in the Greater Mekong Subregion: Regional and Country Profiles. Geneva: Global Malaria Programme, 2010. World Health Organization. Containment of Malaria Multi-Drug Resistance on the Cambodia-Thailand Border: Report of an Informal Consultation. Geneva: Global Malaria Programme, 29-30 January 2007 World Health Organization. Progress on the Containment of Artemisinin Tolerant Malaria Parasites in South￾East Asia (ARCE) Initiative. Geneva: Global Malaria Programme, 2010. World Health Organization. Strategic Plan to Strengthen Malaria Control and Elimination in the Greater Mekong Subregion: 2010-2014. A Mekong Malaria Programme Partnership Initiative Working Document. Geneva: Mekong Malaria Programme. December 2009. Page 67 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Neither of the consultants who prepared this report have any conflicts of interest in relation to USAID or the management of the Malaria Control in Cambodia Project. Page 68 U.S. Agency for International Development/ Regional Development Mission-Asia Athenee Towers, 25th Floor 63 Wireless Road, Pathumwan, Bangkok 10330 Thailand