MID-TERM EVALUATION OF USAID ENHANCING QUALITY OF HEALTHCARE ACTIVITY (EQHA) AND PROMOTING HEALTHY BEHAVIOR (PHB) ACTIVITY – REPORT July 2021 MICHAEL GEBREMEDHIN FOR USAID/CAMBODIA MID-TERM EVALUATION OF USAID ENHANCING QUALITY OF HEALTHCARE ACTIVITY (EQHA) AND PROMOTING HEALTHY BEHAVIOR (PHB) ACTIVITY Mid-Term Evaluation Report July 2021 Contract Number: AID-486-I-14-0001 Task Order: 72048219F0002 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. Authors/Evaluation Team Dr. Beulah Jayakumar, Team Leader Dr. Sovannarith Em, Health Systems Specialist Dr. Amry Ok, Clinical Health Specialist Ms. Sakony Pen, Social & Behavior Change Specialist Ms. Bopha Kong, Research Coordinator / Gender & Social Inclusion Specialist i | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ABSTRACT In 2018, the United States Agency for International Development (USAID) funded Family Health International and Population Services International to implement the USAID Enhancing Quality of Healthcare Activity (EQHA) and the Promoting Healthy Behaviors (PHB) Activity, respectively. EQHA aims to improve the quality and safety of health services and strengthen systems, and PHB, to strengthen social behavior change (SBC) in Cambodia. USAID/Cambodia contracted Social Impact, Inc. to conduct a mid-term performance evaluation of EQHA and PHB in order to assess performance, identify constraints, and recommend actions for improvement. The evaluation team used mixed methods including document review, key informant interviews, and focus group discussions to identify findings that addressed USAID/Cambodia’s evaluation questions. The evaluation found that EQHA’s support for facility-based quality improvement, hospital accreditation, preservice training, private health sector engagement and regulation, and Health Professional Councils are of high quality and are collaborative but require continued support through EQHA and beyond to be institutionalized in the national system. Interventions at the local level and confined to a thematic area will need to be linked to broader impacts on the system. The evaluation also found that PHB has successfully begun to institutionalize its interventions, but further actions are needed to support sub-national entities to raise their profile and funding, engage the National Center for Health Promotion (NCHP) more actively in co-deliver work, and monitor SBC work. PHB needs to consider more substantive roles for Village Health Support Groups (VHSGs). The report makes 21 recommendations directed to EQHA, PHB, and USAID/Cambodia for improvements in implementation. ii | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV CONTENTS Abstract i Acronyms iv Executive Summary vii Evaluation Background and Purpose 1 Activity Background 1 Evaluation Questions 5 Evaluation Methodology and Limitations 6 Data Collection Methods 6 Sampling 8 Data Collection 8 Data Analysis 9 Ethical Considerations 9 Evaluation Limitations 10 Dissemination and Utilization 10 Findings and Conclusions: EQHA 11 Findings and Conclusions: PHB 31 Findings and Conclusions: EQHA and PHB 43 Recommendations: EQHA 44 Recommendations: PHB 46 Annex 1: Additional Assessments of EQHA 47 Annex 2: Evaluation Design and Methods 49 Annex 3: Gantt Chart 55 Annex 4: Data Collection and Analysis Tools 59 Annex 5: Document Review, KII Participants, and Sampling 110 Annex 6: Conflict of Interest Disclosures 116 Annex 7: Team Member CVs 125 Annex 8: Evaluation Statement of Work 144 iii | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV TABLES Table 1: Criteria for Purposive Sampling 8 Table 2: Selected Provinces and ODs 8 Table 3: Remote Data Collection Methods 9 Table 4: EQHA Contributions to the Health System 11 iv | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ACRONYMS AHEAD Action for Health and Development ASSIST Applying Science to Strengthen and Improve Systems BCC Behavior Change Communication BFC Better Factories Cambodia BTB Basic Training Bureau CBE Competency-Based Education CCN Cambodia Council for Nurses CCWC Commune Council for Women and Children CDCS Country Development Cooperation Strategy CENAT National Center for Tuberculosis and Leprosy Control CHAS Cambodia Hospital Accreditation Standards CMA Cambodia Midwives Association CMC Cambodia Midwifery Council CNM National Center for Malaria CPD Continuous Professional Development CPHAS Cambodia Primary Healthcare Accreditation Standards D&D Decentralization and Deconcentration DCC Dental Council of Cambodia DEC Development Experience Clearinghouse DHS Department of Hospital Services DoP Department of Planning DPHI Department of Planning and Health Information EpiC Meeting Targets and Maintaining Epidemic Control Project ET Evaluation Team EQ Evaluation Question EQHA USAID Enhancing Quality of Healthcare Activity FGD Focus Group Discussion FHI 360 Family Health International FP Family Planning GBV Gender-Based Violence GMAG Gender Mainstreaming Action Group H-EQIP Health Equity and Quality Improvement Project HC Health Center HCMC Health Center Management Committee HEF Health Equity Funds HPC Health Professional Council HPU Health Promotion Unit HRDD Human Resource Development Department HRMIS Human Resources Management Information System HSAC Hospital Services Association of Cambodia HSS Health Systems Strengthening I-TECH International Training and Education Center for Health ID Identification IHI Institute for Healthcare Improvement ILO International Labor Organization IP Implementing Partner IRB Institutional Review Board ITOCA Integrated Technical Organizational Capacity Assessment JS Joint-Secretariat v | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV KAP Knowledge, attitudes and practices KII Key Informant Interview LOE Level of effort MCC Medical Council of Cambodia MCH Maternal and Child Health MEL Monitoring, Evaluation, and Learning MoH Ministry of Health MoLVT Ministry of Labor and Vocational Training MRD Ministry of Rural Development MoWA Ministry of Women’s Affairs NCD Non-Communicable Disease NCDD National Committee for Sub-National Democratic Development NCHADS National Center for HIV/AIDS, Dermatology, and STDs NCHP National Center for Health Promotion NGO Non-Governmental Organization NMCHC National Maternal and Child Health Center NNP National Nutrition Program NQEM National Quality Enhancement Monitoring NSSF National Social Security Fund OD Operational District OECD Organization for Economic Cooperation and Development ONA Organizational Network Analysis PC Partners in Compassion PCA Payment Certification Agency PCC Pharmacy Council of Cambodia PDSA Plan-Do-Study-Act PHB Promoting Healthy Behaviors PHD Provincial Health Department PHPU Provincial Health Promotion Unit PPP Public Private Partnership PSF Patient Satisfaction Feedback PSI Population Services International PSL Partnering to Save Lives QAO Quality Assurance Office(r) QI Quality Improvement QIC Quality Improvement Collaborative RGC Royal Government of Cambodia RH Referral Hospital RMNCH Reproductive, Maternal, Newborn, and Child Health RMS Registration Management System RTC Regional Training Center SBC Social and Behavior Change SBCC Social and Behavior Change Communication SI Social Impact, Inc. SOP Standard Operating Procedure TA Technical Assistance TB Tuberculosis TL Team Leader TWG Technical Working Group UNFPA United Nations Population Fund vi | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV USAID United States Agency for International Development VHSG Village Health Support Group WASH Water, Sanitation, and Hygiene WHO World Health Organization WHWG Workplace Health Working Group WRA Women of Reproductive Age vii | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY ACTIVITIES’ OVERVIEW The USAID Enhancing Quality of Healthcare Activity (EQHA), implemented by Family Health International (FHI 360) aims to engage and empower national and provincial leadership as well as public and private healthcare managers and providers to collaboratively improve the quality and safety of health services, strengthen systems, and increase service utilization. Promoting Healthy Behaviors (PHB), implemented by Population Services International (PSI), aims to improve health behaviors among Cambodians and support USAID/Cambodia’s goal to ensure that Cambodians seek and receive quality health care with decreased financial hardship through more sustainable systems. It aims to strengthen public health systems for oversight and coordination of social and behavior change (SBC) and improve the ability of individuals to adopt healthy behaviors. EVALUATION PURPOSE This mid-term performance evaluation assessed the performance of EQHA and PHB to date, identified key bottlenecks and constraints, and made actionable recommendations for improvements to meet the activities’ intended objectives. The findings noted successes, areas for improvement, and potential course corrections. This evaluation looked across both activities, and answered the following questions: 1. How could EQHA and PHB effectively leverage and build synergy with each other? 2. How can EQHA and PHB leverage their respective experiences to institutionalize the activities’ interventions with the local system in the future? 3. What are key challenges in achieving the expected results of EQHA and PHB, and how can they be overcome? 4. How have the EQHA’s and PHB’s structure, management, and internal procedures and sub￾awardees management affected implementation and outcomes, positively and/or negatively? 5. How effective have the EQHA approaches been in engaging the private sector in program implementation? 6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? METHODS This evaluation used a mixed-methods approach and multiple modalities to answer the evaluation questions. The evaluation team assessed documents on EQHA and PHB design, implementation, and evidence, as well as secondary quantitative data, national health policy and strategy, and national and global resources. The evaluation team used multistage purposive sampling to select key informants from the categories of respondents at the national level, from provinces and districts, and for focus group discussion (FGD) participants. In answering the evaluation questions, the team developed sub-themes that structured both the collection and analysis of data. They obtained primary qualitative data through 83 key informant interviews (KIIs) and 16 FGDs held in March-April 2021. These were conducted at the national and sub￾national levels in Battambang, Kampong Chhnang, and Tbong Khmum Provinces. viii | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS: EQHA Support to national programs: EQHA’s efforts to include ten clinical vignettes into the tablet-based National Quality Enhancement Monitoring (NQEM) tool are aligned with the overall direction of EQHA and will continue to be utilized in the health system. EQHA’s support to the Quality Assurance Office (QAO) the Department of Hospital Services (DHS) of the Ministry of Health (MoH) to digitize the NQEM database will require continued support beyond EQHA’s term. EQHA’s support for addressing gender and gender-based violence (GBV) in the healthcare system is also in line with EQHA’s overall system-strengthening efforts. The Activity’s support to the provincial GBV technical working group (TWG) and to the national tuberculosis (TB) and HIV programs in developing and updating guidelines have contributed to strengthening individual thematic areas within the overall health system but not addressed constraints that cut across the system, given how the Activity’s Program Description was designed. Facility-based quality improvement (QI): EQHA’s quality improvement (QI) intervention in target facilities has created a QI “mindset” in the facilities, provincial health departments (PHDs), and operational districts (ODs) covered. The model was implemented through a facility-led process, and it met an expressed need of the facilities to improve their NQEM scores. These enablers ensured its success, and they are likely to ensure its continued use in facilities where it was implemented. QI collaboratives (QICs) provide the supporting structure to facility-level QI activities and help in replication. EQHA recognizes the need to utilize its work in facilities to shape national policy, but its QI intervention is not widely known among national stakeholders. GIZ implements a QI intervention in facilities across three provinces to support the NQEM process. EQHA is best positioned to initiate and lead the effort to blend appropriate features from all models currently being implemented and lead national-level advocacy for its uptake. Hospital accreditation standards: EQHA’s ongoing technical assistance (TA) and financial support for bringing together a diverse team to develop the first edition of the Cambodia Hospital Accreditation Standards (CHAS) helped the MoH reach a crucial milestone. EQHA also fast-tracked the drafting of a framework and minimum standards for licensing of private health facilities. Continued support will be needed through the term of EQHA and beyond to advocate for issuing of legal instruments required to establish the Health Service Accreditation Committee (HSAC), supporting surveyor training, and providing technical support to incorporate CHAS in the next version of NQEM. Support to health professional councils (HPCs): EQHA continues earlier support and has refined the guidelines for continuous professional development (CPD). Concerns regarding low capacity for implementation, raised by the HPCs since the earlier grant cycle, remain unaddressed. HPCs require significant financial and technical support to mobilize their cadres toward registration and CPD, particularly with those in the private sector. This support is needed in the form of staff and the finances needed to support these staff and HPC dissemination activities. EQHA’s support for broad-based collaboration has advanced furthest with Pharmacy Council of Cambodia (PCC). Workplace health: EQHA employed a systems approach to strengthen workplace health in the garment industry, in collaboration with the MoH and the Ministry of Labor and Vocational Development (MoLVT). HPCs will require continued, significant financial and technical support to conduct this assessment, and to continue engaging infirmary-level healthcare providers. Preservice education: EQHA’s high-quality work with the nursing curriculum helped transform the understanding of stakeholders on competency-based education (CBE). Further progress will depend on collaboration, alongside technical support to other efforts to revise curricula for nursing and midwifery. ix | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Engaging private facilities and providers: EQHA has used a systems approach to engaging this rapidly proliferating sector and engaged them in national and sub-national public forums for its activities. EQHA has also piloted strengthening of family planning (FP) service provision in 18 facilities, which is expected to lead to further sub-national engagement, and is linked to EQHA’s system-wide engagement on FP. EQHA has begun supporting PHDs and ODs to conduct QICs in private facilities, and there is need for supporting provincial networks of private facilities and providers, extending to ODs and communes.. Internal structure and management: EQHA is a well-organized consortium with consultative leadership, collaborative work processes and sub-awardee management, and a priority of staff development. EQHA focuses on evidence-based design, technical quality of interventions, and sustainability in its design and operations. Challenges: EQHA’s challenges include restrictions related to COVID-19 and diversion of level of effort (LOE) resources to its response. QI is one of the 11 sets of standards in CHAS, and EQHA reports that it aims to use the platform of its QI work to increase facilities’ appetite for accreditation, and to advocate with MoH that complying with QI will enable facilities to get accredited. However, the sheer effort required to meet accreditation standards could take attention away from QI, and there is also the challenge that facilities meet the QI standard in CHAS and yet do not fully implement the QI activities that EQHA has promoted. Performance monitoring indicators of EQHA could be improved to better reflect the complexity of the work (see illustrative indicators in Annex 1). Funding that is tied to specific thematic areas constrains the ability of EQHA to carry out system-wide activities. Conclusions: EQHA’s support to QAO and its investments in CHAS and minimum standards for private facilities are likely to be institutionalized with continued support from the Activity and later, from stakeholders in the partner community. Interventions at the local level and confined to a single vertical program such as TB and HIV will be difficult for EQHA to scale up considering the overall Activity effort, and they contribute to the strengthening of these thematic areas rather than the overall health system. EQHA’s QI work can be leveraged for institutionalization if the Activity takes the lead to convene stakeholders, develop a QI model, and lead advocacy for its adoption. Interventions with HPCs need increased support in terms of full-time staff and advocacy for funding to implement their strategic plans, to reach a stage where they would be self-sustaining. Interventions for workplace health are a mix of those that are likely to be institutionalized and continue in the system, and those that are at the level of infirmaries that would require significant effort to scale up. EQHA’s work in preservice education is poised to become embedded in the local system but needs continued support to incorporate CBE in other health professional courses, as EQHA is part of the MoH-led effort that is revising other courses. EQHA’s engagement of private facilities has given them a voice at national and sub-national levels, and its work in FP at service provision levels needs to be used as a lever to produce impact broader than FP, in engaging sub-national networks of private facilities and providers. EQHA’s organizational structure and management processes have contributed to its high performance. Its focus on sustainability is likely to enable a thoughtful transition process. FINDINGS AND CONCLUSIONS: PHB Institutionalizing PHB’s interventions in the local system: The National Health Promotion Center (NCHP) and other stakeholders have learned aspects of the social and behavior change communication (SBCC) design process they were engaged in and find it to be relevant. PHB has revived the SBC Forum as part of NCHP’s strategic plan (2018-2022), which can serve as a platform for the SBCC community and NCHP’s engagement, but the Forum guidelines require a costed work plan and transition plan. PHB has begun supporting provincial health promotion units (PHPUs) to lead SBCC work at these levels and will need significantly increased support. NCHP is keen to build its own x | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV capacity to monitor and guide PHPUs. PHB was designed to work through the local sub-awardee organizations and their teams rather than through village health support groups (VHSGs), but the latter are widely recognized as a critical local system for SBC and can be utilized by later programs. Women, men and youth in target communities reported wider practice of key behaviors. PHB’s co-deliver work among ethnic minority populations in its target provinces will help understand the determinants of behavior in these communities. Strengthening public sector systems for SBCC: PHB’s strategy to build the capacity of NCHP for high-quality SBCC delivery does not factor in the NCHP’s current capacity, experience, and contextual understanding. PHB reports state that the IP team discussed the five-step design process with NCHP and identified its complementary with NCHPs’ eight steps for behavior change; the evaluation team saw no evidence that PHB obtained substantive NCHP’s input into the process. During the process, NCHP were engaged more as observers than active participants, in keeping with the Learning Phase of PHB’s strategy to build NCHP capacity. NCHP is a small team with a large mandate and limited funding, and so on-budget initiatives are prioritized. NCHP had not demonstrated that it actively prioritized engagement in PHB activities. Encouraging a more active, decisive role for NCHP in planning and implementing the five-step process could have elicited more interest and more consistent engagement from the NCHP team. Partners have creatively engaged NCHP, including embedding SBCC experts to provide day-to￾day support for specific needs such as procuring specialized services. The Integrated Technical Organizational Capacity Assessment (ITOCA) resulted in an action plan for strengthening NCHP. PHB has addressed actions that were within its capacity, many of which NCHP had raised prior to ITOCA. NCHP is not able to address actions related to staffing and funding. NCHP highly values the infrastructure investments made by PHB and is keen to utilize the equipment provided by PHB. The SBCC Practitioner Guidelines offer expertise not available in Cambodia, but they must be simplified to reach a wider audience and serve as a checklist for NCHP to assess internal and partner capacity. Each national program has its own SBCC team and has been able to procure external technical assistance to design and implement SBCC initiatives that engage community-based structures and are responsive to local epidemiology. NCHP saw their role as harmonizing support from multiple sources. Internal structure and management: PHB’s leadership is highly consultative, and work processes and sub-awardee management are collaborative. PHB has been intentional and successful in building local sub-awardee capacity. Coordination with multiple stakeholders has been challenging for PHB. Challenges: PHB has seen considerable delays in formative activities, leading to reduced time available for co-deliver work. The intensity of co-deliver work is also lower than what is typically required for behavior change outcomes. There was misalignment in the perceptions of PHB vis-à-vis NCHP such as the SBCC Practitioner Guidelines and the engagement of NCHP in the five-step process and national programs in terms of the frequency of program updates. PHB is also unable to track or influence funding for SBCC at the national level, which is one of its stated results. Conclusions: PHB has successfully leveraged its strengths to begin to institutionalize its interventions in local systems, but further actions are required. Areas for this include support for PHPUs and OD health promotion units (HPUs) to advocate for more funding and to raise their profile as SBCC agents, engaging NCHP more actively in co-deliver work, and in supporting their use of production equipment and monitoring of PHPUs. PHB’s design decision to not engage VHSGs warrants revisiting. PHB’s internal structure and management processes have positively affected its performance, despite significant challenges. xi | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS: EQHA & PHB Avenues for collaboration are already being explored and utilized by both EQHA and PHB. Each is likely to benefit by adding a learning component to the ongoing collaboration efforts, including by observing the other’s interventions in person rather than a presentation of what was learned. Additionally, sharing what did not work, and what was harder than expected, would also be helpful for the other team. RECOMMENDATIONS: EQHA AND PHB For EQHA, the evaluation team recommends that: 1. EQHA leverage its position as the thought leader and lead collaborator in the country for health systems strengthening, provide technical expertise under the leadership of the QAO of DHS, MoH to convene all stakeholders engaged in facility-based QI work in the country and develop a harmonized QI model that combines features from all the models that are appropriate for the country’s health systems context. Such a harmonized model stands a better chance at being accepted than any one model, for nationwide scale up through joint advocacy, and later for integration into the other workstreams of EQHA. As this effort is likely to take time to shape up, this activity must be prioritized in the remaining Activity timeframe to be effective. 2. EQHA provide continued technical leadership to lead partners to approach MoH to attain clarity and consensus on the processes and obtain the approvals related to the endorsement and rollout of the minimum standards for licensing of private facilities and also lead the advocacy with the MoH for the required approvals. 3. EQHA intensify and customize its support to HPCs – Cambodia Council of Nurses (CCN), Cambodia Midwifery Council (CMC) and PCC – to identify and address specific challenges to becoming high-performing and self-supporting entities, through a collaborative process with other partners and MoH. These challenges include, but are not limited to advocacy for more funds to implement the strategic plans and continued technical support for the registration process. EQHA to facilitate the process of collaboratively identifying and determining replicable success factors in PCC that have aided its successful engagement of its members and apply the lessons to CCN and CMC. The aim is to strengthen these HPCs’ nationwide reach and effectiveness, which is also critical for EQHA’s private sector and workplace interventions. 4. For those activities across all its workstreams that do not address system-wide constraints and that are not feasible to expand, EQHA to identify ways to connect them to broader impacts on the system. Notable among these interventions are: supporting the provincial TWG on GBV and HIV, FP interventions in private facilities, and infirmary-level interventions in factories including outreach camps and referral linkages for individual infirmaries. 5. EQHA protect and advance the gains made in the nursing curriculum as well as the change brought about in the collective mindset toward CBE by reaching out to partner programs and offering expertise in CBE to collaboratively transform the curricula of other health professions. 6. EQHA revisit its performance indicators and include those that that more directly capture the breadth and complexity of its work, including composite weighted indicators, illustrative examples for which are included in Annex 1. 7. EQHA coordinate with PHB to learn from observing each other’s key activities especially the QIC work of EQHA and the community-level demonstration activities of PHB. 8. EQHA continue to support the adaptation of relevant accreditation standards for institutionalizing in the next version of NQEM tool, while working with relevant partners to advocate for the adoption of the new law, establishment of the accreditation organization and rollout on a voluntary basis after approval. xii | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV For PHB, the evaluation team recommends that: 1. In strategies and work plans, PHB add a more substantive and active role for NCHP in the co￾design and co-deliver phase of its interventions. The effort put into this change would be calibrated against the time available for PHB to reach its targets and COVID-19 restrictions. 2. PHB maximize time remaining by expediting the five-step process for the remaining behaviors and moving to the co-deliver phase. 3. PHB collaborate with NCHP to assist the NCHP in utilizing the equipment that PHB provided them, in order to develop SBCC products for a few thematic areas that are mutually agreed upon. 4. PHB consult with USAID, NCHP, and sub-awardees to redefine the role of the SBCC expert in its team, to be embedded in NCHP as a coach, for incrementally higher involvement of NCHP in the remaining activities of the five-step process for select behaviors. 5. PHB work with NCHP, PHDs, and ODs to develop a plan to train and coach its staff in using Kobo Toolbox™ and set up a system for NCHP to track and support SBCC-related indicators from sub-national levels. 6. PHB partner with the review team (including partner organizations and NCHP) for the SBCC Practitioner Guidelines to revise/reduce the range of theories, incorporate case studies and descriptions of deploying the theories, and also consider recasting the entire document to target sub-national audiences and the requirements of NCHP, in order to serve as a tool to assess internal NCHP and partner capacity. 7. PHB work with USAID and NCHP to include in ITOCA the tracking of the above interventions in order to strengthen the capacity of NCHP to design and deliver SBCC. 8. PHB engage NCHP and partners to develop a costed plan for the SBC Forum and the terms for a rotating secretariat and incorporate them in the Forum Guidelines, and conduct capacity building on parts of the five-step SBCC design process that have gained traction among partners. 9. PHB ramp up its support to PHPUs to develop their work plans, advocate for funding at the province level, collaborate with vertical program staff, and monitor co-deliver activities at demo sites. PHB should consider placing staff in some of its target provinces to oversee this work. 10. PHB provide opportunities for sharing and cross-learning amongst local sub-awardees. 11. PHB coordinate with EQHA to observe each other’s key activities and glean lessons. 12. PHB continue to actively engage ethnic minority populations in their target provinces in order to understand specific factors that influence their adoption of the key behaviors that PHB promotes. 13. Review PHB’s approach to implementing co-deliver interventions at demo sites, consider a substantive role for VHSGs as a cadre working under the direction and support of the local staff of local sub-awardees, and work with commune councils to identify ways to compensate them and maintain their motivation and retention. 1 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EVALUATION BACKGROUND AND PURPOSE USAID Enhancing Quality of Healthcare Activity (EQHA) is a five-year (August 30, 2018, to August 29, 2023) Activity funded by the U.S. Agency for International Development (USAID) and led by Family Health International (FHI 360). EQHA aims to engage and empower national and provincial leadership as well as public and private healthcare managers and providers to collaboratively improve the quality and safety of health services, strengthen systems, and increase service utilization. EQHA contributes to the Development Objective “Improved health and education for vulnerable population” of the USAID/Cambodia Country Development Cooperation Strategy (CDCS) by directly focusing on Sub￾Purpose 3: “Improved quality of public and private health services.” Promoting Healthy Behaviors (PHB) is a five-year (May 2018 to June 2023) Activity funded by USAID and led by Population Services International (PSI). It is designed to improve health behaviors among Cambodians and support USAID/Cambodia’s goal to ensure that Cambodians seek and receive quality health care with decreased financial hardship through more sustainable systems. Both EQHA and PHB work at the national level and in five provinces – Battambang, Kampong Cham, Tbong Khmum, Kampong Chhnang, and Pailin – and in Phnom Penh. The primary purpose of this mid-term performance evaluation is to assess the performance of both EQHA and PHB to date, identify key bottlenecks and constraints, and make actionable recommendations for improvements to meet the activities’ intended objectives. The findings will be used to provide feedback to USAID/Cambodia, the implementing partners (IPs), and relevant stakeholders on the successes, areas for improvement, and any course corrections required for the remainder of the activities. This evaluation is one exercise that looks across both activities. Throughout the report, “IPs” refer to FHI 360 and PSI, and “partners” refers to bilateral and multilateral organizations, and international and national non-governmental organizations (NGOs). ACTIVITY BACKGROUND CAMBODIA HEALTH SYSTEMS CONTEXT Cambodia has made significant improvements in health outcomes over the past two decades, achieving Millennium Development Goals 4 (reduce child mortality) and 5 (improve maternal health) in 2014 and lower middle-income status in 2015. Despite these improvements, challenges remain; access to health services is constrained, with gaps in health infrastructure and the health workforce, and high out-of￾pocket expenditure for health. Non-communicable diseases (NCDs) present a rising disease burden, while that of communicable diseases is ongoing. At the policy level, health priorities focus on all people and cover the entire health sector, and include the following: ● Promoting reproductive health ● Reducing maternal, newborn, and child mortality ● Improving nutrition status among women and young children ● Reducing morbidity and mortality caused by communicable, non-communicable and chronic diseases 2 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● A resilient, responsive, and transparent healthcare system ● Strengthened universal coverage and improved quality of healthcare services throughout the country ● Improved coverage of social protection mechanisms for all people regardless of economy and social status The National Health Strategic Plan – 3 (2016-2020) outlines a strategic framework to strengthen the operation of the entire health system, public and private, and lays out an operational framework to ensure that sector strategies are consistently applied across all programs’ interventions.1 The Ministry of Health (MoH) is mandated by the Royal Government of Cambodia (RGC) to lead and manage the entire health sector, both public and private. The public health system is organized into central, provincial, and operational district (OD) levels. The MoH defines the health policy; develops strategies, plans, and guidelines; and manages and coordinates resources. The 24 Provincial Health Departments (PHDs) along with one Municipal Health Department manage the Provincial Hospitals and link MoH with ODs. The 102 ODs2 implement health objectives through a network of Referral Hospitals (RHs), Health Centers (HCs), and Health Posts. HCs deliver basic health care services as defined in MoH’s minimum package of activities, while the RHs deliver the MoH’s complementary package of activities.3 Private for-profit health services account for the largest share of the total healthcare spending. Health Equity Funds (HEFs) provide financial risk protection to 20 percent of the total population focusing on vulnerable groups. The National Social Security Fund (NSSF) covers workers in the private formal sector. The Health Equity and Quality Improvement Project (H-EQIP) (2016-2022) improves the resourcing and management of HEFs and Service Delivery Grants to PHDs, ODs, and public facilities, and monitors performance through the National Quality Enhancement Monitoring (NQEM) process.4 The next phase, H-EQIP 2, plans to continue the NQEM process with considerable changes to the content and process of the assessment. The private for-profit sector is an important provider of health services and has grown rapidly. While it is mainly concentrated in urban and economically advantaged areas, it is also becoming pervasive in rural areas, largely used for ambulatory treatment. The Cambodia Socio-Economic Survey of 2019-2020 notes that 50 to 70 percent of services are delivered by private facilities, which are three times more numerous than the public facilities. The persistently high out-of-pocket expenditure is mainly attributed to the private healthcare sector. MoH has given a mandate to the five health professional councils (HPCs) and the Department of Hospital Services (DHS) with oversight of private sector facilities and providers at the national and sub-national levels. The workplace is a primary determinant of how and where workers access health care. The 2018 annual report of Better Factories Cambodia (BFC) of the International Labor Organization (ILO) shows that 84.2 percent of factories were not compliant with the guidelines on establishment of enterprise infirmaries. Behavior change has been prioritized in national strategic plans. The National Center for Health Promotion (NCHP) is the lead MoH institution in the development and management of behavior change interventions and is mandated to provide technical assistance to provincial health promotion units 1 Ministry of Health, Department of Planning and Health Information. The Third Health Strategic Plan 2016-2020, May 2016. 2 As of December 2018. 3 Ministry of Health, Department of Planning and Health Information. The Third Health Strategic Plan 2016-2020, May 2016. 4 Health Improvement and Quality Improvement Project (H-EQIP). Fact Sheet 2019. 3 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV (PHPUs) to implement SBC interventions. However, NCHP is one of eight national centers, each belonging to a vertical program. Each has the mandate for SBC interventions in its respective areas, making the coordinating role of NCHP unclear. Funding constraints also make coordination of SBC interventions between NCHP and vertical programs challenging. At the peripheral levels, Commune Committees for Women and Children (CCWC) and HCs work with village health support groups (VHSGs) for health communication, with support from partner initiatives. The RGC undertook a 10-year process of comprehensive governance reform in 2010 to achieve poverty reduction and development, led by the National Committee for Sub-National Democratic Development (NCDD) program, under the Ministry of Interior. The Decentralization and De￾Concentration (D&D) pilot aims to increase the delegation of administrative functions and resources to sub-national levels and is expected to make the system more efficient, accountable, and responsive to local needs. However, the D&D reform has been met with significant operational and political challenges and has been slow to roll out across the country. USAID/Cambodia has played a long-standing role in building sustainable health systems to address issues of child and maternal health, tuberculosis, HIV/AIDS, and malaria, and assisted in the expansion of HEF, and technical support to implement NSSF. A development objective of the 2014-2018 CDCS was to improve the health and education status of vulnerable populations. The 2021 Journey to Self-Reliance Country Roadmap for Cambodia shows the country’s progress to self-reliance and informs USAID’s development strategies and programs for child health, education, and government effectiveness. To respond to the challenges in the provision of quality and accessibility of health services, and those related to the promotion of appropriate behaviors and demand for health services in the population, USAID/Cambodia initiated two activities, EQHA and PHB, to facilitate a coordinated response to supply and demand side challenges and improve health outcomes. ACTIVITY OBJECTIVES AND KEY INTERVENTIONS: EQHA EQHA aims to engage and empower national and provincial leadership as well as public and private healthcare managers and providers to collaboratively improve the quality and safety of health services, strengthen systems, and increase service utilization, by achieving the following objectives: Objective 1: Improve policies, guidelines, and standards for streamlined quality assurance. Objective 2: Increase efficiency and effectiveness of service delivery. Objective 3: Strengthen implementation and enforcement of the regulatory framework. Objective 4: Strengthen preservice public health training. To implement EQHA, FHI 360 works closely with the RGC and in partnership with sub-awardees: PSI, the International Training and Education Center for Health (I-TECH) of the University of Washington, mClinica, and Meridian Group International. ACTIVITY OBJECTIVES AND KEY INTERVENTIONS: PHB PHB aims to foster a vibrant and cohesive social and behavior change (SBC) community of practice to guide collaborative and evidence-based design, implementation, and assessment in order to achieve the following: 4 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Objective 1: Strengthened public sector systems for oversight and coordination of SBC at the national and provincial levels. Objective 2: Improved ability of individuals to adopt healthy behaviors. PSI implements PHB in partnership with 17 Triggers Consulting Asia (a private sector partner), Pact, Partners in Compassion (PC), Khemera, and Action for Health Development (AHEAD) as sub-awardees. CRITICAL ASSUMPTIONS The achievement of EQHA’s objectives and results rest on the following critical assumptions: 1. Political will to continually address gaps found in NQEM assessments, include vertical disease programs into the assessment, and to use evidence to update policies. 2. Willingness of the private sector to collaborate with the MoH to address identified quality issues in health care delivery. 3. Maintenance of the schedule of the approved roadmap to accreditation of facilities. 4. Willingness of public and private academic institutions to collaborate for a rationalized health workforce. The achievement of PHB’s objectives and results rest on the following critical assumptions: 1. NCHP continuing its current mandate and the public sector composition of the SBC community remaining unaffected. 2. NCHP continuing to actively engage, collaborate with, and support PHB. 3. Access to medical commodities and quality health services from private and public facilities. 4. Sustained or increased funding for the MoH from RGC. 5. Collaboration of private sector institutions in SBC. 6. Priority populations not facing barriers to care. 5 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EVALUATION QUESTIONS This evaluation addressed six evaluation questions (EQs), drafted by USAID/Cambodia in the initial Scope of Work with Social Impact, Inc. (SI) (Annex 8: Evaluation Statement of Work) and finalized by the evaluation team (ET) in collaboration with USAID/Cambodia: 1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub-partners, other USAID activities, and other key partners in order to achieve desired health outcomes. 2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? 3. What are key challenges in achieving the expected results of EQHA and PHB, and how can they be overcome? 4. How have EQHA’s and PHB’s structure, management, and internal procedures and sub￾awardees management affected implementation and outcomes, positively and/or negatively? 5. How effective have the EQHA approaches been in engaging the private sector in program implementation? 6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? The themes and dimensions defined in the design matrix under the EQs guided the overall design of the evaluation and provided structure to thematic analysis of data. 6 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EVALUATION METHODOLOGY AND LIMITATIONS DATA COLLECTION METHODS The evaluation team used a mixed methods evaluation methodology consisting of document review, secondary quantitative data, key informant interviews (KIIs), and focus group discussions (FGDs). The approach outlined in this section includes evaluating EQHA and PHB as separate activities as well as together, to respond effectively to all EQs. As such, the scope of each method described below included both activities. Within each method, some participants were included for their potential to contribute to evaluating EQHA, others for PHB, and still others to evaluate both activities together. Please see Annex 2: Evaluation Design and Methods for the evaluation design matrix. As the country went into lockdown due to COVID-19, primary data collection was conducted remotely. Details are provided in a separate section below. DOCUMENT REVIEW Several documents were reviewed to establish evidence related to the EQs. These documents related to the design of the two Activities, reports of baseline and other formative assessments, work plans and monitoring, evaluation, and learning (MEL) plans and those related to implementation including quarterly and annual reports. The ET also reviewed documents related to national policies and strategies, as well as reports of initiatives and literature related to the themes of this evaluation. A list of documents reviewed is included in Annex 5: Document Review, KII Participants. SECONDARY QUANTITATIVE DATA The ET analyzed secondary data from the MEL databases of EQHA and PHB to establish the extent of improvements in the quality of service delivery (for EQHA) and improvements in utilization of services (for EQHA and PHB). Reviewing trends of MEL indicators of EQHA and PHB helped assess gaps, to conduct in-depth inquiry into possible causes for the gaps. Data was disaggregated by geography, gender, and intervention type, where available. PRIMARY QUALITATIVE DATA COVID-19 Precautions for KIIs and FGDs: Considering the Royal Government’s guidelines in place for COVID-19, and based on the recommendation of USAID/Cambodia, the ET conducted all KIIs and FGDs remotely. Key informants came together to participate in group KIIs in some RHs and ODs, while the ET led the meeting remotely. In such instances, the ET recommended that participants wear masks and follow distanced seating and handwashing as part of the informed consent process. All groups complied with the recommendations. FGDs were not conducted in a group per se, but individuals participated from their respective homes and were added to a group call. KIIs: These gathered primary qualitative data on the main themes and sub-themes identified under each EQ. Themes that KIIs explored include the design and metrics used in EQHA and PHB, learning from past and present efforts, the scope and nature of challenges to achieving the objectives of EQHA and PHB, perspectives on effectiveness of approaches used, organizational characteristics, and project management processes. Within each of the themes, specific findings related to vulnerable populations and impacts on men and women were explored. KII participants were selected based on their engagement with EQHA and PHB and the level of detail their perspectives could provide, and included selected individuals from the categories listed in Annex 5. 7 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Please note that the specific samples selected within each category are provided further below, based on criteria for sampling. Informed consent was obtained from every participant, and the interviews were conducted using KII guides developed for each participant category. These instruments were reviewed and approved by SI’s Institutional Review Board (IRB). Each KII lasted from 30 minutes to two hours. Apart from some KIIs at the national level (which were conducted in English), others were carried out in Khmer. The team conducting each KII included a facilitator and a note-taker. The team leader (TL) participated in as many KIIs as feasible. An interpreter was included for KIIs where required to aid the TL’s concise note taking and ability to follow-up with respondents. KIIs were also audio recorded with consent from participants, to ensure that the content of the discussion was captured fully. Key informants who belonged to the same entity were interviewed as a group to obtain comprehensive information on topics, reduce redundancy, and reduce the time required to complete data collection. The ET interacted with each member of the group KII, ensuring the ability of everyone in the group to provide unbiased perspectives, and noted opposing points of view. A description of the sampling approach is presented in the sampling section. FGDs: These were conducted to: 1) gather participant’s perceptions and experiences in working with PHB to improve their health behaviors, 2) understand enablers and barriers to behavior change, and participant’s perspectives on ways in which the convergence of interventions for clinical and behavior change services can better support their journey toward changing behaviors, and 3) collect insight on health systems dynamics and influences on population health outcomes, including any impacts on vulnerable populations. The themes for discussion with the different focus-group respondents were as follows: 1. Mothers of children under two years of age: Perceptions and experiences with PHB’s intervention for water, sanitation, and hygiene (WASH), and perspectives on converging interventions for clinical and behavior change services. 2. Women of reproductive age (WRA) and their male partners (as separate groups): Perceptions and experience with PHB’s intervention for FP, and perspectives on converging interventions for clinical and behavior change services. 3. VHSGs: Perspectives on potential improvements to clinical and behavior change services received. Each FGD was held with four participants in a facilitated discussion of the range of predetermined topics covering health service delivery, health communications, and health behaviors. Each FGD was conducted in Khmer by a facilitator and a note-taker. Participants for FGDs with mothers and young persons were selected from communities where SBC activities of PHB were taking place. The evaluation team obtained this information from PHB’s sub-awardee staff in selected ODs, which enabled the ET to assess exposure to health messaging and any changes for these key populations over time. FGDs for male and female participants were held separately to ensure comfort in discussing topics such as FP. Informed consent was obtained from participants, and protocols approved by the IRB were used to guide the discussions. 8 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV SAMPLING The evaluation team applied multistage purposive sampling to select national-level entities, provinces, and ODs for primary data collection. Table 1 provides the criteria that the evaluation team used to make these selections. Table 1: Criteria for Purposive Sampling NATIONAL-LEVEL PARTICIPANTS PROVINCES ODS ● Intensity of engagement in/collaboration with EQHA and/or PHB activities ● Expertise and experience related to evaluation themes ● Time available for data collection ● Geographic representation ● Interventions of both EQHA and PHB ● Expertise/experience in evaluation themes ● Time available for data collection ● Mix of urban and rural locations ● Interventions of EQHA and PHB expertise/experience in evaluation themes ● Time available for data collection Based on these criteria, the following provinces and ODs were selected. Table 2: Selected Provinces and ODs PROVINCE OD Battambang Thmar Kol and Moung Russey ODs Kampong Chhnang Kampong Chhnang and Boribo ODs Tbong Khmum Memot and Ponhea Kreak ODs Within each chosen OD, one district referral hospital and two communes were selected using lists provided by sub-awardees and the OD health office, based on the location of EQHA and PHB implementation sites. Private facilities and providers were selected from the list of licensed providers available with the PHD team. Participants for FGDs were also recruited through these lists. Deploying the above criteria, the ET obtained a final anticipated sample of 91 KIIs and 16 FGDs. Of these, the ET was able to complete 83 KIIs and 16 FGDs. A total of 183 key informants were interviewed, of whom 64 were women, and there were 53 participants in FGDs, of whom 33 were women. The distribution of the final sample is provided in Annex 5. DATA COLLECTION Following identification of the sampled sites, a detailed data collection plan was finalized that optimally utilized ET members’ expertise and experiences. The Team Leader took part in as many KIIs as feasible, as detailed in Data Collection Methods above. Table 3 gives details of the methods of remote data collection used. 9 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Table 3: Remote Data Collection Methods CATEGORIES VIDEO CONFERENCING CONFERENCE CALL TELEGRAM GROUP CALL/MESSAGE TOTAL National: KIIs 42 1 4 47 Sub-National: KIIs 19 14 6 36 Sub-National: FGDs 0 16 0 16 Total 61 28 10 99 The ET drafted semi-structured protocols that covered the EQs, themes, and sub-themes identified in the evaluation matrix to guide the KIIs and FGDs. As data collection proceeded, the ET prioritized some of the thematic areas for deeper inquiry, based on data already collected. Verbal informed consent included information about the evaluation and the activities being evaluated, the potential use of findings, possible risks and benefits for the participant, and the measures that the ET would take to ensure confidentiality and consent for audio recording. DATA ANALYSIS Data was collected between March 23 and April 21, 2021. Notes and transcripts from KIIs and FGDs and findings from documents and literature were coded manually under themes pre-identified in the evaluation matrix as well as emerging themes. Coding was done in a standard analysis matrix. Relevant secondary quantitative data, drawn from EQHA and PHB, were reviewed to identify findings relevant to the EQs and their sub-themes. Any distinctive findings by gender and geography were actively examined for an analysis of both textual and quantitative data. The ET constructed a matrix of findings, conclusions, and recommendations to capture all data collected, by themes and sub-themes of the evaluation design, by EQ and separately for EQHA and PHB. This ensured that all data is utilized in a meaningful and comprehensive way to arrive at conclusions and recommendations that are appropriate for the context and are actionable. All findings were reviewed through a gender and social inclusion lens to identify trends affecting particular groups and their bearing on recommendations to improve access to health services for all segments of the population. KIIs and FGDs were each given a unique identification (ID) number to protect respondent identity, and the list containing these IDs will stay with the ET until the evaluation report has been approved. Soft copies of transcripts with only IDs will be handed over to SI and will be held until the final evaluation report is uploaded to USAID’s Development Experience Clearinghouse (DEC). Audio recordings reference only the ID and will be destroyed after approval of the final report. The analysis matrix with coded data references only IDs and does not contain names of participants. ETHICAL CONSIDERATIONS All interview protocols and informed consent forms associated with this performance evaluation were submitted to SI’s in-house Institutional Review Board and were approved via expedited review. The evaluation team obtained verbal informed consent from the KII and FGD participants. 10 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Before commencing data collection, the team leader conducted a meeting with the ET to ensure that the team was familiar with the ethical standards laid out in the IRB application, especially COVID-19- related protocols, and agreed on interviewing techniques, such as probing, and engaging with vulnerable populations. This also included ensuring the protection of respondents’ confidentiality and privacy during data collection through procedures for obtaining informed consent to collect data in any medium, and informing respondents about the potential uses and any potential sharing or publication of their data. All members of the ET who were engaged in data collection completed a certificate course on essential elements of ethics for engaging with human subjects prior to the start of data collection. EVALUATION LIMITATIONS Primary data was entirely qualitative and was collected from a purposive sample of respondents and geographic locations. Emphasis was on collecting and analyzing in-depth perspectives that are grounded in participants’ experiences on the topics being evaluated. As such, the design was adequate to meet the purpose of the evaluation but the conclusions from the evaluation were not necessarily representative of all implementation sites of EQHA and PHB. EQHA and PHB are wide and ambitious in their scope and engage a broad range of stakeholders through their interventions. Given the limitations of time, the ET selectively identified participants for primary data collection, prioritizing those with the longest and deepest engagement with the two activities. The list of stakeholders planned for KIIs and FGDs was not exhaustive, and so the perspectives of those not included may not be fully accounted for. The ET has sought to minimize this by conducting a thorough document review. Remote data collection with participants at commune and village levels could only be done through conference calls, and this limited the number of participants in each FGD to four, against the planned number of six to eight per FGD. Therefore, perspectives obtained from these exercises may not be as comprehensive as expected. The ET has sought to minimize this by triangulating data from other sources to arrive at findings. Despite their additional workload brought on by COVID-related duties, KII participants at the national and sub-national levels set aside time to meet with the ET. All but one planned key informant was interviewed. DISSEMINATION AND UTILIZATION The findings from this evaluation are summarized in this report and will be submitted to USAID/Cambodia. Based on review feedback from USAID and from EQHA and PHB IPs, the ET will work to respond adequately to the review comments and prepare a final report for USAID/Cambodia’s approval. At the conclusion of data collection, the ET conducted an internal presentation of preliminary findings to both USAID/Cambodia and to the IPs of EQHA and PHB to communicate results, best practices, and recommendations. When the report has been finalized and approved by USAID/Cambodia, SI will also submit the report to USAID’s DEC. 11 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS: EQHA OVERVIEW OF EQHA INTERVENTIONS We begin this section with an overview of EQHA’s activities and its key stakeholders. This will provide the needed context to the findings and conclusions under each EQ, detailed further below. Key stakeholders: EQHA collaborates with national policymakers at the MoH and its departments and national programs,5 leaders of HPCs, the public-private sector technical working group (TWG),6 the Quality Improvement (QI) TWG, and the Nursing TWG, and Battambang and Kampong Cham Regional Training Centers, and with nursing education institutions. It also works with the Ministry of Labor and Vocational Training (MoLVT), the Workplace Health Working Group (WHWG) and the Ministry of Women’s Affairs (MoWA). EQHA continues the work done by the USAID Activity Applying Science to Strengthen and Improve Systems (ASSIST) in TA for five HPCs and collaborates with other development partners, GIZ, World Health Organization (WHO), and the World Bank. EQHA also assessed USAID support to the Joint-Secretariat (JS) and consulted with the chairman of the HPC’s steering committee before the decision to stop supporting the JSs of HPCs. At the provincial and OD levels, EQHA focuses its efforts on PHD and OD health staff, provincial hospitals, HCs, RHs, Regional Training Centers (Battambang and Kampong Cham), private facilities and providers, provincial councils, and the provincial and district governor’s office. EQHA works in six target provinces, which cover 32 ODs, 32 referral hospitals, and three Regional Training Centers (RTCs). As illustrated in Table 4, EQHA contributes to all levels of the health system and is thus well positioned to institutionalize its QI approach in Cambodia. Table 4: EQHA Contributions to the Health System QI INTERVENTIONS CRITICAL STAKEHOLDERS INTERNATIONAL PARTNERS Health Care Accreditation System MOH (QAO); Quality Improvement WG USAID/EQHA, WB, GIZ Performance-Based Financing/NQEM MOH (QAO; PHD, OD) H-EQIP (World Bank, DFAT, KfW, KOICA) Licensing/relicensing of private health facilities MOH (Hospital Service Department; One-Window Service at sub-national level) USAID/EQHA, GIZ, WHO Facility-based CQI activities MOH (National programs, PHDs, ODs, RHs, HCs) USAID/EQHA, PHD/OD/RH/HC, GIZ 5 These include the DHS and its Quality Assurance Office (QAO), the Payment Certification Agency (PCA), the Department for Planning and Health Information (DPHI), the Human Resource Development Department (HRDD), the National Center for Tuberculosis and Leprosy (CENAT), the National Center for HIV AIDS and STIs (NCHADS), the National Maternal and Child Health Center (NMCHC), the Gender Mainstreaming Action Group (GMAG) 6 TWG includes: HRDD, Regional Training Centers (RTCs), University of Health Science, Institute of Health Sciences of the Royal Cambodian Armed Forces, four private universities, four training hospitals in Phnom Penh and the Angkor Hospital for Children. 12 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV QI INTERVENTIONS CRITICAL STAKEHOLDERS INTERNATIONAL PARTNERS Supportive supervision MOH (National programs, PHDs, ODs, RHs, HCs) Competency-based preservice training MOH (HRDD, UHS, RTCs) USAID/EQHA, WB Registration, licensing, relicensing individual practitioners and continuous professional development Five health professional councils USAID/EQHA, GIZ Strategic purchasing MEF (NSSF, HEF) MOH, H-EQIP, GIZ EQHA aims to reach its goal by achieving the following: Objective 1: Improved policies, guidelines, and standards for streamlined quality assurance. Under this objective, EQHA worked to integrate the national programs into the mainstream health system by: updating policies, guidelines, and standard operating procedures (SOPs) related to TB in partnership with CENAT, an SOP on infection prevention and control in collaboration with the DHS, and piloting the integration of HIV and NCD services and the registration of private health facilities in collaboration with NCHADS. EQHA also initiated innovation in using digital data management for private health facility registration management systems. It supported GMAG to develop a gender curriculum and the Policy and Strategic Plan on Gender Mainstreaming in the Health Sector (2020-2024). EQHA coordinated the development of ten new clinical vignettes, which are fully integrated into the NQEM tool and were rolled out in April 2021. EQHA also assisted the QAO to digitize, analyze, and validate NQEM and ex-post assessment data. EQHA reinstated the WHWG in partnership with MoLVT and other stakeholders and conducted a range of interventions to improve services in infirmaries. Objective 2: Increased efficiency and effectiveness of service delivery. EQHA adapted the Quality Improvement Collaborative (QIC) approach of the Institute for Healthcare Improvement (IHI)7 for RHs and HCs across 32 ODs in six provinces. This approach uses the Plan-Do-Study-Act (PDSA) model for testing, generating, and implementing change ideas for a range of health system issues, and provides clinical capacity building to public health care providers. The PDSA model emphasizes the importance of starting with small-scale improvement efforts and leveraging lessons learned to plan for spread and, later, scale-up. EQHA also piloted a digital patient satisfaction feedback (PSF) mechanism in three RHs and provided clinical capacity building to public health care providers. Furthermore, EQHA conducted site assessments of referral hospitals’ GBV services response. Objective 3: Strengthened regulatory framework, implementation, and enforcement of the regulatory framework. EQHA facilitated the development of the Concept Note and Road Map to Develop the Cambodian Healthcare Accreditation System. After the Road Map was endorsed by the QIWG of the MoH, EQHA led the development of Cambodian Hospital Accreditation Standards (CHAS) and subsequent training of accreditation coordinators and hospital workshops in 13 public hospitals and seven private hospitals to prepare them for CHAS assessments. Establishment of an accreditation organization is awaiting promulgation of the Law on Regulation of Healthcare Services. EQHA also collaborated with GIZ and other partners to draft the minimum standards for licensing of private facilities. EQHA continued the work of USAID ASSIST in developing HPCs, engaging the Cambodia 7 Institute for Healthcare Improvement, Boston, MA, USA. 13 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Council for Nursing (CCN), Cambodia Midwives Council (CMC), the Pharmacy Council of Cambodia (PCC), the Dental Council of Cambodia (DCC), and the Medical Council of Cambodia (MCC). EQHA supported improvements in the registration management system (RMS) to enhance registration and licensing/re-licensing of health professional and private healthcare practitioners. The Activity also developed a five-year strategic plan for the five HPCs, including scope of practice, standard of practice, and the launch and dissemination of the guidelines related to continuous professional development (CPD) required for the licensing and relicensing of healthcare facilities and professionals. For example, this included digital CPD courses through the Swipe Rx app for pharmacists, and modules on TB, COVID-19, and FP rolled out in April 2021. Objective 4: Strengthen preservice public health training. Working with HRDD and the preservice Nursing TWG, EQHA led the Associate Degree in Nursing curriculum revision process based on principles of competency-based education (CBE). This included developing CBE modules for the MoH to use nationwide for all health professional programs, and setting up distance learning hubs in four Regional Training Centers alongside HRDD enabling CPD. And due to COVID-19 restrictions, these hubs have been used for multiple activities including training for faculty, student teaching, TWG meetings, and other activities. High fidelity and simulation equipment were provided to two target RTCs to improve the teaching and learning of nursing and midwifery students. Support is also being provided to strengthen the Human Resource Management Information System (HRMIS) at HRDD and the Department of Planning (DoP). EQHA has engaged professionals and facilities in the private sector through its work with HPCs, CHAS, and curriculum development, and pilots QI work on FP in private facilities. PRESENTATION OF FINDINGS AND CONCLUSIONS In the sections below, we present findings and conclusions according to each EQ. As EQHA focuses on strengthening health systems, EQ2 takes center stage, as this pertains to institutionalizing Activity interventions into the local system. We follow this with relevant findings and conclusions for EQ5, which surround private sector EQHA intervention effectiveness. In response to EQ3, we present challenges that confront EQHA and potential options to mitigate them. Finally, we respond to EQ4, presenting findings and conclusions related to the management structure and processes of EQHA and their impact on performance. Recommendations specific to EQHA are included at the conclusion of this report. EQ2: HOW CAN EQHA AND PHB LEVERAGE THEIR RESPECTIVE EXPERIENCES TO INSTITUTIONALIZE THE ACTIVITIES’ INTERVENTIONS WITH THE LOCAL SYSTEM IN THE FUTURE? This section will discuss findings related to all EQHA interventions, except those related to private facilities and private providers. Given the irreducibly complex nature of EQHA’s range of interventions and the multiple interconnections, we have largely kept to the delineations that EQHA uses in its objectives and reporting: 1) support to national TB and HIV programs, DHS, and the QAO; 2) facility￾based QI and QIC and sub-national capacity building in QIC; 3) regulations related to accreditation and minimum standards; 4) strengthening of HPCs and CPD; 5) workplace health; and 6) preservice training. However, it is important to note that findings and conclusions under one section have implications for other sections, even as EQHA itself is designed to work across these thematic areas. 14 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA’s design has been guided by the framework for high-quality health systems proposed in the landmark publication of the Lancet Global Health Commission,8 with a focus on an equitable, efficient, and resilient system that delivers care focused on improving or maintaining health outcomes and improves confidence in the system. In addition, EQHA views its health systems strengthening (HSS) efforts as “levers” for change that bolster the foundations of a high-quality health system, as described in the EQHA FY 2020 Annual Report and as outlined in a WHO, World Bank, and Organization for Economic Cooperation and Development (OECD) global report on the quality of healthcare published in 2018.9 This approach to HSS demonstrates the interdependence of the levers and avoids a single￾track approach,10 and ensures that continuous QI and regulation of the quality of healthcare services are institutionalized.11 The 2018 Lancet publication on health systems and the 2018 global report on the quality of healthcare referenced above, build on the WHO framework of health system building blocks.12 WHO described HSS in its 2007 framework for action (page 5) where it is stated as “Improving [the] six health system building blocks and managing their interactions in ways that achieve more equitable and sustained improvements across health services and health outcomes.”13 A 2012 assessment of HSS evaluations point to the need for evaluation designs to take into account the characteristics of complex adaptive systems such as interactions between the building blocks.14 Such systems thinking provides a holistic perspective when evaluating complex HSS initiatives15 such as EQHA. A 2019 review of 193 studies on HSS interventions found the scope of an intervention, with effects cutting across disease programs and the WHO building blocks, to be a key criterion of HSS interventions.16 EQHA’s interventions are a mix of inputs into specific vertical programs and interventions that cut across the system and seek to address the more complex interrelationships within the system to improve the quality of healthcare for priority public health issues. In responding to EQ2, the ET reviewed the extent to which EQHA’s interventions strengthened various components of the health system as well as service delivery quality pertaining to individual vertical programs. The ET assessed this using a set of criteria adapted from a 2013 article by Chee et al17 about interpreting HSS interventions: 1. Have cross-cutting benefits beyond a single disease; 8 Kruk, M.E., A.D. Gage, C. Arsenault, et al. High-quality health systems in the Sustainable Development Goals era: Time for a revolution. Lancet Glob Health. 2018;6:e1196-e1252. 9 Delivering quality health services: A global imperative for universal health coverage. Geneva: World Health Organization, World Bank and Organization for Economic Co-operation Development (EODC), 2018. Licensee: CC BY-NC-SA 3.0 IGO. https://apps.who.int/iris/bitstream/handle/10665/272465/9789241513906-eng.pdf?sequence=1&isAllowed=y. 10 Ibid. 11 EQHA Annual Report, FY 2020. 12 WHO provides a framework that describes health systems in terms of six “building blocks”: service delivery, health workforce, information, medicines, financing, and governance. 13 World Health Organization. Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action. Geneva: World Health Organization. 2007. 14 Adam, Taghreed, Justine Hsu, Don de Savigny, John N. Lavis, John-Arne Røttingen, and Sara Bennett. “Evaluating Health Systems Strengthening Interventions in Low-Income and Middle-Income Countries: Are We Asking the Right Questions?” Health Policy and Planning 27 Suppl 4 (October 2012): iv9-19. https://doi.org/10.1093/heapol/czs086. 15 Adam, T., and D. de Savigny. “Systems Thinking for Strengthening Health Systems in LMICs: Need for a Paradigm Shift.” Health Policy and Planning 27, no. suppl 4 (October 1, 2012): iv1–3. https://doi.org/10.1093/heapol/czs084. 16 Witter, Sophie, Natasha Palmer, Dina Balabanova, Sandra Mounier-Jack, Tim Martineau, Anna Klicpera, Charity Jensen, Miguel Pugliese-Garcia, and Lucy Gilson. “Health System Strengthening—Reflections on Its Meaning, Assessment, and Our State of Knowledge.” The International Journal of Health Planning and Management 34, no. 4 (2019): e1980–89. https://doi.org/10.1002/hpm.2882. 17 Chee, G., N. Pielemeier, A. Lion, and C. Connor. (2013). Why differentiating between health system support and health system strengthening is needed. The International journal of health planning and management, 28(1), 85–94. https://doi.org/10.1002/hpm.2122. 15 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 2. Address policy and organizational constraints; 3. Produce permanent, systemic impact beyond the project’s term; and 4. Are tailored to country-specific constraints and opportunities with clearly defined roles for country institutions. In the sections that follow, the ET presents findings related to EQHA’s workstreams, assessing them on their ability to benefit the health system beyond a single disease, their potential to address organizational constraints, and production of lasting systemic impact. We have also included in Annex 1 a table assessing a selection of EQHA interventions against these criteria, along with the feasibility of scaling up the intervention in the light of the overall level of effort of EQHA (Annex 1 Additional Assessment of EQHA Interventions). FINDINGS: NATIONAL TB AND HIV PROGRAMS, GENDER PROGRAMMING, AND THE QAO EQHA supported the national TB and HIV programs in developing/updating guidelines and SOPs, but not their systemwide uptake. CENAT recognized the high-quality technical expertise that EQHA provided to update guidelines on TB diagnostics (based out of national laboratories) and the training curriculum for childhood TB. It also noted there is a need to disseminate the guidelines to service delivery levels and train staff in their use. In addition, EQHA supported SOPs updates for other TB thematic areas led by CENAT. These were expressed needs and the contribution from EQHA was not duplicative of concurrent support to CENAT from other partners. The extent of national program staff engagement in the updating process is not clear, nor is the program’s ability to identify the need for similar future updates and solicitation of partners’ technical support to facilitate this. As part of its facility-based QI work (described below), EQHA supported the improvement of TB screening in all outpatient departments (OPD), which was a facility-wide response with the aim of improving service delivery across multiple verticals. EQHA also supported NCHADS in developing an SOP for integrating management of NCDs into routine HIV services in two pilot facilities, resulting in facilities’ services utilization increase. NCHADS reported that this intervention is also being implemented by a partner in other locations. EQHA developed clinical vignettes on a range of thematic areas, for inclusion in the NQEM tool. Working with QAO and the national programs, EQHA continued earlier assistance given by other partners, and supported the development, testing, and inclusion of ten clinical vignettes into the tablet￾based NQEM tool on a range of thematic areas. The inclusion of vignettes on maternal and child health, FP, TB, HIV, and STIs is a first and was done in close collaboration with the national program. There are four other vignettes, including two on gender-based violence (GBV), that were developed subsequently. Draft documents related to H-EQIP 2 indicate that vignettes will continue to be a part of the assessment in the next phase. Partners believed vignettes only test knowledge and memory and need to be combined with competency-based tests. The revised NQEM tool for H-EQIP 2 includes assessing provider competencies using simulation equipment in RHs. EQHA supported MoH in developing the policy and tools for addressing GBV. It also continued earlier support to a multi-stakeholder effort to mainstream GBV management in one province. IP staff and GMAG members reported the broad collaboration among partners and EQHA’s technical expertise that led to the development and inclusion of four clinical vignettes on GBV case management, the Policy on Gender Mainstreaming in the Health Sector 2020-2024, and EQHA’s contribution in drafting curriculum on gender to be included as a stand-alone subject in the preservice training curriculum for nursing. EQHA is in the process of conducting site assessments on the Referral Hospitals’ GBV Services Response to inform the draft report. EQHA also developed a GBV tracking system (using telegram group and a mobile app) in conjunction with the USAID LINKAGES project and 16 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV later the Meeting Targets and Maintaining Epidemic Control (EpiC) project and piloted it for reporting GBV in Battambang Province. This tracking system has been working at the provincial level but has not taken off due to poor Internet connectivity in rural areas. Therefore, EQHA proposes introducing it into an urban location. EQHA continued to support a multisectoral response to GBV in Phnom Penh that LINKAGES/EpiC initiated and established a provincial TWG for GBV and HIV. However, health managers in the ODs were unclear of the activities of this group. EQHA’s support to QAO builds on earlier efforts. Senior MoH officials, partners, and the EQHA team reported that IT support provided to QAO built on earlier efforts and established an NQEM dashboard for enhanced data visualization and use. This has also made allocation of performance-based payments more efficient. Senior QAO officials reported they are heavily understaffed and will require continued support for this function. CONCLUSIONS: NATIONAL PROGRAMS AND POLICIES The updated guidelines and SOPs for the national TB program will remain in the system but their utilization of these guidelines will depend on their dissemination to sub-national/facility levels. There was no evidence that EQHA’s support to CENAT had features that would enable sustainability beyond EQHA, and it is characterized by vertical program support that does not necessarily have system-wide impact. EQHA’s support to improve TB screening in OPDs improved service delivery facility-wide. Similarly, the pilot for integrating HIV and NCD services has shown early effects on improving utilization. It is important, however, to increase the likelihood of HIV integration being institutionalized through careful documentation of the process and lessons learned. Both these interventions are narrowly focused on specific vertical programs and do not qualify as the levers that give rise to system￾wide impact and bolstering of the foundations of the health system, as the EQHA interventions are described in the FY 20 annual report. EQHA’s support to vertical programs is largely determined by the type of funding it receives. For a more in-depth discussion on this, please refer to the EQ3 Findings section. EQHA’s intervention to increase the clinical vignettes’ range in the NQEM tool, including those for GBV management, is already established in the system. However, the GBV TWG at the province level is still in its infancy and would require a level of effort that EQHA can ill afford, given the overall level of effort needed in the coming two years and COVID-19 impacts. We conclude that EQHA can leverage its transition planning to ensure that work completed on national policies and guidelines remain in the system and continue to receive technical and financial support beyond its term. We also conclude that the local-level interventions that are confined to a single vertical program will be difficult for EQHA to scale in light of the overall Activity effort, unless there is a dedicated team working on that intervention. FINDINGS: FACILITY-BASED QI EQHA's QI effort has several enablers that ensured both its success at the local level and its continued use. EQHA’s QI effort used an evidence-based IHI QI model (including the PDSA testing tool approach under Objective 2) that was adapted to complement the local context of the NQEM process. It promoted facility-based leadership, with a multidisciplinary team identifying specific gaps in service delivery quality as well as solutions. This effort also met a need felt by respondents: facility teams from RHs and HCs and managers at PHDs highlighted the value that the QI process added to improving the NQEM scores of facilities. However, some HCs face considerable constraints with infrastructure, which the facilities need to address along with the PHD for the NQEM scores to improve. Facilities noting (PDSA is) like a roadmap, guiding us where to go, all put together in a single document. No money is needed to do fishbone analysis. Head of an RH 17 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV improvements through QI cycles report they will continue the QI cycles even after EQHA ends. One RH head stated that they plan to scale back direct coaching for QI and focus on coaching the coaches. QICs provide the supporting structure to facility-level QI activities. EQHA describes QIC as a “...multi-level collaborative approach [that] allows for the joint decision-making on priorities, potential solutions, and implementation planning.” PHD and OD teams, RH and HC teams, EQHA staff teams, and reports describe the impact of QICs in helping replicate change ideas across all facilities in the PHD and OD. The Activity’s draft sustainability plan includes building PHDs’ and ODs’ capacity for leading QICs. EQHA’s facility-based QI intervention was not known among two key national stakeholder groups interviewed, and the Activity recognizes the need to utilize its work in facilities to shape national policy. One group of senior MoH officials who are very familiar with EQHA’s contributions to the NQEM tool and its digitization process as well as with the improvements in NQEM scores were not also familiar with EQHA’s QI intervention at the facility level. Another respondent from a partner organization that works with H-EQIP was also not aware of EQHA’s facility-based QI intervention. Recognitions of the approach include scale-up requests initiated by MoH for training in infection prevention and control (which EQHA has completed) and PSF (not scaled up yet), though this does not necessarily represent a full understanding of “what it takes” to set a QI process in motion. A key national level partner stated that they were not aware of the very fact that EQHA supports the QI process in facilities. While this finding pertains to only one interview participant and partner, the ET believes this is significant enough to report, as this is a key stakeholder in the national QI process. As part of its sustainability plan, EQHA has presented findings of its QIC work to the QI TWG and has hired a consultant to develop a handbook for QI. There is at least one other facility-based QI model being implemented, to support the H￾EQIP/NQEM process. Institutionalizing QI necessarily entails blended appropriate features from all models currently being implemented. EQHA is best positioned to initiate and lead the effort. One partner noted that GIZ implements a QI intervention in three PHDs facilities to support the H-EQIP/NQEM process. Partnering to Save Lives (PSL), which ended in 2018, implemented a QI model that supported NQEM though the model focused on reproductive, maternal, newborn, and child health (RMNCH).18 IP teams noted that participants in an accreditation workshop were able to list the QI models that were being implemented in the country, suggesting the presence of more than one additional model currently being implemented. Partners and MoH officials noted the need for collaborative effort, and two national level partners noted that EQHA is best positioned to lead this. IP teams noted that participants in an accreditation workshop were able to recall QI models actively being implemented in the country, though the EQHA QIC model was the most frequently recalled. This suggests that more than one QI model is currently being implemented, in support of H-EQIP/NQEM. Partners and MoH officials noted the need for collaborative effort, and two national-level partners noted EQHA is best positioned to lead this. While EQHA’s sustainability approach for its facility QI intervention seeks to shape national policy, it is not clear if this includes incorporating other QI model features that are operating in the country. Two national level partners noted the need for approaching the MoH “with one voice,” underscoring EQHA is best positioned to take on the role of leader and convener. The Activity has already set its sights on shaping national policy on QI, and rightly so, as it follows that QI efforts underway across the country 18 Partnering to Save Lives. Learning Package 1: Improving service quality. RGC Ministry of Health, Australian Aid, CARE, Marie Stopes International Cambodia, and Save the Children. 18 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV are brought together and assessed for effectiveness and feasibility (with national scale up in mind) and a harmonized model that combines the most relevant features from all the QI interventions is proposed. This will require broad-based collaboration among partners, the MoH, and the private sector, and EQHA has the technical expertise, experience, resources, mandate, and national reputation to lead this process at this juncture. EQHA’s sustainability plan includes integrating QI in its other workstreams. EQHA’s draft plans for sustainability include integration of QI in preservice, including QI efforts in CPD credits, linking accreditation and QI and to set up QICs with private facilities, and procuring the services of a consultant to develop a QI handbook. As discussed earlier, these efforts are more likely to result in institutionalization if EQHA leads the development plans for scale up of the EQHA implemented QIC model and a harmonized QI model that includes other QI model features, particularly those operating in-country and in support of the NQEM process. CONCLUSIONS: FACILITY-BASED QI EQHA’s QIC model in target facilities emerged as “proof of concept” and has created a QI “mindset” within the facilities, PHDs, and ODs covered. This model complements the H-EQIP/NQEM process. A key enabler for intervention success is EQHA meeting the felt need of improving NQEM scores, improving disease outcomes with a proven intervention that puts the facility management team in the lead. Other enablers for success include the QIC model being led by facility teams and the focus on small, actionable steps that resulted in quick wins. The QI process is likely to continue to be used in the facilities that have successfully carried out QI cycles. EQHA has been a key facilitator of the QITWG and a key player in QI in the country. As noted earlier, advocating for a national policy on QI and the adoption of a uniform QI model for facilities is more likely to succeed if all QI players engaged in a coordinated manner. EQHA is best positioned to convene all stakeholders engaged in facility-based QI work in the country and lead the development of a harmonized model that combines features from all appropriate models for the country’s health systems context. This model would complement the NQEM process, specifically its upcoming version that incorporates accreditation standards, leading to the development of a national QI policy. This model stands a better chance at acceptance for nationwide scale up through joint advocacy, and later for integration into other EQHA workstreams. We conclude that EQHA’s interventions in this workstream can be leveraged into institutionalization within the local system if EQHA leads stakeholder convenings, develops a QI model, and leads national level advocacy for its adoption. We also conclude that EQHA’s interventions in the geographies where it currently works are very likely to be institutionalized into the local systems. FINDINGS: ACCREDITATION AND MINIMUM STANDARDS FOR LICENSING Developing a hospital accreditation system has been a long-standing ambition of the MoH, and EQHA’s ongoing TA and financial support has helped reach crucial milestones on that journey. Senior MOH officials and partners note EQHA’s fast-tracking of the hospital accreditation standards development that had been stuck at a preliminary stage for more than a decade. The previously slow progress had been due to the lack of a strong technical voice that could bring together and consolidate diverse perspectives along with global evidence. EQHA met this need in the form of international expertise and a strong support team. MoH officials and partners, as well as the documents reviewed, affirm the high TA quality that EQHA provided. MoH officials and partners describe the process the EQHA team adopted as highly effective. It brought together a large and diverse team of collaborators from the public and private sector and the development community, started with what 19 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV was available in country, and identified gaps against international standards. EQHA assigned each of the 12 sections of CHS to a working group, along with the tools with which to work, to develop the first edition. EQHA also provided TA to the MoH to submit the draft product to be reviewed by the International Society for Quality in Health Care and supported pilot testing of the standards. MoH officials and partners confirm the broad contours of the plan that is taking shape, based on the roadmap drafted by EQHA, in which EQHA is also taking a lead role alongside the QAO, the QI WG, and GIZ. The plan includes actions that have been completed or are ongoing: drafting legislative instruments to establish the accreditation organization, supporting the establishment of a TWG for developing the Cambodia Primary Healthcare Accreditation Standards (CPHAS) for primary healthcare facilities, training of surveyors and accreditation coordinators for facilities, and a strategy to communicate the standards across all facilities. Continued technical and financial support will be needed through the term of EQHA and beyond. Partners, MoH officials, and EQHA note the long road ahead in establishing the accreditation system as a well-functioning mechanism that encompasses public and private facilities at all levels of care. Continued technical assistance from EQHA, as planned in the roadmap will be required. Financial investment (from sources outside USAID) is needed until the accreditation process becomes self￾sustaining. MoH officials and partners stress the need for thoughtful exit strategies for EQHA. Partners from bilateral and multilateral organizations also note that more recently, the MoH has been pushing the boundary on accreditation by incorporating it into the next version of the NQEM tool, to be used under H-EQIP 2. MoH officials, partners, and a draft document related to H-EQIP 2 indicate that this tool will incorporate CHAS alongside an expanded edition of the clinical vignettes for hospitals; EQHA has been asked to develop another set of standards for health centers. However, it is unclear if the present edition of CHAS or if the upcoming CPHAS will be used. It is also unclear how the voluntary nature of accreditation will be maintained when it is linked to performance through the NQEM tool. Continued support will be needed through the term of EQHA and beyond to advocate for legal instruments issuance required to establish the Health Service Accreditation Committee (HSAC), support them to train surveyors, and to provide technical support to incorporate the accreditation standards in the next version of NQEM. Lack of political consensus could make progress slower than planned. Partners and MoH officials note that while consensus has been built at the technical level, largely through EQHA’s efforts, the legislation has still not passed, and the establishment of an accreditation body is dependent on it. Partners note the limited role they have in moving this forward. MoH officials, partners, and EQHA documents report that the Activity fast-tracked the work started by GIZ and WHO in developing a framework and minimum standards for initial licensing, interim check, and relicensing of private health facilities through a TWG. This was required as a precedent to developing the accreditation standards. The minimum standards are yet to be approved by the MoH. The Ministry’s long-range plan is to link the relicensing of private facilities with accreditation standards, but that will require a discussion on the mandatory nature of relicensing and the voluntary nature of accreditation. One partner noted the lack of clarity on what the appropriate sequence of these activities is, as well as the need for a “home” within the MoH for the minimum standards for licensing private facilities. EQHA is a pioneer and has the relative advantage of being in the front seat (for leading the development of accreditation standards). The government is committed but will require support for 10 years down the road. Accreditation is a long road. It’s a culture change. - Senior official in partner organization 20 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV CONCLUSIONS: ACCREDITATION AND MINIMUM STANDARDS FOR LICENSING The MoH’s commitment to see the accreditation system fully operationalized is unlikely to wane. The investment that EQHA has made (and plans to continue to make) in this area has further enhanced the momentum across the public and private sector. However, it will require considerable technical and financial support beyond EQHA’s lifetime to maintain the momentum until the system is fully institutionalized and self-sustaining. EQHA is best positioned to convene partners engaged in this effort and advocate for continued technical and financial support. This could be included as activities in its transition plans. Uncertainty related to passing of the legislation remains an unresolved challenge in fully establishing the accreditation system. In the nearer term, implementing the roadmap EQHA laid out, including the incorporation of minimum standards for licensing of private facilities, will require continued technical leadership of EQHA to convene and lead partners to attain clarity and consensus on these processes and also lead the advocacy with the MoH for the required approvals. We conclude that EQHA’s investments in this area are likely to be institutionalized with continued support from EQHA and later, from the partner community. FINDINGS: DEVELOPMENT OF HPCS AND CPD EQHA continues support to HPCs initiated under USAID ASSIST to improve the effectiveness and cost efficiency of their operations. Building on the efforts of USAID ASSIST to reinforce the registration, licensing, and relicensing of health professionals throughout the country, EQHA has further fine-tuned the support to reinforce individual HPCs. EQHA continued supporting the Joint Secretariat of HPCs as part of the HPC strategic plan of 2015-2020 that USAID ASSIST supported. In FY 2020, EQHA ended this support based on an external review of the functioning of the Joint Secretariat and developed a template for HPCs to develop their individual strategic plans for 2021-2025. EQHA has also upgraded the RMS for four out of the five HPCs with customized features for each HPC to enable each to update them in the future without undermining the stability of the system, and updated legal instruments required to implement the 2016 Law on Regulation of Healthcare Practitioners, except for DCC. The Activity also supported the development of the scope and standards of practice for nursing and midwifery, which senior MoH officials saw as an effective step and believed would be required for all five professions. EQHA supported the launch and dissemination of CPD guidelines developed by GIZ for CCN and CMC, along with promotional material. The Activity also supported the translation of CPD guidelines for midwives (developed prior to the Activity) in English, which is useful for expatriate midwives. EQHA also engaged CCN in its efforts on preservice training efforts for nursing. EQHA has refined the CPD guidelines, but they need further development, finalization, and support for implementation. EQHA refined CPD guidelines using best practices such as active learning. Based on consultations with the HPCs, EQHA developed the Proposed Guidelines for CPD providers, July 2019, which is also meant to serve as a template for HPCs to recruit potential CPD training providers and moved from the earlier credit-based system to a simplified and common system based on actual hours spent in “active learning” translated to points. The Guidelines list recognized CPD providers as well as other potential cadres of providers and includes interprofessional topics such as soft skills. EQHA has also produced promotional material including factsheets for each cadre. EQHA’s support and collaboration has been consistent and fruitful with CCN, CMC, and PCC. Of these, the Guidelines for CCN have been approved and are publicly available, while the others are in varying stages of development. Concerns regarding low capacity for implementation, raised by CCN, PCC, and CMC since the time of USAID ASSIST, remain unaddressed. Senior MoH officials, partners, and HPC 21 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV representatives state the need for support for implementation in the form of staffing and financial support to implement the plans and guidelines. They pointed to the need for a systematic assessment of what it would take for the HPCs to become fully functional and fulfill their mandate and cited the elected status and voluntary nature of positions in HPCs that are “high on commitment, and some capacity,” as a senior MoH officer described them. The HPC representatives themselves stated categorically that they do not have the time to implement their strategic plans, especially reaching out to the ever-growing private and NGO cadres. Thus, key constraints include lack of sources of funding and full-time staff. These unaddressed issues are a major barrier for the rollout of the upgraded system and updated guidelines. According to its reports, EQHA plans to hand over the RMS to the four HPCs in FY 21, but HPCs and partners report that the registration process needs ongoing support. While the cost of hosting the RMS is low, HPCs report that they need support in identifying issues in the system and procuring assistance. It is also likely that there is a need for a longer period of coaching until the HPC teams get used to the system. HPCs are unable to implement their strategic plan without significant additional financial and technical support. HPCs, senior MoH officials, and partners stated that while the strategy and plans developed with EQHA’s support are comprehensive, the gaps in capacity are too wide for the Councils to meet them on their own. They reiterate that the HPCs simply do not have the capacity to scale nationwide. While the guidelines and scope of practice are up to date, the Councils do not have the means to take them to their respective constituencies. The Councils also highlighted that these are issues they raised with USAID ASSIST. They recognize that the strategic plans that they drafted with assistance from EQHA require large-scale dissemination of the requirements of registration including CPD and training their cadre of workers on the scope of practice, but they do not have the capacity to carry these out. Key capacity constraints are staffing and financial support to disseminate and train widely. EQHA is continuing to work to secure the commitment, ownership, and availability of HPCs to finalize the scope and standards of practice and SOPs. This has delayed subsequent activities such as dissemination and training to the sub-national levels. The draft strategic plan of CCN for 2021-25 includes a costed activity plan that will require significant financial and technical assistance to be implemented. EQHA has offered technical assistance in analyzing revenue flows from registration fees and a proposal to ensure that each Council is self-funded with adequate annual revenue to carry out its regulatory responsibilities. A landscape analysis carried out by EQHA also recommended improved coordination between HPCs, in-service training providers, and healthcare professionals. Partners and HPC representatives cited the need for building the capacity of HPC staff teams to implement their strategic plans, and to carefully transition the support and help HPCs find sustainable ways to support themselves over time. The councils were aware of the size of the task of mobilizing their cadres toward registration and CPD, particularly with those in the private and NGO sectors. HPC representatives state that during the Workplace Health workshop that EQHA organized for representatives from MoLVT, CCN, CMC, and MCC, along with healthcare workers in factories, it was clear that these healthcare workers in the private sector as well as their employers were largely unaware that registration was required, and by extension, had no idea about CPD and relicensing. In the words of one HPC representative, “There’s such a big distance between them and the laws.” HPC representatives also highlighted the need to sensitize the employers of these healthcare workers to the legal requirements of registration and relicensing. HPCs also reported the challenges with changing the “We can’t use one system (RMS) for all councils. It doesn’t work. It doesn’t respond to what we each need. It is one house with many rooms, trying to fit many people. It even takes much longer than doing it manually. Manual is faster and we have confidence in it.” - Representative of one HPC 22 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV attitudes of older practitioners toward continuous and lifelong learning in a culture with high deference for age and experience. PCC has progressed further than the other HPCs in disseminating CPD guidelines and enrolling their cadre for CPD through the Swipe Rx intervention. HPC representatives, partners, IP teams, and EQHA documents showed that PCC, the last council to be established, has made further progress with registration and strengthening collaboration with a wide range of stakeholders. With EQHA support, PCC disseminated the CPD guidelines in 25 provinces in collaboration with the PHDs. PCC has also partnered with academic institutions and associations of pharmacists to utilize their training events to provide orientation on registration and CPD, and have begun to issue registrations and two-year licenses, with no external staff support. There are clear areas identified for EQHA’s support, such as obtaining legal advice on disciplinary measures related to registration and licensing, and measures to prevent “ghost attendants” taking the online courses on behalf of the pharmacist. However, there are still issues with the RMS that have persisted since the time of ASSIST, especially in the provinces, as the PCC is unable to pay for the IT support required to resolve the issues. They also report that older pharmacists find it harder to use the online system. In effect, PCC has not yet begun using the online registration system. PCC reported that they had conveyed to EQHA their preference for including core subject matters related to pharmacy rather than interprofessional topics like soft skills, and cover topics that are relevant for the various parts in the overall health system where pharmacists may be employed. At the same time, the mClinica’s SwipeRX study on Knowledge, Attitudes and Practices (KAP) for pharmacists showed that more than a quarter of respondents had unhelpful attitudes toward FP use among youth, indicating that this continues to require work. EQHA and GIZ worked together in the initial phases, but broad-based collaboration has since been limited to PCC’s activities. One partner was unsure of the reasons for transitioning out of the Joint Secretariat. EQHA began a pilot initiative on CPD champions in two provinces from February 2021, using the QI approach it uses for facility-based improvement, and will evaluate it and disseminate findings in July 2021. CONCLUSIONS: DEVELOPMENT OF HPCS AND CPD While each HPC is at differing levels of progress, the present structure and functioning of councils necessitates intensified support that is customized to specific challenges and gaps identified in each council. Strengthening HPCs’ nationwide reach and effectiveness is critical for EQHA’s private sector and workplace interventions. Unlike the work on accreditation, the work with HPCs already has the necessary legal authorization to continue and scale up. HPC capacity development is required for it to become high-performing and self-supporting, and in their current state, the CCN, CMC, and PCC require significant input of financial and technical resources (in the form of full-time staff, funding for the staff and for implementing the strategic plans and technical support for RMS), brought in and managed through a collaborative process with other partners and MoH. Without such input, the HPCs are likely to remain without the capacity to implement their strategic plans. PCC has made further progress than the rest of the HPCs, likely through the combined effect of intensive support from EQHA and the online CPD application as well as other characteristics of the PCC team. This warrants further study to identify enablers that can be adopted for the other HPCs’ contexts. We conclude that EQHA’s interventions with HPCs need increased technical and financial support to reach a stage where they would be able to continue in the system with ongoing support from partners. 23 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FINDINGS: WORKPLACE HEALTH EQHA employed a systems approach to strengthening workplace health, with a focus on the garment industry. Senior MoH and MoLVT officials, HPC representatives, and EQHA staff commended the range of interventions of the Activity in this area. Achievements included reviewing the National Enterprise Infirmary Guidelines; collaborating with NGOs working in the sector, employers, and buyers; and reinstating the WHWG, a mechanism for facilitating workplace health issues (in which EQHA functions as the secretariat and set up an information sharing platform) to advocate for workplace health best practices. EQHA collaboratively assessed the status of implementation of the infirmary guidelines and other practices in 64 factories, identified further actions to improve workplace health, and engaged HPCs and MoLVT in this assessment. The Garment Manufacturers Association of Cambodia is also engaged in the process. MoLVT is keen on further engagement and expansion; EQHA’s province- and infirmary-level activities are set to expand. Senior MoLVT officials pointed out the need for further engagement of buyers to ensure compliance in infirmary practices, like the BFC initiative of the ILO. Future activities lined up include a review of the legal instruments related to infirmary staff, developing additional guidelines to address gaps, and developing a joint work plan with HPCs to ensure registration and licensing of nurses and midwives in infirmaries. MoLVT officials recommended that EQHA scale up its work with infirmaries to all special economic zones in the country and help them establish referral links with local facilities in collaboration with PHDs. EQHA has begun replicating a model employed by another NGO to improve collaboration between infirmaries and local HCs in one province. This collaboration currently includes referral linkages but has the potential to improve registration and licensing of healthcare professionals in infirmaries facilitated by the PHD. The plan for the coming years is to go to scale in this province, and possibly beyond. EQHA plans to integrate its work in this area with its other workstreams. The weakest link and the greatest opportunity in these efforts is the capacity of HPCs. EQHA plans to explore expanding FP training currently offered in private facilities in two provinces to healthcare professionals in the workplace. It will explore extending the use of virtual platforms to offer CPD to these professionals. Finally, it plans to assist CCN and CMC in conducting needs assessments in infirmaries similar to the exercise EQHA did earlier. MoLVT officials emphasized the need to regulate these workers through registration and licensing, and the low coverage of registration was a key finding in the assessment conducted by EQHA. As seen in the Findings: Development of HPC and CPD section, CCN and CMC will require continued, significant financial and technical support to conduct this assessment, and to continue to engage infirmary-level healthcare professionals. Leveraging existing opportunities to institutionalize these interventions would mean that EQHA plays the role of facilitator and highlights the HPCs. CONCLUSIONS: WORKPLACE HEALTH Through a systems approach, EQHA has laid a solid foundation of extending its HSS interventions into the workplace. It has significant involvement in the functioning of the WHWG, which will have to be transitioned well for the interventions to continue. EQHA should touch on the [essential services for] health [of employees] while MoLVT inspectors and BFC assessors focus on occupational safety. -Senior MoLVT official 24 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA’s plans for HPC-led assessments of factory infirmaries and integrating FP in CPD programs for infirmary workers are great opportunities to build the capacity of the HPCs, in collaboration with other partners in this space, and to gradually build their membership base across provinces as well as their working relationships with PHDs. The Activity’s work at the level of infirmaries, linking them with HCs and conducting outreach camps, are useful as pilots, but the work done elsewhere has already shown this to be successful. This does not line up with EQHA’s overall direction of system-wide support. We conclude that EQHA’s work in this area is a mix of interventions that are likely to be institutionalized and continue in the system, and those that are at the local level will require a considerable level of effort to be expanded. FINDINGS: PRESERVICE TRAINING EQHA’s high-quality work with the nursing curriculum has helped transform the understanding of MoH and other stakeholders on preservice education. MoH officials, sub-national health staff, partners, and academics applaud the highly collaborative effort that EQHA undertook in this workstream. EQHA brought in high quality TA on competency-based education for the first time in the country and this resulted in a significant mind shift across the board, not just toward CBE but also toward the sequence of developing the material, and not just in the curriculum for the associate degree in nursing but for curricula in other professions as well. Staff of HRDD, RTCs, public and private academic institutions, and private hospitals appreciated being included in developing the work products, even if it meant additional work for them. EQHA staff and GMAG representatives considered the inclusion of topics on gender and QI as important progress in the curriculum. There is wide consensus that progress with the nursing curriculum will depend on collaborative effort. MoH officials were clear that the process of endorsement, approval, and rollout of the EQHA-led curriculum will be part of a collaborative effort involving multiple partners. Senior MoH officials stated that they will learn from all partners engaged in preservice education, and that EQHA support stands as a foundation, but not a stand-alone. IP staff believed that the focal person for their work in HRDD, MoH will ensure that the next step in the process is for the steering committee to endorse the curriculum, and that they cannot influence the process any further unless there are additional resources to offer support on more curricula. However, senior MoH officials stated that EQHA’s work will need to merge with other efforts taking place for other professions, and engage actively with that process in order to identify what it would take to ensure approval of the nursing curriculum. Partners stated that endorsement of the nursing curriculum might be undermined if it is not a collaborative effort alongside concurrent efforts (currently led by UNFPA and the World Bank-led consortium) to revise curricula for midwifery and medicine. MoH is likely to give the latter charge over revising all curricula into the CBE framework, for which they believe that only EQHA has the needed skillset. Both EQHA staff and partners recognize the political context within which these changes are taking place. EQHA did foundational work. This created additional interest and leveraged support from other partners. -Senior official in a partner organization We will consider all the pieces and see what EQHA can contribute. We recognize what we need…. CBE connects everything. But EQHA needs to work with MOH…. It can’t be one organization’s effort. -Senior MOH official 25 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Beyond endorsement of the curriculum, it will be a long journey before the education transitions fully to CBE. Partners, EQHA staff, senior MoH officials, and RTC staff report that a range of other improvements need to be overcome before CBE can be rolled out. These include preparing the schools for the transition, training and retaining the instructors, improving the overall quality of care in clinical training sites, and increasing the training capacity of preceptors. CONCLUSIONS: PRESERVICE TRAINING EQHA brought in technical expertise previously unavailable in the country and built understanding and consensus on what is considered best practice in preservice education. Due to the donor landscape, its work had to focus relatively narrowly on the nursing curriculum, but the broad collaboration it built for this work is what led to a transformation in the collective understanding of preservice education. We conclude that EQHA’s work in preservice education is poised to become embedded in the local system but needs continued support to prepare local systems for implementing it and to incorporate CBE in other health professional courses. The gains that EQHA made in the nursing curriculum, as well as the change it brought about in the collective mindset toward CBE, can be protected and advanced if EQHA leverages its technical expertise and the goodwill it has built through its HSS work thus far at the national level as a credible provider of high-quality TA, and if EQHA pours in additional resources to offer the expertise to transform curricula of other professions, in collaboration with all relevant players. EQ5. HOW EFFECTIVE HAVE THE EQHA APPROACHES BEEN IN ENGAGING THE PRIVATE SECTOR IN PROGRAM IMPLEMENTATION? In this section, we present findings related to EQHA’s interventions with private facilities and private healthcare professionals (hereafter called providers). Workplace health and development of HPCs (and to a lesser extent the work on accreditation standards) also have a significant component for the private sector; those are addressed as separate sections under EQ2 and will not be repeated here, except for the interlinkages. FINDINGS The proliferation of private facilities and providers is a common concern, as is the lack of coordination mechanisms. EQHA has used a systems approach to engaging them. Representatives of HPCs, MoH officials, and partners note the rapidly growing numbers of private providers of all cadres, making this a large, high-energy group. HPCs in particular pointed out that many providers are not even aware of what HPCs are, or of the need for registration, licensing, and relicensing, and are disconnected from the rest of the healthcare system. As with its other workstreams, EQHA’s interventions carried out so far utilize a systems approach, engaging policymakers and the regulatory mechanism including digitizing the one window service system for registrations of small private facilities at subnational levels, providers, and clients. The Activity also has interventions for these providers across its four objectives. This is a recognition of the scope and size of the problem and the breadth of engagement needed in response to it. EQHA has helped amplify the voice of the private sector in public forums at the national and sub-national levels, but this is just the beginning. EQHA’s and others’ assessments have pointed to the lack of a clear sense of the size of the private sector, and the lack of representation of the private providers in policymaking. EQHA has been intentional in including participants from the private sector in forums and workshops for the drafting of accreditation standards and the drafting of the competency-based curriculum for nursing. Respondents from these groups appreciated the 26 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV opportunity to participate in public initiatives and felt that their concerns were “heard.” The relatively small number of licensed private providers included in the evaluation sample noted their concern about the lack of openness in public facilities to collaborating with them, and their desire for partners to encourage openness in this regard. This was also noted in EQHA’s studies. The providers also reported that most of their fellow providers fear lawsuits for alleged malpractice and detentions without due process. EQHA shifted its focus to the sub-national level due to challenges in engaging national-level bodies. After EQHA’s initial plan for forming an association of private hospitals was halted, it decided to move its engagement to sub-national levels. This move took advantage of EQHA’s established presence in its target provinces. EQHA plans to further engage the sub-TWG for public-private partnerships (PPPs) as a potential mechanism within MoH for the Ministry to partner with the private sector, but partners report that the PPP sub-TWG has been dormant for some time. There is need for engaging networks of HPCs at provinces, ODs and communes through provincial authorities, and large-scale communication efforts on regulations. This effort needs to extend to ODs and communes through provincial authorities. EQHA’s assessments and responses from HPC representatives and private providers provide a window to the need for a large-scale communication effort among private providers to orient them about regulations. Private providers stated that they were in favor of enforcing the legislation on registration and licensing as the only way forward to meaningfully engage and regulate private providers and facilities. This engagement will also open the door for engaging private facilities to comply with minimum standards. The deputy governor of a province reported that they comply with MoH guidelines for issuing licenses and have committees at provincial and district levels to monitor the functioning of the private providers. Commune chiefs reported that they monitor private cabinets and pharmacies and encourage them to register. In the past, they provided information related to private providers to the OD and PHD but have not heard back from the health managers yet. The chief of one commune recounted how the commune council prevented a private provider from continuing the practice as there was no effort to be registered. EQHA’s intervention with private facilities in two provinces has an FP focus and is a facility-based intervention that is part of a larger system-wide FP intervention. Licensed private providers had an appetite for more opportunities to engage with the public system. EQHA’s work with 18 private facilities in two provinces as a pilot, along with the Cambodia Midwives Association (CMA) and provincial Maternal and Child Health units, is focused on FP. This is based on the source of available funds and is part of EQHA’s system-wide effort on FP. Activities include providing basic FP training for providers, which EQHA considers to be an entry point for greater sub-national engagement. The additional resources made available for this were part of the opportunity module built into EQHA’s cooperative agreement with USAID, to respond to needs in specific thematic areas during the life of the Activity. Through this arrangement, EQHA aims to bring about system-wide improvement, beginning with FP. EQHA has also helped link these providers with the respective PHDs, facilitating formal partnerships, which includes ensuring the participation of private providers in PHD-level training events. The trust quotient built through QI work arguably facilitated this process with PHDs. The gaps identified in the private sector show that these facilities and providers need to be engaged more broadly than in FP alone, which is the direction that EQHA plans to take, by engaging these facilities and providers in implementing the minimum standards, and later apply for accreditation. The present level of engagement of these 18 facilities, however, is limited to FP service provision. 27 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Private providers who were familiar with EQHA’s work saw the Activity as a potential bridge between them and the public health system. They also recommended that EQHA work through the existing public health system (PHD and OD) to enhance their reach to private providers. Private providers at the province and OD levels were keen on collaborating with EQHA in QI activities. They also wanted to be part of training events organized by the PHD, especially in the area of waste management and infection prevention. They bemoaned the lack of training opportunities available for private providers. CONCLUSIONS Rather than be overwhelmed by the size of the problem of the unregulated and proliferating private facilities and private providers, EQHA has started to address the issue strategically by engaging actors related to the private sector across the health system. EQHA’s inclusion of the private sector voice in its public engagements is a model for other partners and is the beginning of building confidence among this diverse group. EQHA’s decision to focus on the sub-national levels for its work with private facilities and providers is in line with the overall direction toward which the findings related to HPCs is also pointing. EQHA’s work with HPCs should be linked to its work with private facilities and providers. This would include considerable financial and technical investments in strengthening HPCs at sub-national levels, including their linkages with the province-level system for registration of facilities and extending them to ODs. EQHA plans to support the fundraising efforts of HPCs, as USAID funds cannot be used to directly support government entities like HPCs. EQHA’s work with private facilities and the CMA and provincial MCH teams in FP is part of a larger system-wide initiative to strengthen FP. EQHA expects this pilot effort to be leveraged to strengthen HPCs in the provinces, communes and ODs, going beyond strengthening service delivery for FP. We conclude that all of EQHA’s interventions with private providers and facilities are effective in having the potential to produce intended results. Although the current effort with private facilities is to strengthen FP service delivery, EQHA’s strategy is to use this as an entry point to a range of system￾strengthening interventions in the private sector. EQ3. WHAT ARE KEY CHALLENGES IN ACHIEVING THE EXPECTED RESULTS OF EQHA AND PHB AND HOW TO OVERCOME THE CHALLENGES? FINDINGS In this section, we present and discuss challenges that cut across more than one workstream of EQHA, which threaten the achievement of its objectives, as well as steps that are likely to mitigate them. The impact of COVID-19 extends to all aspects of Activity planning and implementation. Restrictions on travel have led to planned activities being postponed or cancelled at the national and sub-national levels. As meetings and workshops moved to remote platforms, stakeholders adapted to the new environment at varying paces, although most of them had adjusted to the new normal by the time of the evaluation. More importantly, EQHA had to dedicate staff to the COVID-19 response, and the availability of stakeholders in the government and among partners has also been significantly curtailed as every Ministry and organization poured its resources into the pandemic response. EQHA 28 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV will have to recalibrate all its plans for the rest of its term to accommodate these limitations. Recommendations provided below account for the strict prioritizing that EQHA will require. Accreditation of hospitals and primary healthcare centers aim to improve overall quality but could overshadow the facility-based QI initiative of EQHA, especially when accreditation standards become tied to performance. Draft documents related to HEQIP-2 note that the 11 standards of CHAS as well as clinical vignettes will be part of the next version of the NQEM tool, thus mandating compliance with CHAS. QI is one of the 11 sets of standards in CHAS, and EQHA reports that it will utilize this opportunity both ways: to use the platform of its QI work to increase facilities’ appetite for accreditation, and to advocate with MoH that complying with QI will enable facilities to get accredited. Even as the rollout of CHAS is anticipated, it will be important for EQHA to continue its focus on implementing and scaling up the facility-level QI activities that EQHA has promoted. A 2014 study19 to examine the relationship between patient safety practices, as measured by accreditation standards, and patient safety outcomes in U.S. hospitals found that accreditation standards reflecting patient safety practices were related to some outcomes but not others. This shows that certain adverse events may be reduced by protocols that are reflected in accreditation standards, whereas other events may require multifaceted strategies that are less easily translated into protocols. As EQHA plans to advocate for a harmonized QI model and lead the shaping of a national QI policy, care should be taken that these are not overshadowed by the effort needed to roll out CHAS. Another related challenge is regarding public perception of public health services and the extent to which accreditation of facilities could affect that. The concept note on HEQIP-2 states that the MoH is in the process of establishing a national accreditation system as a way of building public confidence in public facilities. However, this does not consider the problem of asymmetry in perceptions of providers and those of clients regarding improvements brought about by this measure, and the extent to which communication efforts around accreditation will help redirect client choices. Performance indicators in the MEL plan do not adequately capture the complexity and scope of EQHA’s interventions. Indicators at the level of objectives are directly related to the deliverables under the objective, but are unweighted numeric measures, such as “Number of policies and guidelines approved with EQHA assistance” and “Number of technical resources developed with EQHA support to strengthen preservice training.” To elaborate, the policies and guidelines included in the actuals when reporting against the first indicator quoted here are likely to have taken varying levels of effort to develop. The relative significance of each on overall clinical practice and patient outcomes is also likely to differ. The indicator currently being used does not offer this nuance. Others capture results that are removed from the interventions of EQHA, such as “Percentage of health professionals (public and private) with a valid license.” Still others do not capture the different intensities of EQHA support within a workstream, but are blanket statements, such as “Percentage of health professionals registered with HPCs.” While there is no evidence that this has affected EQHA’s performance, and therefore does not pose a challenge to the achievement of objectives, a set of indicators that are sufficiently nuanced to track Activity deliverables and capture their proximate effects will be useful to validate the theory of change on which EQHA is founded, adequately showcase its achievements, and help track progress in implementation. Therefore, the absence of such indicators could be seen as a challenge that requires mitigation. 19 Thornlow, D.K., and E. Merwin. Managing to improve quality: the relationship between accreditation standards, safety practices, and patient outcomes. Health Care Manage Rev. 2009 Jul-Sep;34(3):262-72. doi: 10.1097/HMR.0b013e3181a16bce. PMID: 19625831. 29 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV This challenge could be mitigated by including at least one indicator for each objective that is a composite, weighted index20 of the various outputs that are necessary and sufficient to reach the objective. A useful definition for composite indices is that they are “formed when individual indicators are compiled into a single index, on the basis of an underlying model of the multi-dimensional concept that is being measured.” Weights assigned to individual items should be based on the relative importance of that item as a deliverable of EQHA, and what the absence of its achievement would mean for the overall objective. While there are several resources available to develop composite indicators, this exercise should obviously not be an extensive one, but seek to establish a simple but robust way of tracking the multi￾faceted interventions of EQHA, based on a methodology agreed upon by the EQHA team. Composite indices, like the Human Development Index, are most useful when they are used to compare across locations and over time, but they do find a place in tracking multi-faceted initiatives like EQHA in the long term. We include an illustration of a composite index in Annex 1 Additional Assessments of EQHA. Each objective could also have a set of yes/no indicators that track individual outputs. Such indicators are already part of a system to track progress internally. These could also be used at output levels, such as: ● Hospital accreditation standards developed and submitted for endorsement (Yes/No) ● Primary healthcare accreditation standards developed and submitted for endorsement (Yes/No) Funding tied to specific disease programs limits EQHA’s ability to intervene across the health system. The Activity’s strategic approach consists largely of system-wide actions, but funding that is tied to specific thematic areas such as FP and TB requires that performance related to these thematic areas is improved. This limits EQHA's ability to utilize the funds for system-wide activities that affect more than one thematic area. CONCLUSIONS We conclude that EQHA faces a range of challenges in reaching its objectives. These include both the impact of restrictions related to COVID-19 and diversion of resources to its response and the massive effort required to rollout accreditation crowding out the efforts needed in facilities for conducting QI cycles. Performance monitoring indicators of EQHA have room for improvement, to better reflect the complexity of the work that is being done. Funding that is tied to specific thematic areas constrains the ability of EQHA to carry out system-wide activities. 20 Greco, S., et al. On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation and Robustness. Soc Indic Res (2019) 141:61–94 https://doi.org/10.1007/s11205-017-1832-9. 30 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQ4. HOW HAVE THE EQHA’S AND PHB’S STRUCTURE, MANAGEMENT, AND INTERNAL PROCEDURES AND SUB-AWARDEES’ MANAGEMENT AFFECTED IMPLEMENTATION AND OUTCOMES, POSITIVELY AND/OR NEGATIVELY? FINDINGS EQHA is a well-organized consortium. Its leadership is highly consultative, and work processes and sub￾awardee management are collaborative. Sub-awardees report that leadership is diffused across the consortium and across EQHA’s technical areas. The leadership team conducts discussions in an open and collaborative manner, is focused on results, and asks tough questions of sub-awardees. The annual work plan process is highly collaborative and transparent. Sub￾awardees also report that the frequent changes seen in early stages have since settled down, and recognize the many interrelated discussions happening simultaneously, which highlights the need to maintain focus on communication across all the teams. When teams get siloed, they reach out to the senior leadership, who should facilitate communication across the vertical teams. Sub-awardees report that the IP coordinates and leads all external engagements and believe this has resulted in streamlined and efficient relationships with the national programs of the MoH and partners. The IP manages the reporting process and timelines well. The IP has also helped a sub-awardee transition to a fully TA role by absorbing its national staff on its payroll. Sub-awardees also report that while the in-country IP team provides rapid review and feedback on concept notes and financial reports, regional and global levels take longer, and so additional time needs to be built into that process. EQHA ensures evidence-based design and technical quality of its interventions. IP staff state that the core values of FHI 360 on evidence-based design, technical robustness, and social relevance guide the intervention choices and day-to-day interactions. The functional teams have optimal levels of autonomy and at least one subject matter expert, resulting in high performance. There are opportunities to share learning across the organization. EQHA is focused on sustainability in its design and operations. Sub-awardees report that the leadership team also organizes regular discussions around sustainability. The Activity documents outline its emerging plans for transitioning and sustainability. EQHA prioritizes staff development. Sub-national level staff need more support to maintain focus on facilitating rather than doing. EQHA has a strong focus on the professional development of its teams. It intentionally invests in female staff and ensures that their voice is heard. Sub-national staff report that their teams are kept small as their role is to enable change and not implement, but in some locations, they have to “jump in and join the government staff.” The EQHA brief on QICs also noted that the QIC meetings at the PHD and OD levels were mainly led by EQHA staff and planned to identify and equip health managers as QI coordinators. This points to the need to further build the skills of EQHA staff in maintaining their focus on enabling the PHD/OD/facility teams, track the level of effort needed to facilitate and enable change, and calibrate team sizes accordingly. CONCLUSIONS We conclude that EQHA’s organizational structure and management processes have contributed to its high performance and achievements thus far. Its focus on sustainability is likely to enable a thoughtful transition process. We also conclude that paying attention to the capacity of sub-national staff will help further enhance performance. The discussions we have had throughout helped set the context and challenges in the most appropriate language. -An EQHA sub-awardee 31 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS: PHB OVERVIEW OF PHB INTERVENTIONS PHB works in partnership with NCHP, and with SBC focal points in CENAT, NMCHC, the National Center for Malaria (CNM), the National Nutrition Program (NNP), and the Ministry of Rural Development (MRD). Objective 1: Strengthened public sector systems for oversight and coordination of SBC at the national and provincial levels. Under this objective, PHB works to improve coordination and joint planning for SBC among stakeholders, develop a sector-wide strategic framework, strengthen systems for monitoring quality, and increase RGC funding for SBC. PHB conducted a Vision of Perfect exercise, organizational network analysis (ONA), an Integrated Technical Organizational Capacity Assessment (ITOCA), and a visibility exercise with NCHP. It updated NCHP’s website, set up a digital library for SBC material, developed social and behavior change communication (SBCC) Practitioner Guidelines, and organized SBC forums. PHB works with national programs and MRD to develop SBCC products related to their thematic areas. Objective 2: Improved ability of individuals to adopt healthy behaviors. PHB works with NCHP and SBC focal points in NMCHC, CNM, development partners supporting SBCC, PHPUs, OD Health Promotion Units (HPUs) Health Center Management Committees (HCMCs), Commune Councils, and CCWCs to develop and implement interventions using its collaborative five-step process: co-diagnose, co-decide, co-design, co-deliver, and co-assess. Key behaviors targeted include those in the areas of health, nutrition, and WASH. PRESENTATION OF FINDINGS AND CONCLUSIONS The two objectives of PHB are to strengthen public sector systems for coordination and oversight of SBCC, and to conduct high-quality SBCC programming. These are inextricably linked and build on one another. Below we present findings and conclusions organized around EQs that pertain to PHB. We begin with EQ2 and present how the Activity could leverage its present experience to institutionalize its interventions. We then move to EQ6, which is in fact the same as Objective 1 of PHB, and here we present findings related to the role of NCHP in SBCC in the country at the national and sub-national levels. From there, we move to EQ3 to present challenges and mitigation measures that have not been covered in the preceding two EQs, and then to EQ4 to present findings and conclusions related to the management structure and processes of PHB and their impact on performance. Recommendations specific to PHB are included at the conclusion of this report. EQ2: HOW CAN EQHA AND PHB LEVERAGE THEIR RESPECTIVE EXPERIENCES TO INSTITUTIONALIZE THE ACTIVITIES’ INTERVENTIONS WITH THE LOCAL SYSTEM IN THE FUTURE? FINDINGS NCHP and a range of stakeholders have learned and internalized aspects of the SBCC design process they were engaged in and have found to be relevant to their work. The five￾step design process used and promoted by PHB to design, implement, and monitor SBCC interventions has been widely recognized amongst NCHP, other MoH programs, PHB sub-awardees, and partners engaged in SBC work. These stakeholders commended the participatory way PHB engages them. 32 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Partners credited PHB with engaging NCHP, PHDs, and partners in bringing in evidence-based approaches to the process of designing SBCC interventions. PHB documents and reports show that the five-step process is comprehensive and encompasses the design, implementation, and monitoring aspects of SBCC. Engagement of partners has been high in the first three of the five steps, which are about diagnosis, decisions, and designing. Partners recounted their learning from engaging these processes, and as SBCC practitioners, they are likely to continue to integrate and utilize the learning in their respective programs. Staff of NCHP, national programs, and partners emphasized that partner support to SBCC programming in the country has always been crucial for all sub-thematic areas in health, nutrition, and WASH, and will continue to remain so even as government funding for SBCC is unlikely to increase in the near term. Therefore, the capacity built in partner organizations is likely to remain and continue to be utilized to affect the local ecosystem for SBCC. Local sub-awardees report learning much from the latter two steps of the five-step process, those about delivering and monitoring the interventions. They, and PHDs, report their observation that the PHB’s SBCC products help engage populations creatively and this is also likely to be embedded in these organizations. NCHP has put to use the assessment and formative inquiry process that it learned from PHB in its recent work with other partners. This is strong evidence that the learning will remain as part of NCHP capacity after PHB’s term. PHB has revived the SBC Forum as part of NCHP’s strategic plan (2018-2022), and this has the potential to serve as a platform for the SBCC community and NCHP’s engagement with them. MoH officials and partners stated that PHB’s organizing the Forum meetings to share research findings and lessons were beneficial. PHB supports NCHP in hosting the meetings of this Forum, and thus provides technical and financial support. supports NCHP in hosting the meetings of this Forum, and thus provides technical and financial support. It has also drafted Forum guidelines, updating them from an earlier version. The Forum contributes to PHB’s overall objective to support a thriving SBCC community of practice in the country. Establishing this Forum at the national and sub-national levels is also part of NCHP’s strategic plan. A review of the draft guidelines showed that stakeholder tasks are clearly laid out, but it does not have a costed work plan and assigning of costs. At the formative stage of the Forum, NCHP had raised the issue of financial support after PHB. At another meeting with NCHP and national programs, participants recommended adding a budget to the guidelines that includes potential sources for continued financial support for the Forum, and also to have a clear plan for conducting the Forum at sub-national levels. PHB has begun working with PHPUs and OD HPUs for fundraising advocacy and for coordinating with vertical programs. NCHP is keen to build its own capacity to monitor and guide PHPUs. PHD and OD teams highlighted that PHPUs and OD HPUs do not have a sense of direction or work plans, and over time the resources available to them in the form of a clear workplan and budget have dwindled. This also has to do with the fact that vertical programs have individual SBCC tasks, and partner support to SBCC work at sub-national levels tends to be built into the program teams at PHD, rather than into the PHPU. As a result, vertical program teams at PHD are well-supported and self-sufficient, while PHPUs have tried to shift their focus to vertical initiatives of their own, such as tobacco and alcohol control. NCHP’s ability to support PHPUs is limited to a few visits related to Elephant, driver, and path…how to move the elephant forward. They [PHB] brought more attention among practitioners to engage its audience… They ensure engagement in the partners’ group. -Partner staff 33 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV tobacco control. Partners also noted that this is a catch-22 situation: the PHPUs need capacity to absorb and utilize resources but they need to mobilize resources for themselves to build their capacity. Under such a scenario, enhancing their capacity for small but effective actions will help build confidence for more engagement on the part of PHPUs and the supporting entities. In this regard, continued support from PHB to PHPUs to strengthen their SBCC capacity and to showcase their work in SBC Forums will be helpful in raising more funds. PHB has started engaging PHPUs to develop advocacy plans in order to improve funding allocation for their work, and have also helped improve collaboration between the PHPU and the maternal and child health team at the PHD in two provinces. The evaluation also notes that these activities are at an early stage and will need significantly increased support, especially support for PHPUs to advocate in a sustained manner for increased funding leading to better resourced work plans. As noted above, the vertical program teams have their own plans for SBCC, and hence it will require strong support to ensure that the PHPU works in close coordination with these program teams and does not duplicate the efforts of the latter. The evaluation also found that PHB has included PHPUs in training and workshops related to the five-step process for a range of behaviors that PHB works on. PHB has also financially supported their travel and per diems to engage the OD and community levels. Thus, PHB has started to build SBCC-related expertise and initial advocacy for increased funding for PHPUs. PHD teams and partners reported that the D&D context for the health function transfer can be taken advantage of, and that more resources poured into the PHPU and OD HPUs to raise their profile and resources and to gain the attention of partners as a potential resource for SBCC in the provinces. This is part of NCHP’s strategic plan as well. Documents and interviews with NCHP revealed that it is keen to take advantage of the monitoring tools (such as Kobo Toolbox™) introduced by PHB and be equipped to use them to track PHPU activities and support them; PHB plans to support NCHP further in this area. PHB has improved the technical, administrative, and organizational capacity of local sub￾awardees. IP teams, local sub-awardee staff, and PHD teams confirmed that PHB’s investments in the three local sub-awardees have led to enhanced skills and knowledge on SBCC and the health thematic areas. PHB built their monitoring capacity and organizational policies regarding ethics and gender. These organizations are directly responsible for the co-deliver step of the five-step process, which is to implement the SBCC activities in target communities. A local cadre called Interpersonal Communication Agents recruited by the local sub-awardees conducts these activities and enlists the support of VHSGs to mobilize participants for these activities. The local sub-awardees reported that PHB’s SBCC products and delivery methods are powerful and engaging, and that they will continue to use them after the end of the Activity. Earlier cycles of SBC programming did not achieve institutionalization through VHSGs and the local authority, and hence PHB was designed to work through local NGOs. All the same, VHSGs are widely recognized as a critical local system for SBC. MoH officials, USAID, the IP, and partners noted the lack of progress to date on a clear policy for sustaining and developing the VHSG cadre as a well-resourced community-based resource for mobilization and SBC. They also noted the lack of clarity on a “home” for the VHSGs under a particular Ministry. Partner initiatives that The other programs are clear but PHPU is not so clear. PHPU seems not to have clear indicators for evaluation [of their work]. If they have clear indicators, it may be better. -PHD team When PHB ends, the PHPU will not have budget to go to HCs; I have skills after attending the training but no budget to raise awareness and to visit HCs. -PHPU staff at PHD 34 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV engage VHSGs have been able to institutionalize their efforts partly under the commune councils, but the latter are elected bodies and need to be re-engaged at regular intervals. USAID-funded Empowering Communities for Health (2014-2018) aimed to strengthen linkages between VHSGs, commune councils, and HCs. A mid-term performance evaluation highlighted several challenges in working with VHSGs including the voluntary nature of their work, and low compensation leading to low motivation and high turnover.21 Due to these experiences, there was a deliberate decision for PHB to move away from working exclusively through VHSGs to deliver their SBCC interventions, but to work through staff of national NGOs, who are PHB’s local sub-awardees. These national NGOs enlist the support of VHSGs, as well as CCWCs and village chiefs, to mobilize target groups for SBCC activities. Partners, PHDs, local sub-awardees, and HC staff viewed the VHSGs as the most sustainable option available. Engaging VHSGs is almost always done through partner initiatives and channeled through local NGOs. Their work comes to an end when the partner initiative ends, although some of these initiatives have been able to get communes to allocate resources to support the VHSGs. Still, these groups are resources that can be utilized by later programs. Partners also cited other reasons that make it a challenge to engage VHSGs: many are older women and men with low literacy, and have high attrition, including migrating outside the community for employment. Women in the cadre are a good means of ensuring that other women in communities are engaged meaningfully, not only through their work but also through the course of daily interactions. The current World Bank Cambodia Nutrition Program utilizes commune councils and VHSGs to bridge the gap between HCs and communities to provide counseling. Under this initiative, NCHP works with NCDDS and NNP to train, deploy, and compensate VHSGs for delivering a package of activities focused on child nutrition and the first 1,000 days of life. The USAID-funded NOURISH project,22 which ended in 2019, trained VHSGs as “community change agents” to conduct home visits, conduct community dialogue, and support HC staff in conducting growth monitoring in communities. The former project pays VHSGs while the latter did not, but only compensated for travel and other expenses. One partner also recalled that NCHP is known for its strong engagement of VHSGs, for outlining their roles and responsibilities, for a monitoring and remuneration system as part of a strategic reform, and also for partnering with NCDDS for financing, as the “institutional home” of VHSGs moved from MoH to commune councils under D&D. These findings indicate that while relying on VHSGs alone has been shown to be an unwise strategy, not utilizing this cadre optimally is a missed opportunity to not only potentially increase coverage of PHB’s co-deliver activities but also to contribute to the collective understanding of what works with regard to funding of VHSGs through the new mechanism. PHB would not be able to pay VHSGs, and hence it will have to look for other mechanisms such as the commune investment plans for this and identify ways to maintain motivation and retention. Women and men in target locations reported higher levels of observance of handwashing now compared to before the time of PHB’s community-level demo activities. Participants in FGDs recalled messages from activities in WASH demo sites related to handwashing. They also reported that this message was reinforced through communication efforts related to COVID-19, carried out by the local sub-awardees. As a result, they reported wide acceptance of the practice. Some of PHB’s target locations have ethnic minority populations, and PHB has begun working with religious leaders of these communities to seeks ways to promote the key behaviors. Young women reported reluctance to share their perspectives on FP. Younger women in FGDs reported that social and cultural inhibitions made it a challenge for them to openly discuss their 21 Jansen, William, Deborah Thomas, Srey Mony, Pam Putney, Ros Bandeth, Nhu-An Tran, and Mao Bunsoth. “Midterm Evaluation of USAID Health Project and Implementation Activities in Cambodia,” n.d., 184. 22 USAID and Save the Children. Nourish Project Endline Survey Report 2019. 35 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV concerns, questions, and experiences with FP methods, as these meetings included men. This needs to be further explored to ascertain alternative means of conducting these meetings. CONCLUSIONS PHB’s work already involves leveraging its expertise and experience in SBCC to institutionalize best practices within the development community at the national and sub-national levels. Given that SBCC interventions are mostly driven by partner organizations, this is a key measure of progress. It has also begun to be embedded in NCHP. The SBC Forums have the potential to continue to serve as a platform for partners and NCHP to continue to exchange ideas and promote cohesiveness in the SBCC community and will help meet the NCHP’s strategic plan and PHB’s objectives. PHB’s support for PHPUs and OD HPUs to raise their profile and visibility is a strategic move. But this attention will need to be significantly ramped up during the remainder of the project in order to build capacity and resources. This will entail placing PHB staff in the Activity’s target provinces to lead and coordinate the effort closely and in a fine-tuned manner that would not be feasible for national level staff. These staff will also be able to provide closer support to the work of local sub-awardees. Strengthening PHB’s initial investments in NCHP’s information system and M&E capacity will enhance NCHP’s ability to monitor and support PHPUs and utilize their data. PHB has successfully built the capacity of local sub-awardees who will remain in the local system as a resource that can be utilized by partner initiatives in the future. However, PHB’s design decision to not engage VHSGs beyond mobilizing communities warrants revisiting. Current initiatives that successfully work with VHSGs (cited above) could provide key learning for PHB. The evaluation found evidence of changed behaviors in target communities. PHB’s engagement of ethnic minority populations will enable them to further understand determinants of the key behaviors in these communities and promote them. We conclude that PHB has successfully leveraged its strengths to institutionalize its interventions in local government and non-profit systems. There are several actions it could take to further enhance the likelihood that these interventions are sustained beyond its lifetime. EQ6: HOW EFFECTIVE HAVE PHB APPROACHES BEEN IN STRENGTHENING PUBLIC SECTOR SYSTEMS FOR SBC, AND WHAT MEASURES COULD BE CONSIDERED TO IMPROVE PERFORMANCE IN THIS AREA? As observed earlier, EQ6 re-states Objective 1 of PHB, which is to strengthen public sector systems for SBC. This section presents findings and conclusions related to the efforts taken by PHB in this area, and potential measures for scale up in the coming years. FINDINGS PHB’s phased strategy to build the capacity of NCHP to coordinate high-quality SBCC does not factor in NCHP’s current capacity and experience, and the resulting need to build in an active role for NCHP. A central premise of the two objectives of PHB is that the SBCC expertise of NCHP directly affects its ability to coordinate and lead SBCC across all programs. PHB aims to build the capacity of NCHP in SBCC through three phases: Learning (Years 1-3), Proficiency 36 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV (Year 4), and Mastery (Year 5). The PHB MEL plan describes the Learning Phase as “…characterized by PHB exposing NCHP to SBC global best practices; providing them with tools and resources to organize, convene, and lead the coordination and joint planning of SBC activities; storing and sharing information on their SBC Library; and training and taking NCHP through the five-step SBC implementation process to allow them to ‘learn by doing’.” The document also describes the Proficiency Phase as one in which NCHP would take a leadership role in SBC interventions; in the Mastery Phase, NCHP would drive the SBCC program planning assessment and partner management. This design of the Learning Phase points to a series of inputs from NCHP but does not take into account the experience of NCHP in SBC or its current expertise. Key informants observed that while NCHP may need refining of its SBCC approaches, it is far from a novice in the field. This was also noted in PHB documents. While PHB discussed the process with NCHP and identified its complementarity with NCHP’s eight-step process for behavior change communication, there is no evidence that PHB obtained NCHP’s input on the five-step design process before deploying it in order to draw on their experience and understanding of the context. While the five-step SBC process is practiced globally as an SBCC tool, its usefulness is enhanced by contextualizing it. In the case of PHB, it could have benefited from NCHP’s experience and understanding of the country's context. It would also have meant that the five steps did not have to be implemented to its full extent for each of the 12 key behaviors included in PHB’s scope of work. This is explored fully below. During implementation of the five steps, NCHP was engaged as observers, even as the Learning Phase intended it to be. Review of PHB reports and documents and interviews of NCHP, IP teams, and partners showed that this evolved over time, resulting from less than active participation from NCHP and from the design of Learning Phase in which NCHP were engaged as observers. It can be argued that creatively designing a more active/decisive role for NCHP in planning and implementing the five-step process could have elicited more consistent engagement from the NCHP team. Building a more active role for NCHP from the start, such as leading technical or planning sessions could have helped NCHP stay invested to a greater extent in the entire process. While NCHP approves the SBCC material produced, they believe they do not own it as their product or their campaign. PHB reported that it plans to revisit the three-phase strategy to build the capacity of NCHP and discuss ways to strengthen it, in consultation with NCHP. NCHP essentially saw PHB’s two objectives as contradicting each other. They believed the design and implementation of PHB did not give them an opportunity to “prove” the improvements in their capacity and do “real work,” as they put it. It was noted that NCHP clearly expected to play a more substantive role in the implementation of the five-step process. PHB has set up the necessary infrastructure for producing SBCC products, but the implementation of the co-design phase for PHB’s key behaviors was managed by the IP with a sub-awardee. NCHP has low staffing and budget. They prioritize urgent, on-budget activities. Partners creatively engage NCHP, including embedding SBCC experts. The ONA done at the start of PHB showed NCHP as a poorly networked entity. A 2010 evaluation of a co-financing project between the European Commission and UNICEF pointed to poor country ownership due to capacity constraints in NCHP, among other reasons.23 Partners who are on NCHP’s budget as well as those who work with NCHP off-budget reiterated the fact that NCHP is a small team stretched over a large mandate, with limited funding, and that leads to NCHP prioritizing initiatives that are on their budget, as fund disbursements are linked to completion of planned activities and are therefore urgent. NCHP itself recognizes this as a limitation in its engagement with PHB. It is also clear from meeting minutes and PHB documents that different staff members from NCHP participated in successive meetings with PHB on 23 Phase 2 Evaluation of the Paris Declaration. Cambodia Country Study Report, Final Version. December 2010 https://www.oecd.org/dac/evaluation/dcdndep/47082953.pdf (Accessed May 2021). 37 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV the same issue, and there was no handing over of relevant information or decisions made. This led to PHB having to retrace its steps even if they had been agreed upon with NCHP earlier. Partners who provide on-budget support to NCHP embed a senior SBCC expert in NCHP to provide day-to-day technical assistance in NCHP’s SBCC work. The role of PHB’s senior-level staff currently placed in NCHP is communication and external engagement and not providing SBCC technical assistance. Partners identify other specific needs of NCHP including skills for procuring specialized services such as drafting scopes of work and terms of reference and developing and deploying technology tools such as interactive voice response. The ITOCA exercise resulted in an action plan, but NCHP is not able to address all actions related to staffing and funding. NCHP’s SBCC capacity is one of the items in the action plan. PHB used ITOCA to assess NCHP’s organizational capacity, and several other assessments that flowed from it. A review of the ITOCA framework shows that it is a comprehensive, guided self￾assessment and is widely used to empower organizations to identify and address capacity gaps. PHB’s Cooperative Agreement notes that the support the Activity provided to the digital library and linking it to global SBC platforms was decided on with NCHP even prior to the award. In addition, minutes of the PHB-NCHP meeting held before the ITOCA was conducted show that NCHP wanted it to focus only on technical aspects of the organization, and listed their requirements of IT support, digital library, and strengthening M&E and production capacity of the team. PHB reports also state that NCHP was not in favor of addressing aspects like staffing and budget that are not within their scope to change. PHB reports show that in another meeting nearly two years later, in December 2020, NCHP had reiterated the need for PHB to focus on building SBCC-related capacity, both skills and infrastructure. The ITOCA exercise identified a range of gaps in NCHP and drafted an action plan to address them. Although the action plan contained in the ITOCA report does not assign responsibilities, PHB has subsequently begun to address all actions related to infrastructure and training. Building SBCC technical expertise is one of the areas included in the action plan. The items in the assessment will collectively contribute to overall capacity, but individual items are not weighted for their contribution to the overall capacity of NCHP, or the feasibility of improving them. Assessing NCHP capacity using ITOCA is a reporting requirement for PHB, and as such, the same tool is planned to be used to report progress in Objective 1 in the coming years. NCHP also has not yet had an opportunity to utilize the new equipment installed. NCHP highly values the infrastructure investments made by PHB and stated that they continue to anticipate a more active role in the five-step process, including production work, using the upgrades provided by PHB. While they recognize that they do not yet have the capacity to carry out the production themselves, they would like to start small and grow from there. Partners also recounted NCHP’s intentions to grow its expertise in developing automated phone￾based tools for SBCC. With the new equipment, we can do [production] but we also don’t have the capacity. The ideal way is that all production is done by NCHP. If our capacity is not big enough, just start with something small. -NCHP official 38 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV The SBCC Practitioner Guidelines offer expertise not available in the country, but stakeholders expressed the need for it to be simplified to reach a wider audience. PHB drafted the Guidelines to replace the Strategic Framework that was planned earlier. It takes forward similar work begun with the support of other partners in 2008. The Guidelines were finalized after several rounds of consultations with NCHP, national programs, and partners. They are intended to provide NCHP with an updated resource for all stakeholders engaging in SBC that will facilitate NCHP’s efforts to promote effective, evidence-based, and high-quality SBC activities in Cambodia, as well as serve sub￾national audiences. The content includes a range of behavior change theories and theory-based SBC design development, and provides links to web-based resources. Meeting minutes show that during the initial stages of development, NCHP requested that the Guidelines be “simple and practical and serve as a checklist for NCHP to assess internal capacity and partner capacity.” NCHP had also raised concerns about the ability of national and sub-national SBC practitioners to understand and apply the theories and recommended that case studies illustrating the application of the theories be included. Other feedback included the addition of templates and examples on segmenting audiences, crafting messages, and developing indicators. Partners reported that they are excited that the Guidelines bring in a new lens and expertise to SBC but emphasized the need to ensure that many can implement them. NCHP’s coordinating role for SBCC across all programs and the constraint of SBCC being fragmented across vertical programs needs further understanding. Each national program has its own SBCC team and has procured external technical assistance to design and implement SBCC initiatives. Even so, there is still concern that SBCC efforts in health, nutrition, and WASH are fragmented and led by vertical programs, with varying quality of programs and little investment in research. The evaluation found that SBCC work in thematic areas such as malaria and WASH have thriving, high-quality programming that engages community-based structures and is responsive to local epidemiology, but there is insufficient formative research to guide the design and approach. These initiatives are led by the relevant ministries, have financial and technical support from partner initiatives, and engage other national programs or ministries as needed, which makes it appear that these programs have identified and procured SBCC expertise based on their own need. In such a context, NCHP has to be able to position itself as a compelling alternative for high-quality, locally relevant SBCC expertise. NCHP understands its role to make sure that SBCC material and tools are up to a standard, are culturally appropriate and effective, and harmonize support from multiple sources. CONCLUSIONS This evaluation has been able to validate the central premise of PHB that the SBCC expertise of NCHP directly affects its ability to coordinate and lead SBCC across all programs. PHB’s three-phase strategy to improve NCHP’s SBCC capacity (Learning, Proficiency, and Mastery) does not appear to consider NCHP’s current skills and experience. At the same time, the third stage of Mastery, with NCHP leading SBCC sector-wide at the end of Year 5, is ambitious and unrealistic. NCHP is a small team with a large mandate, and therefore they prioritize urgent on-budget activities; those not in NCHP’s budget need to incentivize its engagement creatively and by addressing felt needs, aiming for quick wins early on. Initiatives that have successfully engaged NCHP embedded high-level technical support to facilitate coordination of input from national programs and NCHP. Embedding an It is possible that they [sub-national managers] never learned SBC theories…SBC is not an easy concept; so, when we put so many theories, it is difficult for those who have never known about them. -NCHP official Please simplify; keep it short and simple. -Minutes of consultative workshop, Jan 2021 39 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV SBCC expert in NCHP would provide greater access to NCHP to his or her expertise and also expedite coordination with other national programs. However, this step has the potential for creating dependency on the seconded staff, given the small staff size of NCHP. NCHP’s leadership in decisions in the co-diagnose and co-decide steps of the five-step process has been patchy. There is a lack of continuity in NCHP involvement, leading to inefficiencies in PHB’s activities. A carefully calibrated approach of building a more active and substantive role for NCHP in upcoming activities is likely to increase NCHP’s SBCC capacity and their engagement with PHB. The calibration would take into account the time available for PHB to reach its targets, the restrictions in place for COVID-19, and other considerations. These activities include the co-deliver phase for TB, FP, maternal and child health (MCH), and WASH, the co-design and co-deliver phases for malaria, and the perception of quality of healthcare services. This would require commitment from NCHP to dedicate staff time for this purpose. While PHB will have to conduct ITOCA to be able to report on progress in Objective 1, it is unlikely to add value. Given the other findings related to building the capacity of NCHP in SBCC, it would be important for PHB to collaboratively assess NCHP’s skill gaps around SBCC design, delivery, and M&E that require prioritized attention, and measure them for progress and ensure that these are specifically addressed in the next ITOCA, as these are expressed capacity needs. The infrastructure capacity that PHB built in NCHP is a necessary long-term investment but not sufficient to connect with and coordinate SBCC interventions across programs. Given NCHP’s high level of interest in being able to utilize them, PHB will do well to enhance its support to NCHP in order to enable them to utilize the equipment to design and produce select SBCC materials. As with participation in co-deliver activities, this, too, will require commitment from NCHP to dedicate staff time for this purpose. The SBCC Practitioner Guidelines have the potential to be the legacy of PHB’s SBCC work in Cambodia. The requests to keep the document simple and user-friendly need to be weighed against the need to have sufficient nuance and complexity to aid understanding and use. With existing constraints in staffing and budget, NCHP is likely to require strong technical and financial support beyond PHB, and is unlikely to reach the Mastery Phase, driving SBCC planning and assessment across programs and partners. We conclude that PHB’s approaches have been partly effective in strengthening public sector systems for SBCC, and they can be further strengthened by implementing the recommendations we outline below. EQ3. WHAT ARE KEY CHALLENGES IN ACHIEVING THE EXPECTED RESULTS OF EQHA AND PHB AND HOW TO OVERCOME THE CHALLENGES? FINDINGS PHB has seen considerable delays in formative activities, leading to reduced time available for co-deliver work. Partners and the MoH stated that the design and implementation of formative assessments and the five-step process have been too linear and taken much longer than anticipated. The first three steps of the five-step process for some behaviors took as long as two years. Changes in key staff positions within PHB compounded this challenge. The process of approvals for activities such as workshops and signoffs on draft products from national programs and NCHP is long-winded, and this 40 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV challenge is common to all partners. This process has been made more stringent with restrictions related to COVID-19 in place. These delays have left PHB with reduced time for implementing field-level SBCC interventions, or the co-deliver phase. There is a need to move to the co-deliver phase as early as possible for the remaining behaviors, and not move through all the phases. The intensity of SBCC interventions appears to be lower than what is typically required for the behavior change outcomes that PHB targets. As described under EQ2, PHB made a strategic choice to work through national organizations and their local cadre of staff. Each of them covers up to three HC areas, and directly conducts SBCC activities related to co-deliver work for all key behaviors. They enlist the assistance of VHSGs only to mobilize target groups for events. With such large geographic areas to cover, the frequency of their contact with target populations and the intensity of the work is likely to be low. Coupled with the delays described earlier, the co-deliver work on the ground may not be sufficient to bring about change in the key behaviors and move the needle on these outcomes. Some of the co-deliver activities have not yet been started. Restrictions brought on by COVID-19 have compounded these constraints many times over. PHB reported that the sample size for the midline and endline surveys would be enough to demonstrate changes in the outcomes in specific provinces and ODs. Partners who were members of PHB’s technical consultative group were not aware of the fact that PHB intends to improve the population-level coverage of key behaviors. There was a misalignment in the perceptions of PHB vis-à-vis NCHP and national programs in several areas. As described under EQ2 and EQ6, PHB’s strategy to build NCHP’s capacity for SBCC did not reflect NCHP’s current capacity or an attempt to understand it. The content of the SBCC Practitioner Guidelines, and the extent of engagement of NCHP in the five-step process were not fully aligned with priorities and needs NCHP expressed. Officials of national programs requested that PHB provide regular updates during the co-deliver phase as well, even if interventions are delayed. Virtual platforms for communication in the context of COVID-19 have made communication even more challenging, although government stakeholders are getting used to this “new normal.” PHB is unable to track or influence funding for SBCC at the national level, which is one of its stated results. This challenge is brought on by NCHP’s reluctance to share its annual budget with partners who do not have on-budget activities, and also their reluctance to advocate for additional funding. CONCLUSIONS Delays in initial phases of the five-step design process have limited the time available for co-deliver work. With COVID-19-related restrictions likely to continue, the kind of high-frequency contacts required to increase the coverage of key behaviors becomes even more challenging. PHB could consider a process to review and revise down its targets for population-level outcomes related to key behaviors. This challenge is also likely to be mitigated if PHB moves forward with building the capacity of VHSGs to assist the local cadre to implement co-deliver activities and increase the reach into target communities. Misaligned perceptions have considerably affected the direction of interventions in PHB. This challenge could be mitigated by more active listening to stakeholder concerns and being willing to make fundamental changes to the design and approach of PHB, including staffing changes such as province-level staff to coordinate the work with PHPUs, OD HPUs, vertical program staff, and local authorities, and embedding the SBCC expert in NCHP. 41 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV The expected result to improve funding at the national level is unlikely to be reached. This challenge could be mitigated by enhancing PHB’s work at sub-national levels to improve the profile and visibility of PHPUs and facilitating their advocacy work. EQ4. HOW HAVE THE EQHA’S AND PHB’S STRUCTURE, MANAGEMENT, AND INTERNAL PROCEDURES AND SUB-AWARDEES’ MANAGEMENT AFFECTED IMPLEMENTATION AND OUTCOMES, POSITIVELY AND/OR NEGATIVELY? FINDINGS PHB’s leadership is highly consultative, and work processes and sub-awardee management are collaborative. IP teams and staff of sub-awardee organizations stated that the core values of their respective organizations guide the choice of intervention strategies and also influence their day-to-day interactions. Sub-awardees stated that the IP sets clear expectations and communicates well, and that they develop work plans and budgets collaboratively. They also stated that the senior management team is approachable, highly communicative, and flexible. Initial workflow issues between IP and sub-awardees have been sorted out through open communication. The IP encourages the participation and leadership of female staff members on all teams. It also organizes regular round tables and panel discussions with sub-awardees and has also been careful to bring in and build the right mix of technical expertise in the PHB team. Sub￾awardees also considered PHB’s M&E system robust, providing short-looped feedback to improve implementation. PHB has been intentional and successful in building the capacity of local sub-awardees in a wide range of areas. Staff of local sub-awardee organizations and IP teams stated that sub-awardees are supported, coached, and mentored in technical, financial, administrative, and organizational development areas. The sub-awardees stated that they are being adequately supported to conduct co￾deliver interventions and that their interactions with the core team of PHB have built confidence and mutual trust and respect. Such an environment empowers them to make decisions. Cross learning among sub-awardees is being planned. Coordination with multiple stakeholders has been particularly challenging for the IP. Delays in TA input from PSI regional and global offices and the process of recruitment of local sub-awardees have added to overall delays and constrained performance, even as the IP and sub-awardees continued to maintain communication and were flexible to adapt to the delays. The IP manages all relationships with government stakeholders, as it rightly should, but there have been continual tensions between managing these delays and their implications on sub-awardee budgets. Sub￾awardees also pointed out that there had been challenges with the IP’s procurement team, which could be managed better. CONCLUSIONS We conclude that PHB’s internal structure and management processes have positively affected its performance, despite significant challenges. PHB has empowered local sub-awardees through its management approach and intentional and prioritized capacity building measures. PHB has given us power to make decisions. They do not put pressure on us. They work with the intention to support rather than find mistakes. -Key staff of sub-awardee They [the IP] try really hard to manage all these partnerships, especially signoffs from national programs. They are often caught in the middle. -Key staff of sub-awardee 42 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV PHB has experienced significant delays that are partly internal and partly external, which have strained even the most communicative and agile work processes. They have navigated through most of them, as most pertain to stages in PHB’s interventions that are already completed. Implementing the changes recommended below is likely to further reduce delays and enhance performance. 43 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS: EQHA AND PHB EQ1: HOW COULD EQHA AND PHB EFFECTIVELY LEVERAGE AND BUILD SYNERGY WITH EACH OTHER? In this section, we present findings related to coordination and synergy between EQHA and PHB, and the potential for the coming years. FINDINGS Avenues for collaboration are already being explored and utilized. EQHA and PHB reports, minutes of coordination meetings, and IP teams of EQHA and PHB reported the range of opportunities that both teams have identified and utilized to improve efficiency and reach in their respective activities. Both work in the same provinces, and PHB’s demo sites are in the same communes where EQHA works with HCs. PHB identifies facility clients to participate in EQHA’s workshops. In addition to these ongoing initiatives, both activities plan for more detailed collaboration at these sites for referral of TB symptomatics from pharmacies to HCs. EQHA and PHB are likely to benefit by adding a learning component to the ongoing collaboration efforts. A review of the interventions of both activities and the related processes showed that they will benefit from a focus on learning by observing the other’s interventions in person, rather than a presentation of what was learned. Examples are PHB teams observing facility-level QI meetings and the QIC meeting, and EQHA teams observing activities at demo sites. The senior management team of each Activity will benefit from conducting these observations with a view to gleaning lessons to apply in their own work. Additionally, sharing what did not work and what was harder than expected would also be helpful for the other team. An example is the challenge that the EQHA preservice team faced with normalizing the use of web references by TWG members. There are opportunities to collaborate with other organizations. Both EQHA and PHB collaborate with a wide range of government and partner organizations. Collaborating with the Implementation of the Social Accountability Framework-II is likely to benefit the provider behavior change intervention that is currently underway in two provinces. CONCLUSIONS We conclude that EQHA and PHB have already leveraged each other’s expertise and field presence to increase efficiency and build synergy. We also conclude that this synergy can be further enhanced by identifying opportunities to learn from each other’s best practices and challenges. There are opportunities to expand collaboration with other stakeholders as well. 44 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV RECOMMENDATIONS: EQHA It is pertinent to note that EQHA has begun planning for scale up and transition of many of its workstreams, particularly QI. For EQHA to further leverage its expertise and experience to institutionalize its interventions and to reach its objective of engaging and empowering national and provincial leadership and public and private healthcare managers and providers to collaboratively improve the quality and safety of health services, strengthen systems, and increase service utilization, we recommend the following: 1. EQHA leverage its position as the thought leader and lead collaborator in the country for health systems strengthening, provide technical expertise under the leadership of the QAO of DHS, MoH to convene all stakeholders engaged in facility-based QI work in the country to develop a harmonized QI model that combines features from all the models that are appropriate for the country’s health systems context. Such a harmonized model stands a better chance at being accepted than any one model, for nationwide scale up through joint advocacy, and later for integration into the other workstreams of EQHA. As this effort is likely to take time to shape up, this activity must be prioritized in the remaining Activity timeframe to be effective 2. EQHA provide continued technical leadership to lead partners, to approach MoH to attain clarity and consensus on the processes and obtain the approvals related to the endorsement and rollout of the minimum standards for licensing of private facilities and also lead the advocacy with the MoH for the required approvals. 3. EQHA intensify and customize support to HPCs – CCN, CMC and PCC – to identify and address specific challenges to develop them into high-performing and self-supporting entities, through a collaborative process with other partners and MoH. These challenges include, but are not limited to advocacy for more funds to implement the strategic plans and continued technical support for registration. EQHA should also facilitate the process of collaboratively identifying and determining replicable success factors in engaging with the PCC that have aided its successful engagement of its members and apply the lessons to CCN and CMC. The aim of this set of actions is to strengthen these HPCs’ nationwide reach and effectiveness, which is also critical for EQHA’s private sector and workplace interventions. 4. For those activities across all its workstreams that do not address system-wide constraints and that are not feasible to expand, EQHA to identify ways to connect them to broader impacts on the system. Notable among these interventions are: supporting the provincial TWG on GBV and HIV, facility-level HIV-NCD integration, FP interventions in private facilities, and infirmary-level interventions in factories including outreach camps and referral linkages for individual infirmaries. 5. EQHA protect and advance the gains made in the nursing curriculum as well as the change brought about in the collective mindset toward CBE by reaching out to partner programs and offering expertise in CBE to collaboratively transform the curricula of other health professions. 6. EQHA revisit its performance indicators and include those that more directly capture the breadth and complexity of the work that is being accomplished, including composite weighted indicators (as illustrated in Annex 1) and yes/no indicators. 7. EQHA coordinate with PHB to learn from observing each other’s key activities especially the QIC work of EQHA and the community-level demonstration activities of PHB. 8. EQHA continue to support the adaptation of relevant accreditation standards for institutionalizing in the next version of NQEMT, while working with HEQIP, GIZ and MOH to 45 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV advocate for the new law adoption, establishment of the accreditation organization and rollout on a voluntary basis after approval. 46 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV RECOMMENDATIONS: PHB To further leverage its expertise and experience of the past two years, and to effectively strengthen public sector systems for SBCC as well as to effect behavior change in target populations, we recommend the following: 1. PHB build into its strategies and work plans a more active and substantive role for NCHP in the co-deliver phase for TB, FP, MCH, and WASH, in the co-design and co-deliver phases for malaria, and the perception of quality of healthcare services. The effort put into this change would be calibrated against the time available for PHB to reach its targets and the restrictions in place for COVID-19. 2. PHB to maximally utilize the time remaining by expediting the five-step process for the remaining behaviors and moving to the co-deliver phase. 3. PHB collaborate with NCHP to develop a plan to assist them in utilizing the equipment that PHB has provided, to develop SBCC products for a few thematic areas that are mutually agreed upon. 4. PHB consult with USAID, NCHP, and sub-awardees, in the light of the findings of this evaluation, to redefine the role of the SBCC expert in its team, to be embedded in NCHP as a coach, for incrementally higher involvement of NCHP in the remaining activities of the five-step process for select behaviors. 5. PHB collaborate with NCHP, PHDs, and ODs to develop a plan to train and coach its staff in using Kobo Toolbox™ and set up a system for NCHP to track and support SBCC-related indicators from sub-national levels. 6. PHB work with the review team (comprising of partners and NCHP) for the SBCC Practitioner Guidelines to revise/reduce the range of theories, incorporate case studies and descriptions of deploying the theories in developing designs, and develop templates and resources for themes such as audience segmentation, crafting messages, and identifying indicators. PHB should also consider recasting the entire document to target sub-national audiences and the requirements of NCHP, to serve as a tool to assess internal NCHP and partner capacity. 7. PHB work with USAID and NCHP to include in ITOCA the tracking of the above interventions in order to strengthen the capacity of NCHP to design and deliver SBCC interventions. 8. PHB engage NCHP and partners to develop a costed plan for the SBC Forum and the terms for a rotating secretariat and incorporate them in the Forum Guidelines. PHB should also identify those aspects of the five-step SBCC design process that have gained traction among partners, and conduct capacity building events through the Forum on those thematic areas. 9. PHB ramp up its support to PHPUs to develop their work plans, advocate for funding at the province level, collaborate with vertical program staff, and monitor co-deliver activities at demo sites. PHB should consider placing staff in some of its target provinces to oversee this work. 10. PHB provide opportunities for sharing and cross-learning amongst local sub-awardees. 11. PHB coordinate with EQHA to observe each other’s key activities and glean lessons that can be applied in their own work. 12. PHB continue to actively engage ethnic minority populations in their target provinces in order to understand specific factors that influence their adoption of the key behaviors that PHB promotes. 13. USAID and PHB review PHB’s approach to implementing co-deliver interventions at demo sites, consider a substantive role for VHSGs as a cadre working under the direction and support of the local staff of local sub-awardees, and work with commune councils to identify ways to compensate them and maintain motivation and retention. 47 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 1: ADDITIONAL ASSESSMENTS OF EQHA ASSESSMENT OF EQHA INTERVENTIONS As noted in the main report, the table below assesses a selection of EQHA interventions against criteria outlined in the WHO HSS framework, along with the feasibility of scaling up the intervention in the light of the overall level of effort of EQHA. The yes/no criteria we employ below is an oversimplification of complex interplays between these interventions and the health system. EQHA INTERVENTIONS CROSS CUTTING BEYOND ONE VERTICAL ADDRESSES SYSTEM-WIDE CONSTRAINTS POTENTIAL FOR SYSTEM IMPACT BEYOND EQHA TERM TAILORED TO CONTEXT; ROLE FOR COUNTRY INSTITUTIONS SCALABLE BY EQHA, IN LIGHT OF OVERALL EFFORT NATIONAL PROGRAMS AND POLICIES Updating TB guidelines and SOPs No No Yes Yes No HIV-NCD integration Yes No Yes Yes No Clinical vignettes for NQEM Yes Yes Yes Yes Yes IT support to QAO Yes Yes Yes Yes Yes GBV TWG at province Yes Yes Yes Yes No WORKPLACE HEALTH Reviving WHWG Yes Yes Yes Yes Yes Referral linkages for infirmary Yes No No Yes No HPC-led assessments Yes Yes Yes Yes Yes Outreach camps at infirmaries No No No Yes No SUPPORT TO PRIVATE FACILITIES AND PRIVATE PROVIDERS Strengthening HPCs in provinces Yes Yes Yes Yes Yes FP training and service provision in private facilities Yes No Yes Yes No Including private providers in QICs; facilitating agreements with PHDs Yes Yes Yes Yes Yes 48 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV PROPOSED COMPOSITE INDICATORS: AN ILLUSTRATION An illustrative option for developing a composite indicator for the intervention of strengthening CCN would be as outlined in the table below. The components given in the table are also illustrative, based on the draft strategic plan of CCN for 2021-25 and the findings from this evaluation on what EQHA needs to support. Each component in the composite indicator is given a score on an equidistant scale of 0 to 2. The score obtained is converted into a percentage of the ideal score and then the weights are added to give the result. Benchmarks/targets for these indicators would have to be carefully set based on the progress expected in each component over time and the final result would show the overall progress in the composite indicator. This is most useful when there are several such composite indicators, and as all of them are weighted, there would be a general consistency across all of them in interpreting, say, what it means to go from 2.5 to 2.7. COMPONENT SCORE OBTAINED FOR REPORTING PERIOD [A] 0 = NOT DONE; 1= PARTLY DONE; 2 – FULLY DONE WEIGHTS ASSIGNED [B] UNWEIGHTED RESULT [C] =[A]/2 *100 (2 BEING THE MAX SCORE PER ITEM) WEIGHTED RESULT FOR THE REPORTING PERIOD [D] = [C]*[B] RMS is used on a regular basis and issues are resolved in [x] days 0.18 CCN supported to conduct training and promotional material to six / all provinces, including follow up of CPD for registered providers 0.15 CCN supported to include infirmary workers in [x] factories per province 0.16 CCN supported to conduct six-monthly follow up meetings with registered providers 0.12 CCN trained in organizational development and financial management 0.13 CCN supported to engage potential CPD providers national and in provinces 0.14 CCN supported to assess CPD providers and monitor their performance 0.12 Total 1.00 49 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 2: EVALUATION DESIGN AND METHODS EVALUATION QUESTION MAIN THEMES POTENTIAL QUESTIONS/INQUIRY DATA COLLECTION METHOD(S) DATA SOURCE/TYPE OF RESPONDENT DATA ANALYSIS METHOD EQ1: How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub- partners, other USAID activities, and other key partners in order to achieve desired health outcomes. 1.a. Learning from past and present efforts What efforts have EQHA and PHB made in the past? What was the learning from these efforts? What factors influenced their effect? EQHA & PHB Document review KII FGD EQHA and PHB EQHA and PHB CA, workplans, annual and special reports Literature EQHA teams; PHB team – national and sub-national USAID/Cambodia MoH: DG Health, NCHP Provinces: PHD Team OD Office HC Chief VHSG Populations: Mothers, WRA, Male partners of WRA Thematic analysis of qualitative data using pre-determined and emerging sub-themes and categories; triangulate from multiple sources; map findings on a matrix 1.b. Understanding opportunities and leverage points for synergy and coordination What opportunities exist in formative and design of EQHA and PHB, in their current approaches and strategies and in any formative work they could undertake together in the future? What opportunities exist in the implementation of EQHA and PHB activities, at the national, provincial, and OD levels? What opportunities exist in the evaluative and learning efforts of EQHA and PHB? What common stakeholders and meeting points are there in all of these phases? What are end users’ perspectives on convergence of efforts to improve supply and demand for health services? 1.c. Adaptation and readiness for change What adaptations do EQHA and PHB need to make to actively pursue these opportunities? What challenges and conflicts do they need to preemptively address? What is their readiness for change? 1.d. Metrics and learning What metrics of success should be considered by EQHA and PHB for synergy and coordinated functioning? What learning mechanisms are in place or should be put in place? 50 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQ2: How can EQHA and PHB leverage their respective experiences to institutionalize the activities’ interventions with the local system in the future? 2.a. Learning from past and present effort What evidence exists for national and provincial capabilities being built, and in what areas in EQHA and PHB? What commitment exists in these systems to continue to deploy these capabilities? EQHA & PHB Document review KII FGD EQHA EQHA CA, workplans, annual and special reports EQHA teams – national and sub- national MoH: DG Health, DPHI, DHS, QAO, HRDD MoLVT: Dept of Occupational Health MRD: Dept of Rural Health HPC Coordination Committee, MCC, CCN, PCC, CMC TWGs: Private sector, QI RTCs, Provincial RH OD Office, District RH Partners: WB – H-EQIP, GIZ, UNFPA EQHA and PHB NMCHC, NCHADS, CENAT TWGs: GMAG USAID/Cambodia Provinces: PHD Team, D&D focal point HC Chief PHB PHB CA, workplans, annual and special reports PHB Team – national and sub- national NCHP, CNM, Dept of Rural Health, MRD TWG: SBC Thematic analysis of qualitative data using pre-determined and emerging sub-themes and categories; triangulate from multiple sources; analysis by EQHA/PHB objective 2.b. Strengths and advantages To what extent did EQHA and PHB draw from their own formative work and national data and national priorities to design their interventions? What strengths and advantages exist in EQHA and PHB strategies, intervention mix, and implementation approaches? What weaknesses exist that hinder institutionalizing? What evidence exists for an effective approach or mechanism among stakeholders in order to build synergy? 2.c. Impact of D&D What has the impact of D&D been at PHD and OD levels on past and present attempts to institutionalize the interventions of EQHA and PHB? What steps have been taken to address barriers and utilize enablers caused by D&D? 51 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 2.d. External evidence What evidence exists in other ministries and interventions in Cambodia? What evidence can be brought in from other similar contexts? Partners: WB-Cambodia Nutrition Program, iDE, Marie Stopes Int’l 2.e. Metrics and learning What metrics of success should be considered by EQHA and PHB for successful institutionalizing? What learning mechanisms are in place or should be put in place? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? 3.a. Challenges by objective What challenges did/does EQHA face (by objective)? What challenges did/does PHB face (by objective)? What common characteristics do these challenges have across objectives and across EQHA and PHB? Challenges of the past and present and those anticipated. Challenges intrinsic to EQHA and PHB (cross-reference EQ4) and extrinsic, such as D&D and other systemic issues. EQHA & PHB Document review KII FGD EQHA EQHA CA, workplans, annual and special reports EQHA teams – national and sub- national MoH: DPHI, DHS, QAO, HRDD MoLVT: Dept of Occupational Health MRD: Dept of Rural Health HPC Coordination Committee, MCC, CCN, PCC, CMC Academic institution Private hospitals TWGs: Private Sector, QI RTCs, provincial RH, private facilities Thematic analysis of qualitative data using pre-determined and emerging sub-themes and categories; triangulate from multiple sources; map findings on a matrix 3.b. Scope of the challenges What are the internal and external perspectives about how easy or difficult it would be for these challenges to be overcome? 52 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 3.c. Steps taken so far What steps have EQHA and PHB taken to address the challenges identified and anticipated? How effective have these steps been? What lessons have been learned? OD Office, District RH Partners: WB – H-EQIP, GIZ, UNFPA EQHA and PHB NMCHC, NCHADS, CENAT TWGs: GMAG USAID/Cambodia Provinces: PHD Team, D&D focal point, HC Chief PHB PHB CA, workplans, annual and special reports PHB Team – national and sub- national NCHP, CNM TWG: SBC Partners: WB-Cambodia Nutrition Program, iDE, Marie Stopes Int’l EQ4. How have the EQHA’s and PHB’s structure, management, and internal procedures and sub- awardees management affected implementation and outcomes, positively and/or negatively? 4.a. Organizational features What values and ethos of the IP and sub- awardee organizations have enabled or hindered performance? Are team structures and job roles justified and enabling high functionality in teams? Is the technical expertise on all teams (IP and sub-awardees) relevant and sufficient? Are there gender and equity issues that are affecting performance and coordination with stakeholders? EQHA & PHB Document review KII EQHA and PHB EQHA and PHB CA, workplans, annual and special reports, organograms, job descriptions, management meeting reports, IP visit reports to sub-awardee areas IP and sub-awardee staff (national and sub-national) Thematic analysis of qualitative data using pre-determined and emerging sub-themes and categories; triangulate from multiple sources 53 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 4.b. Project management procedures How were sub-awardees selected? What support does the IP provide the sub- awardee in managing implementation? What steps are in place to ensure transparency and efficiency? USAID/Cambodia EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? 5.a. Measures of effectiveness and quality What measures of effectiveness (of engaging the private sector) does EQHA employ? What are the perspectives of private hospitals and providers on quality of services for better health outcomes? What are the perspectives of EQHA and public players on the challenges in the quality of services in the private sector? EQHA Document review KII EQHA EQHA CA, workplans, annual and special reports EQHA teams – national and sub- national USAID/Cambodia MoH: DPHI, DHS MoLVT – Dept of Occupational Health HPC Coordination Committee, MCC, CCN, PCC, CMC Private hospitals and facilities TWGs: Private sector Partners: GIZ Provinces: D&D focal point, PHD team, private facilities Thematic analysis of qualitative data using pre-determined and emerging sub-themes and categories; triangulate from multiple sources; analysis by EQHA objective 5.b. Effectiveness of EQHA interventions for the private sector What are the perspectives of public and private players on the role of EQHA in linking the public and private sector? What are the perspectives of the private players on the effectiveness of referral linkages? What are the successes, challenges, and lessons from EQHA’s support to the accreditation process, and its engagement of private academic institutions? What is the level of ownership and participation from the private players and how can this be further improved? 5.c. Incentivizing private sector involvement What incentives exist for the private hospital or provider to engage with the public sector, and what is the best value proposition for them? What disincentives exist? How can EQHA address these effectively? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? 6.a. Measures of success How does PHB measure the success of its approaches to strengthen public systems for SBC? EQHA & PHB Document review KII FGD PHB CA, workplans, annual and special reports PHB Team – national and sub- national MoH: Director General Health, NCHP, NMCHC, CENAT, CNM MRD: Dept of Rural Health TWG: SBC, GMAG Partners: WB-Cambodia Nutrition Program, iDE, Marie Stopes Int’l Thematic analysis of qualitative data using pre-determined and emerging sub-themes and categories; triangulate from multiple sources; analysis by PHB objective 6.b. Effectiveness of PHB interventions To what extent have the advocacy approaches to increase RGC funding for SBC been effective, both nationally and sub-nationally? What pragmatic approach could be considered to ensure the sustainability of the funding? What potential challenges are anticipated to 54 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV increase the visibility of SBC in health programs? Provinces: PHD team OD Office Communes: Chief + CCWC HC Chief VHSG Populations: Mothers, WRA, Male partners of WRA 6.c. Lessons learned and knowledge transfer to institutions To what extent have the findings from formative work of PHB utilized and carried forward? How does PHB adapt and use evidence (both global and national)? To what extent is NCHP able to determine best practices? How does the five-step design approach of PHB help NCHP and what is its uptake? What institutional, economic, and sociocultural factors enable or hinder this adaptation? What can be done to maximize their uptake? Have the critical assumptions of PHB been re-assessed? What can be done to ensure that leading and coordinating role of NCHP remains functional? 6.d. Adaptations in PHB and readiness What adaptations should PHB make in order to improve uptake by NCHP? What/which approach could be justified to maximize outcomes of PHB, and why? 55 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 3: GANTT CHART FEBRUARY MARCH APRIL MAY JUNE WEEK START 1 8 15 22 1 8 15 22 3/2 9 5 12 19 26 3 10 17 24 5/3 1 7 14 21 WEEK END 5 12 19 26 5 12 19 26 4/2 9 16 23 30 7 14 21 28 6/4 11 17 28 WEEK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Desk Review/Inbrief Kickoff Meeting 9- Feb Drafting of Evaluation Work Plan & Design Report 23-Feb Submission of Draft Eval Work Plan to USAID 23- Feb USAID First Round Review USAID Comments Received 4- Ma r Inbrief PPT Drafting Submission of Inbrief PPT to USAID 8- Mar 56 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FEBRUARY MARCH APRIL MAY JUNE WEEK START 1 8 15 22 1 8 15 22 3/2 9 5 12 19 26 3 10 17 24 5/3 1 7 14 21 WEEK END 5 12 19 26 5 12 19 26 4/2 9 16 23 30 7 14 21 28 6/4 11 17 28 WEEK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 USAID/Cambodia Inbrief 9- Mar Evaluation Work Plan Revisions Submission of Final Evaluation Work Plan 11- Mar USAID/Cambodia Approval of Final Evaluation Work Plan 17- Mar Data Collection Data Collection Prep Data Collection Begins 17- Mar Data Collection Ends 21-Apr Exit Briefing/Final Evaluation Report 57 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FEBRUARY MARCH APRIL MAY JUNE WEEK START 1 8 15 22 1 8 15 22 3/2 9 5 12 19 26 3 10 17 24 5/3 1 7 14 21 WEEK END 5 12 19 26 5 12 19 26 4/2 9 16 23 30 7 14 21 28 6/4 11 17 28 WEEK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Data Analysis Drafting of Evaluation Report FCR Matrix Development Submission of PPT to USAID 25- Apr Exit Briefing - USAID 26- Apr Exit Briefing - IPs 28- 29- Apr Submission of Draft Evaluation Report to USAID 19- Ma y USAID Review of Report 58 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FEBRUARY MARCH APRIL MAY JUNE WEEK START 1 8 15 22 1 8 15 22 3/2 9 5 12 19 26 3 10 17 24 5/3 1 7 14 21 WEEK END 5 12 19 26 5 12 19 26 4/2 9 16 23 30 7 14 21 28 6/4 11 17 28 WEEK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Comments Received from USAID/Cambodia 2- Jun Revisions to Final Evaluation Report Submission of Final Evaluation 16- Jun Note: All deliverable and report submission dates (data analysis, draft final, final) are in EOD/COB Eastern Time. 59 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 4: DATA COLLECTION AND ANALYSIS TOOLS KEY INFORMANT INTERVIEW CONSENT FORM – Public and private functionaries, Partners (TOTAL 74 KIIs) Title: EQHA and PHB Midterm Evaluation Investigators: Beulah Jayakumar, Em Sovannarith, Ok Amry, Sakony Pen, Bopha Kong Sponsor: USAID/Cambodia Introduction Hello, my name is----------. I am part of an evaluation team from Social Impact (SI). SI is an international consulting company with its headquarters in Virginia, USA and works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. SI has been asked by USAID to conduct an independent evaluation of the Enhancing Quality of Healthcare Services Activity (EQHA) and Promoting Healthy Behaviors (PHB). EQHA is a USAID-funded activity to improve the quality and safety of health services, strengthen systems and increase service utilization and is implemented by FHI 360. PHB is also a USAID-funded activity to improve health behaviors among Cambodians and is implemented by PSI. This evaluation aims to assess the performance of EQHA and PHB, identify the main constraints and make recommendations needed to meet their intended objectives. As independent evaluators, SI does not represent the Royal Government of Cambodia or USAID. We selected you to participate in this evaluation because of your experience with EQHA/PHB and with the health services of Cambodia. I would like to request that you read (or I will read to you) this Consent Form, to make sure that you are fully informed about this evaluation before you decide if you want to participate or not. After I have introduced this evaluation to you and have gone through what is expected of you, and if you agree to participate, I will sign this form indicating your consent. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. Please ask us to explain anything in this form that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to ask you and other key informants about the interventions of EQHA and PHB that you may know, and your perceptions of their achievements and challenges. We will also ask you about how they could be improved to achieve more significant results. We plan to conduct 74 interviews similar to this with individuals or groups, at the national level and across the locations where EQHA and PHB work. Your participation in this interview will take no more than 45 minutes. Your rights You have the right to refuse to answer any questions or to stop the interview at any time. You have the right to talk about this evaluation to whomever you choose. Accepting to participate or ending your participation will not affect your working relationship with EQHA or PHB. Possible risks 60 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV We do not anticipate any significant risks to you or your organization/facility because of your participation in this interview. However, please note that should you choose to participate in this interview, you will be taking time away from your regular activities, which may affect your routine tasks. Also, some aspects of the interview might affect your feelings regarding your work. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making and improve strategies for better health outcomes. Your participation in this evaluation will therefore be beneficial to EQHA and PHB and similar programs. By participating in this interview, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation This interview is voluntary, and you may choose to end your participation at any time. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with EQHA and PHB or other organizations that provide similar services will not be affected at all. Confidentiality The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. We will share the opinions you give us in a report to other entities outside of our team, but all your answers will be combined with those from other interviews, so nothing you share can be directly linked back to our conversation today. We want to record your interview to help us transcribe what you said accurately, but you may choose to participate and not be recorded, with no consequence. If we record this interview, the recordings will be uploaded to a password protected site and will not be made public. Only the evaluation team of 5 people will have access to these recordings, and they will be permanently deleted two months after the evaluation is completed. We will not record your name in our data collection tools or notes, or in this consent form. We will also not indicate your name in the any of the reports we prepare, but only your official designation and place of work in an annex to the evaluation report. Please note that (f your official designation is only one of its kind, mentioning it in the annex amounts to identifying you as a participant in this evaluation. We will not tell your peers, supervisors, or friends about the information you give. Protection from COVID-19 – Only for in-person interviews Given the COVID-19 pandemic there are several reasons you may choose not to participate in this interview. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we would ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as maintaining a distance of six feet during interviews, wearing face masks, utilizing well-ventilated areas, and using hand sanitizer before, during, and after interviews. If you are worried that these measures are not possible or you cannot adhere to these requests, the study team can contact you remotely for an interview instead. Likewise, if you do not feel comfortable the day of the interview for any reason, you can decline to participate or end the interview early without any consequence. Also, please note that due to COVID￾19 we will be keeping a log of all interviews including your name and contact information to facilitate contact tracing should any member of the evaluation team become ill so that we would be able to inform you. We will delete your information from the log two weeks from today. If you have any questions about this evaluation: If you have any questions about this evaluation, you may contact Ms. Kong Bopha. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. 61 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV KEY INFORMANT INTERVIEW CONSENT FORM IPs and Sub awardees (TOTAL 15 KIIs) Title: EQHA and PHB Midterm Evaluation Investigators: Beulah Jayakumar, Em Sovannarith, Ok Amry, Sakony Pen, Bopha Kong Sponsor: USAID/Cambodia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of EQHA and PHB. SI is an international consulting company with its headquarters in Virginia, USA and works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. We believe you are aware of the objectives and interventions of EQHA and PHB. This mid-term evaluation aims to assess the performance of EQHA and PHB, identify the main constraints and make recommendations needed to meet their intended objectives. As independent evaluators, SI does not represent the Royal Government of Cambodia or USAID. We selected you to participate in this evaluation because of your experience with EQHA/PHB and with the health services of Cambodia. I would like to request you to read (or have read to you) this Consent Form, to make sure that you are fully informed about this evaluation before you decide if you want to participate or not. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you to sign this form if you agree to participate. The SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. Please ask us to explain anything in this form that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to ask you and other key informants about the interventions of EQHA/PHB that you may know, and your perceptions of their achievements and challenges. We will also ask you about how they could be improved to achieve more significant results. We plan to conduct a total of 15 interviews similar to this with individuals or groups, across the locations where EQHA and PHB works. Your participation in this interview will take no more than 2 hours to complete. Your rights You have the right to refuse to answer any questions or to stop the interview at any time. You have the right to talk about this evaluation to whomever you choose. Accepting to participate or ending your participation will not affect your working relationship with EQHA or PHB. Possible risks We do not anticipate any significant risks to you or your organization because of your participation in this interview. However, please note that should you choose to participate in this interview, you will be 62 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV taking time away from your regular activities, which may affect your routine tasks. Also, some aspects of the interview might affect your feelings regarding your work. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making and improve strategies for better health outcomes. Your participation in this evaluation will therefore be beneficial to EQHA and PHB and similar programs. By participating in this interview, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation This interview is voluntary, and you may choose to end your participation at any time. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with EQHA and PHB or other organizations that provide similar services will not be affected at all. Confidentiality The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. We will share the opinions you give us in a report to other entities outside of our team, but all your answers will be combined with those from other interviews, so nothing you share can be directly linked back to our conversation today. We want to record your interview to help us transcribe what you said accurately, but you may choose to participate and not be recorded, with no consequence. If we record this interview, the recordings will be uploaded to a password protected site and will not be made public. Only the evaluation team of 5 people will have access to these recordings, and they will be permanently deleted two months after the evaluation is completed. We will not record your name in our data collection tools or notes, or in this consent form. We will also not indicate your name in the any of the reports we prepare, but only your official designation and place of work in an annex to the evaluation report. Please note that if your official designation is only one of its kind, mentioning it in the annex will amount to identifying you as a participant in this evaluation. We will not tell your peers, colleagues, or supervisors about the information you give. Protection from COVID-19 – Only for in-person interviews Given the COVID-19 pandemic there are several reasons you may choose not to participate in this interview. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we would ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as maintaining a distance of six feet during interviews, wearing face masks, utilizing well-ventilated areas, and using hand sanitizer before, during, and after interviews. If you are worried that these measures are not possible or you cannot adhere to these requests, the study team can contact you remotely for an interview instead. Likewise, if you do not feel comfortable the day of the interview for any reason, you can decline to participate or end the interview early without any consequence. Also, please note that due to COVID￾19 we will be keeping a log of all interviews including your name and contact information to facilitate contact tracing should any member of the evaluation team become ill so that we would be able to inform you. We will delete your information from the log two weeks from today. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Ms. Kong Bopha. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 63 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. KEY INFORMANT INTERVIEW CONSENT FORM – USAID/Cambodia (TOTAL 2 KII) Title: EQHA and PHB Midterm Evaluation Investigators: Beulah Jayakumar, Em Sovannarith, Ok Amry, Sakony Pen, Bopha Kong Sponsor: USAID/Cambodia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of EQHA and PHB at the request of USAID. SI is an international consulting company with its headquarters in Virginia, USA and works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. We believe you are aware of the objectives and interventions of EQHA and PHB. This mid-term evaluation aims to assess the performance of EQHA and PHB, identify the main constraints and make recommendations needed to meet their intended objectives. We selected you to participate in this evaluation because of your experience with EQHA/PHB and familiarity with the health services of Cambodia. I would like to request you to read (or have read to you) this Consent Form, to make sure that you are fully informed about this evaluation before you decide if you want to participate or not. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you to sign this form if you agree to participate. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. Please ask us to explain anything in this form that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to ask you and other key informants about the interventions of EQHA/PHB that you may know, and your perceptions of their achievements and challenges. We will also ask you about how they could be improved to achieve more significant results. This is the only/one of two interview(s) we will conduct with USAID/Cambodia. Your participation in this interview will take no more than 2 hours to complete. Your rights You have the right to refuse to answer any questions or to stop the interview at any time. You have the right to talk about this evaluation to whomever you choose. Accepting to participate or ending your participation will not affect your working relationship with EQHA or PHB. 64 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Possible risks We do not anticipate any significant risks to you or your organization because of your participation in this interview. However, please note that should you choose to participate in this interview, you will be taking time away from your regular activities, which may affect your routine tasks. Also, some aspects of the interview might affect your feelings regarding your work. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making and improve strategies for better health outcomes. Your participation in this evaluation will therefore be beneficial to EQHA and PHB and similar programs. By participating in this interview, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation This interview is voluntary, and you may choose to end your participation at any time. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with EQHA and PHB or other organizations that provide similar services will not be affected at all. Confidentiality The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. We will share the opinions you give us in a report to other entities outside of our team, but all your answers will be combined with those from other interviews, so nothing you share can be directly linked back to our conversation today. We want to record your interview to help us transcribe what you said accurately, but you may choose to participate and not be recorded, with no consequence. If we record this interview, the recordings will be uploaded to a password protected site and will not be made public. Only the evaluation team of 5 people will have access to these recordings, and they will be permanently deleted two months after the evaluation is completed. We will not record your name in our data collection tools or notes, or in this consent form. We will also not indicate your name in the any of the reports we prepare, but only your official designation and place of work in an annex to the evaluation report. Please note that as your official designation is only one of its kind, mentioning it in the annex amounts to identifying you as a participant in this evaluation. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Ms. Kong Bopha. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. 65 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FOCUS GROUP DISCUSSION - CONSENT FORM for VHSGs, Mothers, WRA and their male partners (Total 16 FGDs) Title: EQHA and PHB Midterm Evaluation Investigators: Beulah Jayakumar, Em Sovannarith, Ok Amry, Sakony Pen, Bopha Kong Sponsor: USAID/Cambodia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of a project funded by USAID, to promote healthy behaviors among Cambodians called PHB. This project is called PHB. SI is an international consulting company with its headquarters in Virginia, USA and works to improve development effectiveness around the world through evaluation, capacity building, and strategic planning. We do not represent the Royal Government of Cambodia or USAID. Information about the evaluation This evaluation aims to assess the performance of the PHB project, identify any challenges it may have had and make recommendations for the project to best improve health services, behaviors, and communications for the people where you live. We have selected you to participate in this group discussion because we would like to hear about your experience using health services in this location. If you agree to be part of this evaluation, we are going to ask you and others about the interventions of PHB that you may know, and your perceptions of their achievements and challenges. We will also ask you about how they could be improved to achieve more significant results for mothers/youth like you. We plan to conduct a total of 16 group discussions similar to this across the locations where this project works. This group discussion will take no longer than an hour. I would like to request you to read (or have read to you) this Consent Form, to make sure that you are fully informed about this evaluation before you decide if you want to participate or not. After I have introduced this evaluation to you and have gone through what is expected of you, if you agree to participate in this discussion, I will sign the form indicating that. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. Your rights You have the right to refuse to answer any questions or to leave this discussion at any time. You have the right to talk about this evaluation to whomever you choose. Accepting to participate or ending your participation will not affect the services you provide or the support you may be receiving from PHB. Possible risks We do not anticipate any significant risks to you because of your participation in this group discussion However, please note that should you choose to participate in this group discussion you will be taking time away from your regular activities, which may affect your routine tasks. Also, some aspects of the discussion might affect your feelings regarding your health and wellbeing. 66 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Possible benefits The results of this evaluation are expected to inform USAID’s future projects and strategies for better health outcomes. Your participation in this evaluation will therefore be beneficial to PHB and similar programs. By participating in this discussion, you will, however, get no immediate and direct personal benefit. You will be compensated for any expense you incurred in order to participate in this discussion, such as travel, or internet data top-up. If you decide not to participate in this evaluation This study is completely voluntary. This means you may decide not to participate and end your participation in this group discussion at any time. We will not hold anything against you should you choose to leave before the end of the discussion. Your relationship with PHB or other organizations that provide similar services will not be affected at all. Confidentiality The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. We will share the opinions you give us in a report to other entities outside of our team, but all your answers will be combined with those from other group discussions, so that nothing you share can be directly linked back to our discussion today. We want to record this discussion to help us transcribe what you said accurately, but you may choose to participate and not be recorded, with no consequence. If we record this discussion, the recordings will be uploaded to a password protected site and will not be made public. Only the evaluation team of 5 people will have access to these recordings, and they will be permanently deleted two months after the evaluation is completed. We will not record your name in our data collection tools or notes, or in this consent form. We will also not indicate your name in the any of the reports we prepare, but only your age, gender, and the name of your commune in an annex to the evaluation report. We will not tell your peers or friends about the information you give. We also request that all of you not share the contents of our discussion with those outside this group. Protection from COVID-19 Given the COVID-19 pandemic there are several reasons you may choose not to participate in this discussion. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we would ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as maintaining a distance of six feet during the group discussion, wearing face masks, utilizing well-ventilated areas, and using hand sanitizer before, during, and after the discussion. If you are worried that these measures are not possible or you cannot adhere to these requests, the study team can contact you remotely instead. Likewise, if you do not feel comfortable, for any reason, you can decline to participate or end the discussion early without any consequence. Also, please note that due to COVID-19 we will be keeping a separate log of all participants’ names and contact information to facilitate contact tracing should any member of the evaluation team become ill so that we would be able to inform you. We will delete your information from the log two weeks from today. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Bopha Kong. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address: 2300 67 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. KEY INFORMANT INTERVIEW – EQHA and PHB national and subnational teams EQHA and PHB Midterm Evaluation Interviewees: EQHA: Chief of Party, Dy Chief of Party, MEL Director, Finance Director, Gender Point person, Policy team, Regulations team, QI team, Private Sector team, Preservice team, Province teams PHB: Chief of Party, MEL Director, Finance Director, Gender Point person, Province level teams Please note: Some questions are meant for both EQHA and PHB, and others for one of the two. Please re-word the term EQHA/PHB as required. EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. Let’s begin the discussion by looking at the areas in which EQHA/PHB has already built the capabilities of the government systems, including systems and processes. Could you please list those areas? Probe by EQHA/PHB relevant objective: Objective 1: Objective 2: Objective 3: Objective 4: 2. Are there other tools and systems does EQHA//PHB plan to institutionalize during the rest of its implementation? Probe by EQHA/PHB relevant objective: Objective 1: Objective 2: Objective 3: Objective 4: 3. How confident are you that these capabilities will continue to be used after the project ends? What level of commitment exists in these systems? What are the reasons for you to say so? Strengths and advantages 4. What strengths in EQHA/PHB’s interventions, tools, and processes (that you are still trying to institutionalize)? 5. What weaknesses in these interventions might hamper your efforts to institutionalize them? Impact of D&D 6. To what extent do you think the D&D process has impacted your efforts to institutionalize EQHA/PHB interventions, or will in the future, both positively and negatively? Which interventions/objectives of EQHA/PHB are more likely to be impacted and which ones, not so much? 7. To what extent do you think EQHA/PHB can take steps to minimize the negative impact? External evidence – ONLY FOR CORE TEAMS 68 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 8. Has EQHA/PHB considered evidence from within Cambodia and from outside, on good practices for successful institutionalization? What are they, and do you plan to use that evidence in your own efforts to institutionalize your interventions and processes? If so, in what ways? Metrics and learning – ONLY FOR CORE TEAMS 9. What would you consider as a measure of success in institutionalizing an intervention or a process? What process do you have in place, or plan to put in place, to measure, assess, and learn from your efforts? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 10. You already mentioned some of the challenges you face in institutionalizing your interventions and processes. Besides, those, what other challenges does EQHA/PHB face, in achieving its results? Please discuss the challenges you currently face and those that you anticipate in the future. Why do you think these challenges come about? Again, let’s discuss these by objective relevant to you, but please also include those challenges that are common across objectives. Objective 1: Objective 2: Objective 3: Objective 4: Scope of challenges 11. Which of these challenges will EQHA/PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 12. What steps has EQHA/PHB taken to address these challenges to date? How successful have these interventions been? Are there any key lessons have you learned? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities 13. Now let’s talk about how EQHA and PHB can work together for better health outcomes. What opportunities are there for both to work together and build synergy, at national, province, and OD levels? National: Province: OD: Probe for common stakeholders, and other meeting points between EQHA and PHB Adaptations 14. What adaptations do you think EQHA/PHB should make to utilize these opportunities? What challenges and conflicts should you preempt and address? What might the enabling factors be, or what might be needed to ensure success of these opportunities? 69 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Lessons from past efforts 15. What efforts has EQHA/PHB already made to work together and build synergy? How successful were these efforts? What do you think contributed to success or lack of, and why? What was the return on investment? What did you learn from these attempts? National: Province: OD: Metrics and learning – ONLY FOR CORE TEAMS 16. What would you consider as a measure of success in working together? What process do you have in place, or plan to put in place, to measure, assess and learn from your efforts to work together? EQ4. How have the EQHA ’s and PHB’s structure, management, and internal procedures and sub awardees management affected implementation and outcomes, positively and/or negatively? – ONLY FOR CORE TEAMS Organizational features 17. Now let’s talk about EQHA/PHB’s internal structure and processes, beginning with your organization. What are the core values and ethos of your organization that have helped make EQHA/PHB successful thus far? 18. How does the structure of the EQHA/PHB team impact its functioning or operations? 19. What measures have you put in place to ensure that gender or equity issues do not affect performance? Project procedures 20. Could you describe to us the process used to select sub awardees? 21. What support does the IP provide to sub awardees in their implementation? 22. What expectations and requirements are there regarding sub-awardee communication with the IP, reporting, and monitoring? 23. What steps are in place to ensure accountability and transparency? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? ONLY FOR EQHA CORE TEAM AND PVT SECTOR TEAM Measures of effectiveness and quality 24. Now let’s talk about EQHA’s engagement of the private sector. We note that this engagement is built into all four of your objectives. How does EQHA define “success” in engaging the private sector in each of its objectives? Objective 1: Using the NQEM tool in private facilities, developing QI plans Objective 2: Referral linkages, and quality of services Objective 3: Accreditation process Objective 4: Engaging private academic institutions 25. What challenges are there in the private sector (hospitals, providers, and academic institutions) in providing quality services for better health outcomes? Private hospitals and clinics Private academic institutions Effectiveness of EQHA interventions 26. How effective, in your opinion, have EQHA’s interventions been so far, in each of the above areas? 27. What challenges have you faced, or expect to face? 70 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 28. What lessons have been learned? Incentivizing private sector involvement 29. What can be done further to increase the value proposition for the private providers in engaging with the system? What incentivizes their involvement? 30. What disincentives does the private sector have? How can EQHA work to minimize these? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? - Only for PHB teams Measures of success 31. Now let’s talk about PHB’s engagement of the public sector, particularly the NCHP. How does PHB define successful engagement of NCHP? EFFECTIVENESS OF PHB INTERVENTIONS 32. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 33. How well have CCWCs absorbed and utilized the funding? 34. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 35. To what extent has PHB utilized findings from its formative work? 36. Could you tell us about how PHB adapts and uses global and local evidence? Please give examples. 37. To what extent the NCHP is able to identify and use best practices? 38. To what extent is the five-step design approach of PHB taken up by NCHP? What can be done to maximize its uptake? 39. Has PHB reassessed its critical assumptions regarding NCHPs role? What further can be done to ensure that this leading role of NCHP remains functional? Adaptations and readiness of PHB 40. What adaptations should PHB make to improve uptake by NCHP? 41. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or PHB, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – Sub awardee of EQHA and PHB EQHA and PHB Midterm Evaluation Interviewees: EQHA and PHB Sub awardees – focal points for the Activity EQ4. How have the EQHA’s and PHB’s structure, management, and internal procedures and sub awardees management affected implementation and outcomes, positively and/or negatively? Organizational features 71 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 1. Now let’s talk about EQHA’s internal structure and processes, beginning with your organization. What are the core values and ethos of your organization that have helped make EQHA successful thus far? 2. How does the structure of the EQHA team impact the functioning of the team? 3. Are there any gender or equity issues that affect performance? Project procedures 4. Could you describe to us the process used to select sub awardees? 5. What support does the IP provide to sub awardees in their implementation? 6. What expectations and requirements are there regarding sub-awardee communication with the IP, reporting, and monitoring? 7. What steps are in place to ensure accountability and transparency? 8. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA or PHB or about the health system in general? Thank the respondents for their time KEY INFORMANT INTERVIEW – USAID/Cambodia EQHA and PHB Midterm Evaluation Interviewees: USAID/Cambodia Activity managers EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. Let’s begin the discussion by looking at the areas in which EQHA/PHB has already built the capabilities of the government systems, including systems and processes. Could you please list those areas? Could you begin with EQHA and then to PHB? Probe by EQHA/PHB objective: Objective 1: Objective 2: Objective 3: Objective 4: 2. What other tools and systems does EQHA//PHB plan to institutionalize during the rest of its implementation? Again, could you begin with objective 1. Probe by EQHA/PHB objective: Objective 1: Objective 2: Objective 3: Objective 4: 3. What strengths in EQHA/PHB’s interventions, tools and processes might enable their efforts to institutionalize them? 4. What weaknesses in these interventions might hamper these efforts? 5. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? 72 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 73 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Impact of D&D 6. To what extent do you think the D&D process has impacted efforts to institutionalize EQHA/PHB interventions, or will in the future, both positively and negatively? Which interventions/objectives of EQHA/PHB are more likely to be impacted and which ones, not so much? 7. To what extent do you think EQHA/PHB can take steps to minimize the negative impact? Metrics and learning – ONLY FOR CORE TEAMS 8. What would you consider as a measure of success in institutionalizing an intervention or a process? What process do you have in place, or plan to put in place, to measure, assess and learn from your efforts? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 9. You already mentioned some of the challenges you face in institutionalizing the interventions and processes. Besides, those, what other challenges does EQHA/PHB face, in achieving its results? Please discuss the challenges they currently face and also those that they anticipate in the future. Again, let’s discuss these by objective, but please also include those challenges that are common, across objectives. Objective 1: Objective 2: Objective 3: Objective 4: Scope of challenges 10. Which of these challenges will EQHA/PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 11. What steps has EQHA/PHB taken to address these challenges, and how successful have they been, and what lessons have they learned? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities 12. Now let’s talk about how EQHA and PHB can work together for better health outcomes. What opportunities are there for both to work together and build synergy, at national, province, and OD levels? National: Province: OD: 74 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Probe for common stakeholders, and other meeting points between EQHA and PHB Adaptations 13. What adaptations should EQHA/PHB take to utilize these opportunities? What challenges and conflicts should they preempt and address? Lessons from past efforts 14. What efforts has EQHA/PHB already made to work together and build synergy? How successful were these efforts? What was the return on investment? What did they learn from these attempts? National: Province: OD: Metrics and learning 15. What would you consider as a measure of success in EQHA and PHB working together? What process do they have in place, or plan to put in place, to measure, assess and learn from the efforts to work together? 16. What is currently working well within the MEL systems, and why? What, if anything is not working well, and why? EQ4. How have the EQHA’s and PHB’s structure, management, and internal procedures and sub awardees management affected implementation and outcomes, positively and/or negatively? Organizational features 17. How does the structure of the EQHA/PHB team impact the functioning of the team? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? Measures of effectiveness and quality 18. Now let’s talk about EQHA’s engagement of the private sector. We note that this engagement is built into all four of its objectives. What is your view on how EQHA defines “success” in engaging the private sector in each of its objectives? Objective 1: Using the NQEM tool in private facilities, developing QI plans Objective 2: Referral linkages, and quality of services Objective 3: Accreditation process Objective 4: Engaging private academic institutions 19. What challenges are there in the private sector (hospitals, providers, and academic institutions) in providing quality services for better health outcomes? Private hospitals and clinics Private academic institutions 75 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Effectiveness of EQHA interventions 20. How effective, in your opinion, have EQHA’s interventions been so far, in each of the above areas? Why do you think this is? 21. What challenges has it faced, or expect to face? Why do you think this is? 22. What lessons have been learned? Incentivizing private sector involvement 23. What can be done further, to increase the value proposition for the private providers in engaging with the system? What incentivizes their involvement? 24. What disincentives does the private sector have? How can EQHA work to minimize these? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Measures of success 25. Now let’s talk about PHB’s engagement of the public sector, particularly the NCHP. How does PHB define successful engagement of NCHP? What is currently working well with this engagement strategy? What, if anything is not working well, and why? Effectiveness of PHB interventions 26. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 27. How well have CCWCs absorbed and utilized the funding? 28. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 29. To what extent has PHB utilized findings from its formative work? 30. To what extent is the NCHP able to identify and use best practices? 31. To what extent is the five-step design approach of PHB taken up by NCHP? What can be done to maximize its uptake? 32. Has PHB reassessed its critical assumptions regarding NCHPs role? What further can be done to ensure that this leading role of NCHP remains functional? Adaptations and readiness of PHB 33. What adaptations should PHB make to improve uptake by NCHP? 34. We have come to the end of this discussion. Is there anything else you would like to tell us at this time about EQHA or PHB, or about the health system in general? Thank the respondents for their time. 76 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV KEY INFORMANT INTERVIEW – DG Health EQHA and PHB Midterm Evaluation Interviewees: DG Health EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? 1. We would like to talk to you about the USAID project EQHA implemented by FHI 360. Its main purpose is to support MoH and other stakeholders to improve the quality of health services and works in areas of […]. Some of their strategic tools and processes include […]. In what ways can we ensure that Cambodia’s health system incorporates them? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 2. We are also reviewing the USAID project PHB implemented by PSI. Its main purpose is to […] and it supports NCHP in its role of coordinating SBC interventions across all vertical programs. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 3. What challenges do you anticipate in increasing the visibility of SBC in health programs? 4. What is the effective approach for NCHP to work effectively in the collaboration with the national vertical programs? Lessons learned and knowledge transfer 5. What further can be done to ensure that the leading role of NCHP remains functional? Adaptations and readiness of PHB In your projection, what do you expect to see NCHP and the national vertical programs in the next 5 years? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. 6. What can these two projects do to better support coordination between health services and behavior change interventions? Opportunities for leverage 7. Now let’s talk about how EQHA and PHB can work together for better health outcomes. What opportunities are there for both to work together and build synergy, at national, province, and OD levels? National: 77 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Province: OD: Adaptations 8. What adaptations should EQHA/PHB take to utilize these opportunities? What challenges and conflicts should you preempt and address? What should EQHA/PHB do better to ensure the effective coordination with other intervention programs? 9. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about NCHP’s role or about the health system in general? THANK THE RESPONDENT FOR THEIR TIME KEY INFORMANT INTERVIEW – Director, DHS and Chief, QAO, QI TWG EQHA and PHB Midterm Evaluation Interviewees: Director, DHS, Chief of QAO, QI TWG members EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We would like to talk to you about the USAID project EQHA implemented by FHI 360. In what areas has EQHA already built the capabilities of health systems and processes. Could you please list those areas? Objective 1: NQEM process Objective 2: NQEM process in facilities Objective 3: Objective 4: 2. What other approaches including strategic concepts, tools and systems can EQHA plan to include in the health system? Objective 1: Objective 2: Objective 3: Objective 4: 3. What strengths are there in EQHA’s intervention, strategic approaches, tools, and processes that will help in their efforts to institutionalize them? 4. What weaknesses in these interventions might hamper these efforts? 5. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? What lessons have you learned from your own efforts? 78 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Impact of D&D 6. To what extent do you think the D&D process has impacted these efforts, both positively and negatively? Which interventions/objectives of EQHA are more likely to be impacted and which ones, not so much? 7. To what extent do you think EQHA can take steps to minimize the negative impact? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 8. You already mentioned some of the challenges faced in including EQHA strategic approaches, tools in the government system. Besides, those, what other challenges does EQHA face in achieving its results? Please discuss the challenges they currently face and also those that they anticipate in the future. Objective 1: Objective 2: Objective 3: Objective 4: Scope of challenges 9. Which of these challenges will EQHA be able to address, and which ones are likely to remain, and why? Steps taken so far 10. What steps has EQHA taken to address these challenges, and how successful have they been, and what lessons have they learned? EQ5. How effective have the EQHA approaches been in engaging the private academic institution in program implementation? Measures of effectiveness and quality 11. What advantages and challenges are there in the private academic institution in providing quality pre-service training for better health outcomes? Effectiveness of EQHA interventions 12. We want to talk to you about EQHA’s engagement of the private academic institution. How effective, in your opinion, have EQHA’s interventions been so far, in supporting pre-service training? 13. What challenges have you faced, or expect to face? 14. What lessons have been learned? 79 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Incentivizing private academic institution involvement 15. What can be done further, to increase the value proposition for the private academic institution in engaging with the system? What incentivizes their involvement? 16. What disincentives does the private academic institution have? How can EQHA work to minimize these? 17. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondent for their time KEY INFORMANT INTERVIEW – DPHI, MOH EQHA and PHB Midterm Evaluation Interviewees: Director, DPHI EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We would like to talk to you about the USAID project EQHA implemented by FHI 360. EQHA works with MoH systems to improve the quality of health systems and services in the areas of […]. Some of their tools and processes have been taken up by MoH departments, for example […]. How can EQHA improve its efforts to institutionalize its strategic approaches, tools, and processes? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 2. What challenges do projects like EQHA face in making sure that their strategic approaches and tools are successfully absorbed into the system? Scope of challenges 3. Which of these challenges will EQHA be able to address, and which ones are likely to remain, and why? Impact of D&D 1. To what extent do you think the D&D process could impact these efforts, both positively and negatively? 2. What steps EQHA take to minimize the negative impact? EQ5. How effective have the EQHA approaches been in engaging the private academic institution in program implementation? 80 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Effectiveness of EQHA interventions 3. What challenges do you face in engaging private institutions for better health care? 4. What lessons have been learned? Incentivizing private academic institution involvement 5. What can be done further to increase the value proposition for the private academic institution in engaging with the system? What incentivizes their involvement? 6. What disincentives does the private academic institution have? How can EQHA work to minimize these? 7. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondent for their time KEY INFORMANT INTERVIEW – HRDD, MOH EQHA and PHB Midterm Evaluation Interviewees: Director HRDD EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We want to discuss with you the USAID project, EQHA. One of the objectives of EQHA is to strengthen policies, standard and guidelines and system for the human resource development? How has this progressed? Probe for policies/guidelines for the department of HRD - for Pre-service training at TRCs? Training curriculum, Materials? 2. What are the gaps in the HRMIS data system you are facing now? What could EQHA do further to work with you improve the system? 3. How useful is the HRMIS in the supporting the human resource development? How confident are you that the HRMIS will continue to be used after the project ends? What are the reasons for you to say so? 4. What other support has EQHA provided to your department? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Impact of D&D 5. How does the D&D process positively contribute to the implementation of human resource recruitment or required need by the health facilities? 6. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? 81 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Thank the respondents for their time. KEY INFORMANT INTERVIEW – Director, Dept of Occupational health and safety, MoLVT EQHA and PHB Midterm Evaluation Interviewees: Director, Dept of Occupational health and safety, MoLVT EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We would like to talk to you about the USAID project EQHA implemented by FHI 360. What kind of cooperation do you have with the EQHA? How does the cooperation look like and how has it benefited the work of your department? 2. One of the main objectives of the collaboration MoLVT is to review workplace policies/guidelines and SOPs in a bid to strengthen infirmary policies. How has this progressed? 3. How confident are you that these capabilities will continue to be used after the EQHA ends? What are the reasons for you to say so? 4. What are the gaps in strengthening quality of health services in infirmary you are facing now? What should be done to improve the system? Strengths, advantages, opportunities 5. What more can EQHA do that will be useful to improve the quality of the health services and strengthen the health system in the labor industry? What are the reasons for you to say so? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? 6. What challenges are there in the private sector (hospitals, providers, and academic institutions) in providing quality services for better health outcomes? Private hospitals and clinics Private academic institutions Incentivizing private sector involvement 7. What can be done further, to increase the value proposition for the private providers in engaging with the system? What incentivizes their involvement? 8. What disincentives does the private sector have? How can EQHA work to minimize these? 9. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondents for their time. 82 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV KEY INFORMANT INTERVIEW – Puthisastra/RTC EQHA and PHB Midterm Evaluation Interviewees: Representative of Puthissatra University and Director/deputy director of RTC in Battambang and Kampong Chhnang EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 8. What kinds of support or in what areas has EQHA already built the quality and capabilities of academic institutions? Could you please list those areas? Probing for support such as revision of competency-based training curriculum for associate degree of nursing (QI and GBV integration in pre-service training for nurses) etc., and specifically for: a. Puthisastra: distance learning hub equipment, b. BB and KPC RTC: simulation classroom equipment, preceptor training, HRMIS 9. What other approaches including tools and systems can EQHA plan to include in the academic institution? 10. To what extent the support/interventions of EQHA contributed to improving your effort in providing quality pre-service training? 11. What strengths are there in EQHA’s interventions, tools and processes that will help in their efforts to institutionalize them? 12. What weaknesses in these interventions might hamper these efforts? 13. How confident are you that these capabilities will continue to be used after EQHA ends? What are the reasons for you to say so? EQ5. How effective have the EQHA approaches been in engaging the private academic institution in program implementation? (Puthisastra ONLY) Measures of effectiveness and quality 14. What advantages and challenges are there in the private academic institution in providing quality pre-service training for better health outcomes? 15. What are the measures of quality pre-service training? Effectiveness of EQHA interventions 16. We want to talk to you about EQHA’s engagement of the private academic institution. How effective, in your opinion, have EQHA’s interventions been so far in supporting pre-service training? 17. What challenges have you faced, or expect to face? 18. What lessons have been learned? Incentivizing private academic institution involvement 19. What can be done further, to increase the value proposition for the private academic institution in engaging with the system? What incentivizes their involvement? 20. What disincentives does the private academic institution have? How can EQHA work to minimize these? 83 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQ3. What are key challenges in achieving the expected results of EQHA and how to overcome the challenges? Challenges by objective 21. What challenges does EQHA face in strengthening preservice training? Scope of challenges 22. Which of these challenges will EQHA be able to address, and which ones are likely to remain, and why? Steps taken so far 23. What steps has EQHA taken to address these challenges, and how successful have they been? 24. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – HPC, MOH EQHA and PHB Midterm Evaluation Interviewees: SoS, Coordination Committee for HPCs, Secretary General, MCC, President, CCN, Under SoS/President, PCC, President, CMC EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present effort 1. We want to discuss with you the USAID project, EQHA. One of the objectives of EQHA is to strengthen policies, standard and guidelines and system for registration and licensing of health practitioners? How has this progressed? 2. What policies, standards, guidelines have been developed (or drafted) so far? If just in drafting process, when do you think they will be approved? How important are these policies/standards/guidelines/systems useful to improve the R&L process? 3. What are the gaps in strengthening R&L process you are facing now? What should be done to improve the system? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? CHALLENGES BY OBJECTIVE 4. What are the main challenges that EQHA faces in this task? 84 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Scope and nature of challenges 5. To what extent will EQHA be able to address these challenges? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? (Puthisastra ONLY) Measures of effectiveness and quality 6. What advantages and challenges are there in the private sector in providing services for better health outcomes? 7. What are the measures of quality that the private sector uses for its services? Effectiveness of EQHA interventions 8. We want to talk to you about EQHA’s engagement of the private sector. How effective, in your opinion, have EQHA’s interventions been so far in supporting the private sector? 9. What challenges has EQHA faced, or expect to face? 10. What lessons have been learned? Incentivizing private academic institution involvement 11. What can be done further, to increase the value proposition for the private sector in engaging with the system? What incentivizes their involvement? 12. What disincentives does the private sector have in engaging the public system? How can EQHA work to minimize these? 13. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – MRD EQHA and PHB Midterm Evaluation Interviewees: Director, Dept of Rural Health EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We want to discuss with you about the work of two USAID projects EQHA and PHB. EQHA aims to improve the quality of health services and PHB supports NCHP in promoting systems for behavior change. We want to discuss with you about your Ministry’s successful efforts to institutionalize CLTS. What lessons can other initiatives learn about sustaining their interventions, strategic approaches, and tools? 2. What are the strengths in project interventions, that help in their efforts to introduce them into government systems? 85 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 3. What weaknesses in project interventions might hamper their efforts? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 4. We also want to talk to you about your interaction with the USAID project, PHB, in promoting WASH behaviors. To what extent are PHB approaches successful and effective? 5. What are some areas of improvement in PHB approaches to designing and implementing behavior change interventions? 6. What challenges do you anticipate in increasing the visibility of behavior change programming in the health sector? 7. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondent for their time. KEY INFORMANT INTERVIEW - NCHP EQHA and PHB Midterm Evaluation Interviewees: Key staff of NCHP EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. Let’s begin the discussion by looking at the support that PHB has provided NCHP. In what areas has PHB built your capabilities including your systems and processes. Could you please list those areas? Objective 1: Objective 2: Strengths and advantages 2. What strengths in PHB’s interventions, tools and processes might enable its efforts to institutionalize them? 3. What weaknesses in these interventions might hamper its efforts? 4. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? 5. What could PHB do better to ensure the PHB’s experiences and approaches to be institutionalized? Impact of D&D 6. To what extent do you think the D&D process has impacted your efforts to utilize PHB’s tools and processes? Or will D&D affect these efforts in the future, both positively and negatively? 86 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Which interventions/objectives of PHB are more likely to be impacted and which ones, not so much? 7. To what extent do you think PHB can take steps to minimize the negative impact? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 1. Now let’s talk about how EQHA and PHB can work together for better health outcomes. What opportunities are there for both to work together and build synergy, at national, province and OD levels? National: Province: OD: Probe for common stakeholders, and other meeting points between EQHA and PHB Adaptations 2. What adaptations should EQHA/PHB take to utilize these opportunities? What challenges and conflicts should you preempt and address? Lessons from past efforts 3. What efforts has EQHA/PHB already made to work together and build synergy? How successful were these efforts? What was the return on investment? What did you learn from these attempts? National: Province: OD: Metrics and learning 4. What would you consider as a measure of success in working together? What process do you have in place, or plan to put in place, to measure, assess and learn from your efforts to work together? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 5. What challenges do you think PHB faces in achieving its results? Please discuss the challenges they currently face and also those that you anticipate in the future. Objective 1: Objective 2: 87 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Scope of challenges 6. Which of these challenges will PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 7. What steps has PHB taken to address these challenges, and how successful have they been? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 8. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 9. How well have CCWCs absorbed and utilized the funding? 10. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 11. Could you tell us about how PHB adapts and uses global and local evidence? Please give examples. 12. To what extent is the NCHP able to identify and use best practices? 13. To what extent is the five-step design approach of PHB taken up by NCHP? What can be done to maximize its uptake? 14. What further can be done to ensure that the leading role of NCHP remains functional? Adaptations and readiness of PHB 15. What adaptations should PHB make, to improve uptake by NCHP? 16. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – NCHADS, NCMCHC, CENAT, CNM EQHA and PHB Midterm Evaluation Interviewees: SoS/Director and Chief of Technical Bureau, NCHP, Director and National Focal Points for FP and Nutrition, NMCHC, Director, NCHADS, Director and Deputy Director, CENAT, Director and Deputy Director, CNM EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 88 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 1. We want to discuss with you about the two USAID projects, EQHA and PHB. Based on your interaction with EQHA and PHB, what areas have they already built the capabilities of the government systems, including systems and processes. EQHA PHB 2. What strengths in EQHA and PHB’s interventions, tools and processes might enable their efforts to institutionalize them? 3. What weaknesses in these interventions might hamper these efforts? 4. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 5. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 6. What further can be done to ensure that the leading role of NCHP remains functional? Adaptations and readiness of PHB 7. What adaptations should PHB make to improve uptake by NCHP? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 8. Based on your experience, what challenges do EQHA and PHB face in achieving its results? Please discuss the challenges they currently face and also those that they anticipate in the future. EQHA PHB Scope of challenges 9. Which of these challenges will EQHA and PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 10. What steps has EQHA and PHB taken to address these challenges, and how successful have they been, and what lessons have they learned? 11. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? 89 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Thank the respondents for their time. KEY INFORMANT INTERVIEW – SBC TWG EQHA and PHB Midterm Evaluation Interviewees: Members of SBC TWG (Ensure Save the Children and UNICEF members are included) EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. What lessons Let’s begin the discussion by looking at the areas in which PHB has already built the capabilities of the govt systems, including systems and processes. Could you please list those areas? Objective 1: Objective 2: 2. What other tools and systems does PHB plan to institutionalize during the rest of its implementation? Objective 1: Objective 2: 3. What strengths in PHB’s interventions, tools and processes might enable its efforts to institutionalize them? 4. What weaknesses in these interventions might hamper its efforts? 5. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? What lessons have you learnt from your own efforts? 6. To what extent do you think government counterparts are confident to continue to be used after the project ends? What are the reasons for you to say so? What lessons have you learnt from your own efforts? 7. What could PHB do better to ensure the PHB’s experiences and approaches to be institutionalized? Impact of D&D 8. To what extent do you think the D&D process will impact PHB’s efforts to institutionalize PHB interventions, or will, in the future, both positively and negatively? Which interventions/objectives of PHB are more likely to be impacted and which ones, not so much? 9. To what extent do you think PHB can take steps to minimize the negative impact? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 10. What challenges does PHB face, in achieving its results? Please discuss the challenges they currently face and also those that you anticipate in the future. Objective 1: Objective 2: 90 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Scope of challenges 11. Which of these challenges will PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 12. What steps has PHB taken to address these challenges, how successful have you been, and what lessons have you learned? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 13. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 14. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 15. To what extent is the five-step design approach of PHB taken up by NCHP? What can be done to maximize its uptake? Adaptations and readiness of PHB 16. What adaptations should PHB make, to improve uptake by NCHP and other government counterparts? 17. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW –Gender Mainstreaming Action Group (GMAG) EQHA and PHB Midterm Evaluation Interviewees: Gender Mainstreaming Action Group EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We want to discuss with you about the two USAID projects, EQHA and PHB. In what areas have EQHA and PHB already built the capabilities of the govt systems, including systems and processes. 91 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA PHB 2. What strengths in EQHA and PHB’s interventions, tools and processes might enable their efforts to institutionalize them? 3. What weaknesses in these interventions might hamper these efforts? 4. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? 5. What could EQHA and PHB do better to ensure their experiences and approaches to be institutionalized? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 6. What challenges related to gender mainstreaming do EQHA and PHB face in achieving its results? Please discuss the challenges they currently face and also those that they anticipate in the future. EQHA PHB Scope of challenges 7. Which of these challenges will EQHA and PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 8. What steps has EQHA and PHB taken to address these challenges, and how successful have they been, and what lessons have they learned? 9. How could EQHA and PHB support the GMAG to achieve the gender action plan in EQHA and PHB? 10. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA and PHB, or about the health system in general? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 11. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 12. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 13. Could you tell us about how PHB adapts and uses global and local evidence? Please give examples. 14. To what extent are you able to identify and use best practices? 15. What further can be done to ensure that the leading role of NCHP remains functional? 92 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Adaptations and readiness of PHB 16. What adaptations should PHB make, to improve uptake by you? 17. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – PARTNERS for EQHA EQHA and PHB Midterm Evaluation Interviewees: UNFPA, GIZ, H-EQIP EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We would like to talk to you about the USAID project EQHA implemented by FHI 360. Based on your interactions with EQHA, in what areas has it already built the capabilities of health systems and processes. Could you please list those areas? Objective 1: Objective 2: Objective 3: Objective 4: 2. What other approaches including tools and systems can EQHA plan to include in the health system? Objective 1: Objective 2: Objective 3: Objective 4: 3. What strengths are there in EQHA’s interventions, tools and processes that will help in their efforts to institutionalize them? 4. What weaknesses in these interventions might hamper these efforts? 5. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? What lessons have you learned from your efforts? Impact of D&D 6. To what extent do you think the D&D process could impact these efforts, both positively and negatively? Which interventions/objectives of EQHA are more likely to be impacted and which ones, not so much? 7. To what extent do you think EQHA can take steps to minimize the negative impact? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? 93 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Challenges by objective 8. You already mentioned some of the challenges faced in including EQHA tools in the government system. Besides, those, what other challenges does EQHA face, in achieving its results? Please discuss the challenges they currently face and also those that they anticipate in the future. Objective 1: Objective 2: Objective 3: Objective 4: Scope of challenges 9. Which of these challenges will EQHA be able to address, and which ones are likely to remain, and why? Steps taken so far 10. What steps has EQHA taken to address these challenges, and how successful have they been, and what lessons have they learned? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? GIZ ONLY Measures of effectiveness and quality 11. What advantages and challenges are there in the private academic institution in providing quality pre-service training for better health outcomes? 12. What are the measures of quality pre-service training? Effectiveness of EQHA interventions 13. We want to talk to you about EQHA’s engagement of the private academic institution. How effective, in your opinion, have EQHA’s interventions been so far, in supporting pre-service training? 14. What challenges have you faced, or expect to face? 15. What lessons have been learned? Incentivizing private academic institution involvement 16. What can be done further, to increase the value proposition for the private academic institution in engaging with the system? What incentivizes their involvement? 17. What disincentives does the private academic institution have? How can EQHA work to minimize these? 18. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – PHB partners 94 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA and PHB Midterm Evaluation Interviewees: Marie Stopes, IDE, WB Cambodia nutrition program EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. What lessons Let’s begin the discussion by looking at the areas in which PHB has already built the capabilities of the govt systems, including systems and processes. Could you please list those areas? Objective 1: Objective 2: 2. What other tools and systems does PHB plan to institutionalize during the rest of its implementation? Objective 1: Objective 2: 3. What strengths in PHB’s interventions, tools and processes might enable its efforts to institutionalize them? 4. What weaknesses in these interventions might hamper its efforts? 5. How confident are you that these capabilities will continue to be used after the project ends? What are the reasons for you to say so? What lessons have you learnt from your own efforts? 6. To what extent do you think government counterparts are confident to continue to be used after the project ends? What are the reasons for you to say so? What lessons have you learned from your own efforts? 7. What could PHB do better to ensure the PHB’s experiences and approaches to be institutionalized? Impact of D&D 8. To what extent do you think the D&D process will impacted PHB’s efforts to institutionalize PHB interventions, or will, in the future, both positively and negatively? Which interventions/objectives of PHB are more likely to be impacted and which ones, not so much? 9. To what extent do you think PHB can take steps to minimize the negative impact? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? Challenges by objective 10. What challenges does PHB face, in achieving its results? Please discuss the challenges they currently face and also those that you anticipate in the future. Objective 1: Objective 2: Scope of challenges 11. Which of these challenges will PHB be able to address, and which ones are likely to remain, and why? Steps taken so far 95 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 12. What steps has PHB taken to address these challenges, and how successful have you been, and what lessons have you learned? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 13. To what extent have advocacy approaches to increase RGC funding for SBC been successful? What approaches could PHB consider, to ensure that this funding is sustained? 14. What challenges do you anticipate in increasing the visibility of SBC in health programs? Lessons learned and knowledge transfer 15. To what extent is the five-step design approach of PHB taken up by NCHP? What can be done to maximize its uptake? Adaptations and readiness of PHB 16. What adaptations should PHB make, to improve uptake by NCHP and other government counterparts? 17. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – PHD Team and OD Office EQHA and PHB Midterm Evaluation Interviewees: Director/deputy, chief of Technical Bureau and PHPU official of PHD; and OD Chief, Chief of Technical Bureau and Health Promotion Official. EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. We would like to talk to you about the USAID projects EQHA and PHB: In what areas has EQHA built the capabilities of the govt health systems and processes in your province/OD? Could you please list those areas? Probe for training in QIC, the use of NQEM finding for service improvement, ongoing coaching, peer-to-peer learning, strengthening capacity of HC on GBV? EQHA Objective 1: Objective 2: 96 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Objective 3: Objective 4: What about PHB? In what ways has its tools and systems been absorbed into the PHPU and at OD level? Probe for tools/systems using to promote healthy behavior changes on the six diseases/focus (FP, WASH, MCH/Nutrition, TB, Malaria). PHB Objective 1: Objective 2: 2. What other approaches from EQHA and PHB be included in your tools and systems? EQHA Objective 1: Objective 2: Objective 3: Objective 4: PHB Objective 2: 3. What strengths are there in EQHA’s and PHB’s interventions, tools and processes that will help in their efforts to institutionalize them? EQHA PHB 4. What weaknesses in these interventions might hamper these efforts? EQHA PHB 5. How confident are you that these capabilities will continue to be used after EQHA and PHB end? What are the reasons for you to say so? EQHA PHB EQ3. What are key challenges in achieving the expected results of EQHA and how to overcome the challenges? Challenges by objective 6. What challenges do EQHA and PHB face? EQHA PHB Scope of challenges 97 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 7. Which of these challenges will EQHA and PHB be able to address, and which ones are likely to remain, and why? Impact of D&D 8. To what extent do you think the D&D process has impacted these efforts, both positively and negatively? Which interventions/objectives of EQHA and PHB have been impacted by the D&D process? Which ones are more likely to be impacted, and which ones are not so much impacted? EQHA PHB 9. To what extent do you think EQHA and PHB can take steps to minimize the negative impact? EQHA PHB Steps taken so far 10. What steps have EQHA PHB taken to address these challenges, and how successful have they been? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 11. Now let’s talk about how EQHA and PHB can work together for better health outcomes. What opportunities are there for both to work together and build synergy, at province and OD levels? Province: OD: Probe for common stakeholders, key player/actors, and other meeting points between EQHA and PHB Adaptations 12. What adaptations should EQHA/PHB take to utilize these opportunities? What challenges and conflicts should they preempt and address? EQHA PHB Lessons from past efforts 13. What efforts have EQHA and PHB already made to work together and build synergy? How successful were these efforts? What was the return on investment? What did they learn from these attempts? 98 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA PHB: Metrics and learning 14. What would you consider as a measure of success in EQHA and PHB working together? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? ONLY FOR PHD TEAM Measures of effectiveness and quality 15. What are the perspectives of private hospitals and providers on quality of services for better health outcomes? 16. What are the perspectives of the private players on the effectiveness of referral linkages? 17. What are the successes, challenges, and lessons from EQHA’s support to the accreditation process, and its engagement of private facilities? 18. What is the level of ownership and participation from the private players and how can this be further improved? 19. What challenges are there in the private sector (hospitals, providers and academic institutions) in providing quality services for better health outcomes? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 20. What challenges do you anticipate in increasing the visibility of SBC in health programs? Adaptations and readiness of PHB 21. What adaptations should PHB make, to improve uptake in your province/OD? 22. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA or PHB, or about the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW: RH (province, OD) EQHA and PHB Midterm Evaluation Interviewees: RH, Province and OD EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 99 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 1. We would like to talk to you about the USAID projects EQHA and PHB. In what areas has EQHA built the capabilities of the systems and processes in your facility? Could you please list those areas? 2. What other approaches from EQHA and PHB be included in your tools and systems? 3. What strengths are there in EQHA’s and PHB’s interventions, tools and processes that will help in their efforts to institutionalize them? 4. What weaknesses in these interventions might hamper these efforts? 5. How confident are you that these capabilities will continue to be used after EQHA and PHB end? What are the reasons for you to say so? Impact of D&D 6. To what extent do you think the D&D process has impacted these efforts, both positively and negatively? Which interventions/objectives of EQHA and PHB? are more likely to be impacted and which ones, not so much? 7. To what extent do you think EQHA and PHB can take steps to minimize the negative impact? EQ3. What are key challenges in achieving the expected results of EQHA and how to overcome the challenges? Challenges by objective 8. What challenges does EQHA face in strengthening health services in your facility? Scope of challenges 9. Which of these challenges will EQHA be able to address, and which ones are likely to remain, and why? Steps taken so far 10. What steps has EQHA taken to address these challenges, and how successful have they been? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? Measures of effectiveness and quality 11. What is the level of ownership and participation from the private facilities and how can this be further improved? 12. What is the level of referral linkages between your hospital and private facilities? What challenges do you face there? 13. We have come to the end of this discussion. Is there anything else you would like to tell us at this time about PHB, or about SBC in the health system in general? Thank the respondents for their time. 100 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV KEY INFORMANT INTERVIEW – PRIVATE SECTOR TWG, Private facilities EQHA and PHB Midterm Evaluation Interviewees: TWG – Private Sector, Private Hospitals and Providers – national and province EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? Measures of effectiveness and quality 1. What advantages and challenges are there in the private sector (hospitals, providers, and academic institutions) in providing quality services for better health outcomes? Private hospitals and clinics Private academic institutions 2. What measures of quality does the private sector use? How do you measure good health outcomes? Effectiveness of EQHA interventions 3. We want to talk to you about EQHA’s engagement of the private sector. How effective, in your opinion, have EQHA’s interventions been so far, in each of these areas? Objective 2: referral linkages, and quality of services Objective 3: Licensing and relicensing 4. What challenges have you faced, or expect to face? Objective 2: referral linkages, and quality of services Objective 3: Licensing and relicensing 5. What lessons have been learned? Incentivizing private sector involvement 6. What can be done further, to increase the value proposition for the private providers in engaging with the system? What incentivizes their involvement? 7. What disincentives does the private sector have? How can EQHA work to minimize these? EQ3. What are key challenges in achieving the expected results of EQHA and how to overcome the challenges? (ONLY FOR PVT SECTOR TWG) Challenges by objective 25. What challenges does EQHA face in strengthening private health services? Scope of challenges 26. Which of these challenges will EQHA be able to address, and which ones are likely to remain, and why? 101 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Steps taken so far 27. What steps has EQHA taken to address these challenges, and how successful have they been? EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 8. In what areas has EQHA already built the quality and capabilities of private institutions? Could you please list those areas? Objective 1: Using the NQEM tool in private facilities, developing QI plans Objective 2: Referral linkages, and quality of services Objective 3: Accreditation process Objective 4: Engaging private academic institutions 9. What other approaches including tools and systems can EQHA plan to include in the private health system? Objective 1: Using the NQEM tool in private facilities, developing QI plans Objective 2: Referral linkages, and quality of services Objective 4: Engaging private academic institutions 10. What strengths are there in EQHA’s interventions, tools and processes that will help in their efforts to institutionalize them? 11. What weaknesses in these interventions might hamper these efforts? 12. How confident are you that these capabilities will continue to be used after EQHA ends? What are the reasons for you to say so? 13. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about EQHA, or about the private health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – D&D FOCAL POINT AT PROVINCIAL LEVEL Interviewees: D&D focal points in Provincial Governor’s Office, Provincial Council member, Staff of One Window Services Office at Provincial Level EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? EQ3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? 1) What are your key roles and responsibilities in implementation of the health functions transfer? In aspects: 102 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV a) Planning: b) Implementation c) Personnel management d) Quality Control e) Monitoring and evaluation 2) What are the challenges in the supporting the process? Could you list down at least 5 such challenges? 3) Could you share with us key strengths and weaknesses of provincial authority in implementing the health functions transfer. 4) What should PHD do more to strengthen the implementation of health functions transferred? 5) Do you have any standard system to monitor health staff’ performance? Is the performance evaluation used for promotion, reward, or punishment of the health staff? 6) What should the provincial authority do more to effectively implement decentralization and de￾concentration in health sector? Please share with us 5 important suggestions from your experience. 7) What kind of cooperation do you have with the EQHA and PHB? How does the cooperation look like? Are you satisfied with this cooperation? Please explain: 8) Do you think the EQHA and PHB’s initiatives to support subnational level in strengthening health system will still be useful even after the EQHA/PHB activities end? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? Measures of effectiveness and quality 9) What are the successes, challenges, and lessons from EQHA’s support to the licensing process, and its engagement of private facilities? 10) We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW: HC Chief EQHA and PHB Midterm Evaluation Participants: HC chief EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. Have you seen improvements in the quality of health services in this facility? Please provide examples. Why or why not? What do you attribute this to? 103 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 2. What would it take for this to continue? How confident are you that these changes will continue? 3. Have you seen improvements in the health behaviors of people in your commune? Why or why not? What do you attribute this to? 4. What would it take for this to continue? How confident are you that these changes will continue? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 5. Some organizations support improvement in health services, and others work to improve the behavior of people. How can these two organizations work together, so that people benefit more than if they worked separately? 6. What will be necessary to make this successful? What challenges exist to them working together? Metrics and learning 7. What would you consider as a measure of success if both organizations work together? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? Measures of effectiveness and quality 8. What are the perspectives of private hospitals and providers on quality of services for better health outcomes? 9. What are the perspectives of the private players on the effectiveness of referral linkages? 10. What challenges are there in the private sector (hospitals, providers, and academic institutions) in providing quality services for better health outcomes? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 1. How can projects support you to effectively carry out behavior change interventions? 2. In your facility? 3. In the community? 4. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. KEY INFORMANT INTERVIEW – Commune chief/CCWC EQHA and PHB Midterm Evaluation 104 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Interviewees: Commune chief and CCWC EQ2. How can EQHA and PHB leverage their respective experiences to institutionalize the Activities’ interventions with the local system in the future? Learning from past and present efforts 1. What recent improvements have you seen in the quality of health services? Probe for improvement in service delivery at health center/post in BB, KCN, TBK? 2. What would it take for this to continue? How confident are you that these changes will continue? 3. What recent improvements have you seen in the health behaviors of people in your commune? Probe for improvement in FP services/awareness at KCN and TBK and WASH situation in TBK? 4. What would it take for this to continue? How confident are you that these changes will continue? Impact of D&D 5. To what extent do you think the D&D process has impacted these efforts, both positively and negatively? 6. To what extent do you think we can take steps to minimize the negative impact? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 7. Some organizations support improvement in health services, and others work to improve the behavior of people. How can these two organizations work together, so that people benefit more than if they worked separately? 8. What challenges exist to them working together? Metrics and learning 9. What would you consider as a measure of success of both such efforts working together? EQ5. How effective have the EQHA approaches been in engaging the private sector in program implementation? Measures of effectiveness and quality 10. What are the perspectives of private hospitals/clinics/providers on quality of services for better health outcomes? 11. What are the perspectives of the private players on the effectiveness of referral linkages? 12. What challenges are there in the private sector (hospitals, providers, and academic institutions) in providing quality services for better health outcomes? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 105 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 13. How can CCWCs use funding for behavior change interventions? 14. What challenges do you anticipate in increasing the visibility of SBC in health programs? 15. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. FOCUS GROUP DISCUSSION – VHSG EQHA and PHB Midterm Evaluation Participants: VHSG members EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 1. Some organizations support improvement in health services, and others work to improve the behavior of people. How can these two organizations work together, so that people benefit more than if they worked separately? 2. What challenges exist to them working together? Lessons from past efforts 1. What has health promotion practice changed in your commune? 2. What has the quality of health services changed in your commune? 3. What efforts have put from some organizations for the enhancement of health service and health promotion practice? Metrics and learning 4. What would you consider as a measure of success if both organizations work together? EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 5. What challenges do you anticipate in increasing the visibility of SBC in health programs? 6. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. FOCUS GROUP DISCUSSION – Mothers 106 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA and PHB Midterm Evaluation Participants: Mothers of children aged 0-2 years EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 1. Let us talk about the care that mothers and children receive in your community, starting with using of a family planning method within six months of delivery? a. How was this practice five years ago, and what is the practice now? b. Why do you think this change happened? c. How can we help more mothers practice this? d. Where do you hear messages or discussions about this practice? Are there any places that you do not currently hear these messages, but think you should? 2. What about washing hands with soap after using the toilet, before feeding the child, after cleaning the child, before preparing food and before eating? a. How was this practice five years ago, and what is the practice now? b. Why do you think this change happened? c. How can we help more mothers practice this? d. Where do you hear messages or discussions about this practice? Are there any places that you do not currently hear these messages, but think you should? 3. What do you think of the health services in the local clinic? a. Have the services changed in the past few years? If so, how have they changed? b. What can we do to make more women like you access these services? c. Can you tell of an experience about accessing a service from the local clinic? What went well, or not so well? Why do you say this? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes but is not limited to stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 4. Some organizations support improvement in health services, and others work to improve the behavior of people. How can these two organizations work together so that people benefit more than if they worked separately? 5. What challenges exist to them working together? 6. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about PHB or about SBC in the health system in general? Thank the respondents for their time. FOCUS GROUP DISCUSSION – WRA EQHA and PHB Midterm Evaluation 107 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Participants: WRA EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 1. Let us talk about the project’s efforts to increase women’s knowledge regarding reproductive health and family planning. a. How was this knowledge five years ago, and what is the level of knowledge now? b. Why do you think this change happened? c. How can we help more men like you learn more about this? d. Where do you hear messages or discussions about this in your community? e. Can you tell us from your own experience of learning about reproductive health and family planning and how it affected you? f. How did the project help you learn more about FP? g. What might be some ways that learning about this topic could be easier? Is there anything that makes it more difficult to learn or talk about this topic? Why do you think this is? 2. What about starting to use a family planning method? a. How was this practice five years ago, and what is the practice now? b. Why do you think this change happened? c. How can we help more women practice family planning if they are married or sexually active? d. Where do you hear messages or discussions about family planning? Are there any places that you think should provide information, but currently do not? Why do you think this? e. Do you discuss with your spouse about FP and childbearing? How has this changed over the past years? f. What challenges do women face in taking up FP? 3. What do you think of the health services in the local health center? a. Have the services changed in the past few years? If so, how have they changed? b. Can you tell us from your own experience of going to the local health center for learning about reproductive health and family planning services? c. Is there anything that keeps you from using these services? What are they? d. How can we work together to remove those barriers? e. What can we do to make more women like you access these services? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 4. Some organizations support improvement in health services, and others work to improve the behavior of people. How can these two organizations work together so that people benefit more than if they worked separately? 5. What challenges exist to them working together? Thank the respondents for their time. 108 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV FOCUS GROUP DISCUSSION – Male Partners of WRA EQHA and PHB Midterm Evaluation Participants: Male partners of WRA EQ6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? Effectiveness of PHB interventions 1. Let us talk about the project’s efforts to increase men’s knowledge regarding reproductive health and family planning. a. How was this knowledge five years ago, and what is the level of knowledge now? b. Why do you think this change happened? c. How can we help more men like you learn more about this? d. Where do you hear messages or discussions about this in your community? e. Can you tell us from your own experience of learning about reproductive health and family planning and how it affected you? f. How did the project help you learn more about FP? g. What might be some ways that learning about this topic could be easier? Is there anything that makes it more difficult to learn or talk about this topic? Why do you think this is? 2. What about starting to use a family planning method? a. How was this practice five years ago, and what is the practice now? b. Why do you think this change happened? c. How can we help more men practice family planning if they are married or sexually active? d. Where do you hear messages or discussions about family planning? Are there any places that you think should provide information, but currently do not? Why do you think this? e. Do you discuss with your spouse about FP and childbearing? How has this changed over the past years? f. What challenges do men face in taking up FP? 3. What do you think of the health services in the local health center? a. Have the services changed in the past few years? If so, how have they changed? b. Can you tell us from your own experience of going to the local health center for learning about reproductive health and family planning services? c. Is there anything that keeps you from using these services? What are they? d. How can we work together to remove those barriers? e. What can we do to make more men like you access these services? EQ1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub￾partners, other USAID activities, and other key partners in order to achieve desired health outcomes. Opportunities for leverage 4. Some organizations support improvement in health services, and others work to improve the behavior of people. How can these two organizations work together, so that people benefit more than if they worked separately? 5. What challenges exist to them working together? 109 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Thank the respondents for their time. 110 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 5: DOCUMENT REVIEW, KII PARTICIPANTS, AND SAMPLING DOCUMENTS REVIEWED: EQHA & PHB Activity Design: EQHA and PHB Cooperative Agreements with USAID, Monitoring, Evaluation, and Learning (MEL) Plans, and strategy documents related to specific interventions of EQHA and PHB. Baselines/Assessments: Assessments, consultations, mapping exercises, and situational analyses conducted by EQHA and PHB. Activity Implementation: EQHA and PHB annual work plans 2018 – 2020, annual and quarterly reports 2018 – 2020, EQHA-PHB quarterly coordination meeting reports, PHB-NCHP coordination meeting reports, EQHA and PHB training material and workshop reports, strategies, manuals, and sector-specific guidelines developed with assistance from EQHA and PHB, such as the Concept Note and Roadmap for Accreditation, HPC Guidelines for CPD, SBCC Practitioner Guideline, ONA and ITOCA reports . National Policy, Strategies, and Other Resources: MoH documents: Health Strategic Plan 3, National Policy for Quality and Safety in Health, Quality Assurance Strategic Plan, health sub-sector action plans, NCHP Strategic Plan (2018-2023), NSSF Policy, National Guidelines for Managing Violence Against Women and Children in the Health System, National Program Plan for SNDD, Workplace Health Policy of the MoLVT, FY 18 Annual Report of USAID ASSIST, Concept Note, and tools related to H-EQIP. ADDITIONAL DOCUMENTS EQHA (undated). Implementing QI in Cambodia’s Public Sector: Experiences and Lessons Learned EQHA (undated). Implementing collaborative QI in Cambodia’s Public Sector: Experiences and Lessons Learned EQHA (undated). Tuberculosis activities as part of USAID EQHA EQHA (Nov 2020). Next steps for QI team (PowerPoint presentation) EQHA (Nov 2019). Concept Note on Accreditation EQHA (Jan 2021). CHAS Manual Final EQHA (March 2020). Private Health Facility Minimum Licensing Standards EQHA (2019). Decision brief on Joint Secretariat of HPCs CCN and GIZ (2017). Final CPD Guidelines for Nurses EQHA (July 2019). Guidelines for Continuing Professional Development EQHA (July 2019). Guidelines for CPD Training Providers, and Application Forms EQHA (April 2020). Landscape Analysis of Inservice Training Providers 111 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV EQHA (Nov 2020). CPD Champions Program EQHA (undated). First Draft Curriculum on Strengthening Gender Integration in Health USAID (undated). Applying Science to Strengthen and Improve Systems (ASSIST) project Country Summaries USAID (Nov 2009). Alva S et al. Measuring the Impact of Health Systems Strengthening. A review of the literature. Department of Health Protection (2014). National Guideline for Managing Violence Against Women and Children in Health Sector system National Action Plan to Prevent Violence Against Women 2019-2023, Approved by the Council of Ministers on 9th October 2020 Policy and Strategic Plan on Gender Mainstreaming in Health Sector 2020-2024 Support for Survivors of Gender-Based Violence (GBV) in Battambang Cambodia, Service Pathway & Service Directory Principle for Gender-Based Violence Response Service Providers and HIV for General Population and Key Population in Battambang Province, Cambodia, Battambang, September 2019 PHB (May 2019). Guidelines for SBC Forum, Draft World Vision (2020). Implementation of Social Accountability Framework PSL (2018). Partnering to Save Lives: Final evaluation report. FHI 360 (undated) Implementation Brief: Community-based Interventions in Cambodia, Mozambique, and Myanmar. Ozano, K. (2016). Evaluation Report Behavior Change Communication Activities in the Northeast of Cambodia. O’Sullivan, G. A., Yonkler, J. A., Morgan, W., & Merritt, A. P A. (2003). Field Guide to Designing a Health Communication Strategy, Baltimore. MD: Johns Hopkins Bloomberg School of Public Health/Center for Communication Programs. Health Communication Capacity Collaborative (n.d.). Social and Behavior Change Communication Saves Lives. Lhachimi, S., Bala, M. & Vanaga, G. (2016) Evidence-based Public Health. Biomed Res Int. DOI: 10.1155/2016/5681409 C-Change (2012). C-modules: A learning Package for Social and Behavior Change Communication (SBCC). Washington, DC. Compass (n.d.). How to Develop a Communication Strategy. Available at How to Develop a Communication Strategy | The Compass for SBC 112 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV U.S. Department of Health and Human Services, National Cancer Institute (2005). Theory at a Glance: A Guide for Health Promotion Practice. Health Communication Capacity Collaborative (2017). Integrated Social and Behaviour Change Communication Programs Implementation Kit. PSI Cambodia (undated). Standard Operating Procedure (SoP) for PHB: Management Information System (MIS). PACT and PSI Cambodia (2018). Political Economy Dynamics of Social and Behavior Change (SBC) Coordination in Cambodia: Recommendations for the Promoting Healthy Behaviors Project. NCHP (2020). Communication Strategy for Behaviour Change: Intervention on clean hand and safe eating. NCHP (2017). Draft version of Strategic Plan 2018-2022. PHB (2019). NCHP Integrated Technical and Organizational Capacity Assessment. Phnom Penh, Cambodia. PHB (undated). High Responsibility for Health Lives: PHB gender analysis & action plan, USAID Promoting Healthy Behaviour Project (PHB) Pact (2018). Government Performance Index (GPI), a handbook NHCP and PHB Coordination Meeting Minutes. NCHP (2020). Draft version of SBC Practitioner Guide. Reports and Meeting Minute of Development SBC Practitioner Guide 17 Triggers (undated) NCHP Brand Visibility: Strategic Recommendation. PHB (undated) Draft Version of Network Analysis: Initial Findings. Documents related to Formative Study of Forest-Gers’ Malaria Prevention and Care-Seeking Behaviors. PHB (undated). IEC Materials, guides and reports related to Family Planning, Wash and Covid-19 KII PARTICIPANTS Senior Officials, DHS, MoH Senior Official, HRDD, MoH Senior Officials, NCHP, MoH Senior Official, CNM, MoH Senior Official, NMCHC, MoH Senior Officials, CENAT, MoH Senior Officials, NCHADS, MoH 113 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Representatives, CCN, CMC, PCC, MoH Senior Official, QI TWG, MoH Senior Official, Preservice TWG, MoH Senior Official, Private Sector TWG, MoH Senior Official, GMAG, MoH Senior Officials, MRD Senior Officials, MoLVT Senior Staff, UNFPA Senior Staff, WB H-EQIP Senior Staff, GIZ Social Health Protection Project Senior Staff, International Development Enterprises Senior Staff, Marie Stopes International Senior Staff, WB Cambodia Nutrition Program Senior Staff, GIZ, SBC Program Senior Staff, Save the Children Senior Staff, Cambodia Malaria Eradication Program, University Research Co. Senior Manager, Puthisatra University Senior Staff, Sunrise Hospital Senior Officials, USAID/Cambodia Senior Management Team, Policy and Programs Team, Regulations Team, Private Sector Team, QI Team, QI Coordinators and Preservice Team, EQHA Staff of I-TECH, Meridian, mClinica, PSI, Senior Staff of 17 Triggers, Pact, PC, Khamera, AHEAD PHD Teams, Battambang, Kampong Chhnang and Tbong Khmum Health Managers of RH, Kampong Chhnang, Boribo, Ponhea Krek, Memut, and Mong Russey Senior Staff of RTC, Battambang and Kampong Cham Private providers in Battambang, Kampong Chhnang 114 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV OD Teams at Ponhea Krek, Memut, Mong Russey, Kampong Chhnang, Boribo, and Thmar Kol Chiefs of Communes in the above ODs Chiefs of HCs in the above ODs FINAL SAMPLE ACHIEVED CATEGORIES # KIIS PLANNED # KIIS ACHIEVED # PARTICIPANTS National – KIIs MoH Departments – EQHA 5 3 5 MoH TWGs Chairs – EQHA 2 2 2 MoH TWGs – PHB 1 0 0 MoH TWGs EQHA & PHB 1 1 2 MoH National Centers – PHB 1 1 3 MoH National Centers – EQHA & PHB 4 4 6 MoLVT – EQHA 1 1 3 MRD – PHB 1 1 3 HPCs – EQHA 5 4 6 Partners – EQHA 3 3 3 Partners – PHB* 3 6 8 Academic Institutions – EQHA 1 1 1 Private Hospital 1 1 2 IP Teams – EQHA 5 5 25 IP Teams – PHB 1 2 11 Sub-awardees – EQHA 4 4 4 Sub-awardees – PHB 5 5 13 USAID – EQHA 1 1 1 USAID – PHB 1 1 2 Sub-Totals National KIIs 46 46 100 Provinces PHD Team – EQHA & PHB 3 3 11 D&D – EQHA & PHB 3 2 2 RHs – EQHA 2 1 2 115 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV CATEGORIES # KIIS PLANNED # KIIS ACHIEVED # PARTICIPANTS Private Facilities – EQHA 6 4 4 RTCs – EQHA 2 2 4 IP Team – EQHA 3 1 5 Sub-awardees – PHB 3 2 2 Sub-Totals Province KIIs 22 15 30 ODs and Communes OD Team – EQHA & PHB 6 6 21 RHs – EQHA 5 4 14 Commune & CCWC – EQHA & PHB 6 6 12 HC – EQHA & PHB 6 6 6 Sub-Totals OD and Commune KIIs 23 22 53 Total – KIIs 91 83 183 FGD CATEGORIES # FGDS PLANNED # FGDS ACHIEVED # PARTICIPANTS MEN WOMEN Women of Reproductive Age 4 4 0 13 Male Partners of WRA 4 4 12 0 Mothers of Children Under Two Years 2 2 0 8 VHSG Members 6 6 8 12 Totals - FGDs 16 16 20 33 *This included members of the Technical Consultative Group of PHB 116 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 6: CONFLICT OF INTEREST DISCLOSURES 117 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 118 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 119 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 120 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 121 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 122 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 123 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 124 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 125 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 7: TEAM MEMBER CVS BEULAH JAYAKUMAR Team Leader Beulah Jayakumar is a global health consultant with 20 years’ experience in providing strategic technical leadership for high-quality programming, and monitoring, evaluation and learning in health and development. She led the teams that conducted the midterm performance evaluation of a reproductive, maternal, newborn and child health (RMNCH) grant from USAID/India (2018) and the final performance evaluations of two Challenge TB grants from USAID/Ethiopia and USAID/Burma (2019-20), and several district-level RMNCH grant-funded and private-funded programs in India, Bangladesh, Afghanistan, Cambodia and South Sudan. Key responsibilities in these assignments included leading the evaluation team in designing the evaluation and writing the inception report, drafting data collection tools, overseeing primary data collection, review of secondary data, and coordinating data processing and analysis. She also served as the point of contact for the client and other stakeholders and led the presentation of preliminary findings and the drafting of the final report. These evaluations aimed to assess relevance of implementation strategies and their contribution to strengthening national, district and community health systems, and identified promising practices for improving key aspects of these systems such as health worker competencies and supportive supervision mechanisms. She also led the evaluation of the global health programs of an international NGO (2014) focused on the technical content, delivery approaches and effective scale of the programs as well as organizational capacity, in order to strategically invest resources for superior returns. She is skilled in the design of social behavior change interventions using formative tools such as Barrier Analysis and Trials for Improved Practice. While employed with World Vision, she designed Timed and Targeted Counseling (TTC), a household behavior change model focused on the first 1,000 days of life. TTC was later adopted as a core program model in World Vision and is currently implemented in its programs in over 20 countries. She has served as a lead facilitator for TTC for several country teams including Cambodia. In 2013, she documented the intervention fidelity, effect modifiers and confounding factors in a four￾country study that assessed the impact of a behavior change intervention on child health and nutrition outcomes, including in Cambodia. PROFESSIONAL EXPERIENCE Oct. 2010 – Present Freelance consultant in public health Mixed methods evaluations and assessments − Led the final performance evaluation of Challenge TB Myanmar, a US$ 15 million five-year activity funded by USAID/Burma, Social Impact, Inc. (October 2019 – March 2020) − Led the final evaluation of a four-year MNCH project in Kratie Province, Cambodia, Samaritan’s Purse International (February 2020) − Led the mid-term evaluation of a three-year MNCH project in north Bangladesh, World Renew, Inc. (September – October 2019) − Led the final performance evaluation of Challenge TB Ethiopia, a US$ 50 million, five-year activity, funded by USAID, Social Impact, Inc. (January – May 2019) − Led the evaluation of an UKAid-funded Adolescent sexual and reproductive health project in north Bangladesh (May 2018) 126 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV − Led a mid-term performance evaluation of the four year, US$ 24 million Scaling up RMNCH+A Project, USAID/India (February – April 2018) − Assessed the effectiveness of Global Fund-supported programs in TB, Malaria and HIV in Ethiopia (June – July 2017) − Led an evaluation of the global health and nutrition programming in 10 countries (with a focus on stunting reduction) and global organizational/partners’ capacity World Renew (November 2014 – February 2015) − Evaluated a multi-site HIV care and support program, EFICOR, India (December 2014) − Led the end-line evaluation of a USAID-funded child survival and nutrition project in Afghanistan, World Vision (January 2013) − Led the mid-term evaluation of a USAID-funded child survival project in South Sudan, WV (September 2012) − Participated as technical lead in the end-line evaluation of the Market Based Partnerships for Health Project, funded by USAID/India, Abt Associates (April 2012) − Conducted a Health Facility Assessment for a USAID-funded child survival project in Jharkhand, India, World Renew (May 2012) − Participated in audit of all Global Fund grants to Namibia from the Office of the Inspector General, the Global Fund (June – July 2011) and Ethiopia (November 2010 – January 2011) − Led the end-line evaluation an MNCH project in two districts, one each in Bihar and Jharkhand, India, Emmanuel Hospital Association (July 2010) Proposal development − Led the development of a proposal to USAID on COVID-19 response in Malawi, World Renew (October 2020) − Led the development of grant proposals on adolescent health and MNCH for Nigeria and Uganda, World Renew (August 2019) − Led a workshop on Theory of Change for WV India (February 2018) − Led the development of a large grant proposal for Nepal, WV US (June 2017) − Led a workshop on proposal development for WV India (September 2016) − Developed the technical piece of AIM Health Plus, an MNCH project of WV Ireland in four countries in East and West Africa, Irish Aid (July 2016) − Was the principal writer for a large USAID grant for malaria in Myanmar, WV US (March – April 2016) − Developed two proposals for submission to the Global Fund’s New Funding Model for HIV and TB, the Union (April 2014) − Developed a technical application in response to USAID/India on Improving Quality of Health Services, in six states of India, Catholic Health Association of India (October 2013) − Co-wrote a proposal in response to an RFP from USAID/Tanzania, integrating nutrition and agriculture interventions, WV United States (May 2011) − Developed the technical application in response to an RFA from USAID/Child Survival and Health Grants Program for South Sudan, WV United States (March 2010) 127 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Quantitative surveys – design and training − Virtual training and coaching a multi-country team to understand, design and implement Lot Quality Assurance Sampling surveys, World Renew Inc. (July 2020 - present) − Led a multi-site household survey using LQAS methodology for district-level MNCH projects located in 10 sites across Kenya, Tanzania, Uganda, Sierra Leone, and Mauritania, WV Ireland and Irish Aid (May – October 2014) − Conducted a household survey for a USAID Child Survival Project, in South Sudan, WV United States (August – September 2011) − Trained staff in LQAS methodology, EFICOR, India (November 2009 and August 2011) Qualitative research and documentation − Principal investigator for a qualitative study on gender barriers to accessing care for visceral leishmaniasis in highly endemic districts in India, KalaCORE, Mott MacDonald. (October 2017 – August 2018) Published in July 2019: https://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0007457 − Documented intervention fidelity, effect modifiers and confounding factors in the Child Health and Nutrition Impact Study (CHNIS) in Cambodia, Zambia, Kenya, and Guatemala, WV International (April – November 2015) − Documented an operational research project on improving Infant and Young Child Feeding using the timed and targeted counseling (TTC) model in Ethiopia, WV (October 2012) − Co-wrote a paper titled “Combating Antibiotic Resistance: A Call to Action” (published in the Health e-book for the G-20 meet), O3i Global (November 2011) − Wrote a discussion paper on Integrating MNCH Interventions within Global Fund-supported programs, WV International (August 2010) Developing program strategy, formative, and operations research − Technical assistance to program design and implementation for a five-year, five-country MNCH program, World Renew (October 2019 – September 2020) − Facilitated Barrier Analysis of an MNCH project in Nigeria and Malawi, World Renew, (August – September 2018) − Facilitated Barrier Analysis on accessing nutrition and MNCH care in four program countries, WV Ireland (January – March 2017) − Facilitated the designing of an MNCH program structure, logic models and performance evaluation in three countries, World Renew (December 2015 – December 2016). − Led the design and implementation of a national TB civil society program in India, funded by USAID (April 2014) − Conducted qualitative research with key stakeholders in 10 countries and developed a multisectoral strategy to address child stunting, World Renew Asia Regional Office, (October 2013) − Trained managers and partners of a national NGO in the design and monitoring of HIV programs, New Delhi, funded by World Renew (June 2012) − Conducted barrier analysis as part of formative research in India, EFICOR (May 2014), Niger, World Renew (May 2016) Apr. 2004 – Oct. 2009 Associate Director/National Coordinator, Public Health, World Vision, India ● Designed generic logical frameworks for health, M&E systems, implementation protocol and guidance for budgets for Area Development Programs (ADPs) ● Developed and tested the TTC model, now being rolled out across World Vision globally ● Designed and participated in mixed methods evaluations of USAID projects ● Designed, executed and analyzed population surveys and qualitative studies ● Developed training manuals and curricula, and trained trainers in MNCH 128 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Jan. 2001 – Mar. 2004 Program Manager, World Vision India Interim Project Director, TB ACSM Project, in 80 districts across 14 states, with staff strength of 35 (October 2008 – April 2009). Provided technical and managerial oversight to project core team and seven sub-recipients in implementing the project in 80 districts (population 176 million). Designed the M&E systems and work plan. Served as key liaison with MOH (TB division) and USAID. Manager, Expanded Impact Child Survival Project in three districts of Uttar Pradesh, covering a total population of 3.4 million and a staff team of 32 (April 2004 – February 2005) Nutrition Jumpstart Initiative in 26 ADPs (January – September 2009) EDUCATION DNB — Family Medicine, National Board of Examinations, New Delhi, 1997 MBBS —Medicine and Surgery, Bharatiar University, 1993 LANGUAGE ENGLISH – Fluent, HINDI – Fluent, TAMIL – Fluent 129 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV SAKONY PEN Social Behavioral Change Advisor Sakony PEN is a Social Behavior Change Advisor for the Mid-term evaluation of EQHA and PHB activity. Prior to joining the Social Impact, Sakony worked in several positions in road safety, research and consultant. She managed the Research Monitoring and Evaluation project at Handicap International (HI) involving surveys related to road user behaviors- health wearing, drunk driving and speeding in Cambodia. She supervised and conducted the quality control for data collection for the research on Long-term health, economic and social impact of injury in low-and middle-income countries for Johns Hopkins University (JHU) in 2016 funded through HI. From 2014-2015, Sakony served as Road Safety Specialist under the consultancy of the Rural Road Improvement Project I (RRIP I) funded by MRD-ADB, to provide technical advisor on social behavior change on road safety for the community-based road safety activity (CBR) and support the team leader to conduct the baseline, mid-term, end-line evaluation, and capitalization documents. To fulfill the requirement of master degree in public health at the Queensland University of Technology (QUT), Australia, she conducted a qualitative study on “the Adaptation of Australian Best Practices on Speed Management to the Cambodian Context” which was published by the Australian College of Road Safety. At the same time, Sakony worked as a research officer and a research assistant for short-term projects for the Centre for Accident Research and Road Safety-Queensland (CARRS-Q), QUT. Most recently as a Rehabilitation Quality Specialist, she has conducted behavioral needs assessments for persons with disabilities (PWD) during Covid-19 and for rehabilitation services for patients with cerebral palsy, and conducted the lesson-learned of tele￾rehabilitation under the under “Rehabilitation Accessibility promotion for an Inclusive and Supportive Environment for All-RAISE project.” Sakony holds a master’s degree in Public Health with a major in Health Promotion from Queensland University of Technology, Australia. She has also extensive knowledge and experience in behavior change theories, project planning, research, monitoring, and evaluation related to public health. PROFESSIONAL EXPERIENCE Oct. 2010 – Present Rehabilitation Quality Specialist, Humanity and Inclusion (HI), Phnom Penh, Cambodia ● Support international consultants to develop the financial strategy for Physical Rehabilitation Centre (PRCs) managed by Persons with Disabilities Foundation (PWDF). ● Lead to conduct surveys related to persons with disabilities and a study to validate the poverty assessment tool. ● Lead research projects and lesson-learned studies related to COVID-19 and disabilities. ● Ensure the compliance of national standards, including the minimum package of services and physiotherapy (PT) standards in order to guarantee the quality of services amongst organization’s staff and partners. ● Strengthen Physical Rehabilitation Centre (PRC)’s managers and PWDF staff on leadership, management, and governance skills in support of the implementation of quality assurance at the PRC through the Rehabilitation Management System (RMS) tool. ● Develop, review, and implement cost-calculation, and cost allocation tools at PRC in order to support financial planning of the rehabilitation sector. ● Supervise the implementation of HI’s safeguarding policies in PRCs as well as its obligation toward CoWaterSogema. ● Support PWDF to develop and monitor rehabilitation action plan of PRCs. ● July 2019 – Nov. 2018 Research Officer, Queensland University of Technology, Australia 130 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Supported a research fellow to build data of the program which is involved in managing data, merging data, and verifying data. ● Processing data for data analysis using various software such as Excel, SPSS, and ArcGIS. ● Supported a research fellow to conduct literature review related to journal articles and Queensland Road Safety Policies and Practices for the evaluation of the policy impact. Feb. 2019 – Mar. 2019 Public Health and Social Work Peer Mentor, Queensland University of Technology, Australia ● Provided support to new international students on how to have access to QUT learning facilities and services. ● Assisted new international students to understand the expectation of learning from both units and the university. ● Shared successful practical experience in learning and techniques to handle study workload effectively. ● Aug. 2014 – Oct. 2016 Deputy Road Safety Program Manager, Handicap International, Cambodia ● Participated in the development of an overall road safety program strategy, review, proposals, and reports. ● Supported commune councils to develop road safety action plans, taking into consideration the needs of society and best use of local resources. ● Conducted needs assessments, involving community participation, and delivered a series of capacity building sessions. ● Supported commune authorities to conduct road safety awareness and facilitate training and awareness sessions, including monitoring activities. ● Worked with road safety core-working group at the national level to advocate for the approval of new Cambodian Land Traffic law. ● Developed project log frame, work plan, monitoring tools, including baseline/post surveys, and project indicators. ● Prepared contracts for service provision and/or grant agreement and advised the sub-grant implementation. ● Built capacity of partners, commune authorities, youth, and school teachers. ● Supported an international consultant and a research team to conduct quantitative and qualitative data collection for the mid-term and final evaluation of a project. ● Participated in creating a research protocol and methodology, questionnaire, data analysis and report writing for mid-term review and final evaluation project. ● Developed project lesson learned papers and capitalization documents ● Developed weekly, monthly, quarterly and annual reports for donors. Sep. 2014 – Dec. 2015 National Road Safety Consultant, Ministry of Rural Development, Cambodia (Under ADB fund) ● Be a focal point and liaison with the Ministry of Rural Development (MRD) and Sweroad for Community-Based Road Safety (CBRS). ● Worked with an international team leader to design, implement, and monitor including budget plans CBRS program using participatory approaches. ● Monitored and provided technical support to team members to ensure that the high quality of program implementation meets the needs and requirements of MRD and ADB. ● Supported team members to implement CBRS program in collaboration with government officials from MRD, provincial and commune level, ensuring that the program implementation complies ADB, MRD, and Handicap International policies and strategies. 131 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Supported commune chiefs to develop annual road safety action plan which aligned with Commune Investment Plan (CIP) and monitored the commune small grants for implementing road safety activities. ● Developed road safety training manuals, curriculums, awareness materials, and tools for awareness and education programs. ● Developed community-based public awareness campaigns to increase road users’ knowledge in target areas. ● Contributed to the development of Safe School Zone (SSZ) project for CBRS and successfully led the implementation of SSZ project. ● Contributed to the development of survey protocol for baseline and end-line data collection as well as mid-term review program strategies. ● Developed capitalization documents on CBRS achievements and lessons learned and shared successful approaches to regional road safety program. ● Developed weekly, monthly, and quarterly reports with high-quality standards to fulfill ADB’s format and requirements. ● Conducted training of trainers for government officials from MRD, Provincial Department of Rural Development, commune chiefs, and school teachers. ● Worked with team members to collect program data, case studies, and relevant information for program monitoring and evaluation as well as donor reports. Nov. 2012 – July 2014 Awareness and Education Project Manager, Handicap International Belgium, Cambodia ● Developed tools and resources for the project implementation. ● Be Responsible for the development of road safety training curriculums and IEC materials to raise awareness of key road safety issues among civil society stakeholders, local NGOs, and private businesses. ● Conducted public awareness campaigns on high-risk factors along rural roads. ● Established youth groups at universities to support them in the advocacy for policy change on road safety at universities. ● Organized training of trainers to the RS Program’s Partner organizations, Road Safety Support Groups, Commune authorities, and school teachers on Road Safety. ● Supported partner staff to define awareness strategy and project framework, conduct pre/post surveys, and deliver training sessions to agencies in a road safety network. ● Planned and monitored budget monthly, quarterly, and yearly. Nov. 2011 – Nov 2012 Support to Civil Society Officer, Handicap International Belgium, Cambodia ● Conducted training needs an assessment on road safety with government officials, local NGOs, and private sectors to identify the capacity training needs, strengths, and weaknesses to enhance road safety competencies and capacities through developing training manuals. ● Developed training tools and IEC materials for the project. ● Provided training of trainers for government officials, local NGOs, and private sectors on road safety. ● Assisted the coordinator and relevant project managers in developing and monitoring the budget plan, narrative/financial reports, and proposals. ● Facilitated and organized road safety workshops and conferences. July 2008 – Oct 2011 Human Resource Officer, Handicap International Belgium, Cambodia ● Promoted and enforced internal regulations and policies in the organization. 132 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Provided advice to solve staff issues related to policies and benefits. ● Assisted the Human Resource Manager to develop and update policies and organize recruitment. ● Supported the Human Resource Manager and Project Managers to identify strengths and weaknesses of employees through annual appraisal. ● Assisted human resource manager to develop training plans to enhance staff’s capacity according to training needs. Dec. 2007 – June 2008 Receptionist and Administration Assistant, Handicap International Belgium, Cambodia ● Assisted Human Resource Manager to process entry visa and visa extension for international staffs including their dependents to work in Cambodia. ● Monitored and followed up staffs’ annual leave, sick leave, and special leave. ● Managed flight booking and processed visa for Cambodian staffs for their abroad mission. April 2006 – Dec. 2007 Administrator, GFA Consulting Group, Land Management, Flood Management and Mitigation Program, MRC ● Controlled all project expenses and ensured each invoice must be signed by a team leader. ● Managed procurements for all project materials. ● Prepared financial report and submitted to headquarters in Germany. ● Supported project staff to organize workshops/training to project partners and government staff. EDUCATION MPH — Health Promotion, Queensland University of Technology, 2020 B.Ed. — Teaching English as a Foreign Language, The Royal University of Phnom Penh, 2011 B.A. — Hotel and Tourism Management, Norton University, 2004 LANGUAGE ENGLISH – Fluent 133 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV AMRY OK Clinical Health Specialist Dr. Ok Amry holds a Master’s Degree in management (English based – International Program) from Pannasastra University of Cambodia (PUC) and a diploma of medical doctor from University of Health Sciences (UHS) and have more than 15 years of responsible working experience in project management, project monitoring and evaluation (M&E) and social research, and providing capacity building training. During the last two years, he worked as a Health/Nutrition Expert to provide technical support for GIZ Cambodia on a high-profile project: “Multi-sectoral Food and Nutrition Security (MUSEFO), in which the main responsibility was to ensure the effectiveness of the project implementation, with tasks including monitoring capacity building training conducted by master trainers to their respective health center staff, monitoring actual coaching session conducted by master trainers at health center level, as well as monitoring the implementation of Local Subsidy Agreement (LSA) with two provincial health departments (Kampot and Kampong Thom PHDs). Moreover, he worked for Food and Agriculture Organization of the United Nations (FAO) for one year on an intermittent basis to coordinate a multi-country study project to refine the Minimum Dietary Diversity for Women (MDD￾W) data collection tool, with responsibilities including ensuring effective communication between FAO, Project Advisory Group and national research partner in relation to the coordination, planning, implementation, monitoring and reporting results of the project. Besides these, during the course of his more than 12-year consultancy job, he has involved in more than 50 different assignments, mainly in the field of public health, nutrition, reproductive health and child protection, including evaluation missions, baseline surveys, mid-term reviews and team leader of many assignments. He successfully wrote 15 research reports and contributed to around 30 research reports, all in English. Prior to working as a consultant, he managed a USAID-funded Program at the Youth Council of Cambodia (YCC), where he was responsible for ensuring the quality of YCC education and training program to promote life skill, reproductive health, HIV/AIDS, among others, to both in-school and dropout youth, for a period of more than five years, where he gained in-depth experience in project management/supervision and monitoring and evaluation. Sep. 2020 – Jan. 2021 National Evaluator, UNICEF/LATTANZIO KIBS, Cambodia ● Country-led Evaluation of the Action Plan to Prevent and Respond to Violence Against Children 2017- 2021 in Cambodia, for UNICEF. The focus of this position is on properly integrating methodological and content related expertise about child protection aspects in the Cambodian context. Involved in the inception phase, contributing to the design of the framework of the evaluation, including the analysis of the ToC and definition of the ToA, and the preparation of checklists and guidelines for qualitative data collections. An active actor in the data collection phase, at both central and province level, conducting key informants’ interviews and group discussions. Provided inputs, revisions, advice and contributed to the drafting of all the deliverables. Took part in the kickoff meeting, in the presentations of the deliverables and in others meetings with UNICEF, as deemed necessary. Aug. 2020 National Consultant, Asian Development Bank, Cambodia ● Worked as a resource person to administer two different surveys: STEM survey with teachers and a survey with school leaders. The aim is to map the current situation of STEM education in Cambodia by getting feedback from STEM subject teachers as well as ascertain the views of school leaders on aspects of the school ecosystem. Mar. 2020 – May 2020 National Consultant, UNOPS/SBK Research and Development, Cambodia 134 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Cambodia Malaria Survey 2020: Coordinated the field data collection of the Cambodia Malaria Survey 2020 for United Nation Office for Project Services (UNOPS) and National Center for Parasitology, Entomology and Malaria Control (CNM), funded by Global Fund. Jan. 2020 – Feb. 2020 Evaluation Consultant, Save the Children/SBK Research and Development, Cambodia ● Endline SURVEY of The Early Childhood Care and Development (ECCD) for Floating Villages project of Save the Children, funded through a grant under the World Bank-administered Japan Social Development Fund (JSDF), implemented over a period of three years (2015 - 2018). Aug. 2019 – Dec. 2019 National Project Coordinator, FAO, Cambodia ● Coordinated a multi-country study of Food and Agriculture Organization of the United Nations (FAO) to refine the Minimum Dietary Diversity for Women (MDD-W) data collection tool. Key responsibilities including: o Led and coordinated capacity development activities, pre-test and pilot study; o Provided technical oversight on all research activities, including sampling, data collection, management and monitoring, ensuring that activities are implemented according to the project document and research protocol; o Led the management of MDD-W data collected, in close collaboration with the team leader from the national research partner, namely Royal University of Agriculture (RUA); o Contributed to drafting technical reports, guidelines, recommendations, and scientific articles directly related to this project. July 2019 – Aug. 2019 Lead Consultant, Plan International/SBK Research and Development, Cambodia ● Baseline Survey of 1000 Days Project: “Food Security and Increasing Resilience to Improve Mother and Child Health in 51 Villages in Stung Treng” of Plan International Cambodia. Apr. 2019 – July 2019 Lead Consultant, Ministry of Health/ADB/SBK Research and Development, Cambodia ● Baseline Survey for the Ministry of Health (MoH) in three provinces of Cambodia Greater Mekong Sub-region (GMS), ADB-funded Health Security Project. Feb. 2019 – May 2019 Lead Consultant, GIZ/SBK Research and Development, Cambodia ● Follow-up Survey (FUS) for GIZ project: “Multi-sectoral Food and Nutrition Security (MUSEFO) of GIZ in Cambodia, which was part of the global BMZ special initiative “ONE WORLD no hunger.” Nov. 2018 – May 2019 Save the Children/SBK Research and Development, Cambodia ● Endline Survey of NOURISH Project of Save the Children Cambodia with funding from USAID and the U.S. Government’s Global Hunger and Food Security Initiative Feed the Future. Sep. 2018 – Oct. 2018 Lead Consultant, GIZ, Cambodia ● Output Assessment Survey of GIZ’s MUSEFO, “Multi-sectoral Food and Nutrition Security,” which was part of the global BMZ special initiative “ONE WORLD no hunger.” Aug. 2018 – Oct. 2018 Team Leader, National Committee for Disaster Management/ADB/SBK Research and Development, Cambodia ● Endline Survey for the Project: Community Based Disaster Risk Reduction (CDRR), under ADB grant, National Committee for Disaster Management Secretariat (NCDM). 135 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Mar. 2018 – Sep. 2018 Evaluator, Children Future International, Cambodia ● Evaluation of Life Skill Training Program: “Next Generation Initiative (NGI)” of Children Future International (CFI) in Battambang Province. Feb. 2018 – Apr. 2018 Lead Consultant, Plan International/SBK Research and Development, Cambodia ● Baseline Study of THRIVE Project: “Promoting A Healthy Childhood Through Quality ECCD, Nutrition and WASH in Northeast, Cambodia,” of Plan International Cambodia. Dec. 2017 – Feb. 2018 Lead Consultant, Save the Children/SBK Research and Development, Cambodia ● Mid-term Survey: of NOURISH Project of Save the Children Cambodia with funding from USAID and the U.S. Government’s Global Hunger and Food Security Initiative Feed the Future. Dec. 2017 – Jan. 2018 Key National Evaluation Expert, EuroPlus Consulting & Management, s.r.o/Czech Republic Development Cooperation, Cambodia ● Ex-post evaluation of the Czech Republic development Cooperation Project in Cambodia on the project “Social Care and Inclusive Education for Children with Disabilities (SCIE),” which was implemented by Catholic Relief Services (CRS) in partnership with Cambodia Development Mission Disability (CDMD), coordinated by Caritas Czech Republic (CCR) and financed by Czech government, in 18 target schools in three districts (Bati, Prey Kabbas, and Samrong) in Takeo Province. Nov. 2017 – Dec. 2017 Consultant, GIZ, Cambodia ● Coordinated a Farm Competition for farmers participating in the GIZ’s MUSEFO program in Kampot and Kampong Thom provinces. “Multi-sectoral Food and Nutrition Security, (MUSEFO)” was part of the global BMZ special initiative “ONE WORLD no hunger.” Sep. 2017 – Oct. 2017 Lead Consultant, Plan International/SBK Research and Development, Cambodia ● Baseline Survey: Improved Mother and Child Health and Income Generation in 45 villages in Rattanak Kiri Province, Cambodia, of Plan International Cambodia. July 2017 – Nov. 2017 National Evaluator, UNICEF/LATTANZIO Group, Cambodia ● Outcome Evaluation of Education Capacity Development Partnership Fund (CDPF), Phase I and Phase II: The CDPF is a multi-donor trust fund, supported by the European Union (EU), the Swedish International Development Cooperation Agency (SIDA) and UNICEF. The fund was established to specifically support the capacity development objectives of the MOEYS Education Strategic Plan (ESP). The National Evaluator (NE) acted as a National Expert, providing information and advises on the Cambodian socio-economic context, insight and a local perspective in analyzing the Cambodian education sector. The NE was involved in the inception visits, contributed to both methodological issues and the accuracy of the contents, taking care of the organizational and logistic part as well. The responsibilities also included facilitating communication with the stakeholders, data collection, and analysis and organizational issues. EDUCATION M.A. — Management, Pannasastra University of Cambodia, 2013 M.D. — Medical Doctor, University of Health Sciences, 2010 B.A. — Health Sciences, University of Health Sciences, 2003 136 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV 137 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV LANGUAGE KHMER – Fluent, ENGLISH – Fluent, FRENCH – Intermediate 138 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV SOVANNARITH EM Health Systems Strengthening Advisor Nov. 2020 – Dec. 2020 Evaluation Consultant, Comic Relief UK, Myanmar/Lao/Cambodia ● Evaluation consultant to Comic Relief UK, for the malaria and health system project in Myanmar, Lao and Cambodia. 2018 – 2019 Monitoring Consultant, ADB, Cambodia ● Consultant to ADB for the monitoring and evaluation of the project of infectious diseases. Oct. 2018 – Dec. 2018 Evaluation Consultant, Zcech Development Agency, Cambodia ● Consultant to Zcech Development Agency to assess the health system in Cambodia. Mar. 2018 – May 2018 Qualitative Health System Researcher, Palladium/USAID, Cambodia ● Consultant to support URC/USAID funding project to conduct the study of the health system information management in six provinces of Cambodia in which to assess the OD, HC and Community level. Nov. 2017 – Dec. 2017 H-Equip Consultant, GFA/World Bank, Cambodia ● Consultant Team Leader to design H-Equip health system assessment tools for PHD, PRH, OD, HC facilities. Sep. 2017 – Oct. 2017 Study Team Leader, ADB, Preah Vihear ● To conduct the end-line survey of Malaria Project which has been implemented by MoH funded by ADB in Preah Vihear, on the KAP survey, serology and blood testing among high￾risk population and mobile population. Jan 2011 – July 2016 Health Programme Auditor, GFA/WB, Cambodia ● Auditing consultant to support the WB financing to the ministry of health for the project of Health Sector Support Project 2 (HSSP2) – the main purposes are to design the quarterly assessment tools and applied it to assess all health system levels including PHD, OD, PRH, DH, and HC. Jan 2016 – May 2016 Study Team Leader, GIZ, Cambodia ● To support GIZ to conduct the anthropology and Life Span among children aged 0 month to 72 months in Siem Reap and Kampong Thom Provinces. June 2015 – July 2015 Study Team Leader, UNOPS, Cambodia ● To evaluate Global Fund RAI Project, which has been implemented by FHI/UNOPS in the western Thai-Cambodia border for the RAI’s related intervention (qualitative method) Dec. 2014 Evaluation Consultant, UNFPA Country Office, Cambodia ● The objective was to conduct an in-depth analysis of the Country Program 2011-2015 Evaluation Focusing on The Maternal, Reproductive Health programme in Cambodia to assess the country program related to Reproductive Health of UNFPA support to Cambodia. Aug. 2014 – Sep. 2014 Study Team Leader, Denmark Red Cross, Cambodia 139 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● The objective was to conduct a mid-term evaluation regarding Water and Sanitation and Maternal/Child health interventions in two provinces of Kampot and Preah Vihear. (1,200 sample size) Feb. 2014 Study Team Leader, CARE, Cambodia ● The objective was to conduct an end-line Country Program evaluation pertaining to Maternal and Child Health, Nutrition Program funded by GFATM to evaluate the project related to CARE activities of Maternal Child Care financed by GFATM Oct. 2013 – Dec. 2013 Study Team Leader, London Malaria Consortium/UNOPS, Cambodia ● The objective was to support the National Malaria Baseline Study 2013. The main roles are included to recruit research teams- research enumerators/supervisors, coordinated training, monitoring on field work data collection and quality control checks and updating overall reports to donors. The total sample sizes was 15,000 HHs among 22 provinces of Cambodia. July 2013 – Aug. 2013 Evaluation Consultant, Save the Children International, Kampong Cham ● The objective was to conduct an end-term project evaluation of the five-year Maternal, Child Health and Nutrition Program implemented by SCI in Kampong Cham Province. June 2013 – July 2013 Study Team Leader, UNOPS, Thai-Cambodia Provinces, Cambodia ● The overall purpose was to conduct the qualitative study of the Malaria Project that has been implemented by FHI and contracted by UNOPS to conduct an endline project evaluation of Malaria program, which was implemented by FHI. May 2013 – July 2013 Consultant, World Vision Cambodia, Cambodia ● The objective is to develop training packages, curriculums for implementation of a community￾based intervention in regard to the improvement of maternal, child health and nutrition services. To evaluate the five-year project of Save the Children related to maternal and childcare. Nov. 2011 – Feb. 2012 Research Consultant, Macro-International financed by GFATM, Cambodia ● The objective was to assess the overall community health system in Cambodia that was funded by GFATM by using the overall qualitative method in the purpose of strengthening the health system of Cambodia and infectious diseases including Malaria, TB and HIV programs. ● To analyze the situation of community health system in Cambodia that was funded by GFATM in the purpose to strengthen malaria, TB and HIV program and community health system. Jan. 2011 – Feb. 2011 Health System Consultant, USADI Cambodia, Cambodia ● The objective was to assist the Mid-Project Evaluation for 4 partners USAIDS-funded (PSI, URC, RACHA and RHAC) with regards to the program of Maternal and Child Health, HIV/AIDS and Health System Strengthening supported to MoH. ● To evaluate the mid-project evaluation of health programs funded by USAID and implemented by four NGOs, under which they have supported the strengthening of health system in Cambodia with a focus on reducing the Maternal and Child morbidity and mortality. 140 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Jan. 2010 – Nov. 2011 Evaluation Consultant, UNFPA Cambodia, Cambodia ● The objective was to conduct an evaluation of the country’s five-year programs of UNFPA that is funded to MoH, MoWA, MoEYs, MoP, and NGOs- Reproductive Maternal and Child Health and Nutrition Programs. Supported the Team Leader for the five-year country program evaluation. EDUCATION M.P.H. — Public Health, MAHIDOL University, 2003 M.D. — Medical Doctor, University of Medical Science Phnom Penh, 1995 Diploma — Tropical Medicine and Epidemiology, Medical Training Center, 1996 Diploma — Epidemiology, Vietnam Epidemiology Training Center, 1999 LANGUAGE KHMER – Fluent, ENGLISH – Fluent, FRENCH – Advanced 141 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV BOPHA KONG Research Coordinator/ Gender Equality & Social Inclusion Specialist Ms. Bopha KONG holds a master’s degree in private law from the Royal University of Law and Economic in Phnom Penh, with certificates in monitoring and evaluation coaching, child’s rights programming, skillful parenting, and human rights, peace, and security. She has more than five years of experience with gender and social inclusion, and women’s rights and advocacy. She has another 10 years of experience in local and international development donor-funded project implementation and partnership work such as with Educo, Oxfam, and We World. Areas of expertise include primary education, social protection, gender equality, public health, and rural livelihoods. She has extensive experience in Cambodia conducting evaluations with an international team using qualitative and quantitative data collection methods and tools. She has conducted baseline, and mid-term evaluations of donor-funded projects across various regions of Cambodia for a variety of clients including USAID, UNDP, and We Effect. June 2020 – Aug. 2020 Gender Specialist, UNDP, Building Capacity of Civic Engagement, Peace Building and Inclusive Dialogue Toward Inclusive and Participatory Governance, Cambodia ● Conducted key informant interviews, focus group discussions, and consultations with stakeholder including community-based organizations, NGOs, activists, human right defenders, and senior government officials. ● Reviewed and revised project document on relevant sections: result framework, define target beneficiaries, stakeholder analysis, partnership, and work plan. ● Joined in organization validate workshop. ● Joined in written inception and final report. Apr. 2020 – Sep. 2020 Program Assistant, USAID, Mid-term evaluation of the Cambodia Malaria Elimination Project, Cambodia ● Support assessment team with all logistic and administration to allow them to carry out this evaluation to ensure that the processes move forward smoothly. ● Prepare and arrange meeting and appointment with relevant stakeholder of the project for team assessment. ● Translate all evaluation tools into Khmer version. ● Conduct quantitative and qualitative data collection with target groups. ● Report daily progress of field data collection. July 2019 – Dec. 2019 Team Leader, Royal University of Fine Art, Royal University of Fine Art Strategic Development Plan, Cambodia ● Review, assess, and analyze and conduct consultation with RUFA staff and relevant stakeholder. ● Develop-Strategic Plan. ● Present the draft Strategic plan to board and line ministry. ● Publish Strategic Plan. Jan. 2019 – Mar. 2020 Gender and Governance Advisor, SNV Netherlands Development Organization, NOURISH Project, Cambodia ● Oversee the implementation of NOURISH’s pro-poor program including administration of the pro-poor, social and gender inclusion strategies of the program’s water, sanitation, and hygiene (WASH) component in target provinces. 142 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Administer the WASH voucher (pro-poor subsidy) including its coordination, planning, budgeting, and monitoring processes at province, district, and commune levels in collaboration with other project team members and government representatives. ● Provide advice and inputs on the governance, gender, and social aspects of the program, especially relating to the planning and monitoring framework. ● Develop and deliver targeted capacity building to small and medium enterprises in the WASH and agriculture sectors, particularly targeting gender aspects such as female entrepreneurship. ● Continue to develop, refine, and track the voucher program modality in close consultation with relevant Provincial Departments of Rural Development and key stakeholders. ● Work with key government partners to build capacity on gender related aspects of WASH to align with national gender guideline and strategies. ● Promote WASH gender engagement actions to enable greater participation, leadership, and decision making of women in WASH activities. ● Support the procurement processes of WASH service suppliers and ensure quality product delivery and installation at target households. ● Prepare monthly disbursement and verify payments to SMEs. ● Conduct regular field visits to ensure the proper implementation of pro-poor project activities. ● Oversee and ensure that all relevant M&E data are collected and regularly enter into project Interactive Voice Response System. ● Coordinate with team members for regular project planning, implementation, and reporting to the WASH Sector Leader. ● Support other related tasks and assignments as requested by the WASH Sector Leader. 2018 Local Consultant, Viet Rural Development Enterprise Center, Baseline Survey of the Project of Economic Empowerment and Social Inclusion for Farmers and Agriculture Procedures, Cambodia ● Contact with representative staff of We Effect in Cambodia to identify survey locations. ● Assist VietED Team to conduct desk review of relevant document provided by the client/project partners. ● Assist VietED Team to conduct the survey activity in the field at the three partners’ location. ● Lead the field activities at the partners’ location. 2017 – 2018 Program Manager, We World Onlus Cambodia, Education and Child Protection Program, Cambodia ● Plan and execute project’s partner monitoring and evaluation with field visit, revise and approve the project narrative reports, update and maintain M&E tools, and manage the database of project’s monitoring. ● Provide technical support to partner on project management, assist the partner on the knowledge management of project (PCM) including support in the identification of the project, preparing of quarterly plan, reporting and monitoring related to Education and Child Protection. 2016 – 2017 Program Officer, Oxfam Cambodia, Voice Program, Cambodia ● Process and monitoring on grant allocation in the administrative system, responsible of country level reporting. ● Provide technical support as needed to potential grantee in the formulation of grant applications, mentor and provide technical assistance to grantees in the implementation of the project. 2013 – 2016 Project Coordinator, EduCo, Education and Child Protection Program, Cambodia 143 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Activities performed: Responsible for Educo in exploring potential partners’ capacity and synergy with EDUCO’s vision, mission on child rights, child protection, and child education particularly on early child education and care as well as primary education. EDUCATION M.A. — Private Law, Royal University of Law and Economic, Phnom Penh, 2010 B.A. — Mathematics, Royal University of Law and Economic, Phnom Penh, 2000 LANGUAGE KHMER – Fluent, ENGLISH – Fluent 144 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ANNEX 8: EVALUATION STATEMENT OF WORK Service Request Mid-Term Evaluation of Enhancing Quality of HealthCare Activity (EQHA) and Promoting Healthy Behavior (PHB) Activity I. Purpose of the Evaluation The primary purpose of this mid-term performance evaluation is to assess the Enhancing Quality of Healthcare Activity (EQHA) and Promoting Healthy Behavior (PHB) Activity performance to date, identify key bottlenecks and constraints, and make actionable recommendations for improvements needed to meet the activities’ intended objectives. The findings will be used to provide feedback to USAID/Cambodia, the implementing partners, and relevant stakeholders on the successes, areas for improvement, and any course corrections required for the remainder of the activities. This evaluation will be one exercise that looks across the two activities. I. Background The objective of the Enhancing Quality of Healthcare Activity (EQHA) is to improve the quality of public and private health services in a sustainable manner through technical assistance to national and sub￾national health systems. The Activity will support public and private healthcare services to meet acceptable global standards, delivered in an efficient manner, resulting in increased client satisfaction. Efforts will ensure that health services are managed holistically and linkages between departments are maximized to ensure that services are patient-centered, and individuals are not lost within the system. Activity efforts will also support the development of accreditation of health facilities and address preservice and in-service training needs in order to ensure the competency of healthcare providers. EQHA prime partner FHI360 leads the overall program implementation, strategies, and approaches under Objectives 1 and 2, contributes expertise in accreditation, and provides technical expertise for design, execution, and Monitoring and Evaluation (M&E). Population Services International (PSI) is the sub-partner that leads the private sector strategies and contributes tools and best practices across all objectives. The International Training and Education Center for Health, University of Washington (I￾TECH/UW), as another sub-partner, designs and executes activities under objective 4 and parts of Objective 3 about the Registration Management System (RMS) for Health Professional Council (HPCs). Finally, sub-partner Meridian Group International (Meridian) contributes to Objectives 2 and 3 to strengthen linkages between workplace infirmaries and public and private health systems. The purpose of Promoting Healthy Behavior (PHB) is to improve health behaviors among Cambodians and ensure that Cambodians seek and receive quality health care with decreased financial hardship through more sustainable systems. Population Services International (PSI) is the prime partner of PHB that leads the overall implementation of the award. 17 Triggers, a global private sector behavior change lab founded in Cambodia, is the sub-partner that contributes their expertise on using marketing, human￾centered design (HCD), research, and design thinking to develop innovative and effective Social Behavior Change (SBC) strategies. These two activities started in Summer 2018 and are scheduled to end in June (PHB) and August (EQHA) of 2023. II. Program Objectives The Enhancing Quality of Healthcare Activity consists of four objectives listed below: EQHA Objective 1 - Improved policies, guidelines and standards for streamlined quality assurance. Policy frameworks, including accreditation, must reflect global evidence-based practices and solutions, and be implemented effectively at the service delivery level across the public and private sectors. Developing and applying new evidence is a continuous learning process requiring the Royal 145 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Government of Cambodia (RGC) to adopt new findings, and have the capacity and tools to make changes in their implementation policies and guidelines. Expected results are: ● RGC capacity to use evidence and visualize data for policy making and health service delivery improvement strengthened. ● Evidence-based, gender-sensitive policies and guidelines for specific disease areas integrated into the health system. ● Quality improvement (QI) tools and processes integrated, streamlined in health facilities’ day-to￾day work. ● Method-specific, level-appropriate SOPs and job aids developed jointly by public and private providers at the facility level. ● QI plans based on (adjusted) NQEM standards at facility level development. EQHA Objective 2 - Increased efficiency and effectiveness of service delivery. To support the MoH QI Master Plan, and component 1 of the H-EQIP, such as strengthening service delivery for a specific program area (e.g., TB, HIV, malaria, maternal and child health—MCH, RH/FP, nutrition), and support the RGC to establish, manage, replicate and sustain QI efforts on a large scale, EQHA will use the PDSA (Plan-Do-Study-Act) based “Model for Improvement” (MFI) framework and the Collaborative Model described in FHI 360’s handbook. Expected results are: ● Sub-national improvement teams continuously measure and improve disease-specific services. ● Sustained sub-national coaching and quality assurance (QA) by provincial NQEM conducted on a regular schedule. ● Private and public provider networking strengthened and referral from private providers increased. ● Percentage of clients treated according to disease-specific guidelines (e.g., TB, HIV, GBV) increased. ● Satisfaction and engagement of clients in dialogue with the health system increased. EQHA Objective 3 - Strengthened regulatory framework, implementation, and enforcement. EQHA will use a systems approach to strengthen existing national and sub-national governance structures, processes and outcomes via both mandatory and voluntary regulatory quality assurance strategies. FHI 360 will support the MoH Department of Human Resources (DHR), the five national professional Councils (medical, dental, midwives, nursing and pharmacy), the Provincial Governor’s offices and 6 target province PHDs and councils to ensure mandatory professional and facility regulatory systems—including registration and licensure—are harmonized and enforced among public and private providers. EQHA will work to lay the foundation for accelerated progress toward establishing a formal independent accreditation scheme for health care facilities. Expected results: ● A robust health workforce database in place for MoH and professional Councils to track registered/licensed practitioners and maintenance of qualifications, registration, and licensure status. ● Quality recognition system for health facilities building toward a successful accreditation system. ● Improved regulations implemented for licensing private and public facilities and practitioners, tied to quality standards, and aligned to the content of the new law regulating health practitioners. ● “Role modeling” of private facilities meeting quality standards introduced to guide MoH policy development toward establishment of quality recognition/accreditation systems. ● Systems established for professional councils and MoH to monitor professional registration and licensing requirements that build on existing systems. ● Harmonized continuing professional development framework for each of the five national professional councils that helps build and maintain competencies of public and private providers. 146 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● License and registration fee structure strengthened to ensure sustainability of professional councils and facility accreditation program. EQHA Objective 4 -Strengthened preservice public health training. The Basic Training Bureau (BTB) in the Department of Human Resources (DHR) of the MoH is directly responsible for all health￾related preservice training for the country’s 19 health education institutions (six public, 12 private, and one military-run), including the development of national curricula and assessments. The Council of Ministers has the ultimate authority to regulate these institutes and is setting similar standards for both public and private university graduates. A gap remains between the content-focused preservice curricula in these institutions and the skills-based competencies required as graduates confront the realities of health care practice. The EQHA team provides the skills and knowledge needed to fill this gap and seeks to partner with BTB/MoH, the health profession Councils, and other key stakeholders to strengthen systems. Expected results are: ● Capacity of preservice training institutions strengthened to effectively deliver high quality educational programs aligned with national health needs and priorities, and global evidence. ● Capacity of preservice training expanded to accommodate increased student enrollment. ● Faculty demonstrates ability to employ 4 teaching competencies: course design, facilitation of learning, assessments and evaluation, maintenance of professional expertise. ● Preservice curricula incorporating evidence-based approaches (as per Objective 1) and filling other gaps in the six disease areas of focus. ● Adequate access to ongoing professional development in health sciences assured. Promoting Healthy Behavior consist of two main objectives which listed below: PHB Objective 1-Strengthened public sector systems for oversight and coordination of SBC at the national and provincial levels. The implementing partner will build on current experience, and solid working relationships with the MoH and SBC partners to engage high level commitment from the Director General for Health. This will be critical to re-invigorating the National Centre for Health Promotion (NCHP)-led SBC coordination mechanisms and broad SBC stakeholder engagement to achieve national SBC priorities and reinforce NCHP leadership and visibility for coordination, joint planning, advocacy for, design and implementation of effective SBC programs. NCHP visibility will be enhanced through the launch of the NCHP led SBC Forum, evidence review, and SBC Strategic Framework. Expected results are: ● Coordination and joint planning of SBC programming among stakeholders, including the private sector, improved through development of a sector-wide strategic framework. ● Strengthened systems for ensuring quality and ongoing monitoring of the impact of SBC products and activities. ● Increased RGC funding for SBC as an essential element of health programming. PHB Objective 2-Improved ability of individuals to adopt healthy behaviors. In Cambodia, social norms, lack of education, and misinformation often hamper health seeking behaviors. While health service availability and the use of certain services within Cambodia’s public health system have improved significantly, reliance on public providers remains low. Private sector facilities are widely viewed as being superior to public facilities, despite variable quality and inadequate regulation. To provide Sara (the representative archetype of the people the PHB is serving. She could be the wife trying to convince her husband to sleep under the net, she could be the mother who worry about her child fever, she could be the daughter who noticed her father’s persistent cough...etc) with the right SBC approach, in a format she understands and appreciates, at the right time, according to her specific context, demographic, Socioeconomic Status (SES) and other factors to improve her core healthy behaviors, the Team and PHB AB will support provincial and commune stakeholders to implement demonstration activities through the Four Step SBC Activity Planning Process. 147 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Application of the Socioecological Model (SEM) and structured behavioral analysis will help these stakeholders to understand Sara’s journey and influence in specific Cambodian contexts in order to make informed decisions for SBC activities to improve targeted behaviors. The Team will leverage proven global and Cambodia SBC approaches and initiatives, link with other RGC and USAID health initiatives and implement rigorous M&E to assess program progress and implement continuous improvement to ensure that demonstration activities lead to measurable improvements to health seeking behaviors (cross cutting or health areas specific) among priority populations, selected as part of the PHB SF and targeted by demonstration activities. Expected results are: ● Quality, high-priority SBC activities demonstrated by RGC in partnership with SBC service providers. ● Increased self-efficacy to enact positive behaviors and reduce harmful behaviors, including use of health and social services, among priority populations. ● Characteristics of high-quality services perceived more accurately by priority populations. ● Strengthened social norms supporting positive behaviors among priority populations. III. Geographic Coverage EQHA and PHB implementation is working at the national level and provincial level focusing in Kampong Cham, Tbong Khmum, Kampong Chhnang, Battambang, Pailin and Phnom Penh. IV. Purpose, Audience, and Application The primary audience of this evaluation is USAID/Cambodia Office of Public Health and Education. The secondary audiences will be the implementing partners including prime and sub-partners of EQHA and PHB and the Ministry of Health counterparts at the national and sub-national levels. V. Evaluation Questions 1. How could EQHA and PHB effectively leverage and build synergy with each other? This synergy includes, but is not limited to, stakeholder coordination, including among sub-partners, other USAID activities, and other key partners in order to achieve desired health outcomes. 2. How can EQHA and PHB leverage their respective experiences to institutionalize the activities’ interventions with the local system in the future? 3. What are key challenges in achieving the expected results of EQHA and PHB and how to overcome the challenges? 4. How have the EQHA ‘s and PHB’s structure, management, and internal procedures and sub￾awardees management affected implementation and outcomes, positively and/or negatively? 5. How effective have the EQHA approaches been in engaging the private sector in program implementation? 6. How effective have the PHB approaches been in strengthening public sector systems for SBC, and what measures could be considered to improve performance in this area? VI. Deliverables and Reporting Requirements 1. Evaluation Work Plan: The evaluation team will submit the proposed Evaluation Work Plan to USAID according to the agreed timeline. The Evaluation Work Plan must include: (1) The anticipated schedule for the evaluation and logistical arrangements; (2) Proposed submission date of each deliverable in accordance with schedule illustrated in section VIII; (3) The evaluation team members, including names of key personnel delineated by roles and responsibilities; and (4) Evaluation design including: 148 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV a) A detailed evaluation design matrix that links the evaluation questions in the service request to data sources, methods, and the analysis plan; b) All data collection instruments or their main features; c) The list of potential interviewees and organizations; and d) Known limitations to the evaluation design. USAID/Cambodia will provide consolidated comments on the Evaluation Work Plan within seven (7) calendar days (one week) of receipt of it. The evaluation team must submit a revised Evaluation Work Plan to USAID for approval within seven (7) calendar days (one week) of receipt of comments. The evaluation team must have an approved Evaluation Work Plan in place before proceeding with field work. 2. In-briefing: The evaluation team is required to have an in-brief with USAID/Cambodia in person to discuss the team’s understanding of the evaluation, initial assumptions, evaluation questions, methodology, and work plan, and to answer questions the team may have. The in￾brief occur must occur within 2 weeks after the team submitted the draft Evaluation Work Plan. 3. Field Work: The evaluation team must complete all field work activities within five weeks following the approval of the Evaluation Work Plan by USAID. It is expected that the team will be able to finish actual field work within four weeks and they have one more week for data verification if needed. 4. Data Analysis and Reporting: The evaluation team must complete data analysis and reporting within nine weeks following the approval of the Evaluation Work Plan by USAID. 5. Regular Updates on the Field Work: The evaluation team leader must brief USAID on progress made with the field work on a weekly basis, in person or by electronic communication in an informal report to be sent via email, agreed upon in advance. Any delays or complications must be communicated to USAID as early as possible to allow for quick resolution and to minimize any disruptions to the evaluation. 6. Exit Briefings: The evaluation team must hold an exit briefing after field work and prior to leaving the country. The debriefings should include initial findings from the evaluation through a PowerPoint presentation, including a discussion of key evaluation findings only, with preliminary recommendations for possible modifications to program approaches, results, or activities for discussion with USAID staff. The evaluation team must consider USAID/Cambodia and implementing partner comments received during the exit briefings and incorporate them appropriately when drafting the evaluation report. 7. Draft Evaluation Report: The draft evaluation report must be consistent with the guidance provided in Section XI: Evaluation Report Format. The report must succinctly address each of the questions identified in the service request and any other issues the evaluation team considers to have a bearing on the objectives of the evaluation. The evaluation team must submit the draft Evaluation Report to USAID within nine weeks following the approval of the Evaluation Work Plan. USAID/Cambodia will provide comments on the report within 14 calendar days (two weeks) following the receipt of the draft report (one set of comments from USAID/Cambodia and another from the implementing partner). 8. Final Evaluation Report: The evaluation team must address/incorporate all comments/suggestions and submit the Final Report to USAID within 14 calendar days (two weeks) of receipt of comments from USAID on the draft report. VII. Evaluation Team Composition The evaluation team should be composed of an Evaluation Team Leader, one National Evaluation Advisor on health system strengthening, one National Evaluation Advisor on SBC, and Logistics/Program Assistant. Inclusion of a fourth team member specializing in clinical health administration is pending. A mix of gender is strongly encouraged for the team composition. 149 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV Evaluation Team Lead The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation/analytic team, and (5) leading briefings and presentations. International expatriate based in Cambodia is strongly preferred. Qualifications: ● At least an MPH or other health-related graduate degree ● Minimum of 10 years of experience of field experience managing health projects, programs and evaluations ● Demonstrated extensive experience leading large, complicated Health Strengthening System (HSS) and Social Behaviour Change (SBC) project/program evaluation/analytics, utilizing both quantitative and qualitative methods ● Advanced knowledge and skills of program performance evaluation design, methodology and processes. Excellent skills in planning, facilitation, and consensus building ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent team leadership and management skills ● Excellent organizational skills and ability to keep to a timeline ● Proficient in writing and spoken English ● Good writing skills, with extensive report writing experience ● Experience working in the region, and experience in Cambodia is desirable 150 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV National Evaluation Advisor on health system strengthening The National Evaluation Advisor will support the Evaluation Team Leader in designing, planning and implementing the evaluation, including qualitative data collection and analysis. In addition to that, s/he will ensure contextual relevance of the design and implementation of the evaluation. The National Evaluation Advisor must have at least 5 years of experience in conducting program evaluations and/or research. S/he must have at least a Master’s Degree in Public Health, Statistics, Epidemiology, Social Studies, or another related field. As a member of this evaluation, s/he must demonstrate strong contextual knowledge of the health system strengthening programming in Cambodia. Qualifications: ● At least 5 years of experience in M&E procedures and implementation of large, complicate relevant projects ● Experience assessing, monitoring and/or evaluating health programs - Health System Strengthening or Social Behaviour Change is preferred ● At least 5 years managing M&E, including evaluations and/or assessments ● Experience in design and implementation of evaluations and/or assessments ● Strong knowledge, skills, and experience in qualitative and quantitative analytic tools ● Experience implementing and coordinating others to implement surveys, key informant interviews, focus groups, observations and other evaluation and assessment methods that assure reliability and validity of the data. ● Experience in data management ● Able to analyze qualitative data ● Experience using analytic software ● Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data ● Experience conducting secondary analysis of existing quantitative datasets ● Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation ● Strong data interpretation and presentation skills ● Proficient in written and spoken English ● Good writing skills, including experience writing evaluation and/or assessment reports National Evaluation Advisor on SBC The National Evaluation Advisor will support the Evaluation Team Leader in designing, planning and implementing the evaluation, including qualitative data collection and analysis. In addition to that, s/he will ensure contextual relevance of the design and implementation of the evaluation. The National Evaluation Advisor must have at least 5 years of experience in conducting program evaluations and/or research. S/he must have at least a Master’s Degree in Public Health, Statistics, Epidemiology, Social Studies, or another related field. As a member of this evaluation, s/he must demonstrate strong contextual knowledge of the SBC programming in Cambodia. Qualifications: ● At least 5 years of experience in M&E procedures and implementation of large, complicate relevant projects ● Experience assessing, monitoring and/or evaluating health programs - Social Behaviour Change is preferred ● At least 5 years managing M&E, including evaluations and/or assessments ● Experience in design and implementation of evaluations and/or assessments ● Strong knowledge, skills, and experience in qualitative and quantitative analytic tools ● Experience implementing and coordinating others to implement surveys, key informant interviews, focus groups, observations and other evaluation and assessment methods that assure reliability and validity of the data. 151 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Experience in data management ● Able to analyze qualitative data ● Experience using analytic software ● Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data ● Experience conducting secondary analysis of existing quantitative datasets ● Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation ● Strong data interpretation and presentation skills ● Proficient in written and spoken English ● Good writing skills, including experience writing evaluation and/or assessment reports Logistics/Program Assistant The Logistics/Program Assistant will support the Evaluation Team with all logistics and administration to allow them to carry out this evaluation. S/he will have a good command of English and Khmer. S/he will have knowledge of key actors in the health sector and their locations including MOH, donors and other stakeholders. To support the Team, s/he will be able to efficiently liaise with hotel staff, arrange in￾country transportation (ground and air), arrange meetings and workspace as needed, and ensure business center support, e.g. copying, internet, and printing. S/he will work under the guidance of the Assessment Team Leader to make preparations, arrange meetings and appointments. S/he will conduct programmatic administrative and support tasks as assigned and ensure the processes move forward smoothly. VIII. Evaluation Report Format The final evaluation report must include: ● an abstract; ● executive summary; ● background of the local context and the activity being evaluated; ● evaluation purpose and final evaluation questions; ● evaluation methodology and limitations; and ● findings, conclusions, and recommendations. For more detail, see: ● “How-To Note: Preparing Evaluation Reports” (https://www.usaid.gov/sites/default/files/documents/1870/How-to-Note_Preparing-Evaluation￾Reports.pdf) ● A Mandatory Reference for ADS Chapter 201 (https://www.usaid.gov/ads/policy/200/201mah) ● An optional evaluation report template is available in the Evaluation Toolkit (https://usaidlearninglab.org/library/evaluation-report-template). The executive summary must be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions and recommendations. The evaluation methodology must be explained in detail in the report. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology. The annexes to the report must include: ● Evaluation SOW; ● Detailed description of the evaluation design and methods; 152 | ALME CAMBODIA EQHA & PHB EVALUATION REPORT USAID.GOV ● Statements regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if any; ● All data collection and analysis tools used in conducting the evaluation, such as surveys, interview questions, checklists, and discussion guides; ● All sources of information, properly identified and listed (including documents reviewed, sites visited, and key informants, assuming they gave permission to be identified); ● Signed disclosure of any conflict of interest forms for all evaluation team members, by including a statement by evaluation team members that attest to a lack of conflict of interest or describes existing conflicts of interest relative to the project being evaluated; and ● Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the evaluation team lead will make the final evaluation report publicly available through the Development Experience Clearinghouse within 90 days of the approval by USAID.