This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION EVALUATION REPORT February 2023 Tasking N064 Contract No. GS-10F-0033M / Order No. 7200AA18M000 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) II ACTIVITY CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION EVALUATION REPORT February 2023 Tasking N064 Contract No. GS-10F-0033M / Order No. 7200AA18M00016 Submitted to: USAID DRG Center Submitted by: NORC at the University of Chicago Matthew Parry, Program Manager Bethesda, MD 20814 Tel: 301-634-5444; E-mail: parry-matthew@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The evaluation team (ET) would like to thank the current and former staff at FHI360, Nickol Global Solutions LLC, Institute for Development Impact (I4DI), and their implementing partners for their participation and provision of documentation and valuable feedback. The ET also appreciates Indochina Research Ltd (IRL) for their data collection services. The ET also thanks the staff at USAID, including Terith Chy, Lee Forsythe, Socheata Vong, and Jennifer Huynh for their support and feedback through the evaluation process. Finally, the ET would like to thank the stakeholders who spoke with us about the Social Accountability Portfolio and the activities. The support and engagement of these organizations and stakeholders proved invaluable during the evaluation process. iii | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV TABLE OF CONTENTS EXECUTIVE SUMMARY 1 1. INTRODUCTION 4 2. BACKGROUND 5 ISAC OBJECTIVES AND INTERVENTIONS 5 OW4C OBJECTIVES AND INTERVENTIONS 6 T4GC OBJECTIVES AND INTERVENTIONS 7 3. EVALUATION QUESTIONS 9 4. EVALUATION DESIGN AND METHODOLOGY 9 METHODOLOGY AND DATA 10 EVALUATION DESIGN MATRIX 12 LIMITATIONS 14 5. FINDINGS AND CONCLUSIONS 14 EFFECTIVENESS OF TECHNOLOGY & YOUTH ENGAGEMENT 14 FACEBOOK CHATBOT 15 GREEN CAMBODIA CITIZENS APP 19 TECH SUPPORT FOR CSOS 22 YOUTH AMBASSADORS PROGRAM 25 COMMUNITY OUTREACH 29 EFFECTIVENESS OF SOCIAL ACCOUNTABILITY ACTIVITIES 34 ONLINE DASHBOARDS 34 COMMUNITY SCORECARDS 40 PUBLIC FORUMS 44 SOCIAL ACCOUNTABILITY ACTIVITIES AND PROGRESS TOWARDS USAID/CAMBODIA’S DRG OBJECTIVES 47 6. RECOMMENDATIONS 48 ANNEX A. EVALUATION SCOPE OF WORK 51 ANNEX B. YA SURVEY INSTRUMENT 54 ISAC 54 OW4C 57 T4GC 61 ANNEX C. KII PROTOCOLS 65 ISAC 65 OW4C 69 T4GC 74 ANNEX D. YOUTH AMBASSADOR SURVEY RESULTS 79 EFFECTIVENESS OF TECHNOLOGY & YOUTH ENGAGEMENT 79 EFFECTIVENESS OF YOUTH AMBASSADORS / CAF PROGRAM 80 EFFECTIVENESS OF OUTREACH 81 EFFECTIVENESS OF SOCIAL ACCOUNTABILITY ACTIVITIES 83 USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION | iv LIST OF TABLES AND FIGURES Table 1: Number of KIIs by Activity and Respondent Type..................................................................................10 Table 2: Evaluation Design Matrix ...............................................................................................................................12 Figure 1: Evaluation Timeline (Source: Evaluation Team)........................................................................................9 Figure 2: Screenshot of OW4C Facebook Chatbot (Source: OW4C Facebook page).................................15 Figure 3: Demand for OW4C Chatbot - Accessing Information about OWSO Services (Source: OW4C Online Dashboard)..........................................................................................................................................................16 Figure 4: Demand for OW4C Chatbot - Providing Feedback on OWSO Services (Source: OW4C Dashboard)........................................................................................................................................................................17 Figure 5: Green Cambodia Citizens App (Source: T4GC)....................................................................................19 Figure 6: Green Cambodia Citizens App total downloads and sign-ups by province (Source: T4GC Online Dashboard)..........................................................................................................................................................20 Figure 7: Green Cambodia Citizens App reports by province and waste type (Source: T4GC Online Dashboard)........................................................................................................................................................................21 Figure 8: Screenshots of a Facebook Page and a Telegram Channel managed by CSOs (Source: AMARA’s Facebook page / SK’s Telegram channel)...................................................................................................................24 Figure 9: Mobile Loudspeaker used for ISAC’s I4C Outreach (Source: AMARA’s Facebook page) .........30 Figure 10: T4GC YA Outreach Activity Participation (Source: YA Survey).....................................................33 Figure 11: Screenshot of OW4C Online Dashboard .............................................................................................35 Figure 12: Site visits to OW4C Dashboard (Cambodia traffic only) (Source: OW4C).................................36 Figure 13: Screenshot of T4GC Online Dashboard................................................................................................37 Figure 14: Screenshot of T4GC’s Online Dashboard Reporting Feature (Source: T4GC Online Dashboard)........................................................................................................................................................................39 v | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV ACRONYMS 3R CAF CSO DG DIIF DO DRG EQ ET GOC GCC I4C I4DI IP IR IRL ISAC KII LER MCC OW4C OWSO PE PO SBAR SI SWAP T4GC US USAID USG WMA WMS WMSP YA Reduce - Reuse - Recycle Community Accountability Facilitators Civil Society Organization Democracy and Governance Digital Innovation and Information Fund District Ombudspersons Democracy, Human Rights, and Governance Evaluation Question Evaluation Team Government of Cambodia Green Cambodia Citizens App Information for Citizens Institute for Development Impact Implementing Partner Intermediate Result Indochina Research Ltd Innovations for Social Accountability in Cambodia Key Informant Interview Learning, Evaluation, and Research Millennium Challenge Corporation One Window for Citizens One Window Service Office Performance Evaluation Provincial Ombudsperson Small Business Applied Research Social Impact Solid Waste Accountability Platform Tech for Green Cities United States United States Agency for International Development United States Government Waste Management Authorities Waste Management Services Waste Management Service Providers Youth Ambassador 1 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY The purpose of this study is to evaluate the performance of three Activities within USAID/Cambodia’s 2019-2022/2024 social accountability portfolio: 1. Innovations for Social Accountability in Cambodia (ISAC), a five-year program designed to support citizens to improve their capacity for collective action, increase access to information about government accountability, and build public demand for government to develop solutions to address community priorities. Concretely, the ISAC Activity consisted of the Community Accountability Facilitators (CAF) program, tech support for CSOs to increase online engagement, community outreach to raise awareness about government accountability, and community scorecards. 2. One Window for Citizens (OW4C), a three-year pilot program designed to increase the capacity of civil society to engage in political processes for improved public services through the One Window Service Office (OWSO). Concretely, the OW4C Activity consisted of the creation of a Facebook Chatbot to help citizens access information about government services, an Interactive Voice Response (IVR) call-in line, a publicly available, online dashboard to help government officials understand citizen feedback on those services, the Youth Ambassadors (YA) program, community outreach, public forums, and community scorecards. 3. Tech for Green Cities (T4GC), a three-year pilot program designed to improve Cambodia’s public service delivery and waste management quality and transparency. Concretely, the T4GC Activity consisted of the creation of a mobile app to help citizens report waste management issues, an online dashboard to help government officials view and analyze those reports, the Youth Ambassadors program, and community outreach to raise awareness of the app and dashboard. These activities were designed and procured in response to the Democracy, Human Rights, and Governance (DRG) Cambodia Assessment conducted in 2019, which recommended using technology and social media innovations to increase citizens’ ability to demand access to public services, provide feedback on the quality of those services, and ultimately hold local government accountable. The intended primary audience of the evaluation is USAID/Cambodia’s DG Office and the three evaluation questions (EQs) guide this study include: 1. How effective have technology and youth engagement activities been at sharing information about government performance, increasing citizen engagement with local administration and service providers, and raising awareness of government accountability? 2. To what extent have social accountability activities increased government accountability at the local level? 3. How appropriate and/or useful were USAID/Cambodia’s goals and objectives for the social accountability activities in achieving the Mission’s DRG objectives? To answer these questions, this performance evaluation (PE) draws on four sources of data: activity documents, KIIs with activity beneficiaries and stakeholders, a close-ended survey of activity stakeholders (N=299), and web analytics data from the mobile apps and dashboards created under the T4GC and OW4C Activities. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION | 2 Analysis of these data provides three topline findings: • Uptake of the mobile app, Facebook Chatbot, and online dashboards created under the T4GC and OW4C Activities was limited and short-lived. After being introduced to these tools, citizens and government officials quickly stopped using them; by the end of the projects, usage of these tools was negligible. Based on these findings, we conclude that these tools were not viewed by citizens as an effective means to access information about government services or provide feedback on the quality of those services. Similarly, limited usage of the online dashboards by government officials suggests these tools were not viewed as an effective means to access citizen feedback or identify priority issues that need to be addressed. Whether the failure of these tools was the result of incongruence with local conditions and systems, limited outreach and dissemination, or other factors remains uncertain. • Self-reported survey and KII data from YA and CAF participants indicates these programs helped build a corps of accountability leaders within targeted communities. As a result of their engagement, participants report improved public speaking skills, greater teamwork skills, expanded professional networks, and greater valuation of government accountability. Although we cannot rule out the risk that response bias influences these findings, they constitute preliminary but encouraging evidence of program effectiveness. • Survey and KII data from citizens and key project stakeholders provide encouraging evidence that public forums and community scorecards are productive fora for stimulating engagement between citizens and government officials, educating citizens on how to access government services, and soliciting feedback on how those services could be improved. These findings rely on respondents’ own accounts of change, however, and we cannot rule out the risk that response bias influences these findings. For this reason, we view this evidence as preliminary. Based on these findings, we offer three sets of recommendations, all of which are aimed at USAID/Cambodia and its implementing partners (IPs): Recommendations for introducing new, tech-based tools to promote government accountability: • Policymakers should carefully consider how new technology tools would fit within existing accountability processes, how it will build on or improve these processes, and under what conditions demand for these tools among citizens and government officials will be sustained. • When launching these tools, it is essential to conduct extensive outreach to ensure a large and sufficient number of users on the new platform. Only when the platform reaches a certain critical threshold number of users will demand for these products be self-sustaining. • To maximize the saturation of outreach and the uptake of the new tech tool(s), the rollout of these tools should be done sequentially, district by district or city by city. Expansion of the tools beyond the initial pilot location(s) should be conditional on robust, sustained, and growing demand in the initial pilot location(s), as measured in real-time through usage data on the tools themselves. Only if the tool is widely used and adopted in the initial pilot location(s) should it be rolled-out to additional locations. Recommendations for promoting youth engagement and leadership for government accountability: • Based on the encouraging if preliminary evidence that the YA and CAF programs were effective at building leadership skills and promoting the importance of government accountability among 3 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV participants, we recommend that USAID/Cambodia and its IPs continue to pursue these types of programs. • Alongside these efforts, USAID/Cambodia and its IPs should seek to generate more definitive evidence regarding the effectiveness of these types of programs. To this end, USAID/Cambodia and its IPs should design future programming in such a way as to allow for rigorous evaluation. This includes collecting before vs. after information on leadership and accountability outcomes and employing experimental or quasi-experimental, comparison-group evaluation designs to generate credible evidence about the effectiveness of these types of programs. Recommendations for promoting collective action for government accountability • Based on the encouraging if tentative evidence that public forums and community scorecards are productive fora for stimulating engagement between citizens and government officials, we recommend that USAID/Cambodia and its IPs continue to pursue these types of programs. • Alongside these future programming efforts, USAID/Cambodia and its IPs should generate additional, more credible evidence about the effectiveness of these programs by coupling their implementation with rigorous experimental or quasi-experimental evaluation to verify their efficacy in Cambodia. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION | 4 1. INTRODUCTION This study evaluates the performance of USAID/Cambodia’s social accountability programs between 2019 and 2022. During this time, USAID/Cambodia’s social accountability portfolio was composed of one flagship activity under a Leader with Associates award and three Small Business Applied Research (SBAR) activities under contracts. The portfolio encompassed four individual activities: 1. ISAC1, a five-year program (2019-2024) designed to support citizens to improve their capacity for collective action, increase access to information about government accountability, and build public demand for government to develop solutions to address community priorities; 2. OW4C2, a three-year pilot program designed to increase the capacity of civil society to engage in political processes for improved public services through the OWSO; 3. T4GC3, a three-year pilot program designed to improve Cambodia’s public service delivery and waste management quality and transparency; and 4. SWAP4, a three-year program designed to increase accountability and responsiveness in municipal solid waste collection and management. These activities were designed and procured in response to the DRG Cambodia Assessment conducted in 2019,5 which recommended using technology and social media innovations to increase citizens’ ability to demand access to public services, provide feedback on the quality of those services, and ultimately hold local government accountable. Because the SWAP activity is already being evaluated by the DevLab@Duke University through a separate mechanism, the focus of this performance evaluation is on ISAC, OW4C, and T4GC. The intended primary audience of the evaluation is USAID/Cambodia’s DG Office and the primary objectives of this performance evaluation are to: 1. Evaluate the validity of the theories of change underlying the DG office’s social accountability program; 2. Assess the performance of the activities and their effectiveness at improving local government accountability; and 3. Identify actionable recommendations that can inform future ISAC programming and activity/portfolio designs for USAID technical offices. USAID/Cambodia requested that NORC’s DRG-LER II activity design and conduct an independent performance evaluation of ISAC, OW4C, and T4GC activities. The evaluation team includes Dr. Ben 1 ISAC, a $16 million activity, was awarded in October 2019 (Award #: 72044219LA00002). The implementer is FHI360, and their subcontractors are Internews, DAI, Open Development Cambodia (ODC). 2 OW4C, a 3 million activity, was awarded in October 2019 (Award #: 72044219C00001). The implementer is Nickol Global Solutions LLCm and their subcontractors are Advocacy and Policy Institute (API), Innovative Support to Emergencies, Diseases and Disasters I-Lab Southeast Asia (InSTEDD). 3 T4GC, a $3 million activity, was awarded to I4DI in October 2019 (Award #: 2044219C00002). 4 SWAP, a $3 million activity, was awarded in October 2019 (Award #: 72044219C00003). The implementer is Triangle Environmental Health Initiative, and their subcontractors are The Asia Foundation, The DevLab@Duke University. 5 The Cambodia DRG assessment can be accessed at: https://pdf.usaid.gov/pdf_docs/pa00tp92.pdf 5 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV Morse (Co-Team Lead from Social Impact, SI), Hyosun Bae (Co-Team Lead from SI), Bailey Fohr (Research Assistant from SI), Sue Nelson (Advisor), and Bopha Kong (Logistics/Data Collection Support). 2. BACKGROUND In this section, we provide an overview of USAID/Cambodia’s social accountability portfolio. We cover each of the three activities in turn. ISAC OBJECTIVES AND INTERVENTIONS The goal of the ISAC activity is to improve citizens’ capacity for civic engagement, increase their knowledge and access to information around collective action, and assist them in identifying, raising, and solving community needs. ISAC began their activities in six provinces of Cambodia (Banteay Meanchey, Battambang, Kampong Chhnang, Kampot, Kandal, Siem Reap) and expanded to in the capital city of Phnom Penh in Year 2 of activity implementation. The theory of change for the ISAC Activity entails: 1. If Cambodian citizens generate actionable information about public services, spending, and investment, and encourage collective action around initiatives to improve public services at the local government level; 2. Then citizen demand will be stronger, better organized, and better informed and local authorities will be motivated to respond to citizen demands with improved services, and eventually all will benefit from cooperation6. To this end, the ISAC Activity consisted of four main sub-activities: 1) Support for civil society organizations (CSOs) to do outreach via Facebook and Telegram, 2) the CAF program, 3) door-to-door outreach and community meetings to raise awareness of accountability, and 4) community scorecards, involving both identification and implementation of identified priority actions. ISAC supported ISAC’s local IPs and Digital Innovation and Information Fund7 (DIIF) grantees to assist their outreach using social media (Facebook, Telegram channel, and Android, iOS, and web-based applications). The support focused on helping these organizations use social media to promote accountability. It also sought to help them run campaigns and advertisements to disseminate activity information to target audiences. Support provided to the CSOs included: 1) trainings on how to set up and manage a social media account, increase online engagement using text, images, and videos, and ensure digital safety and security, 2) funding to procure external resources to set up digital tools such as Telegram channels and web-based applications, and 3) review of and feedback on digital tool prototypes. 6 ISAC Activity Monitoring, Evaluation and Learning Plan, p. 2. 7 The DIIF was added to ISAC programming in 2020, when the activity developed a set of criteria and a selection process for grantees that would receive technical oversight and support on digital outreach efforts. ISAC provided advisory support to these grantees on citizen-focused user testing for the digital tool prototypes/wireframes. Four DIIF grants have been disbursed at the end of 2021. Two of the grantees were also ISAC’s implementing partners. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION | 6 The second sub-activity, the CAF program, involved the selection of youth from targeted communities to assist the ISAC IPs with implementing community outreach activities. These youth were then trained on ISAC’s campaigns and assisted in Information for Citizens (I4C) dissemination through various outreach components including door-to-door outreach, mobile loudspeaker, community meetings, and information sharing via social media. CAFs also assisted with community scorecard meetings. In addition, the CAF program sought to provide civic skills to build a culture of accountability and promote civic participation among facilitators. The third sub-activity consisted of door-to-door outreach and community meetings, which were implemented by CSOs and CAFs. The purpose of these activities was to raise awareness of civic education, accountability and I4C among citizens. Information shared included information about citizens’ rights, budget and performance of the health center, primary school, and the Sangkat administration. The fourth and final sub-activity, the community scorecard meetings and creation of implementation committee, was also implemented by local CSOs and CAFs. Community scorecards began with a series of participatory town-hall style meetings to identify health, school, and Sangkat administration issues and brainstorm ideas to address them. These meetings were attended by local residents and service providers such as schoolteachers, health workers, and village authorities. The second step in the scorecard process was to hold prioritization meetings in which priority issues for action were identified. In the final steps, the priority needs were consolidated and citizens and service providers met again, this time together, to re-score priority needs from most to least important, and then discuss with the Sangkat administration how to best address these. A committee was then formed to oversee the implementation of the identified activities. These meetings took place over a cycle of four consecutive meetings per year: Inception Meetings, Shared Vision Meetings, Scoring Meetings, and Interface Meetings. OW4C OBJECTIVES AND INTERVENTIONS The goal of the OW4C activity is to test new approaches to increase the capability of civil society to engage the Royal Government of Cambodia in accessing improved public services. The OW4C activity was premised on the following three-step theory of change: 1. If citizens have real-time access to provide and receive reliable information about the Royal Government of Cambodia’s One Window Service Office (OWSO) services through civic tech innovations, 2. And if they are engaged through meaningful processes, 3. Then they will more effectively hold the government accountable to improve public services through the OWSO. The Activity was considered to be an initial pilot of a hybrid approach to programming that links traditional accountability activities such as community scorecards, service improvement plans, and public forums to citizen-centered designed tech tools. The pilot hybrid model was piloted in the provinces of Battambang (Battambang Municipality and the districts of Bavel, Thma Koul, Moung Russie, and Kamrieng), Banteay Meanchey (Serei Sarophon and PoiPet Municipalities) and Kampong Chhnang (Kampong Chhnang municipality). 7 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV The OW4C Activity consisted of six main sub-activities. The first was an online Facebook Chatbot, branded as “Pidor the Smart Villager,” which citizens can use to access information about One Window Service Offices.8 The Chatbot provides information about how to access OWSO services, fees associated with OWSO services, and delivery times. The Chatbot also provides an anonymous feedback system for those who used the OWSO services and want to give feedback. Alongside the Chatbot, OW4C also created an Interactive Voice Response (IVR) call-in line for citizens who lack smartphones or literacy to access information about OWSO services. Due to resource constraints, the IVR component of this activity is not covered by this evaluation. The second component of the OW4C Activity was the Youth Ambassadors Program. YAs are university students that are trained on the roles of the OWSO and ombudsperson, the Facebook Chatbot, and how to engage communities and conduct scorecards. They conducted much of the door-to-door outreach to raise awareness of the Chatbot. In addition, the YA program sought to provide civic skills to build a culture of accountability and promote civic participation. Youth Ambassadors were critical to the third component of the OW4C Activity, which involved community outreach through door-to-door canvassing and community meetings to raise awareness of OWSO and the Facebook Chatbot. The fourth component of the OW4C activity was a public-access online dashboard where citizens and CSOs could access information on citizens’ feedback regarding the services received at the One Window Service Offices.9 In addition to citizens, CSOs, and journalists, the website was used by district ombudspersons (DOs) to track feedback on lower-level OWSO officials. The fifth component of the OW4C Activity were community scorecards, in which OW4C staff facilitated one-off meetings with citizens who used OWSO services to assess their quality and identify priority areas for improvement. OW4C staff then shared this feedback with the DOs, OWSO Chiefs, and district officials, which then led to the development of the OWSO Accountability Working Group that defined action points to act upon citizen suggested improvements. OWSO officials did not attend the meetings. The sixth component of the OW4C Activity was public forums where district and OWSO officials and the DO presented to the public on the status of the OWSO improvement plans and took questions and comments from community members in attendance. Most of the questions from citizens during these forums were related to their personal experiences with OWSO services, such as following-up on the timeline for receiving a response to a service request. T4GC OBJECTIVES AND INTERVENTIONS The goal of the T4GC activity is to improve responsiveness and efficiency of waste management services (WMS) delivery in target locations. The activity was premised on the following five-step theory of change10: 8 See https://www.facebook.com/pidorow4cbot/. Last Accessed August 5, 2022. 9 See https://dashboard.ow4c.info/. Last Accessed August 5, 2022. 10 This Theory of Change is drafted by the ET based on the T4GC MEL Plan Logical Framework. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION | 8 1. If citizens and WMS providers have actionable data on WMS performance, efficiency, and responsiveness; 2. And if stakeholders actively engage in waste management information sharing; 3. And if stakeholders organize/lead accountability networks that constructively support accountability and info mediation; 4. And if stakeholders promote circular economy in Cambodia through efforts that reduce-reuse￾recycle (3R) waste consistent with national 3R strategy; 5. Then WMS delivery will improve in target locations. In the first year of implementation, T4GC was piloted in 28 communes/Sangkats (10 target districts/municipalities) of Battambang and Banteay Meanchey. During the second year, T4GC expanded to two additional locations in Samraong municipality (four Sangkats) in Oddar Meanchey and Kampong Chhnang municipality (four Sangkats) in Kampong Chhnang province. The T4GC Activity consisted of four main sub-activities.11 The first was a mobile application branded as “Green Cambodia Citizens App (GCC)”. The GCC App has four features: 1) “report” feature allows citizens to upload WM issues by tagged with photos and GPS location, 2) “find” feature provides information on waste collection service schedules, routes, and events, 3) “learn” feature displays resources on waste management best practices and 3R, and 4) “marketplace” feature connects users to offer up and buy used goods. The second component of the T4GC activity was an online dashboard allowing access to access real￾time information on citizens reports submitted via the GCC App.12 There are two main users of the Dashboard: Waste Management Service Providers (WMSP) and Waste Management Authorities (WMA). WMSPs refer to waste management companies contracted by the local government to collect and dispose solid waste from paid customers and/or from public areas. WMAs refer subnational government officials at municipalities or districts who are responsible for waste management. The purpose of the Dashboard was to help WMAs monitor the performance of WMSP and track their progress towards addressing WM issues. The third component of the T4GC activity was the Youth Ambassador/Community Empowerment Facilitator (hereafter, YA) Program. Highschool students, university students, and community members were trained on the GCC App, WM best practices, and 3R strategies. They conducted much of the door-to-door outreach to raise awareness of the GCC App. In addition, the YA program sought to provide civic skills to build a culture of accountability and promote civic participation. YAs were critical to the third component of the T4GC activity, which involved community outreach through door-to-door canvassing, community meeting facilitation, school campaigns, community clean￾11 The T4GC activity also included the following sub-activities which were not included in the evaluation: training of CSOs and WMSPs; quarterly stakeholder/coordination meetings; and outreach to large waste producers. The ET determined the sub-activities to be included in the evaluation scope based on the scope of the sub-activities, comparability with other Activities, and feedback from USAID. 12 To view the Dashboard, visit https://i4di.org/reports/t4gc-dashboard/dashboard/dashboard.html. Last Accessed August 29, 2022. 9 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV up campaigns, social media activism, and Facebook and radio talk shows. These outreach activities were all aimed at promoting the GCC App and raising awareness of WM best practices and 3R strategies. 3. EVALUATION QUESTIONS The study seeks to answer the following overarching questions EQ 1: How effective have technology and youth engagement activities been at sharing information about government performance, increasing citizen engagement with local administration and service providers, and raising awareness of government accountability? EQ 2: To what extent have social accountability activities increased government accountability at the local level? EQ 3: How appropriate and/or useful were USAID/Cambodia’s goals and objectives for the social accountability activities in achieving the Mission’s DRG objectives? These overarching EQs are operationalized in the context of specific sub-activities in the Table 2 Evaluation Design Matrix below.14 13: 4. EVALUATION DESIGN AND METHODOLOGY This evaluation began in February 2022 with a series of co-creation workshops and consultations attended by the ET, USAID/Cambodia DG staff, and IP staff from the implementing agencies for each of the three Activities. Decisions made during these meetings regarding the evaluation’s scope and methodology were reflected in the subsequent Evaluation Design and Data Collection Plan, which was reviewed by each IP and approved by USAID/Cambodia. Feb Mar Apr May Jun Jul Aug Co-Creation Meetings Document review Evaluation Design & Data Collection Preparations Data collection Data analysis Figure 1: Evaluation Timeline (2022) During this phase, the ET received over 230 files, reports, and beneficiary contact lists from the IPs. As part of the evaluation design process, the ET reviewed these documents to brainstorm potential evaluation strategies, identify research subjects and key informants, and design data collection instruments. This process resulted in an evaluation designed based on two key sources of data: key 13 A fourth Evaluation Question, “What are some potential strategies for sustainability of the program approaches developed under USG assistance, such as tech tools, youth engagement, etc.?” was included in the Concept Note for this study but subsequently dropped during the co-creation process because the social accountability program’s activities are linked to USAID funding and cannot be expected to continue in the absence of this funding. 14 This report uses the term ‘Activity’ to refer to ISAC, OW4C, and T4GC. We use the term ‘sub-activity’ to refer to specific components of the overarching Activities, examples of which are articulated in the rows of the Evaluation Design Matrix below. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION | 10 informant interviews (KIIs) with project beneficiaries and stakeholders, and surveys with project implementation staff. In addition, during the analysis phase of the project, the ET determined that web analytics data from the mobile apps and dashboards created under the T4GC and OW4C Activities would provide valuable insight into the EQs. Accordingly, the ET also requested these data from the OW4C and T4GC IPs. We discuss each of these three data sources in further detail below. METHODOLOGY AND DATA The EQs are inherently causal in nature – they inquire about the causal impact of the technology and youth engagement sub-activities on citizen engagement and awareness of government accountability (EQ1) and the impact of social accountability sub-activities on government accountability (EQ2). Answering causal questions requires identifying a comparison group that accurately captures the counterfactual – in this case, what levels of citizen engagement and government performance would have been in the absence of ISAC, OW4C, and T4GC activities. But because this performance evaluation covered all three activities, each of which had numerous sub-activities linked to distinct outcomes, it was not possible to conduct an experimental or quasi-experimental impact evaluation with comparison groups for each sub-activity. Instead, the evaluation addresses the EQs using descriptive data about how beneficiaries perceived ISAC, OW4C, and T4GC services, what effect they think these activities had on accountability outcomes, and what evidence they can provide to substantiate these claims. It is also important to note that this evaluation does not cover each sub-activity completed by the three activities but rather prioritizes evaluating the sub-activities that overlap between the three activities and those that are directly related to answer the evaluation questions. To collect this information, the ET conducted two distinct primary data collection activities: a closed￾ended, quantitative survey of YAs (or CAFs, for ISAC), and KIIs with various stakeholders associated with the OW4C, ISAC, and T4GC activities. These data sources and their use to answer the EQs are summarized in our Evaluation Design Matrix (Table 2). Table 1: Number of KIIs by Activity and Respondent Type ISAC Activity Number of KIIs Community Accountability Facilitators 15 CSOs 6 Service providers 10 Citizens 10 OW4C Activity YAs 15 District/Provincial Ombudspersons 6 OWSO officials 8 Public Forum attendees 5 Community Scorecard participants 5 T4GC Activity Youth Ambassadors 15 Waste Management Service Providers 5 Waste Management Municipal Authorities 5 Total 105 11 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION USAID.GOV Data collection began in June 2022 and ran through August 2022. In total, the ET conducted 299 surveys and 105 KIIs across the three activities. For the survey, this sample size provides a margin of error of +/- five percentage points for binary outcomes. All 299 surveys were conducted over the phone.15 The breakdown of KIIs across respondent types is presented in Table 1. 40 percent of the KIIs were conducted in-person; the remaining KIIs were conducted virtually via Microsoft Teams or over the phone. Data was collected by a local data collection firm, Indochina Research, as well as by two in￾country consultants. In addition to the YA survey and KIIs, the ET analyzes site visit data from the OW4C Dashboard, focusing on the total number of visits to the site since its launch, as well as visit patterns over time. The latter is particularly insightful for understanding demand for the Dashboard over time – strong user demand for the Dashboard will result in steady growth in site visits over time. As users tell their colleagues about the sites, who in turn tell their friends, site visits should increase exponentially. Conversely, if the Dashboard site is not valued by users, then we will expect demand to spike during periods of peak outreach but thereafter quickly diminish. Using the same logic, the ET also analyzes patterns of App/Chatbot usage by citizens over time, drawing on data gleaned from the Dashboard. 15 Out of 302 ISAC CAFs, 142 were surveyed. Out of 129 OW4C YAs, 78 were surveyed. Out of 124 T4GC YAs, 78 were surveyed. The ET made at least three attempts to survey the YAs/CAFs. Those who were not interviewed as part of the YA survey either: 1) refused to participate; 2) did not pick up the call after three attempts; 3) could not be reached due to phone connection; or 4) had incorrect phone number. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DRAFT REPORT | 12 EVALUATION DESIGN MATRIX Table 2: Evaluation Design Matrix Sub-Activities ISAC OW4C T4GC Sub-Activity-specific Sub-Evaluation Questions16 Data Sources & Data Collection Methods EQ 1: How effective have technology and youth engagement activities been at sharing information about government performance, increasing citizen engagement with local administration and service providers, and raising awareness of government accountability? Facebook Chatbot  Was the Facebook Chatbot viewed as an effective means of accessing information about OWSO services? Did it make it easier for citizens to access OWSO services and improve their experience accessing those services? YA survey KIIs with YAs Chatbot Usage Data Green Cambodia Citizens App  Was the Green Cambodia Citizens App viewed as an effective means of reporting WM issues to WMSPs? YA survey KIIs with YAs App usage data Support for CSOs to do outreach via Facebook and Telegram  Do CSOs/grantees view social media as a viable avenue to engage with citizens? To what extent did social media outreach generate meaningful engagement from citizens? KIIs with CSOs Youth Ambassadors (YAs) / Community Accountability Facilitators (CAFs)    Did the YA17 program raise awareness about the importance of government accountability and integrity among Ambassadors? Did it increase their participation in civic affairs? YA survey KIIs with YAs Door-to-door outreach    Was door-to-door outreach by YAs effective at raising awareness of the Apps / Facebook Chatbot? Was it effective at raising awareness about the importance of accountability and civic participation? YA survey KIIs with YAs EQ 2: To what extent have social accountability activities increased government accountability at the local level? Online Dashboards   Was the dashboard valued by District and Provincial Ombudspersons / WMAs / WMSPs? Did it help them improve the quality of their services? KIIs with DOs/POs KIIs with WMAs KIIs with WMSPs Dashboard analytics 16 This column indicates how the overarching Evaluation Questions are operationalized in the context of specific sub-activities. 17 To achieve consistency across Activities, we use the term Youth Ambassador rather than Community Accountability Facilitator, even though this was the term used for the ISAC activity. 13 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE DRAFT REPORT USAID.GOV Sub-Activities ISAC OW4C T4GC Sub-Activity-specific Sub-Evaluation Questions16 Data Sources & Data Collection Methods Community Scorecards   Did the Scorecards program increase citizens' access to information about local government performance? Did it increase citizen engagement with local officials and/or service providers? Did it raise awareness about government accountability? Did local officials and/or service providers feel more accountable to citizens as a result of the program? And did the quality of local services improve? YA survey KIIs with YAs KIIs with participating officials KIIs with participating citizens KIIs with participating service providers Public forums  Did the forums increase citizens' access to information about local government performance? Did they increase citizen engagement with local officials and/or service providers? Did they raise awareness about government accountability? Did local officials and/or service providers feel more accountable to citizens as a result of the forums? And did the quality of local services improve? YA survey KIIs with YAs KIIs with participating officials KIIs with participating citizens EQ 3: How appropriate and/or useful were USAID/Cambodia’s goals and objectives for the social accountability activities in achieving the Mission’s DRG objectives? N/A    USAID/Cambodia’s DRG objectives 1. Improved access to information on government decisions and processes, and the planning, spending, and investment of public funds 2. Increased citizen participation and engagement to improve accountability for public services 3. Increased utilization of new or existing technologies enabling citizen engagement to increase accountability for public services 4. Education and outreach to raise awareness of accountability and integrity in public administration Synthesis of findings on EQ1 and EQ2 + insights from KIIs USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 14 LIMITATIONS There are two primary limitations with our evaluation design. The first, as alluded to above, is the absence of an experimental or quasi-experimental design to answer causal questions about the impact of the ISAC, OW4C, and T4GC activities. Throughout the social science community, experimental or quasi-experimental methods are accepted as the only methods suitable for providing reliable evidence on a program’s causal impact.18 These standards are also accepted by USAID’s own standards and guidelines (ADS 201.3.4.9.B.II).19 However, employing experimental or quasi-experimental methods in this evaluation was not feasible due to the expansive breadth of the ISAC, OW4C, and T4GC activities. Instead of employing an experimental or quasi-experimental design, we rely on descriptive assessments of the activities collected through surveys and KIIs with activity beneficiaries and stakeholders. Such evidence provides insights on respondents’ self-assessments of the activities’ impacts, but they are also subject to response bias by participants. This concern looms large in the present case, where many of the respondents were direct or indirect beneficiaries of the program or were directly or indirectly employed by the program. In this setting, respondents may feel obliged to portray the program in a positive light. In addition to the potential for response bias, KIIs may be skewed by outlying anecdotes of impact rather than average impacts across the entire sample. We attempt to mitigate these limitations through two strategies. First, we complement the surveys and KIIs with data on actual use of the mobile Apps/Chatbots and site visits to online Dashboards. Such data provides an objective and real-world behavioral measure of beneficiary demand for, and usage of, the primary products delivered by the OW4C and T4GC activities. Second, we analyze and interpret self￾reported data with potential response biases in mind, upweighting respondent testimony that is substantiated by specific examples of impact and discounting testimony that is general in nature and not supported by examples. We also take care to analyze the totality of KII and survey evidence, to ensure our findings are not skewed by outlying cases. 5. FINDINGS AND CONCLUSIONS We organize our findings according to the Evaluation Design Matrix (Table 2). We start by evaluating the technology sub-activities, specifically the Facebook Chatbot, the Green Cambodia Citizens App, ISAC’s support for CSOs to increase their use of social media to generate citizen engagement. We then proceed to evaluate the two main youth engagement sub-activities: the Youth Ambassadors program and the Community Outreach program. We analyze these two sub-activities separately by ISAC, OW4C, and T4GC because each activity was run independently. EFFECTIVENESS OF TECHNOLOGY & YOUTH ENGAGEMENT 18 See e.g., Angrist, Joshua D., and Jörn-Steffen Pischke. Mostly harmless econometrics: An empiricist’s companion. Princeton University Press, 2009. 19 For an overview of USAID’s Impact Evaluation and Performance Evaluation policies, visit https://www.usaid.gov/project-starter/program-cycle/project-monitor-evaluation-plan/monitor-evaluation-plan￾evaluation-component/impact-evaluation-designs and https://www.usaid.gov/project-starter/program-cycle/project￾monitor-evaluation-plan/monitor-evaluation-plan-evaluation-component/performance-evaluation-designs. Accessed August 30, 2022. 15 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV FACEBOOK CHATBOT This section reports on two metrics of the effectiveness of the Facebook Chatbot at helping citizens access information about OWSO services and providing feedback on those services: usage statistics and feedback from Youth Ambassadors. The former source provides an objective measure of demand for the Chatbot among citizens; the latter provides insights into why citizens may or may not value the Chatbot, but is also subjective to certain response biases given that Youth Ambassadors were integral to the implementation of Chatbot outreach. Figure 2: Screenshot of OW4C Facebook Chatbot (Source: OW4C Facebook page) Regarding usage, we pay particular attention to patterns of usage over time. If a product is valued by citizens, they will continue using that product over time; they will also recommend the product to their friends and family, leading to steady and potentially exponential growth. Conversely, if a product does not prove to be useful, then usage will putter-out after an initial period of advertising or outreach, as citizens try a product once or twice before giving up on it.20 Based on this logic, we interpret growing and sustained usage over time as evidence that the Facebook Chatbot was valued by citizens, helped them access relevant OWSO information, and helped them more easily provide feedback to OWSO officials. Conversely, we interpret “boom-and-bust” trends over time as evidence that that the Facebook Chatbot was not viewed as an effective means to access information or provide feedback regarding OWSO services. Figure 3 and Figure 4 display usage trends over time for both accessing information via the Chatbot (Figure 3) and providing feedback on services received (Figure 4). The results suggest that usage patterns overwhelmingly conform to the “boom-and-bust” pattern of usage. In Battambang province, usage spikes to 2,076/month in November 2020, before dropping off for the remainder of the Activity period. In 20 It is important to keep in mind that we are referring to demand for OWSO services (and by extension, OW4C tools) in the aggregate across entire communities or provinces where the tools were introduced. Although any one individual’s demand may be infrequent (after all, one only needs a new license or business certificate every few years), demand at the aggregate level across entire communities should be steady. Considerable demand at the aggregate level is evidenced in the data by comparatively high levels of usage during “boom” periods where the tools were introduced. If these tools were valued by these initial users, we would expect them to tell their family, friends, and colleagues about the OW4C tech tools, and for at least some of these people to then need these services in the near future, who then go on to tell their friends and family, thereby stimulating sustained demand on an aggregate level across entire communities. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 16 Banteay Meanchey province, use of the Pidor Chatbot is limited until February 2022, whereupon usage spikes to between 1,901/month and 837/month for five months before dropping off in July and early August. And in Kampong Chhnang, usage is high in December 2021, January 2022, March to April 2022, and June 2022, but is otherwise very low. Battambang Province Banteay Meanchey Province Kampong Chhnang Province Figure 3: Demand for OW4C Chatbot - Accessing Information about OWSO Services (Source: OW4C Online Dashboard) 17 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Usage patterns for the Pidor Chatbot’s feedback feature is similarly “spikey” (Figure 4). For Banteay Meanchey province, feedback reports submitted through the Pidor Chatbot spike in May 2021 and February 2022, but are otherwise very limited. In Battambang province, feedback reports spike in November 2020, but remain relatively low thereafter. And in Kampong Chhnang province, feedback reports spike in May 2021, December 2021, and March 2022, but are otherwise low. Battambang Province Banteay Meanchey Province Kampong Chhnang Province Figure 4: Demand for OW4C Chatbot - Providing Feedback on OWSO Services (Source: OW4C Dashboard) One likely explanation for this finding is that boom-and-bust patterns correspond to periods of peak vs. dormant outreach by the OW4C field teams. As of writing, the ET does not have precise date and USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 18 location data information about outreach activities to confirm this interpretation. Regardless, the data is not consistent with the idea that citizens value the Pidor Chatbot as a means to access information about OWSO or provide feedback on its services. Rather, citizens introduced to the Chatbot appear to use it just once or twice before reverting to status quo ante patterns of engagement. Interviews with the YA KIIs provide insight into why usage of the Chatbot was not sustainable. The most common explanations related to three limitations of the Chatbot’s viability: low internet penetration, limited smartphone ownership, and low smartphone literacy (YA Interview 08, 11, 14).21 As one respondent explained, “most people become aware of OWSO through leaflet and poster dissemination rather than through the Chatbot. This is because most people do not have smart phones, especially students and the elderly. Another barrier is the low internet service in their living areas, so it becomes too complicated for them to access the Chatbot (YA Interview 11). Echoing this sentiment, another YA expressed uncertainty about whether demand for the Chatbot would be sustained over time: “During the dissemination, we always remind people to use Pidor the Smart Villager when they want any service of OWSO. But I cannot evaluate whether citizens are still getting this information from their friends or family. As a youth ambassador, I would hope they do not forget Pidor the Smart Villager” (YA Interview 08). Despite the boom-bust patterns of demand for the Chatbot and some YAs reservations about its viability, YAs were otherwise extremely positive about the Pidor the Smart Villager Chatbot. The Chatbot was cited as a more reliable source of information (YA Interview 01, 02), a more comprehensive source of information (YA Interview 05, 12), and an important time saver relative to traveling to OWSO centers in person (YA Interview 03). Respondents also reported that clear information about the fees associated with certain services helped prevent corruption in the form of OWSO employees overstating fees (YA Interview 06, 09). YA feedback from the YA survey was also overwhelmingly positive (Annex D). 95 percent of respondents agreed or strongly agreed that “the Pidor the Smart Villager Facebook Chatbot made it easier for citizens to access OWSO services.” 98 percent said that the Chatbot enabled citizens to get answers to questions they otherwise would not be able to answer. And an overwhelming majority agreed or strongly agreed that people who use the Chatbot once are likely to use it again (97 percent) and to recommend it to their friends (95 percent). It is important to interpret these numbers with caution, however, because of the potential for response bias among informants so closely involved in implementation, and because of the inconsistency of these perspectives with real-world patterns of demand for the Chatbot. YAs were not quite as optimistic about citizens’ willingness to use the Chatbot to provide feedback on OWSO services received. Consistent with the low demand for the OWSO feedback feature of the Chatbot relative to the OWSO information feature, YAs stressed that citizens were not fully assured that their feedback would be anonymous (YA Interview 02, 04, 05, 10, 14). Despite YAs’ efforts to explain the anonymity features of the Chatbot “people still felt afraid that all their personal information would be known to OWSO officials, including their name and residence” (YA Interview 05). Another 21 It is important to note that OW4C foresaw the challenge of low internet penetration and/or low literacy, and for this reason created the Interactive Voice Response call-in hotline, which provides the same set of information. Evaluating this technology was beyond the scope of this evaluation. 19 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV worried that “citizens might not use the feedback feature because they are afraid of speaking wrong even though we explained clearly about anonymity when using it. When talking about feedback or critique of government officials, they feel fear” (YA Interview 14). GREEN CAMBODIA CITIZENS APP We use GCC App usage statistics and feedback from interviews with YAs and WMSPs to evaluate the effectiveness of the GCC App at helping citizens report WM issues and provide feedback to WMSPs. We use longitudinal data on App usage as a measure of demand for the App among residents and interviews with YAs for additional insights into whether and why citizens valued the App. Figure 5: Green Cambodia Citizens App (Source: T4GC) Data on the number of GCC App downloads, sign-ups, and reports indicate usage of the GCC app fluctuated between May 202122 and July 2022. Figure 6 displays the total number of App downloads and the number of sign-ups per province.23 As of August 5, 2022, the GCC App was downloaded 25,759 times and has a total of 18,452 sign-ups. The number of downloads peaked in June, August, September, and December 2021 as well as in June and July 2022. However, new downloads and sign-ups are not sustained overtime; after each peak, they drop-off precipitously. 22 Note that the GCC App launched in November/December 2020 but data on downloads and sign-ups are only available starting May 2021. 23 The number of downloads refer to the number of times the Green Cambodia Citizens App was downloaded from the Apple App store and Google Play store. Downloading the App does not require the user to sign-up. To complete the sign-up process, the App requires a Cambodian phone number and an email address. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 20 Figure 6: Green Cambodia Citizens App total downloads and sign-ups by province (Source: T4GC Online Dashboard) Patterns of citizen reports24 submitted through the GCC App show a similar pattern of short spurts of reporting followed by periods of reduced activity. For Battambang and Banteay Meanchey, the number of reports peaked between May to July 2021 and again in February to April 2022. Between these periods, very few reports were recorded. In Oddar Meanchey and Kampong Chhnang provinces, where the GCC App was not fully launched until July 2021, reports peaked in January 2022 before tapering off entirely. In addition to its boom-and-bust nature, reporting remained low overall throughout the GCC App lifespan, even during periods of peak reporting.25 Across all four provinces, there were only 710 verified reports from the time the App was launched in December 2020 through August 2022, or roughly 1.1 reports per day. Given the population of the targeted provinces and communities (estimated to be over 400,00026) and the scale of the waste management issues in these areas, it is unlikely that the GCC App had a meaningful impact on service providers’ knowledge of priority waste management issues needing attention. Together, the boom-and-bust pattern of reporting and the low overall number of reports suggest that citizens did not view the app as an effective means of providing feedback to WMSPs. Once introduced to the App, citizens used the App just a couple times before returning to ex-ante methods of feedback and reporting. 24 Note that the number of reports presented in this Figure excludes tests and illustrative reports submitted as part of App demonstrations or training. 25 The greatest single-day number of reports was only 22, in Kampong Chhnang province. 26 The estimated population targeted by T4GC (424,485) was calculated using the average population size of sangkats/communes in Battambang and Banteay Meanchey provinces (141,014 and 185,636 respectively) where the T4GC Activity was implemented across 28 communes/sangkats. For the Oddar Meancehy and Kampong Chnnang provinces, the average population size of Samraong municipality (56,755) and the Kampong Chhnang municipality (41,080) sangkats/communes were used. T4GC Activity was implemented in four sangkats out of five within the Samraong municipality and all four sangkats in the Kampong Chhnang municipality. 21 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Battambang Banteay Meanchey Oddar Meanchey Kampong Chhnang Figure 7: Green Cambodia Citizens App reports by province and waste type (Source: T4GC Online Dashboard) Evidence from the YA surveys and KIIs is mixed and partially inconsistent with evidence from the App reporting data. Contrary to what the limited and boom-bust patterns of reporting imply, YAs had USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 22 exceedingly positive views about the App. For example, 100 percent of surveyed youth agreed or strongly agreed with the statement “The GCC App made it easier to report waste management issues to service providers,” and 97 percent of surveyed YAs agreed or strongly agreed that “citizens who used GCC App once were likely to use it again” and that “citizens viewed GCC App as an effective avenue for providing feedback.” During KIIs, YAs cite the many ways the GCC App makes it easier for citizens to report WM issues: reporting takes less time, location information is more precise, and anonymity increases participation. Several youths acknowledged that the App helps citizens report missed waste collection (YA Interview 01, 05, 08). YAs also highlighted the benefits of anonymous reporting (YA Interview 01, 02, 12). “Before the App, they did not want to go and make a report to authorities or service providers in person out of fear that it could cause troubles for themselves,” a YA explained during an interview. “However, with the App, there is nothing for them to fear because their identity is unknown” (YA Interview 01). Overall, these positive perspectives on the GCC App’s effectiveness are inconsistent with the actual usage patterns of the App. But upon further inquiry, YAs also revealed several shortcomings to the App that may help explain its limited and sporadic usage. Several YAs highlighted the challenge of getting citizens to use the App consistently. When YAs were asked during the KIIs if citizens viewed the GCC App as an effective measure to report on WM issues, six out of 15 YAs replied that the App (and smartphones in general) are too difficult for citizens, especially older generations (YA Interview 02, 03, 06, 07, 09, 11). YAs also mentioned that some citizens did not own smartphones in rural areas (YA Interview 01, 03) or found the registration process to be too complicated (YA Interview 11). For instance, while agreeing that the App is an effective method of reporting issue, a YA claimed that “not all of citizens use it, probably because of the difficulties in using the app” (YA Interview 09). Another reason cited by YAs was a lack of demand from the citizens for WM reporting and services (YA Interview 01, 06, 08, 15). According to several YAs, waste management services such as trash pick￾up are very limited, particularly in rural areas; instead, citizens simply burn or bury their trash. In the words of a YA, “In the city, citizens have greater awareness. However, in rural areas, only about 50 percent of them because people simply burn waste on their own property” (YA Interview 01). A third reason why usage of the App was not have been sustained may be that some citizens felt that their feedback and reports were not taken seriously by WMSPs. In several interviews, YAs expressed concerns that their reports were not being acted on in the rural areas (YA Interview 07, 12). A YA explained that “people find [the App] ineffective because the WMSP services are still limited in some markets …they complain sometimes that nobody comes to collect the waste even if they reported to the WMSP already” (YA Interview 07). TECH SUPPORT FOR CSOS A key component of the ISAC Activity was the provision of technical support to CSOs to help them make greater use of social media to generate citizen engagement. This support was provided in two forms: i) as part of on-going support for CSOs who were selected as ISAC’s IPs, and ii) as DIIF grants for select CSOs to develop digital tools. Funding and trainings on a range of topics were offered to help ISAC partner CSOs become more equipped to create engaging and useful content on their social media pages. 23 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Evidence from KIIs with CSO employees suggest that CSOs/grantees believe greater use of social media as a valuable way to engage with citizens. Four out of six CSOs specified that social media makes it easier for citizens to access information and provide feedback (CSO interview 02, 03, 04, 05). Referring to their outreach activities prior to using social media, a CSO said, “Until now, we could access I4C only at the commune hall, primary school, and health center. But now, information is at citizens’ fingertips” (CSO Interview 02). This convenience is especially helpful for reaching populations based in rural areas. According to a CSO, social media is widely used because “even citizens living in remote areas can access the service easily and quickly” (CSO Interview 04). Two CSOs highlighted that the use of social media was particularly helpful during the COVID-19 pandemic because it allowed them to disseminate information without meeting in person (CSO Interview 02, 05). A CSO noted “During the Covid-19 outbreak, we relied on online communication and social media in reaching out our target groups” (CSO Interview 05). Respondents also reported that social media is a useful tool to reach a wider audience than they could through traditional means (CSO Interview 01, 04). For example, one respondent stated, “citizens, not only in the project coverage areas but also outside of project areas, can review and learn about I4C, their rights, and standard service” (CSO Interview 01). Another said that they use social media “to share information on their commune/sangkat and bring the issues raised by citizens in the group to the local authorities because [it] includes all stakeholders” (CSO Interview 04). One CSO shared the interesting view that social media is especially attractive because something they can manage and sustain even after external funding and support expire. “In case the project is phasing out,” a CSO claimed, “the social media still can be managed on this issue and the organization can manage the telegram group and Facebook page to ensure that I4C is still available” (CSO Interview 04). Although overall respondents were enthusiastic about their use of social media under ISAC, respondents did note some limitations to the degree to which it can generate meaningful engagement with citizens (CSO Interview 01, 02, 04). A CSO reported “we also have challenges collecting feedback on social media. Only one to three active citizens out of 100 views leave comments on our post. It is the opposite of face-to-face meetings where people raise their concerns” (CSO Interview 01). While they were optimistic about their social media platform, a CSO shared that “some months, there are two or three pieces of feedback or suggestions, some months only one piece of feedback, but some months, we see no feedback or suggestions” (CSO Interview 04). One CSO we interviewed shared a limitation of their social media platform saying that their social media platform “does not give more chances for [the citizens] to participate in specific activities” (CSO Interview 02). Another challenge of social media is that a substantial portion of the population CSOs are trying to reach only have access to older phones or outdated software (CSO Interview 03, 05). As a result, newly developed apps and social media sites do not always work for them (CSO Interview 03). Another respondent shared the difficulties with requiring certain target populations to use new forms of technology, stating “elderly people are not fast learners, and most of the people who are involved in our project are elderly people in the village” (CSO Interview 05). According to a respondent, social media put pressure on the CSOs to create new content and keep the citizens engaged. Pointing out that their social media outreach has already exceeded the initial target USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 24 number of people reached, the CSO reported “some members also leave the group because we do not have new content to keep their attention” (CSO Interview 04). This is particularly concerning because all of the CSOs are currently relying on content, mainly infographics, created by ISAC for disseminating information on social accountability (CSO Interview 01, 03). It is unclear whether the CSOs will be able to re-implement reliable social accountability outreach via social media platforms in the future without ISAC’s support. Figure 8: Screenshots of a Facebook Page and a Telegram Channel managed by CSOs (Source: AMARA’s Facebook page / SK’s Telegram channel) Upon reviewing the social media platforms managed by the CSOs, we found some limited evidence that social media outreach generated meaningful engagement from citizens. Among those we interviewed, ISAC supported CSOs to establish five social media accounts and two web-based applications. The social media post that went the most viral is a 3-minute video posted on a Facebook page explaining Sangkat administration’s roles and activities. This video was viewed more than 226,000 times, shared 188 times, and liked 389 times. Other Facebook pages also had one or two video posts that gained some traction from citizens and was viewed by more than 100 citizens. These were exceptions, however. The majority of 413 I4C posts uploaded on these Facebook pages had less than 10 likes/loves and users rarely commented on the posts. For instance, the five most recent I4C-related posts on a Facebook page of a CSO had an average of 3.4 likes/loves, 0.6 comments, and 0 shares. Telegram channels have more than 1,400 subscribed members and the channel administrators had shared multiple I4C-related videos and photos. I4C-related posts on Telegram channels were viewed by most of the subscribers but limited engagement; the five most recent I4C-related posts on a Telegram channel received an average of 6.8 reactions and 1.6 comments. We found posts on both the Facebook pages and the Telegram channels to be informative, platform-appropriate, and engaging but many of them were repetitive, which potentially could have resulted in citizen engagements dwindling. The ET also learned of frustrations with the length of time this support lasted. For example, ISAC supported a CSO to develop an entirely new app designed to collect citizen feedback. This individual shared dissatisfaction over the limited timeframe stating that “eight months were not enough for both designing [a new] app and implementation” (CSO Interview 06). The respondent also noted delayed 25 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV feedback from ISAC and said that they would have liked more time to manage and operate the App they created. YOUTH AMBASSADORS PROGRAM Youth Ambassadors were responsible for implementing various aspects of the ISAC, OW4C, and T4GC Activities, including the community outreach for activity-sponsored technology (OW4C, T4GC), the public forums (OW4C, T4GC), the community scorecards (ISAC, OW4C), and other forms of community outreach (T4GC). But apart from the direct impact of these sub-activities on accountability outcomes, which we analyze in other sections of this report, the YA program itself may have strengthened accountability outcomes through its impact on the YAs. Accordingly, this section evaluates whether participation in the YA program instills norms of civic and political participation and leadership that live-on after participation in the YA program ends. We examine the YA programs’ lasting impacts on four specific dimensions of change among YAs: development of new skills, instilling norms of civic and political participation, applying these skills to civic and political leadership, and career influences. ISAC CAF Program ISAC CAFs provided largely positive feedback about the skills they learned through the CAF program. 12 out of 15 CAFs cited building up their public speaking confidence as a direct result of leading numerous meetings with service providers, administration members, and citizens. One CAF reported that “I feel so proud of myself and cut down my fear toward meeting with authorities” (CAF Interview 04). Another CAF shared that they became “more confident in speaking with community members after practicing many times during meetings,” and that they no longer feel the fear they felt before when speaking in front of a large group of people (CAF Interview 11). Along with growing speaking confidence, many respondents also reported an improvement in their facilitation, networking, and leadership skills. They identified new ways they had learned to elicit discussions from participants, such as “using the brainstorm method in order to motivate the participants to speak out” (CAF Interview 06). Additionally, they reported that they felt more equipped to lead discussions around these topics due to their own increased knowledge on how local authorities and service providers work within the community. CAFs also shared that they gained the ability to speak and work with individuals that they had limited interactions with in the past, including with Sangat administration officials and with citizens in villages that they had not previously visited. One respondent summed this up, saying “I have learned a lot from the program. I know more people and extended my network. I am now more than a housewife. I earn more respect from people in my community” (CAF Interview 15). Three respondents also shared that they gained new technological and social media skills, including learning how to take more appealing photos for use in reports and social media, video editing skills, and learning better document management practices (CAF Interviews 13, 14, 15). CAFs also expressed interest in continuing to use these newfound skills outside of the CAF program, describing future instances of casual interactions throughout their community that they might use to continue to build awareness around citizens’ rights and responsibilities and government accountability. For example, one person stated, “I will disseminate to other people in remote areas where I visit during my free time” (CAF Interview 14). This suggests that the CAF program may have a positive legacy on accountability at the grassroots level. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 26 The CAF program seemed to lead to some improvements in CAFs’ understanding of citizens’ rights and performance of local government administration, but few CAFs shared tangible examples of how this knowledge increased their participation in civic affairs. Instead, they spoke more to how they believed this program helped improve accountability norms and civic engagement in general, non-specific terms. Respondents also reported that they gained more understanding on the rights of citizens and on the services provided by each of the sectors, stating for example, “I did not know about my rights to access services from the commune/Sangat administration, health center, and primary school. But now, myself and others understand more about our rights” (CAF Interview 09). They described how they came to understand that citizens could also play an important role in influencing service provisions and community decision-making, both from their perspective and from the perspective of the citizens. Another CAF stated “The program has changed the way I see my role and the responsibility of the service providers in my Sangkat. I have learned more about my rights as citizen and know how to monitor and evaluate the performance of our Sangkat/commune administration” (CAF Interview 15). Respondents provided very limited input on how the program increased their own civic and political participation norms; instead, they mostly spoke from the perspective of the citizens and service providers. For instance, one respondent shared that after conducting the outreach efforts and holding meetings at the different service provision areas, “service receivers and service providers are closer and know each other a bit better, which makes us less afraid of interacting with each other” (CAF Interview 09). Answers regarding this question focused more on the greater knowledge of accountability processes that resulted from the program versus any impacts on engagement. Only a handful of CAFs commented directly on the impact their participation in this program might have had on their careers. Of those that did, most indicated that their participation helped them advance potential careers outside the accountability sector. For instance, two CAFs shared stories about their previous jobs that they found limiting, one being a tuk tuk driver (CAF Interview 05) and the other who was a stay-at-home parent (CAF Interview 09). They indicated that their participation in the program made them realize that they can get involved with things like this that work closely with government actors. Others shared that their participation in the program “encouraged [them] to work for society” (CAF Interview 05), “influenced [their] education” and showed them how they could link what they were learning in school to real-life opportunities (CAF Interview 12). According to the survey results, 98 percent of ISAC CAFs agreed or strongly agreed that their participation in the program had opened new doors and opportunities for them, including ways to change or further their career. OW4C YA Program YAs were nearly unanimous in their appreciation for the communication, outreach, and networking skills they gained as a result of their experience in the YA program. “This program really transformed me,” reported one Ambassador, “before the program, I was a silent, shy, and timid person. But through the outreach activities I became comfortable greeting and approaching people in my community. Moreover, I learned to work as team and to facilitate in community meeting” (YA Interview 02). Another YA reported that they “improved so much in terms of working as team, becoming familiar with different regions and communities in the province, being tolerant in dealing with community people, gaining more knowledge of OWSO fees and services, and being more confident in speaking with strange people” (YA Interview 15). This appreciation for communication and outreach skills was reiterated by all but two of the 15 YAs we interviewed in our YA KIIs. 27 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Other YAs appreciated the leadership skills they gained and the connections they formed with local authorities. As one YA reported “I now know how to build good relations with local authorities and communicate well with them” (YA Interview 06). Another praised three core leadership skills they gained through the program: “First, I now have good communications with community people, youth, local authorities, and program staff. Second, I gained facilitation skills for community organizing. Third, the public speaking skills I gained make me feel encourage enough to speak with local authorities” (YA Interview 10). Beyond communication, outreach, and leadership skills, respondents also mentioned research and survey enumeration skills (YA Interview 05), and problem-solving skills (YA Interview 03, 11, 12). In terms of instilling norms of civic and political participation for government accountability, a select number of respondents said their perspectives on government accountability changed because of the program. In the words of one respondent, “I changed a lot in my attitudes toward local authorities by raising my voice toward what changes I want to see in their service delivery as well as the common interest for public people” (YA Interview 05). Another reported that the program strengthened his belief that “citizens have a right to access the quality of service from the public institution and the government official must provide good quality of service to the citizen” (YA Interview 11). This encouraging evidence is supported by responses to the YA survey, with upwards of 95 percent of respondents agreeing or strongly agreeing that “the YA program inspired me to become a leader in my community” (98 percent), “the YA program taught me how to advocate for change in my community” (96 percent), and “Because of the YA program, I am more engaged in my community than I otherwise would be” (100 percent). Nevertheless, this seemingly encouraging evidence must be interpreted with caution in light of the response bias risks highlighted in our Evaluation Design and Methodology section. Moreover, in our YA KIIs, respondents did not elaborate on these patterns with any mentions of specific civic or accountability activities they became involved in because of the YA program. Turning to the program’s impact on YAs’ careers, the primary benefit of the program was its value as a “resume builder” for YAs (YA Interviews 02, 08, 10, 11). In addition, numerous respondents said that the program led to networking and employment opportunities with other NGOs in Cambodia (YA Interview 02, 03, 04, 11). Consistent with this perspective, nearly 99 percent of respondents to the YA survey agreed or strongly agreed that “the YA program opened new doors and opportunities for me.” The majority of these opportunities were in the NGO and social services sector, and at least two of these NGOs are active in matters of government accountability, suggesting the program may have helped some YAs gain career experience in the accountability space. T4GC YA Program Similar to the ISAC CAF and OW4C YA programs, T4GC YAs commonly reported improvements in their communications and management skills. Nine out of 15 YAs interviewed highlighted improved communications and outreach skills from participating in the YA program. Referring to their community outreach experience, a YA recalled that “the important skills I have learnt is communication and dissemination of information” (YA Interview 03). Several YAs also appreciated acquiring management and teamwork skills from the YA program: “Another thing is that we learnt how to manage our tasks to carry out projects…We learnt how to make a plan and strategies to train citizens and students” (YA Interview 01). Mentioning new opportunities to participate in other activities, another YA reported that USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 28 “I learn how to work in a group and get more experiences from the people I work with. That also makes me braver when I speak in public. Some of the opportunities include the new projects that I was asked to join by some of the people I know” (YA Interview 06). Several respondents also said that the YA program helped them gain leadership skills. “Leadership doesn’t necessarily refer to leadership of big leaders like the prime minister,” reported a YA referring to a leadership course they had taken as part of the program, “but it is about leading yourself. When we can lead ourselves, we also know how to gather and lead people to achieve something. This is an important skill set for us because it is not taught at schools” (YA Interview 07). Both interviews and the survey suggest that YA program raised awareness about government accountability and civil rights. As one YA noted “Before the program, I did not really know about their [authorities and municipal government] roles and responsibilities. After I entered the program, I learn and notice more about their work and roles in the society” (YA Interview 05). Specifically referring to the role of government in waste management, another YA shared that “Before I didn’t think waste management issues are related to local officials. I thought they were related to NGOs, environmental organizations, or waste collection companies. After becoming a YA, I know each commune deals closely with waste management issues” (YA Interview 03). Reflecting this positive change, more than 97 percent of respondents have indicated that they either agree or strongly agree that “the YA program inspired me to become a leader in my community” (97 percent) and “the YA program taught me how to advocate for change in my community” (100 percent). Similarly, several YAs reported positive change in their understanding of civil rights. When asked about their understanding of citizens’ rights, a YA responded “Before I joined the program, I did not know what people’s rights were. But after I joined the program, I know that people’s rights refer to how people express their opinions and be able to ask questions freely” (YA Interview 04). Another YA referred to the youth’s participation in civic affairs, saying: “Before becoming a youth ambassador... I didn’t know what rights I have and what I can do as a citizen. After becoming a YA, I know my rights and duties to make decisions and to get involved in activities. Before this, I thought I am young and not capable of doing things” (YA Interview 01). Some YAs, however, noted that they had good understanding of citizens’ rights prior to joining the program (YA Interview 10, 07) while others put more emphasis on citizens’ responsibility in waste management (YA Interview 03, 14). In terms of civic participation, YAs expressed mixed views during the interviews. Some youths said they are more engaged in their community as a result of the program. A YA referred to a number of specific examples, stating “I participate in more activities in the society…For example, with the election campaigns earlier. I participate in the development of the communes or districts, and road development” (YA Interview 02). The YA survey results corroborate this finding, with all respondents saying that they agree to the statement “Because of the YA program, I am more engaged in my community than I otherwise would be.” During the interviews, however, we observed several YAs who indicated that their civic participation level did not change after the YA program (YA Interview 07, 06, 10, 11). Yet, some YAs who indicated increased community engagement limited their participation to the environmental sector (YA Interview 04, 05). “I notice myself that I am more open to talking, both on social media and in public,” said a YA, “mostly regarding the environmental challenges and issues. As for other issues, I don’t really discuss those” (YA Interview 05). Relatedly, most of the interview 29 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV respondents (11 out of 15) revealed their improved understanding of waste management and environmental issues. As for influence on YAs’ careers, many YAs interviewed expressed their gratitude for gaining experience through the YA program (YA Interview 03, 06, 07, 09, 12) with some referring specifically to the certificate they received in recognition of their participation (YA Interview 03, 07). Another major benefit cited by YAs was networking opportunities the program offered. 12 out of 15 YAs interviewed mentioned that their network has improved after participating in the YA program and they were hopeful that their expanded network will offer new opportunities in the future. The survey of the YAs also reflects this enthusiasm with 99 percent of the respondents agreeing that the YA program opened new opportunities for them. Despite these positive impressions from the survey, only one YA from our KIIs was able to mention a specific new employment opportunity that emerged from the YA program: “I know someone who introduced me to another job. He asked me if I can do it or not, and he also sees my ability” (YA Interview 05). In addition, the KIIs with YAs came just a few months after their program had ended, and it is likely given the optimism expressed in the survey that others will find opportunities through their YA program connections in the coming months. An important observation from the YA interviews is that some youths may have had varying levels of engagement and experience from participating in the program. For instance, a youth volunteer based in Battambang who had participated in the program for a year said, “Honestly, we didn’t work together much because I live in [name of the district] which is far from the city, and I seldom go to the city... For me, I didn’t interact with them much, but for other members, it might have been different” (YA Interview 11). A focal youth based in Kampong Chhnang who had participated in the program for several months noted the limited training they received and said, “I did not go through any sort of training so I did not know my roles or what I should do” (YA Interview 07). The same YA also shared their disappointment in the unprofessionalism of the organizers managing the YA program. While these references could be isolated incidents, it raises a concern that the participating partners of the T4GC activity may have had different approaches to their work and required additional oversight during Activity implementation. Heterogenous impacts of the YA / CAF programs by gender Across all measures of the YA program’s effectiveness and all three programs (ISAC, OW4C, and T4GC), females were just as likely as males to respond positively. For example, female YAs were just as likely to say that the “YA program inspired me to become a leader in my community” (98 percent) as males; they were also just as likely to agree or strongly agree with the statement that “the YA program taught me to advocate for change in my community” (98 percent) and with the statement that “the YA program opened new doors and opportunities” (99 percent). Lastly, they were nearly as like as males to agree or strongly agree that “Because of the YA program, I am more engaged in my community than I otherwise would be” (96 percent compared to 99 percent). COMMUNITY OUTREACH ISAC The ISAC activity incorporated three major forms of community outreach for the I4C campaign: 1) community meetings; 2) mobile loudspeaker; and 3) door-to-door outreach. The CAFs assisted in the USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 30 execution of each of these components, working to ensure these activities were accessible to citizens. In our interviews with CAFs, they shared various perspectives on how effective each of these components were in raising awareness about the importance of accountability and civic participation. Regarding the face-to-face community meetings, respondents felt that this was the most effective form of community outreach within the I4C program as it brings citizens and administration members together to discuss community needs in person. CAFs suggested that the in-person nature of these meetings made it easier for fruitful discussion to ensue between the parties, and for citizens to “have a chance to question back” and make it more of a conversation they could learn through (CAF Interview 13). In these meetings, as citizens grew in their knowledge of service provision, they felt more empowered to “raise out their issues…and suggest solutions” to the Sangat administration and providers (CAF Interview 06). CAFs indicated that they felt the community meetings led to the most tangible impacts, both on citizen knowledge of their rights and responsibilities, knowledge about government accountability, and on improvements on service provision. This was supported by the ISAC CAF survey, with 95 percent of respondents reporting that community meetings made citizens more likely to raise issues with the local government. CAFs had mixed opinions on the effectiveness of the mobile loudspeaker component. On the one hand, some respondents felt that it was a very effective way to raise citizen awareness and described the usefulness of presenting this “downtown with many citizens” around (CAF Interview 05). One CAF shared that “When the people heard our messages, they wonder what accountability is and they run to us with questions of what are their rights and what are the Sangkat council’s rights?” and from there CAFs were able to answer their specific questions, share resources and information pamphlets with them, and encourage future engagement (CAF Interview 08). The CAFs who felt this method was most effective seemed to think so because it allowed the I4C campaign to reach the most individuals at one time, versus meetings or door-to-door which typically took longer amounts of time to reach less individuals. Figure 9: Mobile Loudspeaker used for ISAC’s I4C Outreach (Source: AMARA’s Facebook page) 31 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV On the other hand, some CAFs felt that the mobile loudspeaker was not an effective method for disseminating information. Some respondents seemed to indicate that citizens were off put by the “noisy voice” of the loudspeaker, which potentially acted as a deterrent to getting them involved in the program (CAF Interview 03). Others shared that one problem with the loudspeaker is that they had no real way to gauge which citizens have listened to the message, or which citizens fully understand the point of the message, due to its mobile nature. One respondent predicted that “its effectiveness is only 20 percent because citizens could not get the full meaning of the message” that was being broadcasted (CAF Interview 10). Another said that CAFs “do not know how much [citizens] understand the message” and are unable to provide clarification or additional information to citizens unless they approach the loudspeaker station and show interest in continuing the conversation (CAF Interview 13). CAFs failed to specifically speak to the ways they thought that the mobile loudspeaker increased citizen awareness about importance of accountability and civic participation. CAFs had limited evidence to base their observations of the door-to-door outreach component on as the ISAC activity completed just one round of this. However, responses indicated that door-to-door outreach was an effective way to “make vulnerable people aware of their rights and know about the standard service” (CAF Interview 12). CAFs felt that this was a good way to reach people who would not otherwise join community meetings or hear about the activity through the mobile loudspeaker. One respondent summarized the advantages of this outreach component, saying “we can provide information directly and clearly to citizens and they have an opportunity to question us on points they do not understand well” (CAF Interview 10). The survey of the CAFs indicates a large amount of enthusiasm for the door-to-door outreach component, with 100 percent of respondents answering that door-to￾door outreach was an effective way to introduce I4C and 94 percent indicating that citizens are more likely to raise issues to local government because of this outreach effort. CAFs did not share many negatives about this type of outreach other than to say that fewer people were reached over a longer period of time due to the nature of door-to-door outreach. OW4C YAs were overwhelmingly positive regarding the effectiveness of door-to-door outreach at raising awareness of the Pidor the Smart Villager Chatbot. In the YA survey, 100 percent of respondents agreed or strongly agreed that “door to door outreach is an effective strategy to introduce citizens to the Facebook Chatbot”. Ninety-nine percent agreed or strongly agreed that “During door-to-door outreach, citizens were interested in the Chatbot”, and 100 percent agreed or strongly agreed that “During door-to-door outreach, YAs had enough time to introduce the Facebook Chatbot” and were not rushed. These positive perceptions of the effectiveness of door-to-door outreach are supported by data from the KIIs. As one YA described, during door-to-door outreach, “citizens were excited to hear news of Pidor the Smart Villager … [they] really like to use it because it saved them from having to go back and forth to OWSO just to certify their document. Citizens liked it so much that they sometimes asked their children to join and try to access the information to certify their children’s educational certificate” (YA Interview 09). Another YA reported that door-to-door is effective because it allows him to spend more time with citizens who need additional support to understand the Chatbot, relative to outreach through community meetings: “with door-to-door, sometime we spend more time to teach them compared to public forums, but the people learn a lot about the Chatbot and OWSO and this makes it USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 32 easy for them” (YA Interview 07). Another expressed the view that door-to-door outreach was more effective than outreach through community meetings or posters alone (YA Interview 06). Comparing door-to-door to outreach concerts, one YA reflected “I would say that door-to-door is the most effective way to disseminate the information of Chatbot, because we have time to explain them through leaflet and practical on their phone or phone call for IVR” (YA Interview 13). Despite these positive impressions, YAs identified some key challenges with door-to-door outreach. The most commonly cited limitation to door-to-door outreach was the issue of scale --- although effective, door-to-door outreach is inherently limited in how many people it can reach. As one YA explained, “because we just established this new technology and are only working within a few provinces and not at a widespread scale yet, citizens generally do not have a good familiarity with the Chatbot” (YA Interview 08). Another highlighted the challenge of accessing remote towns and villages: “if people knew about it, I think they would use it but at the moment knowledge of OWSO and the Chatbot is not widely disseminated in the rural areas” (YA Interview 04). Picking up on this concern, a YA suggested more actively advertising the Chatbot through social media: “if we continue to disseminate through social media, in the future, people will become widely aware and use it because the activity start in small scale of dissemination and right now only a small amount of people are aware of it” (YA Interview 03). The second challenge to door-to-door outreach identified by YAs was that of building citizens’ trust. “We faced the challenge of lack of participation from citizens because there are so many scammers out there that scares them and when we go to them, they do not trust us” (YA Interview 12). Another YA described how “when we did the door-to-door outreach activities … we faced some challenges, for example, at least 30 percent did not welcome our door-to-door activities due to concerns about talking politics” (YA Interview 02). Comparing outreach in villages to downtown commercial centers, one YA described how “it is difficult to conduct outreach activities downtown and at the market because they do not welcome us, and they are busy at their businesses” (YA Interview 02). Echoing this sentiment, another YA said that getting residents to take time away from their day to learn about the Chatbot was difficult, opining that “they do not have enough time for us” (YA Interview 10). T4GC YAs engaged in six types of outreach activities aimed at sharing information on WM best practices and introducing the GCC App: door-to-door, community meetings, school campaigns, social media, Facebook and radio talk shows, and clean-up campaigns. Figure 10 presents the types of outreach activities T4GC YAs participated in based on the YA survey results. The three main outreach activities YAs engaged in are door-to-door campaigns, community meetings, and school campaigns. 33 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Figure 10: T4GC YA Outreach Activity Participation (Source: YA Survey) YAs found the door-to-door outreach, the most utilized approach, to be generally effective at raising awareness of the App albeit some challenges. As part of this outreach, youths gathered at a pre￾determined commune, engaged with the commune chiefs for approval and assistance, and visited households in groups of two or three YAs. During the interview, six YAs ranked door-to-door outreach as the most effective means of raising awareness primarily because they could engage directly with the citizens and support them to install the App on their phone (YA Interview 04, 05, 07, 08, 12, 13). “I think [the door-to-door outreach] is effective,” explained a YA during an interview, “because I can meet them directly and talk to them and see that they follow me” (YA Interview 08). Echoing this optimistic view, all YAs agreed to the statement that “D2D outreach was an effective strategy to introduce tech/I4C” for the survey. A few YAs, however, found the door-to-door activities to be challenging because the citizens they met during this outreach tended to be older and some showed reluctance when the YAs visited their houses. As noted earlier in the report about difficulties associated with using smartphone and the App, a YA noted that “Some go to school while others go to work so mostly old people are at home…they don’t know how to use the App so this is difficult for them to access Green Cambodia app” (YA Interview 06). Some YAs experienced difficulty gaining trust from the citizens during the door-to-door outreach, especially during the COVID-19 pandemic (YA Interview 03, 07, 09). Apprehension among the YAs visiting citizens directly for the outreach may have resulted in only 89 percent of the YA survey respondents agreeing to the statement that “During D2D, YAs had enough time to introduce tech/I4C.” It is also noteworthy that YAs did not have a clear plan when conducting the door-to-door visits. Some seemed targeted (arranged by commune chiefs or choosing houses with a lot of waste) while others selected houses randomly. During the interviews, YAs perceived the community meetings to be relatively effective as well. Community meetings were arranged by commune chiefs and local authorities who convene 30 – 40 citizens and gave an opportunity for the YAs to explain about WM practices and introduce the GCC App. Four YAs interviewed for the evaluation ranked community meetings to be the most effective outreach method (YA Interview 3, 9, 11, 13). According to the YA interviews, factors that contributed to positive outcomes from community meetings were the authority of the commune head to convene a large group of diverse citizens (YA Interview 03, 09, 11). Recalling the difficulties they had with door-to￾door outreach, a YA highlighted that “When village or commune officials were present, they sat and USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 34 listened. They respected us. When we introduced ourselves on our own, they didn’t think we were credible because they didn’t know us” (YA Interview 03). 100 percent of YAs interviewed for the survey also agreed to the statement that “community meetings were an effective strategy to introduce tech/I4C.” To improve this outreach approach, a YA recommended increasing the frequency of community meetings (YA Interview 11). School campaigns were also identified by the YAs as an effective approach to disseminate information about the GCC App. For the school outreach campaigns, YAs contacted school officials and trained 50 – 100 students on WM practices and the App. Four YAs selected school campaigns to be the most effective way to raise awareness of the T4GC technology (YA Interview 11, 13, 14, 15) citing students’ receptiveness to technology and learning as the main reason for their success. In contrast to the door￾to-door outreach, several YAs believe that students are more likely to learn and use the App (YA Interview 06, 11, 14, 15). All of the YAs interviewed for the survey either agreed or strongly agreed to the following statements: “school campaigns were an effective strategy to introduce tech/I4C” and “During school campaigns, citizens were interested in tech/I4C.” Three other outreach activities include social media outreach (13 percent participation), Facebook and radio outreach (13 percent participation), and clean-up campaigns (7 percent participation). None of the YAs we interviewed identified these approaches as an effective approach to disseminating information about the App. The survey of YAs shows that 78 percent agreed to the statement that “During FB/radio talk show, youths had enough time to introduce tech/I4C.” While 78 percent is high, it is relatively low compared to the responses on other outreach types where 93 percent of the YAs agreed to the same statement that they had enough time to introduce tech/I4C during the outreach, which suggests that the talk show programs could have allocated more time to fully elaborate on the GCC App. EFFECTIVENESS OF SOCIAL ACCOUNTABILITY ACTIVITIES ONLINE DASHBOARDS OW4C Evidence regarding the utility and effectiveness of the OW4C online dashboard was mixed. On the one hand, several DOs reported that they receive a monthly report from the OW4C Dashboard, and that they use this information to report to the district council on the types of services citizens are requesting through OW4C’s tools and the feedback on those services (DO Interviews 01, 03, 04). As one DO described, “The automatic report really helped me understand the number of citizens accessing OWSO services, the types of services people are using, and their feedback on those services. So, I use it as an evidence source for me to develop my monthly report to district council about the performance of OWSO and my office” (DO Interview 04). 35 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Figure 11: Screenshot of OW4C Online Dashboard On the other hand, DO respondents were quick to point out the limitations of information conveyed through the Dashboard. Multiple respondents indicated that the high-level summary of citizen usage and feedback is less helpful than feedback on specific issues that need to be addressed. “The Dashboard does not have space for citizens to write their feedback or suggestions, so it does not help much because I cannot address their specific concerns” reported one official (DO Interview 01). The respondent added that feedback through the Dashboard was among the least effective ways to receive feedback: “the most effective ways of complaints are phone call and verbal complaint, followed by complaints through the OWSO mailbox” (DO Interview 01). Another respondent agreed with this assessment, stating that “when citizens have problems related to OWSO, they always call directly to us, and I can say that that is one of the most effective aways for a complaint to be addressed” (DO Interview 02). DOs generally viewed the OW4C Dashboard as useful for background knowledge on OWSO services, but they stopped short of using it to inform any specific policy decisions. For example, when asked if they knew of any specific policy or procedural decisions that results from the Dashboard helped inform, none of the six DOs responded affirmatively. And only one of the six DO officials reported sharing the dashboard with other DO officials or colleagues. DOs also expressed concern about the sustainability and management of the OW4C dashboard: “For the sustainability of dashboard, what does the program do regard to this technology? Does it hand over to district administration to manage it or not? In case of hand over to district administrative to manage, how about the maintenance and upgrading of this technology for future use?” (DO Interview 03). Another respondent agreed, stating that “it would be good to handover this dashboard to the Ministry of Interior for future management and long-term sustainability” (DO Interview 01). Unfortunately, the OW4C IP was unable to transfer their technology to the Government of Cambodia due to funding restrictions. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 36 Data on site visits to the dashboard also reveal mixed but generally limited demand for the Dashboard. From the launch of the Dashboard in July 2021 through August 17, 2022, there were 4,084 visits to the Dashboard from 2,038 unique IP addresses. Of the 4,084 visits, 2,945 were from Cambodia, 726 were from the USA, and the rest were from various other countries. Within Cambodia, 90 percent (2,663 / 2,945) of the site visits were from IP addresses located in Phnom Penh. Figure 12: Site visits to OW4C Dashboard (Cambodia traffic only) (Source: OW4C) Site visit patterns over time reveal sporadic but generally dwindling demand for the OW4C Dashboard (Figure 12). Traffic was highest immediately after the launch of the Dashboard in July 2021, when it reached a single-day high of 110 visits. This spike in demand was likely the result of OW4C led outreach. Traffic then dwindled through the fall of 2021 before spiking again in March and May 2022, possibly due to OW4C led outreach. By June and July of 2022, site traffic dwindled to less than 10 visits per day. Overall, the data is inconsistent with the idea that citizens, researchers, journalists, OWSO officials, DOs and Provincial Ombudsperson (POs) value the Dashboard, visit it regularly, and share the Dashboard with their colleagues – if this were the case, we would expect to see site visits grow over time. Instead, the site visit patterns are more consistent with the idea that individuals introduced to the Dashboard use it just a few times before returning to ex-ante ways of monitoring OWSO services. T4GC The T4GC App and Dashboard were launched in 10 districts in Battambang and Banteay Meanchey in December 2020 and ran until August 2022. In July 2021, the App and Dashboard were launched in two additional districts in Kampong Chhnang and Oddar Meanchey, respectively, and ran until August 2022. In total, the App and Dashboard were active for a total of 204 district-months. Over this period, a total of 710 reports were submitted to the GCC App and uploaded to the Dashboard. These figures imply that on average, only 3.4 reports were made per month per district, a number that pales in comparison to the scale of the waste management problems in these districts. Based solely on these calculations, we 37 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV can conclude that the App and Dashboard did not reach the scale required to make a meaningful dent in the quality of waste management service provision in the targeted districts. Site visits to the T4GC Dashboard were similarly limited. Due to an oversight, the IP did not collect data on site visits for the first seven months of its launch. But available site visit data for the final six months of the Activity period indicate that from March to August 2022, the site was visited only 455 times, or about 2.5 times per day across all 12 districts. Recognizing that much of this traffic may have been associated with training events for WMSPs or site visits from outside Cambodia, such as those by the ET, these figures indicate very low uptake of the T4GC Dashboard. Figure 13: Screenshot of T4GC Online Dashboard Despite the limited uptake of the GCC App and Dashboard, evidence from the KIIs suggests that the system did help WMAs and WMSP address waste management problems in isolated cases. The primary benefit of the dashboard cited by the WMAs was improved efficiency of identifying waste management issues. Four out of five WMAs praised the Dashboard for allowing them to identify issues and their location quickly and accurately. Before the T4GC activity, officials reported that it was challenging to locate the issues reported over the phone (WMA Interview 04). With the Dashboard and the GCC App, the WMAs were able to identify the WM issue using the GPS information and the photos without having to travel to the location, allowing them to “work faster and better” (WMA Interview 01). All but one WMSP interviewed also shared the same view and lauded the geolocation feature of the Dashboard. The reporting function is especially helpful, according to a WMSP manager, when “there is waste that has no owner, so the company does not know about it, and nobody cares” (WMSP Interview 01). WMAs and WMSPs welcomed how the Dashboard encouraged them to deliver improved WM services by flagging an issue as ‘resolved’ (WMA Interview 02, 04). Noting the increased visibility of WM issues and government responses on the Dashboard, a WMA explained that “it makes the waste companies pay more attention to their work because we know when there are waste issues. It makes them work better and become more responsible” (WMA Interview 02). A WMSP also noted how the Dashboard encouraged accountability and explained that they “work faster because there is a technology that can see if [they] don’t collect the waste” (WMSP Interview 05). USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 38 Some WMAs found the Dashboard to be particularly helpful for planning WM activities. The Dashboard is user friendly and displays information for each district on WMSP services, citizens they are serving, and WM reports disaggregated reports by type of waste and resolution status overlayed on a map. Several WMAs had used the Dashboard to report on WM status to their supervisors (WMA Interview 02, 05). One WMA provided an example when they used the Dashboard to inform their WM decisions: “We also registered to provide services for people in two districts now. Once we received more information, we expanded our services for them. Because we received the information from Dashboard, we know we can provide our services to more people” (WMA Interview 01). Another crucial benefit cited by WMAs was their improved ability to monitor WMSP’s performance. All but one WMA interviewed shared that the T4GC Dashboard has made it easier for them to oversee the WMSPs (WMA Interview 01, 02, 04, 05). One official highlighted the strengths of the Dashboard and said, “I like the function that allows the citizens to report to the administration, which allows us to inform the waste management companies to take action right away” (WMA Interview 05). For the WMSPs, improved communication is a benefit of the Dashboard quoted from several users (WMSP Interview 01, 03, 04). WMSPs noted improved communication with the people they serve (WMSP Interview 03), local authorities (WMSP Interview 01), and fellow WMSPs participating in the T4GC activity (Interview 04). Nevertheless, Dashboard users also shared some challenges they faced. Some WMSPs and WMAs reported weak incentives to use the Dashboard. One official, who does not use the Dashboard at all, said “I never use it because no one will solve the problems. I am the person who solves the problems” (WM Interview 03). Similarly, one WMSP shared “to be honest, I only use it when I am told” (WMSP Interview 04). Reflecting on how citizens used to report WM issues prior to the T4GC activity, a WMA and two WMSPs recalled that receiving feedback in-person or over the phone was not too challenging and hence the use of the Dashboard is not as helpful (WM Interview 02, WMSP Interview 01, 02). One critical point raised by a service provider was that WMSPs have no incentive to use the technology T4GC developed even though much of the effort of keeping the Dashboard updated falls on them. WMSPs are asked to provide key data for the Dashboard such as service routes, service schedules, and number of WMSP employees. As one WMSP explained, “the App does not help the company much because we are the ones who provide reports on data and statistics in the database. We do not get any benefits by uploading data there at all …The benefits do not come to the company. They are more for the local authorities” (WMSP Interview 04). This suggests that collecting feedback and addressing reports from citizens was not a priority for the WMSPs who listed several other challenges in their operations including difficulty collecting fees, limited understanding of WM best practices from the citizens, inadequate resources, and maintenance of equipment. Even though most of the respondents were pleased with the Dashboard’s features, several WMAs and WMSPs alike saw that ‘resolving’ issues on the Dashboard could be improved. While acknowledging that they were trained to “click solve to update that [they] have cleared the spot” and that “it does not take a lot to update,” a WMA stated that this step is rather time-consuming and burdensome for their team already strapped for resources (WMA Interview 01). Another WMSP expressed frustrations that the ‘resolve’ function is limited to the WMAs. Citing this issue as their reason for not using the Dashboard regularly, a manager of a WMSP said “I can see the waste issues and tell my workers to clear the waste problems, but I cannot update them on the Dashboard because there is no function for me to use to 39 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV update the problems after I have solved them … If they use the data in that Dashboard, it does not reflect my actual work” (WMSP Interview 04). Regarding the ‘resolve’ function of the Dashboard, the WMSP also added that having a green dot on the dashboard for ‘resolved’ cases hinders their ability to focus on the new issues that are reported (WMSP Interview 04). Another related concern expressed by the dashboard users was the delay in updating the data displayed on the online dashboard (WMSP Interview 05, WMA Interview 05). Figure 14: Screenshot of T4GC’s Online Dashboard Reporting Feature (Source: T4GC Online Dashboard) Some WMSPs also noted that the Dashboard would be more useful to them if there were more GCC App users. For instance, when we asked about the utility of the Dashboard, one WMSP replied “to make everything work, the number of users should be more. The more people use it, the more reports we can receive. It does not depend on us alone, so people should know about it and report it…The important thing is to have more users” (WMSP Interview 02). This important point of feedback is consistent with the low and inconsistent usage patterns documented above in the GCC App subsection. WMAs also expressed concerns on the sustainability and management of the Dashboard and the GCC App. Two WMAs interviewed for the evaluation assumed that the technology developed under the T4GC Activity would be transferred to them to manage and maintain once the Activity is completed (WMA Interview 01, 02). Yet it is uncertain whether the Activity has prepared concrete steps to transfer over the technology to the WMAs. Referring to their most recent engagement with the T4GC management team, a WMA said, “I want another meeting before the app is officially launched and transferred to us” (WMA Interview 02). Although sustainability and transfer to the GOC was a concerned expressed by respondents, transferring the technology to the GOC was never within the mandate of the T4GC IP, and funding for this transfer was not available. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 40 Overall, our interviews provide anecdotal evidence that WM services improved due to the online dashboard and the GCC App, but the results may have varied between urban and rural areas.27 Feedback from the YA survey was generally positive with 93 percent of respondents agreeing or strongly agreeing that “most issues submitted through GCC App were acted on by SPs and officials.” During the KII interviews, both WMSPs and WMAs recounted improved services due to the Dashboard and the GCC App. For instance, a WMSP recalled an issue reported in their service area: “we saw it on the Dashboard and took action immediately,” the manager said describing the exact location where the issue was reported, “and we got a compliment for that, so the Dashboard allows us to work in the right location, fast, with a satisfying outcome” (WMSP Interview 01). Some YAs also reported instances when the WMSPs and WMAs took action in response to feedback from the Dashboard (YA Interview 03, 15). A YA shared their experience and said, “after reporting the problems, about half a day or one day later the service providers came to collect it” (YA Interview 15). However, several youths reported during their interviews that they did not observe change in WM services. As elaborated in the Green Cambodia Citizens App section above, WMSPs or WMAs did not respond to the reports submitted on the App especially in the remote areas (YA Interview 07, 12). COMMUNITY SCORECARDS ISAC Feedback on the ISAC Community Scorecards program was mixed. In general, service providers were more optimistic about the program having tangible impacts than citizens. CAFs also had mixed opinions on the success of the scorecard itself, with some feeling it was one of the most successful components of the ISAC activity and others feeling like it did not lead to many tangible impacts. Notwithstanding these mixed perceptions, most respondents were optimistic that the program could be effective if certain improvements were made to future iterations of the program. Citizen Perspective Overall, citizens stated they had positive interactions with the Community Scorecards program. Many showed interest in the process and stated that they enjoyed engaging in the scorecard program (Citizen Interview 03, 05, 08, 09, 10). These participants felt that they were able to voice their needs, that their needs were listened to by authorities, and that they learned new information about how to access to services and their own rights and responsibilities. This more positive perspective was overwhelmingly supported in the responses to the CAF survey, where 99 percent of respondents agreed or strongly agreed that participation in the community scorecard activity enhanced citizens’ access to information about the quality of local government service. In the CAF KIIs, one individual shared: It was a great experience for me to engage in the community scorecard program. It was the first time that I got to know more about citizens’ rights and responsibilities. I also learned about the obligations and accountability of the local authorities in my commune. In the past, I dared not to visit and talk to the commune council member. However, in 27 T4GC Online Dashboard shows resolution status of reports and complaints submitted. However, as of August 29, 2022, the resolution statuses of reports are marked as “N/A” for the four T4GC provinces. 41 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV this program I have more confidence to talk to them and raise my concerns. (Citizen Interview 09) Many of these respondents expressed interest in continuing to engage in the community scorecard process, and stated that they would like to see the program continued and expanded to be open to as many participants as possible. CAFs supported this viewpoint, with 10 out of 15 CAF respondents reporting that the scorecard process successfully increased citizens’ knowledge on their rights and responsibilities regarding civic engagement. One respondent stated “after our dissemination, some citizens understood I4C well and could discuss and talk with authorities” thanks to the knowledge they gained through this experience (CAF Interview 11). Some of the needs expressed by citizens focused on the quality of teaching, unclean communal environment (including lack of clean toilets and a need for more trash bins), the need for better infrastructure to prevent flooding of roads, and the need for better service at local healthcare centers (Citizen Interview 03, 04, 05, 08, 09, 10). Citizens shared that these areas had seen some improvements since their involvement in the community scorecard activity, including increased staff at the local health center (Citizen Interview 01), road repairs (Citizen Interview 10), and upgrades to school bathrooms (Citizen Interview 10). In addition to addressing needs raised by citizens, the community scorecard activity also increased citizen engagement with local officials and service providers and helped to inform citizens of their rights and responsibilities within this relationship. For instance, one citizen participant stated: From the scorecard meetings, I have learnt more about my rights and responsibilities. I also shared the knowledge with other villagers. I observe that the relationship between villagers and service providers is getting closer. Some villagers have more confidence to request or advocate for improvement of services. Since participating in the Community Scorecard meetings, villagers have more interactions with the Sangkat/commune administration or other local service providers (schoolteachers and director, or health center nurses and managers). They go to the Facebook page or call the service providers directly. Some villagers contact our youth team to convey their messages or concerns to relevant authorities. (Citizen Interview 10) Respondents shared that they had observed continued regular meetings between citizen groups and members of the Sangkat administration after the community scorecard activity had been completed (Citizen interview 09, 10). Some participants believed that fellow citizens had felt more empowered to voice their concerns and needs to authority figures thanks to their participation in the community scorecard activity, a perspective that was also shared by CAF respondents. Regarding increased access to information about local government performance, citizens did report some improvement in their knowledge, but not an overwhelming amount. For instance, respondents who spoke to this aspect of the activity reported that they learned about where to voice their complaints and needs, and had a marginally better understanding of how budgeting and service provision works (Citizen Interview 05, 08, 09). However, multiple respondents stated that they “learned nothing” during their participation in the activity, suggesting that access to information was not significantly impacted by this process (Citizen Interview 03, 04). USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 42 Some citizens expressed confusion over their involvement in the scorecard process. Respondents reported that they were not sure why they had been selected to participate and were unclear how the scorecard process was meant to work (Citizen Interview 01, 03). Others expressed skepticism over the fidelity of the program implementation, with concerns ranging from too few participants, failure of the facilitators to share important information (like budget plans and expenses), and a skewed prioritization of needs raised. One respondent stated, “most participants are relatives to local authorities, there has not been much voice for any change according to the real needs of citizens” (Citizen Interview 04). This was confirmed by CAF respondents, who shared that citizens faced difficulty in advocating for their priority issues to be duly prioritized by organizers and local officials (CAF Interview 10). This lack of trust in the process likely resulted in the failure to achieve citizen buy-in, which was evident in the number of respondents that stated they would no longer participate in the process and had not bothered to follow up on outcomes and results from the meeting. It should be noted that citizens were not as thorough as would be ideal in their KII responses. Even when citizens expressed positive experiences with the community scorecard process, they often did not provide specific examples of how their engagement in this activity led to tangible improvements, so this seemingly encouraging evidence must be interpreted with caution in light of the response bias risks highlighted in our Evaluation Design and Methodology section. This is an issue common in Cambodian social accountability programs, and evidence shows that lack of funding for implementation of certain improvement actions has been a challenge faced by several implementers Service Provider Perspective Service providers generally felt that the community scorecard activity was successful and led to positive impacts on the community and the quality of service provision. Their experience participating in the scorecard activity was largely positive. Many service provider respondents said they thought it was good that they were able to hear more from the community about their needs, as well as share their own perspectives and responses with participants (Service Provider Interview 02, 03, 06, 09). One service provider felt “it is good that I can raise any issues…I can express my thoughts and concerns, and can exercise my right to demand the full participation of parents” in school activities (Service Provider Interview 03). Despite these positive experiences, some respondents again shared confusion over the activity, saying “I do not know what the purpose of the community scorecard is. To me, AMARA works like a mystery” (Service Provider Interview 07) and “I did not know what they were talking about” (Service Provider Interview 08). These perspectives suggest the implementation of the program may have lacked adequate awareness raising. Conversely, they may reflect the fact that some of the service providers sent different people to different meetings, resulting in the absence of a continuous focal person for follow-up and action. Such high turnover in service provider attendance may in turn reflect the lack of prioritization they give to the activity. Service providers also raised complaints about logistical issues associated with the training, such as short notice for meetings, the lack of per diem and the lack of reimbursement for transport costs. According to service providers, citizen engagement with local officials and service providers seemed to increase after their participation in the activity. Respondents reported positive impacts like “after meeting, both sides understand each other” (Service Provider Interview 06) and “since participating in the community scorecard meetings, the interaction between our school and parents has become closer” 43 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV (Service Provider Interview 09). Another service provider stated “the communication between parents and schools is much better” (Service Provider Interview 02). Overall, respondents indicated that the two groups understood each other better and had more transparent and sustained interactions after the close of the activity. Respondents reported some improvements in service provision as a result of the community scorecard activity, but many of them fail to provide tangible evidence to substantiate their claims. A common view was that citizens and providers are “working closely to address common issues,” along with other general statements like “we strengthened our service delivery according to what was agreed to” but without concrete examples (Service Provider Interview 05). A few respondents described improved school services and health center services in more detail, stating that “this meeting also makes schools improve their school campus environment and friendly manner” (Service Provider Interview 03) and that the health center staff had been welcoming more patients and providing more services, such as vaccinations (Service Provider Interview 06). But these reports remain anecdotal. Respondents also mentioned considerable challenges with scorecard meetings, including difficulty having either the right citizens attend (i.e. people who actually utilize the services in question), or getting the same people to attend multiple meetings. This made it difficult to have consistent discussion around priority issues and action items. Some reported that parents who no longer had children enrolled in school attended scorecard meetings focused on school improvements, or that the audience was mostly made up of school management even when the discussion was meant to focus on the needs raised by parents of students (Service Provider Interview 07). Others reported that there wasn’t enough time between the announcement of the scorecard meeting and the date of the event, so it was difficult to get many citizens to join (Service Provider Interview 04). One respondent reported on response bias on the side of service providers during scorecard meetings, stating “the supply side always scores their performance highly, which is opposite to the score given by the demand side or social accountability committee,” and mentioned that they had to take time to have the service providers re-do their scorecards to achieve more accurate answers (Service Provider Interview 01). As with all evidence drawn from self-reported impacts, it is important to balance our interpretation of these findings with the risk of response bias. For example, when CAFs were asked about the impact the community scorecard activity had on service providers, they either showed skepticism that service provision had improved or they mentioned that they felt it had improved but were unable to provide concrete examples of these improvements. Responses to the survey were also overwhelmingly positive, but these questions did not ask respondents to provide specific details. OW4C In contrast to the ISAC Community Scorecard model, OW4C scorecards consisted of a single meeting with OW4C staff to score OWSO on the quality of their services and brainstorm areas for improvement. OW4C staff then forwarded this feedback on to OWSO officials for their consideration and potential action. Also in contrast to the ISAC model, the OW4C scorecards were implemented in only a minority of OW4C communities. While this more limited model cannot be expected to result in major improvements to the quality of OWSO services, it does have the potential to spur modest changes in response to specific points of feedback relayed through the scorecard meeting. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 44 Our interviews indicate that this potential was realized in most, but not all, of the communities where the scorecard was implemented. Where the scorecards fell short, implementation appears to have been a problem. In particular, implementation fell short of beneficiaries’ expectations due to limited turnout (Citizen Interview 01, YA Interviews 10, 12, 13), citizen reticence to criticize government officials (Citizen Interview 02, YA Interview 14), the non-participation of OWSO officials (Citizen Interview 01), and disorganization during the meeting itself (Citizen 03). When well-organized and well-attended, however, the scorecard model appeared to be effective at communicating feedback to OWSO officials. As one citizen relayed, “the community scorecard meeting is a great opportunity to give constructive feedback for better service of OWSO for community people” (Citizen Interview 02). Echoing this sentiment, another citizen said that “I am really interested in this community scorecard because it allows me to remind public official on their performance and their role to serve the public” (Citizen Interview 04). Common points of constructive feedback for OWSO officials included the quality of latrines, disrespect of citizens by OWSO officials, discrimination against the poor, and excessive delays in receiving documents like birth certificates, deeds, or ID cards. During the scorecard meetings, citizens also learned a lot about the services offered by OWSO and how to navigate them. When asked what they learned during the scorecard meetings, one citizen replied, “Now, I clearly understand the process and fee of the land title from OWSO, Office of Land Management, Tax Department, and that they are not allowed to charge more than the fee defined by law” (Citizen Interview 04). Other common topics and learning points include how to register a new business, how to buy or sell property, how to transfer property deeds, and how to obtain ID cards and birth certifications (Citizen Interview 01, 02, 05). By and large, OWSO officials appear to have taken the community scorecard feedback seriously. Reflecting on improvements made since the scorecard meetings, one citizen recounted “After the community scorecard, [the OWSO staff’s] performance has improved. They now work a full working day and there is always staff stationed at each counter” (Citizen Interview 04). Two additional citizens reported that the feedback they provided during the meetings was taken seriously by OWSO officials and acted upon. YAs were also in agreement with this assessment. As one YA recounted, “OWSO officials take seriously the concerns and suggestions from the scorecards and they respond back clearly to make sure that citizens understand the actions that were taken to address them” (YA Interview 06). PUBLIC FORUMS Evidence from the KIIs and YA survey suggests that the public forums were a productive venue in which citizens learned important information about OWSO services and engaged with OWSO officials in meaningful, constructive, and sometimes critical dialogue. Ninety-six percent of YAs agreed or strongly agreed with the statement “During Public Forums citizens received accurate information about OWSO services.” Ninety-five percent agreed or strongly agreed that citizens actively participated, and ninety- 45 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV three percent agreed or strongly agreed that OWSO officials took citizen feedback seriously (YA Survey Results, Annex D).28 In the KIIs, both citizen participants and OWSO officials expressed an appreciation for the constructive nature of the dialogue. They felt that on many occasions, citizen feedback was critical of OWSO officials which led a few of those interviewed to make changes, some of which were visible later to forum participants while others noted there had been none (Citizen Interview 01, 06). Some officials felt many questions were out of their jurisdiction (such as land issues) while others within their jurisdiction dealt with the quality and accuracy of services provided by OWSO and the timeliness of their work. As one official described, “in the forum, we got a lot of critical feedback on the hospitality of OWSO staff, unclean office, and the speed of service delivery … [in response], we asked citizens to understand that we are short staffed and cannot always give a warm welcome at the front desk. But for the rest of feedback, we followed up by looking for a cleaner to clean the office and providing citizens with contact numbers for service follow up” (OWSO Official Interview 08). Another OWSO official described the meetings as a “win-win” for both sides, citizens and OWSO officials: “Citizens are made aware of OWSO services from public officials at the Forum … and OWSO officials receive comments, feedback, and suggestions for improvement of services in order to meet the clients’ demand. So, the public forum is a win-win for both sides, demand and supply sides” (OWSO Official Interview 06). Another official described the question-and-answer session as “very active” and explained this made the Forum “very productive” for the purposes of understanding citizens’ feedback and concerns (OWSO Official Interview 04). Not all officials valued the Forums, however. One official stated that “we are very busy at the office and do not want to spend much energy for the Forum. (OWSO Official 03). The official further stated that “participants in Forum do not pay much attention to the information we deliver because most of them are elderly, and the people demanding the services do not participate in the forum,” which raises the issue of targeting, similar to those raised in the scorecard process. For their part, citizens appreciated the opportunity to communicate with officials about their concerns (Citizen Interview 02, 05). They described the forums as “an open floor for question and answer” (Citizen Interview 02), while recognizing that citizens also have a role to play in helping OWSO officials address problems: “the objective of the Forum is to let citizens raise questions and concerns … but the people and authorities should work together to address these concerns” (Citizen Interview 01). By and large, citizens reported being able to express their views openly and without fear of reprisal: “everyone can ask questions to get more information from OWSO. … from what I saw in the forum, people asked questions without fear” (Citizen Interview 02). But some were still reticent. According to one respondent, when she returned home from the meeting, her husband asked her why OWSO charges a fee (bribe) for a land filling, to which the respondent replied “I still do not know, because I was afraid to ask this question in public” at the forum (Citizen Interview 05). The youth ambassadors also 28 Although YAs viewed the in-person forums as effective venues for facilitating engagement between OWSO officials and citizens, they were somewhat less enthusiastic about virtual forums used during the COVID-19 pandemic --- only 72 percent agreed or strongly agreed that virtual forums were just as effective as in-person forums. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 46 noted the apprehension of some citizens to ask questions in public because the “majority are afraid of speaking out.” (YA Interview 03, 12, 14, 15). According to participants, the most common issues raised during the meetings was birth registration, construction permits, document registration, and land registration. For many of these issues, OWSO officials had only limited ability to make improvements. As one official described, “the priority issue raised in forums across all the communes … is the time and cost of acquiring a land title. [Citizens] always suggest to cut down the period of accessing as well as the fee. But we could not address it because the procedure is defined by the Ministry. Another issue raised was the difficulty of accessing construction permits … we could not assist them because it is required by the law, and our office does not have leeway in enforcing the requirements” (OWSO Official Interview 02). The intractability of some of the concerns was cited by at least two other OWSO officials as a core challenge to converting citizen feedback into meaningful change. “Some of the more complicated issues raised in the Forums were difficult resolve: … for land issues, we tried to deal with citizens’ complaints but it was too complicated” and infringed on the jurisdiction of the Ministry of Interior (OWSO Official Interview 05). The (understandably) limited agency of OWSO officials over matters governed by law or regulation is likely one reason why citizens saw limited downstream impact on the quality of services: “I do not know of any change in the performance of public officials, to me it looks the same” (Citizen Interview 01). For matters within their control, however, some OWSO officials report making modest but important changes in response to citizen feedback during the Forums. Describing his response to citizens’ complaints about the rudeness of OWSO staff, one official stated “on the hospitality of staff towards people accessing services, I tried to lobby my superiors to get more staff to address understaffing and I informed the staff about the feedback … and asked them change their behavior” (OWSO Official Interview 03). Another described how “after each forum, I always report to my staff on the points raised by citizens and I reflect to address our gaps. My staff also tries their best to correct it. Even though we have not changed 100 percent, we show good signs of improving” (OWSO Official Interview 04). Several other officials described taking action in response to the forums, albeit in general and non￾specific terms: “the forum is really fruitful for citizens’ understanding of OWSO, and for OWSO’s understanding of citizens’ concerns. The citizens provide constructive feedback, and we make the correction for better service delivery” (OWSO Official 06). At the same time, several participants noted the value of the information provided during the forum and especially the ability to approach OSWO directly, without the need for an intermediary who charged for this service, cited at 50,000 Riel (about $12 USD) (OWSO Official 02). The public meeting and information provided brought the citizens closer to the officials and allowed for more direct lines of communications between them, reducing the role of intermediaries. In terms of areas of improvement for public forums in the future, multiple respondents noted the need to accurately target the forums towards those who recently used the OWSO services, or who are likely to use them in the future. The following complaint from an OWSO official is illustrative: “the participants in the forum did not pay much attention to the information we delivered because most of them are elderly, whereas the people that demand OWSO services did not actually participate in the forums. So in the future, the project should more carefully consider the target audience for the forums to produce the most fruitful results (OWSO Official 03). Some of the youth and others also suggested the use of visual aids to make the presentations more interesting and to maintain participant interest (OWSO Official 08). 47 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV SOCIAL ACCOUNTABILITY ACTIVITIES AND PROGRESS TOWARDS USAID/CAMBODIA’S DRG OBJECTIVES This section addresses EQ 3: How appropriate and/or useful were USAID/Cambodia’s goals and objectives for the social accountability activities in achieving the Mission’s DRG objectives? The Mission’s DRG objectives center around protecting human rights and advancing government accountability (USAID Cambodia CDCS 2020-2025, p. 6). In particular, USAID/Cambodia seeks to strengthen the defense of constitutional and legal rights (IR 3.1), increase civic participation and action for reforms (IR 3.2), and strengthen public oversight of government institutions (IR 3.3). The ISAC, OW4C, and T4GC Activities are most relevant to the latter two objectives. Under these Activities, USAID/Cambodia sought to achieve IRs 3.2 and 3.3 through five distinct sub￾activities: i) mobile Apps / Chatbots for citizens linked to online dashboards for government officials, ii) door-to-door and other forms of community outreach to raise awareness of government accountability, iii) the YA / CAF program, iv) support for accountability-oriented CSOs to generate engagement via Facebook and Telegram, and v) community scorecards and public forums. The remainder of this section evaluates the goals and objectives of each of these sub-activities vis-a-vis USAID/Cambodia’s DRG objectives. The goal of the GCC App and OW4C Chatbot, in tandem with their associated dashboards, was to make it easier for citizens to engage with local government officials, access information about their services, and provide feedback on the quality of those services. In turn, greater information about citizens’ needs and feedback was expected to improve the quality of government services. Contrary to these aspirations, patterns of demand for and usage of these technology tools indicate they were not fully valued by citizens and government officials: after being introduced to these tools, citizens and government officials quickly stopped using them. Although our KIIs documented cases in which citizens and government officials reported that the tools were useful for accessing information on services, providing feedback, and addressing specific service requests, these cases remain anecdotal and conflict with actual usage patterns of these tools, suggesting they are the exception rather than the norm. The failure of the technology tools to stimulate sustained engagement raises an important set of questions: was this failure the result of unrealistic objectives? Was it the result of incongruence between these tools and local conditions, or limited buy-in from government officials? Or was it the result of weak or limited outreach and dissemination by the IPs? Would a more robust roll-out of the technology tools, stronger buy-in from government officials, and more intense outreach and dissemination have achieved sustained demand and usage of these tools? Or would citizens and government officials have resisted adopting this technology regardless of the intensity of outreach and dissemination? Unfortunately, it is impossible to know the answers to these counterfactual questions. All three Activities engaged in community outreach through door-to-door canvassing, community meetings, and loudspeaker tours. The goal of these activities was to raise awareness of the technology tools (in the case of OW4C and T4GC) and increase citizens’ awareness of government accountability and citizen participation. Although feedback from the YA KIIs was overwhelmingly positive about the effectiveness of most forms of community outreach, it was ultimately unsuccessful at stimulating sustained demand for the OW4C and T4GC technology tools. One potential explanation for these seemingly disparate findings is that outreach was effective at stimulating initial demand, but citizens quickly stopped using them because they were difficult to use, ineffective, incongruent with local USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 48 conditions, or otherwise undesirable. Conversely, the overwhelmingly positive feedback from the YAs may reflect response bias because YAs felt obliged to provide rosy reflections on activities they were involved in implementing. The downstream impact of the informational campaign on services, government accountability, civic participation, and better waste management is unknown due to the lack of tracking data. The objective of the YA and CAF programs was to facilitate implementation of the outreach, community scorecard, and public forums. But designers of the program also intended for the program to serve as a standalone intervention to help grow a corps of accountability leaders and activists within targeted communities. Taken as a whole, our analysis suggests this was an appropriate and useful goal for the program and would contribute to USAID’s objectives. Across all three Activities, most of the youth understood the purpose for the sub-activities, felt empowered by the effort, became better at community outreach, public speaking, and learned to work as a team. They were empowered by the information, training, and real-world experience received. Participation in the program expanded their horizons, networks, and skills, all of which they hoped to use in the future - whether for civic engagement, business or education. Regarding the effectiveness of ISAC’s support for CSOs to better engage their audiences through social media towards advancing the Mission’s DRG objectives, our results indicate that beneficiary CSOs viewed social media as an important way to engage with citizens and appreciated the support provided by ISAC. Social media also helped them reach diverse audience that they otherwise might not, especially during the COVID-19 pandemic. A small number of posts generated significant engagement online, including an I4C video viewed more than 226,000 times, but these cases were exceptional. The majority of 413 I4C posts uploaded on ISAC CSOs Facebook pages had less than 10 likes/loves and users rarely commented on the posts. Sustainability is also an issue, because the CSOs are reliant on ISAC to create content. Evidence from the KIIs and YA survey regarding the effectiveness of the public forums and community scorecards suggests that these fora are constructive venues for stimulating engagement between citizens and government officials. Service providers generally felt that community engagement provided valuable feedback that subsequently led to tangible improvements in the quality of services. Citizens also viewed these forums as a productive venue for learning important information about services and providing critical feedback to government officials. These topline results suggest that creating public spaces for citizens and authorities to engage constructively may be a promising avenue for advancing USAID/Cambodia’s DRG objectives. Further evidence is needed to verify this potential, however, because these results are based entirely on anecdotal, self-reported testimony from beneficiaries and project stakeholders. 6. RECOMMENDATIONS We organize our recommendations according to three intervention categories: new technology innovations to promote accountability; programs to promote youth leadership and civic participation; and programs to promote collective action and civic participation at the community level. All recommendations are aimed at USAID/Cambodia and its implementing partners. 49 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV INTRODUCING NEW, TECH-BASED TOOLS TO PROMOTE GOVERNMENT ACCOUNTABILITY • When considering whether to introduce new, tech-based tools to promote accountability, policymakers should carefully consider how the new tool would fit within existing accountability processes, how it will build on or improve these processes, and under what conditions demand for these tools among citizens and government officials would be self-sustaining. • When launching new tech-based accountability tools, it is essential to conduct extensive outreach to ensure a large and sufficient number of users on the new platform. Only once a platform reaches a certain threshold number of regular users will the crowd-sourced information posted to the platform be viewed as accurate, up-to-date, relevant, and comprehensive. Below this threshold, there will not be enough useful information on the platform to attract a steady audience. • Rather than launching the tool in multiple locations simultaneously, roll-out should be staggered so implementers can achieve a high level of outreach saturation and achieve a locality-specific, self-sustaining user base. Implementers should also condition the subsequent roll-out of the tool to new locations on evidence of robust and sustained demand in the initial pilot locations, as measured in real-time through site analytics data on the tools themselves. Only if the tool is widely used and adopted in the initial pilot location(s) should it be rolled-out to additional locations. • At the end of the outreach feedback module of YA/CAF Survey (N=299), we provided respondents with an opportunity to reflect on their involvement in the outreach activities and offer open-ended suggestions for improvements that could be made to future outreach efforts. Here, we provide a summary and synthesis of these open-ended responses: o Give prior notice to the authorities (commune or village leaders, school directors and teachers) regarding the outreach (date, time, outreach objective and plan, and number of YAs/CAFs participating) and engage them to obtain their trust and support in the outreach process. Encourage their participation in the outreach activities and obtain support in disseminating information. o Adopt a phased approach by disseminating general information to the public using various channels (social media, leaflets, loudspeakers, tuk tuks, and radio) before engaging in outreach activities. o Allocate sufficient time to fully elaborate on the technology/activity and to answer questions as needed using terms that are clear and easy to understand for everyone. o Include more in-person demonstrations of technology or practices to help the citizens understand the content and the practices better. o Provide more support to the YAs/CAFs conducting the outreach (D2D, school campaigns, etc.) including regular in-person visits of Activity staff. o Increase frequency of outreach to regularly communicate with citizens and build their trust. PROMOTING YOUTH ENGAGEMENT AND LEADERSHIP FOR GOVERNMENT ACCOUNTABILITY • This evaluation finds encouraging if tentative evidence that the YA and CAF programs were effective at building leadership and communication skills among youth participants. Based on USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 50 these findings, we recommend that USAID/Cambodia and its IPs continue to pursue these types of programs. • To generate additional, more definitive evidence on the effectiveness of these programs, we recommend collecting before vs. after information on leadership, accountability, and other outcomes of interest. Combining this information with an experimental or quasi-experimental, comparison-group evaluation design would be especially valuable for strengthening the credibility of available evidence regarding these programs’ effectiveness in Cambodia. PROMOTING COLLECTIVE ACTION FOR GOVERNMENT ACCOUNTABILITY • This evaluation finds encouraging if tentative evidence that public forums and community scorecards are productive fora for stimulating engagement between citizens and government officials, educating citizens on how to access government services, and soliciting feedback on how those services could be improved. Based on these results, we recommend that USAID/Cambodia and its IPs continue to pursue these types of programs. In tandem, USAID/Cambodia should commission a rigorous experimental or quasi-experimental impact evaluation of these programs to verify their efficacy in Cambodia. 51 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV ANNEX A. EVALUATION SCOPE OF WORK USAID DRG-LER II Tasking Request: N064 Cambodia Social Accountability Portfolio MT PE Date of Request: January 25, 2022 Type of Task: • Developing Learning Questions & Agendas • Indicator Selection • Country Data Portrait • Evidence Gap Map • Literature Review • Systematic/Evidence Review • Public Opinion Survey • Case Study ✔ Performance Evaluation • Impact Evaluation • Metaketa • Auxiliary Study (Other Research) • Knowledge Dissemination and/or Utilization Statement of Work (SOW): The social accountability program under the Democracy and Governance Office consists of a five-year, flagship activity under a Leader with Associates award and three Small Business Applied Research (SBAR) activities under contracts. These activities were designed and procured as a concerted effort in response to the Democracy, Human Rights, and Governance Assessment conducted in 2019, to utilize technology and social media innovations to increase citizens’ access to public services offered by Cambodia’s local government entities, establish platforms for them to provide feedback on the services they receive, and hold their government accountable for the quality of services. Further details of each activity are as follows. Innovations for Social Accountability in Cambodia (ISAC) - [10/01/2019 - 09/30/2024] ISAC, implemented by FHI 360, uses technology to expand and enhance citizen engagement on issues of pressing concern for the Cambodian public through increased access to credible information, strengthened coalitions for collective actions, improved evidence-based policy analysis and dialogue, and the development of effective social accountability tools using cutting edge information and communications technologies (ICT) to provide citizens with venues in which to hold the Cambodian government accountable. Geographical coverage: Battambang, Banteay Meanchey, Siem Reap, Kampong Chhnang, Phnom Penh, Kandal, and Kampot. One Window for Citizens (OW4C) - [10/01/2019 - 09/30/2022] OW4C, implemented by Nickol Global Solutions LLC , aims to increase the capacity of civil society to engage and hold the government accountable through the One Window Service Offices using a series of carefully planned and tested interventions, including design and deployment of civic tech tools, USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 52 outreach/engagement activities, monitoring activities, and other complementary activities supporting innovation, success, and sustainability. Geographical coverage: Battambang, Banteay Meanchey, and Kampong Chhnang. Solid Waste Accountability Platform (SWAP) - [10/01/2019 - 09/30/2022] The SWAP activity, implemented by Triangle Environmental Health Initiative LLC, works to improve the ability of citizens to monitor waste management services and implements Waste Voice, an innovative community-monitoring system, in order to increase opportunities for Cambodians’ to hold their government accountable. Geographical coverage: Siem Reap, Kampong Chhnang, and Kampong Cham. USAID Tech for Green Cities (T4GC) - [10/01/2019 - 9/30/2022] The T4GC activity, implemented by the Institute for Development Impact, seeks to improve the transparency of service delivery by increasing access to data and creating opportunities for Cambodians to hold their government accountable. T4GC provides channels for citizens to report waste issues and hazards in real-time and promotes a circular economy in Cambodia by introducing and strengthening the Reduce-Reuse-Recycle strategy. Geographical coverage: Battambang, Banteay Meanchey, Kampong Chhnang, and Oddar Meanchey. Evaluation Purpose The primary purpose of this performance evaluation is to validate the theory of change of DG’s social accountability program; to assess the performance of the activities and effectiveness of the mechanisms and approaches to improve the local government’s accountability to its constituents; and to identify actionable recommendations that can inform any course correction for the remaining years of ISAC and for future activity portfolio designs for DG and other USAID technical offices. The following are the proposed evaluation questions: 1. How effective have the approaches been, including using technology and youth engagement, in sharing information, increasing citizen engagement with local administration and service providers, and in raising awareness of government accountability? 2. To what extent have the social accountability activities increased government accountability at the local level? 3. How appropriate and/or useful were USAID/Cambodia’s goals and objectives for the social accountability activities in achieving the Mission’s DRG objectives? 4. What are some potential strategies for sustainability of the program approaches such as tech tools, youth engagement, etc. developed under USG assistance? The evaluation will cover the 2.5-year period from 10/01/2019 - 04/01/2022 of all activities. This is the final year for the three SBARs, and the implementation midpoint for ISAC. Deliverable(s): The primary deliverable is a performance evaluation, which will include the following specific deliverables: 1. Post-award Kickoff Meeting 2. Evaluation Work Plan 53 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV 3. Draft Data Collection Tools and Testing Protocol 4. In-briefing 5. Field Work – Quantitative and Qualitative Data Collection 6. Data Analysis and Reporting 7. Regular Updates on the Field Work 8. Findings and recommendation workshop 9. Exit Briefings 10. Draft Evaluation Report 11. Matrix illustrating evaluation questions, findings, conclusions, and recommendations 12. Final Evaluation Report and Cleaned Data 13. Draft Post-Evaluation Action Plan Dates of performance and timeline: The Date of Performance is no later than April 4, 2022. The following schedule will be based on the Start of Performance Date: Activity Schedule Desk review Week 1 Submit draft Evaluation Design Week 2 USAID/Cambodia review of draft Evaluation Design Week 3 In country fieldwork (in-briefing, data collection, data analysis, out-briefing) Week 4-9 Draft evaluation report preparation and submission Week 10-11 USAID/Cambodia review of the draft report (14 days) Week 12-14 Final evaluation report preparation and submission Week 15-16 Required Personnel: ● Sr. Team Leader ● An international expert in DRG evaluation ● Local Expert in civic engagement and social accountability ● Logistics/Program Assistant ● Translator￾Submission Instructions: You will have fifteen (15) business days to submit a brief concept note and budget. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 54 ANNEX B. YA SURVEY INSTRUMENT ISAC varname Question text Answer options respid Enter respondent ID Numeric consent1 Thank you for taking the time to speak with us today. My name is [TEAM MEMBER NAME]. I am a researcher working with Indochina Research Ltd., a research company based in Phnom Penh, collaborating with Social Impact, a research company based in the United States. We are conducting an evaluation of USAID's Innovations for Social Accountability (ISAC) project. This study was commissioned by USAID's office in Cambodia. The purpose of this evaluation is to better understand the effectiveness of USAID’s work with local governments here in Cambodia. The results of this evaluation will be used to improve USAID’s work in Cambodia in the future. The phone survey will take 15 to 20 minutes. The survey will cover the following topics: the effectiveness of technology developed by the activity, door-to-door outreach, and community scorecards, and the extent to which the Community Accountability Facilitator (CAF) increased participation in civic affairs. We are contacting you today because you are a CAF in the ISAC program, or you were previously a CAF. Your participation in this survey is voluntary, and you are under no obligation to participate. If you start the interview and wish to stop at any time for any reason, or if you do not want to answer any questions, you may do so without consequence. consent2 The information that you and other CAF respondents provide will be used to write a report evaluating the USAID activity. This report will be shared with USAID and made publicly available via USAID’s Development Experience Clearinghouse (DEC). However, your name and position will not appear in any reporting, and it will be impossible to attribute any specific findings to you. The anonymized data from this survey may be made publicly available for other researchers to use and analyze. However, the anonymized data will not contain any information that could be used to identify you, such as your name or job title. There are no significant risks to your participation in this interview. You will not receive any direct benefit or compensation for participating in this interview. Although this evaluation will not benefit you personally, we hope that our results will improve potential future USAID programming. If you have any questions or concerns about your rights as a participant, you may contact the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884, or Bopha Kong at +855 92 665 993. consent3 Do you have any questions? [If yes, enumerator should answer questions] 0-No 1-Yes consent4 Do you agree to participate in this study? [If no, end interview] 0-No 1-Yes 55 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV varname Question text Answer options Effectiveness of door-to-door and other forms of outreach at disseminating I4C information outreach_txt1 [Select all that apply] As a CAF, what type of outreach to introduce citizens to the Information for Citizens (I4C) were you involved in? 1-Door to door 2-Community meetings 88-Other (please specify) outreach_txt2 For each type of outreach reported: [outreach_type]_effective [OUTREACH TYPE] was an effective strategy for introducing citizens to the Information for Citizens (I4C) messages. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_effective_nowhy If disagree or strongly disagree: Why do you think [OUTREACH TYPE] was not an effective at introducing I4C information? Open response [outreach_type]_citizens_engaged During [OUTREACH TYPE], citizens were genuinely interested in learning about the Information for Citizens (I4C) messages? Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_enough_time During [OUTREACH TYPE], facilitators like yourself had enough time to fully introduce the Information for Citizens (I4C) messages. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_accountability As a result of the [OUTREACH TYPE], citizens have become more likely to raise issues or concerns to local government officials in the future. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_improvements_what Reflecting on your involvement in [OUTREACH TYPE] to disseminate Information for Citizens (I4C) messages, what improvements do you think could have been made to the process? Open response Effectiveness of Community Scorecards cs_enhance_info The Community Scorecard program enhanced citizens' access to information about the quality of local government services. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_ctzns_active During the Community Scorecard meetings, citizens actively participated by asking questions and voicing their priorities and identifying areas for improvement. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_govt_listen During the Community Scorecard meetings, government officials listened to citizens and took their feedback seriously. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 56 varname Question text Answer options 4-Strongly disagree 97-Do not know 99-Not applicable cs_govt_change As a result of the Community Scorecard meetings, government officials made changes and improvements that they otherwise would not have made. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_raise_future As a result of the Community Scorecards, citizens have become more likely to raise issues or concerns to local government officials in the future. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_govt_qual_improve As a result of the Community Scorecards, the overall quality of local government services improved in areas where the Community Scorecards were held. Agree or disagree. 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable Impacts of volunteering in the Community Facilitators program ya_inspr_ldr Volunteering in the Community Facilitator program has inspired me to become a leader in my community. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_advc_chg Volunteering in the Community Facilitator program taught me how to advocate for change in my community. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_new_doors Volunteering in the Community Facilitator program opened new doors and opportunities for me. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_more_engaged As a result of my participation in the Community Facilitator program, I am more engaged in my community than I otherwise would be. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_impact Please describe in your own words the impact the CAFs program has had on you. Open response 57 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV OW4C varname Question text Answer options respid Enter respondent ID Numeric consent1 Thank you for taking the time to speak with us today. My name is [TEAM MEMBER NAME]. I am a researcher working with Indochina Research Ltd., a research company based in Phnom Penh, collaborating with Social Impact, a research company based in the United States. We are conducting an evaluation of USAID's One Window for Change (OW4C) project. This study was commissioned by USAID's office in Cambodia. The purpose of this evaluation is to better understand the effectiveness of USAID’s work with local governments here in Cambodia. The results of this evaluation will be used to improve USAID’s work in Cambodia in the future. The phone survey will take 15 to 20 minutes. The survey will cover the following topics: the effectiveness of technology developed by the activity, door-to-door outreach, and community scorecards, and the extent to which the Youth Ambassador increased participation in civic affairs. We are contacting you today because you are Youth Ambassador in the OW4C program, or you were previously a Youth Ambassador. Your participation in this survey is voluntary, and you are under no obligation to participate. If you start the interview and wish to stop at any time for any reason, or if you do not want to answer any questions, you may do so without consequence. consent2 The information that you and other youth ambassador respondents provide will be used to write a report evaluating the USAID activity. This report will be shared with USAID and made publicly available via USAID’s Development Experience Clearinghouse (DEC). However, your name and position will not appear in any reporting, and it will be impossible to attribute any specific findings to you. The anonymized data from this survey may be made publicly available for other researchers to use and analyze. However, the anonymized data will not contain any information that could be used to identify you, such as your name or job title. There are no significant risks to your participation in this interview. You will not receive any direct benefit or compensation for participating in this interview. Although this evaluation will not benefit you personally, we hope that our results will improve potential future USAID programming. If you have any questions or concerns about your rights as a participant, you may contact the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884, or Bopha Kong at +855 92 665 993. consent3 Do you have any questions? [If yes, enumerator should answer questions] 0-No 1-Yes consent4 Do you agree to participate in this study? [If no, end interview] 0-No 1-Yes Effectiveness of the Facebook Chatbot cb_txt1 For each of the following statements, please tell me if you agree or disagree. USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 58 varname Question text Answer options cb_easier_report The Pidor the Smart Villager Facebook Chatbot made it easier to citizens to access OWSO services. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cb_use_again Citizens who used the Pidor the Smart Villager Facebook Chatbot once were likely to use it again. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cb_use_recommend Users of the Pidor the Smart Villager Facebook Chatbot were likely to recommend it to their friends. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cb_new_answers The Pidor the Smart Villager Facebook Chatbot enabled citizens to get answers to questions about OWSO services that they otherwise would not have been able to answer. 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cb_effective_feedback Citizens viewed the Pidor the Smart Villager Facebook Chatbot as an effective avenue for providing feedback to OWSO officials on the quality of services they received from OWSO and areas for improvement. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cb_issues_serious Citizen feedback submitted through the Pidor the Smart Villager Facebook Chatbot were taken seriously by OWSO officials. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable Effectiveness of door-to-door at raising awareness of the app outreach_type [Select all that apply] As a youth ambassador, what type of outreach to introduce citizens to the Pidor the Smart Villager Facebook Chatbot and Interactive Voice Response (IVR) were you involved in? 1-Door to door 2-Community meetings 88-Other (please specify) outreach_txt1 For each type of outreach reported: [outreach_type]_effective [OUTREACH TYPE] was an effective strategy for introducing the Facebook Chatbot and IVR to a large number of citizens. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_effective_nowhy If disagree or strongly disagree: Why do you think [OUTREACH TYPE] was not an effective outreach strategy? Open response 59 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV varname Question text Answer options [outreach_type]_citizens_engaged During [OUTREACH TYPE], citizens were genuinely interested in learning about the Facebook Chatbot and IVR? Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_enough_time During [OUTREACH TYPE], canvassers like yourself had enough time to fully introduce the Facebook Chatbot and IVR. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_improvements_what Reflecting on your involvement in [OUTREACH TYPE] during the OW4C activity, what improvements do you think could have been made to the door-to-door outreach process? Open response Effectiveness of Public Forums pf_txt1 For each of the following statements, please tell me if you agree or disagree. pf_enhance_info During the public forums, citizens received accurate information about the quality of local government services. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable pf_ctzns_active During the public forums, citizens actively participated by asking questions and voicing concerns. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable pf_govt_listen During the public forums, government officials listened to citizens and took their feedback seriously. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable pf_govt_change As a result of the public forums, government officials made changes and improvements that they otherwise would not have made. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable pf_raise_future As a result of the public forums, citizens have become more likely to raise issues or concerns to local government officials in the future. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable pf_virtual_effective Virtual forums were just as effective as in person forums. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 60 varname Question text Answer options 97-Do not know 99-Not applicable pf_virtual_effective_why If disagree or strongly disagree: Why do you think virtual forums were not as effective as in-person forums? Open response Effectiveness of Community Scorecards cs_enhance_info The Community Scorecard program enhanced citizens' access to information about the quality of OWSO services. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_ctzns_active During the Community Scorecard meetings, citizens actively participated by asking questions, voicing their priorities, and identifying areas for improvement. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_govt_change As a result of the Community Scorecard meetings, OWSO offices made changes and improvements that they otherwise would not have made. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_raise_future As a result of the Community Scorecards, citizens have become more likely to raise issues or concerns to OWSO offices in the future. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable cs_govt_qual_improve As a result of the Community Scorecards, the overall quality of OWSO services improved in areas where the Community Scorecards were held. Agree or disagree. 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable Impacts of volunteering in the Youth Ambassadors program ya_inspr_ldr Volunteering in the Youth Ambassadors program has inspired me to become a leader in my community. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_advc_chg Volunteering in the Youth Ambassadors program taught me how to advocate for change in my community. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_new_doors Volunteering in the Youth Ambassadors program opened new doors and opportunities for me. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 61 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV varname Question text Answer options 97-Do not know 99-Not applicable ya_more_engaged As a result of my participation in the Youth Ambassador program, I am more engaged in my community than I otherwise would be. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_impact Please describe in your own words the impact the Youth Ambassadors program has had on you. Open response T4GC varname Question text Answer options respid Enter respondent ID Numeric consent1 Thank you for taking the time to speak with us today. My name is [TEAM MEMBER NAME]. I am a researcher working with Indochina Research Ltd., a research company based in Phnom Penh, collaborating with Social Impact, a research company based in the United States. We are conducting an evaluation of USAID's Tech for Green Cities (T4GC) project. This study was commissioned by USAID's office in Cambodia. The purpose of this evaluation is to better understand the effectiveness of USAID’s work with local governments here in Cambodia. The results of this evaluation will be used to improve USAID’s work in Cambodia in the future. The phone survey will take 15 to 20 minutes. The survey will cover the following topics: the effectiveness of technology developed by the activity, door-to-door outreach, and community scorecards, and the extent to which the Youth Ambassador increased participation in civic affairs. We are contacting you today because you are Youth Ambassador in the T4GC program, or you were previously a Youth Ambassador. Your participation in this survey is voluntary, and you are under no obligation to participate. If you start the interview and wish to stop at any time for any reason, or if you do not want to answer any questions, you may do so without consequence. consent2 The information that you and other youth ambassador respondents provide will be used to write a report evaluating the USAID activity. This report will be shared with USAID and made publicly available via USAID’s Development Experience Clearinghouse (DEC). However, your name and position will not appear in any reporting, and it will be impossible to attribute any specific findings to you. The anonymized data from this survey may be made publicly available for other researchers to use and analyze. However, the anonymized data will not contain any information that could be used to identify you, such as your name or job title. There are no significant risks to your participation in this interview. You will not receive any direct benefit or compensation for participating in this interview. Although this evaluation will not benefit you personally, we hope that our USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 62 varname Question text Answer options results will improve potential future USAID programming. If you have any questions or concerns about your rights as a participant, you may contact the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884, or Bopha Kong at +855 92 665 993. consent3 Do you have any questions? [If yes, enumerator should answer questions] 0-No 1-Yes consent4 Do you agree to participate in this study? [If no, end interview] 0-No 1-Yes Effectiveness of the Green Cambodia Citizens App gc_txt1 For each of the following statements, please tell me if you agree or disagree. gc_easier_report The Green Cambodia Citizens App made it easier to citizens to report waste management issues to waste management service providers. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable gc_use_again Citizens who used the Green Cambodia Citizens App once were likely to use it again. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable gc_use_recommend Users of the Green Cambodia Citizens App were likely to recommend it to their friends. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable gc_new_reports The Green Cambodia Citizens App encouraged citizens to report waste management issues that they otherwise would not have reported at all (e.g. unauthorized dumping, burning, or disposal). Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable gc_effective_feedback Citizens viewed the Green Cambodia Citizens App as an effective avenue for providing feedback to Waste Management service providers and officials. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable gc_issues_serious Most waste management issues submitted through the Green Cambodia Citizens App were acted on by Waste Management service providers and officials. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable Effectiveness of door-to-door and other forms of outreach at raising awareness of the app outreach_type [Select all that apply] As a youth ambassador, what type of outreach to introduce citizens to the Green Cambodia Citizens App were you involved in? 1-Door to door campaigns 2-Facebook and radio talk-show 63 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV varname Question text Answer options 3-Social media outreach 4-Outreach in schools 5-Clean-up campaigns 6-Community meetings 88-Other (please specify) outreach_txt1 For each type of outreach reported: [outreach_type]_effective [OUTREACH TYPE] was an effective strategy for introducing the Green Cambodia Citizens App to a large number of citizens. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_effective_nowhy If disagree or strongly disagree: Why do you think [OUTREACH TYPE] was not an effective outreach strategy? Open response [outreach_type]_citizens_engaged During [OUTREACH TYPE], citizens were genuinely interested in learning about the Green Cambodia Citizens App? Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_enough_time During [OUTREACH TYPE], canvassers like yourself had enough time to fully introduce the Green Cambodia Citizens App. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable [outreach_type]_improvements_what Reflecting on your involvement in [OUTREACH TYPE] during the T4GC activity, what improvements do you think could have been made to the process? Open response Impacts of volunteering in the Youth Ambassadors program ya_inspr_ldr Volunteering in the Youth Ambassadors program has inspired me to become a leader in my community. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_advc_chg Volunteering in the Youth Ambassadors program taught me how to advocate for change in my community. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_new_doors Volunteering in the Youth Ambassadors program opened new doors and opportunities for me. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 64 varname Question text Answer options ya_more_engaged As a result of my participation in the Youth Ambassador program, I am more engaged in my community than I otherwise would be. Agree or disagree? 1-Strongly agree 2-Agree 3-Disagree 4-Strongly disagree 97-Do not know 99-Not applicable ya_impact Please describe in your own words the impact the Youth Ambassadors program has had on you. Open response 65 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV ANNEX C. KII PROTOCOLS ISAC CSOS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Respondent organization: # of months respondent(s) has worked with the organization: Sex of respondent: QUESTIONS 1. What is your organization’s Mission and how does it relate to the work of USAID’s ISAC Activity? (Probes: main activities, geographic focus) 2. How was your organization involved in the USAID ISAC Activity? (Probe: What kind of support did you receive from ISAC? Were you involved in the Information for Citizens (I4C) campaign? Did you receive support from the Digital Innovation and Information Fund, and if so, how did this help your organization? Did you receive other kinds of training or support?) Do CSOs/grantees view social media as a viable avenue to engage with citizens? 3. How has ISAC training and support for the use of social media (e.g. Facebook and Telegram) helped advance your social accountability program or your organization’s Mission? 4. Reflecting on your organization’s use of social media so far, what have been the most significant challenges? (Probe: how have you tried to address these challenges?) 5. What do you think is the potential for social media to help advance social accountability or your organization’s Mission moving forward? (Probe: are there engagement strategies that your organization hasn’t done yet but should be considered?) To what extent did social media outreach generate meaningful engagement from citizens? 6. To what extent has social media been useful to inform citizens and engage with them? (Probe: Has social media helped you share information with citizens? Has it helped you collect feedback? Does it help raise awareness of your organization?) 7. Could you provide any specific examples of how social media generated a lot of hype or engagement about social accountability, citizen’s rights, your organization, or its causes? 8. How has the training you’ve received through USAID’s ISAC program helped your organization conduct outreach through social media? (Probe for specific changes, most significant achievements). 9. How has the funding you’ve received through USAID’s ISAC program helped your organization conduct outreach through social media? (Probe for specific changes, most significant achievements). USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 66 Wrap Up 10. Could you please share with me your organization’s social media pages on Facebook, Telegram, and any other relevant outlets? [Interviewer: please include those links in the interview notes]. 11. Is there anything else you would like to mention or discuss about ISAC support for social media or your organization’s use of social media, or the USAID ISAC Activity? COMMUNITY ACCOUNTABILITY FACILITATORS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Respondent organization: Sex of respondent: QUESTIONS Introduction 1. Tell me a bit about the Community Accountability Facilitator program. How were you recruited into the program? What kind of training did you receive? (Probes: What kind of oversight and interaction did you have with ISAC staff? How frequently did you have trainings, meetings, or other CAF events? What types of activities did you do as part of the program?) Effectiveness of door-to-door outreach, community meetings, and other forms of outreach at disseminating I4C information 2. Tell me about the outreach activities that you were involved in to raise awareness of the Information for Citizens (I4C) campaign. What were they and how did they work? (Probe: were some methods more effective than others?) 3. Overall, do you think these outreach campaigns were effective at raising awareness of the I4C messages? Is there anything you think could have been improved? (Probes: Were the meetings well-attended? Were people genuinely interested and engaged during the meeting?) 4. Is there anything you think could have been improved about the outreach activities? 5. After these events, how do you think the I4C campaign impacted attendees? What new activities or actions did they take as a result of I4C that they would not otherwise have taken? Impacts of volunteering in the Community Accountability Facilitators 6. What do you think are the most important skills that you developed while working as a CAF? (Probe: what new things did they learn? How did the program advance your professional development?) 7. Tell me a bit about how the Community Accountability Facilitators program has influenced your career. (Probe: Did it introduce you to new networks of people? Did it open new opportunities for you? What were those opportunities)? 8. How has the program influenced how you think about citizens’ rights and the performance of your Sangkat/commune administration? (Probe: has the program led you to become more involved in local government activities? Has the program led you to become more engaged in your community?) 67 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV 9. Have you or will you use the skills and knowledge you gained as a CAF in other environments? Effectiveness of Community Scorecards 10. Tell me about the Community Scorecard program. How did that work? (Probe: who participated? what specific steps were involved in the Scorecard process? What were some of the challenges?) 11. As far as you know, were the improvements called for in the Joint Accountability Action Plan actually implemented? (Probe: what improvements were called for in the JAAP, and how specifically were they addressed?) 12. One of the goals of the program was to teach citizens about the importance of monitoring the performance of the Sangkat/commune government, including primary schools and health centers, and holding them accountable for improvements. Do you think the program was successful at this? (Probe: do you think citizens will be more likely to demand improvements from the Sangkat/commune administration, schools, and health centers in the future?) 13. By the same token, the program also sought to demonstrate to public officials that they should be responsive to the issues and concerns raised by citizens. Do you think the program was successful at this objective? (Probe: Why or why not? Do you think that Sangkat/commune government officials, schools, and health centers will be more responsive to citizens in the future as a result of the program?) Wrap Up 14. Is there anything else you want to mention about the CAF program, I4C campaign, or the community scorecards? COMMUNITY SCORECARDS – PARTICIPATING CITIZENS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Village, Commune, District, Province: Sex of respondent: QUESTIONS Introduction 1. Tell me about the Community Scorecard program. How did that work? (Probe: what was the objective of the program? Who participated? What specific steps were involved in the Scorecard process?) 2. How were you recruited to participate in the community scorecard meetings? (Probe: Which meeting did you participate in (inception meeting, shared vision meeting, scoring meeting, interface meeting)? 3. How was your experience participating in these meetings? (What was interesting about the meetings? What did you learn during the meetings? How was your voice heard? Do you feel like your concerns were heard or paid attention to?) Impacts on citizen engagement with local government officials, knowledge of local government and services, and awareness of accountability USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 68 4. During the Scorecard meetings, were you able to voice your views and opinions? What types of issues did you raise during the meetings? 5. During the Scorecard meetings, what did you learn about your rights and responsibilities as a citizen? 6. During the Scorecard meetings, what did you learn about the Sangkat administration/government and/or local service providers (schoolteachers and directors, or health center nurses and managers)? (Probe: What did you learn about the quality of their services? Did you learn about areas for improvement? Did you learn about their budgets?) 7. Reflecting on your community’s experience during the Community Scorecard meetings, do you know if how your community interacts with the Sangkat/commune administration or other local service providers (schoolteachers and director, or health center nurses and managers) changed? (Probe: Did the program increase your community’s ability to organize to advocate for improved services? Is your community more likely to organize in the future as a result of the program? Would you be interested in engaging in community meetings in the future?) 8. How has the way you interact with the Sangkat/commune administration or other local service providers (schoolteachers and director, or health center nurses and managers) changed since participating in the Community Scorecard meetings? (Probe: What’s changed? Frequency of interaction? Mode of interaction?) Impacts on the quality of government services [questions not applicable to Phnom Penh citizens] 9. As far as you know, were the improvements called for in the Joint Accountability Action Plan actually implemented? (Probe: what improvements were called for in the JAAP, and how specifically were they addressed?) 10. Do you think the Community Scorecard will have any downstream impact on the quality of services your government receives in the future? (If so, why? By what channel(s) will the Scorecard process have a lasting impact on the quality of services in your community? If not, why not?) Wrap Up 11. Is there anything else you would like to mention or discuss about the Community Scorecard process? COMMUNITY SCORECARDS – SERVICE PROVIDERS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Location of interview (Sankat, District, Province): Sex of respondent: QUESTIONS Introduction 1. Tell me about the Community Scorecard program. How did it work? (Probe: what was the objective of the program? Who participated? What specific steps were involved in the Scorecard process?) 69 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV 2. What was your role in the Community Scorecard program? 3. How was your experience participating the Community Scorecard meetings? (Did you find them interesting? What did you learn during the meetings? Did you feel like your voice was heard?) Impacts on citizen engagement with municipal government officials and local government accountability 4. During the meetings, what did you learn about the needs and priorities of citizens? (Probe: what priority issues did citizens want the Sangkat/commune administration and service providers (schools and health centers) to address?) 5. How has the way you interact with citizens changed since participating in the Community Scorecard meetings? (Probe: What’s changed? Substance of the interactions? Frequency of interaction? Mode of interaction? Your accountability to citizens?) Impacts on the quality of government services 6. As far as you know, were the Joint Accountability Action Plan improvement actually implemented? (Probe: what improvements were called for in the JAAP, and how specifically were they addressed?) 7. Do you think the Community Scorecard will have any downstream impact on the quality of government services? (If so, why? By what channel(s) will the Scorecard process have a lasting impact on the quality of services? What improvements do you anticipate? If not, why not?) 8. Reflecting back on the Scorecard meetings, what were some of the challenges? Conversely, what were some of the successes? Wrap up 9. Is there anything else you would like to mention or discuss about the Community Scorecard process? OW4C YOUTH AMBASSADORS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Respondent location: Sex of respondent: QUESTIONS Introduction 1. Tell me a bit about the Youth Ambassadors program. How were you recruited into the program? What kind of training did you receive? (Probes: What kind of oversight and interaction did you have with OW4C staff? How frequently did you have trainings, meetings, or other events? What types of activities did you do as part of the program?) Effectiveness of door-to-door and other forms of outreach at disseminating I4C information USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 70 2. Tell me about the outreach activities that you were involved in to raise awareness of the Pidor the Smart Villager Chatbot. What were they and how did they work? (Probe: were some methods more effective than others? If so, why?) 3. Overall, do you think these outreach campaigns were effective at raising awareness of the Chatbot? (Probes: Were people genuinely interested and engaged during the outreach events? Do you think people reached through outreach were actually likely to use the Chatbot in the future?) 4. Do you think citizens viewed the Chatbot as an effective avenue for accessing information about OWSO services? (Probe: Was getting information through the Chatbot easier than just asking friends or family in their community about OWSO services?) 5. Sometimes the Chatbot was used by citizens after they returned from the OWSO office, to give feedback on the quality of services they received. Do you think these citizens viewed the Chatbot as an effective means for providing feedback to OWSO officials on the quality of services they received from OWSO and areas for improvements? (Probe: when citizens provided feedback on the quality of OWSO services through the Chatbot, did they actually expect OWSO officials to act on that information?) Effectiveness of Community Scorecards 6. Tell me about the Community Scorecard program. How did that work? (Probe: who participated? what specific steps were involved in the Scorecard process? What were some of the challenges?) 7. One of the goals of the program was to teach citizens about the importance of monitoring the performance of the municipal government, and holding them accountable for improvements. Do you think the program was successful at this? (Probe: do you think citizens will be more likely to demand improvements from the municipal government in the future?) 8. Similarly, the program also sought to demonstrate to public officials that they should be responsive to the issues and concerns raised by citizens. Do you think the program was successful at this objective? (Probe: do you think that municipal government officials will be more responsive to citizens in the future?) Effectiveness of Public Forums 9. Tell me about the Public Forums. How did they work? (Probes: who attended? What types of issues were discussed?) 10. During the Public Forums, did citizens actively participate by asking questions and voicing concerns? Did government officials take these questions and concerns seriously? 11. Do you think that participating in the Public Forums made citizens more likely to raise issues or concerns to local government officials in the future? (Probe: why or why not?) Impacts of volunteering in the Youth Ambassadors program 12. What do you think are the most important skills that you developed while working as a YA? (Probe: what new things did they learn? How did the program advance your professional development?) 13. Tell me a bit about how the Youth Ambassadors program has influenced you and/or your career. (Probe: Did it introduce you to new networks of people? Did it open new opportunities for you? What were those opportunities)? 14. How has the program influenced how you think about citizens’ rights and the performance of your municipal government? (Probe: has the program led you to become more involved in local government activities? Has the program led you to become more engaged in your community?) 15. Have you or will you use the skills and knowledge you gained as a YA in other environments? Wrap Up 71 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV 16. Is there anything else you want to mention about the YA program, the Pidor the Smart Villager Chatbot, or the Community Scorecards? DASHBOARD – DISTRICT/PROVINCIAL OMBUDSPERSONS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Location of interview (Sankat, District): Sex of respondent: QUESTIONS Introduction 1. Can you tell me a bit about your work as a District / Provincial Ombudsperson? (Probe: What are your primary duties and responsibilities? What are some of the challenges you face in performing these duties?) 2. How were you first introduced to the OW4C online dashboard (https://dashboard.ow4c.info/)? (Probe: When were you first introduced? Was the introduction done virtually or in-person? Were other people from your office also introduced? Have you received any additional information about the OW4C Dashboard since you were first introduced?) Value of the Dashboard 3. How do you use the OW4C online dashboard? (Probe: frequency, what do you use it for, which information is most useful?) 4. How has the OW4C dashboard influenced how you go about your duties as a District/Provincial Ombudsperson? 5. Have you shared the Dashboard site with anyone else from your office or other government offices? 6. Before the OW4C online dashboard was available, how did you collect citizens’ feedback or complaints about OWSO services? (Probe: method, frequency, intensity, and content of citizen feedback collected without the dashboard) 7. What do you think is the potential for the Dashboard? Do you think it could be used more widely by government officials? Are there any changes or improvements that you would recommend making to the Dashboard? Could it be adapted to other government agencies or sectors? 8. How did local government officials respond to information from the Dashboard? Do you know of any specific actions they took in response to information from the Dashboard? Impact of the Dashboard on the quality of their services 9. Has the information you’ve received through the Dashboard helped you make any specific changes to OWSO services? (Probe: If yes, please explain). Wrap Up USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 72 10. Is there anything else you would like to mention or discuss about the USAID OW4C Activity? COMMUNITY SCORECARDS – PARTICIPATING CITIZENS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Village, Commune, District, Province: Sex of respondent: QUESTIONS Introduction 1. Tell me about the Community Scorecards. How did they work? (Probe: what was the objective of the Forums? Who participated?) 2. How were you recruited to participate in the Community Scorecard meeting? 3. How was your experience participating in the Community Scorecard meeting? (Did you find the meeting interesting? What did you learn during the meeting? Did you feel like your voice was heard?) Impacts on citizen engagement with OWSO officials, knowledge of OWSO services, and accountability of OWSO Officials 4. During the Community Scorecard Meeting, what did you learn about One Window Service Office services? 5. During the Community Scorecard Meeting, were you able to voice your views and opinions? What types of issues were raised raise during the meetings? 6. During the Community Scorecard Meeting, did you ever fear reprisal from public officials, community leaders, or facilitators for voicing your opinions? 7. During the Community Scorecard Meeting, what did you learn about the performance of OWSOs or the quality of their services? (Probe: Did you discuss areas for improvement?) 8. Reflecting on your experience during the Community Scorecard Meeting, in what ways did your participation change how you interact with OWSO officials? (Probes: Did your participation make you more likely to use OWSO services in the future? Did it change what you expect from OWSO officials?) Impacts on the quality of OWSO services 9. As far as you know, were any of the improvements or action items discussed at the Community Scorecard Meeting actually implemented? (Probe: what improvements were discussed, and how specifically were they addressed?) 10. Do you think the Community Scorecard Meeting will have any downstream impact on the quality of services offered at the OWSO? (If so, why? By what channel(s) will the Forums process have a lasting impact on the quality of OWSO services? If not, why not?) Wrap Up 11. Is there anything else you would like to mention or discuss about the Community Scorecard Meeting? 73 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV PUBLIC FORUMS – PARTICIPATING CITIZENS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Village, Commune, District, Province: Sex of respondent: QUESTIONS Introduction 1. Tell me about the Public Forums. How did they work? (Probe: what was the objective of the Forums? Who participated?) 2. How were you recruited to participate in the Public Forums? 3. How was your experience participating in the Forum(s)? (Did you find the meetings interesting? What did you learn during the meetings? Did you feel like your voice was heard?) Impacts on citizen engagement with OWSO officials, knowledge of OWSO services, and accountability of OWSO Officials 4. During the Forums, what did you learn about One Window Service Office services? 5. During the Forums, were you able to voice your views and opinions? What types of issues were raised during the meetings? 6. During the Forums, did you ever fear reprisal from public officials for voicing your opinions? 7. During the Forums, what did you learn about the performance of OWSOs? (Probe: What did you learn about the quality of their services? Did you discuss areas for improvement?) 8. Reflecting on your experience during the Forums, in what ways did your participation change how you interact with OWSO officials? (Probes: Did your participation make you more likely to use OWSO services in the future? Did it change what you expect from OWSO officials? Impacts on the quality of OWSO services 9. As far as you know, were any of the improvements, action items, or issues discussed at the Public Forums actually addressed? (Probe: what improvements were discussed, and how specifically were they addressed?) 10. Do you think the Public Forums will have any downstream impact on the quality of services offered at the OWSO? (If so, why? By what channel(s) will the Forums process have a lasting impact on the quality of OWSO services? If not, why not?) Wrap Up 11. Is there anything else you would like to mention or discuss about the Public Forums? PUBLIC FORUMS – OWSO OFFICIALS DEMOGRAPHIC INFORMATION Date: Start time: USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 74 End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Location of interview (Sankat, District, Province): Sex of respondent: QUESTIONS Introduction 1. Tell me about the Public Forum(s) you participated in. How did it work? (Probe: what was the purpose of the Forum? Who participated? What was your role in the Forum?) 2. How was your experience participating the Public Forum? (Did you find them interesting? What did you learn during the meeting(s)? Did you feel like the discussions were productive?) Impacts on citizen engagement with OWSO officials, knowledge of OWSO services, and accountability of OWSO Officials 3. During the meetings, what did you learn about the needs and priorities of citizens requesting services from OWSOs? (Probe: what priority issues did citizens want the OWSO officials to address?) 4. How has the way you interact with citizens changed since participating in the Public Forums? (Probe: Have you attempted to host additional public forums on your own? Have you pursued any other channel to get citizens’ feedback on OWSO services?) 5. What did you do with the feedback you received through the public forums? Impacts on the quality of OWSO services 6. As far as you know, were any changes or improvements made based on the feedback received through the Public Forum(s)? (Probe: If yes, what specific improvements or changes were made? If no, why were no action items from the meeting implemented?) 7. Do you think the Public Forums will have any downstream impact on the quality of OWSO? (If so, why? What specific improvements do you anticipate? If not, why not?) Wrap Up 8. Reflecting back on the Forums, what were some things that could have been improved? 9. Is there anything else you would like to mention or discuss about the Public Forums? T4GC YOUTH AMBASSADORS DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: Primary notetaker name: Respondent name: Respondent title: Respondent location: 75 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Sex of respondent: QUESTIONS Introduction 1. Tell me a bit about the Youth Ambassadors program. How were you recruited into the program? What kind of training did you receive? (Probes: What kind of oversight and interaction did you have with T4GC staff? How frequently did you have trainings, meetings, or other events? What types of activities did you do as part of the program?) Effectiveness of door-to-door and other forms of outreach at disseminating information 2. Tell me about the outreach activities that you were involved in to raise awareness of the Green Cambodia Citizens App. What were they and how did they work? (Probe: were some methods more effective than others? If so, why?) 3. Overall, do you think these outreach campaigns were effective at raising awareness of the App? (Probes: Were people genuinely interested and engaged during the outreach events? Do you think people reached through outreach were actually likely to use the App in the future?) 4. Do you think citizens viewed the App as an effective avenue for reporting waste management issues such as illegal dumping and open burning to waste management authorities? (Probe: For example, when reported illegal dumping through the App, did they expect the authorities to act on that information) 5. Do you think citizens viewed the App as an effective avenue for reporting waste management issues such as missed waste collection to waste management service providers? (Probe: For example, when reported missed collection through the App, did they expect service providers to act on that information?) 6. As far as you know, did waste management service providers actually use the App / Dashboard and act on citizens’ reports? (Probe: Why or why not?) Impacts of volunteering in the Youth Ambassadors program 7. What do you think are the most important skills that you developed while working as a YA? (Probe: what new things did they learn? How did the program advance your professional development?) 8. Tell me a bit about how the Youth Ambassadors program has influenced your career. (Probe: Did it introduce you to new networks of people? Did it open new opportunities for you? What were those opportunities)? 9. How has the program influenced how you think about citizens’ rights and the performance of your municipal government? (Probe: has the program led you to become more involved in local government activities? Has the program led you to become more engaged in your community?) 10. Have you, or will you, employ the skills and knowledge you gained with T4GC in other environments? Wrap Up 11. Is there anything else you want to mention about the YA program, the Green Cambodia Citizens App, or the outreach activities? WASTE MANAGEMENT AUTHORITIES DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 76 Primary notetaker name: Respondent name: Respondent title: Respondent organization: # of months respondent(s) has worked with the organization: Sex of respondent: QUESTIONS Introduction 1. Can you tell me a little bit about your work as a Waste Management Authority? (Probe: What are your primary duties and responsibilities? What are some of the challenges you face in performing these duties?) 2. Before the T4GC online dashboard or the Green Cambodia App were available, how did you collect feedback or complaints on waste management services from the citizens? (Probe: What were the most significant obstacles receiving feedback? How often did you collect feedback?) Value of the dashboard 3. Do you use the T4GC online dashboard? (If the answer is no, skip to the question on wrap up) How were you first introduced to the dashboard? (Probe: When did you start accessing the dashboard? What training have you received? Were other people from your office also introduced? Have you received any additional training since you were first introduced?) 4. How do you use the T4GC online dashboard? (Probe: frequency, what do you use it for? which information is the most useful?) 5. What is your experience using the T4GC online dashboard? (Probe: strengths and weaknesses of the dashboard) 6. How has the T4GC online dashboard influenced how you perform your duties? (Probe: value added, opportunities with using the dashboard, specific examples) 7. Have you shared the Dashboard site with anyone else from your company? 8. What do you think is the potential for the Dashboard? (Probe: Do you think it could be used more widely by the waste management service providers? Are there any changes or improvements that you would recommend making to the Dashboard? Could it be adapted to other government agencies or sectors?) Impact of the Dashboard to improve the quality of services 9. Has the information you have received through the Dashboard helped you make any specific changes to waste management services? (Probe: If yes, please explain with specific examples) 10. Has any other T4GC activity other than the Dashboard influenced the quality of waste management services? (Probe: If yes, please explain with specific examples) Wrap Up 11. Is there anything else you would like to mention or discuss about the USAID T4GC Activity? WASTE MANAGEMENT SERVICE PROVIDERS (T4GC) DEMOGRAPHIC INFORMATION Date: Start time: End time: Interviewer name: 77 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV Primary notetaker name: Respondent name: Respondent title: Respondent organization: Location of organization (Village, Sankat, and District): # of months respondent(s) has worked with the organization: Sex of respondent: QUESTIONS Introduction 1. Can you tell me a little bit about your work as a Waste Management Service Provider? What exactly does your organization do? (Probe: What are your primary duties and responsibilities? What are some of the challenges you face in performing these duties?) 2. Before the Green Cambodia Citizens App and the T4GC online dashboard were available, how did you collect feedback or complaints on waste management services from the citizens? (Probe: What were the most significant obstacles receiving feedback? How often did you collect feedback?) Value of the Dashboard 3. Do you use the T4GC online dashboard? (If the answer is no, skip to the question on the App) How were you first introduced to the dashboard? (Probe: When did you start accessing the dashboard? What training have you received? Were other people from your company also introduced? Have you received any additional training since you were first introduced?) 4. How do you use the T4GC online dashboard? (Probe: frequency, what do you use it for? which information is the most useful?) 5. What is your experience using the T4GC online dashboard? (Probe: strengths and weaknesses of the dashboard) 6. How has the T4GC online dashboard influenced how you perform your work? (Probe: value added, opportunities with using the dashboard, specific examples) 7. Have you shared the Dashboard site with anyone else from your company? 8. What do you think is the potential for the Dashboard? (Probe: Do you think it could be used more widely by the waste management service providers? Are there any changes or improvements that you would recommend making to the Dashboard?) Impact of the Dashboard to improve the quality of services 9. Has the information you have received through the Dashboard helped you make any specific changes to your waste management services? (Probe: If yes, please explain with specific examples) Effectiveness of the Green Cambodia App for reporting waste management issues 10. Do you use the Green Cambodia Citizens App? (If the answer is no, skip to the wrap up question) How were you introduced to the App? (Probe: When did you start using the App? What training have you received? Were other people from your company also introduced? Have you received any additional training since you were first introduced?) 11. How do you use the Green Cambodia Citizens App? (Probe: frequency, what do you use it for? which information is the most useful?) 12. What is your experience using the Green Cambodia Citizens App to get reports on waste management issues? (Probe: strengths and weaknesses of the App) Value of feedback provided through the App 13. How has the Green Cambodia Citizens App influenced how you perform your work? (Probe: value added, opportunities with using the App, specific examples) USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 78 14. Have you shared the App with anyone else from your company? What do you think is the potential for the App? (Probe: Do you think it could be used more widely by the waste management service providers? Are there any changes or improvements that you would recommend making to the App?) Impact of the App to improve the quality of services 15. Has the App helped you make any specific changes to your waste management services? (Probe: If yes, please explain with specific examples) Wrap up 16. Is there anything else you would like to mention or discuss about the USAID T4GC Activity? 79 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV ANNEX D. YOUTH AMBASSADOR SURVEY RESULTS EFFECTIVENESS OF TECHNOLOGY & YOUTH ENGAGEMENT Facebook Chatbot OW4C Current Retired Female Male Percent who agree or strongly agree that: FB Chatbot made it easier to access OWSO services 98% 97% 100% 100% 94% Citizens who used FB Chatbot once were likely to use it again 98% 97% 100% 96% 100% Users of FB Chatbot were likely to recommend it to their friends 98% 98% 94% 98% 97% FB Chatbot enabled citizens to get answers to questions about OWSO services 99% 98% 100% 98% 100% Citizens viewed FB Chatbot as an effective avenue for providing feedback 98% 97% 100% 98% 97% Feedback submitted through FB Chatbot were taken seriously by OWSO officials 95% 97% 88% 98% 92% Observations 80 63 17 44 36 Green Cambodia Citizens App T4GC Current Retired Female Male Percent who agree or strongly agree that: GCC App made it easier to report waste management issues to SPs 100% 100% 100% 100% 100% Citizens who used GCC App once were likely to use it again 97% 98% 96% 98% 96% Users of GCC App were likely to recommend it to their friends 91% 94% 87% 89% 96% GCC App encouraged citizens to report waste management issues 83% 83% 83% 82% 84% Citizens viewed GCC APP as an effective avenue for providing feedback 97% 98% 96% 96% 100% Most issues submitted through GCC APP were acted on by SPs and officials 93% 96% 87% 91% 96% Observations 69 46 23 44 25 USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 80 EFFECTIVENESS OF YOUTH AMBASSADORS / CAF PROGRAM Youth Ambassador Program Full Sample OW4C T4GC ISAC Current Retired Female Male Percent who agree or strongly agree that: YA program inspired me to become a leader in my community 98% 98% 97% 99% 99% 95% 98% 98% YA taught me how to advocate for change in my community 98% 96% 100% 99% 99% 98% 98% 98% YA opened new doors and opportunities for me 99% 100% 99% 98% 100% 95% 99% 98% Because of YA, I am more engaged in my community than I otherwise would be 98% 100% 100% 97% 99% 96% 96% 98% Observations 299 80 69 150 217 82 179 113 81 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV EFFECTIVENESS OF OUTREACH Outreach: Typology of outreach activities Full Sample OW4C T4GC ISAC Current Retired Female Male Percent who participated in: Door-to-door outreach 65% 88% 52% 59% 68% 57% 67% 63% Community meeting outreach 68% 51% 46% 86% 67% 68% 69% 64% Social media outreach 5% 5% 13% 1% 6% 2% 5% 4% Facebook and radio talk show 3% 0% 13% 0% 2% 6% 4% 2% School campaigns 10% 0% 42% 0% 7% 16% 11% 9% Clean-up campaigns 2% 0% 7% 0% 1% 4% 2% 2% Other outreach activities 0% 0% 0% 1% 0% 1% 1% 0% Observations 299 80 69 150 217 82 179 113 Outreach: Effectiveness of Door-to-door Outreach Full Sample OW4C T4GC ISAC Current Retired Female Male Percent who agree or strongly agree that: D2D outreach was an effective strategy to introduce tech/I4C 100% 100% 100% 100% 100% 100% 100% 100% During D2D, citizens were interested tech/I4C 98% 99% 97% 98% 98% 98% 96% 99% During D2D, YAs had enough time to introduce tech/I4C 95% 100% 89% 94% 96% 94% 95% 96% Because of D2D, citizens are more likely to raise issues to local gov't 94% N/A N/A 94% 96% 90% 95% 93% Observations 194 70 36 88 147 47 119 71 Outreach: Community meeting Full Sample OW4C T4GC ISAC Current Retired Female Male Percent who agree or strongly agree that: Comm. meetings were an effective strategy to introduce tech/I4C 100% 100% 100% 100% 100% 100% 100% 100% During comm. meetings, citizens were interested in tech/I4C 97% 95% 94% 98% 98% 93% 97% 97% During comm. meetings, youths had enough time to introduce tech/I4C 96% 98% 91% 97% 98% 91% 95% 97% Because of comm. mtgs, citizens are more likely to raise issues or local gov't 95% N/A N/A 95% 97% 91% 95% 96% Observations 202 41 32 129 146 56 124 72 Outreach: Social media Full Sample OW4C T4GC ISAC Current Retired Female Male Percent who agree or strongly agree that: Social media was an effective strategy to introduce tech/I4C 100% 100% 100% 100% 100% 100% 100% 100% USAID.GOV CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PE EVALUATION DESIGN REPORT | 82 During social media outreach, citizens were interested tech/I4C 100% 100% 100% 100% 100% 100% 100% 100% During social media outreach, youths had enough time to introduce tech/I4C 100% 100% 100% 100% 100% 100% 100% 100% Because of social media outreach, citizens likely to raise issues to gov't 100% N/A N/A 100% 100% N/A 100% N/A Observations 14 4 9 1 12 2 9 5 Outreach: T4GC outreach activities T4GC Current Retired Female Male Percent who agree or strongly agree that: FB/radio talk show was an effective strategy to introduce tech/I4C 100% 100% 100% 100% 100% During FB/radio talk show, citizens were interested in tech/I4C 89% 100% 80% 86% 100% During FB/radio talk show, youths had enough time to introduce tech/I4C 78% 75% 80% 71% 100% Observations 9 4 5 7 2 School campaigns were an effective strategy to introduce tech/I4C 100% 100% 100% 100% 100% During school campaigns, citizens were interested in tech/I4C 100% 100% 100% 100% 100% During school campaigns, youths had enough time to introduce tech/I4C 97% 100% 92% 95% 100% Observations 29 16 13 19 10 Clean-ups were an effective strategy to introduce tech/I4C 100% 100% 100% 100% 100% During clean-ups, citizens were interested in tech/I4C 80% 100% 67% 100% 100% During clean-ups, youths had enough time to introduce tech/I4C 100% 100% 100% 100% 100% Observations 5 2 3 3 2 83 | CAMBODIA SOCIAL ACCOUNTABILITY PORTFOLIO PERFORMANCE EVALUATION DESIGN REPORT USAID.GOV EFFECTIVENESS OF SOCIAL ACCOUNTABILITY ACTIVITIES Community scorecard Full Sample OW4C ISAC Current Retired Female Male Percent who agree or strongly agree that: CS enhanced citizens' access to info about quality of local gov't services 99% 99% 99% 99% 98% 100% 97% During the CS meetings, citizens actively participated 97% 97% 97% 99% 93% 97% 98% Because of CS, gov't officials made improvements 93% 93% 94% 94% 92% 93% 94% Because of CS, citizens are more likely to raise issues to local gov't 93% 95% 91% 93% 92% 92% 93% Because of CS, overall quality of local gov't services improved 97% 100% 95% 99% 92% 96% 99% During the CS meetings, gov't officials took citizen feedback seriously 95% N/A 95% 99% 86% 98% 92% Observations 230 80 150 171 59 135 88 Public Forums OW4C Current Retired Female Male Percent who agree or strongly agree that: During PFs, citizens received accurate information about local gov't services 96% 97% 94% 96% 97% During PFs, citizens actively participated 95% 98% 82% 98% 92% During PFs, gov't officials took citizen feedback seriously 93% 94% 88% 91% 94% Because of PFs, gov't officials made improvements 94% 92% 100% 91% 97% Because of PFs, citizens are more likely to raise issues to local gov't 94% 95% 88% 93% 94% Virtual forums were just as effective as in person forums 72% 68% 88% 68% 78% Observations 80 63 17 44 36