JANUARY 2020 DISCLAIMER: The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. LOCAL ORGANIZATIONS—MOVEMENT TOWARDS SELF-RELIANCE ACTIVITY Impact Evaluation Design Report PHOTO CREDIT: MICHAEL GEBREMEDHIN / USAID Written and prepared by Aleta Starosta and Dr. Jeremy Springman, with contributions from Dr. Erik Wibbels, Lucy Right, and Aidan Schneider. This document was produced for review by the United States Agency for International Development Cambodia Mission under the Local Organizations—Movement Towards Self-Reliance (LO—MSTR) Activity contract: 72044219C00004. Implemented by: The Cloudburst Group 8400 Corporate Drive, Suite 550 Landover, MD 20785-2238 Tel: 301-918-4400 CONTENTS ABBREVIATIONS 3 1.0 INTRODUCTION 1 2.0 BACKGROUND 3 RELIANCE ON INTERNATIONAL FUNDING 3 LACK OF SOCIAL EMBEDDEDNESS 4 DIVISIONS WITHIN CIVIL SOCIETY 4 INSUFFICIENT CSO CAPACITY 5 3.0 INTERVENTIONS AND THEORETICAL FRAMEWORK 6 OVERVIEW OF THE LOCAL ORGANIZATIONS—MOVEMENT TOWARDS SELF RELIANCE ACTIVITY 6 INTERVENTIONS 7 THEORETICAL FRAMEWORK 9 4.0 HYPOTHESES, DATA SOURCES, & INDICATORS 13 RESEARCH HYPOTHESES (H) 13 DATA SOURCES 13 CSO IMPACTS 15 5.0 RESEARCH & SURVEY METHODOLOGY 17 CSO SAMPLING METHODOLOGY 18 MATCHING 19 LIMITATIONS 19 6.0 POWER ANALYSIS 20 7.0 CONCERNS AND CONSIDERATIONS 23 INSUFFICIENT SAMPLE SIZE 23 MATURATION 23 SPILLOVER 23 NON-COMPLIANCE AND UNEVEN TAKEUP OF THE INTERVENTION 23 ATTRITION 24 HUMAN SUBJECTS PROTECTION 24 ELECTIONS AND POLITICAL CLIMATE 24 8.0 TIMELINE AND TEAM COMPOSITION 25 9.0 DELIVERABLES 27 BASELINE & ENDLINE REPORT 27 CLA WORKSHOP 27 PRE-ANALYSIS PLAN 27 DATA SET & CODEBOOK 27 DISSEMINATION 27 10.0 REFERENCES 29 ABBREVIATIONS σ Mean (average) μ Standard Deviation ATE Average Treatment Effect CDCS Country Development Cooperation Strategy CSO Civil Society Organization CLR Cluster Level Reliability CPP Cambodia People’s Party DEC Development Experience Clearinghouse GIS Geographic Information System HI High Intensity ICC Intra-Class Correlation IRB Institutional Review Board FGD Focus Group Discussions KII Key Informant Interview LANGO Law on Associations and NGO LO-MTSR Local Organizations—Movement Towards Self-Reliance LI Low Intensity M&E Monitoring & Evaluation MDES Minimum Detectable Effect Size N Number of Respondents NGO Non-Government Organization PPS Probability Proportionate to Size R+ Resiliency+ RCT Randomized Control Trial RGC Royal Government of Cambodia ROCCS Resilient Organization in Closing Civic Space SBAR Small Business Applied Research USAID U.S. Agency for International Development USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 1 1.0 INTRODUCTION This report describes an Impact Evaluation (IE) design for work being conducted under USAID/Cambodia’s Local Organizations—Movement Towards Self-Reliance (LO-MTSR) Activity. The LO-MTSR is a three￾year, $2,230,510 USD project funded by the United States Agency for International Development (USAID). The period of performance is October 1, 2019–September 30, 2022. LO-MTSR was awarded as part of the Small Business Applied Research (SBAR) pilot mechanism. This evaluation will focus on the impact of an innovative Resiliency+ program on the financial resiliency and organizational capacity of 60 Civil Society Organizations (CSOs) in Cambodia. In Cambodia, nearly 5,000 local and international CSOs fill a critical role as service providers in the health, education, democracy, governance, and agriculture sectors, especially in remote areas and communities. For USAID/Cambodia to achieve its Development Objectives, it is critical for CSOs to be resilient and self-reliant. As donor funding becomes more limited or tied to specific governmental objectives, competition for these resources threatens to divide local organizations working towards development goals. While Cambodian organizations have “graduated” to become direct and compliant USAID grantees, they have failed to thrive as independent, mission-oriented organizations with diverse funding sources to insulate them from external shocks. Furthermore, while governments around the world work to shrink civic space and restrict the operations of organizations promoting accountability and good governance, reduced access to international funding will decrease the independence of CSOs and erode their ability to provide checks on governments (Khieng & Dahles, 2015). Encouraging CSOs to shift to more sustainable financing models is important for them to continue fulfilling this purpose. The LO-MTSR Activity aims to increase the organizational resiliency of Cambodian civil society organizations across the health, education, food security, agriculture, and democracy, human rights, and governance sectors by increasing their capacity to expand their networks and tap into new markets and revenue streams, thus decreasing their reliance on aid from foreign donors and empowering them on their journey to self-reliance. The impact evaluation proposes a framework for measuring the key development impacts of the LO￾MTSR Activity in Cambodia. The overarching policy question underlying the evaluation is: Does building organizational resiliency increase the ability of CSOs to strategically plan for and expand network connections outside of large donors, ultimately increasing their financial diversity? Based on the overarching policy question, a number of research objectives have been developed to focus the evaluation activities. Specifically, the evaluation will investigate the extent to which the activities under the LO-MTSR Activity generate the following outcomes and impacts: 1. Increased organizational capacity to protect against external closing space threats and organizational vulnerabilities that impact resiliency. 2a. Decreased CSO reliance on funding from USAID and other international donors. 2b. Decreased CSO reliance on funding from the Cambodian government and large international donors who are at odds with USAID’s CDCS objectives. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 2 3. Increased network connections within and across sectors. These three evaluation objectives form the basis for a series of testable development hypotheses and indicators on the impact of the LO-MTSR Activity, as well as for measuring the magnitude of that impact. The primary development hypothesis driving the LO-MTSR Activity is that by increasing the organizational resiliency of targeted CSOs through the Resiliency+ suite of activities, organizations will be better positioned to seek out and gain access to funding streams outside of traditional international donors, such as USAID, or other funding that may be at odds with USAID’s CDCS objectives. This increased financial diversity will empower local organizations to be increasingly independent of donor funds, further advancing USAID’s Journey to Self-Reliance goals. The evaluation will provide an evidence base for improved policy making and programming by testing the development hypotheses andenabling the LO-MTSR Activity theory of change to be validated and adjusted as needed to develop a program that could be implemented on a larger scale, both in Cambodia and around the globe. What follows in this report is an exploration of the theoretical underpinnings of the proposed program intervention, the theory of change, and the impact evaluation design. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 3 2.0 BACKGROUND This section provides background information on historic trends and challenges facing Cambodian civil society, including reliance on international funding, lack of social embeddedness, divisions within civil society, and insufficient capacity. In recent years, interference in CSO activities by the Royal Government of Cambodia (RGC) has intensified. While this increased scrutiny has resulted in burdensome registration requirements and invasive monitoring practices that affect the entire sector, the brunt of this interference has been targeted toward CSOs receiving foreign support for rights-based advocacy and democracy promotion, as well as local organizations focused on land rights and environmental protection. Much of this increased interference has found a legal basis in the Law on Associations and NGOs (referred to as LANGO) enacted in August 2015 (Curley, 2018), which was met with widespread criticism from civil society and the international community. Among concerns with the LANGO are mandatory registration for all domestic and international associations, unfettered discretion by the Ministry of Interior over registration, and the requirement that all associations and organizations be “politically neutral”. Legal restrictions on civil society have been part of a broader push by the Cambodia People’s Party (CPP) to stifle dissent to their 30-year rule and undermine political competition. LANGO was accompanied by similar legal restrictions on the media and the freedoms of expression and assembly, including the use of social media. In recent years, the RGC has used legal channels to limit the main opposition party, expel foreign NGOs, and eliminate independent media critical of the ruling party. These moves come despite increased pressure from the European Union and other international actors to discourage the CPP’s turn toward closing. While these actions represent a purposeful closure of civic space, four related characteristics make CSOs in Cambodia especially vulnerable to this type of restrictive legislation. RELIANCE ON INTERNATIONAL FUNDING Foreign support was critical in getting Cambodia civil society off the ground after the 1991 Paris Peace Accords ended the country’s decades long period of conflict. However, this massive influx of foreign support created a sector that is now widely regarded as “docile” and donor-facing (Öjendal & Sedara, 2006). While this perception of CSOs as “cowed into apathy by authoritarian politics and belittled by a patronizing donor community” is likely exaggerated, financial support for civil society organizations is dominated by external funding (Öjendal & Sedara, 2006, pg 519). According to the CSO Sustainability Index, 85% of CSO funding in Cambodia comes from foreign donors, itself a highly volatile source of funding that has been steadily decreasing since Cambodia’s transition to lower middle-income status in 2015 (USAID, 2017). According to original calculations by DevLab@Duke based on a nationally representative survey of Cambodian CSOs conducted by Suarez & Marshall (2014) in 2011, 78% of 137 Cambodian CSOs reported receiving funding from at least one foreign source. While essential to the continued operations of many CSOs, this reliance on foreign funding creates several problems for the sector. First, the problem of reliance on external donors for sustaining resources is exacerbated by a lack of revenue diversification. According to a 2011 survey of 312 CSOs across five provinces, 71% of local and 78% of international CSOs in Cambodia rely on a single source of funding (Khieng, 2014). Widespread reliance on a small number of international donors makes organizations especially vulnerable to closing civic spaces. According to a global dataset of legislation regulating civil society between 2009 and 2019 USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 4 provided by the International Center for Not-for-Profit Law, original calculations by DevLab@Duke suggest that restricting access to foreign funding is one of the most common legislative approaches to undermining civil society, while restrictions targeting domestic sources of funding are comparatively rare. In fact, Bromley (2019) report that 1/3 of the world’s countries have adopted legislation that restricts CSO access to foreign funding. LACK OF SOCIAL EMBEDDEDNESS This reliance on foreign donors also creates problems for CSO legitimacy, rooted in genuine concerns about “incentives to cater to donors’ programmatic priorities” rather than “grassroots constituencies” (Malena & Chhim 2009, pg. 8). It is well known that donor priorities have a strong influence on CSO programming (Bush, 2015; Khieng & Dahles, 2015; Khieng, 2014). According to Coventry (2016), donor dependence is a primary reason why the civil society sector is widely seen as “driven by the imperatives of international donors, owing their existence to the influence and financial support… and accountability to service users or other constituents is low." These two challenges facing CSOs are inherently linked, as a lack of perceived legitimacy by the broader Cambodian society hampers the shift to local sources of funding and perpetuates reliance on highly volatile and vulnerable donor flows. Relatedly, this lack of legitimacy reduces the social costs paid by the Cambodian state when restricting civil society organizations. According to Malena & Chhim (2009), the resulting “inability to demonstrate popular support undermines NGOs’ credibility and influence and has caused government officials to question their legitimacy and representativity” (Malena & Chhim 2009, pg. 8). In fact, attacking the sector as serving foreign rather than domestic interests has been a common tactic for the regime. In 2017, both foreign and domestic CSOs with links to the US were accused by the government of participating in a “US interference network” (USAID, 2017). This justification is common in countries where legal restrictions on CSOs are adopted (Bromley, 2019) DIVISIONS WITHIN CIVIL SOCIETY The vulnerability of civil society organizations to government interference also creates tensions between CSOs primarily focused on service delivery and those focused on more contentious political issues such as human rights, democracy promotion, and land rights. While CSOs engaged in the delivery of basic services are generally seen by government as valuable development partners1, those engaging in rights￾based advocacy are generally seen as opponents of the incumbent government (Coventry, 2016; Malena & Chhim, 2009). Interviews suggest that many service delivery CSOs believe that political issues are not relevant to their work and feel that “trouble-making” by “advocacy” NGOs draws unnecessary government scrutiny to the sector, hampering the ability of many CSOs to fulfill their primary missions. Similarly, advocacy NGOs are often critical of service delivery NGOs for complying with heavy-handed government monitoring, even when these monitoring tactics have no legal basis (Malena & Chhim, 2009). Despite these tensions, shrinking civic space affects both advocacy and service delivery NGOs by increasing the costs of compliance with government regulations and threatening to restrict access to international funding. 1 https://www.icnl.org/resources/civic-freedom-monitor/cambodia USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 5 INSUFFICIENT CSO CAPACITY Finally, despite an abundance of donor sponsored capacity building programs and the increased professionalization required to comply with donor reporting requirements, Cambodian CSOs remain characterized by “low technical capacity linked to poor skills in critical thinking and poor coordination” (Coventry, 2016, pg 6). According to original calculations by DevLab@Duke based on a nationally representative survey Cambodian CSOs conducted by Suarez & Marshall (2014) in 2011, a quarter of local CSOs reported having no strategic plan, 40% did not go through independent audits of their financial records, and 51% did not have a designated staff member focused on Monitoring & Evaluation (M&E). Furthermore, while capacity building initiatives have focused on a broad range of organizational practices, very few of these programs focus on revenue diversification or explicitly address concerns around closing civic spaces. This suggests a great deal of room to improve capacity strengthening programming aimed at the rapidly changing civic space context in Cambodia. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 6 3.0 INTERVENTIONS AND THEORETICAL FRAMEWORK The information provided below presents a synthesis of the LO-MTSR Activity. OVERVIEW OF THE LOCAL ORGANIZATIONS—MOVEMENT TOWARDS SELF RELIANCE ACTIVITY The LO-MTSR Activity Project Goal is to build the organizational resiliency of targeted CSOs so they are better able to strategically plan for and expand network connections outside of primary donors. In doing so, organizations will lessen reliance on government or large international donor funding that may be at odds with USAID’s Country Development Cooperation Strategy (CDCS) objectives. The primary development hypothesis driving the LO￾MTSR Activity is that by increasing the organizational resiliency of targeted CSOs through the Resiliency+ suite of activities, including the Resilient Organization in Closing Civic Space (ROCCS) workshop and assessment, one-on-one coaching, resources for leadership development, and training on entrepreneurship, communications, and social media, organizations will be better positioned to seek out and gain access to funding streams outside of traditional international donors such as USAID, or other funding that may be at odds with USAID’s CDCS objectives. This increased financial diversity will empower local organizations to be increasingly independent of donor funds, further advancing USAID’s Journey to Self￾Reliance goals. Figure 1 outlines the causal model approach to implementing performance metrics the LO-MTSR Activity. The primary activity of the LO-MTSR Activity is the implementation of an innovative Resiliency+ (R+) Framework, which uses adaptive strategies, more effective narratives, alternative organizational and funding models, stronger mechanisms of transparency and accountability, and wider networks across the public, private, and non-profit sector to increase organizational resiliency. It was developed through a rigorous, evidence-based learning process to increase financial resilience and decrease reliance on variable donor funding for local organizations operating in challenging environments. FIGURE 1. CAUSAL MODEL USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 7 The R+ Framework provides local organizations with the strategies, tactics, tools, and peer-to-peer assistance needed for addressing obstacles to self-reliance through four main steps: 1. Conduct an environmental assessment of the operational landscape to understand the threat, challenges, and opportunities facing local organizations. 2. Organize and prioritize identified threats and challenges, and chose tactics, tools, and strategies that can be implemented to address, mitigate, or respond to them. 3. Carry out an organizational model and life cycle self-assessment to determine internal strengths, weaknesses, threats, and opportunities related to the organizations’ existing business model and life stage. 4. Devise an intervention plan based on the outcomes from the three previous steps. R+ utilizes trained facilitators, called Coaches, to help organizations create customized interventions that mitigate the major threats identified in the framework, all of which are present to some degree in Cambodia. The core innovation of R+ is its contextual adaptability. By building in activities designed to customize the framework locally, R+ programming is uniquely equipped to address the challenges and needs of organizations operating in diverse contexts. INTERVENTIONS The focus of the impact evaluation is the bundle of Resiliency+ interventions. These interventions are offered to two different cohorts at two different intensities: High Intensity (HI) and Low Intensity (LI). One of the largest differences between these two levels of the intervention will be the accompaniment of the program activities with active coaching and mentoring. Because assignment to HI or LI will be random, our intent is that the design will enable a careful cost-benefit analysis of the additional value created by the resource-intensive coaching component. The relatively small sample size may result in an analysis that is statistically underpowered, but a more speculative evaluation of the relative impact of these two levels of the intervention will still be informative. RESILIENCY 101 WORKSHOP All organizations will send multiple staff representing various levels of seniority to participate in a Resiliency 101 Workshop. The three-day workshops will introduce organizations to the R+ Framework, including discussions of the most prevalent civic space threats and common vulnerabilities that make organizations susceptible to civic space shifts. During the workshops, CSOs will implement the Resilient Organizations in Changing Civic Space (ROCCS) Self-Assessment Tool. By taking this self-assessment, CSOs will gain insights into which areas of their organizations present potential vulnerabilities that could either be affected by external civic space threats or prevent an organization from capitalizing on opportunities that may arise in changing civic space. ROCCS is grounded in both research and insights from the latest in academic and practitioner thinking within the fields of organization-level crisis management, disaster management, organizational resilience across various industries, complex operating environments, and civil society organizational models and capacities. ROCCS reveals an organization’s capacity and ability to plan for, respond to, and recover from impacts of changing civic space. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 8 RESILENCY ROADMAP Following the Resiliency 101 workshops, each of the 60 CSOs will create a Resiliency Roadmap, which will be comprised of strategies, tools, tactics, and approaches that correlate to the challenges identified through ROCCS. The number of hours of project support will vary by HI and LI cohorts. COACHING AND MENTORING R+ Coaches will work one-on-one with each organization to provide on-going mentoring support for interventions detailed in each organization’s respective resiliency plan. R+ Coaches will schedule regular check-ins and be available on-call to assist with specific ad hoc challenges or difficulties organizations may face during the implementation process. The number of hours of project support will vary by HI and LI cohorts. RESILENCY TOOLKIT LO-MTSR will curate a virtual collection of tools and resources to help CSOs implement their Resiliency Roadmaps. These tools will include information on financial diversification, funding sources in Cambodia, and information on various trainings offered by our partner organization, PartnersGlobal, all translated into Khmer and adapted for the Cambodian context. This toolkit will be available to both the HI and LI cohorts. GROUP TRAININGS AND WORKSHOPS HI Cohort CSOs will be invited to participate in group trainings and workshops that address organizational needs identified in their ROCCS assessment. At least six group training/workshop events will be conducted during the invention timeframe. These workshops will be conducted by a mix of PartnersGlobal experts and/or locally hired subject matter experts fluent in Khmer. An illustrative sample of topics that may be covered at workshops is highlighted in Table 1 below. TOPIC DESCRIPTION / PURPOSE Civil Society Coalition Building Participants would do a civil society mapping of relevant actors in their operating space, conduct awareness raising on the importance of collective action and how to build coalitions based on common interests, and undertake an intensive coalition building activity to create and leverage new partnerships. Knowledge Building for CSO Sustainability To build understanding of the legal issues impacting CSO financial health, we would hold a workshop on regulations that affect CSOs’ ability to access funding. This may include raising awareness of anti-money laundering efforts and bank de-risking practices that unnecessarily target CSOs, restrictions on foreign funding of civil society, and challenges in obtaining tax-exempt status. TABLE 1—SAMPLE WORKSHOP TOPICS USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 9 Social Media and Marketing Raising awareness across organizations around shared issues and causes in addition to raising the profiles of individual organizations can be essential to building lasting change and increasing investments in CSO funding. Organizations and their staff would receive a training on developing a social media strategy, including how to use photography, video, and hashtags that align with best practices to increase their ability to use social media as a resource to expand their audience and amplify their message. The workshop would help organizations to identify influencers, citizen journalists, and cross-promote each other to further build these organizations influence and capacity in line with fundraising goals. Digital Security Knowledge and Expertise Effectively operating in today’s civic space requires the effective use of online tools and resources. Digital security is a critical yet often overlooked underlying factor to any organization as they transfer money, maintain an online presence and identity, store information on staff or stakeholders, and plan project-related activities. A digital security workshop would instill the importance of digital security in the minds of participants, present common security threats, and introduce various tools that are available. RESOURCES FOR LEADERSHIP DEVELOPMENT AND INSTITUTIONAL SUPPORT The HI Cohort will receive funds for leadership development and institutional support needed to implement their Resiliency Roadmaps. R+ Coaches will help organizations identify appropriate leadership and organizational coaching opportunities including professional conferences/training, network events, courses, and/or trainings by specialized resource partners or consultants. Intervention will be focused on building organizational capacity or helping implement identified improvements. THEORETICAL FRAMEWORK A large body of research suggest strong complementarities between democracy and development. Higher levels of economic development and citizen welfare make countries more amenable to the adoption and survival of democratic institutions (Przeworski et al., 2000), and conversely, the adoption and strengthening of democratic institutions leads to increased rates of economic growth and increased government investments in health and education (Acemoglu et al., 2019). Civic knowledge and press freedoms, both key elements of a healthy civil society, are also strongly related to improved government performance and reduced corruption (Adsera, Boix and Payne 2003). These complementarities between democracy, civic space, and development underscore the importance of CSOs operating across a broad range of sectors—including both the delivery of basic services and the promotion of political accountability—to the broader goal of democracy promotion. However, twin forces are working to undermine the ability of these organizations to perform their critical role in economic and political development. First, governments around the world are working to shrink civic space and restrict the operations of organizations that promote accountability and good governance. While regulations on civil society are frequently enacted to hinder the operations of CSOs engaged in advocacy, they typically generate similar burdens for CSOs engaged in more regime-compatible activities such as service delivery (Heiss, 2017). Second, shifting donor landscapes and uncertainty over international funding inhibit the ability of these organizations to invest in their resilience to closing spaces. Cambodia is a country where both forces are at work with particular intensity. As a result, the Resiliency+ intervention is aimed at strengthening CSOs operating across a range of critical sectors through coaching in both revenue diversification and organizational management and resiliency. By helping organizations to access new sources of sustainable funding while strengthening inter-organizational networks, Resiliency+ USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 10 aims to help CSOs adapt to environmental challenges that pose an existential threat to Cambodian civil society. Conducting a rigorous impact evaluation of this intervention will answer important research questions about the types of practices that contribute to CSO resilience in closing civic spaces. One of the greatest threats to the continued efficacy of the civil society sector in Cambodia has been increased competition over limited funding from international donors, as detailed in Section 2, Background. Reduced access to international funding threatens to undermine the independence of the civil society sector and erode its ability to act as a check on government. As funding from traditional donors becomes increasingly scarce, reliance on funding from the Cambodian government may continue to increase, or organizations may turn to alternative international donors working against USAID’s CDCS objectives. To avoid such a loss of autonomy, alternative resource mobilization strategies are desperately needed, and recent trends suggest that successfully employing alternative revenue generation strategies is possible. According to Khieng (2014), 75% of Cambodian CSO funding in 2011 came from grants and donations coming almost entirely external funding from abroad. However, the share decreased substantially from 92% in 2006, while the share of funding from earned income increased from 6% to more than 21% (24% for local NGOs). Successful earned-income ventures include a broad range of activities related to tourism and hospitality, agriculture and fisheries, publications and the media, training courses, service provision, and/or microcredit. CSO directors also reported that earned income provided a less volatile source of funding (compared to grants) and that they expected the share of funding coming from earned-income to continue to increase over the next five years. Local NGO directors also reported increasing attempts to secure donations from private companies and individuals through sponsorships and fund-raising events. This dramatic shift in CSO funding provides encouraging evidence that CSOs in Cambodia are willing and able to adapt their revenue mobilization strategies to changes in their environment. According to Khieng & Dahles (2015), the strategic responses employed by local CSO leaders to reduce external resource dependence entail a paradigm shift from external control to local embeddedness and increased autonomy. Future declines in bilateral aid mean that CSOs will need to secure funding from new sources if they are to survive. While only 4% (3% for local CSOs) of CSOs in Cambodia reported receiving funding from the RGC in 2011, this share doubled from 2% in 2006 (Khieng & Dahles, 2015). While all resource mobilization strategies impose some constraints on CSO activities, Khieng & Dahles (2015) find that funding from the government of Cambodia had a stronger goal displacement effect than other sources of CSO funding. As funding from traditional donors becomes increasingly scarce, reliance on funding from the Cambodian government may continue to increase, or organizations may turn to alternative international donors working against USAID’s Country Development Cooperation Strategy (CDCS) objectives. In addition to the increased stability and increased autonomy that it promises, a shift to less volatile local sources of funding could also increase the legitimacy of CSOs and insulate them from government targeting. In 2017, both foreign and domestic CSOs with links to the US were accused by government of participating in a “US interference network” (USAID, 2017). In addition to strengthening the resiliency of CSOs to civic space shocks indirectly by coaching organizations in new approaches to resource mobilization, Resiliency+ also aims to strengthen resiliency directly by guiding organizations through a process of strategic planning, providing them with training in USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 11 the necessary skills to implement their plans, and investing in leadership development. Evidence from academic research and received wisdom among practitioners suggests that budget transparency and capacity, public-facing narratives about organizational legitimacy (Coventry, 2016; Malena & Chhim, 2009), and within and across sector network density are especially important resiliency factors. Combining detailed financial data on CSO budgets with survey data on beneficiary satisfaction with CSO performance, Dang et al. (2019) find that capacity is strongly and positively associated with greater citizen satisfaction. CSO networks are a critical determinant of organizational resilience. In their cluster analysis of 135 CSOs in Cambodia, Suarez & Marshall (2014) find that membership in networks and an emphasis on external relations is one of the best predictors of local CSO capacity. Using data on citizen and politician social networks in the Philippines, Cruz et al. (2017) find that networks connect their members to material, legal, and political resources and facilitate the flow of information between constituents. Through these mechanisms, denser networks are likely to increase the ability of CSOs to share critical information about their environment, exchange the most effective strategies for managing environmental threats, and connect them with alternative sources of funding. This emphasis on networks also has the potential to amplify the effects of the intervention by producing positive spillovers to organizations outside of those directly receiving the programming—if the CSOs that receive Resiliency+ disseminate best practices through broad CSO and citizen networks, the programming will benefit a much larger set of actors than indicated in our technical design. Indeed, an important advantage of our big data approach to CSO networks is that it will allow us to measure potential positive spillovers to CSOs formally outside the programming. While there is substantial evidence that coaching and management training can be effective at improving outcomes at the individual and organizational level (Giorcelli, 2019, Banerjee et al., 2015; Blattman et al., 2016; McKenzie & Woodruff, 2013), there are currently no capacity strengthening models that have been rigorously proven to help organizations protect themselves against civic space threats, enable financial diversification, and achieve greater CSO resiliency and performance. Prior research suggests that individual coaching can effectively relieve household poverty by creating accountability for individuals trying to adopt good financial practices, sharing technical skills and knowledge, and increasing individual self-confidence (Banerjee et al., 2015). Similarly, research suggests that organizational training and coaching improves outcomes by helping firms to manage human resources, improve and expand marketing, and maintain accurate records, among other advantages. This project aims to implement and rigorously evaluate the impact of an intervention modeled on the Resiliency+ Framework and designed to counteract the two primary threats faced by Cambodian civil society. This unique framework leverages a nuanced understanding of how civil society organizations and actors operate, adapt, and thrive in response to multi-faceted and multi-dimensional changes to environmental dynamics with a focus on organizational adaptive capacities and resiliency in the face of closing civic space. In particular, the impact evaluation will examine the casual link between promoting financial diversification, building CSO networks and CSO performance in challenging environments. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 12 Drawing on the body of literature described above, the results framework for the LO-MTSR Activity is depicted below in Figure 2. The theoretical framework and the results framework form the basis for the hypothesis tested by the impact evaluation2. Figure 2. LO-MTSR Results Framework Drawing on this theoretical framework and the LO-MTSR results framework, the subsequent section lays out the research hypotheses that the impact evaluation is designed to test. 2 The LO-MTSR Activity also has a fourth objective tied directly to the IE, but outside of the results framework driven by the R+ Intervention. The fourth objective is to increase understanding of the effectiveness of the R+ Intervention on organizational capacity to move towards self￾reliance through comparison of treatment and control groups. The IE itself is the objective’s result. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 13 4.0 HYPOTHESES, DATA SOURCES, & INDICATORS RESEARCH HYPOTHESES (H) The design of the impact evaluation will allow us to answer a set of incredibly valuable and understudied questions that can help increase the resiliency of local civil society organizations operating in challenging political environments. At the CSO level, we expect that the intervention will: H1. Increase organizational, administrative, and financial capacity H2. Decrease in organization’s reliance on funding from USAID and other interventional donors, the RGC, and large international donors at odds with USAID’s CDCS objectives H3. Increase in the density of organization’s CSO network within and across sectors Each of these three primary hypothesis will be modified and expanded into multiple sub-hypothesis in the pre-analysis plan, submitted in Year 2. DATA SOURCES To test these hypotheses, the evaluation will use three primary sources of CSO level data to measure changes in the capacity of organizations, their funding sources and diversification, their ability to manage these funds, and their capacity to generate linkages with citizens and other organizations. This will allow us to precisely estimate the impact of the intervention on a variety of outcomes related to organizational self-reliance and resiliency. Our project data sources include (1) baseline and endline surveys of local civil society organizations, (2) baseline and endline budget data from these organizations, and (3) social media data on the size of each organization’s networks and their position within those networks both before and after the intervention. Baseline data collection will take place in the spring of 2020 prior to any R+ Intervention activity. After the R+ intervention is complete, an endline survey will be conducted to measure changes in the capacity of organizations, their funding diversification and levels of non-donor funding, their ability to manage these funds, and their capacity to generate linkages with citizens and other organizations. This will allow us to precisely estimate the impact of the intervention on a variety of outcomes related to organizational self￾reliance and resiliency. Each data source is described in detail below. 1. CSO MAIN SURVEY (N=240) Frequency: Baseline & Endline The primary survey of local organizations will draw heavily on two large, successful, and highly informative academic surveys of representative samples of NGOs operating in Cambodia and Uganda. The team will also draw heavily on questions used in two academic surveys of NGOs in Cambodia (Khieng, 2014; Suarez & Marshall, 2014). The surveys will include modules on organizational characteristics, networks, revenue￾generating activities, and employee perspectives of political and financial independence. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 14 The survey will be answered by three representatives of each organization. There are several reasons that it is important to collect this data from multiple individuals within each organization. First, individuals occupying different roles within an organization may have differential access to information about how it functions. For this reason, we hope to collect information from an individual in an executive position (ex. executive director) and an individual that is familiar with daily operations and finances (ex. full time staff member). Second, collecting information from multiple sources will allow for validation of the information that we collect. Discordant responses on important questions can be reconciled in follow-up interviews, potentially reducing measurement error dramatically. Finally, variation in responses to the same questions from different individuals within the same organization can serve as an interesting indicator itself. We expect that organizations with better internal communication to have more similar responses to the same questions about their organization’s structure and activities than those with poor internal communication. 2. CSO BUDGET SURVEY (N=120) Frequency: Baseline & Endline In addition to the main CSO survey, one respondent at each of the 120 CSOs will complete an online budget survey that collects provide basic, highly aggregated data on the organization’s budget. The survey will request organizations fill in figures including their sources of funding (international donors, membership fees, etc.), program costs, per diems, and wages. The Cloudburst team will draw on an in￾depth survey of NGOs’ budgets conducted by Dang et al (2019) and the application of Benford’s law to measure the accuracy of this reporting. This survey will allow us to measure several key features of CSO finances. First, this budget data will give us essential information on the funding diversification and levels of non-donor funding for each organization. This will serve as our measure on one of the primary outcomes of interest to the study. Second, this data will allow us to measure the financial professionalism and capacity of each organization. This outcome will be measured primarily by the ability of each organization to complete the budget form completely and accurately. As Dang et al. (2019) show, the misreporting of financial data by non-profits (both inaccuracies in and missing data from financial reports) is driven more frequently by a lack of capacity than an attempt to hide underperformance. We will deploy an accounting technique known as Benford’s law to detect inaccuracies in self-reported financial data by local organizations.3 3. SOCIAL MEDIA DATA SCRAPING (N=~120) Frequency: Continuous and aggregated to the monthly level As part of the baseline, we will also collect data on each organization’s web presence, including their social media account handles. In addition to providing basic data on the extent of each organization’s web presence, this data can be combined with social network analysis to provide a measure of each organization’s linkages with citizens, the for-profit private sector, and other civil society organizations. Social network analysis (SNA) utilizes data from a broader population (a network) to assess how connected certain units within the network are to other units. This approach is frequently applied to social 3 This technique compares the distribution of the leading digits of all numeric figures in an organization’s self-reported financial data with the theoretical Benford distribution, which holds that the observed distribution of leading digits in unmanipulated financial data will follow a logarithmic distribution. The greater the deviation of self-reported leading digits from this theoretical distribution, the greater the level of misreporting in the data. This technique has been extensively validated in the accounting literature, and tests have shown that individuals are unable to fabricate datasets that adhere to the Benford distribution even when explicitly instructed to do so. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 15 media data, which contains large volumes of information about online connections and interactions within the online population. Data on the number of social media followers and the number of interactions with an organization’s original online content can be used to quantify both the size of each organization’s online network and their “centrality” in that network. To quantify network size and centrality, SNA data expands outward through an organization’s network to measure the network size and centrality of other members of the organization’s network, giving an accurate measure of an organization’s online connectedness relative to other actors in the network. Furthermore, this information can be used to identify more nuanced data on each organization’s online networks, including their connectedness to international organizations vs local organizations, non-profit vs for-profit organizations, and private citizens. CSO IMPACTS CSOs receiving the R+ Intervention will see: H1: INCREASED ORGANIZATIONAL AND FINANCIAL CAPACITY Indicators: a. Increased staffing b. Increased geographic reach c. Increased ability to engage in advocacy d. Increased engagement in advocacy e. Less staff turnover f. Decreased within-organization variation between survey responses g. Increased budget reporting accuracy Data Sources: A. CSO survey B. Budget survey H2: DECREASE IN ORGANIZATIONS’ RELIANCE ON FUNDING FROM USAID AND OTHER INTERNATIONAL DONORS, THE RGC, AND LARGE INTERNATIONAL DONORS AT ODDS WITH USAID’S CDCS OBJECTIVES Indicators: a. Increase in number of funding sources pursued b. Increase in number of funding sources c. Increase in number of domestic funding sources pursued d. Increase in number of domestic funding sources e. Increase in revenue generating activities f. Increase in funding from revenue generating activities g. Increase (or no change) in amount of funding h. Decrease (or no change) in share of funding from USAID i. Decrease (or no change) in share of funding from Cambodian government j. Decrease (or no change) in share of funding from large international donors who are at odds with USAID’s CDC objectives USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 16 Data Sources: A. CSO survey B. Budget survey H3: INCREASE IN THE DENSITY OF CSO NETWORKS WITHIN AND ACROSS SECTORS Indicators: a. Number of formal partnerships b. Number of cross-sectoral formal partnerships c. Number of SM connections d. Number of SM interactions e. Density of SM networks Data Sources: A. CSO survey B. Social Media Scraping CONTROL VARIABLES Indicators: a. Director level of education b. Number of employees with college degree c. Size of organization budget d. Organization sector e. Organization type f. Organization activities g. Years since founding h. Number of employees i. Share of staff that is Cambodian j. Number of partner organizations k. Number of network organization memberships l. Number of previous capacity building interventions m. Organization governance (governing body, designated accountant, formal annual budget, external auditing) n. Experience receiving grants from international donors or foundations o. Experience receiving subawards from international donors or foundations p. Experience giving subawards to other organizations q. Experience using social enterprise or other revenue generating activities r. Dedicated employee in charge of M&E s. Use of external M&E consultant t. Share of budget spent on programming u. Self-reported relationship with government v. Gender of director Data Sources: A. CSO survey B. Budget Survey C. Social Media Scraping USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 17 5.0 RESEARCH & SURVEY METHODOLOGY The LO-MTSR Activity IE will be implemented as a randomized control trial (RCT). RCTs estimate the impact of an intervention by comparing outcomes for treated units against outcomes for a “counterfactual” group that was randomly selected to not receive the treatment. This technique gives implementers and donors a straightforward way to understand program effects and how outcomes would be different if the intervention had not taken place. The random assignment of treatment and control units is the most scientifically rigorous way to establish a causal relationship between an intervention and outcome; it is considered the “gold standard” in policy evaluation. Random assignment is also a normatively fair method for assigning CSOs to programming given that limited budget necessarily implies that the programming can only be provided to a modest number of CSOs. There have been very few rigorous evaluations of programming on CSOs, so this approach provides an enormous opportunity for USAID to learn crucial, rigorous lessons that might help civil society programming writ large. The IE is designed as a tiered intervention with two treatments cohorts, one receiving a low intensity treatment and one receiving a high intensity treatment. A third group serves as a control. Thirty CSOs will be assigned to the low intensity cohort, 30 CSOs will be assigned to the high intensity cohort, and 60 will be assigned to the control group. A description of the intervention for each cohort is described in Table 2, below. This design allows for the pooling of cohorts to evaluate the impact of receiving any R+ intervention against outcomes in the control group and for the comparison of outcomes for each cohort￾specific intervention against outcomes in the control group. In other words, the distinct impact of each tier of the Resiliency+ Framework will be measured. This tiered approach serves two purposes: First, it will provide insight into whether, and to what extent, CSO resilience can be improved with a light touch, or if it requires larger, more sustained programming and coaching; second, it will allow for a careful cost-benefit analysis of low- vs. high-intensity programming. To the extent USAID promotes CSOs across the developing world, this careful cost-benefit analysis will provide crucial insight for technical teams and help to identify how much of the effects detected by the evaluation were the result of the most resource-intensive components of the intervention. COHORT DESCRIPTION OF INTERVENTION Cohort 1: Low￾Intensity Resiliency+ Intervention (N=30) This cohort will participate in a Resiliency 101/ROCCS assessment workshop and receive four days of coaching and mentoring to create customized R+ plans, including access to an online repository of tools and resources to implement those plans. Organizations will also complete baseline and endline surveys and quarterly M&E data collection, supported by light-touch check-ins by R+ coaches. Cohort 2: High￾Intensity Resiliency+ Intervention (N=30) This cohort will participate in a Resiliency 101/ROCCS assessment workshop. Organizations will be assigned an R+ coach to provide ongoing mentoring support for execution of each organization’s respective resiliency plan. Organizations will also participate in group trainings and workshops, and will receive $3,000 for leadership development. Organizations will also complete baseline and endline surveys and quarterly M&E data collection. Cohort 3: Control Group (N=60) This cohort will receive no intervention. They will complete only baseline and endline surveys and M&E data collection and will participate in the final Peer Learning Workshop. TABLE 2. R+ COHORTS USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 18 The Cloudburst team will collect data from all three cohorts at two time points; baseline, prior to R+ implementation, and endline, after the R+ implementation has concluded. The evaluation data sources include (1) baseline and endline surveys of local civil society organizations, (2) baseline and endline budget data from these organizations, and (3) social media data on the size of each organization’s networks and their position within those networks both before and after the intervention. After the R+ intervention is complete, the CSO survey and the Budget survey will be conducted again with the same respondents to measure changes in the capacity of organizations, their funding diversification and levels of non-donor funding, their ability to manage these funds, and their capacity to generate linkages with citizens and other organizations. This will allow us to precisely estimate the impact of the intervention on a variety of outcomes related to organizational self-reliance and resiliency. CSO SAMPLING METHODOLOGY To secure a sufficiently large sample of local organizations, we will recruit at least 120 organizations through a call for applications distributed by our partner organizations in Cambodia. All applications will be screened based on basic criteria of organizational development, sector, and ability to commit staff time. 4 After filtering all CSOs who apply to the call to include only those that meet these basic criteria, at least 120 organizations will be selected to take part in a baseline survey in April 2020. After completing the baseline survey, organizations will be randomly selected using a statistical matching process, detailed below, to be part of the HI cohort (N=30), the LI Cohort (N=30), or the control group (N=60). The sampling process is illustrated in Figure 2, below. The inclusion of additional control organizations would likely increase the precision of the study considerably at a minimal cost. If more than 120 eligible CSOs apply to participate in the program, 60 organizations will be randomly selected to receive the intervention while the “surplus” organizations will serve as additional controls. As shown in Figure 3 in Section 6, Power Analysis, increasing the size of the control group can add valuable statistical power to this design; statistical power refers to our capacity to identify a program effect. Figure 3. CSO Sampling Methodology 4 USAID/Cambodia staff will have the opportunity to view the final list pre-randomization to remove any organizations that are considered inconsistent with USAID’s CDCS objectives. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 19 MATCHING Before assigning local organizations to intervention cohorts, we will use baseline data to match organizations on a range of characteristics that are correlated with important outcomes of interest, such as sector, organizational size and capacity, budget size and composition, and network size and centrality. We will then use statistical matching techniques such as coarsened exact matching to produce matched matched pairs that are as similar as possible to one another on these dimensions. 5 Once organizations are grouped into pairs of similar organizations, half of the pairs will be assigned to the high intensity cohort and half to the low intensity cohort. Finally, one CSO from each pair will be randomly assigned to the control group, which we will continue to gather data on but which will not receive R+ programming. Matching organizations on relevant characteristics and randomizing within these matched groups will allow us to compare outcomes across very similar organizations that range in the intensity of the intervention that they receive. This technique can dramatically reduce variance and increase statistical power. In short, this matching exercise will ensure that our R+ cohorts and control group are as similar as possible, thereby improving our capacity to measure program effects. LIMITATIONS While RCTs have a number of limitations when conducting research, they are the ideal tool for evaluating the effectiveness of programs and interventions. Like any approach to rigorous policy evaluation, they require a sufficiently large sample to make valid inferences about the effect of a given intervention. For this reason, the primary limitation of this design will be the relatively small sample size, especially when conducting the cost-benefit analysis. The chief constraint on our sample size is budgetary. For a more detailed discussion, see Section 6. 5 Absent initial matching, randomization could produce very different groups of R+ and control groups simply as a result of bad luck. This potential problem is particularly stark with sample sizes below 300. See Bruhn and McKenzie (2008) and Greevy et al. (2004). USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 20 6.0 POWER ANALYSIS This section presents power calculations for the LO-MTSR IE. Here, power refers to the capacity to detect an impact if one does exist; the associated power calculations indicate the sample size required for an evaluation to detect a given effect size. Because we do not yet have data on the mean or standard deviation of our outcome variables for CSOs in Cambodia, this power analysis is highly speculative. For this reason, we conduct power analysis using the fewest number of assumptions possible and aim to be conservative in the assumptions that we do make. We do not account for the design elements meant to increase statistical power, such as matching. For similar reasons, we also only present calculations for an analysis that pools both treatment arms (high and low intensity) and compare the effect of this single treatment group against the control group. After the collection of baseline data is complete, this analysis will be updated to incorporate the true parameter values from our sample, the additional design elements meant to reduce variance and boost power, and explore our ability to detect the independent effects of the high and low intensity treatment arms. The key assumptions in this power analysis are the mean and standard deviation of the outcome variable, the effect size of the intervention, and the alpha (see Box1 for a definition of these key terms). Using the DeclareDesign package in R (Blaire et al., 2018), we conduct power analysis on 500 simulations of our outcome variables drawn from a normal distribution (mean = 5, standard deviation = 1). In this IE, we know that 60 organizations will receive the treatment, but we cannot know the number of control organizations in advance. As described in Section 5, because the costs of surveying additional control organizations is very small, we can increase our statistical power by including as many of these organizations as possible given the number or organizations that apply and which meet the selection criteria. In the figures below, we assume a mean on our outcome of interest of 5 and a standard deviation of 1. For example, if the average number of funding sources for CSOs in our sample is the outcome of interest, this would mean that the CSOs in our sample have an average of 5 funding sources as baseline, and about 68% of the CSOs in the sample have between 4 and 6 sources of funding. Figure 1 shows the power of the IE design when varying the effect size of the intervention and the number of control organizations when using only endline data (we exclude baseline data in this analysis to minimize the number of assumptions being made). Assuming 60 treatment CSOs and the minimum number of control CSOs specified by the design (n=60), we can detect an effect size of between 0.5 and 0.6. In figure 2, we make an additional assumption of a 0.5 correlation between pre- and post-test outcomes in order to perform calculations that condition outcomes on their baseline values. As expected, this increases our power over a design only including endline values (show by the blue line for comparison). Assuming 60 treatment CSOs and the minimum number of control CSOs specified by the design (n=60), we can now detect an effect size of between 0.45 with 80% power. Recalling the discussion from Section 5, we also hypothesize that the high intensity intervention will have a large effect on our outcomes of interest when compared with the low intensity intervention. The analysis above suggests that such an analysis is likely to be statistically underpowered. Therefore, any investigation of the relative impact of these two levels of the intervention is likely to be speculative. However, even if it is not possible to detect a difference between the effect of high and low intensity interventions with statistical precision, a descriptive comparison of the size of the effect of each treatment arm will provide useful information about how much more effective the high intensity intervention was. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 21 Figure 4. Statistical Power Across Number of Control CSOs and Effect Size Figure 4: This figure shows how statistical power varies according to the number of control CSOs and the size of the population average treatment effect when using only endline data (while ignoring values from the baseline). This is not the most efficient approach, but it requires the smallest number of assumptions and represents a very conservative estimate of our statistical power. The shaded regions represent confidence intervals based on 500 simulations of the data, assuming 60 treatment organizations, an outcome mean of 5, a standard deviation of 1, and a significance level of 0.05. Figure 5. Statistical Power Across Effect Size and Estimator USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 22 Figure 5: This figure shows how statistical power varies according to the size of the population average treatment effect and the estimator being used. The red line indicates statistical power when conditioning on. The shaded regions represent confidence intervals based on 500 simulations of the data, assuming 60 treatment organizations, 60 control organizations, an outcome mean of 5, a standard deviation of 1, a correlation of 0.5 between baseline and endline outcome values for each organizations, and a significance level of 0.05. BOX 1: KEY POWER CALCULATION PARAMETERS The following describes the key parameters used to conduct the power analysis and sample size requirements for this impact evaluation. Standard Deviation is a numerical value that indicates how much outcome values vary across a sample of units. For example, if CSOs in our sample have an average of 5 sources of funding, but some in the sample have many fewer sources and some in the sample have many more sources, the variable measuring the number of funding sources will have a high standard deviation. An experiment where outcome variables have a low standard deviation will have more statistical power than an experiment where outcome variables have high standard deviation. Effect Size is the magnitude of impact that can be detected for a given sample. For example, if the CSOs in our sample have an average of 5 sources of funding at baseline, but CSOs in the treatment group have an average of 5.5 sources of funding after receiving the intervention, the effect size would be 0.5. An experiment where the treatment has a small effect size will have less statistical power than an experiment where the treatment has a large effect size. Power is the probability of detecting an impact if one has occurred. The power of a test is equal to 1 minus the probability of a type II error, ranging from 0 to 1. Popular levels of power are 0.8 and 0.9. High levels of power are more conservative and decrease the likelihood of a type II error. An impact evaluation has high power if there is a low risk of not detecting real program impacts, that is, of committing a type II error. Power calculations indicate the sample size required for an evaluation to detect a given minimum desired effect. Power calculations depend on parameters such as power (or the likelihood of Type II error), significance level, variance, and intra-cluster correlation of the outcome of interest. Alpha is the Type I error and is also referred to as the p-value in statistics. Generally speaking, this is the probability of concluding there was an impact when no impact actually exists. Typical values of alpha are 0.1, 0.05, and 0.01 with lower values indicating greater confidence in results (that is, less chance of concluding there is a program effect when there is none). USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 23 7.0 CONCERNS AND CONSIDERATIONS This section describes factors that present risks to the validity of the research design and evaluation methodology, as well as additional concerns raised by USAID/Cambodia and other stakeholders regarding the research. INSUFFICIENT SAMPLE SIZE There are a large number of indicators, and the size of sample required to detect impact will depend on the parameters of those indicators (i.e. mean and variability) along with the expected impact. Since these parameters cannot be known prior to the collection of baseline data, it is impossible to determine in advance the precise sample size required to detect program impact for every outcome of interest. However, we have tried to design the program so that under reasonable assumptions, we should be able to detect any meaningfully large effect of the intervention on the primary outcomes of interest. MATURATION Some of the impacts may take a much longer time period to materialize than what is currently available in terms of time between the baseline and end line data collection. To allow for this, the survey data will collect information on anticipated changes for key proximate outcomes in cases where enough time will not have passed to measure a meaningful change in a more distal development indicator. For example, questions about the number of grants or the number of revenue-generating activities or ventures (such as fundraising events, social enterprises, etc.) a CSO has planned will give us critical information on expected resources that will not yet be reflected in the CSO’s budget. SPILLOVER Given the R+ program’s focus on increasing CSOs’ networks, there is a risk that the treatment CSOs and control CSOs will interact, and the treatment CSOs will diffuse what they have learned from the R+ program to the control CSOs. While this spillover effect is ultimately good for building CSO resiliency overall, it would reduce our capacity to detect a program effect. However, if both treatment and control CSOs improve, we will be able to measure it, and our data on CSO networks will allow us to understand how the effects are diffusing. This capacity to measure positive spillovers is an important advantage of our data collection and evaluation efforts. NON-COMPLIANCE AND UNEVEN TAKEUP OF THE INTERVENTION In all policy evaluations, there is concern that citizens, CSO, firms or whatever the target of the policy do not comply with the policy design and objectives. For instance households that do not formally qualify for a conditional cash transfer might otherwise gain access to it, or households that do qualify can fail to actually gain access. This “noncompliance” makes it more difficult to estimate program effects. In our case, we expect targeted CSOs will agree to, and fully participate in, each component of the intervention. However, the team is aware that environmental forces may influence the degree to which all organizations are able to fully participate in the programming. The intervention team will collect detailed administrative data about organizational participation in trainings, hours of coaching received, etc. that will allow the evaluation team to control for these effects during analysis. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 24 ATTRITION This refers to a reduction in CSO sample size in the context of a panel due to an organization not participating in endline data collection, either because the organization ceased operations during the duration of the study, or the organization otherwise declines to participate in either the program or in endline data collection. To minimize the risk of organizations not participating in the endline survey, we will pay respondents $10 in exchange for completing the survey. In addition, we will work to maintain contact with both targeted and control organizations over the lifecycle of the project through check-ins and by encouraging these organizations to participate in the Expert Monitoring Survey associated with the machine learning component of the project, which provides respondents with interactive data on past and predicted future changes to civic space. HUMAN SUBJECTS PROTECTION Because the study participants are organizations, not individuals, the study does not require approval from an Institutional Review Board (IRB). However, all data collection activities will adhere to professional ethical stands for the treatment of human subjects. This includes ensuring that research produces do not pose more than a negligible risk to the participant subjects and to assure there adequate safeguards to protect subject’s rights, welfare, and dignity. Researchers will (1) inform subjects about the purpose, risks, and benefits of the study so they can make an informed decision about whether or not to participate in the search, and (2) protect the anonymity of subjects and the confidentiality of the data. The evaluation will confirm to the legal and other requirements governing research in Cambodia. To confirm to the Development Data Library Policy, the informed consent protocols for the study will include language highlighting the study’s plan for public data sharing after public identifiable information has been removed. ELECTIONS AND POLITICAL CLIMATE Cambodia’s commune elections in 2022 and national elections in 2023, and the increased surveillance and crackdowns on public gatherings that will likely accompany them, may influence NGO’s willingness to freely express opinions and engage with the project. This is especially true for NGOs operating in the provinces where local authorities tighten control prior to commune elections. To mitigate this risk, we will prioritize building open and trusting relationships with NGOs in the activity from the very beginning of our engagement. By the time of the commune elections, the majority of the project activity will have taken place, and we will have built a rapport with the NGOs in the program. We hope this relationship, as well as clear communication about how the information they share with us will be used for, will minimize this risk. A related challenge is potential tensions inherent in bringing service delivery and advocacy NGOs together to talk about restrictions on civic space and other political issues. The perception among service delivery CSOs that political issues are not relevant to their work and draws unnecessary government scrutiny to their organization may cause some reservations about participation in the Resiliency+ program (Malena & Chhim, 2009). For this reason, the intervention emphasizes the relevance of politics to all CSOs. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 25 8.0 TIMELINE AND TEAM COMPOSITION LO-MTSR proposes the following composition of the Impact Evaluation team: Impact Evaluation Manager Aleta Starosta (Cloudburst) Evaluation and Learning Specialist Erik Wibbels (DevLab@Duke) Civil Society Subject Matter Expert Jeremy Springman (DevLab@Duke) Machine Learning Subject Matter Expert Scott DeMarchi (DevLab@Duke) Methods Specialist Nancy Hite-Rubin (Cloudburst) Research Analysts Lucy Right (DevLab@Duke) Aidan Schneider (Cloudburst) The baseline data collection for both the CSO and budget surveys will occur after the participating CSOs have been selected, so it is expected to take place in April 2020. The implementation will begin in June 2020 and continue to November 2021. During the implementation, the IE team will conduct continuous social media surveying and web scraping. The endline data collection for the CSO and budget surveys, as well as a final social media survey and web scrape, is planned for November 2021. Table 3, on the next page, provides a detailed timeline for the LO-MTSR Activity. USAID.GOV LO-MSTR ACTIVITY: YEAR ONE WORK PLAN (OCT 2019–SEP 2020) | 26 Table 3. Project Timeline Activity Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Produce a Design Report Design Survey Instruments Survey Programming, Testing, and Translation Baseline CSO Survey Baseline Budget Survey Social Media Survey & Ongoing Web Scraping Analysis Matching Analysis for Organization Randomization of CSOs Into 3 Cohorts Baseline Data Cleaning and Analysis Baseline Report CLA Event Intervention Launch: Initial outreach to Resiliency 101 Workshop R+ Intervention, Low-Intensity Cohort R+ Intervention, High-Intensity Cohort Ongoing Monitoring and Evaluation Data Collection Pre-analysis Plan Revise Baseline Instruments for Endline Endline Survey Programming, Testing, and Translation Endline CSO Survey Endline Budget Survey Endline Social Media Scraping Endline Data Cleaning and Analysis Endline Report Intervention Close Out 2-pager of Evaluation Results Powerpoint of Evaluation Results Learning Summit for All CSOs Presentation of Findings to USAID/Cambodia Final Reporting 2019 2020 2021 2022 ACTIVITY 3: BASELINE ASSESSMENT, RANDOMIZATION, AND ANALYSES ACTIVITY 4: R+ INTERVENTION ACTIVITY 5: ENDLINE ANALYSIS ACTIVITY 6: CLOSEOUT, PEER LEARNING, AND DISSEMINATION USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 27 9.0 DELIVERABLES6 BASELINE & ENDLINE REPORT After the baseline surveys are completed, the data will be cleaned and analyzed. The LO-MTSR team will develop a baseline report that provides descriptive data on the participating CSOs, detailing the matching and randomization process and flagging any potential imbalances between the cohorts. The first draft will be completed by August 17, 2020, and the final version will be submitted by September 18, 2020 after USAID review. After the endline surveys are completed and the data is cleaned, a comprehensive endline report on the impact evaluation findings will be developed in June 2022. CLA WORKSHOP After baseline data collection, the team will convene for an internal half-day CLA workshop to discuss lessons learned during data collection and from initial analyses. The team will discuss potential revisions to the baseline instruments, as well as any recommended changes in the analysis design or endline data collection activities. A one-page summary of the CLA event will be produced for USAID’s internal use. PRE-ANALYSIS PLAN Prior to collecting endline data, the LO-MTSR team will develop a pre-analysis plan. The pre-analysis plan will guide the strategy for the endline anlaysis. It entails a substantial work-up of the baseline data and is used to determine the most rigorous, feasible, and appropriate analytic strategy for estimating program impacts. It involves revisiting and strengthening research hypotheses, comprehensive balance checks across treatment and control groups, construction and validation of intended outcome indicators, confirmation of study power for each intended outcome and sub-group analysis, identification of key confounders and selection biases, and a detailed specification of the analytic strategy and regression approaches that will be used to test impacts and causal mechanisms, as well as the robustness checks and sensitivity tests that will be used to gauge confidence in the results. DATA SET & CODEBOOK Following the baseline data collection, the LO-MTSR team will submit a fully cleaned and documented data set and codebook, with all identifiers removed, for the CSO and budget surveys. The CSO survey data will be submitted to USAID by September 1, 2020, and the budget survey data will be submitted by September 15, 2020. Endline datasets for both the CSO and budget surveys will be submitted in July 2022. All datasets that are developed under the LO-MTSR Activity will be cleaned of personally identifiable information and submitted to the Data Development Library to be used and shared at USAID’s discretion. DISSEMINATION To share the findings of the impact evaluation, the LO-MTSR team will develop a PowerPoint and a 2- page brief. These will provide summaries of the results from the LO-MTSR Activity. Results will be shared 6 The due dates for each deliverable are dependent on completion of all tasks leading up to them. These dates assume no extenuating circumstance delay program or evaluation activities. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 28 to USAID/Cambodia, participating CSOs, and other interested stakeholders. The PowerPoint and the 2- page brief will be developed in July 2022. If there are available resources in the later phases of the LO-MTSR Activity, the LO-MTSR team will consider additional dissemination activities, such as an in-country presentation to USAID/Cambodia and other local stakeholders, or a presentation of the baseline findings. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 29 10.0 REFERENCES Acemoglu, D., Naidu, S., Restrepo, P., & Robinson, J. A. (2019). Democracy does cause growth. Journal of Political Economy, 127(1), 47-100. Banerjee, A., Duflo, E., Goldberg, N., Karlan, D., Osei, R., Parienté, W., ... & Udry, C. (2015). A multifaceted program causes lasting progress for the very poor: Evidence from six countries. Science, 348(6236), 1260799. Blair, G., Cooper, J., Coppock, A., Humphreys, M., & Fultz, N. (2018). DeclareDesign. Software package for R, available at http://declaredesign. org. Blattman, C., Green, E. P., Jamison, J., Lehmann, M. C., & Annan, J. (2016). The returns to microenterprise support among the ultrapoor: A field experiment in postwar Uganda. American Economic Journal: Applied Economics, 8(2), 35-64. Bromley, P., Schofer, E., & Longhofer, W. (2019). Contentions over World Culture: The Rise of Legal Restrictions on Foreign Funding to NGOs, 1994–2015. Social Forces. Coventry, L. (2016). Civil Society in Cambodia: Challenges and Contestations. In The Handbook of Contemporary Cambodia (pp. 73-83). Routledge. Curley, M. (2018). Governing civil society in Cambodia: implications of the NGO law for the “Rule of Law”. Asian Studies Review, 42(2), 247-267. Dang, C. T., Burger, R., & Owens, T. (2019). Better Performing NGOs Do Report More Accurately: Evidence from Investigating Ugandan NGO Financial Accounts. Economic Development and Cultural Change. Giorcelli, M. (2019). The long-term effects of management and technology transfers. American Economic Review, 109(1), 121-52. Heiss, A. (2017). Amicable contempt: The strategic balance between dictators and international NGOs (Doctoral dissertation, Duke University). Khieng, S. (2014). Funding mobilization strategies of nongovernmental organizations in Cambodia. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 25(6), 1441-1464. Khieng, S. & Dahles, H, (2015). Commercialization in the Non-Profit Sector: The Emergence of Social Enterprise in Cambodia. Journal of Social Entrepreneurship. 6(2):1–26). Malena, C., & Chhim, K. (2009). Linking citizens and the state: An assessment of civil society contributions to good governance in Cambodia. McKenzie, D., & Woodruff, C. (2013). What are we learning from business training and entrepreneurship evaluations around the developing world?. The World Bank Research Observer, 29(1), 48-82. USAID.GOV LO-MTSR ACTIVITY: IE DESIGN REPORT | 30 Öjendal, J., & Sedara, K. (2006). Korob, Kaud, Klach: in search of agency in rural Cambodia. Journal of Southeast Asian Studies, 37(3), 507-526. Przeworski, A., Alvarez, R. M., Alvarez, M. E., Cheibub, J. A., & Limongi, F. (2000). Democracy and development: political institutions and well-being in the world, 1950-1990 (Vol. 3). Cambridge University Press. Savun, B., & Tirone, D. C. (2018). Foreign aid as a counterterrorism tool: More liberty, less terror?. Journal of Conflict Resolution, 62(8), 1607-1635. Savun, B., & Tirone, D. C. (2011). Foreign aid, democratization, and civil conflict: how does democracy aid affect civil conflict?. American Journal of Political Science, 55(2), 233-246 USAID (2017). Civil Society Organization Sustainability Index. United States Agency for International Development.