EVALUATION REPORT EX-POST PERFORMANCE EVALUATION OF USAID/RDMA SAPAN PROGRAM July 2017 This publication was produced for review by the United States Agency for International Development by Social Impact, Inc. JEAN-CAMILLE KOLLMORGEN/ SOCIAL IMPACT/FEBRUARY 2017 ii EX-POST PERFORMANCE EVALUATION OF USAID/RDMA SAPAN PROGRAM USAID/RDMA AID-486-I-14-00001/ AID-486-TO-16-00007 Submitted July 12, 2017 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Coeli Barry and Jean-Camille Kollmorgen on behalf of Social Impact, Inc. Contact: Social Impact, Inc. 2300 Clarendon Blvd., Suite 1000 Arlington, VA 22201 Sierra Frischknecht, Project Manager sfrischknecht@socialimpact.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii TABLE OF CONTENTS Tables and Figures ....................................................................................................................... iv Acronyms....................................................................................................................................... v Executive Summary....................................................................................................................... i I. Introduction........................................................................................................................... 1 1.1 Background..............................................................................................................................................................1 1.2 Evaluation Purpose and Audience......................................................................................................................1 1.3 Evaluation Questions............................................................................................................................................2 1.4 Team Composition................................................................................................................................................2 II. Evaluation Design.................................................................................................................. 3 2.1 Data Collection Methods.....................................................................................................................................3 2.2 Methodologies........................................................................................................................................................4 2.3 Sampling....................................................................................................................................................................5 2.4 Data Analysis...........................................................................................................................................................7 2.5 Design Limitations and Mitigation Strategies...................................................................................................7 III. Findings and Conclusions...................................................................................................... 8 Evaluation Question 1: ...............................................................................................................................................8 Evaluation Question 1.1: ...........................................................................................................................................21 Evaluation Question 1.2: ..........................................................................................................................................24 Evaluation Question 2: ..............................................................................................................................................26 Evaluation Question 2.1: ...........................................................................................................................................29 Evaluation Question 3: ..............................................................................................................................................31 Evaluation Question 3.1: ...........................................................................................................................................35 Evaluation Question 4: ..............................................................................................................................................38 Evaluation Question 5: ..............................................................................................................................................41 IV. Recommendations .............................................................................................................. 43 Annex A: Case Studies ............................................................................................................... 45 Annex B: Sapan Performance Indicator Analysis .................................................................... 51 Annex C: Full Listing of References and Reports Utilized ...................................................... 54 Annex D: Desk Review Bibliography......................................................................................... 58 Annex E: Data Collection Protocols.......................................................................................... 59 Annex F: Evaluation Design Matrix ........................................................................................... 80 Annex G: Full Listing of Persons Interviewed .......................................................................... 84 Annex H: Evaluation Statement of Work ................................................................................ 91 Annex I: Disclosure of any Conflicts of Interest..................................................................... 100 iv TABLES AND FIGURES Table 1: Bellwether Contributors .................................................................................................................................5 Table 2: Case Studies and Selection Criteria..............................................................................................................6 Table 3: Objective A Output Indicator Targets and Actuals..................................................................................9 Table 4: Objective A Outcome Indicator Targets and Actuals............................................................................10 Table 5: Objective B Output Indicator Targets and Actuals.................................................................................13 Table 6: Objective B Outcome Indicator Targets and Actuals.............................................................................13 Table 7: Sapan Partner Organizations.........................................................................................................................15 Table 8: Objective C Output Indicator Targets and Actuals................................................................................19 Table 9: Objective C Outcome Indicator Targets and Actuals............................................................................19 Table 10: CSO Capacity Scorecard Results..............................................................................................................39 Figure 1: Respondents by Type ......................................................................................................................................3 Figure 2: Respondents by Region...................................................................................................................................3 Figure 3: Respondents by Sex and Type ......................................................................................................................4 Figure 4: Objective A Result Achievement..................................................................................................................9 Figure 5: Objective B Results and Indicator Achievement.....................................................................................12 Figure 6: Objective C Results and Indicator Achievement....................................................................................18 Figure 7: “Book Republic” Bookstore, Home of Café Democracy.....................................................................28 Figure 8: Percentage of Seats Held by Women in National Parliament in Thailand........................................37 ACRONYMS AMW Association for Muslim Women CSNM Center for Civil Society and Non-Profit Management CSO Civil Society Organization DAI Development Alternatives, Inc. DOS Department of State ECT Election Commission of Thailand EQ Evaluation Question ER Expected Results ET Evaluation Team FCEM Foundation for Community Educational Media GI Group Interview IA Independent Agency IDI In-depth Interview IQC Indefinite Quantity Contract KII Key Informant Interview LGBT Lesbian, Gay, Bisexual, and Transgender MFA Ministry of Foreign Affairs MLC Media Learning Center NACC National Anti-Corruption Commission NGO Non-Governmental Organization NHRC National Human Rights Commission OAG Office of the Auditor General OBJ Objective OTI Office of Transition Initiatives PGI Provincial Governance Index PMP Performance Monitoring Plan RDMA Regional Development Mission for Asia RFP Request for Proposals RTG Royal Thai Government SGD Small Group Discussion SOW Statement of Work SWOT Strengths, Weakness, Opportunities, and Threats [Analysis] USAID United States Agency for International Development USG United States Government WDF Women’s Development Fund i EXECUTIVE SUMMARY PROJECT BACKGROUND The United States Agency for International Development’s Regional Development Mission for Asia (USAID/RDMA) awarded the contract for the five-year Citizen Engagement and Reconciliation Program, known post-award as Sapan, to Development Alternatives, Inc. (DAI). Sapan operated from March 2010 through September 2015 with a total budget of $29,343,629. The project had three main objectives: (A) strengthen the capacity of independent agencies (IAs) to conduct government oversight; (B) strengthen the capacity of civil society organizations (CSOs) and media to serve as checks and balances for political processes and policy; and (C) support civic peacebuilding efforts to diminish potential for escalation of violent conflict and radicalization in the Deep South. The Sapan theory of change posited that these objectives will lead to the program’s development goal of fostering constructive civil society engagement with the Royal Thai Government (RTG) to build consensus for democratic political processes and mitigate extremism. Sapan engaged core partners in Chiang Mai, Phitsanulok, Khon Kaen, Ubon Ratchathani, Nakorn Sri Thammarat, Pattani, Yala, and Bangkok. EVALUATION PURPOSE AND QUESTIONS The purpose of this ex-post evaluation was to assess the effectiveness and sustainability of Sapan one year after its conclusion in September 2015. This report answers the following questions posed by USAID/RDMA: 1. To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? In addition to the overall results, the evaluation must also assess: 1.1.Did Sapan’s theory of change, especially the roles of the independent agencies, hold true throughout the course of Sapan’s life? 1.2.How did the changes in the initial assumptions that formulated the theory of change, if any, affect Sapan’s ability to meet the objectives? 2. What interventions were more successful and/or had a greater contribution to Sapan’s objectives? 2.1. What, if any, management shifts adopted in response to the deteriorating political environment proved effective in contributing to Sapan’s objectives? 3. What are observable positive/negative changes in the capacity of the targeted groups, i.e. IAs, CSOs, media, and civic peacebuilding leaders, as a result of Sapan? What are the factors that helped or hindered such changes? 3.1. Were there observable differences in the involvement of, or impact on, men and women?1 4. If any, what and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? 5. What policy changes, during or after the life of Sapan, are observable as a result of Sapan? 1 Sub-question proposed by the evaluation team to ensure gender dimensions are appropriately considered. The team originally proposed a similar sub-question for EQ4. However, the team did not discover gender differences related to sustainability and thus did not present that sub-question. ii EVALUATION METHODS AND LIMITATIONS The evaluation team (ET) consisted of two core team members, research support staff, and home office evaluation management staff. Field work in Thailand lasted approximately six weeks (January 6 to February 21, 2017). Evaluation Approach The evaluation utilized a mixed-methods approach that included several qualitative data collection methods and an analysis of quantitative monitoring data: • Desk Study of project-related documents provided by USAID and DAI as well as other secondary reports and analyses as relevant to Sapan’s objectives and the evaluation questions. • In-depth Interviews (IDIs) and Group Interviews (GI) with Key Informants using semi-structured interview protocols tailored to respondent groups. Key informant types included representatives from USAID, DAI, local Sapan partners, IAs, and independent policy experts known as “bellwethers.” The ET conducted 37 IDIs/GIs, and spoke with 23 of 25 Sapan partners. • Small Group Discussions (SGDs) with program beneficiaries using semi-structured discussion protocols. The ET conducted eight SGDs. To augment the data collection methods described above, the ET implemented additional evaluation approaches to provide strategic focus on certain evaluation questions: • Case Study Methodology. The ET conducted a total of six case studies of organizations that either received capacity-building training or were used to build the capacity of other CSOs. The ET used data from the desk review, IDIs/GIs, and SGDs to elucidate the specific contextual and programmatic aspects of these organizations, and the outcomes of their capacity-building experience. • Bellwether Methodology. The ET conducted IDIs with influential actors or thought leaders (“bellwethers”) who were external to Sapan. 2 The ET queried bellwethers on public policy and social changes occurring during Sapan’s implementation. The evaluation incorporated a gender-responsive design that included interviews with men and women. SGDs were sex-segregated when possible and appropriate based on the number of SGDs and availability of respondents. The ET included gender-specific questions in interview guides. Sampling The ET applied both purposive sampling and snowball sampling to identify respondents for the IDIs/GIs. The ET used purposive and convenience sampling to select SGD participants. The ET purposively selected the six case study CSOs to reflect a variety of organizations based on selected criteria.3 Data Analysis For primary data, the ET first categorized raw notes and findings from secondary data and document reviews into an aggregated Excel spreadsheet organized by evaluation question. The ET analyzed the content of the aggregated data for recurring themes both within and across different respondent groups, sex, and regions. The ET used a tally sheet to quantify themes identified, and to help determine major 2 Blair, E, Evaluating an Issue’s Position on the Policy Agenda: The Bellwether Methodology. The Evaluation Exchange Volume XIII, Number 1 and 2, (Spring 2007). 3 (1) Geographic diversity; (2) Sapan objectives or expected result areas; (3) Capacity Scorecard performance (high, medium, and low) for core partners; (4) capacity-building involvement (giving vs. receiving); (5) Sapan target beneficiaries (including media, youth-focused, and women-led groups); and (6) logistical feasibility. iii findings. Finally, the ET triangulated the data from the different sources, noting where data sources agreed and disagreed. The ET used a gender lens in data collection and analysis. Design Limitations and Strengths Because random sampling was not used, there is a risk of selection bias among the respondents. Because the evaluation was conducted 16 months after Sapan ended in September 2015, respondent answers may also be subject to recall bias, but as this was an ex-post evaluation, the remaining perceptions of the respondents are important. The ET worked through a translator, so there may have been a translation bias. However, the Thai-English capability of the Team Leader may have mitigated this challenge. Respondent attrition in the case of youth participants was a challenge. Because of its small sample size, the evaluation is limited in its external validity. Because there was no defined control group included as part of the evaluation, it is difficult to attribute outcomes to the Sapan program alone. To compensate for this limitation, the ET used data triangulation and the inclusion of bellwether voices to ground-truth the perceptions of program stakeholders. At the same time, the ET interviewed over 90 percent of Sapan partners. The ET members were present during all interviews and conducted data analysis together, strengthening the evaluation’s inter-rater reliability. CONCLUSIONS Evaluation Question 1: To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? • Among the three objectives, Sapan was most effective at contributing to Objective B (CSO capacity building), having increased the organizational and technical capacities of CSO and media partner organizations through effective training and mentoring interventions. Providing checks and balances on the government was something only some Sapan partners had been engaged in prior to the start of Sapan in 2010. Project documentation noted organizations showing signs of increased technical capacities. The ET found that most organizations could not actively provide checks and balances on the government after the 2014 coup. • Sapan made some progress toward Objective C (peacebuilding and reducing violent conflict in the Deep South), though more so in terms of increasing the connectivity of organizations in the Deep South to those in other parts of the country, rather than direct peacebuilding. • Sapan was least effective at contributing to Objective A (strengthening IAs) for both programmatic and political reasons. • Sapan’s relationship and network building interventions contributed to progress to all three objectives. Evaluation Question 1.1: Did Sapan’s theory of change, especially the roles of the independent agencies (IAs), hold true throughout the course of Sapan’s life? - Hypotheses related to the role of IAs, media, and university partners changed over time. Assumptions about the neutrality and motivation of IAs proved to be untrue. The project also adapted to changes in assumptions regarding the role universities and local media organizations could play in Sapan programming. - Sapan’s logic of working with both government and civil society was valid, but the project could have better chosen the number and type/level of government and civil society actors with whom to work. - Sapan’s three objectives were intended to be mutually reinforcing as well as necessary and sufficient to achieve Sapan’s larger objective of fostering constructive civil society engagement with government. The theory of change linking Objectives A and B as mutually reinforcing iv elements was valid for Sapan, whereas Objective C as a standalone result was not; Objective C could have been wholly subsumed under Objective B. - The theory of change related to how Sapan interventions in the Deep South were to lead to a reduction in violent conflict proved questionable as Sapan under Objective C did not focus on direct peacebuilding interventions. There was a disconnect between the indicators chosen to measure Objective C’s higher-level outcome (related to youth economic and social opportunities) and the interventions Sapan actually implemented. Evaluation Question 1.2: How did the changes in the initial assumptions that formulated the theory of change, if any, affect Sapan’s ability to meet the objectives? - Incorrect assumptions related to IAs and the political environment had a negative effect on project achievement. - Sapan’s assumptions regarding the capacity of CSOs in Thailand were incorrect. Sapan effectively adapted its approach to focus on organizational capacity-building efforts, yet this pivoting also meant that Sapan was limited in its ability to do higher-level governance work given the current capacities and comforts of partner organizations. Evaluation Question 2: What interventions were more successful and/or had a greater contribution to Sapan’s objectives? • Sapan’s organizational capacity building interventions were successful. • Specific interventions that could be applied were more useful and contributed to Sapan’s objectives. • Some successful interventions were not part of the original design, but came up in the middle of implementation or were opportunities that Sapan seized. • Interventions for building relationships between and among CSOs and other actors, either through formal networks or relationship building or exchange, e.g., things like community forums, proved successful in contributing to all objectives, but especially Objective B and Objective C. Relationship building also had particular significance for connecting organizations in the Deep South to those in the rest of the country. Evaluation Question 2.1: What, if any, management shifts adopted in response to the deteriorating political environment proved effective in contributing to Sapan’s objectives? - Sapan implemented some management shifts in response to the deteriorating political environment, ranging from the discontinuation of interventions under Objective A to more cautionary work with partners, though it is unclear what effect, if any, these shifts had on attaining program results. - Though some interventions of Sapan partners were affected by the political environment, many Sapan interventions were not affected and no management shifts were made. Evaluation Question 3: What are observable positive/negative changes in the capacity of the targeted groups, i.e. IAs, CSOs, media, and civic peacebuilding leaders, as a result of Sapan? • Sapan increased the organizational and technical capacities of targeted groups, as well as their understanding of democracy and citizen engagement. • The tangible skills participants acquired proved transferable to settings beyond governance and democracy work. • Respondents, especially women, reported an increased sense of empowerment and confidence as a result of participating in Sapan interventions. v Evaluation Question 3.1: Were there observable differences in the involvement of, or impact on, men and women? - USAID’s Policy on Gender Equality and Female Empowerment was communicated and institutionalized by partners at the output level, through recognizing the need to include both men and women in Sapan interventions. - There was an increase in women’s leadership in informal settings that respondents reported as due to Sapan. An increase in women’s leadership in formal roles was reported but not confirmed. - The ability of women in the Deep South to take part in Sapan-supported interventions was shaped by the sociocultural character of the Deep South: Muslim social norms affected the movement of women, especially unaccompanied, unmarried women. Evaluation Question 4: What and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? • Although many interventions ceased after the project, capacities related to personal empowerment, organizational capacities and technical skills persisted to varying degrees. Evaluation Question 5: What policy changes, during or after the life of Sapan, are observable as a result of Sapan? • Although public policy change was not a core objective of Sapan, CSOs were meant to engage public policy in their “checks and balance” role. There was a major policy intervention consisting of a national-level advocacy campaign involving five core partners that focused on policies associated with the Women’s Development Fund (WDF). • Policy change action, when it occurred, was in response to opportunities seized by both local partners and DAI. RECOMMENDATIONS Evaluation Questions 1, 1.1, 1.2: To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? A. In future design of similar projects, USAID should consider working with the RTG but focus on local-level administration in addition to central government (keeping in mind the level of centralization or decentralization in the country of implementation). These projects should be designed collaboratively with the government entity to facilitate buy-in and shared understanding of objectives. B. Within Thailand, other USAID projects and technical sectors may consider using local media outlets and universities as intermediaries with CSOs and communities. C. Future civil society engagement programs may consider engaging other parts of civil society, such as the private sector, and not just CSOs. Evaluation Questions 2, 2.1: What interventions were more successful and/or had a greater contribution to Sapan’s objectives? A. USAID might consider the use of contracts, rather than cooperative agreements, in sensitive environments to exert more control over program activities, though only after evaluating the pros and cons of different mechanisms and making sure it is a good fit. This type of contract is best vi paired with a flexible and consultative management style. Flexible management should provide space for local partners to propose new interventions not included in the original program design. B. Considering the positive effect of Sapan’s work with organizations across the Thai political spectrum, future USAID governance projects in divisive or polarized environments should work with both sides of the political spectrum and emphasize objectivity. C. USAID projects that introduce tools (governance or otherwise) to CSOs or other local partners should focus on tool application rather than lecture, and should incorporate mentoring and feedback from the implementing partner. D. USAID projects that incorporate organizational capacity-building work with local organizations should use a capacity-building scorecard tailored to the country context as a framework for encouraging and assessing progress. E. USAID projects that include any type of capacity building (organizational or technical) should build in opportunities for partners to network, build relationships, and learn from one another. Evaluation Question 3: What are observable positive/negative changes in the capacity of the targeted groups, i.e. IAs, CSOs, media, and civic peacebuilding leaders, as a result of Sapan? A. In future projects, capacity building that strengthens the professionalism of CSOs should be promoted, since professional skills learned and put into practice are highly valued by these organizations. In particular, planning for the transfer of organizational leadership could help prevent the overdependence on founders and leaders that hurts many CSOs. B. USAID and implementing partners should consider identifying trusted local intermediaries (e.g., Sapan’s use of universities) to help with the delivery of trainings; these can be vital to bridging resources between Sapan and the CSOs and their beneficiaries. C. USAID and implementing partners should develop or utilize preexisting technical tools, such as community scorecards, that reflect the needs of the organization and community and are suitable to the skillsets of the intended users. D. Because Sapan partners used governance tools effectively in a range of sectors (e.g., health, social welfare), USAID programs in other sectors may consider using public discussion forums, community scorecards, Strengths, Weakness, Opportunities, and Threats (SWOT) analysis and other easy-to-learn tools to tackle governance-related obstacles or corruption in technical sectors. E. Future USAID capacity building interventions and trainings should focus on developing participants’ soft skills such as leadership, confidence building, and public speaking. These skills are especially relevant for women-led organizations. F. Bellwether recommendations for further work to be done in Thailand include a focus on civil society strengthening programming in Thailand to support critical thinking and civic mindsets and to prepare the younger generation for when civil society is less threatened in the country. Evaluation Question 4: What and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? A. To increase the sustainability of technical skills, USAID and implementing partners should allow sufficient time for local organizations to practice and implement newly learned skills so they can internalize them sufficiently to be able to use them. B. USAID and implementing partners should develop interventions and skills trainings focusing on issues of relevance to the community's wellbeing. vii C. USAID should identify partners with high personal or organizational motivation to sustain their work, even at a reduced level, when donor assistance is no longer available. D. USAID and implementing partners should introduce fundraising skills to capacity building interventions as early as possible, because continuity of work depends upon the organization's ability to get external support. Evaluation Question 5: What policy changes, during or after the life of Sapan, are observable as a result of Sapan? A. If public policy change is to be an objective of a USAID activity or project, the design needs to clearly identify what level of policy is to be affected, what is highest priority for type of change, and who are the most suitable project partners. B. If USAID wants implementing partners to make an impact on public policy, stakeholder mapping skills, advocacy skills, and planning for campaigns to influence policymakers and processes need to be introduced to partners early on and supported for an extended period of time. 1 I. INTRODUCTION 1.1 BACKGROUND A middle-income country, Thailand has experienced various military interludes in its parliamentary democracy, including coups in 2006 and 2014. A constitutional referendum occurred on August 7, 2016, and a general election is anticipated for late 2017. In the context of tumultuous national governance, conflict around the separatist insurgency continues in Thailand’s Muslim-majority Deep South region, though violence levels have declined in recent years and the current military government has mobilized civilian volunteer counter-insurgency units in the region.4 Civil society—including the media—has developed and evolved with Thailand’s democracy. Civil society organizations (CSOs) have engaged with the Royal Thai Government (RTG) on various political, economic, and service delivery issues. Service delivery CSOs tend to be stronger than other types of CSOs and have undertaken advocacy related to their fields, such as health governance. However, civil society is weaker in Thailand than might be expected for a middle-income country and has not been able to adequately fulfill its watchdog role over government and public policy. In the 2015 CSO Sustainability in Asia report by the United States Agency for International Development (USAID), Thailand was rated a 4.7 out of 7 on the CSO Sustainability Index. Though this rating places Thailand within the “sustainability evolving” category, the country is only a few points from being categorized as a “sustainability impeded” country.5 USAID awarded a contract for a five-year Citizen Engagement and Reconciliation Program, known post￾award as Sapan, to Development Alternatives, Inc. (DAI). Sapan operated with a period of performance from March 2010 through September 2015, with a total budget of $29,343,629. The project had three main objectives: (A) strengthen the capacity of independent agencies (IAs) to conduct government oversight; (B) strengthen the capacity of CSOs and media to serve as checks and balances for political processes and policy; and (C) support civic peacebuilding efforts to diminish potential for escalation of violent conflict and radicalization in the Deep South. The Sapan theory of change posited that these objectives would lead to the program’s development goal of fostering constructive civil society engagement with the RTG to build consensus for democratic political processes and mitigate extremism. Sapan covered all regions across Thailand, with its core partners located in Chiang Mai, Phitsanulok, Khon Kaen, Ubon Ratchathani, Nakhon Sri Thammarat, Pattani, Yala, and Bangkok. 1.2 EVALUATION PURPOSE AND AUDIENCE The purpose of this ex-post evaluation was to assess the effectiveness and sustainability of Sapan one year after its September 2015 conclusion. It focused on evaluating the extent to which the program contributed to its stated goal, objectives, and selected expected results (ERs); examined the validity of the project’s critical assumptions and theory of change; and identified any sustainable changes in Thai civil society and/or governance to which the project contributed. The primary audience and key user for this evaluation is the USAID Regional Development Mission for Asia (RDMA), specifically the Office of Economic Growth and Vulnerable Populations. Secondary audiences include other technical offices at USAID/RDMA and other USAID missions, as well as other interested parties (e.g., other donors and academics) following USAID’s approved release of the report. 4 International Crisis Group. Southern Thailand’s Peace Dialogue: No Traction. Crisis Group Asia Briefing 148 (2016). 5 USAID. CSO Sustainability Index for Asia (2015). CSO Sustainability Index is rated on a scale from 1 to 7, denoting with a 1–3 score denoting an advanced civil society sector (“sustainability enhanced”), a 3.1–5 score as a medium-developed civil society sector (“sustainability evolving”), and a 5.1–7 score as a fragile civil society sector (“sustainability impeded”). 2 1.3 EVALUATION QUESTIONS USADI/RDMA tasked Social Impact with answering the following five evaluation questions (EQ). 1. To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? In addition to the overall results, the evaluation must also assess: 1.1.Did Sapan’s theory of change, especially the roles of the independent agencies, hold true throughout the course of Sapan’s life? 1.2.How did the changes in the initial assumptions that formulated the theory of change, if any, affect Sapan’s ability to meet the objectives? 2. What interventions were more successful and/or had a greater contribution to Sapan’s objectives? 2.1. What, if any, management shifts adopted in response to the deteriorating political environment proved effective in contributing to Sapan’s objectives? 3. What are observable positive/negative changes in the capacity of the targeted groups, i.e., IAs, CSOs, media, and civic peacebuilding leaders, as a result of Sapan? What are the factors that helped or hindered such changes? 3.1. Were there observable differences in the involvement of, or impact on, men and women?6 4. If any, what and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? 5. What policy changes, during or after the life of Sapan, are observable as a result of Sapan? 1.4 TEAM COMPOSITION The Social Impact evaluation team (ET) consisted of two core team members, research and logistical support staff, and home office evaluation management staff. Dr. Coeli Barry served as Senior Team Leader and as the Senior Governance Specialist. She has over 20 years of experience in governance, civil society strengthening, and democracy with a strong focus in Southeast Asia. Jean-Camille Kollmorgen served as the mid-level Monitoring & Evaluation Specialist, providing the team with expertise in qualitative and quantitative analysis techniques, sampling strategies, and other technical elements of the evaluation design and implementation. Amanda Stek completed an initial desk review in support of the core team. The team was supported throughout data collection by an in-country logistician, Onuma Chaisumrej, and an interpreter, Tutiya Buabuttra. 6 Sub-question proposed by the ET to ensure gender dimensions are appropriately considered. The ET originally proposed a similar sub-question for EQ4. However, the ET did not discover gender differences related to sustainability and thus did not present that sub-question. 3 II. EVALUATION DESIGN 2.1 DATA COLLECTION METHODS Field work in Thailand lasted approximately six weeks (January 6 to February 21, 2017), including data collection in all eight Sapan core partner locations7 plus an in briefing at the start of field work and a presentation of preliminary findings to USAID/RDMA after field work. The evaluation utilized a mixed-methods approach that included several qualitative data collection methods and an analysis of quantitative monitoring data. The ET obtained informed consent from all respondents prior to primary qualitative data collection. Data collection methods included: Desk Study. Prior to and during in-country fieldwork, the ET completed a desk review of project-related documents provided by USAID and DAI, including: annual and final work plans and narrative reports; the contract and modifications; the performance monitoring and evaluation plan and tools; data quality assessment reports; periodic progress reports and success stories; and program evaluation and audit report. The team additionally reviewed other non-project-related secondary reports and analyses as relevant to Sapan’s objectives and the evaluation questions. Desk review documents provided contextual background for the evaluation for work planning and tool development purposes and also served as a data source. A list of reviewed documents is included in Annex C. In-depth Interviews (IDIs) and Group Interviews (GI) with Key Informants. The ET interviewed key informants either individually (IDI) or in small groups (GI), using semi-structured interview protocols tailored to respondent groups. Key informant types included representatives from USAID, DAI, local Sapan partners, IAs, and independent policy experts known as “bellwethers” (see Methodologies description below). The ET conducted a total of 37 IDIs and GIs, and spoke with 23 out of 25 Sapan partners. Small Group Discussions (SGDs). The ET used semi-structured discussion protocols to interview small groups of program beneficiaries (e.g. students who participated in Sapan youth￾development interventions) and community members/local leaders who were involved in or affected by interventions (e.g., participants in a media intervention). The ET worked with Sapan partners to recruit SGD participants and specifically requested a mix of male and female participants. Groups made up of men and women were sex￾segregated if two or more individuals of the same sex were present in the group. To ensure confidentiality and safety for participants, especially women, the ET conducted SGDs in a private room at Sapan partner offices and gave participants a transportation stipend to allow them to take a safe transportation method of their choosing during daylight hours. The ET conducted a total of eight SGDs, of which two were sex segregated. 7 Sapan activities covered all regions in Thailand, and core partners are headquartered in eight provinces: Chiang Mai (North), Khon Kaen and Ubon Ratchathani (Northeast), Bangkok and Phitsanulok (Central), Nakhon Sri Thammarat (South), and Pattani and Yala (Deep South). Partners may also operate in nearby provinces, either directly or indirectly through their networks. Figure 1: Respondents by Type Figure 2: Respondents by Region 4 For four SGDs or GIs that were not segregated, but during which respondents of a certain sex or authority status dominated the conversation, the ET conducted follow-up interviews with individuals who were not as vocal during the group discussion. Two SGDs (not sex-segregated) were conducted with youth over the age of 18 years. Respondent consent forms and data collection instruments for IDIs/GIs and SGDs are included in Annex E. An evaluation design matrix can be found in Annex F. In total, the evaluation included 45 combined IDIs/GIs/SGDs, encompassing 127 total respondents (77 women and 50 men). The evaluation included more female than male respondents, in part because of the inclusion of women-led organizations whose beneficiaries are exclusively women. 2.2 METHODOLOGIES To augment the data collection methods described above, the ET implemented additional evaluation approaches to provide strategic focus on certain evaluation questions: Case Study Methodology. The ET utilized a case study approach to address EQs 3 and 4, related to the efficacy and sustainability of Sapan’s capacity-building efforts. The ET conducted a total of six case studies of organizations that either received capacity-building training, or were used to build the capacity of other CSOs. The ET utilized data from the desk review, IDIs/GIs, and SGDs to elucidate the specific contextual and programmatic aspects of these organizations, and the outcomes of their capacity-building experience. Bellwether Methodology. As part of addressing EQ 5, and also as a means of validating Sapan stakeholder claims on other EQs, the ET conducted IDIs with influential actors or thought leaders (“bellwethers”) who were external to Sapan. 8 The ET queried bellwethers on public policy and social changes occurring during Sapan’s implementation, without directly referencing Sapan’s efforts, and triangulated these responses with those heard from project stakeholders. Table 1 describes the bellwether respondents. Gender-responsive Design. The evaluation incorporated a gender-responsive design. This included understanding how Sapan’s gendered frame affected outcomes, and sampling and interviewing both men and women. The ET was attuned to cultural norms such as social perceptions of women in positions of leadership and to what extent women-led organizations focus on conventional, gendered issues and responsibilities such as caring for children. These issues were further explored through key informant interviews (KIIs) and SGDs. SGDs were segregated by sex when possible and appropriate based on the number of SGDs and availability of respondents. The ET included gender-specific questions in interview guides with all relevant stakeholder groups to evaluate the potential differential impacts of Sapan on males and females. Both data collection methodologies (KIIs and SGDs) considered the privacy and confidentiality of respondents and included gender-responsive questions. The ET was also diligent in recording findings that indicated situations in which gender dynamics were not considered a defining component in the delivery of Sapan interventions. IDIs with bellwethers sharpened the ET's gender 8 Blair, E, Evaluating an Issue’s Position on the Policy Agenda: The Bellwether Methodology. The Evaluation Exchange Volume XIII, Number 1 and 2 (Spring 2007). Figure 3: Respondents by Sex and Type 5 analysis. The ET further triangulated findings through secondary sources, including a "Needs Assessment for Women’s Participation in Local Governance in Thailand." The assessment was commissioned by DAI, and drawing on it allowed the ET to bring out a thorough and nuanced understanding of gender issues and governance, democracy, and conflict reduction in Thailand. The team was guided by a Gender Advisor who reviewed the deliverables and data collection tools to ensure gender issues were appropriately incorporated in the evaluation design and analysis. Table 1: Bellwether Contributors Region Respondent Sex Affiliation Selection Rationale Bangkok/National Male Technical Specialist, Political Science & Lecturer Journalist and expert in media/civil society Bangkok/South Male Independent Security Analyst Expert on conflict and security in the southern provinces Northeast/ National Female & Male Mahasarakham University Expert in governance and civil society/social movements in Northeast Thailand North Male Chiang Mai University Expert in Northern Thailand civil society and Southern Thailand socio￾cultural/gender issues Deep South/National Female Prince of Songkla University Expert in gender and lesbian, gay, bisexual, and transgender (LGBT) in Southern Thailand 2.3 SAMPLING The ET applied both purposive sampling and snowball sampling to identify respondents for the IDIs and GIs. Respondents were chosen based on their involvement with Sapan or knowledge of Sapan’s sociopolitical operating environment. Some of these respondents, in turn, suggested other respondents who could knowledgeably comment on Sapan. With the assistance of Sapan partners, the ET used purposive and convenience sampling to select SGD participants. For all qualitative methods, the ET requested to interview both male and female respondents, as well as youth over the age of 18, recognizing that men and women of varied ages may be affected by program interventions differently. The ET selected the case study CSOs to reflect a variety of organizations based on the following criteria: 1. Geographic diversity 2. Sapan objectives or expected result areas 3. CSO Capacity Scorecard performance (high, medium, and low) for core partners 4. Capacity-building involvement (giving versus receiving) 5. Sapan target beneficiaries (including media, youth-focused, and women-led groups) 6. Logistical feasibility Table 2 below shows the proposed case study CSOs and how they map to the selection criteria. 6 Table 2: Case Studies and Selection Criteria Selection Criteria Partner Location Region Project Objective (Obj.) Designation or Affiliation Targeted Beneficiaries CSO Capacity Scorecard9 Foundation for Community Educational Media (FCEM)/Prachatai Bangkok Central Strengthen Civil Society (Obj. B) Core Partner; Capacity￾building CSO Media High Sri Song Kwai Women's Group Phitsanulok Central Strengthen Civil Society (Obj. B) Core Partner Gender Low Media for Happiness Ubon Ratchathani Northeast Strengthen Civil Society (Obj. B) Core Partner Media High CSNM/Khon Kaen University Khon Kaen Northeast Strengthen Civil Society (Obj. B) Capacity-building CSO/University N/A N/A10 Association for Muslim Women (AMW) Nakhon Si Thammarat South Peace Building in Deep South (Obj. C) Core Partner Gender Medium People’s College Pattani Deep South Strengthen Civil Society (Obj. B); Peace Building in Deep South (Obj. C) Capacity-building CSO Youth N/A 9 Ranking classifications were designated by the ET, not Sapan. Scores of 13–15 = High; 10–12 = Medium; and <10 = Low, based on the CSO Scorecard Criteria, which includes 15 elements ranging from legal registration and financial policies to fundraising, gender and diversity, and organizational structure. 10 N/A = Not applicable. Denotes that CSNM and People’s College did not participate in the capacity scorecard process as part of Sapan. These organizations were formed through the assistance of Sapan toward the end of the contract period and did not exist early enough to participate in CSO Capacity Scorecard assessments. However, these organizations provided capacity-building support to other local actors. 7 2.4 DATA ANALYSIS Prior to fieldwork, the ET extracted relevant qualitative and quantitative information from the desk review documents and organized them into a summary Excel spreadsheet organized by question. For qualitative primary data, the ET conducted iterative data analysis throughout data collection, meeting regularly to triangulate new data and discuss emerging findings. Raw notes were first categorized into an aggregated Excel spreadsheet. The ET then analyzed the content of the aggregated data for recurring themes both within and across different respondent groups, sexes, and regions. The ET used a tally sheet in Excel to quantify themes identified and to help determine major findings. Finally, the ET triangulated the data from the various sources, noting where data sources agreed and disagreed. 2.5 DESIGN LIMITATIONS AND MITIGATION STRATEGIES Because random sampling was not used, there is the possibility of selection bias on the part of the ET, and desirability bias on behalf of respondents. Because the evaluation was conducted 16 months after Sapan ended in September 2015, respondent answers may also be subject to recall bias. The ET conducted interviews in Thai via a translator, which may have introduced translation bias. The dual English and Thai language capabilities of the Team Leader helped mitigate against potential translation bias, as the Team Leader could cross-check her interpretation against that of the translator’s. Respondent attrition in the case of youth participants was a challenge given that many youth participants were university students at the time of Sapan and had since graduated and were harder for the ET and Sapan partners to locate. As a result, the respondent sample includes only five youth participants. As with many primarily qualitative designs with a small sample size both overall and within each respondent type, in some regards the evaluation is limited in its external validity. Furthermore, because there was no defined control group included as part of the evaluation, it is difficult to attribute outcomes to the Sapan program alone. To compensate for this limitation, the ET used data triangulation and the inclusion of bellwether voices to ground-truth the perceptions of program stakeholders. At the same time, the ET interviewed 23 of the 25 (over 90 percent) Sapan partners, thus collecting information representative of most Sapan partners. The ET members were also present during all interviews and conducted data analysis together, strengthening the evaluation’s inter-rater reliability. The ET originally included a mixed-gender, three-person team (one male and two female members). However, the evaluation was primarily carried out by the two female team members as the male team member withdrew during the second week of in-country fieldwork. Because most of the evaluation was conducted with female-only evaluators and translator, it is possible but unknown if this gender dynamic biased some respondents’ answers. Given the proportion of female respondents in the sample and gender norms in Thailand, it is also possible that the all-female ET had a positive effect and allowed for more candid responses from female respondents. 8 III. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: TO WHAT EXTENT AND HOW DID SAPAN REACH ITS THREE OBJECTIVES AS LAID OUT IN THE SAPAN PROGRAM FRAMEWORK? EQ 1 FINDINGS: OBJECTIVE A. STRENGTHEN THE CAPACITY OF INDEPENDENT AGENCIES (IAS) Sapan worked with four IAs with limited success and eventually discontinued work toward Obj. A in 2014. Sapan worked with four IAs: the Election Commission of Thailand (ECT), the National Anti-Corruption Commission (NACC), the National Human Rights Commission (NHRC), and the Office of the Auditor General (OAG). Interactions with the IAs occurred at the national level between DAI, USAID, and senior officials in Bangkok, as well as between Sapan partners and regional or provincial IA representatives.11 Qualitative data from respondent interviews and monitoring data results demonstrated that Sapan made the least headway toward Objective (Obj.) A: Enhance capacity of key independent agencies to provide effective government oversight. As Figure 4 shows, Sapan did not meet its targets for three out of five output indicators for ER A.1 Stronger independent government agencies, with linkages with civil society and academia that effectively oversee government action and implementation of public policy. 12 Sapan also did not meet its targets for the overall outcome indicators for Obj. A (1.3.1 Specialist perception [and scoring] of IA government oversight capacity and 1.3.2 Public perception and knowledge of IAs), although there was a slight increase in reported citizen confidence in NACC and OAG over the life of the project. Monitoring data for these indicators is displayed in Table 3 and Table 4. Additional analysis of indicator targets versus actuals is in Annex B. 11 2010 Sapan Work Plan Version 2 12 Sapan did exceed or meet its targets for indicators 1.1.1 [Number of] IAs assisted and 1.2.2 [Number of] people attending independent agency-CSO joint oversight awareness raising campaigns. 9 Table 3: Objective A Output Indicator Targets and Actuals (Source: Sapan Final Report, 2015) Objective A (IAs): Enhance capacity of key independent agencies to provide effective government oversight Life of Program (3/2010–9/2014) Target Actual % Achieved (Actual / Target) ERA.1: Stronger independent government agencies, with linkages with civil society and academia, that effectively oversee government action and implementation of public policy 1.1.1 Number of IAs supported/assisted 43 65 151% 1.1.2 Number of IA staff trained in transparency and accountability 462 249 54% 1.1.3 Number of IA staff trained in outreach and communications 393 136 35% 1.2.1 Number of IA-CSO joint oversight awareness raising campaigns 80 32 40% 1.2.2 Number of people attending IA-CSO joint oversight awareness raising campaigns 3,200 10,175 318% Figure 4: Objective A Result Achievement 10 Table 4: Objective A Outcome Indicator Targets and Actuals (Source: Sapan Final Report, 2015) Objective A (IAs): Enhance capacity of key independent agencies to provide effective government oversight Life of Program (3/2010–9/2014) Baseline Target Actual 1.3.1 Specialist perception [and scoring] of IA government oversight capacity: Independent Agency supports CSO and Media usage for campaign on Government Oversight (Panel Objective 4) Not Presented in Sapan Final Report (Change from baseline) North = +22% Northeast = +22% South = +22% (Change from baseline) North = -11% Northeast = -6% South = -6% 1.3.2 Public perception and knowledge of IAs Survey Question: Which organizations should oversee the work of the government? OAG = 8% NACC = 19% OAG = 10% NACC = 23% OAG = 13% NACC = 21% 1.3.2 Public perception and knowledge of IAs Survey Question: What is your level of confidence in this IA? OAG = 48% NACC = 54% OAG = 58% NACC = 66% OAG = 68% NACC = 62% According to USAID respondents, working with IAs as part of Sapan’s design occurred after prolonged discussions among stakeholders. During the design process, USAID and relevant contacts at the United States (US) Embassy in Bangkok initially debated the involvement of any aspect of the RTG in Sapan. USAID also conducted an initial assessment of the political landscape and had preliminary discussions with IAs. The results of these processes factored into Sapan’s design and formed the basis of many of its assumptions. For example, the discussions with IAs led USAID to believe that working with IAs would be a productive alternative to working with the RTG, which was experiencing discord between its political parties. Furthermore, although the political analysis recognized the fractioned nature of the Thai political landscape, the analysis did not predict a coup, but instead predicted relative stability. USAID documented the outcomes of stakeholder discussions and potential risks related to working with the RTG and IAs in Sapan’s Activity Approval Document. Though USAID did not receive the RTG’s official approval of Sapan—and its engagement of IAs—prior to Sapan’s launch,13 the US Ambassador to Thailand at the time, Eric G. John, informed the former Prime Minister of Thailand, Samak Sundaravej, about Sapan through a diplomatic note. One USAID respondent also cited examples of the Thai Ministry of Foreign Affairs (MFA) acknowledging Sapan activities, such as the MFA chairing a meeting for USAID to update Thai security officials on Sapan’s progress, as well as the attendance at some Sapan interventions of two former Directors of the MFA’s North America Division. However, per USAID and DAI respondents, work with IAs proved more difficult than envisioned. According to annual reports and respondents from USAID and DAI, Sapan experienced the most traction at the national level with the OAG, including a series of “open house” events in 2012 to increase citizen understanding of OAG. These events, though fewer than planned, drew more participants than expected. On the local level, Sapan held a series of training courses with IA and CSO participants, which “provided a rare opportunity for IA and CSO representatives to interact, share experiences and learn together.”14 13 Section 634 (o) of the US Consolidated Appropriations Act, 2008 (https://www.gpo.gov/fdsys/pkg/PLAW￾110publ161/html/PLAW-110publ161.htm) and Section 7034(k) of US Omnibus Appropriation Act, 2009 (https://www.gpo.gov/fdsys/pkg/PLAW-111publ8/pdf/PLAW-111publ8.pdf) both state that “with respect to the provision for democracy, human rights, and governance assistance, the organization implementing such assistance and the specific nature of that assistance shall not be subject to prior approval by the government of any foreign country.” 14 Sapan Final Report, pg. 15 11 A second training series on fraud and corruption brought together IA officials and partner universities at the local level. 15 In general, however, USAID and DAI respondents stated they had difficulty obtaining commitment from IA leadership (see EQ 1.3).16 Except for Sapan partners in one province in the South and one partner in the Central region who provided positive examples of working with the ECT and NACC, most Sapan partners interviewed did not find their work with IAs to be productive, with one respondent noting that IAs attended the events as a “check the box” exercise. Annual reports and respondents stated several factors contributing to the limited achievement of Obj. A, including: • lack of IA desire and commitment to work with civil society; • IAs being personality driven and bureaucratic; IAs not as independent as originally thought partly due to IAs being perceived as being controlled by political parties during periods with elected governments. At the very least, IAs were often staffed by former government officials, further affecting their independence. • ideological gaps between Sapan objectives and those of IAs; • benefits of working with Sapan not communicated between Bangkok and the provinces; • the 2014 coup itself, which halted any opportunities to pursue planned activities with IAs; and, • lack of a recent relationship between the United States government (USG) and RTG regarding development projects. Because of these challenges, Sapan formally discontinued its work with IAs in mid-2014. 17 EQ 1 FINDINGS: OBJECTIVE B. STRENGTHEN THE CAPACITY OF CSOS AND MEDIA Sapan increased partners’ organizational capacity and technical knowledge of tools and strategies for engaging government Quantitative data show mixed results for Obj. B: Strengthen the capacity of CSOs and media to serve as checks and balances for political processes and public policy. Sapan met or exceeded targets for 7 out of 10 output indicators for ER B.1: Strengthened CSO research and advocacy capacities to enable constructive dialogue with the RTG on key public policy issues, indicating achievement of this expected result. However, Sapan did not achieve ER B.2: More sustainable community media outlets, improved capacities of journalists to cover policy and conflict issues and a more open media environment, meeting the target for only one of its four indicators. 18 It is worth noting, however, that original targets for the four indicators were based on assumptions—which proved to be incorrect—regarding the volume of community media outlets, and thus the targets were overly ambitious. For Obj. B overall, values increased slightly for two outcome indicators (2.3.1 Specialist perception [and scoring] of CSO oversight capacity and 2.3.3 Specialist perception [and scoring] of community media capacity to strengthen democracy), though target results were not met. This monitoring data suggests 15 Sapan Final Report, pg. 16 16 Sapan Final Report, pg. 15 17 “As the political situation deteriorated in late 2013 and into 2014, opportunities for the Sapan Program to work with the IAs diminished. Any possibility to work with the IAs on activities presented in the Work Plan effectively ended with the May coup.” Sapan Workplan 6 Addendum 2014, pg. 1 18 ER B.2 originally had five indicators but dropped indicator 2.2.2 Media outlet staff trained in management, communications and planning as part of the Sapan Performance Monitoring Plan Version 2, September 2011. 12 there was progress towards but not achievement of Obj. B (see Table 5 and Table 6). Additional analysis of indicator targets versus actuals can be found in Annex B. Figure 5: Objective B Results and Indicator Achievement 13 Table 5: Objective B Output Indicator Targets and Actuals (Source: Sapan Final Report, 2015) Objective B: Strengthen the capacity of civil society organizations and media to serve as checks and balances for political processes and public policy Life of Program (3/2010-9/2015) Target Actual % Achieved (Actual / Target) ER B.1: Strengthened CSO research and advocacy capacities to enable constructive dialogue with the RTG on key policy issues 2.1.1 Number of civil society organizations trained in management, communications and planning 189 238 126% 2.1.2 Number of civil society organization staff members [/persons] trained in management, communications and planning 749 1,705 228% 2.1.3 Number of civil society organizations trained in governance issues 230 253 110% 2.1.4 Number of civil society organization staff members [/person] trained in governance issues 762 1,394 183% 2.1.5 Number of civil society organizations trained in research and advocacy 296 207 70% 2.1.6 Number of civil society org staff members [/persons] trained in research and advocacy 839 1,218 145% 2.1.7 Number of research grants awarded 343 318 93% 2.1.8 Number of good governance, oversight and advocacy campaigns 157 161 103% 2.1.9 Number of people attending good governance, oversight and advocacy campaigns 7,800 33,930 435% 2.1.10 Number of CSOs engaging in advocacy and oversight functions 160 188 118% ER B.2: More sustainable community media outlets, improved capacities of journalists to cover policy and conflict issues and a more open media environment 2.2.1 Number of non-state news outlets assisted 180 70 39% 2.2.2 Number of media outlet staff trained in management, communications and planning Indicator Dropped 2.2.3 Number of journalists trained on policy, freedom of information laws and conflict issues 446 142 32% 2.2.4 Number of journalists trained on transparency and accountability issues 440 93 21% 2.2.5 Number of media CSOs assisted 31 37 119% Table 6: Objective B Outcome Indicator Targets and Actuals (Source: Sapan Final Report, 2015) Objective B: Strengthen the capacity of civil society organizations and media to serve as checks and balances for political processes and public policy Life of Program Baseline Target Actual 2.3.1 Specialist perception [and scoring] of CSO oversight capacity Civil Society Organizations have strong organizational competency (Panel Objective 2.3) Not Presented in Sapan Final Report (Change from baseline) North = +22% Northeast = +22% South = +22% North = +5% Northeast = 0% South = +26% Outcome: CSO Capacity Scorecard 0 partners have all fifteen elements present 15 partners have all fifteen elements present 6 partners have most elements present (average = 13 elements); 14 partners have all elements present but some not standard quality 14 Additional quantitative and qualitative data complicate the picture shown by Performance Monitoring Plan (PMP) data. Though actuals achieved were lower than expected for three out of four output indicators for ER B.2 (the exception being “[Number of] media CSOs assisted)”, CSO capacity scorecard results and interviews with Sapan respondents provide evidence that both CSO and media partners increased their organizational and technical capacities, though some made more progress than others. Of Sapan’s 25 partner organizations, 15 were selected as “core partners” who received organizational capacity building interventions—in addition to participating in other interventions—and were assessed on 15 criteria19 using a CSO capacity scorecard (see Table 7 for core partners who used the CSO capacity scorecard).20 Of the 15 core partners whose progress was tracked by the CSO capacity scorecard, six partners reached the requirements for all 15 criteria; three partners reached 13-14 criteria; three partners reached 10-12 criteria; and three partners reached nine or fewer criteria. This is notable given most partners (11 out of 15) met three or fewer criteria at first. 21 All interviewed Sapan core partners stated appreciation for these organizational capacity building efforts and believed they are stronger because of them. The ET did not use any quantitative measures for assessing CSO respondents’ technical skill acquisition, but in qualitative interviews with USAID, DAI, and partners, respondents claimed an increase in CSOs’ technical knowledge and described concrete ways in which youth CSOs, women-led CSOs, media partners, and university partners put the tools and skills they learned to use (see EQ3 for more detail). According to annual reports, Sapan’s success at increasing the organizational and technical capacities of partners was primarily achieved through training (provided through subcontractors) and direct mentoring of the 15 core partners during field interventions using a “learning by doing” approach, as well as using the core partners to subsequently provide training for other local CSOs, resulting in “organizational capacity building [of] nearly 100 CSOs during each year of the Program.” 22 Training for core partners was grouped into three themes of management, governance, and research/advocacy, with the first wave of trainings focusing on management. In 2012, the focus changed to governance, and in 2013 to research/advocacy. In addition to trainings, technical capacity development also involved workshops, conferences, and exchange visits to neighboring countries in the region. Sapan helped establish four organizations—Café Democracy, People’s College, Center for Civil Society and Non-profit Management (CSNM) at Khon Kaen University, and Media Learning Center (MLC) at FCEM/Prachatai—as providers of training services to CSOs. 23 Qualitative interviews with respondents confirmed the effectiveness of these capacity building techniques: USAID and DAI respondents mentioned all types of capacity-building interventions. Sapan partners’ responses were different: workshops or exchange visits were not singled out; rather, Sapan partners tended to focus on the value of training and mentoring. 19 CSO Capacity Scorecard Criteria: (1) Overall goal or purpose statement; (2) Legal registration; (3) Finance policies; (4) Administrative policies; (5) Procurement policies; (6) Personnel policies; (7) Gender and diversity policies; (8) Anti-corruption policy; (9) Organizational structure; (10) Inventory; (11) M&E system; (12) Job descriptions; (13) Annual budgeting process; (14) Public outreach; (15) Fundraising strategy. 20 Six of Sapan’s seven university partners (the exception being Khon Kaen University due to the presence of its Research Group on Wellbeing and Sustainable Development (WeSD)) and four partner IAs were not designated as “core partners” and their organizational capacities were not assessed using the scorecard. 21 Sapan Final Report, pg. 67 22 Sapan Final Report, pg. 19 23 Sapan’s role in establishing these organizations was described in several Sapan Annual Reports: Café Democracy (April– September 2011); People’s College (October 2012–September 2013); CSNM and MLC (October 2013–September 2014). 15 Table 7: Sapan Partner Organizations CSOs in different sectors are primarily using tools to engage government and non￾government actors at the community level, often for improving service delivery but not acting as checks and balances for political process and public policy. Sapan partners interviewed all provided examples of using specific governance tools they learned in important ways, though not to the level of monitoring political process and changing public policies in the way Sapan envisioned in its results statements. Sapan partners mentioned using the community scorecard to help community members prioritize issues among themselves and decide which issues to raise to the government. Partners and beneficiaries also provided examples of using tools to peacefully negotiate and mitigate conflict with local administrations involved in service delivery and the private sector. For example, one beneficiary partner in the Central region described how the local government originally planned to contract a private construction firm to build a community water system. However, the community used the Community Charter tool to express that community members should be involved in the design and Sapan Partner Organizations Partner Type 1 Foundation for Community Educational Media (FCEM)/Prachatai* Media 2 Friends of Women Foundation* Women-led 3 Luukrieng Group* Youth 4 Media for Happiness Foundation* Media 5 Media Selatan* Media 6 Muslim Women’s Association * Women-led 7 Nature Care Foundation* Women-led 8 Office of Health and Social Development (OHSD)* Women-led 9 Pattani Forum* Media 10 Prachathum* Media 11 Research Group on Wellbeing and Sustainable Development (WeSD) at Khon Kaen University* University 12 Sri Song Kwai Women Ordinary Partnership* Women-led 13 Voluntary Women’s Group* Women-led 14 Women for Peace Association (WePeace)* Women-led 15 Youth Cares Thailand* Youth 16 People's College Youth 17 Café Democracy Youth 18 Center for Civil Society and Non-Profit Management (CSNM) at Khon Kaen University Youth 19 Southern Community Media Association Media 20 Ubon Ratchathani University University 21 Naresuan University University 22 Chiang Mai University University 23 Kasetsart University University 24 Yala Rajabhat University University 25 Nakhon Sri Thammarat Rajabhat University University * Designates "core" partner status 16 development process of the water system, and successfully convinced the local government to hire local people to perform the construction, thus injecting income at the grassroots level. Another beneficiary partner in the Northeast region described that community members were upset by the long queues and rude treatment they received at a health promotion hospital. The community used the Community Scorecard to peacefully express their discontent and engage in a dialogue with the health promotion hospital, which eventually took steps to address these issues. A few partners provided examples of attempting to monitor local government budgets using social audit tools, though the partners stated that these efforts were not very productive (see EQ 2). In addition to the interviewed women-led CSOs who joined together to work on Women’s Development Fund (WDF) policy, three of the 14 core partners interviewed provided examples of attempting to change policies (e.g., land use policy in Northeast) through tools and public policy campaigns. Sapan partner respondents stated the following factors influenced and continued to influence their ability to engage government actors: government sensitivity to some issues (e.g., rights issues or budget inquiries) more than others, laws passed after the 2014 coup prohibiting the gathering of people and suppressing media, government officials’ receptiveness to community participation and criticism, and provincial government office receptivity to local government officials who do want to work with communities. Sapan made some progress on strengthening community media outlets but was strongly affected by the political situation. As previously mentioned, Sapan fell short of its indicator targets for ER 2.2. The Sapan Final Report describes challenges related to working with media organizations in Thailand, including media organizations’ hesitance to go beyond themes they have investigated in the past, a tendency to be politically polarized and biased, and the difficulty of finding unbiased partners interested in developing their capacity. 24 However, qualitative data suggests that Sapan did strengthen the technical capacities of the five media partners it managed to work with. Media partner respondents provided examples of ways in which they became more sensitive in their reporting on conflict (e.g. changing the terminology, such as refraining from using the terms “Southern terrorist”), included more voices in their media coverage to provide more perspective and objectivity, incorporated the voices of women and marginalized groups, and brought community issues to the attention of local government officials and policy makers (see EQ 3). Technical strengthening of media partners occurred through trainings and, in 2014, the establishment of the MLC at FCEM/Prachatai to “build the capacity of journalists and social activists in the use of media technologies, news reporting, modern communication methods and government oversight.” 25 In qualitative interviews, media partners from around the country specifically stated how much they learned from FCEM/Prachatai. Though the political situation after the 2014 coup affected several Sapan partners, media partners appeared to have been particularly affected, as all five media partners interviewed mentioned military scrutiny into their interventions and needing to self-censor to avoid further military action against them. 24 Sapan Final Report, pg. 27 25 Sapan Final Report, pg. 22 17 Sapan engaged youth participants in governance, providing some youth with skills and appreciation for governance or non-governmental organization (NGO) work. Sapan engaged youth through three primary interventions: research grants to university students; study of governance and democracy issues through Café Democracy, Freedom Zone, and the School of Good Citizenry at People’s College; and OPERACY, a leadership and personal empowerment training.26 Sapan’s PMP did not include indicators tracking results related to governance outcomes for youth, but in the two SGDs conducted with youths who participated in Café Democracy (two youth respondents) and School of Good Citizenry (two youth respondents), youth credited Sapan’s governance courses with increasing their analytical skills and interest in social activism. The ET was not able to interview students who received research grants, though three university partners and one women-led CSO mentioned students who were involved in Sapan subsequently getting involved in NGO work, and one youth interviewee from the Deep South currently worked for the NHRC. This last point was important given a DAI respondent’s view that youth and their families often did not consider NGO work a viable career path (opting instead to go into the private sector), and that one of the challenges the Thai NGO community faces is a lack of incoming young, new staff and activists with requisite skills. The ET was unable to adequately assess the effects of the OPERACY training; the Sapan annual report states that “hundreds” of youth participated in the 5-day OPERACY course, and one USAID respondent provided three anecdotes about how OPERACY training empowered some participants from Sapan partners, including a university professor whose colleagues reportedly commented that this person’s demeanor changed after the training. However, the ET did not interview any youth who had participated in the training, and only two Sapan partners (one in the Northeast and one in the Deep South) mentioned OPERACY at all. EQ 1 FINDINGS: OBJECTIVE C. SUPPORT CIVIC PEACEBUILDING EFFORTS AND REDUCE VIOLENT CONFLICT IN THE DEEP SOUTH Data show a mixed picture of whether Sapan interventions had a direct impact on peacebuilding and diminishing potential for violent conflict in the Deep South, though qualitative data suggests that Sapan did increase connections between organizations in the Deep South and those in the rest of the country. According to monitoring data, Sapan reached its targets for all output indicators under ER C.1: Increased capacity and visibility of advocacy NGOs and civic leaders working to promote peace, and ER C.2: Reduced pool of recruitable youths available to insurgent groups and increased social and economic opportunities for youth, suggesting achievement of these ERs. Monitoring data was inconclusive regarding whether Obj. C was achieved, since the outcome indicator measuring youth perceptions of social and economic opportunities increased slightly but did not meet its target (see Table 8 and Table 9). 26 OPERACY is not an acronym, but the name of the training program developed by Christopher Lee. https://www.facebook.com/OperacyTraining/ 18 Figure 6: Objective C Results and Indicator Achievement 19 Table 8: Objective C Output Indicator Targets and Actuals (Source: Sapan Final Report, 2015) Objective C: Support civic peacebuilding efforts and diminish the potential for radicalization and escalation of violent conflict in Southern Thailand Life of Program (3/2010–-9/2015) Target Actual % Achieved (Actual / Target) ER C.1: Increased capacity and visibility of advocacy NGOs and civic leaders working to promote peace 3.1.1 Good governance, oversight and advocacy campaigns (South) 26 36 138% 3.1.2 Persons attending good governance and advocacy campaigns (South) 1,925 8,151 423% ER C.2: Reduced pool of recruitable youths available to insurgent groups and increased social and economic opportunities for youth 3.2.1 Youth-focused conflict transformation campaigns 22 19 86% 3.2.2 People attending youth-focused conflict transformation campaigns 1,890 9,404 498% 3.2.3 Deep South youth trained in management, communications and planning 30 49 163% 3.2.4 Deep South youth trained in governance issues, research and advocacy 110 161 146% Table 9: Objective C Outcome Indicator Targets and Actuals (Source: Sapan Final Report, 2015) Objective C: Support civic peacebuilding efforts and diminish the potential for radicalization and escalation of violent conflict in Southern Thailand Life of Program (3/2010–-9/2015) Baseline Target Actual 3.3.3 Youth perceptions of social and economic opportunities Survey question: Youth get sufficient support from government 37% agree 45% agree 44% agree 3.3.3 Youth perceptions of social and economic opportunities Survey question: Youth are active in community activities 21% state “to a great extent”; 41% to some extent” 26% state “to a great extent”; 50% to some extent” 16% state “to a great extent” 70% to some extent” 3.3.3 Youth perceptions of social and economic opportunities Survey question: Youth know of CSOs working to improve the prospects of youth 1.7% know of such CSOs 3% know of such CSOs 2% know of such CSOs Qualitative data also complicated whether Sapan’s larger objective of reducing the potential for conflict was achieved. Per the Sapan final report, interventions under this objective “brought together people from all sides of the conflict in ‘safe’ spaces to facilitate collaborative design and implementation of activities to promote peace” (pg. 31), but respondents from DAI, USAID, Sapan partners, and bellwethers talked about how interventions under Obj. B—not only Obj. C— had similar outcomes in other parts of the country. These respondents described Sapan’s Obj. C interventions as an important “stepping-stone” to peace, but could not directly claim that Sapan did peacebuilding work. Respondents from USAID, DAI and Sapan partners did provide anecdotal evidence that Sapan widened Deep South issues beyond the Deep South—an important achievement according to these stakeholders. These successes included changing the way some large media organizations report on the conflict in the 20 Deep South (see EQ 5), and helping organizations in the Deep South understand how organizations in the North and Northeast have dealt with conflict. Qualitative data suggests that Sapan created relationships among civil society actors, which is an important contribution to all three objectives. According to interviews with USAID and DAI respondents, the relationship-building network Sapan helped create also contributed to successes under multiple objectives, and has made the CSO community stronger as a result. This claim is supported by interviews with Sapan partners, the majority of whom mentioned the value of their relationships and networks with Sapan partners and other government and non-government actors. Although Sapan may not have conclusively achieved higher-level outcomes related to civil society acting as a check and balance for the government, respondents emphasize that the relationship-building successes are an important achievement in themselves and cannot be discounted, although they take up a considerable amount of program time and resources. Per USAID respondents, this idea of the foundational importance of relationships being overlooked can be summed up in a metaphor: “If you look at the skyscrapers, they’ll spend months on a foundation, then all of a sudden the building is built—but until the building gets built no one is going to believe us.” EQ 1 CONCLUSIONS • Sapan was most effective at contributing toward Objective B (CSO capacity building), having increased the organizational and technical capacities of CSO and media partner organizations through effective training and mentoring interventions. Providing checks and balances on the government was something only some Sapan partners had been engaged in prior to the start of Sapan. Project documentation noted organizations showing signs of increased technical capacities. The ET found that most organizations were not able to actively play a role in providing checks and balances on the government after the 2014 coup. • Sapan made some progress toward Objective C (peacebuilding and reducing violent conflict in the Deep South), though more so in terms of increasing the connectivity of organizations in the Deep South to those in other parts of the country, rather than direct peacebuilding. • Sapan was least effective at contributing toward Objective A (strengthening IAs) for reasons both programmatic and political. • Sapan’s relationship- and network-building interventions contributed to progress in all three objectives. 21 EQ1.1: DID SAPAN’S THEORY OF CHANGE, ESPECIALLY THE ROLES OF THE INDEPENDENT AGENCIES, HOLD TRUE THROUGHOUT THE COURSE OF SAPAN’S LIFE? EQ 1.1 FINDINGS The role of IAs changed due to lack of success engaging IAs and increasing lack of independence on the part of IAs. Sapan designers originally theorized that “the independent agencies—if strengthened and linked with CSOs and academics—could play a key role in providing oversight of government and promoting a more democratic culture of checks and balances.” 27 This strengthening would occur through technical assistance and training to IA staff, along with Sapan initiatives that brought together IAs, CSOs, academics, and others to jointly address problems with law and policy. 28 Work with IAs proved more difficult than envisioned. Per annual reports and respondents from USAID and DAI, Sapan experienced the most traction at the national level with the OAG, including a series of “outreach events” in 2012, but in general had difficulty obtaining commitment from IA leadership. 29 The initial traction with the OAG was due to the receptivity of the initial Auditor General, but when that person left, Sapan activities with the OAG ceased. According to USAID and DAI respondents, Sapan designers believed that it needed to work with IAs at the national level because of the IAs centralized administration, with the rationale that local-level IA representatives would not be able to collaborate with Sapan without the support of senior officials in Bangkok. However, because of the lack of traction with IAs at the national level, local-level IA staff were instructed to attend Sapan events without understanding why. The independence of IAs, and thus their ability to monitor other government entities, also became more questionable over time according to USAID, DAI, and Sapan partner respondents. According to one Sapan partner, IA staff are often former government staff and therefore do not act impartially to other government bodies. Sapan’s media strategy changed from working with community media30 and individual journalists to working with local media outlets. The role of universities also changed to involve more interaction with CSOs and communities. Sapan learned early on that its original theory that community media and local journalists would be an “effective channel” for governance work was “overly optimistic” in part due to their lack of credibility among Thai citizens, 31 as fewer independent journalists were willing to work with Sapan than expected according to a DAI respondent. Sapan shifted its design accordingly to working with local media outlets.32 Similarly, Sapan altered its design to expand the role universities played. Sapan found few large traditional NGOs working directly on the governance issues relevant to the program. 33 Per DAI and USAID respondents, Sapan identified universities as being well positioned to fill this gap due to Thai citizens’ 27 Sapan Contract, Section C: Statement of Work (SOW), pg. 12 28 Sapan SOW, pg. 13 29 Sapan Final Report, pg. 15 30 Generally, in Thailand, the term community media refers to the use of non-professionals. 31 Sapan Final Report, pg. 26 32 Local media outlets, or “Media CSOs” refers to not-for-profit organizations that transmit feature stories and news to the public through various distribution channels, including the internet, television, radio and printed material [Sapan Performance Monitoring Plan Version 2, 2011] 33 Sapan Annual Report 2010–2011 22 perception of them as more politically neutral. DAI also observed that universities had weak relationships with communities, other than as research subjects, and that Sapan interventions could help bridge this gap. Consequently, universities were used to identify and train CSOs, as well as leading interventions with communities. In interviews with the ET, two out of five university partners included in the sample agreed that they had difficulty working with communities prior to Sapan, but, through Sapan interventions, their relationships with communities improved. Stakeholders question Sapan’s theory of change under Obj. C, and the link between Sapan and reducing violent conflict and extremism in the Deep South. Sapan was flexibly tasked to “develop activities to…build trust within the community and between citizens and the state, rather than further aggravating the conflict, responding to opportunities as they emerged. 34 As described in EQ1, notable interventions under Obj. C focused on information sharing between actors and youth in the Deep South, and connecting the Deep South to other regions. But respondents from DAI and Sapan partners in the Deep South questioned the validity of Sapan’s theory on how to meaningfully do peacebuilding work and diminish the potential for radicalization, especially since respondents also note that—due to political sensitivities—USAID instructed Sapan not to engage in direct peacebuilding activities (e.g. working with human rights activities). These respondents consequently described Sapan’s interventions under Obj. C as an important “stepping stone” to peace, or “indirectly” working toward the peace process, but not directly addressing peacebuilding in the way that other donor programming does. But many of Sapan’s constituency-building, governance-training, and media-strengthening interventions reported under Obj. C with Deep South actors were not significantly different from interventions conducted under Obj. B with CSOs, universities, and media groups in other parts of the country. One DAI respondent described interventions under Obj. C as “Component B in a different place.” Though Obj. C interventions showed positive results in terms of reducing prejudicial thinking and increased understanding between stakeholders (see EQ1), Sapan partners implementing interventions under Obj. B (e.g. discussion forums, media programs) also described similar changes in other parts of the country, intimating that perhaps Sapan’s interventions were more appropriate for achieving conflict resolution, though not peacebuilding. Some Sapan partners in the Deep South, as well as bellwethers, said real peace cannot be had without tackling issues like language, education, and cultural rights. Sapan seemed aware of this in its original design: embedded in Sapan’s theory of change is the hypothesis that increasing youth’s social and economic opportunities will reduce their risk of engaging in violent conflict. ER C.2 in the project roadmap is “Reduced pool of ‘recruitable’ youths available to insurgent groups and increased social and economic opportunities for youth” (italics added by ET), including supporting intended outcomes of “vocational training made available to at-risk youth” and “Advocacy for social/econ opportunities for youth strengthened.” Similarly, an outcome indicator 3.3.3 for Obj. C measures youth perceptions of social and economic opportunities. Despite expressing this linkage and planning relevant interventions in its work plans,35 Sapan’s interventions in the Deep South as reported in annual reports and by respondents had little if any focus on increasing social and economic opportunities for youth. Rather, they focused on youth empowerment. 34 Sapan SOW, pg. 16 35 “Sapan will support an employment opportunity and career-building convention to increase youth awareness of employment opportunities in, and appropriate approaches to and skill sets required for obtaining gainful employment in the public and private sectors. The event will include sessions on private and public-sector job opportunities, what potential employers are looking for; the legal and regulatory frameworks under which businesses and NGOs operate; cross border trade opportunities and challenges; and resume writing and interview techniques.” [2012 Sapan Work Plan Version 4, pg. 16] 23 USAID and DAI respondents offered explanations for this disconnect, including earmarked funding streams that necessitated Obj. C being standalone in the project design with its own associated indicators, as well as US–Thai political relations, which limited Sapan’s ability to work with human rights activists or other sensitive actors in the peacebuilding process. Key informants believe that strengthening both government (Obj. A) and civil society (Obj. B) were essential and complementary to reaching Sapan’s overall goal, but that the project could have been better scoped. USAID and DAI respondents mentioned that working with government, in addition to civil society, was an appropriate aspect of Sapan’s design. The flaw, as some respondents saw it, was working with the wrong type and number of government stakeholders, though respondents had different ideas of which and how government entities would have been more ideal. Two USAID respondents felt that working with IAs was the correct approach, but that factors outside of the Sapan’s control (e.g., internal conflict within IAs) limited the effectiveness of the approach. Other USAID and DAI stakeholders felt that work with IAs was overly ambitious altogether, given their reportedly entrenched behaviors and mindset (see EQ1.2 for more detail). One DAI respondent would have preferred to target one, not four, IAs, while other respondents, including Sapan partners, believed that interventions to formally engage local administrators may have yielded more results than focusing on IAs at all: “Local government is the best organization that we have to work with … local government is the front line that has to solve the problem” (University Partner). At the same time, a USAID respondent suggested that Obj. A was too narrow in scope by limiting itself to IAs, and should have included engagement with government “at any level.” With regards to Obj. B, key informants also differed in their opinions of which and how many civil society actors Sapan should have worked with. Sapan partners in the Northeast felt that working directly with communities rather than CSOs is better as they believed CSOs are too politically polarized, and though CSOs attend donor meetings, it is communities who act to solve problems. Some DAI and bellwether respondents suggested that Sapan’s work focused on changing civil society in the provinces, rather than targeting other influential civil society actors like middle-class citizens and the private sector. On the other hand, a USAID respondent stated that targeting CSOs rather than communities was necessary to issue sub-grants, facilitate networking, and promote sustainability. DAI respondents commented that though it was correct to target CSOs, Sapan worked with too many types of CSOs and thus was too “scattered” in its approach, making it difficult to target the right partners within each CSO type: “Successful theory of change can only happen where there is already good local community, with good local administration.” At the same time, while acknowledging the scoping challenges, one DAI respondent pointed out the appropriateness of the civil society groups Sapan chose to work with: “We did everything but try to change the habits of the rich and elite. But we also did not go too granular trying to change individuals. We reached the middle ground. Scope is as good as it was going to get in Thailand.” EQ 1.1 CONCLUSIONS • Hypotheses related to the role of IAs, media and university partners changed over time. Assumptions about the neutrality and motivation of IAs proved to be untrue. The project also adapted to changes in assumptions regarding the role universities and local media organizations could play in Sapan programming. • Sapan’s logic of working with both government and civil society was valid, but the project could have better chosen the number and type/level of government and civil society actors with whom to work. • Sapan’s three objectives were intended to be mutually reinforcing, as well as necessary and sufficient to achieve Sapan’s larger objective of fostering constructive civil society engagement 24 with government. The theory of change linking Obj. A and B as mutually reinforcing elements was valid for Sapan, whereas Obj. C as a standalone result was not, and could have been subsumed under Obj. B in the project framework. • The theory of change related to how Sapan interventions in the Deep South were to lead to a reduction in violent conflict proved questionable as Sapan interventions under Obj. C. did not focus on direct peacebuilding. There was a disconnect between the indicators chosen to measure Obj. C’s higher-level outcome (related to youth economic and social opportunities) and the interventions Sapan actually implemented. EQ1.2: HOW DID THE CHANGES IN THE INITIAL ASSUMPTION THAT FORMULATED THE THEORY OF CHANGE, IF ANY, AFFECT SAPAN’S ABILITY TO MEET THE OBJECTIVES? EQ 1.2 FINDINGS The assumption that the political environment would hold stable did not hold true, which affected Sapan’s ability to conduct some interventions but not others. As articulated in its yearly work plans, one of Sapan’s critical assumptions was that the “political environment is conducive to design and implementation of program interventions throughout the country,” 36 but this assumption did not hold true. Respondents from USAID, DAI and the Department of State (DOS) confirm that though USAID conducted a governance assessment prior to launching Sapan and understood the fragility of the political environment, stakeholders did not anticipate a coup to occur during Sapan’s lifetime. The deteriorating political situation—culminating in the May 2014 coup—affected Sapan’s ability to work with IAs and make progress toward Obj. A. (see EQ 1). It also affected progress towards Obj. B and C, as military crackdowns and new laws affected some but not all CSOs abilities to conduct certain interventions (see EQ 2.1 for more detail on how the political environment affected Sapan’s management and implementation). The incorrect assumption that IAs would want to work with Sapan limited Sapan’s ability to strengthen government. Sapan assumed that “independent agencies are willing and motivated program partners,” 37 but this assumption did not hold true. Despite some initial traction with the OAG, USAID, and DAI respondents state IAs’ lack of commitment to Sapan interventions, a challenge noted in Sapan’s annual reports. This change in assumption limited the extent to which Sapan could work with IAs, and negatively affected Sapan’s progress toward Obj. A (see EQ I). Qualitative interviews suggest some factors that affected this assumption. USAID and DAI respondents point to the lack of a recent US-Thai foreign assistance relationship prior to Sapan, as well as the fact that, for political reasons, Sapan was not designed in collaboration with the government, as reasons why it was difficult and time consuming to build relationships with the MFA and IAs. Furthermore, a DAI respondent stated that Sapan overestimated the extent to which IAs feel accountable to citizens, and would thus be interested in Sapan’s objectives: “The mentality that IAs should work with the people just doesn’t exist.” Some Sapan partners believe the same to be true of other government actors as well, as “in certain instances the local administration [is] accountable to the center, not the local people.” 36 2010 Sapan Work Plan Version 2, pg. 22 37 2010 Sapan Work Plan Version 2, pg. 22 25 Some assumptions around work with CSOs were incorrect, while others held true. An incorrect assumption about CSO capacity led Sapan to focus on organizational capacity building interventions. Three of Sapan’s assumptions underlying its work with CSOs included: (1) “CSOs, IAs, and the media are willing and motivated to work together”; (2) “Capable partners will be found in all focus regions and themes, and for all interventions”; and (3) “CSOs are part of or are willing to form alliances to advocate on themes relevant to Sapan.” The third assumption held true, as all Sapan partners provide examples of engaging in project-initiated coalitions and networks (see EQ2), such as the women’s coalition convened to advocate around the WDF (see EQ5). The first assumption partially held true. Though IAs participated in events organized by CSO and media partners, their motivation and buy-in is questionable (see EQ1). However, CSOs, media partners, and universities proved willing to work together, as evidenced by the number and types of interventions involving multiple Sapan partners. The second assumption did not hold true according to USAID and DAI respondents. Sapan undertook a 9-month long Learning Process in Project Year 1 to understand the CSO landscape and select its 25 CSO partners.38,39 Yet even with the assistance of universities to find suitable CSO partners, DAI respondents and the Sapan Annual Report 2011–2012 state that the pool of CSOs capable of working on governance issues was smaller than initially envisioned. Two DAI respondents commented that the internal management capacity of Thai CSOs was low. Sapan did find “CSO partners that have the interest and motivation to [work on governance]” and shifted its work with partners “to develop the necessary technical and administrative capacity for successful action.” 40 With Sapan’s interventions focusing on building the capacity of CSOs—setting the foundation for governance work—some DAI and USAID respondents expressed disappointment with some CSO partners, such as the women-led organizations, to “go to the next level,” that is, act beyond their comfort zones and advocate for governance issues outside of direct service delivery. However, other USAID respondents clarified that although some partners had governance goals as part of their grant proposal to Sapan, Sapan never developed a formal Memorandum of Understanding or other document with its partner organizations that specified an “end state” of what the organization would look or act like by the end of the project. One USAID respondent furthermore explained that funding cuts to Sapan, just as NGOs were poised to move towards “higher-level” work, may have affected their progress. EQ 1.2 CONCLUSIONS • Incorrect assumptions related to IAs and the political environment had a negative effect on project achievement. • Sapan’s assumptions regarding the capacity of CSOs in Thailand were incorrect. Sapan effectively adapted its approach to focus on organizational capacity building efforts, yet this pivoting also meant that Sapan was limited in its ability to do higher-level governance work given the current capacities of partner organizations, as well as the generally undefined or inadequately agreed upon governance goals between these organizations and Sapan. 38 Learning Process Final Report 39 Selection criteria for the CSO partners included: Well-established and respected as leaders in their geographic and thematic area; Active, with full-time staff and on-going projects; Large network, working with many other NGOs to enable spreading of Program impacts; Motivated to learn and grow; and Politically neutral [Sapan Final Report, pg. 18] 40 Sapan Annual Report 2011–12, p 13 26 EVALUATION QUESTION 2: WHAT INTERVENTIONS WERE MORE SUCCESSFUL AND/OR HAD A GREATER CONTRIBUTION TO SAPAN’S OBJECTIVES? EQ 2 FINDINGS Organizational capacity-building training and the associated CSO capacity scorecard were successful. All Sapan partners interviewed that received organizational capacity building expressed that this intervention was valuable. 41 Similarly, partners appreciated the CSO capacity scorecard tool by which they could measure their progress. Sapan partners most frequently cited financial management training and legal registration as useful organizational capacities. For the former, some Sapan partners admitted that they were at first put off by financial management training, but eventually realized that it was the most useful of all the trainings. Organizational capacity building efforts successfully increased CSOs capacity to work in a more strategic and professional way, and increased their potential for sustainability though there is still room for organizations to improve further (see EQ 3 and EQ 4). Certain governance tools (e.g., community scorecards and public discussion forums) were more successful than others (e.g., budget literacy and the Provincial Governance Index). Sapan introduced partners to several governance tools, including community scorecards, budget literacy and monitoring, the Provincial Governance Index (PGI), citizens’ charters, social audit, public discussion forums, and participatory analyses. The community scorecard was one of the most successful tools,42 with the final report claiming its use “more than 65 times in 16 provinces,” 43 and at least nine combined Sapan partners and beneficiaries interviewed mentioned using this tool. Sapan partners also cited effectively using public discussion forums and participatory analyses. One partner in a province in the North described a particularly effective use of the citizens’ charter with a health promotion hospital. The ET confirmed the productive and continued use of the citizen charter in its SGD with associated beneficiaries (including a health promotion hospital representative) of that intervention. According to Sapan partners, budget literacy and monitoring and social audit were not as useful because they were too technically complex for communities and CSOs—one DAI respondent said that budget monitoring was complicated and better suited for those with better financial skills (i.e., college graduates)—and because most government officials were not amenable to having their finances scrutinized. Although the Sapan final report describes PGI as well as budget literacy and monitoring as successful interventions, 44 Sapan partners interviewed said that these tools were less useful to them. All respondents 41 The partners who received organizational capacity building and used the CSO capacity building scorecard are: FCEM/Prachatai, Friends of Women Foundation, Luukrieng, Media for Happiness, Media Selatan, Muslim Women’s Association, Nature Care Foundation, OHSD, Patani Forum, Prachathum, WeSD, Sri Song Kwai Women Ordinary Partnership, Voluntary Women’s Group, WePeace, and Youth Cares Thailand. 42 Community Score Card is an instrument to elicit social and public accountability and responsiveness from service providers. See: http://www.betterevaluation.org/en/resources/guides/community_mapping/general_methodology_note 43 Sapan Final Report, pg. 11 44 Sapan Final Report, pg. 11-12 27 from university partners described PGI as being useful in theory. These respondents appreciated that Sapan initially asked the entire network of university partners to work together to contextualize the index to the Thai context. However, Sapan tasked the final iteration of the tool to one university partner. University respondents believe that the final tool developed by this single partner did not reflect the network’s contextualization and was not an effective measure of provincial governance in Thailand. Consequently, university partners did not use the tool because they did not find it applicable to the Thai context. Creating different networks and networking opportunities with CSO partners, within and across CSO types, and across regions was successful. A major focus of Sapan interventions was relationship building among stakeholders through partner networks (e.g. Thai Women Coalition, coalition of media partners, university coalition and Deep South Youth Congress) and networking events. All Sapan partners interviewed spoke about their involvement with a Sapan-facilitated network, and how their involvement in these networks contributed to their capacity building. CSO partners of the same type but located in different regions learned from each other (e.g., smaller local media learned techniques from Prachatai), and partners of one type learned from other partner types (e.g., women-led organizations learned from universities) (see EQ 3 for more detail). Café Democracy, operating through a “Book Republic” bookstore, is a successful intervention that was not originally in Sapan’s design but became a replicable model. In 2011 Sapan helped establish Café Democracy in Chiang Mai. This intervention was not part of Sapan’s original design, but turned out to be a “highly successful hub for youth socio-political activism.”45 Café Democracy organized panel discussions, book launches, workshops, film screenings, and other interventions dealing with democracy and good governance. Café Democracy also created a “Democracy School” with training courses that focused on increasing advocacy skills for youths. 46 Per key informants from USAID and Café Democracy, as well as Café Democracy beneficiaries, the project succeeded at increasing youth capacities to analyze social structures and governmental policies and encouraging public exchange of ideas about governance. According to a beneficiary, Democracy School created a network among many people with different views and created a social movement. After people joined this school they created a network with other networks—such as the New Democracy Movement (NDM), and another group called “media art”—tackled issues for freedom of speech. Another youth beneficiary noted that through Democracy School students learned how to raise questions about things they had not previously understood. “We’ve been eating rice everyday” but we don’t have any idea … why farmers came out to protest. But after [Democracy School] we realized that there are factors causing the issues. The government kept the price of rice artificially low. We think more about these issues. And we start thinking about the social structures that caused these problems.” 45 Sapan Final Report, pg. 83 46 Sapan Final Report, pg. 22 28 Café Democracy’s approach was so successful that Sapan attempted to replicate its interventions by establishing “Freedom Zone” at Ubon Ratchathani University to convene public discussion forums, and People’s College in Pattani ran the “School of Good Citizenry” (modeled on Democracy School). 47 Key informants from People’s College and youth beneficiaries who participated in the School of Good Citizenry both gave examples of positive results of this intervention. For example, youth beneficiaries explained that they learned how to analyze before acting—i.e., how to first understand the history, cultural context, and power dynamics in which communities are immersed before identifying the activities and tools one should use. They also learned facilitation tools to help “find a common area between two parties.” However, key informants from Ubon Ratchathani and Café Democracy comment that Freedom Zone was not successful for several reasons, including motivation on the part of the Sapan project leader48 and the location of the Freedom Zone Center.49 EQ 2 CONCLUSIONS • Sapan’s organizational capacity building interventions were successful. • Specific interventions that could be applied were more useful and contributed to Sapan’s objectives. • Some successful interventions were not part of the original design, but came up in the middle of implementation or were opportunities that Sapan seized. • Interventions for building relationships between and among CSOs and other actors, either through formal networks or relationship building/exchange like community forums, proved critical for contributing to all Sapan objectives, but especially Obj. B and Obj. C. Relationship building also had particular significance for connecting organizations in the Deep South to those in the rest of the country. 47 Sapan Final Report, pg. 22 48 According to Sapan partners, Café Democracy’s founder wanted to start a bookstore as a hub of activism, whereas Sapan management (DAI and USAID) asked the project lead at Ubon Ratchathani University to implement Freedom Zone without being entirely vested in the idea. 49 According to Sapan partners, Café Democracy functions out of “Book Republic,” a commercial enterprise accessible in a public area and accessible to different people. In contrast, Freedom Zone was established inside Ubon Ratchathani University due to financial constraints and this limited its accessibility to the greater public. Figure 7: “Book Republic” Bookstore, Home of Café Democracy 29 EQ2.1: WHAT, IF ANY, MANAGEMENT SHIFTS ADOPTED IN RESPONSE TO THE DETERIORATING POLITICAL ENVIRONMENT PROVED EFFECTIVE IN CONTRIBUTING TO SAPAN’S OBJECTIVES? EQ 2.1 FINDINGS Stakeholders believe that the contracting mechanism for this program was appropriate for the context and allowed USAID/Sapan to make management shifts as appropriate. Sapan was issued as a contract rather than a cooperative agreement. The decision to use a contract occurred after USAID conducted a lengthy analysis of different funding mechanisms, including direct contract, cooperative agreement, Leader with Associate, Indefinite Quantity Contract (IQC), and direct grant. Per USAID respondents, due to the rapidly evolving political solution in Thailand during the design phase, USAID ultimately decided on a contract because designers wanted to integrate best practices in contract management from USAID’s Office of Transition Initiatives (OTI), since this office has considerable experience implementing in unstable environments. The intention behind the contract was for USAID to be an active part of the Sapan team, and that Sapan would embrace a culture of responsible risk-taking and employ innovative means to get funding to local entities (e.g., Sapan directly funding the cost of meeting rooms rather than the partner procuring it themselves). This contract design also allowed for USAID, as well as DOS, to have more involvement in implementation given Thailand’s sensitive political environment. For the most part, DAI and Sapan partners did not perceive the contracting mechanism and USAID’s involvement to negatively affect project implementation because of DAI and USAID’s flexible management approach with partners. Sapan discontinued, delayed, or tailored some interventions because of the deteriorating political environment, including but not limited to the 2014 coup. Sapan also increased its transparency through additional reporting. Sapan made the management decision to discontinue interventions with IAs owing to the view (expressed in the Risk Mitigation section of the November 2012 Workplan) that there was a lack of adequate buy-in by key leaders, particularly of IAs, and because of “political crises at the national level due to polarization or competition for control of independent agencies.”50 Though this meant sacrificing progress toward Obj. A, one USAID respondent noted that it allowed Sapan to redirect resources to other interventions. Three Sapan partners (two in the North and one in the Northeast) stated they had to delay or stop some interventions post-coup (e.g., public discussion forums in the case of one partner, media-related interventions in the case of another), out of concerns that these interventions could affect US-Thai relations. One USAID respondent also spoke about a Sapan-supported conference scheduled in the Northeast in 2014 that was eventually canceled due to scrutiny from USAID. These respondents feel that Sapan could have potentially achieved more—such as creating more awareness among the public or fostering peer exchange—if these interventions were not canceled. However, it was not possible for the ET to verify the “counterfactual”—that is, what might have happened if the interventions had been implemented as planned. At the same time, USAID, DAI, and DOS respondents note that the political situation throughout the life of Sapan was always tense and that some management shifts did not originate from DAI, Sapan partners, or USAID. For example, Sapan adjusted to concerns from DOS and increased its transparency by having more frequent meetings with MFA, as well as creating “fortnightly updates” as part of its reporting to USAID. Some of these respondents claim pressure from USAID senior management and DOS at various 50 Sapan Workplan 5 (Version 19, November 2012), pg. 3 30 points throughout the project, including directives about which local partners Sapan could and could not work with, which governance issues Sapan could tackle, and who could be invited to speak or participate in certain public Sapan interventions. Sapan and partners made other management shifts related to being more careful, though it is difficult to ascertain what effect these cautionary steps had on achievement of Sapan objectives. According to respondents from USAID, DAI, and some Sapan partner respondents in the North and Deep South, Sapan and partners communicated the need to proceed more carefully after the coup. In one partner’s case, this involved getting a waiver to remove USAID branding from Sapan-supported materials so as not be directly associated with USG funding. In other cases, this involved Sapan staff asking partners to communicate that Sapan is a governance project that does not take sides, but rather works with actors from across the Thai political spectrum. In the highly polarized political environment in which Sapan worked, the emphasis on objectivity was meant to protect both Sapan partners and USAID from accusations that Sapan was interfering in Thai politics, potentially alienating supporters. For some stakeholders, the coup itself had no immediate effect. Seven Sapan partners, primarily media partners in the North, Northeast, and Central regions were affected by the coup (i.e., with interference from or additional scrutiny by Thai government), while five Sapan partners report that the coup had no direct effect on the implementation of Sapan interventions and therefore there were no management shifts noticeable on the part of DAI. According to the latter respondents, Sapan interventions had already ceased or were winding down by mid-2014 as part of Sapan’s scheduled work plan. According to one SGD with Sapan beneficiaries of a university partner, the coup happened “far way” and therefore had no effect on Sapan interventions at the village-level. One partner in the Deep South believed the coup had little effect on Sapan in that region because “we face worse conflict,” meaning that Sapan partners dealt with the violent conflict and military presence in the Deep South more than with anything that happened because of the coup. A DAI respondent confirms that Sapan made no changes to its management approach in the Deep South because the project was working with “strong leaders” who were experienced at working in militarized political situations. EQ 2.1 CONCLUSIONS • Sapan implemented some management shifts in response to the deteriorating political environment, ranging from the discontinuation of interventions under Obj. A to more cautionary work with partners, though it is unclear what effect, if any, these shifts had on attaining program results. • Though some interventions of Sapan partners were affected by the political environment, many Sapan interventions were not affected and no management shifts were made. OVERVIEW OF EQ3 AND EQ4 Though EQ3 (pertaining to capacity outcomes) and EQ4 (pertaining to the sustainability of capacity built) are distinct, in some instances the relationship between capacity building and sustainability is quite close. For example, one media partner made scaling contributions to Thai Public Broadcasting Service (Thai PBS) after Sapan ended—an increase in sustainability of technical skills—due to the production training and equipment received through Sapan—organizational and technical capacity building. 31 Similarly, for sake of clarity, the ET presented the organizational capacity building findings separately from the technical capacity building ones, but there are instances where respondents’ answers testify to the interconnectedness of these two aspects of capacity. A media partner role in providing training courses drew on its organizational capacities built through support from Sapan as it trained other journalists in a range of technical skills related to monitoring, conflict-sensitive reporting, and investigation techniques for fraud and corruption. A media partner in the South said that through Sapan trainings on monitoring tools for transparency and accountability, it realized its internal governance could also be improved in these areas as well. The ET conducted case studies of six Sapan partners to better understand their distinct experiences regarding which capacities they developed and how. The findings for EQ3 and EQ4 below incorporate some comparative themes from across the case studies and the experience of other Sapan partners, though the full individual case studies can be found in Annex A. EVALUATION QUESTION 3: WHAT ARE OBSERVABLE POSITIVE/NEGATIVE CHANGES IN THE CAPACITY OF THE TARGETED GROUPS, I.E., IAS, CSOS, MEDIA, AND CIVIC PEACEBUILDING LEADERS, AS A RESULT OF SAPAN? WHAT ARE THE FACTORS THAT HELPED OR HINDERED SUCH CHANGES? EQ 3 FINDINGS Sapan’s organizational capacity development interventions helped core partners work more professionally, systematically, and strategically. Among the observable changes in organizational capacities, professionalism of CSOs from different sectors and across the regions of Thailand stands out among the findings. Seven out of the 14 core partners interviewed described themselves as being more “professional” as a result of Sapan. Respondents linked professionalism to legal registration in some instances—most partner organizations were categorized as a “People’s Organization” prior to Sapan—and in other cases implied both a change in working style and the establishment of internal policies within organizations. For example, a core media partner in the Northeast reported that working with Sapan raised their management quality from ad hoc “Thai style” to a more formal “international style” with higher standards. The partner stated that staff became more punctual and had more accuracy in terms of paperwork. In some cases, respondents noted that this formalization and professionalism gave them more “credibility” with government officials, implying that these changes were also recognized by outside actors. Sapan partners and beneficiaries increased their ability to operate more systematically and improve their “internal governance,” in the words of a university partner. Partners specifically mentioned having better personnel management, a financial management system, a formal organizational structure, and transparent processes as a result of Sapan. Per the 2011 Sapan Learning Process Final Report, Thai CSOs are often leader-driven and their direction, effectiveness, and sustainability can depend on the characteristics of the leader. One university partner explained that having formalized systems in place ensured that the organization could continue to work in an organized way if their leader left, thus illustrating how capacity building efforts influenced organizational sustainability. Two partners stated that Sapan helped them think more strategically and have long-term goals as an organization. As one partner put it: “Before Sapan, we didn’t have a clear strategic direction or objective. We only had an idea to solve certain issues. Working with the Sapan [we] had an opportunity to brainstorm and come up with a strategic direction, so we see ourselves clearly and what importance we 32 have by being in this location.” Another youth-focused partner in the Deep South claimed they now do better project planning and implementation because of Sapan’s assistance. Some of the factors that were present in organizational capacities building include how trainings were delivered and how useful or applicable the skills were for the organization. For example, DAI responded to feedback that one facilitator for a training was not suitable for the participants by finding better trainers; Sapan partners felt this improved what they learned from the trainings. Partners reported being able to reinforce what they learned in trainings through application and implementation in their organizations. According to respondents, factors that contributed to the success of organizational capacity building interventions also included a leader’s perception of the benefits the organization could derive from applying what they learned. Key factors that influenced the effectiveness of Sapan’s organizational capacity building interventions were the motivation of partner leadership to pursue the CSO capacity scorecard standards, and the perceived benefits or usefulness of the capacity building element. A leader of a youth￾focused organization in the Deep South reported learning the value of transparency after trainings on monitoring tools, and realizing their own organization would be strengthened by practicing transparency internally. Factors that constrained success included the attitude of the organization as persuading staff that they had to set up an internal system was sometimes hard. Sapan partners admitted that one of the challenges to organizational capacity building was partners’ initial resistance or dislike of some CSO capacity scorecard elements. For example, partners spoke about how learning and setting up financial systems was hard at first, though they eventually saw the value of it. Another challenge Sapan had to overcome was the mindset among the CSO community that professionalism is a characteristic of private sector businesses, but not necessarily CSOs that base their work on the “volunteer heart” of their staff and network members. 51 Sapan increased targeted groups’ technical skills to promote governance. In addition to building organizational capacities, respondents from USAID, DAI, and Sapan partners believed that Sapan increased partners’ technical skills to promote good governance within communities and engage both community and government actors. These technical skills and tools proved transferable to settings beyond governance and democracy work. For example, participants in two SGDs with beneficiaries reported learning communication and critical thinking skills that can be applied to a range of contexts and issues. Within the public health sphere, for example, beneficiaries of a women-led partner in the Central region and beneficiaries of a media partner in the Northeast independently reported using the problem-solving and media skills learned from Sapan to deal with issues internal to the community, such as addressing unwanted teen pregnancy. As previously mentioned, beneficiaries of partners in the North and Northeast also talked proudly about engaging administrators and officials at local health promotion hospitals to improve services for the poor and marginalized members of the community who were often made to wait long hours for medical care. Beneficiaries reported having approached administrators to inform them of the long wait and request that they take measures to allow those seeking medical care to have a shorter wait. Respondents attributed the willingness of administrators and officials to their application of peaceful tools and facilitation skills learned through Sapan. Lastly, youth involved in Sapan interventions gained a “new perspective”—according to a university partner—and learned a working approach for all contexts that integrated democracy-building concepts (e.g. participation, 51 Sapan Annual Report 2014, pg. 7 33 respect, living together, being part of a team). One student beneficiary stated: “We learned how to use reason and openness to listen and exchange ideas.” According to interviews with Sapan partners and beneficiaries, several factors contributed to skill growth and tool application, including: (1) the perceived usefulness of the skill or tool, (2) good training delivery and facilitators, (3) partners’ collaborative working relationship with DAI, (4) focus on learning processes for analyzing situations and applying tools after training, and (5) learning from other partners in the Sapan network. As previously mentioned, beneficiaries found the tools to be useful because something like the community scorecard offered a peaceful, non-aggressive approach to advocate and engage with government officials. A DAI stakeholder reiterated this idea, saying that “people came back to tell us the [community] scorecard was very useful,” and that there was positive engagement on the part of communities/CSOs rather than "just submitting complaint letters and protesting.” Partners reported being able to reinforce what they learned in trainings and networks through practice in their organizations, such as a Deep South women-led partner that reported increased capacities of female peacebuilding leaders as they became more experienced working in conflict and militarized zones. These women exhibited a willingness and ability to adapt to the working environment as needed. For example, to get past the suspicion of the police stationed in community, the partner organization invited police to their trainings: "They see we can invite people for trainings. We have influence over community members." Lastly, in terms of learning from other partners in Sapan networks, examples include a small and recently￾established media outlet learned writing and investigative journalism skills by working with a Bangkok￾based media partner. In other instances, Sapan partners learned from other partner types, such as CSOs learning governance tools from universities. Sapan partners and beneficiaries also identified factors that enable them to apply the newly learned skills and tools, such as: (1) the relationship of the organization to the local community and community motivation or experience, (2) the motivation of the organization’s leaders and staff to engage in certain issues, (3) the use of trusted intermediaries such as universities or students, and (4) use of a foreign funding source. On the last point, a Sapan partner clarified that it is easier to use foreign funding sources, such as USAID monies, rather than Thai funding, to support democracy building work, as Thai funders may not have the same understanding or motivation to promote democracy in Thailand. Specifically, in the case of youth, university partners and student beneficiaries noted that selecting youth who were already student leaders and youth who were already motivated to get involved in democracy issues (especially those motivated to work on Deep South conflict issues), positively affected Sapan’s youth interventions. Partners and beneficiaries also noted factors that detracted from technical skill growth or made tool application challenging. Some Sapan partners and beneficiaries reported being selective in how they used tools both during and after Sapan because government officials were resistant to the tools or CSOs feared reprisal. The community scorecard was offered as an example: if an issue was too sensitive or contentious, such as illicit drugs, roadworks, or land disputes, the tools would not be used. Sapan partners of all types and in all regions stated that the “short-term nature” of Sapan support—with certain interventions spanning only 6-months to 1-year at a time—hindered the full development of their technical skills. Respondents expressed a need for more sustained and continuous training and mentoring, followed by time to practice the skills, before being trained further or implementing other interventions. Sapan helped partners widen their understanding of citizen engagement and democracy. Partners in different regions of the country reported changes in how they and their beneficiaries understood the importance of citizen engagement and what it means to engage in democracy. Regarding 34 citizen engagement, a respondent from a media partner in the South said: “The result from [Sapan’s] awareness raising activities that we’ve seen is community people realize the role of their citizenship and learn from election commission more. One thing we see clearly is raising citizen engagement, and providing political knowledge using technology and communication.” In at least three interviews with Sapan partners and three SGDs with beneficiaries, respondents spoke about how people’s understanding of democracy became broadened because of Sapan. A beneficiary and female leader working with a women-led organization in southern Thailand stated: “Having rights and freedom to live—this was our pre-Sapan understanding [of democracy]. After Sapan, we feel stronger in democracy, in [using] governance indicators and in participation in public policy at local level. [I have] more confidence in helping community take part more.” In another example, respondents from another women￾led organization in the Central region stated that prior to Sapan women equated democracy with elections, but after Sapan they now know that democracy also encompasses women’s rights and gender equality. Through Sapan’s interventions, target groups experienced an increase in confidence, empowerment, and leadership. In five of 23 IDIs/GIs with Sapan partners, three of 8 SGDs with beneficiaries, and one IDI with USAID, respondents reported that Sapan positively affected participants’ confidence and sense of empowerment. Although increased confidence and empowerment is not an explicit expected result in the Sapan results framework, it is an important unexpected result because respondents linked these personal changes to their ability to address community issues and engage Thai authorities. For example, the beneficiaries of a media partner in the Northeast reported that community members became more confident in responding to local issues after they participated in Sapan-supported interventions. The respondents highlighted how important it was to them that the media partner used language they could understand. When the beneficiaries approached the media partner there was “instant response using simple terms understood by community members.” Local female leaders—beneficiaries of a women-led organization—stated that the community members they work with developed “courage to express their opinion … to tell government officials what to improve.” Additionally, respondents in the two SGDs conducted with youth beneficiaries spoke about how Democracy School and School of Good Citizenry students felt empowered after learning public speaking skills, including specific anecdotes about quiet colleagues who became more outspoken overall after the course. Although both male and female respondents mentioned confidence building, female respondents spoke of it more often, and mentioned how being involved in capacity strengthening and civic engagement interventions was sometimes felt at a deep and personal level. For example, all women-led organizations and beneficiaries of these organizations who were interviewed gave examples of women undergoing personal empowerment changes, resulting in concrete actions such as speaking up in public or taking on informal or formal leadership positions. Respondents attribute these changes both to information gained (e.g. new knowledge about women’s rights led to a desire to claim those rights) as well as structured opportunities to practice using—and thus become comfortable with—the new skills and governance tools. The words of a beneficiary of a women-led organization capture this: “Democracy means women have rights. We see that women can work the same as men do, we’re not different. We are important in the community. Before, women are the followers, ‘the hind legs of the elephant.' We try to encourage wives in the community to raise their voices.” According to a women-led organization in the North, “After [women] practice … and they see the result that they are able to do it, their confidence increases.” 35 According to respondents, program factors that contributed to an increase in individuals’ confidence included: (1) Sapan trainings that emphasized how to “work as a change leader,” (2) participation in Sapan networks—especially for local female leaders who were beneficiaries of women-led CSOs—where participants could support and encourage each other in their leadership endeavors, (3) stakeholders’ belief in women’s capacity to affect change, and (4) opportunities to apply newfound knowledge and skills and witness results of their efforts. EQ 3 CONCLUSIONS • Sapan increased the organizational and technical capacities of targeted groups, as well as their understanding of democracy and citizen engagement. • The tangible skills participants acquired proved transferable to settings beyond governance and democracy work. • Respondents, especially women, reported an increase sense of empowerment and confidence as a result of participating in Sapan interventions. Q3.1: IS THERE AN OBSERVABLE DIFFERENCE BETWEEN THE INVOLVEMENT OF AND IMPACT ON MEN AND WOMEN? 52 EQ 3.1 FINDINGS Sapan partners incorporated the USAID policy on Gender Equality and Female Empowerment through sex disaggregation of data and including both men and women participants in Sapan interventions. However, Sapan interventions did not address lesbian, gay, bisexual, and transgender (LGBT) issues. By the second and third years of implementation, USAID’s Policy on Gender Equality and Female Empowerment had been communicated to partners, and annual reports from 2012 onward noted that Sapan's own management was following the policies. The CSO capacity scorecard included gender and diversity indicators, and Sapan reporting to USAID required disaggregation of data according to sex. In qualitative interviews, Sapan partner respondents confirmed that they needed to try to have a gender balance in Sapan interventions, although Sapan did not impose quotas for male and female participants. Partners also confirmed that both men and women did participant in Sapan events, though the nature of their participation sometimes varied (see below finding). Though Sapan emphasized equal participation of men and women, few Sapan interventions focused on LGBT issues or intentionally encouraged LGBT participation. Through interviews with Sapan partners, the ET learned of only two interventions under Sapan that had an LGBT component or participation, and these were partner-driven: one partner in the North hosted public discussion forums around LGBT rights, and a women-led organization in the Deep South led a project during the last year of Sapan that aimed to increase communities’ acceptance of LGBT youth (the partner stated that the project ended once Sapan funding was discontinued). When asked about the fact that interventions around sexual identity and LGBT issues were absent from Sapan, a DAI stakeholder said that these were urban issues and did not concern the rural areas. One gender equality and democracy bellwether stated that younger Thais regard sexual identity as an important component of gender equality and are not drawn to organizations that do not prioritize sexual orientation and gender identity issues. Qualitative interviews reveal that most of the 52 Sub-question proposed by the ET to ensure gender dimensions are appropriately considered. 36 Sapan partner women-led organizations were led by women aged 40 years and above. While some beneficiaries were younger than this, the situation of women-led organizations in Thailand is a more acute illustration of what was observed by a DAI stakeholder commenting on CSOs more widely in Thailand: that younger people—some of whom are more likely to take up sexual identity issues than older Thais— are not going into CSO work. For some interventions, men and women differed in the nature of their participation. Though men are the majority in formal leadership positions, some stakeholders noted increasing numbers of women in these positions. When the ET queried about which sex participated more in Sapan interventions, Sapan partners across types overwhelmingly stated that more women than men participated. According to these respondents, men have more participation in official meetings (e.g., meetings with university representatives), but more women are engaged in action, implementation, and data collection on the ground. Respondents offered different explanations for why this is the case, such as how women stay in the communities to work while men travel outside of the communities and are not as available, and that in some regions women get involved because it is more dangerous (in terms of government targeting and harassment) for men to do so. According to a women-led partner in the Deep South: “We are softer. We are mothers. Find different ways to compromise. Women have become leaders. [They] can use their rights, negotiate with officials.” However, a beneficiary of one core partner told the ET: “Women participate more, [but I’m] not sure if women’s voices are really heard.” Students who participated in SGDs state that more male than female students participated in “Democracy School” and “School of Good Citizenry” courses. One women-led organization in the Central region stated that, in their experience, the Sapan interventions in which men were interested included Action-Research, WDF, and preventing unwanted pregnancy. Respondents from Sapan partners and beneficiaries disagreed on whether the sex of a facilitator, leader, or CSO staff was an influential factor in project success. Some respondents believed that government officials were more receptive to working with women rather than men, while other respondents said that some community members accept female facilitators while others do not. In the case of universities, respondents spoke about how a researcher’s language ability, age, or socio-economic status (rather than a researcher’s gender) had more influence on whether that person could establish rapport with local community members. Qualitative interviews with Sapan partners reveal instances of women moving into formal leadership positions, although these respondents also say that there are still more men than women in formal positions. Promising instances include women in communities becoming the assistant to the village chief as well as a president of a women-led organization who was considering running for public office. Sapan's work with women-led organizations reflected the programmatic importance placed on strengthening the role of women in governance. Many women play active role in communities though they may be less concerned with legal and constitutional rights. In 2015, Sapan commissioned a "Needs Assessment for Women’s Participation in Local Governance in Thailand." The assessment notes that “it is doubtful that most Thai women think that constitutional provisions are the answer to alleviating their most urgent problems." 53 The report further clarifies that the inactivity of the formal women’s movement does not, however, mean women are not active and engaged at the local level. The report asserts that many individuals and organizations are actively engaged in assisting victims of violence, providing education and capacity for women in micro‐economic areas, supporting reconciliation efforts, and providing local 53 Needs Assessment for Women’s Participation in Local Governance in Thailand, pg. 5 37 services to women-headed families. This claim was confirmed by two bellwethers, one in the North and one in the South, who said that women have long existed as leaders in informal settings. Figure 8 below suggests that the percentage of women who hold elected seats in national parliament is growing. However, what this trend might mean for the advancement of women's well-being in democracy and peacebuilding is not clear. In the words of one policy expert based in Northern Thailand but with extensive research experience in the Deep South, "Women are gaining more formal positions in government, but I'm not sure if their involvement is really changing policies. These are quantitative indicators, but we also need data looking at the substance of their policies." Figure 8: Percentage of Seats Held by Women in National Parliament in Thailand (Source: World Bank) Women’s participation in Sapan interventions and the space for women involved in peace and governance work in the Deep South was shaped by religious norms. In the three southern provinces, referred to as the Deep South, women’s participation in Sapan-sponsored interventions was more influenced by religious norms than in other parts of the country. Muslim norms sometimes discouraged young women from taking part in events (such as the Democracy School) where they would be attending in the company of men they did not know. In some instances, male Muslim youth were resistant to female youth leaders: a youth programs in high schools had to have sex-segregated elections for student officers. As a bellwether stated, there is a reluctance on the part of women in the Deep South to pursue gender equality under the banner of feminism: "women in Deep South say they work on women’s issues, but don’t call themselves “feminists” as they see it as a western idea." The sociocultural character of Deep South influenced whether young men and women could participate in the same way in some Sapan interventions. But the landscape is not uniform. Religious-influenced gender roles shaped women’s participation in other Sapan interventions, though not always. A beneficiary of a core partner in the South felt that the work of this organization had indirect effects on gender equality, since the public discussion forums sponsored by this organization talked about human rights and democracy. In some cases, women were freer to take part in some sensitive Sapan governance interventions than men because a militarized environment forces men into background roles (e.g. working 5.6 5.6 5.6 4.8 9.2 9.2 9.2 8.8 10.6 8.7 11.7 11.7 13.3 13.3 15.8 15.8 15.8 4 6 8 10 12 14 16 18 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Percentage (%) Year 38 in high conflict zones). Here is how a core partner women-led organization described their work in the 'red zones' (most dangerous areas) of the Deep South: "We were among only groups allowed to enter and be accepted. Initially the community members dare not participate. We made house visits, said we wanted to talk with them about the effects of violence. Even if there was a bombing the day before, we would still go ahead with the forum." Though they have faced many years of violence in the Southern provinces, some women partners took up the domestic violence issue when asked whether their work had contributed to reducing violent conflict. "We’re having an effect on domestic violence. The community sees that these women [members of a Deep South women’s organization] are in the area working on childcare center and that might help indirectly. Then those who participate in Sapan activities start talking to each other and that might be helping reduce domestic violence." EQ 3.1 CONCLUSIONS: • USAID’s Policy on Gender Equality and Female Empowerment was communicated and institutionalized by partners at the output level, through recognizing the need to include both men and women in Sapan interventions. • There was an increase in women’s leadership roles in informal settings that respondents reported as due to Sapan. An increase in women’s leadership roles in formal roles was reported but unconfirmed. • The ability of women in the Deep South to take part in Sapan-supported interventions was shaped by the socio-cultural character of the Deep South: Muslim social norms affected the movement of women, especially unaccompanied, unmarried women. EVALUATION QUESTION 4. IF ANY, WHAT AND TO WHAT EXTENT DID THE INCREASED CAPACITY OF THE IAS, CSOS, MEDIA, AND CIVIC PEACEBUILDING LEADERS AS A RESULT OF SAPAN STILL REMAIN AND SEEM LIKELY TO REMAIN IN THE FUTURE? WHAT ARE KEY SUPPORTING FACTORS TO SUSTAIN SUCH CAPACITY? EQ 4 FINDINGS Sapan partners are still using some of the organizational development skills learned, but gaps in organizational capacity still exist. All 15 core partners, except for Youth Cares, were still in existence one year after Sapan, and core partner respondents state that they continue to practice elements learned during organizational capacity building, for example financial management. In another example, as a result of improving their fundraising skills through Sapan, a women-led organization in the North showed the ET a recently completed proposal that they planned to submit to a Thai agency to request funding from the Gender Equality Act—a piece of legislation approved in 2014. They noted that their work with Sapan encouraged them to feel confident and able to write the proposal. Some factors contributing to Sapan partners’ sustainability included formalization of organizations through legalization and the benefits that the organizations experienced from the organizational capacity improvements in things such as financial management. But as the CSO capacity scorecard results below reveal, some skills were still somewhat weak by the time Sapan finished. The top three standards not met including fundraising strategy, monitoring and assessment system, and annual budgeting process. The ET 39 did not assess whether the CSO capacity scorecard results were still valid at the time of the evaluation. Only one organization (Youth Cares) had not received legal registration by the end of Sapan (see Table 10). A DAI stakeholder noted that Sapan did not force any organization to implement specific systems as laid out in the CSO capacity scorecard. Rather, Sapan engaged in discussions with organization regarding the benefits of organizational improvement, and left it up to the partners themselves to make changes with support from Sapan. Most organizations saw the benefits to organization improvement and thus worked toward increasing their scores, whereas other organizations chose not to pursue certain elements of the scorecard. Regardless of organizations’ individual progress, the DAI stakeholder emphasized that one of the successes of the CSO capacity scorecard was that it allowed CSOs to consider their strengths and weaknesses. Table 10: CSO Capacity Scorecard Results (Source: Sapan Final Report) Organization Baseline Score End Line Score Scorecard Standard Not Met at End Line FCEM/Prachatai 3 15 None Friends of Women 7 15 None Luukrieng 4 15 None Nature Care Foundation 2 15 None OHSD 4 15 None WePeace 1 15 None Media for Happiness 2 14 Organizational structure WeSD 5 14 Fundraising strategy Media Selatan 1 13 Fundraising strategy; monitoring & assessment system Muslim Women’s Association 2 12 Fundraising strategy; finance policies; procurement policies Prachathum 2 12 Fundraising strategy; organizational structure; annual budgeting process Pattani Forum 2 10 Fundraising strategy; monitoring & assessment system; annual budgeting process; procurement policies; administrative policies Voluntary Women’s Group 2 9 Fundraising strategy; monitoring & assessment system; annual budgeting process; procurement policies; personnel policies; public outreach Youth Cares 1 8 Fundraising strategy; monitoring & assessment system; annual budgeting process; finance policies; personnel policies; administrative policies; legal registration Sri Song Kwai 1 8 Fundraising strategy; monitoring & assessment system; annual budgeting process; organizational structure; personnel policies; procurement policies; administrative policies 40 The political environment in Thailand since 2014 has not been conducive to the advancement of the technical work of the Sapan partners. Political environment is a crucial factor that helps determine the capacities built as well as the sustainability of organizations and the skills individuals learned from Sapan. A bellwether felt that the current military government has succeeded in weakening CSOs overall in Thailand and that civil society is a victim of the ‘lack of a judicial system’ in the country. A lecturer from a university partner in the North said, "You couldn't do anything after the coup using the words democracy or governance." Sustainability of partners’ interventions has been affected in various ways by the coup in May 2014 and by the ending of Sapan itself which occurred, in some instances, not long after important interventions had been introduced. The Bangkok-based media partner reported how security forces came to their offices and that military and police would come to activities they held, especially if the topic was related to human rights and democracy. In July 2016, a journalist with a media organization was arrested in the lead-up to the constitutional referendum (held in August 2016) for violating Article 61 Clause 2 of the Referendum Act. News coverage of the event described how activists from the New Democracy Movement travelled to a police station to support red shirt villagers who had been summoned for opening a referendum monitoring center. The villagers were accused of unlawful assembly and of a political gathering involving more than five people, an action that violates National Council for Peace and Order, Head Order 3/2558. The journalist published an account of his arrest, stating that “an official claimed that there was reason to suspect that I was participating in the activist’s activities since I was riding in the same car, even though I explained that I was a journalist.” 54,55 The case against the journalist is still proceeding as of the writing of this evaluation report. In 2014 Sapan helped its women partners establish a Thai Women Coalition with seven women-led CSOs to increase participation of women in governance and policy making. 56 The ET found evidence of informal relationships across this coalition still being maintained, as in the case of women led organizations in the Deep South continuing to talk with others in the coalition from other regions. But without financial support, it was not possible for them to attend events or participate in activities of members of the Coalition. Many interventions initiated during Sapan did not continue post-Sapan, although some did remain. For example, a youth organization in the Deep South continued to organize discussion forums and produce newsletters. Stakeholders cite evidence of continuing to use some governance tools in local governance related to service delivery (such as the community scorecard and the people’s charter). Another youth organization in the Deep South continue to use tools learned from Sapan, though they have been adapted (e.g. “community analysis” is the name given to the community scorecard tool). This same group reported that, because of limited financial resources, after Sapan ended they had to change some of their interventions and reduce the range of people they could include. They were also more limited in the places they can cover—not much farther than 30-40 kilometers. Although the partner did not elaborate on this point, in the view of one university partner in the North, however, some governance tools were too complex to be learned and used in a sustainable way. 54 ABC News Australia. Thai journalist charged under draconian charter law. 11 July 2016. Accessed from: http://www.abc.net.au/news/2016-07-11/thai-police-detain-four-for-opposing-junta-draft-constitution/7587096 55 Human Rights Watch. Thailand: Activists, Journalists Arrested for Vote-No Campaign. 12 July 2016. Accessed from: https://www.hrw.org/news/2016/07/12/thailand-activists-journalist-arrested-vote-no-campaign 56 Sapan Final Report, pg. 23 41 EQ 4 CONCLUSION • Although many interventions ceased after the project, capacities related to personal empowerment, organizational capacities and technical skills persisted to varying degrees. EVALUATION QUESTION 5: WHAT POLICY CHANGES, DURING OR AFTER THE LIFE OF SAPAN, ARE OBSERVABLE AS A RESULT OF SAPAN? EQ 5 FINDINGS Though changing policy was not one of its core objectives, Sapan partners did attempt to address some policy issues, with few cases of success. The effect of the Sapan project on policy change regarding IAs, CSOs and peacebuilding leaders and groups in the Deep South is not easy to discern. Public policy appears as an expected result in Obj. A, Obj. B, and Obj. C, but Sapan interventions reveal little that is particular to public policy. In the case of Obj. B, some interventions in the first few years of Sapan aimed at capacity building were linked with media coverage of policy rather than advocacy about media policy. Journalists received training in 2011 and 2012 to enable them to cover both policy and conflict. The statement “Good advocacy compels policy makers to listen and to act”57 reveals expectations for and an understanding of policy change that is integral to project’s objectives, yet DAI does not describe Sapan as having a policy change objective. One USAID respondent distinguished between wanting to change policy and wanting to implement it stating that Sapan did not set out to change government policies; rather, the project focused on the implementation of policy. Although policy change was not core objective of Sapan, CSOs in their checks and balances and monitoring role were to engage the government on policy advocacy. CSOs in different sectors primarily used tools to engage government at the community level mainly for service delivery, not acting as a check on political processes and public policy. Policy change action aimed at national-level policy, occurred in response to opportunities seized by both partners and DAI, with government and non-governmental actors. One notable example was the advocacy trainings of women-led organizations to equip them with the skills to advocate for the establishment of the WDF. In the case of a core partner working with women-led organizations, there were some findings of Sapan-supported interventions being directed at national policy change. One concerned female investigators to deal with cases of domestic violence. The leaders of a women-led organization were proud to be able to say that now there are 500 female investigators and that currently the police cadet school receives 77 female applicants per year. This same women-led organization has advocated for a change in government policy that would mandate a quota for women in local administration positions, but to date they have not been successful. A Sapan media partner in the North built capacity of NGO networks in Chiang Mai to push local issues to the policy level (e.g. mass transit). In the South, another partner worked with local and national media to have policy on improved reporting on the Deep South. There was also informal policy change under the initiative of a partner that drew on governance tools learned to respond to a community whose ability to fish had been lost when laws were introduced that 57 Sapan Annual Report 2013, pg. 13 42 restricted access to areas that had been used for fishing in the past. The local authorities allowed local fishing techniques even though they are against the law. EQ 5 CONCLUSIONS 1. Although public policy change was not a core objective of Sapan, CSOs were meant to engage public policy in their “checks and balance” role. There was a major policy intervention consisting of a national-level advocacy campaign involving five core partners that focused on policies associated with the WDF. 2. Policy change action, when it occurred, was in response to opportunities seized by both partners and DAI. 43 IV. RECOMMENDATIONS EVALUATION QUESTIONS 1, 1.1, 1.2 A. In future design of similar projects, USAID should consider working with government but focus on local level administration in addition to central government (keeping in mind the level of centralization/decentralization in the country of implementation). These projects should be designed collaboratively with the government entity to facilitate buy-in and shared understanding of objectives. B. Within Thailand, other USAID projects and technical sectors may consider using local media outlets and universities as intermediaries with CSOs and communities. C. Future civil society engagement programs may consider engaging other parts of civil society, such as the private sector, and not just CSOs. EVALUATION QUESTIONS 2, 2.1 A. USAID can consider use of contracts, rather than a cooperative agreements, in sensitive operating environments to exert more control over program activities, though only after evaluating the pros and cons of different mechanisms and making sure it is a good fit. This type of contract is best paired with a flexible and consultative management style. Flexible management should provide space for local partners to propose new interventions not included in the original program design. B. Based on the positive effect of Sapan working with organizations across the Thai political spectrum, future USAID governance projects in divisive/polarized environments should work with both sides of political spectrum and emphasize objectivity. C. USAID projects that introduce tools (governance or otherwise) to CSOs or other local partners, should focus on tool application rather than lecture and should be paired with mentoring and feedback from the implementing partner. D. USAID projects that incorporate organizational capacity building work with local organizations should use a capacity building scorecard tailored to the country context as a framework for encouraging and assessing progress. E. USAID projects that include any type of capacity building (organizational or technical) should build in opportunities for partners to network, build relationships, and learn from one another. EVALUATION QUESTIONS 3, 3.1 A. In future projects, capacity building that strengthens the professionalism of CSOs should be promoted, since the professional skills learned and put into practice are highly valued by the organizations. In particular, planning for the transfer of organizational leadership could help prevent the over￾dependence on founders and leaders that hurts many CSOs. B. USAID and implementing partners should consider identifying trusted local intermediaries (e.g. Sapan’s use of universities) to help with the delivery of trainings as these can be vital to bridging resources between management and the CSOs and their beneficiaries. C. USAID and implementing partners should develop or utilize pre-existing technical tools, such as community scorecards, that reflect the needs of the organization/community and are suitable to the skillsets of the intended users. D. Because Sapan partners used governance tools effectively in a range of sectors (e.g. health, social welfare), USAID programs in other sectors may consider using public discussion forums, community scorecards, Strengths, Weakness, Opportunities, and Threats (SWOT) analysis, and other easy-to￾learn tools to tackle governance related obstacles or corruption in technical sectors. E. Future USAID capacity building interventions and trainings should focus on developing participants’ soft skills, such as leadership, confidence building, and public speaking. These skills are especially relevant for women-led organizations. 44 F. Bellwether recommendations for further work to be done in Thailand include a focus on civil society strengthening programming in Thailand to support critical thinking and civic mindsets, and to prepare the younger generation for when civil society is less threatened in the country. EVALUATION QUESTION 4 A. To increase the sustainability of technical skills, USAID and implementing partners should allow sufficient time for local organizations to practice and implement newly learned skills so that the organization can internalize them sufficiently to be able to use them. B. USAID and implementing partners should develop interventions and skills trainings that focus on issues of relevance to the community's wellbeing. C. USAID should identify partners with high personal or organizational motivation to continue with their work, even at a reduced level, when donor assistance is no longer available. D. USAID and implementing partners should introduce fundraising skills to capacity building interventions as early as possible, because continuity of work depends upon the organization's ability to get external support. EVALUATION QUESTION 5 A. If public policy change is to be an objective of a USAID activity or project, the design needs to clearly identify what level of policy is to be affected, what is highest priority for type of change, and who are the most suitable project partners. B. If USAID wants implementing partners to make an impact on public policy, stakeholder mapping skills, advocacy skills and planning for campaigns to influence policy makers and processes need to be introduced to partners early on and supported for an extended period of time. 45 ANNEX A: CASE STUDIES CASE STUDY Association of Muslim Women (Nakhon Si Thammarat) The Association of Muslim Women (AMW) of Nakhon Si Thammarat (NST) formally registered as a CSO in 2004 and began working with Sapan in 2013 on the Engaging Citizen in Governance intervention. To improve women’s political participation, AMW conducted trainings and public discussion forums for local participants. AMW later joined the Thai Women’s Coalition established by Sapan in 2014 to further increase participation of women in governance and policy making. AMW is one of the first organizations of Muslim women in the Deep South to work on development issues, and is now growing to work with the Federation of Associations of Muslim Women, a group with nationwide membership. CAPACITY BUILDING: AMW members stated that they learned and used several governance tools, including the community scorecard, citizen charter (which the community renamed sanya jai - “Heart Commitment”), social audits, and how to conduct public discussion forums. They liked the citizen charter because “t’s a tool we can use for claiming things from the government that is not against the law.” According to one AMW respondent, AMW’s work with Sapan helped people “find agreement with government and the community on what is missing” because AMW “acts as a mediator between the two to ask for better service.” Another key contribution of AMW is the role it played in enhancing people’s understanding of and confidence to do democracy-building work. As one AMW respondent noted, “Having rights and freedom to live - this was the pre-Sapan understanding [of democracy]. After, we feel stronger in democracy … in participation in public policy at the local level. [We are] more confident in helping the community take part [in democracy]. Sub-district level officials have to adjust themselves in response to [Sapan’s] community scorecard interventions. On the grassroots’ level, communities changed their understanding of democracy.” Many AMW respondents said that local people used to see democracy in terms of their rights and entitlements, but now people are more aware of their roles and responsibilities to participate as part of the democratic process. More importantly, communities are “more confident to use public discussion forums and to express themselves in meetings.” AMW respondents believed that they are now stronger in terms of working with local government and communities. With regards to organizational development, AMW was proud of their new financial management and procurement skills, though funding remains a challenge. CSO Capacity Scorecard (out of 15): Baseline Score – 1; End line Score – 12 ENABLING/CONSTRAINING FACTORS: AMW respondents said that having local volunteers and an available budget through Sapan helped their work. On the other hand, AMW had difficulty with Sapan’s detailed paperwork requirements. Working with government officials was also difficult at first. For instance, one respondent stated that to ensure their own safety during working with a government official, they had to “assure the official that AMW would not inspect him, and to clarify what is missing in the area and how working with Sapan would be beneficial to him.” SUSTAINABILITY: Aspects of Sapan’s efforts remain. According to one AMW member: “We built trust. We can work with people from other religions, and other local government people were surprised since they never saw any Muslim organization working with other religions.” Respondents stated that they worked with local government to improve a child care center and that the improvements are still in practice after Sapan concluded. They also stated they keep in touch with other Sapan partners via a popular online messenger service called LINE. They use this platform to share information about what different organizations are doing so others who are interested can implement these ideas elsewhere. 46 CASE STUDY Media for Happiness (Ubon Ratchathani) Media for Happiness (MfH) began working with Sapan in 2011 to strengthen community capacity in idea sharing. Subsequent Sapan interventions focused on using media to promote citizen involvement in oversight of local government, and greater civic participation in governance. MfH started working as a people’s organization in 1990 and registered in 2007. CAPACITY BUILDING: MfH staff and local community members who participated in Sapan interventions reported that communities’ willingness and capacity to engage with government have changed. Through its Sapan-supported “Life￾shared” intervention (60-minute democracy and governance TV programs), MfH broadcasted local issues such as land rights and flooding so that these issues became widely known by the public, and promoted a televised space for discussion between citizens, government officials, academics, and other experts. One interviewee pointed out that through this idea-sharing platform people became more confident in making demands of government. Additionally, MfH trained community members on different communication techniques (e.g., documentaries) to promote issues relevant to their villages. Community members stated that operating their own media is an effective way to express their opinion. For example, communities living along the river would become isolated during the flooding times. Without access to community media it was hard for government officials and other outsiders to learn the real situation inside the flooded areas and figure out how to solve problems. In discussing media’s importance, one community member commented, “It is hard for us to access justice, but it is easy for us to access media.” MfH’s organizational capacity increased as well. Because of Sapan’s capacity building support, MfH staff stated that their management style is now “more international and with more quality.” Examples included being a registered foundation, being punctual, and having more accurate financial paperwork. CSO Capacity Scorecard (out of 15): Baseline Score - 1; End line Score – 14. ENABLING/CONSTRAINING FACTORS: Many MfH beneficiary respondents considered the non-profit nature of MfH as one of the contributing factors to its success, as “communit[ies] recognized that MfH wasn’t working for its own benefit; they worked with the community and continued to follow up on the issues.” MfH also appreciated that they were able to write intervention proposals for Sapan in Thai, rather than in English. Though MfH described its collaboration with DAI as generally very good, early changes in DAI personnel and unclear standards of financial management affected capacity building at first. SUSTAINABILITY: MfH staff said that they still use the financial management system as well as the physical infrastructure received from Sapan. Sapan interventions stopped once Sapan funding ended, but as a result of increased organizational and technical capacities MfH is now working with Thai PBS to re-establish community TV programming. MfH is currently receiving funding from the government-issued Health Development Fund, but fundraising difficulties remain a large threat to MfH’s sustainability. Sapan-supported Interventions (Period) Life Shared, Year 1 (December 2011 – June 2012) Life Shared, Year 2 (December 2012 – December 2013) Public Hearing Before Election (Feb 2014 – April 2014) Community TV (March 2014 – November 2014) Equipment Provision (May 2015 – July 2015) 47 CASE STUDY Sri Song Kwai Women Ordinary Partnership (Phitsanulok) Sri Song Kwai (SSK) was formed in 1995 and began working with Sapan in 2012. SSK collaborated with Friends of Women, another Sapan partner, to monitor the Women’s Development Fund (WDF), and in late 2014 became part of Sapan’s Thai Women’s Coalition. SSK additionally implemented Sapan interventions to raise awareness on local governance issues. SSK works in six sub-districts of Phitsanulok, and each sub-district has a cadre of five volunteer women leaders. CAPACITY BUILDING: SSK members and volunteer women leaders noted that they learned and used several governance tools, including the community scorecard, public discussion forums, social audits, direct observation, walking map, budget literacy and monitoring, citizen charters, and others. Women leaders successfully used these tools in different communities to bring changes to a local health-promotion hospital and stop a private pig farm from polluting a water source. Respondents stated that Sapan interventions widened their understanding of democracy. As one respondent noted, “Most of the time [people] think that governance is for the government, not the citizens, and now we realize that all the money from the government is from the citizens so we have to be involved.” Women leaders also expressed a change in their understanding of gender dynamics, and increase in their confidence, particularly with regards to public speaking: “[There were] older ideas, [such as] men in front and women follow, but after Sapan women don’t walk behind, they walk together,” and “We’re housewives [and] didn’t have opportunity to do public speaking. [We] used to shake with the microphone in hand, but now we can do it.” In terms of organizational capacity building, SSK members believed they are stronger and more “formal” since they became formally registered in 2012 (with support from Sapan) and have association rules and regulations. CSO Capacity Scorecard (out of 15): Baseline Score - 1; End line Score – 8 ENABLING/CONSTRAINING FACTORS: One of the enabling factors for building the capacities of women leaders was the opportunity to meet other women leaders, creating a support network as they implemented governance tools in their own locales. Respondents have mixed opinions about working with government officials. One SSK member believed that SSK’s longstanding, pre-Sapan relationships with government officials enabled Sapan’s work. Another women leader complained that the government was only interested in “giving people fish instead of teaching people how to fish,” and that the government was willing to address “only certain issues, not the sensitive ones such as monitoring budgets for roads.” SUSTAINABILITY: SSK members and women leaders commented that they still use Strengths, Weakness, Opportunities, and Threats (SWOT) analysis, problem tree analysis, and especially community scorecards in their work, though one respondent caveated, “[Women leaders] are not using the tools often now but they know how to use it.” The current political environment in which CSOs are under more government scrutiny and tend to self-censor makes it difficult to use the tools. Women leaders stated that their confidence, leadership, and management skills remained after Sapan ended, and that people still organize public discussion forums to address local problems. During Sapan, SSK had adequate funding to support staffing during Sapan, but lost their staff after Sapan concluded. SSK said their sustainability is also affected by their lack of young members. Current SSK members (administrators and women leaders) are all volunteers, and because SSK cannot afford to offer compensation, the organization has trouble attracting students or younger employees who would rather be paid for their work. 48 CASE STUDY Foundation for Community Educational Media/Prachatai (Bangkok) Foundation for Community Educational Media (FCEM)/Prachatai, established in 2004, is one of the four CSO training service providers brought together through Sapan. FCEM/Prachatai aims to provide professional media coverage and responsible analysis of public interest issues in Thailand and to enhance linkages between media and the public and private sectors. During Sapan, FCEM/Prachatai conducted a series of training courses for journalists to improve their capacity to use multimedia techniques to promote democracy. FCEM/Prachatai provided various training and courses including, such as Fellowships for Journalists (10 participants), Conflict-Sensitive Reporting (21 participants), and Investigating Fraud and Corruption (37 participants). FCEM/Prachatai conducted a series of training courses for 15 of their affiliate journalists to improve their capacity to use multimedia techniques to promote democracy. It also conducted media conferences to promote an open media environment, and trained youth on media literacy and journalism design. In early 2014, with Sapan’s help, FCEM/Prachatai opened the Media Learning Center (MLC) as a media training resource. CAPACITY BUILDING: As a key capacity-building partner, FCEM/Prachatai played an important role in providing technical support to other Sapan-affiliated media partners, and connecting media partners in the Deep South with other media groups located outside of the Deep South through trainings and other networking events. A media partner in the Deep South stated, “We can cover 50,000 people before, but [FCEM/Prachatai] helped expand [our] coverage…the [government’s] harassment towards local media [in the Deep South] reduced as well.” Media partners also comment that FCEM/Prachatai taught them strategies on how to report news effectively without being shut down by the government. In addition to building the technical capacities of media partners, FCEM/Prachatai received organizational capacity-building support from Sapan. As a result, FCEM/Prachatai stakeholders said they are more professional and able to “seek funding from other funders with more credibility.” FCEM/Prachatai staff also reported that Sapan helped them think more strategically and make their own plan as an organization. CSO Capacity Scorecard (out of 15): Baseline Score - 3; End line Score – 15. ENABLING/CONSTRAINING FACTORS: With Sapan’s support in September 2014, FCEM/Prachatai reconstructed and refurbished their offices, including new space for instruction and seminars. The provision of these physical assets helped FCEM/Prachatai grow. Threats from the military (e.g. journalists being monitored and arrested for alleged participation in activist activities) and insufficient financial support now limit the sustainability of FCEM/Prachatai’s programming and sustainability. The internal motivation of FCEM staff in the face of these challenges is an internal enabling factor. One FCEM/Prachatai staff stated, “If we didn’t do it, who would do it?” SUSTAINABILITY: FCEM/Prachatai continues to operate a program called Prachatai News and plans to grow its presence with the launch of Prachatai WEB TV. However, FCEM/Prachatai staff said that the MLC has not really been active after Sapan due to the withdrawal of Sapan funding. Sapan media partners around the country stated that they are still part of the media network with FCEM/Prachatai and some have contributed stories to Prachatai News. FCEM/Prachatai staff said they are still using the tools and systems learned during Sapan’s organizational capacity building process. 49 CASE STUDY Center for Civil Society and Non-Profit Management (Khon Kaen) With Sapan’s help, Khon Kaen University launched the Center for Civil Society and Non-Profit Management (CSNM) in late 2013 to support civil society leaders and nonprofit organizations in building managerial capacity in Thailand and Southeast Asia. CAPACITY BUILDING: During Sapan, CSNM served as a platform for community members to gain knowledge and hold discussions on policy issues. For example, community leaders interviewed described a land use problem in which the government ordered communities to vacate national park lands that the communities had been occupying for a long time. Community members received knowledge from CSNM about how they could communicate with government to address this issue. CSNM staff said that learning governance tools empowered marginalized people to know their rights and strengthened their capacity to monitor governance work. These respondents also said that Sapan interventions benefited young people because youths had not previously been given the opportunity to initiate something for the community. Leadership and support from Sapan enhanced the role of young people in raising awareness among communities, monitoring government work, and participating in local media. In 2014, CSNM’s organizational capacity was high enough to receive direct funding from USAID ($1.44 million grant for a 3-year period). This funding helped CSNM launch the Center’s Civil Society Partnerships Project, which allowed CSOs and experts in Thailand, Vietnam, Cambodia, Lao PDR, and Burma to come together and learn from one another about the challenges of managing CSOs. CSO Capacity Scorecard (out of 15): Not applicable. CSNM was formed through the assistance of the project toward the end of the contract period and did not exist early enough to participate in CSO Capacity Scorecard assessments. ENABLING/CONSTRAINING FACTORS: One of CSNM challenges was and continues to be conflicting ideology with CSOs about the purpose of nonprofit management. According to CSNM staff, some people believed “that if you become a professional working with NGOs, you are selling out and becoming a capitalist,” whereas other people believed that “problems in the Northeast are because of weak civil society compared to government and the private sector.” Resistance and suspicion from government officials was also a challenge according to CSNM staff: “When we got new tools from Sapan, like the Provincial Governance Index, we needed to explain to the officials and invite them to participate. Government officials think it’s an inspection program, so we have to spend more time to do constructive communication.” SUSTAINABILITY: CSNM continues to operate management courses, though organizations must approach CSNM to order one. With regards to capacity building of youth, one respondent reported that these young people “still continue their support through local administration and local media.” He believed that the outcome has contributed greatly to the engagement of youth. 50 CASE STUDY People’s College (Pattani) People’s College was established in 2010 by a group of young civil society activists. In 2015 Sapan formed a network of four CSO capacity-building service providers, including People’s College, CSNM, FCEM/Prachatai, and Café Democracy. During Sapan, People’s College implemented the “School of Good Citizenry” – courses for youth on good governance, active citizenry, and peacebuilding. CAPACITY BUILDING: Youths interviewed who took part in the School of Good Citizenry believe that Sapan built their critical thinking and structural analysis skills: “We learned theories and practice that we can apply in our real life … We learned how to conduct community analysis, know who’s who, and it gives a better impact on selecting activities to satisfy the needs.” Students from the School then worked with communities and local government officials, using tools like the community scorecard and public discussion forum. According to the youth respondents, local government officials were impressed with youths’ participation and increased capacity, and asked to send youths from their areas to the School of Good Citizenry. However, the school is no longer open due to lack of budget. But the People’s College staff reported that some members of the army in the area wanted to enroll in some of the classes and even proposed creating courses jointly. But at the time of the evaluation, nothing had come from those proposals. These students also took part in advanced courses at the School. People’s College staff said that the advanced course focused on how to work with CSOs and claimed it was so popular that the students nicknamed it “NGO School.” Youth respondents also stated that participation in the School increased youths’ confidence and changed their understanding of democracy: “[I] saw engagement as voting. But after Sapan [I] understand that engagement is not only as voting, but also being a part of the community development process.” CSO Capacity Scorecard (out of 15): Not applicable. People’s College was formed through the assistance of the project and did not exist early enough to participate in CSO Capacity Scorecard assessments. ENABLING/CONSTRAINING FACTORS: Youth respondents stated that an enabling factor in Sapan was using youth facilitators to promote governance work is. They believed youth are effective facilitators due to their level of educational attainment relative to others: “In the community, we can see that those who possess knowledge are youth because we have more education than our parents. Parents aren’t highly educated, so for [facilitation] tasks it’s better to use those of a certain education level.” Another enabling factor was youths’ perceived neutrality as students: “Youth work better because we are in [student] uniform and can gain better access to the community because we are wearing university uniforms – it’s more acceptable.” Another respondent clarified that there are a lot of conflicts in communities, and students are not regarded by community members as people who takes sides in the conflict situation. SUSTAINABILITY: People’s College staff said they found it difficult to continue some of Sapan’s interventions after funding ended. They said that they still run interventions similar to what was being done under the School of Good Citizenry (they now call them “Grassroots Schools”). However, they are more limited now in their geographic reach and number of students: “We invite youth leaders from communities. We don’t have a budget, but participants. Participants come from the three [Deep South] provinces, there. There is no charge [for the school] so they just pay for their transport expenses. We ask our own alumni to help serve as facilitators. It is challenging because, as it is voluntary, our facilitators are not necessarily able to come.” 51 ANNEX B: SAPAN PERFORMANCE INDICATOR ANALYSIS Objective A (IAs) (Enhance capacity of key independent agencies to provide effective government oversight Period 1 (03/30/10 - 09/30/10) Period 2 (10/01/10 - 09/30/11) Period 3 (10/01/11 - 09/30/12) Period 4 (10/01/12 - 09/30/13) Period 5 (10/01/13 - 09/30/14) Period 6 (10/01/14 - 09/30/15) Total Target Actual Target Actual Target Actual Target Actual Target Actual Target Actual Target Actual ERA.1: Stronger independent government agencies, with linkages with civil society and academia, that effectively oversee government action and implementation of public policy 1.1.1 [Number of] IAs supported[/assisted] 0 0 4 11 22 33 12 15 5 6 0 0 43 65 1.1.2 [Number of] IA staff trained in transparency and accountability 0 0 101 91 156 121 175 3 30 34 0 0 462 249 1.1.3 [Number of] IA staff trained in outreach and communications 0 0 10 0 165 87 218 49 0 0 0 0 393 136 1.2.1 [Number of] IA-CSO joint oversight awareness raising campaigns 0 0 4 4 30 11 46 17 0 0 0 0 80 32 1.2.2 [Number of] people attending IA-CSO joint oversight awareness raising campaigns 0 0 200 724 1,500 7,529 1,500 1,922 0 0 0 0 3,200 10,175 Objective B (Strengthen the capacity of civil society organizations and media to serve as checks and balances for political processes and public policy) Period 1 (03/30/10 - 09/30/10) Period 2 (10/01/10 - 09/30/11) Period 3 (10/01/11 - 09/30/12) Period 4 (10/01/12 - 09/30/13) Period 5 (10/01/13 - 09/30/14) Period 6 (10/01/14 - 09/30/15) Total Target Actual Target Actual Target Actual Target Actual Target Actual Target Actual Target Actual ER B.1: Strengthened CSO research and advocacy capacities to enable constructive dialogue with the RTG on key policy issues 2.1.1 [Number of] civil society organizations trained in management, communications and planning 0 0 38 30 42 77 47 48 37 47 25 36 189 238 2.1.2 [Number of] civil society organization staff members [/persons] trained in management, communications and planning 0 0 114 259 114 656 188 413 148 211 185 166 749 1,705 2.1.3 [Number of] civil society organizations trained in governance issues 0 0 25 23 73 81 87 81 20 30 25 38 230 253 52 2.1.4 [Number of] civil society organization staff members [/person] trained in governance issues 0 0 100 96 146 429 261 544 160 181 95 144 762 1,394 2.1.5 [Number of] civil society organizations trained in research and advocacy 0 0 25 9 88 43 123 82 60 69 0 4 296 207 2.1.6 [Number of] civil society org staff members [/persons] trained in research and advocacy 0 0 100 62 174 154 264 420 261 532 40 50 839 1,218 2.1.7 [Number of] research grants awarded 0 0 63 63 155 194 115 43 10 18 0 0 343 318 2.1.8 [Number of] good governance, oversight and advocacy campaigns 0 0 10 12 30 28 50 45 50 60 17 16 157 161 2.1.9 [Number of] people attending good governance, oversight and advocacy campaigns 0 0 500 1,359 1,500 6,649 2,500 16,788 2,500 8,042 800 1,092 7,800 33,930 2.1.10 [Number of] CSOs engaging in advocacy and oversight functions 0 0 10 34 48 46 60 57 25 25 17 26 160 188 ER B.2: More sustainable community media outlets, improved capacities of journalists to cover policy and conflict issues and a more open media environment 2.2.1 [Number of] non-state news outlets assisted 0 0 9 10 51 16 92 14 14 16 14 14 180 70 2.2.2 [Number of] media outlet staff trained in management, communications and planning Indicator Dropped 2.2.3 [Number of] journalists trained on policy, freedom of information laws and conflict issues 0 0 18 0 137 23 241 66 30 32 20 21 446 142 2.2.4 [Number of] journalists trained on transparency and accountability issues 0 0 18 24 121 9 241 3 40 36 20 21 440 93 2.2.5 [Number of] media CSOs assisted 0 0 3 0 9 11 7 8 7 10 5 8 31 37 53 Objective C: Support civic peacebuilding efforts and diminish the potential for radicalization and escalation of violent conflict in Southern Thailand Period 1 (03/30/10 - 09/30/10) Period 2 (10/01/10 - 09/30/11) Period 3 (10/01/11 - 09/30/12) Period 4 (10/01/12 - 09/30/13) Period 5 (10/01/13 - 09/30/14) Period 6 (10/01/14 - 09/30/15) Total Target Actual Target Actual Target Actual Target Actual Target Actual Target Actual Target Actual ER C.1: Increased capacity and visibility of advocacy NGOs and civic leaders working to promote peace 3.1.1 Good governance, oversight and advocacy campaigns (South) New Indicator 5 5 8 9 8 16 5 6 26 36 3.1.2 Persons attending good governance and advocacy campaigns (South) New Indicator 250 1,453 400 3,144 800 3,050 475 504 1,925 8,151 ER C.2: Reduced pool of recruitable youths available to insurgent groups and increased social and economic opportunities for youth 3.2.1 Youth-focused conflict transformation campaigns 0 0 2 0 7 6 5 4 5 5 3 4 22 19 3.2.2 People attending youth-focused conflict transformation campaigns 0 0 100 0 340 7,116 250 800 700 701 500 787 1,890 9,404 3.2.3 Deep South youth trained in management, communications and planning 10 15 20 34 30 49 3.2.4 Deep South youth trained in governance issues, research and advocacy 60 104 50 57 110 161 54 ANNEX C: FULL LISTING OF REFERENCES AND REPORTS UTILIZED 1. AR Research. (2012). Citizen's Perceptions and Knowledge of Thailand's Independent Agencies (Rep.) 2. Chen, P. H. The Vulnerability of Thai Democracy: Coups d'etate and Political Changes in Modern Thailand. 3. Pottebaum, D. (2011, February 19). Sapan Program Fortnightly Highlights-- 7 to 18 Feb 2011 [E￾mail to M. Bak, C. Felley, M. J., H. Silvey, B. Spake, J. McCarthy, et al.]. 4. Bak, M. (2011, March 22). Sapan Program [E-mail to O. Carduner, B. Krell, K. L. Foster, B. B. Soderberg, R. D. Bennett, J. Dunbar, et al.]. 5. Bak, M. (2011, April 18). Sapan Program (Thailand) Update (28 March to 15 April 2011) [E-mail to G. P. Kent, B. B. Soderberg, E. W. Kneedler, K. L. Foster, N. Ransaeva, & K. Kneedler]. 6. Bak, M. (2011, March 28). Sapan Program (Thailand) Update (14 - 25 March 2011) [E-mail to O. Carduner, B. Krell, G. P. Kent, E. W. Kneedler, N. Ransaeva, S. N. Stevenson, et al.]. 7. Bak, M. (2011, May 9). Sapan Program (Thailand) Update (18 to 29 April 2011) [E-mail to G. P. Kent, B. B. Soderborg, E. W. Kneedler, K. L. Foster, & W. Harrity]. 8. Bak, M. (2011, June 6). Sapan Program (Thailand) Update (16-27 May 2011) [E-mail to G. P. Kent, B. B. Soderberg, E. W. Kneedler, N. Ransaeva, & S. Hoffman]. 9. Bak, M., & Pottebaum, D. (2011, June 18). Sapan Program Fortnightly Highlights #14 -- 30 May to 10 June 2011 [E-mail to C. Felley, M. Jukic, S. Pasong, J. McCarthy, H. Silvey, & B. Spake]. 10. Bak, M. (2011, February 17). RE: Sapan Program (Thailand) Update (24 January 2011 - 04 February 2011) [E-mail to O. Carduner, B. Krell, G. P. Kent, B. B. Soderborg, E. W. Kneedler, K. L. Foster, et al.]. 11. Bak, M. (2010, November 18). Sapan Program (Thailand) Update [E-mail to O. Carduner, B. Krell, G. P. Kent, B. Soderborg, E. W. Kneedler, K. L. Foster, et al.]. 12. Bak, M. (2010, November 18). Sapan Program (Thailand) Update [E-mail to O. Carduner, B. Krell, G. P. Kent, B. Soderborg, E. W. Kneedler, K. L. Foster, et al.]. 13. Bak, M. (2010, December 27). Sapan Program (Thailand) Update [E-mail to O. Carduner, B. Krell, G. P. Kent, B. B. Soderborg, E. W. Kneedler, K. L. Foster, et al.]. 14. Bak, M. (2011, January 19). Sapan Program (Thailand) Update [E-mail to O. Carduner, B. Krell, G. P. Kent, B. B. Soderborg, E. W. Kneedler, K. L. Foster, et al.]. 55 15. Bak, M. (2011, February 8). Sapan Program (Thailand) Update [E-mail to O. Carduner, B. Krell, G. P. Kent, K. L. Foster, J. P. Schutte, W. Harrity, et al.]. 16. Data Quality Assessment Checklist [USAID/Regional Development Mission Asia (RDMA)]. (2011, October 17). Thailand, Bangkok. 17. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, March 29). Contract 486-C-00-10-00001 [Award/Contract]. Thailand, Bangkok. 18. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 4]. Thailand, Bangkok. 19. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 5]. Thailand, Bangkok. 20. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 6]. Thailand, Bangkok. 21. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 8]. Bangkok, Thailand. 22. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 9]. Bangkok, Thailand. 23. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, March 29). Amendment of Solicitation/ Modification of Contract [Mod. 10]. Bangkok, Thailand. 24. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 11]. Bangkok, Thailand. 25. Development Alternatives Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 12]. Thailand, Bangkok. 26. Development Alternatives Inc. (February 14). Event Participant Attendance List. Thailand, Bangkok. 27. Development Alternatives Inc. (2011, September). Sapan - Strengthening Thai Democracy [Performance Monitoring Plan (PMP)]. Thailand, Bangkok. 28. Development Alternatives Inc. (2012, January). Sapan - Strengthening Thai Democracy [Performance Monitoring Plan (PMP) Addendum]. Thailand, Bangkok. 29. Development Alternatives Inc. (2011). Sapan - Strengthening Thai Democracy Annual Report (revised) March 2010 to March 2011 (pp. 1-34, Rep. No. No. 486-C-00-10-00001). United States Agency for International Development. 30. Development Alternatives Inc. (2011). Sapan - Strengthening Thai Democracy Annual Report April to September 2011 (pp. 1-40, Rep.). United States Agency for International Development. 56 31. Development Alternatives Inc. (2014). Sapan - Strengthening Thai Democracy Annual Report October 2013 - September 2014 (pp. 1-62, Rep. No. No. 486-C-00-10-00001). United States Agency for International Development. 32. Development Alternatives, Inc., & Regional Office of Procurement, USAID. (2010, September 9). Amendment of Solicitation/ Modification of Contract [Mod. 1]. Thailand, Bangkok. 33. Development Alternatives, Inc., & Regional Office of Procurement, USAID. (2010, September 29). Amendment of Solicitation/ Modification of Contract [Mod. 2]. Thailand, Bangkok. 34. Development Alternatives, Inc. (2012). Sapan - Strengthening Thai Democracy Annual Report October 2011 - September 2012 (pp. 1-72, Rep. No. No. 486-C-00-10-00001). United States Agency for International Development. 35. Development Alternatives Inc. (2011, September). Sapan - Strengthening Thai Democracy [Performance Monitoring Plan (PMP)]. Thailand, Bangkok. 36. Development Alternatives Inc. (2012, May). Sapan - Strengthening Thai Democracy [Work Plan 4]. Thailand, Bangkok. 37. Development Alternatives Inc. (2014, August). Sapan - Strengthening Thai Democracy [Performance Monitoring Plan (PMP) Addendum 2: Revised Indicators and Targets]. Thailand, Bangkok. 38. Development Alternatives, Inc. (2013). Sapan - Strengthening Thai Democracy Annual Report October 2012 to September 2013 (pp. 1-54, Rep. No. No. 486-C-00-10-00001). United States Agency for International Development. 39. Development Alternatives Inc. (2010, December 1). Sapan - Strengthening Thai Democracy [Performance Monitoring Plan (PMP)]. Thailand, Bangkok. 40. Berenshot, W., Nordholt, H. S., & Bakker, L. (n.d.). Social, Economic and Political Studies of the Middle East and Asia (Vol. 115) (D. F. Eickelman, Ed.). Boston. 41. Lacina, B. (2014, April). Public Opinion and the Costs of Civil War: Evidence form the southern Thailand insurgency [Presented at the annual meeting of the International Studies Association]. 42. McCargo, D. (2014, April). Southern Thailand: From conflict to negotiations? [Lowy Institute for International Policy]. 43. Mercy Corps. (2015). Outcome Harvesting Evaluation Final Report Broadening Participation Through Civil Society, Iraq (pp. 1-28, Rep.). 44. Munger, F. (2007, Spring). Cornell International Law Journal [Constitutional Reform, Legal Consciousness, and Citizen Participation in Thailand]. 57 45. Pottebaum, D. (2014, October 18). USAID Sapan Program PMP Addendum 2 (revised indicators and targets) [E-mail to S. Raviprapa, C., P. Kangkun, D. Wales, J. Go, M. Jukic, et al.]. 46. Pottebaum, D. (2014, October 18). USAID Sapan Program PMP Addendum 2 (revised indicators and targets) [E-mail to S. Raviprapa & C. Felley]. 47. Sapan - Strengthening Thai Democracy [Work Plan 5]. (2012, October). Thailand, Bangkok. 48. Sapan - Strengthening Thai Democracy [Performance Monitoring Plan (PMP)]. (2013, February). Thailand, Bangkok. 49. Sapan Program Performance Monitoring Indicators [Program Outcomes]. (2012, January 13). Bangkok, Thailand. 50. Section C - Statement of Work [Assessment of countering-trafficking-in-persons situations in Thailand and Sapan ex-post evaluation]. (n.d.). Thailand, Bangkok. 51. Southern Thailand's Peace Dialogue: No Traction [International Crisis Group]. (2016, September 21). Bangkok/ Brussels. 52. The World Bank. (2014). Cases in Outcome Harvesting (pp. 1-108, Rep. No. 90172). Washington, DC: The World Bank. 53. United States Agency for International Development. (n.d.). Data Quality Assessment: Thailand OU/ Governance and Vulnerable Populations Office. 54. USAID Sapan Program - Strengthening Thai Democracy [Work Plan 6 - Addendum]. (2014, July). Thailand, Bangkok. 58 ANNEX D: DESK REVIEW BIBLIOGRAPHY 1. Chen, Pei-Hsiu. The Vulnerability of Thai Democracy: Coups D’état and Political Changes in Modern Thailand. In Contemporary Socio-Cultural and Political Perspectives in Thailand, Chapter 12 pp. 185-207. Springer Netherlands, 2014. 2. McCargo, Duncan. 2014. Southern Thailand: From Conflict to Negotiations? Lowy Institute for International Policy, viewed 04 November 2016, . 3. Ferrara, Federico. The Political Development of Modern Thailand. Cambridge University Press. 26 March 2015. 4. International Crisis Group. Southern Thailand’s Peace Dialogue: No Traction. Crisis Group Asia Briefing N 148. 21 September 2016. 5. Lacina, Bethany. April 2014. Public Opinion and The Costs of Civil War: Evidence from The Southern Thailand Insurgency. University of Rochester. 6. Munger, Frank. Constitutional Reform, Legal Consciousness, and Citizen Participation in Thailand. Cornell International Journal of Law. V. 40 Issue 2. Spring 2007. 7. Pangsapa, P. (2015), When Battlefields become Marketplaces: Migrant Workers and the Role of Civil Society and NGO Activism in Thailand. International Migration, 53: 124–149. doi:10.1111/j.1468- 2435.2009. 00559.x 8. Schaffar, Wolfram. 2016. Citizenship and Democratization in Southeast Asia: Citizenship, Rights, and Adversarial Legalism in Thailand. Pp 238 – 264. 9. United States Agency for International Development. 2015 CSO Sustainability Index for Asia. Country Reports, 59. https://www.usaid.gov/sites/default/files/documents/1866/2015_Asia_CSOSI.pdf 59 ANNEX E: DATA COLLECTION PROTOCOLS SAPAN Ex-post Evaluation การประเมินโครงการสะพานหลังปิ ดโครงการ Consent Form for KII and SGD Participants แบบฟอร์มค ายินยอมเข้าร่วมการประเมินโครงการ ส าหรับผู้ให้ข้อมูลคนส าคัญ (KII) และผู้เข้าร่วมอภิปรายกลุ่มยอย่ (SDG) Participant or SGD ID: ______ เลขที่: ______ Purpose Hello, thank you for taking the time to meet with us. My name is [your name], and my colleagues are [names]. We are working for Social Impact who has been contracted by USAID to do an ex-post evaluation of the Sapan project. The purpose of the Sapan project is to: A. strengthen the capacity of Thai Independent Agencies (IAs) to conduct government oversight; B. strengthen the capacity of civil society organizations and media to serve as checks and balances for political processes and policy; and C. support civic peacebuilding efforts to diminish violent conflict and radicalization in the Deep South. These objectives will lead to the program development goal to foster constructive civil society engagement with the Royal Thai Government (RTG) as a means to build consensus for democratic political processes and to mitigate extremism. As an ex-post evaluation, we will explore the effectiveness and sustainability of the Sapan Program one year following its conclusion with questions around:  the extent to which Sapan reached its three objectives  interventions contributing to better achievements of its objectives  observable positive/negative changes in the capacity of targeted groups and factors contributing to those changes  increased capacity that remains and likely to sustain in the future and its key supporting factors; as well as  observable policy changes as a result of Sapan project during or after its life. We will be speaking to many different stakeholders to understand how effective and sustainable Sapan has been in Thailand. Since [stakeholder group] are intended [stakeholder or beneficiary] of this project, we would like to speak with you about your perceptions of [civil society] in Thailand, and your perspective about those issues related to Sapan’s implementation. Social Impact intends to use the evaluation finding to inform USAID, other USG programs that support to civil society in Thailand, and to share evaluation learning with other USAID Democracy and Governance (DG) officers working to connect citizens and their government in the region and globally for the better program development in the future. 60 วัตถุประสงค์ สวัสดี(ครับ/ค่ะ) ผม/ดิฉันชื่อ [ชื่อผู้สัมภาษณ์] และทีมงานชื่อ [ชื่อทีมงาน] พวกเราต้องขอขอบพระคุณที่ท่านได้สละเวลามาพูดคุยในครั้งนี้พวกเราเป็ นทีมงานจากบริษัท Social Impact ซึ่งเป็ นบริษัทที่ปรึกษา ได้รับมอบหมายจากองค์กรเพื่อการพัฒนาระหวางประเทศ ่ (USAID) ในการจัดท าการประเมินโครงการสะพานหลังปิ ดโครงการ วัตถุประสงค์หลักของโครงการสะพานคือ เพื่อเสริมสร้างความเข้มแข็งให้กบั (1) องค์กรอิสระ เพื่อสอดส่องความละเลยและข้อผิดพลาดของรัฐบาล (2) องค์กรภาคประชาสังคม และสื่อมวลชน เพื่อท าหน้าที่ตรวจสอบและถ่วงดุลในกระบวนการทางการเมืองและด้านนโยบาย และ (3) เพื่อสนับสนุนความพยายามสร้างสันติภาพทางการเมือง เพื่อกาจัดความขัดแย้งรุนแรงและการก ่อการร้ายในพื้นที่ภาคใต้ โดยวัตถุประสงค์ที่กล่าวมานี้จะน าไปสู่จุดมุ่งหมายการพัฒนาโครงการเพื่อส่งเสริมความร่วมมืออยางสร้างสรรค์ที่จะเก ่ ิดขึ้นระห วางภาคประชาสังคมและรัฐบาลไทย ่ ซึ่งถือเป็ นวิธีการสร้างฉันทามติในกระบวนการทางการเมืองระบอบประชาธิปไตยและบรรเทาปัญหาความรุนแรงวิธีหนึ่ง วัตถุประสงค์ของการประเมินโครงการหลังจากที่ได้มีการสรุปปิ ดไปแล้วเป็ นเวลา 1 ปี นั้นกเพื่อจะส ารวจประสิทธิภาพและความยั ็ งยืนของโครงการที่เก ่ ิดขึ้นและเหลืออยูอย่ างแท้จริง ่ ทีมงานได้วางแผนที่จะสัมภาษณ์ผู้มีส่วนได้ส่วนเสียกบโครงการหลายฝ่ าย ั เนื่องจาก [กลุ่มผู้มีส่วนได้ส่วนเสีย] ได้รับการเลือกให้เป็ น [ผู้มีส่วนได้ส่วนเสีย หรือผู้รับผลประโยชน์] ของโครงการ ทางทีมงานจึงอยากจะขอสอบถามความคิดเห็นจากท่านในฐานะที่เป็ นส่วนหนึ่งของภาคประชาสังคมในประเทศไทย โดยจะมีค าถามเกี่ ยวกบมุมมองในเรื่องความส าเร็จของโครงการสะพาน ั การเปลี่ยนแปลงที่เกิดขึ้นทั้งเชิงนโยบายและความสามารถของกลุ่มเป้ าหมาย ปัจจัยที่ส่งเสริมการเปลี่ยนแปลงนั้นๆทั้งทางบวกและทางลบ รวมถึงจะขอเรียนถามถึงข้อแนะน าเพื่อน าไปพัฒนาการด าเนินงานโครงการให้ประสบความส าเร็จมากยิงขึ ่ ้น ทาง Social Impact มีความตั้งใจที่จะน าสิ่งที่ได้เรียนรู้จากการประเมินในครั้งนี้ไปแบ่งปันกบองค์กรเพื่อการพัฒนาระหว ั างประเทศ ่ (USAID), โครงการภายใต้การสนับสนุนของรัฐบาลสหรัฐอื่นๆในประเทศไทยที่สนับสนุนภาคประชาสังคม, และเจ้าหน้าที่รับผิดชอบโครงการประชาธิปไตยและธรรมาภิบาล (Democratic Governance) ท่านอื่นๆที่ท างานเพื่อเชื่อมโยงประชาชนและภาครัฐในระดับภูมิภาคและในระดับโลก เพื่อการพัฒนางานที่ดียิงขึ ่ ้นต่อไป 61 Procedures/Confidentiality If you agree to participate, we will ask you a series of questions taking about [1-1.5 hr.], of your time. Although we will record your name, your name will not be attached to any specific finding or quote in the report without your prior written permission. The other participants in the group will be asked to keep what we talk about private, but this cannot be assured. ขั้นตอน / การปกปิ ดความลับ ถ้าท่านตกลงที่จะเข้าร่วมให้ความคิดเห็นในการประเมินครั้งนี้ทางทีมงานจะมีรายการค าถามเพื่อสัมภาษณ์ท่าน ใช้เวลาประมาณ [1-1.5 ชัวโมง ่ ] การจดบันทึกรายละเอียดส่วนตัว เช่น ชื่อ-นามสกุล ที่เกิดขึ้นจะไม่ได้ถูกน าไปใช้หรือปรากฎในการน าเสนอสิ่งที่พบเห็น หรืออ้างอิงค าพูดในรายงานโดยไม่ได้รับการอนุญาตจากผู้ให้สัมภาษณ์ก่อน ส าหรับผู้เข้าร่วมการประเมินโครงการท่านอื่นๆในกลุ่ม ทางทีมงานขอให้เก็บข้อมูลที่พูดคุยกนไว้เป็ นส ั ่วนบุคคล และเราเชื่อวาจะไม ่ ่น าไปเผยแพร่ที่ใด Risks/Benefits There is no large risk involved in your participation in this interview. The questions will not involve sensitive or personal information, and you can refuse to answer any question. Although this study may not benefit you personally, we hope that our results will add to the knowledge about strengthening civil society and local governance. ความเสี่ยง/ผลประโยชน์ จะไม่มีความเสี่ยงใดๆเข้ามาเกี่ยวข้องสืบเนื่องมาจากการเข้าร่วมให้สัมภาษณ์เพื่อการประเมินโครงการของท่านในครั้งนี้ ค าถามที่น ามาสัมภาษณ์ไม่มีความเกี่ยวข้องกบข้อมูลเชิงอ ั ่อนไหวหรือเป็ นส่วนตัวแต่อยางใด ่ โดยท่านสามารถปฏิเสธที่จะตอบค าถามก็ได้หากรู้สึกไม่สะดวกใจ แม้วาการศึกษาในครั ่ ้งนี้อาจจะไมมีผลประโยชน์ก ่ บตัวท ั ่านเป็ นส่วนตัว ทางทีมงานมีความหวังวาผลที่เก ่ ิดจากการศึกษาของเราจะมีส่วนเพิ่มพูนความรู้เกี่ ยวกบการเสริมสร้างความเข้มแข็งภาคประชาสั ั งคมและการบริหารจัดการที่ดีตามหลักธรรมมาภิบาลในระดับท้องถิ่นได้ Voluntary Participation Participation in this interview is completely voluntary. You do not have to agree to be in this study, and you may change your mind and stop at any time. ความสมัครใจเข้าร่วม การเข้าร่วมให้สัมภาษณ์ในครั้งนี้ทางทีมงานมุ่งเน้นให้เป็ นความสมัครใจของผู้ให้สัมภาษณ์โดยสมบูรณ์ ท่านไม่จ าเป็ นต้องตกลงเข้าร่วมการศึกษาครั้งนี้หากไม่เต็มใจ หรือสามารถเปลี่ยนใจ และหยุดการสัมภาษณ์ในทุกขณะ Permission to Proceed Are you willing to join the interview/discussion?  Yes  No ขออนุญาตเพื่อด าเนินการต่อ ท่านมีความเต็มใจที่จะเข้าร่วมให้สัมภาษณ์/การสนทนาในครั้งนี้หรือไม่?  เต็มใจ  ไม่เต็มใจ ______________________________ _______________________________ ________________ (Name) (ชื่อ) (Signature) (ลายเซ็น) (Date) (วันที่) 62 USAID Sapan Performance Evaluation การประเมินผลการด าเนินงาน โครงการสะพาน ภายใต้การสนับสนุนขององค์กรเพื่อการพัฒนาระหว่างประเทศของสหรัฐฯ (USAID) Key Informant Interview (KII) Protocol – Bellwethers เกณฑ์วิธีการสัมภาษณ์ผู้ให้ข้อมูลคนส าคัญ (Bellwethers) Date: ________________ วันที่: Interviewer Name: _________________________ ชื่อผู้สัมภาษณ์: Primary Note Taker Name: ____________________ ชื่อผู้จดบันทึกหลัก: KII Code (first three letters of project location and KII number [e.g. UBON3]): _________________ รหัสผู้ให้ข้อมูลส าคัญ (พยัญชนะ 3 ตัวแรกของพื้นที่โครงการ ตามด้วยล าดับของผู้ให้สัมภาษณ์): Interviewee Name: _____________________ ชื่อผู้ให้สัมภาษณ์: Interviewee Organization: _________________ ชื่อองค์กร: Interviewee Title at Organization (duration): ____________________ ต าแหน่งหน้าที่ในองค์กร (พร้อมระบุระยะเวลา): Project Location: ___________________________ Region: ____________________________ พื้นที่โครงการ: ภาค: Respondent Group: ___USG ____DAI ___Sapan Partner ____Policy Expert ____Other ประเภทกลุ่ม: ____รัฐบาลสหรัฐ ____องค์กร DAI ____องค์กรของโครงการสะพาน ____ผู้เชี่ยวชาญด้านนโยบาย ____อื่นๆ Sex of Interviewee: ___ Male ____ Female เพศ: ____ชาย ____หญิง 63 Theme 5: Policy Changes ประเด็นที่ 5: การเปลี่ยนแปลงเชิงนโยบาย EQ5: What policy changes, during or after the life of Sapan, are observable as a result of Sapan? ข้อ 5: การเปลี่ยนแปลงเชิงนโยบายใด (ที่เกิดขึ้นระหวางหรือภายหลังการด าเนินงานโครงการสะพาน ่ ) ที่เห็นได้วาเป็ นผลของโครงการ ่ ? 1. Do you know of any important changes in national or local policies contributing to or detracting from the strengthening of democracy of which you are aware [from 2010-until now]? Probes: คุณคิดวามีการเปลี่ยนแปลงส าคัญใดบ้างที่เก ่ ิดขึ้นกบนโยบายสนับสนุนหรือเบี่ยงเบนความตั ั ้งใจสร้างความเข้มแข็งในระบอบป ระชาธิปไตย ทั้งในระดับประเทศและระดับท้องถิ่น (นับจากปี 2553 จนถึงปัจจุบัน)? แนวค าถาม a. Please describe the change. Can you provide some examples that illustrate the change? กรุณายกตัวอยางและอธิบายการเปลี่ยนแปลงที่เก ่ ิดขึ้นดังกล่าว b. What caused or influenced the change? อะไรเป็ นสาเหตุหรือผลักดันให้เกิดการเปลี่ยนแปลงนั้น c. Did the change affect women, men, youth, or other groups differently? If so, how and why? การเปลี่ยนแปลงดังกล่าวมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชน หรือกลุ่มคนกลุ่มอื่นที่แตกต่างกนหรือไม ั ่? ถ้ามีความแตกต่าง กรุณาอธิบายวาแตกต ่ ่างกนอย ั างไร และอะไรคือสาเหตุ ่ ? 2. Do you know of any changes in national or local level policies aimed at diminishing radicalization and reducing violent conflict in Southern Thailand [from 2010-until now]? คุณคิดวามีการเปลี่ยนแปลงส าคัญใดบ้างที่เก ่ ิดขึ้นกบนโยบายลดทอนการก ั ่อความรุนแรงและลดความขัดแย้งในพื้นที่ ภาคใต้ของประเทศไทย ทั้งในระดับประเทศและระดับท้องถิ่น (นับจากปี 2553 จนถึงปัจจุบัน)? a. Please describe the change. Can you provide some examples that illustrate the change? กรุณายกตัวอยางและอธิบายการเปลี่ยนแปลงที่เก ่ ิดขึ้นดังกล่าว b. What caused or influenced the change? อะไรเป็ นสาเหตุหรือผลักดันให้เกิดการเปลี่ยนแปลงนั้น c. Did the change affect women, men, youth, or other groups differently? If so, how and why? การเปลี่ยนแปลงดังกล่าวมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชน หรือกลุ่มคนกลุ่มอื่นที่แตกต่างกนหรือไม ั ่? ถ้ามีความแตกต่าง กรุณาอธิบายวาแตกต ่ ่างกนอย ั างไร ่ และอะไรคือสาเหตุ? 3. Do you know of any changes in policy concerning civil society organizations and media, either at the national or local level [from 2010-until now]? คุณคิดวามีการเปลี่ยนแปลงส าคัญใดบ้าง ที่เก ่ ิดขึ้นกบนโยบายด้านองค์กรประชาสังคมและสื่อ ั ทั้งในระดับประเทศหรือระระดับท้องถิ่น (นับจากปี 2553 ถึงปัจจุบัน)? a. Please describe the change. Can you provide some examples that illustrate the change? คุณคิดวามีการเปลี่ยนแปลงส าคัญใดบ้าง ที่เก ่ ิดขึ้นกบนโยบาย ั 64 b. What caused or influenced the change? อะไรเป็ นสาเหตุหรือผลักดันให้เกิดการเปลี่ยนแปลงนั้น c. Did the change affect women, men, youth, or other groups differently? If so, how and why? การเปลี่ยนแปลงดังกล่าวมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชน หรือกลุ่มคนกลุ่มอื่นที่แตกต่างกนหรือไม ั ่? ถ้ามีความแตกต่าง กรุณาอธิบายวาแตกต ่ ่างกนอย ั างไร และอะไรคือสาเหตุ ่ ? 4. Do you know of any changes in local or national level policies directed at Independent Agencies relating to strengthening democracy and/or civic participation [from 2010-until now]? คุณคิดวามีการเปลี่ยนแปลงส าคัญใดบ้างที่เก ่ ิดขึ้นกบนโยบายการเสริมสร้างความเข็มแข็งในระบอบประชาธิปไตยและ ั /หรื อการมีส่วนร่วมของภาคประชาชน ทั้งในระดับประเทศและในระดับท้องถิ่น ที่มุ่งเน้นมีผลบังคับใช้กบองค์กรอิสระโดยเฉพาะ ั (นับจากปี 2553 จนถึงปัจจุบัน)? a. Please describe the change. Can you provide some examples that illustrate the change? คุณคิดวามีการเปลี่ยนแปลงส าคัญใดบ้าง ที่เก ่ ิดขึ้นกบนโยบาย ั b. What caused or influenced the change? อะไรเป็ นสาเหตุหรือผลักดันให้เกิดการเปลี่ยนแปลงนั้น c. Did the change affect women, men, youth, or other groups differently? If so, how and why? การเปลี่ยนแปลงดังกล่าวมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชน หรือกลุ่มคนกลุ่มอื่นที่แตกต่างกนหรือไม ั ่? ถ้ามีความแตกต่าง กรุณาอธิบายวาแตกต ่ ่างกนอย ั างไร และอะไรคือสาเหตุ ่ ? 65 Key Informant Interview (KII) Protocol - Sapan Partners เกณฑ์วิธีการสัมภาษณ์ผู้ให้ข้อมูลคนส าคัญ – ภาคีความร่วมมือโครงการสะพาน Date: ________________ วันที่: Interviewer Name: _________________________ ชื่อผู้สัมภาษณ์: Primary Notetaker Name: ____________________ ชื่อผู้จดบันทึกหลัก: KII Code (first three letters of project location and KII number [e.g. UBON3]): _________________ รหัสผู้ให้ข้อมูลส าคัญ (พยัญชนะ3 ตัวแรกของพื้นที่โครงการ ตามด้วยล าดับของผู้ให้สัมภาษณ์): Interviewee Name: _____________________ ชื่อผู้ให้สัมภาษณ์: Interviewee Organization: _________________ ชื่อองค์กร: Interviewee Title at Organization (duration): ____________________ ต าแหน่งหน้าที่ในองค์กร (พร้อมระบุระยะเวลา): Project Location: ___________________________ Region: ____________________________ พื้นที่โครงการ: ภาค: Respondent Group: ___USG ____DAI ___Sapan Partner ____Policy Expert ____Other ประเภทกลุ่ม: ____รัฐบาลสหรัฐ ____องค์กร DAI ____องค์กรของโครงการสะพาน ____ผู้เชี่ยวชาญด้านนโยบาย ____อื่นๆ Sex of Participant: ___ Male ____ Female เพศ: ____ชาย ____หญิง 66 Theme 1: Outcome/Output Achievement ประเด็นที่ 1: การบรรลุผลสัมฤทธิ์/ผลลัพธ์ EQ1: To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? ข้อที่ 1: โครงการสะพานประสบความส าเร็จตามวัตถุประสงค์หลัก 3 ข้อ ที่ได้วางไว้ในกรอบการด าเนินงานโครงการได้อยางไร ่ ? กรุณาอธิบายขอบเขตความส าเร็จของโครงการสะพาน EQ2: What interventions were more successful and/or had a greater contribution to Sapan’s objectives? ข้อที่ 2: กิจกรรมใดของโครงการที่ประสบความส าเร็จดีและ/หรือมีส่วนส่งเสริมให้การด าเนินงานโครงการประสบความส าเร็จตามวัตถุ ประสงค ได้มากกวาก่ ิจกรรมอื่น 1. Please explain how you were involved in the design or implementation of Sapan. กรุณาอธิบายวาคุณมีส ่ ่วนร่วมในการออกแบบหรือการด าเนินงานโครงการอยางไร ่ o Who was your main POC at Sapan (DAI) and what was your relationship like? ใครคือผู้ติดต่อหลักโครงการสะพานประจ าองค์กร DAI ของคุณ และความสัมพันธ์ในการด าเนินงานเป็ นอยางไร ่ ? 2. “Fostering constructive civil society engagement with the RTG to build consensus for democratic political processes and to mitigate extremism” was SAPAN’s ultimate goal. This was broken down into three main objectives: “การส่งเสริมให้ภาคประชาสังคมมีส่วนร่วมกบภาครัฐ ั บาลไทยอยางสร้างสรรค์เพื่อสร้างฉันทามติทางกระบวนการการเมือ ่ งระบอบประชาธิปไตยและเพื่อบรรเทาความรุนแรง” เป็ นเป้ าหมายสูงสุดของโครงการสะพาน โดยสามารถแบ่งออกเป็ นวัตถุประสงค์ยอยหลักได้ ่ 3 วัตถุประสงค์คือ • Objective A: Enhancing the capacity of key IAs to provide effective government oversight วัตถุประสงค์ที่ 1: เพื่อเสริมสร้างความเข้มแข็งให้กบองค์กรอิสระ ั เพื่อสอดส่องความละเลยและข้อผิดพลาดของรัฐบาล • Objective B: Strengthening the capacity of CSOs and media to serve as checks and balances for political processes and public policy วัตถุประสงค์ที่ 2: เพื่อเสริมสร้างความเข้มแข็งให้กบองค์กรภาคประชาสังคม ั และสื่อมวลชน เพื่อท าหน้าที่ตรวจสอบและถ่วงดุลในกระบวนการทางการเมืองและด้านนโยบาย และ • Objective C: Support civic peacebuilding efforts and diminish the potential for radicalization and violent conflict in the Deep South วัตถุประสงค์ที่ 3: เพื่อสนับสนุนความพยายามสร้างสันติภาพทางการเมือง เพื่อกาจัดความขัดแย้งรุนแรงและการก ่อการร้ายในพื้นที่ภาคใต้ In your experience, did you see change in any of these areas or in the ultimate goal? จากประสบการณ์ของคุณ คุณเห็นการเปลี่ยนแปลงเกิดขึ้นตามวัตถุประสงค์ยอยหรือตามเป้ าหมายสูงสุดของโครงการหรือไม ่ ่? 67 [Note to facilitator: have participant state the outcome – or change – that they observed. Re-confirm that you have captured how they would ‘define’ that change and later compare to the description developed from desk review] [หมายเหตุส าหรับผู้ด าเนินการสัมภาษณ์: ผู้เข้าร่วมสัมภาษณ์ได้กล่าวถึงผลสัมฤทธิ์ – หรือการเปลี่ยนแปลง – ที่พวกเขาสังเกตเห็นหรือไม?่ ยืนยันซ ้าอีกครั้งวา่ คุณได้จับประเด็น“นิยาม”ความหมายของค าวาการเปลี่ยนแปลงและเปรียบเทียบก ่ บค าบรรยายที่พัฒนาขึ ั ้นจากการทบทวนงานวิ จัย (desk review)] Probes: แนวค าถาม: a. If yes, how much change? Please describe the changes. ถ้ามีการเปลี่ยนแปลงมีมากน้อยแค่ไหน? กรุณาอธิบาย b. Why do you think the change was possible? ท าไมคุณถึงคิดวาการเปลี่ยนแปลงนั ่ ้นเป็ นไปได้? c. What role, if any, did Sapan play in the change? โครงการสะพานมีบทบาทที่ท าให้เกิดการเปลี่ยนแปลงนั้นอยางไร ่ ? d. What else, besides Sapan, contributed to the change? นอกจากโครงการสะพานแล้ว อะไรคือสิ่งที่ส่งเสริมให้เกิดการเปลี่ยนแปลง? e. Did the change affect women, men, youth or other groups differently? If so, how and why? การเปลี่ยนแปลงที่เกิดขึ้นมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชนหรือกลุ่มอื่นๆต่างกนหรือไม ั ่? หากมีผลกระทบ ผลกระทบเกิดขึ้นอยางไร ่ ? และสาเหตุที่ท าให้เกิดคืออะไร? 3. To what extent were other changes achieved outside of those listed above? [Note to facilitator: use same probes as Q2] นอกจากการเปลี่ยนแปลงที่เกิดขึ้นดังกล่าวมาแล้วนี้ยังมีการเปลี่ยนแปลงอยางอื่นเก ่ ิดขึ้นด้วยหรือไม่? อยางไร ่ ? [หมายเหตุส าหรับผู้ด าเนินการสัมภาษณ์: ให้ใช้แนวทางการถามเหมือนข้อที่ 2] 4. In your opinion, which of Sapan’s interventions or activities were most successful? Probes: ในความคิดเห็นของคุณ กิจกรรมใดของโครงการสะพานประสบความส าเร็จมากที่สุด? แนวค าถาม: a. What factors made these interventions/activities successful (coordination, personalities, adequate funding, motivation, political will, community mobilization, etc.)? ปัจจัยใดมีส่วนท าให้กิจกรรมของโครงการประสบความส าเร็จ (เช่น การประสานงาน, บุคลิกภาพ, เงินทุนที่เพียงพอ, แรงจูงใจ, เจตจ านงทางการเมือง, การระดมชุมชน, และอื่นๆ)? b. Did the change affect women, men, youth or other groups differently? If so, how and why? การเปลี่ยนแปลงที่เกิดขึ้นมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชน หรือกลุ่มอื่นๆแตกต่างกนหรือไม ั ่? หากมีผลกระทบ ผลกระทบเกิดขึ้นอยางไร ่ ? และสาเหตุที่ท าให้เกิดคืออะไร? 68 Theme 2: Project Theory of Change ประเด็นที่ 2: ทฤษฎีการเปลี่ยนแปลงโครงการ EQ1a: Did Sapan’s development hypothesis, especially the roles of the independent agencies, hold true throughout the course of Sapan’s life? ข้อที่ 1: สมมติฐานการพัฒนาของโครงการสะพาน โดยเฉพาะอยางยิ ่ งในเรื่องบทบาทขององค์กรอิสระ ่ ถือเป็ นจริงตลอดระยะเวลาของโครงการหรือไม่? EQ1b: How did the changes of the initial assumption that formulated the development hypothesis, if any, affect Sapan’s ability to meet the objectives? ข้อที่ 2: การเปลี่ยนแปลงของการสันนิษฐานเบื้องต้นที่เป็ นตัวกาหนดสมมติฐานการพัฒนาของโครงการ (ถ้ามี) มีผลกระทบต่อความสามารถของโครงการสะพานในการด าเนินงานเพื่อบรรลุวัตถุประสงค์หรือไม่? อยางไร ่ ? 5. [Note to facilitator: Summarize Sapan’s theory of change] How and why, if at all, did the program theory of change evolve over the course of implementation? [หมายเหตุส าหรับผู้ด าเนินการสัมภาษณ์: ให้สรุปทฤษฎีการเปลี่ยนแปลง] ทฤษฎีการเปลี่ยนแปลงโครงการมีการพัฒนาระหวางการด าเนินงานโครงการอย ่ างไร ่ (ถ้ามี)? และสาเหตุคืออะไร? 6. What effects did the change in assumptions/theory of change have on program implementation and management? การเปลี่ยนแปลงในข้อสันนิษฐาน/ทฤษฎีการเปลี่ยนแปลงมีผลกระทบอะไรบ้างต่อการด าเนินงานและบริหารงานของโครง การ? a. How did the project or scheduled activities change over time and why? How, if at all, did the changes affect men and women involved in Sapan activities? โครงการหรือกิจกรรมที่กาหนดไว้มีการเปลี่ยนแปลงอย างไรบ้าง ่ ? สาเหตุของการเปลี่ยนแปลงคืออะไร? การเปลี่ยนแปลงที่เกิดขึ้นมีผลกระทบต่อผู้ชายและผู้หญิงที่เข้ามาเกี่ยวข้องกบกั ิจกรรมโครงการสะพายอยางไร ่ ? b. What support did you receive from DAI? Did the support change over the course of implementation? How and why? ความช่วยเหลือใดที่คุณได้รับจากองค์กร DAI? ความช่วยเหลือมีการเปลี่ยนแปลงไปตลอดระยะเวลาด าเนินงานโครงการหรือไม่? เปลี่ยนแปลงไปอยางไร ่ ? และสาเหตุของการเปลี่ยนแปลงคืออะไร? 7. To what extent, if at all, did changes to the project assumptions positively or negatively influence project outputs/outcomes? การเปลี่ยนแปลงที่เกิดขึ้นกบข้อสันนิษฐานโครงการมีอิทธิพลด้านบวกและลบก ั บผลลัพธ์ ั /ผลสัมฤทธิ์ของโครงการในระดั บ? กรุณาอธิบายขอบเขตการเปลี่ยนแปลง 69 Theme 3: Management Adaptations to Political Context ประเด็นที่ 3: การปรับตัวด้านการจัดการเพื่อให้สอดคล้องต่อสถานการณ์ทางการเมือง EQ2: What, if any, management shifts adopted in response to the deteriorating political environment prove effective in contributing to Sapan’s objectives? ข้อที่ 2: มีการเปลี่ยนแปลงด้านการบริหารจัดการใดที่เกิดขึ้น (ถ้ามี) เพื่อเป็ นการตอบสนองสภาวะความวุนวายทางการเมือง ่ ที่ได้รับการพิสูจน์แล้ววามีประสิทธิภาพในการส ่ ่งเสริมการด าเนินงานโครงการสะพานให้บรรลุวัตถุประสงค์? 8. Can you please describe how the political environment changed over the course of Sapan implementation? กรุณาอธิบายการเปลี่ยนแปลงสภาวะทางการเมืองในช่วงระยะเวลาการด าเนินงานโครงการสะพาน [Note to facilitator: give examples of changes to the political context from the desk review if respondent is not forthcoming] [หมายเหตุส าหรับผู้ด าเนินการสัมภาษณ์: ยกตัวอยางการเปลี่ยนแปลงในบริบททางการเมืองที่ได้จากการทบทวนงานวิจัย ่ (desk review) ถ้าผู้ให้สัมภาษณ์ไม่สามารถให้ข้อมูลได้] 9. Did Sapan’s management adapt to the changing political environment? If yes, how? ผู้บริหารโครงการสะพานมีการปรับตัวต่อการเปลี่ยนแปลงตามสภาวะทางการเมืองหรือไม่? ถ้ามี, มีการปรับตัวอยางไร ่ ? 10. Of the management techniques you mentioned, which were most/least successful and why? จากเทคนิคการบริหารจัดการที่คุณได้กล่าวมาข้างต้น เทคนิคใดประสบความส าเสร็จมากที่สุด? และน้อยที่สุด? สาเหตุคืออะไร? 70 Theme 4: Capacity-building Changes and Sustainability ประเด็นที่ 4: การเปลี่ยนแปลงจากการสร้างเสริมศักยภาพ และการพัฒนาอยางยั ่ งยืน ่ EQ3: What are observable positive/negative changes in the capacity of the targeted groups, i.e. IAs, CSOs, media, and civic-peacebuilding leaders, as a result of Sapan? What are the factors that helped or hindered such changes? ข้อที่ 3: การเปลี่ยนแปลงทั้งด้านบวกและลบที่สังเกตเห็นได้ในศักยภาพของกลุ่มเป้ าหมาย (เช่น องค์กรอิสระ องค์กรภาคประชาสังคม สื่อ และผู้น าด้านการสร้างสันติภาพ) อันเป็ นผลจากโครงการสะพานนั้นมีอะไรบ้าง? EQ4: If any, what and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? ข้อที่ 4: (ถ้ามี) ศักยภาพที่เพิ่มขึ้นในองค์กรอิสระ องค์กรภาคประชาสังคม สื่อ และผู้น าด้านการสร้างสันติภาพ อันเป็ นผลจากโครงการสะพานนั้นมีอะไรบ้าง? ศักยภาพใดที่ยังคงอยูและดูเหมือนว ่ ายังคงอยู ่ ในอนาคต ่ ? อะไรคือปัจจัยสนับสนุนที่ส าคัญในการรักษาศักยภาพดังกล่าวให้ยังยืนต ่ ่อไป? 11. What capacity-building support did you receive from Sapan? คุณได้รับความช่วยเหลืออะไรบ้างจากโครงการสะพาน เพื่อส่งเสริมการสร้างเสริมศักยภาพ? 12. What changes do you see, if any, in your/your organization’s capacity due to your involvement with Sapan? Probes: การเปลี่ยนแปลงใดที่คุณเห็น (ถ้ามี) วาเก่ ิดขึ้นกบศักยภาพของตน ั เอง/องค์กรของคุณ อันเป็ นผลเนื่องมาจากการมีส่วนร่วมในโครงการสะพานของคุณเอง? แนวค าถาม: a) To what extent and how did you or your organization apply knowledge and skills acquired from Sapan’s support? คุณหรือองค์กรของคุณได้น าความรู้และทักษะที่ได้รับจากการช่วยเหลือของโครงการสะพานมาใช้อยางไรบ้าง ่ ? กรุณาอธิบายถึงขอบเขตการน าไปใช้ประโยชน์ b) What have you been able to achieve as a result of the capacity-building support? หลังจากที่ได้รับการอบรมพัฒนาศักยภาพจากโครงการสะพานแล้ว คุณได้น าไปปฏิบัติใช้และประสบความส าเร็จอยางไรบ้าง ่ ? c) What have you not been able to achieve? อะไรคือสิ่งที่คุณยังไม่สามารถบรรลุวัตถุประสงค์ได้? 13. What elements of Sapan’s capacity building efforts were most helpful and why? องค์ประกอบใดในการสร้างศักยภาพภายใต้ความช่วยเหลือของโครงการสะพานที่คุณคิดวามีประโยชน์มากที่สุด ่ ? และสาเหตุคืออะไร? a) To what extent, did the support meet your (capacity building) needs? 71 ความช่วยเหลือที่โครงการสะพานให้เพื่อเสริมสร้างศักยภาพนั้น สามารถตอบสนองความต้องการของคุณได้หรือไม่? กรุณาอธิบายขอบเขตของความต้องการ b) What needs remain unmet? ความต้องการใดในด้านการเสริมสร้างศักยภาพที่ยังไม่ได้รับความช่วยเหลือ? 14. What can be done to ensure that the organizational capacity changes you described continue in the future? สิ่งใดที่คุณคิดวาเราจะสามารถท าได้เพื่อให้การเปลี่ยนแปลงด้านศักยภาพองค์กรที่ได้กล ่ ่าวมาแล้ว สามารถด าเนินต่อไปได้ในอนาคต? Theme 5: Policy Changes ประเด็นที่ 5: การเปลี่ยนแปลงเชิงนโยบาย EQ5: What policy changes, during or after the life of Sapan, are observable as a result of Sapan? ข้อที่ 5: การเปลี่ยนแปลงเชิงนโยบายใดบ้างที่เกิดขึ้นระหวางหรือภายหลังโครงการสะพาน ่ และสามารถสังเกตเห็นได้วาเป็ นผลจากโครงการสะพาน ่ ? 15. Have you observed any changes in policy at the local or provincial level as a result of Sapan? คุณสังเกตเห็นการเปลี่ยนแปลงเชิงนโยบายในระดับท้องถิ่นหรือจังหวัด ซึ่งเป็ นผลจากโครงการสะพานหรือไม่? Probes / แนวค าถาม: a) If yes, how much change? Please describe the changes. ถ้ามี, เปลี่ยนแปลงไปมากแค่ไหน? กรุณาอธิบายถึงการเปลี่ยนแปลงที่เกิดขึ้น b) Why do you think the change was possible? สาเหตุที่ท าให้คุณคิดวา่ การเปลี่ยนแปลงนั้นเป็ นไปได้คืออะไร? c) What role, if any, did Sapan play in the change? บทบาท (ถ้ามี) ของโครงการสะพานที่มีต่อการเปลี่ยนแปลงคืออะไร? d) What else, besides Sapan, contributed to the change? นอกจากโครงการสะพานแล้ว อะไรคือปัจจัยที่ส่งเสริมให้เกิดการเปลี่ยนแปลง? e) Did the change affect women, men, youth or other groups differently? If so, how and why? การเปลี่นแปลงดังกล่าวมีผลกระทบต่อผู้หญิง ผู้ชาย เยาวชน หรือกลุ่มอื่นๆต่างกนหรือไม ั ่? ถ้ามี, มีผลกระทบอยางไร ่ ? และสาเหตุของผลกระทบคืออะไร? 72 USAID Sapan Performance Evaluation Key Informant Interview (KII) Protocol - USAID/DAI Date: ________________ Interviewer Name: _________________________ Primary Note Taker Name: ____________________ KII Code (first three letters of project location and KII number [e.g. UBON3]): _________________ Interviewee Name: _____________________ Interviewee Organization: _________________ Interviewee Title at Organization (duration): ____________________ Project Location ___________________________ Region ____________________________ Respondent Group: ___USG ____DAI ___Sapan Partner ____Policy Expert ____Other Sex of participant ___ Male ____ Female Theme 1: Outcome/Output Achievement EQ1: To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? EQ2: What interventions were more successful and/or had a greater contribution to Sapan’s objectives? 1. Please explain how you were involved in the design or implementation of Sapan. • Probe for Sapan partners: Who was your main POC at Sapan (DAI) and what was your relationship like? 2. “Fostering constructive civil society engagement with the RTG to build consensus for democratic political processes and to mitigate extremism” was SAPAN’s ultimate goal. This was broken down into three main objectives: • Objective A: Enhancing the capacity of key IAs to provide effective government oversight • Objective B: Strengthening the capacity of CSOs and media to serve as checks and balances for political processes and public policy • Objective C: Support civic peacebuilding efforts and diminish the potential for radicalization and violent conflict in the Deep South In your experience, did you see change in any of these areas or in the ultimate goal? [Note to facilitator: have participant state the outcome – or change – that they observed. Re-confirm that you have captured how they would ‘define’ that change and later compare to the description developed from desk review] Probes: a. If yes, how much change? Please describe the changes. b. Why do you think the change was possible? c. What role, if any, did Sapan play in the change? d. What else, besides Sapan, contributed to the change? e. Did the change affect women and men differently? If so, how and why? 3. To what extent were other changes achieved outside of those listed above? [Note to facilitator: use same probes as Q2] 73 4. In your opinion, which of Sapan’s interventions or activities were most successful? Probes: a. What factors made these interventions/activities successful (coordination, personalities, adequate funding, motivation, political will, community mobilization, etc.)? b. Did the activity involve men, women, boy, and girls differently? If yes, what effect did that have on the beneficiaries? Theme 2: Project Theory of Change EQ1a: Did Sapan’s development hypothesis, especially the roles of the independent agencies, hold true throughout the course of Sapan’s life? EQ1b: How did the changes of the initial assumption that formulated the development hypothesis, if any, affect Sapan’s ability to meet the objectives? 5. [Note to facilitator: Summarize Sapan’s theory of change] How and why, if at all, did the program theory of change evolve over the course of implementation? 6. What effects did the change in assumptions/theory of change have on program implementation and management? a. How did the project or scheduled activities change over time and why? How, if at all, did the changes affect men and women involved in Sapan activities? b. How did you respond to the mid-term evaluation and Inspector General reports regarding the program’s theory of change? 7. To what extent, if at all, did changes to the project assumptions positively or negatively influence project outputs/outcomes? Theme 3: Management Adaptations to Political Context EQ2: What, if any, management shifts adopted in response to the deteriorating political environment prove effective in contributing to Sapan’s objectives? 8. Can you please describe how the political environment changed over the course of Sapan implementation? [Note to facilitator: give examples of changes to the political context from the desk review if respondent is not forthcoming] 9. Did Sapan’s management adapt to the changing political environment? If yes, how? 10. Of the management techniques you mentioned, which were most/least successful and why? 74 Theme 4: Capacity-building Changes and Sustainability EQ3: What are observable positive/negative changes in the capacity of the targeted groups, i.e. IAs, CSOs, media, and civic-peacebuilding leaders, as a result of Sapan? What are the factors that helped or hindered such changes? EQ4: If any, what and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? 11. How, if at all, have you seen Sapan partners (CSOs, media, local leaders) change as a result of Sapan’s capacity building activities? [Notes to facilitator: summarize or reference elements of CSO capacity as needed: (1) Overall goal or purpose statement; (2) Legal registration; (3) Finance policies; (4) Admin policies; (5) Procurement policies; (6) Personnel policies; (7) Gender and diversity policies; (8) Anti-corruption policy; (9) Organizational structure; (10) Inventory; (11) M&E system; (12) Job descriptions; (13) Annual budgeting process; (14) Public outreach; (15) Fundraising strategy Probes: a. Which capacity building activities were most and least successful? Why? b. To what extent, if at all, did the capacity building activities affect women and men differently? Why and how? 12. To what extent do you believe these organizations have sustained their increased capacity? Do you think this improvement is likely to remain? a. What will be most critical to ensuring these improvements remain? b. What are the biggest potential challenges? Theme 5: Policy Changes EQ5: What policy changes, during or after the life of Sapan, are observable as a result of Sapan? 13. Policy changes are not expressed as an expected result of Sapan in program documents, but Sapan partners worked on policy-related issues. To what extent and how did Sapan expect to affect policy change? a. To what extent was this goal presented to implementing agencies or partners? b. What opportunities did Sapan have to contribute to policy change? 75 USAID Sapan Performance Evaluation การประเมินผลการด าเนินงาน โครงการสะพาน ได้รับการสนับสนุนโดยองค์กรเพื่อการพัฒนาระหวางประเทศของสหรัฐฯ ่ (ยูเสด) Small Group Discussion (SGD) Guide แนวทางการจัดอภิปรายกลุ่มยอย่ Date: ________________ วันที่: Facilitator Name: _________________________ ชื่อผู้ด าเนินการจัดอภิปราย: Primary Notetaker Name: ____________________ ชื่อผู้จดบันทึกหลัก: SGD Code (first three letters of project location and SGD number [e.g. UBON3]): ______________ รหัสกลุ่มอภิปรายยอย่ (พยัญชนะ 3 ตัวแรกของพื้นที่โครงการและเลขที่กลุ่มอภิปรายยอย่ [เช่น อุบล3]): Interview Location: ___________________________ Region: ________________________ สถานที่อภิปราย: ภาค: Type of SGD Participants (CSO members, community leaders, etc.): __________________________ ประเภทของผู้เข้าร่วมอภิปราย (สมาชิกภาคประชาสังคม, ผู้น าชุมชน, อื่นๆ): Number of Participants: Males _______ Females_________ Total_____________ จ านวนผู้เข้าร่วมอภิปราย: เพศชาย เพศหญิง รวม SGD Code ______________ 76 # Sex เพศ Age อายุ Transport Reimbursement ยอดเงิน เบิกจ่ายค่าเดินทาง Male ชาย Female หญิง Below 25 ต ่ากวา่ 25 25-45 46-65 65+ ____ Baht (บาท) 1 2 3 4 5 6 7 8 9 10 Total (รวม) NOTES FOR FACILITATOR: The questions below are structured under the core evaluation topics. The facilitator should let the conversation move organically within the discussion topics, and use the question probes as needed to clarify or to keep the conversation moving. Evaluation Question 1: To what extent and how did Sapan reach its three objectives as laid out in the Sapan program framework? • Objective A: Independent Agencies—enhance the capacity of key IAs to promote effective, transparent and accountable government; • Objective B: Civil Society—strengthen the capacity of CSOs and media to serve as checks and balances for political processes and public policy; and • Objective C: Peacebuilding in the Deep South—support civic peacebuilding efforts and diminish the potential for radicalization and escalation of violent conflict in Southern Thailand. วัตถุประสงค์: เพื่อเสริมสร้างความเข้มแข็งให้กบั (1) องค์กรอิสระ เพื่อสอดส่องความละเลยและข้อผิดพลาดของรัฐบาล 77 (2) องค์กรภาคประชาสังคม และสื่อมวลชน เพื่อท าหน้าที่ตรวจสอบและถ่วงดุลในกระบวนการทางการเมืองและด้านนโยบาย (3) เพื่อสนับสนุนความพยายามสร้างสันติภาพทางการเมือง เพื่อกาจัดความขัดแย้งรุนแรงและการก ่อการร้ายในพื้นที่ภาคใต้ Evaluation Question 2: What interventions were more successful and/or had a greater contribution to Sapan’s objectives? Evaluation Question 3: What are observable positive/negative changes in the capacity of the targeted groups, i.e. IAs, CSOs, media, and civic peacebuilding leaders, as a result of Sapan? What are the factors that helped or hindered such changes? Discussion Topic: Nature and outcomes of involvement with Sapan (including capacity building) (15 minutes) หัวข้ออภิปราย: ธรรมชาติและผลสัมฤทธิ์จากการมีส่วนร่วมกบโครงการสะพาน ั (รวมถึงการเสริมสร้างขีดความสามารถ) (15 นาที) Discussion Q1: Can you describe your involvement with Sapan? ค าถามที่ 1: กรุณาอธิบายการมีส่วนร่วมของคุณในโครงการสะพาน Examples of possible involvement: • Training on management and technical themes, mentoring and coaching • Organizations Established to Train CSOs in Thailand and the Lower Mekong Sub region. • Comprehensive training on local government processes, participatory and gender sensitive budgeting, communication and facilitation, and advocacy (for women-led CSOs); participation in Thai Women Coalition. • Research on perceptions and priorities of youth, community-level activities to engage youth on real democracy and governance issues, and “Youth Theatre” to facilitate expression and discussion of concerns and priorities by youth among their peers and with local leaders; Youth Congress. • Activities with CSOs, women and youth culminated in a series of large conferences that facilitated regional sharing of experiences and joint planning for future activities Discussion probes: a) What type of support/activities did you receive from Sapan and for how long? b) How was the support provided to you? Who was your main POC at Sapan? c) Do you see any differences in the way that the activity involved men, women, boy, and girls? If so, how did their involvement differ, and what effect did that have? d) To what extent, did the support meet your (capacity-building) needs? What needs remain unmet? e) To what extent, did the support/activity meet the different (capacity-building) needs of men, women, boys and girls? Why and how? 78 Discussion Q2: Of the Sapan activities you participated in, which, in your opinion, were the most/least successful? Why? ค าถามที่ 2: ในกิจกรรมของโครงการสะพานที่คุณได้มีโอกาสเข้าไปมีส่วนร่วม ในความคิดของคุณ กิจกรรมใดประสบความส าเร็จมากที่สุด? และกิจกรรมใดประสบความส าเร็จน้อยที่สุด? กรุณาให้เหตุผล Discussion probes: a) Which activities did you find the most/least helpful? Why? b) What factors made the activities successful/unsuccessful (coordination, personalities, adequate funding, motivation, political will, community mobilization, etc.)? c) To what extent and why were different activities more or less successful for women vs. men? Discussion Q3: What changes personally, with your organization, or in your community have you observed (good or bad) as a result of your involvement with Sapan? What contributed to those changes? ค าถามที่ 3: โดยส่วนตัวแล้ว คุณสังเกตเห็นวาองค์กรหรือชุมชนของคุณเก ่ ิดความเปลี่ยนแปลงในด้านบ้าง (ทั้งดีและไมดี่ ) อันเป็ นผลมากจากการที่คุณไปมีส่วนร่วมในโครงการสะพาน? ปัจจัยที่ส่งผลให้เกิดความเปลี่ยนแปลงดังกล่าวคืออะไร? Discussion probes: a) What, if any, differences did you observe in the way the support/activity affected men, women, boys, and girls? b) To what extent did the changes you observed align with your expectations? Why, why not? c) What have you been able to achieve as a result of the capacity-building support? What have you not been able to achieve? d) To what extent, if any, is there an observable difference between the impact on men and women of the change in capacity? Why? e) To what extent and how did you or your organization apply knowledge and skills acquired during Sapan-supported trainings? f) To what extent and how have the attitudes and beliefs of staff members, or community members changed regarding civil society engagement or peacebuilding? To what extent and how do men and women differ in their beliefs, if at all? g) Were there any negative or positive outcomes from the activity/capacity-building support that you did not expect? 79 Evaluation Question 4: If any, what and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? Discussion topic: Sustainability of capacity built (15 minutes) หัวข้ออภิปราย: การเสริมสร้างขีดความสามารถเชิงการพัฒนาอยางยั ่ งยืน ่ Discussion Q4: Of the capacity changes described, how have they continued (and evolved) since you stopped receiving Sapan support? ค าถามที่ 4: ดังที่กล่าวมาแล้วข้างต้นวามีการเปลี่ยนแปลงอันเป็ นผลจากโครงการเก ่ ิดขึ้น การเปลี่ยนแปลงเหล่านี้มีการด าเนินการต่อไป (และพัฒนาอยางต ่ อเนื่อง ่ ) หลังจากความช่วยเหลือจากโครงการหยุดไปหรือไม? ่ อยางไร ่ ? Discussion probes: a) Do you feel that your/your organization/your community’s capacity has gotten better/worse or stayed the same after Sapan support ceased? How different? Why? b) To what extent, if any, is there a difference in the way that the capacities of women, men, boys and girls have changed after Sapan support ceased? How different? Why? Discussion Q5: In your opinion, what factors caused the changes to continue or not continue? ค าถามที่ 5 ในความคิดของคุณ ปัจจัยอะไรเป็ นสาเหตุให้การเปลี่ยนแปลงดังกล่าวนั้นคงอยูหรือไม ่ ่คงอยู?่ Discussion probes: a) What has enabled you/your org/your community from continuing the ‘positive’ capacity change? What do you think the project did well to help foster sustainability? b) What has prevented the changes from continuing? c) What should the project have done in order for the changes to continue? d) What capacity related challenges do you continue to face and why? 80 ANNEX F: EVALUATION DESIGN MATRIX Evaluation Questions Indicators OR Question Topic Data Sources Data Collection Methods Data Analysis Methods 1a) To what extent and how did Sapan reach its three objectives as laid out in Sapan’s program framework? Outcome Indicator 1.3.1: Specialist perception and scoring of IA government oversight capacity Outcome Indicator 1.3.2. Public perception and knowledge of IAs Outcome Indicator 2.3.1. Specialist perception and scoring of CSO oversight capacity Outcome Indicator 2.3.2. Public perception of partner CSOs Outcome Indicator 2.3.4. Capacity Scorecard Outcome Indicator 2.3.5. Specialist perception and scoring of community media capacity to strengthen democracy Outcome Indicator 3.3.1. Public perception of CSOs to promote peace Outcome Indicator 3.3.3. Youth perceptions of social and economic opportunities Sapan reports; monitoring data (outcomes); evaluation reports (external and internal) USAID staff; Sapan program staff; external experts (donors and political analysts) Desk Study IDIs/GIs (purposive) Quantitative data: Trend analysis Qualitative data: Content analysis Pattern analysis 81 Q1b) Did Sapan’s theory of change, especially the roles of the independent agencies, hold true throughout the course of Sapan’s life? Q1c) How did the changes in the initial assumption that formulated the theory of change, if any, affect Sapan’s ability to meet the objectives? •Underlying critical assumptions and risks to project log frame 1.Sapan SOW; Sapan Activity M&E documents and data 2.USAID; Sapan program staff; external experts (donors and political analysts) 1.Desk Study 2.IDIs/GIs (purposive) Qualitative data: •Content analysis •Pattern analysis Q2 What interventions were more successful and/or had a greater contribution to Sapan’s objectives? Output indicators: IA staff members trained in outreach and communications IA-CSO joint oversight awareness raising campaigns People attending IACSO joint oversight awareness raising campaigns GJD 4.1. CSOs improving internal capacity GJD 4.1. CSO advocacy campaigns GJD 4.1. CSOs engaging in advocacy and watchdog functions GJD 4.2. Non-state news outlets assisted Sapan reports; monitoring data; evaluation reports (external and internal) Sapan core CSO partners; Media outlet partners; youth groups Desk Study IDIs/GIs (purposive) Small group discussions (purposive) Quantitative data: Trend analysis Qualitative data: Content analysis Pattern analysis 82 GJD 4.2. Number of journalists trained GJD 4.2. Number of media CSOs assisted PS 6.2. Events geared toward strengthening understanding among conflict affected groups PS 6.2. People attending events geared toward strengthening understanding among conflict affected groups Q3 What are observable positive/negative changes in the capacity of the targeted groups; i.e. IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan? What are the factors that helped or hindered such changes? OCA results against CSO Operations and Administration Minimum Standards Sapan reports; monitoring data; evaluation reports (external and internal) Sapan core CSO partners; Media outlet partners; youth groups Case Studies Desk Study IDIs/GIs (purposive) Small group discussions (purposive) Quantitative data: Trend analysis Qualitative data: Content analysis Pattern analysis Q4 If any, what and to what extent did the increased capacity of the IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? Qualitative follow up of OCA results Sapan reports; monitoring data; evaluation reports (external and internal) Sapan core CSO partners; Media outlet partners; youth groups Case Studies Desk Study IDIs/GIs (purposive) Small group discussions (purposive) Qualitative data: Content analysis Pattern analysis 83 Q5 What policy changes, during or after the life of Sapan, are observable as a result of Sapan? Qualitative follow up of outputs: CSO advocacy campaigns; CSOs engaging in advocacy and watchdog functions Sapan reports; monitoring data; secondary documents and analyses on Sapan-supported advocacy or policy issues “Bellwethers” External experts (donors, analysts, and/or journalists) Sapan partner advocacy NGOs (region/issue) Bellwether Methodology Desk Study IDIs/GIs (purposive) Small group discussions (purposive) Qualitative data: Content analysis Pattern analysis 84 ANNEX G: FULL LISTING OF PERSONS INTERVIEWED ID Type Respondent Group Org Title Sex Province Region BANG1 IDI Bellwether N/A Independent Security Analyst/Bellwether Male Bangkok Central BANG2 IDI USAID USAID Former COR Female Bangkok Central BANG3 IDI Bellwether N/A Technical Specialist, Political Science & Lecturer Male Bangkok Central BANG4 GI Sapan Partner Prachatai Business Manager Female Bangkok Central BANG4 GI Sapan Partner Prachatai Journalist / Director of Media Learning Center Male Bangkok Central BANG4 GI Sapan Partner Prachatai Project Coordinator Female Bangkok Central BANG5 GI USAID USAID Former COR Female Bangkok Central BANG5 GI USAID USAID Former COR Female Bangkok Central BANG5 GI USAID USAID Former Alt-COR Female Bangkok Central BANG6 GI Bellwether Department of State Political Specialist Female Bangkok Central BANG6 GI Bellwether Department of State Cultural and Educational Officer Female Bangkok Central BANG6 GI Bellwether Department of State Cultural Affairs Specialist Female Bangkok Central BANG7 IDI Sapan Partner The Asia Foundation Country Representative to Thailand Male Bangkok Central BANG8 IDI Sapan Partner Friends of Women President Female Bangkok Central BANG9 IDI DAI DAI COP Male Bangkok Central BANG10 IDI DAI DAI Regional Manager Male Bangkok Central BANG11 IDI DAI DAI Program Officer Female Bangkok Central BANG12 IDI DAI DAI Project Coordinator Female Bangkok Central BANG13 IDI Sapan Partner Youth Cares Project Leader Female Bangkok Central BANG14 IDI USAID USAID COR Male Bangkok Central 85 CM1 SGD Beneficiaries Book Republic (Beneficiaries) Book Republic / University Student Male Chiang Mai North CM1 SGD Beneficiaries Book Republic (Beneficiaries) Book Republic / University Student Female Chiang Mai North CM2 IDI Sapan Partner Book Republic Co -founder Female Chiang Mai North CM3 GI Sapan Partner Prachatham Director Female Chiang Mai North CM3 GI Sapan Partner Prachatham News Reporter Male Chiang Mai North CM3 GI Sapan Partner Prachatham Editor Male Chiang Mai North CM4 GI Sapan Partner Chiang Mai University Project Leader Male Chiang Mai North CM4 GI Sapan Partner Chiang Mai University Head of Project Coordination Male Chiang Mai North CM5 SGD Beneficiaries Voluntary Women's Group Women Leader / Volunteer Female Chiang Mai North CM5 SGD Beneficiaries Voluntary Women's Group Women Leader / Volunteer Female Chiang Mai North CM5 SGD Beneficiaries Voluntary Women's Group Women Leader / Volunteer -Sarapee district Female Chiang Mai North CM5 SGD Beneficiaries Voluntary Women's Group Women Leader / Volunteer -Mae Rim District Female Chiang Mai North CM6 GI Sapan Partner Voluntary Women's Group Finance and Accounting Assistant Female Chiang Mai North CM6 GI Sapan Partner Voluntary Women's Group Project Coordinator Female Chiang Mai North CM6 GI Sapan Partner Voluntary Women's Group Finance and Accounting Officer Female Chiang Mai North CM6 GI Sapan Partner Voluntary Women's Group President Female Chiang Mai North CM7 IDI Bellwether N/A N/A Male Chiang Mai North KHON1 GI Bellwether Mahasarakham University Lecturer (Bellwether) Female Mahasarakham Northeast KHON1 GI Bellwether Mahasarakham University Lecturer (Bellwether) Male Mahasarakham Northeast KHON2 GI Sapan Partner CSNM - KKU Director of CSNM Male Khon Kaen Northeast KHON2 GI Beneficiaries CSNM - KKU Director, Community radio station Male Khon Kaen Northeast 86 KHON2 GI Sapan Partner CSNM - KKU Chairman of Health Assembly, Khon Kaen province Male Khon Kaen Northeast KHON2 GI Sapan Partner CSNM - KKU Project Coordinator Female Khon Kaen Northeast KHON2 GI Sapan Partner CSNM - KKU Deputy Director of CSNM Male Khon Kaen Northeast NST1 IDI Bellwether Walailuck University Lecturer (Bellwether) Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women Muslim Network Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women President, Federation Association of Women (Thailand) Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women President and Sapan Project Leader Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women Volunteer Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women Core Leader, Sapan Project Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women Finance Officer, Sapan Project Female Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women Advisor Male Nakhon Si Thammarat South NST2 GI Sapan Partner Association for Muslim Women Advisor Male Nakhon Si Thammarat South NST3 SGD Beneficiaries Provincial Election Commission Supervisor - Election Organization Section Male Nakhon Si Thammarat South NST3 SGD Beneficiaries Community Welfare Fund, Larnsaka Municipality Treasurer Female Nakhon Si Thammarat South NST3 SGD Beneficiaries Sub -district Administrative Office, Larnsaka sub -district Deputy Chief Female Nakhon Si Thammarat South NST3 SGD Beneficiaries Southern Community Media Head of Social Communication Female Nakhon Si Thammarat South 87 NST3 SGD Beneficiaries Southern Community Media Office Manager Male Nakhon Si Thammarat South NST4 GI Sapan Partner Rajabhat Nakorn Si Thammarat PGI Team Member and University Lecturer Male Nakhon Si Thammarat South NST4 GI Sapan Partner Rajabhat Nakorn Si Thammarat Project Coordinator Female Nakhon Si Thammarat South PAT1 GI Sapan Partner People's College Executive Director Male Pattani Deep South PAT1 GI Sapan Partner People's College Curriculum Development and Training Manager Male Pattani Deep South PAT1 GI Sapan Partner People's College Curriculum Development and Training Manager Male Pattani Deep South PAT1 GI Sapan Partner People's College Admin Officer Female Pattani Deep South PAT2 SGD Beneficiaries People's College Students from Nong -Jik district Female Pattani Deep South PAT2 SGD Beneficiaries People's College Students from Nong -Jik district Male Pattani Deep South PAT3 GI Sapan Partner Media Selataan Admin Officer Female Pattani Deep South PAT3 GI Sapan Partner Media Selataan Radio Presenter Female Pattani Deep South PAT3 GI Sapan Partner Media Selataan Radio Station Manager Male Pattani Deep South PAT3 GI Sapan Partner Media Selataan Director Male Pattani Deep South PAT3 GI Sapan Partner Media Selataan Technician Male Pattani Deep South PAT3 GI Sapan Partner Media Selataan Webmaster Male Pattani Deep South PAT4 GI Sapan Partner Pattani Forum Project Manager Male Pattani Deep South PAT4 GI Sapan Partner Pattani Forum Finance Officer Female Pattani Deep South PAT5 SGD Beneficiaries Pattani Forum (Beneficiaries) Lecturer (Prince Songkhla University, Humanity and Social Sciences), Technical Expert for Sapan Project Female Pattani Deep South PAT5 SGD Beneficiaries Pattani Forum (Beneficiaries) Journalist (Freelance), Studying Master Degree Male Pattani Deep South 88 PAT5 SGD Beneficiaries Pattani Forum (Beneficiaries) Radio Presenter Male Pattani Deep South PAT6 IDI Sapan Partner Lukrieeng President Female Yala Deep South PAT7 IDI Sapan Partner Yala Rajabhat University Chief Information Officer Male Yala Deep South PAT8 IDI Sapan Partner WePeace President Female Yala Deep South PAT9 GI Sapan Partner WePeace Working group member Female Yala Deep South PAT9 GI Sapan Partner WePeace Working group member Female Yala Deep South PAT9 GI Sapan Partner WePeace Working group member Female Yala Deep South PAT9 GI Sapan Partner WePeace Admin Officer Female Yala Deep South PAT9 GI Sapan Partner WePeace Project Coordinator Female Yala Deep South PAT10 IDI Bellwether N/A Lecturer Female Pattani Deep South PHIT1 GI Sapan Partner Naresuan University Project Leader/Deputy Dean for Graduate Studies Female Phitsanulok Central PHIT1 GI Sapan Partner Naresuan University Team Leader/Deputy Project Leader Male Phitsanulok Central PHIT1 GI Sapan Partner Naresuan University Project Officer - PGI & LG Male Phitsanulok Central PHIT1 GI Sapan Partner Naresuan University Former Project Leader Male Phitsanulok Central PHIT1 GI Sapan Partner Naresuan University Project Assistant not involved during Sapan Female Phitsanulok Central PHIT2 SGD Beneficiaries Naresuan U - Sukhothai (Thachai sub -district) Project Leader Female Sukhothai Central PHIT2 SGD Beneficiaries Naresuan U - Sukhothai (Thachai sub -district) Advisory Board to Sapan Project/Provincial Representative (District 3) Male Sukhothai Central PHIT2 SGD Beneficiaries Naresuan U - Sukhothai (Thachai sub -district) Director of Local Health Promotion Hospital Male Sukhothai Central 89 PHIT3 GI Sapan Partner Sri Song Kwai Women's Association Assistant Treasurer Female Phitsanulok Central PHIT3 GI Sapan Partner Sri Song Kwai Women's Association Secretary Female Phitsanulok Central PHIT3 GI Sapan Partner Sri Song Kwai Women's Association Vice Chairwoman Female Phitsanulok Central PHIT3 GI Sapan Partner Sri Song Kwai Women's Association Chairwoman Female Phitsanulok Central PHIT3 GI Sapan Partner Sri Song Kwai Women's Association Public Relations Female Phitsanulok Central PHIT3 GI Sapan Partner Sri Song Kwai Women's Association Treasurer Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Public Health Volunteer, Na Bua sub -district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Team Leader, Wang Nok Aen sub -district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Public Health Volunteer, Baan Krang sub -district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Public Health Volunteer, Na Bua sub -district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Team Leader, Wang Kong sub - district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Women's Leader, Na Bua sub -district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Public Health Volunteer, Na Bua sub -district Female Phitsanulok Central PHIT4 SGD Beneficiaries Sri Song Kwai Women's Association Public Health Volunteer, Na Bua sub -district Female Phitsanulok Central 90 UBON1 GI Sapan Partner Media for Happiness Secretary to the Organization Male Ubon Ratchathani Northeast UBON1 GI Sapan Partner Media for Happiness Foundation Manager Female Ubon Ratchathani Northeast UBON1 GI Sapan Partner Media for Happiness Producer Male Ubon Ratchathani Northeast UBON1 GI Sapan Partner Media for Happiness Coordinator of Community Project Female Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Female Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Female Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Male Ubon Ratchathani Northeast UBON2 SGD Beneficiaries Media for Happiness - SGD Beneficiary Female Ubon Ratchathani Northeast UBON3 GI Sapan Partner Ubon Ratchathani University Project Leader (2013 - 2014)/Lecturer Female Ubon Ratchathani Northeast UBON3 GI Sapan Partner Ubon Ratchathani University Project Coordinator/Lectu rer Female Ubon Ratchathani Northeast UBON3 GI Sapan Partner Ubon Ratchathani University PGI Leader/Lecturer Female Ubon Ratchathani Northeast UBON3 GI Sapan Partner Ubon Ratchathani University Project Leader (2012) Male Ubon Ratchathani Northeast UBON4 IDI Sapan Partner Nature Care Project Leader Female Ubon Ratchathani Northeast 91 ANNEX H: EVALUATION STATEMENT OF WORK SECTION C – STATEMENT OF WORK C.1 TITLE OF ACTIVITY Assessment of [REDACTED] and Sapan ex-post evaluation. STATEMENT OF WORK C.1.1 Purpose [REDACTED] The second purpose of this TO is to evaluate the USAID/RDMA Sapan Program implemented by Development Alternatives, Inc. (DAI) from March 2010 to September 2015. The evaluation is a performance evaluation focusing on the effectiveness and sustainability of the Sapan program, its interventions and its overall approach to foster civil society engagement in governance in Thailand. In general, it will evaluate the extent to which the program has contributed to advancing its stated goal, objectives and selected Expected Results (ERs) (see 2.2 in this section below). In particular, it will assess the effectiveness of the program in key areas and objectives, namely; a) enhancing capacity of key Independent Agencies (IAs) to provide effective government oversight, b) strengthening capacity of civil society organizations and media to serve as checks and balances for political processes and public policy, and c) supporting civic peacebuilding efforts and diminished potential for radicalization and escalation of violent conflict in southern Thailand. The evaluation will seek to answer a central question – to what extent did the program contribute to fostering constructive civil society engagement with the Royal Thai Government (RTG) as a means to build consensus for democratic political processes and to mitigate extremism? It will also seek to identify any sustainable changes in Thai civil society and/or governance that resulted from the intervention. The audience for this evaluation includes USAID staff and broader constituencies should there be interest outside of USAID, based on approved release of the evaluation by designated USAID authorities. Background Sapan Ex-Post Evaluation The USAID Sapan Program (hereafter referred to as “Sapan”) was initiated in March 2010 and ended in September 2015. The contract was awarded to DAI for a base period of three years ($19.5 million) and two option years ($10.7 million). The overall development goal of this program was to foster constructive civil society engagement with the RTG as a means to build consensus for democratic political processes and mitigating extremism. Sapan’s Development Hypothesis is “If independent agencies, civil society and media’s capacity is strengthened, to provide effective government oversight and peacebuilding efforts across Thailand, then civil society’s engagement with the RTG will be more constructive, as a means to build consensus for democratic political processes and to mitigate extremism.” The development problem the project was to address was the lack of political consensus building processes in Thailand and the existence of extremism in the country, principally in Thailand’s Deep South. 92 93 94 95 As the general political situation in Thailand deteriorated in late 2013 and into 2014 because of frequent demonstrations against the government, opportunities for Sapan to work with the Independent Agencies (IAs), which are constitutional organizations of Thailand, under Objective A further diminished. Any possibility to work with the IAs effectively ended with demonstrations by pro-government demonstrators at their offices at the beginning of 2014 and most certainly with the May 2014 coup. At this time, USAID determined that further work on this objective 96 was not possible and decided to stop activities with the IAs. Program targets were revised to reflect this situation. Sapan addressed the development problem by employing a flexible mechanism to provide technical assistance, training, material, and other resources as needed to a broad range of civil society organizations and stakeholders working to promote peace building and stable democratic governance in Thailand. Since the political situation in Thailand was not stable during the life of Activity, Sapan needed to use a flexible mechanism to respond to needs as they emerged and opportunities as they arose, especially for the Deep South as violence could occur anytime and lead to changes in conducting interventions. Sapan was defined and guided by three principles. First, bring together a critical mass of leaders and managers from all sides to design and guide a process of reform. Second, actively link and build relationships built on trust and confidence between civil society, media and government. Third, pursue opportunities that keep these groups in sustainable creative interaction. Sapan activities covered all regions across Thailand, of which its core partners are located in Chiang Mai, Phitsanulok, Khon Kaen, Ubon Ratchathani, Nakorn Sri Thammarat, Pattani, Yala and Bangkok. While Sapan was in its third year of the five-year implementation, USAID/RDMA contracted independent evaluators to conduct a mid-term evaluation. The evaluation examined the period of March 2010 to July 2013. The mid-term evaluation found that Sapan had achieved modest success in reaching the project goals. IAs staff received empowerment training and assistance in creating events to heighten public awareness of their role as a check and balance on corrupt elected officials and civil servants as mentioned in the Constitution of Thailand. Sapan, through universities and CSOs, introduced new tools for participatory democracy in Thailand, some of which were highly appreciated by Sapan’s partners. In the South, Sapan’s support gave residents daily access to radio programming that allows people to discuss and share ideas on conflict and peacebuilding in the local language. These achievements, however, were reported by the mid-term evaluation to be undermined by managerial issues centered around the highly directive approach of Sapan (despite of Sapan’s claim of employing a flexible mechanism in providing services to its beneficiaries), and structural challenges in Thailand which were outside Sapan’s control. TASKS This Task Order consists of two parts as described below. C.3.1. CTIP Assessment [REDACTED] C.3.2 Sapan Ex-Post Evaluation Sapan ex-post evaluation will focus on providing detailed answers for its effectiveness and sustainability. The evaluation must answer all questions listed below: 1) To what extent and how did Sapan reach its three objectives as laid out in Sapan’s program framework? In addition to the overall results, the evaluation must also assess: Did Sapan’s development hypothesis, especially the roles of the independent agencies, hold true throughout the course of Sapan’s life? How did the changes of the initial assumption that formulated the development hypothesis, if any, affect Sapan’s ability to meet the objectives? 2) What interventions were more successful and/or had a greater contribution to Sapan’s objectives? 3) What are observable positive/negative changes in the capacity of the targeted groups; i.e., IAs, CSOs, media, and civic peacebuilding leaders as a result of Sapan? What are the factors that helped or hindered such changes? 4) If any, what and to what extent did the increased capacity of the IAs, CSOs, media and civic peacebuilding leaders as a result of Sapan still remain and seem likely to remain in the future? What are key supporting factors to sustain such capacity? 97 5) What policy changes, during or after life of Sapan, are observable as a result of Sapan? Given the nature of the evaluation questions set forth in this SOW, as well as the complexity of the project, it is anticipated that a combination of mixed methods will be applied for this evaluation. It is anticipated that samples of targeted sites and population groups will be required for each evaluation question. In addition to multiple levels and types of respondents/informants, a combination of sound quantitative and qualitative research methods (e.g., surveys, case studies, interview and focused group discussion with appropriate statistical and qualitative data analysis methods for each type of data collected) shall also be developed for each evaluation question as deemed appropriate. Non-conventional evaluation methods such as the Most Significant Change and/or Outcome Harvesting may be applied for selected evaluation questions as appropriate. However, different evaluation questions may be combined in one tool/method for specific targeted groups as appropriate. Attempts shall be made to collect data from different sources by different methods for each evaluation question and findings be triangulated to draw valid and reliable conclusions. Data shall be disaggregated by sex where possible and appropriate. The ex-post evaluation of Sapan must comply with the USAID Evaluation Policy. The evaluation must be undertaken in a manner that ensures credibility, lack of bias, transparency, and the generation of high quality information and knowledge. The contractor must use sound social science methods and include the following basic features: Establish a team with the appropriate methodological and subject matter expertise to conduct a credible, evidence-based ex-post evaluation. Written design, including key and sub-question(s), detailed methods, data collection instruments, and data analysis plans. Incorporate relevant gender sensitive indicators and sex disaggregated data in the evaluation design and analysis. An approach that encourages participation by Sapan core partners, grantees and sub-grantees, national counterparts, USAID and State Department, and evaluators throughout the process. Use of data collection and analytic methods that ensures, to the maximum extent possible, that if a different, well qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. Application and use to the maximum extent possible of internationally-recognized social science methods and tools that reduce the need for evaluator specific judgments. Standardized recording and maintenance of records from the evaluation (e.g., interview and focus group transcripts). Collection of data on variables corresponding to technical and programmatic results. Collection of data from relevant sources other than Sapan’s partners and beneficiaries to obtain comprehensive information to support findings, conclusions, and recommendations. Evaluation findings that are based on facts, evidence and data. This precludes relying exclusively upon anecdotes, hearsay and unverified opinions. Findings should be specific, concise and supported by triangulation of quantitative and qualitative information derived from various sources to ensure reliability, validity and generalizability. Evaluation reports that include action-oriented, practical and specific recommendations assigning or designating the implementer. The recommendations may be built upon successes and lessons learned from relevant project to be evaluated and/or other similar programs or projects implemented by other organizations funded by USAID and/or other donors, as well as derived from the Evaluation Team’s own evidence-based innovative or other solutions. A final evaluation report incorporating the criteria outlined in USAID’s Evaluation Policy (http://www.usaid.gov/evaluation), including all annexes, supporting data and records. A matrix of key findings and recommendations that improve the evidence-base for effectiveness and sustainability should be summarized in an additional annex. The Evaluation Report must: represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; address all evaluation questions included in the scope of work, and meet the objectives and purpose of the evaluation; at least include the following documents as annexes, 98 o the statement of work, o all evaluation tools, o description of the design and methodology, o All sources of information, o list of documents reviewed clearly explain evaluation design and methodology; properly identify sources of information in the report findings; disclose limitations, with particular attention to the methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) and efforts to mitigate bias and improve quality; present findings as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence; provide recommendations that are supported by conclusions derived from a specific set of evidence-based findings, action-oriented, practical and specific, with defined responsibility for the action; not exceed 50 pages, not including covers, executive summary, and annexes; be written in correct English grammar, readable, flow logically, and be written in an appropriate style and tone. Any gaps in information should be reported; and include an Executive Summary of 3-4 pages that is stand alone and provides a sufficient summary of the evaluation background, methodology, findings, conclusion and recommendations. 99 SCHEDULE C.4.1 CTIP Assessment [REDACTED] C.4.2 Sapan Ex-Post Evaluation Week 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1. Contract signed 2. Team planning meeting 3. Entrance briefing with USAID 4. Work plan submission and approval 5. Data collection/Field activities for internal and external consultations and data collection, including stakeholder and key informant interviews and focus groups. 6. Data analysis 7. Monthly oral briefing 8. Presentation of final findings 9. Report writing and submission 10. Submission of final report, assessment tools and other materials in hard copies and CD 100 ANNEX I: DISCLOSURE OF ANY CONFLICTS OF INTEREST 101 102 103 104 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523