Civil Society: Mutual Accountability Project (CS:MAP) Mid-term Performance Evaluation September 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International. i CIVIL SOCIETY: MUTUAL ACCONTABILITY PROJECT (CS:MAP) MID-TERM PERFORMANCE EVALUATION SUPPORTING NEPALI CIVIL SOCIETY AND MEDIA ORGANIZATIONS TO IMPROVE THEIR LEGITIMACY, ACCOUNTABILITY, AND RESILIENCE September 2018 Project Number: Evaluation Mechanism Number: AID-367-A-15-00005 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii TABLE OF CONTENTS Acronyms and Abbreviations .........................................................................................................iv Team Members and Acknowledgements .......................................................................................vi Abstract ........................................................................................................................................viii Executive Summary ........................................................................................................................ix Evaluation Purpose .....................................................................................................................ix Project Background ....................................................................................................................ix Evaluation Questions and Methodology ......................................................................................x Findings and Conclusions.............................................................................................................x Recommendations........................................................................................................................x Recommendations for USAID/NEPAL .......................................................................................xi 1.0 Evaluation Purpose ............................................................................................................. 1 2.0 Project Background ............................................................................................................. 1 3.1 Evaluation Questions and Methodology ............................................................................. 4 3.2 Framing the Evaluation .................................................................................................... 5 3.3 Methodology ................................................................................................................... 6 3.3.1 Data Collection Methods .......................................................................................... 6 3.3.2 Sampling Strategy ..................................................................................................... 7 3.3.3 Limitations ................................................................................................................ 8 4.1 Findings and Conclusions .................................................................................................... 9 4.2 Evaluation Question 5: Relevance of Project to Changed Governance Structures ....... 10 4.2.1 Findings and Conclusions 1: Theory of Change and Program Design .................... 10 4.2.2 Findings and Conclusions 2: Project Modalities ..................................................... 12 4.2.3 Finding and Conclusion 3: Project Structures ........................................................ 12 4.2.4 Findings and Conclusion 4: Program Approaches .................................................. 13 4.2.5 Findings and Conclusion 5: Lack of Technical Leadership ...................................... 14 4.3 Evaluation Question 1: Most and Least Effective Activities ............................................... 15 4.4 Evaluation Question 2: Effectiveness in Resolving Issues .............................................. 16 ii 4.5 Evaluation Question 3: Fostering Trust ......................................................................... 17 4.6 Evaluation Question 4: Marginalized Populations and GESI ......................................... 18 4.7 Evaluation Question 6: Sustainability ............................................................................ 19 5.0 Recommendations ................................................................................................................. 21 Appendix I: Evaluation Statement of Work .................................................................................. 23 Appendix II: Evaluation design ..................................................................................................... 33 Appendix III: Getting to Answers Matrix ...................................................................................... 50 Appendix IV: Detailed Methodology ............................................................................................ 52 Appendix V: Data Collection Instruments .................................................................................... 60 Appendix VI: Disclosure of Any Conflicts of Interest .................................................................... 72 Appendix VII: new State Structure of Nepal ................................................................................. 77 Appendix VIII: RUKUM Case Study ............................................................................................... 81 Appendix IX: Summary Table of Evaluation Findings ................................................................... 85 Appendix X: GESI Approach to CS:MAP: Critical Observations ………………………………………………..89 5 ACRONYMS AND ABBREVIATIONS ARI CA Advocacy Readiness Index Common Assembly CAC Citizen Awareness Centers CAG Content Advisory Group CBOs Community-based Organizations CDCS Country Development Cooperation Strategy CDO Chief District Officer CMC Conflict Mediation Centre CSOs CS:MAP Civil Society Organizations Civil Society Mutual Accountability Project DDRC District Disaster Relief Committees DEC Development Experience Clearinghouse DRG Democracy, Human Rights and Governance DO Development Objective EA Equal Access FGDs Focus Group Discussions FNJ Federation of Nepali Journalists FY Fiscal Year GESI Gender Equality and Social Inclusion GGB GI Good Governance Barometer Group Interview GON Government of Nepal HEAD Health, Education, Agriculture, Disaster Risk Reduction KIIs Key Informant Interviews KLL Kathmandu Living Labs ICNL The International Center for Not-for-Profit Law INSEC Informal Sector Service Sector ISO Intermediary Service Organization LDAG Listening, Discussion and Action Group LDO Local Development Officer LGBTI Lesbian, Gay, Bisexual, Transgender and Intersex LOE Level of Effort MITRA Samaj Measures for Intervention Training Research and Action Samaj MOFALD Ministry of Federal Affairs and Local Development MOWCSW Ministry of Women, Children and Social Welfare NEOC Nepal Election Observation Committee NEPAN Nepal Participatory Action Network NFN NGO Federation of Nepal NGO Non-governmental Organizations 6 NNSWA Nepal National Social Welfare Association NRA National Reconstruction Authority OCA Organization Capacity Building PMP Performance Monitoring Plan POC Public Outreach Campaign PPWG Public-Private Working Group SAP South Asia Partnership - Nepal SOW Scope of Work SWC Social Welfare Council TL Team Leader USAID United States Agency for International Development 7 TEAM MEMBERS AND ACKNOWLEDGEMENTS Team Members Kourtney Pompi, Evaluation Team Lead Sudhindra Sharma, National Team Lead/Governance Expert Indu Tuladhar, National Team Member, GESI Expert Prakriti Adhikari, Field Researcher Bijeta Acharya, Field Researcher Sumana Devkota, Field Researcher Akriti Rana, Translator and Field Researcher USAID’s Monitoring, Evaluation, and Learning (MEL) Support Manorama Adhikari, Deputy Chief of Party Basan Shrestha, Monitoring and Evaluation Specialist vii Acknowledgements This evaluation could not have been possible without the contributions of dedicated USAID, implementing partner, and CSO and media partner staff whose inputs were instrumental to the evaluation team’s efforts. The team expresses its deepest gratitude to Agreement Officer’s Representative Krishna Pathak and the Monitoring and Evaluation Specialist Bishwas Rana for their tremendous support and guidance from the evaluation’s inception to its close-out. The evaluation team also wishes to acknowledge the support it received from CS:MAP activity staffs. Specifically, the team expresses its appreciation for the generosity of Chief of Party Mr. Bishnu Sapkota and his extensive team for their willingness to welcome the evaluation team, provide critical input to numerous inquiries, supply project data, and facilitate introductions to partner organizations. The evaluation team would like to convey its appreciation for the staff of each organization and partner sampled for this evaluation, as well as the beneficiaries who made themselves available to participate in focus group discussions and interviews. These staff and beneficiaries consistently greeted the team with warmth and enthusiasm, and willingly provided the information upon which this evaluation and its findings, conclusions, and recommendations are based. 8 ABSTRACT This mid-term performance evaluation of the USAID/Nepal’s Civil Society: Mutual Accountability Project (CS:MAP) employed a mixed method approach to assess the interventions, processes, and approaches in the changed governance context and recommend future actions for course correction for optimum utility of the remaining project period. The evaluation team reached out to a total of 248 respondents; 176 males and 72 females using key informant interviews, group interviews, focus group discussions, and mini-surveys to answer six evaluation questions. Based on the evidence collected, the evaluation team concludes that while the current activities have had some demonstrable successes, CS:MAP has yet to contribute substantially to achieving the project goal: “to foster a more legitimate, accountable and resilient Nepali civil society that is capable of advancing the public interest.” This may be due to the evolution of the changing governance structures and thereby the gap generated between the original vision of the project and the changed reality. The shift from a long-standing centralized system of governance to a decentralized federalist system required elections for sub-national officials and a considerable shift in authority, decision-making and bureaucratic processes. The evaluation team finds, however, that CS:MAP leadership has yet to adjust the project to maximize opportunities or capitalize on leverage points across objectives that may have further entrenched or reinforced project goals and objectives. Based on the evaluation team’s findings, the team recommends that USAID and CS:MAP leadership undertake a review of the current project’s scope, including partnerships, activities and locations (districts and wards) to determine if a more effective design could be developed to more closely achieve results and objectives as stated in the current project logical framework. 9 EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the evaluation is to assess approaches and interventions carried out, and results achieved in support of the four project objectives so as to generate learning and recommendations to inform USAID’s future civil society and media strengthening programming. The evaluation focused analysis on how recent changes to the state have affected current program and how it may impact future programming going forward. The evaluation provides evidence to: a) capture project achievements, and reflect on the factors that made them possible; b) identify project limitations and shortcomings, as well as the challenges that CS:MAP has encountered to date; and c) build on the above to infer lessons that relate to the conceptualization, design, management, and implementation of further years of implementation for the CS:MAP program with a focus on potential pitfalls, opportunities, and suggested good practices to which future programming should remain alert. The evaluation examined the integration and participation of marginalized populations, including women. The evaluation provides recommendations to address the findings in the above areas of inquiry. PROJECT BACKGOUND CS:MAP’s theory of change (TOC) states: “If Nepali civil society and media organizations actively participate in organizational and technical capacity-building efforts to improve their legitimacy, accountability and resilience, then they would be able to advance the public interest.” Guided by the TOC, CS:MAP set as a goal to foster a more legitimate, accountable, and resilient Nepali civil society that can advance the public interest. The goal is to be achieved through establishing a favorable environment, enhancing the capacity of Nepali CSOs and media to constructively engage, enhancing coordination and oversight of public resource use and service delivery, and strengthening organizational capacity and sustainability of CSO and media. CS:MAP was developed prior to the passage of the Constitution of Nepal 2015 and the transition to federalism. To achieve project objectives, CS:MAP provides technical assistance and financial support to civil society and media to build their capacity to identify and implement projects that advance the public’s interest – including initiatives that provide overall capacity development, seek to strengthen the enabling environment for civil society and media, and support advocacy and government oversight activities that actively engage government on local issues. 10 EVALUATION QUESTIONS AND METHODOLOGY The methodology for this mid-term evaluation (MTE) was primarily qualitative. To answer six evaluation questions, the evaluation team collected data through the following methods: review of program documents, semi structured key informant interviews (KIIs), group interviews (GIs), and focus group discussions (FGDs) of people representing the same institution as well as beneficiary communities, and mini-surveys, reaching out to a total of 248 respondents; 176 males and 72 females. This methodology built on existing data and included a variety of respondent groups including implementing partners (IP), government stakeholders, and community members. Perceptions of a wide range of stakeholders ensure that Exhibit 1: Key Evaluation Questions To what extent is the design of CS:MAP’s hypothesis, theory of change and operating modalities structures, and approaches relevant to the new structure of the state? b) What are the modifications and/or re￾scoping required in the activity and modality, if any, given the changes in the governmentalstructures? biases were mitigated, with quantitative data allowing the MTE team to triangulate evidence. Data collection took place over the course of five weeks from Early May to mid-June 2017. Field work was conducted in four districts and Kathmandu. The evaluation team consisted a seven-person team (one expatriate evaluation team leader, one national team leader/governance expert, one national team member/GESI expert and four field researchers). The evaluation team followed a rigorous, consultative, participatory and inclusive approach to evaluation. The team collaborated closely through daily de-briefs and frequent analysis sessions. FINDINGS AND CONCLUSIONS Following more than four weeks of document review and in-person data collection, it is evident that the CS:MAP team and partners have put a lot of effort into the project. The project team and its partners worked hard to engage stakeholders at the national level on issues affecting the operating environment for civil society and media. They also have identified target communities and rural municipalities in 34 districts to engage on issues they care about. That work should not go unrecognized. The evaluation team found, through its mini-survey of the 14 district-based partners, that CS:MAP technical support was very useful. The evaluation team commends the CS:MAP team for their work to-date. Based on the evidence collected, the evaluation team determines, however, that while the current activities have some demonstrable successes, overall, CS:MAP efforts have yet to contribute substantially to the project’s results framework (see exhibit 4 in the main text). The changing political context in Nepal occurred just as the project was getting underway. The shift from a long-standing centralized system of governance to a decentralized federalist system required elections for sub-national officials and a considerable shift in authority, decision-making and bureaucratic processes. These changes were well outside of the control of CS:MAP’s implementing partners and delayed project start up. The evaluation team finds, however, that CS:MAP leadership did not adjust the project to maximize opportunities or capitalize on leverage points across objectives that may have further entrenched or reinforced project goals and objectives. RECOMMENDATIONS Based on the evaluation team’s findings, the team recommends that USAID and CS:MAP leadership undertake a review of the current project’s scope, including partnerships, activities and locations 11 (districts and wards) to determine if a more effective design could be developed to achieve results and objectives as stated in the current project’s results framework. Key recommendations include: 1. Partnerships across program objectives should be reviewed to maximize program outcomes in achieving CS:MAP’s Theory of Change. 2. CS:MAP should re-examine ward level engagement to take advantage of new municipal level decision-making power. 3. As CS:MAP is stretched too thinly across 34 districts, CS:MAP should re-consider the number of districts it works (both for partner reach and depth of work) and think strategically about where to work. The evaluation team suggests selecting a limited number of districts on which to focus CS:MAP efforts and selecting districts by explicit criteria agreed upon with USAID for maximum effect. 4. Consider working more deliberately and actively at the municipal level. For example, if local level entities think a specific municipal office is the decision maker for given issue, CS:MAP should work with partners to engage with that office to go through real-time solution/advocacy efforts. 5. There should be greater, deliberate involvement of CS:MAP’s national and local CSOs to work on issues identified at the local level. This could include leveraging national partners to work on issues that also have local impact. 6. CS:MAP should be encouraging organic interactions; not creating new structures that are outside of typical civil society interactions. It should foster what is already there, only create something new if there is a significant gap in engagement. 7. GESI partners need to be more fully integrated across all CS:MAP components so that they can provide adequate support to help GESI issues be integrated in a transformative manner. 8. CS:MAP should work with local partners to identify appropriate methods of support that help them link issues to appropriate decision-makers and then to provide technical expertise as they learn how to use those mechanisms to resolve local issues. 9. CS:MAP is encouraged to focus at the local level, where relationships can grow and strengthen faster than at a central level. 10. CS:MAP should encourage activities that build strong, positive relationships between civil society, media and government and should deemphasize activities that pit civil society against government, which social accountability tools can often do. 11. CS:MAP and USAID should build the capacity of, increase participation and provide specific intervention for marginalized populations. 12. CS:MAP should work with local CSO partners to identify and engage with existing, organic structures to provide demand-side led civic engagement opportunities. 13. Appropriate resources, both technical and financial should be provided to CSOs to help in the implementation of their GESI policies. 14. National GESI partners should be encouraged to engage with other national partners to mainstream GESI perspective in policy analysis and agenda setting of national advocacy efforts. 15. CSOs should be encouraged to mainstream GESI at the community level. 16. CS:MAP objectives and indicators should be reflective of GESI issues, including meaningful targets that measure gender equality and social inclusion. RECOMMENDATIONS FOR USAID/NEPAL 1. USAID should work closely with the IP of CS:MAP to determine the proper course correction for the remaining life of project. CS:MAP Mid-term Performance Evaluation 1 1.0EVALUATION PURPOSE In the spring of 2018, USAID/Nepal issued a statement of work (SOW – Appendix I) for mid-term performance evaluation of the CS:MAP. The overall goal of the evaluation was Exhibit 1: Evaluation Objectives to assess CS:MAP’s approaches, interventions, and results carried out in support of the four objectives noted in Exhibit 2 below. The primary purpose was to generate learnings and recommendations to inform the project’s remaining period and USAID’s future civil society and media strengthening programming particularly in the changed governance context. The evaluation provides evidence to: a) capture project achievements, and reflect on the factors that made them Assess the interventions, processes, and approaches in the changed governance context. Recommend future actions for course correction for optimum utility of remaining activity period. possible; b) identify limitations, shortcomings, and challenges that CS:MAP has encountered to date; and c) build on the above to infer lessons that relate to the conceptualization, design, management, and implementation of further years of implementation for CS:MAP with a focus on potential pitfalls, opportunities, and suggested good practices that future programming should consider adopting. The evaluation also examined the integration and participation of marginalized populations, including women. It is important to note that while the context for civil society and media in Nepal is changing dramatically, many lessons from CS:MAP’s experience will remain relevant going forward, and that identifying these lessons and taking them into account has much potential to strengthen future programming in this area. 2.0PROJECT BACKGROUND Through a cooperative agreement with FHI 360, USAID Nepal is supporting the implementation of CS:MAP. It is a $15 million, five-year civil society strengthening project that focuses on the objectives presented in Exhibit 2. CS:MAP is currently implemented in 34 districts, including 20 West, Mid-West and Far-West districts and 14 earthquake-affected districts of Nepal. CS:MAP aims to foster a more legitimate, accountable, and resilient Nepali civil society, capable of advancing the public interest. The project collaborates with a wide range of CSOs, including non-governmental and non￾profit organizations, non-governmental media outlets, member-based organizations, associations, unions, and federations. Exhibit 2: CS:MAP Objectives Objective One: Establish a favorable environment in which Nepali civil society and media operates. Objective Two: Enhance the capacity of Nepali CSOs and media to collaboratively and constructively engage in policy advocacy. Objective Three: Enhance coordination and effective civil society and media oversight of public resource use and service delivery. Objective Four: Strengthen organizational capacity and sustainability of CSO and media working in USAID priority sectors. CS:MAP was conceived, designed, and commenced before the passage of the constitution of Nepal 2015. Prior to then, Nepal’s system of governance was highly centralized with limited authority at the subnational level. CS:MAP was designed with that in mind. The theory of change for CS:MAP as stated in project documents is: “If Nepali civil society and media organizations actively participate in CS:MAP Mid-term Performance Evaluation 2 organizational and technical capacity-building efforts to improve their legitimacy, accountability and resilience, then they would be able to advance the public interest.” The CS:MAP theory of change addresses the problem statement that “Nepali civil society and media lack the desired legitimacy, accountability and resilience required to effectively advance the public interest.” As delineated in Exhibit 3 below, USAID and FHI 360 identified several root causes at the outset of this project. Exhibit 3: Root Causes for Lack of Desired Legitimacy, Accountability, and Resilience Root Causes for Lack of Desired Legitimacy Root Causes for Lack of Desired Accountability: Root Causes for Lack of Desired Resilience • The public does not perceive civil society and media as sufficiently legitimate and often associates them with political agendas. • Inability to influence public policy due to 1) lack of access to decision-makers; 2) lack of skills, such as ability to formulate and articulate policy recommendations based on evidence; and 3) lack of coordination. • Government of Nepal does not perceive civil society and media as sufficiently legitimate. • Lack of representation and inclusion of marginalized communities in project activities and internal policies. • Marginalized communities do not perceive CSOs and media as sufficiently legitimate. • Lack of internal self-regulation and governance mechanisms that contributed to the politicization of civil society and media sector that are beholden to political party and donor agendas rather than to the constituents they represent. • Heavy reliance on donor funding. • Lack of institutional capacity to effectively manage projects and resources and affect change in their communities. • Regulatory frameworks that often constrain civil society and media activities. To address these challenges, CS:MAP provides technical assistance and financial support to civil society and the media in Nepal to build their capacity in identifying and implementing projects that advance the public’s interest – including initiatives that provide overall capacity development, seek to strengthen the enabling environment for civil society and the media, and support advocacy and government oversight activities that actively engage government on local issues. In addition to direct technical assistance, CS:MAP provides grants to national, district, and local-level Nepali non-governmental organizations. It also sub-contracts local and international organizations to provide specific services and support. CS:MAP is implemented in 20 West, Mid-West, and Far-West districts and 14 earthquake-affected districts of Nepal. The 14 earthquake affected districts are Bhaktapur, Dhading, Dolakha, Gorkha, Kathmandu, Kavrepalachowk, Lalitpur, Makwanpur, Nuwakot, Okhaldhunga, Ramechhap, Rasuwa, Sindhuli, and Sindhupalchowk, all of which are USAID Nepal’s Country Development Cooperation Strategy (CDCS) focused districts. The other 20 CDCS focused districts include Achham, Baitadi, Dadeldhura, Doti, Kailali and Kanchanpur in the Far-West; Banke, Bardiya, Dailekh, Dang, Jajarkot, Rolpa, Rukum, Salyan and Surkhet in the Mid-West and Agrakhanchi, Kapilvasu Gulmi, Palpa, and Pyuthan in the West. Following the state restructuring, these CDCS focused districts fall under provinces 5, 6, and 7. The logic of the CS:MAP is based on the results framework that appears below in Exhibit 4. CS:MAP Mid-term Performance Evaluation 3 Exhibit 4: CS:MAP Results Framework CS:MAP Mid-term Performance Evaluation 4 3.1EVALUATION QUESTIONS AND METHODOLOGY Exhibit 5: Key Evaluation Areas • To assess the interventions, processes, and approaches in the changed governance context. • Recommend future actions for course correction for optimum utility of remaining project period. The full set of evaluation questions, sub-questions, measures, and data sources appear in table format in Appendix III: Getting to Answers Matrix. During the in-brief with USAID, the evaluation team learned that the Democracy, Human Rights and Governance (DRG) Office at USAID/Nepal was most interested in understanding what have been the activity’s most effective actions so far and what could the activity do going forward for program beneficiaries in the changed governance context. Subsequently, the evaluation team reframed the evaluation design to better understand the impact of the changed governance context on CS:MAP outcomes. The evaluation team proposed the evaluation questions be re-ordered from the agreed statement of work to better capture this new focus and so moved evaluation question five to the top as outlined below: EQ5. To what extent is the design of CS:MAP’s theory of change and operating modalities, structures, and approaches relevant to the new state structure? 5b) What are the modifications and/or re-scoping required in the activities and modalities, if any, given the changes in the governmental structures? Through this lens, the evaluation team sought data to answer the remaining evaluation questions: EQ1. Which components and approaches of CS:MAP are the most and least effective to date in fulfilling activity objectives, and why? 1b) Which activities/interventions have been the most effective and why? EQ2. To what extent and how have CSOs and the media been effective in supporting identification and resolution of issues of greater public interest? EQ3. To what extent and how have CS:MAP activities contributed to fostering trust among government, political, and civil society stakeholders? EQ4. To what extent and how is participation and representation of marginalized groups improving at the community level? EQ6. How is the activity progressing on long-term self-reliance through its sustainability and exit strategies? CS:MAP Mid-term Performance Evaluation 5 3.1 FRAMING THE EVALUATION The evaluation team understands that having informed citizens and an engaged civil society is essential to effective democratic governance as represented in Exhibit 6. As designed, the project components would influence and impact one another in opening the space necessary for meaningful engagement of civil society and the media to help identify and resolve Exhibit 6: Evaluation Framework Informed citizens National level policy and issues of public concern. Through these efforts, civil society and the media would, in turn, begin to change public perceptions of their roles in a democratic system of government as trusted agents of change focused on the common good. For example, as the legal framework and operating environment becomes more conducive to civil society and the media freely being able to press the government for greater accountability legal framework development Institutional infrastructure (necessary for oversight) Democratic governance Civil society engagement Inclusive local governance and transparency, citizens become more informed and demand government to be more accountable to issues of public concern. Their government institutions, in turn, must become more responsive and be able to play a more significant and positive role in social accountability. CS:MAP Mid-term Performance Evaluation 6 3.2 METHODOLOGY 3.2.1 DATA COLLECTION METHODS The evaluation team applied a mixed-methods approach, relying on qualitative and quantitative data to answer the evaluation questions. The evaluation team collected data through a review of program documents, semi structured key informant interviews (KIIs), group interviews (GIs), and focus group discussions (FGDs) of people representing the same institution as well as beneficiary communities, and mini-surveys. This methodology built on existing data and information and used different tools and techniques (KIIs, FGDs, and document review) for collecting qualitative evidence. Perceptions of a wide range of stakeholders ensured that biases were mitigated allowing the mid-term evaluation (MTE) team to triangulate evidence. A consultative, participatory and inclusive approach was followed. The evaluation team collaborated closely through daily de-briefs and frequent analysis sessions. It applied the following data collection methods to each of the identified respondent groups: KIIs or GIs with GON officials: Interviews were held with appropriate government officials at the national and district levels who interacted with the CS:MAP program activities. Refer to the separate deliverable on field instruments for the interview guide for key informant interviews. KIIs or GIs with CS:MAP staff: Interviews were held with the representatives of FHI 360 and two other implementing partners, Equal Access and the International Center for Non-profit Law, and included individuals involved in overall management, technical assistance, gender equity and social inclusion (GESI) issues, grants management, and monitoring and evaluation (M&E). GIs with Technical and Financial Assistance Recipients: The evaluation team conducted GIs with the several national coalitions, such as the Federation of Nepali Journalists (FNJ) and NGO Federation, and key national non-governmental organization (NGO) partners [i.e., Informal Sector Service Center (INSEC), Good Governance (GoGo) Foundation, Freedom Forum, South Asia Partnership (SAP) Nepal, Samjhauta Nepal with Media Advocacy Group (MAG)] and Intermediary Service Organization (ISO) partners [i.e., Nepal Participatory Action Network (NEPAN), Shtrii Shakti, Kathmandu Living Labs (KLL), Nepal National Social Welfare Association (NNSWA), and Mitra Samaj]. At the district level, the evaluation team held GIs and FGDs with CS:MAP district-level partners [Human Right Awareness and Development Center (HURADEC), Human Right Protection and Legal Services (HRPLSC), Media for Development and Social Change (FMDC), Development Exchange Center Nepal (DEC-Nepal) ]. Gls were also held with members of Public Private Working Groups (PPWGs) in the districts visited by the team. FGDs with beneficiary communities: The evaluation team conducted FGDs with local CS:MAP entity Common Assemblies (CAs), Content Advisory Group (CAG), Listening, Discussion, and Action Groups (LDAGs) that received technical and/or financial assistance under the activity. The evaluation team also held FGDs with CSOs not involved in CS:MAP’s activities to get a broader understanding of the environment for civil society and the media. CS:MAP Mid-term Performance Evaluation 7 Exhibit 7: Overview of Data Collection effort Type N KIIs- 23 GIs and FGDs 44 Mini Survey (phone￾based) 14 Total Respondents Men Women 248 176 72 3.2.2 SAMPLING STRATEGY The evaluation team conducted 23 key informant interviews, 44 group interviews and focus group discussions, and 14 mini-surveys. A total of 248 respondents, including 176 males and 72 females, provided feedback. The evaluation team conducted mini-survey via telephone with all district based CSOs (n=14) to better understand the direct technical assistance received from CS:MAP’s implementing partners and intermediary support organizations (ISOs). District Selection: CS:MAP was initially designed to be implemented in 20 core districts. Following the 2015 earthquake, an additional 14 districts from the earthquake affected areas were added. Given the available resources and limited time to complete the evaluation, the evaluation team limited data collection to four districts. It used the following sampling criteria to select the districts: • Accessibility to the evaluation team. • Mix of core and earthquake districts (with a priority focus on core districts). • Presence of marginalized communities. • Presence of mixed ethnic populations. • Intensity and innovation of activity interventions. • Perspectives of USAID and its IP. In applying the above criteria, the evaluation team selected two core districts and two earthquake affected districts, as presented in Exhibit 8. Exhibit 8: Sample District Province Rationale for Selection Kailali Province 7 • Core district • Presence of marginalized communities Rukum Province 5 and 6 • Core district • Presence of marginalized communities Dolakha Province 3 • Recovery district • Received especially intense activity interventions • Presence of mixed ethnic population Dhading Province 3 • Recovery district • Ease of access for evaluation team • Use of technology tools in CS:MAP activities In addition to these four districts, the evaluation team also interacted with CS:MAP’s CSO partner in Dang district, Rural Women Development Center (RWDC), which was headed by a female chairperson and whose executive committee and general assembly members comprised exclusively of women. CS:MAP Mid-term Performance Evaluation 8 3.2.3 LIMITATIONS There are a few limitations to the methods used and data collected during this evaluation which are listed below along with the evaluation team’s mitigation strategies. • Recall bias occurs when respondents provide inaccurate or incomplete recollections about past experiences. This can happen for a variety of reasons, including when informants participate in more than one activity intervention. This was the case for CS:MAP as most of the IPs and CSOs had multiple interventions and found it difficult to attribute some of the contributions from CS:MAP. To remind and refocus the respondents back to CS:MAP, the evaluation team probed and asked the same question differently as a means of validation of their responses. • Response bias is the risk that key informants may have been motivated to provide the evaluation team with responses that would be considered socially desirable or influential in obtaining donor support. The team expected that some respondents might have behaved this way knowing that that this evaluation would shape future activity opportunities and funding. The evaluation team strictly adhered to “do no harm” principles and made sure not to raise the expectation of the respondents. It is important to note that district-based CSO partners coordinated, and were present in some of the KIIs, GIs and FGDs. While the evaluation team believes that no harm was meant and the local CSO was merely interested in the responses of the participants, it may have influenced responses. • Selection bias is an inherent risk when implementers help to facilitate contact with activity beneficiaries because they may inadvertently select the most active, responsive, or engaged candidates of a project who may have a more positive view of the project than less engaged participants. For this evaluation, CS:MAP team provided most of the respondent groups and participating grantees’ and assisted in scheduling most of the appointments. It is possible that the IP may have selected respondents and grantees that it thought would provide more positive feedback about the effectiveness of CS:MAP. • Time Constraint: the evaluation team reviewed the documents that USAID and IP thought would be most useful and provided to the team. The evaluation team also reviewed other documents and reports provided by various partner organizations and those available in public platform and teased out the relevant information from these sources. Due to time limitation, the evaluation team could not review the documents beyond those sources. The evaluation team addressed the threat of bias through several measures by: 1) using multiple sources of data to triangulate data for each evaluation question; 2) combining information found in documents and interviews with multiple sources; and 3) ensuring, to the extent possible, that IP staff were not present for any KIIs, GIs, or FGDs during data collection. Consequently, the evaluation team is confident in the quality of the data. The responses were consistent across all respondent groups; there was no response that constituted outlier from the other information collected. CS:MAP Mid-term Performance Evaluation 9 4.0FINDINGS AND CONCLUSIONS Following more than four weeks of document review and in-person data collection, the evaluation team found that the CS:MAP implementer and its partners have engaged stakeholders at the national level on issues affecting the operating environment for civil society and the media. They have also identified target communities and rural municipalities to engage on issues their constituents care about. The evaluation team found, through its mini-survey of the 14 district-based partners, that CS:MAP technical support was considered very useful. Based on the evidence collected, the evaluation team concludes that while the current activities have had some demonstrable successes, CS:MAP has yet to contribute substantially to achieving the activity goal: “to foster a more legitimate, accountable and resilient Nepali civil society that is capable of advancing the public interest.” This may be due to the evolution of the country context and thereby the gap generated between the original vision of the project and the changed reality. The new federal structure called for the change in approach for program implementation, which the CSMAP team did not seem to realize as evidenced from the interviews with the team and the findings generated during data collection. (see Appendix VII: New State Structures of Nepal). The changing political context in Nepal occurred just as CS:MAP was getting underway. The shift from a long-standing centralized system of governance to a federal system with strong local level governments required a considerable shift in authority, decision-making, and bureaucratic processes. Moreover, elections at the local, province and federal level meant that the contextual changes were well outside the control of the project, but it affected in delaying the project’s start-up. Based on discussions with more than 200 respondents, the evaluation team came to learn that the effects of CS:MAP’s activities are very different than those described by the CS:MAP team. The evaluation team also identified several missed opportunities, particularly regarding the engagement of the key stakeholders and champions at both the national and new subnational levels. These missed opportunities also include several entry points with influencers and decision-makers, as well as different engagement tactics than those employed. For example, engaging subnational policy makers could help enact legislation and policy thereby reinforcing advocacy efforts conducted at the national level. Another example is including grassroots advocacy efforts with municipal level officials that could be carried up to district, provincial and national decision makers through CS:MAP partners. CS:MAP Mid-term Performance Evaluation 10 Exhibit 9: Evaluation Team’s Understanding of Project Framework Findings and conclusions regarding the five evaluation questions are listed below. 4.1 Evaluation Question 5: Relevance of Project to Changed Governance Structures Question: To what extent is the design of CS:MAP’s hypothesis, theory of change and operating modalities structures, and approaches relevant to the new structure of the state? Sub-question: What are the modifications and/or re-scoping required in the activity and modality, if any, given the changes in the governmental structures? 4.1.1 FINDINGS AND CONCLUSIONS 1: THEORY OF CHANGE AND PROGRAM DESIG Exhibit 10: CS:MAP Theory of Change The CS:MAP theory of change remains valid under the changed political context and should continue to guide the project. Since the core activities that Nepali civil society and the media are expected to undertake will not be different in the new structures of the state, the theory of change does not need to be reformulated. “If Nepali civil society and media organizations actively participate in organizational and technical capacity￾building efforts to improve their legitimacy, accountability and resilience, then they would be able to advance the public interest.” CS:MAP Mid-term Performance Evaluation 11 CS:MAP was developed to help achieve USAID/Nepal’s CDCS Development Objective (DO) 1: “More inclusive and effective governance.” The DO1 development hypothesis is presented in Exhibit 11. While not tasked with evaluating the validity of the DO with respect to the government changes, the evaluation team did not find any evidence to indicate that it should change. Exhibit 11: USAID CDCS Development Hypothesis Effective and inclusive governance, combined with investments in human capital and economic growth, will result in a more democratic, prosperous, and resilient Nepal. However, the evaluation team did find that the program design has hindered the project’s ability to be more successful. While the original design remains sound, the CS:MAP team, as of the timing of the evaluation, did not implement any changes to that design to consider the changing governance structures. For example, many decisions that affect the day- to-day life of Nepalis are now made at the municipal and district level, but CS;MAP continued to focus most of their subnational activities at the ward level i.e., the former Village Development Committee (VDC) level. The evaluation team finds that in practice, the activity is siloed – both in terms of national versus district-based work and within each of its four objectives. CS:MAP’s partners have limited interactions across program objectives – with national level partners not taking advantage of new entry points at a subnational level with district-based CS:MAP partners. This has resulted (with some exceptions) because coalition partners and national level NGOs are mandated to work on the first project objective to strengthen the enabling environment for civil society and the media. For example, while several of the national level partners mentioned working with district-based partners – either with CS:MAP’s partners or their respective network partners – none of them described how they have worked on specific advocacy efforts at a subnational level. Based on the construct of the new state structures, there are several opportunities where national level efforts can influence subnational efforts and vice versa. An example of a CS:MAP issue that has implications at all levels of government is registration of CSOs. CSOs and media partners working on the legal framework and regulatory environment related to civil society and media registration are only working on this issue at the national level. In the group interviews and KII, two national level partners mentioned they had shared sample legislation prepared by the Ministry of Federal Affairs and Local Development for subnational government (particularly to Local Level Government) entities to consider, but they did not conduct any advocacy or outreach. At the same time, district-based partners are engaging on issues that could be supported by working with CS:MAP’s partners at the national level. The evaluation team finds this to be a missed opportunity of the CS:MAP partner to engage on key issues across all levels of decision-making. At the same time, CS:MAP should have encouraged partners to engage on issues at all levels of government. The evaluation team also found some partners’ work to be duplicative of others based on their similar work scopes. FNJ and Freedom Forum, NGO Federation and INSEC, and NGO Forum and GoGo Foundation seem to be duplicating each other’s work. Also, advocacy is a major tactic being used by national level partners, but the evaluation team found no evidence that any advocacy was taking place at the State or Local Level. Finally, CS:MAP partners expressed concerns that the targeting of 34 districts is beyond the capacity of CS:MAP alone to effect sustainable change at the district level. The evaluation team concludes CS:MAP is stretched too thinly across 34 districts without sufficient strategic overlap with other USAID activities to be able to make a noticeable, let alone measurable, effect. The evaluation team concludes that CS:MAP’s partners’ failure to coordinate their activities with stakeholders at different levels of government is constraining CS:MAP from achieving its objectives. CS:MAP Mid-term Performance Evaluation 12 Because CS:MAP tends to operate in silos, its activities fail to take advantage of opportunities where partners could learn from and reinforce one another across objectives, topical issues, and levels of governance. For example, similar interventions under each of the four objectives rarely interact with one another, severely limiting the relationships CS:MAP can build with partners and decision-makers across all levels of government. 4.1.2 FINDINGS AND CONCLUSIONS 2: PROJECT MODALITIES With the changes in the structures of the state, the unit at which CSOs previously engaged and was envisaged in the original program design, were replaced by municipalities and wards within each municipality. Despite this change, CS:MAP chose to continue working at the ward level as planned, which under the new structure was equivalent to the previous VDC. By not working at the municipal level, the activity precluded itself from working with the primary decision-makers at the local level. For instance, CS:MAP formed CAs at the ward level. Though the CA can identify pertaining issues related to health, education, agriculture, or disaster risk reduction for the ward in which it is established, the identified issues may not be relevant for other wards of the municipality and, therefore, may not be addressed by municipal level decision-makers. This was specifically mentioned as a problem by district￾level CSO partners. Additionally, the evaluation team found that there were important missed opportunities for CS:MAP to influence municipal level policy framework (e.g. legislation, policies, guidelines, manuals, directives, etc.). Under the new constitution the Local Level governments have certain exclusive rights and authority to pass laws related to those rights (see Appendix VII: New State Structures of Nepal). During KIIs with local officials, the evaluation team learned that municipalities planned to deliberate legislation during their assembly meetings in July, August, and September. Several individuals that the evaluation team met with indicated that they would welcome support on drafting legislation and policy that would benefit their communities. Had CS:MAP worked with partners at the municipal level during the months preceding the scheduled assembly meetings, it could have influenced such legislation. 4.1.3 FINDING AND CONCLUSION 3: PROJECT STRUCTURES 1 The CA is the closest CS:MAP structure for the communities under the previous system of governance. The members of a CA are comprised of former Ward Citizen Forum members, former Citizen Awareness Center members, elected local officials, and members of marginalized communities. As described by the CS:MAP team, the CA is the body established to identify and address pressing issues at the local level and, if possible, to resolve them at that level. The CA is organized by district-based CSOs and assembles at local meeting spaces that include internet access. Within the CA, there are committees and point persons focused on identifying health, education, agriculture, and disaster risk reduction (HEAD) issues to take to relevant ward level officials for consideration. The evaluation team found that the CAs are bringing HEAD issues to the attention of ward level officials. Although CS:MAP staff and its CSO partners claim that CAs are involved in advocacy work to persuade ward level officials to address these issues, discussions with various stakeholders in several districts produced no evidence that the CAs or their committees were engaged in genuine advocacy. 1 For this evaluation, the evaluation team defines structures as the activity’s created entities: common assembly (CA), listening and discussion group (LDAG), content advisory groups, innovation centers, and private public working group (PPWG). CS:MAP Mid-term Performance Evaluation 13 Additionally, the evaluation team found that the CS:MAP structures were created in all 34 target districts regardless of whether that made sense for the local community context. The evaluation team found that this top-down approach was different from the method that district-based CSO partners used to determine which communities to engage. Rather than reinforcing the work of the CSOs in support of the project, CS:MAP’s formulaic approach to community engagement often weakened it. Another body established to engage officials at the district level is the PPWG. The group is overseen by the chairperson of the District Coordination Committee (DCC). Public sector members represent the government offices of health, education, agriculture and disaster assistance. Private sector members come from the district level Federation of Nepalese Chamber of Commerce and Industry (FNCCI) chapters 2. Although the evaluation team did not meet with PPWG as a group, it interacted with PPWG members and learned that due to the transition in state structures, the PPWG as an entity had not convened during the past few months. Because of the reduced position in the new constitution of the district level entities, including PPWG and others, their roles remain uncertain. During interviews with community members, the evaluation team also found that the structures created by CS:MAP to engage at the community level, such as CAs, LDAGs, , may not have been the structures preferred by community members. While respondents were not opposed to working through these entities, they could not confirm that these structures were their preferred mode for engaging local leaders. The evaluation team found that while some of the structures created by CS:MAP at the subnational level align with some of the new GON structures, others do not. For instance, the CAs and PPWGs, continue to focus on issues at the ward level and district level respectively whereas it is the municipal level where local decision-making authority is now vested. 4.1.4 FINDINGS AND CONCLUSION 4: PROGRAM APPROACHES The evaluation team found that CS:MAP has a large training component to strengthen the capacity of district-level, partner CSOs. The training plan was developed using several tools to assess the partners’ training needs, such as the Organizational Capacity Assessment (OCA), Advocacy Readiness Index (ARI), and the Good Governance Barometer (GGB). CS:MAP’s ISO partners trained partner CSOs in organizational development, evidence-based advocacy, social accountability, and GESI related topics. Some ISOs expressed concerns that the number of training topics and training days left little time for participants to apply their newly learned skills before participating in the next training session. ISO representatives also mentioned mentoring CSO partners to reinforce the skills they were taught during the training sessions. Based on interviews with CSO partners, the evaluation team found no evidence of mentoring by ISOs to CSO partners. Cascading also is a common approach used to share new knowledge with other similar organizations. The evaluation team found no evidence that CS:MAP or its ISO partners had helped district-based CSOs partners share newly acquired skills and knowledge with other CSOs based in the same district. Additionally, the evaluation team examined how GESI was incorporated in the project. A GESI assessment of Civil Society and Media of Nepal was carried out during its design. However, CS:MAP does not include 2 During the mid-term evaluation team’s fieldwork, CSOs were unable to bring all members of PPWG together – though they were able to fix meetings between the team and the concerned DCC officials – which points to problems in convening PPWGs. The team also noted that the FNCCI was not a partner of the CS:MAP activity and did not encounter its work in any other CS:MAP component. CS:MAP Mid-term Performance Evaluation 14 any GESI specific objectives or intermediate results (IRs) that findings from the GESI assessment would contribute towards. GESI-specific activities are included as a part of CS:MAP, such as: trainings on GESI, developing GESI assessment toolkits, research, mentoring, internal GESI policy development for each CSO partner. There are two achievement indicators set under IR 2.3, on the participation of women and marginalized groups. The MEL plan of CS:MAP developed two additional Intermediary Results to monitor GESI component: but the framework lacks indicators to determine if there is a change in women’s and socially excluded groups' access to resources, their meaningful engagement and/or participation in the project; or qualitative indicators to track the empowerment and inclusion of women and marginalized groups. The project has also developed a CS:MAP specific GESI Approach and Action Plan, which successfully identifies the social and political gaps in inclusion faced by people of Nepal of different gender, caste and religion. CS:MAP seeks to reduce such disparities among the marginalized populations by improving their access to and participation in local governance. By including women and marginalized groups in feedback committees, increasing their access to online platforms to raise their awareness of the social accountability mechanism, and ensuring inclusivity throughout its activities, CS:MAP is working to give women and marginalized groups equal space for voicing their interests to local decision makers. The evaluation team notes there is a GESI point person on the CS:MAP team and there is a concerted effort to integrate GESI issues in their program activities. Specialized GESI and ISO partners are providing technical support to local CSOs on GESI related issues. With the support of national level ISOs, all 14 district-based CSO partners conducted the GESI Audit. In response to the audits findings, several CSOs reported drafting internal policies for tackling GESI issues within their own organizations. However, the evaluation team found no evidence that the CSO partners were implementing those policies in the earnest. The evaluation team also finds that using social accountability tools as an approach to be concerning. In theory social accountability tools can be effective in holding government accountable, to increase transparency in government processes, and to raise public awareness on these issues. It is also noteworthy that these tools could produce interesting results given the structural changes in Nepal and the shift in budgetary resources to the sub-national level, as district-based CSOs should have access to information they could use in their social accountability related activities. The evaluation team concludes, however, that social accountability may not be the best approach to take, or if so, it should be well thought out for each organization and for each specific issue at hand. The evaluation team could find no evidence this was the case. In fact, the evaluation team was told that each of the 14 district-based partners were trained on 6 social accountability tools and were required to pick 2 to implement, with no discussion of which tools were appropriate for which situation. This is also concerning as social accountability activities often create an antagonistic, adversarial relationship between civil society and government. In a project where an objective is to foster a healthier perception of civil society, social accountability activities can run counter to that, as it may create strained relations between civil society and the government. The evaluation team, therefore concludes, that as designed, CS:MAP is not the right vehicle for furthering social accountability through its district-based CSO activities. The evaluation team concludes that CS:MAP’s current approach to changing the perceptions of civil society and media is not robust and the evaluation team saw no evidence of perceptions changing. 4.1.5 FINDINGS AND CONCLUSION 5: LACK OF TECHNICAL LEADERSHIP After conducting several GIs with the core CS:MAP team, it is evident that CS:MAP team members are knowledgeable about their areas of responsibility. They have a rich background in the theory of CS:MAP Mid-term Performance Evaluation 15 democracy, governance, human rights and inclusion. These are critical skills necessary for the success of a project like CS:MAP. While the evaluation team finds that the CS:MAP team is skilled in some technical areas, there is a significant skill gap in citizen advocacy and good governance programming. Many of the gaps in implementation, including readily identifying and taking advantage of openings for collaboration and reinforcement; developing strategic advocacy initiatives across objectives and layers of governance; and fostering and encouraging a civic culture that is engaged in the political process could be addressed by a capable civil society expert on the CS:MAP core team who has firsthand experience conducting such activities. The evaluation team also saw no evidence of synergies on topics that could have incorporated advocacy across national and subnational levels related to operating environment and space (registration for example or industry specific regulations). The evaluation team was informed about the advocacy letters at the national level (but not real advocacy). Advocacy is also an engagement tactic that can promote accountability and transparency in a way that can be more broad-based and less controversial than the social accountability tools that were being mobilized in the project. Advocacy and awareness raising on a variety of topics also means that civil society is occupying political space that may otherwise close if they are not engaged. This is an important tactic that the evaluation team found was missing from the overall implementation strategy of the CS;MAP. The evaluation team recognizes that there is a concerted effort to integrate GESI issues across objectives. However, the approach throughout this project on GESI has been accommodative rather than transformative. Through several KIIs and GIs with CS:MAP staff, national partners and ISOs, the evaluation team concludes that CS:MAP adopted an approach to GESI that acknowledges the role of existing gender norms and inequities and seek to develop actions that adjust to compensate around existing social norms. CS:MAP has not tried to challenge discriminatory/gendered nature of social norms and unequal power relations and gender roles – which would be more transformative in the longer term towards achieving desired GESI attitudes and behaviors. 4.2 EVALUATION QUESTION 1: MOST AND LEAST EFFECTIVE ACTIVITIES Question: Which components and approaches of CS:MAP are most effective (given the government structural changes) and which are less effective to date in fulfilling activity objectives, and why? Sub-question: Which activities/interventions have been the most effective and why? Given the above concerns that emerged from EQ 5, the evaluation team decided to focus on what was working well for this specific evaluation question. The evaluation team found that the media component at the subnational level is effective. Through speaking with media partners at the national and local level, the evaluation team found that the Sajha Boli radio program is providing much needed information on the new government structures and the key issues surrounding those changes. By design, Sajha Boli radio program is a powerful outlet for raising community needs to a large audience of elected officials and topical experts that have the authority and knowledge to help address those needs. Based on several GIs with local media participants (local radio staff, community reporters, and producers), the evaluation team concludes that value could be added if additional resources were put into hiring more experienced producers that could link community topics to subnational and national legal and policy issues. There was some evidence that this was done on a very local level, but the evaluation team concluded that it could be replicated across the entire CS:MAP project. CS:MAP Mid-term Performance Evaluation 16 The evaluation team found that CS:MAP and its partners are aligning current legal and policy reform efforts to international standards and practices at the national level. Through GIs with national coalition partners and CS:MAP implementing partner; the International Center for Not-for-Profit Law (ICNL); the evaluation team learned how the partners are incorporating international standards and best practices into the Nepali context. The evaluation team learned through GIs, that CS:MAP’s legal and policy analysis is focused solely at the national level, covering Right to Information (RTI) policies and those relating to the policy environment in which media and civil society operate. By not working on this component below the national level, CS:MAP is severely limiting its ability to effect policy changes at the subnational and municipal levels. Likewise, another limitation that the evaluation team noted was that though national level partners developed a checklist for policy analysis from GESI perspective, its ownership is limited among other national level partners. Through GIs, the evaluation team heard that partners report incorporation of improved self-governance into their organizational structures. This was largely self-reporting related to internal policy reviews and revisions to standard operating procedures around decision-making and financial reporting. Time will tell if these steps are meaningful enough in practice to begin to change perceptions of civil society’s purpose and functionality. Due to the delayed start of CS:MAP and the competing interests of other project objectives, the evaluation concludes that more time is needed to see if self-governance efforts are fruitful. The evaluation team conducted a mini-survey of all fourteen district-based partners to gather feedback on their specific activities under CS:MAP. More than half of the district-based CSOs (57%) reported that OCA was the most useful assistance they received. Thirty-five percent of CSOs reported advocacy training as the second most useful form of assistance. Only one of the fourteen CSOs reported training in social accountability as the most useful. This indicates that the project may need to re-evaluate the needs of selected partners and where they are as organizations with the goals and objectives of the project. If OCA is the most useful technical support local partners have received, local partners are likely not at a capacity level to take on higher capacity efforts, such as strategic advocacy or policy interventions. 4.3 EVALUATION QUESTION 2: EFFECTIVENESS IN RESOLVING ISSUES Question: To what extent and how have CSOs and the media been effective in supporting identification and resolution of issues of greater public interest? CS:MAP developed structures, such as CAs and LDAGs, are identifying issues in their assigned wards that respond to the needs of their ward. For example, Dolakha was focused on disaster risk reduction as it was an earthquake recovery district, while Rukum was focused primarily on health due to problems in accessing quality health care. As a forum, the CAs can create space where concerned people come together and talk about issues that need to be resolved in their communities. Issues that came up during the evaluation team’s fieldwork were the irregularity of teachers, unavailability of quality seeds for farmers, lack of free medicines at health posts, and delays in the earthquake relief compensation. Although the CAs and LDAGs have shown to be effective for identifying and raising issues of concern to local officials, their record of proposing solutions to the issues they raise is less evident. However, in a few cases, the CA was able to get an issue resolved by holding the proper authorities accountable. For CS:MAP Mid-term Performance Evaluation 17 instance, in Tribeni municipality of Rukum district, the CA was able to pressure local leaders to provide free medicines at the local health center. The evaluation team was unable to find sufficient evidence where CS:MAP structures, such as the CA, LDAG, and PPWG, were able to make decisions at the local level more transparent and accountable to the public, both of which are foundational elements of good governance and an integral part of CS:MAP’s goal. Through discussions with CSO partners, the evaluation team found that many of the issues raised by local level CS:MAP structures remain unresolved for two principal reasons: 1) some issues were too complex to resolve quickly; and 2) leaders at the ward level that were the recipients of the issues raised by CAs and LDAGs do not have the decision making authority to resolve the issues, as such authority is now vested with municipal officials under the new government structure. It is important to note that LDAGs’ links to radio programs has enabled the groups to influence radio program content to ensure that local issues are made aware to the larger public, including technical experts and local, subnational, and national level elected officials. The evaluation team found that the CA may be unable to resolve more complex issues because the government officials involved with the CA, typically ward chairpersons, are influencers and not actual decision-makers. The actual decision-making body at the local level is the municipality and since the evaluation team found no evidence of the CA engaging with the municipality level, there was no evidence of the CA engaging actual individuals that have the authority to resolve the types of public concerns that are being raised at the CA. The evaluation team found that current CS:MAP structures do not map on to new GON decision￾making structures. The CAs and LDAGs engage at the ward level. Though the ward level is the lowest level of local government, one where government structures interface with the people, it is not the appropriate local government decision-making body. Based on the evaluation teams’ understanding of the new governance structures (See Appendix VII: New State Structures of Nepal), the appropriate local decision-making authority is at the municipality level. CSMAP structures, however, has not engaged or interacted with this level of government, based on both program design and implementation. The evaluation team finds that the most important problem here is how issues of concern are identified within CS:MAP structures (i.e., the CA or LDAG) but current CS:MAP modalities do not lead to the right decision-maker. Likewise, the PPWG has similar problems. In circumstances where district level entities do not have decision-making authority, and where their role is mainly limited to monitoring and coordination, PPWG is not effective. This has also been explained in the section on evaluation question 5. 4.4 EVALUATION QUESTION 3: FOSTERING TRUST Question: To what extent and how have CS:MAP activities contributed to fostering trust among government, political, and civil society stakeholders? Building and strengthening relationships, especially ones with a history of distrust and misunderstanding, takes time. As CS:MAP has been implementing activities only for the last 18 months, the evaluation team believes it is still too early to know the extent to which trust has increased among government, political, and civil society stakeholders. The evaluation team did find evidence that both civil society and media partners are building relationships through their activities with government, political parties, and civil society stakeholders at CS:MAP Mid-term Performance Evaluation 18 the national and subnational levels. CSO partners also are working on their internal governance aspects rigorously so that they could be more transparent and trustworthy organizations. Further, through the work of CS:MAP structures, there is greater interaction between civil society and government officials at the ward level and at times also with municipal officials. Moreover, CS:MAP’s CSO partners also are working at the national level on policy issues that affect the operating environment for civil society and the media. The evaluation team finds, through both GIs and FGDs, that district-based CSOs and CS:MAP structures are engaging at the local level. The team finds this to be a step in the right direction in improving trust. . However, at this stage of CS:MAP implementation, it remains too early to measure the extent to which trust levels have changed among the three groups. The evaluation team developed a case study based on meetings with the district-based partner assigned to Rukum. Please see Appendix VIII: Rukum Case Study for additional insights into building trust between civil society and the government under the new federal structure. 4.5 EVALUATION QUESTION 4: MARGINALIZED POPULATIONS AND GESI Question: To what extent and how is participation and representation of marginalized groups improving at the community level? The evaluation team found that CS:MAP has put considerable effort in integrating GESI in the project. The evaluation team reviewed the GESI Assessment of Civil Society Organizations and Media in Nepal, which included the project’s overall problem statement that "Nepali civil society and media lack the desired legitimacy, accountability, and resilience required to effectively advance the public interest." As per the CS:MAP documents reviewed by the evaluation team, marginalized communities do not perceive CSOs and media as sufficiently legitimate, and, therefore, are reluctant to use them as vehicles to represent their interests. After a thorough review of the project documents and through KIIs, the evaluation team finds that CS:MAP is responsive to GESI issues. However, the overall focus on the GESI issues is found to be limited in certain aspects and heavily guided towards women and excluded groups’ representation in its mechanism. The evaluation team reviewed CS:MAP’s GESI Action Plan, which successfully identifies the social and political constraints that collectively lead to systematic marginalization of certain groups. The aim of CS:MAP is to improve the participation of marginalized groups in the political process. The new government structures which devolve authority to the local level provide an opportunity for CS:MAP to reach out to marginalized populations and bring them into the CS:MAP community-based structures. Other means by which CS:MAP is integrating GESI include reviewing existing legal norms and recommending changes to ensure equal rights for marginalized groups, increasing their access to online platforms for awareness raising of available accountability mechanisms, conducting GESI audits and taking actions after its findings, and ensuring and implementing inclusivity throughout all stages of CS:MAP programming. There is serious lack of discussion and agenda setting among CAs, LDAGs and PPWGs regarding the constitutional and human rights perspective of women and marginalized groups. Also, largely, members are not aware about these aspects, so discrimination, inequality against women and disadvantage groups in the field of health, agriculture, education and disaster reduction is less integrated within the larger discussion and agenda setting on GESI issues. Hence, CS:MAP structures must actively engage in discussion and agenda setting based on the gender and marginalized people’s perspective for enhancing inclusive advocacy agenda at national to local level. CS:MAP Mid-term Performance Evaluation 19 Throughout data collection, it became evident to the evaluation team that women participants were much less vocal and willing to speak than their male counterparts. Women expressed more difficulty than men (mainly related to family duties) in participating in CS:MAP activities and structures and were concerned that they did not have the requisite leadership skills to participate more actively. One of the CS:MAP’s ISO provides technical support exclusively on GESI, with its major responsibility of capacity building and mentoring the CSOs on GESI. In lieu of that it has developed the GESI Audit Tools. These tools were used to conduct GESI Audit for all14 CSO’s which aided the development of the GESI Policies. As a result, all the CSOs have GESI policy in place. The evaluation team was able to review some sample GESI policies developed by CSOs and all of them were found to be accommodative in their approach. They adopted the approaches that acknowledge role of existing gender norms and inequities and sought to develop actions that adjust to compensate around existing social norms. They did not however, seek to challenge discriminatory/gendered nature of social norms and unequal power relations and gender roles. Challenging the existing gender norms would have made these policies transformative. While the current GESI policies are positive initiatives, they do suffer from implementation challenges through a lack of resource planning and negligible budget, which hinders partners’ ability to implement. Another critical observation is that, technical support and mentoring on GESI is limited to CSOs at the district level. District based CSOs and national partners are yet to mainstream this approach within CS:MAP structures such as CAs, LDAGs and PPWGs as the evaluation team found these institutions too have limited understanding about GESI. 4.6 EVALUATION QUESTION 6: SUSTAINABILITY Question: How is the activity progressing on long-term self-reliance through its sustainability and exit strategies? The CS:MAP team believes that sustainability is ensured through continuity of the activities’ gains and achievements. When asked, the exit strategy was not well articulated by the CS:MAP team to the extent that it would lead to a clear path of sustainability. CS:MAP could be, however, sustained through different mechanisms. For example, some level of sustainability is ensured through government ownership of local bodies, such as CAs, and through engagement with grass root organizations in civic advocacy. Given the delayed start of the project due to the earthquake and local elections, the evaluation team finds that it is too soon to tell the lasting effects of the CS:MAP activity’s efforts, such as trainings. Through GIs with CSO partners, the evaluation team did learn that many have resource mobilization strategies in place, which ensures a certain level of continuity of some of the activities that the project is implementing. The evaluation team learned that several gaunpailkas were interested in absorbing the CAs into their own official government structures – so that the CAs became government entities. The CAs are currently organized and facilitated through CS:MAP district-based partners and CS:MAP’s Innovation Centers. This is a positive development, but CS:MAP would need to identify alternative, and hopefully more organic methods of involving civil society and citizens in local governance issues, which would ensure greater sustainability of the project’s gains and achievements. For this to occur however the current resource mobilization plan will need to be reviewed and likely revised to better align with the revised project design. CS:MAP Mid-term Performance Evaluation 20 Identifying champions and allies and providing them space for their active participation and engagement is key to ensuring the sustainability of a civil society project like CS:MAP. The evaluation team did not find the engagement of civic champions, such as civil society leaders, academics, civic activists, or any other influencers who could organically drive and take leadership to advocate on the community issues. The evaluation team finds this as a missed opportunity for leveraging like-minded actors to further the project’s goals. The evaluation team concludes that civic champions could be better engaged at the subnational level to further CS:MAP’s objectives. This would provide CS:MAP and its partners additional points to leverage in pressing decision makers to be more transparent and accountable to public concerns. Engaging champions not only ensures local advocacy issues are addressed more organically, but also serves the purpose of a point of continuity after the activity phases out. The project should play a facilitative role among civil society actors at the national and subnational levels in creating an enabling environment for raising and advocating for issues of public interest. Moving forward, CS:MAP should devise an exit strategy so that its gains and achievements are not lost after the activity phases out. The evaluation team concludes, based on information gleaned in project documents and through interviews that if the CA is absorbed by local government structures, there will be a gap of demand-side led civil society engagement. This could be remedied if the revised CS:MAP design encouraged civil society to engage communities and decisionmakers using more traditional methods that they have been using for years (with perhaps some enhancements from CS:MAP to strengthen those existing skills and behaviors). CS:MAP Mid-term Performance Evaluation 21 5.1 RECOMMENDATIONS 1. Partnerships across program objectives should be reviewed to maximize program outcomes. 2. CS:MAP should consider actively working at the municipal level, where local decision-making authority now resides. 3. As CS:MAP is stretched too thinly across its targeted 34 districts, diluting results, CS:MAP should consider a demonstrative model narrowing their geographical focus to a limited number of districts. The evaluation team recommends presence and strength of the local CSO partners as one of the important criteria CS:MAP should agree upon with USAID for district selection. 4. There should be greater coordination among CS:MAP national and local CSOs to work together on national issues that have a local impact, and local issues that can only be resolved by national level authorities. 5. For future work at the local level, CS:MAP should consider building on existing structures for civil society engagement with local officials rather than imposing CS:MAP developed structures that may not be supported by the community. 6. GESI partners need to be more fully integrated across all CSMAP components to further enhance GESI integration. 7. CS:MAP should identify appropriate technical support to fill gaps in building meaningful civic engagement at the national, subnational, and local levels. 8. CS:MAP should recruit an experienced, local civil society expert to fill an identified skill gap within the CS:MAP core team. Such an expert should have firsthand experience in working with CSOs to build and carry out advocacy campaigns, building and managing coalition and networks for common goals, and working at the grassroots to mobilize communities to action. 9. CS:MAP should provide additional technical support to help media partners connect community concerns to decision-makers at the appropriate level of government. 10. CS:MAP should provide technical support to partners working at subnational levels to assist them to more effectively engage decision-makers at the appropriate levels of government. 11. The project should immediately begin mentoring and providing individualized technical assistance to the CSO that have completed CS:MAP trainings to help them put into practice the new acquired skills and knowledge. 12. CS:MAP should encourage opportunities for civil society to be viewed as a resource partner to government. This could be done through policy recommendations, collaborative events with government officials, or sharing of other country experiences with efforts to engage civil society in government decision-making. 13. CS:MAP should reconsider its support to CSOs for social accountability activities as these CSOs are also delivering key services to communities which could create real or perceived conflicts of interest. 14. The project should revise sustainability plans based on recommended changes to CS:MAP’s scope and focus. 15. CS:MAP partners should identify and engage civil society champions at all levels of government to leverage their experience in furtherance of CS:MAP’s objectives. 16. “Sajha Boli” radio program should hire more experienced producers to link community topics to nub national and national policy issues to help elevate their importance to subnational and national decision-makers.” 17. CS:MAP should work with local CSO partners to identify and engage with existing, organic structures to provide demand-side led civic engagement opportunities. CS:MAP Mid-term Performance Evaluation 22 18. Appropriate resources – including technical and financial support should be provided to CSOs to help them implement their GESI policies effectively. 19. National GESI partner should be encouraged to engage with other national partners to mainstream GESI perspective in policy analysis and agenda setting of national advocacy efforts. 20. CSOs should be encouraged to mainstream GESI at the community level. 21. CS:MAP objectives and indicators should be reflective of GESI issues, including meaningful targets that measure gender equality and social inclusion. CS:MAP Mid-term Performance Evaluation 23 APPENDIX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK (SOW) Mid Term Evaluation for USAID/Nepal Civil Society: Mutual Accountability Project (CS:MAP) CS:MAP Identification Data Project Title: Civil Society: Mutual Accountability Project Cooperative Agreement No: AID-367-A-16-00007 Life of Project: April 2016 – April 2021 Implementing partner: FHI 360 Project Funding: $14,984,085 Agreement Officer’s Representative: Krishna Pathak BACKGROUND The presence and role of civil society organizations (CSOs) increased after the democratic movement and the 1990 constitution’s articles in support of civil society. Since then, CSOs emerged as a key influencer in instilling democratic values in the society, strengthening good governance practices, and becoming a voice for Nepal’s poor and marginalized groups. As democracy advances in Nepal, the roles of CSOs are becoming increasingly important to provide unbiased oversight, advance public interests, and engage with political actors and government authorities in advocacy based on evidences. Similarly, Nepali media continues to grow as an influential sector carrying the mandate of monitoring ongoing democratic process and informing citizens. Unfortunately, the decade-long armed conflict and royal rule enforced stringent media self-censorship. Threats and aggressive assaults on media personnel still go unchecked, making Nepal’s media only “partially free” as per the 2014 Freedom House’s Global Press Freedom Rankings which places Nepal as 118 out of 197 countries. The new constitution promulgated in September 2015 places greater importance on civil society. The constitution’s provisions advance civic freedom and liberties, strengthening the basis of participatory democratic governance for greater inclusion. Nepal’s civil society and media, thus, have stronger responsibilities to increase people’s awareness of their new rights, duties and enable them to increase their participation in governance processes. Even while the democratization process unfolds, Nepal’s government and people need to meet the social inclusion and governance challenges exacerbated by the country’s April 2015 CS:MAP Mid-term Performance Evaluation 24 earthquake. As of June 2015, the GON’s Post-Disaster Needs Assessment Executive Summary reported that the earthquake caused 8,790 deaths and 22,300 injured because of the quake, while country-wide, Nepal experienced 881, 986 damaged and destroyed private houses and public buildings. In total, 8.1 million people were affected by the disaster and the National Planning Commission had stated that an additional 700,000 people slipped into poverty because of their losses. Civil society played an immediate and vital role in responding to the disaster and holds potential to ensure equitable and responsive post-disaster assistance to those who need it most. As Nepal’s democratization process advances, civil society simultaneously is influenced by the process and plays a key role in overseeing how it progresses. Nepal’s transitional process has the potential to lead to significant and unpredictable changes and opportunities in the constitutional, political and legislative landscape of the nation. Civil society must be prepared to be flexible and responsive in reaction to these changes, from new forms of government with significant impacts on governance, and service delivery as well as elections. In this context, creation of a civil society prepared for active participation in Nepal’s transition, recovery, and its post-constitutional era is very crucial. USAID designed Civil Society: Mutual Accountability Project (CS:MAP), a five-year, $15 million project, to address above mentioned needs. CS:MAP has been implemented in 20 West, Mid￾West and Far-West districts and 14 earthquake-affected districts of Nepal. CS:MAP aims to foster a more legitimate, accountable, and resilient Nepali civil society, capable of advancing the public interest. The project will collaborate with a wide range of CSOs including non￾governmental and non-profit organizations, non-governmental media outlets, member-based organizations, associations, unions, and federations. PROJECT OBJECTIVES The purpose of the CS:MAP is to strengthen the enabling environment for Nepali civil society and media, enhance the capacity of Nepali Civil Society Organizations (CSO) to collaboratively and constructively engage in policy advocacy, and support broad-based, cross-sectoral initiatives that monitor, report, and demand greater effectiveness and accountability in governance and public service delivery. For the purposes of this Project, civil society includes non-governmental and non-profit organizations, non-governmental media outlets, member￾based organizations, associations, unions, and federations. USAID recognizes, therefore, that civil society is not a monolithic entity and that varying strategies and support options will need to be developed to engage the variety of actors in the sector, although the diverse actors in civil society also face similar challenges and rely on similar tools, legal frameworks and channels of communication to fulfill their role in a democratic system. The goal of this Project is to foster a more legitimate, accountable, and resilient Nepali civil society that is capable of advancing the public interest. USAID will support civil society strengthening activities, including core capacity development of select local partners, to achieve the following set of interrelated and self-reinforcing objectives: Four objectives are to; 1. Establish a favorable environment in which Nepali civil society and media operates. 2. Enhance the capacity of Nepali CSOs and media to collaboratively and constructively engage in policy advocacy. 3. Enhance coordination and effective civil society and media oversight of public resource use and service delivery. CS:MAP Mid-term Performance Evaluation 25 4. Strengthen organizational capacity and sustainability of CSO and media working in USAID priority sectors. KEY ACTIVITIES ● Strengthen CSO and media’s legal and policy frameworks to adhere to international standards. ● Initiate constructive engagement between CSOs and the Government of Nepal in policy work, planning, and performance across sectors. ● Strengthen capacity of CSOs and media to conduct participatory and evidence-based research and advocacy. ● Establish mechanisms and processes to improve bottom-up coordination between community, district, and national CSOs engaged in public service oversight. ● Raise the awareness of Nepalis including informing them on the use of available GON social accountability mechanisms. ● Train CSOs and media on greater engagement in the use of media and Information and Communications Technology (ICT) tools to strengthen public participation and oversight. ● Develop institutional governance standards for CSOs and media organizations. Figure 1. CS:MAP Results Framework 26 EVALUATION PURPOSE AND USE The purpose of this Mid-term Evaluation of the CS:MAP project is to determine whether the activity is on track to accomplish its intended goals and objectives and recommend course corrections if this is not the case and/or if adjustments should be made given Nepal’s state restructuring and political transition. The evaluation will also assess the degree to which there is evidence of sustainability of project results, and whether the approach appears to be advancing the Mission’s GESI policy. This evaluation will examine the effectiveness of approaches and relevance of project hypothesis and theory of change both in 20 CDCS districts and 14 earthquake affected districts. The evaluation will also draw lessons on how effective the project has remained in the changed political context and how should it go ahead in the remaining life of the activity. The mid-term evaluation will be primarily for the use of USAID and the CS:MAP Implementing Partner in terms of identifying best practices and lessons learned to guide project implementation. USAID will also use evaluation findings and recommendation to inform future programming decisions. EVALUATION QUESTIONS The following evaluation questions will guide the midterm evaluation team. In answering these questions, the evaluation team must analyze how the project is advancing benefits to the vulnerable groups with public resource and service delivery. 1. Which components and approaches of CS:MAP are most effective, and which are less effective to date in fulfilling activity objectives, and why? 1b) Which activities/interventions have been the most effective and why? 2. To what extent and how have CSOs and the media been effective in supporting identification and resolution of issues of greater public interest? 3. To what extent and how has the CS:MAP activities contributed in fostering trust among government, political and civil society stakeholders? 4. To what extent and how is participation and representation of marginalized groups improving at the community level? 5. To what extent the design of CS:MAP’s hypothesis, theory of change and operating modalities structures, modalities and approaches are relevant to the new structure of state? 5b) What are the modifications and/or re-scoping required in the activity and modality, if any, given the changes in the governmental structures? 6. How is the activity progressing on long-term self-reliance through its sustainability and exit strategies? 27 In addition to identifying unexpected opportunities and challenges, the evaluation will also identify, where possible, how changes to Nepal’s governance structure has affected CS:MAP’s design, interventions and expected outcomes. EVALUATION DESIGN AND METHODOLOGY The evaluation will follow a largely qualitative analytical research design. It will mainly rely on information from document review and qualitative information collected from key stakeholders and marginalized populations. Data collection methodologies • Document review including, Original program documents, including USAID’s solicitation, Implementing Partner’s final proposal; CS: MAP monitoring reports, special studies, and activity learning assessments. • Key informant interviews, focus group discussions, and/or other qualitative methods suggested by the evaluation team with key stakeholders and marginalized populations. The evaluation team may utilize short quantitative surveys with participating CSOs as needed. o Information should be collected from key stakeholders at the national level. o In-depth information should be collected preferably from at least two districts. Districts should be selected to represent a variety of situations such as: ▪ Geographic diversity (i.e. hill, plain) ▪ Presence of specific marginalized populations ▪ CDCS areas and earthquake affected areas ▪ Mix of high and low performing districts ▪ Coverage of CSOs across districts ▪ Inclusion of youth engagement activities The evaluation team will develop a description that maps data collection methodologies to evaluation questions via the questions-to-answers matrix, which builds on the illustrative table below. Evaluation question Illustrative methodology 1. Which components and approaches of CS:MAP are most effective, and which are less effective to date in fulfilling activity objectives, and why? 1b) Which activities/interventions have been the most effective and why? • Key Informant Interviews (KIIs); • Focus Group Discussions (FGD); including participatory exercises as appropriate • Desktop review of project documents • Mini-survey 2. To what extent and how have CSOs and the media been effective in supporting identification and resolution of issues of greater public interest? • Key Informant Interviews (KIIs); • Focus Group Discussions (FGD); including participatory exercises as appropriate • Desktop review of project documents • Mini-workshop 28 Evaluation question Illustrative methodology 3. To what extent and how has the CS:MAP activities contributed in fostering trust among government, political and civil society stakeholders? • Key Informant Interviews (KIIs); • Focus Group Discussions (FGD); including participatory exercises as appropriate • Mini-workshop 4. To what extent and how is participation and representation of marginalized groups improving at the community level? • Key Informant Interviews (KIIs); • Focus Group Discussions (FGD); including participatory exercises as appropriate • Desktop review of project documents 5. To what extent the design of CS:MAP’s hypothesis, theory of change and operating modalities structures, modalities and approaches are relevant to the new structure of state? 5b) What are the modifications and/or re￾scoping required in the activity and modality, if any, given the changes in the governmental structures? • Key Informant Interviews (KIIs); • Focus Group Discussions (FGD); including participatory exercises as appropriate • Desktop review of project documents 6. How is the activity progressing on long-term self-reliance through its sustainability and exit strategies? • Key Informant Interviews (KIIs); • Focus Group Discussions (FGD); including participatory exercises as appropriate • Desktop review of project documents Data analysis methodologies may include: • Comparative analysis of plan versus actual. • Cause and effect analysis/mapping on component-wise contributions. • Descriptive analysis of information received from stakeholders. • Stakeholder mapping EVALUATION TEAM The evaluation team should be composed of experts comprising a range of skills directly relevant to the purpose of the CS:MAP mid-term evaluation. USAID proposes the following composition, although relies on the Team Leader and MEL to make a final recommendation on the composition to fit the task: Team Leader: The Team Leader has demonstrated leadership and team management skills. S/he should have a postgraduate degree in related field, with at least 10 years of experience leading and/or evaluating development activities for USAID or other donor agencies, preferably with some experience in Asia. S/he has broad technical experience with the function and operation of civil society and media and their advocacy work, civil society and media institutional governance, capacity development and sustainability. S/he has specific knowledge of 29 Democracy, Human Rights and Governance (DRG) transition, and has profound analytical experience, which equips her/him to conduct high-quality and in-depth analysis of the civil society space and governance. The ideal candidate will have excellent oral and written communication skills in English; writing samples from recent evaluations or reports, and three professional references, will be requested. CSO/Governance Expert (National Team Leader): The CSO/Governance Expert must have extensive experience working with USAID or other donor agencies. A Nepali citizen, the expert must have a minimum of master’s degree in management, social science, development studies or relevant subject area, and at least 7 years of experience as a member of an evaluation team including national lead in at least one of the evaluation team. Experience leading or serving as senior technical expert in civil society and media’s work in the areas of use of social accountability tools, public policy analysis and advocacy, governance reforms/oversight and CSO’s own governance reforms and capacity development is required. The ideal candidate will have excellent oral and written communication skills in English; writing samples, and three professional references, will be requested. GESI expert (National Team Member): Gender Equality and Social Inclusion (GESI) expert must have a minimum of master’s degree in social science or relevant subject area, and at least 5 years of experience as a member of an evaluation team. The GESI expert must have an extensive knowledge about the GESI issues in Nepal. GESI expert will work with the evaluation team to develop evaluation methodology to assess the how the project has affected women and vulnerable groups. The expert should have a good knowledge of local governance and development sustainability. The ideal candidate will have excellent oral and written communication skills in English; writing samples, and three professional references, will be requested. Enumerators/Field Researchers (3): The team will include three (3) Nepali Field Researchers with a master’s degree or equivalent in social sciences, organizational development including governance/women’s studies/ development studies, from a recognized national or international university. They must have at least three years of field experience working in the development sector with a focus on governance, civil society activities and capacity building. The researchers will be selected considering their gender, social background and language prevalent in the proposed sample districts. 30 KEY DOCUMENTS FOR REVIEW Project documents, such as the following: • Original USAID Solicitation • Final proposal from Implementing Partner • Annual work plans • CS: MAP M&E plan • Quarterly and Annual reports to USAID from IP • Baseline and formative assessment reports • PPWGs and Sajha Sabhas reviews • CS:MAP central level database reports • Semi-annual review of indicators reports • Reports of continuous cycle of action and reflection • Quarterly rolling assessment reports • EA quarterly reports • Grantee quarterly reports • MeroReport portal reports • Annual OCA/OPI reports • Quality assessment reports Thematic reports such as: • Public perception surveys • EA’s broadcast audience survey of 2014 • COSI (Civil Society Organization Sustainability Index) survey • Gender equality and social inclusion audit • IGP (Integrated Governance Project) midterm evaluation KEY STAKEHOLDERS TO BE CONSULTED • USAID staff • CS:MAP FHI360 staff (including program, M&E, and management) • Local implementing partners (i.e. Equal Access - EA); Nepal National Social Welfare Association -NNSWA and Measures for Intervention Training Research and Action Samaj - MITRA Samaj) • CSOs (grant recipients, other groups) • Government officials (central line agencies and local governments) o Given the changes to local governance structures, careful consideration must be given to which local levels should be included (i.e. provincial, departmental, municipal, Gaunpalika, community) • Community members • Marginalized populations • USAID-funded partners operating locally • Other relevant donors • Other relevant NGOs and INGOs 31 • Individual civil society leaders/champions at district (and central) level TIMELINE FOR EVALUATION The following is a tentative timeline for the evaluation tasks; the detailed timeline will be developed by evaluation team as part of finalizing the evaluation plan. The evaluation is expected to start in April 2018 and be completed by September 2018. Main Activities March April May June July Aug Sept Recruit and onboard TL and national experts Finalize SOW Conduct design meetings with key stakeholders Review secondary data, reports, any other relevant materials for drafting the design and tools Draft evaluation design Finalize evaluation design Recruit field researchers Draft data collection tools Finalize data collection tools In-briefing with USAID Train use of tools Data Collection Collate, compile, and analyze data Provide initial, topline findings to USAID & IP following field work X Drafting report, internal review, revision, initial copy editing, Submit draft report to USAID X Receive consolidated comments from USAID, meeting to prioritize Revise final report; internal review and revisions; final copy editing; submit to USAID for approval X DELIVERABLES The following is a set of evaluation deliverables, with the expected timing of each adhering to the above Evaluation Timeline. 1. An in-briefing with USAID/Nepal upon the Team Leader’s arrival in Nepal 2. An evaluation design containing the proposed evaluation outline; the evaluation methodology, implementation schedule and deliverables; to be delivered within one week of the Team Leader’s arrival in Nepal 32 3. A set of data collection instruments; and field work schedule, to be delivered within one week after receiving USAID/Nepal’s feedback on evaluation design 4. A presentation to USAID (and another to senior CSMAP management) of the field work, to be delivered within 12 weeks of the Team Leader’s arrival in Nepal. Preliminary, topline findings will be presented to USAID before the Team Lead’s departure from Nepal. 5. A draft evaluation report of not more than 30 pages in length, single-spaced in Gill Sans MT or Arial 11-point font, excluding annexes, with an executive summary of 3-5 pages in length, within 16 weeks of the Team Leader’s arrival in Nepal 6. A final evaluation report of not more than 30 pages in length, single-spaced in Gil Sans MT or Arial 11-point font, excluding annexes, with an executive summary of 3 to 5 pages in length, within three weeks of agreement on prioritized, consolidated comments in electronic format from USAID. 7. A final presentation by MEL for USAID/stakeholders after the evaluation report is finalized. EVALUATION REPORT OUTLINE The following page numbers should serve as approximate guidelines with a maximum page count of 30 pages. I. Executive Summary (3-5 pp.) II. Evaluation Purpose and Questions (max 1 pp.) III. Project Background (max 1 pp.) IV. Evaluation Methods and Limitations (3 pp.) V. Findings by each evaluation questions (20 pp.) VI. Conclusions (max 2 pp.) VII. Recommendations (3 pp.) Annexes: A. Evaluation Statement of Work (SOW) B. List of documents Reviewed C. List of Organizations Met D. Data Collection Instruments (e.g. Interview guides, data extraction formats) E. Description of Data Utilized in the Evaluation F. Disclosure of Any Conflicts of Interest G. Unresolved Issues from Evaluation Review H. Statement of Differences [only if applicable] 33 APPENDIX II: EVALUATION DESIGN USAID/Nepal – Civil Society: Mutual Accountability Project (CS:MAP) Mid-term Performance Evaluation Design May 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International. 34 ACRONYMS AND ABBREVIATIONS CA Common Assembly CAC Citizen Awareness Centers CAG Content Advisory Group CBOs Community-based Organizations CDCS Country Development Cooperation Strategy CDO Chief District Officer CMC Conflict Mediation Centre CSOs Civil Society Organizations DDRC District Disaster Relief Committees DEC Development Experience Clearinghouse DO Development Objective EA Equal Access FGDs Focus Group Discussions FNJ Federation of Nepali Journalists FY Fiscal Year GESI Gender Equality and Social Inclusion GON Government of Nepal HEAD Health, Education, Agriculture, Disaster Risk Reduction KIIs Key Informant Interviews KLL Kathmandu Living Labs ICNL The International Center for Not-for-Profit Law INSEC Informal Sector Service Sector ISO Intermediary Service Organization LDAG Listening, Discussion and Action Group LDO Local Development Officer LGBTI Lesbian, Gay, Bisexual, Transgender and Intersex LOE Level of Effort MITRA Samaj Measures for Intervention Training Research and Action Samaj MOFALD Ministry of Federal Affairs and Local Development MOWCSW Ministry of Women, Children and Social Welfare NEOC Nepal Election Observation Committee NEPAN Nepal Participatory Action Network NFN NGO Federation of Nepal NGOs Non-governmental Organizations NNSWA Nepal National Social Welfare Association NRA National Reconstruction Authority PMP Performance Monitoring Plan POC Public Outreach Campaign 35 PPWG Public-Private Working Group CS:MAP Civil Society: Mutual Accountability Project SOW Scope of Work SWC Social Welfare Council TL Team Leader USAID United States Agency for International Development VDC Village Development Committee 36 I. INTRODUCTION PROJECT BACKGROUND Through a contract with FHI 360, USAID Nepal is supporting the implementation of the Civil Society: Mutual Accountability Project (CS:MAP), a $15 million, five-year civil society strengthening activity that focuses on the objectives presented in Exhibit 2. CS:MAP is currently implemented in 20 West, Mid-West, and Far-West districts and 14 earthquake-affected districts of Nepal. CS:MAP aims to foster a more legitimate, accountable, and resilient Nepali civil society, capable of advancing the public interest. The project collaborates with a wide range of civil society organizations (CSOs), including non-governmental and nonprofit organizations, non-governmental media Exhibit 1: Civil Society: Mutual Accountability Project – Activity Objectives Objective One: Establish a favorable environment in which Nepali civil society and the media operate. Objective Two: Enhance the capacity of Nepali CSOs and the media to collaboratively and constructively engage in policy advocacy. Objective Three: Enhance coordination and effective civil society and media oversight of public resource use and service delivery. Objective Four: Strengthen organizational capacity and sustainability of CSOs and the media working in USAID priority sectors. outlets, member-based organizations, associations, unions, and federations. Exhibit 2: CS:MAP Results Framework 37 Exhibit 3: CS:MAP – Hypothesis/Theory of Change The following statement represents the CS:MAP theory of change (TOC): If Nepali civil society and media organizations actively participate in organizational and technical capacity￾building efforts to improve their legitimacy, accountability, and resilience, then they would be able to advance the public interest. The CS:MAP TOC follows from the problem statement and the root causes underlying the problem – all of which were identified during the design stage of the project: Problem Statement: The Nepali civil society and media lack the desired legitimacy, accountability, and resilience required to effectively advance the public interest. Root Causes for Lack of Desired Legitimacy: Politicization: • Political party agendas, as opposed to public needs, drive advocacy agenda and media reporting → public does not perceive civil society and media as sufficiently legitimate. • Inability to influence public policy due to: 1) lack of access to decision-makers; 2) lack of skills, such as the ability to formulate and articulate policy recommendations based on evidence; 3) lack of coordination → Government of Nepal does not perceive civil society and media as sufficiently legitimate; and lack of representation and inclusion of marginalized communities in project activities and internal policies → Marginalized communities do not perceive CSOs and media as sufficiently legitimate. Root Causes for Lack of Desired Accountability: • Lack of internal self-regulation and governance mechanisms that contributed to politicization of the civil society and media sector that are then beholden to political party and donor agendas rather than to the constituents they represent. Root Causes for Lack of Desired Resilience: • Heavy reliance on donor funding. • Lack of institutional capacity to effectively manage projects and resources and affect change in their communities. • Regulatory frameworks that often constrain civil society and media activities. To achieve these objectives, CS:MAP provides technical assistance and financial support to civil society and the media to build their capacity to identify and implement projects that advance the public’s interest – including initiatives that provide overall capacity development, seek to strengthen the enabling environment for civil society and the media, and support advocacy and government oversight activities that actively engage government on local issues. In addition to direct technical assistance, CS:MAP relies on grants to national-level Nepali non-governmental organizations (NGOs) as well as to district- and local￾level ones. It also involves sub-contracts to local as well as international organizations to provide specific services and support. CS:MAP is implemented in 20 West, Mid-West, and Far-West districts and 14 earthquake-affected districts of Nepal. The 14 earthquake affected districts are Bhaktapur, Dhading, Dolakha, Gorkha, Kathmandu, Kavrepalachowk, Lalitpur, Makwanpur, Nuwakot, Okhaldhunga, Ramechhap, Rasuwa, Sindhuli, and Sindhupalchowk. The 20 target districts in the Western region also are the country development cooperation strategy (CDCS) focus districts. The districts in the Far-Western region include Achham, Baitadi, Dadeldhura, Doti, Kailali, and Kanchanpur. Likewise, the target districts in the Mid￾Western region are Banke, Bardiya, Dailekh, Dang, Jajarkot, Rolpa, Rukum, Salyan and Surkhet. Agrakhanchi, Kapilvasu Gulmi, Palpa, and Pyuthan. Post the state restructuring, these Western CDCS districts fall under provinces 5, 6, and 7. 38 EVALUATION PURPOSE In the spring of 2018, USAID/Nepal released a scope of work (SOW – Appendix 1) for a mid-term performance evaluation of the CS:MAP activity. The overall goal of the evaluation is to assess approaches, interventions, and results carried out in support of the four objectives noted in Exhibit 1to generate learning and recommendations to inform USAID’s future civil society and media strengthening programming. The evaluation will provide evidence to: a) capture project achievements, and reflect on the factors that made them possible; b) identify project limitations and shortcomings, as well as the challenges that CS:MAP has encountered to date; and c) build on the above to infer lessons that relate to the conceptualization, design, management, and implementation of further years of implementation for the CS:MAP program with a focus on potential pitfalls, opportunities, and suggested good practices to which future programming should remain alert. The evaluation also will examine the integration and participation of marginalized populations, including women. Further, the evaluation will explore how recent changes to the state have affected current programming and how it may impact programming going forward. The evaluation will provide recommendations to address the findings in the above areas of inquiry. The evaluation is predicated on the assumption that while the context for civil society and media in Nepal is changing rapidly and dramatically, many lessons from CS:MAP’s experience will remain relevant going forward, and that identifying these lessons and taking them into account can strengthen future programming in this area. 39 II. FRAMING THE EVALUATION CONCEPTUAL AND OPERATIONAL FRAMEWORK CS:MAP was developed to help achieve USAID/Nepal’s CDCS Development Objective (DO) 1: “More inclusive and effective governance.” The DO1 development hypothesis is presented in Exhibit 4. CS: MAP was expected to contribute to DO 1 through mutually complementary gains in civil society, media, cooperation with government, and gender equality and social inclusion (GESI). To respond to the GESI analysis, the evaluation may consider how the program affected the participation and representation of marginalized groups 3 in the following areas: Exhibit 4: USAID CDCS Development Hypotheses CDCS Development Hypothesis Effective and inclusive governance, combined with investments in human capital and economic growth, will result in a more democratic, prosperous, and resilient Nepal. • CS-MAP GESI policy and framework. • Program design (GESI related program and budget). • Specific program intervention related to GESI (mainstream and stand-alone) and identification of those interventions and approaches that were most effective in empowering the selected marginalized populations. • Constraints in the program process (critical issues related to GESI during the program intervention and monitoring). • Monitoring mechanism (how that ensured the GESI objective of the program intervention achieved). Revised Evaluation Framework For CS:MAP Mid-Term Performance Evaluation During the May 4th in-brief with USAID, the evaluation team learned that USAID/Nepal’s DRG office is most interested in knowing, given the changes to the governance structures in Nepal since the CS:MAP project began, what have been the most effective project activities in terms of those changes and what could the project do going forward for program beneficiaries with the changes in mind. Considering this, and in agreement with the Mission, this mid-term performance evaluation reframes the evaluation design to focus on understanding the changes. Through the lens of the recent government structural changes at the national and subnational levels, the evaluation team re-ordered the evaluation questions to better capture this focus. Thus, the mid-term evaluation will not be the standard inventory of what has worked/not worked to date, but rather what interventions have been most useful given the governance changes and what the project partners could use more of given the changes. By using Evaluation Question (EQ) 5 as a guiding frame, the evaluation team will focus areas of inquiry to better understand the activities that CS:MAP has implemented: which are most and least useful and what approaches may be more helpful to CSOs and media partners for the remainder of the CS:MAP project. This will provide the evaluation team 3 Based on "A Common Framework for Gender Equality & Social Inclusion." GESI Working Group, 2017, published by Gender Equality and Social Inclusion Working Group, International Development Partners Group, Nepal. The definition of who the excluded in program districts will be drawn from Sajhedari documents and aligned with the list provided in the Common GESI Framework. 40 with the necessary data to provide recommendations for the project going forward and any course corrections as appropriate. Therefore, the evaluation team presents below the re-ordered evaluation questions, with slight modifications to text found in the approved SOW, as indicted in red (please also refer to Appendix 2 for the evaluation questions). Primary question for the CS:MAP Mid-Term Performance Evaluation is: EQ5. a. To what extent is the design of CS:MAP’s hypothesis, theory of change and operating modalities, structures, and approaches relevant to the new structure of the state? b. What are the modifications and/or re-scoping required in the activity and modality, if any, given the changes in the governmental structures? Through this lens, the evaluation team will seek data to answer the remaining evaluation questions: EQ1. a. Which components and approaches of CS:MAP are most and least effective (given the government structural changes) to date in fulfilling activity objectives and why? b. Which activities/interventions have been the most effective and why? EQ2. To what extent and how have CSOs and the media been effective in supporting identification and resolution of issues of greater public interest? EQ3. To what extent and how has the CS:MAP activities contributed in fostering trust among government, political, and civil society stakeholders? EQ4. To what extent and how is participation and representation of marginalized groups improving at the community level? EQ6. How is the activity progressing in long-term self-reliance through its sustainability and exit strategies? 41 III. METHODOLOGY AND SAMPLING METHODOLOGY AND METHODS The evaluation team’s plan is to primarily use a qualitative methodology. However, it also may conduct quantitative data collections via a mini-survey, likely over the phone, to capture additional data and extend the research of the sample to capture specific data from CSO partners. To answer the evaluation questions presented in Exhibit 5, the evaluation team will collect data through the following methods: review of program documents, semi structured key informant interviews (KIIs), focus group discussions (FGDs) of people representing the same institution as well as beneficiary communities, and mini-surveys. This methodology builds on existing data and information and uses different tools and techniques (KIIs, FGDs, and document review) for collecting qualitative evidence. Perceptions of a wide range of stakeholders ensures that biases are addressed, and quantitative data and information enables triangulation of evidence. A systematic and structured approach for information compilation and processing ensures that field work evidence is well organized and effectively used for the evaluation report. A consultative, participatory and inclusive approach will be followed. The team will closely collaborate with the Mission for guidance as needed. The team’s data collection processes are outlined in Appendix 3. National Level: Data collection in Kathmandu will include semi-structured interviews, consultation meetings, and FGDs with USAID, CS:MAP staff, national-level implementing partners, different ministries of the government, other government entities 4 , provincial government representatives, other donor partners working in the civil society and media strengthening space, and the National Reconstruction Authority (NRA). In addition, the evaluation team will undertake semi-structured interviews with key civil society leaders at the national level who are not linked with CS:MAP. Appendix 4 has the proposed national sample. District Level: Field work is planned in three districts. A six-person team (one expatriate evaluation team leader, one national team leader/governance expert, one national team member/GESI expert, and three field researchers), will conduct the field work. The team will meet and interact with the concerned CSO partner, CDO or representative from the CDO office, mayor/deputy mayor of the municipality, other elected officials, Public Private Working Group, innovation centers, content advisory groups, and other prominent civil society members, champions, and media based at the district level. The inputs of these entities working on governance and GESI will be sought regarding CS:MAP's work and lessons for CS:MAP specifically, and civil society strengthening programming generally, going forward. At the field level, interactions will be held with members of the Common Assembly, including health, education, Agriculture and Disaster Risk Reduction (HEAD) focal points; listening, discussion, and action groups (LDAG); and elected ward chair-persons/representatives. Evidence from the field will provide insights on the experiences of a diverse group of stakeholders regarding CS:MAP's achievements to date and challenges. 4 This may include the Ministry of Information and Communication, Information Commission, Ministry of Women, Children and Social Welfare, Social Welfare Council, Ministry of Law, Justice and Parliamentary Affairs, and others in consultation with USAID and the IP. 42 Data Collection Types and Targets: Due to time limitations, data collection types and targets may differ somewhat across districts but may include the following five types of data collection: KIIs or GIs with GON Officials: Interviews and meetings will be held with appropriate government officials at the national and district levels that have interacted with the CS:MAP program activities, including HEAD. (Appendix 5 lists the proposed officials/offices at the national and subnational levels with whom the evaluation team intends to meet.) Refer to the separate deliverable on field instruments for interview guide for the KIIs. KIIs or GIs with CS:MAP staff: Interviews and meetings will be held with appropriate CS:MAP staff at the national and regional levels and will include those involved in overall management, technical assistance, GESI issues, grants management, and M&E (Appendix 4 lists the proposed people the evaluation team intends to meet at various levels). KIIs/FGDs with Implementing Partners: Interviews and meetings will be held with appropriate implementing partners (EA and ICNL) at the national and regional levels and will include those involved in management, capacity building and technical assistance, GESI issues, grants management, and M&E (Appendix 4 lists the proposed people the evaluation team intends to meet at various levels). FGDs and GI with Technical and Financial Assistance Recipients: The evaluation team plans to conduct GIs and FGDs with main national coalitions, including the Federation of Nepali Journalists (FNJ) and NGO Federation, main national NGO partners [i.e., Informal Sector Service Center (INSEC), GoGo Foundation, Freedom Forum, South Asia Partnership (SAP) Nepal, Samjhauta Nepal with the Media Advisory Group (MAG)] and Intermediary Service Organization (ISOs) [i.e., Nepal Participatory Action Network (NEPAN), Stri-Shakti, Kathmandu Living Labs (KLL), Nepal National Social Welfare Association (NNSWA), Mitra Samaj, etc.]. At the district level, KIIs will be held with partner CSOs, PPWG, innovation centers, and content advisory groups. As appropriate, the evaluation team will conduct short interviews with recipients to capture data unique to their experience. FGDs with Beneficiary Communities: The evaluation team will hold FGDs with local entities, such as CAs and LDAGs, that received technical and/or financial assistance. As time permits, it also will hold FGDs with CSOs beyond the CS:MAP project to get a broader understanding of the environment for civil society and media. Exhibit 6: Overview of Data Collection Effort Type N HQ District and Local Level KIIs/ Meetings TBD GON officials, CS:MAP staff, USAID, and implementing partners GON officials, CSOs, elected officials, implementing partners, and sub-awardees GIs or FGDs TBD CS:MAP staff and implementing partners Elected officials, sub-awardees, Common Assembly members, and implementing partners Mini Survey (phone-based) TBD CSOs 43 SAMPLING STRATEGY District Selection: The activity was initially implemented in 20 core districts. An additional 14 were added in the earthquake affected areas. Given available resources, data collection for the evaluation will be limited to four districts. The evaluation team used the following sampling criteria to select the districts: • Accessibility to evaluation team. • Mix of core and earthquake affected districts. • Presence of marginalized communities. • Presence of mixed ethnic populations. • Intensity of activity interventions. To make the best use of the time and resources available for this evaluation, two districts will be from the core districts and two will be from the earthquake affected districts. Given the above criteria, the evaluation team will select a total of four districts, which are presented in Exhibit 7. Exhibit 7: Illustrative Sample District Province Rationale for Selection Kailali Province 7 • Core district • Presence of marginalized communities • Intensive activities Rukum Province 5 and 6 • Core district • Presence of marginalized communities • Hill Representation Dolakha Province 3 • Recovery district • Received especially intense activity interventions • Mountain representation Dhading Province • Recovery district • Received especially intense activity interventions (mob app) • Ease of access for evaluation team In addition to thorough data collection in these four districts, the evaluation team will stop in Dang to meet the women-headed CSOs there to compare with other CSO activities. VDC/Municipality Selection: Local Group Selection: In the implementation districts, CS:MAP’s partner CSO works in one urban municipality ward and one rural municipality ward. In the sampled districts (i.e., Dolakha, Rukum, and Kailali), the evaluation team will undertake fieldwork in the concerned urban and rural municipality ward that CS:MAP’s partner CSO is currently working in. At the local level, interactions will be held with Common Assembly (CA) representatives as well as members of LDAGs. The evaluation team will identify other relevant local stakeholders, including civil society activists and champions at the district and local levels, and will conduct KIIs with them. 44 Limitations As should be expected in any social research project, the evaluation team expects to encounter biases and other limitations that must be addressed through methodological or analytical means. • First, recall bias may be present, such as key informants responding to team questions with answers related to a different assistance project. A similar problem that may arise is participants in multiple capacity building projects may blend their experiences into a composite memory or response. For example, individuals may have received training on several topics and subsequently do not distinguish them as separate activities in their responses. • Second, response bias is a common problem for project evaluations. For instance, respondents may give the interviewer positive remarks about an activity like capacity building trainings because they would like to participate in additional training in the future. The team fully expects that key points of contact, trainees, and sub-grantees may understand that this evaluation will shape future project opportunities and funding. • Third, selection bias in the form of contacts provided by the implementers can mean that the team only hears from people with positive experiences. The evaluation team may ask for a larger set of potential respondents than they will meet with to try to partially offset this issue. In other cases, there will be only one or two potential respondents in a category, reducing the selection bias. To offset this bias to a certain degree, the evaluation team also will engage with the CSOs and media at the district level, which are not beneficiaries/partners of CS:MAP. • The evaluation team also may need to rely on CS:MAP personnel to coordinate some of the field logistics. To reduce selection bias, criteria and standards will be communicated to the field coordinators. It is not possible to interview all the different groups and mechanisms with which CS:MAP works. The evaluation team has consciously selected to understand in-depth the ones who have significant linkage with the four objectives of CS:MAP. The most effective approach to combating bias is to use multiple sources of data to triangulate on an evaluation issue, as is often accomplished through qualitative reliability matrices. By combining data found in documents or KIIs from multiple sources, any one piece of biased data would not skew the analysis. Another approach that pertains specifically to KIIs is the inclusion of key informants from organizations that do not directly benefit from the assessed project, and the use of questions about specific examples of knowledge use. In addition, our team members will use direct observation based on nuanced knowledge of the evaluation environment to assist in identifying potential bias in responses. Another concern is the availability of contacts. We anticipate that data collection with direct and indirect beneficiaries may be difficult to schedule because of existing demands on their time. To mitigate this concern, our team will remain flexible to the extent possible to accommodate as many key informants and FGD participants as possible. In addition, mini-surveys will offer an efficient mode of data collection that introduces comparatively less burden on key informants. 45 IV. DATA ANALYSIS AND REPORTING The Kathmandu level meetings held by the Team Leader will be transcribed and shared before field work starts. At the end of each day of fieldwork, team members will compile their interview notes and discuss the day’s findings with each other. The two teams (see Appendix 4 for data collection procedures) will share notes with one another and compare findings as appropriate throughout the data collection process. They will exchange impressions and draw each other’s attention to noteworthy results as well as challenges or opportunities related to data gathering. The Team Leader will coordinate and oversee that process, which will inform subsequent interviews and provide a preliminary and partial entry into the data analysis process. Detailed notes (which are clear and understandable) as per the guides need to be prepared for KIIs and FGDs completed by the team member(s) and submitted to the National Team Leader/Governance Expert for compilation using simple computer software. Transcription from taped conversations and translation from local languages or Nepali to English will need to be managed by responsible team members. Caste/ethnicity and sex profile of each informant will be recorded as per the formats provided. (Refer the separate deliverable on field instruments for the formats). Upon completion of the fieldwork, the team will regroup in Kathmandu. Over a two-day period, field researchers, the team members, the USAID representative, and the National Team Leader/Governance Expert and National Team Member/GESI Expert will be given an opportunity to make progress toward finalizing their notes. During that period as well, with logistical assistance from CAMRIS, the Team Leader will run a workshop aimed at enabling brainstorming among team members and a more systematic mining of field data. Through a skype or phone conversation, the team leader will discuss preliminary field findings with the senior CAMRIS headquarter monitoring and evaluation technical advisor and other key stakeholders as appropriate. The Team Leader will return to Washington, DC, but will work closely with the national team members as they begin the process of systematically analyzing the findings against the evaluation questions and objectives. National team members will provide written analysis for inclusion in the final written report. Organization and presentation of these findings will be facilitated by a report template developed by the Team Leader and CAMRIS headquarter Senior Monitoring and Evaluation Technical Advisor. That template will be shared beforehand with USAID/Nepal. The team leader, with assistance from the CAMRIS headquarter Senior Monitoring and Evaluation Technical Advisor, will craft a draft report to be submitted to USAID/Nepal. Following receipt of USAID/Nepal’s comments, the Team Leader will incorporate USAID and partner feedback and develop a final evaluation report. That document will include an assessment of data reliability and integrity and will note any limitations in the data collection and analysis process. Data collection tools will be added as an addendum to the final report. 46 V. EVALUATION MANAGEMENT ETHICS The American Evaluation Association Guiding Principles for Evaluators guide CAMRIS’s evaluation practices. 5 All interview and FGD participants will be read an informed consent statement (in the local language) to ensure that they understand that participation in data collection is voluntary. Participants also will be informed that they do not have to answer questions that make them uncomfortable and can halt their participation at any time. Representative quotations from data collection will be used in the report to add context to findings. However, participants will not be identified by name. Protection of research participants is paramount in an evaluation and the evaluation team will minimize any risks associated with participation in data collection activities. Confidentiality procedures will be explained to all participants involved in the data collection. RESEARCH PHASES The evaluation team envisions a three-phased approach to this study: Phase 1 – Define: The evaluation team facilitated a series of initial meetings to identify the purpose of the evaluation and how findings, conclusions, and recommendations will be used. As part of the define and design process, the team lead, upon arrival in Kathmandu, will finalize the evaluation design and data collection tools with the USAID mission and FHI 360. As part of this phase, the evaluation team reviewed key activity documents, including work plans, the performance monitoring plan (PMP), and quarterly and annual reporting documents. Phase 2 – Design: This evaluation design represents the deliverable of the design process, which includes the reframing of the evaluation questions and development of the data collection methodology. The design phase includes submission of the draft data collection tools (which are presented in a separate deliverable). Phase 3 – Conduct: Implementation of the evaluation will include the following key tasks: • Review of Secondary Documents: The evaluation team will review key documents, such as CS:MAP project documents, work plans, the PMP, annual and semi-annual reports, CS:MAP assessments/survey reports and datasets, and other related technical documents and studies, GESI strategies, and GESI tools. It also will study relevant government documents, such as local development plans for sample districts, regulation policies of CSOs and media organizations, media operation directives, the Constitution of Nepal, Local Level Governance Operation Act, 2074, advocacy plan, and draft integrity plan will be studied (refer to Appendix 5 for a list of documents to review). To understand the district context, the evaluation team will consult reports and other write-ups outlining the civil society and media landscape of the concerned district along with the challenges these sectors face. Additionally, it will collect from project staff basic demographic and socio-economic data of the study districts disaggregated by social group and sex (where available) and progress against program objectives (refer to Appendix 4 for data requirements). 5 The American Evaluation Association’s Guiding Principles for Evaluators can be found online: http://www.eval.org/p/cm/ld/fid=51 47 • Preparation of Instruments and Team Orientation: The evaluation team will develop the required tools for the investigations and assessments, which will be based on secondary document review. It will ensure that the objectives of the study are achieved, and that in-depth qualitative analysis can be done. The evaluation team will develop interview guides for KIIs, FGDs, and GIs. The entire evaluation team, including field researchers, will be oriented on the use of the different tools. • Interactions/Consultations in Kathmandu: The evaluation team, as required, will conduct semi￾structured interviews, consultation meetings, and workshops with USAID, CS:MAP staff, GON, national level implementing partners, and other bilateral and multilateral donor partners (refer to Appendix 5 for the list of people to meet in Kathmandu). • Coordination and Logistics Arrangements for Field Work: CAMRIS will prepare all logistics for field research but may request assistance from CS:MAP staff as necessary. The National Team Lead/Governance Expert and NMEL staff will coordinate with the appointed CS:MAP staff to ensure that all preparatory work is completed smoothly. CAMRIS will make all arrangements (as far as possible) before the arrival of the evaluation team to the field (refer to Appendix 4 for tasks of the CS:MAP staff who will work as the field work coordinator). • Field Visit: The entire team together (the International Team Leader, National Team Leader/Governance Expert, GESI Expert, and three field researchers) will visit the Dolakha district (May 15 – 20th), after which the team will split and work concurrently. One team made up of a Governance Expert and field researcher will travel to Kailali or Rukum (May 22 – 27) and another team made up of GESI Expert and field researcher will go to Kailali or Rukum (May 22 – 27) while the Team Leader and one field researcher will continue working in Kathmandu. A list of key activities in the field appear in Appendix 4. • Information Compilation, Processing, and Analysis: All team members who conduct KIIs and FGDs will prepare detailed notes as per the areas of enquiry. Each note will be numbered and organized by a data analyst so that analysis of the information can be done more effectively and references to different pieces of evidence can be tracked with ease. The field level information analysis, including demographic analysis, will be compiled and organized and a document prepared with the analysis. • Draft Report and Finalization: The evaluation team will organize a workshop with team members to discuss findings and analysis of the evidence immediately following the field research. After the field work, toward the end of May, it will complete an informal debriefing to update USAID on topline, preliminary findings from the field research. The Team Leader will prepare a draft report using the field analysis, secondary document review findings, and inputs of national level meetings, which will be shared for feedback; the evaluation team will incorporate relevant comments into the final report (refer to Section 6 on data analysis and reporting for details). DELIVERABLES The MEL team acknowledges the following key deliverables for this evaluation: • Draft and Final Evaluation Design: The evaluation team is submitting this evaluation design in advance of the implementation of the evaluation. • In-brief with USAID/Nepal Upon the Team Leader’s Arrival in Nepal: The evaluation team will review the evaluation questions and USAID’s needs and expectations for the evaluation, review the evaluation timeline and deliverables, review data collection tools, and clarify any logistical and administrative procedures for the assignment. • Data Collection Instruments: The evaluation team will submit the data collection instruments to USAID for their review and feedback prior to training the field researchers. The 48 tools will be revised to incorporate the Mission’s feedback and will appear as an appendix to the evaluation design. • Presentation of Preliminary Findings from the Field: The Team Leader, along with the National Team Leader/Governance Expert and National Team Member/GESI Expert will present preliminary findings and reflections during an in-person meeting at the mission. • Draft Evaluation Report: The evaluation team will submit a draft evaluation report that adheres to USAID reporting guidelines and Criteria to Ensure the Quality of the Evaluation Report. • Final Evaluation Report: The evaluation team will submit the final evaluation report within 10 working days of receipt of consolidated comments in electronic format from USAID, including a matrix explaining how each comment on the draft report was addressed. The evaluation team will submit in MS Word and PDF versions of the final approved evaluation report, and will post it to the Development Experience Clearinghouse (DEC). Knowledge Products: The evaluation team will produce one- to two-page info graphics, a flyer, and so on, highlighting the key findings and lessons from the evaluation that are useful for organizations working on civil society issues. Team Composition The evaluation team will comprise of four professionals, including an International Team Leader and a National Team Leader. Team Leader: The Team Leader has demonstrated leadership and team management skills. She has broad technical experience with the function and operation of civil society strengthening, government accountability, and transparency programming. She has specific knowledge of democracy, human rights, and governance (DRG) issues in transitioning political environments. She also has experience as a team leader in several USAID performance evaluations. National Team Leader/Governance Expert: The Governance Expert has profound knowledge and experience in political/social science. He has broad technical experience with the function and operation of local governance, development sustainability, including specific and extensive knowledge on Nepal, particularly on DRG transition. National Team Member/GESI Expert): For this evaluation, the GESI Expert will also serve as a national team member. She has decades of experience in the design, monitoring, and evaluation of democracy and governance related projects, with a focus on GESI. She has broad technical experience with the function and operation of GESI, GESI assessment tools, and GESI approaches, including specific and extensive knowledge of Nepal. She has good knowledge of local governance and development sustainability. The experts’ team will be accompanied by three field researchers who are adept in local governance and GESI issues as well as qualitative evaluation methodology. The researchers will be selected considering their gender, social background, ability to speak the language prevalent in the proposed sample districts, and English language and writing skills. One translator/interpreter will be with the Team Leader throughout the field research to help her in translation and interpretation as required. 49 WORKPLAN AND TIMELINE ESTIMATES The work plan and timeline appear in Appendix 5. MEL is working on the LOE estimates for each consultant and field researcher as well as the budget. MEL will submit LOE and budget as a separate document. 50 APPENDIX III: GETTING TO ANSWERS MATRIX Evaluation Questions Illustrative Measures Sources Methods of Verification Context and Design 5a. To what extent is the design of CS:MAP’s hypothesis, theory of change and operating modalities structures, and approaches relevant to the new structure of the state? 5b. What are the modifications and/or re￾scoping required in the activity and modality, if any, given the changes in the governmentalstructures? The project: a) Provides interventions to project beneficiaries that they find useful given the changes to GON structures b) Effectively course corrects its activities in response to changes within the GON structures • GON officials • CS:MAP program staff • CS:MAP partners – international, national, and local KIIs, FGDs, GIs Design 1. Which components and approaches of CS:MAP are most and least effective (given the government structural changes) to date in fulfilling activity objectives, and why? The project: a. Improved legal and regulatory framework for civil society and media to operate b. Improved self-regulation of civil society and media c. Improved public perception of civil society and media d. Strengthened relationships among civil society, among civil society and media, and among civil society, media, and government e. Improved capacity of civil society and media for public service oversight f. Improved participation of marginalized populations • GON policies and plans • C:SMAP project description • CS:MAP program progress reports • Sub-awardee progress reports Government documents, CSMAP documents, KIIs, GI/FGDs, and mini-survey Implementation 2. To what extent and how have CSOs and the media been effective in supporting The project: a. Improves public perception of civil society and media • GON policies and plans, • CS:MAP project description, Government documents, 51 identification and resolution of issues of greater public interest? 3. To what extent and how has the CS:MAP activities contributed to fostering trust among government, political, and civil society stakeholders? 4. To what extent and how is participation and representation of marginalized groups improving at the community level? b. Develops and delivers civil society and media led public accountability initiatives c. Develops effective coalitions involving civil society and media d. Civil society and media can effectively provide oversight to public service delivery, and where possible, make those services and government responsive to citizens needs and feedback e. Marginalized populations are fully incorporated into the design, implementation and analysis of civil society and media projects f. Service delivery projects include the needs of marginalized populations and include marginalized populations in their projects • CS:MAP program progress reports • Sub-awardee progress reports • Sample of activity deliverables (media spots, advocacy initiatives, and so on) CS:MAP documents, KIIs, and GI/FGD s Sustainability 6. How is the activity progressing on long￾term self-reliance through its sustainability and exit strategies? The project: a. Strengthens civil society and media’s ability to function effectively b. Empowers citizens, including the marginalized to press for greater government accountability on service delivery c. Makes local government more responsive, transparent and accountable to citizens based on inputs by civil society and media partners. d. Increases trust between citizens, media and government e. Increases the overall capacity for civil society and media to effectively engage • Government of Nepal (GON) policies and plans, • CS:MAP project description, • CS:MAP program progress reports • Sub-awardee progress reports Government Documents CS:MAP documents KIIs, GI/FGDs Mini-survey 52 APPENDIX IV: DETAILED METHODOLOGY Sample Selection Methodology: Stakeholders to be included for the study were selected based on mapping exercise of the current stakeholders. Once the stakeholders at both the national and district levels were mapped, the sample was selected to include diverse set of actors from the national level civil society activists to community groups. The list of stakeholders consulted during the evaluation are listed in the table below: List of Group Interviews SN Date Organizations/ Groups Venue 1 06 May, 2018 Chief Commissioner and Commissioner of National Information Commission (NIC) NIC Office, Anamnagar, Kathmandu 2 09 May, 2018 Country Director/ COP, Civil Society Director, GESI Advisor, M&E Manager of FHI 360/ CS:MAP, FHI 360 office, Balwatar 3 09 May, 2018 Media Director for CS:MAP, Country Representative of Equal Access (EA), FHI 360 office, Balwatar 4 10 May, 2018 Treasurer and Program Manager for CS:MAP Federation of Nepali Journalists (FNJ), FNJ Office, Sinamangal 5 10 May, 2018 NGO Federation’s Chairperson, Treasurer, ICT Officer and Finance Officer and Focal Person for CS:MAP NGO Federation office, Baneshwor 6 11 May, 2018 Chair-person, GoGo Foundation, Executive Director CAMRIS NMEL Office, Balwatar 7 11 May, 2018 Executive Director and Project Officer of SAP Nepal CAMRIS NMEL Office, Balwatar 8 11 May, 2018 Executive Director and Project Officer of Samjhauta Nepal, Media Advocacy Group (MAG) CAMRIS NMEL Office, Balwatar 9 11 May, 2018 Executive Director and Project Manager of Freedom Forum CAMRIS NMEL Office, Balwatar 53 List of Group Interviews SN Date Organizations/ Groups Venue 10 13 May, 2018 Program Officer and Capacity Building Specialist of MITRA Samaj CAMRIS NMEL Office, Balwatar 11 13 May, 2018 Director, Capacity Developing Specialist and Program Officer of Nepal Participatory Action Network (NEPAN), CAMRIS NMEL Office, Balwatar 12 13 May, 2018 Project Manager and Project Officer of Shtrii Shakti CAMRIS NMEL Office, Baluwatar 13 14 May, 2018 Department Head and Professors of the Central Department of Journalism and Mass Communication, Tribhuvan University Ratna Rajya Campus, Bhrikutimandap 14 15 May, 2018 District Coordinator and Project Manager of Human Right Awareness and Development Centre (HURADEC), Horse Shoe Resort, Mude, Sindhupalchowk 15 16 May, 2018 Chairperson, Vice Chairperson, and Chief Administrative Officer of Sailung Gaupalika (Rural Municipality) Sailung Gaupalika (Rural Municipality), Dolakha 16 17 May, 2018 NGO Federation, Dolakha Chapter Charikot, Dolakha 17 20 May, 2018 DCOP of CS:MAP CAMRIS NMEL Office, Balwatar 18 21 May, 2018 Chairperson, Executive Director, Program Coordinator, Advocacy Officer of Rural Women Development Center (RWDC), Dang RWDC Office, Ghorahi, Dang 19 21 May, 2018 Agreement Officer Representative (AOR), Program Development Specialist, Monitoring and Evaluation Officer, and Democracy and Governance Officer of USAID CAMRIS NMEL Office, Balwatar 54 List of Group Interviews SN Date Organizations/ Groups Venue 20 21 May, 2018 Grant Manager, Finance Officer, and two capacity building officers of CS:MAP Regional Team and Equal Access in Nepalgunj Nepalgunj 21 21 May, 2018 Chairperson, Vice-Chairperson, and Project Manager of Human Rights Protection and Legal Service Centre (HRPLSC) Asian Hotel, Salle, Rukum 22 21 May, 2018 FHI 360/ CS:MAP, Country Director/COP, DCOP, Civil Society Director, GESI Advisor, and M&E Manager FHI 360 Office, Balwatar 23 22 May, 2018 Head, Program Officer, and Finance Officer, Far West Media Development Center (FMDC), Media for Development and Social Change (MDSC) Kailali 24 22 May, 2018 NEPAN Director, Capacity Developing Specialist, and Program Officer NEPAN Office 25 23 May, 2018 Ward Chairpersons (wards 1 to 5) of Tribeni Gaupalika Tribeni Gaupalika, Rukum 26 23 May, 2018 Health, Education, Agriculture, and Disaster (HEAD) Focal Person, Tribeni Gaupalika. Tribeni Gaupalika, Rukum 27 24 May, 2018 Nepal National Social Welfare Association (NNSWA) Kanchanpur 28 24 May, 2018 HEAD (District) Rukum 29 25 May, 2018 COP, DCOP, and Civil Society Director of CS:MAP/ FHI 360 FHI 360 Office, Balwatar 30 7 June, 2018 District Coordinator, Project Director of Development Exchange Center (DEC) Nepal DEC Office, Dhading Besi 31 7 June, 2018 Deputy Head and Coordinator of DCC, Dhading DCC Office, Dhading 55 List of Group Interviews SN Date Organizations/ Groups Venue 32 8 June, 2018 FNJ/Radio Dhading: Station Manager/FNJ Secretary and Program Coordinator/Journalist Radio Dhading Office, Dhading 33 8 June, 2018 Mayor, Neelkantha Municipality, Dhading Neelkantha Municipality Office, Dhading Total number of participants: 107 Male: 86 Female: 21 List of FGDs SN Date Name of the Groups Venue 1 16 May, 2018 Common Assembly/ Innovation Centre Sailung Gaupalika, Dolakha 2 16 May, 2018 Listening Discussion and Action Group (LDAG), Ward 3 Sailung Gaupalika, Dolakha 3 17 May, 2018 Radio Sailung and Content Advisory Committee Charikot, Dolakha 4 22 May, 2018 Common Assembly (CA) Kailali 5 22 May, 2018 LDAG Kailali 6 22 May, 2018 Radio Group (Radio Sani Bheri) Rukum 7 23 May, 2018 PPWG Dhangadi, Kailali 8 23 May, 2018 Private FM Dhangadi, Kailali 9 23 May, 2018 Common Assembly of Tribeni Nagarpalika Tribeni Nagarpalika (Urban Municipality), Rukum 10 23 May, 2018 LDAG, Radio Bisaune, Rukum Musikot Nagarpalika, Rukum 11 7 June, 2018 CA and LDAG, Siddhalek Rural Municipality and Neelkantha Municipality, Dhading Dhading Besi, Dhading Total number of participants: 104 Male: 60 Female: 44 56 List of KIIs SN Date Designation/ Organization Venue 1 06 May, 2018 Acting Director of Social Welfare Council (SWC) Lainchaur, Kathmandu 2 06 May, 2018 Joint Secretary of Ministry of Communication and Information Technology, Singha Durbar, Kathmandu 3 09 May, 2018 Legal Advisor of International Center for Not-for-Profit Law (ICNL) CAMRIS NMEL Office, Balwatar 4 11 May, 2019 Informal Sector Service Center (INSEC) CAMRIS MEL Office, Balwatar 5 13 May, 2018 Kathmandu Living Labs (KLL) CAMRIS MEL Office, Balwatar 6 16 May, 2018 Ward Chairprson of Ward 3, Sailung Gaupalika Sailung Gaupalika, Dolakha 7 17 May, 2018 Deputy Mayor of Bhimeshwor Municipality, Dolakha Bhimeshwor Municipality, Dolakha 8 17 May, 2018 District President of FNJ, Dolakha chapter Charikot, Dolakha 9 17 May, 2018 District Administration Office, Chief District Officer (CDO) of Dolakha Charikot, Dolakha 10 17 May, 2018 Chairperson and Member of District Coordination Committee (DCC), Dolakha DCC Office, Charikot, Dolakha 11 17 May, 2018 HEAD Coordinator on DRR, Dolakha Charikot, Dolakha 12 20 May, 2018 DCOP- USAID CAMRIS MEL Office, Balwatar 13 21 May, 2018 Deputy Mayor of Nepalgunj sub￾metropolitan City. Nepalgunj, Banke 14 22 May, 2018 Civil Society Leader, Director of Alliance Social Diaogue (ASD) ASD Office, Baneswore 15 22 May, 2018 Vice Chairperson of Tribeni Gaupalika Asian Hotel, Salle, Rukum 16 22 May, 2018 Information Officer, Tribeni Gaupalika Asian Hotel, Salle, Rukum 17 22 May, 2018 Mayor of Gauriganga Municipality Kailali 18 22 May, 2018 Vice President of Gauriganga Municipality Kailali 19 23 May, 2018 Community Action Researcher Rukum 20 24 May, 2018 CDO of Kailali Kailali 21 24 May, 2018 Mayor of Dhangadi sub￾Metropolitan, Kailali Kailali 22 24 May, 2018 CDO of Rukum Rukum 23 8 June, 2018 ICT Officer, Dhading Neelkantha Municipality, Dhading Total number of participants: 23 57 Male: 19 Female: 4 Documents Reviewed Documents Reviewed Title Author(s)/Organization Date Request for Appraisal and Program Document Notice of Funding Opportunity Number: RFA￾367-15-000003 Shirley Baldwin, USAID/Nepal September 3, 2015 AID-367-A-16-00007 Civil Society: Mutual Accountability Project (CS:MAP), Attachment B - Program Description Project Description Final Annual Work Plan Annual Work Plan (Year III) October 1, 2017 to September 30, 2018 CS:MAP/ FHI 360 August 31, 2017 Annual Work Plan (Year II) October 1, 2016 to September 30, 2017 CS:MAP/ FHI 360 September 30, 2016 Annual Work Plan (Year 1) April 4, 2016 to September 30, 2016 CS:MAP/ FHI 360 June 03, 2016 Monitoring, Evaluation and Assessment Post-DQA Action Plan CS:MAP/ FHI 360 May 2018 Data Quality Assessment Report Bishwas Rana, Krishna Pathak, Reshma Thapa, Ganga Maharjan, USAID/ Nepal April 2018 CS:MAP Data Quality Assessment 2018 Debriefing notes Feb 8, 2018 Assessment Summary Sheet of MEL WHEEL assessment January 12, 2018 Monitoring and Evaluation (M&E) Plan, Revised CS:MAP/ FHI 360 August 31, 2017 Activity M&E Plan Review Sheet July 14, 2016 Monitoring and Evaluation (M&E) Plan CS:MAP/ FHI 360 July 1, 2016 Gender Equality and Social Inclusion Audit, GESI Auditing: ISO Partner Shtrii Shakti Progress Report Quarterly Progress Report, October 1- December 31, 2017 CS:MAP/ FHI 360 January 31, 2018 Annual Progress Report October 1, 2016-September 30, 2017 CS:MAP/ FHI 360 October 30, 2017 Quarterly Progress Report April 1-June 30, 2017 CS:MAP/ FHI 360 July 27, 2017 Quarterly Progress Report January 1 -March 31, 2017 CS:MAP/ FHI 360 April 28, 2017 Quarterly Progress Report October 1 - December 31, 2016 CS:MAP/ FHI 360 January 30, 2017 Annual Progress Report April 4 – September 30, 2016 CS:MAP/ FHI 360 October 28, 2016 Quarterly Progress Report April 4 – June 30, 2016 CS:MAP/ FHI 360 July 29, 2016 Guidelines, Manual and Tool Strategic Guide CS:MAP/ FHI 360 December 2017 58 Documents Reviewed Title Author(s)/Organization Date Implementation Guide Good Governance Barometer (GGB) CS:MAP/ FHI 360 April 2017 Grants Manual FHI 360 Dec 2015 Advocacy Readiness Index (ARI) Facilitator Guide, Advocacy Capacity Building Assessment Tool for Civil Society Organizations (CSOs) CS:MAP/ FHI 360 Organizational Capacity Assessment Facilitator Guide CS:MAP/ FHI 360 GESI Approach and Action Plan Memo Recommendation for Final Selection of the CS:MAP Implementing Agencies for Policy Advocacy, Government Engagement and Oversight Grants Program Prem Raj Ojha et al., CS:MAP/FHI 360 February 06, 2017 Recommendation for Pre-award Assessment of Organizations to work Under Policy Advocacy, government Engagement, and Oversight Grants Focused on Achieved Results under Objectives Two, Three and Four of CS:MAP Prem Raj Ojha et al., CS:MAP/FHI 360 January 17, 2017 Recommendation for Final Selection of the CS:MAP Implementing Agencies for Intermediary Service Organization (ISO) Grants Program Sajana Maharjan et al., CS:MAP/FHI 360 December 21, 2016 Recommendation for Pre-award Assessment of Organizations to Work under Intermediary Service Organization (IS) Grants Program on Objective[s] Two, Three and Four of CS:MAP Sajana Maharjan Amatya et al., CS:MAP/ FHI 360 November 16, 2016 Recommendation for Final Selection of the CS:MAP Implementing Agencies for Strengthening Enabling Environment Grants Program under Funding Stream One and Funding Stream Two Sajana Maharjan et al., CS:MAP/FHI 360 November 14, 2016 Recommendation for Pre-award Assessment of Organizations to Work for Strengthening Enabling Environment Grants Programs under Objective One of CS:MAP Sajana Maharjan Amatya et al., CS:MAP/FHI 360 October 21, 2016 Evaluation Process The evaluation team started with an in brief with the Mission followed by the in brief with the IP. The in briefs laid the groundwork for the stakeholder mapping and selection for further interviews. The evaluation team conducted national level meetings, including with the ISOs and consortium partners, in Kathmandu during the first week of commencement of the evaluation. The field researchers were trained on the tools for the key informant interviews, focus group discussions, and group interviews conducted at the district level. The team launched the field work in a district starting from Dolakha, whereby all the team members went to the field together. However, in the field, the team divided 59 themselves into two groups, so that they could talk to all the potential stakeholders in the district. Every day after the field work, the team got together for the debrief session for the day. The summary of top level findings was noted and the plan for the next day was reviewed to ensure that there was no duplication and none of the stakeholders were left behind. The team divided into two groups for data collection in the remaining districts. Each of the national experts led the team in two districts in parallel. Informed Consent and Confidentiality The American Evaluation Association Guiding Principles for Evaluators guide CAMRIS’s evaluation practices. 6 All interview and FGD participants were conducted following an informed consent statement (in the local language) to ensure that respondents understand that participation in data collection is voluntary. Participants were informed that they do not have to answer questions that make them uncomfortable and can halt their participation at any time and that the representative quotes from data collection will be used in the report to add context to findings. However, participants will not be identified by name. Protection of research participants is paramount in an evaluation and the evaluation team has taken every step to minimize any risks associated with participation in data collection activities. Confidentiality procedures were explained to all participants involved in the data collection. ANALYSIS Analysis of data started by analyzing the data generated through secondary sources. There were several longer sessions to review in-depth data collected from implementing partners before conducting district￾based data collection to ensure that core questions were still valid, and all team members were aware of nuances that may come up with each respondent group. The evaluation team developed the analysis and reporting plan. High level codes were generated based on the analysis plan. The evaluation team conducted group coding and analysis during a three-day workshop after the data collection was over in three districts. This provided the team with preliminary analysis for the out-brief session with USAID. The evaluation team also used this opportunity to develop a parallel presentation for the IPs so that they were aware of the evaluation team’s preliminary findings. The analysis phase included a re-review of KII, GI and FGD notes taken by team members, the mini￾survey results and a review of key program documents. All analysis is based on information hard by more than one respondent, and blanket generalizations were not included in the narrative report. 6 The American Evaluation Association’s Guiding Principles for Evaluators can be found online: http://www.eval.org/p/cm/ld/fid=51 60 APPENDIX V: DATA COLLECTION INSTRUMENTS Interview Details Key Informant Details Date of Interview: Name, Title: Time Start: Time End: Affiliation: Team: Sex: Male / Female (circle one) Interviewer(s): Location: Introduction 1. What is your role in the CS:MAP project (either as an individual or as an organization)? 2. What are the activities that your organization is doing in CS:MAP? GON Changes and Effectiveness (EQ5+EQ1) 3. There have been widespread changes to the governance structures in Nepal in the last year. How have these changes affected the activities you are conducting through CS:MAP? 4. Of the support that you have received through the CS:MAP project, what has been the most useful given these changes? a. What support would you like to receive more of from the CS:MAP project? 5. Of the support you have received through the CS:MAP project, what has been less helpful given these changes? Media and CSO effectiveness (EQ2) 6. Can you name any opportunities or barriers, given these new government structures, to civil society and media being more effective in their work? 7. How has CS:MAP helped the media and civil society to identify and raise issues of public interest? Please describe. Improved Relationships (EQ3) 8. How have the new government structures helped or hindered civil society’s and the media’s ability to build trust with the government and political stakeholders? 9. How have relations between CSO, the media, and the government changed? (Have relations become better, worse, or stayed the same?) a. Can you attribute any of this improvement to CS:MAP? b. What CS:MAP activities were positive contributors to these strengthened relationships? Marginalized Groups (EQ4) 10.Has your organization been able to integrate GESI and marginalized populations into the program in a way that produced meaningful participation and representation? Alternatively, could you have done something differently to increase impact? 61 Sustainability (EQ6) 11.How do you incorporate sustainability in the activities you conduct under the CS:MAP project? 12.How could the activities of civil society and media be more sustainable after CS:MAP support ends? 62 USAID/Nepal CS:MAP Evaluation Coversheet for Focus Group Discussion Protocols Interview Details Key Informant Details Date of Interview: Name, Title: Time Start: Time End: Affiliation: Team: Sex: Male / Female (circle one) Interviewer(s): Location: Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are working with USAID/Nepal to conduct a mid-term evaluation of the Civil Society: Mutual Accountability Program (CS:MAP) implemented by FHI 360. The evaluation is intended to provide an informed assessment of progress to date and recommend potential modifications, if needed. Our team has reviewed some background documents to get a better sense of the design and implementation of the project. However, documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, to help us better understand how CS:MAP operates “on the ground.” Confidentiality Protocol • We will take note of the people we speak with, but for security precautions, we will not provide a list of key informants in our final evaluation report. • We may include quotations from respondents in the evaluation report, but we will not link individual names, organizations, or personally identifiable information to those quotations, unless express written consent is granted by the respondent. Should the team desire to use a quotation, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purposes of this evaluation and will not be shared with other audiences or used for any other purpose. • Your participation in this interview is voluntary and if you do not feel comfortable answering a question, please let us know and we will simply go on to the next question. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? Inform interviewee we may follow-up with additional questions at the end of fieldwork. 63 Focus Group Discussion Guide DIRECT CS:MAP BENEFICIARIES (DIRECT GRANTEES, PARTNER ORGANIZATIONS, ETC.) TIME: UP TO 90 MINUTES Be sure to: • Set up the room to facilitate a participatory discussion. • Introduce yourselves. • Introduce SI and explain why you are conducting the focus group session. • Ask the participants to do the same. • Give a verbal agenda and length of the meeting. • Set goals. • Clarify your role as a facilitator. Evaluation’s Objective To better understand what has been accomplished, document any challenges, and identify opportunities for program improvement. Purpose of Discussion: To hear from CS:MAP beneficiaries about their experience with the CS:MAP activity. Focus Group facilitator’s guidelines for effective discussions: • Everyone is clear on the topic. • Everyone participates; no one dominates the discussion - No speeches! • One person talks at a time. • Comments and discussion stay on the topic. • Comments should be to the whole group - no side discussions. • Respect time limits. • Write down unanswered questions. • No divisive or confrontational language or tone. • Take notes of discussion, comments and observations so that you can write report. 64 1. Introduction (5 min.) • Reaffirm points of the meeting. • Welcome participants. • Set a 90-minute timeline. • Introduce the evaluation, give verbal agenda, objectives (as stated above). 2. Setting rules (5 min.) Before the participants begin dealing with issues and ideas, the participants should agree on a set of rules that define how a group will function and how the participants will interact. Sample rules • Each group member has the right to participate. • The opinion of each group member is important and should be respected. • Group members should be tolerant of different ideas. • Each group member is important. 3. Tell us about your involvement with the CS:MAP project (15 min.) (EQ1) Ask participants: • What activities did you participate in through (local organization) and CS:MAP? • How long was your involvement with this activity? • Did you receive any training or direct support from CS:MAP? o If yes, what types of training or support did you receive and what did you learn? o Did you find the support to be helpful? Facilitator note: explore with the group why trainings were or were not helpful. Try to get concrete examples of how they utilized the information or skills they learned. 4. Tell us about the changes you saw because of participating in the CS:MAP project (15min.) (EQ1) Ask participants: • What were the results of the activities that you took part in? • What changes did you see in your community because of these activities? • Have these activities carried on even after the CS:MAP project ended? o If not, what type of additional support is needed to make these activities continue? Facilitator note: Try to explore beyond requests for additional financial support. 65 5. Effectiveness of CSOs and Media (10 min.) (EQ2) Ask participants: • Have you seen or experienced any instances where civil society or the media help identify and resolve local issues of concern? • Can you provide any examples? 6. Participation of Marginalized Populations (20 min.) (EQ4) Ask participants (adjust wording as needed if participants are male or female, youth or non￾youth): • What efforts does CS:MAP take to promote women’s involvement your organization’s activities? • Did you feel this was a good approach? If no, how might they have done this better? • What efforts does CS:MAP take to promote youth involvement your organization’s activities? • Did you feel this was a good approach? If no, how might they have done this better? • What could CS:MAP do differently to encourage more marginalized groups to become involved your organization’s work? • Did you feel this was a good approach? If no, how might they have done this better? 7. Contextual Factors (EQ3+5) (10 min.) Ask participants: • How has the change in government structures affected your ability to get issues of concern resolved? • Have you seen a change in how the government, civil society, and the media work together because of CS:MAP or changes in the government structure? Please explain. 8. Conclusion / Close (5 min.) • Summarize the session. • Ask if everyone agrees to what was discussed, offer any final comments, and thank all those who participated. 66 Focus Group Discussion Guide NON-CS:MAP CSOS, CIVIL SOCIETY CHAMPIONS, ETC TIME: UP TO 90 MINUTES Be sure to: • Set up the room to facilitate a participatory discussion. • Introduce yourselves. • Introduce SI and explain why you are conducting the focus group session. • Ask the participants to do the same. • Give a verbal agenda and length of the meeting. • Set goals. • Clarify your role as a facilitator. Evaluation’s Objective To better understand what has been accomplished, document any challenges and, identify opportunities for program improvement. Purpose of Discussion: To hear from CS:MAP beneficiaries about their experience in the CS:MAP Program. Focus Group facilitator’s guidelines for effective discussions: • Everyone is clear on the topic. • Everyone participates; no one dominates the discussion - No speeches! • One person talks at a time. • Comments and discussion stay on the topic. • Comments should be to the whole group - no side discussions. • Respect time limits. • Write down unanswered questions. • No divisive or confrontational language or tone. • Take notes of discussion, comments and observations so that you can write report. 1. Introduction (5 min.) • Reaffirm points of the meeting. • Welcome participants. • Set a 90-minute timeline. • Introduce the evaluation and provide a verbal agenda, give the objectives (asstated above). 67 2. Setting rules (5 min.) Before the participants begin dealing with issues and ideas, the participants should agree on a set of rules that define how a group will function and how the participants will interact. Sample rules • Each group member has the right to participate. • The opinion of each group member is important and should be respected. • Group members should be tolerant of different ideas. • Each group member is important. 3. Have you heard about the CS:MAP project (10 min.) (EQ1) Ask participants: • What have you heard? What’s your opinion? 4. Results of CS:MAP (30 min.) (EQ1) Ask participants: • Have you seen any changes in the work of civil society in the last several years? Please describe. • Have you seen changes in the media in the last several years? Please describe. • Have you seen changes in how civil society or the media interacts with the government? • Has public perception about the role of civil society or the media changed in the last few years? • What are some of the barriers or obstacles? • What are some entry points that could be further capitalized upon? Facilitator note: Try to explore beyond requests as appropriate. 5. Participation of Marginalized Populations (20 min.) (EQ4) Ask participants (adjust wording as needed if participants are male or female, youth or non￾youth): • Has there been a change in the levels of participation of women, youth, or other minorities in the community in addressing issues of concern? Please describe why or how an increase took place. 7. Contextual Factors (EQ3+5) (15 min.) Ask participants: • How has the change in the government’s structures affected your ability to get issues of concern resolved? • Have you seen a change in how the government, civil society and the media work together because of CS:MAP or changes in government structure? Please explain. 68 8. Conclusion / Close (5 min.) • Summarize the session • Ask if everyone agrees to what was discussed, offer any final comments, and thank all those who participated. 69 USAID/Nepal CS:MAP Evaluation Coversheet for Mini-Survey Protocol Introduction: Hello. I am calling from CAMRIS International, an independent research company based in Bethesda, Maryland, USA. We are working on behalf of USAID/Nepal to collect data about USAID’s Civil Society: Mutual Accountability Program (CS:MAP) implemented by FHI 360. Our evaluation is intended to provide an informed assessment of progress to date and recommend potential modifications, if needed. CS:MAP provided your name as a person who is informed on this topic. I have a few questions for you. Could I speak with you for no more than ten (10) minutes? Confidentiality Protocol • We will take note of the people we speak with, but for security precautions, we will not provide a list of survey respondents in our final evaluation report. • All data gathered will be used for the sole purposes of this evaluation and will not be shared with other audiences or used for any other purpose. • Your participation in this survey is voluntary and if you do not feel comfortable answering a question, please let us know and we will simply go on to the next question. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? 70 USAID/Nepal Mini-Survey Protocol (Est. Time ~10 min.) CSO Partners Date: Organization Name Gender Male 1 Female 2 Districts Covered 1. How many grants did you receive from CS:MAP? 2. How long (in months) was your grant(s)? 3. What was the main activity you conducted under the award you received from CS:MAP? (Interviewer read out options – Have the respondent pick ONE answer that corresponds to the main duties they conducted under CS:MAP. If they want to choose more than one, make a notation and let’s discuss afterwards). Public Hearing 1 Public Expenditure Tracking (PET) 2 Social Audit 3 Citizen Charter 4 Advocacy 5 Participatory Evidence-based Action Research (PEAR) 6 Media/information sharing 7 Service delivery 8 GESI 9 Other [Specify] 95 Don’t know 99 4. What is the most important type of support provided to your organization by CS:MAP? (EQ1+EQ6) [Interviewer Note: Please read out options. Ask respondent to only choose ONE answer] Training 1 Funding 2 Advocacy to promote civil society participation in public interest topics 3 Support to networking and coalitions 4 Other [Specify] 95 Don’t know [Do not Read] 99 5. Did you receive any training (or technical support) under CS:MAP? No Yes Don’t know 0 1 99 6. [If yes to Q5], how do you describe the quality of trainings? [Interviewer reads out options? (EQ1+EQ6)] Very poor 1 Poor 2 Satisfactory 3 Good 4 Very good 5 71 Not Applicable 77 7. [If yes to Q5], Thinking about the work you have conducted since the government changes, what was the most useful training (or technical support) you received? (EQ1)[Interviewer: Read out options] OCA 1 Advocacy 2 Participatory Evidence-based Action Research (PEAR) 3 Social Accountability 4 GESI 5 Other (specify) 99 Not Applicable 77 8. Thinking about the changes in the government’s structures and the support you have received so far from CS: MAP, what additional support would you find useful? Training (please describe) 1 Funding 2 Information on new government structures and lines of authority/responsibilities/decision-making 3 Advocacy to promote civil society participation in public interest topics 4 Support to networking and coalitions 5 Lawmaking Process 6 Other (Specify) 95 Don’t know (Do not Read) 99 9. How do you rate your organization’s inclusion of GESI in the activities you have conducted through the CS:MAP project? (EQ4) [Interviewer read out options] Very poor 1 Poor 2 Satisfactory 3 Good 4 Very good 5 10. How would you rate the overall relevance of CS:MAP services or activities in helping your organization achieve its goals? (EQ1+EQ6) [Interviewer reads out options] Irrelevant 0 Somewhat relevant 1 Very relevant 2 Extremely relevant 3 11. Thinking about the work you did because of the support your organization received from CS:MAP, did a lot of that work need to change because of the new governmental structures (EQ5a). No Yes Don’t know 0 1 99 12. What is the top/main challenge you have faced in implementing your CS:MAP activities? 13. Is there anything else that you would like for us to know about your CS:MAP project? End of the Interview (Thank you very much for your time) 72 APPENDIX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name KourtneyPompi Title Consultant Organization CAMRISInternational Evaluation Position? √ Team Leader Team member Evaluation Award Number (contract or another instrument) AID-367-C-15-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) CS Map Mid-term Evaluation I have real or potential conflicts of interest to disclose. Yes √ No 73 If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s)whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whoseprojects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s)whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with 74 75 76 77 APPENDIX VII: NEW STATE STRUCTURE OF NEPAL As the evaluation team moved through the data collection phase, the team members found it critical to have a common understanding of the new state structures in Nepal. The evaluation team developed this analysis and used it as a reference throughout the analysis phase to better understand the changes taking place within Nepal’s new federalist structures. The evaluation team includes this as an appendix, as a reference, and as context to the team’s understanding of the changes that CS:MAP program is now operating within. 1. Introduction of State Structure in Nepal The state structure in Nepal are divided into three levels: the federation, state (province), and local levels. The federation level is the highest level in the hierarchy followed by the state level and then the local level. The unique powers of the federation, state, and district are enumerated in Schedules 5 (List of Federal Power) and 6 (List of Provincial Power) of the Constitution of Nepal 2015 respectively. The concurrent powers are enumerated in Schedule 7 (Federal and State Power). The powers of the Local Level are enumerated in Schedule 8 and the concurrent powers between all three levels in Schedule 9. Any laws made in furtherance of the concurrent powers at any level should not be inconsistent with that of the higher level and if it so does, such law shall be deemed to be invalid to the extent of such inconsistency, as provided by Article 56 (6)(7) of the Constitution. Local level government is authorized to make laws and policy on that enumerated in Schedule 8. While it may make such laws, the laws should not be inconsistent with the Constitution of Nepal. All three levels may impose taxes on matters falling within their respective jurisdictions and collect revenue from these sources as provided under Article 60. 2. Structure at Local Level The local level means the rural municipality or municipality. This term shall also include district assembly. (Section 2 (S) of Local Level Election Act, 2073 (2017) There are now a total 753 local bodies in Nepal 78 that include 460 rural municipalities, 276 urban municipalities, 11 sub-metropolitans, and 6 metropolitans, all comprised within 77 districts. Each rural municipality is comprised of a minimum of five wards and a maximum of 21 wards. Regarding urban municipalities (that include sub-metropolitans and metropolitans), there will be a minimum of nine and a maximum of 35 wards. The key structures of the local bodies are as follows according to the Constitution of Nepal, 2015: District Coordination Other Members District Assembly Rural Municipality Urban Municipality Village Assembly, i.e., Legislative Body Village Executive Judicial Committe e Committees/units/ departments Committees/units/ departments Ward Committees • Village Assembly (rural municipality assembly) (Art. 222) comprises of: A chair, vice-chair, and ward chairs. Each ward will have four members, including at least two women (among them one should be a Dalit woman) and two Dalit or minority community members elected by the Village Assembly from the Dalit and minority communities. • Village Executive (rural municipality executive) (Art. 215) comprises of: A chair, vice-chair, ward chairs elected from each ward, four women members elected by the members of the Village Assembly from amongst themselves, and two members elected by the Village Assembly from the Dalit or other minority communities. 79 • Urban Municipality/Sub-Metropolitan and Metropolitan Assembly (Art. 223) comprises of: A mayor, deputy mayor, ward chairs, and four ward members, including two women and three Dalit or minority community members elected from the Municipal Assembly. • Municipal Executive (Art. 216) comprises of: A mayor, deputy mayor, ward chairs, five women members elected by the Municipal Assembly from amongst themselves and three from Dalit or minority communities elected by the Municipal Assembly. • District Assembly (Art. 220) comprises of: Chairs and vice-chairs of Village Executive and mayors and deputy mayors of Municipal Executives within the district. • District Coordination Committee [Art. 220(3)] comprises of a maximum of nine members, including a chief, deputy chief, at least three women, at least one Dalit or minority, and other members. A member of Village Assembly or Municipal Assembly within the concerned district shall be eligible to be a candidate for the office of chief, deputy chief or members of the District Coordination Committee. 3. Functions, Duties and Rights of Rural Municipality and Municipality Schedule 8 of the Constitution provides for the exclusive rights of the rural municipality and municipality. List of Powers/Jurisdiction for Local Level Serial Number Subjects 1. Municipal police 2. Cooperatives 3 FM operation 4 Local tax (property tax, house rent tax, fee on registration of houses and land, vehicle tax), service fee, tourism fee, advertisement tax, business tax, land tax (land revenue), fines, entertainment tax 5. Management of local services 6. Local statistics and record keeping 7. Local development projects and programs 8. Basic and secondary education 9. Basic health and sanitation 10. Management of local markets, environment conservation and biological diversity 11. Local roads, rural roads, agriculture roads, irrigation 12. Village assembly, Municipal assembly, district assembly, local courts, dispute settlement and mediation 13. Management of local records 14. Distribution of land, building ownership certificates 15. Farming and livestock, agriculture production management, livestock health, cooperative 16. Management of senior citizens, people with physical disability and disabled 17. Collection of statistics of unemployed people 18. Management, operation and control of agriculture extension 19. Drinking water, small electricity projects, alternative energy 20 Disaster management 21. Conservation of Watershed, wetland, wildlife, mines and minerals 22. Preservation and development of language, culture and fine arts 80 The rural municipality or municipality is required to develop and enforce laws, policies, plans, standards, and procedures while exercising the function, duties, and rights stated above. Based on the Constitution of Nepal, Local Level Governance Operation Act, 2073 elaborated the provisions. 4. Functions, Duties and Rights of Ward Committee The functions, duties, and rights of the ward committee is prescribed by the executive. The functions, duties and rights of the ward committee shall be the formulation, implementation, and monitoring of plans within the ward, data updating and preserving, development work, regulation work, and recommendation and certification. The ward committees are only granted implementation roles as of now. Further responsibilities of policymaking and development planning must also be bestowed upon these committees for grass-root level effectiveness (detail is prescribed in Sec 12, Local Governance Operation Act, 2073). 5. Functions, Duties and Rights of the District Coordination Committee The functions, duties and rights of the District Coordination Committee are ensured by the Constitution of Nepal Art. 220 (6) as follows: a) Coordinate among the Rural Municipalities and Municipalities and Province within the District. b) Carry out monitoring so as maintain balance in development and construction works, c) Maintain coordination among the Federal and Provincial government offices and Village Council and Municipality within the district. d) Perform other tasks as provided for in the Provincial law. In addition to these duties, the Local Governance Operation Act also bestows certain functions, duties and rights to the district assembly, such as coordination within the district in managing issues of development construction, monitoring of various development plans, and ensuring sustainable development; coordination of efforts of disaster management and natural calamity handling in the district; reporting within two months of the end of the fiscal year to the ministry of the provincial government overseeing its local matters; and organization of an annual review program reviewing its development in the year and seeking advice and suggestions from members of the parliament. Apart from these functions, the district assembly shall also carry out other functions as well including tasks assigned by the government of Nepal. 6. CS: MAP Structure at Local Level Based on the above analysis, it is clear that, CS:MAP’s existing project structure seems not competent to address theory of change envisioned by project. • CA is limited to ward level. Ward level committee have certain rights and duties; however, ward level committees have no higher lever decision-making authorities to make laws and policies and endorse a plan of action and budget. Hence, CA may be unable to resolve multifaceted issues which required local level legislative and policy intervention. • Most of the district level offices already have been merged or mainstreamed to rural municipality and urban municipality. So, the purpose of PPWG cannot be served at the district level. The role of District Level Coordination Committee is not a district level decision making body, though they might be influential. 81 APPENDIX VIII: RUKUM CASE STUDY As part of the CS:MAP Mid-Term Performance Evaluation, members of the evaluation team visited the Rukum district to meet with CS:MAP program participants. CS:MAP partner HRPLSC provided a good example of how the relationships between civil society and governmental authorities can be positive. This case study provides a few lessons on how a local organization expanded their scope to support local communities while also earning the trust of government, ultimately becoming a relied upon resource for newly elected officials in the Rukum district. GENESIS AND EVOLUTION Shanti Jagaran (which translates to peace awakening) was a radio program that gained a lot of momentum in Rukum and its neighboring districts during the period of Maoist conflict (1996-2006). One of the main objectives of Shanti Jagaran was to advocate for a peaceful resolution of the conflict. It also functioned as a watch-dog against the excesses of both the rebels and the state security forces. Human Rights Protection and Legal Service Center (HRPLSC), the local CSO in Rukum, is co-founded by the same members who started and ran the Shanti Jagaran radio program. Ordinary people were caught up between the state and the Maoists during the conflict period. Rukum, being a base area of the Maoists, was particularly hit hard. There were significant human rights violations on numerous occasions from both sides. The youth were especially vulnerable, as the state suspected them of being Maoist supporters, while the Maoists regarded them as informers. Moreover, the Maoists launched a massive recruitment drive. The saying in Nepali shows the predicament of youth of the area, “Ki ta militia ki ta Malaysia” (“Either join the Maoist militia or leave the area and go to Malaysia as laborers”). Shanti Jagaran urged the youth against migrating, resist being recruited by the Maoists and work for peace promotion. The programs aired by Shanti Jagaran touched the heart of many in Rukum and surrounding districts. During the period of Maoist’s conflict, it was difficult to post letters since the letters would be intercepted by Maoists, if not by the state security forces. The radio program was a useful way to pass on personal information, such as letters. Shanti Jagaran used to air programs where the personal letters would be read-out by the program conductor – letters that used to be sent by the families to their relatives. Additionally, patriotic songs and poems used to be aired urging people to stand against the odds. Shanti Jagaran was loved by the people and because of its espousal of humane values, later became tolerated by both the Maoist rebels and the state security forces. FROM RADIO PROGRAM TO CSO Two streams of people came together to form HRPLSC. The first constituted individuals who founded and ran Shanti Jagaran. The other set constituted lawyers from the area, many of whom were in Dang. These individuals came together and registered HRPLSC in the year 2000 as an NGO to work in human rights protection and promotion, through awareness-raising activities, advocacy campaigns, and community mobilization. HRPLSC’s central office is in Rukum while its regional office is in Nepalgunj. Presently, there are a total 88 members in HRPLSC, while 41 staff work in the organization. HRPLSC continued with the radio program Shanti Jagaran and engaged the public directly to create awareness on human rights issues, advocating for victims to whom justice had been denied and empowering the victims to seek justice. 82 From the days of the insurgency to the present political times, the journey embarked upon by HRPLSC is commendable. HRPLSC has been able to create a positive image of itself in Rukum and surrounding districts affected by the conflict. Over time HRPLSC has expanded its scope from human rights advocacy, to include local governance oversight, promotion of reconciliation, women’s empowerment, and improved participation of the marginalized community. HRPLSC is also involved in voters’ awareness raising programs – especially so on the eve of the local elections held in 2017. Sunita Magar Sunita Budha Magar is currently the vice chairperson at HRPLSC, and she met with the CSMAPCS:MAPCSMAP mid-term evaluation team during the team’s fieldwork. Sunita originally hails from Rukum but now lives in Dang. She recounted to the evaluation team about how difficult it was to work during the Maoist insurgency period. She told the team about one time when she was abducted by the Maoist who accused her of supporting the state. They kept her for five days but let her go because one of the Maoist leaders in the group was aware of Shanti Jagaran. Eventually, as Shanti Jagaran started getting more popular, the Maoist accepted the noble tenet of the radio program; as the program was able to remain neutral by neither siding with Maoist rebels nor the state security forces. As a result, the Maoists stopped capturing members from Shanti Jagaran and others associated clubs. WORKING UNDER CS: MAP HRPLSC is of one of the 14 district-based CSOs in the CS:MAP activity and works in four CS:MAP districts (Rukum, Rolpa, Pyuthan, and Salyan). The goal of CS:MAP activity is to strengthen the effectiveness of civil society and media oversight over public resource use and public service delivery. The project is underway in achieving mutual accountability at the local level. The organization is involved in encouraging the local level citizens to participate in governance aspects, including utilizing social accountability tools, such as public hearing, and public audits. CS:MAP established various structures at both the local level to the district level. At the local level, it created the Common Assembly (CA) where the members include former Ward Citizen Forum members, former Citizen Awareness Center members, elected local officials and members of marginalized communities. Together with the presence of the media, pressing issues at the local level are raised, and, if possible, resolved at that level. The CA focuses on health, education, agriculture, and disaster (HEAD) issues. Once identified, HEAD issues can then be taken to representatives of local level officials responsible for these functions who can then play a proactive role in addressing the issues once they are raised in the CA. HRPLSC is working in selected wards within the gaupalikas of Tribeni and Bafikot. Of the four HEAD-related concerns, the concern that has been prioritized in Rukum is health. Though CS:MAP intended to only work in one ward of the Tribeni gaupalika, due to the request of the elected representatives, the program soon expanded into five wards – ward numbers 1, 2, 3, 4 and 5. 83 INTERFACING WITH THE ELECTED LOCAL GOVERNMENT Tribeni gaupalika, during the period of the Maoist conflict, used to be a base area of the Maoists where they had formed Jana Sarkar, or people’s government. In the 2017 local level elections, most of the ward chairpersons and members who won the elections in this rural municipality were formerly associated with Jana Sarkar. Because of the indoctrination, the Maoist cadres during the conflict period looked upon the NGOs and INGOs as agents of imperial powers. Added to this was a widespread perception in rural areas that NGOs engage in what is called dollar kheti, or dollar harvesting. Both factors prompted local government representatives to curtail NGOs’ and CSOs’ scope and work. With this perception in mind, the government representatives sought not only to reduce the role of civil society, but also to increase their own mandate in working with their respective municipalities. After the July 2017 local elections, local elected officials demanded all NGOs and CSOs working in the area to register in the rural municipality itself. This move was opposed by HRPLSC and other NGOs working in the district who argued that according to state’s legal provisions, they need to be registered at the office of the Chief District Officer while they need to be renewed at the Social Welfare Council (SWC); not at the rural municipality. After much back and forth on the issue a compromise was reached where it was agreed between the elected local government bodies and the NGOs working in Rukum that NGOs should simply be listed at the rural municipality level, but that neither registration nor renewal at the rural municipality was required. NGOs based and working in Rukum, such as HRPLSC, agreed to this provision since they realized that it was important for local governments to be aware of which NGO is undertaking what kind of activities in their area. Keeping in mind how NGOs and CSOs were portrayed back in the Maoist insurgency, the fact that the CSOs and the local government were able to find a middle ground regarding the regulation of NGOs and CSOs is a positive step. HRPLSC contributed significantly to reaching this compromise partly because of its involvement in Shanti Jagaran and its track record of being able to maintain neutrality in the past. Elected local representatives soon realized that they lacked the technical expertise needed to work on draft laws that rural municipalities are mandated to enact before commencing work. As per the new constitutional provisions, local governments have exclusive rights in certain domains and each urban/rural municipality is required to pass the concerned laws related to these domains. Elected local representatives also realized that they do not have the know-how and capacities – be it related to management or administration or communication – that they need to have in fulfilling their functions effectively. Due to these reasons, the elected officials soon realized that they need to build bridges with CSOs if they are to be able to function well. CSOs like HRPLSC had reached out to the elected officials from early on, and in these circumstances, the engagements between local governments and HRPLSC deepened. Local elected officials soon began to appreciate the work that HRPLSC was doing in some wards – for instance, the work related with CAs – and requested that this be replicated in all 10 wards of the rural municipality. Thus, from a situation of wanting to do away with CSOs, the local governments in Rukum, as exemplified by the Tribeni case, want to engage with CSOs and NGOs and want to form partnerships with them. District level representatives of the federal government, the chief district officer, as well as the elected local representatives, do believe that CSOs, like HRPLSC, provide services that are helping the local 84 communities to understand their rights and duties. HRPLSC, in turn, has assured local elected leaders that it will inform the local government about their activities, working modalities and will keep them informed about their work. In this way, the case of Tribeni gaupalika and the work of HRPLSC there and elsewhere in Rukum district shows that space has been created to allow for dialogue and engagement between CSOs and local government bodies. 85 APPENDIX IX: SUMMARY TABLE OF EVALUATION FINDINGS Key Area Evaluation Questions Answers Generated by Evaluation Design EQ5. To what extent is the design of CS:MAP’s theory of change and operating modalities, structures, and approaches relevant to the new state structure? 5b) What are the modifications and/or re-scoping required in the activities and modalities, if any, given the changes in the governmentalstructures? Theory of Change and Program Design The CS:MAP theory of change remains valid under the changed political context and should continue to guide the project. The theory of change states: “If Nepali civil society and media organizations actively participate in organizational and technical capacity-building efforts to improve their legitimacy, accountability and resilience, then they would be able to advance the public interest.” However, the evaluation team did find that the program design hindered the project’s ability to be more successful. While the original design remains sound, the CS:MAP team, as of the timing of the evaluation, did not implement any changes to that design in consideration of the changing governance structures. For example, many decisions that affect the day-to-day life of Nepalis are now made at the municipal and district level, but CS:MAP continued to focus most of their subnational activities at the ward level (i.e., the former VDC level). The evaluation team finds that in practice, the project is siloed – both in terms of national versus district-based work and within each of its four objectives. CS:MAP’s partners have limited interactions across program objectives – with national level partners not taking advantage of new entry points at a subnational level with district-based CS:MAP partners. Finally, CS:MAP partners expressed concerns that the targeting of 34 districts is beyond the capacity of CS:MAP to effect sustainable change at the district level. The evaluation team concludes that CS:MAP partners’ failure to coordinate their activities with stakeholders at different levels of government is constraining CS:MAP from achieving its objectives. Modalities With the changes in the structure of the state, the VDCs, the unit at which CSOs previously engaged and was envisaged in the original program design, were replaced by municipalities and wards within each municipality. Despite this change, CS:MAP chose to continue as planned at the ward level, which under the new structure was equivalent to the previous VDC. By not working at 86 the municipal level, the activity precluded itself from working with the primary decision-makers at the local level. Structures The evaluation team found that the CS:MAP structures were created in all 34 target districts regardless of whether that made sense for the local community context. The evaluation team found that this top-down approach was different from the method that district-based CSO partners used to determine which communities to engage. Rather than reinforcing the work of the CSOs in support of the project, CS:MAP’s formulaic approach to community engagement often weakened it. During interviews with community members, the evaluation team also found that the structures created by CS:MAP to engage at the community level may not have been the structures preferred by community members. While respondents were not opposed to working through these entities, they could not confirm that these structures were their preferred mode for engaging local leaders. The evaluation team found that while some of the structures created by CS:MAP at the subnational level align with some of the new GON structures, others do not. Approaches The evaluation team found that CS:MAP has a large training component to strengthen the capacity of district-level partner CSOs. The training plan was developed using several tools to assess the partners’ training needs. Some ISOs expressed concerns that the number of training topics and training days left little time for participants to apply their newly learned skills before participating in the next training session. Based on interviews with CSO partners, the evaluation team found no evidence that any mentoring by ISOs to CSO partners had occurred. The evaluation team finds that the CS:MAP program is taking many steps to integrate GESI issues into the program. The team notes, however, that the CS:MAP approach to GESI is accommodative and not transformative in its implementation. Technical Leadership While the evaluation team finds that the CS:MAP team is skilled in some technical areas, there is a significant skill gap in citizen advocacy and good governance programming. Many of the gaps in implementation, including readily identifying and taking advantage of openings for collaboration and reinforcement; developing strategic advocacy initiatives across objectives and layers of governance; and fostering and encouraging a civic culture that is engaged in the political process could be addressed by a capable civil society expert on the CS:MAP core team who has firsthand experience conducting such activities. Design EQ1. Which components and approaches of CS:MAP are the The evaluation team found that the media component at the subnational level is effective. Through speaking with media partners at the national and local level, the evaluation team found that the 87 most and least effective to date in fulfilling activity objectives, and why? 1b) Which activities/interventions have been the most effective and why? Sajha Boli radio program is providing both much needed information on the new government structures and the key issues surrounding those changes. The program is a powerful outlet for raising community needs to a large audience of elected officials and topical experts that have the authority and knowledge to help address those needs. The evaluation team found that CS:MAP and its partners are aligning current legal and policy reform efforts to international standards and practices at the national level. The evaluation team learned through GIs that CS:MAP’s legal and policy analysis is focused solely at the national level, covering ‘right to information’ policies and those relating to the policy environment in which the media and civil society operate. By not working on this component below the national level, CS:MAP is limiting its ability to effect policy changes at the subnational and municipal levels. More than half of the district-based CSOs (57 percent) reported that the OCA was the most useful assistance they received. However, if OCA is the most useful technical support local partners have received, local partners are likely not at a capacity level to take on higher capacity efforts, such as strategic advocacy or policy interventions. Implementation EQ2. To what extent and how have CSOs and the media been effective in supporting identification and resolution of issues of greater public interest? Although the CAs and LDAGs have shown to be effective for identifying and raising issues of concern to local officials, their record of proposing solutions to the issues they raise is less evident. The evaluation team was unable to find sufficient evidence where CS:MAP structures, such as the CA, LDAG, and PPWG, were able to make decisions at the local level more transparent and accountable to the public, both of which are foundational elements of good governance and an integral part of CS:MAP’s goal. The evaluation team found that current CS:MAP structures do not map on to new GON decision￾making structures. The CAs and LDAGs engage at the ward level. Though the ward level is the lowest level of local government, one where government structures interface with the people, it is not the appropriate local government decision-making body. Based on the evaluation teams’ understanding of the new governance structures, the appropriate local decision-making authority is the municipality level. Implementation EQ3. To what extent and how have CS:MAP’s activities contributed to fostering trust among government, political, and civil society stakeholders? The evaluation team did find evidence that both civil society and media partners are building relationships through their activities with government, political parties, and civil society stakeholders at the national and subnational levels. The evaluation team finds, through both GIs and FGDs, that district-based CSOs and CS:MAP structures are engaging at the local level. The team finds this to be a step in the right direction in 88 improving trust. However, at this stage of the CS:MAP implementation, it remains too early to measure the extent to which trust levels have changed among the three groups. Implementation EQ4. To what extent and how is participation and representation of marginalized groups improving at the community level? The evaluation team found that CS:MAP has put considerable effort into integrating GESI in the project. After a thorough review of the project documents and through KIIs, the evaluation team found that CS:MAP is responsive to GESI issues. However, the overall focus on the GESI issues is found to be limited in certain aspects and heavily guided towards women and excluded groups’ representation in its mechanism. The evaluation team reviewed CS:MAP’s GESI Action Plan, which successfully identifies the social and political constraints that collectively lead to systematic marginalization of certain groups. There is serious lack of discussion and agenda setting among CS:MAP’s local structures from a constitutional and human rights perspective of women and marginalized groups and in general. Throughout data collection, it became evident to the evaluation team that women participants were much less vocal and willing to speak than their male counterparts. Women expressed more difficulty than men (mainly related to family duties) in participating in CS:MAP activities and structures and were concerned that they did not have the requisite leadership skills to participate more actively. The tools used to conduct GESI Audit aided the development of the GESI Policies. As a result, all the CSOs have GESI policy in place. The evaluation team was able to review some sample GESI policies developed by CSOs and all of them were found to be accommodative in their approach. They did not however, seek to challenge discriminatory/gendered nature of social norms and unequal power relations and gender roles. Sustainability EQ6. How is the activity progressing on long-term self￾reliance through its sustainability and exit strategies? Given the delayed start of the project due to the earthquake and local elections, the evaluation team found that it is too soon to tell the lasting effects of CS:MAP’s efforts, such as trainings. Through GIs with CSO partners, the evaluation team did learn that many have resource mobilization strategies in place, which ensures a certain level of continuity of some of the activities that the project is implementing. The evaluation team learned that several gaunpailkas were interested in absorbing the CAs into their own official government structures – so that the CAs became government entities. The CAs are currently organized and facilitated through CS:MAP district-based partners and CS:MAP’s Innovation Centers. This is a positive development. 89 Appendix X: GESI Approach of CS:MAP: Critical Observations Findings In 2016, during the design phase of CS:MAP, USAID/Nepal conducted a stand-alone Gender Equity and Social Inclusion (GESI) Assessment of Civil Society Organizations and Media in Nepal. The assessment identified the social and political gaps in inclusion that are faced by Nepalis of different genders, castes, and religions due to various social norms that amalgamate to create a past, present, and future of systematic marginalization. In response, as part of the CS:MAP project, USAID/Nepal developed the GESI Approach and Action Plan, which seeks to close the gaps in gender and social inclusion through several steps, including reviewing existing legal norms and suggesting changes, and providing technical support to national and local civil society organizations (CSOs) to assist them in drafting internal policies on GESI. The evaluation of CS:MAP found, however, that several CSOs are facing difficulties in implementing the policies due to poor planning and lack of resources. In addition, the evaluation found that technical support and mentoring on GESI is limited to CSOs at the district level. CSOs and national partners have yet to provide GESI training to community entities, such as the Common Assembly (CA), the Listening, Discussion, and Action Group (LDAG), and Public￾Private Working Group (PPWG). Hence, GESI related issue identification, agenda formulation, and resource allocation were found to be very limited at the local-level structure. The Common Assembly, the platform where CA members meet and discuss local issues, does not normally address the problems specific to women and marginalized groups, in part, because they are not well represented in the CA. Such groups are also under represented in the LDAG and PPWG, which limits their ability to push those groups to promote GESI issues at the community and national levels. At the national level, CS:MAP's national partners have reviewed through a GESI lens federal legislation related to media and civil society and have been engaging in policy advocacy to amend federal laws so that women and marginalized groups have greater opportunities to participate in governance through civil society organizations and media. discussion and agenda setting of CA, LDAG, and PPWG from constitutional and human rights. The CS:MAP mid-term evaluation also found that the project’s Monitoring, Evaluation and Learning (MEL) Plan lacked indicators to determine: • If there is a change in women's and socially excluded groups' access to resources and opportunities in health, education, and agriculture. • If women and socially excluded groups participated meaningfully in the project and were engaged in activities. • If women and marginalized groups perceived themselves as more prepared and empowered to actively participate in civic affairs. Conclusion Although, CS:MAP has adopted GESI approaches that acknowledge the role of gender norms and inequities and has developed an action that seeks to challenge those existing social norms through training and technical support to national and local partners, change to improve the participation of women and marginalized groups in civic affairs and in gaining access to employment opportunities and services has been slow and uneven. 90 Recommendations • National GESI partners should be encouraged to engage with other national partners to mainstream a GESI perspective in policy analysis and agenda setting of national advocacy efforts. • CSOs should be encouraged to provide GESI training to CA, LDAG, and PPWGs to ensure they include GESI issues in their advocacy efforts. • CS:MAP structures at the local and district level should encourage the active engagement of women and other marginalized groups to ensure their perspectives are included in agenda setting and discussions. Further, CS:MAP should integrate critical awareness raising and training programs on GESI for CS:MAP staff members.