EVALUATION CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Social Impact, Inc., under the Liberia Strategic Analysis activity. PHOTO CREDIT: SINCLAIR CORNELL FOR SOCIAL IMPACT CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID/Liberia Liberia Strategic Analysis Contract No: AID-669-C-16-00002 June 2019 Contact: Liberia Strategic Analysis Social Impact, Inc. UN Drive Adjacent Gate 3 US Embassy Monrovia, Liberia POC: Carla Trippe, ctrippe@socialimpact.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV TABLE OF CONTENTS Abstract......................................................................................................................................... iii Acronyms...................................................................................................................................... iv Executive Summary..................................................................................................................... vi Program Background .................................................................................................................................................. vi Evaluation Purpose ...................................................................................................................................................... vi Evaluation Design and Methodology ....................................................................................................................... vi Findings and Conclusions.......................................................................................................................................... vii Recommendations....................................................................................................................................................... ix Introduction................................................................................................................................... 1 Context........................................................................................................................................................................... 1 Activity Background..................................................................................................................................................... 1 Evaluation Purpose and Audience............................................................................................................................. 2 Evaluation Questions................................................................................................................................................... 2 Methodology .................................................................................................................................. 3 Overview ........................................................................................................................................................................ 3 Evaluation Design.......................................................................................................................................................... 3 Data Analysis.................................................................................................................................................................. 5 Biases and Limitations.................................................................................................................................................. 5 Findings and Conclusions.............................................................................................................. 6 Current Strategy Design............................................................................................................................................. 6 Implementation ........................................................................................................................................................... 11 Sustainability................................................................................................................................................................. 20 Future Strategy Design.............................................................................................................................................. 24 Recommendations ...................................................................................................................... 26 Utilization Plan............................................................................................................................ 28 Annex A: Desk Review Documents Utilized ............................................................................ 29 Annex B: Illustrative Key Informant Interview Questions...................................................... 31 Annex C: Sample Focus Group Discussion Protocols ............................................................. 35 Annex D: Management Capacity Assessment Matrix ............................................................. 41 Annex E: Quantitative Survey Questionnaire.......................................................................... 51 Annex F: Evaluation Statement of Work ................................................................................. 66 Annex G: Disclosure of Any Conflicts of Interest .................................................................... 76 USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | ii Annex H: Full Listing of Persons Interviewed .......................................................................... 81 Annex I: Quantitative Survey Summary Report...................................................................... 82 Annex J: Statements of Difference............................................................................................ 93 iii | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV ABSTRACT This report presents findings and recommendations from the whole-of-Intermediate Result (IR) evaluation of activities generated from the USAID Civil Society and Media Project Appraisal Document (PAD) created in 2015. The evaluation included two current activities—Liberia Accountability and Voice Initiative (LAVI) and Liberia Media Development (LMD)—and incorporated information and interviews from the Civil Society and Media Leadership (CSML) activity, which predated the PAD. The evaluation determined how USAID activities made progress toward achieving the Project Purpose—Increased influence of citizens in the governance of public goods and services—and provided recommendations to inform development of a future Civil Society and Media strategy. The evaluation found that the PAD contributed to increasing opportunities for civil society organizations (CSOs) and media outlets to play a “watchdog” role over governance. However, evidence is lacking that (1) CSOs can sustain the coalition approach beyond current USAID support, (2) CSOs can use the coalition approach to mobilize citizens more broadly to engage with government decision makers, and (3) media outlets can increase the quality of news and information disseminated to citizens. Critical gaps exist in making connections among the three stakeholder groups within governance accountability systems: (1) Political Leaders, (2) CSOs and Media Outlets, and (3) Citizens. A key priority for a future PAD would be to create interactive information and activity loops that better connect political leaders to citizens, using CSO coalitions and media outlets as channels through which citizens directly engage with government institutions. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | iv ACRONYMS AOR Agreement Officer’s Representative CDCS Country Development Cooperation Strategy CODRA Community Development and Research Agency COP Chief of Party COR Contracting Officer’s Representative CSDF County Social Development Fund CSML Civil Society and Media Leadership CSO Civil Society Organization DCOP Deputy Chief of Party DO Development Objective DRG Democracy, Rights, and Governance ECOWAS Economic Community of West African States ELBC (Call sign for the) Liberian Broadcasting Corporation EQ Evaluation Question FGD Focus Group Discussion FOI Freedom of Information GOL Government of Liberia GUC Grant under Contract IIC Independent Information Commission IP Implementing Partner IR Intermediate Result KAK Kamara Abdullah Kamara KII Key Informant Interview LAVI Liberia Accountability and Voice Initiative LD Liberian Dollar LEAP Liberia Electoral Access and Participation LISGIS Liberia Institute of Statistics and Geo-Information Services LMC Liberia Media Center LMD Liberia Media Development LOE Level of Effort LSA Liberia Strategic Analysis MFDP Ministry of Finance and Development Planning MTN Mobile Telephone Networks v | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV NAYMOTE National Youth Movement for Transparent Elections NGO Non-Governmental Organization NMC National Media Council PAD Project Appraisal Document PEA Political Economy Analysis PIDS Performance Indicator Data System PMC Project Management Committee PMP Performance Management Plan PTA Parent-Teacher Association PUL Press Union of Liberia SAVI State Accountability and Voice Initiative SI Social Impact, Inc. SOW Statement of Work USAID United States Agency for International Development VOA Voice of America USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | vi EXECUTIVE SUMMARY Through the Liberia Strategic Analysis (LSA) activity, Social Impact conducted a whole-of-Intermediate Result (IR) evaluation of the activities generated from the Civil Society and Media Project Appraisal Document (PAD) created in 2015. This evaluation’s primary objective is to identify results and lessons learned from implementation and provide succinct, actionable recommendations to inform development of a Civil Society and Media strategy in the new Country Development Cooperation Strategy (CDCS) design. The whole-of-IR evaluation examined the progress, synergies, and sustainability of activities implemented under the Civil Society and Media PAD. At the project level, the evaluation analyzed the development hypothesis, its relevance, and underlying assumptions. PROGRAM BACKGROUND USAID designed the activities in the Civil Society and Media PAD with the goal of achieving Development Objective (DO) 1 of USAID/Liberia’s CDCS: More effective, accountable, and inclusive governance. The project is designed to deliver on IR 1.4 (Strengthen the capacity of civil society and the media to exercise their advocacy and oversight roles) and contribute to two sub-IRs—1.4.1 (Civil society strengthened to play an effective watchdog role) and 1.4.2 (Increased accuracy and impartiality of target media). The developmental hypothesis theorized that increased influence of citizens and media in the governance of public goods and services will result if: (1) there are stronger multi-stakeholder partnerships advocating for and monitoring policy and accountability reforms and (2) citizens’ access to independent and reliable information is increased. The five-year, $29 million PAD comprised two activities: Liberia Accountability and Voice Initiative (LAVI) and Liberia Media Development (LMD). Predating the PAD was the Civil Society and Media Leadership (CSML) activity. The goal of the $19.2 million CSML was to sustain peace in Liberia through greater inclusion, giving a voice to, informing, and engaging Liberian citizens. EVALUATION PURPOSE The purpose of this whole-of-IR evaluation was to examine the entire Civil Society and Media portfolio, including all its constituent activities and progress toward achieving the Project Purpose, “Increased influence of citizens in the governance of public goods and services.” The Project Purpose is the highest-level result to be achieved by a project, supporting the mission’s CDCS Results Framework at the IR level. USAID/Liberia will use findings from this whole-of-IR evaluation to inform adaptive management of the PAD and design of the new CDCS. EVALUATION DESIGN AND METHODOLOGY LSA used a mixed-methods approach to answer the following evaluation questions: 1. Current Strategy Design: To what extent did the programmatic and contextual assumptions of the Civil Society and Media development hypothesis address the underlying barriers to accountable governance of public goods and services? A. Did the new Civil Society and Media PAD design in 2015 improve USAID/Liberia’s approach to achieving IR 1.4? B. How well did the strategy adapt to Liberia’s changing context and capitalize on needs and opportunities? 2. Implementation: How much has the Civil Society and Media portfolio contributed to more effective, accountable, and inclusive governance in Liberia at the DO level? What elements worked or have not worked, and why? A. To what extent have the activity-level objectives contributed to the broader objectives aimed at strengthening civil society and media’s ability to exercise their advocacy and oversight roles at the IR level? vii | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV B. Were there missed opportunities? C. What have been the unintended outcomes—positive and negative—of the portfolio, if any, and how have these influenced progress toward IR 1.4 and DO 1? 3. Sustainability: To what extent is the Civil Society and Media portfolio building the individual and collective capacity of organizations to sustain their work beyond USAID support? 4. Future Strategy Design: Based on these findings, what are the strategic priorities for DO 1 to incentivize Government of Liberia (GOL) reform through Civil Society and Media in the new CDCS design? A. What are new, applicable innovations and approaches being used in other contexts to engage civil society and media in government reform? The evaluation methodology primarily comprised qualitative methods, supplemented by some quantitative data analysis. The evaluation team carried out a comprehensive document review; conducted a total of 68 key informant interviews (KIIs) with USAID, implementing partners (IPs), Liberian service provider organizations, and direct beneficiary Civil Society Organizations (CSOs), associations, and media outlets; conducted six focus group sessions with Liberian citizens in Monrovia, Buchanan, and Gbarnga; and commissioned a quantitative survey of media consumption behaviors and citizen perceptions. Qualitative data collection took place in March 2019, and the quantitative survey was administered in April 2019. FINDINGS AND CONCLUSIONS CURRENT STRATEGY DESIGN Overall, the evaluation showed that many of the assumptions included in the 2015 PAD continue to hold, particularly with space existing for civil society and media sectors to engage and report on democracy and government accountability issues. The USAID approach of “working with the grain” has helped focus on realistic results and incremental changes while finding entry points to engage existing, albeit limited, political will for policy reforms. Passage of the Kamara Abdullah Kamara (KAK) Press Freedom Act has helped to solidify press freedoms within legislation, yet implementation of such reforms remains nascent, with limited public education of key stakeholder groups on the reforms passed within the Act. By intentionally focusing establishing national coalitions to implement coordinated advocacy efforts for targeted reforms, the new PAD directly improved USAID’s approach to achieving IR 1.4. The PAD did not adequately incorporate the impact of national elections on CSOs and media outlets as an underlying assumption. However, LAVI and, particularly, LMD capitalized on the opportunity provided by the 2017 elections by pivoting the portfolio to substantial support for district-level candidate debates nationwide. While there are examples of specific gains through policy changes—including passage of a county social development fund (CSDF) bill and incremental increases in the national education budget— progress remains slow with respect to government accountability in delivery of public goods and services. The strategy outlined in the PAD remained consistent over the performance period and contributed to creation of increasing opportunities for CSOs and media outlets to play a “watchdog” role over governance. However, evidence is lacking that (1) CSOs can sustain the coalition approach beyond current USAID support, (2) CSOs can use the coalition approach to mobilize citizens to directly engage with government decision makers, and (3) media outlets can increase the quality and quantity of news and information that citizens receive about critical public issues. The USAID change under the new PAD to divide civil society and media activities between two IPs has generated greater confidence within USAID that the respective partners can effectively adapt to USAID priorities. Furthermore, USAID stakeholders believed that use of the contract mechanism for LAVI allowed for a more interactive and adaptive response to USAID development priorities. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | viii IMPLEMENTATION Beginning in Year 2, both LAVI and LMD made incremental progress toward creating opportunities for more accountable and inclusive governance. The LAVI-supported CSDF and Education coalitions started engaging senior government officials on the importance of transparent budgeting and project implementation, with significant prospects for passage of CSDF reform legislation and an increase in the national education budget. The LMD effort to support 126 district-level candidate debates during the 2017 elections—bolstered by intentional project design and debate formatting contributed by a Harvard University research team—showed increased citizen demand for election information and increased candidate knowledge about districts’ issues due to participation in the debates. LAVI CSOs made initial steps mobilizing citizens and communities in advocacy efforts, yet LAVI and CSO partners describe continuing challenges in building coalitions, including difficulties implementing coalition activities at the county and community levels, lack of experience implementing approaches to influence policy makers, and lack of experience incorporating media outlets within advocacy campaigns. Media outlets still struggle to improve the quantity and quality of news and information content essential for citizen participation in advocacy and oversight functions. A prominent missed opportunity is the lack of citizen engagement in the CSDF process. Focus group information indicates that a significant gap remains in engaging community members in the CSDF process, with participants having little or no information about County Sittings and Project Management Committee decisions. One widely acknowledged negative outcome of the lack of progress working with the Independent Information Commission, combined with LMD ending the subcontract with Albany Associates, led to the need to “go around the Commission” by working with journalists rather than government institutions to improve use of Freedom of Information procedures. SUSTAINABILITY LAVI and LMD have yet to incorporate a more holistic definition of ‘sustainability’ that encompasses all those management functions critical to the ability of organization to operate effectively. CSOs require effective implementation of strategic planning, project management, human resources, financial management, and revenue-generation planning systems in order to operate beyond LAVI support. Media outlets require effective implementation of editorial management, program scheduling, program formatting, human resources, financial management, and technical maintenance systems—in addition to sales and marketing skills—to continue successful broadcasting after LMD interventions conclude. LAVI training of CSO partners has improved implementation of coalition interventions and has generated progress upgrading some internal management systems. However, significant training on developing advocacy strategies and management practices would allow CSOs to be more effective in implementing their advocacy and government oversight functions. While LMD has trained individual journalists in subject-matter reporting and provided some media management training support, media outlets struggle to implement adequate editorial and management practices, including the use of audience consumption data to create program schedules and program formats and to allocate station resources. Reliance on short-term training workshops, periodic mentoring visits, and short-term follow-up sessions may not help CSOs or media outlets internalize strong management systems and organizational policies that allow daily operations and longer-term advocacy campaign initiatives to continue beyond LAVI and LMD support. Electricity continues to be a critical obstacle for media outlets. The lack of reliable power sources undermines the ability of stations to reach audiences, preventing them from providing the information to audiences necessary to motivate citizens to participate in government decision making processes. FUTURE STRATEGY DESIGN Critical gaps still exist in making connections among the three key stakeholder groups within governance accountability systems: (1) Political Leaders, (2) CSOs and Media Outlets, and (3) Citizens. A key priority ix | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV for the new Civil Society and Media PAD should be to create interactive information and activity loops that connect political leaders to citizens, using CSO coalitions and media outlets as channels through which such information and activities mobilize citizens to engage with government institutions. USAID can incorporate lessons from innovative activities implemented in Nigeria when designing the next Civil Society and Media PAD. The United Kingdom Department for International Development-funded State Accountability and Voice Initiative (SAVI) in Nigeria relies on internal staff that play a hands-on, behind-the-scenes role to facilitate change on citizen priority issues—creating networks among citizens, CSOs, media outlets, state legislatures, and executive government stakeholders to establish coalitions for change. Within the media sector in Nigeria, two alternative-format programs—the multi-actor scripted radio drama “Story Story: Voices from the Market” and the news satire program “The Other News”—engage audiences on key political and social issues such as local elections, electoral fraud, and government corruption. Such programs can be centrally produced and distributed for broadcast over community radio stations in multiple counties. RECOMMENDATIONS 1. Quantitative and qualitative research should be prioritized within both USAID activities and included in activity design from the solicitation phase and throughout implementation. All research proposed by IPs should be first and foremost designed to be utilized as tools to inform CSO and media outlet managers. Secondly, the same research should help inform the IPs when reviewing thematic interventions. Finally, the same research should assist USAID to monitor and evaluate the progress of Civil Society and Media Portfolio activities. 2. USAID should seek to build capacity at the best-positioned CSOs and media outlets. By increasing technical assistance to fewer direct CSO and media outlet partners, the new activities will have a greater opportunity to improve systems at leading organizations and set standards for “modeling behavior” within the CSO and media sectors. 3. Both future activities should improve the use of management capacity assessments as management tools. IPs should include CSO and media outlet managers in the development of assessment methodologies in order to increase buy-in for the assessment process and post-assessment utility. USAID should require year-on-year assessments for all core-supported CSOs and media outlets so that managers can better use assessment findings while USAID uses the capacity assessments to inform activity evaluation efforts. 4. USAID should continue to incorporate the problem-driven iterative, adaptive programming model launched under LAVI for the future CSO development activity. 5. USAID should include a dedicated component to provide solar power electricity generation units to core supported media outlets under the new media development activity. 1 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV INTRODUCTION CONTEXT Liberia has made important strides toward improving its democratic governance over the last decade, although major challenges remain. There is increasing space for citizens, civil society organizations (CSOs), and the media to engage in public policy and service delivery, but there are few mechanisms for holding public institutions accountable. Citizens, especially in rural areas, lack widespread access to information and thus do not have the means to understand and monitor their government. USAID is helping Liberian efforts to meet these democratic governance challenges by providing support for, among other things, the transparent and accountable management of public resources, civil service reforms, free and fair elections, the rule of law, domestic resource mobilization, decentralization, stronger CSOs, and a more professional media sector. While many USAID/Liberia activities support civil society’s ability to deliver services, USAID/Liberia’s 2012–20191 Country Development Cooperation Strategy (CDCS) Intermediate Result (IR) 1.4 focuses specifically on the advocacy and oversight role of civil society: Civil society and media exercise their advocacy and oversight roles. Interventions aimed at achieving IR 1.4 include the Civil Society and Media Project Appraisal Document (PAD) under the Democracy, Rights, and Governance (DRG) Development Objective (DO). The five-year, $29 million PAD aims to increase the influence of citizens and media in the governance of public goods and services. It comprises two activities: Liberia Media Development (LMD) and Liberia Accountability and Voice Initiative (LAVI). Predating the PAD was the Civil Society and Media Leadership (CSML) activity, a $19.2 million activity aimed at sustaining peace in Liberia through greater inclusion, giving a voice to, informing, and engaging Liberian citizens. ACTIVITY BACKGROUND OVERVIEW OF CIVIL SOCIETY AND MEDIA PAD The Civil Society and Media PAD activities aim to contribute to DO 1’s goal of USAID/Liberia’s CDCS: More effective, accountable, and inclusive governance. The purpose of the PAD is increased influence of citizens and media in the governance of public goods and services. The project is designed to deliver on IR 1.4: Strengthen the capacity of civil society and the media to exercise their advocacy and oversight roles; and to contribute to two sub-IRs—1.4.1 (Civil society strengthened to play an effective watchdog role) and 1.4.2 (Increased accuracy and impartiality of target media). The underlying developmental hypothesis is that increased influence of citizens and media in the governance of public goods and services will result if: (1) there are stronger multi-stakeholder partnerships advocating for and monitoring policy and accountability reforms and (2) citizens’ access to independent and reliable information is increased. Through two components, one focused on civil society and the other on the media, the project aims to support the following outputs: • Increased horizontal and vertical linkages among actors engaged in similar issues; • Increased organizational capacity of targeted CSOs to participate in issue-based reforms; • Ongoing capacity development services available on the local market; • Learning and methodologies shared and applied by other development actors; • Strengthened sustainability of media houses; • Plurality of voices represented in Liberian media; • Professionalism of media improved; • Enabling environment for media improved; and 1 The CDCS was originally designed to cover 2012–2017 but was extended. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 2 • The Government of Liberia’s (GOL’s) ability to respond to Freedom of Information requests from citizens and media establishments strengthened. OVERVIEW OF CIVIL SOCIETY AND MEDIA ACTIVITIES TABLE 1. CIVIL SOCIETY AND MEDIA ACTIVITIES ACTIVITY CONTRACT # ACTIVITY DATES FUNDING IMPLEMENTING PARTNER TARGETED INSTITUTIONS GEOGRAPHIC FOCUS Liberia Accountability and Voice Initiative (LAVI) AID-669-C￾16-00003 Nov. 25, 2015– Nov. 25, 2020 $17 million DAI CSOs; Training Service Provider Organizations All 15 counties Liberia Media Development (LMD) AID-669-A￾15-00003 Sept. 18, 2015– Sept. 18, 2020 $11 million Internews Media Outlets; Media Sector Advocacy Associations All 15 counties Civil Society and Media Leadership (CSML) AID-669-00- 09-00104-00 Feb. 26, 2010– Nov. 30, 2015 $19.2 million IREX CSOs; Media Outlets, Media Sector Advocacy Associations, Training Service Provider Organizations All 15 counties EVALUATION PURPOSE AND AUDIENCE The purpose of this whole-of-IR evaluation is to inform USAID/Liberia’s adaptive management of the PAD and the new design of the CDCS. In doing so, it examines the entire Civil Society and Media portfolio, including all its constituent activities, and progress toward achievement of the Project Purpose. The Project Purpose is the highest-level result to be achieved by a project. This evaluation supports USAID/Liberia’s Country Development Cooperation Strategy (CDCS) Results Framework at the IR level. USAID/Liberia will use findings from this whole-of-IR evaluation to inform adaptive management of the PAD and new design of the CDCS. EVALUATION QUESTIONS The evaluation focused on the following specific evaluation questions (EQs): 1. Current Strategy Design: To what extent did the programmatic and contextual assumptions of the Civil Society and Media development hypothesis address the underlying barriers to accountable governance of public goods and services? A. Did the new Civil Society and Media PAD design in 2015 improve USAID/Liberia’s approach to achieving IR 1.4? B. How well did the strategy adapt to Liberia’s changing context and capitalize on needs and opportunities? 2. Implementation: How much has the Civil Society and Media portfolio contributed to more effective, accountable, and inclusive governance in Liberia at the DO level? What elements worked or have not worked, and why? A. To what extent have the activity-level objectives contributed to the broader objectives aimed at strengthening civil society and media’s ability to exercise their advocacy and oversight roles at the IR level? B. Were there missed opportunities? C. What have been the unintended outcomes—positive and negative—of the portfolio, if any, and how have these influenced progress toward IR 1.4 and DO 1? 3 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV 3. Sustainability: To what extent is the Civil Society and Media portfolio building the individual and collective capacity of organizations to sustain their work beyond USAID support? 4. Future Strategy Design: Based on these findings, what are the strategic priorities for DO 1 to incentivize GOL reform through Civil Society and Media in the new CDCS design? A. What are new, applicable innovations and approaches being used in other contexts to engage civil society and media in government reform? METHODOLOGY OVERVIEW The evaluation methodology primarily involved qualitative methods, with some secondary quantitative data analyzed. Primary data collection included document review; key informant interviews (KIIs) including site visits for direct observations in Montserrado, Grand Bassa, Margibi, Bong, and Nimba counties; CSO and Media Management Capacity Assessments; focus group discussions (FGDs) in Monrovia, Buchanan, and Gbarnga; and a quantitative survey measuring media audience consumption and citizen participation behaviors. Upon direction from USAID, the evaluation dedicated the majority of resources to investigate progress following the 2015 PAD. As such, while some note of CSML is included, the evaluation did not cover this activity in as much depth as the others. EVALUATION DESIGN QUALITATIVE DATA Desk Review This evaluation began with a preliminary document review of key project documents, including USAID strategy documents, CSML, LAVI, and LMD agreements and performance reports, and additional reports and research conducted during LAVI and LMD interventions. This preliminary review allowed the evaluation team to familiarize itself with the country context and project background and was instrumental in contextualizing EQs, scope, and tools. Once in country, the evaluation team continued with document review, delving into more details about the various components and agreements of the projects and the roles of stakeholders and focusing on trends and information that required further inquiry. The evaluation team utilized desk review findings to inform design of data collection protocols and presented these findings to USAID. The 37 documents reviewed (see Annex A) also served as a source of triangulation for primary data collected during fieldwork. Key Informant Interviews Using the information derived from the desk review, the evaluation team developed semi-structured qualitative data collection instruments (see Annex B) designed to solicit in-depth knowledge from key stakeholders on strategic approaches, programmatic implementation, and outcomes of USAID Civil Society and Media activities. The KII questions were tailored for each respondent type (e.g., USAID staff, implementing partners [IPs], beneficiary organizations, local resource organizations, and other stakeholders). Questions generated narrative evidence of impact of CSM activities, investigated challenges to activity implementation, and provided commentary on successes and missed opportunities. Information gleaned from KIIs provided context not found in documents such as PADs, IP quarterly and annual reports, and indicator data. The evaluation team conducted 68 KIIs, using a purposive sampling strategy to identify key informant categories, in consultation with USAID/Liberia and IPs and informed by the desk review. This consultative process ensured that the evaluation team conducted KIIs with individuals with deep knowledge of implementation activities in Bong, Nimba, Grand Bassa, Margibi, and Montserrado counties, which USAID identified as priority locations, and accessible during the field data collection period. Key informants USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 4 included senior managers and staff members at IPs, local CSO resource organizations or service provider organizations, direct CSO and media sector beneficiaries, and national- and local-level government officials.2 KIIs were held one-on-one or in small groups with representatives of the target groups below: TABLE 2. KEY INFORMANT INTERVIEWS INFORMANT GROUP CATEGORY TOTAL KIIS USAID 4 IP staff (DAI, Internews, and IREX) 13 National Government Stakeholders 7 Local Government Stakeholders 5 LAVI Advisory Council Members 2 LAVI CSO Service Providers 6 LAVI CSO Coalition Partners 13 LMD Media Support Organizations 5 Media Sector Advocacy Associations 3 LMD Media Outlet Partners 10 Total 68 Focus Group Discussions FGDs with Liberian citizens complemented existing data by discerning the reasons behind behaviors that individuals take. More specifically, they examined the reasons why citizens do or do not participate in advocacy or community-level governance events and the reasons why citizens do or do not consume news and information content from local media outlets. FGDs queried participants on their interaction— or lack of interaction—with local decision making structures and local media outlets in their communities. The evaluation team facilitated six FGDs with 39 citizens not directly benefiting from the two current Civil Society and Media activities to discuss perceptions of citizen engagement and media outlet content. To keep the FGDs focused and comparable, all FGDs followed a common format using FGD protocols (Annex C) addressing and derived from the EQs. To allow for open and active discussion, three FGDs included female participants only, disaggregated by age group, while the remaining three sessions included male participants according to the same age-group stratifications. An informed consent statement prefaced all focus groups and can also be found in Annex C. The statement explained the study objectives, purpose of the FGD, and confidentiality measures. Organization Management Capacity Assessments Concurrent with KII visits to selected LAVI-supported CSOs and LMD-supported media outlets, the evaluation conducted Organization Management Capacity Assessments that identified the existence of key management tools and the performance of critical management functions that promote medium- and long- 2 Some LAVI and LMD beneficiary organizations and service providers included for KIIs also benefited from CSML support. 5 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV term successful operations at CSOs, radio stations, and television stations. In order to generate some indication of sustainable measures within beneficiary organizations, the Management Capacity Assessment Matrix (Annex D) completed at each selected organization verified the presence and implementation of strategic plans, human resource and finance functions, revenue-generation strategies, target audience/community participation research, and recruiting strategies, as applicable for CSOs and media outlets, respectively. The information generated by the Organization Management Capacity Assessments provided important insights into the potential sustainability of CSOs and media outlets supported through Civil Society and Media activities. QUANTITATIVE DATA Media Consumption and Citizen Perception Survey A quantitative survey commissioned in April 2019 through GeoPoll measured audience consumption behaviors for radio, television, and the internet, identified audience perceptions regarding radio and television programming, measured citizen behaviors regarding participation in governance decision making structures such as County Social Development Fund (CSDF)3 County Sittings and community-level education meetings, and identified citizen perceptions regarding the value of their participation in governance structures. The questionnaire used for the survey is included as Annex E, and a summary report including topline data from the survey is included as Annex I. DATA ANALYSIS The data analysis plan was built around the EQs and themes, ensuring that the data collected could be effectively assessed against the EQs. During data collection, the evaluation maintained a SharePoint database to include KII interview notes, focus group transcripts, and organizational management assessment matrices. This database also included information from the document review and site visit schedules. After site visits, the evaluation team discussed information collected to synthesize implications and meaning. The team charted these as preliminary findings, conclusions, and recommendations into a matrix organized by EQ, which was reviewed weekly by the evaluation team, project manager, and project director. This ongoing process helped to uncover remaining gaps in knowledge, identify trends, and highlight areas for further investigation. With qualitative and quantitative data organized by source and EQ, the team was able to triangulate and verify findings, feeding into verifiable conclusions and realistic recommendations. Qualitative data presented in this report was deidentified to ensure respondent confidentiality and was aggregated but not quantified. BIASES AND LIMITATIONS The evaluation encountered the following limitations and biases during the evaluation, listed in the table below with the mitigation strategy employed to ensure data quality and a holistic evaluation design: TABLE 3. LIMITATIONS, BIASES, AND MITIGATION STRATEGIES LIMITATION/BIAS RESULT MITIGATION STRATEGY The evaluation team visited five counties. The evaluation did not reach remaining counties, as well as many Selection Bias: The team analyzed monitoring data disaggregated by selected sites to predict differences between counties. The team 3 The CSDF is a county-level benefit-sharing mechanism that utilizes revenue from natural resource management concessions to support community-level development projects, the first effort in Liberia to use benefits from public￾private partnerships to finance community-driven and managed development initiatives. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 6 remote implementation sites, owing to limited time and human resources. Organizational capacity of selected CSOs and media outlets may have varied from community to community and from organization to organization. Organizations in more urban locations included higher capacity staff and greater resources than organizations in more rural locations. struck a balance between more urban and more rural communities, prioritizing counties targeted by USAID for development assistance. IPs recommended CSOs or media outlets that have shown more successful performance with generating citizen participation or radio stations claiming greater performance and impact. Selection Bias: There is a risk that the team may have only heard from key informants who report positive experiences. The team coordinated with IPs and stakeholder partners to purposively sample a variety of interview participants and location experiences, including utilization of snowball identification of individuals selected for KIIs, where appropriate. LMD and LAVI beneficiaries may have responded to questions with answers that blended their experiences into a composite memory. Beneficiaries who may have participated in more than one activity or in similar activities conducted by several implementers over many years may not distinguish them as separate activities. Recall Bias: Experiences may have been misattributed to CSML, LAVI, or LMD activities. Some interviewees had difficulty recalling specifics of interventions from previous years. The team employed strong data collection tools that specified exactly the intervention being evaluated, and triangulated data, matching information from different sources to corroborate varied experiences. Respondents may have provided positive or negative remarks about an activity because s/he would like to benefit from future USAID-funded interventions, or IPs may have believed that evaluation findings could result in changes to assistance opportunities from USAID/Liberia. Response Bias: Some responses might have been skewed more positively in order to promote CSML, LAVI, and LMD successes. The team was particularly sensitive to the risk of response bias and conducted all beneficiary interviews without IP or USAID personnel present during FGDs and KIIs. The team triangulated data to corroborate data collected during fieldwork with monitoring data and third-party data. SI was a subcontractor to IREX under CSML providing Monitoring and Evaluation support, posing a potential conflict of interest as the firm managing this evaluation. Reporting Bias: SI might have skewed results more positively to promote subcontracting work under CSML. The evaluation did not conduct a detailed analysis of CSML implementation. The evaluation team consisted primarily of independent consultants, especially the Team Leader, and the SI staff directly on the evaluation team had no experience working under CSML. Consultants examined SI support critically and independently, and the Team Leader found no attempts by SI to alter SI-related findings to USAID. FINDINGS AND CONCLUSIONS CURRENT STRATEGY DESIGN EQ 1: TO WHAT EXTENT DID THE PROGRAMMATIC AND CONTEXTUAL ASSUMPTIONS OF THE CIVIL SOCIETY AND MEDIA DEVELOPMENT HYPOTHESIS ADDRESS THE UNDERLYING BARRIERS TO ACCOUNTABLE GOVERNANCE OF PUBLIC GOODS AND SERVICES? Findings The development hypothesis is that increased influence of citizens and media in the governance of public goods and services will result if: (1) there are stronger multi-stakeholder partnerships advocating for and monitoring policy and accountability reforms and (2) citizens’ access to independent and reliable information is increased. 7 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV In the PAD, USAID identified several assumptions that would address underlying barriers to accountable governance of public goods and services: Purpose level (contextual): • Demand for increased accountability does not create an unmanageable backlash from the GOL; • The project is seen as Liberian-led and owned; • There is relative political stability, and there are no major security threats; • Space remains open for civil society and media to engage around and report on democracy and accountability issues; and • Donor support to civil society and media is sufficiently harmonized to avoid duplication of efforts and build synergies. Sub-Purpose 1 (programmatic): • Government response and engagement are sufficient to avoid CSO disillusionment; • Multi-stakeholder partnerships and issue-based advocacy will increase influence on decisionmakers; • Tensions over reform proposals do not preclude joint action by partners; • CSOs are already working on core issues, avoiding artificial diversion of organizations from core purpose; • There is sufficient demand for ongoing capacity development services in the local market; • Partners commit to systematic learning individually and collectively; • Activity management does not exclude non-traditional CSOs; and • CSOs are willing to receive technical assistance from regional/local organizations. Sub-Purpose 2 (programmatic): • Levels of political influence on media content are reduced or remain the same as in 2014; • Access of Liberian citizens to information and communication technologies continues to expand; • The Information Commission continues to implement, and other GOL bodies continue to comply with, the Freedom of Information Act; • Media associations continue to exist and to be relevant; • Press freedom continues at same levels or expands; • Media associations are willing to receive technical assistance; and • Media institutions and advertisers respond to market research data. Overall, KIIs and activity documentation showed that many of these assumptions continue to hold, particularly with space existing for civil society and media sectors to engage and report on democracy and government accountability issues. Even with the tensions associated with the 2017 national elections, political stability has remained in place over the past five years, allowing CSOs and media outlets to develop their “watchdog” roles and engage public institutions on reforms. KIIs indicated that the GOL has at times acknowledged the benefits of the CSO coalition activity, particularly disseminating information about, and advocating for, policy reforms. LAVI program efforts encouraged Liberian civil society ownership and direction of CSO coalition activities under the CSDF and Education thematic windows. A LAVI advisory board established these thematic windows in consultation with USAID. The USAID approach of “working with the grain”—described in the PAD as engaging on issues or reforms with some existing level of support from political leaders and communities—has helped LAVI by intentionally focusing on results and incremental changes, while also finding entry points to engage the existing, albeit limited, political will regarding targeted policy reforms. Key informants from USAID held that the connection between increased citizen participation in government decision making and enhanced accountability of government services would contribute to stability within communities across Liberia. As USAID sees persistent poverty and lack of perceived government ability to address obstacles to economic growth—including corruption—USAID finds that USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 8 civil society and media development activities can play a role in promoting stability through improved citizen engagement with decision making institutions. While there is a thirst among media outlets for training, technical assistance, and access to market research data to better address the barriers to accountable governance, KIIs described unmet expectations as to the volume and quality of training provided by LMD to outlets. Many outlet senior managers have had little to no access to audience research data and so currently have little ability to apply research to station operations. The media sector has seen gains regarding press freedom through passage of the Kamara Abdullah Kamara (KAK) Press Freedom Act, which codified the decriminalization of defamation, yet implementation of the new media freedom measures remains to be seen, with limited efforts to educate key stakeholders (police, prosecutors, judges, politicians, and media practitioners) and the public. The PAD, however, did not adequately incorporate the underlying assumption that national elections would have significant influence on interventions across the Civil Society and Media portfolio. KIIs described the acknowledgement in late 2016 that the upcoming elections would have great influence on citizens nationwide and therefore would affect the performance of media outlets and the lives of citizens working with CSOs in all 15 counties. Only after new leadership took hold at LMD did the portfolio pivot toward encouraging citizen engagement in the electoral process through candidate debates organized throughout all electoral districts. The recognition of national elections as a key opportunity to promote citizen interaction with this important decision making process also allowed LMD and LAVI to coordinate on the nationwide debate interventions, giving the two activities their first avenue to combine their respective subject matter expertise and improve cross-activity implementation. LAVI and LMD later coordinated interventions surrounding the CSDF and Electoral Reform thematic windows. Although there are examples of LAVI activity gains through policy changes—including potential passage of the CSDF bill and increases to the national education budget—KIIs described slow progress in regard to government accountability and delivery of public goods and services. Underlying barriers are deeply political and rooted in power dynamics, elitism, and a system of patronage. As a result, there remains limited room for meaningful civic advocacy and participation in political processes, with few opportunities for constituents to contribute input leading to more effective and accountable government institutions. Building civic participation and CSO influence to hold government to account has started to challenge and raise attention around budget issues in relation to CSDF and Education. For example, the reformed CSDF county sitting participant selection process allows citizens to elect Project Management Committee (PMC) members, but local CSOs and government officials reported that participatory election of members is not taking place across all PMCs. Delegates are often handpicked by county authorities or by Representatives of the House. County-level CSO partners organized events to encourage community involvement through town-hall format meetings, discussions on CSDF projects and budget allocations, and venues to solicit citizen input regarding education budget increases and teacher/student absenteeism. A few CSO partners reported greater understanding from community gatherings, describing youth as feeling more empowered after having received information about the CSDF, the county sitting process, and the CSDF bill. These examples represent early stages of CSOs demonstrating some degree of effectiveness through collective action and establishing dialogues with government on policy reforms. However, CSO strengthening through a coalition-led multi-stakeholder approach has yet to bring about an increase in citizen influence. The development hypothesis assumes that a more strengthened and effective civil society utilizing dedicated coalitions to promote change will lead to increased citizen influence. The evidence from LAVI reporting and KIIs suggest that gaps still exist that prevent citizens from influencing political processes—in particular, influencing the powerful political elites who maintain a hold on decision making. In addition, citizen influence and civil society strengthening alone will not lead to greater responsiveness and better services without increased government interaction with CSOs and media outlets. 9 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV EQ 1A: DID THE NEW CIVIL SOCIETY AND MEDIA PAD DESIGN IN 2015 IMPROVE USAID/LIBIERA’S APPROACH TO ACHIEVING IR 1.4? Findings Through the Civil Society and Media PAD, USAID/Liberia sought to dedicate separate IP resources toward coordinated and improved efforts that would lead to civil society and media exercising their advocacy and oversight roles. After managing the DRG portfolio from 2010 to 2015, USAID KIIs described the CSML activity, which combined civil society and media development programming under one IP, as less effective than the activity designs included in the latest PAD that separated two subject-matter development interventions between LAVI and LMD. In the 2015 PAD, USAID sought to avoid having one IP attempt to “do it all”—at times not implementing either programming stream effectively. In addition, the DRG office sought to increase flexibility of the two activities to respond to USAID requests for adaptive activity interventions. USAID described an appreciation for the separate subject matter expertise that DAI and Internews brought to LAVI and LMD, respectively, in particular the leadership of the two respective Chiefs of Party (COPs) who started in Year 2. USAID saw more constructive linkages between LAVI and LMD after the experience of partnership implementing elections-related interventions, where LAVI complemented LMD support for organization of district-level debates by training journalists and debate moderators on the CSDF process, allowing the first of five debate questions to focus on CSDF so that citizens would learn about candidate knowledge and influence on this local development mechanism. Both LMD and LAVI later absorbed the initiative to engage on electoral reform in support of USAID and DRG priorities, with LAVI adding a third thematic window to incorporate the existing Elections Coordination Committee as a new coalition effort and LMD adding dedicated electoral reform journalism training for 16 fellows so that news reports will be generated across Liberia through September 2019. The 2015 PAD was intentionally focused on establishing country-wide coalitions that combine the efforts and expertise of several Liberian CSOs in coordinated efforts to influence targeted government policy reforms. KIIs described the national coalition approach implemented through the CSDF and Education thematic windows as improving their ability to advocate collectively on policy and budget issues at the national level. KIIs with the IP- and LAVI-supported CSO coalition members described incremental progress in addressing underlying barriers to accountable governance. However, they have not consolidated the ability to engage citizens directly in oversight of, and advocacy for, public goods and services. The coalition approach has resulted in a small number of tangible results from Education and CSDF coalition efforts that can validate the effectiveness of campaign interventions.4 In addition, there is limited evidence that citizen engagement has produced influence over governance of public goods and services. Over the next few months, potential passage of CSDF reform legislation and a new increase in the education budget may help to demonstrate to citizens that the CSO coalition approach can increase accountability and responsiveness from the government regarding delivery of services and management of public goods. Sustained progress would be influenced by the CSO coalition and citizen engagement in the oversight of implementation of CSDF reforms and demonstrated allocation of increased education budget resources down to the community level. 4 Evidence of this includes the 710 multi-party meetings conducted by LAVI as of FY 2019 Q2. In addition, there were 283 consensus-building meetings. These include town hall meetings, legislative engagement meetings, and district advocacy meetings that brought together opinion leaders, political leaders, government officials, CSOs, media representation, and the general public to discuss key issues around policy reforms around CSDF, advocacy on education budget increment, and the implementation of Education Reform Act 2011. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 10 The PAD led to LAVI being awarded a contract, allowing USAID to be more prescriptive through approvals of activity interventions, thematic windows, and Grants under Contract (GUCs). While day-to￾day USAID management burden is increased under a contract, the mechanism allowed for more rapid and adaptive IP response to priorities communicated by USAID. Priorities for thematic windows targeted through LAVI-supported coalition development allowed LAVI to respond to USAID programming priorities across the DRG portfolio and other USAID/Liberia portfolios such as Education and Economic Growth. USAID had direct involvement in the selection of thematic windows, and the DRG team convened USAID/Liberia-wide technical management teams that contributed to development of LAVI’s thematic windows drawn from relevant technical offices. According to USAID, the intentional design within the contract mechanism allowed USAID to develop necessary synergies across offices and incorporate activity-specific information contributing to the cross-cutting value of CSO advocacy for all USAID programming. EQ 1B: HOW WELL DID THE STRATEGY ADAPT TO LIBERIA’S CHANGING CONTEXT AND CAPITALIZE ON NEEDS AND OPPORTUNITIES? Findings Overall, the strategy outlined in the PAD remained consistent over the performance period. In particular, the strategy implemented through the LAVI contract and the LMD cooperative agreement allowed for interventions to adapt as opportunities arose, as described above through the pivot toward national elections debate programming in 2016–2017. In the case of LMD, much of the ability to address emerging opportunities was described as a conversational process between USAID and Internews senior leadership after COP and Deputy Chief of Party (DCOP) changes in 2016. USAID KIIs highlighted the relationship with the Internews COP as allowing for development of evolving interventions over the past three years. The LAVI coalition model takes a problem-driven analysis approach during coalition-building activities to test original PAD contextual assumptions. LAVI partners review constraints and weaknesses in coalition strategies and then adapt to changes in context, government agendas, and CSO coalition activities. LAVI incorporates social network analysis to understand key stakeholders, identify key champions, address government administrative bottlenecks such as within the Ministry of Education, and highlight constraints facing community-level CSDF implementation. In Year 4, LAVI-supported coalitions identified a key point of contact in the Ministry of Finance and Development Planning (MFDP) critical to implementation of reforms, particularly under the CSDF coalition, and engaged MFDP during strategic stages. Engagement exercises have also occurred within the Education theme, but the Education coalition continues to learn ways to best engage with MFDP. While USAID and LAVI CSO partners recognized certain political constraints and underlying barriers to advocacy efforts, national and local CSOs have yet to determine how these barriers can be effectively addressed. For example, local CSOs and stakeholders describe difficulties asking citizens to actively demand government accountability where they recall a history of fear of challenging government officials and the risk of retribution from representatives or local authorities. KIIs with local CSOs and community stakeholders found that simply increasing awareness on CSDF or education issues will not change these historical power dynamics. LMD media outlet partners participated in public events to increase information made available to the public regarding the CSDF process, the potential for increased education resources, and other issues of public interest. However, media managers and stakeholders decried the influence of politicians who interfere with accurate reporting on issues connected to political power. LAVI CSOs and local stakeholders cannot yet describe how they plan to continue advocacy efforts past current LAVI support, in order to capitalize on the opportunities made in their effort after passing new legislation or a new national education budget. Without intentional continuation of CSO coalition efforts that oversee implementation of new reforms, there is limited understanding of how to sustain the effort to hold government accountable for implementing an increased education budget (for example, increasing 11 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV teacher salaries, increasing resources for schools, and addressing poor attendance of teachers and students) or implementation of the CSDF bill (including increasing stakeholder and citizen participation in County Sittings, consistent participatory selection of PMC members, transparency of PMC decision making procedures, and transparency of CSDF budgets). Conclusions Overall, the strategies outlined by USAID in the 2015 Civil Society and Media PAD contributed to increasing opportunities where CSOs and media outlets can play the “watchdog” role over governance. However, evidence is lacking that (1) CSOs can replicate the coalition approach beyond current USAID support, (2) CSOs can use the coalition approach to mobilize citizens to directly engage with government decisionmakers, and (3) media outlets can increase the quality and quantity of news and information that citizens receive about critical public issues. As a result, the ability to engage citizens directly in oversight of, and advocacy for, public goods and services has not yet been consolidated. Citizens have yet to see tangible results from CSDF advocacy efforts that validate the effectiveness of CSO coalition campaign interventions and provide evidence that citizen engagement can influence governance. Proposals of reforms have reached the national legislature, and LAVI-supported coalition interventions have resulted in certain changes to current legislation and the national budget, specifically through amendments to the 2018 budget law. However, at the time of the evaluation, other significant expected efforts to change legislation were yet to be achieved. Potential final results (passage of legislation and budgets, followed by implementation) will help to demonstrate to citizens that the coalition approach can contribute to accountability and responsiveness from government regarding delivery of services and management of public goods. KIIs with coalition CSOs showed that there is not yet a public outreach strategy for combining civil society advocacy and media outlet reporting efforts that inform and engage citizens on implementation of potential reforms—for example, in the summer of 2019 when both the CSDF bill and a new education budget may be passed by the legislature. The PAD strategy has not yet led to a process where CSO coalitions—incorporating media outlets as channels for interacting with target audiences—move beyond simple “awareness raising” to actually engage citizens in efforts to address underlying barriers to exercising political influence. The USAID change under the new PAD to divide civil society development and media development activities between two distinct IPs generated greater confidence within USAID that the respective partners can effectively adapt to USAID priorities. In particular, the shift of LAVI to a contract mechanism allowed for intentional flexibility when determining strategy and tactical implementation, both through implementation of the CSDF and Education thematic windows and in the adoption of Electoral Reform as a third thematic window through 2020. Similarly, based largely on the USAID Agreement Officer’s Representative (AOR)–COP relationship developed with Internews senior management since 2016, LMD integrated USAID priorities through dedicated journalist training on budget transparency, CSDF accountability (utilizing journalist access to public information through the Freedom of Information Act), and electoral reform reporting beginning in 2019. IMPLEMENTATION EQ 2: HOW MUCH HAS THE CIVIL SOCIETY AND MEDIA PORTFOLIO CONTRIBUTED TO MORE EFFECTIVE, ACCOUNTABLE, AND INCLUSIVE GOVERNANCE IN LIBERIA AT THE DO LEVEL? WHAT ELEMENTS WORKED OR HAVE NOT WORKED, AND WHY? Findings Based on key informant and focus group data, while CSOs and media practitioners improved some skills that contribute to creating opportunities for more effective, accountable, and inclusive governance, significant gaps remain within the Liberian context. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 12 As of February 2019—over three years into the activity—the first LAVI thematic window was nearly complete (CSDF), the second thematic window had been one year in progress (Education), and a third (Electoral Reform) was just beginning. Two other potential coalition efforts (Human Rights and Forestry Resources) were under initial development. Looking back to the LAVI start, the IP and service providers found that even the most developed Liberian CSOs had little to no experience operating within the coordinated coalition model, necessitating significant process-oriented training and organizational development. Both the CSDF and Education coalitions have begun to engage legislators and government officials on the importance of transparent budgeting and project implementation. The More4Education campaign contributed to a proposed increase in the national education budget from 13 percent to 14 percent of the overall government 2018 budget. The coalition found champions in the legislature for amendments around the Education Reform Act, but results have not yet been consolidated, with a lack of information about how local advocacy efforts in counties are contributing to national-level engagement on the Act. Successes by Year 4 include lawmaker efforts to review and pass the CSDF accountability legislation, which would potentially allocate funds from concessions directly to county-level project management control and oversight of CSDF projects, as well as legislative efforts to increase the national budget for education by as much as 20 percent for the next fiscal year. Both efforts remained in progress as of March 2019. An excellent example of successful implementation through “working with the grain” resides with the LAVI thematic window and coalition advocating for an increase in resources allocated to schools through the national education budget. Education is an issue that directly touches the lives of families in all communities, and so the LAVI-supported advocacy effort took advantage of citizens already primed to engage to improve conditions and teaching in their children’s schools, as indicated by focus group participants: I attended PTA meetings because my son attends that school . . . Children call the school “government school” because many students are not willing to learn and some teachers feel that since they are on the state payroll, they don’t care about teaching. [PTA meetings] allow me to express my feelings. – Female, 25–34 years of age, Gbarnga I went to PTA meetings to get more information about our Community College, because the association told us that the President said everyone should go to school for free. We went to the meeting and talked about school business. But since then, we have asked again, but people tell us that the President has not yet sent the [education] budget that would make school free of charge, so we continue to wait. – Female, 25–34 years of age, Gbarnga I attended a recent PTA meeting to hear from the administration. They said that teachers from across Liberia were on strike demanding for increase in salary, so they called parents to let them know that there would be an increase in school fees. – Male, 18–24 years of age, Buchanan I attended several PTA meetings because I have my children in school. I usually go there to find out how our child makes an effort in school. Normally, we send our children to school, but they go somewhere else [absenteeism]. At the PTA meeting we get information about whether our children attend school regularly or not. . . . The meeting helps to strengthen me [as a parent] to hold them with an iron hand and make sure that they do better in school. – Female, 35–55 years of age, Monrovia Participation in the PTA structure is a successful example of how citizens across gender and age groups included in FGDs described effectively engaging in inclusive governance. Local county-level PTAs provide a venue through which key stakeholders—school administrators, teacher, parents, community leaders— address critical issues such as teacher and student absenteeism, report on student enrolment, and discuss payment of teacher salaries. CSOs and education-focused representatives report changed attitudes of lawmakers around education, based on successful relationship-building during the advocacy campaign. 13 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV Multiple stakeholders, including USAID, perceive that the potential budget increase would not have moved forward as an issue without the LAVI Education coalition advocacy campaign. Another highlight from the Civil Society and Media portfolio was the effort surrounding national elections in 2017. As described above, with the arrival of the long-term Internews COP in 2016, LMD initiated a coordinated effort to support 126 district-level debates for candidates competing in House of Representative elections across all 15 counties. The initiative was bolstered by the USAID-driven inclusion of the Harvard University research team, in particular the Harvard team’s incorporation of consistent and intentional design of the debate formats that would allow for pre-election and post-election measurement of voter participation. All debates included two common questions from moderators, one specifically asking candidates about their knowledge of the CSDF process, and then included questions on local district issues. LAVI expertise was brought to train debate moderators about CSDF issues to inform questioning. Debates were broadcast live and rebroadcast to expand voter access to debate content prior to elections, with local radio stations collectively airing the district debates over 570 times across Liberia. The Harvard research found that citizens increased their demand for political information through radio listening, discussion of politics within social networks, and demand for other sources of political information. Community members discussed the debates and coordinated vote choices more with their friends and family as a result of their interaction with debate content. Forty-five percent of candidates reported that they were forced to learn more about their districts owing to their participation in the debates. Led by LMD support, in combination with the Harvard research effort, the election debate programming represented an outstanding example of intentional intervention design that not only promoted interaction between voters and candidates through public events and media channels but also allowed for dedicated measurement of the impact of USAID support for interventions intended to promote inclusive governance, where voters can increase informed participation in elections as a critical national decision making process. EQ 2A: TO WHAT EXTENT HAVE THE ACTIVITY-LEVEL OBJECTIVES CONTRIBUTED TO THE BROADER OBJECTIVES AIMED AT STRENGTHENING CIVIL SOCIETY AND MEDIA’S ABILITY TO EXERCISE THEIR ADVOCACY AND OVERSIGHT ROLES AT THE IR LEVEL? Findings Building the skills of selected Liberian CSOs and training journalists made some progress in improving CSO and media sector advocacy and oversight functions through USAID activities. However, significant skill and resource gaps remain in ensuring that CSOs and media outlets can replicate and continue these core functions with or without USAID support. Through the first thematic window supported by LAVI, DAI invited proposals from individual CSOs through a request for proposals. After review, LAVI requested that eight organizations form a coalition as a method of increasing potential impact. The eight CSOs—Sustainable Development Initiative, Liberia Education Network, Platform for Peace for Development, National Youth Movement for Transparent Elections (NAYMOTE), Rural Human Right Activist Program, and Citizens United for Peace in Liberia— formed the initial group but, while knowing each other, had never worked together in the form of a coalition. Originally seeking a broad focus on the natural resource extraction sector, LAVI encouraged the group to narrow the advocacy effort down to an examination of the CSDF process and a call for CSDF reform. The LAVI approach to coalition building included training to reinforce collective action, incorporating problem-driven adaptive and strategic thinking, quarterly learning meetings, county inputs, and awareness-raising interventions to influence county government. Several LAVI CSO partners described successful improvement in their institutional skills and ability to organize advocacy campaigns as a result of LAVI training and capacity building. LAVI support to CSO USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 14 partners was structured around needs identified during participatory capacity assessment sessions, which included development of monitoring and evaluation systems, strategic planning, financial management, communication and advocacy, and gender inclusivity. KIIs with LAVI-supported CSOs described progress with professional skills that allowed them to be more effective. Even with the process-intensive approach to capacity building within the LAVI coalition interventions, participating CSOs reported still having limited knowledge regarding the administration of government concessions and natural resource management generally. For example, three different organizations recalled challenges engaging with local government authorities who also lacked information on government concessions, including a reluctance from local officials to participate in discussions about natural resource management and concessions awarded to private companies. LAVI CSO partners have made initial steps to improve mobilization of citizens and communities into advocacy campaign efforts at the county level. In early 2018, a new natural resource management strategy was developed to guide CSO interventions, including identification of entry points to engage the national legislature to kickstart CSDF reform. Awareness-raising events allowed CSOs to inform parents, youth, chiefs, and local associations about the CSDF process and the prospective CSDF bill. The events also described the role and selection of the PMCs. For example, NAYMOTE facilitated two community reporting events for CSDF stakeholders in Maryland and River Gee counties in February 2018, and other coalition CSOs have organized county-level events to discuss how to engage political leaders on CSDF issues and how to better incorporate women in the CSDF process. At the same time, LAVI and CSO partners described several challenges with the coalition-building process: 1. Lack of CSO knowledge of building effective strategies for influencing change through the coalition￾building approach, including influencing policy makers and legislators; 2. Underestimation of the amount of technical support and resourcing necessary to “hand-hold” coalitions in order to be effective and have impact; 3. Difficulty of local CSOs with implementing coalition activities at the county and district level, including effective engagement with citizens, communities, and local organizations; 4. Difficulty developing research materials and evidence to engage senior government policymakers; 5. Lack of experience within coalition members using media outlets as partners in advocacy efforts, particularly where outlets avoid accurate reporting in fear of retribution by political leaders; and 6. Inability to implement a larger number of thematic window-based coalitions without additional funding and human resources. According to KIIs with LAVI and coalition partners, the challenges above contributed to a limitation in number of thematic windows addressed to date through LAVI, as well as limitations to the ability of the existing CSDF and Education coalitions to effectively deliver greater results in supporting national efforts to advocate for government reforms. While some LAVI-supported CSOs used tools such as community scorecards and government leader pledge cards as a first step to holding political leaders and government officials accountable for delivery of services or management of public goods, the same CSOs described a lack of knowledge as to how they would follow up with the scorecard/pledge card tool in order to mobilize citizens to continue efforts to hold their government representatives accountable. LAVI CSO partners continue to face challenges applying strategic thinking and advocacy engagement skills in order to influence national-level political will. Political economy analyses (PEAs) within the CSDF, Education, and Electoral Reform thematic windows, combined with associated learning events, have contributed to LAVI partner understanding of how to improve traction with senior government leaders on thematic windows. However, national-level coalition members and local CSOs reported that they would benefit from extending analysis from the county level through to the national level regarding how to identify (1) avenues for citizen engagement, (2) entry points to influence national political leaders, and 15 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV (3) methods to increase responsive behavior by government, where government finances rarely trickle down to counties and districts. From the perspective of citizens on media outlets, limited progress has been made regarding the ability of media to effectively participate in advocacy and oversight of governance. Media focus group participants across gender and age groups described a thirst for media outlets that provide balanced and accurate news and information: Stations should capture the facts and not just repeat from the [main] actors within an event because the actor might be talking in his/her own interest, and therefore mislead the public. . . . If journalists capture the side of the normal person, then stories will present the facts for the public to judge. – Male, 35–55 years of age, Gbarnga Stations need to do research properly and get facts from that information before they bring it to the public. . . . Everyone can tune in to your radio station . . . but if you don’t do your research properly, people will turn away because they will not trust you. – Male, 25–34 years of age, Monrovia First and foremost, radio stations need to void itself of politics. . . . Then need to report the facts. Once you are balanced in your reporting people will always listen to you. The thing about radio is not just about music, but about giving information. If you see journalists taking sides, people will always shut their ears. [Stations] need to be balanced and get out of politics. – Male, 35–55 years of age, Gbarnga Citizens from focus groups reiterated the challenge described in USAID’s 2015 PAD regarding the damage to station credibility from political influence on outlets and individual media practitioners: I feel two radio stations are biased [in my county …]. During the elections period, [one candidate] vied for the senatorial post against [another]. The two radio stations were divided: [one station] was in favor of [one candidate] while [the other station] was in favor of [the other candidate]. As a result, [each station] preached more information about their own candidates. – Male, 35–55 years of age Our radio stations in Liberia are not trustworthy because of conflict of interest. If someone calls in to debate with you, they insult you on the radio. Some callers get money from politicians to go on-air and defend them. When you call, [the talk show host] cuts you off, or allows the [paid agent] to insult you. Most of the radio stations in Liberia do that now. – Male, 25–34 years of age, Monrovia Media outlets both in Monrovia and in counties interviewed during the evaluation described struggles with increasing quantity and improving quality of news and information content. Station managers regularly cite the challenges of training staff, acquiring and retaining staff, and finding adequate financial resources required to increase newsgathering and production. Those media outlets supported by IREX during the CSML activity recalled the benefit of resources provided for reporting and production of news and information content, in particular the resources made available to increase reporting and information disseminated during the Ebola outbreak. Since 2015, however, the media outlets interviewed described support provided through LMD as largely limited to training of individual journalists, periodic (but few) onsite training visits to stations, and targeted equipment (computers, digital recorders, microphones, and in some cases replacement of damaged transmitters). Where training was provided, participants described difficulty with continually using transferred skills due to a lack of financial resources available to replicate new newsgathering practices. As a result, stations rarely add new programs or augment program formats, citing the need for additional human, technical, and financial resources. Similarly, LMD has struggled to improve media management practices at the 21 supported media outlets. The Liberia Media Center (LMC), used by both IREX and Internews over the past nine years of media development assistance as a primary local provider of journalism, media management training, and advisory services, is widely considered to have low capacity itself, a situation exacerbated by recent turnover in senior leadership. LMD-supported media outlets report participation in sporadic training workshops with USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 16 minimal follow-up by LMC or Internews to assist with implementing new journalism, production, technical, and management skills based on individual station needs. Even though LMD reports that all 21 supported media outlets have created business plans, all but one of the stations visited either do not recall such plans, do not have access to those business plans, or only refer to the business plans when receiving requests from LMD or LMC connected to reporting. Business plans that outlets could produce were largely based not on the reality of day-to-day media operations but instead focused on aspirational proposals for side businesses that would generate alternative revenue. However, these plans are unrealized, as they require significant up-front capital investment for which media outlets do not have the financial resources. Similarly, the media capacity assessments conducted by LMD and LMC were considered to have little relevance to the actual operations at media outlets and were not used to inform station managers when making decisions about station programming or planning. Development of media sector support associations continues to suffer from poor institutional capacity despite nine years of support through CSML and LMD. The Press Union of Liberia (PUL) participation in the National Media Council (NMC) is improving, and the NMC itself is developing as a nascent alternative dispute resolution mechanism to address accusations against journalists and media outlets. However, the PUL leadership cites a lack of resources and skills to effectively perform basic outreach and advocacy functions, including maintaining an up-to-date website/social media presence, and has yet to secure resources to implement proposed training and advocacy around the recently passed KAK Press Freedom Act, which decriminalized defamation. Some CSOs describe making use of meager resources to implement activities, including promotion of women within media outlets as journalists, radio presenters and (ideally) in senior management positions. They utilized basic LMC monitoring data on female journalists, female sources, and gender-focused topics appearing in media items from Monrovia-based outlets to advocate for additional gender balance through media production and content. However, such CSOs fail to practice basic management skills through organizational strategy, organization accounting, and planning, primarily depending on project-to-project financing to maintain core operations. While these CSOs have received significant support from USAID activities dating back to 2010, capacity within both associations remains quite low, diminishing their ability to play an active role in developing the media sector enabling environment. EQ 2B: WERE THERE MISSED OPPORTUNITIES? Findings Considering the overarching purpose of the 2015 Civil Society and Media PAD—“Increased influence of citizens and media in the governance of public goods and services”—there are indications that while the CSO coalition-building approach and training of journalists may have increased the capacity of targeted CSOs and media practitioners, significant gaps remain as missed opportunities to increase the ability of advocacy and accountability efforts to mobilize citizens themselves as a method of influencing government decision making. Feedback from focus group sessions indicated that citizens had some degree of awareness of CSO activity in their local communities, in particular knowledge of organizations that implement activities directly connected to daily lives of families: The Grand Bassa Students Union is a group that advocates for the rights of students. They promote programs that bring students from across various schools to participate in debates, quiz programs, football games, and student retreats. They make students feel proud, associate with other students, and help high school dropouts go back to school. – Male, 18–24 years of age, Buchanan The National Youth Movement for Transparent Elections [NAYMOTE] had a recent activity to train young women in governance and politics, hoping to engage women in politics five or ten years from now. Women are expected to learn how to organize campaign activities, build self-esteem, and build more confidence 17 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV in themselves so they can take leadership positions. I like that the program will increase the number of women in our leadership structure and also bring about development. In Bong County here there are a limited number of women in elected positions. – Female, 25–34 years of age, Gbarnga We have a youth organization in our community called WE CARE. They organize events to clean up around the community. Every Saturday, we go in the community to clean up in the area. I have been to so many meetings where we discuss drug abuse, violence against women, and teaching children the importance of going to school and behave well in the community. I have children and I don’t want to spoil their future. – Female, 35–55 years of age, Monrovia However, there is less progress indicated regarding engaging citizens in the CSDF process, particularly in regard to mobilizing citizens to participate in County Sittings and other events where projects are prioritized, selected, or shared with community residents. Many focus group participants had received little or no information about CSDF county sittings or other meetings that discuss CSDF development projects: I have not attended a County Sitting, but I would like to go so I can know how the funds that are being allotted to the county are being distributed, and what projects are being prioritized. I would be able to check the things they promised they were going to do, and if they did those projects, I would thank them for their work. – Female, 25–34 years of age, Gbarnga [I have not been to a County Sitting], but I saw in the newspaper where the PMC described all they did for the county that year. For example, they described painting our pavilion saying that it cost close to 75 thousand Liberian Dollars (LD) just for painting. But when you see the paint work, you can’t say it was more than 10 thousand LD. I want to know what is unfolding in my county and how people can benefit from the social development fund. – Male, 18–24 years of age, Buchanan I never attended a County Sitting, but people who did attend say that they heard at the County Sitting that the funds are not really put to use in our county. One project, our main street, has only seen work off and on. Back in 2012/2013, the committee was bringing youth together to sponsor them to get scholarships to attend colleges like Cuttington University or other high schools. But now, nothing like that is coming from the Sitting. These days we are not hearing anything, like they just put a stop to everything. – Female, 25–34 years of age, Gbarnga In part, while the nascent coalition-building approach developed by LAVI has built skills within key national or larger CSOs, the coalitions have not yet brought issue-oriented advocacy efforts down to the level of communities and citizens on a consistent basis. LAVI interventions to date have not yet been able to effectively coordinate and hold respective coalitions together across national- to county-level organizations that are directly connected to citizens “on the ground.” Work at the county level is not carried through to build consistent relationships across relevant local actors that can build momentum and effort with LAVI county and national level partners within one unified coalition effort. The challenge of fixing the capacity gap for effective advocacy requires not only the supply of timely information to all levels of the coalition but also creation of a common understanding of the advocacy process and incentives behind advocacy efforts. In one positive example, the Bong County-based CSO Community Development and Research Agency (CODRA) conducted an assessment that found that beyond the attraction of donor funding to facilitate advocacy efforts of civil society, CSOs and community￾based organizations reported that advocacy engagement was not carefully explained to all county-level actors. Hence, some very important steps for joint action were overlooked. Promoting cohesion within current LAVI-supported coalitions remains weak, in particular the methods through which national-level coalition leaders share information with county-level CSOs responsible for organizing local advocacy events. As a result, the closer coalitions get to communities, the more coalitions reflect “marriages of convenience” that are difficult to sustain. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 18 While PEAs are a useful method of informing the broader coalition approach at the national level, these research instruments have not yet provided in-depth analysis at the county level. By extending the PEA process down to the county level, the LAVI coalition efforts could have better mapped out barriers to citizen engagement in local advocacy efforts through identification of key local-level actors, entry points for interaction with local political structures, and opportunities to mobilize reformers on behalf of CSDF and education oversight efforts. Generating PEAs from the county level would better inform national-level policy efforts, while at the same time increasing coalition reach to—and participation from—local constituents. The LAVI Education Coalition has yet to develop an effective engagement strategy for senior government leaders not already associated with education as an issue, such as the senior national legislature leadership critical to facilitating all legislative debate in both houses. Certain House Representatives have been brought into public awareness activities on education issues; however, the legislators targeted were primarily limited to those already focused on education. Other senior executive branch officials and legislature leadership have yet to be engaged. KIIs with private sector firms having provided capacity building services to LAVI-supported CSOs also indicated that there has been no attempt to analyze and understand how to incorporate the private sector within civil society advocacy efforts. Stakeholders interviewed suggested the potential to include private sector associations into thematic areas that address corruption or promote a better business and enabling environment for economic growth. Engagement and strategic thinking around private sector engagement are inconsistent. Service provider stakeholders described a significant opportunity to influence the private sector by advocating for changes that add value to accountability measures, including reforms surrounding employment practices, procurement, or grand corruption. The focus on developing national-level CSO coalition and advocacy skills within the LAVI-supported coalition-building effort missed the opportunity to reach down to the traditional family level, with common citizens unaware of greater advocacy efforts. For example, some families do not allow their children to attend school, and CSOs cited poverty, tradition, and vulnerability as also preventing parents from acknowledging the relevance of education. Activities within the education advocacy campaign did not take such family apathy toward schooling into account. The focus on policy or the national budget fails to meet community-level concerns, with stakeholders citing a need to include both the national and community￾level contexts when developing a coherent strategy to both reach impoverished citizens to increase demand for education and seek an increased national education budget. One longstanding missed opportunity within past USAID-supported media development interventions involves qualitative media content monitoring. The Liberian organization performing media content analysis for the past nine years received media monitoring training through both CSML and LMD, and yet both Internews and USAID cite the organization as having a poor understanding of actionable media monitoring research and standard methodologies. As a result, the monthly product created by the Liberian media monitoring organization has little application from the perspective of media outlets, particularly as a method of improving editorial content. The senior management of the organization believe that their reports are solely used by Internews for sharing with USAID. EQ 2C: WHAT HAVE BEEN THE UNINTENDED OUTCOMES—POSITIVE AND NEGATIVE—OF THE PORTFOLIO, IF ANY, AND HOW HAVE THESE INFLUENCED PROGRESS TOWARD IR 1.4 AND DO 1? Findings The evaluation found only two specific unintended outcomes from the USAID civil society and media development activities, (1) a positive spillover effect of the LAVI coalition intervention and (2) the limited results supporting GOL institutions charged with leading Freedom of Information efforts. 19 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV LAVI-supported research for the education campaign helped CSO partners identify particular issues at the county level. CODRA used the evidence-gathering process to examine how the organization might address local problems such as delays with teacher salary payments, which contribute to teacher absenteeism. The PTA in Bong County, with CODRA, supported an advocacy effort with the Ministry of Education, local school boards, and the County Council to seek payments for teachers, resulting in increased teacher attendance. Without the education campaign-associated research, CODRA believed that these teachers would never have had their salary problems addressed, a positive outcome not directly associated with Education coalition interventions. A dedicated component of both the PAD and the LMD activity focused on promotion and use of the 2010 Freedom of Information (FOI) Act, in particular interventions designed to build GOL capacity through the Independent Information Commission (IIC) to enforce, oversee, and monitor implementation of FOI regulations. The original LMD activity included a subcontract with Albany Associates to provide training and other capacity support to the IIC, which previously had benefited from technical assistance from the Carter Center under the CSML activity. However, KIIs with various stakeholders found that the relationship with the IIC failed to produce opportunities to enhance the role of the IIC within the GOL as a promoter of the FOI process. LMD and USAID could not meet IIC expectations for material and financial support, which did not appropriately fall within the scope of USAID assistance, while the commission leadership found problems with the lack of an established understanding of what assistance LMD might provide and how that assistance would be delivered. Furthermore, across government institutions, many ministries and agencies do not have dedicated Public Information Officers in place, or, where they do exist, they are not provided adequate resources to respond according to FOI regulations. After two years of limited interventions, LMD and USAID found that the sizable financial resources needed to continue Albany engagement with the IIC would not produce expected results, and therefore agreed to end the subcontract with Albany. The unintended outcome led to LMD “going around the Commission” according to USAID, by instead directing training on FOI request procedures to a targeted group of media practitioners. Conclusions As a relatively new approach in Liberia, there is a significant lack of experience and skills among CSOs working within coalitions. Developing coalitions to strategically address key issues took longer than expected. However, by Year 4, the coalition approach utilized by the CSDF and Education thematic windows has shown successes with policy recommendations delivered to the legislature, with strong prospects for passage of policy reforms. In addition, LAVI will incorporate Electoral Reform as a thematic window in Year 4—an example of the intentional design of the 2015 PAD and LAVI to fill USAID programming gaps across the DRG portfolio. The LAVI experience through nearly four years shows that building relationships among CSOs, lawmakers, and government ministries is a time-intensive process. The coalition approach brings a collective voice and effort of civil society stakeholders around policy changes, for which legislators interviewed described their appreciation regarding the ability to access evidence from coalition activities and inform legislative debate. However, significant challenges remain with accessing county-level information on CSDF budgets, where information often may not exist. Limited CSO access to information and knowledge of how concessions contribute to CSDFs and inconsistent application of decision making procedures result in a lack of clear information available to citizens. LAVI is progressively building CSO capacity through implementation of advocacy campaign interventions and events. However, there is less evidence regarding how potential implementation of new national-level CSDF legislation, or an increased national education budget will provide local communities with the tangible results that in turn motivate citizens to become more engaged in holding government accountable for delivery of local development and education services. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 20 Establishing a functional strategy where CSO coalitions can directly engage with senior government legislators and executive branch officials remains a serious challenge to the ability of LAVI coalitions to affect national policies. LAVI coalitions struggle to have impact on political will and government responsiveness—connected to the need for greater coalition strategic planning and experience addressing bottlenecks in legislative and executive administration processes. The LMD approach to training individual journalists has allowed the activity to introduce subject-matter journalism to several media practitioners across the country. However, without resources required to systematize newsgathering skills within daily programming, from the audience perspective the number of improved reports available are few and not broadcast in the volume necessary to reach listeners. Similarly, the limited training opportunities for media outlet managers make it difficult for stations to replicate new skills and instill management practices within daily operations. Moreover, the effort and resources dedicated to media monitoring and analysis through LMC throughout the past nine years have not been applied by editorial managers to improve journalism content. As a result, the combination of sporadic training and little-used media content analysis has had limited impact on improving the daily news and information programming required for media outlets to play a more effective role in holding government accountable for performance and delivery to constituents. The inability to provide effective support to building capacity of the IIC to lead FOI efforts within the government affects both government transparency efforts and the ability of media outlets to play their critical watchdog role. The experience through CSML and LMD indicates that in order to give the approach of embedding advisors with government institutions the best chance to succeed, USAID must dedicate appropriate resources and require an IP approach that improves interaction with government institutions, including having written memoranda of understanding between the implementer and government institutions that define the scope of activity contributions to beneficiary institutions and set clear (written) expectations between the IP and the government entity – understanding that not all GOL institutions may have the political will in place to embrace expert technical assistance. USAID described a recent intervention where a long-term embedded international advisor improved institutional capacity at another government institution, but unlike the Albany Associates example, that advisor maintained a more sustained physical on-site presence. SUSTAINABILITY EQ 3: TO WHAT EXTENT IS THE CIVIL SOCIETY AND MEDIA PORTFOLIO BUILDING THE INDIVIDUAL AND COLLECTIVE CAPACITY OF ORGANIZATIONS TO SUSTAIN THEIR WORK BEYOND USAID SUPPORT? Findings LAVI seeks to support sustainability of individual organizations and coalitions through training and mentoring on advocacy skills and organizational management practices. The primary capacity-building effort is through effective “learning by doing,” namely, development of the coalition approach where CSOs work together to tackle CSDF and education issues both nationally and at the county level. LAVI has developed a pool of strong service providers that work with LAVI partners on internal systems, project management, and financial management skills. LAVI utilizes a participatory assessment tool (the Institutional and Advocacy Capacity Assessment) to help CSO partners understand their organizational capacity regarding specific management skill areas and monitors progress against results. LAVI partners described some progress in gender inclusivity, monitoring and evaluation, budgeting and financial management, and advocacy strategy development. However, partner CSOs do not report progress toward long-term sustainability, with additional training and skills development required to embed these management practices within daily operations. Strategic planning. CSO Management Capacity Assessments conducted during the evaluation highlighted clear weaknesses with institutional systems. The large majority of CSOs continue to rely on project- 21 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV specific budgets, with limited or no financing available for larger organizational needs. There is no sense of business planning that establishes strategies for larger revenue generation or fundraising beyond simple responses to individual requests for proposals. The majority of CSOs have developed organizational vision and mission statements but do not have strategic plans aligned to those visions. KIIs described a great deal of training still needed to support development of strategic planning frameworks. CSO managers also cited the importance of connecting citizen priorities to organizational initiatives. Although CSOs have made progress in developing approaches to advocacy, organizations seek training on how to link research evidence on citizen participation to advocacy efforts that mobilize citizens to engage in events regarding policy reform and delivery of public services. CSO technical capacity. KIIs with LAVI CSO partners indicate an improved understanding of advocacy practices around policy reforms, understanding of legislation, and linking of evidence to policy debates and legislative discussions. However, these are relatively new skills for the majority of national-level CSO partners. Key gaps exist in understanding the national budget process, how to engage with the Ministry of Finance and Development Planning, and methods to better interact with the national legislature. CSOs and stakeholders cited a need for strategies that inform the public about the efforts of Education and CSDF campaigns, including outreach to illiterate audiences. Currently CSOs within the CSDF and Education coalitions feel constrained by the limits of IP financing and donor project timelines. CSOs are not certain how to move forward with CSDF and Education coalition activities in order to oversee implementation of forthcoming legislation and increased education budgets, in particular how to engage citizens in holding government accountable for reforms and budgets approved by the Legislature. CSOs report struggling with how coalitions might sustain engagement with communities beyond dedicated LAVI support. CSO partners predict that growing community awareness on education and CSDF issues may falter without continued engagement. LAVI-supported CSOs also reported concerns with the ability of advocacy efforts to sustain citizen buy￾in and influence. In many locations, CSOs show up and hold events to increase community awareness on CSDF or education issues but struggle to engage with citizens, who in turn struggle with their own poverty. CSOs suggest that through livelihoods and income-generating activities, women, men, and youth might be more likely to engage in advocacy efforts more consistently. The balance between family economic livelihoods and how citizens exercise rights is a critical component for sustainable advocacy strategies. Media revenue generation and finances. Overall, KIIs described “sustainability” within USAID activities as connected to the ability to secure organizational financing. This is particularly emphasized with media outlets where LMD efforts toward sustainability have concentrated on assisting stations with the development of plans to create side businesses which, while not associated with media operations, might generate potential income. However, the media outlets visited during the evaluation had made little progress toward creation of these side businesses. Many ideas from stations visited were purely aspirational, others more limited, but all unrealized. The ambitious plans required significant capital investment—renovation of office space, purchase of office equipment, purchase or rental of land, hiring of additional staff—which are currently impossible when stations struggle to find the minimal revenue necessary simply to cover basic daily expenses and maintain broadcast operations. Media Outlet Management Capacity Assessments found that some radio stations keep paper financial ledgers, and many have opened bank accounts to receive payments, but most have not started to use computer accounting applications. Business plans, as mentioned above, are primarily focused on describing aspirational side businesses and include station budgets that are not rooted in the realistic description of day-to-day expenses. While most stations interviewed have assigned at least one person to collect money for announcements and dedications, there are no plans for generating airtime sales revenue with targets and potential sponsorship opportunities. Stations often wait in hope that international non-governmental organizations (NGOs) will approach the outlet to purchase airtime for broadcast of programs or public USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 22 service announcements—income that covers the most basic of daily expenses—but none of the stations interviewed have proactive strategies for seeking airtime sales revenue. Media management. More importantly, according to KIIs and the Media Outlet Management Capacity Assessments, basic management systems essential to sustaining operations are rudimentary at best at most media outlets. Station managers, media sector associations, and stakeholders report that media outlets rarely practice basic editorial management functions, including daily editorial meetings with reporters to establish the daily news agenda, newscast and program review sessions, and consistent editorial review of reports. Station program schedules are a mix of call-in programs, discussion programs, a few dedicated newscasts, rebroadcasts of newscasts from the radio stations ECOWAS or ELBC, and a few acquired programs from organizations such as Voice of America (news and feature programs), Talking Drum Studio (a radio drama program produced by Search for Common Ground), or other international NGOs using formats that stations do not have internal finances or skills to produce themselves. Audience targeting. Most stations have little understanding of audience targeting, and learning about audience preferences is limited to hearing comments during call-in programs. Radio station schedules consist of a list of individual programs but without descriptions of each program or a logic behind sequencing of programs. The PAD and the LMD activity included a dedicated component to commission media audience research to generate listenership and viewership data. However, the data captured through the GeoPoll SMS￾platform media audience research initiative is primarily unknown to most media outlet managers interviewed during this evaluation. The one station director interviewed who was familiar with the GeoPoll research found little utility to the data, which the director described as not providing the detail necessary to determine target audience listenership of programs within time blocks. Both the PAD and LMD describe the primary value of audience research to be through application in establishing sales and marketing strategies, advertising prices, and rate-cards, as opposed to using audience research as a fundamental management tool to inform development of program schedules, time blocks, and program formats, as well as allocation of station human, technical, and financial resources. KIIs with LMD and partner outlets found that no stations have been introduced to or use the internationally practiced tool of audience focus group research to inform editorial and program decisions. Rare positive examples of self-starting media outlet teams do exist. Voice of Tappita station managers described a recent effort to go door-to-door to ask residents about their listening preferences. A common response from listeners was that the station newscasts often ran overtime, cutting into popular call-in programs that follow the news. As a result, the station is imposing greater time discipline on newscasts. Comments from agricultural workers about music genres played in the early morning led to a change in music format during the breakfast time block. Voice of Tappita once sponsored a local football tournament that generated over 56,000 LD in contributions, but the station has not successfully replicated this approach. Electricity. The single greatest obstacle to the ability of media outlets to sustain broadcast operations is access to an independent and reliable power supply. Many stations that have their own generators often suffer from equipment malfunction, leading to the station going off-air for days or weeks. Inconsistent daily care and maintenance of electrical equipment has led to transmitter malfunctions, again halting broadcast ability for days or weeks. Only one radio station out of nine outlets visited reported no major problems with its power supply, while one had been off air for more than three months. Several LMD-supported media outlets have secured reciprocal agreements with the mobile network operator Orange Liberia, allowing access to cellular tower transmission infrastructure—namely placement of radio antennas on cell towers and access to electrical supply or generator fuel. However, even these beneficial commercial partnerships are subject to change, with one station reporting that Orange plans to move its cell tower to a new location, a second station reporting that the national government may take 23 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV over ownership of the cell tower for government communication services (forcing Orange to move their equipment to a new location), and a third station having recently moved to a location in part due to the possibility of gaining access to Orange infrastructure but finding that the mobile network operator was not willing to connect the station to the Orange generator or allow access to the cell tower. The lack of access to reliable power sources undermines the ability of stations to reach audiences, essentially preventing stations from providing the news and information to audiences necessary to motivate citizens to participate in government decision making processes or advocacy efforts. Stations that go off air for extended periods see revenue from announcements and dedications going to competing media outlets. Stations that have poor equipment care and maintenance practices go off air for extended periods when electricity surges damage transmitter or studio equipment. Those outlets that have reciprocal contracts with Orange to provide electricity supply are prevented from seeking advertising revenue from the competing mobile network operator, Lonestar Cell/Mobile Telephone Networks (MTN), eliminating a key potential airtime sales customer. Conclusions LAVI and LMD have yet to incorporate a more holistic definition of “sustainability” that encompasses all those management functions critical to the ability of an organization to operate effectively. CSOs require effective implementation of strategic planning, project management, human resources, financial management, and revenue-generation planning systems in order to operate beyond LAVI support. Media outlets require effective implementation of editorial management, program scheduling, program formatting, human resources, financial management, and technical maintenance systems, in addition to sales and marketing skills, to continue successful broadcasting after LMD interventions conclude. Understandably within the Liberian context, and taking into account the existing gaps with sustainable management systems, much work remains before CSOs and media outlets can progress toward sustainability. Internalizing these management systems helps address the issue of staff retention. Currently, through training of individual managers or journalists, skills transferred often depart when staff members depart for new opportunities. If management systems are put in place and replicated by CSO or media outlet teams, the systems can continue to function as individual personnel come and go. LAVI training of CSO partners has improved implementation of coalition interventions and has generated progress with upgrades of some internal management systems. However, significant training on developing advocacy strategies and management practices would allow CSOs to be more effective implementing their advocacy and government oversight functions. While LMD has trained individual journalists in subject￾matter reporting and provided some media management training support, media outlets struggle to implement adequate editorial and management practices. Furthermore, stations lack the skills to both acquire audience consumption data and apply that data to making decisions on program schedules, program formats, and internal resource allocation. Reliance on short-term training workshops, periodic mentoring visits, and short-term follow up sessions have yet to help USAID-supported CSOs or media outlets internalize the strong institutional practices, robust management systems, and organizational policies that would allow daily operations and longer￾term advocacy campaign initiatives to continue beyond LAVI and LMD support. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 24 FUTURE STRATEGY DESIGN EQ 4: BASED ON THESE FINDINGS, WHAT ARE THE STRATEGIC PRIORITIES FOR DO 1 TO INCENTIVIZE GOL REFORM THROUGH CIVIL SOCIETY AND MEDIA IN THE NEW CDCS DESIGN? Findings Overarching Accountability System Gaps The current Civil Society and Media portfolio has focused activities primarily on improving skills at CSOs and media outlets, including interventions to extend the reach of both to citizens through local-level advocacy events, community forums, and media reporting. However, critical gaps still exist in making connections among the three key stakeholder groups within governance accountability systems. Currently, CSO coalitions and media outlets struggle to effectively interact with senior political leaders both at the national and county levels. Similarly, CSOs and media outlets have not yet instituted practices to consistently engage with citizens in local communities. As a result, CSO coalitions and media outlets have yet to overcome obstacles to informing and mobilizing citizen participation in government decision making processes and thereby influencing government reforms. A key priority for the new Civil Society and Media PAD would be to create interactive information and activity loops that better connect political leaders to citizens, using CSO coalitions and media outlets as channels through which such information and activities mobilize citizens to engage with their government institutions and increase citizen demand for government reform. KIIs with CSO coalition partners and stakeholders show that through nearly four years, the LAVI activity has primarily supported CSOs to raise awareness on CSDF and Education thematic policy initiatives, and then to advocate for government reforms and policies that encourage greater response to those dedicated priorities. CSO coalition work has not yet adapted strategies to bring both senior government officials and citizens explicitly into the coalition effort. Coalition strategies can better integrate contextual obstacles to successful advocacy, including pervasive poverty throughout local communities, limited citizen understanding of the potential role within civic life, lack of senior government political will, and poor allocation of public resources to local communities. Similarly, KIIs with media outlets and local media support associations showed a lack of editorial and management skills at media outlets needed to effectively and consistently provide audiences with the credible news and information that encourage citizens to actively participate in civic life. Audiences do not yet have daily access to the information that can motivate citizens to take part in events that help influence government decision making and hold institutions accountable for delivery of goods and services. Training Strategy Both LAVI and LMD have relied heavily on the workshop training model, followed by mentoring sessions with individual beneficiaries. LMD subject-matter journalism training is tied to individual reporters Figure 1. Stakeholder Engagement within Accountability Systems 25 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV producing a finite number of articles or packages post-training. Because LMD training support has been more focused on training individual journalists, the impact is limited. Journalists having received training describe returning to their stations unable to continually replicate newly acquired reporting skills due to lack of station financial resources. Senior news editors and station managers retain decision making authority over broadcast content and require editorial management training. In many cases, journalists are left on their own to perform daily newsgathering functions without management establishing news agendas, and, at best, managers have minimal interaction with journalists prior to submission of news reports. Turnover of journalists is expectedly pervasive at radio stations, with LMC trainers and station managers describing journalists that have received training leaving stations in as little as three months. CSO Sector Champions Contributing to weaknesses within the civil society sector, according to USAID, Liberia lacks strong CSO research institutions that can provide stakeholders with quality PEA and policy research products. USAID identified an opportunity where a follow-on to LAVI might include mentoring and shaping key champions of the political adaptive approach integrated across a few existing LAVI CSO partners. Over the long term, organizations identified during the future civil society development activity could advocate for national issues of concern, while at the same time supporting the GOL and international donors with policy analysis needed to open debate and discussion around national priorities. Developed civic research organizations can act as a bridge between CSO advocacy efforts, government priorities, and donor support. EQ 4A: WHAT ARE NEW, APPLICABLE INNOVATIONS AND APPROACHES BEING USED IN OTHER CONTEXTS TO ENGAGE CIVIL SOCIETY AND MEDIA IN GOVERNMENT REFORM? Findings In order to further improve the coalition-building approach post-LAVI, USAID might consider incorporating lessons from the strategy and interventions implemented by the United Kingdom Department for International Development-funded State Accountability and Voice Initiative (SAVI) in Nigeria. SAVI Nigeria utilized a non-grant model where CSOs organize collective action to engage CSOs, government, media outlets, and citizens. SAVI Nigeria-supported CSOs facilitated a partnership approach that establishes home-grown and self-sustaining coalition relationships to influence reforms. SAVI Nigeria staff members in each state (personnel drawn from past experience working in government, at CSOs and media outlets, and in the State House of Assembly) played a hands-on, behind-the-scenes role to facilitate change on issues already acknowledged as priorities by citizens and CSOs. SAVI Nigeria facilitation improved understanding and better working relationships among citizens, CSOs, media outlets, State Houses, and executive government stakeholders around issues of mutual concern and key democratic processes, generating coalitions for change. State governments now engage citizens in certain planning, budgeting, and monitoring processes. Overall, State Houses, civil society groups, and media houses demonstrate increased effectiveness as agents of citizen change in all SAVI Nigeria target states.5 As described above, the LMD-supported media outlets have meager technical and financial resources, compounded by limited reporting and production skills. A successful method to improve ability of media outlets to engage audiences is to provide dedicated packages of support for creation of new programs for broadcast. Packaged support would include a combination of (1) two- to four-week on-site technical assistance and training from senior radio/television professionals, (2) financial support for episode and series production costs, and (3) targeted equipment (where necessary). Other media development activities have included dedicated assistance packages for creation of station newscasts and current affairs programs. In Macedonia, the USAID Macedonia Support Initiative provided on-site training, production support, and equipment to leading television outlets in order to improve news 5 For additional information on SAVI Nigeria, see http://savi-nigeria.org/results/. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 26 and current affairs programs. The resulting productions set the stations apart from competitors and drew new and increased audiences to improved programs. TV SITEL launched a news-magazine program promoting cross-ethnic group cooperation in an effort to address historic community tensions; similar news-magazine format programs can be produced for radio regarding government-focused topics. TV SITEL used USAID-bundled training and equipment support to reengineer their news operation by instituting new management practices: structured daily editorial management meetings, production rehearsals, technical quality standards, improved news package production, and disciplined newscast formatting.6 Another innovative approach to engaging citizens is to support production of alternative-format programs for radio and television. In Nigeria, BBC Media Action supported the creation of “Story Story: Voices from the Market,” a multi-actor scripted radio drama based in an open city marketplace that focused on characters with storylines incorporating timely political, social, and economic issues, including electoral fraud, natural resource management, and government corruption. The program was centrally produced and distributed to radio stations throughout Nigeria and garnered millions of listeners during its run.7 Alternative-format productions can also include programs that target specific audiences. The Nigeria￾based news satire program “The Other News” (supported by the Open Society Initiative for West Africa and produced locally by Pilot Media Initiatives for Channels Media Group) brought the famous Nigerian comic actor Okey Bakassi to present a weekly political satire newscast. Produced for the youth audience, “The Other News” introduced audiences not normally attracted by news programming to comedy-themed current affairs topics facing the country, particularly young-adult females. Similar news satire programs can be produced for either radio or television; a radio version could be centrally produced and distributed for broadcast over community radio stations in multiple counties.8 Conclusions A future Civil Society and Media PAD can build on the successes of current USAID activities. The LAVI coalition approach can take advantage of the recognized increasingly open space in which Liberian CSOs operate. The future activity can deepen strategic engagement with existing CSDF and Education coalitions, incorporating research and input from district-level communities through local CSOs. Government actors in all interviews expressed the need for CSOs to support reform agendas. Based on existing relationships from LAVI interventions, the CSDF and Education coalitions can extend their reach upward to incorporate additional senior government leaders (legislators and ministry officials) as part of follow-on CSDF and Education coalition activities. RECOMMENDATIONS Quantitative and qualitative research should be prioritized within both USAID activities. All research proposed by IPs should be first and foremost designed to be utilized as tools to inform CSO and media outlet managers. Secondly, the same research should help inform the IPs when reviewing thematic interventions. Finally, the same research should assist USAID to monitor and evaluate the progress of Civil Society and Media Portfolio activities. 6 For additional information on the USAID Macedonia Support Initiative program, see “Macedonia Support Initiative (MSI) Final Report” at https://pdf.usaid.gov/pdf_docs/PA00TJ3M.pdf. 7 For additional information on Story Story: Voices from the Market, see “BBC Media Action’s governance research: emerging evidence and learning,” pp. 24–25 at http://bit.do/ePJQn and “The story of Story Story: 13 years of drama making a difference in Nigeria” at http://bit.do/ePJRF. 8 For additional information on The Other News, see “Using Comedy to Strengthen Nigeria’s Democracy” at https://www.newyorker.com/magazine/2018/01/22/using-comedy-to-strengthen-nigerias-democracy. 27 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV Accordingly, the new civil society development activity should mandate use of PEAs during the first year of the activity in order to increase and understand prospects for passage of key policy and legislative changes. PEAs should identify opportunities for engaging with legislators and senior ministry officials as efforts to advocate for legislative reforms and successful implementation of reforms passed through approved laws and regulations. In addition, PEAs that address county-level issues would provide additional evidence of community priorities and inform interventions that would close the gaps between citizens and their government institutions. Research should also inform Advisory Board deliberations regarding adoption of new thematic windows. Similarly, the media development IP should operationalize the use of media focus group research to complement quantitative audience consumption data and inform media managers regarding the reasons behind perceptions of station content and performance. USAID should seek to build capacity at the best positioned CSOs and media outlets. By increasing technical assistance to fewer direct CSO and media outlet partners, the new activities will have a greater opportunity to improve systems at leading organizations and set standards for “modeling behavior” within the CSO and media sectors. The new civil society development activity can target increased training and GUC support to a more focused group of national-level CSOs from the outset of Year 1. Using revised media outlet selection criteria that balance analysis of political influence at stations with quantitative audience consumption research, the new media development activity should reduce the number of stations supported to those with existing audiences in USAID-selected locations, including Monrovia. Through this selection process, USAID-supported stations will have a greater likelihood of delivering improved news and information to greater target audiences. USAID might consider commissioning nationwide audience research using a face-to-face interview methodology and a minimum 2,500 respondent sample (rather than the existing LMD-supported SMS survey platform) in order to generate data at the county level to inform selection of stations for the next media development activity. Both future activities should improve the use of management capacity assessments as management tools. Current CSO capacity assessments under LAVI have provided some information to CSO managers as to where system improvements can be made, and the participatory assessment process had some value for CSO managers. Media outlet managers reported little to no utility from the capacity assessments conducted by LMC under LMD. Both the new civil society and media development IPs should include CSO and media outlet managers in the development of assessment methodologies in order to increase buy-in for the assessment process and post-assessment utility. USAID should require year-on-year assessments for all core-supported CSOs and media outlets so that managers can better use assessment findings while USAID uses the capacity assessments to inform activity evaluation efforts. USAID should continue to incorporate the problem-driven iterative adaptive programming model launched under LAVI for the future CSO development activity. Based on current LAVI interventions, coalitions can utilize strategic learning events with government and local community stakeholders as a method of incorporating interactive learning to make informed and evidence-based improvements during advocacy campaigns. The iterative approach to applying feedback loops, reviews, and strategy testing can improve CSO coalition ability to make mid-course corrections with new or adjusted interventions. Through the adaptive model, the new activity can take advantage of the experience of coordinated activities by preexisting CSDF and Education coalitions. USAID should include a dedicated component to provide solar-power electricity generation units to core supported media outlets under the new media development activity. With countrywide access to reliable and independent electricity sources representing the most critical obstacle to broadcast operations—and therefore media outlet sustainability—provision of solar power units to USAID￾supported radio stations would help guarantee that stations remain on air, providing news and information to local target audiences. Support for purchase and installation of solar power units should be combined with extensive and repeated training on daily care and maintenance in order to improve continuous USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 28 operations. Stations would be able to maintain connections to own generators or continue reciprocal partnerships with mobile network operator infrastructure in order to retain backup power capacity. UTILIZATION PLAN The results of this evaluation and its recommendations should be used to help USAID in 2019 as it begins design of a new Civil Society and Media PAD within the forthcoming 2020–2025 CDCS. When crafting the solicitation for both the new civil society development activity and the new media development activity, USAID could include case studies that require offerors to apply the activity approaches included in each respective USAID solicitation design within a Liberia-like context. The case study would help USAID understand how prospective offerors understand the intentional USAID design and require offerors to include a roster of expert broadcast media or CSO advocacy trainers that would be engaged as part of their response to the case study objectives. 29 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV ANNEX A: DESK REVIEW DOCUMENTS UTILIZED DAI/LAVI. 2016. FY2016 Workplan Final. Monrovia: USAID/LAVI. DAI/LAVI. 2017. FY2017 Workplan Final. Monrovia: USAID/DAI. DAI/LAVI. 2017. FY2018 Workplan Final. Monrovia: USAID/DAI. DAI/LAVI. 2018. FY2019 Workplan Final. Monrovia: USAID/DAI. DAI/LAVI. 2018. LAVI Updated AMELP 2018-03-26.pdf. Monrovia: UDAID/DAI. Governance, U. o. 2015. USAID 1.4 CSM PAD.pdf. Monrovia: USAID/Liberia Civil Society and Media. Impact/LSA, S. 2017. LMD_DQA Report_Final.pdf. Monrovia: USAID/Liberia Strategic Analysis/LMD. Internews/LMD. 2015. LMD Year 1 Activity Monitoring Evaluation Learning Plan_REVISED_SUBMIT.docx. Monrovia: USAID/LMD. Internews/LMD. 2015. LMD Year 1 Implementation Plan_REVISED.docx. Monrovia: USAID/LMD. Internews/LMD. 2016. LMD Year 2 Implementation Plan_Final Version_22Dec2016_CLEAN.docx. Monrovia: USAID/LMD. Internews/LMD. 2016. LMD Year 3 Implementation Plan_25 OCT 2017.docx. Monrovia: USAID/LMD. Internews/LMD. 2017. LMD Year 2 Annual Report.pdf. Monrovia: USAID/LMD. Internews/LMD. 2018. LMD Year 3 Annual Report.pdf. Monrovia: USAID/LMD. Internews/LMD. 2018. LMD Year 4 Implementation Plan_31OCT2018. Monrovia: USAID/LMD. IREX. 2016. CSML Final Life of Project Report for USAID.pdf. Monrovia: USAID/CSML. IREX/CSML. n.d. CSML_PMP USAID APPROVED.docx. IREX. Liberia Institute of Statistics and Geo-Information Services (LISGIS). August 2017. Household Income and Expenditure Survey 2016. LISGIS. USAID/CSML. n.d. Cooperative Agreement; CSML_PMP USAID APPROVED.docx. IREX. USAID/LAVI. 2017. LAVI CSDF PEA Final Report.pdf. Monrovia: USAID/LAVI. USAID/Liberia. 2013. USAID Liberia CDCS 2013-17.pdf. USAID/Liberia. USAID/Liberia. 2015. AID-669-A-15-00003-09182015-JSchramm_signed.pdf. Washington DC: USAID/Liberia/LMD. USAID/Liberia. 2015. AID-669-C-16-00003 11.25.2015.pdf. Monrovia: USAID/LAVI. USAID/Liberia. 2016. LAVI FY2016 Annual Report Revised Final. Monrovia: USAID/LAVI. USAID/Liberia. 2017. LAVI FY2017 Annual Progress Report. Monrovia: USAID/LAVI. USAID/Liberia. 2017. LAVI MOD 04 - SIGNED AND EXECUTED - RDCH 11162017.pdf. Monrovia: USAID/DAI Global LLC/LAVI. USAID/Liberia. 2017. USAID Meta-Analysis Final Report.pdf. Monrovia: USAID/Liberia. USAID/Liberia. 2018. LAVI FY2018 Annual Report. Monrovia: USAID/LAVI. USAID/Liberia. 2018. LAVI MOD 05 - SIGNED AND EXECUTED - RDCH - 02052018.pdf. Monrovia: USAID/LAVI. USAID/Liberia. 2018. LAVI Mod 06.pdf. Monrovia: USAID/DAI Global LLC/LAVI. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 30 USAID/Liberia. 2018. LAVI Mod 07.pdf. Monrovia: USAID/DAI Global LLC/LAVI. USAID/Liberia. n.d. LAVI - Mod 03 .pdf. Monrovia: USAID/DAI Global LLC/LAVI. USAID/LMD. 2016. LMD Information and Assessment Report Feb 2016.pdf. Monrovia: USAID/LMD. USAID/LMD. 2016. LMD Year 1 Annual Report.pdf. Monrovia: USAID/LMD/Internews. USAID/LMD. 2017. LMBMA Media Content Baseline Assessment Narrative_8Dec2017.pdf. Monrovia: USAID/LMD. USAID/LMD. 2018. LMD Year 4 Activity Monitoring Evaluation Learning Plan.docx.pdf. Monrovia: USAID/LMD. USAID/L-MEP. 2012. CSML Mid-Term Evaluation Final Report (October 2012) .pdf. Monrovia: USAID/Liberia/CSML. USAID/LSA. n.d. IPs Consolidated Performance Indicator Data.xls. Monrovia: USAID/LSA/CSML/LAVI/LMD. USAID/Macedonia. 2018. Macedonia Support Initiative (MSI) Final Report. Https://pdf.usaid.gov/pdf_docs/PA00TJ3M.pdf. 31 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV ANNEX B: ILLUSTRATIVE KEY INFORMANT INTERVIEW QUESTIONS Introduction: Thank you for taking the time to meet with us today. We are researchers from Social Impact (SI) and we have been contracted by USAID to evaluate the LAVI/LMD activity, an initiative funded by USAID and implemented by DAI/Internews. We will ask you questions that will inform our analysis of whether LAVI/LMD is meeting its activity-level objectives contributed to the broader objectives aimed at strengthening civil society and media’s ability to exercise their advocacy and oversight roles. Today’s interview is expected to last approximately 1 hour(s). Risks and benefits: We do not anticipate that you will incur any risk or direct benefit from participating in this interview. Confidentiality: Your responses in this interview will be kept in confidence by the evaluation team. Only the evaluation team will have access to the notes that are taken. Your name or position will not appear in any reporting. Voluntary participation: Your participation is voluntary. If you do not want to participate or to answer specific questions you do not have to. Should you choose to participate, please know that you may change your mind at any point during our discussion. There will be no consequences to your decision not to participate. Report: We expect to speak with approximately 70 people as part of this review. The information that you and others provide will be used to write report. This report will be shared with USAID and other stakeholders for comment and eventually be made public. [Note: Only if permission is sought to record should the italicized text be read] Recording: To ensure the accuracy of our notes, we request your permission to record this interview. Recordings will be uploaded to a protected server and then deleted from the device on which they were recorded. You may choose not to have this interview recorded but still proceed with this interview. Do you have any questions for us before we get started? Are you willing to participate in this interview? KII – ______ KII ID DATE/TIME: COUNTY/INTERVIEW LOCATION: RESPONDENT NAME: RESPONDENT GENDER: RESPONDENT ORGANIZATION/TITLE: INTERVIEWER NAME: NOTETAKER NAME: Permitted to Record THE INTERVIEW? USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 32 USAID • Describe how USAID is working/has worked within the media and civil society environment in Liberia? How did this programming fit into the overall USAID strategy for Liberia? • What results have you seen from these efforts? • What factors have contributed to these results? • How did the media and civil society strategy evolve over the past four years? What events or other influences led to those strategy changes? • How did your projects measure the output and impact of these results? • How effective do you believe LAVI and LMD have been in increasing Liberian citizen influence over governance of public goods and services? Can you provide examples of increased engagement? • How have your projects attempted to include women as participants and beneficiaries? • How did your projects measure the ability of local beneficiary partners to sustain activities after conclusion of your assistance interventions? • What have been the biggest implementation challenges to civil society and media development programming in Liberia over the past four years? How were they addressed/overcome? • What lessons/best practices have you learned about programming in the Liberian context? • Were there any unintended consequences of programming, positive or negative? Implementing Partner Staff – DAI/Internews • What are your main roles and responsibilities? − Within your organization? − On which activities? • How are your interventions/activities designed? • How did you select local partner organizations (CSOs, Service Providers, Media Outlets)? • How useful were baseline capacity assessments conducted for the local partner organizations? Can you give examples of how assessment data changed or created new interventions? • Please describe any training/mentoring activities implemented during the project(s), including curriculum, trainers, and follow up? − How are training topics selected? − How are training sessions designed? − How are individual participants selected? − Is baseline data collected from participants regarding specific skills/knowledge of training topic? − How are impact and results of training activity measured? − What type of follow-up is conducted with participants after training interventions? • What other forms of capacity-building assistance has been provided to beneficiaries/local partners? • How have local partner organizations (CSOs/Service Providers/Media Outlets) used the skills and knowledge transferred during training in their work? What evidence do you have? • How did you incorporate women into project activities as beneficiaries? • How did your projects utilize quantitative or qualitative survey research during the projects? • What evidence do you have of local partner organizations (CSOs/Service Providers/Media Outlets) leading to changed engagement with government/public institutions or changed popular attitudes towards national or local government institutions? • How did your projects measure the ability of local beneficiary partners to sustain activities after conclusion of your assistance interventions? • Can you give examples of new or altered interventions based on events or contextual changes at the local or national level? 33 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV • Have there been any unintended effects of LAVI/LMD activities? • What obstacles did you face with implementation of project activities? How did those obstacles affect ability to deliver project assistance? Local Partner Organizations (LAVI/LMD Subcontractors, Core-Funded CSOs and Service Providers) • Describe how your project activities are working/have worked within the civil society/media environment in Liberia? • What specific project activities did you support? What was your specific role during those activities? • How did these activities contribute to overall LAVI/LMD program objectives? • What results have you seen from these efforts? • What factors have contributed to these results? • How did your projects measure the output and impact of these results? • How are your programs managed? How could operational management be improved? • (For training activities) How were results of training activity measured? • Did you conduct any quantitative or qualitative research to measure change in community behaviors? • How have your projects attempted to include women as participants and beneficiaries? • How did your projects measure the ability of local beneficiaries to sustain operations after conclusion of your assistance interventions? • What are the major challenges for building capacity of Liberian beneficiaries? • What are the biggest implementation challenges to programming in Liberia? How were they addressed/overcome? • What lessons/best practices have you learned about programming in the Liberia context? Liberian CSO Beneficiaries • Please describe the activities (campaigns/events) implemented through support from LAVI • Please describe the support you received from LAVI (training, mentoring, equipment, financial) • What is/was your organizations involvement in the topics selected for training and support, design of training activities, and selection of individual participants (where relevant)? • Do you (or your staff) use the skills, knowledge and tools provided by the organization and trainers? If so, can you provide some examples? • Has your involvement with and support from LAVI impacted your organization? If so, how? Can you provide specific examples? • Does your organization continue to organize advocacy events/campaigns developed through training/support from LAVI? Please describe. • How does LAVI follow-up with your organization regarding impact of specific training/assistance? • Would you be able to continue advocacy activities without ongoing LAVI support? • Are you satisfied with the quality and quantity of technical support provided? Why or why not? • What additional support does your organization need? • How do you address gender issues in your organization and programming? • How would you describe your role regarding engagement with government officials and institutions? USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 34 Liberian Media Outlets • Please describe the types of programs that you produce and broadcast through support from LMD • Please describe the different types of support you received from LMD (training, mentoring, equipment, financial) • What is/was your involvement in the topics selected for training and support, design of training activities, and selection of individual participants (where relevant)? • Do you (or your staff) use the skills, knowledge and tools provided by the organization and trainers? If so, can you provide some examples? • Has your involvement with and support from LMD impacted your organization? If so, how? Can you provide specific examples? • Does your organization continue to produce and broadcast programs, or use production systems developed through training/support from LMD? Please describe. • How does LMD follow-up with your organization regarding impact of specific training/assistance? • Would you be able to continue productions without ongoing LMD support? • Are you satisfied with the quality and quantity of technical support provided? Why or why not? • What additional support does your organization need? • How do you address gender issues in your organization and programming? • How would you describe your role regarding engagement with government officials and institutions? • Do you have any programs that address national level political issues? Please describe the format and content. • Do you have programs that address local or municipal issues? Please describe the format and content. 35 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV ANNEX C: SAMPLE FOCUS GROUP DISCUSSION PROTOCOLS Focus Group Discussion research focusing on media will provide background reasoning behind audience perceptions and attitudes of the Liberian citizens toward their local radio stations. The data generated by focus group sessions will provide evidence-based and specific qualitative insights to inform media management decisions regarding media programming and resource allocation and contribute to analysis provided to USAID/Liberia regarding the performance of media outlets supported by LMD interventions. Focus Group Discussion sessions that focus on civil society participation will provide background reasoning behind perceptions and attitudes of Liberian citizens toward advocacy efforts that have been implemented in their communities. The data generated by these focus group sessions will provide evidence-based insights that can inform CSO management decisions regarding organization of campaign and other governance-related events and will contributed to the analysis provided to USAID/Liberia regarding the performance of CSOs supported by LAVI interventions. In addition, research data collected can in the future assist trainers and consultants as they advise media and CSO managers on the development of editorial content, program scheduling, campaign design and advocacy efforts that adapt to the needs and demands of audiences and local communities. The focus group discussion data will help media professionals and civil society activists better understand specific attitudes of citizens regarding how programs and campaigns are perceived in terms of satisfaction and trust. The focus group data will allow us to understand areas of strength and areas of perceived negativity and underperformance. In addition, the data will provide essential information which can be used to measure against potential future research to gauge the developing impact of overall program perception. Proposed FGD Model and Schedule Number of FGD Sessions: 6 (Three Radio FGDs / Three Citizen Participation FGDs) Sessions should generate a rounded gender and age-balanced perspective and provide a differentiation between audience perceptions urban centers and rural areas. Focus Group Sample Size: 8 participants Sample Demographics: Gender: Three Female-only Groups; Three Male-only Groups Age: 18-24 Years of Age (1 Female Radio Session / 1 Male Citizen Participation Session) 25-34 Years of Age (1 Male Radio Session / 1 Female Citizen Participation Session) 35-55 Years of Age (1 Male Radio Session / 1 Female Citizen Participation Session) The purpose of the focus group discussion should be explained to the participants at the outset, presented as an opportunity for the participants to influence program content development (for media), and inform options for interactions with local governance institutions in the future. Moderators should assist with the free flow of conversation amongst session participants, while not diverting from the subject of each specific question. The research should generate an English-Language Summary Report from each focus group discussion session, summarizing responses for each question, followed by 3-5 quotes from participants for each question. RADIO FOCUS GROUP RESEARCH MODERATOR GUIDE USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 36 Instructions to Focus Group Participants: Welcome. We have asked you to participate in this focus group discussion to share your perceptions about radio programming in Liberia. The information and opinions that you share with us today will influence the type of programs that you may see in the future on domestic radio stations and inform the content of programs available to audiences in Liberia. All your comments will remain anonymous. Which radio stations do you listen to most? Describe specifically what you like most about the radio stations to which you listen most often? (Ensure each participant answers both parts of the question, probe for additional detail) Name the one radio station you would describe as your FAVOURITE radio station. Why is that station your favourite? [PROBE RESPONDENT] Which of the following types of radio programs do you MOST LIKE to listen to each day, and which do you LEAST LIKE to listen to each day? THE LIST BELOW SHOULD BE WRITTEN IN ADVANCE ON A WHITE BOARD OR FLIPCHART EACH PARTICIPANT SHOULD BE ASKED TO NAME THE “MOST LIKED” THREE CATEGORIES OF PROGRAMS, THEN THEIR “LEAST LIKED” THREE CATEGORIES OF PROGRAMS (Give the participants a few moments to look at the list, and then determine their most and least liked three types of shows). MODERATOR WILL ASK EACH PARTICIPANT IN THE GROUP FOR VERBAL RESPONSES Music Comedy / Satire Call-in / Talk Shows Current Affairs / Political Discussions Culture Shows Newscasts Drama Serials / Soap Operas Live Sports Coverage / Sports Programs Educational Programs / Documentaries Which radio station do you most trust for accurate and honest news coverage? Why do you trust that radio station news coverage the most? [PROBE RESPONDENT] Which radio station has the best presenters? Can you name any presenters that you specifically like? Describe what you like about them? [PROBE RESPONDENT] Do you feel that any specific radio station’s news is biased in favour of any political party or faction? If yes, give examples of stories you have seen to make you think that the news is biased from that specific radio station, and why you believe those news stories were biased in favour of a specific political party? [PROBE RESPONDENT] Which of the following types of news and information are most important to you when you listen to radio news programs? THE PROGRAM CATEGORIES BELOW SHOULD BE LISTED ON AN A4 SHEET WITH A SEPARATE BOX FOR INDIVIDUAL PARTICIPANT TO PERSONALLY RANK EACH PROGRAM CATEGORY FROM 1 (MOST IMPORTANT) TO 12 (LEAST IMPORTANT) 37 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV Social & Economic (prices of goods and services, items that affect daily living conditions) Crime Political News Sports News International News (news from around the world but affecting Liberia) Corruption Human Interest Stories (stories that show examples of the lives of individual people) Business news Investigative stories Health issues Positive news stories about Liberia Entertainment and Show Business news (Movies, Television, Music, Celebrities, etc.) Which radio station do you believe has the most listeners? Why do you think that station is so popular? [PROBE RESPONDENT] If you were an advisor to the management of your local radio station, what changes would you tell the managers to make that would make you listen to the station more regularly? How would you rate the performance of the local radio stations from your community in the following areas (1 being poor, 5 being excellent)? Each individual should be provided the options below on a separate A4 sheet and rank the areas from 1 to 5. The categories below must be identified and clearly defined for each participant, so we can cross-tab results. 1) Quality of Presenters/Hosts 1 2 3 4 5 2) Trust in News and Information Offered 1 2 3 4 5 3) Selection and Quality of News and Information on Local Stations 1 2 3 4 5 4) Ability to Use Information Offered on News Programs 1 2 3 4 5 5) My Local Radio Station Meets My Needs for News and Information 1 2 3 4 5 USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 38 CITIZEN PARTICIPATION FOCUS GROUP RESEARCH MODERATOR GUIDE Introductions to Focus Group Participants: Welcome. We have asked you to participate in this focus group discussion to share your perceptions about community governance in Liberia. The information and opinions that you share with us today will inform development of activities that support the efforts of citizens to participate in governance in their local communities. All your comments will remain anonymous. Can you name the most influential Civil Society Organization in your community? Describe specifically what you like most about that CSO’s activities? (Ensure each participant answers both parts of the question, probe for additional detail) Have you participated in an event (public meeting, public discussion, demonstration, etc.) organized by that CSO? Why did you take part? [PROBE RESPONDENT] Which local government institutions do you most trust to deliver services to you and your family? Why do you trust that government body the most? [PROBE RESPONDENT] Do you feel that any specific CSO is biased in favour of any political party or faction? If yes, give examples of something have seen to make you think that the CSO is biased, and why you believe that CSO is biased in favour of a specific political party? [PROBE RESPONDENT] Have you attended a meeting of the CSDF? Why did you decided to attend? Would you consider attending a CSDF meeting again? Why? [PROBE RESPONDENT] Have you attended a community school monitoring group meeting Why did you decide to attend? Would you consider attending a community school monitoring group meeting again? Why? [PROBE RESPONDENT] How would you rate the performance of your CSDF in the following areas (1 being poor, 5 being excellent)? Each individual should be provided the options below on a separate A4 sheet and rank the areas from 1 to 5. The categories below must be identified and clearly defined for each participant in order to cross-tab results. 1) Quality of CSDF Members 1 2 3 4 5 2) Trust in Information Offered at the CSDF Meeting 1 2 3 4 5 3) Selection and Quality of Projects 1 2 3 4 5 4) Ability to Use Information Offered by the CSDF Members 1 2 3 4 5 5) CSDF Projects Address My Family’s Needs 1 2 3 4 5 39 | CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION USAID.GOV Consent Form Focus Group Discussion – Radio Introduction: Thank you for taking the time to participate in our focus group discussion. We are researchers from Social Impact (SI) and we will ask you questions about your perceptions on radio programming that will inform our analysis about radio in Liberia. The information and opinions that you share with us today will influence the type of programs that you may hear in the future on domestic radio stations and inform the content of programs available to audiences in Liberia. All your comments will remain anonymous. Today’s discussion is expected to last approximately 2 hour(s). Risks and benefits: We do not anticipate that you will incur any risk or direct benefit from participating in this interview. Confidentiality: Your responses in this interview will be kept in confidence by the evaluation team. Only the evaluation team will have access to the notes that are taken. Your name or position will not appear in any reporting. Voluntary participation: Your participation is voluntary. If you do not want to participate or to answer specific questions you do not have to. Should you choose to participate, please know that you may change your mind at any point during our discussion. There will be no consequences to your decision not to participate. Recording: To ensure the accuracy of our notes, we request your permission to record this interview. Recordings will be uploaded to a protected server and then deleted from the device on which they were recorded. You may choose not to have this interview recorded but still proceed with this interview. Date/Time: Location: Respondent Name: Respondent Gender: Do you have any questions for us before we get started? Are you willing to participate in this discussion? Yes_______ No_______ Signature: ______________________________ USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 40 Consent Form Focus Group Discussions – Citizen Participation Introduction: Thank you for taking the time to participate in our focus group discussion. We are researchers from Social Impact (SI) and we will ask you questions about your perceptions regarding community governance in Liberia. The information and opinions that you share with us today will inform development of future activities that support the efforts of citizens to participate in governance in their local communities. All your comments will remain anonymous. Today’s discussion is expected to last approximately 2 hour(s). Risks and benefits: We do not anticipate that you will incur any risk or direct benefit from participating in this interview. Confidentiality: Your responses in this interview will be kept in confidence by the evaluation team. Only the evaluation team will have access to the notes that are taken. Your name or position will not appear in any reporting. Voluntary participation: Your participation is voluntary. If you do not want to participate or to answer specific questions you do not have to. Should you choose to participate, please know that you may change your mind at any point during our discussion. There will be no consequences to your decision not to participate. Recording: To ensure the accuracy of our notes, we request your permission to record this interview. Recordings will be uploaded to a protected server and then deleted from the device on which they were recorded. You may choose not to have this interview recorded but still proceed with this interview. Date/Time: Location: Respondent Name: Respondent Gender: Do you have any questions for us before we get started? Are you willing to participate in this discussion? Yes_______ No_______ Signature: ______________________________ ANNEX D: MANAGEMENT CAPACITY ASSESSMENT MATRIX CSO Management and Financial Viability Indicators CSO Management Indicator Level 1 Level 2 Level 3 Level 4 1 CSO has a clear and effective mission statement. The vision and/or mission is not a clearly stated description of what the organization aspires to achieve or become. No formal vision and/or mission statement is in place. Although the vision and/or mission of the organization is generally understood it is rarely used as a foundation for setting priorities or decision-making. A clear and specific mission statement is in place, but staff are not aware of organizational objectives. It is sometimes cited as justification for business decisions. The mission statement is a clear and specific proclamation of what the organization wants to achieve and is consistently used as foundation for strategic planning and operational decisions. Baseline: Level __ Justification: 2 CSO has established and implemented an effective strategic / business plan. No written plan in place. A vague 1 to 2-year outline exists but is not easily understood without additional explanation. The plan may contain goals and objectives but not timelines, detailed responsibilities or linkages to the budget. A basic plan of goals, objectives, timelines and responsibilities exists but is not shared with all staff, is not always consistent with the budget and does not undergo periodic evaluations and updates. A 1 to 2-year operational plan includes clearly stated goals, measurable objectives, timelines and responsibilities. It is shared with the entire staff, is consistent with the budget and is regularly evaluated and updated. Baseline: Level __ Justification: USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 42 CSO Management Indicator Level 1 Level 2 Level 3 Level 4 3 CSO is able to meet its financial obligations. Income in the prior 12 months covered up to 30% of expenditures (excluding capital expenses Income in the prior 12 months covered up to 50% of expenditures (excluding capital expenses). Income in the prior 12 months covered up 75% of expenditures (excluding capital expenses). Income in the prior 12 months covered all expenditures (excluding capital expenses). Baseline: Level __ Justification: 4 CSO maintains a complete and accurate budget that reflects all company expenses and revenues. The organization does not have an established budget and there are no financial planning procedures although simple budgets for some activities may be in place. Spending is based on available cash and no core-cost budget is in place. Periodic efforts to establish an organizational budget have been undertaken but there is no systematic process to develop detailed annual budgets which are aligned with strategic and operational plans. Individual activity budgets do not account for core costs or reflect the organization’s overall expenses and revenue. An effective budgeting process accounts for most anticipated expenses and revenue, is generally consistent with the strategic plan but is not strictly followed or evaluated. Activity budgets address overhead expenses, may not ensure full cost recovery and are not always evaluated with correct documentation or methodology. All anticipated incomes and expenses are compiled into a complete and appropriate master budget document according to purpose and category of expense. This document is routinely updated and is used by management for the purposes of planning, cost control and cash flow. A core-cost budget is aligned with the strategic plan and regularly reviewed. Each activity budget includes a share of overhead and office expense based on consistent methodology that is documented and ensures full recovery of costs. Baseline: Level __ Justification: CSO Management Indicator Level 1 Level 2 Level 3 Level 4 5 CSO maintains written human resources and financial management policies that are implemented by dedicated staff with appropriate skills. Without designated managers for human resources and finance, duties are performed ad hoc by other staff who have no specific expertise in these areas. Although HR and financial management duties are assigned to specific staff members as secondary duties, they have no specific expertise and there are no policies or methodologies to guide them. HR and financial management functions are assigned to specific managers or staff. Some have special expertise in the field but mainly perform other functions. Some policies and procedures are established. HR and financial management duties are assigned to specific staff who have expertise in the field and have policies and procedures to guide their work. Baseline: Level __ Justification: Fundraising, Revenue Generation, and Marketing Indicator Level 1 Level 2 Level 3 Level 4 1 Revenue generation and fundraising goals and strategies are developed and implemented. The organization has set neither revenue goals nor strategies on how to increase advertising income. Although the organization occasionally identifies possible sources of new revenue, effective and enduring goals and strategies are neither formulated nor communicated effectively to staff. The organization has identified a potential source of revenue from at least one source, formulated and implemented strategy and established goals which are effectively communicated to staff. The organization has developed sales goals, identified potential new revenue from multiple sources, and formulated and implemented effective strategies. Goals and strategies are communicated to staff and are periodically reviewed to determine their effectiveness. USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 44 Baseline: Level __ Justification: 2 Capable fundraising staff employ best professional practices. No professionally capable fundraising personnel are on the staff. fundraising activities are done on an ad hoc basis by unqualified employees. Although fundraising duties are assigned to specific staff as secondary duties, they have no specific expertise and there are no policies or methodologies. Fundraising functions are assigned to specific managers or staff who have some experience but who also perform other functions. Some policies and procedures are set. Fundraising duties are assigned to dedicated staff who have expertise in the field. Clearly defined goals, policies and procedures guide their work. Baseline: Level __ Justification: 3 Citizen participation research is conducted, with results informing management decisions. The organization neither conducts nor uses research in planning or decision￾making. Although the organization attempts to conduct basic research on participation behaviors, management decisions are not relying on and do not reflect outcomes of the market research. The organization plans and systematically conducts research on citizen participation in advocacy initiatives, but management is not fully aware of how to use data. Research on citizen participation in advocacy initiatives is conducted systematically, and is routinely used in decision making. Baseline: Level __ Justification: 4 Strategic planning and decision-making are based on an identified and clearly defined target audience. The organization is unable to identify a specific target audience. The organization has a general idea about who they are trying to reach but the target audience is not well-defined and is not considered in The organization has defined a target audience. Strategic planning and decision￾making do not fully consider The target audience is well￾defined and is the foundation for effective strategic planning and decision making. The staff is fully informed and trained on how to focus their work on meeting the planning or decision￾making. the target audience profile and the staff is not informed. needs and interests of the target audience. Baseline: Level __ Justification: 5 Effective awareness and recruitment campaigns focus on a defined target audience. The organization is not working to increase awareness and volunteerism among target audiences. The organization has plans and discussions on the ways to increase awareness of their issues and activities among target audiences, but no tangible activities are undertaken. The organization has done at least a couple of awareness campaigns to increase awareness of their activities. The campaign lacks focus, and its impact is minimal or not measured. The organization is working on an awareness campaign in a systematic manner, effects are measured, and some results are achieved. Baseline: Level __ Justification: 6 Decisions about advocacy initiatives are consistent with the interests of the target audience. Decisions about advocacy events and campaigns are not based on the needs and interests of the target audience. The organization uses internal resources such as staff input to determine how events and campaigns can focus more effectively on target audience interests. The organization uses external resources such as input through websites, social media, focus groups and research to determine the interests of target audiences and makes decisions on events and campaigns accordingly. The organization expands its collection of external inputs on target audience interests, feedback on past events and campaigns, and creates an institutional culture that prioritizes meeting target audience needs so that all decisions are driven by community preferences and demands. Baseline: Level __ Justification: USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 46 Media Management and Financial Viability Indicators Media Management Indicator Level 1 Level 2 Level 3 Level 4 1 Media outlet has a clear and effective mission statement. The vision and/or mission is not a clearly stated description of what the organization aspires to achieve or become. No formal vision and/or mission statement is in place. Although the vision and/or mission of the organization is generally understood it is rarely used as a foundation for setting priorities or decision-making. A clear and specific mission statement is in place, but staff are not aware of organizational objectives. It is sometimes cited as justification for business decisions. The mission statement is a clear and specific proclamation of what the organization wants to achieve and is consistently used as foundation for strategic planning and operational decisions. Baseline: Level __ Justification: 2 Media outlet has established and implemented an effective strategic / business plan. No written plan in place. A vague 1 to 2-year outline exists but is not easily understood without additional explanation. The plan may contain goals and objectives but not timelines, detailed responsibilities or linkages to the budget. A basic plan of goals, objectives, timelines and responsibilities exists but is not shared with all staff, is not always consistent with the budget and does not undergo periodic evaluations and updates. A 1 to 2-year operational plan includes clearly stated goals, measurable objectives, timelines and responsibilities. It is shared with the entire staff, is consistent with the budget and is regularly evaluated and updated. Baseline: Level __ Justification: Media Management Indicator Level 1 Level 2 Level 3 Level 4 3 Media outlet is able to meet its financial obligations. Income in the prior 12 months covered up to 30% of expenditures (excluding capital expenses Income in the prior 12 months covered up to 50% of expenditures (excluding capital expenses). Income in the prior 12 months covered up 75% of expenditures (excluding capital expenses). Income in the prior 12 months covered all expenditures (excluding capital expenses). Baseline: Level __ Justification: 4 Media Outlet maintains a complete and accurate budget that reflects all company expenses and revenues. The organization does not have an established budget and there are no financial planning procedures although simple budgets for some activities may be in place. Spending is based on available cash and no core-cost budget is in place. Periodic efforts to establish an organizational budget have been undertaken but there is no systematic process to develop detailed annual budgets which are aligned with strategic and operational plans. Individual activity budgets do not account for core costs or reflect the organization’s overall expenses and revenue. An effective budgeting process accounts for most anticipated expenses and revenue, is generally consistent with the strategic plan but is not strictly followed or evaluated. Activity budgets address overhead expenses, may not ensure full cost recovery and are not always evaluated with correct documentation or methodology. All anticipated incomes and expenses are compiled into a complete and appropriate master budget document according to purpose and category of expense. This document is routinely updated and is used by management for the purposes of planning, cost control and cash flow. A core-cost budget is aligned with the strategic plan and regularly reviewed. Each activity budget includes a share of overhead and office expense based on consistent methodology that is documented and ensures full recovery of costs. Baseline: Level __ Justification: USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 48 Media Management Indicator Level 1 Level 2 Level 3 Level 4 5 Media outlet maintains written human resources and financial management policies that are implemented by dedicated staff with appropriate skills. Without designated managers for human resources and finance, duties are performed ad hoc by other staff who have no specific expertise in these areas. Although HR and financial management duties are assigned to specific staff members as secondary duties, they have no specific expertise and there are no policies or methodologies to guide them. HR and financial management functions are assigned to specific managers or staff. Some have special expertise in the field but mainly perform other functions. Some policies and procedures are established. HR and financial management duties are assigned to specific staff who have expertise in the field and have policies and procedures to guide their work. Baseline: Level __ Justification: Sales and Marketing Indicator Level 1 Level 2 Level 3 Level 4 1 Revenue generation and sales goals and strategies are developed and implemented. The organization has set neither revenue goals nor strategies on how to increase advertising income. Although the organization occasionally identifies possible sources of new revenue, effective and enduring goals and strategies are neither formulated nor communicated effectively to staff. The organization has identified a potential source of revenue from at least one source, formulated and implemented strategy and established goals which are effectively communicated to staff. The organization has developed sales goals, identified potential new revenue from multiple sources, and formulated and implemented effective strategies. Goals and strategies are communicated to staff and are periodically reviewed to determine their effectiveness. Baseline: Level __ Justification: 2 Capable sales and marketing staff employ best professional practices. No professionally capable sales and marketing personnel are on the staff. Sales activities are done on an ad hoc basis by unqualified employees. Although sales and marketing duties are assigned to specific staff as secondary duties, they have no specific expertise and there are no policies or methodologies. Sales and marketing functions are assigned to specific managers or staff who have some experience but who also perform other functions. Some policies and procedures are set. Sales and marketing duties are assigned to dedicated staff who have expertise in the field. Clearly defined goals, policies and procedures guide their work. Baseline: Level __ Justification: 3 Market research is conducted, with results informing business decisions. The organization neither conducts nor uses market research in planning or decision-making. Although the organization attempts to conduct basic market research, management decisions are not relying on and do not reflect outcomes of the market research. The organization plans and systematically conducts market research, but management is not fully aware of how to use data. Market research is done systematically and is routinely used in decision making. Baseline: Level __ Justification: 4 Strategic planning and decision-making are based on an identified and clearly defined target audience. The organization is unable to identify a specific target audience. The organization has a general idea about who they are trying to reach but the target audience is not well-defined and is not considered in planning or decision￾making. The organization has defined a target audience. Strategic planning and decision￾making do not fully consider the target audience profile and the staff is not informed. The target audience is well￾defined and is the foundation for effective strategic planning and decision making. The staff is fully informed and trained on how to focus their work on meeting the needs and interests of the target audience. Baseline: Level __ USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 50 Justification: 5 Effective promotional campaigns focus on the target audience. The organization is not working to increase awareness among target audiences. The organization has plans and discussions on the ways to increase awareness of their product among target audiences, but no tangible activities are undertaken. The organization has done at least a couple of awareness campaign to increase awareness of their product. The campaign lacks focus, and its impact is minimal or not measured. The organization is working on an awareness campaign in a systematic manner, effects are measured, and some results are achieved. Baseline: Level __ Justification: 6 Programming and content production decisions are consistent with the interests of the target audience. Content production and/or programming decisions are not based on the needs and interests of the target audience. The organization uses internal resources such as staff input to determine how content production and programming can focus more effectively on target audience interests. The organization uses external resources such as input through websites, social media, focus groups and market research to determine the interests of target audiences and makes programming and content decisions accordingly. The organization expands its collection of external inputs on audience interests, feedback on programming and content, and creates an institutional culture that prioritizes meeting audience needs so that all content production, programming and related decisions are driven by consumer preferences and demands. Baseline: Level __ Justification: 51 ANNEX E: QUANTITATIVE SURVEY QUESTIONNAIRE AUDIENCE BEHAVIOR AND PERCEPTION SURVEY Methodology and Sample Size: Total Sample Size: 1000 Respondents Sample Demographic Targets: Male 50% / Female 50% 18–65 Years of age Data Collection Period: Data Collection Method: CATI (Computer Assisted Telephone Interview) Interviewer Introduction Hello, my name is _______________ and I represent the research organization GeoPoll. We are conducting a survey on media audience behaviors, and citizen interaction with government and civil society organizations. The main objective of this survey is an analysis of public perceptions in Liberia that will be used to help improve programming available to Liberian audiences. We would like to ask you several questions about how you use media, and questions regarding your interactions with government and Civil Society Organizations. We guarantee the confidentiality and anonymity of your answers. This questionnaire includes four sections, and will last approximately 30 minutes. We thank you very much for taking the time to answer. SQ1. Do you or any member of your family have any personal connection with a radio or television station? 1. Yes  [If the answer is YES, discontinue the survey] 2. No INTERVIEWER, COMPLETE THE TABLE BELOW A1 County 1. Bong 2. Grand Bassa 3. Margibi 4. Montserrado 5. Nimba A2 Respondent full name Name___________________________________ A3 Respondent Mob. Phone / E-mail Mob. Phone______________________________ E-mail__________________________________ A4 Time started interview (24 hour format) A5 Time ended interview (24 hour format) A6 Total time of interview (in minutes) A7 Interviewer code A8 Supervisor code A9 Date of interview ____/____/2019 USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 52 DEMOGRAPHIC INFORMATION, PERSONAL DETAILS In this first section, I would like to ask some questions about yourself. Q1. What level of education have you reached? 1. No formal education 2. Some Primary 3. Completed Primary 4. Some Secondary 5. Completed Secondary 6. Specialized Secondary Education 7. Some Higher Education 8. Completed Higher Education 9. Other ____________________________ 98. Refuse to answer 99. Don’t know Q2. Gender /INTERVIEWER, WRITE DOWN THE GENDER WITHOUT ASKING THE RESPONDENT/ 1. Male 2. Female Q3. Age 1. 18-25 2. 26-35 3. 36-45 4. 46-55 5. 56-65 98. Refuse to answer Q4. Marital Status 1. Single 2. Married 3. Separated / Divorced 4. Widow / Widower 5. Refused to Answer 53 SECTION A – MEDIA GENERAL Q5. What sources of information do you use to keep informed on current events? /INTERVIEWER, DO NOT READ RESPONSES. MULTIPLE RESPONSES ALLOWED. NOTE FIRST MENTIONED IN Q5 ROW AND OTHER MENTIONED IN Q5A ROW/ Q6. And of these sources of information you mentioned, which is the one source that is the most important to you? /INTERVIEWER, RECORD ONE ANSWER ONLY/ Q7. Which of these sources do you believe is the one most reliable source of information about current events in this country? /INTERVIEWER, RECORD ONE ANSWER ONLY/ Q8. How often do you access these sources of information? /INTERVIEWER, ASK FOR EACH SOURCE USED (FROM Q5), ONE AFTER THE OTHER. ONE ANSWER FOR EACH SOURCE/ Radio Television Newspaper and Magazines Internet Word of mouth (from friends, family members) Religious Institution Mobile phones/ Text (SMS) message Billboards / posters Facebook / Social Media Do not know Refuse to answer Q5 First source 1 2 3 4 5 6 7 8 9 98 99 Q5A Other sources of information 1 2 3 4 5 6 7 8 9 98 99 Q6 Most important 1 2 3 4 5 6 7 8 9 98 99 Q7 Most reliable 1 2 3 4 5 6 7 8 9 98 99 Q8 Frequency 1. Every day 1 1 1 1 1 1 1 1 1 2. 2-3 times a week 2 2 2 2 2 2 2 2 2 3. Once a week 3 3 3 3 3 3 3 3 3 4. A few times a month 4 4 4 4 4 4 4 4 4 5. Once a month 5 5 5 5 5 5 5 5 5 6. Less often than once a month 6 6 6 6 6 6 6 6 6 98. Don’t know 98 98 98 98 98 98 98 98 98 99. Refuse to answer 99 99 99 99 99 99 99 99 99 Q9. Do you listen to the radio, regardless of how often and where? (in a minibus, in the car, at home, at work; on the radio, on a cellular telephone, via the Internet)? 1. Yes  ASK Q10 AND BELOW 2. No  SKIP TO TELEVISION (SECTION C / Q21) USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 54 SECTION B – ASK THOSE WHO LISTEN TO RADIO In this section, I would like to ask you about how you listen to radio. Q10. SPONTANEOUS AWARENESS: Can you please tell me the names of all the radio stations you can think of, even if you listen to them or not? /INTERVIEWER, DO NOT READ VARIANTS. NOTE ONE ANSWER IN «FIRST MENTION» COLUMN AND OTHER ANSWERS IN «OTHER MENTIONED» COLUMN. Q11. PROMPTED AWARENESS: I will now read to you a list of radio stations available in your local market. /INTERVIEWER, READ FROM PREPARED LIST OF RADIO STATIONS SERVING THE GEOGRAPHIC MARKET AREA/ Apart from those that you have just mentioned to me, which of these radio stations that I mentioned have you ever heard of? /INTERVIEWER, MORE THAN ONE ANSWER ACCEPTED/ Q12. FAVORITE: Which radio station would you consider to be your favorite? /INTERVIEWER, ONE ANSWER ONLY Q13. WEEKLY REACH: Which radio stations have you listened to during the past 7 days? /INTERVIEWER, RECORD THE NAMES/CODES FOR THE STATIONS MENTIONED. MORE THAN ONE ANSWER ACCEPTED/ No Number of question Q10 Q11 Q12 Q13 Station name / Frequency First Mention Other Mentions Prompted Awareness Favorite Past 7 days 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 6 7 7 7 7 7 7 8 8 8 8 8 8 9 9 9 9 9 9 10 10 10 10 10 10 Other__________________ Other__________________ Other__________________ Other__________________ 98 Refuse to answer 98 99 Do not know 99 100 Didn’t listen to the radio over the past 7 days 100 /INTERVIEWER, IF THE RESPONDENT DID NOT LISTEN TO ANY RADIO STATON (100) WITHIN THE LAST 7 DAYS OR REFUSED TO ANSWER (98) OR DOESN’T KNOW (99)  GO TO Q16/ Q14. Did you listen to the radio yesterday? (whether you were in a minibus, in a car, at home, at work. Listening is accepted whether using a radio, a cellular telephone, or using the Internet) 1. Yes  GO TO Q15 2. No  GO TO Q16 Q15. Thinking back over the previous 24 hours, what were all the times when you listened to the radio yesterday? / INTERVIEWER: BEGIN AT 06:00 AND GUIDE THE RESPONDENT THROUGH EACH TIME INTERVAL AND NOTE THE TIME OF LISTENING / What were the specific radio stations that you listened to at those times? / INTERVIEWER: OPPOSITE EACH TIME INTERVAL, NOTE THE CODES OF THE RADIO STATIONS TO WHICH THE RESPONDENT 55 LISTENED. PROBE REPEATEDLY TO CAPTURE ALL STATIONS LISTENED TO DURING EACH TIME INTERVAL. MULTIPLE STATIONS FOR EACH INTERVAL ALLOWED / Q15 Time CODES OF RADIO STATIONS 1st Mention 2nd Mention 3rd Mention 4th Mention 5th Mention 1 06:00-07:59 2 08:00-09:59 3 10:00-13:59 4 14:00-17:59 5 18:00-19:59 6 20:00-23:59 Q16. PLACE OF LISTENERSHIP: Where do you most often listen to the radio? /INTERVIEWER, READ THE ANSWERS. ONE ANSWER ACCEPTED/ 1. At home 2. At a friend’s house 3. In a public park or outdoor informal meeting place 4. At my place of work 5. At a bar, café or restaurant 6. In a motor vehicle (car, bus, etc.) 7. While walking around 8. At school / university 9. While shopping 10. Other______________________ 98. Refuse to answer 99. Don’t know Q17. What type of radio program is your favorite? /INTERVIEWER, ONLY ONE ANSWER ACCEPTED. MARK THE ANSWERS IN COLUMN Q17 IN THE TABLE SHOWN AFTER Q18/ Q18. What other types of programs are you most interested in listening to on the radio stations you listen to? /INTERVIEWER, MARK THE ANSWERS IN Q18 COLUMN. WRITE DOWN THE CODES: MORE THAN ONE ANSWER ACCEPTED/ Type Q17 (Favorite) Q18 (Other Mentions) News 1 1 Weather 2 2 Politics 3 3 Talk/Discussion/Call-in Show 4 4 Sports 5 5 Music 6 6 Religious Content 7 7 Education 8 8 Health 9 9 Drama Series 10 10 Don’t know 99 99 USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 56 Q19. What type of news are you most interested in hearing on the radio? / INTERVIEWER: READ LIST OF TYPES OF NEWS CONTENT. ONLY ONE ANSWER ACCEPTED / Type of news Q19 News about my county (including coverage of local government and county governance issues) 1 News about the local security situation 2 News about politics and elections 3 International News (including coverage of Liberia from overseas) 4 Social & Economic News (news about consumer issues, the price of goods, food and services) 5 Crime News 6 Human interest stories – Stories about ordinary citizens affected by the current situation 7 Health stories 8 Sports News 9 Culture and Entertainment 10 I do not listen to news 98 I do not know 99 Q20. Overall when you think of all the radio stations you know or have heard about…, which radio station would you say has the following qualities: /INTERVIEWER, ONE ANSWER FOR EACH QUALITY/ QUALITY Radio Station Name Don’t Know 1. Which radio station do you believe is the most popular? 99 2. Which radio station do you believe plays the best music? 99 3. Which radio station do you believe is a source that provides the least reliable information? 99 4. Which radio station do you believe is a source that provides the most reliable information 99 5. Which radio station do you believe gives you the best information about national government activities? 99 6. Which radio station do you believe gives you the best information about county (local) government activities? 99 7. Which radio station do you believe has the best presenters? 99 8. Which radio station do you believe provides the most up-to-date news? 99 9. Which radio station do you believe is the best source of news about your city / county? 99 57 SECTION C – ASK THOSE WHO WATCH TELEVISION In this section, I would like to ask you about television channels and programs. Q21. Do you ever watch television? 1. Yes  ASK Q22 AND BELOW 2. No  SKIP TO SECTION D ABOUT INTERNET /Q34/ Q22. SPONTANEOUS AWARENESS: Can you please tell me the names of all the television stations/channels you can think of even if you never watch them? /INTERVIEWER, DO NOT READ LIST OF TELEVISION STATIONS. MARK THE CODE FOR THE FIRST STATION MENTIONED IN COLUMN «FIRST MENTIONED», AND MARK ALL THE OTHERS IN «OTHER MENTIONS» COLUMN, WHERE MORE THAN ONE ANSWER ACCEPTED/ Q23. PROMPTED AWARENESS: I will now read to you a list of television stations/channels available in your local market. /INTERVIEWER, READ FROM PREPARED LIST OF TELEVISION STATIONS SERVING THE GEOGRAPHIC MARKET AREA/ Apart from those that you have just mentioned to me, which of these have you ever heard of? /INTERVIEWER, MORE THAN ONE ANSWER ACCEPTED/ Q24. FAVORITE: Which television station/channel would you consider to be your favorite? /INTERVIEWER, ONE ANSWER ONLY Q25. WEEKLY REACH Which television stations among those you mentioned have you watched during the past 7 days? /INTERVIEWER, SEVERAL ANSWERS ACCEPTED/ № Television Station / Channel Name Q22 Q23 Q24 Q25 First Mention Other Mentions Prompted Awareness Favorite Past 7 days 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6 6 6 6 6 6 7 7 7 7 7 7 8 8 8 8 8 8 9 9 9 9 9 9 10 10 10 10 10 10 11 11 11 11 11 11 12 12 12 12 12 12 13 13 13 13 13 13 14 14 14 14 14 14 15 15 15 15 15 15 Other___________________ Other___________________ Other___________________ Other___________________ 98 Refuse to answer 98 98 98 98 98 99 Don’t know 99 99 99 99 99 100 I did not watch television during the past 7 days 100 USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 58 /INTERVIEWER, IF THE RESPONDENT DID NOT WATCH ANY TELEVISION CHANNEL (100) WITHIN THE LAST 7 DAYS OR REFUSED TO ANSWER (98) OR DOESN’T KNOW (99)  GO TO Q28/ Q26. Did you watch television yesterday? (whether you were at home, at work, at a friend’s house, in a restaurant, in a bar, in a café) 1. Yes  GO TO Q27 2. No  GO TO Q28 Q27. Thinking back over the previous 24 hours, what were all the times when you watched television yesterday? / INTERVIEWER: BEGIN AT 06:00 AND GUIDE THE RESPONDENT THROUGH EACH TIME INTERVAL AND NOTE THE TIME OF VIEWING / What were the specific television channels that you watched at those times? /INTERVIEWER: OPPOSITE EACH TIME INTERVAL, NOTE THE CODES OF THE TELEVISION CHANNELS THAT THE RESPONDENT VIEWED. PROBE REPEATEDLY TO CAPTURE ALL CHANNELS WATCHED FOR AT LEAST 5 MINUTES DURING EACH TIME INTERVAL. MULTIPLE CHANNELS FOR EACH INTERVAL ALLOWED / Time CODES OF TELEVISION CHANNELS 1st Mention 2nd Mention 3rd Mention 4th Mention 5th Mention 1 06:00-07:59 2 08:00-09:59 3 10:00-13:59 4 14:00-17:59 5 18:00-19:59 6 20:00-23:59 Q28. Where do you usually watch television? /INTERVIEWER, READ THE ANSWERS. ONLY ONE ANSWER ACCEPTED/. 1. At home 2. At a friend’s house 3. In a public park or outdoor informal meeting place 4. At my place of work 5. At a bar, café or restaurant 6. At school/college 7. Other ____________________________ 98. Refuse to answer 99. Don’t know Q29. What type of television do you have access to? /INTERVIEWER, MULTIPLE ANSWERS ACCEPTED/ 1. Terrestrial TV 2. Cable TV 3. Satellite TV 4. Internet/Online Viewing 5. Other ___________ Q30. What is your favorite type of television program? /INTERVIEWER, ONLY ONE ANSWER ACCEPTED. MARK THE ANSWER IN Q30 COLUMN IN THE TABLE SHOWN AFTER QUESTION Q31/ 59 Q31. What other programs are you interested in watching on television? /INTERVIEWER, READ FROM LIST OF PROGRAM CATEGORIES. MARK THE ANSWERS IN Q31. SEVERAL ANSWERS ACCEPTED/ Type Q30 (Favorite Type) Q31 (Other Mentions) News 1 1 Current affairs/Discussion programs 2 2 Commercials / Advertisements 3 3 Music videos 4 4 Soap operas (melodramas) 5 5 Drama serials 6 6 Comedy serials 7 7 Sports 8 8 Religious programs 9 9 Educational / Documentary programs 10 10 Movies 11 11 Children’s programs 12 12 Talk shows 13 13 Comedy programs 14 14 Don’t know 99 99 Q32. Overall, when you think of all the television stations you know or have heard about…: /INTERVIEWER, ONLY ONE ANSWER FOR EACH QUALITY/ QUALITY Code/Name of Television Channel Don’t know 1. Which television station do you believe is the most popular? 99 2. Which television station do you believe plays the most entertaining programs? 99 3. Which television station do you believe is a source that provides the least reliable information? 99 4. Which television station do you believe is a source that provides the most reliable information? 99 5. Which television station do you believe gives you the best information about national government activities and services? 99 6. Which television station do you believe gives you the best information about county (local) government activities and services? 99 7. Which television station do you believe has the best presenters? 99 8. Which television station do you believe provides the most up-to-date news? 99 9. Which television station do you believe is the best source of news about your city / county? 99 Q33. What type of news are you most interested in watching on television? / INTERVIEWER: READ LIST OF TYPES OF NEWS CONTENT. ONLY ONE ANSWER ACCEPTED / Type of news Q33 News about my county/city (including coverage of county and city governance issues) 1 News about the local security situation 2 News about politics and elections 3 International News (including coverage of Liberia from overseas) 4 Social & Economic News (news about consumer issues, price of goods, food, and services) 5 Crime News 6 Human interest stories (Stories about ordinary citizens affected by the current situation) 7 Health stories 8 Sports News 9 Culture and entertainment news 10 I do not watch news programs 98 Don’t know 99 USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 60 SECTION D – INTERNET In this section, I would like to ask about if and how you use the Internet. Q34. Have you ever used the Internet? (via cell phone, personal computer, in Internet cafes, at home, at work) 1. Yes GO TO Q35 2. No GO TO SECTION E (Q45) Q35. (IF YES) Where do you access the Internet? /INTERVIEWER, RECORD ANY THAT APPLY/ 1. At home 2. At an Internet café 3. At work 4. Other(specify)_______________________ 99. Don’t know Q36. Do you have a mobile phone? 1. Yes 2. No  GO TO INTERNET USE QUESTION (Q38) Q37. Do you access the internet through your mobile phone? 1. Yes 2. No Q38. When you use the Internet, for what do you use it? /INTERVIEWER, MULTIPLE ANSWERS ALLOWED. TICK ANY THAT APPLY/ 1. Personal e-mail 2. Looking for jobs 3. For news or information 4. For music 5. To watch videos (movies, humorous video clips, etc.) 6. Particularly for news about Libya 7. For Social Media (Facebook, Twitter, Instagram, etc.) 8. To Make International Calls 9. For my work/job 10. For my school/studies 11. For information about Humanitarian Assistance (Food, Water, Shelter, etc.) 12. Chat 13. Particularly for news about my county 14. Research 99. Don’t know Q39. How often do you use the Internet (on a computer or on your mobile phone)? /INTERVIEWER, ONLY ONE ANSWER/ 1. Every day  GO TO Q40 2. Several times a week  GO TO Q40 3. Several times a month  GO TO SECTION E (Q45) 4. Several times a year  GO TO SECTION E (Q45) 99. Don’t know  GO TO SECTION E (Q45) Q40. On average how much time per day do you spend using the Internet (on a computer or on your mobile phone)? /INTERVIEWER, ONLY ONE ANSWER/ 1. 0-30 minutes per day 2. 31-60 minutes per day 3. 1 to 2 hours per day 4. 2 to 4 hours per day 5. 4 to 6 hours per day 6. More than 6 hours per day 98. Don’t know 99. Refused to answer Q41. Do you trust the information you receive from News Websites on the Internet? /INTERVIEWER, ONLY ONE ANSWER/ 1. Absolutely trust 2. Somewhat trust 3. Somewhat distrust 4. Absolutely distrust 5. Don’t Know 6. Refuse to Answer 61 Q42. How often do you share information you receive from News Websites on the Internet when speaking with family and friends? /INTERVIEWER, ONLY ONE ANSWER/ 1. Always, I talk about information I receive from news websites in daily conversations 2. Once in a while, only when information I receive from news websites is useful to me 3. Never, I do not believe information from news websites is useful 4. Don’t Know 5. Refuse to Answer Q43. Do you trust the information you receive from Facebook and other Social Media Platforms (Twitter, Instagram, etc.)? /INTERVIEWER, ONLY ONE ANSWER/ 1. Absolutely trust 2. Somewhat trust 3. Somewhat distrust 4. Absolutely distrust 5. Don’t Know 6. Refuse to Answer Q44. How often do you share information you receive from Facebook and other Social Media Platforms when speaking with family and friends? /INTERVIEWER, ONLY ONE ANSWER/ 1. Always, I talk about information I receive from Facebook and other social media in daily conversations 2. Once in a while, only when information I receive from Facebook and other social media is useful to me 3. Never, I do not believe information from Facebook and other Social Media is useful 4. Don’t Know 5. Refuse to Answer USAID.GOV CIVIL SOCIETY AND MEDIA WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION | 62 SECTION E – GOVERNANCE AND CIVIL SOCIETY In this section, I would like to ask you about your perceptions of and interactions with government and civil society. Q45. Let's start with your general view about the current direction of our country. Some people might think the country is going in the wrong direction. Others may feel it is going in the right direction. So let me ask you about the overall direction of the country: Would you say that the country is going in the wrong direction or going in the right direction? /INTERVIEWER, ONLY ONE ANSWER/ 1. Going in the Wrong Direction 2. Going in the Right Direction 3. Do Not Know 4. Refuse to Answer Q46. Looking back, how do you rate economic conditions in the country compared to 12 months ago? 1. Much worse 2. Worse 3. The Same 4. Better 5. Much Better 6. Do Not Know 7. Refuse to Answer Q47. Have you ever received information about the County Social Development Fund (CSDF) in your area? 1. Yes /Go to Q48/ 2. No /SKIP TO Q51/ 3. Do Not Remember/SKIP TO Q51/ 4. Refuse to Answer /SKIP TO Q51/ Q48. How did you receive information about the County Social Development Fund (CSDF)? / INTERVIEWER, MULTIPLE RESPONSES ALLOWED / 1. Attended a public meeting / event 2. Listened to a radio program 3. Watched a television program 4. Received information on Facebook or social media sites 5. Read information on an internet website 6. Read information in a newspaper 7. Saw information on posters or billboards 8. Do Not Remember 9. Refuse to Answer Q49. Have you ever attended a public discussion of the County Social Development Fund (CSDF) or attend a CSDF County Sitting? 1. Yes /GO TO Q50/ 2. No /SKIP TO Q51/ 3. Do Not Remember /SKIP TO Q51/ 4. Refuse to Answer /SKIP TO Q51/ Q50. Do you feel you were able to influence and participate in decisions made about your County Social Development Fund (CSDF)? 1. Yes 2. No 3. Do Not Remember 4. Refused to Answer Q51. Have you heard of the “More for Education” campaign? 1. Yes 2. No /SKIP to Q52/ 3. Do Not Remember /SKIP to Q52/ 4. Refuse to Answer /SKIP to Q52/ 63 Q52. How did you receive information about the “More for Education” campaign? / INTERVIEWER, MULTIPLE RESPONSES ALLOWED / 1. Attended a public meeting / event 2. Listened to a radio program 3. Watched a television program 4. Received information on Facebook or social media sites 5. Read information on an internet website 6. Read information in a newspaper 7. Saw information on posters or billboards 8. Do Not Remember 9. Refuse to Answer Q53. Have you attended a Parent-Teacher Association meeting or School Community Monitoring Group meeting? 1. Yes 2. No /SKIP TO Q55/ 3. Do Not Remember /SKIP TO Q55/ 4. Refuse to Answer /SKIP TO Q55/ Q54. Do you feel you were able to influence and participate in decisions made by the Parent￾Teacher Association or School Community Monitoring Group? 1. Yes 2. No 3. Do Not Remember 4. Refused to Answer Q55. Have you heard of the Elections Coordination Committee? 1. Yes 2. No /SKIP to Q57/ 3. Do Not Remember /SKIP to Q57/ 4. Refuse to Answer /SKIP to Q57/ Q56. How did you receive information about the Elections Coordination Committee? / INTERVIEWER, MULTIPLE RESPONSES ALLOWED / 1. Attended a public meeting / event 2. Listened to a radio program 3. Watched a television program 4. Received information on Facebook or social media sites 5. Read information on an internet website 6. Read information in a newspaper 7. Saw information on posters or billboards 8. Do Not Remember 9. Refuse to Answer Q57. How have conditions changed in the following sectors over the last 12 months on a 5-point scale, where 5 – significantly better, 1 – significantly worse. / INTERVIEWER, ONE ANSWER PER LINE/ Significantly better Somewhat better Not changed Somewhat worse Significantly worse I do not know 1. Roads and Traffic 5 4 3 2 1 99 2. Agriculture and food production 5 4 3 2 1 99 3. Construction of housing, offices, government buildings 5 4 3 2 1 99 4. Health Care 5 4 3 2 1 99 5. Business Services (Trade, repair, hotels, restaurants, etc.) 5 4 3 2 1 99 6. Education and Schools 5 4 3 2 1 99 7. Government Services 5 4 3 2 1 99 Q58. What is your opinion of the quality of government services you receive on a 5-point scale, where 5 – the highest quality/excellent, 1 - the lowest quality/very bad. /INTERVIEWER, ONE ANSWER PER LINE/ Service Excellent Good Satisfactory Bad Very bad I do not know 1. Health Care 5 4 3 2 1 99 2. Education 5 4 3 2 1 99 3. Housing 5 4 3 2 1 99 4. Electricity 5 4 3 2 1 99 5. Police and Security 5 4 3 2 1 99 6. Water 5 4 3 2 1 99 65 Q59. What is your opinion right now of the work of key public institutions? /INTERVIEWER, READ OPTIONS, ONE ANSWER PER LINE/ Very Positive Somewhat Positive Average (Neither Positive nor Negative) Somewhat Negative Very Negative I do not know 1. National Government 5 4 3 2 1 99 2. County Government 5 4 3 2 1 99 3. CSDF Project Management Committee 5 4 3 2 1 99 4. School Community Monitoring Groups 5 4 3 2 1 99 5. National Legislature 5 4 3 2 1 99 6. Ministry of Education 5 4 3 2 1 99 7. Elections Coordination Committee 5 4 3 2 1 99 Q60. Over the past twelve months, what have you done to help make changes in your community? /INTERVIEWER, MULTIPLE ANSWERS ACCEPTED/ 1. Voted in an election 2. Participated in protest movements to demand changes in my community 3. Participated in public meetings with the national government 4. Participated in public meetings with the county government 5. Talked to my county government leaders privately 6. Talked to teachers or county school officials privately 7. Volunteered with a Civil Society Organization (CSO) 8. Volunteered at a radio station 9. It is not possible for me to have influence and change things, it doesn’t matter 98. I do not know 99. Refused to respond Q61. Over the past twelve months, what were the most effective ways that you influenced change in your community through the following actions? /INTERVIEWER, MULTIPLE ANSWERS ACCEPTED/ 1. Voting in an election 2. Participating in protest movements to demand changes in my community 3. Participating in public meetings with the national government 4. Participating in public meetings with the county government 5. Talked to my county government leaders privately 6. Talked to teachers or county school officials privately 7. Volunteered my time at a Civil Society Organization (CSO) 8. Volunteered my time at a radio station 9. It is not possible for me to have influence and change things, it doesn’t matter 98. I do not know 99. Refused to respond THANK RESPONDENT AND FINISH INTERVIEW ANNEX F: EVALUATION STATEMENT OF WORK CIVIL SOCIETY AND MEDIA PROJECT WHOLE-OF-INTERMEDIATE RESULT (IR) EVALUATION STATEMENT OF WORK 1. Background This Statement of Work (SOW) describes the terms of reference for an external whole-of￾Intermediate Result (IR) evaluation of USAID/Liberia’s 2012-20199 Country Development Cooperation Strategy (CDCS) IR 1.4: “Civil society and media exercise their advocacy and oversight roles.” Interventions aimed at achieving IR 1.4 include the Civil Society and Media Project Appraisal Document (PAD) under the Democracy, Rights, and Governance (DRG) Development Objective (DO). The five-year, $29 million PAD aims to increase influence of citizens and media in the governance of public goods and services. It comprises two activities: Liberia Media Development (LMD) and Liberia Accountability and Voice Initiative (LAVI). Predating the PAD was the Civil Society and Media Leadership (CSML) activity. The goal of the $19.5 million CSML was to sustain peace in Liberia through greater inclusion, giving a voice to, informing and engaging Liberian citizens. The purpose of this whole-of-IR evaluation is to examine the entire Civil Society and Media portfolio, including all its constituent activities and progress toward the achievement of the Project Purpose. The Project Purpose is the highest-level result to be achieved by a project. It supports the Mission’s Country Development Cooperation Strategy (CDCS) Results Framework at the IR level. The Mission will use findings from the whole-of-IR evaluation to inform adaptive management of the PAD and new design of the CDCS. Overview of Civil Society and Media PAD The activities in the Civil Society and Media PAD have been compiled with the goal of achieving DO 1 of USAID/Liberia’s CDCS: More effective, accountable, and inclusive governance. The purpose of the PAD is increased influence of citizens and media in the governance of public goods and services. The project is designed to deliver on I.R. 1.4: Strengthen the capacity of civil society and the media to exercise their advocacy and oversight roles; and contribute to two sub-intermediate results: 1.4.1: Civil society strengthened to play an effective watchdog role, and 1.4.2: Increased accuracy and impartiality of target media. The underlying developmental hypothesis is that increased influence of citizens and media in the governance of public goods and services will result if: (1) there are stronger multi-stakeholder partnerships advocating for and monitoring policy and accountability reforms; and (2) citizens’ access to independent and reliable information is increased. Through two components, one focused on civil society and the other on the media, the project aims to support the following outputs: 9 The CDCS was originally designed to cover 2012–2017 but was subsequently extended. 67 ● Increased horizontal and vertical linkages among actors engaged in similar issues; ● Increased organizational capacity of targeted CSOs to participate in issue-based reforms; ● On-going capacity development services available on local market; ● Learning and methodologies shared and applied by other development actors; ● Sustainability of media houses is strengthened; ● Plurality of voices is represented in Liberian media; ● Professionalism of media is improved; ● Enabling environment for media is improved; and ● GOL’s ability to respond to Freedom of Information requests from citizens and media establishments is strengthened. Overview of Civil Society and Media Activities Activity Contra ct # Activity Dates Funding Implementin g Partner AOR/COR Alternate AOR/COR Geographic Focus Liberia Accountability and Voice Initiative (LAVI) AID￾669-C￾16- 00003 Nov. 25, 2015 – Nov. 25, 2020 $17 Million Development Alternatives Incorporated (DAI) Michael Haines Mary Zell All 15 counties Liberia Media Development (LMD) AID￾669-A￾15- 00003 Sept. 18, 2015 – Sept. 18, 2020 $11 Million Internews Michael Haines April O’Neill All 15 counties Civil Society and Media Leadership (CSML) AID￾669-00- 09- 00104- 00 Feb. 26, 2010 - Nov. 30, 2015 $19.5 million IREX Louise Fahnbulleh Michael Haines All 15 counties The USAID/Liberia Results Framework is provided in Annex I. To link the Civil Society and Media PAD to the overall USAID/Liberia objectives, refer to the Civil Society and Media PAD Results Framework in Annex II. The LAVI Results Framework at the activity level is provided in Annex III, and the LMD Results Framework is provided in Annex IV. While many USAID/Liberia activities support civil society’s ability to deliver services, IR 1.4 focuses specifically on the advocacy and oversight role of civil society. 2. Purpose of Evaluation The Civil Society and Media PAD was a new addition to the current CDCS. As a cross-cutting intervention, there is much to be learned about how to integrate Civil Society and Media into project design to maximize impact and effectiveness. To ensure that the activities are strategically contributing to Project Purpose in the most effective way, this whole-of-IR evaluation will examine their progress, synergies, and sustainability. At the project level, the evaluation will test the development hypothesis, its relevance, and underlying assumptions. The evaluation will identify results and lessons learned from implementation and will provide succinct, actionable recommendations to inform development of a Civil Society and Media strategy in the new CDCS design. The evaluation should also identify if USAID/Liberia can better measure the effects of Civil Society and Media activities. Evaluation findings and recommendations will be shared and discussed with USAID/Liberia, the implementing partners, and relevant partners. 3. Evaluation Questions 1. Current Strategy Design: To what extent did the programmatic and contextual assumptions of the Civil Society and Media development hypothesis address the underlying barriers to accountable governance of public goods and services? 10 a. Did the new Civil Society and Media PAD design in 2015 improve the Mission’s approach to achieving IR 1.4? b. How well did the strategy adapt to Liberia’s changing context and capitalize on needs and opportunities?11 2. Implementation: How much has the Civil Society and Media portfolio contributed to more effective, accountable, and inclusive governance in Liberia at the DO level? What elements worked or have not worked, and why?12 a. To what extent have the activity-level objectives13 contributed to the broader objectives aimed at strengthening civil society and media’s ability to exercise their advocacy and oversight roles at the IR level? b. Were there missed opportunities? c. What have been the unintended outcomes – positive and negative – of the portfolio, if any, and how have these influenced progress toward IR 1.4 and DO 1? 3. Sustainability: To what extent is the Civil Society and Media portfolio building the individual and collective capacity of organizations to sustain their work beyond USAID support?14 4. Future Strategy Design: Based on these findings, what are the strategic priorities for DO1 to incentivize GOL reform through Civil Society and Media in the new CDCS design?15 a. What are new, applicable innovations and approaches being used in other contexts to engage civil society and media in government reform? 4. Evaluation Methods The evaluation is expected to apply both quantitative and qualitative methods for data collection and analysis. The evaluation team will conduct a desk review of available literature including 10 In the recommendations, identify what new assumptions and adaptive measures need to be considered given the current government administration. 11 Analysis should include CSML. 12 “Elements” could mean USAID’s coordination of constituent activities, for example. In the recommendations, identify opportunities to improve linkages between (e.g. pathways to change in the ToC) and measurement of the effects of Civil Society and Media activities. 13 E.g. changes to the operations of County Social Development Funds and improving the government’s transparency in budgeting and expenditures. 14 In the recommendations, suggest specific interventions that could be incorporated into current activities or new designs to make civil society and media activities more sustainable, especially in terms of transitioning from donor￾driven to a more locally-driven (and resourced) model. 15 In the recommendations, speak to whether the Civil Society and Media strategy should continue to be integrated into a PAD, across the DG Office’s DO, or a cross-cutting, Mission-wide DO for the new CDCS design. 69 activity and project documents. The evaluation team will also look at sectoral assessments that were used to determine interventions. Visits to the relevant partners will provide primary qualitative data for analysis through methods such as in-depth and key informant interviews, focus groups, and direct observation. In addition, the evaluation team will collect primary quantitative data through mini-surveys. These are different from large-scale quantitative surveys in that, typically, questions are limited to 15- 30, questionnaire can be completed within 30 minutes, and sample size is kept small (25-70 cases). A mini-survey will allow for rapid statistical analysis of beneficiary perceptions, benefits, and challenges. It is expected that the evaluation team will use data provided by the implementing partners in regular quarterly and annual reports, performance reporting, and studies for supplemental quantitative data. The team will also draw on international and national datasets such as the Liberia Electoral Access and Participation (LEAP) Survey media questions, Harvard Debate Impact Study, and Media Sustainability Index. Primary collection of quantitative data through large-scale structured surveys is beyond the scope of this evaluation USAID recommends that the evaluation considers Bong, Nimba, Grand Bassa, Lofa, Grand Gedeh, and Maryland counties as field visit options. These are the common counties in which LAVI, LMD, and CSML work(ed). Partner lists provided by IPs will be further used to select field visit sites that cover the breadth of interventions being reviewed by the evaluation. The USAID/Liberia DRG team will provide documents for the desk review, contact information for prospective interviewees, and innovative case study options for Evaluation Question 4a. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation, as well as additional contacts. Illustrative data sources include but are not limited to: 1. Civil Society and Media PAD 2. LAVI Contract 3. LMD Cooperative Agreement 4. CSML Cooperative Agreement 5. Project Monitoring and Evaluation Plan 6. Activity Monitoring and Evaluation Plans 7. Annual Work Plans 8. Quarterly and annual progress reports 9. Performance Indicator Database System data 10. DQA reports 11. Baseline assessments 12. Organizational assessments and tools 13. USAID/Liberia CDCS 14. CDCS Meta-Analysis 15. CSML Midterm Evaluation 16. Agenda for Transformation 17. Harvard Debate Impact Study 18. LAVI Stocktaking reports 19. LAVI PEA reports 20. LEAP Surveys 21. Annual Media Sustainability Index 22. Civil Society Sustainability Index 2017 23. Other related national data and reports To engage the key evaluation users in analysis of the preliminary findings and help shape the recommendations, the evaluation team will facilitate the debriefs in a more collaborative, half or full-day session. During the session, preliminary findings will be shared, and the evaluation users will add interpretation to the “so what” and “now what” of the evaluation results. This can also help the evaluation team identify areas where more information is needed (e.g. through additional document review or remote interviews). This may take place separately or jointly with USAID, Implementing Partners, and CSOs depending on the nature of the evaluation results. 5. Deliverables and Timeline Evaluation deliverables include: a. Evaluation Team Planning Meetings b. Inbrief with USAID/Liberia and Implementing Partners c. Inception Report with work plan and data collection instruments d. Participatory analysis session with USAID/Liberia, Implementing Partners, and CSOs - Session that allows for stakeholders to engage with the preliminary evaluation findings and help shape the recommendations prior to report drafting. e. Formal Debrief – Presentation to USAID/Liberia prior to team departure. f. Draft Evaluation Report - A draft report should be submitted to LSA for review, and LSA must submit the draft report to USAID/Liberia within three weeks after the in-country work is conducted. The written report should clearly describe findings, conclusions, and recommendations and conform to USAID requirements outlined below. USAID will provide comments on the draft report within five working days of submission. Then, LSA will submit the draft report to the Implementing Partners for another five working days to provide comments and disclose a statement of difference, if applicable. g. Final Report - LSA will submit a final report that incorporates USAID and Implementing Partners’ comments no later than ten days after final, written comments on the team’s draft report have been submitted by all parties. h. Learning Event – LSA will facilitate a learning event one to two months after the report has been finalized to engage USAID/Liberia, Implementing Partners, CSOs, and other relevant stakeholders in the utilization of recommendations. This will be combined with an After-Action Review, prescribed as good learning practice per ADS 201, to support USAID in the incorporation of recommendations into work plans, project or activity design, and the next Country Development Coordination Strategy. The draft evaluation report should meet the following criteria: i. The report should be in line with USAID Evaluation Policy (see Appendix I – Criteria to Ensure the Quality of the Evaluation Report) and USAID Secretariat Style guide. ii. The report should be no longer than 30 pages, excluding executive summary, table of contents, and annexes. iii. The report should include a three to five page Executive Summary highlighting findings and recommendations. iv. The report should represent a thoughtful and well-organized effort to objectively respond to the evaluation questions. v. The report shall address all evaluation questions included in the SOW. 71 vi. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides shall be included in an Annex in the final report. vii. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). viii. Evaluation findings and conclusions should be specific, concise and supported by strong quantitative or qualitative evidence. ix. Recommendations should be action-oriented, practical, specific, and evidence-based. x. The final report should be edited and formatted. xi. Liberia Strategic Analysis must submit the final evaluation to the Development Experience Clearinghouse. The report will be submitted electronically. The final report will be edited/formatted by the contractor and provided to USAID/Liberia five working days after the Mission has reviewed the content and approved the final revised version of the report. The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy: http://www.usaid.gov/ sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Timeline The evaluation should follow the timeline for producing deliverables outlined in the Gantt chart below. The evaluation is estimated to begin January 2019. Performance Evaluation Gantt Chart Activity # of days Week 1-2 Week 3-5 Week 6-7 Week 8 Week 9 Week 10 Week 11 Week 12 Week 13-14 Week 15-16 Week 17-18 Week 19-20 USAID shares SOW with LSA 1 LSA publishes call for CVs 14 LSA recruits and onboards team members 21 Desk Review 7 Team travels to Liberia 1 Team planning meeting - internal 1 Planning Meetings with USAID and IPs 2 Submit draft Inception Report to USAID Inbrief with USAID and IPs 1 Prepare for field work 1 Submit final Inception Report to USAID Field Work and preliminary analysis 21 Participatory analysis sessions with USAID, IPs, and CSOs 1.5 Official debrief with USAID 0.5 Travel from Liberia 1 Analysis and report drafting 13 LSA quality control and revisions 6 LSA submits Draft Report to USAID USAID and IP review the draft report 10 Evaluation team incorporates comments and feedback 5 LSA quality control 5 LSA submits Final Report to USAID LSA facilitates Learning Event with USAID, IP, CSO, and other stakeholders 1 1-2 months after PE Learning event prep work by Team Leader and national consultant 1 73 6. Team Composition The evaluation team will be composed of six individuals: team leader, three technical experts, an evaluation specialist, and a logistics coordinator. The evaluation team will be supported by LSA staff, including a Project Manager based in Monrovia to facilitate client relations and a Project Director based at HQ to enhance quality of deliverables. LSA staff may be used to fill the evaluation team roles if available. 1. Team Leader/Senior Evaluation Specialist: S/he will be responsible for coordinating the overall activities of the evaluation team. The Team Leader will approve the final whole￾of-project evaluation design, oversee the development of evaluation instruments, integrate the findings of different team members, and coordinate the preparation of the final reports. The Team Leader should have at least ten years of experience in the evaluation and administration of DRG interventions in developing countries. Knowledge of Civil Society and Media programming is required, with technical expertise complementing the other international technical expert. The candidate should have a background implementing USAID whole-of-project evaluations and supporting CDCS design. S/he should have experience in managing multi-disciplinary teams and developing and conducting mixed-method evaluations. It is essential that the candidate has the ability to conceptualize and structure whole-of-project evaluation activities and produce high quality evaluation reports in English. The candidate will hold a post graduate degree in public policy, public administration, international development, research, or related field. 2. Social Accountability Advisor: S/he should be a technical expert with at least ten years of experience creating an enabling environment for civil society and media with an emphasis on advocacy techniques in developing countries. S/he must have deep knowledge of the relationship between civil society, media, and government institutions and how to engage actors across horizontal and vertical linkages toward issue-based reforms. Technical sectoral expertise will complement the Team Leader’s, either in civil society or media development. The candidate must have exceptional organizational, analytical, writing, and presentation skills. Knowledge of USAID rules, regulations, and procedures in this sector is highly desirable. The candidate will hold an advanced degree in public policy, public administration, social sciences, international development, or related field. 3. Civil Society Capacity Development Specialist: The specialist should have at least eight years of experience working effectively with civil society organizations, including organizational strengthening, capacity building, sub-granting and network building. S/he should have demonstrated experience with heuristic tools to measure capacity of civil society. The specialist must have strong social analytical skills, specifically in designing and evaluating similar programs that strengthen the advocacy and oversight role of civil society. The incumbent must be able to conduct interviews and focus group discussion and analyze the resulting data. Strong American English language speaking and writing skills are required. The candidate will hold a degree in international development, public administration, organizational development, social sciences, or related field. 4. Media Specialist: The specialist should have at least eight years of implementation experience related to professionalization and sustainability of media houses in developing countries. S/he must have demonstrated expertise in communications, advocacy, and capacity building for media and CSO partners. S/he should have experience in policy reform processes and good governance in Liberia. Knowledge of fundamental concepts, practices, and USAID regulation for media is required. In-depth knowledge of the Liberian media and information landscape is preferred. The candidate will hold an advanced degree in media, communication, journalism, or related field. 5. Evaluation Specialist: The candidate should have at least five years relevant experience in qualitative and quantitative data collection methods. The specialist must have experience evaluating civil society and media programs. S/he will contribute to the overall drafting of the evaluation framework and participate in the desk study, interviews, and other data collection; and analyze the data with input from team members and USAID/Liberia to draft the evaluation report. Some experience in gender analysis and managing or implementing research programs is preferable. The incumbent must be able to conduct interviews and focus group discussions and analyze the resulting data. Strong analytical and writing skills are essential. 6. Logistics Coordinator: The candidate should have at least five years relevant experience in logistics planning for large evaluations or research studies. S/he must have experience in fleet management, scheduling meeting, setting up appointments, and coordinating team travels. The candidate must also have experience in qualitative data collection methods to support interviews as necessary. S/he will serve as the main logistical coordinator for the evaluation, and work with local partners to plan travel, data collection, interviews, and evaluation activities as required. Collectively the team members must have experience in conducting both quantitative and qualitative data collection and analysis. Prior to their arrival in Liberia, all team members are required to familiarize themselves with USAID’s Evaluation Policy, with USAID’s publication outlining a good evaluation report, and with USAID’s checklist for assessing an evaluation report. Additionally, all team members should possess a strong familiarity with the political, economic, policy and educational context in Liberia. USAID may propose internal staff from USAID/Liberia or from Washington to accompany the team in this evaluation as observers. As observers, their role will be to provide, when asked, background information and to reply to the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected evaluation team. 7. Logistics and Level of Effort The evaluation team is anticipated to visit and conduct consultations and data collection visits in the greater Monrovia area with travel to the field. USAID and the Implementing Partners can assist in providing contact information. LSA is responsible for making meeting and logistical arrangements for the evaluation team including hotel, air travel, and local transportation arrangements. Evaluation team members are authorized and expected to work a six-day week. Travel over weekends may be necessary. Work should commence as soon as practicable, but no later than January 2019. For planning purposes, contractors should be aware of Liberian and US holidays during the evaluation time frame. The evaluation should follow the illustrative level of effort given below. 75 Task/Deliverable Estimated time (Days) Team Leader Int’l Team Member Lib Team Member Review background documents & preparatory work (offshore) 7 7 5 Travel to Liberia 2 2 0 Team Planning Meetings in Monrovia with USAID/Liberia and IPs 2 Inbrief with USAID/Liberia and IPs and prepare for field work Submit Inception Report 3 Field work: Data collection and on-going data analysis 21 (including field travel) Participatory analysis sessions with USAID/Liberia, IPs, and CSOs 2 Analysis and report drafting 13 10 7 LSA performs quality assurance check and edits draft report 5 LSA submits draft report to USAID/Liberia 1 USAID/Liberia and IPs provide comments on draft report 10 Team Leader revises draft report to incorporate comments and submit final report 5 2 1 LSA performs quality assurance check and submits final report 5 LSA facilitates learning event 1 0 1 Total time required 77 Working days for consultants 56 49 42 ANNEX G: DISCLOSURE OF ANY CONFLICTS OF INTEREST 77 79 81 ANNEX H: FULL LISTING OF PERSONS INTERVIEWED The full listing of persons interviewed was submitted separately in line with data de-identification policies. Please contact Carla Trippe, ctrippe@socialimpact.com, to request the data. ANNEX I: QUANTITATIVE SURVEY SUMMARY REPORT MEDIA CONSUMPTION AND CITIZEN PERCEPTION SURVEY Social Impact commissioned a quantitative survey in April-May 2019 through GeoPoll to measure audience consumption behaviors for radio, television, and the internet; identify audience perceptions regarding radio and television programming; measure citizen behaviors regarding participation in governance decision making structures such as County Social Development Fund (CSDF) County Sittings and community-level education meetings; and identify citizen perceptions regarding the value of their participation in governance structures. The survey employed computer-assisted telephone interview (CATI) methodology, where Liberian enumerators interviewed respondents from a call center in Monrovia following a prescribed questionnaire. The interviews were computer-assisted because call center operators captured respondent answers using a tablet application. The survey was conducted in Liberian English and targeted a total sample size of 1000 respondents from the five counties targeted during the evaluation. The sample was intentionally stratified for location and age based on population data provided in the Liberia Institute of Statistics and Geo￾Information Services (LISGIS) 2016 household survey, detailed below: GENDER MALE FEMALE TOTAL AGE 18-24 25-34 35-44 45-65 18-24 25-34 35-44 45-65 Bong County 20 18 18 14 20 18 18 14 140 Grand Bassa County 14 13 13 10 14 13 13 10 100 Margibi County 14 13 13 10 14 13 13 10 100 Montserrado County 66 59 59 51 66 59 59 51 470 Nimba County 27 24 24 20 27 24 24 20 190 Totals 141 127 127 105 141 127 127 105 1000 In order to guarantee female representation in the survey, the sample set a 50% female to 50% male respondent quota. Data collection took place over a three-week period from April 17 to May 3, 2019. The CATI methodology targeted mobile phone users, and resulted in the following reported education level from survey respondents: EDUCATION PERCENT COUNT Completed Secondary 32% 324 Some Higher Education 32% 319 Completed Higher Education 12% 121 Some Secondary 11% 113 Specialized Secondary Education 6% 63 No Formal Education 2% 23 Completed Primary 2% 21 Some Primary 1% 10 Other 1% 6 Total 100 1000 83 RESEARCH CAVEATS CATI methodology is reliant in Liberia upon respondents with access to mobile phones. This fact contributed to survey results generating a majority respondent profile which does not reflect the general profile of the Liberian population. The resulting respondent sample was heavily skewed toward highly￾educated internet users: • 44% of the sample either completed university or had completed some university education • 32% of the sample completed secondary education • 85% of the sample reported being internet users, with 48% of the sample reporting using internet every day As we know from alternative survey data on education and literacy rates, the sample does not represent poorly-educated or illiterate adults in Liberia. The LISGIS 2016 household survey reports that 6.8% of Liberians have completed university education, with 26.3% having completed senior high school. In addition, from triangulating internet use statistics from publicly accessible quantitative research, access to internet across Liberia is extremely low (as low as 11% according to the 2019 Hootsuite internet usage survey which pulls data from multiple 2018-2019 sources).16 The data below also includes citizen perceptions of media and reported seven-day radio listening and television viewing behaviors from respondents. However, the survey was not successful in generating 24- hour recall for listenership and viewership. Limitations with survey firm and call center administration of the 24-hour recall questions generated unreliable data, in particular for television viewing. Recommendation to USAID: Based on the experience with this quantitative research effort, the evaluation team recommends that USAID should conduct a baseline survey on citizen perceptions and behaviors in regard to media and political participation to inform design of future Civil Society and Media activities. However, that baseline research should employ face-to-face interview methodology, using a standard household sample selection in order to better represent the Liberian citizen population throughout the country. Face-to-face interviews in households will allow for greater confidence in respondent understanding of questions and in turn lead to greater credibility of respondent sample profiles and data collected. The evaluation team also recommends that USAID bring in independent expert management of quantitative baseline research to oversee compliance with design of methodology, training of research supervisors and enumerators, survey administration, data collection, and analysis. Upon request, LSA can provide support for design, management, and oversight of a baseline quantitative research survey. SURVEY RESULTS: CITIZEN PERCEPTIONS AND BEHAVIORS Overall Political Perceptions: A significant majority of respondents perceived the country as going in the “wrong direction” and reported their economic situation as worse than one year ago. 16 The Hootsuite Digital 2019 report for Liberia can be found at: https://www.slideshare.net/DataReportal/digital￾2019-liberia-january-2019-v01 IS LIBERIA GOING IN THE WRONG DIRECTION OR RIGHT DIRECTION? PERCENT COUNT Going in the Wrong Direction 64% 637 Going in the Right Direction 31% 307 Do Not Know 5% 45 Refused to Answer 1% 11 Total 100 1000 ARE ECONOMIC CONDITIONS BETTER OR WORSE THAN 12 MONTHS AGO? PERCENT COUNT Worse 38% 380 Much Worse 34% 340 Much Better 9% 93 Better 8% 82 The Same 7% 74 Do Not Know 3% 29 Refused to Answer 0% 2 Total 100 1000 Correspondingly, respondents also felt that their ability to enact change in their communities was better rooted in individual action – voting in an election or volunteering time at a local CSO – rather than direct interaction with national or local government decision makers (either through structured meetings or individual interactions). MOST EFFECTIVE WAYS TO AFFECT CHANGE IN MY COMMUNITY (MULTIPLE RESPONSES ALLOWED) PERCENT COUNT Voting in an election 44% 443 Volunteering time at a Civil Society Organization [CSO] 42% 416 Participating in protest movements to demand changes in my community 12% 122 It is not possible for me to have influence and change things 10% 97 Talking to teachers or county school officials privately 9% 90 Participating in public meetings with the county government 6% 59 Talking to my county government leaders privately 6% 56 Participating in public meetings with the national government 3% 32 Volunteering time at a radio station 2% 20 I do not know 1% 7 Respondents provided significantly positive perceptions of government institutions that focus on education issues. Conversely, institutions linked to public resourcing such as CSDF Project Management Committees and the National Legislature generated negative favorability ratings. FAVORABILITY OF INSTITUTIONS NET FAVOR Ministry of Education +21% School Community Monitoring Groups +17% Elections Coordination Committee +7% National Government +2% County Government -/+0% CSDF Project Management Committees -17% National Legislature -20% County Social Development Funds: Less than half of respondents had ever received information about the CSDF in their community, with those that received some information predominantly accessing 85 that information through radio. However, of those that did have information about their local CSDF, only 16% had ever attended a meeting where the CSDF was discussed. HAVE YOU EVER RECEIVED INFORMATION ABOUT THE COUNTY SOCIAL DEVELOPMENT FUND (CSDF) IN YOUR AREA? N=922 PERCENT COUNT Yes 42% 387 No 58% 531 Do Not Remember 0% 3 Refuse to Answer 0% 1 Total 100 922 HAVE YOU EVER ATTENDED A PUBLIC DISCUSSION OF THE COUNTY SOCIAL DEVELOPMENT FUND (CSDF) OR ATTEND A CSDF COUNTY SITTING? N=426 PERCENT COUNT Yes 16% 67 No 84% 359 Total 100 426 HOW DID YOU RECEIVE INFORMATION ABOUT THE COUNTY SOCIAL DEVELOPMENT FUND? N=426 (MULITPLE RESPONSES ALLOWED) PERCENT COUNT Listened to a radio program 91% 389 Read information in a newspaper 10% 44 Received information on Facebook or social media sites 8% 33 Attended a public meeting / event 7% 30 Watched a television program 6% 27 Read information on an internet website 4% 16 Saw information on posters or billboards 1% 3 DO YOU FEEL YOU WERE ABLE TO INFLUENCE AND PARTICIPATE IN DECISIONS MADE ABOUT YOUR COUNTY SOCIAL DEVELOPMENT FUND (CSDF)? N=67 PERCENT COUNT Yes 69% 46 No 31% 21 Total 100 67 Education: Few respondents could recall the “More4Education” campaign, primarily receiving that information through radio. However, respondents overall had significantly positive perceptions of local school oversight bodies (42% positive / 25% negative) and the national Ministry of Education (49% positive / 28% negative). HAVE YOU HEARD OF THE “MORE FOR EDUCATION” CAMPAIGN? PERCENT COUNT Yes 17% 174 No 82% 823 Do Not Remember 0% 2 Refused to Answer 0% 1 Total 100 1000 HOW DID YOU RECEIVE INFORMATION ABOUT THE “MORE FOR EDUCATION” CAMPAIGN? N=174 (MULTIPLE RESPONSES ALLOWED) PERCENT COUNT Listened to a radio program 81% 141 Received information on Facebook or social media sites 13% 22 Read Information on an internet website 9% 15 Read information in a newspaper 8% 14 Watched a television program 5% 9 Attended a public meeting / event 5% 8 Saw information on posters or billboards 2% 3 WHAT IS YOUR OPINION OF SCHOOL COMMUNITY MONITORING GROUPS? PERCENT COUNT Very Negative 9% 88 Somewhat Negative 16% 157 Average (Neither Positive nor Negative) 25% 253 Somewhat Positive 37% 372 Very Positive 5% 51 I do not know 8% 79 Total 100 1000 WHAT IS YOUR OPINION OF THE MINISTRY OF EDUCATION? PERCENT COUNT Very Negative 9% 87 Somewhat Negative 19% 185 Average (Neither Positive nor Negative) 23% 231 Somewhat Positive 42% 415 Very Positive 7% 73 I do not know 1% 9 Total 100 1000 Perceptions of Government Institutions: Other government institutions did not generate positive perceptions from respondents, in particular regarding county government and CSDF project management committees. WHAT IS YOUR OPINION OF COUNTY GOVERNMENT? PERCENT COUNT Very Negative 18% 182 Somewhat Negative 21% 207 Average (Neither Positive nor Negative) 21% 208 Somewhat Positive 33% 326 Very Positive 6% 59 I do not know 2% 18 Total 100 1000 WHAT IS YOUR OPINION OF THE CSDF PROJECT MANAGEMENT COMMITTEE? PERCENT COUNT Very Negative 24% 240 Somewhat Negative 17% 169 Average (Neither Positive nor Negative) 19% 193 Somewhat Positive 20% 200 Very Positive 4% 40 I do not know 16% 158 Total 100 1000 87 WHAT IS YOUR OPINION OF THE NATIONAL GOVERNMENT? PERCENT COUNT Very Negative 21% 206 Somewhat Negative 19% 187 Average (Neither Positive nor Negative) 19% 185 Somewhat Positive 31% 305 Very Positive 11% 106 I do not know 1% 11 Total 100 1000 WHAT IS YOUR OPINION OF THE NATIONAL LEGISLATURE? PERCENT COUNT Very Negative 35% 350 Somewhat Negative 16% 157 Average (Neither Positive nor Negative) 17% 167 Somewhat Positive 26% 255 Very Positive 5% 45 I do not know 3% 26 Total 100 1000 WHAT IS YOUR OPINION OF THE ELECTIONS COORDINATION COMMITTEE? PERCENT COUNT Very Negative 12% 123 Somewhat Negative 14% 137 Average (Neither Positive nor Negative) 22% 218 Somewhat Positive 26% 262 Very Positive 7% 72 I do not know 19% 188 Total 100 1000 SURVEY RESULTS: MEDIA AUDIENCE CONSUMPTION BEHAVIORS AND PERCEPTIONS By far, survey respondents look to radio as their first and most important source of information, considering the news they received from radio as most reliable compared to other media sources. The sample also considered the internet as a significant information source – understanding the caveat described above regarding the significant reported internet usage of this particular respondent sample. FIRST SOURCE OF INFORMATION MOST IMPORTANT SOURCE OF INFORMATION MOST RELIABLE SOURCE OF INFORMATION PERCENT COUNT PERCENT COUNT PERCENT COUNT Radio 68% 680 63% 626 74% 740 Internet 21% 207 25% 252 15% 153 Facebook/ Social Media 5% 54 6% 57 4% 36 Word of Mouth [From Friends/Family Members] 2% 23 1% 14 1% 13 Newspaper and Magazines 1% 12 1% 13 2% 19 Television 1% 11 2% 17 2% 18 Mobile Phones/ Text [SMS] Message 1% 10 1% 10 1% 7 Religious Institution 0% 3 1% 10 1% 11 Do Not Know 0% 0 0% 1 0% 3 Total 100 1000 100 1000 100 1000 Radio Station Awareness and Reach: The total respondent sample identified ELBC and Truth FM (Monrovia) – both in terms of spontaneous recall of station names, and through reported seven-day reach (respondents recalling having listened to a particular station during the previous seven-day period). REPORTED RADIO LISTENING 7-DAY REACH – TOP TEN RADIO STATIONS N=890 (MULTIPLE RESPONSES ALLOWED) PERCENT COUNT ELBC 31% 279 Truth FM 19% 172 BBC 13% 116 ECOWAS 12% 110 OK FM 10% 87 ELWA 8% 73 Fabric 7% 65 Sky 6% 53 Radio Gbarnga 5% 46 Radio Nimba 5% 43 Radio Audience Perceptions: Correspondingly, the government-operated ELBC and private Truth FM were also considered the most reliable sources of information by listeners, both in general and in regard to coverage of national and local government activities. This data corresponds to focus group participation descriptions specifically about ELBC and their correspondents in various counties that provide daily contributions to ELBC news coverage. RADIO STATION SPONTANEOUS AWARENESS RADIO LISTENERS – TOP TEN RADIO STATIONS N=890 (MULTIPLE RESPONSES ALLOWED) PERCENT COUNT ELBC 57% 510 Truth FM 39% 347 HOTT FM Nimba 14% 121 HOTT FM 13% 117 OK FM 13% 114 ECOWAS 12% 106 Radio Gbarnga 11% 99 Fabric 11% 98 Super Bongese 11% 96 Sky 9% 84 89 MOST RELIABLE INFORMATION TOP TEN RADIO STATIONS PERCENT COUNT ELBC 34% 302 Truth FM 11% 94 ECOWAS 8% 73 BBC 6% 53 Radio Nimba 3% 28 Kergheamahn FM 3% 24 Fabric 3% 24 ELWA 3% 23 OK FM 2% 20 Radio Gbarnga 2% 18 LEAST RELIABLE INFORMATION TOP TEN RADIO STATIONS PERCENT COUNT Kings 10% 93 HOTT FM 4% 40 Magic 4% 34 Fabric 3% 26 ELBC 2% 19 Peace 2% 17 ELWA 2% 15 Lacsa FM 2% 14 Radio Saclepea 1% 13 Voice FM 1% 11 BEST INFORMATION ABOUT NATIONAL GOVERNMENT ACTIVITIES TOP TEN RADIO STATIONS PERCENT COUNT ELBC 50% 448 Truth FM 9% 76 ECOWAS 7% 59 ELWA 3% 23 Radio Nimba 3% 23 Kergheamahn FM 2% 20 Fabric 2% 17 Radio Gbarnga 2% 17 OK FM 2% 14 Radio Gbehzon 1% 13 BEST INFORMATION ABOUT LOCAL GOVERNMENT ACTIVITIES TOP TEN RADIO STATIONS PERCENT COUNT ELBC 44% 395 Truth FM 8% 73 ECOWAS 7% 58 Radio Gbarnga 5% 48 Radio Nimba 3% 29 ELWA 3% 27 Kergheamahn FM 3% 25 Radio Kakata 2% 16 Radio Gbehzon 1% 13 Fabric 1% 13 MOST UP-TO-DATE NEWS TOP TEN RADIO STATIONS PERCENT COUNT ELBC 37% 325 Truth FM 10% 88 BBC 8% 71 ECOWAS 8% 69 ELWA 3% 25 Kergheamahn FM 3% 23 Radio Nimba 3% 23 OK FM 2% 20 Fabric 2% 18 Radio Gbarnga 2% 18 Television Station Awareness and Reach: Television respondents, primarily from Montserrado county (with 224 of 358 total reported television viewers residing in Montserrado), identified the government-operated LNTV and private stations Sky TV and Real TV as market leaders both through spontaneous recall and in reported viewership during the previous week. SPONTANEOUS AWARENESS TOP TEN TELEVISION STATIONS N=358 PERCENT COUNT LNTV 80% 288 Sky TV 48% 172 Real TV 37% 134 Power TV 24% 85 Hot TV 16% 56 KMTV 11% 41 CNN 8% 30 BBC World News 6% 23 SuperSport 6% 21 Light TV 4% 13 REPORTED TELEVISION VIEWING 7-DAY REACH – TOP TEN TELEVISION STATIONS N=358 PERCENT COUNT LNTV 33% 118 Sky TV 19% 67 Real TV 12% 44 KMTV 9% 34 Power TV 8% 28 SuperSport 9 6% 20 SuperSport 4 5% 18 Light TV 4% 15 Television Audience Perceptions: Correspondingly, LNTV and Sky TV were also considered by respondents to be the most reliable sources of information. However, slightly more respondents perceived Sky TV as an unreliable source of information. As with radio, the state broadcaster LNTV was by far considered the best source of information regarding national and local government activities. MOST RELIABLE INFORMATION TOP TEN TELEVISION STATIONS PERCENT COUNT LNTV 49% 176 Sky TV 10% 35 KMTV 7% 26 BBC World News 6% 23 Real TV 6% 22 CCTV News Channel 4% 14 Base TV 3% 10 Power TV 2% 8 SuperSport 9 1% 4 SuperSport Blitz 1% 3 LEAST RELIABLE INFORMATION TOP TEN TELEVISION STATIONS PERCENT COUNT Real TV 13% 48 Sky TV 11% 41 Power TV 9% 34 KMTV 6% 23 Base TV 5% 18 LNTV 3% 11 Sound City Channel 327 2% 6 CCTV News Channel 409 2% 6 Light TV 1% 4 One Gospel Channel 331 1% 4 MOST UP-TO-DATE NEWS TOP FIVE TELEVISION STATIONS PERCENT COUNT LNTV 51% 184 KMTV 9% 33 Sky TV 9% 31 BBC World News Channel 400 6% 21 Real TV 5% 17 91 BEST INFORMATION ABOUT NATIONAL GOVERNMENT ACTIVITIES TOP FIVE TELEVISION STATIONS PERCENT COUNT LNTV 68% 242 Sky TV 8% 27 KMTV 7% 26 Real TV 5% 17 Base TV 4% 13 BEST INFORMATION ABOUT LOCAL GOVERNMENT ACTIVITIES TOP FIVE TELEVISION STATIONS PERCENT COUNT LNTV 65% 233 Sky TV 9% 31 KMTV 8% 29 Real TV 5% 17 Base TV 4% 15 SURVEY RESULTS: INTERNET USAGE Internet Use: As described above, this specific respondent sample skewed heavily in favor of regular internet users, with 95% of internet users reporting frequent use (daily or several times per week), and 96% having internet access via their personal mobile phones. Of these frequent internet users, over two￾thirds reported using the internet and social media for between 30 minutes to two hours each day, with an additional 17% online for two to four hours per day. INTERNET USERS PERCENT COUNT Yes 85% 850 No 15% 150 Total 100 1000 MOBILE PHONE INTERNET USERS PERCENT COUNT Yes 96% 812 No 4% 33 Total 100 1000 INTERNET – FREQUENCY OF USE PERCENT COUNT Every Day 57% 484 Several Times A Week 38% 323 Several Times A Month 4% 38 Several Times A Year 0% 3 Don’t Know 0% 2 Total 100 850 INTERNET – USE PER DAY N=806 PERCENT COUNT 1 To 2 Hours Per Day 34% 274 31-60 Minutes Per Day 33% 264 2 To 4 Hours Per Day 17% 134 4 To 6 Hours Per Day 8% 66 More Than 6 Hours Per Day 5% 38 0-30 Minutes Per Day 4% 30 Internet User Perceptions and Behaviors: Over 90% of these frequent internet users expressed trust in the information they received via news websites, with 78% of users having confidence in what they found on social media platforms, such as Facebook. Nearly all of these users would share the information they received with family or friends periodically or through daily conversations. TRUST IN INTERNET NEWS WEBSITES N=806 PERCENT COUNT Somewhat Trust 69% 559 Absolutely Trust 23% 188 Somewhat Distrust 6% 49 Absolutely Distrust 1% 6 Don’t Know 0% 4 HOW OFTEN DO YOU SHARE INFORMATION FROM NEWS WEBSITES? N=806 PERCENT COUNT Once in A While - Only When Information I Receive Is Useful to Me 66% 529 Always - I Talk About Information I Receive in Daily Conversations 31% 252 Never - I Do Not Believe Information from News Websites Is Useful 3% 23 Don’t Know 0% 2 TRUST IN FACEBOOK/SOCIAL MEDIA N=807 PERCENT COUNT Somewhat Trust 70% 563 Somewhat Distrust 16% 128 Absolutely Trust 8% 61 Absolutely Distrust 7% 55 HOW OFTEN DO YOU SHARE INFORMATION FROM FACEBOOK/SOCIAL MEDIA? N=807 PERCENT COUNT Once in A While - Only When Information I Receive Is Useful to Me 72% 578 Always - I Talk About Information I Receive in Daily Conversations 22% 174 Never - I Do Not Believe Information from Facebook And Social Media Is Useful 7% 55 93 ANNEX J: STATEMENTS OF DIFFERENCE DAI PROGRAM OVERVIEW The Liberia Accountability and Voice Initiative (LAVI) is a five-year USAID program, implemented by DAI Global. The overarching goal of the program is to strengthen multi-stakeholder partnerships to advocate for and monitor policy and accountability reforms. LAVI aims to develop and implement a portfolio of initiatives with the potential for cumulative impact, clustered around issues which have the greatest potential for change, and which are responsive to locally defined priorities. The four broad objectives of LAVI included 1) Increase horizontal and vertical linkages among actors engaged in similar issues; 2) Increase organizational capacity of targeted CSOs to participate in issue-based reforms; 3) Promote the development of on-going capacity development services on the local market; and 4) Ensure that learning and methodologies are shared and applied by other development actors. LAVI is implemented across Liberia using an issues-based approach to address accountability and policy reforms in Liberia. LAVI proposes and implements a method for identifying, defining and selecting priority themes in collaboration with USAID and the Advisory Council. Once the themes have been selected, LAVI supports civil society organizations working on specific themes, and brings together different actors across sectors to strengthen the vertical and horizontal linkages among communities and organizations. Based on capacity development needs identified through assessments, LAVI assists each partner to develop and implement a capacity development plan. Capacity building interventions include but are not limited to specialized technical assistance, training, mentoring and coaching; establishment of communities of practice or peer-to-peer learning for information sharing, and efforts that match organizations with appropriate donors to meet continuous funding needs. In this way, the program is creating an enabling environment for information sharing and continuous learning by promoting the development of the local market for capacity development. INTRODUCTION The Civil Society and Media Whole-of-Intermediate Result (IR) evaluation was conducted by Liberia Strategic Analysis (LSA) for USAID. The evaluation process was a valuable learning opportunity for the LAVI team, allowing them to pause and reflect on the project’s performance, achievements, and challenges to date. LSA’s final report hits the mark in several respects, but in some instances, lacks the necessary context for evaluating LAVI activities objectively. This Statement of Differences has therefore been prepared by LAVI to clarify a few inaccuracies. Several sections of the final evaluation report are excerpted below. Passages highlighted in light blue represent the Differences that LAVI has identified. This is followed by a response with additional information or facts that were missed in making the key findings upon which the evaluation team based its conclusions. CONCERNING THE KEY FINDINGS 1. On Roman Page iii, Paragraph 3; Roman Page Viii, Paragraph 1; and Page 24, Paragraph 3: Critical gaps exist in making connections among the three stakeholder groups within governance accountability systems: (1) Political Leaders, (2) CSOs and Media Outlets, and (3) Citizens. A key priority for a future PAD would be to create interactive information and activity loops that better connect political leaders to citizens, using CSO coalitions and media outlets as channels through which citizens directly engage with government institutions. Comment on highlighted section: The project team would like to clarify that LAVI’s approach to implementation has focused on building CSO advocacy coalitions (e.g., the Natural Resources Management (NRM) Coalition, and the More-for-Education Campaign team among others) which directly connect with citizens, political leaders, media, and government institutions. The result has been meaningful engagement with and between 66 Legislators, nine political parties, seven media outlets, 11 Government Ministers, 39 Ministry of Internal Affairs (MIA) Commissioners, 19 MIA Superintendents, and 59 County & District Education Officers, among others. Refer to Annex 2: Legislators & Other Government Authorities Engaged by Coalitions. Somehow the methodologies of both the NRM and the More for Education CSO coalitions were not fully captured in the final evaluation report, specifically how they function to connect citizens with political leaders and media outlets. For instance, the NRM coalition is working to build an advocacy campaign that would achieve greater accountability and transparency in the management of County Social Development Funds (CSDF) through citizen-centered sustainable management practices; and to establish adequate and long-term mechanisms for effective and participatory CSDF governance. In 2016 the NRM coalition conducted a CSDF Review and comprehensive information-gathering exercise which targeted more than 2,000 citizens in all counties using a range of research methodologies including face-to face surveys and focus group discussions. This background research was done before seeking citizens’ views on reforming CSDF management. Out of those opinions and recommendations, the NRM CSO Coalition developed a strategy using four frameworks to guide their advocacy activity design, namely: a Policy Advocacy Framework; Inclusive Citizen’s Compliance Monitoring Framework; CSO Capacity Strengthening Framework; and a Media Capacity Strengthening Framework. Using these frameworks, the Coalition engaged citizens, legislators, and key members of government at the national and county level. As a result of this process, well-informed recommendations were made for amending Liberia’s Budget Law. In July 2018, five of the 17 amendments to the current CSDF law proposed by the Coalition were passed by the Legislature (2018-2019 Section (9) Budget Law). These amendments give citizens and local authorities more control of CSDF management and require greater transparency and accountability in the use of the CSDF. The NRM CSO Coalition’s legislative reform achievements were the result of a process they undertook to solicit input from relevant stakeholders which included: consultations with citizens; working with Ministry of Internal Affairs (MIA) and Ministry of Finance and Development Planning (MFDP) officials; hosting number of radio talk shows to generate wider media coverage around the CSDF reform agenda; hosting a meeting of local media outlet managers to provide them with an overview of NRM Coalition activities and goals, as well as background on CSDF reporting and coverage. The Coalition also printed and distributed 3,000 posters, 3,000 fliers, 35 banners and 600 t-shirts to CSOs, Community Based Organizations (CBOs), law makers, media, and community members to promote visibility of the CSDF policy reform campaign; organized CSDF Journalist Network reporting trips to cover NRM Coalition 95 activities; engaged with officials from 9 political parties on their thoughts and commitment to the CSDF policy reforms; and organized meetings with legislative caucuses, CSOs, and citizens in all 15 counties. Similar approaches were used by LAVI’s Education Coalition partners when they collaborated and implemented the “More for Education” advocacy campaign in 15 counties of Liberia. Their efforts yielded results fairly quickly, influencing an increase in the Government of Liberia’s (GOL) funding share for education from 14% to 15% of the FY 2018-2019 national budget. This represented a 7% increase in the amount of funding allocated to the education sector. The Coalition’s success was an outcome of a process that dramatically increased engagement between the “More for Education” campaign members, legislators, citizens, and other education stakeholders. Part of LAVI’s mandate and contractual responsibility is to measure the connectivity of Liberian civil society organizations (CSOs) to political leaders and citizens over time through three outcome performance indicators (see below table for baselines and targets). The performance data summarized in the table below demonstrates that the project is achieving the intended result – to strengthen multi-stakeholder partnerships to advocate for and monitor accountability reforms. LAVI 1: Quality of engagement between civil society and the government Disaggregation Baseline FY16 Target FY16 Actual FY17 Target FY17 Actual FY18 Target FY18 Actual FY19 Target NRM Coalition 65% N/a 65% 70% 61% 75% 73% 75% Education Coalition 46% N/a N/a N/a N/a 46% 46% 60% LAVI 2: Civil society’s satisfaction with their voices being heard in decision making and monitoring processes NRM Coalition 36% N/a 36% 41% 68% 46% 69% 70% Education Coalition 42% N/a N/a N/a N/a 42% 42% 65 LAVI 3: Increased capacity of CSOs to foster citizen engagement NRM Coalition 77% N/a 77% 82% 69% 85% 79% 85% Education Coalition 84% N/a N/a N/a N/a 80% 84% 85% N/A indicates that the period for collecting data for that particular thematic window had not yet arrived. This data is evidence of LAVI’s efforts, which included a series of multi-stakeholder meetings hosted by Coalition partners to strengthen linkages among legislators, GOL officials, CSOs, media outlets, and citizens. Performance was tracked by the number of multi-party meetings conducted (FY2017 = 310 meetings; FY2018 = 212; Q1 & Q2 of FY2019 = 188) and consensus-building meetings held (FY2017 = 164; FY2018 = 113; Q1 & Q2 of FY2019 = 6). These numbers included town hall meetings, district advocacy meetings, and 113 legislative engagement meetings (NRM/CSDF = 77; Education = 33). For further background, refer to Annex 2: Engagement Meetings with the Liberian Legislators. 2. Roman Page viii (Executive Summary Section), Paragraph 3: A prominent missed opportunity is the lack of citizen engagement in the CSDF process. Focus group information indicates that a significant gap remains in engaging community members in the CSDF process, with participants having little or no information about County Sittings and Project Management Committee decisions. Comment on highlighted section: The final evaluation report states a lack of citizen engagement in the CSDF process as a prominent missed opportunity. This conclusion is inconsistent with the quantitative and qualitative evidence available. In FY2017, the NRM CSO Coalition trained more than 1,800 community representatives in advocacy and their rights related to concession agreements - of which several groups initiated advocacy actions. Additionally, the Coalition reactivated 25 District Development Councils (DDCs) and 15 Community Monitoring Groups (one per county) to engage community members in monitoring local projects funded by the CSDF. It was citizens who took responsibility for ensuring that existing mechanisms for local project monitoring were operational and functioning well enough to follow up on CSDF issues. For example, when President Ellen Johnson Sirleaf’s visited Pleebo, Maryland County, citizens of that County requested her intervention in the implementation of the agreement made with Maryland Oil Palm Plantation to employ people locally (LAVI Annual Report FY2017). A second example took place in the entire period of Quarter 1- FY2019. LAVI staff and the Field Based Monitoring Consultants (FBMCs) individually monitored twelve (12) County Sittings using a checklist to track if they were being implemented in line with the amended Budget Law. The Sittings were held in Montserrado, Bomi, Bong, Grand Bassa, Cape Mont, Grand Kru, Lofa, Margibi, Nimba, Rivergee, Sinoe, and Rivercess counties. Major observations drawn from the County Sittings included massive attendance on the part of CSOs, demonstrating that citizens were galvanized by, and making use of, the CSDF reforms achieved by the NRM Coalition. Another observation was that the lawmakers attending showed willingness to address the considerations of citizens presiding over County Sittings over the Caucus Chairs of the legislature. For example, Grand Cape Mount and Montserrado Counties allowed delegates to preside over the Sittings, rather than the legislators. Such a gesture indicates other legislators are likely to relinquish their presiding role to citizens, as proposed in the Draft Stand-alone CSDF bill. LAVI agrees that participants in the focus group discussions during the evaluation exercise may have had little or no information about County Sittings and Project Management Committee decisions. County Sittings are development agenda planning meetings, held annually per fiscal year to allow citizens, county lawmakers, and local government officials to decide on how the county allotment will be utilized. However, not all citizens attend those meetings; just those selected by the Superintendent Council in consultation with the Legislative Caucus. To promote citizen engagement, LAVI worked with local partners to ensure there was massive publicity in advance of every County Sitting. (Ref: Nimba Prepares to Hold First County Council Sitting in Three Years). A list of documents, including LAVI’s quarterly and annual reports were provided to LSA’s evaluation team, however the final report does not recognize the many instances that have been documented of citizen engagement in the CSDF budget law amendment process. For example, citizens are now involved in the selection of Project Management Committees (PMC); the management of CSDF during County Sittings; and the removal of PMC members as the budget law stipulates. Throughout all phases of the CSDF amendment process, multiple citizen engagement meetings were held by the NRM Coalition to solicit citizens’ views on the Budget Law amendment process and the subsequent drafting of the CSDF Stand￾Alone bill, which currently awaits passage. 3. Roman Page viii, Paragraph 1; and On Page 23, Paragraph 2: LAVI and LMD have yet to incorporate a more holistic definition of ‘sustainability’ that encompasses all those management functions critical to the ability of organization to operate effectively. CSOs require effective implementation of strategic planning, project management, human resources, financial management, and revenue-generation planning systems in order to operate beyond LAVI support. Comment on highlighted section: The word sustainability in the context of the Evaluation Scope of Work (SOW) refers to “what extent is the Civil Society and Media portfolio building the individual and collective capacity of organizations to sustain their work beyond USAID support” in the context of transitioning from donor-driven to a more locally- 97 driven (and resourced) model. LAVI’s sustainability objectives are focused on (1) strengthening CSOs’ organizational systems and structures to advocate for policy reforms; (2) supporting the development of CSOs’ institutional policies and practices in support of policy reforms; and (3) developing CSOs’ staff skills in key technical advocacy areas. The work of LAVI’s Capacity Development Team under Objective 2 and 3 of the LAVI Results Framework was designed to help CSOs and local service providers learn how to perform high-quality advocacy and policy reform support. It is accurate to say that institutional development of these organizations to achieve broader operational effectiveness and financial sustainability was never a part of LAVI’s Scope of Work or mandate. Rather, the theory of change for this project is that our partners will retain the skills, knowledge, systems, and networks to continue performing effective policy advocacy beyond the LAVI intervention, learning from these experiences to adopt and adapt new strategies well after the project ends. The extent to which our partners institutionalize these capabilities and continue delivering valuable services to their clients and constituents will impact their ability to attract and utilize direct funding from donors and investors, which is a requirement for sustainability. The specific ways that LAVI does contribute to organizational sustainability is summarized below: LAVI assesses the institutional capacity of all its partner CSOs to identify gaps within these organizations and provide capacity building support in the areas of Advocacy, Monitoring and Evaluation, Leadership Management, and Financial Management. LAVI has also worked with local firms and Subject Matter Experts to form a Business Consortium of Service Providers which will exist beyond LAVI to provide high-quality capacity development services to CSOs and government institutions. In addition, LAVI supports A-Lab and I-Lab to manage I-Campus as the place where learning activities under Objective 4 are implemented. As part of their support, LAVI is helping them develop a business plan for generating income through renting of rooms and fees for their services. 4. Roman Page viii, Paragraph 2; and On Page 23, Paragraph 4: LAVI training of CSO partners has improved implementation of coalition interventions and has generated progress upgrading some internal management systems. However, significant training on developing advocacy strategies and management practices would allow CSOs to be more effective in implementing their advocacy and government oversight functions. Comment on highlighted section: We agree with the report’s conclusion that LAVI training of CSO partners has improved implementation of coalition interventions and has generated progress in upgrading some internal management systems. However, the statement above does not fully capture the fact that LAVI has conducted several capacity development trainings on advocacy strategies and management practices for its partners, and continues to identify capacity gaps and provide necessary training to fill them. The Institutional and Advocacy Assessment (IACAT), which was conducted for all LAVI-supported CSOs, primarily focuses on strategic management; project management; monitoring and evaluation; financial management; advocacy goals, plans, and strategies; conducting advocacy; advocacy avenues; and public policy processes. The capacity development support plan is co-designed with LAVI partners where they have an opportunity to prioritize which areas will receive support. In general, CSOs and local partners in Liberia prioritize implementation of project activities and fundraising over institutional capacity development. Significant and longer trainings, which should include the organization’s executives, are in most cases impossible because staff simply do not have time for it. Additionally, there is limited capacity in Liberia of CSOs to receive and implement training. LAVI has worked closely with USAID to design activities that have meaningful and sustainable impact, consistent with the realities on the ground at the current stage of CSO development. 5. Under Evaluation Question (EQ) 1 and 1a: EQ 1: TO WHAT EXTENT DID THE PROGRAMMATIC AND CONTEXTUAL ASSUMPTIONS OF THE CIVIL SOCIETY AND MEDIA DEVELOPMENT HYPOTHESIS ADDRESS THE UNDERLYING BARRIERS TO ACCOUNTABLE GOVERNANCE OF PUBLIC GOODS AND SERVICES? On page 8, paragraph 3: Although there are examples of LAVI activity gains through policy changes—including potential passage of the CSDF bill and increases to the national education budget—KIIs described slow progress in regard to government accountability and delivery of public goods and services. Underlying barriers are deeply political and rooted in power dynamics, elitism, and a system of patronage. EQ 1A: DID THE NEW CIVIL SOCIETY AND MEDIA PAD DESIGN IN 2015 IMPROVE USAID/LIBIERA’S APPROACH TO ACHIEVING IR 1.4? On page 9, paragraph 4: In addition, there is limited evidence that citizen engagement has produced influence over governance of public goods and services. Over the next few months, potential passage of CSDF reform legislation and a new increase in the education budget may help to demonstrate to citizens that the CSO coalition approach can increase accountability and responsiveness from the government regarding delivery of services and management of public goods. Comment on highlighted sections: Concerning the two sections on page 8 and 9 highlighted above, LSA’s report does not take into account the fact that the NRM Coalition scored a major policy victory in July 2018 by amending sections of the current Budget Law. This achievement is distinct from the goal of passing a proposed CSDF Stand-Alone bill, a fact which was pointed out to the evaluation team during the draft report review period. The 2018- 2019 Amendments of Section (9) of the Budget Law by the Liberian Legislature bring new transparency and accountability to the use of the CSDF, thus reducing the underlying barriers to accountable governance of public goods and services in the long-term. Potential passage of the CSDF Stand-alone bill continues to be pursued by the NRM Coalition in order to permanently adopt the changes in the Budget Law, which is voted on every year. This strategy is an example of ‘step by step’ reform, which is realistic in the Liberian context (i.e., passing amendments to the Budget Law, Section 9 to build momentum for the CSDF Stand-alone bill, which will institutionalize and expand those reforms). Meanwhile, the impact of these gains has started to manifest gradually; for example: citizens are presiding over the County Sittings unlike in the past; CSOs are participating in the County Sittings to monitor the process; and Superintendent Councils are conducting extensive publicity about the County Sittings. During FY 2018, for example, upon the passage of amendments to the Budget Law, 14 out of 15 counties held their annual County Sittings. This is a significant improvement from before, when several counties had not had County Sittings for more than two years (Refer to story: Nimba Prepares to Hold First County Council Sitting in Three Years). Likewise, the report did not recognize the Education CSO coalition partners’ efforts and achievements when they collaborated to implement the “More for Education” advocacy campaign in 15 counties of Liberia. 6. Page 10 (under EQ 1B) Overall, the strategies outlined by USAID in the 2015 Civil Society and Media PAD contributed to increasing opportunities where CSOs and media outlets can play the “watchdog” role over governance. However, evidence is lacking that (1) CSOs can replicate the coalition approach beyond current USAID 99 support, (2) CSOs can use the coalition approach to mobilize citizens to directly engage with government decision makers, and (3) media outlets can increase the quality and quantity of news and information that citizens receive about critical public issues. As a result, the ability to engage citizens directly in oversight of, and advocacy for, public goods and services has not yet been consolidated. Citizens have yet to see tangible results from CSDF advocacy efforts that validate the effectiveness of CSO coalition campaign interventions and provide evidence that citizen engagement can influence governance. Comment for highlighted section: LAVI would like to clarify that the coalition approach is the current trend adopted by many CSOs in Liberia, evidenced by the increase in the number of coalitions being formed around different thematic areas. Current evidence includes the NRM Coalition; More-for-Education Coalition; Liberia Election Observation Network; Election Coordination Committee; and the Decentralization Platform. We have recently learned that several organizations working in health have formed a coalition mimicking More for Education to advocate to the GOL for an increase in the budget allocation for health. In addition, the final evaluation report fails to recognize the NRM CSO coalition efforts. As already indicated above, in July 2018, the NRM Coalition amendments for reform to the CSDF law in the 2018- 2019 Section (9) Budget Law were passed by the legislature, giving citizens and local authorities more control of CSDF management, and providing new transparency and accountability to the use of the CSDF. The amendments created formal opportunities for citizens to be engaged which did not exist prior to that. LAVI FBMCs visibly observed evidence of this during the most recent County Sittings held from August to December 2018, compared to prior calendar years. For example, County Superintendent and the County Superintendent Council, in consultation with the Legislative Caucus, are now in charge of convening the County Sitting meetings where decisions are made on which CSDF projects to fund. Previously, this power rested entirely with the Legislative Caucus. Supporters of the change thought that arrangement gave too much control of the process to lawmakers and not enough to the community. Delegates to the County Sitting are now selected by the Superintendent Council in consultation with the Legislative Caucus, which gives local officials the power to decide who will be invited. Previously this was the responsibility of the Legislative Caucus. The law also says, for the first time, that delegates should reflect a broad representation of the community as determined by the local administration of the district. Members of the County Sittings now have the authority, by a two-thirds vote, to remove PMC members who they believe are not performing satisfactorily or have been involved with corruption or improprieties. The Legislative Caucus had that power before. This amendment promotes independence and reduces the risk of PMC members being subjected to legislative influence (refer to Annex 1). 7. Under EQ 1B – Page 11, paragraph 2: While proposals of reforms have reached the national legislature, LAVI-supported coalition interventions have yet to successfully result in changes to current legislation and the national budget. Potential final results (passage of legislation and budgets, followed by implementation) will help to demonstrate to citizens that the coalition approach can contribute to accountability and responsiveness from government regarding delivery of services and management of public goods. KIIs with coalition CSOs showed that there is not yet a public outreach strategy for combining civil society advocacy and media outlet reporting efforts that inform and engage citizens on implementation of potential reforms—for example, in the summer of 2019 when both the CSDF bill and a new education budget may be passed by the legislature. Comment for highlighted sections: We suggest the evaluation team revise the statement above. Coalition interventions resulted in amendments to the Budget Law in 2018, a 7% increase in the education sector budget and a resurgence of the County Sittings with greater citizen participation. In addition, public outreach has been conducted on the amended Budget Law, Section 9 in collaboration with Internews/LMD. 8. Under EQ 2: HOW MUCH HAS THE CIVIL SOCIETY AND MEDIA PORTFOLIO CONTRIBUTED TO MORE EFFECTIVE, ACCOUNTABLE, AND INCLUSIVE GOVERNANCE IN LIBERIA AT THE DO LEVEL? WHAT ELEMENTS WORKED OR HAVE NOT WORKED, AND WHY? On page 12, paragraph 2: The coalition found champions in the legislature for amendments around the Education Reform Act, but results have not yet been consolidated, with a lack of information about how local advocacy efforts in counties are contributing to national-level engagement on the Act. Successes by Year 4 include lawmaker efforts to review and pass the CSDF accountability legislation, which would potentially allocate funds from concessions directly to county-level project management control and oversight of CSDF projects, as well as legislative efforts to increase the national budget for education by as much as 20 percent for the next fiscal year. Both efforts remained in progress as of March 2019. Comment on highlighted sections: The Education Campaign conducted by the More for Education Coalition has already led to an increase in the education budget allotment of the national budget, to 20% by year 2020, and in holding the GOL accountable for implementation of the already existing 2011 Education Reform Act by the Ministry of Education. The More for Education advocacy campaign team’s legislative engagements created a space for LAVI partners to engage and develop good relationships with influential lawmakers. (Ref: More-for-Education Mid-Term Evaluation 2019). For instance, UMOVEMENT, one of the Education Coalition’s members, was able to rapidly obtain signatures on pledge cards for the 20% increment from both Senators and Representatives of Montserrado, Sinoe, Nimba, Margibi, and RiverCess Counties. The pledge cards demonstrated a lawmaker’s commitment to working with their colleagues in the Legislature on this reform, helping to raise awareness about the campaign and increase buy-in. As a result, political will for this reform has increased and some lawmakers, most especially those on the influential Ways, Means, Finance, and Judiciary of both Houses (i.e., upper and lower) advocated for the 20% increment during the budget formulation hearings. We recommend again that the evaluation team include passage of the Amendments to the Budget Law, Section 9 as a success in the report (Refer to evidence: Legislature Amends Budget Law to Improve Accountability). 9. Under EQ 2B: WERE THERE MISSED OPPORTUNITIES? On page 18, Paragraph 2: The LAVI Education Coalition has yet to develop an effective engagement strategy for senior government leaders not already associated with education as an issue, such as the senior national legislature leadership critical to facilitating all legislative debate in both houses. Certain House Representatives have been brought into public awareness activities on education issues; however, the legislators targeted were primarily limited to those already focused on education. Other senior executive branch officials and legislature leadership have yet to be engaged. Comment on the highlighted section: This statement is inaccurate. The LAVI-supported Education Coalition has engaged up to 63 lawmakers/legislators in both Houses (i.e. Representatives and the Senate), which exceeds the number of members of the House Committees on Education (Refer to Annex 2). Additionally, the Education Coalition engaged other senior government officials from the Executive; including Ministers and Deputy Ministers in the Ministry of Education; Ministers and Deputy Ministers in the Ministry of Internal Affairs; and senior officials of the Ministry of Finance and Development Planning. 101 10. On page 18, Paragraph 3: KIIs with private sector firms having provided capacity-building services to LAVI-supported CSOs also indicated that there has been no attempt to analyze and understand how to incorporate the private sector within civil society advocacy efforts. Stakeholders interviewed suggested the potential to include private sector associations into thematic areas that address corruption or promote a better business and enabling environment for economic growth. Engagement and strategic thinking around private sector engagement are inconsistent. Service provider stakeholders described a significant opportunity to influence the private sector by advocating for changes that add value to accountability measures, including reforms surrounding employment practices, procurement, or grand corruption. Comment on the highlighted section: LAVI by design is not a private sector-driven project; and private sector entities in Liberia are hesitant to engage in political activities that could potentially threaten long-term business interests with the GOL. The exception is LAVI support to Coalition advocacy efforts through the Consortium of Business Development Service Providers of Liberia (CBDSPL). Private sector businesses who are members of this consortium provide institutional and advocacy capacity building support to Coalitions and their members. And in certain instances, LAVI has reached out to concession companies to supplement the advocacy efforts of its Coalition partner projects. On page 18, Paragraph 4: The focus on developing national-level CSO coalition and advocacy skills within the LAVI-supported coalition-building effort missed the opportunity to reach down to the traditional family level, with common citizens unaware of greater advocacy efforts. For example, some families do not allow their children to attend school, and CSOs cited poverty, tradition, and vulnerability as also preventing parents from acknowledging the relevance of education. Activities within the education advocacy campaign did not take such family apathy toward schooling into account. The focus on policy or the national budget fails to meet community-level concerns, with stakeholders citing a need to include both the national and community￾level contexts when developing a coherent strategy to both reach impoverished citizens to increase demand for education and seek an increased national education budget. Comment on the highlighted section: LAVI’s national-level Education Coalition reached down to the traditional family level by training the Parent Teacher Associations (PTAs). Those that had limited knowledge regarding their roles and responsibilities in the school system received training, coaching, and mentorship from Education Coalition partners to strengthen their understanding of the roles and responsibilities of PTAs in the education sector, most especially in monitoring schools and families to ensure children attend school. As a result, there have been improvements in the relationship between PTAs and school administrators. County-based CSO Education partners have encouraged PTAs to be more effective in reaching out to families and recruiting or bringing students to schools. As a result, schools have started observing increased school attendance. In addition, students are no longer missing school to work in their teachers’ small agricultural farms because PTAs are monitoring the schools. Teachers now know that if students are not found in schools, the PTAs will follow up, report their absence, and track them down (Ref: More-for-Education Mid-Term Evaluation 2019). 11. On page 20, Paragraph 1: Establishing a functional strategy where CSO coalitions can directly engage with senior government legislators and executive branch officials remains a serious challenge to the ability of LAVI coalitions to affect national policies. Comment on the highlighted section: We would argue that a functional strategy to engage senior government legislators and executive branch officials does exist. LAVI-supported NRM Coalition partners developed an engagement strategy that lays out approaches and steps in identifying relevant stakeholders, both at the levels of the national legislature and the executive body of the government. Within the national legislature, strategic committee members relevant to the NRM sector were engaged firstly to amend Budget Law, Section 9 and to form legislative champions for the passage of the Draft Stand-alone CSDF bill. Similarly, at the executive level, the LAVI NRM Coalition engaged the Ministry of Finance and Development Planning and the Ministry of Internal Affairs, which are the two most important key players in the management and implementation of the CSDF (Please refer to Annex 2). 12. On Page 24, Paragraph 3: KIIs with CSO coalition partners and stakeholders show that for nearly four years, the LAVI activity has primarily supported CSOs to raise awareness on CSDF and Education thematic policy initiatives, and then to advocate for government reforms and policies that encourage greater response to those dedicated priorities. CSO coalition work has not yet adapted strategies to bring both senior government officials and citizens explicitly into the coalition effort. Comment on the highlighted section: Most of the multi-stakeholder dialogues, including county and district-level legislative engagement meetings held by the NRM and Education Coalition partners, bring together diverse stakeholders, including civil society leaders, youth, women, and policy makers such legislators, ministers, and government authorities to discuss common issues of concern in the NRM and Education sectors. LAVI’s decision to bring all of these actors into policy discussion and dialogue was informed by the thematic window intervention strategy developed by each technical lead, which takes into consideration various stakeholders that are essential for advocacy efforts to be effective. One of LAVI’s objectives is to ensure that: “Horizontal and Vertical Linkages among Actors Engaged in Similar Issues are Strengthened - (Ref. LAVI AMEP, IR-1). LAVI measures its performance in this area by the “Number of multi-party meetings held to strengthen the horizontal and vertical linkages amongst partners.” To collect this data, the project tracks the number of policy meetings supported and which stakeholders take part (e.g.; senior government officials, citizens, etc.). During the evaluation LAVI reported 522 meetings had already been convened for the purpose of strengthening collaboration between partners, getting them to hear and respond to one another, and come to agreement on strategies for moving forward. See data below: LAVI 1.2: Number of multi￾party meetings held to strengthen the horizontal and vertical linkages amongst partners Disaggregation Baseline FY16 Target FY16 Actual FY17 Target FY17 Actual FY18 Target FY18 Actual FY19 Target Parties 0 0 0 20 310 200 212 230 103 INTERNEWS 105 107 U.S. Agency for International Development – Liberia 502 Benson Street Monrovia, Liberia