1 CIVIL SOCIETY AND MEDIA LEADERSHIP (CSML) PROGRAM MID-TERM EVALUATION REPORT OCTOBER 2012 October 2012 This publication is made possible by the generous support of the American people through the United States Agency for International Development (USAID/Liberia) under the Liberia Monitoring and Evaluation Program (L-MEP) implemented by The Mitchell Group, Inc., Contract Number 669-C-10-00-00181. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Mid-term Evaluation Report CIVIL SOCIETY MEDIA LEADERSHIP (CSML) PROGRAM Implemented by International Research and Exchanges Board (IREX) Evaluated by Liberia Monitoring and Evaluation Program (L-MEP) Submitted to USAID October 2012 Evaluation Team: Robert J. Groelsema, Team Leader Denis J. Hynes, Training Specialist Karen Louise Boothe, Media Specialist Kanio Bai Gbala, Civil Society Specialist ii Table of Contents Executive Summary ...................................................................................................................... vii Introduction and Development Problem ......................................................................................... 1 Theory of Intervention ................................................................................................................ 1 Project Design ............................................................................................................................. 2 Purpose of the Evaluation ............................................................................................................... 4 Research Design and Evaluation Methodology .............................................................................. 4 Findings, Conclusions and Recommendations ............................................................................... 6 Result Area 1 (RA1): CSOs have measurably increased organizational capacity and culture to serve as a vehicle for representing their members’ interests ....................................................... 6 Findings ................................................................................................................................... 6 Conclusions ............................................................................................................................. 8 Recommendations ................................................................................................................... 8 Result Area 2: Advocacy CSOs in 7 counties have the capacity to monitor government policy, performance, and expenditures. .................................................................................................. 9 Findings ................................................................................................................................... 9 Conclusions ........................................................................................................................... 11 Recommendations ................................................................................................................. 11 Result Area 3: Perceptions of CSOs and government of each other are enhanced .................. 11 Findings ................................................................................................................................. 12 Conclusions ........................................................................................................................... 13 Recommendations ................................................................................................................. 13 Result Area 4: Independent Media operates more professionally............................................ 14 Findings ................................................................................................................................. 14 Conclusions ........................................................................................................................... 16 Recommendations ................................................................................................................. 17 Result Area 5: Community Radio Stations in 7 target counties engage with community, diversify listening audience and improve reach ........................................................................ 18 Findings ................................................................................................................................. 18 Conclusions ........................................................................................................................... 19 Recommendations ................................................................................................................. 20 Result Area 6: SMS technology is utilized as a link between people and their leaders ............ 21 Findings ................................................................................................................................. 21 Conclusions ........................................................................................................................... 22 Recommendations ................................................................................................................. 22 iii Result Area 7: The right to access to information is advanced through the project’s engagement with civil society and the media............................................................................ 22 Findings ................................................................................................................................. 23 Conclusions ........................................................................................................................... 24 Recommendations ................................................................................................................. 24 Result Area 8: Media is strengthened for transparent elections ................................................ 24 Findings ................................................................................................................................. 24 Conclusions ........................................................................................................................... 25 Recommendations ................................................................................................................. 25 Overall Media Conclusions........................................................................................................... 26 Project Management ..................................................................................................................... 26 Recommendations ..................................................................................................................... 28 Concluding Statement ............................................................................................................... 28 iv List of Figures Figure 1: CSML Targeted Counties ................................................................................................ 2 Figure 2: Cascading Training Delivered in Three Tiers ................................................................. 3 Figure 3: BAWODA Work Plan in Progress .................................................................................. 7 Figure 4: BAWODA Mission Statement ........................................................................................ 8 Figure 5: Advocacy Planning in Action.......................................................................................... 9 Figure 6: CRS in Sanniquellie – The transmitter was down and sent to Monrovia for repair...... 14 Figure 7: Diversifying Media Revenue Streams ........................................................................... 18 Figure 8: Publicizing FOI and the Right to Know ........................................................................ 23 Figure 9: Reporters on Wheels ..................................................................................................... 26 Figure 10: Poster in a Women's Empowerment NGO .................................................................. 28 List of Annexes Annex 1: CSML Training and Mentoring Process Annex 2: Mid-Term Evaluation SOW Annex 3: CSML Evaluation Inception Report and Evaluation Plan - Revised Annex 4: Final Draft Survey Instrument CSOs and Media Annex 5: Interview List Annex 6: CSML Partners Contact List Annex 7: CSML Logic Model Outcomes for Media Component Annex 8: CSML Logic Model Outcomes for CSOs Annex 9: Summary of Media Interviews Annex 10: M&E Results Chart Annex 11: Focus Group of 2011 Journalism Mentorship Graduates Annex 12: SOW Questions and Data Sources Revised Annex 13: List of CSOs and MOs interviewed with selection criteria Annex 14: Table of CSOs and CRSs with Selected Findings Annex 15: Call-in Segment on Super Bongese Radio Annex 16: Questions for RPs - YMCA v Acronyms ALICOR Association of Liberian Community Radio ANPPCAN African Network of Prevention and Protection Against Child Abuse and Neglect BAWODA Bassa Women’s Development Association BCC Behavior Change Communications CBO Community Based Organization CODRA Community Development and Research Agency COS Constituency Outreach System CRS Community Radio Station CSML Civil Society and Media Leadership Program CSO Civil Society Organization EI Extractive Industry FOI Freedom of Information Law GoL Government of Liberia HQ Headquarters Office HR Human Resources INGO International Non-Governmental Organization IP Implementing Partner IREX International Research and Exchanges Board IVA Inter-Visionary Artists KSAs Knowledge, Skills, and Abilities LIWOMAC Liberia Women Media Action Committee LMC Liberia Media Center L-MEP Liberia Monitoring and Evaluation Program LMTU Liberia Motorcycle Transport Union LWDR Liberia Women’s Democracy Radio MPs Main Partners M&E Monitoring and Evaluation MO Media Outlet MOU Memorandum of Understanding MT Master Trainer NARDA New African Research and Development Agency vi OD Organizational Development PANS Partnering, Advocacy and Networking System PIRS Performance Indicator Reference Sheet PMP Performance Management Plan RA Result Area RP Resource Partner SI Social Impact SOW Scope of Work TA Technical Assistance TCC The Carter Center TOT Training of Trainers USAID United States Agency for International Development WANEP West Africa Network for Peace building WONGOSOL Women NGO’s Secretariat of Liberia YMCA Young Men’s Christian Association of Liberia vii Executive Summary The Civil Society Media Leadership (CSML) Program is a 5-year $11.9M project implemented by IREX – the prime – and its partners Social Impact (SI) and The Carter Center (TCC). The Request for Applications (RFA) was issued in June of 2009; the award was made to IREX early in 2010. Start-up work began in the field in March 2010 and the project was officially launched on June 4, 2010. The project’s goal is to sustain peace in Liberia through greater inclusion, giving a voice to, informing, and engaging Liberian citizens. CSML encompasses eight result areas (RAs) and activity sets, which IREX and its partners implement in seven counties. The RAs range from civil society organization (CSO) and community radio station (CRS) capacity building in organizational development and advocacy to professionalization of the media and increased access to information. USAID/Liberia requested that the Liberia Monitoring and Evaluation Program (L-MEP) conduct a mid-term evaluation of CSML to assess progress and make recommendations for mid-course adjustments. The evaluation team (The Team) views the CSML program as indirect assistance delivered in three tiers through cascading training. Implementing partners IREX, TCC and SI deliver knowledge, skills, and abilities (KSAs) to four Resource Partners (RPs) and two media partners at the second level, who together constitute the Main Partners (MPs), and who in turn deliver training and mentorship to CSOs and CRSs at the third level. A robust grants component largely administered through RPs as part of their capacity-building permits the CSOs and CRSs to reach out to citizens, government, and member constituencies through forums, roundtables, town hall meetings, and public square activities, thereby putting into practice their training and mentoring. This approach, in the Team’s view, strengthens the sustainability of outcomes and impacts. We found ample evidence of capacity-building and institutional strengthening during our site visits and interviews with key informants. In the Team’s view, mentoring was the project’s distinguishing feature. The mentor’s role combines aspects of a collaborator, monitor, coach, and trainer/facilitator, which lends itself to moral engagement and professional reciprocity. We saw it as critical to reinforcement of training, and to the effective transfer of KSAs. Mentoring was transforming attitudes in the workplace about work culture, and the moral engagement evident between mentor and mentee was changing work habits, stimulating productivity, and building reciprocal professional relationships. Although no explicit central hypothesis of the project was articulated in the Program Description or the Performance Management Plan (PMP), it might be stated as: If organizational and advocacy training are reinforced by mentoring and practicums, and if this learning approach is delivered to targeted CSOs and media outlets (MOs) indirectly through Liberian Main Partners, then individual and institutional capacity will be built, strengthened, and sustained. Thus, a society characterized by exclusion, dependence, and helplessness will transform into one that is governed democratically, inclusive of all citizens – youth, women, and marginalized groups – who, in the aftermath of war, are becoming more involved and engaged in their political processes and whose voice is increasingly heard in important development policy debates regarding the future of their country. The Team employed a combination of methods for the evaluation, including a desk review of relevant project documents such as the RFA, the revised 2011 program description, quarterly reports, assessment reports, the capacity self-assessment tool, the 2011 elections report, and M&E materials. The document review served to inform the Team’s key informant interviews viii with USAID, IREX and its international partners, six MPs, 17 CSOs, seven CRSs, a commercial radio station, two newspapers, national and county government officials, and other project stakeholders across four counties. The Team also conducted three focus groups; a listener call-in program on CRS Super Bongese, in Gbarnga; and utilized its in-depth field observations at sub￾grantee and partner radio stations, offices and sites to draw inferences about the extent, pace and direction of change attributable to the project. The empirical and qualitative research was triangulated with statistical and quantitative data generated by the results of two citizens’ perceptions surveys of government and civil society, and CSML’s M&E data. Among the key findings are: 1) RA1 - increased organizational capacity at both levels and in both cohorts of Resource Partners (RPs) and CSOs, but CSML indicator data show mixed results; 2) RA2 – RPs and CSOs/CRSs are actively advocating on issues and monitoring government, however, this finding cannot be confirmed by CSML available indicator data; 3) RA3 - Despite improved information exchange between citizens and government, a deep mistrust of and lack of confidence in government persist; 4) RA4 – Media professionalism and independence are improving, but self-sustainability remains a distant prospect; 5) RA5 – Community radio is consolidating its role as a public service for democratic governance; 6) RA6 – SMS was highly successful as a real-time elections tool, but momentum to widen SMS media applications needs to be sustained; 7) RA7 - Increased assistance for GoL responsiveness to Freedom of Information (FOI) requests would enhance citizens’ perceptions of government; 8) RA8 – Elections investments were effective, but benefits – unless sustained - are at risk of dissipating prior to the next round of elections. Among key recommendations are: 1) RA1 – Establish project logbooks with copies of assessment tools, scores, and handouts; 2) RA2 – Increase grant support to monitor government compliance with the FOI, county development funds, extractive industry (EI) royalties, and other budget lines; 3) RA3 – Involve government in public information and constituent services training and create awareness among CSOs about government functions; 4) RA4 – Develop income generating potential and further revise the Media Monitoring Checklist; 5) RA5 – Track media attrition rates and trends; 6) RA6 - Monitor the gender gap of SMS use and visualize new media for the long-term; 7) RA7 - Monitor and report on FOI requests and outcomes; 8) RA8 - Promote women journalists to cover political beats and ensure their mobility on the road. To conclude, some of the team’s recommendations may fall outside CSML’s SOW, but could be instructive for an extension or follow on to the project. Five years are relatively few in number to address such extensive needs. Our observations and analyses point to staying the course with adjustments that enhance the effectiveness and staying power of the capacity building, emphasize and validate the role of women in media and civic engagement, and further connect citizens with their government through access to quality information and civic forums. This ensemble of activities will help build a social contract between people and their leaders. In sum, the project’s foundation is solid, the fundamentals sound, and change is trending positively. CSML is on track to achieving the desired objectives, but full data from the project’s M&E system is needed to connect all the dots. The evaluation mission has produced many possible recommendations, several of which were adopted by IREX and its partners during the recent CSML Mid-Term Partners’ Conference. Project implementers should ensure that M&E data collection and reporting are current, and ensure rigorous application of the capacity self￾assessment tool so that the project will be able to tell its story with statistical confidence and better manage for results. 1 Introduction and Development Problem The Civil Society Media Leadership (CSML) program (AID-669-A-00-10-00074-00) is a 5-year $11.9M project implemented by IREX – the prime – and its partners Social Impact (SI) and The Carter Center (TCC). The Request for Application (RFA) was issued in June of 2009; the award was made to IREX early in 2010; start-up work began in the field in March 2010, and the project was officially launched on June 4, 2010. The project’s goal is to sustain peace in Liberia through greater inclusion, giving a voice to, informing and engaging Liberian citizens and encompasses eight specific results areas. USAID/Liberia requested the Liberia Monitoring and Evaluation Program (L-MEP) to conduct a mid-term evaluation of CSML to assess progress made and make recommendations for mid-course adjustments. IREX revised its program in early 2011 to place greater emphasis on mentoring, reduce the number of CSOs, increase the number of CRSs, reduce the number of Resource Partners, and increase their grant amounts for more effective implementation of the program. Liberia has emerged from 14 years of armed conflict and a long legacy of human rights violations; the country faces significant challenges in achieving a balance between government and non-state actors with respect to governance. The 2003 Comprehensive Peace Agreement carved out an unprecedented role for civil society and produced many frameworks, including the Liberia Governance and Economic Management Assistance Plan (GEMAP) and the Truth and Reconciliation Commission. Significantly, for the first time in Liberia’s history, civil society organizations were allotted senior posts in the National Transitional Government of Liberia (NTGL) and seven seats in the National Transitional Legislative Assembly (NTLA). This arrangement gave civil society the dual role of leading the governmental reform effort from within as well as advocating for policy changes and greater accountability and transparency from the outside. As underscored by the Request for Applications (RFA), one of the many challenges of post￾conflict reconstruction is to re-establish trust in governmental institutions. Civil society, though far from robust, needs to contribute positively to Liberia’s democratic transition. Civil society and the media can effectively deliver civic and voter education. Their development is vital to rebuilding citizens’ trust in government and the political processes in which they should be the primary participants. Theory of Intervention In view of the foregoing, the stated objective of CSML is “Peace in Liberia sustained through citizen voice, engagement, information, and inclusion.” This objective contributes to USAID’s goal in Liberia, which is “Civil society and government are more effective partners in achieving good governance.” The project has identified two sub-objectives: 1) “CSOs have the capacity to foster democratic development and good governance,” and 2) “Independent media has the capacity to foster democratic development and good governance.” Although each of these sub￾objectives is important in its own right for building a democratic society, the Mission viewed them as interdependent with each relying substantively on the other to secure the dividends of peace, and to more fully and completely engage Liberians to support the post-conflict transition to democratic representative government. The selection of seven of 15 counties over a five-year period aimed to reinforce key USAID investments in the target counties, while developing a critical mass of CSOs and MOs with greater professional, organizational, technical, and sustainable capacity. 2 Project Design CSML’s two main components – civil society and independent media – are focused on fostering democratic development and good governance. In the original Results Framework (RF) there are seven Intermediate Results (IRs): three under the CSO component and four under the media component. In anticipation of the 2011 constitutional referendum and the national elections, USAID added an eighth IR, which strove to strengthen media for transparent elections. Under these eight IRs are 16 sub-IRs, seven under the CSO component, and nine under the media component. The project reports on 31 performance indicators, nine of which fall under the CSO component, and 22 under media. Five of the 31 are FACTS indicators. The project’s approach to achieving its goal is predicated on the transfer of KSAs through training and mentoring to some 60 civil society organizations (CSOs); 19 Community Radio Stations (CRS); one commercial and three non-profit Monrovia-based radio stations; one television station; and four newspapers across Montserrado, Grand Gedeh, River Gee, Grand Bassa, Lofa, Nimba, and Bong counties. Approximately 40% of the beneficiary organizations are located in Monrovia and the rest upcountry. Although the PMP did not articulate a central hypothesis, it might be stated as: If organizational and advocacy training is reinforced by mentoring and practicums, and if this learning approach is delivered to targeted CSOs and media outlets (MOs) indirectly through Liberian partners, then individual and institutional capacity will be built, strengthened, and sustained toward transforming a society characterized by exclusion, dependence, and helplessness, into one that is governed democratically, inclusive of all citizens – youth, women, and marginalized groups – who, in the aftermath of war, are becoming more involved and engaged in their political processes and whose voice is increasingly heard in important development policy debates regarding the future of their country. The Team views the program as indirect assistance delivered in three tiers through cascading training (Figure 2). Implementing partners IREX, TCC and SI deliver KSAs to Resource/Main Partners at the second level, who in turn deliver training and mentorship to CSOs and CRSs at the third level. Grants perform a facilitative role covering all project counties and nearly all partners. In the second year of operation, CSML made 45 grants to 22 CSOs for awareness￾Figure 1: CSML Targeted Counties 3 raising on the national referendum followed by grants to 23 CSOs to promote peace during the presidential elections. A large number of grants were scheduled for the third and fourth quarters of 2012. As part of their capacity building, RPs administer grants. In addition, CSML has made grants directly to five CSOs that were RPs before the number of RPs was reduced. The grants component helps CSOs and CRSs put into practice what they have learned through training and mentoring. They permit CSOs to reach out to citizens, government, and member constituencies through forums, roundtables, town hall meetings, and public square activities. Figure 2: Cascading Training Delivered in Three Tiers In the view of the Evaluators, this approach appropriately centers responsibility for development on the beneficiaries, builds their self-confidence and self-reliance, internalizes skills sets, and enhances sustainability. The mentoring helps make the training stick, and to the extent that mentoring is followed by practicums, that is, opportunities via grants or other means to apply knowledge and skills in the real world, it should result in stronger outcomes and more lasting impact. In February 2011, IREX proposed a revision to the CSML strategy, which USAID approved. The purpose of the revision was to focus limited human and financial resources more strategically for sustainable results. Hence, the program reduced the number of RPs to four and increased their grant amounts. The four RPs were WONGOSOL, YMCA, WANEP, and NARDA, which provided CSO leadership for women and youth, peace building and organizational development. The number of targeted CSOs was reduced from 120 to 60 to allow more time for mentoring and in-depth assistance. The number of CRSs was increased to 21 with RPs assigned for their organizational capacity building. The heart of the technical assistance delivery package is the mentoring system, which assumed greater prominence after administering organizational assessments and reviewing the history of prior training for RPs and Cohort I CSOs during 2010 and early 2011. The notion of mentoring surfaced in the RFA in the context of training journalists. IREX expanded the notion to include management training. Mentoring now constitutes the crucial technical and attitudinal assistance vehicle to CSOs and CRSs under the CSML project. Grant support for the RPs provides for one Master Trainer and two mentors. All counties except River Gee have one half-time position or more for a resident mentor, supplemented by Monrovia-based mentors who travel to counties. Mentors participate in CSML trainings to RPs; they are further trained in-house in facilitation International Partners IREX Carter Center Social Impact M&E Organizational Development Training, Mentoring, Grants Resource Partners LIWOMAC-Media Partner LMC-Media Partner NARDA-Resource Partner WANEP-Resource Partner YMCA-Resource Partner WONGOSOL-Resource Partner Organizational Development Training/Mentoring/ Grants CSOs/Media Outlets 60 CSOs 19 CRSs 4 Commercial Radios 1 TV Station 4 Newspapers Linking, Networking, Forums, Advocacy with Citizens, Communities and Government 4 skills and in mentoring, which can be understood as a technical role that combines features of a collaborator, coach, monitor, and trainer/facilitator. As reinforced by the grants component, this approach has proven effective in strengthening CSO capacity. IREX and its implementing partners offer capacity building approaches from standard process or management training, which relies heavily on experiential learning techniques, to technical training in areas such as policy advocacy and gender sensitivity, to roundtables and forums which promote a more comprehensive understanding and awareness of government policies and activities by citizens. The core strategy is to strengthen RPs to a level where they will be able to continue providing benefits for the CSOs that they are mentoring beyond the life of the project. Purpose of the Evaluation Per the evaluation SOW, the purpose of the evaluation is to assess progress toward the project’s expected results. The findings of the evaluation are intended to help inform USAID, the prime and sub-partners as well as other stakeholders on how well the project’s planned activities and strategy are contributing to the achievement of its expected results. The evaluation is meant to provide further guidance for the effective operation of the project over the remaining project period to help ensure the achievement and sustainability of results. Research Design and Evaluation Methodology The members of the evaluation team are: Karen Louise Boothe, Media Specialist; Denis J. Hynes, Training Specialist; Kanio Bai Gbala, Civil Society Analyst and Country Expert; and Robert J. Groelsema, Civil Society Analyst and Team Leader. Ms. Boothe has consulted on media programs in Africa, Eastern Europe and Asia, has implemented a major media - including radio - strengthening program in Kosovo, and has assessed the role of media in promoting transparency during elections. Mr. Hynes has conducted training as a Master Trainer for several NGOs and has some 40 years of experience implementing capacity-building development programs in West Africa. Mr. Gbala has a law degree from the Grimes School of Law in Liberia and currently oversees implementation of civil society programs for Trust Africa in Liberia. Dr. Groelsema, a specialist in democracy and governance (DG) and civil society programming, has led teams in conducting DG-related assessments and project evaluations in the U.S. and in Africa. The Team spent a total of six weeks on the exercise with four of those weeks in country (September 2012), including seven days upcountry. For this evaluation, the Team utilized a mixed methods research design featuring:  Desk review of relevant project documents including the RFA and original proposal, the revised program description of 2011, quarterly reports, assessment reports (including the capacity self-assessment tool for CSOs), the RF, the Performance Indictor Reference Sheets (PIRS), the Data Quality Analysis (DQA), and a study by IREX on the electoral process of 2011;  Key informant interviews with USAID, IREX and its implementing partners; six MPs; 17 CSOs and 9 CRSs; 1 commercial radio station; 2 newspapers; 1 CRS advocacy association (ALICOR); key national and county government officials; and select project stakeholders;  Focus groups – one with a women’s radio listener group in Buchanan, a second with 5 County Dispute Resolution Monitors (CDRMs) covering Bong and Lofa counties, and a third with graduates of the specialized consulting editorial training program for Monrovia-based journalists;  Listener call-in program on CRS Super Bongese in Gbarnga hosted by the Team’s Media and Civil Society Specialists;  Field observations at sub-grantee and partners’ offices, radio stations and project sites;  Review of the citizens’ perceptions survey of government and civil society conducted by L-MEP; and  Review of the CSML project’s M&E data and reporting systems. The data collection for key informant interviews, focus groups and on-site observations, was guided by five semi-structured questionnaires - two for key informant interviews with MPs, one for CSOs, one for CRSs, and one for a journalists’ focus group (see Annex 4). The CSOs we visited were engaged in various activities including child protection, youth federation, women’s rights, environmental protection, road safety, and the performing arts. Because not all project beneficiaries could be visited, the Team selected a representative sample based on six criteria (see Annex 13):  Type of CSO (e.g., umbrella, community-based, mission-driven);  Service delivery sector and demographic (e.g., health, advocacy, youth, women);  Geographic location across four of the project’s seven counties: Montserrado, Bong, Nimba and Grand Bassa;  Rating of the CSO/CRS by IREX as “high,” “moderate” or “low” capacity;  MP mentor organization of the CSO/CRS; and  Cohort 1 vs. Cohort 2 CSOs/CRSs. The evaluation encountered some limitations. Due to impassible roads, the Team was unable to visit all of the CSOs and CRSs selected. In some instances, the Team found a replacement, but in a few cases target entities had to be dropped, reducing the original sample from 20 to 17 CSOs. Because of the breadth of training and activities, not all interviews covered the same material with equal emphasis. In some instances not all mentors and mentees were available, and in some cases interviewees attended some but not all of the CSML trainings. Nonetheless, the range and the sum of data we collected on the background of the participating CSOs and CRSs, their organizational culture, their staffing tenure and turnover, their self-assessments, staff participation in trainings and mentoring, and in advocacy, watchdog, networking and partnering activities provided the Team enough information to draw inferences and conclusions that were reasonably generalizable to the project’s target population. The Team includes in the annexes three charts: 1) matching of selection criteria to beneficiaries; 2) alignment of the SOW questions with data type and source; and 3) progress by CSOs and CRSs. 6 Findings, Conclusions and Recommendations This section of the report addresses the questions posed in the evaluation SOW. It presents and analyzes our key findings, draws conclusions that interpret the findings, and offers actionable recommendations that follow from the findings and conclusions. We have structured the findings, conclusions, and recommendations by Result Area (RA) so that each of the eight RAs (also referred to as Intermediate Results) is addressed separately. Result Area 1 (RA1): CSOs have measurably increased organizational capacity and culture to serve as a vehicle for representing their members’ interests The RFA underscores the centrality of capacity building to CSML, which is measured in Results Area 1 by the OD capacity self-assessment tool. As noted in the RFA, studies on the Liberian CSO sector indicate that the low OD capacity of civil society is the main reason why civil society is not as effective as it could be. Therefore, the RFA recommends a program of “intensive capacity building” for CSOs across a number of components including leadership, strategic planning, staff development, financial management, constituency outreach and member service, advocacy, women’s advancement, coalition-building, networking, institutional sustainability, communications technology and new media. The Team’s view is that this RA is foundational for success in other areas since CSOs must first have the proper work attitudes, habits, and norms necessary to be able to perform other functions. Achievement of this result is tied directly to the effectiveness of the training and mentoring (T/M) components, and consequently we place a fairly substantive analytical focus on T/M. Findings The Evaluation SOW requested that the Team assess the progress of CSO organizational capacity building using the organizational assessments completed in year one as a baseline. The Team’s findings to specific questions follow. Using the organizational assessments of the CSOs completed in year one as a baseline, what progress has been made to strengthen CSO organizational processes and culture, advocacy, and networking? Scores on the second assessment (B2) – a year after the baseline (B1) - were inconclusive. Of the four indicators for which data were available, two showed slight declines from the baseline (B1), one showed mixed results and one showed a slight increase. These findings were somewhat at odds with the positive changes we observed in revised mission, vision and core values statements, personnel policy formulation, financial management, proposal preparations, and other organizational processes. According to IREX and its MPs, a probable explanation for the anomaly is that CSO staff inflated their responses on the B1 self-assessment to improve their chances for inclusion in the program. The Team posits that other threats to reliability could exist such as variance in B1 and B2 populations during administration of the tool. Only one CSO was able to show the Team its baseline assessment scores; none of the CSOs was conversant with the scores or results of the assessments. According to CSOs and CRSs, has the training provided by Master Trainers been effective to enhance their organizational competencies? CSO and CRS staff voiced their wholehearted appreciation for organizational training including budgeting, record-keeping, report writing, and proposal preparation. We received feedback from 7 several CSO executive directors that report writing by program officers had dramatically improved. According to CSOs and CRSs, has the mentoring they have received helped them enhance their organizational competencies? Personnel we interviewed unanimously believed the mentoring process had enhanced their organizational competence. The Bassa Youth Caucus said that financial management mentoring had helped them become more accountable to their funders, and an environmental protection NGO in Bong County, CODRA, said that CSML training and mentoring had helped them submit a successful proposal for funding. According to the RPs, has the organizational change mentoring provided to them by the IREX Civil Society Specialist provided them the skills required to be effective mentors? All of the RPs praised CSML for providing them mentorship skills. NARDA, the pre-eminent training organization, wants to retain its MT and mentors after the project ends, but expressed its concern at the lack of a clear exit plan to sustain its mentoring operations. To what extent have the CSOs implemented gender strategies and with what impact on gender integration? CSML has addressed gender sensitivity in its CSO and CRS training, which is visible in media news and information content, CSO/CRS hiring practices and retention of women. Of the 17 CSOs the team interviewed, 25% were women’s organizations, committed to women’s empowerment and gender integration. In the CSOs interviewed women occupied leadership staff positions, but still were outnumbered by men. The executive directors we interviewed acknowledged the importance of gender in service delivery, and said they were mainstreaming gender in their activities. The trend toward increasing female participation in the media is positive with the percentage of female journalists in Liberia increasing from 13/16% in 2010 to 22% in 2012. To what extent have the grants successfully been employed to support advocacy campaigns; human rights; peace building and reconciliation and gender awareness activities; youth activities; networking with other CSOs and with CRSs; work with local government; and other community driven projects? The grants program has enabled both the MPs and selected CSOs to implement advocacy campaigns and activities directly linked to the knowledge and skills acquired in CSML trainings. The major drawback to the grants program is the low financial ceiling for grants. NARDA and WONGOSOL indicated that their mentoring would be more effective if there were more grants to incentivize CSO mentees to put into practice their KSAs. Figure 3: BAWODA Work Plan in Progress 8 How has the project benefitted CSOs and CRSs outside of Montserrado County? CSOs and CRSs upcountry have participated equally with their Monrovia cohorts in training, although bad roads, travel time, poor communications, and limited budgets make it more difficult for them to receive the same amount of attention including constant contact through mentoring. Nonetheless, we saw examples upcountry of improved Vision, Mission and Core Values statements, a reconfiguration of an inactive Board of Directors, personnel policies where none had existed previously, a financial management system in at least one instance where none had existed before, and numerous instances of revised project documents as a result of CSML templates and mentoring on use of the templates. CSML has trained, mentored and equipped CRSs upcountry resulting in broader geographic reach and larger listening audiences. Conclusions CSML introduced program changes in Year 2, which reduced the number of target CSOs and emphasized mentoring as the key to reinforcing training concepts and to improving the sustainability of impacts. Although the statistical scores on the B2 capacity self-assessment are mixed in terms of indicating capacity building progress, we believe plausible explanations for those results exist, and we remain confident that our observations and interview data confirm organizational progress. We do question, based on reports from the RPs, whether current mentoring staffing levels and coverage are sufficient to meet the mentoring needs for the full complement of 60 CSOs and 19 CRSs, especially for the more remote CSOs and isolated radio stations that require intensive mentoring. Also, without a project logbook on-site to document assessments and to consolidate project training and mentoring, it is difficult for beneficiaries to understand, track, analyze and to set targets for progress in organizational and institutional capacity; it is also more difficult for IREX and its RPs to conduct M&E. Finally, given high attrition rates among women journalists, it is doubtful whether the trend toward gender parity in CRSs can be sustained. Recommendations Grants Management. For USAID/IREX - devote more project resources to grants. Introduce a training module on grants management followed by mentoring. Target grants at encouraging and retaining women CSO and CRS staff. Figure 4: BAWODA Mission Statement 9 Training of Trainers (TOT). For IREX - offer additional TOT training for mentors to address special challenges encountered by different types of CSOs, e.g., mass membership organizations vs. a mission-driven NGO. The OD needs for the Liberia Motorcycle Transport Union (LMTU) will be different from those of the Inter-Visionary Artists (IVA), both of which are mentored by the YMCA in Bong County. Log Book for Mentors and Mentees. For IREX/RPs - design and distribute a Project Log Book for all mentors and mentees. The Log Book should be a large binder. Contents would include calendars, copies of training handouts, assessment tools and scores, mentoring check lists, indicator collection sheets, targets and goals, project memoranda and training modules. Annual Calendar. For CSOs/CRSs - distribute a yearly calendar, month by month, with plenty of space for each day for each CSO and CRS site. The mere visual impact of this capacity-building tool for planning will enhance performance. More Mentors. For IREX - post an additional regional mentor to counties, and increase the number of mentors for CRSs. Utilize cutting-edge station managers as visiting mentors, e.g., Super Bongese, to advise CRSs on how to generate private sector revenue. Result Area 2: Advocacy CSOs in 7 counties have the capacity to monitor government policy, performance, and expenditures. RA2 aims to help civil society realize its potential to monitor policy development, implementation, and enforcement, which is critical for the growth of strong, accountable Liberian institutions. In the RFA, USAID deemed it important that local governmental authorities and national legislators be engaged by CSOs to improve policy recommendations and enhance collaboration between CSOs and lawmakers. The RFA recommended that national level advocacy organizations receive training to strengthen their organizational and advocacy capacity, and that civil society and media programming build on previous USAID investments in legislative strengthening. Findings The Evaluation SOW requested that the Team assess whether CSML had helped CSOs to advance their advocacy and networking skills and increase their effectiveness as government and policy watchdogs. The Team’s responses to specific questions follow: According to CSOs, to what extent has advanced capacity building in advocacy and networking provided to them increased their effectiveness as watchdogs in monitoring national and decentralized policy development, implementation, and enforcement? What further training do they say they need? The six advocacy CSOs interviewed provided a strong endorsement for the relevance of the policy advocacy training received under CSML, indicating that it has made a dramatic impact on their monitoring of county social development funds. One Cohort 2 CSO, the Figure 5: Advocacy Planning in Action 10 African Network of Prevention and Protection against Child Abuse and Neglect (ANPPCAN) said that it had launched an activity to protect Ivoirian refugees in Butuor as a direct result of a CSML June 2012 advocacy training. All mission-driven CSOs interviewed indicated that they found the tools, tactics, and approaches helpful to their understanding of how to build an advocacy campaign that gets results. The advocacy and networking training was particularly appreciated by CSOs upcountry that wanted to be able to engage local officials and legislators more effectively. What evidence is there that CSOs have been engaged in effective monitoring supports their progress as watchdogs? Especially in areas such as government expenditures and performance, gender based violence, and child trafficking? In 2011, several of the CSOs we interviewed played critical watchdog roles in the national referendum and the national elections. RPs and CSOs have taken a keen interest in the national government’s commitment to keep its 150 days deliverables promise. One ministry official said that as a result of LMC pressure, the President of the Republic called a cabinet meeting to admonish her ministers to comply with FOI. Ears for the Masses in Nimba County has collaborated with a CRS to pressure the mining giant, ArcelorMittal, and government to make public the details of mining contracts, and to account for $1.5M in annual revenue to Nimba County (see mini case study). A women-based NGO, United Muslim Women’s Advocacy and Empowerment Organization, was conducting meetings for some 500 members of all faiths across three counties to create awareness of human and civil rights laws and to educate women on their roles and responsibilities under the law. According to RP trainers, have CSOs developed the required skills to serve as watchdogs? RP trainers, most notably the MT at YMCA, praised the project’s policy advocacy training. The YMCA is employing an advocacy action plan to monitor progress of its CSO mentees. Although RPs said they thought CSOs were better able as a result of CSML to promote good governance, protect human rights, and to advocate for development in Liberia, they – in particular WANEP and NARDA - were concerned that the project had not devoted sufficient attention to institutional support and to sustaining the project’s actions beyond the LOP. From another angle, County Dispute Resolution Monitors (CDRMs), who receive training support under CSML, declared that CSOs, particularly in Bong County, have been highly proactive, and have given “no easy ride to MPs.” They had similar praise for community radio in Bong County. According to media specialists, have CSOs developed the required skills to serve as watchdogs? Media managers and journalists asserted that CSOs were becoming stronger watchdogs, especially on elections management and on monitoring government policies and promises. Media managers credited CSOs with having vision and mission statements that explicitly advocated greater transparency in public affairs, and that included protections for human and civil rights, including those of marginalized peoples. Media specialists we spoke with said that CSOs were being strengthened by their collaboration with media and that this dual approach was instilling greater public confidence in CSOs, and, to some extent, greater confidence in government. 11 Ears for the Masses Ears for the Masses, based in Sanniquellie and Radio Nimba (not a CSML partner) have been working together to create public awareness surrounding a 20% earmark of funds paid each year by the mining company ArcelorMittal to compensate property owners whose lands had been confiscated or adversely affected by mining operations. EARS broke the story on Radio Nimba that these funds – 20% of $1.5m for 6 years – had not been forthcoming, and demanded that government provide an explanation. The story first was aired in 2011 and was still being debated hotly. When we raised this issue with the Deputy Superintendent for Nimba County, he acknowledged the impact of the EARS advocacy campaign, and said that government was going to provide an answer. To what extent has CSO ability to network been strengthened by the project? The team found that networking was being encouraged through joint training, and through the CSO-CRS joint proposal grants. Conclusions The advocacy training provided by IREX and its partners appears to have strengthened CSOs’ capacity to monitor government policy, performance, and expenditures. CSOs’ activities bear out this finding, and the RPs and national- and county-level officials we spoke with acknowledged that CSML had been instrumental in this respect. However, of the three indicators that contribute to this RA (indicators 7, 8 and 9) only indicator 8, “% of CSO staff who correctly respond to questions regarding the Liberian policy making cycle” had complete data to support it, and the actuals for Year 3 of this indicator show nearly a 50% drop off from Year 2 actuals (71% to 38.9%). The project’s own M&E therefore, is not adequately supporting what the Team believes to be favorable progress toward CSO advocacy and monitoring of government performance. Recommendations Media Sensitization. For IREX - increase training to enable advocacy CSOs to better to utilize media as a public service ally, which could also help advocacy CSOs to tell their story more effectively. M&E System. For IREX/SI - update indicator data on monitoring CSO capacity; Grants. For USAID/ IREX - increase grant support to monitor government compliance with the FOI, to monitor the use of county development funds, and to monitor extractive industry (EI) contracts with GoL and counties. Sustainability. For USAID/CSML partners – plan for ways and means to sustain training and capacity-building activities beyond the LOP. Result Area 3: Perceptions of CSOs and government of each other are enhanced RA3 aims to gauge whether trust is improving in government and civil society. As noted in the RFA, one of the many challenges of post-conflict reconstruction is the re-establishment of trust in governmental institutions. A 2008 International Republican Institute (IRI) poll found that public confidence in government was limited to President Johnson-Sirleaf. CSOs, however, enjoyed a comparatively positive image and a degree of moral authority unmatched by government or politicians. To better link programming to perceptions, USAID requested CSML to conduct surveys of citizens’ opinions about government and civil society, and government leaders’ opinions about civil society and its role in a newly democratic, developmental, post- 12 conflict state. In addition to generating perceptions data, the survey questions contribute data to measure eight of the project’s indicators. CSML conducted the first survey in 2010; L-MEP conducted the second survey in July-August 2012. The evaluation SOW requested the Team to review the surveys as part of assessing overall progress. To enhance government capacity to represent community interests, The Carter Center’s Access to Justice Program provides partial support for TCC staff to work with CSML partners and the Ministry of Internal Affairs and the National Traditional Council to provide capacity building trainings for traditional leaders to improve linkages among traditional leaders, citizens and the government. The principal funding for this work stems from USAID’s “Strengthening Citizen Participation in Government: Access to Justice and Information.” Trainings include a focus on rule of law, conflict resolution, social accountability, and the role traditional leaders can play in advocating for the provision of basic social services in their communities. To date, the TCC has conducted more than 20 of these trainings in the target counties. The TCC has also worked with prominent female traditional leaders to convene the first regional conference for female traditional leaders (December 2011) followed by three regional events for women in 2012. Findings The Team did not interview chiefs, but it held a focus group with County Dispute Resolution Monitors (CDRMs), who cover five of the project counties, and who conduct three to four town hall meetings per chiefdom. The purpose of the meetings is to disseminate legal information and to air citizen agendas through open discussion about budgets, social services, security and land issues. CDRMs have received training from TCC in advocacy, law and government. According to the CDRMs, citizens are far more likely to have contact with traditional authorities than with their legislators since lawmakers spend as much as 90% of their time in Monrovia. The relatively weak capacity of customary institutions and county governments means that at this stage of Liberia’s transition public servants rely on assistance from the CDRMs to help educate citizens on the laws and rights and responsibilities under the law. The Team’s findings to specific questions are as follows: How effective has the training of county government officials on participatory democracy been to improve their perceptions of the role of CSO and the media? Without isolating variables, it is difficult to say how much impact the training has had on positive perceptions of CSOs and media by county government officials. According to the citizens’ perception survey, 84% of county officials have a positive impression of CSOs. County Dispute Resolution Monitors claim that training of officials is improving government’s understanding and perception of CSOs and the media. How effective has training of CSOs on participatory democracy been to improve their perceptions of the government? The citizens’ perception survey did not measure CSO attitudes or perceptions toward government. CSOs that the Team interviewed did not hold high opinions of government officials or praise government performance. What outstanding issues still need to be addressed by the project to achieve this result? 13 The Team recommends that the project direct more effort toward helping CSOs understand the mission, mandate and workings of government. Officials we interviewed in Sanniquellie indicated that they felt misunderstood by CSO personnel, who they thought had little notion or appreciation of the challenges they faced. How successful have Roundtables been to improve perception and collaboration? The roundtables have only recently been conducted, and more are in the planning stage. More time and further interventions are required to obtain a sense of their impact on perceptions and collaboration. The need for roundtables, town hall meetings and similar forums is critical, however. In our CDRM focus group discussion, the monitors asserted that the citizen-official relationship is still characterized by fear and is viewed as one of “peon” vs. “boss.” According to the monitors, most people did not see themselves as employers of public servants; rather, they were bound by a class mentality that prevented them from seeking access to their government. To what extent have citizens improved their perception of CSOs and government? Citizens have improved their perception of CSOs, but not of government. Sixty-seven percent (67%) of the population holds a positive perception of CSOs. Specifically, 69% expressed satisfaction in CSOs representation of their interests. Only 31% of the population in the seven targeted counties had a positive perception of county officials. To what extent have radio shows improved perceptions? The citizens’ perception survey indicated that 81% of citizens were being reached by CSML’s targeted CRSs. Although direct attribution is difficult to confirm, it is likely that news reports, community information, interviews, and call-in programs are contributing to improved perceptions. Our interviewees said they were getting more access to government owing to call-in programs. In Bong County callers on our show responded positively to Radio Super Bongese talk shows and call in programs saying that these venues gave them opportunities to interact with officials. One caller said, “It was like having town hall meetings on the air.” To what extent has education regarding access to public information improved perceptions? Anecdotally, education is having a positive impact on improved perceptions. The Deputy Minister of Information and county officials in Nimba had highly positive opinions about the effectiveness of CSML’s Access to Information program implemented by TCC. Interviewees confirmed that public service announcements, media exposure, and CSO advocacy were contributing to enhanced public awareness of FOI, and that such awareness contributed to improved perceptions. Conclusions Citizen/government perceptions of CSOs are highly positive. The challenge in the remainder of CSML will be to maintain such a high level of trust and confidence. Conversely, two out of three citizens distrust government meaning that a gulf separates the credibility of civil society and the public sector. At some stage this imbalance will need redress. Media appears to be performing a highly beneficial role in this respect. Recommendations To improve perception and trust, the L-MEP survey report recommended that 1) CSOs provide 14 regular awareness on their activities through community meetings, occasional newsletters, radio and other media outlets to ensure greater appreciation of their work; 2) CRSs be more attentive to listener interests; and 3) CRSs improve their reporting and editorial skills. The report also recommended that CSML require CSOs that receive grants to consult with and provide regular feedback to stakeholders as part of their grant requirements. We acknowledge the relevance of these recommendations, although some of them imply additional financial resources. As for improving perceptions of county governments, L-MEP recommended establishing governmental-CSO/CRS partnerships to disseminate county government policies and programs. Additionally, the Team recommends: For USAID/IREX – conduct public information and constituent services training for government; consider establishing a public information office (PIO) in each county. For IREX – work with CSOs to emphasize more balanced reporting on government performance. For IREX/TCC - create greater awareness among CSOs about government functions. Result Area 4: Independent Media operates more professionally RA4 aims to increase the capacity of MOs and professionals to provide reliable information through activities such as journalism training, media management, and business development. As the RFA noted, the democratic transition in Liberia opened up the freedom of the press. In the aftermath of war, newspapers and radio stations blossomed, but media were hampered by economic constraints, low pay, a shortage of skilled investigative journalists, and poor internet connectivity outside Monrovia and Buchanan. Low literacy, but the widespread use of cell phones opened up new possibilities to access information. Findings The Team was asked in the SOW to investigate the extent to which CSML initiatives have improved the professional operations of targeted media outlets, and what progress has been made in the professional and independent behavior of media. The Team’s observations confirmed that CRSs still lag in the area of business acumen; financial records are inconsistent; some CRSs have done audience research, but CRSs do not market themselves well to the business community. CSML has attempted to address these issues via trainings and mentoring. The Team’s findings to specific questions are as follows: To what extent has the specialized consulting, business development and sustainability initiative improved the professional operations of targeted media outlets? Figure 6: CRS in Sanniquellie – The transmitter was down and sent to Monrovia for repair 15 Some CRS managers believe that CSML training has improved their bookkeeping and proposal￾writing skills, and will help them win grants. One station remarked that it now collects and files sales receipts for purchased airtime, but we did not see the receipts and financial books were not opened to us. We see much room for improvement on the business-side of the media sector, such as marketing, sales, subscription services, private sector advertising, program sponsorships, and other revenue-generating endeavors. The same holds true for commercial radio stations and newspapers. Although they are selling more ad time and space, a large portion of those sales comes from the INGO sector, and they remain indirectly reliant on donor-support. The station manager at commercial SKY FM (commercial radio and TV) believes financial independence is a decade away. What progress has been made in the professional and independent behavior of media? Human resources trainings have helped media managers do a better job in recruiting and diversifying their staff. Some CSML trainees including women have been promoted to higher editorial levels in their media outlets. CRSs have made programming adjustments to better serve listener needs based on informal audience needs assessments. We heard that while media managers would like to generate more revenue, they rely on donors for “seed money.” What are citizen perceptions of independent media? We did not conduct a citizen perceptions survey of independent media. However, our interviews with CSOs and our caller program in Gbargna indicated that citizens hold positive perceptions of the media. In Buchanan, the NGO, BAWODA, sponsors a women’s radio club to encourage broader radio listenership among women. In Gbarnga, a waitress (in her 20s) at our guesthouse said she favors Super Bongese radio because she likes their music and the news and information programming. The 11 callers (all male) on our call in program on Radio Bongese said they have noticed improvement in the station’s performance in news and information programming as well as better linkage of listeners to government officials via call-in shows. The CSML referendum and elections activities appeared to have enhanced citizen perceptions of independent media. What progress has been made in the establishment of pilot subscription and distribution services? CSML is endeavoring to resolve transportation problems that hinder newspaper distribution outside Monrovia. However, little progress has been made in fostering the development of radio, on-line, SMS subscription news and information services. SMS in particular shows promise as a revenue-generating tool for broadcast, print, and on-line media within Liberia and across borders. Based on the on-line elections experience, the Liberian diaspora constitutes an untapped market. What progress has been made in the Women’s Media Initiative (WMI)? Approximately 400 women have been direct beneficiaries of the WMI: Scholarships (4), trainings of two national women media groups (200), mentees (17), and 4 community women groups (150). Media monitoring indicates growth in the number of women cited as sources (24%) in news and information reporting in both broadcast and print thanks to CSML gender￾sensitivity journalist trainings. However, women journalists constitute only about 13%-22% of all journalists in Liberia and only about 5% work in the higher editorial or senior reporting ranks in newsrooms. Women are less apt to be assigned to government/political reporting, are less likely to be given airtime or radio program slots with the highest levels of listenership or front 16 page above-the-fold space in newspapers. Women journalists/program hosts are contributing as little as 10% of the overall programming and written content on the editorial side. The LWDR station is completing a radio studio teaching lab that will help advance women in media. What progress has been made in the Young Journalists Initiative? This initiative has not progressed as expected. IREX recently signed a grant agreement with the University of Liberia Mass Communications Department, but must conduct outreach and recruitment before programming begins. Have the investigative journalism training workshops had an impact on improving the reporting skills of journalists? Journalists in CSML asserted that their skills in digging deeper than the usual press release have been enhanced by project training. One reporter stated, “My eyes were opened to printing stories in two or more parts.” In one instance, a reporter at the Public Agenda newspaper uncovered biased admission practices at JFK hospital that discriminated along tribal and religious lines. The outcome of this reporting was that the hospital removed discriminatory admissions questions from its admissions forms. Has the on-line media resources clearinghouse been used, and if so, what impact has this had on users in terms of expanding their knowledge and awareness and skills? The online media resources clearinghouse remains a work in progress. IREX explained that despite initial advances during the referendum and elections, low capacity and bureaucracy have slowed development of this tool. We saw efforts toward content development for the site, but the content population of the site is not yet complete. The clearinghouse has no revenue-generating business-development plan despite the potential for its success within and outside the country. Conclusions Evidence from the 2011 elections and from our interviews indicates that independent media are maturing and are operating more professionally. The Young Journalists Initiative represents the type of assistance that could promote media literacy and lay the foundation for a future generation of media content providers, technicians, engineers, IT specialists, and social media citizen journalists. Progress in economic self-sustainability is noticeable, but most media outlets, especially CRSs, are years away from financial independence. To be self-sufficient and financially independent, media will need to be able to pay family-sustaining salaries, maintain What Matters is Often Quantifiably Immeasurable A woman journalist at Magic FM was asked how either her way of working or the content of her work had changed following her participation in CSML trainings. Although she had worked at the CRS for more than 8 years, she said her interviewing skills had improved. When asked to elaborate, a story emerged that pointed to her newly found “voice” as a woman journalist. She said, “My self-esteem is higher. I realized through the training that as a journalist—as a woman—I have every right to ask questions of officials and expect answers when I interview them.” The interpersonal “shift” seemed transformational. The 17 expensive broadcast infrastructure and replace depreciating equipment. The problem is particularly acute among CRSs. One potential solution is to move further toward subscription￾based services, and market programs that specifically attract advertising and private-sector sponsorships. There remains heavy reliance among newspapers on the INGO community for income-generation from job announcements and procurement notices. Recommendations Leverage success. For IREX - utilize successful media managers, such as those at Super Bongese radio to mentor other media managers. Mentor marketing. For IREX - continue to teach and mentor media marketing, advertising sales, and promotion to media managers and assist them in skills-building to create more effective advertising and marketing schemes that are not INGO-generated. Monitor strategic planning and the business side of media. For IREX - require media outlets to open their books and work with mentors to discern trends in advertising sales, program sponsorship, and subscription-based programming. Expand Human Resources topics. For IREX - address the workplace environment and issues of sexual harassment, equal pay for equal work, and promotion of women in newsrooms. Re-energize the Young Journalists Initiative. For IREX - focus more clearly on fewer select individuals, and identify potential gender gaps. Women’s and youth initiatives could get high school youth, especially girls, excited about technology via activities such as engineering, and technology. One idea would be to assist girls’ radio listening clubs to build radio sets. The Young Journalists Initiative should lay the groundwork for the economic potential of this sector and jobs creation in technology, engineering, and IT. Further develop the media monitoring checklist tool. For CSML partners - monitor the gender of the program hosts and reporters on newscasts and programs. It is not enough to monitor the gender of sources cited in programming content; it is equally important to monitor whether women are contributing editorial content. Super Bongese Radio: An emerging success story Since 2005 Super Bongese has been serving the citizens of Bong County. It sells ads and is establishing a membership club composed mainly of businesses willing to pay 100-200 Liberian Dollars monthly in exchange for weekly mentions on “Fan Day” as well as other member perks. The station publishes a newspaper twice monthly featuring income-generating ads. Although some of its marketing strategies precede CSML, station leadership said the business and management training they have received from CSML is sharpening their business and marketing skills. Super Bongese now markets a weekly health program to the local hospital, which pays to sponsor the show. A “trade-out” with Cellcom allows Cellcom to pay for fuel to operate the station while the station airs advertisements for Cellcom. Nonetheless, the station remains reliant on donor funding to pay staff salaries. 18 Expand the women’s scholarship fund. For USAID/IREX - include a first-year paid apprenticeship at a media outlet in an outlying area so that skills and knowledge are shared in outlying media outlets and opportunities to gain professional experience in a smaller market are enhanced. Educated reporters will help build professional capacity at CRSs. Track attrition rates, especially of women. For IREX - determine Who, What, Where, When and Why media professionals are leaving the sector and where they are going to work. Replicate the Monrovia-based media-monitoring group in other counties. For IREX - make the same benefits available in Monrovia to outlying areas; foster IT interest among girls to avoid a potential gender gap in technological/IT jobs for women in the media. Establish a business plan. For IREX - continue to develop the on-line media clearinghouse with content that is subscription-based, and marketable to other media, CSOs, philanthropic interests and business/private sectors, both locally and internationally. Result Area 5: Community Radio Stations in 7 target counties engage with community, diversify listening audience and improve reach RA5 aims to support the organizational and financial sustainability of 19 CRSs to help them become model stations in the project’s operating areas. The objective is to enrich CRS program content and expand coverage, enabling CRSs to reach isolated communities. Importantly, the RFA underscored that the purpose of community radio is to foster local ownership, to share information pertinent to a geographic area, including public service announcements of general interest, to patch in news from services to give local communities a connection to the outside world, and to provide a forum for community discussion and debate. Toward this end, the RFA recommended procurement of equipment and production assistance for CRSs. Findings The evaluation SOW asked the Team to explore questions ranging from the extent to which model radio stations were operating more effectively because of IREX support, training, and mentoring, and to what extent the grants were impacting radio station capabilities. The Team’s findings to specific questions are as follows: To what extent are the model radio stations operating more effectively on behalf of their audiences because of IREX’s support, training, and mentoring? Lessons learned from IREX trainings, MP trainings/mentoring and needs assessments conducted by media outlets indicated that CRSs were operating more effectively. We saw programming that was addressing issues of importance to women, to Muslims, to language-specific communities. Figure 7: Diversifying Media Revenue Streams 19 We did not see from our review of program logs at CRSs much programming geared toward children. To what extent are gender-sensitive topics being more equitably covered? Based on the IREX media monitoring quarterly reports, all indicators are showing improvement, including gender-sensitive programming. The latest monitoring report (September 2012) showed rankings around 3.0, an improvement over the average of 2.0-2.5 in the previous report. To what extent are the radio stations hosting shows that encourage citizen participation in representative government? The CRSs we observed conduct listener call-in programs that cover a range of civic and policy issues. In Bong County, the Team conducted a half-hour call-in show to explore this question. Each of the 11 callers (all male) reported they thought citizens were better able to access government leaders via call-ins on the local CRS and via community forums jointly hosted by a local CRS and CSOs. The University of Liberia station, LUX FM, hosts a weekly call-in show with the president of the university. The station’s new 500 watt transmitter has increased access so that students residing in outlying areas can now participate. What additional skills does IREX need to help develop in the media/journalism sector? The core tenets of journalism must be covered in trainings and mentoring. The need is compounded by high attrition in the profession—especially among women. Other skills include business acumen, marketing and public relations, technical assistance/engineering skills, digital editing skills, writing skills, and social media skills (e.g., blogging, podcasting). How do radio stations assess the assistance received by IREX to improve their skills? What do they say they need? The radio station personnel we visited appreciated IREX’s business and marketing training, technical support and all suggested more assistance in funding their revenue-generating endeavors. Some journalists we interviewed expressed interest in learning social media skills. What equipment has been provided and how has this helped radio stations? What equipment is still lacking? IREX has equipped new studios, provided studio-equipment, higher-powered transmitters, computers and field recording gear as well as motorbikes during the elections that remain in use today. Cohort 2 stations need many of the same upgrades and technical training/support as Cohort 1 stations. What has been the impact of the grant program on improving radio station capabilities? CRSs have improved their technical capabilities and will be partnering with CSOs on proposal writing for grants to support jointly produced media and civic engagement programs. Most community forums thus far appear to have been single events, and have not utilized approaches to monitor meaningful outcomes along the lines of Behavior Change Communications (BCC). Conclusions As evidenced by their program logs and confirmed by our conversations with CSOs, the CRSs we observed have been making a concerted effort to take into account local audience needs and to accommodate diversity in gender, language, and religious beliefs. The community forums, 20 whereby CRSs partner with CSOs to conduct and broadcast public meetings on important local issues, have occurred in six locales. They seem to provide a useful vehicle to conduct extended public information and advocacy campaigns, although they should include business and government more fully. Without a baseline and benchmark assessment it is impossible to measure change along a meaningful scale such as the Behavior Change Communications (BCC) model. Business development and social media skills are lagging and unless they come up to speed they threaten to deter other gains. Recommendations Train media to conduct audience surveys. For IREX - community needs assessments and ratings surveys serve community needs and, in particular, women’s and other marginalized population interests. Diversify programming and staff to better meet audience needs. For IREX - programming aimed at women, Muslims, and local language-speaking audiences exists, but should be mainstreamed in newscasts and in broad-based programming. Diversity in staffing can help broaden audience demographic diversity. Increase programming for children and youth. For IREX - our review of radio program logs and newspapers revealed little children’s programming. Newspapers can be useful tools for teachers in the classroom. In the U.S., newspapers have partnered with schools and school districts on educational activities and programs. Newspapers can be literacy tools. Close the gender gap in radio program participation. For IREX - some CRSs had a dedicated phone line to encourage women callers. The gender imbalance can be addressed by mainstreaming gender-sensitive reporting, promoting prime time female program hosts, providing portable radios to women, financial support for SMS participation, call-in programs that are female schedule-friendly, and women’s radio listener-clubs. Mainstream gender-sensitive programming/reporting. For IREX - according to project M&E reports, Liberian media are failing to include, address, and serve women’s interests. Newspapers perform slightly better than broadcast media. Gender-sensitive programming needs to be more than a bloc of time, or relegated to a women’s radio station (e.g., LWDR). Expand collaboration with FeJAL. For IREX - FeJAL is an important voice for women working in the media and can help address gender gaps in pay, promotion, fair and safe workplace environments, and passage of a gender-sensitive policy for media. Track attrition rates among female journalists. For IREX - it is unclear whether the increase in female journalists to 22% is temporary bump or an aggressive trend. Attrition rates among women journalists should be tracked in order to confirm gender balance in the sector. Continue to upgrade equipment. For USAID/IREX - high-powered transmitters and mobile technology for Cohort 2 CRSs are needed. Utilize MOUs to establish CRS performance benchmarks. For IREX - equipment upgrades should be tied to performance, which could be incentivized through stiffer requirements in MOUs. 21 Result Area 6: SMS technology is utilized as a link between people and their leaders RA6 aims to support mobile phone-based information distribution systems that utilize SMS and voice messaging to allow communities, their leaders, and journalists to collect and share information across the country. As detailed in the RFA, the popularity and relative affordability of mobile phones, mobile phone-based information distribution or interactive systems, SMS￾based news services, and interactive voice response could be a revolutionary way of disseminating news, public service announcements, and market information for farmers and small business people in Liberia. This technology exists where cellular networks are in place. Findings The evaluation SOW requested the Team to assess the impact that the Mobile Phone Innovation Conference has had on promoting SMS technology. SMS is being utilized at all CRSs to gather information. A reporter from Public Agenda said he received a news tip via SMS that launched an investigative series of reports. SMS questions are regularly posed to officials on radio call-in programs. One drawback is the inability to monitor the caller’s gender or location. The Team’s findings to specific questions are as follows: What impact has the Mobile Phone Innovation Conference had on promoting SMS technology? The conference resulted in progress, but gains made by CSML’s elections/referendum projects stand to lose impact if the original investment in training, monitoring elections results, and marketing to a global audience is not valued. What impact has the innovation fund had to support small-scale pilots? We did not observe impact; although much potential exists, it is another three to five years away. How much progress has been made in the use of SMS technology? We see significant opportunity, although the potential for innovation and growth is three to five years out. CSML is moving forward with a local firm, MWETANA to develop an SMS platform that will merge two technologies to offer a large scale integrative texting service for news content distribution and the collection of information on various issues. The first technology will be the SMS system, which will gather and disseminate information to journalists and the public. The SMS System will function on two levels. At the national level, a system similar to the one used during the elections will be established at the LMC and will 1) gather and distribute short news bits to journalists and citizens; 2) track national development efforts and register complaints from citizens about development projects or the lack thereof nationally; and 3) facilitate public opinion polling. The second technology, Freedom Fone, is an open source voice messaging technology designed to augment the existing system. Voice messaging allows users to submit and receive larger volumes of information that can be sent and received via SMS. Voice-messaging technology can also be used by all regardless of literacy level. Like the SMS System, Freedom Fone works through GSM networks and requires no Internet connection. 22 Conclusions Mobile phones are portable, personal, and a modern tool for communications. Professional journalists, citizens, and CSOs can capitalize on the power of SMS/mobile technology to produce and consume information. Mobile media, new/social media, digital media, and traditional media are integral parts of the media landscape. Looking ahead, SMS can be useful for media and audiences, and can generate income for media outlets and other users. Although 4G now exists in Liberia, market saturation will take years. However, social media and citizen journalism will likely blossom soon. A 2011 graduate of the CSML program said he is excited about learning to blog, Twitter, and use mobile technology in his reporting. His smartphone is part of his newsgathering toolbox, but he is not utilizing it to the fullest capacity. Some reporters are ready to make this leap now. Recommendations Prepare to capitalize on SMS potential. For IREX - continue to instruct media on SMS in both content-generation and for-profit potential - subscription-based news and information, and polling. Once 4G takes off and smartphones saturate the market, mobile news gathering and citizen journalism will become commonplace. New areas of media law not currently addressed in Liberia will demand attention. CSML can begin positioning to advise the GoL on how leaders can better utilize SMS and other social media in order to enhance transparency and engage constituents and/or conduct service-related communications. Monitor the SMS gender gap. For IREX/L-MEP - women’s access to public information and interactions with government officials via SMS are important indicators of gender parity in technology. Perhaps the citizens’ perception survey could be broadened to include questions about how SMS is utilized to participate in media programs. Create a subscription news service. For IREX – explore a subscription news service that can provide content to media outlets and perhaps SMS subscription news and information. Envision long-range activities for social media. For IREX – challenge RPs to visualize an active social media environment in Liberia of blogging, Twitter, Mobile News and Information gathering and reporting by both professionals and citizens. Offer basic social media training. For IREX – challenge journalists to expand their reporting repertoire through use of social media. Result Area 7: The right to access to information is advanced through the project’s engagement with civil society and the media The aim of RA7 is to engage civil society to advance the establishment of the right to access to information. As envisioned in the RFA, USAID intended to propose a broad program of support to the media to achieve Freedom of Information (FOI) objectives. The RFA called for an updated analysis of the enabling legal and institutional environment for independent media, including analysis of the status and prospects of media, and FOI draft legislation. The Liberian legislature passed the FOI law in 2010. Through the Carter Center, CSML offers training and technical assistance to promote understanding of the recently enacted law and encourages communities to invoke it. 23 Findings The evaluation SOW tasked the Team with assessing progress made since the passage of FOI, the extent to which citizens understand the law, have access to information, and are actually requesting information; the extent to which progress has been made in establishing the broadcast Regulation Commission and the impact that IREX assistance has had on the entity; and progress that has been made in media freedom constituency building. The Team noted that TCC has provided technical assistance and training on fundraising, reporting and networking to the Liberia Freedom of Information Coalition (LFIC), which constitutes a network of CSOs to coordinate and spearhead actions on FOI across counties. In Bong County the LFIC comprises some 20 CSOs that also have received funding from the Open Society Initiative (OSI). TCC has convened leading CSOs and media with the GoL to advance the right to information in support of good governance. As noted earlier in this report, a senior government official affirmed the growing influence of FOI advocacy citing a cabinet meeting at which ministers were admonished to respond to requests regarding the government’s 150-day deliverables promises. The Team’s findings to specific questions are as follows: What progress has been made since the passage of the Freedom of Information law and to what extent do citizens understand this law, have access to information, and are actually requesting information? We observed increased awareness of FOI among media leaders, journalists, and CSOs. We heard a number of anecdotes of FOI application. We gathered evidence of both GoL compliance and shortcomings concerning FOI requests. Although CSML activities have helped media and civil society hold officials accountable on budgets and expenditures, FOI has been less successful in obtaining disclosure of officials’ private assets. What progress has been made in establishing the Broadcast Regulation Commission (BRC) and what impact has IREX assistance had on this entity? IREX reports that the BRC is mired in red tape, and per CSML’s revised program description is no longer part of the program. What progress has been made in media freedom constituency building? Awareness-raising, education and training among media and civil society are advancing. Presently, media freedom appears highly respected, but access to government officials in the counties remains restrictive given unequal power relations and the tendency of some government officials to act opaquely. LFICs constitute a viable structure for promoting and building media freedom constituencies. The establishment of a FOI Commissioner post represents a positive step toward enforcing self-monitoring/reporting on FOI Figure 8: Publicizing FOI and the Right to Know 24 requests by government. Conclusions Despite increasing transparency owed to FOI, the legal framework that would promote more proactive media advocacy for FOI remains mixed in Liberia. Some laws limit full independence of media and specifically CRSs. Current law stipulates that income generated by CRSs, owned and operated by and for citizens/communities, must be used solely for operational expenses. That does not prevent CRSs from generating revenues, but they cannot become self-sustaining businesses. In the U.S., the public broadcasting system in the wake of government funding cutbacks has had to generate more income and has created for-profit subsidiaries such as Minnesota Public Radio’s Greenspring Corporation, Prairie Home Companion, and other marketing schemes. Although perhaps outside the scope of CSML, the enabling environment for media requires attention. A governmental framework for funding of community radio stations presently under consideration by the Ministry of Information, Culture, and Tourism, needs to be completed. On the positive side, the project has broadened awareness of FOI by CSOs and the media, and the Team encountered examples of project-supported NGOs that have utilized FOI to actively pressure county governments to disclose details about their social development funds, and to advocate to the national government to provide information on progress toward achieving its 150-day deliverables. Recommendations FOI awareness. For IREX/TCC – continue, in partnership with the PUL, to create awareness of and train media and civil society to utilize FOI; CSOs, media outlets and government should track and self-monitor FOI requests and outcomes. Expand Media Law. For USAID - help academia and CSOs to foster the expansion of Media Law, as a discipline, examining media-related laws such as libel, defamation, licensure, business-regulation, and media ethics. New legislation needs to be passed that strengthens an overall legal framework for media independence. Strengthen ties with ALICOR. For IREX - continue to build ALICOR’s organizational capacity as a media advocacy organization representing CRSs on issues including, but not limited to, FOI, media law, ethics, and the media business-enabling environment. Result Area 8: Media is strengthened for transparent elections The purpose of RA8 is to ensure transparent elections process and keep Liberia on the path to peace. As IREX noted in its Lessons Learned from the “Strengthened Media for Transparent Elections” the international community worked with local media to establish conditions for transparent elections. IREX held discussions with the LMC, LIWOMAC, the PUL, and a number of international partners, including the Accra-based African Elections Project. The discussion resulted in a concept paper for the initiative, which was presented to and approved by USAID. Findings The evaluation SOW requested the Team to examine whether IREX-assisted media helped to assure transparent elections in 2011 and what impact the capacity building of debate moderators had on the electoral process. The Team’s findings to specific questions are as follows: 25 Motorbikes Provide Reporters Increased Access to Assignments in Outlying Areas: The CSML program launched add-on activities during the 2011 elections and referendum, which included motorbikes to select CRSs in order to expand the “reach” of geographical coverage of the campaigns. But how many women journalists took to the road on two wheels to report on politics, government, and the elections in outlying areas? Unequal access to road travel could widen the gender gap in political reporting—especially in outlying areas. What impact did capacity building of debate moderators have on the electoral process? Skills were enhanced in elections reporting, debate and moderating. We heard anecdotal evidence of how media and civil society actors believed the work by CSML had achieved greater transparency in the elections process as well as assisted in mitigating violence. What impact did IREX’s support to the Election Reporting Center have on the electoral process? The end result was greater public confidence in government due to the comparable elections results reported by the IREX activities (online clearinghouse) and the official GoL elections results. Additionally, there was evidence, via media monitoring, of the effectiveness of training media practitioners on the skills of elections reporting, effectively moderating debates and call-in shows so that candidates were held accountable for their claims. What was the overall impact of IREX assistance to the media on the transparency of elections and referendum? IREX assistance enhanced the capacity of media to report on the process, track the results process, and achieve greater fairness and balance in programming. The ultimate outcome was enhanced public confidence in government. Conclusions Although the elections provided an excellent venue for journalist training, the chapter on media strengthening for elections and governance reporting is still being written. High turnover in the journalism sector creates an ongoing need to build capacity to cover governance, politics, and public policy. The elections demonstrated that Liberian MOs, with trained journalists and civil society actors and INGO support, were able to report elections results in a timely and accurate manner. However, M&E of the fairness, accuracy, and balance of media elections coverage was not conducted on a wide scale. Nevertheless, it was successful, and bodes well for the future. Recommendations Build on momentum. For USAID/IREX – continue to validate gains from the elections program by engaging trained elections observers in CSOs to be government observers on other levels. Some well-informed CSO leaders could be poised to be credible political and government media commentators. Re-capture the elections-results on-line audience and build upon it not only for audience-share but for potential revenue-streams and other kinds of support from diaspora philanthropists, business leaders, and potential news and information subscribers. Encourage media to utilize SMS technology. For IREX – utilize SMS for other reporting purposes such as audience polls, and measuring opinions on governance and elected leaders’ performance. Train more women journalists in government/political reporting. For IREX – continue to build 26 gender parity to balance a male-dominated reporting beat. Overall Media Conclusions Media outlets in Liberia are years away from becoming fully independent, thriving, financially stable, and self-sustainable. The outlook for professional journalism looks much the same. If journalists cannot hope to earn family-sustaining wages then the recruitment and retention of trained and experienced practitioners will remain a challenge. The CSML program has already played a significant role in its five main objectives for media (Appendix 7). Incrementally, the project has fostered change—albeit in fits and starts. If the CSML program lives beyond the current five year mandate—and we believe that it should—then consideration should be given to developing a third sector - the actors who populate the jobs in Public Affairs/Information, Public Relations, Marketing and Constituency Services. Presently, some government agencies are filling PIO positions with persons who lack training and skills in the discipline. To ignore this sector—while investing in the development of media and civil society, is to ignore an important partner to the overall objective of enhancing access to information in a democratic society. USAID should consider funding the development of this sector beyond its investments in media and civil society. Project Management The evaluation SOW mentions four project management components: 1) development hypothesis, 2) project management, 3) sustainability, and 4) lessons learned. The Team has offered its opinions on the development hypothesis and sustainability elsewhere in the report. In general, the hypothesis is still relevant and appropriate for the evolving socio-political context of Liberia, and the project’s indirect capacity-building assistance and cascading training approach have sustainability built into them. What is less clear is the project’s exit strategy, which has not been articulated in the project documents we reviewed, and which IREX and its partners have only begun to discuss. It is not too early to envision a project exit strategy, which, in the Team’s opinion, should consider follow on options. As for project design and staffing, IREX has quite successfully implemented an ambitious activity schedule by outsourcing to its MPs initiatives such as the Women’s Media Initiative and the Young Journalists Initiative. The program directors in charge of civil society and the media and their MPs have worked collaboratively across component areas. Although certain lessons can be drawn – particularly from the elections - we believe it is useful to address lessons more fully in the ex-post impact evaluation. Mentoring According to IREX, there is a total of 4.25 full time staff based in all counties except for River Gee, which is covered by the mentor resident in Grand Gedeh; the total number of CSOs and Figure 9: Reporters on Wheels 27 CRSs to be mentored is 56, which is covered by resident mentors according to the following estimates of resident staff workloads:  Nimba: 1.35 full-time staff  Lofa: 0.6 full-time  Bong: 0.5 full-time  Grand Bassa: 1.0 full-time  Montserrado: 2.5 full-time (available only for Montserrado CSOs and CRSs)  Grand Gedeh: 0.8 full-time  River Gee: no resident mentor We consider the practical implications of this staffing scheme by looking at the resident mentor in Nimba, which has the highest county-based percentage of staff time (1.35). The Master Sheet for CSOs and CRSs shows six (6) Nimba-based CSOs and five (5) CRSs, which translates to 11 visits per month in Nimba from a team of 1.35 persons. Let us calculate two days per visit for those CRSs and CSOs outside of Ganta (total= 7) in the dry season, and three days per visit during the rainy season; and let us assume over a year 2.5 days x 7 or 17.5 days per month. We then add 4 days/month for the Ganta-based CSOs and CRSs, for a total of 21.5 days per month of mentoring visits divided by 1.35 or roughly 16 days per month per mentor (about 75% of staff time in any given month, the rest in administrative work at his or her home office).This is a reasonable scenario for Nimba County (1.35), but dramatically lower staffing ratios apply to Bong (0.6), Lofa (0.6) and Grand Gedeh (which also covers River Gee; 0.8). The team feels that the Nimba ratio of mentors to mentees provides adequate minimal coverage based on a budget that only allows one visit per month. However, Category C and Cohort 2 beneficiaries would greatly benefit from additional monthly visits. Monitoring and Evaluation More vigilance is advised to maintain a robust M&E reporting system. The PMP and the PIRS we examined were incomplete. Some indicators were missing explanations of self-assessment scores, baselines, actuals and targets, and did not adequately explain the performance expectations over the life of the project. More complete and robust PIRS will serve the project better as management tools, and up to date indicator data will help CSML tell its story. The Team believes that it is a story well worth telling. Indicator #3 is arguably the most comprehensive indicator for measuring institutional change. Part of the difficulty with this indicator is that there are 104 questions directed at ten organizational development areas, while far fewer questions measure partnering, advocacy and networking. Concerning the actual data at our disposal, the Team notes that the inability of the organizational development capacity self-assessment scores to confirm the Team’s empirical findings raises methodological concerns. We feel the self-assessment tool is generally sound despite the distinct imbalance in the number of questions relating to the COS/PANS component compared with the OD component. We would expect that if the same persons were assessed in Years 1 and 2, and that if each of them had received the full treatment of training and mentoring, that they would score collectively higher on both COS/PANS and OD components. IREX’s explanation that grades may have been inflated on BI, combined with the likelihood that the second round assessments overlooked attrition and included individuals who had not benefitted from the full 28 menu of CSML training and mentoring, would help explain the discrepancy between self￾assessment scores and the Team’s on-site observations. Recommendations We believe that the project should undertake three distinct assessment activities:  Self-assessments must include those staff who have demonstrated some longevity throughout the project life span; we assume scores will improve over time and should reach an agreed-upon threshold level score of 2.3 or higher for an organization aspiring to attract funding from an international donor.  A second self-assessment by incoming staff after they have had time to gain an appreciation of the institutional culture, how it is affecting them and their view of the organization; we assume a lower score than those posted by the staff who have remained on board, but still higher than the baseline in 2010 for Cohort 1.  A custom assessment by an MT in project months 1, 30 and 55 to give yet another perspective of sustained institutional change and progress (baseline, mid-project, impact). This assessment is separate from and in addition to the capacity self-assessment. Its purpose would be to obtain an informed opinion by a MT of organizational performance, and it would offer a means to triangulate the data generated by the capacity self￾assessment tool.  In addition, CSML should articulate a development hypothesis within the PMP; update the PIRS quarterly; report on indicators quarterly instead of annually where possible; establish a rigorous protocol for capacity self-assessments and train mentors accordingly; and identify and address weaknesses in the M&E reporting system. Concluding Statement The team is confident that CSML’s foundation is solid, the fundamentals sound, and change is trending in the right direction. Complete data from the project’s M&E system will help connect the dots and more methodological rigor in administering the capacity self￾assessment will strengthen its reliability. The evaluation mission produced many recommendations, some of which were adopted at the CSML Partners Conference. Our advice is to conclude updates and improvements to the M&E data collection and reporting systems so that the project will be able to tell its story with greater statistical confidence, and better manage for results. Figure 10: Poster in a Women's Empowerment NGO ANNEXES Annex 1: The CSML Training and Mentoring Process 1 The CSML Training and Mentoring As the RFA noted, the CSML project is fundamentally an effort to administer a capacity￾building program for sustainable benefits through the Resource Partners; consequently the training and mentoring process are the keys to achieving results. The team felt it needed to examine this system in depth, but could not do so within the parameters of the report text. But we hope this Annex is treated as an integral part of the report. We’ve looked at the three major TA vehicles: management training, which we’ve called “organizational development”; technical training ; and mentoring. The following constitute our findings in these areas: A. Organizational Development (OD) To build stronger CSOs, CSML offers standard management training workshops, at both the Resource Partner (RP) and CSO levels. For example, training addresses a CSO’s constitution or bylaws; the project management cycle, financial management, personnel and administration, monitoring and evaluation and proposal writing. What unites these topics is an emphasis on establishing viable systems. Other training includes seminars and technical training workshops (also facilitated with participatory group methods) on subjects such as advocacy, gender awareness and mainstreaming, elections monitoring, and the Freedom of Information (FOI) act which is now signed into law. IREX highlights three “competency tracks” among CSOs: 1) constituency outreach along with partnering, advocacy and networking; 2) organizational development; and 3) promoting a more professional and independent media. All tracks are supported by training, mentoring and grants. The grants program is considerable, and was originally planned to include $1.5m. In 2011 more funds were made available to CSML for elections monitoring, bringing the grants envelope to $2.3m of which approximately $1.3m has been expended. The project’s administration also mirrors this capacity-building concern: the selection process of the RPs was a capacity-building exercise, as candidates had to be not only eligible, but also fulfill nine basic criteria for selection. Many more candidates applied than were retained as partners. The same observation holds for the CSO selection process. With implementation experience and difficulties encountered among the RPs – for example, it required many re-writes of proposals for grants before they were accepted – IREX began to scale back its original plan to work with up to 130 CSOs, while it decided to increase support for CRSs. The project is working with six main partners, two of whom are media-focused (Liberia Media Center or LMC and the Liberia Women Media Action Committee or LIWOMAC). These, like the organizational development-focused RPs, are all membership agencies with mission statements expressing a primary purpose to build the capacity of their members. The four RPs focusing on organizational development assistance are: the Liberia branch of the regional West Africa Network for Peace Building (WANEP); the New African Research and Development Agency; Women NGOs Secretariat of Liberia (WONGOSOL); and YMCA Liberia. In terms of sector and thematic expertise these six main partners respond well to the project’s central concerns: WANEP is a major regional player in conflict resolution and peace building; NARDA Annex 1: The CSML Training and Mentoring Process 2 is well known as a training institution with a special focus on organizational development; WONGOSOL is committed to women’s empowerment, while YMCA is perhaps the country’s leading youth-focused CSO. All RPs have undergone the participatory capacity self-assessment in both constituency outreach/partnering, advocacy and networking, as well as the comprehensive organizational development self-examination. All have commented on how important this procedure was for building their own capacity. These six partners in turn mentor the CSO and MO beneficiaries. The team reviewed for content and for facilitation process modules and materials at both levels including the July 2012 module for organizational development aimed at RPs; RP modules for proposal writing; an RP Facilitator’s manual for Project Cycle Management; handouts for M and E training, budget preparation, and proposal templates. The team interviewed Master Trainers at both the IP and RP levels; indeed one RP Master Trainer described the impact of CLMS training as “revolutionary.” Among many examples of change which CSOs (many outside Monrovia) shared with us as evidence of the project’s continuing institutional impact and evolving organizational culture include a new personnel policy; a new financial management system; a wholly revised vision, mission and core value statement; a new board election policy; and proposals rewritten for donor submission and in some cases actual outcomes of funding from other donors. All interviewees attributed these changes to CSML training and mentoring. One area of potential concern was the sequencing of the OD training program: on the assumptions that (1) core OD capacity is required before technical skills can be mastered (e.g., policy advocacy) and (2) the amount of questions in the self-assessment tool addressed to basic OD categories (from constitution to sustainability, more than 100 questions) compared to less than 15 covering constituent outreach and partnering, advocacy and networking, the team would have imagined that a broad refresher course in OD should have initiated the CSML capacity building training program. Instead, the project waited until July 2012 to provide a basic OD module for RPs. B. Technical As used in this report, technical training covers specialty areas such as business development, gender awareness, and policy advocacy. The use of the term “technical” does not imply that the delivery of these sessions is not influenced by adult learning techniques and facilitated by standard process training features such as small group work, case studies, and role play; but it places a greater emphasis on knowledge transfer (as opposed to skills acquisition). In some instances – for example, computer training – individual learning replaces more classic management training models which depend on team-building and achieving consensus in small groups. As noted above the advocacy and networking training were especially successful among RPs and even among Cohort 2 CSOs: the team noted one Monrovia advocacy CSO which appropriated knowledge and skills obtained in a CSML training on advocacy policy and adopted this learning for an intervention among Ivoirian refugees in Butuor. Annex 1: The CSML Training and Mentoring Process 3 C. Mentoring Feedback from all CSOs and RPs confirmed the appreciation of mentoring; as one Master Trainer affirmed in an interview: “Once the project ends, we’ll continue to use mentors.” The mentorship program works in part because of the immediacy of follow-up from training event to application; in part because the mentor is involved in all facets of the capacity-building pathways, from assessment to on-the-job skills practice; in part because the attitudinal posture of the mentor is supportive, rather than supervisory: the emphasis is always on what the mentee is doing right, which builds confidence and instills a sense of moral obligation to improve performance. The team examined the Mentoring Facilitation Guide and interviewed mentors both at headquarters and in the field; we also requested feedback from executive CSO staff and technical staff (financial and program staff, in particular) on the proposal writing and budget preparation trainings and follow-up mentoring. All unanimously endorsed the mentorship process as the turning point in their organizational learning in the CSML project. Although the team has not seen a written protocol, according to discussions with IREX, the process goes like this: the MT and the mentor initially together administer the assessment tool to approximately five staff members, one of whom should be the Executive Director, another chief program officer, and a finance person, and perhaps two program officers or a secretary. The scores are then merged into a mean score for the organization. However, in the subsequent assessments, conducted by mentors, there is no certainty that the same five people will be available and be interviewed; hence there could be a drop in a given capacity self–assessment score through assessments of different individuals. The team proposes that there be a revised method for assessments: that there is an assessment of only those staff who have been interviewed in Year 1 through the life of the project, which will provide a complete picture of individual growth. Expected scores from this group will be assumed to rise every year. If new personnel come on board, the expectation is that they will benefit from the changes in organizational culture and within several months should be given an assessment. These will score lower than the staff who’ve been with the organization continuously, but presumably higher than the initial Cohort 1 staff from the initial assessment. The mean of these two scores will provide a better – though still incomplete – gauge of institutional progress. In addition, the team recommends that the MT conduct an independent assessment of OD capacity by drawing on approximately 20 of the more than 100 questions asked; these independent assessments should be undertaken at the beginning of the project, at the midway point, and at the end. This will give yet another, more complete view of institutional progress over time. The mentoring process is as follows: CLMS grants to RPs provide the support for the Master Trainer and 2 mentors per RP; in at least two counties the team saw mentors assigned to the field (1 mentor per county). Mentors participate in all RP staff trainings. Mentors are also trained in the self-assessment exercise, and administer the tool to staff in the CSOs; the mentor and staff then participate in the scoring and staff reach consensus on their mean scores in the outreach/advocacy test as well as the far more comprehensive OD test. The mentor then engages staff to reflect critically on the assessment and propose OD areas for improvement; mentors do Annex 1: The CSML Training and Mentoring Process 4 not propose, do not suggest, do not convince. Instead, they listen. Once the staff selects a topic, the mentor will ask the select a date which is convenient to them; then the mentor will return with handouts and other documents relevant to the topic selected. If the topic happens to be too technical – e.g., budget preparation using Excel spreadsheets – the mentor will return with a specialist from HQ. The session which transpires is interactive: the mentor asks what is not understood, but applauds and encourages what is understood. The mentor does not supply answers to questions nor solutions to problems, but seeks to engage the staff to arrive at a solution. But there are times when mentors help out in a very practical manner: they can photocopy documents; they can sometimes repair equipment; and at one RP in Monrovia, HQ mentors are either providing on-line help with specific questions or referring clients to regional mentors for follow-up. Mentoring combines the roles and attitudes of a collaborator, monitor, coach and trainer/facilitator. It appears to tap into a cultural norm of reciprocity: the mentor’s visit is not purely a technical affair, because the staff of the CSO being mentored has bought in to the idea of engagement: the mentor’s visits come with a sense of moral obligation to deliver on the tasks which have been agreed upon. This approach differs from classical supervision; it is not judgmental, and for that reason it seems to work. Annex 2: Mid-Term Evaluation SOW 1 MID-TERM EVALUATION OF THE CIVIL SOCIETY AND MEDIA LEADERSHIP PROGRAM (CSML) Scope of Work (SOW) I. INTRODUCTION USAID/Liberia has requested that the Liberia Monitoring and Evaluation Program (L-MEP) conduct a mid-term evaluation of the “Civil Society & Media Leadership Program” (CSML). CSML is a five-year project which seeks to strengthen Liberian civil society organizations and the independent media as a means to include and engage the Liberian population in civil society, thereby contributing to the creation of rapid, inclusive, peaceful, and sustainable development. The overall purpose of the evaluation is to assess progress toward the project’s expected results. The findings of the evaluation will help inform USAID, the prime and sub-partners as well as other stakeholders on how well the project planned activities and strategy are contributing to the achievement of its expected results; and help provide further guidance for the effective operation of the project over the remaining period to help ensure the achievement of the project results and the sustainability of those results. The evaluation will take place over a period of six weeks in September and October 2012. II. BACKGROUND CSML is a five-year USAID funded project launched in 2010. The project is aimed at sustaining peace in Liberia through greater inclusion, giving a voice to, informing and engaging Liberian citizens. Specifically, CSML provides strategic trainings, one-on-one mentoring, small grants, and other resources to Liberian civil society organizations and independent media institutions to include and engage the Liberian population in the creation of inclusive, peaceful and sustainable development. The project is also working towards changing the institutional culture of Liberian civil society organizations and independent media institutions. The International Research Exchange Board (IREX), the prime Implementing Partner, works with The Carter Center (TCC), Social Impact and the Liberia-based Liberia Media Center (LMC) and the Liberia Women Media Action Committee (LIWOMAC) to carry out the project’s activities. After a year of implementation in 2011, the project modified its implementation strategy. Some of the changes included the reduction in the number of Resource Partners and targeted Civil Society Organizations (CSOs); and an increment in the number of targeted Community Radio Stations (CRS). The amount of grant given to the Resource Partners was also increased. IREX currently works through four local organizations – New African Research and Development Agency (NARDA), Women Non-governmental Organization Secretariat of Liberia (WONGOSOL), West Africa Network for Peace (WANEP) and the Young Men Christian Academy (YMCA) – which serve as Resource Partners (RP). IREX provides intense one-on-one mentoring and strategic training to the Resource Partners, which in turn build the capacity of CSOs and Community Radio Stations. The project also worked with the Liberian media to enable them to cover the 2011 national elections. Annex 2: Mid-Term Evaluation SOW 2 The project also currently targets sixty CSOs, twenty-one Community Radio Stations, and ten media outlets for the entire duration of the project. These institutions receive training, mentoring, and small grants as a way of building their capacities. The project is carried out in seven counties – Montserrado, Bong, Nimba, Lofa, Grand Bassa, River Gee and Grand Gedeh. III.PURPOSE OF THE EVALUATION The overall purpose of the evaluation is to assess the level of progress towards the project’s eight expected results and to provide recommendations for improving project implementation and/or management, as well as for altering project design if current activities are not contributing to the accomplishment of project results. Evaluators should also determine whether the project is effectively implementing a sustainability plan so that project results can be sustained by CSOs, CRSs, and RPs. The project results include: 1. CSOs have measurably increased organizational capacity and culture to serve as a vehicle for representing their members’ interests; 2. Advocacy CSOs in seven counties have the capacity to monitor government policy, performance, and expenditures; 3. Perceptions of CSOs and government of each other are enhanced; 4. Independent media operates more professionally; 5. Community Radio Stations in 7 target counties engage with community, diversify listening audience, and improve reach; 6. SMS technology is utilized as a link between people and their leaders; 7. The right to access to information is advanced through the project’s engagement with civil society and the media; 8. Media is strengthened for transparent elections. IV.EVALUATION QUESTIONS Key evaluation questions identified include the following: 1) How effective has the training been in enhancing the specific skills, competencies, and knowledge/understanding/perceptions targeted and what additional training is required? 2) How effective have the program’s mentoring activities been in providing the anticipated skills and competencies? What unexpected opportunities and/or challenges have surfaced? 3) To what extent are the IREX-supported community radio stations operating more effectively because of IREX support, training, and mentoring? (IREX currently has 19 community radio station partners in 7 targeted counties.) 4) Have organizational processes and culture, advocacy, and networking been strengthened as a result of program activities to date? Annex 2: Mid-Term Evaluation SOW 3 5) To what extent have citizens improved their perception of CSOs and government as a result of program activities thus far? 6) What challenges and/or opportunities to the effectiveness IREX’s support to the Election Reporting Center can be observed thus far? 7) What progress has been made in the professional and independent behavior of media? 8) Have CSOs developed the required skills to serve as watchdogs? 9) To what extent have gender strategies been implemented (including more equitable coverage of gender issues), and with what impact on gender integration? 10) To what extent have the grants successfully been employed to date? 11) Is the project logic still relevant in light of the shifting socio-political and economic context and are the expected results still applicable in each of the component areas of Civil Society and Media? What unexpected opportunities and/or challenges have surfaced? 12) Is the project designed and adequately staffed to work collaboratively and effectively across all component areas? 13) Are there programmatic or organizational gaps that hinder the achievement of results? 14) How can program activities be enhanced to increase the likelihood of CSO long-term sustainability? A more detailed list of evaluation questions is provided in ANNEX A; however, the evaluation does not have to be limited to the questions provided here or in the annex. The evaluation should be structured to address the degree to which program activities implemented to date have met anticipated time-frames and the degree to which adaptive management has been effective where required, and make specific recommendations for the remainder of the program, including eventual modification of the program. V. DATA COLLECTION AND ANALYSIS METHODS The evaluation study design will be primarily descriptive, with some retrospective comparison in the absence of quantitative data collected at baseline. The evaluation will employ a mixture of qualitative and quantitative primary and secondary data collection and analysis. Specifically, the methods for data collection will include:  Review of relevant documents;  Semi-structured interviews with key stakeholders;  In-depth field observation visits to select sub-grantees and partners sites; Annex 2: Mid-Term Evaluation SOW 4  Focus group meetings with key stakeholders; and  Citizen mini-surveys.  Key stakeholders and selected sub-grantee and partner sites will be identified according to specific selection criteria, which will be detailed in the evaluation report. It is anticipated that a statistically representative sample (simple or systematic random selection of a specifically targeted sampling frame) will be utilized in identifying individual responses to the citizen minisurveys. Measures will be undertaken in order to ensure protection and confidentiality during data collection; these will be detailed in the evaluation report. VI.PROJECT INFORMATION SOURCES Information on the project can be obtained from the following available source documents:  Revised Project Description (Approved by USAID)  Performance Management Plan (PMP)  Quarterly and Annual Reports  Data Quality Assessment Report (2011)  L-MEP Assessment of RP M&E Systems.  Surveys of Perception of CSOs, government, and citizens.  Organizational Assessments completed of CSOs, CRSs, and RPs.  Skill Needs Assessments  Training Reports (Pre and Post Training Evaluations)  Training Manuals and Agendas  Mentoring Guides The evaluation team may request other relevant documents as available. VII. EVALUATION TEAM COMPOSITION AND PARTICIPATION The Evaluation Team will be comprised of three expatriate consultants and one Liberian Consultant, each with at least ten years of experience in their specialties. One of the expatriate consultants needs to be a recognized expert in strengthening independent media, including radio, in developing countries and experience with the role of media to help promote transparency during elections. The media expert also needs to have experience with issues related to access to information. This expert will focus on the progress made by IREX toward helping Liberia build an independent media. The second expatriate needs to be an expert in strengthening civil society organizations in developing countries. The third expatriate consultant should be an expert in training design and execution and mentoring and will focus on the effectiveness of the training and mentoring materials and in the training events in imparting desired knowledge, skills, and abilities. Ideally, all of these experts would have previous experience in Africa and documented. The Liberian consultant must have had at least five years experience working in civil society and/or media. Annex 2: Mid-Term Evaluation SOW 5 The L-MEP M&E Specialist for Democracy and Governance will manage the evaluation team and participate to the extent practical, given his other responsibilities. VIII. LEVEL OF EFFORT The evaluation team will have six weeks to conduct the evaluation and provide a final draft evaluation report. One week will be spent in the U.S. reviewing project documents and crafting an evaluation approach to be approved by L-MEP and USAID. Four weeks shall be spent in Liberia collecting data and reaching preliminary results and providing a draft evaluation report. One follow-up week will be provided to finalize the report after comments from USAID and L￾MEP have been received. The evaluation will begin during the second week of September with the team arriving in Liberia during the third week. To the extent possible, the evaluation team should be arriving in country in order to participate. DELIVERABLES The deliverables include: 1. An Evaluation plan: The plan should include (a) the overall design strategy for the evaluation, (b) the data collection and analysis plan and tools, (c) a list of the team members, and (d) the team’s work schedule for the evaluation. Due prior to arrival in Liberia. 2. Inception Report: This report should be submitted immediately after the document review. Due prior to arrival in Liberia. 3. A Draft evaluation report: The report should not exceed 30 pages excluding annexes and an executive summary of not more than two pages. The report should be presented electronically and in hard copies. Due during week four of the team’s time in Liberia. 4. Oral presentation of the evaluation: The evaluation team will be required to give an oral presentation of the evaluation findings to key stakeholders of the program including USAID, the Implementing Partner, L-MEP and selected beneficiaries of the project, if possible. Due during week four of the team’s time in Liberia. 5. Final evaluation report: The final report should also not exceed 30 pages including annexes and an executive summary of not more than two pages. A total of five original hard copies as well as electronic copy of the report should be presented to L-MEP. Due one week after departing Liberia. 6. A Dissemination Plan: The plan should detail how the evaluation findings could be disseminated to stakeholders through various means. Due one week after departing Liberia. Annex 2: Mid-Term Evaluation SOW 6 EVALUATION SCHEDULE No. ACTIVITY DATE 1. Document Review Mon, Aug 27 – Tues, Aug 28, 2012 2. Inception Report Thu, Aug 30 – Fri, Aug 31, 2012 3. Field Activities (Focus Group Discussions, Interviews, Field Visits, etc.) Mon, Sep 3, 2012 – Fri, Sep 28, 2012 4. Draft Report Sun, Sep 30, 2012 5. Oral Presentation of Evaluation Findings Fri, Sep 28, 2012 6. Final Report Tues, Oct 16, 2012 BUDGET TMG Headquarters will determine the evaluation budget at a later date. Annex 2: Mid-Term Evaluation SOW 7 ANNEX A DETAILED EVALUATION QUESTIONS The evaluation team must answer at a minimum the questions presented below. In their evaluation plan, they can include additional questions as required. Assessment of Project Results: A number of evaluation questions will be posed for each of the project eight results. These include: 1. CSOs have measurably increased organizational capacity and culture to serve as a vehicle for representing their members’ interests. IREX is working to strengthen CSOs and Community Radio Stations (CRS) ability to advocate and network in order to represent their constituent’s interests and influence the policy process and at the local and national levels. IREX is doing this through a cadre of Resource Partner Master Trainers and Mentors and Mentoring by the IREX Civil Society Specialist and Program Officer. Organizational assessments were conducted to serve as a baseline for organizational capacity development. In order to assess progress to date, the evaluation team should answer the following questions:  Using the organizational assessments of the CSOs completed in year one as a baseline, what progress has been made to strengthen CSO organizational processes and culture, advocacy, and networking?  According to CSOs and CRSs, has the training provided by Master Trainers been effective to enhance their organizational competencies?  According to CSOs and CRSs, has the mentoring they have received helped them enhance their organizational competencies?  According to the RPs, has the organizational change mentoring provided to them by the IREX Civil Society Specialist provided them the skills required to be effective mentors?  To what extent have the CSOs implemented gender strategies and with what impact on gender integration?  To what extent have the grants successfully been employed to support advocacy campaigns, human rights, peace building and reconciliation and gender awareness activities, youth activities, networking with other CSOs and with CRS, work with local government and other community driven projects? 2. CSOs in 7 counties have capacity to monitor government policy, performance, and expenditures. In order to evaluate progress toward this result, evaluators need to answer the following questions:  According to CSOs, to what extent has advanced capacity building in advocacy and networking provided to them increased their effectiveness as watchdogs Annex 2: Mid-Term Evaluation SOW 8 monitoring national and decentralized policy development, implementation, and enforcement? What further training do they say they need?  What evidence that CSOs have been engaged in effective monitoring supports their progress as watchdogs? Especially in areas such as government expenditures and performance, gender based violence, and child trafficking?  According to RP trainers, have CSOs developed the required skills to serve as watchdogs?  According to media specialists, have CSOs developed the required skills to serve as watchdogs?  To what extent have CSO ability to network been strengthened by the project? 3. Perceptions of CSOs by government and the government and CSOs by citizens enhanced. Information needs to be better channeled between citizens and the government, connecting people’s views to the policy process. IREX conducted an initial survey of perceptions and is required to conduct a follow-up survey. This survey will be completed prior to the evaluation. Evaluators should review these surveys in their overall evaluation of progress made to date in the project. Evaluators should answer the following questions:  How effective has the training of county government officials on participatory democracy been to improve their perceptions of the role of CSO and the media?  How effective has training of CSOs on participatory democracy been to improve their perceptions of the government?  What outstanding issues still need to be addressed by the project to achieve this result?  How successful have Roundtables been to improve perception and collaboration?  To what extent have citizens improved their perception of CSOs and government?  To what extent have radio shows improved perceptions?  To what extent has education regarding access to public information improved perceptions? 4. Independent Media Operates More Professionally IREX is working to raise the capacity of media outlets to provide reliable information in a post-conflict environment through activities integrating journalism training, media management and business development, enabling environment reform, build-up of community radio both institutionally and programmatically and targeted support to other media, and piloting of mobile-telephone-based and citizen journalism models. Evaluators should answer the following questions:  To what extent have the Specialized Consulting and Business Development Initiative and the Sustainability Initiative improved the professional operations of targeted media outlets? Annex 2: Mid-Term Evaluation SOW 9  What progress has been made in the professional and independent behavior of media?  What are citizen perceptions of the independence of media?  What progress has been made in the establishment of pilot subscription and distributions services?  What progress has been made in the Women’s Media Initiative?  What progress has been made in the Young Journalists Initiative?  Have the investigative journalism training workshops had an impact on improving the reporting skills of journalists?  Has the on-line media resources clearinghouse been used and if so, what impact has this had on users in terms of expanding their knowledge and awareness and skills? 5. Community Radio Stations in 7 Target Counties engage with community, diversify listening audience, and improve reach. IREX is working to build 8 model radio stations in 7 targeted counties. The evaluation should address the following questions:  To what extent are the 8 model radio stations operating more effectively because of IREX support, training, and mentoring?  To what extent are gender issues being more equitably covered?  To what extent are the radio stations hosting shows that encourage citizen participation in representative government?  What additional skills does IREX need to help them develop?  How do radio stations assess the assistance received by IREX to improve their skills? What do they say they need?  What equipment has been provided and how has this helped radio stations? What equipment is still lacking?  What has been the impact of the grant program on improving radio station capabilities? 6. SMS Technology is utilized as a link between people and their leaders.  What impact has the Mobil Phone Innovation Conference had on promoting SMS Technology?  What impact has the innovation fund had to support small-scale pilots?  How much progress has been made in the use of SMS technology? 7. The Right to Access to information Advanced by the engagement of civil society and media.  What progress that has been made since the passage of the Freedom of Information Act and to what extent citizens understand this law, have access to information, and are actually requesting information. Annex 2: Mid-Term Evaluation SOW 10  What progress has been made in establishing the broadcast Regulation Commission and what impact has IREX assistance had on this entity?  What progress has been made in media freedom constituency building? 8. Strengthened Media for Transparent Elections IREX assisted media help assure transparent elections in 2011 and the evaluation team should assess the success of this effort. Specifically:  What impact did capacity building of debate moderators have on the electoral process?  What impact did IREX’s support to the Election Reporting Center have on the electoral process?  What was the overall impact of IREX assistance to the media on the transparency of the elections? Project Design and Management: Development Hypothesis 1. Is the project logic still relevant in light of the shifting socio-political and economic context? 2. Are the expected results still applicable in each of the component areas of Civil Society and Media? Project Management 1. Is the project designed and adequately staffed to work collaboratively and effectively across all component areas? 2. Are the monitoring, evaluation and reporting systems adequate to track project performance? 3. Are there programmatic or organizational gaps that hinder the achievement of results? 4. Is information being effectively captured and used internally and externally? 5. How effective is the CSML structure in engaging civil society, Media, government, USAID and other stakeholders? Sustainability 1. What are the sustainable elements to the CSML that will remain after project completion? 2. How can program activities be enhanced to increase the likelihood of CSO long-term sustainability? 3. What is the most appropriate program exit strategy that will embed benefits into local organizations? Lessons Learned 1. What (if any) are the lessons learned from project design, implementation and monitoring? Annex 3: CSML Evaluation Inception Report and Evaluation Plan 1 CSML Mid-Term Evaluation Inception Report and Evaluation Plan September 5, 2012 (revised September 13, 2012) Preface This Inception Report covers the Civil Society and Media Leadership (CSML) evaluation team’s plans for conducting the mid-term formative evaluation of the CSML program implemented by the Information and Research Exchange Board (IREX) and its partners the Carter Center (TCC) and Social Impact (SI). The 5-year program, funded by USAID/Liberia and managed by the Democracy and Governance (DG) Team, was launched in February 2010. CSML seeks to strengthen Liberian civil society organizations (CSOs) and independent media as a means to include and engage the Liberian population in civil society, thereby contributing to accelerated, inclusive, sustainable peace and development. The evaluation findings will shed light on project performance and relevance of activities toward achieving program objectives in eight result areas. The exercise aims to provide USAID, the prime and sub-partners as well as other stakeholders’ recommendations for program adjustments that may enhance project implementation and sustainability. The Mitchell Group, Inc. (TMG’s) Liberia Monitoring and Evaluation Program L-MEP) is USAID/Liberia’s monitoring and evaluation partner. L-MEP, in consultation with USAID/Liberia’s Program and DG Offices, selected four evaluators for the team. They are: 1] Robert Groelsema: Team Leader & Civil Society Expert: 0776024986 2] Denis Hynes: Capacity Building and Training Expert: 0777452412 3] Karen Louise Boothe: Media Expert: 0777452374 4] Kanio Bai Gbala, Local Consultant: 0886528619 The L-MEP M&E Specialist for Democracy and Governance, Baward Johnson, is coordinating the exercise; while overall supervision is provided by Randal Thompson, L￾MEP Chief of Party (COP). The evaluation spans a period of six weeks between August 27, 2012 and October 15, 2012. Background-CSML Project Logic CSML aims to sustain peace in Liberia through greater inclusion, giving a voice to, informing and engaging Liberian citizens. Specifically, CSML provides technical assistance, strategic trainings, one-on-one mentoring, small grants, and other resources to Liberian civil society organizations and independent media institutions to include and engage the Liberian population in the creation of inclusive, peaceful and sustainable development. The project is also working towards changing the institutional culture of Liberian civil society organizations and independent media institutions. Annex 3: CSML Evaluation Inception Report and Evaluation Plan 2 The International Research Exchange Board (IREX), the prime Implementing Partner (IP), works with international partners, The Carter Center (TCC) and Social Impact (SI), to carry out the project’s activities. After a year of implementation in 2011, the project modified its implementation strategy. Some of the changes included the reduction in the number of Resource Partners and targeted Civil Society Organizations (CSOs); and an increment in the number of targeted Community Radio Stations (CRS). The amount of grants given to the Resource Partners was also increased. IREX and its partners work through six local organizations – the Liberia Media Center (LMC), the Liberia Women Media Action Committee (LIWOMAC), New African Research and Development Agency (NARDA), Women Non-governmental Organization Secretariat of Liberia (WONGOSOL), West Africa Network for Peace (WANEP) and the Young Men Christian Academy (YMCA) – which serve as Media and Resource Partners (RP). IREX provides one-on-one mentoring and strategic training to these partners, which in turn builds the capacity of CSOs and Media Outlets (MOs). The project received an additional $1 million from USAID/Liberia to work with the media to enable them to cover and add transparency and accountability to the 2011 national elections. CSML currently targets 60 CSOs and 23 Community Radio Stations for the duration of the project. These entities receive training, mentoring, and small grants to strengthen their capacities. The project covers seven counties – Montserrado, Bong, Nimba, Lofa, Grand Bassa, River Gee and Grand Gedeh. Implementing Partners IREX Carter Center Social Impact M&E Organizational Development Mentoring Main Partners LIWOMAC-Media Partner LMC-Media Partner NARDA-Resource Partner WANEP-Rersource Partner YMCA-Resource Partner WONGOSOL-Resource Partner Organizational Development Training/Mentoring Beneficiaries 60 CSOs 23 CRSs 1 TV Station Newspapers Links/Networking to Citizens and Communities Annex 3: CSML Evaluation Inception Report and Evaluation Plan 3 Anticipated Project Results The desired project results include: 1. CSOs have measurably increased organizational capacity and culture to serve as a vehicle for representing their members’ interests; 2. Advocacy CSOs in seven counties have the capacity to monitor government policy, performance, and expenditures; 3. Perceptions of CSOs and government of each other are enhanced; 4. Independent media operates more professionally; 5. Community Radio Stations in seven target counties engage with community, diversify listening audience, and improve reach; 6. SMS technology is utilized as a link between people and their leaders; 7. The right to access to information is advanced through the project’s engagement with civil society and the media; 8. Media is strengthened for transparent elections. Purpose of the Evaluation The CSML evaluation is a mid-term formative evaluation designed to assess assistance mechanisms and implementation modes and to evaluate how they have contributed toward achieving results in the project’s eight result areas. It also provides the evaluators’ recommendations - based on the data and evidence they have collected - for making mid-term adjustments. The evaluation will also assess the project’s exit strategy and sustainability plan. Roles and Responsibilities of the Evaluators The evaluators perform the following roles: Denis Hynes – is the Training and Capacity Building Specialist. He will focus on the effectiveness of the training and mentoring materials and of the training events to impart desired knowledge, skills and abilities (KSAs). He will pay special attention to adult education methods for participatory management training, and will also assess the adequacy of the mentoring process and monitoring of it. Karen Louise Boothe – is the Media Expert. She will focus on the progress made by IREX toward helping Liberia build an independent media. She will pay special attention to the nexus of media and civil society. Kanio Gbala – is the Country Expert. He will focus on civil society and media and will pay special attention to the relevance of project logic in light of the shifting socio￾political and economic context. Robert Groelsema – is the Team Leader and Civil Society Strengthening Expert. He will focus on the project’s emphasis to enhance civil society’s role in democratic governance. He will pay special attention to evaluation design and methodological issues and to the sustainability of the program. Annex 3: CSML Evaluation Inception Report and Evaluation Plan 4 Desk Review – CSML Strengths, Assumptions and Risks The team received soft and hard copies of several project documents for review. The documents reviewed included: 1. Revised Project Description – 21 February 2011 2. IREX/CSML Partners 3. Revised PMP and Results Framework – 30 March 2012 4. Start-up assessments – August 2010 5. Baseline Perception/Attitudes Survey – August 2010 6. Community Radio Station Needs Assessment – August 2010 7. Data Quality Assessment Report – 12 September 2012 8. Second Report CSML Media Partner News content 12/2011-3/2012 9. Baseline Report CSML Media Partner News Content 10. Strengthened Media for Transparent Elections – Lessons Learned 11. Understanding the Media and Advertising Market in Liberia – July 2011 12. Quarterly Reports: a. April-June 2012 b. January-March 2012 c. October 1-December 31, 2011 d. July 1-September 30, 2011 e. April 1- June 30, 2011 f. January-March 2011 g. October-December 2010 h. July-September 2010 i. April-June 2010 13. Resource Partner Organizational Capacity Self-Assessment Tool 14. Government Officials attitudes toward CSOs survey 15. Citizens’ attitudes toward CSOs and county or government officials survey 16. Media Component Start up Assessment Plan The team also requested: 1. Original project document – technical section/SOW 2. First and Second Year Work Plans 3. Training plans, strategies, methods, modules, participant lists 4. Success stories 5. Community radio station program analysis and their internal M&E plans 6. USAID M&E policy – Forward 7. Map of project sites 8. “ABC” ratings, and color-coded designations of beneficiaries Our desk analysis of project documents indicates that the CMSL program would appear to be an ideal model of how a broad-based media and civil society project should work— in tandem—to effect long-lasting change. The design favors a holistic approach involving awareness creation and transfer of KSAs via indirect assistance. Training cascades from the IPs to the RPs to the beneficiary CSOs and CRSs, who engage and Annex 3: CSML Evaluation Inception Report and Evaluation Plan 5 involve their citizens and their communities. The cascade effect builds and strengthens capacity at multiple levels. The CSML theory of change might be expressed as: “A selection of CSO and media entities believed to make good partners, equipping them with tools, methods and skills, and enabling them to better inform and engage citizens to sustain peace in Liberia through greater inclusion.” As described in the Revised Program Description 21 February 2011, the IPs noted that after less than a year into the program more deliberate, consistent and responsive follow up was needed to build individual capacity and to strengthen institutional change. With USAID approval, this change was operationalized as a mentorship component. Similar to on-the-job training, mentorship takes place in the workplace of the trainee/participants. The assumption is that in the long run this approach will add value to training in organizational development (OD) and technical areas by reinforcing KSAs. Nonetheless, the evaluators draw attention to the following risks of this approach, which we will explore and test during our in-country interviews, data collection and analysis: 1. Several years are needed, usually, before independent media and the journalism profession itself are large and stable enough to become self-sustaining; 2. CSML assistance mechanisms include FOGs, which are known to be difficult to administer, and require a great deal of monitoring from the grant-making institution; 3. “Mentorship” has multiple understandings and applications including the existence and quality of organizational development documents; 4. Mentoring drills down deep, but its narrow scope it excludes other members of targeted entities as well as other entities, and therefore sacrifices breadth for depth; 5. Outputs are not easily transformed into outcomes. How will the project ensure that women trained and mentored as journalists sustain their roles and advance in their nascent careers after the project ends? Will more women, in fact, be employed and contribute editorial content for radio and television and media outlets? In addition, the team observed that the first two questions in the Evaluation SOW relate directly to the effectiveness of training and other modes of capacity building: (1) How effective has the training been in enhancing the specific skills, competencies, and knowledge/understanding/perceptions targeted and what additional training is required? (2) How effective have the program’s mentoring activities been in providing the anticipated skills and competencies? What unexpected opportunities and/or challenges have surfaced? We note that these questions presume there has already been evidence of effectiveness both for training and mentoring, and that there are reasonable expectations for “additional” training or other kinds of capacity building approaches. Annex 3: CSML Evaluation Inception Report and Evaluation Plan 6 Therefore, one key evaluation question is: to what extent is this standard process training methodology reproduced (both in terms of training skills, but also in terms of the required attitudes of a trainer) as we move down the scale from International Partners, to Resource Partners, to CSOs and CRSs, and to the level of citizens and communities. Based on experience we would advance the following hypothesis: The level of attention to process training methods decreases dramatically after the international partner level because the notion of adult education is culturally biased to a western sense of egalitarianism where group work is more highly prized, where working in groups is the norm, and hence where hierarchy does not play a dominant role. To the extent that the process model weakens at successive training levels, we would expect more pedagogical emphasis on lecturing, much less on practical exercises, and hence lesser learning. Evaluation Plan A) Design Strategy for Evaluation & Data Collection and Analysis Plan B) Team’s Schedule for Evaluation C) Team’s Media Interviews D) Team’s CSO Interviews A) Design Strategy and Tools for Evaluation and Data Collection and Analysis Plan The evaluation methodology includes: 1) desk study review; 2) identification of stakeholder groups for interviewing and surveying; 3) development of tools and survey instruments; 4) conduct of interviews with key stakeholder groups; 5) trainee/participants focus groups; 6) citizen focus group/s; and 7) radio listener call-in program/s. The team initially planned to conduct citizen mini-surveys. However, after further consideration of data needs and given time and logistical constraints, the team decided to replace the mini-surveys with citizen focus groups, e.g., a women’s radio listener group, and CRS listener call-in program/s – (numbers 6 and 7 above). For example, a CSML participant CSO has established a community women’s radio listener group in Grand Bassa to discuss women’s concerns that are carried on one of the IREX-supported CRSs. Similarly, IREX-supported CRSs feature listener call-in shows in their program mix. The team plans to participate in at least one live call-in show, which will ask an overarching question such as “How has this station served your needs to better access public information that formerly was off-limits, unavailable or unattainable?” These exercises will provide a notion of CSML reach down to the grassroots, and the project’s challenges and effectiveness in involving and engaging ordinary citizens in democratic governance and development. In addition, the team will assess and utilize the findings of the citizen perception survey recently conducted by L-MEP to gauge program impacts vis-à-vis citizen perceptions of CSO and government performance in meeting citizen needs, as well as governmental perceptions of CSOs as development partners in providing relief, development and other forms of assistance and services to citizens. Annex 3: CSML Evaluation Inception Report and Evaluation Plan 7 1) Desk study document review (cf. above list and discussion) 2) Identification of stakeholder groups for interviews:  L-MEP and M&E Officer in D&G  IREX staff  Carter Center staff  Social Impact M&E Officer  USAID AOR and M&E Officer  GOL interlocutors (Min of Information)  Program RP, CSO and Media Outlet participants 3) Development of tools and survey instruments (cf. Annex 1 Main Partner questionnaire) 4) Key stakeholder group interviews:  Liberia Media Center CSML staff  LIWOMAC CSML staff  NARDA  YMCA  WONGOSOL  WANEP  20/60 CSOs; 8/23 radio stations; 1 newspaper – about half of the CSOs and radio stations will be in Monrovia and the remainder will be in Montserrado, Grand Bassa, Bong and Nimba counties. 5) JPC and The Carter Center community monitors focus group – to be held in Ganta; Mentorship graduate focus group to be held in Monrovia; 6) Citizen focus group/s: focus group of citizen beneficiaries to be held in each of Grand Bassa, Bong and Nimba counties as possible; 7) Radio listener call-in program/s to be held in Gbargna (and an additional Community Radio Station TBD as possible). A note on selection criteria: The sample of CSOs and CRSs to be interviewed is purposive. The team has attempted to balance six selection criteria in order to cover different types of experience and challenges associated with the CSO beneficiaries of CSML. Only four of these criteria apply to the CRSs. The six criteria are:  Type of CSO – network/umbrella, mass-member CBO, and mission-driven NGOs. Illustrative examples of these types are: The Bassa Youth Caucus, the Liberia Motorcycle Transport Union, and Ears for the Masses. Our supposition is that different types of CSOs will require different training and mentoring strategies, approaches, plans and techniques (not applicable to CRSs);  Thematic area and mission orientation – e.g., advocacy, dispute resolution, environment, research, member-service, youth, arts, women’s issues, etc. Thematic variation will shed light on differing needs, challenges, networks and relationships by assessing multiple sectors (not applicable to CRSs);  Cohort – IREX has two CSO and CRS cohorts. Cohort 1 began treatment in 2010 and is now in its third year. Cohort 2 began treatment in 2012 and is about 7 months into its program. Assessment of CSOs and CRSs from both cohorts Annex 3: CSML Evaluation Inception Report and Evaluation Plan 8 will provide a more rounded picture of the CSML assistance effort and of IREX’s adaptive management style;  Resource Partner (RP) Mentor – CSOs are mentored by four different RPs – NARDA, WANEP, WONGOSOL, and YMCA. CRSs are mentored by two different Media Partners – LMC and LIWOMAC. The team has selected CSOs and CRSs mentored by the different Media Partners and RPs, which will help assess the effectiveness of indirect assistance training by these partners.  Self-assessment rating – IREX and its partners have conducted capacity assessments of the participating CSOs and CRSs and have grouped them into three categories based on their scores. We have selected CSOs and CRSs from across the three (A, B, C) ratings to include a sampling of high, medium and low performers;  Location – About half of the selected CSOs and CRSs for the evaluation are located in Monrovia; the other half is located upcountry (Grand Bassa, Bong and Nimba counties). A few substitutions in the upcountry selections may be necessary owing to accessibility and road conditions. As possible, the team will conduct the interviews on site at RP and CSO offices and in the radio stations of the CSML participants in order to observe institutional culture, staffing dynamics, and status of physical office infrastructure and equipment. B] Team’s Schedule for Evaluation (note: timeline and schedule of activities are subject to revision.) TIMELINE Liberia Civil Society & Media Evaluation Activity Timeline Review LMEP/Civil Society & Media Background Materials Mon, Aug 27 – Tues, Aug 28, 2012 Travel to TMG/DC Wed, Aug 29, 2012 (1) Team Building Exercise (2) Develop Evaluation/Work Plan which includes: (a) overall design strategy for the evaluation, (b) the data collection and analysis plan and tools, (c) a list of the team members, and (d) the team’s work schedule for the evaluation (3) Create Inception Report - include preliminary findings of the desk review (review of project documents). The inception report should also include the Evaluation Plan which should specify the evaluation team’s approach to the evaluation methodology and questions contained in the SOW; determining, at the minimum, the data collection and analysis plan and tools, limitations to the evaluation, and the team’s work schedule for the evaluation 4) Conduct DC-Based interviews. IREX and Social Impact contacts. Thu, Aug 30 – Fri, Aug 31, 2012 Annex 3: CSML Evaluation Inception Report and Evaluation Plan 9 Travel to Liberia Sat, Sept 1, 2012 10:00 a.m. - Meet with L-MEP and also with Liberian member of evaluation team to go over their SOW and Work Plan and proposed travel schedule. 2:00 p.m. – Meet with IREX at IREX Office. Mon, Sept 3, 2012 9:00 a.m. – Meet with L-MEP COR Laura Arntson at L-MEP 4:30pm – Meet with DG Team at USAID Tues, Sept 4, 2012 PPT Prepared and Submitted to USAID/Liberia/IREX & L￾MEP Fri, Sept 21, 2012 DRAFT REPORT prepared and submitted to USAID/IREX/L-MEP for review Mon, Sept 24, 2012 PPT Presented to USAID/Liberia, IREX & IREX’s Stakeholders Tues, Sept 25, 2012 IREX Mid Term Partners’ Conference Evaluation Team meets with USAID Democracy & Governance Team for feedback on the Draft Report Wed, Sept 26, 2012 IREX Mid Term Partners’ Conference Draft report submitted to USAID & IREX for review and comments Thu, Sept 27, 2012 Exit debriefing powerpoint to USAID based on DRAFT Report and preliminary feedback) Fri, Sept 28, 2012 Team Departs Liberia (submits revised draft evaluation report) Sun, Sept 30, 2012 Final comments received from USAID/IREX/L-MEP Fri, Oct 5, 2012 Dissemination Plan, showing in detail how evaluation findings could be disseminated to stakeholders through various means Fri, Oct 8, 2012 Final Evaluation Report Submitted to USAID/IREX/LMEP for approval Fri, Oct 15, 2012 Team Travel Schedule: September 9-15 Grand Bassa: Buchanan Sept.9-10 (returning to Monrovia the 10th) Monrovia Sept. 10-11 (departing 11th) Bong: Gbargna Sept. 11 Nimba: Ganta, Saclpea-Sinniqueullie, Sinniqueullie, Sept. 12 (arriving) to 14 (departing) Bong: Gbarnga Sept. 14 (arriving) to 15 (departing the 15th) Monrovia: Sept. 15 Return to Monrovia Annex 3: CSML Evaluation Inception Report and Evaluation Plan 10 C) Team’s Media Interviews Media Interviews (8 Community Radio Stations and 2 Newspapers) Monrovia/Montserrado No. CRS City Cohort 1 or 2 Ranking Comments 1 LWDR Monrovia 1 A ranking 2 LUX FM Montserrado 1 B ranking 3 Radio Veritas Monrovia 1 A ranking 4 Public Agenda Monrovia 1 5 Inquirer Monrovia 1 Grand Bassa 6 Magic FM Buchanan 1 C ranking Bong 7 Radio Jorwah Jorwah 1 C ranking To be substituted due to impassible roads 8 Super Bongees Gbarnga 2 A ranking Nimba 9 Voice of Tappita Tappita 1 C ranking To be substituted due to impassible roads 10 Radio Kergheamahn Ganta 2 B ranking A ranking= Strong B ranking= Doing Relatively Well C ranking= Needs Extra Attention Annex 3: CSML Evaluation Inception Report and Evaluation Plan 11 D) Team’s CSO Interviews CSO Interviews 20 of 60 Monrovia/Montserrado No. CSO Name City Cohort 1 or 2 Ranking Comments 1 Federation of Liberian Youth Montserrado 1 A ranking 2 Concerned Christian Concerned Montserrado 2 B ranking 3 United Muslim Women Montserrado 1 A ranking 4 Prison Fellowship Liberia Montserrado 2 A ranking 5 Voice of the Voiceless Montserrado 2 C ranking 6 African Network of Prevention and Protection against Child Abuse and Neglect Montserrado 2 A ranking 7 Society for Women and AIDS in Africa/Liberia Montserrado 2 B ranking U Grand Bassa 8 BAWODA Buchanan 1 B ranking 9 Bassa Concerned Citizen Movement Buchanan 1 C ranking 10 Bassa Youth Caucus Grand Bassa 1 B ranking Nimba 11 SEARCH Saclpea￾Sanniqeullie 1 A ranking 12 Ears for the Masses Sanniqeullie 1 B ranking 13 Aiding disadvantaged and traumatized women and girls ADWANGAH Tappita 1 C ranking To be substituted due to impassible roads 14 Inter-visionary Artists (IVA) Ganta 1 B ranking 15 The Botanical Product Assoc. of Liberia Zorgowee 2 B ranking To be substituted due to impassible roads 16 Liberia Motorcycle Transport Union Ganta 2 B ranking Annex 3: CSML Evaluation Inception Report and Evaluation Plan 12 17 YMCA Ganta RP N/A Bong 17 Dev. Education Network-Liberia (DEN-L) Gbarnga 2 C ranking 18 Community Dev. and Research Agency Gbarnga 2 A ranking 19 Assoc. of Liberian Community Radio Stations Gbarnga 1 B ranking 20 Catholic Justice and Peace Commission Gbarnga 2 C ranking A ranking= Strong B ranking= Doing Relatively Well C ranking= Needs Extra Attention Annex 1: Semi-Structured Questionnaire for Resource Partners Semi-structured Q’s for Main Partners 1) Have you had management training in the media sector under IREX? A: If yes, what kind eg. Managerial or technical/journalistic? B: If no, have you requested? 2) What capacity building activities are you conducting for media? 3) How do you see your work with media organizations as supporting civil society? 4) In your work with media outlets and/or journalists can you share with us a special success story and/or special outcome? 5) In your work with media outlets and/or journalists can you share with us a lesson￾learned from unsuccessful intervention? 6) How has the mobile phone innovation conference and resulting SMS technology affected your work? Do you use SMS technology? And, do you see further applications for its development? Tell us about the innovation fund? Did you access it? 7) Tell us how you think media have been strengthened for increasing transparency in elections? 8) Can you share with us an instance of a successful training you conducted which resulted in a measureable improvement of organizational development of a CSO/CSR? 9) Can you share with us an instance of successful mentorship that resulted in a measureable improvement of organizational and/or individual development of a CSO/CSR? 10) In what ways has the IREX training/mentoring strengthened YOUR training/mentoring of CSOs/CRSs? 11) Can you share with us your understanding of the way in which your grant from IREX has built your capacity? Annex 3: CSML Evaluation Inception Report and Evaluation Plan 13 12) Please tell us about the training and mentoring you provided to your CSOs and CRSs? A: Can you share with us, a success story resulting from this? B: What do you see as the work still needed in building this capacity? 13) Have you have assisted CSOs and CRSs in order increase their capacity to monitor government? A: Please share with us a success story. 14) If you were to suggest ways to the project (IREX) to further improve your capacity, what kind of assistance would you require? Annex 2: Semi-Structured Questionnaire for CSOs and CRSs Design guide: to the extent possible, the responses to the following questions should be able to address the following--  The project’s eight Intermediate Results;  The substantive concerns of the SOW questions for CSOs;  Specific indicators for civil society strengthening;  Evidence of CSO and media synergies for promoting good governance;  The effectiveness of training and mentoring;  Professionalization of independent media;  Efforts to engage communities and audience expansion;  SMS technology as a link between people and their leaders;  The effectiveness of activities promoting the right to access to information;  Strengthening of the media to promote accountability and transparency. GENERAL INFORMATION 1. Describe how and why you were selected by the CSML project? 2. Were there any findings in the assessment which surprised you? 3. How many OD assessments have you had by CSML? 4. When is your next OD assessment scheduled? 5. What were the results, and can you share them with us? 6. How long have you been working with the CSML? 7. Who are your members1 and how do they support your work as a CSO? 8. How do you support your members’ interests? 1 For CBOs and umbrella groups Annex 3: CSML Evaluation Inception Report and Evaluation Plan 14 For CRSs and Newspapers: 9. Are you a member of ALICOR (radio stations)? If yes, how many programs originate from network/Monrovia? How many programs originate locally at the CRS? 10. Radio Stations: Do you use a Program Log? If yes, it records what kind of information? 11. For newspapers: what is the focus of your content? News, Information, Entertainment, Other? 12. Total Number of Staff? Paid Staff? Volunteers? 13. Number of Program Hosts? Reporters? Management? Support staff? 14. Staffing: Men? Women? 15. Number of paid staff? Men? Women paid? UNDERSTANDING CSML 16. How do you understand your role in the CSML project? 17. What do you see as the overall goal of the CSML project? 18. How does your organization utilize media to promote good governance? 19. How do you see your media work as supporting civil society? ORGANIZATIONAL DEVELOPMENT/INSTITUTIONAL CAPACITY BUILDING 20. Please tell us what changes have occurred in your organization since you have become a CSML partner, specifically with reference to: Your organization’s Constitution—what has changed? Financial management—what has changed? Personnel management—what has changed? Resource mobilization—what has changed? Program delivery—what has changed? Monitoring and Evaluation—what has changed? (For media): Capacity to monitor governmentwhat has changed? 21. Do you have a particular success story from this training that you’d like to share with us? 22. Please describe any of the managerial training for organizational capacity building you received from CSML (this would include training on the constitution, project design and proposal writing, budget preparation, training in supervision, and monitoring and evaluation, either from IREX or a RP). 23. Please give us a ranking of the three most useful ODCB training exercises you participated in with CSML (IREX and/or RP) staff. 24. For each of these trainings which you participated in, what can your organization do now that it could not do before receiving this training from the CSML project? Annex 3: CSML Evaluation Inception Report and Evaluation Plan 15 25. Concerning the most useful training you received and the new skill(s) you have identified, what was it about the training which enabled you to learn this or these new skill(s)? 26. What was the most useful knowledge you learned during your training(s) under the CSML project? 27. How have you put this new knowledge into practice? 28. Do you have a particular success story from this training that you’d like to share with us? TECHNICAL TRAINING 29. Please describe any technical training you have received from IREX (NB: this would include advocacy, journalism, ability to represent member interests, diversifying constituents, capacity to better understand and influence policy, and engage more effectively with government; also on computer usage, new technologies, and new equipment). 30. Of the technical training you have received from IREX, which subject area has been the most useful? 31. For each of these trainings which you participated in, what can your organization do now that it could not do before receiving this training from IREX? 32. Concerning the most useful training you received and the new skill(s) you have identified, what was it about the training which enabled you to learn this or these new skill(s)? 33. What was the most useful knowledge you learned during your technical training(s) with CSML? 34. How have you put this new knowledge into practice? 35. Do you have a particular success story from this training that you’d like to share with us? MENTORING 36. Please describe the mentoring process which IREX and its main partner has designed for your organization. 37. Please rank in term of most useful the topics or issues for which your organization received or receives mentoring. 38. For the topic you’ve identified as most useful, was the mentoring provided by one Mentor to one Mentee, or did the learning take place in a group format? 39. Was the Mentor assistance a new kind of assistance for your organization? What is your appreciation of this assistance? How do you understand the difference between a trainer and a mentor? 40. Who among your staff has shown the most progress under the Mentoring program? Annex 3: CSML Evaluation Inception Report and Evaluation Plan 16 41. What can your organization do now that it could not do before the mentoring assistance began? 42. How has your increased skills and knowledge under the mentoring program enabled you to bring citizens and government closer together? 43. (For media): how have your journalists who have been mentored by the CLMS program contributed to or affected the kind of programming and/or reporting your station and/or newspaper can now offer your audience? 44. Can you share with us an example of how this capacity building through mentorship of your and/or your reporters’ reporting skills has changed the editorial content of your product? 45. Do you have a particular success story from this training that you’d like to share with us? Can you point to an example of how your coverage of an issue or story influenced subsequent discussions, dialogues and actions of your readers and/or listeners and of other key stakeholders (for example, elected officials)? ASSISTANCE MECHANISMS (GRANT) 46. What are the main activities which your organization conducts with the grant you have received from IREX or through a main partner? 47. Please explain how you manage the grant process: -- what kind of grant do you have? Do you receive an advance and then submit expenditures? Or do you have to accomplish certain tasks before you receive payment? -- who on your staff administers the grant (reviews expenditures, authorizes payments, tracks receipts, and reviews results)? -- in what ways, if any, did this grant make your organization stronger? What can you do now that you could not do before receiving this grant? MONITORING AND EVALUATION 48. Are you familiar with the Project’s Performance Monitoring Plan (PMP)? 49. Are you aware that your organization’s performance data contribute to a specific result in the project’s PMP? 50. How does your organization collect information on the progress of the activities which your organization sponsors? 51. What does your organization do with this information? 52. Did your organization have an M&E system in place before you received training from the CSML project? 53. What can your organization do now that it could not do before in M&E as a result of training from the CSML project? Annex 3: CSML Evaluation Inception Report and Evaluation Plan 17 SUSTAINABILITY 54. What is your organization’s plan for maintaining the benefits and strengthening you have obtained from the CLMS project once the project ends in 2014? 55. Is your organization able to retain talented staff? Does it have a problem of high turnover? LESSONS LEARNED 56. In your experience so far with the CSML project managed by IREX, what is the most useful lesson (or lessons) you have learned in being a partner in this project? Have you experienced any disappointments so far? How can the partnership between you and IREX and the main partners become more useful to your organization? MOVING FORWARD 57. What is the single most important change you would make to the way in which the project has been working so far with your organization? Annex 4: CSML Mid-Term Evaluation 1 Questionnaire for Civil Society Organizations1 and Community Radio Stations and Media Outlets Design guide: to the extent possible, the responses to the following questions should be able to address the following--  The project’s eight Intermediate Results  The substantive concerns of the SOW questions for CSOs  Specific indicators for civil society strengthening  Evidence of CSO and media synergies for promoting good governance  The effectiveness of training and mentoring  Professionalization of independent media;  Engage communities and audience expansion;  SMS technology is utilized as a link between people and their leaders;  The right to access to information is advanced;  Media is strengthened to promote accountability and transparency GENERAL INFORMATION 1. Describe how and why you were selected by the CSML project? 2. Were there any findings in the assessment which surprised you? 3. How many OD assessments have you had by CSML? 4. When is your next OD assessment scheduled? 5. What were the results, and can you share them with us? 6. How long have you been working with the CSML? 7. Who are your members2 and how do they support your work as a CSO? 8. How do you support your members’ interests? For CRS and MOs: 9. Are you a member of ALICOR (radio stations)? If yes, how many programs originate from network/Monrovia? How many programs originate locally at the CRS? 10. Radio Stations: Do you use a Program Log? If yes, it records what kind of information? 11. For newspapers: what is the focus of your content? News, Information, Entertainment, Other? 1 Includes national and local NGOs, CBOs, and umbrella groups 2 For CBOs and umbrella groups Annex 4: CSML Mid-Term Evaluation 2 12. Total Number of Staff? PAID Staff? Volunteers? 13. Number of Program Hosts? Reporters? Management? Support staff? 14. Staffing: Men? Women? 15. Number of paid staff? Men? Women paid? UNDERSTANDING CSML 16. How do you understand your role in the CSML project? 17. What do you see as the overall goal of the CSML project? 18. How does your organization utilize media to promote good governance? 19. How do you see your media work as supporting civil society? ORGANIZATIONAL DEVELOPMENT/INSTITUTIONAL CAPACITY BUILDING 20. Please tell us what changes have occurred in your organization since you have become a CSML partner, specifically with reference to: Your organization’s Constitution—what has changed? Financial management—what has changed? Personnel management—what has changed? Resource mobilization—what has changed? Program delivery—what has changed? Monitoring and Evaluation—what has changed? (For media): Capacity to monitor governmentwhat has changed? 21. Do you have a particular success story from this training that you’d like to share with us? TECHNICAL TRAINING 22. Please describe any technical training you have received from IREX (NB: this would include advocacy, journalism, ability to represent member interests, diversifying constituents, capacity to better understand and influence policy, and engage more effectively with government; also on computer usage, new technologies, and new equipment). 23. Of the technical training you have received from IREX, which subject area has been the most useful? 24. For each of these trainings which you participated in, what can your organization do now that it could not do before receiving this training from IREX? Annex 4: CSML Mid-Term Evaluation 3 25. Concerning the most useful training you received and the new skill(s) you have identified, what was it about the training which enabled you to learn this or these new skill(s)? 26. What was the most useful knowledge you learned during your technical training(s) with CSML? 27. How have you put this new knowledge into practice? 28. Do you have a particular success story from this training that you’d like to share with us? ORGANIZATIONAL CAPACITY BUILDING TRAIN ING (ODCB) 29. Please describe any of the managerial training for organizational capacity building you received from CSML (this would include training on the constitution, project design and proposal writing, budget preparation, training in supervision, and monitoring and evaluation, either from IREX or a RP). 30. Please give us a ranking of the three most useful ODCB training exercises you participated in with CSML (IREX and/or RP) staff. 31. For each of these trainings which you participated in, what can your organization do now that it could not do before receiving this training from the CSML project? 32. Concerning the most useful training you received and the new skill(s) you have identified, what was it about the training which enabled you to learn this or these new skill(s)? 33. What was the most useful knowledge you learned during your training(s) under the CSML project? 34. How have you put this new knowledge into practice? 35. Do you have a particular success story from this training that you’d like to share with us? MENTORING 36. Please describe the mentoring process which IREX and its main partner has designed for your organization. 37. Please rank in term of most useful the topics or issues for which your organization received or receives mentoring. 38. For the topic you’ve identified as most useful, was the mentoring provided by one Mentor to one Mentee, or did the learning take place in a group format? 39. Was the Mentor assistance a new kind of assistance for your organization? What is your appreciation of this assistance? How do you understand the difference between a trainer and a mentor? 40. Who among your staff has shown the most progress under the Mentoring program? 41. What can your organization do now that it could not do before the mentoring Annex 4: CSML Mid-Term Evaluation 4 assistance began? 42. How has your increased skills and knowledge under the mentoring program enabled you to bring citizens and government closer together? 43. (For media): how have your journalists who have been mentored by the CLMS program contributed to or affected the kind of programming and/or reporting your station and/or newspaper can now offer your audience? 44. Can you share with us an example of how this capacity building through mentorship of your and/or your reporters’ reporting skills has changed the editorial content of your product? 45. Do you have a particular success story from this training that you’d like to share with us? Can you point to an example of how your coverage of an issue or story influenced subsequent discussions, dialogues and actions of your readers and/or listeners and of other key stakeholders (for example, elected officials)? ASSISTANCE MECHANISMS (GRANT)* 46. What are the main activities which your organization conducts with the grant you have received from IREX or through a main partner? 47. Please explain how you manage the grant process: -- what kind of grant do you have? Do you receive an advance and then submit expenditures? Or do you have to accomplish certain tasks before you receive payment? -- who on your staff administers the grant (reviews expenditures, authorizes payments, tracks receipts, and reviews results)? -- in what ways, if any, did this grant make your organization stronger? What can you do now that you could not do before receiving this grant? (*only for the five former Main Partners and selected CSOs) MONITORING AND EVALUATION 48. Are you familiar with the Project’s Performance Monitoring Plan (PMP)? 49. Are you aware that your organization’s performance data contribute to a specific result in the project’s PMP? 50. How does your organization collect information on the progress of the activities which your organization sponsors? 51. What does your organization do with this information? 52. Did your organization have an M&E system in place before you received training from the CSML project? 53. What can your organization do now that it could not do before in M&E as a result of Training from the CSML project? Annex 4: CSML Mid-Term Evaluation 5 SUSTAINABILITY 54. What is your organization’s plan for maintaining the benefits and strengthening you have obtained from the CLMS project once the project ends in 2014? 55. Is your organization able to retain talented staff? Does it have a problem of high turnover? LESSONS LEARNED 56. In your experience so far with the CSML project managed by IREX, what is the most useful lesson (or lessons) you have learned in being a partner in this project? 57. Have you experienced any disappointments so far? 58. How can the partnership between you and IREX and the main partners become more useful to your organization? MOVING FORWARD 59. What is the single most important change you would make to the way in which the project has been working so far with your organization? Annex 5: Interview List 1 Persons Interviewed by Evaluation Team 1 Kevin Rubio Sr. Technical Advisor IREX/DC 8/31 2 Danielle de Garcia Performance Management Specialist Social Impact-DC 8/31 3 Tilly Reed Chief of Party IREX CSML 9/3 4 Bill Burke Sr. Media Specialist IREX CSML 9/3 5 Cerue Konah Garlo Civil Soc. Program Officer IREX CSML 9/3 6 Joshua Kpelewah M&E Specialist Social Impact/CSML 9/3 7 Lancedell Mathews Executive Director NARDA 9/4 8 Monyou Culu Resource and Documentation Associate NARDA 9/4 9 Arthur Berrian Associate Program Associate/Master Trainer-CSML NARDA 9/4 10 Laura Arntson M&E Specialist USAID/Liberia 9/4 11 Kristin Joplin Democracy and Governance Officer USAID/Liberia 9/4 12 Louise Fahnbulleh DG Specialist USAID/Liberia 9/4 13 Morpue Speare Executive Director WONGOSOL 9/5 14 Monah Kelly Master Trainer WONGOSOL 9/5 15 Emmanuel Bryant Mentor WONGOSOL 9/5 16 Rev. Bartholomew B. Colley Executive Director WANEP 9/5 17 Samuel Darpolor Master Trainer WANEP 9/5 18 James Benson Mentor WANEP 9/5 19 Chelsea Payne Country Representative Carter Center Liberia 9/5 20 Nigbalu Warner Information Project Lead Carter Center Liberia 9/5 21 Pewee Flomoku Senior Project Coordinator, Access to Justice Carter Center Liberia 9/5 22 Alphonsus Zeon Project Carter Center 9/5 Annex 5: Interview List 2 Coordinator, Access to Information Liberia 23 Counselor Reeves Legal Advisor Carter Center Liberia 9/5 24 Norris Tweah Deputy Minister Administration Ministry of Information, Cultural Affairs & Tourism 9/6 25 Lamil Kpargoi Program Director LMC 9/6 26 Marcus Shamu Broadcast Technician LMC 9/6 27 Wede Kobbah￾Wureh Chairperson/Asst. Professor Dept. of Mass Comm. Univ of Liberia/Monrovia 9/6 28 Duplex Tchouente Project Manager LIWOMAC 9/7 29 Eric Mortor M&E Officer LIWOMAC 9/7 30 Weedor Ballayan Finance Manager LIWOMAC 9/7 31 Eliza Dahn Program Director LIWOMAC/LWDR 9/7 32 Euriahs Togar Station Manager LUX FM 9/7 33 Saturday H. Seke Deputy Station Manager LUX FM 9/7 34 Sydney L. Nicol Jr. News Director LUX FM 9/7 35 Winnie Saywah Core Editor/Sr. Reporter Inquirer 9/7 36 Marietta Williams Executive Director United Muslim Women’s Advocacy and Empowerment Organization (UMWO) 9/7 37 Jimmy B. Zimnah Project Officer UMWO 9/7 38 Samuka V Konneh News Editor/reporter Public Agenda 9/8 39 Dominic Farley Reporter Public Agenda 9/8 40 Martin Browne Sr. President/CEO Sky Communications Radio/TV 9/8 41 Christopher Yarwoe Station Manager Magic FM 9/10 Annex 5: Interview List 3 42 Numen Wheaton Reporter/Producer Magic FM 9/10 43 Victoria Momo Reporter/Producer Magic FM 9/10 44 Martha F. Karnga Executive Director Bassa Women’s Development Organization (BAWODA) 9/10 45 Julie Flangy Admin. Assistant BAWODA 9/10 46 Louise Gbarowheon Sec’y and Social Worker BAWODA 9/10 47 Wahde Toe President Bassa Youth Caucus (BYC) 9/10 48 Samuel G. Wilson VP Operations BYC 9/10 49 Abraham Sandoe Financial Sec’y BYC 9/10 50 Victor Flomo Sec’y General BYC 9/10 51 Troko Kreangar Member BYC 9/10 52 David S. Mehdeh Executive Director BUCCOBAC 9/10 53 Mark S. Sherman Administrator BUCCOBAC 9/10 54 Rebecca Swegaye Program Coordinator BUCCOBAC 9/10 55 Abel R. Karr Student BUCCOBAC 9/10 56 Esther Jarou Master Trainer Den-L 9/11 57 Alpina Clay II Sports Coordinator Den-L 9/11 58 Elisha L. Bloe Master Trainer Den-L 9/11 59 Peter Dolo Human Resources Trainer Den-L 9/11 60 Augustine S. Tweh Coordinator Den-L 9/11 61 Lasanah Dukuly Executive Director CODRA 9/11 62 Joseph B Dorbor Admin./Finance CODRA 9/11 63 J Samuel Kplaiuure Radio Manager & Program Director Radio Super Bongese 9/12 64 James Dorbor General Manager Radio Super Bongese 9/12 65 William Quaire President ALICOR 9/12 66 Nyahn Flomo Volunteer Organizer Radio Kergheamahn 9/12 67 Jess Cole Regional Coordinator Catholic Justice and Peace Commission 9/12 Annex 5: Interview List 4 68 Eleane P.K. Keamue Community Monitor Catholic Justice and Peace Commission 9/12 69 Linda Paye￾Tokpah Community Monitor Catholic Justice and Peace Commission 9/12 70 Lorweimon Dokie County Dispute Resolution Monitor, Bong The Carter Center 9/12 71 Ismail Kromah County Dispute Resolution Monitor, Lofa The Carter Center 9/12 72 Johnny Ndebe National Coordinator Community Dispute Resolution Monitor The Carter Center 9/12 73 Daniel Gboe Master Trainer YMCA/Ganta 9/13 74 Nyahn Flomo Journalist Radio Kergheamahn 9/13 75 Nyahn M. Gongloe Executive Director IVA 9/13 76 Aloysius Seakel President Liberian Motorcycle Transport Union (LMTU) 9/13 77 Othello Fayiah Member IVA 9/13 78 Siah K. Nyamah Member IVA 9/13 79 Janjay D. Cole Member IVA 9/13 Wilfred M. Paye, Jr. Darius K. Biago Karbeah Kotee Project Officer Construction Technician Protection Manager SEARCH* [*NB: the interview could not be completed as the Executive Director and Senior Program Manager were not available] 9/13 80 Laurence Tokpa Project Manager Ears for the Masses 9/13 81 Olamdo Kollie Melling Director Radio Totota 9/14 82 Josh Tarbee Station Manager Radio Totota 9/14 Annex 5: Interview List 5 83 D. Sunkule Technician Radio Totota 9/14 84 Matthew Caine Station Manager Radio Nimba 9/14 85 Hananiah Zoe Standard Bearer Liberian Empowerment Party 9/14 86 Prince Karsieh Deputy Station Manager Radio Senhwai 9/14 87 F. Moham Kromah Deputy Assistant and Political Liaison Officer Nimba County 9/14 88 George Karmie Assistant to the Superintendent Nimba County 9/14 89 Whittington Kweti Budget Officer Nimba County 9/14 90 David Foday Executive Director Christian Concerned Community (CCC) Monrovia 9/17 91 Frances R. Greaves Executive Director Voice of the Voiceless, Monrovia 9/17 92 Lorma Baysah Executive Director Rural Human Rights Assistance Program (RHRAP), Monrovia 9/18 93 Sumo Harris Project Assistant RHRAP 9/18 94 Issac Parochue Field Monitor RHRAP 9/18 95 Sandee Cole Deputy Director Administration RHRAP 9/18 96 Ruby Zoe Folly Gender Coordinator ANPPCAN 9/19 97 Haje I. Paasewe Child Protection Coordinator ANNPPCAN 9/19 98 Mr. Clement Stephens Executive Director ANNPPCAN 9/19 99 Jane Kiplaght Exchange Participant (from Kenya) ANNPPCAN 9/19 100 Mabel M.K. Kear Master Trainer YMCA 9/24 101 Jerry Paye Mentor YMCA 9/24 102 Troy Johnson Program Coordinator IREX CSML 9/24 103 Jackie Eastman Grants Officer IREX CSML 9/24 Annex 5: Interview List 6 104 Korpo Watson Grants Manager IREX CSML 9/24 105 Alvin Attah Deputy National Authorizing Officer Ministry of Planning and Economic Affairs 9/25 106 Thomas Norris Data Officer, NGO Unit Ministry of Planning and Economic Affairs 9/25 107 Frank Suku Deputy Secretary, Programs Federation of Liberian Youth (FLY) 9/26 108 Mark Whitehouse Vice President for Media IREX/DC 10/11 Annex 6: CSML Partners Contact List CSML Civil Society and Media Leadership Program IREX/CSML PARTNERS MAIN PARTNERS Resource Partners New African Research and Development Agency (NARDA) Email: narda@nardanet.org Tel: 231 (0)777-013-630; (0)88-668-7779 Chief Executive: Lancedell Mathews 231 (0)777-020-338 Mentor CSOs and Community Radio Stations in: Montserrado and Bong Counties West Africa Network for Peacebuilding (WANEP) E-mail: wanepliberia@yahoo.com Chief Executive: B.B. Colley bbcolley@yahoo.com Tel: 231-(0)88-6551-520 Mentor CSOs and Community Radio Stations in: Lower Nimba, Grand Gedeh and River Gee Counties Women NGOs Secretariat of Liberia (WONGOSOL) Email: libngowomen@yahoo.com Tel: 231-(0)88-067-8701 Chief Executive: Marpue M. Speare mspeare2@yahoo.com Tel: 231-(0)88-653-8043 Mentor CSOs and Community Radio Stations in: Grand Bassa and Montserrado Counties YMCA Liberia National General Secretary: Edward Gboe Email: eegboe.ymcalr@gmail.com Tel: 231 (0)88-654-7249 Mentor CSOs and Community Radio Stations in: Upper Nimba and Lofa Counties MEDIA PARTNERS Liberia Media Center (LMC) Executive Director: Lawrence Randal Email: lmclib@yahoo.com Tel: (0)88-640-0206 Liberia Women Media Action Committee (LIWOMAC) Executive Director: Estella Nelson Email: tovianj@yahoo.com Tel: (0)88-081-0536 Annex 6: CSML Partners Contact List CIVIL SOCIETY ORGANIZATIONS Organization Contact Telephone Email County/Town First Cohort of CSOs Actions for Genuine Democratic Alternatives (AGENDA) Jasper Cummeh, Chief Executive jascum2001@yahoo.com 088-654-2933 077-714-4237 jascum2001@yahoo.com Montserrado, Monrovia Aiding Disadvantaged and Traumatized Women and Girls (ADWANGA) Amelia M. Cooper 088-651-1979 077-751-1979 adwangaproject@yahoo.com Nimba, Tapita AMU WULU Garley Marh 088-681-3655 garleymarh@yahoo.com Lofa, Zwedru Association of Liberia Community Radio (ALICOR) William M. Quire, President 088-659-3090 088-655-1594 alicorradio@yahoo.com Bong/Gbarnga Bassa Concern Citizen Movement Estella Palvey 088-652-4947 bassaconcim@yahoo.com Grand Bassa, Buchanan Bassa Women Development Association (BAWODA) Martha F. Karnga 088-684-5349 088-655-1263 bawodabuch@yahoo.com Grand Bassa, Buchanan Bong Youth Association (BYA) Gonda Seton 088-665-3490 byassociation2004@yahoo.com Bong, Gbarnga Buchanan Child Community Based Care (BUCCOBAC) David S. Mehdeh 077-735-6748 buccobac@yahoo.com Grand Bassa, Buchanan Catholic Justice and Peace Commission (JPC) Rev.Fr. Patrick M. Kabba, Chief Executive pamustakabba@yahoo.com 088-655-2620 088-650-0800 justiceandpeacecommission@yahoo.com Bong, Gbarnga Center for Justice and Peace Studies (CJPS) Joseph D. Howard, Chief Executive Jhowardsam2@yahoo.com 088-652-6151 088-652-6151 088-658-7343 088-652-0539 cjpstudies@yahoo.com Montserrado, Monrovia Center for Media Studies & Peace Building (CEMESP) Malcolm W. Joseph, Chief Executive malcolmjoseph2000@yahoo.com 088-651-4357 088-651-4357 centerforpeacebuilding@yahoo.com Montserrado, Monrovia Children Assistance Program Deroe A. Weeks 088-651-7134 daweeks2002@yahoo.com Montserrado, Monrovia Common Action Matthew Siakor, Jr. 088-688-8849 siakolism1@yahoo.com Lofa, Voinjama Development Education Network￾Liberia (DEN-L) Dorothy Toomann, Chief Executive 088-653-3140 088-655-7593 dev_edunet@justemail.net Bong, Gbarnga Ears for the Masses Darius Dan Wehyee 088-644-9818 earsforthemasses@yahoo.com Nimba, Sanniquellie Federation of Liberian Youth (FLY) Mohammed A. Nasser 088-653-7175 monasser2002@yahoo.com; mohammed.nasser@flyliberia.org Montserrado, Monrovia Inter-visionary Artists (IVA) Emmanuel A. A. Sarty 088-644-6485 emmanuelsarty@yahoo.com Nimba, Ganta Liberian Democratic Institute (LDI) Dan T. Saryee Sr. 088-6449-818 dsaryeeldi@yahoo.com Grand Gedeh, Zwedru Lofa Youth Education Awareness Zarballee K. Woiwor 088-682-8628 zarballeekwoiwor@yahoo.com Lofa, Zorzor Annex 6: CSML Partners Contact List MODIA Oscar Dolo 088-654-2628 snyepudolo@yahoo.com Bong, Gbarnga National Adult Education Association Liberia (NAEAL) Roye B. Bloh, Jr. 088-644-6231 naealiteracy@yahoo.com Montserrado, Monrovia Prison Fellowship Liberia Rev. Francis Kollie 088-656-6720 088-664-4108 Justiceliberia_access@yahoo.com Montserrado, Monrovia Rural Agricultural Alternatives Tom S. Koenig, Sr. 088-654-9319 tjkoenigi99@yahoo.com Lofa, Voinjama Save My Future Foundation Dekontee Zulu 0886-766-165 Samfu1@yahoo.com River Gee, Fish Town SEARCH Sondah Geepea-Wilson 088-658-6411 sondah-geepea@yahoo.com Nimba, Saclepea & Sanniquellie SEWODA Helena Turoh 088-655-4218 Sewoda2004@yahoo.com River Gee, Fish Town Village Development Fund Albert M. Cole 088-653-3354 villagedevelopmentfund@yahoo.com Voinjama United Muslim Women Marietta M. Williams 088-655-8859 umwaeo@yahoo.com Montserrado, Monrovia Second Cohort of CSOs Association Of Evangelicals Of Liberia Victoria H. Wollie 088-651-6699 aelliberia@yahoo.com Bong, African Network of Prevention and Protection Against Child Abuse and Neglect (ANNPCAN) Clement O. Stephens 088-659-2905 088-667-8679 anppcan.liberia@yahoo.com Montserrado Ballet Gedeh Rooter Culture Troupe Melvin Cleon 088-000-1219 ggculture2010@gmail.com Grand Gedeh, New Zwedru Bassa Youth Caucus T. Francis M. Garliah 088-664-3039 bassaycaucus@yahoo.com Grand Bassa, Buchanan Botanical Product Association Of Liberia Oliver P. Geh 077-782-8141 077-793-7112 Nimba, Zorgowee Community Safety Initiative Inc. Samuka B. Sannoh 088-649-4481 054-944-81 077-749-4481 ssannoh12@yahoo.com/csinitiative2005 @yahoo.com Lofa, Voinjama Community Development and Research Agency Lansana A. Dukuly 088-648-9687 088-646-3010 077-687-8956 codraliberia@yahoo.com, codraliberia@hotmail.com Bong, Gbarnga Community Empowerment and Sustainable Program James F Kamanda II jamesk@cespliberia.org 088-654-9997 jamesk@cespliberia.org Montserrado and Lofa Concerned Christian Community David G. Foday 088-633-6692 Montserrado, Bong, Lofa Consolidated Youth For Peace & Development Jerrison M. Davis 088-664-5483 Coyped.buchanan@gmail.com Montserrado Feeding House Shaka Romeo 088-804-4383 fhorganization@yahoo.com Grand Gedeh Flomo Theatre Peter Ballah 088-645-8745 flomotheater@yahoo.com/ Montserrado Fore-Runners of Children’s Universal Rights for Survival Anthony L.M. Boikal 088-653-6186 088-655-7820 088-053-2014 focusliberia@yahoo.com Montserrado Gbarzon Youth Abraham T. Chorbun 077-637-8148 077-795-1743 Liberia@respectrefugees.org Grand Gedeh, Polar Gbarzon Annex 6: CSML Partners Contact List Liberia Christian Handicapped Organization Fallah S. Boima Cymbian 088-658-3284 liho@licho-lib.org Lofa, Foya Liberia Motorcycle Transport Union Racck Martin Dolo 088-681-2043 Nimba, Ganta Lofa County Network of Local Non￾Governmental Organization Elijah M. Kwaytah 088-080-7929 lofanet2009@yahoo.com Lofa, Voinjama National Empowerment Program Sianeh Sanoe Omeze spomeze@yahoo.com 088-651-2609 077-654-6023 naepro@yahoo.com spomeze@yahoo.com Montserrado National Student Christian Council Isatu Ville 088-653-3846 isatuville@yahoo.com Montserrado, National Youth Action Alfred C. Kargbo, Jr. 088-656-0005 nyaliberia2009@aol.com Montserrado NAYMOTE Partners For Democratic Development Eddie D. Jarwolo 088-651-0479 naymoyeliberia@gmail.com Montserrado, Bong Peace Building Resource Center Stanley K. Beyan 088-647-0698 077- 747-0698 pbrcliberia@yahoo.com Lofa, Voinjama Rebuild Africa Williams Massaquoi 088-653-2075 billmass82@gmail.com Montserrado, Monrovia & Lofa Refugees Educational Sponsorship Program Enhancing Communities Together Shatha Karma 0886-203-757 sheben22@yahoo.com Grand Gedeh, New Zwedru Restoring Our Children’s Hope Kaifala Kromah 088-651-2997 roch.liberia2009@yahoo.com Lofa, Voinjama Rural Human Rights Activists Programme Lorma Baysah 088-651-5953 rhrapliberia@yahoo.com lormarhrap@yahoo.com Bong, Nimba, Montserrado Self- Help Initiative for Sustainable Development Jeremiah T. Burgress jburgess@shifsd.org 088-679-5895 shifsdliberia@gmail.com Grand Gedeh,Zwedru Society for Women and AIDS in Africa/Liberia Confidence Wallace 088-640-9360 swaa.liberia@yahoo.com Montserrado Sustainable Agricultural Program for Liberia Senehn Karlo 088-008-83827 liberiasap@yahoo.com Grand Gedeh, Zwedru Voice of the Voiceless Frances Greaves 088-651-6996 voiceofthevoiclessint505@yahoo.com Montserrado Volunteers To Support International Effort in Developing Africa Kortu Timothy, H. 077-706-1116 088-684-5344 info@vosiedafricaonline.org Montserrado, Bong,Lofa We-Care Foundation T. Michael Weah 077-702-4251 wecarelib@yahoo.com Montserrado Young Women Organized for Sustainable Development Frances Rogers 088-686-40354 ywoso2010@yahoo.com Monrovia, Montserrado Zorzor Women in Action for Assistance and Development Hannah S. Tamba 088-684-6540 077-770-5402 markelee2005@yahoo.com Lofa, Zorzor Annex 6: CSML Partners Contact List Media Organization Contact Telephone Email County/Town Female Journalists’ Association of Liberia (FEJAL) Torwon Sulenteh Brown Siatta Scott Johnson 088-651-4911 077-751-4911 088-657-4931 trowon@yahoo.com siatta74@yahoo.com Montserrado, Monrovia Clay Street Media Women Center for Development and Democracy. (MEWOCEDE) Gradiah Walker 088-655-6290 gradiahwalker@yahoo.com gradiahwalker@gmail.com Montserrado, Monrovia Clay Street Press Union of Liberia (PUL) Peter Qua Qua 088-652-9611 pqua31@yahoo.com Montserrado, Monrovia Clay Street Specialized Consulting and Business Development Initiative Partners Heritage Newspaper Mr. Mohammed Kanneh Managing Editor 088-653-3098 mmkanneh2004@yahoo.com Montserrado, Monrovia Broad Street Public Agenda Newspaper Mohammed Kamara, Managing Editor 088-680-5257 agendapublic@yahoo.com Montserrado, Monrovia Carey and Gurley Women Voices Ms. Helena Nah helenanah201@yahoo.com 088-651-5093 helenanah201@yahoo.com Montserrado, Monrovia Mechlin Street Informer Newspaper Mr. Darkollie Sumo darkollie2003@yahoo.com 088-651-9515 darkollie2003@yahoo.com Montserrado, Monrovia Johnson Street Sky FM and TV Mr. Martin Browne martinbrown54@yahoo.com 088-651-0487 martinbrown54@yahoo.com Montserrado, Monrovia Broad Street United Methodist Radio Mr. Julu Swen juluswen@yahoo.com 088-652-7042 juluswen@yahoo.com Montserrado, Sinkor 12 Street Non Profit Partners Liberia Women Democracy Radio Estella Nelson President/LIWOMAC Eliza Dahn Acting Program Director 088-0810536 088-652-4124 tovianj@yahoo.com elizadahn@yahoo.com Montserrado, Congo Town Radio Veritas Ms. Weade Kekulah Station Manager 088-657-9222 Adewede@gmail.com Montserrado, Monrovia Ashmun Street First Cohort of Community Radio Stations Lux FM Euriahs Togar 088-659-4314 Togar_euriahs@yahoo.com Montserrado, Monrovia Magic FM Christopher Yarwoe 088-682-4264 chrisyarwoe76@yahoo.com Grand Bassa, Buchanan Radio Gee George Kayee 088-645-6860 River Gee, Fish Town Radio Jorwah Tiletor T. Folonkpala /Mustaphar 088-689-0083 077-713-5784 Bong, Jorwah Radio Kintoma John Gayflor 088-643-8704 jgayflor@yahoo.com Lofa, Voinjama Radio Life Quaqua Miller 088-647-0467 tquaqua2255@yahoo.com Lofa, Zorzor Radio Peace Johnson Sharty 077-791-1697 Grand Gedeh, Toe Town Voice of Saclepea Robert Martally 088-645-3272 royoma2006@yahoo.com Nimba, Saclepea Annex 6: CSML Partners Contact List Second Cohort of Community Radio Stations Bong Mines Radio Alexander T. Kukah 088-644-8694 bongminecommunityradio@gmail.com Bong, Gbarnga Radio Karn Emmanuel Y. Karlea/Moore Tuah 077-759-6935 088-659-6935 077-728-0047 radiokarn@yahoo.com Nimba, Karnplay Radio Kerghemahn Nyahn Flomo 088-641-2908 077-63-46929 nayahnflomo@yahoo.com Nimba, Ganta Radio Sehnwai Lawrence Tokpah Jr. 088-660-6759 077-760-6759 Lawtokjr1982@yahoo.com Nimba, Saniquellie Radio Tamba Taikor Samuel N. Kemayah 088-665-8853 Lofa, Foya Radio Totota George P.S. Tarlee Sr. 088-625-3158 077-710-5555 radiototota@yahoo.com Bong, Totota Radio Vahun Abdus K. Sannoh 088-689-6389 Lofa, Vahun Super Bongese James Borbor Sao 088-648-7072 077-748-7072 054-87-072 Jdsao2005@yahoo.com Bong, Gbarnga Voice of Rural Montserrado Kassia N. Porte 077-703-75491 088-683-4493 Montserrado, Bensonville Voice of Tappitta Jusu Kpandwah 088-647-3103 077-777-77980 vottappita@yhoo.com Nimba, Tippita Voice of Webo Fahn Allison 088-629-2595 River Gee, Webo Annex 7: CSML Logic Model Outcomes for Media Component KEY CSML Outputs & Outcomes for Media Component: Project Resources Project Activities Project Outputs Short-term Outcomes (1 yr.) Intermediate Outcomes (2-3 yrs.) Long-term Outcomes (4-5 yrs.) IREX Staff Consultants Main Partners Trainers Mentors Technical Assistance (in media that means technicians also) Equipt.(computers, radio equipt., transportation, training materials, small grants etc.) Trainings Mentoring Elections￾related media programs CSO events in partnership with CRSs/Media Surveys/Polls # People Trained Men/Women (On￾going) # People Mentored Men/Women/CRSs (On-going) # CRSs with improved technology and other assets (On-going) # Elx results web hits/SMS utilized Expected Outcomes: *Awareness Change *Knowledge Change *Actions Journalists’ application of improved professional skills have resulted in content change at media level Audiences (geographic reach & those reached) have grown for community radio stations (no reported increases by newspapers we met with.) More women Journalists have been trained & have influenced/increased gender-related content in media along qualifying inputs but neither numerically quantifiable nor measureable against a quantifiable baseline measurement of content. CSOs and Media have strengthened networks and collaboration. Citizen access to government and other leaders is increased via media (and SMS in Elections add-on project). Media is strengthened and ability to monitor government & elections (and elections) is increased. (Elections add-on project). Expected Outcomes: *Awareness Change *Knowledge Change *Opinion Change *Attitude Change *Actions *Public Policy Change Project replicated in more media outlets. Journalists’ application of improved professional skills has resulted in both change in media content and some qualified change at institutional, gov’t and civil society levels. There are examples of greater dialogue between citizens & officials. Communities (audiences) more engaged via CSO/Media outreach and activities and exhibit more awareness on certain issues. Media skills ability toward more financial independence realized to some extent via increase in ability to manage finances, seek revenue streams (grants, advertising, community support, etc.) Note: financial independence not fully realized. There are new anecdotal actions among citizens due to influence of IREX CSML project but the highest levels of the theory of change model (attitudes and beliefs) have neither been measured along a baseline nor benchmarked on, for example, socio￾cultural lines, eg. Shifts in Attitudes and Beliefs over issues like: violence against women, religious identity etc., that intersect with CSML project results in policy, governance, etc. Expected Outcomes: *Awareness Change *Knowledge Change *Opinion Change *Attitude Change *Actions *Behavior Change *Belief Change: final stage of BCC Theory of Change and can take years to measure. *Public Policy Change (To be determined against an impact analysis). Note: No Baseline was conducted by which to do final/definitive measurement of attitude/behavior change along any particular socio-cultural levels and/or issue(s). Annex 8: CSML Logic Model Outcomes for CSOs KEY CSML Outputs & Outcomes for CSO Component: Project Resources Project Activities Project Outputs Short-term Outcomes (1 yr.) Intermediate Outcomes (2-3 yrs.) Long-term Outcomes (4-5 yrs.) IREX Staff Consultants Main Partners Trainers Mentors Technical Assistance Operational grants (RPs) Small grants (CSOs) In-kind grants (RPs) Assessment/Sel ection of CSOs Training (to RPs and CSOs) technical management (OD) Mentoring (to RPs and CSOs) Grants to RPs Grants-funded activities in advocacy reconciliation forum (with media) roundtable participatory democracy Capacity building for traditional leaders Support for FOI coalition # of training events mentoring visits  staff trained men/women (on￾going)  staff mentored men/women (on-going) grants awarded indirect beneficiaries from grant activities improved service delivery of CSOs Expected Outcomes: # CSOs showing institutional change in constitution proposal design capacity project management capacity reporting capacity advocacy capacity networking capacity Expected Outcomes: # CSOs showing institutional change in proposals funded by other donors range of core activities collaboration with media capacity to monitor government capacity to render greater services to members/constituents organizational culture Expected Outcomes: #CSOs with sustainable capacity to obtain funding directly from USAID and other donors initiate income￾generating activities (e.g., social enterprise) engage directly with media and government for promoting peace and good governance Annex 9: Summary of Media Interviews Media Interviews (9 Community Radio Stations, 1 commercial radio station, 2 Newspapers & ALICOR/Media Advocacy) Monrovia/Montserrado No. CRS City Cohort 1 or 2 Ranking Other 1 LWDR/LIWOMAC Monrovia 1 A ranking Has 2 2012 J Mentees & 1 2011 grad 2 LUX FM Montserrado 1 B ranking Has 3 2012 J Mentees & 1 2011 grad 3 SKY FM Monrovia 1 A ranking Has 3 Mentees 2012 4 Public Agenda Monrovia 1 N/A Has 3 2012 J Mentees & 2 2011 grads 5 Inquirer Monrovia 1 N/A 2 2011 J grads Grand Bassa 7 Magic FM Buchanan 1 C ranking Bong 8 Radio Totota Totota 2 C ranking 9 Super Bonges Radio Gbranga 2 A ranking Also conducted call-in show Nimba 10 Radio Senhwal Saniquelle 2 C ranking 11 Radio Nimba Saniquelle N/A Not incl. 12 Radio Kergheamahn Ganta 2 B ranking 13 ALICOR (media advocacy CSO) Monrovia/Ganta 1 N/A A ranking= Strong Capacity B ranking= Doing Relatively Well C ranking= Needs Extra Attention Annex 10: M&E Results Chart Title: No. Indicator RP1/2; CSO 1/2; Media1/2 B T A NM M E Comments 1 Average score on “Constituency Outreach System” and “Partnering, Advocacy and Networking System” in CSOs’ organizational capacity assessment 2 % of Liberian population who expressed satisfaction in CSOs representation of their interests 3 Average score on CSOs’ organizational capacity assessment 4 # of CSOs using USG assistance to improve internal capacity* 5 % of marginalized population who expressed satisfaction in CSOs representation of their interest 6 % of marginalized population who have attended or been engaged in civil society activities 7 # of USG assisted CSOs that engage in advocacy and watchdog functions at least two times in the last 6 monthly reports* 8 % of CSO staff who correctly respond to questions regarding the Liberian policy￾making cycle 9 % of CSOs who report that they have communicated/coordinated with government officials in at least 3 of the last 6 monthly reports 10 # of forums or meetings held with government in the 7 counties or at the national level by CSML or a CSML partner CSO 11 % of government officials who have positive impression of CSOs 12 % of population who responds positively to attitudinal surveys regarding perceptions of government (disaggregated) 13 Relative score of NTC trainees on index of key representational skills 14 % of Liberian population reached by targeted community radio partners 15 # of non-state news outlets supported by USG* 16 # of journalists trained with USG assistance* 17 Average score of content analysis from a sample of news reports and articles released by partner media outlets 18 Amount of advertising and additional fee revenue generated by participating outlets Annex 10: M&E Results Chart 19 % of participating outlets with a 3-5 year financial plan in place 20 # of media outlets that received USG supported training to promote financial sustainability* 21 Average score of content analysis from a sample of news reports and articles released by targeted media 22 % of female journalists in 7 target counties 23 % of marginalized population who expressed interests to be represented by community radio 24 Estimate for radio listening density per population of coverage area 25 % of station airtime devoted to community information, dialogue, or engagement 26 Average score of content analysis from a sample of news reports and articles released by targeted media 27 # of visitors accessing an online centralized media resource facility 28 # of people in target areas who access the mobile-based information distribution system and interactive tools per month 29 Number of measures taken by targeted groups to monitor and promote the Freedom of Information law 30 Number of people attending FOI awareness events 31 Number of FOI capacity building events held by TCC Annex 11: Focus Group of 2011 Journalism Mentorship Grads 1 2011 GRADUATES OF JOURNALIST TRAINING/MENTORING INVITED FOR A FOCUS GROUP SEPTEMBER 17, 2012 Mamba Pt. Hotel, Monrovia List of Graduates: Institution Telephone 1. Samuka V. Konneh : Public Agenda 0886422711 2. J. Dominic Farley Public Agenda 0886591661 3. William Sandi Kamara LWDR 0886976913 4. Rufus S. Davies LUX FM 0885840439 5. Winnie Saywah Inquirer 0886549526 6. Morrison O. G. Sayon Inquirer 0886511509 7. Emmanuel M. Weedee Heritage 0880450404 8. Theophilus S. Seeton Heritage 0886646005 9. Sam Z. Zota, Jr. News Newspaper 0886474563 10.Konah Browne Radio Veritas 0776040620 11. Joaquin M. Sendolo Daily Observer 0886838535 12. Varnetta M. Johnson LWDR 0886354421 13. Moses Owen Browne LWDR 0886493370 Note: BOLD denotes those who attended=9 Graduates. Of the 13 graduates, 3 were women. All 3 women attended the focus group. SEMI-STRUCTURED INTERVEW QUESTIONS FOR FOCUS GROUP: UNDERSTANDING CSML 1. How do you understand the role you served in the CSML project? 2. What do you see as the overall goal of the CSML project? 3. How do you see your reporting work, and the work of your media organization, as supporting good governance and civil society? ORGANIZATIONAL DEVELOPMENT INDIVIDUAL CAPACITY BUILDING 4. Please tell us what changes you have helped foster in your news organization since you have graduated from the CSML program. Program delivery—what has changed? Monitoring and Evaluation—what has changed? Capacity to monitor governmentwhat has changed? 5. Do you have a particular success story about your reporting that you’d like to share with us? Annex 11: Focus Group of 2011 Journalism Mentorship Grads 2 6. For the journalism training/mentoring which you participated in, what can you do, as an individual reporter, now that you could not do before receiving this training/mentoring from the CSML project? 7. What was the most useful knowledge you learned during your training(s) under the CSML project? 8. How have you put this new knowledge into practice? TECHNICAL TRAINING 9. Please describe any technical training you have received from IREX (NB: this would include advocacy, journalism, ability to represent member interests, diversifying constituents, capacity to better understand and influence policy, and engage more effectively with government; also on computer usage, new technologies, and new equipment) 10. SMS news reporting? Blogging? Twitter? Podcasts? MENTORING 11. Please describe the mentoring process that IREX and its main partner had designed for you. 12. Was the Mentor assistance a new kind of assistance for you? What is your appreciation of this assistance? How do you understand the difference between a trainer and a mentor? 13. How have you influenced the reporting your station and/or newspaper can now offer your audience? 14. Can you point to an example of how your coverage of an issue or story influenced subsequent discussions, dialogues and actions of your readers and/or listeners and of other key stakeholders (for example, elected officials)? SUSTAINABILITY 15. What is your organization’s plan for maintaining the benefits and strengthening you obtained from the CSML project once the project ends in 2014? 16. Are you able to pass along skills as a mentor yourself? 17. Sustainability of staff? Women reporters? Paid? Unpaid? LESSONS LEARNED 18. In your experience with the CSML project managed by IREX, what is the most useful lesson (or lessons) you have learned in being a partner in this project? Did you experience any disappointments that you like to see corrected for this year’s group of journalist mentees? MOVING FORWARD 19. New Media? Mobile Phone, citizen participation in content generation via mobile technology? 20. What is the single most important change you would make to the way in which the project works with individual journalist mentees. Annex 12: SOW Questions and Data Sources 1 SOW Questions for Results Areas Data type Data Source Commentary RA 1 Self￾administered organizational capacity assessment of the COS/PANS and OD M&E reports from IREX for performance indicators 1 and 3 Statistical results are mixed; team felt that evidence from interviews was not born out by statistical scores, and hypothesized that the data collection process was flawed Using the organizational assessments of the CSOs completed in year one as a baseline, what progress has been made to strengthen CSO organizational processes and culture, advocacy, and networking? According to CSOs and CRSs, has the training provided by Master Trainers been effective to enhance their organizational competencies? Questionnaire for CSOs and CRSs Key Informants, especially executive directors and program officers CSO responses uniformly positive about MT performance According to CSOs and CRSs, has the mentoring they have received helped them enhance their organizational competencies? Questionnaire for CSOs and CRSs Key Informants, especially program officers and financial officers CSO responses singled out report writing, budget preparation and project proposal writing as key skills acquired According to the RPs, has the organizational change mentoring provided to them by the IREX Civil Society Specialist provided them the skills required to be effective mentors? Questionnaire for Main Partners Master Trainers of Resource Partners Master Trainers responded positively to the mentoring process as an “engagement.” To what extent have the CSOs implemented gender strategies and with what impact on gender integration? Interview questions; team observations Key Informants, especially executive directors Strategies were informal except for women’s organizations – integration is nascent. To what extent have the grants successfully been employed to support advocacy campaigns, human rights, peace building and reconciliation and gender awareness activities, youth activities, networking with other CSOs and with CRS, work with local government and other community￾driven projects? MP and CSO Questionnaires Key informants, including grants managers at IREX; Directors of MP; CSO executive directors; examples of grants agreements Both cost recovery and fixed￾obligation grants (FOG) have proven effective in enabling CSOs to implement activities using KSAs from training and mentoring. How has the project benefitted CSOs and CRSs outside of Montserrado County? CSO and CRS Questionnaires; team observations Key informants in Bong, Bassa, and Nimba county CSOs and CRSs Training, mentoring, grants and elections assistance contributed to capacity of upcountry beneficiaries. RA 2 CSO questionnaires Key informants, especially executive directors Policy advocacy training has been effective in enabling CSOs to understand how to construct advocacy campaigns; some examples of monitoring government expenditures (e.g., in Nimba County); they need grants to put knowledge into practice. According to CSOs, to what extent has advanced capacity building in advocacy and networking provided to them increased their effectiveness as watchdogs in monitoring national and decentralized policy development, implementation, and enforcement? What further training do they say they need? What evidence that CSOs have been engaged in effective monitoring supports CSO questionnaires; team Key Informants Team has included a mini case study on Ears for the Masses (Nimba County) Annex 12: SOW Questions and Data Sources 2 SOW Questions for Results Areas Data type Data Source Commentary their progress as watchdogs? Especially in areas such as government expenditures and performance, gender based violence, and child trafficking? observations; case study data (e.g., Ears for the Masses) According to RP trainers, have CSOs developed the required skills to serve as watchdogs? Questionnaires for RP Master Trainers at RPs Yes. Policy advocacy training emphasized how to access officials and mount an advocacy campaign; taking action needs emphasis now till end of CSML According to media specialists, have CSOs developed the required skills to serve as watchdogs? Questionnaire for MPs in media; MO observations Master Trainers Yes – but continued emphasis on watchdog skills and application required. To what extent have CSO ability to network been strengthened by the project? Questionnaires for RPs, CSOs and CRSs Key Informants Networking is nascent, and improving, but needs more emphasis; “networking grants” are a useful tool toward this end. RA 3 Survey questionnaires Citizen Perception Survey According to the survey, 84% of county officials have a positive perception of CSOs and media. How effective has the training of county government officials on participatory democracy been to improve their perceptions of the role of CSO and the media? How effective has training of CSOs on participatory democracy been to improve their perceptions of the government? No data No data Citizen Perceptions Survey did not measure CSO perceptions of government What outstanding issues still need to be addressed by the project to achieve this result? Semi-structured questionnaire Key informant interviews More CSO-government and citizen exchanges required – e.g., joint training, roundtables, public forums. How successful have Roundtables been to improve perception and collaboration? Semi-structured questionnaire Key informant interviews More time is required to conduct roundtables and to ascertain impact. To what extent have citizens improved their perception of CSOs and government? Survey questionnaires Citizen Perception Survey 67% of citizens hold a positive perception of CSOs; 31% hold a positive perception of government To what extent have radio shows improved perceptions? Semi-structured questionnaire Key informants; listener call-in program News, interviews and call-in programs are contributing to improved perceptions. To what extent has education regarding access to public information improved perceptions? Semi-structured questionnaire Key informant interviews Information provided by government and transmitted through media is improving perceptions, but more effort and emphasis on access to information is needed. RA 4 CSO/CRS Key Informants Few stations are adequately pursuing private sector revenue-generating activities To what extent have the Specialized Annex 12: SOW Questions and Data Sources 3 SOW Questions for Results Areas Data type Data Source Commentary Consulting and Business Development Initiative and the Sustainability Initiative improved the professional operations of targeted media outlets? questionnaire though some examples exist (Super Bongese and Radio Gee). Donor-dependency persists. What progress has been made in the professional and independent behavior of media? CSO/CRS questionnaire Key Informants Media monitoring indicates consistent rise in performance levels now reaching the 3.0 range of 5.0 in latest report. What are citizen perceptions of the independence of media? CSO/CRS questionnaire Key Informants According to perceptions survey it is increasing. A boost in media confidence (and confidence in government) came from CSML elections results reporting and other related elections media activities. What progress has been made in the establishment of pilot subscription and distributions services? CSO/CRS questionnaire Key Informants Little activity achieved at this point. What progress has been made in the Women’s Media Initiative? CSO/CRS questionnaire Key Informants/Media Monitoring reports Overall, the largest indicators include: more women working in media (22%) some women promoted to editorial posts after CSML trainings and gender-sensitive training conducted with media managers/journalists as well as increased numbers of women utilized as sources in news and information reporting. What progress has been made in the Young Journalists Initiative? CSO/CRS questionnaire Key Informants Agreement signed with the Univ. of Liberia Mass Comm. Dept.— Initiative not yet being implemented. Have the investigative journalism training workshops had an impact on improving the reporting skills of journalists? CSO/CRS questionnaire Key Informants/review of newspaper investigative reports (Public Agenda) Evidence of increased investigative reporting by CSML participants—we found a couple examples of significant outcomes due to reporting. Has the on-line media resources clearinghouse been used and if so, what impact has this had on users in terms of expanding their knowledge and awareness and skills? CSO/CRS questionnaire Key Informants Development of this tool stalled post-elections project and is slowly grinding toward more completion. RA 5 CSO/CRS questionnaire Key Informants/CRS bookkeeping records Evidence of: gender￾sensitivity in staffing and programming/reporting, better To what extent are the model radio stations operating more effectively Annex 12: SOW Questions and Data Sources 4 SOW Questions for Results Areas Data type Data Source Commentary because of IREX support, training, and mentoring? (in few cases) bookkeeping in some cases, audience needs addressed in programming, better HR practices etc. To what extent are gender issues being more equitably covered? CSO/CRS questionnaire Key Informants/media monitoring reports/program logs Overall improvement among media outlets being monitored. Performance numbers have incrementally moved up on a checklist scale of evaluations. To what extent are the radio stations hosting shows that encourage citizen participation in representative government? CSO/CRS questionnaire Key Informants/program logs Call-ins remain a form of radio programming, CRSs increasingly collaborating with CSOs on public programming/participation but gender gap in participation persists. What additional skills does IREX need to help them develop? CSO/CRS questionnaire Key Informants New Media, Social Media, business acumen, etc. How do radio stations assess the assistance received by IREX to improve their skills? What do they say they need? CSO/CRS questionnaire Key Informants CRSs give high marks to CSML. Many request more equipment upgrades, seed￾money for revenue streams and some J’s request new media training. What equipment has been provided and how has this helped radio stations? What equipment is still lacking? CSO/CRS questionnaire Key Informants Equipment upgrades to some stations with cohort 2 pending: higher watt transmitters, motorbikes, new production studios, field recording gear etc. What has been the impact of the grant program on improving radio station capabilities? CSO/CRS questionnaire Key Informants In partnership with CSOs, collaborative grants have enhanced programming. RA 6 CSO/CRS questionnaire Key Informants SMS initiative progressing with organizational set-up however it has not yet translated into acitivies by CSOs/CRSs etc. What impact has the Mobile Phone Innovation Conference had on promoting SMS Technology? What impact has the innovation fund had to support small-scale pilots? CSO/CRS questionnaire Key Informants No pilots yet underway. How much progress has been made in the use of SMS technology? CSO/CRS questionnaire Key Informants/IREX qtly reporting Little, beyond elections project. RA 7 What progress that has been made since the passage of the Freedom of Information Act and to what extent citizens understand this law, have access to information, and are actually requesting information? CSO/CRS questionnaire Key Informants/TCC Gov’t implementation of FOI mixed, but examples of compliance exist. Media and CSOs report increase awareness and more examples of invoking it etc. No M&E being collected nationally to track FOI requests/outcomes. Annex 12: SOW Questions and Data Sources 5 SOW Questions for Results Areas Data type Data Source Commentary What progress has been made in establishing the broadcast Regulation Commission and what impact has IREX assistance had on this entity? CSO/CRS questionnaire Key Informants/Ministry of Information Law remains pending. No longer part of CSML program. What progress has been made in media freedom constituency building? CSO/CRS questionnaire Key Informants Greater work can be done to build coalitions of CSOs, Media Advocacy Orgs to continue to push for media reforms—especially in legal environment. RA 8 What impact did capacity building of debate moderators have on the electoral process? CSO/CRS questionnaire Key Informants/IREX post-elections project report Journalists and Media Moderators trained in effectively management debates, keeping sources accountable for statements etc. What impact did IREX’s support to the Election Reporting Center have on the electoral process? CSO/CRS questionnaire Key Informants IREX post-elections project report Results included: greater transparency of process, more connections between citizens and leaders, elections reporting. What was the overall impact of IREX assistance to the media on the transparency of the elections? CSO/CRS questionnaire Key Informants IREX post￾elections project report Most impactful outcome was increased public trust in government due to elections results that mirrored early results reported by CSML elections interventions. Appendix 13: List of CSOs and Media Outlets Interviewed and Criteria for Selection 1 Name of CSO/MO Type Sector Location Capacity Rating Mentoring organization Cohort African Network of Prevention and Protection against Child Abuse and Neglect (ANPPCAN) Chapter of International Umbrella - member/service delivery Youth Montserrado/Monrovia A NARDA 2 Association of Liberian Community Radio (ALICOR) Umbrella, member service organization Media Montserrado/Monrovia B NARDA 1 Bassa Youth Caucus Mass membership Youth Grand Bassa/Buchanan B WONGOSOL 2 Bassa Women’s Development Association (BAWODA) Mission/service delivery NGO Women's Empowerment Grand Bassa/Buchanan A WONGOSOL 1 Buchanan Child Community-based Care (BUCCOBAC) Mission/service delivery NGO Children's rights Grand Bassa/Buchanan A WONGOSOL 1 Catholic Justice and Peace Commission (JPC) Faith-based NGO Human & civil rights Montserrado/Monrovia N/A N/A N/A Community Development and Research Agency (CODRA) Mission/Service Delivery NGO Environment Bong/Gbarnga A NARDA 2 Concerned Christian Community (CCC) Faith-based NGO Human rights Montserrado/Bong B NARDA 2 Den-L Membership Human rights & Training Bong/Gbarnga N/A N/A N/A Ears for the Masses (EARS) Mission/service delivery NGO Human rights Nimba/Sanniquellie B YMCA 1 Federation of Liberian Youth Mass Membership Youth Montserrado/Monrovia A NARDA 1 Inquirer Newspaper Media Monrovia, Montserrado N/A N/A 1 Appendix 13: List of CSOs and Media Outlets Interviewed and Criteria for Selection 2 Name of CSO/MO Type Sector Location Capacity Rating Mentoring organization Cohort Inter-Visionary Artists (IVA) Mission driven & membership Performing Arts Nimba/Ganta B YMCA 2 Liberia Motorcycle Transport Union (LMTU) Union/mass membership Transportation & Safety Nimba/Ganta B YMCA 2 LUX FM Community service Media Monrovia, Montserrado (University of Liberia) B WONGOSOL 1 LWDR Community service; women’s rights Media Monrovia, Montserrado A LMC 1 Magic FM Community service Media Buchanan, Grand Bassa County C WONGOSOL 1 Public Agenda Newspaper Media Monrovia, Montserrado N/A LMC 1 Radio Kergheamahn Community service Media Ganta, Nimba County B YMCA 2 Radio Nimba Community service (not in CSML) Media Sanniquellie, Nimba County N/A N/A N/A Radio Sehnwai Community service Media Sanniquellie, Nimba County C YMCA 2 Radio Totota Community service Media Totota, Bong County C NARDA 2 Rural Human Rights Activist Program (RHRAP) Mission/service delivery Human Rights Montserrado/Monrovia A WANEP 1 SKY FM Commercial Station Media Monrovia, Montserrado A LMC 1 Super Bongese Radio Community service Media Gbarnga, Bong County A NARDA 2 United Muslim Women Advocacy and Empowerment Organization Membership, mission, service delivery Women’s Empowerment Montserrado/Monrovia A WONGOSOL 1 Voice of the Voiceless Mission/Service delivery Young Women’s Rights and Welfare Montserrado/Monrovia C WONGOSOL 2 Annex 14: CSO and CRS Data on Improved Organizational Development Capacity Key Issues for Capacity Building in the CSML project Federation of Liberian Youth United Muslim Women Voice of the Voiceless African Network of Prevention and Protection against Child Abuse and Neglect Association of Liberian Community Radio stations Community Development and Research Agency Rural Human Rights Activist Program BAWODA BUCCOBAC Bassa Youth Caucus Ears for the Masses Inter-Visionary Artists Liberia Motorcycle Transport Union Catholic Justice and Peace Commission Den-L Concerned Christian Community Magic FM Radio Totota Super Bongese Radio Radio Sehnwai Radio Kergheamahn Lux FM LWDR Sky FM Inquirer Public Agenda 1. Organizational Development Improved Mission, Vision, Core Values 1                     Improved personnel system 2    Improved financial management system 3    Improved report writing 4              Improved proposal preparation (including budget) 5              Improved operational planning 6   Improved M and E system 7   2. Training and Mentoring KSAs in policy advocacy 8      KSAs in gender sensitivity 9               KSAs in basic journalism 10          KSAs in project cycle management 11       KSAs in proposal writing          3. Sustainability Existence of a strategic plan 12   Income generation plans 13          Proposals for other donors 14             Commentary 10 IREX data show that journalists improved (by 25% of subject mastery to 38%) in pre- and post-tests for basic journalism trainings. 11 The team found that Executive Directors in particular learned much in these trainings. 12 Only two CSOs had a strategic plan; many requested further training in this area by CSML. 1 For improved mission, vision and core value statements, the finding is that these exist, have been improved by CSML, and for the most part were visible. Not all statements were in what the team would judge to be a final form; there were instances of far too many activities (e.g., BAWODA), and others had difficulty in conceiving of what a mission and a vision statement was (e.g., LMTU). But as a general finding, CSML has built capacity in this key area, for the most part across the board. 2 For improved personnel systems, several CSOs (including Ears for the Masses and LMTU in Nimba County) could show that they had no system before CSML training and mentoring; for others the evidence was anecdotal. The team did not examine personnel policies in depth in each CSO; it looked for evidence of change as a result of CSML training. 3 The team did not examine financial records at each CSO, but asked instead what CSOs and CRSs did differently in terms of financial management because of CSML. As with personnel systems, there were cases where a given CSO had no financial management system in place (e.g., LMTU); but all financial officers and Executive directors who attended the CSML trainings in financial management could point to improvements in their existing systems. 6 By operational planning we mean typical work plans which are project-specific. 14 Perhaps the single most important capacity buildin skill imparted by CSML across the board. 13 The team did not count proposal writing from CSOs in this category; it did look for examples of social enterprise or other income-generating ideas; in Bassa the team found several CSOs who planned to rent out their offices for meetings. 4 Many CSOs and all CRSs reported improved report writing. 7 Very few CSOs outside of Monrovia had M and E systems. 8 These skills were most pronounced among the advocacy CSOs. 9 The gender sensitivity gains were most pronounced among the CRSs. 5 This was another feature of the CSML training in which most CSOs affirmed improvement in proposal preparation, and of which the team saw visual evidence of newly completed proposals. There was even one instance (African Network of Prevention and Protection Against Child Abuse and Neglect) of a CSO which had attended a CSML training in proposal preparation, and had used the knowledge to revise a proposal for a UN agency, and which was successfully funded. this was a clear instance of an outcome attributable to CSML training. Annex 15: Call-in Segment on Super Bongese Radio 1 Call-in Segment on Super Bongese radio in Gbarnga September 14, 2012 8:30 a.m. to 9:00 a.m. The radio station agreed to offer a thirty-minute segment during its regularly scheduled ‘Sunrise’ morning program to two of the evaluation team members. Together, with the morning host, a so-called, ‘Question of the Day ‘ was asked of listeners. It was, “Is this particular radio station—and perhaps media in general in Bong County—doing a better job at creating a space for greater government transparency.” Below is a short synopsis of what callers said as well as gender identification. 1] Yes. Caller especially likes the program, ‘What’s on Your Mind,’ and says the chance to directly interact with officials is like a town hall meeting on the air. 2] Yes. Caller said he wants to see the government at the national and local level to provide budgetary support for radio—especially since government also relies on media as a tool. 3] Yes. Caller discussed the FOIA saying it is a new concept in Liberia and as people grasp it, learn about it and begin to put it to the test then information and media literacy also will grow. 4] Yes. Caller said even media are often at a disadvantage, like citizens, in accessing information from government without having to put in real effort which takes savvy, training, expertise and even sometimes invoking FOIA. If media are well-trained then access to information to citizens increases thanks to the information sharing by media. 5] Yes. Caller thanked evaluators for evaluating the media just as the media need to evaluate government. He said media monitoring is as necessary as government monitoring. Caller responded to a follow-up question about whether the media cover male and female officials with the same balance by saying that women in politics is still new in Liberia even though great strides have been made statistically in terms of the numbers of women representatives. He noted that while this results, then, in greater media attention given to women in leadership he believes there is a gender imbalance in terms of who participates in politics at the citizen level. (note, while the program host encouraged women callers, only males called in to the program during the thirty-minute timeframe.) 6] Yes. Caller said he has noticed more officials being featured on radio call-in programs. He noted that since Bong is geographically close to Monrovia, national office holders are more apt to be in their home area of Bong and thus more accessible. Annex 15: Call-in Segment on Super Bongese Radio 2 7] Yes. Caller said local radio recently exposed issues over mis-management of county funds and citizens were made aware of the issues and given the chance to voice their concerns. 8] Yes. Caller referred to recent media coverage over the national budget and recalled a meeting called by various CSOs that was covered by media and the Bong County Superintendent was called upon to disclose assets and information about the county’s development budget etc. 9] Yes. Caller said thanks to FOIA local officials had to disclose more information about the county’s social development fund that previously were not being disclosed before CSOs invoked FOIA. Caller said more awareness about FOIA is needed. 10] Yes. Caller said the work of both radio stations in Gbarnga make listeners feel like they are in Monrovia—since now information and good programming is happening here (Gbarnga) and not being created there (Monrovia.) He said he likes the morning ‘Sunrise’ program on Super Bongese radio. *Note: Since callers were asked for their opinions, and not questions, no text messages arrived from radio listener/participants. On the issue of a previous SIM card benefit provided by Cellcom to radio stations, allowing for free texts to be submitted by listeners wishing to ask questions, the evaluation team was told Cellcom switched to swapping fuel costs for advertisements due to abuses of the SIM cards by radio stations’ and/or staff who used the SIM cards for personal use as well. Even so, while the fuel cost support is welcome to the radio stations and an indirect benefit to listeners it has no direct benefit to listeners who previously had access a free call-in line. Super Bongese radio has not noticed a change in number/gender of callers to programs however. Annex 16: Questions for RPs - YMCA 1 Questions for RPs- YMCA (24 September 2012) Background Questions 1. Tell me about the YMCA – chapters, structure, activities, membership, and goals? 2. Why did you join the CSML project? What benefits did you foresee? How has your role evolved? Training Received by YMCA 1. Communications and Media Plan Development (December 2010) 2. Gender Awareness (February 2011) 3. M&E Training (August-September 2011) 4. OD Training (July 2012) Who attended the trainings? How have you applied the KSAs from these trainings in your work, and in the mentoring to your CSOs? A few examples, please? How have these trainings impacted activities in your annual work plans? Training for Master Trainers 1. TOT Adult Learning and Facilitation (October 2010) 2. TOT Policy Advocacy (May 2011) Questions for MTs 1. What do you remember from your Policy Advocacy Training? 2. How do you impart policy advocacy to your mentees? 3. What results are you seeing at the CSO level in their advocating capacity, and in their monitoring government performance? (Seek a separate interview with MTs). Result Area 1 – RP 5. When (date) did you complete your self-assessment capacity tool? a. OD b. Constituency System and Partnering, Advocacy and Networking System? 6. What were your baseline scores on these assessments? 7. Who administered the assessment – what’s your appreciation of the tool and the process? Had you ever undergone a process like that before? 8. Can you show me OD training module of July 2012 (YMCA conference room)? What did you learn that you didn’t know previously? How do you plan to apply this knowledge? Did you receive other OD training previously from IREX/CSML? If so, when? 9. If no, then how did you mentor your CSOs in OD? Annex 16: Questions for RPs - YMCA 2 CSOs 10. Which CSOs do you mentor? 11. When did you administer the self-assessment capacity tool? a. OD b. Constituency System and Partnering, Advocacy and Networking System? 12. What were the mean scores of your CSOs in the Baseline under each of these categories? 13. How much progress in your view have your CSOs made in OD, culture advocacy and networking? How do you measure and report on it? Can you give some examples of progress? 14. Tell me about the OD skills you acquired on mentoring from IREX - how have you used them? What can your mentees do now in OD that they couldn’t do before? 15. Can you show me Result Area 2 – RPs 1. For the RP Trainer: what are the skills you have imparted to CSOs to help them perform watchdog functions? How do you know whether they have acquired the skills they need to monitor government performance? For example, measurement, observations, stories, etc. What’s the evidence? 2. Can you show me a training model for policy advocacy training that you use with CSOs? Other 1. Grants – tell me which grants you’ve received, and the purposes they’ve served – how useful are they? How have they contributed to helping you achieve your goals? 2. FOI – any experience with this new law? Tell me about your work/mentoring of EARS. How have you supported the work that they are doing in Nimba County? 3. Media – tell me about your media partnerships? How do you utilize media to further your aims? Has media told your story on the air or in print? Does media engage you in any way, and if so, how?