DRG Learning, Evaluation, and Research Activity: Mid-Term Performance Evaluation of the USAID Media Strengthening Program Contract No. GS-10F-0033M/AID-OAA-M-13-00013 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP) Final Report (August 18, 2015) Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013, Tasking N022 Submitted to: USAID Submitted by: Susan Abbott (Principle Investigator) Audra Grant (Evaluation Specialist) Renée Hendley (Project Manager) Contractor: NORC at the University of Chicago Attention: Renée Hendley Bethesda, MD 20814 Tel: 301-634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 1 TABLE OF CONTENTS ACRONYMS ...................................................................................................................................................... 2 EXECUTIVE SUMMARY ................................................................................................................................. 3 1. EVALUATION OVERVIEW ................................................................................................................ 10 1.1 EVALUATION PURPOSE ................................................................................................................. 10 1.2 EVALUATION QUESTIONS ........................................................................................................... 10 2. PROGRAM BACKGROUND ............................................................................................................ 11 2.1 PROGRAM CONTEXT .................................................................................................................... 11 2.2 PROGRAM OBJECTIVES .................................................................................................................. 12 2.3 PROGRAM DESIGN AND MANAGEMENT ............................................................................... 13 3. EVALUATION METHODS ................................................................................................................. 14 3.1 EVALUATION MANAGMENT ....................................................................................................... 14 3.2 STUDY DESIGN ................................................................................................................................. 14 3.3 TARGET POPULATIONS ................................................................................................................ 15 3.4 LIMITATIONS ...................................................................................................................................... 16 4. FINDINGS,CONCLUSIONS AND RECOMMENDATIONS ..................................................... 17 4.1 EVALUATION QUESTION 1 (Result 1): ..................................................................................... 17 4.2 EVALUATION QUESTION 2 (Result 2): ..................................................................................... 25 4.3 EVALUATION QUESTION 3 (Result 5): ..................................................................................... 30 4.4. PRELIMINARY FINDINGS ON GENDER ................................................................................... 34 ANNEXES ....................................................................................................................................................... 43 ANNEX A: MSP ORGANIZATIONAL CHART ................................................................................ 44 ANNEX B: SCOPE OF WORK.............................................................................................................. 45 ANNEX C: DATA COLLECTION INSTRUMENT ........................................................................... 53 ANNEX D: OBSERVATIONS ON THE MCAT DATABASE ........................................................ 59 ANNEX E: SOURCES OF INFORMATION ....................................................................................... 62 MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 2 ACRONYMS APP Application CAICC Centro de Apoio a Informacao e Comunicacao CSO Civil Society Organization DAI Development Associates International DfID Department for International Development DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research Activity ECA Eduardo Mondlane University, School for Communications and Arts FOIA Freedom of Information Act FRELIMO Mozambican Liberation Front HQ Headquarters ICT Information Communications Technology ICT4D Information Communications Technology for Development IT Information Technology KII Key Informant Interview LAC Latin America and the Caribbean M&E Monitoring and Evaluation MCAT Media Content Analysis Tool MISA Media Institute of Southern Africa NGO Non-governmental Organization NORC National Opinion Research Center PMP Performance Management Plan RENAMO Mozambican National Resistance SOW Statement of Work USAID United States Agency for International Development USG United States Government MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 3 EXECUTIVE SUMMARY This report presents the findings of the Mid-Term Performance Evaluation of the Mozambique Media Strengthening Program (MSP), funded by the United States Agency for International Development (USAID). The MSP project, with a period of performance of June 10, 2012-June 11, 2017 and a budget of $9,978,124 million, is being implemented by the International Research and Exchange Board (IREX). The MSP is the largest activity in USAID/Mozambique’s Democracy, Human Rights, and Governance (DRG) portfolio. Launched by IREX in June 2012, its goal is “[a] free, open, diverse, and self-sustaining Mozambican media sector providing high-quality information to citizens that promotes debate, accountability, and transparency.” The MSP contributes to USAID broader objective of strengthening democratic governance of Mozambican institutions and second- and third-order aims of more effective civil society participation in governance processes and citizens becoming better informed of their rights and responsibilities. This evaluation of the MSP seeks to achieve two purposes. First, the assessment attempts to validate interventions that are valued by stakeholders and that contribute to desired results, and second, the study aims to generate evidence-based recommendations for improved imple￾mentation of MSP over its final two years. While the MSP identifies a number of result areas associated with this broader objective (professional capacity building of individual journalists; sustainability of media businesses; journalism education; community radios; media advocacy; and gender/ gender based violence), the current evaluation at USAID’s request, focuses on three main questions and their rubrics, specifically: (1) Increasing the level of professionalization of the Mozambican media sector To what extent have MSP capacity-building activities enabled media professionals to improve their capacity in targeted areas? Which components of the MSP intervention have been perceived as most useful and least useful for promoting professionalization and quality media content? And how responsive has MSP assistance been to the needs of the media sector according to target beneficiaries? (2) Improving the financial capacity of Mozambique’s media sector Do organizations perceive that their business management capabilities have improved as a result of working with the MSP, and if so, in what areas? Where do weaknesses still lie? Do organizations perceive that their financial viability has improved as a result of working with MSP? What specific changes have been implemented as a result of collaboration with MSP to improve business management capabilities and financial viability? What critical issues must outlets still address to strengthen financial viability? How could MSP more effectively contribute to greater financial viability of outlets? (3) Mozambican media organizations display an increased ability to advocate for press freedom and for a legal environment that enables media freedom How can the MSP strengthen activities to improve advocacy for media freedom and an improved legal environment? In other words, is the MSP exploiting all appropriate approaches possible to achieve Result 5? Are there international best practices that have MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 4 been effective elsewhere that may be applicable for the Mozambique context (of particular interest are approaches which involved building the advocacy capacity of local organizations)? This report details findings resulting from review and analysis of regular MSP and USAID reporting, face-to-face in-depth interviews with MSP project local partners and beneficiaries, site visits, and focus groups and small-group discussions among beneficiaries. KEY FINDINGS MSP is the first-of-its kind: no other media development program has offered a similar range of assistance in Mozambique. The comprehensive nature of MSP’s program design and the inter￾relationships among the result areas make for a robust strategy for media assistance. The Maputo-based focus of the program, combined with the Nampula field office’s support for community media in the Northern Provinces appears to have created a favorable foundation for future MSP support of media development activities. The mid-term assessment of the MSP finds that donor assistance is needed to support independent media professional development, organization growth, and sustainability. There is a great demand for media sector building activities and there are no Mozambican institutions with the current or near-term potential capacity to provide the kind of assistance provided by IREX, in order for the independent media sector to grow and flourish. Personnel changes resulted in delays in MSP implementation during the first two years (mainly in the start-up of the community radio and advocacy related activities), and the development of a strong, unified and qualified team took time. Despite this hurdle, the evaluation team finds efforts well-organized and well–coordinated at the MSP Maputo office. Similarly, observations of the MSP Nampula office reveal a tightknit team that is seemingly held in high regard in the community. It is fair to say, here, that despite initial reticence to IREX – as a non-Mozambican organization new to the country and in some cases outright resistance to IREX and MSP – IREX has earned high esteem among the media community, not just in Nampula, but throughout the country. This was accomplished by providing media professionals and outlets quality technical assistance and strategic provision of much needed material assistance (e.g. offering equipment grants). This support not only improved operations, but it also increased awareness of the need for improved professional standards of reporting and better business management creating demand for yet more capacity-building opportunities. MSP journalism training is lauded by online, print, TV and radio professionals interviewed across the board. The evaluation team notes that in every interview, subjects mention how valuable the MSP trainings are for significantly enhancing skills and capacity. IREX has raised the bar of professional training opportunities organizing not just effective training seminars but also offering mentorships and internships within IREX and media outlets which are not readily available elsewhere. Moreover, there is demand for MSP-trained journalists. Media managers interviewed expressed their preference for journalists that have completed the MSP journalism training as opposed to journalists who have not. Media professionals continue to face many challenges to improve the quality of media content. Respondents repeatedly highlight the lack of basic equipment. They laud MSP’s provision of equipment as essential for not just improving content, but also business viability. Media outlets, print, TV and community radio emphasize the value of MSP’s provision of laptops, furniture, MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 5 recorders, cameras, and other forms of small equipment and modernizing technology. It has helped to modernize media practices and streamline day-to-day operations in terms of both content production and business management. The NORC evaluation team observed a wide variety of types of equipment and technology used by media outlets. To put the need in perspective, some outlets that the NORC team visited were still using FAX machines and working from hand-me-down desks that they had found along the way. Community radio stations were in most desperate need for equipment. Support is also high among beneficiaries for MSP activities related to business and financial viability. Media outlets are enthusiastic about the opportunities to improve their practices and approaches to the business dimension of media company operation. In particular, many media organizations looked to MSP for guidance with learning about up-to-date, efficient business practices, such as effective collection and use of marketing data, sales trends, audience and circulation figures, etc. General managers and heads of media outlets are receptive to training more staff in these areas. While many media outlets need more staff and resources devoted to business operations to more substantially augment staff expertise on the non-journalistic aspects of running a media outlet, nevertheless, nearly all report that MSP’s assistance improved their financial viability and helped their operations significantly. Financial viability persists as a very serious issue for independent media outlets which they struggling to address. Although, the print and TV organizations interviewed seek domestic and international1 investors, and generate income from selling advertising locally and through subscription services, revenue is generally insufficient to meet business needs. There are a number of key obstacles for media organizations to overcome, chief among them developing suitable business models that will allow them to grow and sustain their operations. In addition to the areas already being covered under MSP activities, access to market and audience research data is a necessity. Yet, currently, sound, trusted, and accessible market and audience research data is not widely available. The advocacy component of MSP has had less impact than desired. This stems is no small part from the relatively meager state of the civil society advocacy for press freedom and access to information, hence reinforcing the need for MSP’s involvement in this area. Of the roughly 10 advocacy associations devoted to media freedom in Mozambique, all can be described as weak as underscored by international and local experts working on media- and communications development-related projects in Mozambique. Adding to this, to date, MSP advocacy activities have been both numerous and diverse. Some activities have not had a clear link to the core goal of increasing Mozambican organizations’ ability to advocate for freedom of speech and an improved legal enabling environment for media. MSP stands to make significant contributions with regard to gender in terms of both women’s participation in the media sector as well as integration of a gender perspective into content coverage and content selection. According to beneficiaries interviewed, women’s participation is lacking in the Mozambique’s media sector and change in this area is expected to be very gradual. Although a formal assessment of the MSP gender activities and results is beyond the scope of this evaluation, preliminary findings from interviews suggest that journalists and                                                                  1 By law, foreign investment in media outlets may not exceed 20%. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 6 managers at media outlets are widely supportive of female participation in the media sector at all levels. The ability to leverage monitoring and evaluation (M&E) data is limited at this time. Improved data collection is essential for more rigorous analysis of impact. While the evaluation team found qualitative data indicating that MSP activities are achieving its objectives of improving journalism and business viability, it is not yet possible to quantify precisely the extent of that change. The MSP has faced many challenges in administering IREX’s Media Content Analysis Tool (MCAT) due to a number of factors including the difficulty of tailoring a research framework suitable for Mozambique, the initial lack of well-trained cadre of coders to maintain the data collection over time, and the complexity of putting together a useful sampling frame that would allow IREX to track the progress of its interventions against different training and media types. In addition, the M&E plan initially did not have an effective means of tracking progress on business operations. MSP is working to address these issues. It retooled the MCAT, developed a more comprehensive business viability matrix comparable to its model community radio station matrix and is currently seeking to ensure more effective implementation of these and other M&E tools. CONCLUSIONS Qualitative findings indicate that MSP is contributing to improved journalism responding to a wide range of needs. While MSP has provided extensive training at the introductory, mid- and senior-levels as well as across a variety of types of outlets (i.e. print, TV, online, and community radio), it has not yet saturated the need for continued work to strengthen content quality. Not only does the need for MSP-like support persist, but there are no other organizations – local or international – in a position to readily respond to that demand. With regard to financial capacity, the NORC team found that MSP has also helped to improve the business sustainability of partner outlets who readily acknowledge MSP’s contributions to their successes. Despite an effective MSP approach, MSP partners continue to face some significant challenges to their financial capacity and long-term viability, especially for independent TV. While most interviewees stated that their organizations would find a way to continue operations, even amidst financial difficulties, sustained donor support in this area would assist with their ability to continue basic roles and functions (e.g. permit attention on quality and editorial independence). MSP has just scratched the surface of its work in building financial capacity, and the MSP team appears to have developed the types of relationships needed to work closely with independent media to enhance long-term financial viability and business management practices. MSP needs to be given time and space to grow those ties and continue its work with its select media houses in a more robust and targeted way to yield yet greater gains. In addition, because media outlets have limited access to capital and the expense of technology in Mozambique, there also remains a significant role for donors and NGOs like IREX to support media outlets’ professional and financial evolution through providing modern equipment and digital technology when strategically complemented by journalism and business management training. Research findings also indicate a need for IREX to continue, if not expand yet further, its efforts to promote increased women’s participation in the media sector and application of a gender perspective in the production of media content. Not only are women media professionals eager to work with MSP and most male counterparts interviewed are supportive of their MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 7 expanded participation, but also reporting would be further strengthened by the application a gender perspective when selecting content and deciding how to cover it. An opportunity exists to collaborate with Mozambican women media leaders. These women can serve as role models for women seeking a career in journalism and highlighting these individuals can encourage women and men to reconsider stereotypes about who should be in a newsroom and managing media outlets. Another area for possible attention is gender policies within media outlets. The lack of gender policies in individual MSP partner media organizations makes it particularly hard for supportive media professionals to impel change within their outlets. With regard to M&E, while important improvements have been made, further refinements are needed. The tools created using benchmarks and milestones to track changes in business viability for Maputo-based outlets and community radio stations address a key need and are promising. This different approach will give much better data on how MSP is contributing (or not contributing) to achieving desired results. The evaluation team believes it will be essential for MSP to have a common list of core areas of development that give definition to the milestone and benchmark tool. The milestone / benchmark indices that will be used for core MSP partners receiving strategic support in the final years of the MSP will give perspective on these differences. Content analysis is a vital means of understanding shifts in professional capacity among journalists participating in MSP activity but doing it well remains a challenge. Mitigating coder errors and bias will be an issue for any content analysis exercise regardless of the barometers used. Lastly, M&E data could be better disaggregated. At this time, it is difficult to compare data, as each media outlet MSP works with is distinct. The MSP engages small, medium, and large outlets, and also PDF / FAX-based, online, and TV media outlets. These basic features shape the nature of MSP’s work with outlets as does outlets’ varied baseline capabilities. M&E data that distinguishes between the characteristics of media outlets, in addition to, demographic characteristics of journalists would be useful. RECOMMENDATIONS There are a number of obstacles for media organizations to overcome, chief among them developing suitable business models that will allow organizations to grow and sustain their operations. In this regard, having access to get market and audience research data is a necessity. Currently, sound, trusted, and accessible market and audience research data is not widely available to private media. Mozambican partners would, therefore, benefit from assistance with developing or acquiring market and audience research data. USAID should explore with IREX if and how this issue might be addressed. Mozambican print media, particularly from the independent sector, struggle for a variety of reasons, namely from lack of advertising revenue, and lack of access to a reliable, affordable, and local printing press. Currently, most independent print media travel to South Africa to have their products printed. Some journalists the evaluation team spoke with contend that it would be better if they could have access to an affordable, local printing press for their use. Research and analysis by MSP is recommended as a next step to assess whether this would be desirable, and how such an initiative would boost growth and sustainability for independent print media. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 8 IREX should push forward with the revised MCAT, but senior IREX management must closely supervise the training and monitoring of coders. The current method for administering the MCAT, collecting and analyzing data, and integrating the findings into program management and strategy shows some limitations and weaknesses. At the time of this evaluation, the MSP was shifting between an original MCAT tool that used a Likert scale-type of measurement instrument, and a new MCAT tool that scored articles dichotomously. This shift was due to dissatisfaction with the scoring results obtained using the first MCAT. The evaluation team finds that regardless of the tool, the MSP should be more open to the tendency of varying quality of data coming in, and that efforts to get reliable information from the MCAT will require patience, time, and training. Some degree of error is expected with virtually any kind of content analysis, which is commonly used among social scientists measuring journalist capacity. To minimize the error and frustration experienced in doing the content analysis so far, MSP’s use of the MCAT would benefit from increased and periodic training for staff as well as expert supervisory support for coders. These changes may go a long way in minimizing production of unreliable data. Given the expressed utility of the MSP training, limited university-level journalism education, and what appears to be a nascent demand for MSP-trained journalists among hiring managers, the evaluation team recommends that MSP journalism practitioner training continue. The trainings – considered one of the most impactful types of MSP assistance – should include more instruction offered on special topics such as health, gender and corruption. Specialized instruction was specifically mentioned by MSP beneficiaries, who explained these would address the needs and issues that matter most to them and, importantly, are related to the type of news and information they find most important. Gender issues came up frequently during interviews and site visits. To a good extent, the MSP already offers this type of tailored, and issue-focused training. On matters of health and corruption, as noted by several journalists interviewed, the new information law will open opportunities to learn more about HIV/AIDS statistics, governance and transparency issues, and allow access to information on other matters. The evaluation team also finds that MSP’s now more narrowed focus on a select group of journalists and outlets is beneficial to accomplishing program goals effectively. However, IREX should selectively continue activities that are more open to avoid over selection, ensure opportunities for new partners and demonstrate IREX’s commitment to the sector as whole. IREX should continue its efforts to engage Mozambican partners to not only inform them of MSP activities, but also to actively bring them into the design process. To that end, expanding various forms of participant feedback will help MSP further improve its efforts to date. The MSP could also contemplate ways of more consistently gauging feedback from journalists, editors, and owners / managers; they all will have different viewpoints regarding priorities and concerns related to MSP assistance. A more holistic approach to M&E will be required in order to adequately answer questions on whether the MSP has achieved desired results. That said, station managers and CEOs from STV, Debate, Echo TV, TV TIM all conveyed that having MSP is helpful to their efforts to run their media outlet as a business. Further, there is really no other organization in the country offering the type of consultations and services as does the MSP. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 9 The evaluation team recommends that the MSP streamline its efforts to aggressively focus on supporting initiatives and organizations that have a clear directive on media freedom and access to information advocacy, policy reform, and research. Fortunately, given the revitalization of Mozambican activism on these issues with the recent defamation cases of Castel-Branco and Mbanze and passage of the right to information law, MSP has a window of opportunity that it should continue to seize. Finally, to further strengthen MSP’s contributions to gender equality, MSP should not only continue its current efforts but also explore additional avenues. One, MSP should explore how to highlight female media sector leaders so that they can serve as role models for younger journalists, female and male, as well as explore the feasibility of some type of mentoring program. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 10 1. EVALUATION OVERVIEW 1.1 EVALUATION PURPOSE This Mid-Term Performance Evaluation of the USAID Media Strengthening Program (MSP) seeks to achieve two purposes. First, the assessment attempts to validate interventions that are valued by stakeholders and that contribute to desired results, and second, the study aims to generate evidence-based recommendations for improved implementation of MSP over its final two years. 1.2 EVALUATION QUESTIONS In order to encourage a more open, diverse and capable media sector in Mozambique, the MSP design focuses on achieving five main results.2 The current USAID, evaluation, however, focuses on three main questions and their rubrics, which are outlined below.3 (1) Increasing the level of professionalization of the Mozambican media sector To what extent have MSP capacity-building activities enabled media professionals to improve their capacity in targeted areas? Which components of the MSP intervention have been perceived as most useful and least useful for promoting professionalization and quality media content? And how responsive has MSP assistance been to the needs of the media sector according to target beneficiaries? (2) Improving the financial capacity of Mozambique’s media sector Do organizations perceive that their business management capabilities have improved as a result of working with the MSP, and if so, in what areas? Where do weaknesses still lie? Do organizations perceive that their financial viability has improved as a result of working with MSP? What specific changes have been implemented as a result of collaboration with MSP to improve business management capabilities and financial viability? What critical issues must outlets still address to strengthen financial viability? How could MSP more effectively contribute to greater financial viability of outlets? (3) Mozambican media organizations display an increased ability to advocate for press freedom and for a legal environment that enables media freedom How can the MSP strengthen activities to improve advocacy for media freedom and an improved legal environment? In other words, is the MSP exploiting all appropriate approaches possible to achieve Result 5? Are there international best practices that have been effective elsewhere that may be applicable for the Mozambique context (of particular interest are approaches which involved building the advocacy capacity of local organizations)?                                                                  2 These include: increased professional capacity of the Mozambican media sector; strengthened business management/organization capacity of media organizations to improve long-term financial viability; improved journalism education at Eduardo Mondlane University’s School for Communications and Arts (ECA); increased ability of community radio stations to provide more and better information to listeners; and an increased ability of Mozambican organizations to advocate for press freedom and an improved legal enabling environment for media. Additionally, MSP aims to achieve cross-cutting results related to gender and public health and HIV/AIDS. 3 Result 3 and Result 4 concerning improvement of the journalism education at Eduardo Mondlane University and the increased ability of community radio stations to provide information to listeners will not be examined, as there has not been enough progress along these objectives. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 11 2.PROGRAM BACKGROUND 2.1 PROGRAM CONTEXT The Media Strengthening Program (MSP) is the largest activity in USAID/Mozambique’s Democracy, Human Rights, and Governance (DRG) portfolio. MSP was launched by IREX in June 2012 with the goal of, “[a] free, open, diverse, and self-sustaining Mozambican media sector providing high-quality information to citizens that promotes debate, accountability, and transparency.” The MSP contributes to USAID goals of strengthening democratic governance of Mozambican institutions and second- and third-order aims of more effective civil society participation in governance processes and citizens becoming better informed of their rights and responsibilities. UNESCO notes, as part of the organization’s efforts to support the post-2015 Sustainable Development Goals, that good governance is generally understood by many observers and practitioners as a public’s ability and willingness to promote rule of law, free speech, and government accountability and transparency.4 The protection and practice of freedom of speech is considered to be a cornerstone of good governance, but press freedom is equally important to a society’s ability to foster good governance. After all, a free and independent media provides citizens with a platform with which to contribute to, monitor, discuss, par￾ticipate and influence decision-making that affects society and its well-being. However, the extent to which press freedom can contribute to good governance depends on the extent to which the public has free and easy access to a variety of information sources and even platforms. Press freedom and good governance are therefore mutually supportive in promoting the public good. Relatedly, journalists, as members of the media, are conduits for this process. Journalists and the outlets that convey their reporting are essential for the transmission of information and ideas, providing an opportunity for discussion of a range of topical issues. The pre-emption of journalists’ ability to further information and debate inhibits an environment where good governance and political and economic development can occur. The implementation of the MSP occurs at a critical period in Mozambique’s continued post￾colonial transition. The legacy of Mozambique’s Marxist, one-party state created at independence state is still felt today in the media sector, despite the two decades since the inauguration of multi-party politics. The effects of this legacy continue to reverberate across all sectors of society, amid tension between state-led and market-, private-sector approaches to managing the Mozambique’s key institutions. State-run media, which many feel is biased in how it covers news, including TV, radio and print, dominate the Mozambican media landscape; however, private media in Mozambique has expanded rapidly over recent years. In addition, state media institutions increasingly face competition from the independent, private sector media, especially with the emergence of digital media platforms, and increasing use and availability of social media, especially Facebook. Media penetration is more limited in the                                                                  4 See Free Media Contribute to Good Governance, Empowerment and Eradicating Poverty, available at http://www.unesco.org/new/en/unesco/events/prizes‐and‐ celebrations/celebrations/international‐days/world‐press‐freedom‐day/2014‐themes/free‐ media‐contribute‐to‐good‐governance‐empowerment‐and‐eradicating‐poverty/. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 12 country’s rural sector, however. Mozambique’s legal framework, as articulated in the country’s constitution, provides protection for freedom of expression, freedom of the press and access to information. However, libel laws are an obstacle to freedom of expression. Despite constitutional protection for free speech and the right to information law, advocacy and the practice of press freedom is dampened by historical legacies that have silenced critical voices and a media landscape that makes it hard for independent, private sector media to compete with state media that is often revered as biased and partial to those in power. The legacy of Carlos Cardoso, a key figure in Mozambican politics and journalism, also continues to resonate with the public. Murdered more than 15 years ago for allegations he made as a journalist while investigating a corruption and banking scandal, Cardoso embodies much of what independent journalists aspire to – truthful, open expression, integrity, and a drive to report on matters of public interest that will bring about social change. Media freedoms are challenged by limited protection, state interference, and polarization, due to the ongoing struggle between Mozambique’s main political parties, the ruling Liberation Front of Mozambique or Mozambican Liberation Front (FRELIMO) and oppositionist, Mozambican National Resistance (RENAMO). It is within this context that USAID endeavours to promote a more vibrant, robust and diverse media sector through the implementation of the MSP. The March 2015 assassination of Gilles Cistac, a well-known constitutional lawyer and advocate for decentralization of power to Mozambique’s provinces, may also stifle advocacy and practice of media freedoms in the country, as will the defamation trial of Carlos Nuno Castle Branco and related prosecution of news publication editors Fernando Mbanze and Fernando Veloso.5 More recently, in August 2015, the assassination of veteran Paulo Machava, who most recently worked as the editor of the online publication Diário de Moçambique, signals a very worrisome threat to press freedom and efforts by programs like the MSP to support the development of free and independent media. 2.2 PROGRAM OBJECTIVES The MSP completed two years and 10 months of implementation by April 2015, or just over half the life of the project, which is the time during which the mid-term evaluation started. The first four months of MSP were considered the initiation or “start-up” phase, which is included in Work Plan1a. This period consisted of largely setting up the facilities and forming the initial implementation team. Work Plan1b corresponded to FY2013, during which MSP activities increased in number and intensity across the program result areas. As such, this midterm evaluation only focuses on a subset of 3 of 5 program objectives, as described, below. (1) Increased levels of professionalization of the Mozambican media sector. (2) Improved financial capacity of Mozambique’s media sector. (3) Mozambican media organizations display an increased capacity to advocate for press                                                                  5 Academic Carlos Nuno Castel-Branco published a critical Facebook post in November 2013, calling for the resignation of President Armando Guebuza. The post was re-printed in Mediafax and Canal de Mocambique news publications, run by editors Fernando Mbanze and Fernando Veloso. While Castel-Branco is being accused of crimes against national security, the editors are accused of abusing media freedoms. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 13 freedom and for a legal environment that enables media freedom. The MSP program design and activities focus on independent media, though state media is welcome to attended trainings and has attended workshops, events, and other forums. 2.3 PROGRAM DESIGN AND MANAGEMENT PROGRAM MANAGEMENT AND OVERVIEW International Research and Exchange Board (IREX) Funded by USAID, MSP is being implemented by IREX. It is a 5-year program that began June 10, 2012, and ends on June 11, 2017. The MSP currently employs 32 staff, including 28 nationals, 2 expats and 2 long-term consultants (one international consultant and one local consultant). Of the 32, some 10 may be considered senior staff (director / specialist / program manager / program coordinator) and 22 junior or support staff. All junior staff are being systematically trained to take on progressively more responsibility. One of the main hurdles to MSP implementation has been staffing / personnel changes. The MSP had a few different staffing changes that caused some disruption and delay to program execution in the first year of the project. IREX’s general intention is to make no significant staffing changes to this over the next 18 months, 6 except that media specialist Selma Inocencia, will be migrating over to STV while maintaining a part time role with MSP and continuing her collaboration with SOICO/STV from the other side. A main strategy change, going forward, is a narrowing of focus to a smaller group of high potential young journalists (30-40 people), who IREX intends to bring up to international standard through intensive training. IREX also intends to reduce the number of media outlets it provides strategic and targeted assistance to, with the aim that by concentrating on those most committed outlets that are more likely to put systems in place that will lead to sustainable, commercially viable media by the project’s end date. Program Implementation Team The IREX/MSP team is managed by a team headed by a Chief of Party, Director of Administration and Finance, Director of Capacity Building, Director of Sustainability, Program Manager for Nampula office, and M&E Coordinator (see Organizational Chart, Annex A). The main office for the MSP is based in Maputo, while the field office is located in Nampula.                                                                  6 As of the mid-term, the MSP has made a number of changes to how it has organized its staff. The main change is that they have decided to re-focus Ricardo Mendes on “direct” journalism activities (i.e., activities that involve capacity building of journalists, reporting and the production of journalistic material) and put Sergio Chusane in charge of indirect journalism activities to maximize the value of Mendes’ 30 years of journalism experience. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 14 3.EVALUATION METHODS To gather data required for this evaluation, NORC’s evaluation team approach included several techniques which entailed a mix of mutually reinforcing qualitative and quantitative methods that reflect the program logic, research questions being addressed, and indicators. The strategy combined the results of each technique to capture the diversity of opinions and perceptions of beneficiaries and stakeholders about the impact the MSP has had thus far, its strengths and weaknesses, and the degree of satisfaction from participants of the program. The qualitative analysis – which draws primarily on KIIs and focus groups – provides information about the local context and provides concrete examples that elucidate in greater detail the quantitative findings. Our approach to selecting the appropriate methodology is based on the USAID Evaluation Policy as well as our experience conducting evaluations in the field. The NORC evaluation team conducted the evaluation in a participatory manner which involved engaging the USAID, implementing partner IREX, program beneficiaries, and other stakeholders. A complete list of documents the evaluation team reviewed and individuals interviewed is included in Annex E, Sources of Information. 3.1 EVALUATION MANAGMENT One international expert conducted the evaluation with the assistance of local coordinators in Mozambique. NORC also provided management support and analysis for the evaluation. 3.2 STUDY DESIGN Key Informant Interviews The first main component of the evaluation was a series of key informant interviews (KIIs) with active MSP participants. The KII questions focused on the interactions with the program, views of MSP interventions, and the impact of those interventions (KII interview guides are provided in Annex C of the report.) The KIIs during field visits were conducted in-person either at the MSP beneficiary and partner location, MSP offices, or at MSP training events. Focus Group Discussions The second feature includes the conduct of four focus group discussions in Nampula. The groups were composed of members from independent print media, independent radio, and media associations.7 Respondents for most of the groups were chosen based on their active participation in MSP program activity. Each group lasted approximately 90 minutes. The Media Content Analysis Tool (MCAT) A third methodological dimension of the evaluation is analysis on data and information from the MSP’s MCAT, in order to assess changes in the level of professionalization of MSP beneficiaries, specifically, journalists who participated in MSP trainings and other activities. There are two MCAT databases designed by IREX to measure change in journalists’ professional capacity over time. The original database is intended to measure professionalism of the Mozambican media sector among other goals8 is comprised of articles written by                                                                  7 The evaluation team attempted to conduct two focus groups in Maputo, but responses were poor. 8 The MCAT database was also intended to measure financial capacity of media organizations. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 15 journalists that have been exposed to MSP assistance and those who have not. The focus was not on MSP-trained journalists. The overarching goal of using the MCAT is to measure and account for any changes (or lack thereof) to improvements related to quality of content. This database tracks progress from 2012 to 2014, according to four criteria: impartiality, relevance, sources and structure. Articles are randomly selected by four coders that rate the materials in each category using a five-point scale. All told, there are 6,056 records. A newer database, created in 2015 focuses on articles written by journalists who have participated in MPS activities. This newer database, under construction at the time of the evaluation analysis, also assesses progress along the same four criteria. The difference for this newer database is that, for each criteria, there are three to seven questions that are answered dichotomously rather than by a Likert scale. Each criterion is then the average of the question scores comprising the criterion, and, so, can vary from 0 to 1. The NORC analysis first includes a probit regression and then estimates an ANCOVA regression to analyze whether MSP training had an effect on journalist performance over time that could be attributed to the program (i.e., had an “impact”). This approach required making some assumptions about certain variables and information given in the new MCAT dataset, which was being constructed during the evaluation. 3.3 TARGET POPULATIONS The target population for the KIIs included individuals from various parts of Mozambique’s media sector in order to capture diversity of the program. Interviewees represented several types of media organizations engaged with the MSP, as well as those who serve in various capacities, ranging from journalists, to sales and marketing representatives, editors, and chief executive officers (See Table 2, below). TABLE 2. Target Population for KIIs Institution Type Total Print Media 4 Radio Stations 1 Community Radio 7 Online Newspapers/Magazines 2 Marketing and Sales Personnel 1 International NGOs or other Implementing Partners 2 International Donors 2 USAID Project Personnel 1 MSP/IREX Staff 5 25 USAID and IREX provided to the NORC team, the list of potential respondents, based on existing exposure to MSP interventions. The initial list included over 600 participants from state and private media with various levels of involvement in the MSP, which offered a robust number of interview subjects to select from. The MSP assisted with respondent selection based on their level of active versus non-active participation, and to ensure the sample of KIIs were conducted in geographic locations that reflected MSP level of effort. For this reason, KIIs focused on Maputo and Nampula. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 16 3.4 LIMITATIONS Due both to the performance evaluation approach and circumstances encountered during the evaluation, the following limitations apply:  Possible biases in data collection. To identify program beneficiaries for the KIIs and focus groups, the evaluation team relied on assistance from MSP staff. Due to a large number of beneficiaries and availability of information about the beneficiaries, MSP assistance was required. Introductions were also necessary because of the local context.  Low response rates. The response rates for focus group participation were low, but this did not preclude the vibrant conversation and useful data collection. Low response rates may be due to the following: 1) the need for more advanced planning and time to organize the focus groups; 2) the need for more dialogue with USAID and IREX in advance of organizing the focus groups would have helped improve the selection of participants; 3) timing of when to conduct the focus groups taking into consideration journalists’ working schedules and other commitments; and 4) the decision not to use material incentives that might have made it more attractive for invited focus group participants to commit to taking part in the focus group. Inability to precisely measure quantitative progress of program goals. Though the evaluation team has acquired qualitative information from the KIIs and focus group discussions, quantitative analysis of changes in the level of professionalization is limited by documentation in the new MCAT dataset provided to the evaluation team. The implementer was in the process of repopulating the database. A more detailed description of data quality analyses challenges are in Annex D. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 17 4. FINDINGS,CONCLUSIONS AND RECOMMENDATIONS 4.1 EVALUATION QUESTION 1 (Result 1): To what extent have MSP capacity-building activities enabled media professionals to improve their capacity in targeted areas? Which components of the MSP intervention have been perceived as most useful and least useful for promoting professionalization and quality media content? And how responsive has MSP assistance been to the needs of the media sector according to target beneficiaries? FINDINGS MSP assistance has augmented journalists’ skills and ability to produce sound journalism. The evaluation team held, in Maputo and Nampula, in-depth interviews, focus groups and small-group discussions among journalism students, working journalists, editors, media owners, and civil society leaders regarding their impressions of the MSP contributions. Interviewees and focus group participants, both attribute MSP training to improvement of their skills and professional development. Nearly all of the interviewees who took part in some aspect of MSP journalism training programs felt there was significant improvement in skills as a result of the training. Managers and editors that had staff involved in MSP journalism training echoed this sentiment, noting a change in ability. Basic skills training was by far the most popular, followed by skills training in information communication technologies (ICTs) for journalists, such as the suite of Google applications, Dropbox, Frontline Short Message Service (SMS), and Facebook training. Focus group and interviews both reveal that basic skills training deemed useful and that MSP continue to offer such assistance as a core part of the program. The training is perceived as more practical, useful, and helpful for the future work of journalists. Of note, the training and expertise provided through MSP assistance are not available at local university journalism programs. Thus the MSP, at this time, is the only source for such instruction. Regionally, particularly in South Africa, there are strong, credible journalism studies options, but costs are prohibitive for most Mozambicans. The MSP trainings as proximate, and moreover, offers its trainings in Portuguese, which is essential as the main language of instruction. In feedback on MSP trainers, the MSP brings in specialists who have the life and work experience that journalists need in this new era of digital journalism. MSP trainings on more specialized topics on issues like corruption, environment, gender, and health are also valued. In site visits to two of the community stations that MSP supports station managers and volunteers relayed that IREX support for equipment is invaluable. MSP’s provision of studio equipment, computers, and other types of technology are put to constructive use. For instance, at the Community Radio of Monapo, the computers provided by the MSP are used not just by station staff, but by the whole of the community. In this regard, the station has created a valuable resource for those who are able to access the computer center. Furthermore, as noted by interviews with the volunteers and coordinators, anytime they get training through MSP, they return to Monapo and share what they have learned. The need for equipment is similarly felt in Maputo, with needs ranging from recorders, to laptops, printers and camera equipment. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 18 Digital media trainings provided by MSP are also deemed instrumental in day-to￾day functions. Learning new digital media techniques and other technologies and online tools for journalists, additionally, were also cited as key to helping enhance journalist capacity, as these tools help journalists adapt to and keep current with the rapidly changing digital age. The MSP training, combined with the trainees’ ability to use more current digital techniques and technology, is linked to what may appear to be a nascent demand for MSP-trained journalists, a conclusion elaborated on in the section below. The training is made available in various ways in Maputo to TV, radio, and print media outlets. In the Nampula office, the trainings are offered to community radio stations throughout the Northern Provinces. New equipment is highly appreciated and put to good use. Site visits to media outlets in Maputo and community radio stations in the Northern Provinces indicate that MSP supported media greatly appreciate the computers, desks, radio equipment, monitors, records and other equipment MSP provides. Clear identification of MSP and USAID support was noticeable in all stations visited. Mangers and CEOs were able to point out equipment, how it was used, and how it has made a difference in daily operations. Based on the modest amount of time that the evaluation team spent with each partner station, it is evident that the smallest infusions of equipment support (e.g. desks, chairs, upright PCs, digital media recorders) significantly improve the operations of independent media outlets. The market is saturated with old computer equipment, and journalists, students, and community members who have access to digital equipment and technology benefit from the use of modern apparatuses. For instance, at the offices of Magazine, a newspaper produced in Maputo, the offices were sparse, computers were old and slow, and staff used old FAX equipment and worked from wobbly desks, and borrowed chairs. They do as best they can under the circumstances, but conveyed to the NORC evaluation team that any provision of modern, digital equipment will be used and appreciated by their staff. At Debate, the equipment and digital technology used was of a mixed variety – some old and some new. TV TIM says, the MSP equipment is “fundamental” and that it “makes a huge difference.” Even Radio Mozambique made an appeal for more equipment from the MSP, a request that signals demand and desire for more modern equipment from all types of media outlets in the country. The biggest request, by and far, among beneficiaries is for laptops, digital recorders, tablets and mobile technology that could help journalists get online, access tools and technologies, and be part of the digital journalism revolution. The largest barrier for journalists and other media professionals in acquiring the technology is being able to afford the cost. For most, it is simply out of reach. Training may create at least a level of broader demand for MSP-trained journalists. Importantly, media managers commented explicitly on their preference to hire and work with journalists that have completed the MSP journalism training as opposed to journalists who have not. The evaluation team finds that many of the journalists trained by MSP were hired immediately at media outlets. For instance, the manager at STV, one of the leading independent TV stations, remarked that the MSP partnership has brought talented staff to the organization. The last journalists hired came from the MSP MediaLab TV and included three women. Similarly, TV TIM’s satisfaction has translated into hiring MSP trainees and managers there assert the trainees bring new approaches to the newsroom.  “The training is very good, because its multiplatform, covers different issues, and opens up the different possibilities for what journalists can do. The MSP training is up to the moment. The Brazilian journalists who offer the teaching are great. This type of help is MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 19 fundamental to [TV TIM] growing.” During site visits to TV TIM and STV in Maputo, where MSP trainees worked, station managers there confirmed that the MSP has created more skilled young journalists who are conscientious about the quality of their work. Both station managers and directors say that MSP trainees were preferable to other candidates, because trainees experienced the type of on-the-job training that resulted in a young professional who showed up ready to work on day one, and who had the skills and wherewithal to take direction and function in the newsroom or media outlet. MSP trainees also had instruction on how to use equipment, apply basic journalism standards and international best practices, and were exposed to leading conventions and ways of thinking about how to work as a journalist.  "IREX uses people who come from media houses, and they can help the institution to improve its approach, because these people know the problems that media in Mozambique are facing. IREX recommended for us, one correspondent in each capital that can make the sales and the distribution, but also write the local news. Then, people can be more interested in buying the newspaper. As I said, the people that work there [at IREX] they know the real media problems" (Media Coop Group). MSP trainings are considered useful; skills provide a concrete career pathway. Responses to MSP interventions around promoting professionalization and improvement of quality of media content amongst the majority of journalists interviewed indicated that MSP training opportunities have a positive effect on their aspirations as journalists and that through IREX they have gained skills and opportunities to get jobs. Notably, younger journalists, fresh out of university, or in the early stages of their career, all identified the MSP as a beneficial and important gateway to acquiring a career in journalism. The MSP, they conclude, offers practical, professional opportunities that cannot be gained from the more theoretical, abstract training received as university students. Potential employers at media houses interviewed also explain job candidates that experienced the MSP training or internship programs were considered more advanced in their skills, as previously mentioned. Some media houses, like STV, will only consider candidates from MSP programs, based on interviews with their senior staff. When asked if there were notable changes in output of journalists after they worked with MSP, on staff member at STV comments, “Yes, there has been a change. The appointment of staff is based on specific issues, based on their ability to report. IREX advised STV on the best type of support for their staff. The challenge, however, is on staff doing full participation, i.e. taking an entire day or week out to do training. It would be better for MSP to offer on-the-job training. People should come to STV to do the training. This arrangement has just started.” STV indicated a strong sense of satisfaction for IREX trainers and consultants who work with them. They talked about how the Portuguese and Brazilian trainers offer “added value to them, because when they address TV they know what they are talking about.” Local NGOs and other organizations funded by international donors that focus on communications – such as IBIS (funded by Sweden) and DIALOGO (implemented by Development Associates International with Department for British International Development funding) – also have positive opinions of the MSP training program.  “Web journalism training on how to do story for different platforms is greatly needed, because skills are so low in Mozambique.” (DIALOGO) MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 20 In fact, DIALOGO felt that the MSP program was also quite complimentary to their organization’s own efforts and both programs were well-suited to reinforce and support the intended outcome and impact they were designed to deliver. In both Maputo and Nampula, respondents were enthusiastic for more MSP basic training. Focus groups in Maputo with students from UEM who are participating in MSP internship program, assert that the skills and knowledge acquired from the training helps our integration in the labor market. The internship participants believe their school’s (the ECA) partnership with numerous companies provide opportunities for future employment. A shortcoming, however, is that the company’s focus may not match students’ desired vocation.  “The IREX internship concept allows them to adjust the course themes with our vocations” (female, 18-29, focus group, Nampula). Criticisms of MSP trainings are varied. Asked what can be improved about the MSP trainings, respondent views are varied, though dependent on age. More mature journalists with over 10 years of experience, for example, were more skeptical of MSP instruction, saying they do not like “one-off” trainings that lacked a specific focus or follow-up. They also think that trainings should be uniquely fitted to the needs of the specific media house. One long-time media researcher and media law advocate, who is also owner of a veteran of media development in Mozambique, commented that the MSP might consider focusing on the organizational level, restructuring specific media houses, rather than training individual journalists. The emphasis should be on strengthening specific media outlets and assisting them with developing concrete strategies for platforms as appropriate. He says:  “The investment made by IREX doesn’t correspond to the impact. The training contents don’t match with the real need of our journalists and training one journalist isn’t going to solve the problem of the media house. These institutions have structural problems that you cannot solve only with the training of one journalist. …Sometimes when the journalist goes back to the institution, he isn’t well received. He faces problems to impose news ideas. The training has to be focused in the context in which the journalist works.” The evaluation team notes, however, that this a common critique of many media development journalism training programs. MSP appears to be addressing such concerns to the best extent possible in efforts to focus and streamline program support to among five to six key media outlets. Consideration will have to be given to what type of basic skills and support is needed to nurture and grow journalists. Interview feedback from the more senior journalists suggests in their view, MSP could do more to be more responsive to their needs and development. Some cite lack of consultation with media houses. Other critiques of the MSP journalism training from veteran journalists, include frustration over the design of the activities around Result 1. They wished for more consultation, and some were concerned that students receiving per diem for taking the courses sets a bad precedent. Their criticism seems to stem from the general appeal and open call for joining in training programs, internships and courses that MSP used to recruit participants in the first years of the program. The newer approach that MSP plans to pursue should mitigate some of the criticism of how the MSP training is designed. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 21 The MSP Program is responsive to needs of the media sector. Based on the qualitative research and site visits, the evaluation team concludes that the MSP is responsive to the needs of target beneficiaries. The program experienced challenges, especially in the startup period. The MSP also had to establish trust and confidence among local leaders and stakeholders in the media and communication space, as well as construct relationships with government and other public officials. That USAID was the source of the funding raised some concern in a few circles, but interviewees by and large were not overly worried or critical of the U.S. presence ultimately. For a few, more transparency would be welcome. A number of interviewees remarked that the MSP could be more transparent and open in terms of how it has developed its program design, overall project concept, and how it chooses its beneficiaries – this sort of criticism came mainly from senior journalists or media owners. The evaluation team was able to ascertain that the MSP has done its due diligence in terms of meeting with a wide range of local journalists, media houses, and by and large strives to keep an open door and make its programs and resources as available as possible. The MSP has a Facebook presence, Twitter feed, and a YouTube channel. Moreover, through its Debate program, anyone who is interested is available to come and join in the programs. Quantitative M&E tools are limited in their ability to measure progress of and change of programmatic goals, but results indicate improvement of journalists’ articles could be statistically attributed to MSP training. The MCAT is the MSP’s primary data source for measuring and observing progress in professionalization of media. It is a quantitative instrument that relies on content analysis to generate evidence-based information on shifts in journalist behavior that may be attributed to exposure to MSP capacity-building activities. An analysis of the new MCAT database provides an indication of how impactful MSP training is on journalist professionalism in terms of article writing. According to the results, ANCOVA regressions exploring the relationship between training and actual improvement in quality of articles matches qualitative assessments. Specific findings are highlighted below.9  On average, the participation in training across journalists in the small sample provided has a statistically significant impact at the 5-percent level, raising scores by 9.7 percent.  The evaluation team, however, finds a coding / scoring bias of approximately 2 percentage points among the two scorers IREX employs for this task.  On average and across all journalists, longer articles tend to have higher scores.  An article written for a Maputo news outlet was likely to be an average quality of 2.8 percentage points better than the average for articles at all other outlets in other geographic areas.10 These are not an exhaustive list of possible comparisons that can be analyzed. Moreover, the specific estimation results presented are also only illustrative of those found by the NORC                                                                  9 There were few women trained according to the dataset provided. As a consequence, an analysis of gender could not be conducted. 10 Note that this does not mean that Maputo articles were the best or better than, say, those written for international outlets. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 22 evaluation team. Other specifications would yield slightly different effect strengths, though not changing the fundamental conclusions. As the representativeness of the small sample size is unknown, however, it is not certain that the results are generalizable. Nevertheless, the evaluation team has reviewed the newer MCAT database, and offers some initial thoughts on the instruments and approaches, in addition to comments outlined in the annex on the contents of the database itself. Weigh carefully, the merits of shifting from Likert-type to dichotomous measures. A robust content analysis tool is useful for understanding how media development interventions offered through training, mentoring and other programs, improve capabilities. The scales used in both MCAT databases come with advantages and disadvantages, which should be carefully considered against MSP program objectives. A list of advantages and disadvantages of using the scales is outlined briefly below (please see Annex D for further observations on the MCAT database):11  Dichotomous (yes/no) options, used for the new MCAT tool, are easy to score.  Dichotomous options are also good for screening respondents.  They are also useful for assessing attributes (male, female, employment, training attendance)  Beyond “yes” and “no,” dichotomous measures, reveal little about “how little” or “how much” a dynamic changes.  The Likert-style scale, used for the previous MCAT tool, allows for greater sensitivity in measurement. In other words, Likert scales provide indications of by what degree outcomes may vary.  Likert scales also lend themselves toward many different kinds of methods and analyses of, for example, the degree of progress among individual journalists, similar comparison of media types, and progress of media development capacity between men and women, as well as geographic considerations.  Because the distance is not always perceived as equal between points on a scale, the difference between the points may be perceived differently.  Dichotomous options may leave less room for error, particularly if the measures themselves are simple and not very subjective. Additional coder support and training might mitigate bias and errors. Given the advantages and disadvantages of each type of coding barometer, initial conclusions are that a multi-point scale may be more suitable for those program objectives measuring change in journalists’ behavior (i.e. levels of professionalization). Content analysis can indeed be a very subjective process, as coders may score materials differently, resulting in unreliable and thus, invalid scoring. IREX may consider having at least two coders per article and averaging the results. However, more intense and periodic training for coders combined with greater                                                                  11 Importantly, the literature on scales and their associated biases / errors is vast. It is a complex matter. The evaluation team’s comments are not meant to suggest that one should simply be chosen over the other. In fact, other scales may also be used dependent on the suitability of the material to the scale. For example, one could use a categorical measure, as well as a dichotomous, and a Likert-type of scale. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 23 supervision might mitigate some of those errors. Adjudication procedures are useful as well. In instances where there are questions about coding decisions or differences in scoring, an adjudication process can be established where an individual makes the final decision regarding what score should be assigned. Upon observation, the new MCAT does incorporate some objective and factual benchmarks into its dichotomous measures (e.g. does the article have at least 3 sources?). The other criteria, however, still require a fair amount of coder judgment and deliberation. Scales, furthermore, are among the most commonly used formats in survey research, and the previously used scoring criteria from original MCAT is closer to research designs used by leading methodologists that employ content analysis to study messaging, narratives, and media. In this regard the evaluation team suggests:  Continuing efforts to make scoring criteria more objective;  Use Likert coding, but during analysis, generate the associated dichotomous variables using different cut-offs12 and then check whether the results change significantly;  Consider multiple coders and then average the scores. Alternatively, one could analyze which criteria within a score are most subject to coder variation and have those criteria scored multiple times);  Always test for the existence of coder bias and adjust scores accordingly when they are to be compared; and  Always have the same person code the same journalists over time. The MSP should continue with its mixed-method approach. While the MCAT has utility as a data source for the M&E purposes associated with Result 1 of the MSP, it is not the only tool that can provide information and insight about the impact of capacity building work for journalists. In addition to the MCAT scoring, the MSP project has at its disposal, assessments and reports from staff and consultants that reveals data related to the progress and changes perceived among journalists and other media professions trained by MSP. In spite of the statistically significant, though small bias found among coders, the site visits to the IREX office and meetings with the MSP M&E officer and coding team, suggest that there appears sufficient capacity and expertise to conduct the MCAT content analysis in a manner that is consistent with best practices used by social scientists who employ content analysis to understand the impact of journalism training.13 CONCLUSIONS (1) The basic, foundational skills courses offered by the MSP are well-received, and the project should continue to provide these in-house to the targeted media outlets that they been prioritized. MSP should continue with its approach of offering balanced foundational skills training with more advanced subject-specific training, as well as its focus on strategically selecting partner outlets and a core group of promising journalists. (2) MSPs technical training has also been impactful, boosting the operational day-to-day skills of journalists, while preparing their longer-term capacity to function in the rapidly changing era                                                                  12 For example, on a scale of 1-to-5, using 3 as a cut-off, 1-to-3 might be coded “0”, and 4-to-5 might be coded “1.” 13 See, “Christoph Spurk and Jan Lublinski, “Content Analysis: Measuring the Success of Journalism Capacity Building,” DW Akademie (10/2014). MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 24 of digital technology. (3) The practical job experience gained through the internship program and the MediaLab has been a vital source of support to the emerging generation of Mozambican journalists. The opportunities that the MSP offers are not available anywhere else and no other institution is poised in the near-term to be in a position to provide what MSP offers. (4) Digital media training and digital literacy opportunities are popular in both Maputo and Nampula. Virtually all media houses and all levels of staff requested more and deeper opportunities to receive training on information and communication technologies, and tools for journalists and the online newsroom. (5) There was some frustration expressed that the MSP was not as transparent with the program content and goals as preferred. This sentiment was conveyed by some journalists, who tended to be older veterans of the field, but was not exclusively confined to this segment. The evaluation team notes that the MSP has endeavored to be consultative and informative. Still, MSP should continue to perform outreach to key stakeholders and media managers so that they perceive they are kept informed and involved the program’s design and vision. (6) The structure of the new MCAT database can be improved with clearer documentation and organization of its information. This would contribute to the MCAT’s utility as an instrument that yields empirically-based evidence that measures progress of program objectives (see Annex D). RECOMMENDATIONS (1) The internship program and MediaLab program seem very promising as activities that can endure post-MSP. There are ways that the MSP could work with local civil society and area universities to develop this into an activity that could be maintained through local organizations once the MSP ends. Training and support for how to run an effective internship and job placement program would be useful as a supporting activity in this regard. (2) The overwhelming feedback from all interviewed at TV, radio, print, and online media is that programs like the MSP are needed to help create a professional media sector in Mozambique. The skill-level and overall individual capacity of those interested in journalism and the media profession remains very weak compared to where station managers and editors would like to see the state of media in Mozambique. Their view is that a solid professional media cannot be created overnight – or in a few years. Managers urged a long-term vision, and a three- to five￾year plan for where USAID would recommend taking a project like the MSP. (3) The current MCAT tool might better assess increased professionalization of the media professionals with the application of a more sensitive measurement instrument, and more reliable coding which could be facilitated through training. The evaluation team, through its statistical analysis of Objective 1, furthermore identifies some potential weaknesses with the structure of the new MCAT database. All of these can be addressed in the shorter term to enable easier and improved evidence-based analysis. The Annex provides a more expansive discussion on how an MCAT-like tool might of better use for future programming post-MSP. (4) Some journalists in the new MCAT database have a large number of articles scored, while others have none, or have only a baseline or end-line score. In order to create a database that MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 25 can be used for substantive evaluation purposes, the evaluation team recommends a review of the scoring strategy in order to address this imbalance. This can be done in the short-term (5) The inclusion of demographic data (gender, education, age, geographic origin/residence, etc.) will permit interesting analyses of heterogeneity and also improve comparison of cases. This data should be captured in the MCAT database and can also be done in the short-term. (6) The M&E coordinator could work with a media specialist and program leaders on make more dynamic use of the monitoring data. For additional feedback, focus groups could also be conducted periodically with interns and other MSP beneficiary populations. The MSP could also use other digital tools like SurveyMonkey or other forms of e-surveys to send out periodic (perhaps one a year) surveys to beneficiary journalists in order to have a more comprehensive and structured approach to assessing the impact of MSP activities in the area of professional capacity building. If time and resources are not immediately available to the MSP, these adjustments might be addressed as part of a longer term M&E strategy after the end of the MSP project. (7) Otherwise, the evaluation team suggests that the MSP continue its mixed-methods approach – employing content analysis, focus groups, and the project reports to document – in order to capture both performance monitoring data and data needed to assess program impact over the long-term. Focus groups, for example, might be conducted on a periodic basis as a means of assessing perceived quality of media content. The focus groups could be comprised of “the public” as the reaction and perceptions of the reading, listening, watching public/ the audience is really crucial to measuring the long-term impact of the MSP. The qualitative research will also give implementers a better sense of how trainings are used, why it is or is not useful, and how trainings might be improved to meet the needs of target beneficiaries. Focus groups will be especially useful for providing supplementary information that will complement the new MCAT tool. (8) MSP should continue its outreach efforts to promote the program and ensure that local journalists and other interested parties are aware of opportunities for how they can participate and get involved in the activities and opportunities the program affords. IREX should to explore where possible additional avenues exist to solicit quality feedback on the program. 4.2 EVALUATION QUESTION 2 (Result 2): Has organizations’ business management capacity improved as a result of working with the MSP, and if so, in what areas? Where do weaknesses still lie? Do organizations perceive that their financial viability has improved as a result of working with MSP? What specific changes have been implemented as a result of collaboration with MSP to improve business management capabilities and financial viability? What critical issues must outlets still address to strengthen financial viability? How could MSP more effectively contribute to greater financial viability of outlets? Given the relative newness of independent, private-sector and commercial media in Mozambique, it is not surprising that issues surrounding financial capacity were of great interest MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 26 to nearly all respondents interviewed during the mid-term evaluation. The need for capacity building around business skills and general know-how was expressed at all levels of those interviewed – from journalists, to editors, to managers and CEOs. Both private-sector and state media expressed the need for targeted assistance to help with their financial capacity, and acknowledged that the MSP is important for providing assistance in this area. FINDINGS MSP assistance has contributed to improvement of financial capacity of media organizations. Nearly all media houses provided examples of the way in which MSP support has expanded their organizational capacity. Examples include: work with Canal on the redesign of the newspaper layout to improve readability, information design and available space for advertising; intensive training for media professionals on improving media marketing, sales and distribution; and targeted in-house consultancies to select media outlets in a variety of areas designed to support financial capacity and overall long-term business viability. These views are borne out in quarterly program reporting. 14 Interview responses also underscore this sentiment:  One employee at the Social Communication Institute in Nampula and who is the community radio coordinator said, “The MSP helps make community media sustainable by offering trainings, marketing, and management courses. It takes time to produce results, but things are changing. It’s difficult to quantify. It’s difficult to say radio stations are more sustainable, but things are changing. With the improvement of programs because of training with MSP, our audience is increasing, the community has a greater interest in putting on announcements, and our money has increased as a result, as compared to a few years ago.”  According to a staff member at TV TIM, “We need to grow our audience. If TIM doesn’t grow, we won’t succeed. To grow an audience we need transmitters and coverage. Moreover, the media environment gets more complicated with new technology, especially with the transition from analogue to digital, which has implications on the financial capacity and viability for a station. Digitalization is a process, but it’s in limbo. It’s difficult to make media sustainable, having the MSP program opens doors.” Furthermore, as reported in the MSP’s work with Mocuba Community Radio, “Now we are thriving,” says Rosalina Caetano, coordinator of the Mocuba Community Radio and the television station, RTVC Mocuba. “Our operations are more dynamic, and we are more sustainable than we were before.” With training in marketing and business development by MSP over the last year, the station has increased revenue from advertising, which is generated primarily through partnerships with local merchants and organizations. Funds enabled the station has been able to buy three motorcycles, which are now being used by community radio volunteers and reporters to extend their reach within the district. As a representative from TIM TV explains, the media business is new in Mozambique, only roughly 20 years-old. In a context where there is a dearth of local expertise in media business management, there is a great deal of knowledge to acquire related to operating and, perhaps more even critically, maintaining a media business. MSP training has also been essential for this                                                                  14 Ibid. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 27 gap. The TIM TV employee notes:  Miguel Billa, Savana’s Marketing and Sales Coordinator says, "Through IREX, I've learned about capacity building, marketing and sales distribution, and administration and coordination. My colleagues go to IREX training, too. IREX training is useful for two reasons: one, what we are learning – it’s an added value. We don’t learn such things at school. Two, media in general face financial limitations." Normally, we wouldn’t use such training. …IREX helps us to see the future, so that’s good.  From Joao R, Director General TIM, "There is no specific funding system [for media]. Media is not a traditional business in Mozambique. There's not a commercial bank that you can just apply to for funds. It's difficult to apply for funds. It is difficult to make media sustainable. Having the IREX program opens doors. They are open and transparent." MSP engagement has resulted in tangible changes. The evaluation team observes that a number of specific changes have been implemented to improve business management capabilities and financial viability as a result of working with the MSP. First and foremost, at a very basic level, the MSP partner media outlets have begun to put more robust and business￾oriented systems in place to support operating the management side of their media outlets. Interviews among MSP partners suggest that partner needs and preferences are, for the most part, aligned with the new IREX/MSP strategy of offering more targeted, tailored support that is customized to the needs and skill levels of each media outlet. The core limitation is that many of the media outlets interviewed, especially the print media, have very small staffs and therefore, everyone has to wear many hats. In other words, any new tailored MSP support would have to consider whether there is staff available to do the tasks and whether they are capable of carrying out the work. Media outlets must address several critical issues in order to strengthen financial viability. Learning to think of media as a business and to develop strategies and practices that are consistent with modern media business skills, systems, and strategies remains a vital and often missing skill from the media, according to interviewees. Many of the CEOs and founders of the media outlets identify as journalists first, and therefore, operating a business and navigating the complexity of a media environment that operates with old-fashioned, legacy media practices, while at the same time embraces new media, and the fruits of digital and social media platforms, is indeed complicated. The MSP faces the challenging reality of working with outlets that may provide some information regarding the financial health of their organizations, but not always a complete picture of the investments and its overall financial realities. The MSP can, and does, pursue a strategy that seeks to impart key business skills to the outlets. This is accomplished through general workshops and seminars that are open to partner institutions to attend as well as through customized, in-house tailored consultancies. Fundamental issues that media outlets must still address in order to strengthen financial viability, include: a) Credible market research: gaining access to reliable and trustworthy market research (In addition to other organizations, this request was also made even by the CEO of Radio Mozambique, which says this information is a much needed part of Mozambique’s media landscape and should be an organizational development priority). MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 28 b) Audience research: similarly, gaining access to audience research would help stations to better leverage their market position with advertisers, as well as to better understand their audiences in terms of needs, preferences and thus improve their overall long-term growth. c) Know-how for planning and sales: this is especially key for print media, who express that they often do not have the appropriate number of print runs. d) In-depth knowledge of the advertising market: a deepened sense of the overall advertising market: There is a feeling amongst independent media that they are at an unfair advantage to state media when it comes to advertising and sales. They believe they have to make do with the “leftovers” in terms of what advertising revenue is available in the country. Having a more concrete evidence-based study or analysis of what the market can bear and how versatile and vibrant the overall advertising market is in Mozambique would be useful. Such research might consider questions of whether it is indeed the case that the state media has maximized the best forms of advertising, and also explore the potential of online and other emerging funding streams for media businesses in Mozambique. A study or mapping of existing research, intelligence and briefings on advertising and marketing in Mozambique and the region may also be useful, especially if made available in Portuguese and shared with MSP partners. e) Web-based marketing and advertising: interviewees report they are not that familiar with the possibilities in this area. They felt that given the low internet penetration in Mozambique, it was not a priority, and thus did not know a lot about how such advertising. Yet, it seems a potential avenue for growth, and one that warrants exploration. f) Printing presses: The ability for independent print media to stay competitive absent a local printing press is difficult. Media organizations would benefit from assistance with acquiring their own printing presses. CONCLUSIONS (1) Data-driven examples demonstrative of how the MSP has impacted financial viability of media organizations, meanwhile, were limited at the time of this evaluation. The MSP also relied on the MCAT as its primary data source for assessing results under Result 2 of its monitoring and evaluation plan (M&E).15 The implementing staff is taking steps to use other data sources to evaluate the assistance delivered through Result 2, however. A matrix developed for use in Nampula with the MSP’s 32 partner community radio stations is being modified and adapted to the same media houses that MSP works with in Maputo. Towards this end, there are a number of steps the MSP can take to modify its current approach measuring changes in progress of goals related to financial capacity. Possibilities include: a) using more of an organizational capacity building approach and related M&E tools to support this work; b) maintaining better records of financial viability benchmarking with MSP partner media outlets; and c) developing customized matrixes for each media house. Based on evaluation team interviews with implementing personnel, the MSP is pursuing each of these strategies. With the proper monitoring systems in place, the MSP M&E officer can work with Result 2 staff and management to capture lessons learned,                                                                  15 IREX, the implementer, determined that the MCAT was not an appropriate instrument with which to measure changes in financial capacity and viability over time. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 29 gauge benchmarks and milestones, and start developing a sense of how to best support financial capacity development of Mozambican media. (2) The in-house consultancies and tailored approach to the MSP’s Result 2 activities is in-line with the feedback that the mid-term consultant team was given. MSP should pursue this strategy and ensure that detailed reports on organizational capacity and financial viability milestone indices are created for each of the core partners (3) The MSP cannot provide targeted customized assistance to all media houses. If the MSP can continue to offer some general trainings and workshops on business basics, sales and marketing for media businesses, and training on digital tools and web-based technologies for media businesses, these types of courses would be very appreciated by the media community as a whole. RECOMMENDATIONS (1) The MSP is in a position to lead a discussion on market and audience research – two tools that are considered essential to meeting the needs of independent media houses in terms of their pursuit of financially viable operations. This information is expensive and hard to come by. The research that exists is largely done in house by State media, and this is not trusted or accepted as a suitable form of information for independent media. Some initial steps should be made under the remaining time of the MSP to assess and map out what research and data sources are available and what type of research and information would best serve the needs of the independent media houses with which MSP works. (2) The MSP could consider a Training-of-Trainers (ToT) approach to some instruction and support offered through the MSP’s Result 2 activities, especially in terms of business skills and financial capacity building. This could be done in partnership with local organizations, providing there is one that is acceptable to work with and capable of carrying out this type of work. Some consideration should be made for long-term plans on who locally can carry on with the mentoring and expert level consulting related to running media as a business and helping journalists who have excellent ideas take their plans and visions to a level that they can receive funding and run their media outlets as viable businesses. (3) Feedback from interviews reveal that a printing press would be very useful for the long￾term survival and growth of local independent media. An assessment of needs, pricing, and who would have access to using a press would be a good next step – and a discrete deliverable that would be possible to develop and share during the next two years of the MSP. Questions related to whether there is demand for a local printing press and whether the private sector could provide for this, indeed, to merit further consideration, and the MSP is well-positioned to look into this suggestion that came up from several of the print journalists who were interviewed. Canal and Savana are both printed in South Africa. It is believed by the journalists interviewed that access to a local printing press is an important foundation for their development, freedom of expression, and developing local media businesses. It is recommended that MSP conduct a needs assessment of how independent media in Mozambique would benefit from a printing press as well as a feasibility assessment for financing. (4) The MSP can provide assistance with establishing financial viability. As mentioned in the previous section, in addition to the acquisition of a printing press for print media, MSP could focus on support in the following areas: web-based marketing and advertising; reliable MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 30 market research as well as audience research; applying planning and sales statistics; and providing knowledge and understanding of overall advertising markets. The program strategy pursued within the purview of its most recent Work Plan (Version 3), combined with its revised more targeted approach of working with fewer outlets in a more customized way, will go a long way in effectively contributing to the financial viability of MSP partner media organizations. Bringing in expertise and perspectives of key media business leaders and investors in the region would be one approach to consider. (5) A workshop or conference – closed- or open-door – geared towards the business practices of media in transitioning African countries could also have potential. Documenting lessons learned and bringing in perspectives from other implementer programs in this arena could also be quite useful, especially as IREX has a long memory of media transitions from the 1990s, onward, and a roster of business and financial management experts who are skilled at working with small- to medium-sized print, radio and TV operations. (6) In addition, exploring ways to better work with the state may also be helpful. The reality for southern and east African media, in general, is probably one where state media will continue to receive their budget allocations from the state and benefit from advertising as well. The cost of running media houses is too high and demanding to depend on license fees alone. That said, state or public media benefit from a rich and multi-player environment in the long run. It is better for Mozambique’s state media institutions to have counterparts that are commercial, private sector media operations. A convening of key stakeholders together with the right mix of financial and business sector actors to think through the next phase of Mozambique’s media and business development agenda may be worth pursuing. Leveraging some of the expertise and lessons learned from South Africa would also be appealing and fit into the current program design. There may be ways to work with others in the region on this issue as it is likely to be a condition that will remain over the long term, though may be of interest. Here, South Africa has many lessons learned with regards to the transformation of its state broadcaster into what is now the South Africa Broadcast Corporation (SABC). The financing of the SABC is particularly interesting. The civil society activism and advocacy related to democratic, free, and independent media in South Africa is also very robust and serves as a model for the region – if not globally. There may be ways to work with others in the region on this issue as domestic conditions in Mozambique will remain over the long term, though may be of interest. 4.3 EVALUATION QUESTION 3 (Result 5): How can the MSP strengthen activities to improve advocacy for media freedom and an improved legal environment? In other words, is the MSP exploiting all appropriate approaches possible to achieve Result 5? Are there international best practices that have been effective elsewhere that may be applicable for the Mozambique context (of particularly interest are approaches which involved building advocacy capacity of local organization)? Of the three result areas address by this mid-term evaluation, Result 5 was the most challenging to assess. It is the area that seems to most lack focus and a strategy in terms of what the MSP would like to do and accomplish. To some extent this is understandable, especially since the space for freedom of expression and access to information advocacy and activism is in an uncertain space. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 31 Nonetheless, there have been some important victories related to freedom of expression. Last year, a Right to Information law was finally passed after being under consideration for many years. 16 While an important step, some elements of the act have been criticized for not adequately and fully protecting right.17 The MSP has held roundtables, events, and assembled a press pack on the new FOIA law and the development of the regulations that will implement the law. While there has been some written analysis of the draft law, there is not a great deal of written analysis on the new law and any short-comings it may have in terms of how it might stack up to international standards and best practices. In meetings with journalists, media managers and researchers in Maputo and Nampula stakeholders express interest for more public discussion and debate on what the law is about, how it will be implemented, and what journalists need to know in order to make use of their rights of access to information. Moreover, while the law passed, it still needs to be regulated. In general, editors and media managers interviewed, as well as some journalists, are interested in more robust training on media law and policy issues. Editors and media owners, in particular, believe their journalists lack knowledge about journalists’ rights to freedom of expression and access to information. For instance, the owner of the community radio station on the Island of Mozambique said;  "Access to official information has been a problem. Local authorities put a lot of pressure on [a person.] For instance, with regards to HIV numbers for the islands – simple information – he cannot get it.” Journalists could also use more information surrounding legal resources they can turn to and rights to counsel. The MSP has addressed this by creating a RCC to help with navigating the legal enabling environment in Mozambique. The MSP and others comment that this resource has not lived up to its original design, and the MSP is now taking steps to assess whether the RRC has a future.18 As the main organization responsible for freedom of expression and access to information advocacy, the Mozambique chapter of the Media Institute of Southern Africa (MISA) was mentioned in nearly every interview. The local MISA chapter, which has gone through a number of personnel changes, financial mismanagement and a crisis of leadership, is currently undergoing a process of transformation. MISA, while recognized as a leading player, was criticized for its lack of consistent leadership and current stature in the Mozambique advocacy and policy-making space. Since MISA remains the organization most identified with advocacy for media freedom despite organizational problems, it may be worth exploring how the MSP                                                                  16 The Bill has been on the policy agenda since 2005 when the Mozambican Chapter of MISA (MISA Mozambique) facilitated a media seminar that crafted the framework for a draft Bill. It seeks to create greater transparency and generate public participation in Mozambique, however it has also been the subject of some criticism with respect to whether it adequately enforces the right in all its dimensions. See Article 19, a global leader on media law and policies issues for their analysis and overview of the new FOIA law in Mozambique and their recommendations for next steps. Also see FreedomInfo.org, a global network of Access to Information lawyers, activists and scholars -- http://www.freedominfo.org/2014/11/mozambique-assembly-oks-freedom-information-law/. 17 See MISA overview of the law and its efforts to get it passed at http://misa.org/component/k2/item/2938- mozambique%E2%80%99s-parliamentary-assembly-passes-final-ati-bill?Itemid=101 18 The evaluation team tried to meet with staff affiliated with the RRC several times while in Nampula and Maputo, but representatives from the RRC were not available for an interview. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 32 might support MISA locally and regionally to strengthen MISA’s operations in Mozambique. There is no existing strong organization to fill the vacuum. Selekani and JOINT are active, but unable to do this alone. MISA and its broader network have access to numerous tools, resources, pro bono counsel and other forms of support that would benefit the MSP’s efforts surrounding Result 5. As noted by rom Luis Nachote:  "MISA is a good association. On the law of the Right to Information, MISA was leading more than other syndicates. The syndicate needs to be more together and independent – unified. The Rapid Response Committee (RCC), they’re helpful. They help expose abuses. They are a very new organization." Like the new decision to take a more strategic approach to fewer outlets and journalists put forward for Result 1 and 2, the strategy going forward for Result 5, is to consider how the MSP can streamline its support to civil society organizations working to advance freedom of expression and access to information in Mozambique. FINDINGS The purpose of Result 5 of the MSP is to support the increased ability of Mozambican organizations to advocate for press freedom, and to promote an improved legal enabling environment for media. MSP has sought to partner with civic organizations to strengthen their advocacy on behalf of media. This is achieved through activities that support advocacy for media freedom and an improved enabling environment, as well as by helping media freedom and access to information to gain broad citizen support. Interest in learning more about the information law is high. There is great deal of interest, particularly in Nampula, among print journalists and community radio stations that the evaluation team interviewed with regarding the new information law. Most say MSP-facilitated trainings, open discussion forums, and debates would be useful to encourage more information about the law and how to apply it. MSP-supported debates are popular. The debates offered by the MSP, which air on TV and YouTube, have broad appeal amongst interviewees. This part of the advocacy strategy ad program is a winning one, and is worth continuing and investing more time and attention. The debates may also provide a means of facilitate partnerships with other CSOs and actors working to advance media development or information communications technology for development in Mozambique (ICT4D), i.e. IBIS, DIALOGO, or Centro de Apoio Informacao e Comunicacao (CAICC). Media have engaged in MSP-supported media law advocacy with success. Mozambique’s Freedom of Information Law was passed after nearly a decade in the Parliament, due in part because of changes of government. The activism and journalists and civil society actors also contributed to movement on the law, in the view of respondents. Several mentioned MISA-Mozambique. The law, however, still faces challenges of implementation, and still needs to be regulated. A number of stakeholders expressed interest in more public debate and programming that would focus on raising awareness about the law and its implications. Respondents consistently expressed a desire for more information about the new freedom of information law and how it will work in practice. They say they would like to debate whether it is a good law, how it compares to other laws around the world, and what it means for their work as journalists. Nampula-based journalists and media advocates also say that they would MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 33 like to actually see the law and understand how to use it in their daily work. Focus groups and interviews findings reinforce the interest found among journalists in wanting to learn more about how the law and having more training on legal issues relevant for the practice of journalism, such as how to navigate the defamation law. Digital media and online journalism pose new threats and challenges. During the course of fieldwork, it was revealed that there were several government or political attempts to disrupt or takedown online content, censor o block social media messages, and to close down websites of online independent media. The picture emerging is internet freedom and digital rights are at risk among journalists, bloggers and media outlets in Mozambique. The emerging free speech environment for online media is an area for growth, and as a consequence, programs like MSP should work with editors and journalists to ensure their safety and digital rights and freedoms well into the future. CONCLUSIONS (1) Encouraging journalists and civil society to use their access to information rights will be a challenge, but could become an area of focus for advocated of press freedom. Given that the regulations are currently in draft, this provides an advantageous juncture to shape implementation of the new law. (2) There appears to be some stock-taking at the MSP Maputo headquarters about the future of the MSP’s work related to advocacy and the program’s support for legal enabling environment issues. This new direction is promising on several fronts because the MSP office in Maputo is mapping out different civil society groups, associations, and other interested organizations who could help to champion the interests of freedom of expression and access to information in Mozambique. (3) There seems to be existing organizations like Media Institute of Southern African (MISA) and others based at the university or within civil society groups that could be leveraged and further supported. (4) MISA, while having some short-comings, is recognized throughout Mozambique as a leader in advocacy of the media sector. A renewed look at the organization and how it can fit into the MSP program is warranted. Yet, whatever the role determined, greater Mozambican leadership will be required and beneficial for sustainability. (5) Even with a relatively low internet penetration rate, Mozambique’s media scene is moving online. Mobile access to the internet is on the rise and Facebook is growing in popularity throughout the country. MSP can help the local media scene navigate the challenges posed by online journalism in terms of digital rights, liberties and freedoms. RECOMMENDATIONS (1) The uncertainty and excitement among journalists and media advocates related to the future of the law creates and opportunity for the MSP to implement more programming around Result 5, which might entail cultivate interest from key stakeholders who work in the area of media law and policy advocacy and activism. One possibility could be hosting debates on laws. (2) MSP can strengthen advocacy efforts through a more coherent strategy. The MSP project can strengthen its efforts around advocacy for media freedom and an improved MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 34 legal environment by putting forward a clearer, more focused strategy. At the moment, a wide assortment of ideas and strategies are under consideration, but it is not clear that there is an end game or goal in mind. Nor is it clear how all the pieces fit together. (3) In addition to the MSP’s advocacy-related work with legacy media like print, radio, and TV, the activities and programs of Result 5 should increasingly consider digital rights and open internet issues, such as digital security safety and capacity. In this regard, there are a number of toolkits, publications and potential forms of partnership that the program can tap into that are available in Portuguese, especially since digital rights activists in Brazil have been among world leaders for efforts to keep the internet free and to support journalism safety and improved multi-stakeholder understanding of online free expression and access to information issues. (4) Sufficient interest in Nampula was expressed around media law, policy issues and ensuring that area journalists, editors and media outlets have a voice in country-wide advocacy. It is recommended that MSP provide support for its partners in the Northern Provinces so they can work towards enabling nvironment for free and independent media issues. 4.4. PRELIMINARY FINDINGS ON GENDER Presenting both a challenge and an opportunity for the MSP, is the gender lens of media institutions as well as efforts to disseminate information and media content on gender issues in Mozambique. As gender is a core focus-area of MSP activities, the evaluation team, at USAID’s behest, integrated questions on gender into the study. Since a mid-term evaluation of MSP gender activities is not required as part of the original scope of work, the team informally assessed program of some components of gender activity. The section below provides preliminary information that may aid MSP design and planning in the remaining years of programming. Cultural norms and conventions are obstacles for Mozambican women. Female representation in media outlets is quite low, as is female business ownership and occupation of key upper-level positions in media institutions. On the one hand, the MSP has accomplished great strides with ensuring MSP training programs and interventions are inclusive. The program has also adopted a robust gender strategy. On the other, the MSP faces some steep challenges in their efforts to encourage media houses to improve hiring practices and increasing employment opportunities for women inside media houses, in addition to boosting media content related to gender issues. Gender integration is considered a vital, but arduous task. The evaluation team used interviews and focus group discussions as opportunities to conduct an initial exploration of beneficiary views regarding media content that addresses key gender concerns – such as, trafficking in persons, health issues like HIV/ AIDS, child marriage – and of the MSP strategy for improving integration of women working in key positions of news outlets. Most interviewees agreed that gender equality and empowerment are key challenges in Mozambique. There was the widespread perception that women would face difficulty securing a foothold in the newsroom, due to the commonly held perception that public space is men’s space, while female space is confined to the private sphere, where women concentrate on maintaining family and the household. In the media industry, according to interview results, this translates into the view that “journalism is man’s work,” and a woman’s place is in the home. Beneficiaries would like to see a gender policy for individual organizations. Beneficiary respondents gave several suggestions that may facilitate promotion of gender MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 35 mainstreaming and empowerment of women in top-level media positions. Several say that more can be done to advocate for a gender policy within individual media institutions. The MSP, for instance, could offer guidance on what such a policy might look like, or facilitate discussion with key media owners and management personnel on how to better address the issue of making newsrooms and other media-related jobs more inviting to female applicants. Retention of female talent is another possible point of future discussion. As one woman respondent said:  “The environment for a newsroom is made for men, but if a woman wants to be a journalist, she can be, but it would help if there was a gender policy in place at news outlets” (female journalist, Maputo). The results find it notable – and perhaps a sign of potential progress – that, two of the media owners interviewed were women. Further, at some of the Maputo-based media houses, such as STV, women occupy important positions of authority. Yet, figures related to gender in the media organizations this evaluation engaged show that men outnumber women by a wide margin. Many interviewed were unaware of the precise number of women employed and only gave vague percentages or ranges when prompted. Moreover, most of the organizations were unable to cite any women in key positions of leadership or management. During site visits to Nampula and the Northern Provinces, the evaluation team observed a training on gender and media for community radio stations. Radio station volunteers and managers appeared to be committed to covering gender issues in their media content and providing an environment where women would not only feel welcome, but would also be offered a voice and a seat at the table. This receptive attitude, however is met with more tempered reflections about putting support into action, with the respondents less certain of whether they could actually cover gender issues like child marriage, HIV/AIDS, and domestic violence. Broadening media content to consider gender issues requires someone being available with the expertise to cover such stories. There also has to be a willingness on the behalf of stations to air female-centric stories. A respondent previously involved in a UNESCO media development program believes that gender equality and mainstreaming in media organizations needs much more attention:  “Even with the UNESCO project, we had a gender component. [But this has not moved forward for years.] The only media house successful in gender is Radio Mozambique. Print media, in particular, are not good, not even Savanah.” Women leaders can be mentors and models of emulation in Mozambique. Of note, three of the women interviewed offered to be mentors and models of leadership for Mozambican women seeking to enter the media profession. Each faced significant gender barriers throughout their careers, and have blazed their own path so-to-speak having accomplished major career milestones. One is the female founder of the digital media platform, Catembe, another is the newsroom manager of STV, and the third is from a newspaper called Debate, who is also a founder and general manager of the publication. These kind of individuals could make important contributions as part of an MSP mentoring project and through participation in workshops, conferences, or trainings. Importantly, the example of indigenous leadership would likely resonate more positively, than models of leadership from abroad. Gender policy notwithstanding, also important is the creation of a newsroom and organizational culture that allows all talent to flourish. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 36 5. SUGGESTIONS FOR FUTURE USAID PROGRAMMING The evaluation team offers the following suggestions to both improve MSP implementation and guide future programming: The MSP has established itself in Mozambique, and has gathered momentum and a vision for a robust media development initiative that should be continued. Most local partners and beneficiaries in Maputo and Nampula remarked that the MSP is the only program of its kind in that it combines journalism training with beneficial financial and management training specially designed to support the business side of media operations. The MSP has managed in a relatively short period of time to establish a program that is designed to serve the multifaceted and multi-stakeholder nature of modern media and journalism. The MSP has begun the process of forging plans for working with a targeted number of independent media outlets that are capable of offering professional, quality content, and that are in a position of being sustainable, financially viable media businesses. Their work with the media scene in Mozambique stands to make a lasting impact if supported and monitored adequately for the next several years, as this will allow the MSP’s work with media beneficiaries to bear fruit, and to provide an opportunity for several journalists and other media professionals the MSP works with to become more solidly part of the media industry. At the same time, more mature media figures and outlets will have a chance to make the changes needed to become more firmly established, competitive with their state-run counterparts, and to put systems in place that will ensure their financial well-being and chances for survival. In addition, building trust networks, and understanding and adapting to local situations requires time. It would be counter￾productive to allow existing MSP-supported networks to disintegrate. For this reason, the current MSP team should largely be encouraged to continue and capitalize on their expert roles within their respective communities. The newer needs-based and customized approach that will now shape the MSP program should allow partners to develop unique organizational strategies and capacity. This evaluation finds this more focused method useful for many beneficiaries, and should, therefore, be retained and expanded in future programming. Consequently, the MSP should modify its existing monitoring and evaluation plan to adjust to this new strategy, especially for work undertaken as part of Results 1 and 2. The milestones and benchmarks for media outlets for which the MSP plans more customized support should be reflected in the new PMP, with the goal of establishing a clear vision and sense of what meaningful changes for each station look like over a two-year period. Furthermore, the MSP, at the moment, tracks 16 key indicators, most of which are output￾oriented, or reliant on changes in scoring to the IREX Media Sustainability Index (MSI).19 While the insight and information gained from the MSI is interesting and useful to consider for the MSP, it is difficult to rely on as the primary measure to assess the impact of MSP. In other words, causality and attribution are difficult to prove, and the MSI does not readily measure specific MSP programmatic interventions. It is, therefore, recommended that the MSP maintain tracking of the MSI as part of an overall trends analysis of the media landscape in Mozambique.                                                                  19 The research and data collection for the MSI is conducted independently from the MSP. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 37 Also, qualitative and quantitative M&E activities should be continued that will offer sufficient information and insight related to how MSP is working towards is intended results and outcomes. In accordance, the evaluation team suggests the following changes to the M&E side of the MSP, which will offer insight and perspective useful for both management and programming related activities. The MSP could reconsider the design of its MCAT. MSP capacity building activities, as one of the project’s primary features, are designed to improve the quality and quality of media content in Mozambique to counter the current media landscape which is described by respondents as “biased,” “politically tainted,” and “unprofessional.” To better answer questions related to MSP contributions towards improving journalist capacity and the state of reporting in Mozambique amongst target MSP beneficiaries, adequate systems of monitoring and analysis of impact need to be put in place. In its current form, the MCAT does not allow for observable change in programmatic objectives. The previous MCAT, while flawed because of inconsistent coding and lack of focus on individual trainees, was sensitive to change and variation over time. The four-point scale enables more sophisticated kinds of analysis. The binary measure, while simple and more likely to avoid coding errors, will provide less information on how much improvement has occurred. It will simply convey whether change has occurred or not. The evaluation team recommends returning to a 4-point scale of some sort with clearly defined categories, and perhaps carefully selecting coders, increasing monitoring of coders, intensifying training for coders, applying an adjudication process, or eliminating coders who tend to be inconsistent in their judgments of articles. The challenge of the MCAT lies not with the tool or complexities of content analysis. (In addition to content analysis being one of the most commonly used research forms for media development programs, it is similarly used by media houses and journalism organizations around the world.) Challenges with applying the original MCAT might have been related to turnover in MSP M&E staff. For example, there have been a few different individuals that served as M&E coordinator, therefore, creating challenges to consistent management of the MCAT tasks. The measurement objective notwithstanding, the NORC evaluation team finds some merits with some of the approaches used for the original MCAT tool. It is beneficial to keep track of journalists who are not in the program in order to compare competencies of non-MSP journalists and MSP-trained journalists, in addition to gauging improvement of MSP-trained journalists’ capability over time within this group and in comparison to others. The evaluation teams notes that the MSP will include some non-MSP journalists in the new MCAT database, which will be beneficial for capturing Result 1 objectives. Inconsistent scoring and article selection, which were cited as issues with the original tool, may be resolved with application of more rigorous oversight to the coding process. The training, background, and overall research management of the coders could be a source of inconsistency in the coding process. It is recommended that the MSP have three, and that coders receive training and regular support, check-ins and data quality assessments from perhaps a content analysis specialist. In addition, the MSP could hire a consultant, who is a seasoned content analysis expert to periodically work with the team as an expert adviser. Such a consultant would be able to complement the efforts of the current M&E staff working from the Maputo MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 38 office in a way to offer counsel and mentoring related to the content analysis, and to ensure that it is not just data being collected for data collection’s sake, but that it will yield the insights and information that will help bolster understanding of how MSP’s capacity building activities are improving (or not) the target beneficiaries. The evaluation team suggests the MSP establish a system for “M&E check-ins” monthly in order to assess the progress and quality of the MCAT scoring, and to ensure that the program is on target and able to meaningfully conduct the data collection and research required for analysis of MSP-supported activities focused on improving media professionalization. This would be a useful way of ensuring support for the management and quality of media development M&E, and should be relatively low cost and easy to maintain. The implementer will be able to identify and mitigate any scoring problems. The previously recommended content analysis expert could be available on retainer to offer support as needed as well as a check in with the team in Maputo for about 10 days per year – all of the work can be done remotely, with the aid of Skype or Google Hangouts. Making such adjustments to the management and design of the MCAT should yield more meaningful information related to the two key intermediary results related to Result 1, which are presently listed in the performance management plan (PMP) as: 1) media professionals have expanded skills; and 2) media professionals produce more and better quality content. The indicators presently used to monitor the progress of Result 1 include: the number and percentage of targeted journalists that demonstrate improvements in the professional quality of media content they produce. The MCAT is the best tool to provide information for these indicators. However, to complement the MCAT data, the MSP could also periodically run focus groups or have key informant interviews with the journalists the program engages with in order to acquire better qualitative data related to questions of how and why the MSP makes a difference (or not) to improving their professional capacity. Changes to the M&E approach and indicators used for Result 2 are encouraged. In the first years of the MSP, the implementer used content analysis as the main means of verification for the activities related to Result 2, improving the financial capacity of Mozambique’s media sector. The two intermediate results that MSP seeks to accomplish include: 1) media organizations improve organizational capacity; and 2) media organizations strengthen financial viability. The MCAT is not designed to assess organizational capacity or financial viability, which the MSP implementation staff acknowledged in interviews. The MSP is currently developing assessment tools for organizational capacity assessments. These are modeled after the Milestone Tool used for the program’s Result 4-related work with community radio stations in the Northern Provinces. This change should yield more information and insight on MSP contributions to specific media outlets with regards to improvements to business and financial viability questions. Of note, the media outlets interviewed as part of this evaluation, shared before-and-after stories about how the MSP impact on their business operations and efforts to become financially viable. This sort of information should be better captured and more consistently monitored over the next two years in order to fully appreciate the impact of the program, as well as to offer insight and learnings to program management responsible for Result 2 activities. It will also give important insight to the differences between the types of media outlets that MSP works with, and will offer an evidence-based perspective on the financial viability for independent commercial media MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 39 based out of Maputo. Moreover, utilizing the Milestone Index approach for Result 2 will offer improved insight on improvements to capacity of media outlets as well as long-term prospects towards sustainability. Additionally, it is recommended that benchmarks for each of the core media outlets that IREX will provided targeted assistances to be established so that a meaningful sense of how MSP unique contributes to each station’s strengthened business management, organizational capacity, and any overall improvements to financial viability can be measured. During the course of the evaluation, USAID personnel noted that they have organizational capacity tools that they can share with the MSP implementation team. As such any resource sharing in this area makes a lot of sense, and will benefit the Mission as well as media development activities in the long-run. As IREX is the only NGO offering support to media in the area of business and financial management practices, such information could also be useful if shared with the rest of the donor community. The information would help shed light on the how the business side of media benefits from donor support. Review whether current public relations and marketing of the MSP is done in the most optimal way. Key informant interviewees suggest some dissatisfaction or frustration with what they perceive as a lack of transparency in the MSP program, as well as insufficient opportunity to communicate their ideas and opinions about how to best utilize the resources of the MSP. The evaluation team found that the MSP undertakes a variety of public dissemination efforts – both online and offline – and that key MSP management regularly meet with and make themselves available to MSP beneficiaries and the wider Maputo and Nampula communities. Nonetheless, due to the number of respondents who felt that the program could be more transparent and open to what Mozambicans think and specifically what long-term media professionals feel should be the priorities for media development, it is suggested that the MSP re-assess how it communicates its offerings and successes. Is there any room for change or improvement? Perhaps the program could re-vamp how it disseminates information to partners -- would this be best achieved through creation of a newsletter and re-tooling how it shares information online? These are the sorts of questions that may be included in MSP conversations with partners. A digital newsletter showcasing best practices and skills that the MSP imparts to target beneficiaries might be one way forward. It could offer tips and suggestions in the areas of digital literacy, digital security, and current trends in journalism. It could also be used as a platform for showcasing the lessons learned from the program as well as to shine a spotlight on success stories from the project. This is done to a great extent by the Newsflashes. Perhaps these could be compiled monthly and shared out through a newsletter to all key beneficiaries and key stakeholders of the project, in addition to some key skills and tips from the top staff at IREX. Continue the practice of offering basic skills training for journalists. The MSP has hit a nerve with the current schedule and training offerings as part of Result 1. MSP beneficiaries interviewed have a largely favorable view of the MSP’s basic skills classes, remarking that the starting points of journalism education and improving the quality of reporter were fairly low – both senior MSP implementation staff and several interviewees from STV and Debate, mentioned that the MSP’s entry point into Mozambique is all the harder due to the lack MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 40 of skills and professional competencies of those entering the journalism profession. Respondents espoused the view that MSP should continue its activities (i.e. offering more basic journalism training, more courses on introduction to the internet-based journalism, and more hands-on internship and mentoring opportunities), because the program fills a gap in available trained entry-level journalists. There did not appear to be a feeling of oversaturation, or that there was diminished return on the investment of training. As one journalist in Nampula commented, “…We need to do more to identify problems of media to help resolve the issues that affect their ability to be professionals in their area. We need more training of journalists, because a man without information cannot inform the other people in his community.” Given the high degree of satisfaction and positive response to the MSP journalism training offerings, it is suggested that the MSP maintain the mix of activities that are presently part of Result 1. This will require some balancing of resources as the implementer intends to work with a narrower range of outlets and offer more targeted, specialized support to a smaller pool of beneficiaries. The MSP is encouraged to continue offering general classes, internships, and training opportunities, however, to a broader population of students and young professionals inclined to pursue journalism as profession. Offer more tailored and mentored approaches to business support. Outlets interviewed all indicated that they benefit from MSP assistance related to financial viability and business training. Mozambique’s existing establishment offers few resources for such instruction. Additionally, banks and other options to secure investment are scarce. Beyond the potential of advertising revenue, the media outlets need to assess other forms of revenue and other business models to secure their long-term survival. As many independent media have been started and are currently run by journalists who have had no to little business or management training, the work of MSP serves an important need in Mozambican media development. The MSP is encouraged to think through and offer recommendations for who and what could fill the role that they are playing in terms of providing business and financial training when the MSP ends. The media will continue to have needs related to Result 2 well beyond the lifespan of the MSP, and a future vision for what to do after the program ends would be very beneficial. Explore ways to improve access to market and audience research. Media outlets generally reported that they either did not have staff or resources to pay for audience research or that the market research or that what they did have access to was not of sufficient or of a reliable quality for them to make meaningful use of. For the Mozambican media commercial, independent media market to grow and prosper in the years ahead there will need to be improvements in terms of the availability and affordability of market and audience research in Mozambique. One deliverable that the IREX MSP could contribute to this exploration could be a research paper that maps and documents the existing resources, researchers, and insights that media companies, advertisers, and investors have access to as well as the types of questions and information that media outlets, advertisers and investors would like more of. The growing use of internet based media will create additional sources of advertising revenue for media outlets. It would be great if any activity related to market and audience research could also take this into consideration. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 41 Review opportunities for more North/ South cooperation and collaboration. While in Nampula, the evaluation team learned of the isolation stakeholders face. Their ability to travel to Maputo and other parts of Mozambique is limited. Interviewees expressed great interest in meeting up with their counterparts in Maputo or other locales in the country. They say that learning from each other had great potential, and that peer-to-peer learning opportunities, sharing of best practices, and having a chance to hear about Mozambican innovations in media in journalism would all be of great value. Due to costs of this type of experience, a program like MSP would be perhaps provide the only means of making this happen. This type of in-country North-South collaboration – in other words, programs and opportunities that would bring together journalists and other media professionals from different regions of Mozambique – would facilitate sharing of technical skills, serve as a platform for the exchange of media content and programs, and cultivate a cadre of journalists and media professionals that are representative of all of Mozambique as well as homegrown opportunities for training, education, and mentoring support. Bring focus and strategy to Result 5 related activities and programs. MSP activities implemented as part of Result 5 – Mozambican media organizations display an increased ability to advocate for press freedom and for a legal environment that enables media freedom – according to the MSP personnel, is being streamlined and will have a greater focus going forward into the next phase of the program execution. The MSP team will spend more time focusing on association building and revamping its legal and security support. This re￾focused effort is welcome by interviewees, based on feedback from the KIIs, which reveal that the most recognized association are the Mozambique chapter of the MISA (which respondents thought has the most potential, but needs to be revitalized), and the National Journalist Union (which respondents considered weak and too politically aligned). Many journalists and especially editors felt that journalists were not very familiar with basic media law and policies and that providing this type of training would be beneficial to the media sector. Thus, the evaluation team suggests that the MSP incorporate more training on enabling environment issues, including offering more training on defamation, libel, and the new access to information law as part of its outreach and support to journalists and media outlets in Mozambique. It is understood that the RCC, an initiative of the MSP, is under review and that the program will decide on how to best utilize this network and resource. As a potentially positive sign, most respondents indicate familiarity with this mechanism, and are open to discussing how to improve its operation and structure. Based on interviewee feedback, the RRC could be part of a future strategy, but there needs to be more outreach related to how it works and how journalists can take advantage of it, and how it fits in to the overall strategy of developing stronger associations an advocacy organizations to represent freedom of expression and access to information in the country. As an important aside, the advocacy effort seems to have strong potential with regards to developing strategic advocacy efforts with North- South partners. This may be an area that the MSP could develop better countrywide advocacy collaboration, especially since there are MISA chapters in both regions. As a follow-up recommendation it would be useful to think of ways to support advocacy leaders from across Mozambique in an effort to come together in partnership with the MSP to plan and coordinate their activities. Leaders could also receive joint and coordinated training related to their advocacy objectives. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 42 Make advocacy programs more participatory and collaborative. The MSP could take advantage of other USAID programs. This is especially relevant for Result 5, according to comments received from USAID personnel. The MSP could work with other civil society advocacy programs supported by USAID. Indeed, as the core focus of the MSP’s Result 5 is to improve press freedom and access to information related advocacy initiatives, coordination and collaboration with other CSO advocacy and activism programs may be beneficial, especially if there are opportunities for some of the media advocacy organizations to be involved and linked to CSOs working in other areas. Exploring with USAID ways that the MSP could tap into other work that USAID supports in the areas of advocacy capacity building; developing civil society movements and organizations that result in success; and, helping NGOs develop into sustainable, long-term organizations, would be an added value to MSP. There appears to be opportunity for resource sharing and collaboration between different advocacy efforts, and it would be worthwhile for the MSP staff working on Result 5 to explore possible synergies between their efforts and other USAID programs. This type of collaboration and coordination might be explored with IBIS and Dialogo. While MSP occasionally works with these two programs, interviews with both organizations suggested that there may be additional means of engagement, especially around Result 5 related work. This could take the form of coordinated and collaborative training, workshops, conferences, or research. Given the relatively low opinion interviewees had of journalism unions and advocacy organizations meant to represent the interests of journalism and a free press, deeper exploration between the international partners working to support civil society in Mozambique seems useful. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 43 ANNEXES MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 44 ANNEX A: MSP ORGANIZATIONAL CHART MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 45 ANNEX B: SCOPE OF WORK Mid-Term Evaluation USAID Media Strengthening Project (MSP) This form is meant to help DCHA/DRG Learning Team clarify goals and interests in designing an evaluation. It asks broad questions to help frame the overall purpose of the evaluation and then some detailed questions about the program. This is meant purely to help make sure that we are all focusing on the questions USAID missions care about most and to help the evaluation team think through important design elements for the evaluation. If there are existing program documents that provide an answer to any of the questions, please email them as attachments. STATEMENT OF WORK MEDIA STRENGTHENING PROGRAM MID-TERM PERFORMANCE EVALUATION Prepared by: USAID/Mozambique Evaluation Mechanism: Democracy, Human Rights, and Governance Learning, Evaluation and Research (DGR-LER) Indefinite Quantity Contract DESCRIPTION OF ACTIVITY TO BE EVALUATED Activity Name: Media Strengthening Program (MSP) Cooperative Agreement: AID-656-A-12-00001 Implementing Partner: International Research and Exchange Board (IREX) Period of Performance: June 11, 2012 – June 10, 2017 Total USAID Amount: $9,978,124 ($5,726,895 sub-obligated into award) Cost Share Amount: $500,000 Location: Main Office in Maputo, Mozambique with Field Office in Nampula, Mozambique BACKGROUND The Media Strengthening Program (MSP) is the largest activity in USAID/Mozambique’s Democracy, Human Rights, and Governance (DRG) portfolio. Under the Mozambique Country Development Cooperation Strategy (CDCS) 2014-201920, MSP contributes to the following:  Development Objective 1: Democratic governance of Mozambican institutions strengthened o Intermediate Result 1.1: More effective civil society participation in governance processes  Sub-Intermediate Result 1.2: Citizens better informed of their rights and responsibilities                                                                  20 See Annex A for the complete CDCS Results Framework. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 46 Launched by IREX in June of 2012 with the goal of a free, open, diverse, and self-sustaining Mozambican media sector providing high quality information to citizens that promotes debate, accountability, and transparency, the MSP design has five main result areas: 1. Increased professional capacity of the Mozambican media sector 2. Strengthened business management/organization capacity of media organizations to improve long-term financial viability 3. Improved journalism education at Eduardo Mondlane University’s School for Communications and Arts (ECA) 4. Increased ability of community radio stations to provide more and better information to listeners 5. Increased ability of Mozambican organizations to advocate for press freedom and an improved legal enabling environment for media Additionally, MSP aims to achieve cross-cutting results related to gender and public health/HIV/AIDS. The cooperative agreement is partially funded by the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) and its the Gender Based Violence Initiative (GBVI). USAID will also sub-obligate Counter Trafficking in Person (C-TIP) funds to MSP in FY2015. These non-DRG funds are used to advance the objectives of the different funding sources via programming designed to achieve MSP’s primary results. MSP will have completed two years and eight months of implementation by the time this evaluation begins in February 2015, or just over half the life of the project. The first four months of MSP were considered the start-up phase and included in Work Plan 1a, consisting of largely setting up the facilities and forming the initial team. Work Plan 1b corresponded to FY2013, during which MSP activities increased in number and intensity across the result areas. Under Result 1, increased professional capacity of the Mozambican media sector, MSP offers a variety of activities to support journalists and media professionals of different experience levels. These include a structured internship program for young professionals and mentorships for aspiring and experienced journalists designed to provide real-world learning opportunities. In addition to specialized trainings on topics such as election journalism and digital media, MSP also provides training on basic journalism skills such as writing, photography, and English. Experts from around the world are routinely brought in to transfer knowledge and skills to activity participants. Programming under Result 2, strengthened business management/organization capacity of media organizations to improve long-term financial viability, has focused on working directly with media outlets. IREX has provided business consulting services to TV, radio, print, and digital outlets. Additionally, MSP awarded sub-grants to three media outlets to implement action plans designed to improve their operations and implement new strategies to improve their financial viability. It is now in the process of awarding a second round of sub-grants. Result 2 also includes the intensive multi-day Emerging Media Leader course for mid-level media professionals. IREX plans to increase support for media entrepreneurs and new digital media business ventures. As part of Result 3, improved journalism education at Eduardo Mondlane University School for Communications and Arts (ECA), IREX has partnered with Rhodes University of South Africa to support ECA management and teaching staff in improving journalism curriculum. The two MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 47 schools engaged in a series of exchanges and curriculum development workshops over the first two years of MSP. IREX staff also taught courses at ECA and worked with faculty and staff to create a student-produced newspaper and online publication. IREX and Rhodes have designed a graduate-level scholarship program for ECA professors, which will be just beginning at the time of the mid-term evaluation. MSP supports selected community radio stations in the Northern Mozambique provinces of Nampula, Cabo Delgado, and Zambezia as part of Result 4, increased ability of community radio stations to provide more and better information to listeners. Initial activities focused on improving strategic planning and business operations at beneficiary radios and introducing low-cost technologies such as Frontline-SMS to enhance listener engagement. Several activities have been co-implemented with the sub-grantees Forum of Community Radios (FORCOM) and Community Information and Communication Support Center (CAIC). The current programming focus is capacity building in basic journalism, radio production, and management, which is to be closely aligned with an equipment upgrade program and mobile training lab for participating stations. Result 5 is increased ability of Mozambican organizations to advocate for press freedom and an improved legal enabling environment for media. In this area, MSP has supported media associations to strengthen their organizations and improve their advocacy skills via training. MSP made legal and security support available for journalists and supported the creation of the Rapid Response Commission (RRC) to take action when journalists’ rights are violated. MSP has also worked to engage stakeholders in debate around issues related to media through monthly debates and online outreach. Half way into FY2014 and Work Plan 2, IREX carried out an internal assessment and strategic adjustment exercise, in close coordination with USAID. As a result, IREX decided to strengthen the MSP team to better respond to beneficiary demand, by adding new Media Specialists, Junior Staff / Trainees, an Expat Community Radio Manager, an Information Technology (IT) expert, and part-time experts for advocacy and linguistics. This reinforcement was designed to allow IREX to rely less on external consultants and play a larger role in co￾implementing activities with partners and sub-grantees. Following IREX’s strategic adjustment exercise, USAID conducted a data quality assessment of several MSP indicators. While the assessment team found the standard indicators related to number of beneficiary individuals and organizations to be of good quality, it also identified the need to improve the data quality for indicators using the Media Content Analysis Tool (MCAT) to measure journalism quality.21 To complement the data quality assessment, USAID and IREX conducted a complete review of MSP’s Monitoring and Evaluation (M&E) Plan. As part of the exercise, the team worked to streamline the activity’s indicators by eliminating indicators that were tied to unused programmatic approaches (such as use of audience research) and strengthening indictors to better measure results (like the MCAT indicators and an improved community radio milestone indicator). Given the data quality issues and recent adjustments to the M&E Plan, the new data generated from this mid-term evaluation will be used to complement the data MSP has gathered to date, offering a more complete measure of activity performance.                                                                  21 The MCAT, developed by IREX and Social Impact,rsid S> MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 48 Additional background materials to be made available for review:  MSP Cooperative Agreement and Modifications  MSP Monitoring and Evaluation Plan (original and revised)  MSP Work Plans, including Gender Based Violence Initiative Work Plan  MSP Quarterly Reports  USAID MSP Data Quality Assessment  USAID Full Finance and Administrative Review  MSP Strategic Adjustment Summary  IREX 2011 Gender Assessment  MSP Email Newsflashes  MSP Activity Videos (on IREX Mozambique YouTube channel)  Annual MSP Publications (Gender in the Media, Health in the Media, Investigative Journalism, and Media Sustainability Index)  USAID Country Development Cooperation Strategy 2014-1019 PURPOSE The purpose of the evaluation is two-fold: 1) validate interventions that are valued by stakeholders and contributing to desired results, and 2) generate evidence-based recommendations for improved implementation of MSP over its final two years. The evaluation will focus on the result areas related to professional capacity building, business sustainability, and advocacy, along with cross-cutting initiatives on gender equity and building local capacity via sub-grants. The primary audience for this evaluation is USAID and IREX staff directly involved in MSP implementation. The recommendations from the evaluation will be used as the basis for adjustments to the MSP technical approach and implementation during the final two years of the five-year activity. The evaluation report will also serve as a resource for design of any future USAID/Mozambique media-related interventions. Secondary audiences include MSP stakeholders in Mozambique, USAID staff involved in media programming worldwide and other donors and implementing partners working in media strengthening. The final evaluation report will be made available on the USAID Development Experience Clearinghouse. EVALUATION QUESTIONS Increased professional capacity of Mozambican media sector (Result 1) 1. To what extent have MSP capacity building activities enabled media professionals to improve their professional capacity in targeted areas? To what extent are MSP capacity building activities contributing to improvements in the professional capacity of media professionals in areas targeted by the program? MSP has trained hundreds of journalists and media professionals via a host of capacity building activities, including internships, mentorships, and specialized training. USAID is particularly interested in understanding to what degree the different interventions have responded to the needs of the sector, expanded participants’ skills, which resulted in more and better quality media content. The evaluation should be designed using methods that will capture the MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 49 perceptions of participants, the managers of the media outlets that hire and employ them, as well as media professionals and stakeholders who have not participated in MSP. It is important for evaluators to capture details of specific and concrete experiences to demonstrate evaluation participants’ perceptions of successes and shortcomings of the program activities. The evaluators will use available data and evidence such as material produced by participants to substantiate the perceptions and analysis shared by the participants and stakeholders. Please see the Methods section for a matrix of suggested groups of capacity building activities and data sources. Strengthened business management/organization capacity of media organization to improve long-term financial viability (Result 2) 2. To what degree have MSP beneficiary media outlets improved their financial viability22 as a result of working with MSP? To what extent is MSP contributing to an improvement in the financial viability23 of media outlets participating in the program? MSP has reported providing customized assistance to nine non-state media outlets since it started. This assistance has included business consulting, one-on-one coaching, and courses for emerging media leaders. Additionally, three of these outlets have received sub-grants to implement plans to improve their organizational capacity and financial viability (three additional sub-grants are under development). USAID is interested in which types of assistance are perceived to be the most valuable. Some of these outlets have been reluctant to share the financial details of their businesses. The evaluation shall seek to substantiate and evaluate, to the degree possible, any stated improvements in financial viability by beneficiary media outlets. For example, identification of new systems, and/or approaches in doing business, that have been institutionalized by media outlets, as a direct result of their participation in MSP activities. Also, the evaluation should capture and highlight challenges, if any, that media outlets may be experiencing as they strive towards financial viability.0. Increased ability of Mozambican organizations to advocate for press freedom and an improved legal enabling environment for media (Result 5) 3. How can MSP strengthen activities to improve advocacy for media freedom and an improved legal enabling environment for media? MSP has hosted debates on media topics, worked with journalists to form a rapid response commission to respond acts against journalists, and provided capacity building support to media associations among other advocacy related activities. This evaluation question seeks to make sure MSP is using all appropriate approaches possible to achieve Result 5. The evaluators should compare MSP advocacy-related activities to donor-funded media activities that were determined via independent evaluation to have contributed to concrete improvements to national-level media policy formulation, revision, and/or implementation. Of particular interest are approaches which involved building the advocacy capacity of local organizations. METHODS                                                                  MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 50 The evaluation team shall begin the evaluation with a desk review of relevant documents, including but not limited to the documents listed in the Background section above. In developing the evaluation design, the evaluation team is asked to suggest the most appropriate methods for answering each evaluation question given the evaluation’s budget. For example, the first evaluation question may entail self-assessments on competencies required by skilled and high performing journalists. These assessments may be completed by MSP-trained journalists to assess their current skillsets and can then be contrasted against manager(s) of the journalist(s) or future employer(s) looking for skilled professionals. The methods for this evaluation question may also require review of work samples and evaluation of journalist work through objective means and subject matter expert review (see Table 1 below for an example). Similarly, methods should be presented across each evaluation question as different questions may require distinct methods. In line with the USAID Evaluation Policy, data will be collected through a mixed method approach, by collecting both quantitative and qualitative data collection methods. Examples of important qualitative data sources may include key informant interviews, focus group discussions, content analyses, and quantitative data sources may include organizational assessments, surveys, and program monitoring data if available, etc. Data should be triangulated using this mixed-methods approach and where possible, quantitative data should also be used to support qualitative analysis. For example, the following matrix, or a modified version approved by USAID, shall be used to structure the research and analysis related to evaluation question 1. The contractor is expected to develop a similar matrix for questions 2 and 3, as part of the evaluation design, prior to field data collection. Table 1: Matrix of Capacity Building Activity Groups and Potential Data Sources Participant perceptions and self-assessment Media manager perceptions and assessment Relevant data and materials produced by participants Basic Skills Training: English, journalistic writing, photography, and graphic design. Mostly journalism students and young journalists. (Focus groups, key informant interviews, and self￾assessment questionnaire/mini survey24) Editors, managers, and supervisors where the participants work. (Key informant interviews and/or objective assessments) Any pre and post￾tests conducted by IREX; and materials produced before, during, and after the training. (Content / work product assessment tool) Internships: MSP intern program and funded media outlet interns. Primarily recent journalism graduates and young journalists. (Focus groups, key IREX supervisors and internship program coordinator. supervisors at Performance evaluations; work products including journalistic materials. (Content                                                                  MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 51 informant interviews, and self￾assessment questionnaire/mini survey)) participating media outlets. (Key informant interviews) / work product assessment tool) Mentorships and story stipends: investigative journalism, economic journalism, trafficking in persons, gender based violence, and health fellows. Young, mid-level, and experienced journalists (Focus groups, key informant interviews, and self￾assessment questionnaire/mini survey)) Mentors, editors and supervisors in the media outlets where the participants work, counterparts from civil society organizations (Key informant interviews) Materials produced prior to, during, and after the mentorships (Media Content Analysis Tool) Media labs: multimedia and television. Primarily recent journalism graduates and young journalists (Focus groups, key informant interviews, and self￾assessment questionnaire/mini survey)) MSP Media Specialists in charge of media labs, editors and supervisors in the media outlets where the participants work (Key informant interviews) Materials produced prior to, during and after the media labs (Content / work product assessment tool) Specialized trainings: elections reporting / peace journalism, extractive industries, neglected sub￾tropical diseases, gender sensitivity Young, mid-level, and experienced journalists. (Focus groups, key informant interviews, and self￾assessment questionnaire/mini survey)) Editors and supervisors in the media outlets where the participants work, counterparts from civil society organizations. (Key informant interviews) Materials related to the relevant topics produced before and after the specialized training (Media Content Analysis Tool) DELIVERABLES AND TIMELINE All deliverables are subject to USAID/Mozambique acceptance and approval. Evaluation Design and Work Plan The written evaluation design shall include details of the methods, data collection instruments, and data analysis that will be used to answer the evaluation questions and make related recommendations. The evaluation design shall be accompanied by a work plan detailing the tasks to be accomplished, the people responsible for each task, and a schedule for when the tasks will be completed. A Gantt chart should be used to present the schedule. The evaluators MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 52 should reference the USAID Evaluation Policy for guidance on evaluation design. This will be completed and a draft submitted to the mission for review prior to the arrival of the team in Mozambique. In-brief materials Upon arrival in Mozambique, the evaluation team will offer an in-brief for relevant USAID staff. The evaluators will present the evaluation design and work plan, offering participants copies of the documents as well as a one-page briefing document on the evaluation. Out-brief materials The evaluation team will present their initial findings and analysis to relevant USAID staff prior to the team leader’s departure from Mozambique. The evaluators will present the attendees with six slides summarizing the initial findings by the five evaluation questions and providing the evaluators’ overall thoughts on MSP performance. Draft Report The report shall adhere to the structure, standards of quality, length, and requirements detailed in the USAID How-to Note Preparing Evaluation Reports. The evaluators must complete the data analysis according to the evaluation plan and incorporate it into the draft report prior to submitting the draft report to USAID. All conclusions and recommendations in the report must be supported by the findings of the evaluation. It shall be presented to USAID for comment 20 working days after completion of the field work. The evaluation reports will be reviewed by USAID against the Evaluation Policy’s “Criteria to Ensure the Quality of the Evaluation Report” as described in Appendix 1 of the USAID Evaluation Policy. Final Report The final report must adhere to the USAID guidance mentioned above. Additionally, it must address USAID’s comments to the draft report. Special attention should be paid to the branding and marking requirements. The revised report must be sent to USAID within five working days of the evaluators receiving USAID’s comments to the draft document. Once the final report is approved, the evaluation should present USAID a digital copy of all data compiled for the evaluation and digital copies of the evaluation report and annexes in Word and PDF. TEAM COMPOSITION All team members must be familiar with the USAID Evaluation Policy. Experience working on USAID evaluations is highly preferred. All team members will be required to provide a signed statement attesting that they have no conflict of interest, or describing any existing conflict of interest. Team Leader One international evaluation expert with experience in media project evaluations will serve as the team leader. An advanced academic degree in a related field is required. The expert must have experience leading independent evaluations of donor-funded media projects similar to MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 53 MSP. Portuguese language skills are strongly preferred. Fluency in English is required. Experience working in the Southern Africa is a plus. Local Evaluation Specialist One local evaluation specialist will support the team leader. The evaluator must have a strong understanding of the Mozambican media sector and general governance context. It is preferable that the individual have experience carrying out evaluations of governance projects in Mozambique. Fluent Portuguese skills are essential, as the local evaluation specialist will serve as an interpreter for the team leader as necessary. Professional proficiency in English is required. USAID Media Expert The core evaluation team may be assisted by one media expert from USAID/Washington, depending on availability. SCHEDULE AND LOGISTICS The DRG-LER contractor’s core team shall prepare the evaluation design, in coordination with USAID, by February 14, 2015. The expected period of performance for the evaluation is March 1, 2015 – June 15, 2015. The field work in Mozambique is expected to take place over approximately two weeks sometime between March 15, 2015 and April 15, 2015. BUDGET $80,000 ADDITIONAL RESOURCES USAID ADS Chapter 203: Assessing and Learning USAID Evaluation Policy, January 2011 USAID How-to Note Preparing Evaluation Reports ANNEX C: DATA COLLECTION INSTRUMENT KEY INFORMANT INTERVIEW GUIDE (General) IREX Key Staff Interview Guide DRG Learning, Evaluation, and Research Activity Mid-term Performance Evaluation of USAID Media Strengthening Program May 2015 INTRODUCTION Hello, my name is Susan Abbott, and I’m an independent consultant working with the National Opinion Research Center, at the University of Chicago, based in the United States, and we are conducting an evaluation of the Media Strengthening Program, which is carried out by the International Research & Exchanges Board. The MSP seeks to improve the capacity of the MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 54 journalism sector in Mozambique. The program and this evaluation are funded by USAID and implemented by IREX. You have been invited to provide an in-depth interview because we are interested in your opinions and feedback on media in Mozambique, your views on the practices and standards of journalism, and your personal experiences with the Media Strengthening Program. Your opinions are considered as valuable regarding media in Mozambique and in the design of future independent media strengthening programs. I’d like to ask you about your experiences since you completed the training program. Many of the questions I’ll ask are open-ended, that is, designed to allow you to answer in your own words. I’d like to make a digital recording of the conversation if you don’t mind so I have the full details of your responses. May I begin with my questions now? It is possible for us to complete the interview in about 1 hour, but I don’t want you to feel rushed in your responses. Objectives of the project: The Media Strengthening Program (MSP) is the largest activity in USAID/Mozambique’s Democracy, Human Rights, and Governance (DRG) portfolio. Launched by IREX in June 2012 with the goal of, “[a] free, open, diverse, and self-sustaining Mozambican media sector providing high-quality information to citizens that promotes debate, accountability, and transparency.” The MSP contributes to broaden USAID goals of strengthening democratic governance of Mozambican institutions and second- and third-order aims of more effective civil society participation in governance processes and citizens becoming better informed of their rights and responsibilities. In order to encourage a more open, diverse and capable media sector in Mozambique, the MSP design focuses on achieving five main results: (1) Increased professional capacity of the Mozambican media sector (2) Strengthened business management/organization capacity of media organizations to improve long-term financial viability (3) Improved journalism education at Eduardo Mondlane University’s School for Communications and Arts (ECA) (4) Increased ability of community radio stations to provide more and better information to listeners (5) Increased ability of Mozambican organizations to advocate for press freedom and an improved legal enabling environment for media Additionally, MSP aims to achieve cross-cutting results related to gender and public health and HIV/AIDS.25 The purpose of the midterm performance evaluation, therefore, is two-fold. The evaluation seeks to: (1) validate interventions that are valued by stakeholders and contributing to desired results, and (2) generate evidence-based recommendations for improved implementation of MSP over its final two years. The evaluation will focus on the five areas                                                                  MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 55 of result areas related to professional capacity building, business sustainability, and advocacy, along with cross-cutting initiatives on gender equity and building local capacity via sub￾grants. The purpose of this interview is to find out about your own views and to gain insight from your experience in the area. Topic: Background 1. Tell me about what it’s like to be Chief of Party (other title) for the IREX MSP? Probe: What has been the biggest surprise you have encountered? Probe: What aspects of running the program do you find most challenging? Probe: What aspects of running the program do you find most rewarding? 2. What do you perceive as the greatest challenge to media development in Mozambique? Probe: Does Mozambique face any challenges that you would describe as unique to the country? Probe: Where do you see Mozambique’s media sector in the next five to ten years? 3. Who are the key actors? Probe: Are there mostly local media actors or is there much influence from regional or international media? Probe: Do you think that radio, TV, print or digital media will shape the media environment in Mozambique in the years to come? Probe: What is the appetite for advertisers in Mozambique? Do you media stand a good chance in attracting advertisers? 4. Can you comment on the differences between state, commercial/private, and community media? Probe: Can you give me some examples of the differences between the types of content – both news and entertainment that they deliver? Topic: Professional Capacity Building 5. Can you tell me about Result Area 1? Probe: What are the Strengths, Weaknesses, Opportunities and Threats to Journalists in Mozambique in terms of their professional competencies, and their ability to practice professional journalism? Probe: What is the journalistic culture like in Mozambique? 6. What types of opportunities do Mozambicans have to pursue journalism education and advanced training that may be required to work in a 21st century newsroom? 7. To what extent have MSP capacity building activities enabled media professionals to improve their capacity in the outlined target area? MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 56 8. How have professionals demonstratively changed their behavior (have professionals become less biased in their reporting? Has the accuracy of reporting improved? Can you comment on journalists’ use of varied and trusted sources? Probe: Can you comment on how journalists area impacted by being involved in MSP? What is the effect of the training, mentorship and other capacity building activities on the journalists? Can you offer any examples, i.e. a before and after type of story about some of the journalists who have taken part in the MSP so far? Probe: What are the most significant accomplishments the project has made with regards to the scope of work for Result 1? Probe: What have been the most and least useful interventions the project has made so far in terms of promoting professionalization and quality media content? 9. To what extent are there strategic gaps in MSP assistance that are essential to promoting professionalize and quality media content? Topic: Business Sustainability/ financial capacity of Mozambique’s media sector 10. Can you tell me about Result Area 2? Probe: Tell me about the types of issues or areas of concern that media outlets need to address in order to strengthen financial viability? 11. Do organizations' perceive that their business management capabilities have improved as a result of working with the MSP, and if so, in what areas? Where do weaknesses still lie? 12. Do organizations perceive that their financial viability has improved as a result of working with MSP? Probe: how is this measured? What indicators or metrics are used to gauge this? 13. What specific changes have been implemented as a result of collaboration with MSP to improve business management capabilities and financial viability? 14. How could MSP more effectively contribute to greater financial viability of outlets? 15. What are the most significant accomplishments the project has made with regards to the scope of work for Result 2? Topic: Advocacy, Press Freedom and Access to Information 16. Can you tell me about Result Area 5? Probe: What is the scope of the problem that MSP is trying to address through the activities and assistance offered in Result 5 of the program? Probe: Do journalists, editors, bloggers, etc. that you work with have a basic understanding of media law and policy? Probe: Do you journalists, editors, and bloggers have a chance to learn about legal aspects of journalism and possess a basic understanding of their legal rights? MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 57 Probe: Can you give me some examples? 17. Who are the local groups/ actors engaged in working on these problems associated with the Enabling Environment for Free and Independent Media? Probe: To what extent have your worked with any/ all of these groups? Probe: What have these groups/ initiatives accomplished? Probe: What have these groups/ initiatives failed to accomplish? Probe: Is there someone at group X (civil society, university, other advocacy related group) that you recommend that I talk to? 18. How can the MSP strengthen activities to improve advocacy for media freedom and an improved legal environment? In other words, is the MSP exploiting all appropriate approaches possible to achieve Result 5? 19. Are there international best practices that have been effective elsewhere that may be applicable for the Mozambique context (of particular interest are approaches which involved building the advocacy capacity of local organizations)? Probe: are local advocacy efforts tapped into international advocacy networks? Regional networks? Or Local Networks? For example, Media Legal Defense Initiative, International Bar Association, Committee to Protect Journalists? Probe: Are there examples of court cases or rulings, legal actions, or charges against journalists relevant to freedom of expression or access to information – did decision or argument made reference legal norms, international standards or best practices? Probe: Do regional legal and regulatory aspects of media and internet regulation affect how Mozambique treats journalists, bloggers and media outlets, and does this affect freedom of expression or access to information in any way? 20. What are the most significant accomplishments the project has made with regards to the scope of work for Result 5? Probe: How do you measure the success of advocacy efforts? What are the benchmarks and milestones that signal progress? Probe: Can you give me some examples? 21. For results under this component where there has been identifiable progress, can that progress be sustained without USAID support? Topic: Problems in the area: 22. What kind of effort should be made to improve the priority to X topics (to be determined by the flow of the interview) or area of media development? Please explain why/ or which areas. Follow up / probe: What kinds of topics do you think might need further development in relation to X? (Please explain why you take this view.) 23. In your experience, which problems in the MSP (media law, etc) concern people the most? MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 58 Follow up/ Probe: How did you learn about these topics/ problems? Follow up/ Probe: What is the scope of the problem? Follow up/ Probe: Why is this considered a problem? Topic: Cross-Cutting Issues 24. Who are the local groups/ actors engaged in working on these problems (Health, Gender, etc)? Follow up/ Probe: What have these groups accomplished? Follow up/ probe: To what extent have you worked with any/ all of these groups? Follow up/ probe: What have these groups failed to accomplish? Follow up/ probe: Is there someone at group X you recommend I talk to? Topic: Future Development 25. What, in your opinion, needs to be done to address the future of media development in Mozambique? Follow up/ probe: Who should be involved in addressing the problem? Follow up/ probe: What resources will be needed to address the problem? Follow up/ probe: Can independent media exist without donor support? Topic: Lessons Learned 26. What are most important lessons learned in respect to donor (in particular, USAID) support to Mozambique in terms of media assistance? 27. What, if any, would you describe as innovative features or best practices of donor activity? Wrap up Is there anything else you would like to discuss that we have not talked about? Thank you very much. Please feel free to email me if you have anything else to add or would like to clarify any of your remarks. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 59 ANNEX D: OBSERVATIONS ON THE MCAT DATABASE Observations on the MCAT Database The NORC evaluation team, in this Annex, provides an overview of some of the strengths and weaknesses of the new MCAT database. As one of the intentions of USAID programs is to develop M&E information systems that are suitable for evidence-based assessment of program change and progress, the following comments might be considered for future reference. The team finds that suggested areas of improvement of the new MCAT database are related to database structure, documentation, and, to a lesser degree, data quality and content.26 1. Database structure A. To make the MCAT database easier to use, NORC recommends that it comprise the following datasets:27 i. Course dataset: Fields (variables) with unique course ID, course name, or a categorical variable with topic (e.g., Saudé), start date and end date. These must be in separate fields and in a format suitable for analysis (i.e., adding/subtracting), number of sessions or hours. Note that each time a course is given, a different course ID should be applied, though the course name/topic should be the same. Also, the fiscal quarter of the course is not necessary, since it is implied by the course dates; ii. Journalist dataset: Fields (variables) with journalist ID, name, gender, approximate age, city of residency, highest education attained, K dichotomous variables (where K is the maximum number of courses any participant took) indicating which, if any, courses the journalist has taken; iii. Article dataset: The present record structure of this dataset is satisfactory. B. One challenge in using the current MCAT database lies with identifying which courses a journalist takes before an article is written (and, likewise, which articles come before each course taken by the author). Adopting the above might mitigate most of this ambiguity. 2. Documentation Critically, the IREX data would be even more useful for generating evidence-based research and analysis, if the documentation were improved. NORC recommends that documentation for the MCAT database be added in a manner that permits analysis to be easily conducted by anyone accessing the database. A non-exhaustive set of examples that present challenges for analysis follow. A. Unclear and unexplained training information i. It is not clear when the trainings took place. Courses have time periods in fiscal-year quarters as well as in month ranges and these periods are sometimes conflicting. The                                                                  26 The team’s comments are related to what was provided in the MCAT database on August 28, the time of the evaluation analysis, acknowledging that some elements may have been changed or added.   27 The list of variables is by no means meant to be comprehensive.   MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 60 inconsistencies, below, make it difficult to know when trainings ended and whether the “end-line” score came later. ii. Training data in new MCAT are inconsistent. For example, the tab, “Resumo de Mentorados” indicates that Hermenegildo Langa took “J. Invest” between August and Sept 2014, but “Progresso de Mentorados” indicates he started training in January of 2015. iii. In the tab, “Resumo de Mentorados” column, “FY15-Q3” is the course “SAÚDE (Out13-Set15)”. Ignoring the year contradiction, there is no information that conveys to which the range conveys? Is it a particular course sequence followed by a fixed cohort? There is also an overlapping course listed as “SAÚDE (Jan15- Dec15)” under FY15-Q2 with different journalists. There is no information that explains this organization. B. The above inconsistencies make it difficult to easily know when training ended and whether the “end-line” score came later. C. It is not clear in which month the fiscal year begins. D. The color coding schemes should be clearly explained and defined. E. It is unclear whether “Date” refers to the coding date or the article date. If the latter (which is preferred for analysis), then it is unusual what entries for a given journalist tend to group around the same dates. We need to assume, therefore, that this because the medium is weekly or monthly. 3. Data quality A. The implementer voiced a concern that scoring was not developed to be used for eval￾uation but for monitoring. NORC did not have the budget to do a dedicated data quality review of the score data. We did utilize the aggregated scores of each article in our analysis so where the reader disagrees with any of the analytic results it could be due to data measurement, aggregation, or coding errors. One result that should be of specific interest in this regard is that scoring of articles was biased in that the scores depended on the person conducting the scoring.28 B. Spelling i. Many entries in the database had slightly different spellings for the same item or person, even when within the same spreadsheet tab. To make the spellings consistent, it takes a great deal of effort to make the data useable. NORC found this was necessary, however, as the small number of observations in the database provided was already very small. C. Fewer than 576 of the 1,023 scored records could not be used for NORC’s analysis, because:                                                                  28 See Section 4 of the main report for a presentation of this analysis. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 61 i. There are 447 scored records without journalist names attached ii. There are names in the scored-data tab that do not appear to be in the training tab (for example, Raimundo Moiane and Alfredo Langa), therefore, the training status cannot be determined for these and some of the scored journalists 4. Content A. The current MCAT would have been a richer data source with many analytic possibilities for the present analysis, if, for example, 100 trainee journalists and 100 untrained journalists were scored for 2 to 3 articles before and 2 to 3 articles after taking a course. B. Articles scored i. Some journalists in the new MCAT database have a large number of articles scored, while others have none, or have only a baseline or end-line score. In order to create a database that can be used for substantive evaluation purposes, NORC recommends a review of how many articles are scored. ii. Some 17 journalists that were marked as trained had no articles scored. This further decreased the sample. C. Demographics i. The inclusion of demographic data (gender, education, age, geographic origin/residence, etc.) will permit interesting analyses of heterogeneity and also improve comparison of cases. MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 62 ANNEX E: SOURCES OF INFORMATION SUPPORTING DOCUMENTATION United States Agency for International Development, 2012. Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001. Washington, DC: USAID, 2012. United States Agency for International Development, 2014. Country Development Cooperation Strategy: 2014-2019. Washington, DC: USAID, 2012. United States Agency for International Development, 2013. FY2013 Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001, First Quarterly Report, 1 October – 31 December 2013. Washington, DC: USAID, 2013. United States Agency for International Development, 2014. FY2014 Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001, Second Quarterly Report, 1 January – 31 March 2014. Washington, DC: USAID, 2014. United States Agency for International Development, 2014. FY2014 Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001, Third Quarterly Report, 1 April – 30 June, 2014. Washington, DC: USAID, 2014. United States Agency for International Development, 2014. FY2014 Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001, Fourth Quarterly Report, 1 July – 30 September, 2014. Washington, DC: USAID, 2014. United States Agency for International Development, 2013. FY2013 Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001, Performance Management Plan, Annex, Revised December 18, 2013. Washington, DC: USAID, 2013. United States Agency for International Development, 2014. FY2014 Mozambique Media Strengthening Program, Cooperative Agreement 656-A-12-0001, Revised Performance Management Plan, October 2014. Washington, DC: USAID, 2014. United Nations Education and Scientific and Cultural Development Organization, 2011. International Program for the Development of Communication, Assessment of Media in Mozambique: Based on UNESCO’s Media Development Indicators, Paris, France: UNESCO, 2011. MOZAMBIQUE MEDIA STRENGTHENING PROGRAM DOCUMENTATION Media Strengthening Program, Data Quality Assessment Report, 2014. Maputo, Mozambique: MSP, 2014. Media Strengthening Program, Media Content Analysis Tool, Baseline Report. Maputo, Mozambique: MSP, 2014. Media Strengthening Program, Media Content Analysis Tool, October-December Report. Maputo, Mozambique: MSP, 2012. Media Strengthening Program, Media Content Analysis Tool, January-March 2013 Report. Maputo, Mozambique: MSP, 2013. Media Strengthening Program, Media Content Analysis Tool, April-September 2013 Report. Maputo, Mozambique: MSP, 2013. Media Strengthening Program, Media Content Analysis Tool, Original. Maputo, Mozambique: MSP, 2014. Media Strengthening Program, Media Content Analysis Tool, Revised, Maputo, Mozambique: MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 63 MSP, 2014. Media Strengthening Program, Media Content Analysis Tool, Categories 2013 Report. Maputo, Mozambique: MSP, 2013. Media Strengthening Program, Monitoring and Evaluation Plan. Maputo, Mozambique: MSP Media Strengthening Program, Mozambican. Media Sustainability Index, 2013, Maputo, Mozambique: MSP, 2013. Media Strengthening Program, Media News Flashes, Maputo, Mozambique: MSP, 2015. Media Strengthening Program, Work Plan, Year 2, Maputo, Mozambique: MSP, 2014. Media Strengthening Program, Work Plan, Year 3, Maputo, Mozambique: MSP, 2015. INDIVIDUALS INTERVIEWED THROUGH KIIS NAME INSTITUTION/POSITION Susan Jay Marcia Mantenja Mary Ellen Duke Sheila Zacarias Oquisso Ryan Raleigh USAID Senior Democracy, Human Rights and Governance Advisor USAID Program Development Specialist USAID Gender Advisor/Health USAID Development Program Specialist/ M&E USAID Project Development Officer Arild Drivdal IREX Program Director Ryan Morris Lola Martinez IREX Program Manager IREX Field Coordinator Helder Martins Jonas Mussa Tito Raimundo Leticia Carmo Multimedia Center and Community Radio of Monapo Multimedia Center and Community Radio of Ilha de Moçambique Radio Watana Radio Encontro Juvito Novela Henrique Silva Fatima da Costa Radio Eheli Community Radio of Maganja da Costa Community Radio of Cuamba Cantifula de Castro Augusto Madeira Pires Andre Marques Costa Abdul Paulo Radio Encontro Whampula Fax Radio Vida Nova Era newspaper Julio Paulino Assumane Tiago Frederico Ines da Silva Francisco Jornal A Verdade Nova Era (online magazine) Jornal Ikweli Helder Caetano Community Radio of Monapo Jonas Ali Mussa Hafir Jamu Community Radio of Ilha de Moçambique Djaffar Moussa-Elkhadum UNESCO Representative in Mozambique Nelson Rafael MISA representative in Nampula Elias Nabarraga Sitoi Lutxeque AJINA Network of Independent Journalists of Nampula Irene Mucule Director of jornal Debate MID-TERM PERFORMANCE EVALUATION OF THE USAID MEDIA STRENGTHENING PROGRAM (MSP)| 64 NAME INSTITUTION/POSITION Leonardo Costa ECO - TV Polly Gaster CAICC Selma Inocencia Arsenio Manhice Rui Lamarques Sergio Chusane IREX media specialist IREX media specialist IREX media specialist IREX media specialist, M&E Ricardo Mendes IREX adviser Anilda Manjate TIM journalist Joao Ribeiro TIM Director Tomas Vieira Mario Lecturer, former MISA and UNESCO Luis Nhachote Independent journalist Miguel Bila SAVANA, marketing and sales Fernando Lima SAVANA, CEO Lourenco Jossias Magazine Independent, director Augusto Uamusse IBIS, Program Officer Access to Information Faruko Sadique Radio Moçambique, CEO Matias Guente Canal de Moçambique, editor Olivia Massango STV, SOICO group Conceiçao Vitorino Jornal Catembe U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523