USAID/SERBIA FLICKR FINAL PERFORMANCE EVALUATION OF STRENGTHENING MEDIA SYSTEMS ACTIVITY December 2021 DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. This publication was produced at the request of the United States Agency for International Development. It was prepared independently by IMPAQ International, LLC, under Contract No. AID-167-I-17-00004, Task Order No. 72016921F00002. Implemented by: IMPAQ International 1325 G Street, NW Suite 900 Washington, DC 20005 ABSTRACT: In 2017, USAID/Serbia awarded the four-year, $6.5 million SMS Activity to IREX, Inc., to increase media independence and create a more competitive and self-sustaining media environment in Serbia. By enhancing the sustainability of Serbia’s independent media and strengthening the legislative, regulatory, and market environments, the Activity aims to strengthen media outlets’ capacity to provide citizens with quality and accurate information—empowering them to engage in informed citizenship and hold their government accountable. Leveraging the groundwork and lessons learned under USAID/Serbia’s media programming extending back to the 1990s, the Activity focuses on (1) identifying key regulatory and media market development issues to facilitate a dialog and cooperation between the media and related industries, and (2) supporting innovative business approaches, content monetization, development and implementation of new digital tools, and business networking and cooperation. In 2021, IMPAQ International (IMPAQ) was awarded a contract from USAID/Serbia to conduct a final performance evaluation of the SMS Activity. IMPAQ conducted a rigorous, utilization-focused evaluation that integrates quantitative and qualitative methods to provide USAID/Serbia with actionable recommendations to improve the efficiency and sustainability of its media programming moving forward. i | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV CONTENTS EXECUTIVE SUMMARY 1 INTRODUCTION 7 ACTIVITY BACKGROUND 7 EVALUATION PURPOSE 7 EVALUATION DESIGN 8 DATA COLLECTION 8 DATA ANALYSIS 9 KNOWN LIMITATIONS AND MITIGATION STRATEGIES 9 FINDINGS AND CONCLUSIONS 10 EVALUATION QUESTION 1 10 EVALUATION QUESTION 2 23 EVALUATION QUESTION 3 29 EVALUATION QUESTION 4 32 EVALUATION QUESTION 5 36 RECOMMENDATIONS 41 Annex A. EVALUATION STATEMENT OF WORK 43 Annex B. EVALUATION MATRIX 46 Annex C. DATA COLLECTION INSTRUMENTS 49 Annex D. ACTIVITY BACKGROUND 72 Annex E. DETAILED METHODOLOGY 74 Annex F. ANALYSIS OF KEY PERFORMANCE INDICATORS 82 Annex G. LIST OF DOCUMENTS REVIEWED 83 ii | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ABBREVIATIONS AND ACRONYMS AAS Aggregated advertising sales AMELP Activity Monitoring, Evaluation, and Learning Plan ASMEDI Association of Serbian Media BIRN Balkan Investigative Reporting Network BMAP Balkans Media Assistance Program CDCS Country Development Cooperation Strategy COP Chief of Party CSO Civil society organization EASA European Advertising Standards Alliance EQ Evaluation question EU European Union FGD Focus group discussion FY Fiscal year GESI Gender equity and social inclusion IREX International Research & Exchanges Board IT Information technology KII Key informant interview NAESA National Association for Ethical Standards in Advertising NGO Nongovernmental organization OSCE Organization for Security and Co-operation in Europe ROI Return on investment SMS Strengthening Media Systems SNA Social network analysis SOW Scope of work USAID United States Agency for International Development USG United States government iii | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID) engaged IMPAQ International, LLC (IMPAQ) to conduct a final performance evaluation of the Strengthening Media Systems (SMS) Activity in Serbia implemented by International Research and Exchanges Board, Inc. (IREX). The $6.5 million, five￾year SMS Activity started in September 2017 and was extended to February 2022. The purpose of SMS was to increase media independence and create a more competitive and self-sustaining media environment in Serbia so that Serbian citizens can be informed and hold their government accountable. To achieve its purpose, the Activity had two intermediate objectives: ● Support the development of a more enabling legal, regulatory, and market environment for media in Serbia ● Support the development of financial sustainability, innovation, and partnerships in the Serbian media sector The evaluation team assessed the extent to which SMS achieved intended results; identified key successes, challenges, and gaps; and documented good practices and lessons learned. The results of the evaluation will help USAID/Serbia, IREX, and other stakeholders involved in promoting media freedoms to design future actions in Serbia and elsewhere in the region. FINDINGS AND CONCLUSIONS EQ1: Has the Activity demonstrated results that are consistent with its theory of change, and to what degree has it achieved its stated objectives? SMS interventions enhanced most media partners’ income-generation strategies, which resulted in the adoption of innovative practices and business models and in new revenue streams. After receiving SMS assistance to implement new business models, most media partners successfully implemented one or more new revenue-generating activities. These produced new revenue to support partners’ activities, albeit with varying absolute and relative values. Most media partners reported that SMS assistance resulted in increases of 10 percent or less in annual revenue and in more diversified revenue streams. Existing data is not sufficient to measure the extent of these changes relative to media partners’ financial situations prior to receiving SMS assistance. The 15 grantees structured as firms reported substantial annual increases in sales and donations, suggesting that these firms were able to capitalize on SMS assistance to raise revenues to a greater extent than the nonprofits could. SMS grants also had a positive multiplier effect, with the value of new revenue exceeding the value of grants disbursed by 50 percent. Moreover, returns will likely increase over time, as some revenue streams are recurring. Even in the case of one-off donations, media partners developed capacity to implement new fundraising campaigns and therefore are likely to see increased revenue from such activities in the future. SMS was successful in helping media partners to generate new revenue using innovative business models and improvements to existing strategies. Crowdfunding/donation and memberships/subscription models, both of which rely mainly on online payment platforms, were innovative approaches for Serbia. These strategies proved successful when implemented by partners of varying sizes and capacity. Although relatively fewer media partners tried to generate revenue from complementary business activities, this strategy also resulted in revenue increases and is an approach that may be replicated with other outlets. Several grant partners were also successful in increasing revenue generated through advertising. However, many local media partners, especially local media, reported that they faced significant challenges in increasing their advertising revenues because of difficult advertising market conditions including the small size of the advertising market and large number of 1 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV media outlets vying for advertising income, unfair distribution of government grants for public interest media projects, and bias in allocation of advertising contracts by government and pro-government businesses in favor of state affiliated media. SMS found effective ways to improve independent media business strategies within Serbia’s current legal, regulatory, and market constraints. Through consultations, research, and direct support to media, SMS identified areas for practical improvements in the business environment. It piloted ways forward by facilitating access to e-commerce and donation management systems, proposing alternative advertising market measurements, and promoting ethical standards in advertising and personal data protection. SMS’s public awareness activity highlighted the value of supporting independent media and encouraged members of the public to make financial contributions to their media outlet of choice. This innovative effort demonstrated an area of untapped potential in Serbia’s media market. SMS effectively adapted its implementation strategy to comply with USAID’s post-award decision to focus SMS interventions primarily on reinforcing the financial sustainability of independent media outlets. However, the Activity Monitoring, Evaluation, and Learning Plan (AMELP) indicators were not sufficiently updated to reflect changes and measure results. SMS was effective in redirecting resources for planned interventions, whose purpose was to promote general legal and regulatory reforms, to focus on efforts to improve the business enabling environment for media and to pilot innovative business strategies with independent media partners. For the purposes of evaluation, to support SMS decision making, and to inform future activity design, it would have been useful to revise the SMS theory of change and associated objectives to reflect more explicitly how results were intended to be achieved and to articulate the key SMS assumptions and risk factors. Although the AMELP indicators were revised, they did not adequately measure how and why diverse interventions affected media partners’ financial sustainability. SMS’s practice of collecting data monthly on changes in grantees’ revenue and audience reach was very useful. Nevertheless, other data, including data on revenue changes, organizational changes, and gender and social inclusion metrics, was needed to establish financial and organizational benchmarks to enable fuller and more holistic assessment of changes in media partners’ financial sustainability. EQ2: To what extent did SMS identify, and address challenges and opportunities in the media sector in Serbia, in particular digitization, approaches to content monetization, innovative business models, and new ways to increase financial sustainability or independence media? SMS successfully tested innovative approaches to address challenges and seize opportunities related to digitization, content monetization, and business development. These approaches helped many partner media outlets to adapt to global trends and improve their revenue and audience outreach. The Activity helped media partners update their digital platforms, making them more competitive in Serbia’s media market. Notably, the Activity’s support for Podcast.rs enabled the expansion of this relatively new medium in Serbia and facilitated the arrival of fresh, independent voices to the county’s independent media scene. SMS assistance to enable its partners to strengthen their audience outreach and use data to better understand the needs of their audience inspired many media partners to reevaluate, reinforce, and capitalize on their loyal user community—an essential step forward for both mobilizing audience contributions and attracting advertising revenues. The SMS aggregated advertising sales (AAS) model attempted to overcome constraints created by volume-based advertising metrics that put smaller, typically local, independent media outlets at a disadvantage. This model was a creative way to tackle constraints related to audience size. Access to Serbia’s digital advertising revenues was a significant challenge for USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 2 most independent media outlets, given their relatively small audience and high levels of competition, together with reported political interference favoring state-affiliated media in the Serbian advertising market. Highlighting the value of audience loyalty using Smart Octo metrics was innovative for Serbia and potentially useful to attract advertisers. Although its recent first sale is promising, the AAS model still needs to be proven. Moreover, although SMS was successful in boosting its media partners’ competitiveness by enabling them to upgrade their digital platforms, further investments to improve partners’ digital platforms are needed as well. To improve sustainability, independent media partners also need mentoring and technical capacity building on content management, use of data analytics, and online audience engagement. EQ3: How well did SMS adapt to the changing operational environment, including incorporating adaptive management and continuous learning in planning and delivering its assistance? Adaptive management was evident in how SMS designed and delivered its technical assistance packages, including in the evolution of its assistance mechanisms to be more flexible and responsive to media partners’ needs. SMS successfully enabled partners to adopt new business models and practices. Media partners expressed high levels of satisfaction with SMS services despite contextual challenges, including the difficult political environment and COVID-19. These results are a positive testament to the SMS team’s management and the relevance of Activity assistance to partners’ needs. Moreover, the Activity’s willingness to start small, try new approaches, accept failure, and scale successful business models was critical to its success in promoting innovation. Evaluation findings and conclusions for EQ1 and EQ2 also highlighted examples of SMS adaptive management. For example, in its answer to EQ1, the evaluation team concluded that SMS responded effectively to USAID’s post-award directive to repurpose resources and retool strategies to focus on improving the business enabling environment for media and helping media partners to develop their financial sustainability. EQ1 and EQ2 findings and conclusions highlighted the Activity’s largely successful efforts to pilot innovative solutions in response to opportunities and challenges associated with digitization and content monetization; these too are examples of agile management. SMS assistance modalities did not sufficiently address some media partners’ needs as they transitioned to greater financial sustainability. Rather than using old assistance models, which provided subsidies for media partners’ general operating expenses such as rent, equipment and payroll, SMS focused on business development so that in the long run, media partners could cover these costs without external support. This approach was designed to reduce the dependence of independent media outlets on government and international donors funding and to increase their financial resilience. Nevertheless, media partners argued convincingly that future USAID activities should consider providing at least some short-term support for operational costs (mainly salaries) to partners that commit to developing their media in line with business principles. Providing at least some short-term support to help cover operational costs would help alleviate the resource burden that many media face and allow them to invest resources into testing and scaling innovative business models. Most importantly, the time and human resources required to test and scale new business models, such as crowdfunding campaigns, while continuing to produce high-quality content and maintain a loyal user base is a significant barrier to many media partners, particularly under-resourced investigative journalism and local media outlets. 3 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV EQ4: To what extent has SMS influenced changing approaches —both in the donor community and among beneficiaries —to media development in Serbia? SMS was successful in changing its partners’ negative perceptions of the feasibility of generating new revenue from innovative business models and in building media managers’ capacity to run their organizations as businesses. To varying degrees, partners embraced approaches such as online crowdfunding campaigns, subscription/membership models, and the use of data analytics to understand audience needs. SMS’s media outlet mentoring approach, which capitalized on the experience of media professionals who understood the Serbian media market, was effective in enabling media partners to acquire and practice new skills—more effective, according to respondents, than SMS group trainings. Moreover, the Activity fostered productive new business networks and partnerships among independent media, media associations, and media service providers; these networks also contributed to changing practices. Several international donors have changed their types and modes of media assistance in recent years. Some donors used approaches like that of SMS, aiming to improve media partners’ financial sustainability. The evaluation team did not find sufficient evidence to attribute observed changes to SMS rather than to other contextual and institutional factors. The European Union (EU), the largest donor for media support activities in Serbia, has been moving away from providing large project-style grants but continues to offer funding for a variety of purposes based on partners' needs, as do other smaller donors. SMS exerted influence on media assistance programs through information sharing and coordination with other USAID Activities as well as through its efforts to encourage donor coordination on activities to promote media freedom. These efforts by SMS and USAID effectively promoted complementarity and avoided duplication of effort, particularly within USAID’s media portfolio. EQ5: Based on the experience and knowledge from this Activity, what approaches are most likely to result in gender equity improvements in the area(s) of focus? SMS efforts to support media partners that respect or promote Gender Equity and Social Inclusion (GESI) values, including its support for the creation of the National Association for Ethical Standards in Advertising (NAESA), was an effective approach to address discriminatory cultural norms and promote greater gender equity. SMS built the capacity of media with gender-sensitive policies and practices, including some which reported regularly producing content promoting feminism and diversity. By building the capacity of these media partners, SMS boosted their financial sustainability and contributed to the continuity of their efforts and subsequently the sustained promotion of positive gender norms. Similarly, NAESA’s commitment to promoting ethical advertising, specifically plans to educate its members on how to ensure content is sensitive and does not perpetuate gender and ethnic minority stereotypes, has the potential to diminish prejudice and promote greater gender equity and social inclusion over time. Identifying and working with female media managers and experts to lead project activities and capacity building efforts helped elevate female voices and promote positive gender norms. As noted in the Organization for Security and Co-Operation in Europe’s (OSCE) gender analysis, despite making up a large percentage of the media workforce, women are underrepresented at the highest managerial levels in journalism. SMS purposefully selected female media managers and experts to lead workshops and mentor media partners. This approach helped promote positive female role models in the media sector and potentially opened new avenues for women-in-leadership. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 4 Targeting local and niche media outlets was an effective approach for reaching women-led and minority-language media. As noted in the USAID Gender Analysis Report, women leaders are more commonly found in local and niche media in Serbia, and minority-language media outlets are often located in communities with mixed ethnic group populations. Despite a lack of a GESI analysis and explicit targets and indicators, SMS’ was effective in reaching women-led and minority-language media by targeting local and niche media outlets during partner selection and ensuring that SMS technical assistance was tailored to the needs of those media. Specifically, tailoring its grant and technical assistance mechanisms to meet the diverse needs of different sizes and types of media organizations located across Serbia, contributed to its success in reaching women-led and minority-language media. While generating significant knowledge and learning on financial sustainability for independent media in Serbia, SMS did not identify specific business constraints, if any, facing women-led and minority media outlets or define approaches to overcome these challenges. SMS was largely noncompliant with USAID directives to conduct a GESI-sensitive needs assessments and measure results using GESI-sensitive indicators. Because of these gaps, SMS did not specifically test approaches or generate significant learning about approaches to improve gender equity in media business development and financial sustainability. RECOMMENDATIONS Based on the findings and conclusions, the evaluation team makes the following recommendations for future USAID activities to support independent media in Serbia. Build on SMS achievements in supporting the collective efforts of media outlets and media support organizations to improve the media business climate. ● Monitor the experience of media partners and media buyers or advertising agencies with the AAS model. Evaluate whether and how the model generates new advertising revenues for AAS media partners. If feasible, help to improve the model. If it proves successful, help scale and/or replicate the AAS model to include other media partners or new partnerships among other media organizations. ● Continue to foster wider acceptance of ethical standards in advertising, including standards related to gender equity and tolerance. Future support may include promoting partnerships between independent media outlets and multinational companies doing business in Serbia to pilot ethical advertising campaigns showcasing the firms’ commitment to responsible business values like tolerance, green development, and democratic values. ● Support advocacy and coalition building among media partners and others in Serbia’s business community to facilitate better regulation for e-commerce, increased access to innovative payment systems (fintech), and improvements related to other media financing issues. To promote transformative change, consider offering grants and technical assistance for financial management, marketing, and organizational development of media partners that meet minimum business readiness requirements. ● Even relatively large independent media outlets lack human resources with expertise in sales, marketing, and financial management. Provide short-term grant and technical assistance to enable these partners to hire new personnel as they restructure and to train and mentor staff in these areas. Anchor grant assistance packages, for example, could help meet this need. ● For niche and local media partners that are too small to absorb this kind of assistance, test collective solutions to enable them to implement innovative business ideas and access needed technical expertise. Solutions might include sharing part-time staff in areas like information 5 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV technology (IT), sales and marketing, or business development—building on the SMS sustainability hub approach. ● To reduce dependence on international donors, continue to condition grant or investment disbursements on meeting outcome benchmarks including increased revenue, diversified revenue, and increased audience reach. Ensure that prospective implementers conduct a GESI analysis during start-up and to include explicit GESI-responsive strategies in their implementation plan. ● Use explicit gender and social inclusion criteria when selecting partners, for example prioritize media led by women and ethnic minorities and/or that have organizational values that promote gender equality and non-discrimination. ● Conduct research to identify specific financial sustainability needs and opportunities facing women-led and ethnic minority media. ● Include GESI-sensitive indicators and disaggregate data by gender. ● Promote the capacity and leadership of women media managers. Further sensitize all media managers to principles of gender-sensitive management and journalism. Develop a monitoring and evaluation framework that measures business outcomes aligned to the Activity intervention strategy. ● Continue to follow SMS’s practice of tracking changes in revenue and audience size, as well as return on investment, to monitor media partners’ progress towards financial sustainability and to measure the efficiency of USAID support. In addition, establish and monitor indicators that measure changes in annual revenue, profitability (among for-profit media outlets), and the proportion of income generated through business activities and audience donations versus international donor contributions. ● Periodically assess changes in organizational capacity. Relevant organizational capacity indicators include changes in the numbers and capacity of media outlet personnel with marketing, sales, data analysis, and financial management skills and, for larger media partners, the extent to which business functions have been separated from editorial functions. ● Reduce reporting frequency and provide technical support for data collection, when feasible, to help media partners collect relevant financial and organizational capacity data. Many partners— particularly small, local, and niche media outlets—face human resource capacity constraints. Data collection and reporting support would improve the Activity’s access to quality data. This can be done through annual or quarterly surveys rather than monthly data collection. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 6 INTRODUCTION ACTIVITY BACKGROUND Despite having a constitution that guarantees freedom of the press, Serbia ranks 93rd out of 180 countries on the 2021 World Press Freedom Index compiled by Reporters without Borders1; it is 89th out of 161 countries in the 2019–2020 Global Expression Report2. Pressure from political and economic interests, threats to independent journalists, and out-of-date and unsustainable business models are among the challenges reported to hinder the ability of media outlets in Serbia to produce unbiased and timely information for the Serbian public. Understanding the challenges facing independent media in Serbia, the United States Agency for International Development (USAID) funded the Strengthening Media Systems (SMS) Activity. SMS, implemented by International Research and Exchanges Board, Inc. (IREX) was a five-year (2017–2022) activity with the purpose to foster a more competitive and self-sustaining media environment in Serbia so that Serbian citizens can be informed and hold their government accountable. To achieve this objective, the SMS Activity was structured around two intermediate objectives: (1) support the development of a more enabling legal, regulatory, and market environment for media in Serbia and (2) support the development of financial sustainability, innovation, and partnerships in the Serbian media sector. Based on discussions between USAID and IREX during the first year of SMS implementation (fiscal year 2017–2018), the Activity decided to shift activities under Objective 1 to focus more on supporting the development of the business enabling environment for media.3 Starting in 2019, USAID/Serbia funded the Organization for Security and Co-operation in Europe (OSCE) to promote broad legal and regulatory reforms as part of its activities to support the Media Strategy development process.4 Under Objective 2, SMS continued planned support to enable independent media outlets to develop innovative business approaches, including the design and implementation of new digital tools and the promotion of business networking and cooperation. The $6.5 million SMS Activity began in September 2017 with an anticipated end date of September 2021. Due to COVID-19–related delays, USAID/Serbia extended the implementation period until February 2022. More background on SMS is included in Annex E. EVALUATION PURPOSE USAID/Serbia commissioned IMPAQ International, LLC (IMPAQ) to conduct a final performance evaluation of the SMS Activity. This evaluation provides the Mission with a full, evidence-based, and independent assessment of the Activity’s performance, including an in-depth analysis of the extent to which SMS achieved intended results. The evaluation identifies key successes, unintended results, challenges, gaps, and contributing factors; it also documents good practices and lessons learned. 1 2021 World Press Freedom Index: Journalism, the vaccine against disinformation, blocked in more than 130 countries. Reporters Without Borders. https://rsf.org/en/2021-world-press-freedom-index-journalism-vaccine against-disinformation-blocked-more-130-countries ­ 2 The Global Expression Report 2019/2020: The state of freedom of expression around the world. ARTICLE 19. https://www.article19.org/wp-content/uploads/2020/10/GxR2019-20report.pdf 3 IREX SMS Year One Workplan. 2017. IREX. 4 IREX SMS Year Two Workplan. 2018. IREX. 7 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV In conducting the final performance evaluation, the team focused on the five evaluation questions (EQs) specified in the evaluation scope of work (SOW): 1. Has the Activity demonstrated results that are consistent with its theory of change, and to what degree has it achieved its stated objectives? 2. To what extent did SMS identify and address challenges and opportunities in the media sector in Serbia, in particular digitization, approaches to content monetization, innovative business models, and new ways to increase financial sustainability of independent media? 3. How well did SMS adapt to the changing operational environment, including incorporating adaptive management and continuous learning in planning and delivering its assistance? 4. To what extent has SMS influenced changing approaches—both in the donor community and among beneficiaries—to media development in Serbia? 5. Based on the experience and knowledge from this Activity, what approaches are most likely to result in gender equity improvements in the area(s) of focus? EVALUATION DESIGN To conduct the final performance evaluation of the SMS Activity, the evaluation team used a utilization￾focused approach, incorporating complementary qualitative and quantitative methods of data collection and analysis. Before collecting data, the evaluation team conducted a thorough review of documentation related to the EQs to gain an in-depth understanding of SMS activities and to assess the availability and quality of SMS monitoring and evaluation data to answer the EQs. DATA COLLECTION From September 18 through October 8, 2021, the evaluation collected data on the perspectives and experiences of key stakeholders involved in the SMS Activity. The evaluation team conducted key informant interviews (KIIs) and focus group discussions (FGDs) with a total of 59 participants, 26 female and 33 male. Key informants were selected based on their involvement with, knowledge of, and awareness of the Activity and of other media development efforts in Serbia. FGD participants were identified from a list of media partners that received SMS support. The evaluation team also delivered a short online survey to 52 current and former SMS grantees and partners. Due to COVID-19 restrictions, all data collection was conducted remotely by telephone and online conferencing technology. ● Key informant interviews. The evaluation team conducted online interviews with 31 individuals (11 female and 20 male) including representatives from (1) USAID, (2) the SMS Activity, (3) SMS media outlet partners and grantees, (4) other international donors supporting media development in Serbia, (5) market research firms, (6) the Government of Serbia, and (7) media experts. ● Focus group discussions. The evaluation team conducted seven online FGDs with a total of 28 participants (15 female and 13 male), each representing a media partner or grantee with which SMS worked. FGD respondents represented all geographic zones in which SMS operated. ● Online survey. The evaluation team delivered an online survey to 52 current and former SMS grantees or partners in the media sector. Twenty-five individuals (13 female, 10 male, and 2 undisclosed) completed the survey, for a response rate of 48 percent. Survey respondents included representatives from seven nongovernmental organizations (NGOs), five media associations, three national media outlets, nine regional or local media outlets, and one “other.” USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 8 For the detailed methodology, refer to Annex F. For the data collection protocols, refer to Annex C. DATA ANALYSIS The evaluation team employed several data analysis methods and approaches to identify key findings from the data, draw conclusions, and make recommendations. The evaluation team triangulated SMS grant monitoring data, including data on changes in media outlet grantees’ revenue, audience reach, and size, with qualitative data collected through document review, KIIs, FGDs, and the online survey. Evaluation team members independently reviewed evaluation notes and tallied occurrences of key themes, comparing data from the various sources and noting where sources converged or diverged. When data sources diverged in their findings, the evaluation team noted the differences and included them in this report. The evaluation team conducted content analysis by reviewing and coding KII and FGD data to identify and highlight notable examples of SMS’s success, or lack of success, that contributed to, or hindered, achievement of Activity objectives. Throughout the analysis, the evaluation team disaggregated all KII and FGD data by sex and analyzed for possible differences in respondents’ perspectives associated with gender. The team found no major differences. KNOWN LIMITATIONS AND MITIGATION STRATEGIES The evaluation team identified several challenges and limitations in conducting the evaluation. Cognitive bias. Key informants and FGD participants constitute a key source for answering all EQs, but interview data is prone to cognitive biases, including recall and social desirability bias. For example, USAID has conducted several activities in the media sector, some of which are ongoing; key informants might have confused one activity with another. To mitigate this risk, the evaluation team reminded participants of key SMS intervention strategies and probed answers to help with recall. Another possibility is that media partners avoided being critical of the Activity because of possible effects on future access to assistance. To mitigate this risk, the evaluation team introduced each KII and FGD by highlighting the confidentiality and anonymity of the discussion and asking participants to provide frank feedback. In addition, the evaluation survey was anonymous. Limitations in the Activity Monitoring, Evaluation and Learning Plan (AMELP). The AMELP SMS developed during its first year fit the Activity’s original implementation strategy.5 From Year 2 onwards, following its change in focus, SMS modified its AMELP indicators to align with changes in the implementation strategy—but with significant gaps, especially for Objective 1. The revised AMELP eliminated all outcome indicators under Objective 1, so that it did not clearly outline what changes in Serbia’s business environment SMS was expected to bring about or how these changes would be measured. To mitigate this limitation, the evaluation team relied on SMS progress reports and interviews 5 Under Objective 1, “Support the development of a more enabling legal, regulatory, and market environment for media in Serbia,” the original AMELP identified key results as follows: diverse media involved in policy dialogue, collaboratively designed reform agenda, evidence used to design and prioritize strategy and develop policy recommendations, development of effective advocacy campaigns, and the establishment of system(s) to monitor (aspects) of the media reform process with implementation and monitoring capacity developed among media system actors. Under Objective 2, “Support the development of financial sustainability, innovation, and partnerships in the Serbian media sector,” the SMS AMELP identified as the main expected results greater diversity of innovative content offered online to an expanded audience and sustainable and forward-looking media outlets with business plans that incorporate sustainable digital strategies and models to monetize content. 9 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV with Activity staff and other key informants to determine what SMS sought to accomplish under Objective 1 and what it actually accomplished. The revised AMELP expanded outcome indicators under Objective 2, a change that was consistent with the refocused implementation strategy.6 Although these indicators aligned with SMS interventions and measured some expected changes, the evaluation team found important gaps. For example, SMS defined increased financial sustainability as an increase in revenue without measuring the extent of the increase. Data on the extent of revenue diversification, which would have been useful to assess changes in donor dependence, was not systematically collected. Moreover, the AMELP did not include data on changes (or lack thereof) in SMS partners’ organizational structures; this information would have been useful to assess changes in media partners’ capacity to manage the new business models introduced by the Activity. Finally, little information was provided to assess the extent of gender equity and social inclusion (GESI) efforts. The evaluation team mitigated these constraints using qualitative data from KIIs, FGDs, quantitative grant monitoring data provided by SMS, and the survey. Incomplete and inconsistent data. SMS collected data on its grantees’ monthly changes in revenue from various income sources (advertising, subscriptions, donations) and in audience size (measured by page views, users, social media followers, listeners) and provided this data to the evaluation team7. However, the evaluation team found some inconsistencies in the data. For example, the types of data provided are not consistent across all SMS media partners, presumably because of variances in media partners’ activities and in the types of assistance provided by SMS. The data provided to the evaluation team aligns with grant periods that differ by grantee, with the earliest available data starting in April 2019 and the latest ending in August 2021. These differences, which were largely justified, affected data comparability. To mitigate this limitation, the evaluation team grouped media partners that reported on the same indicators before conducting its data analysis. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 Has the Activity demonstrated results that are consistent with its theory of change, and to what degree has it achieved its stated objectives? To frame the performance analysis in response to EQ1 and subsequent EQs, this report first describes SMS’s intervention strategy. It then provides an overview of SMS accomplishments, followed by an assessment of whether and how Activity interventions contributed to achievement of its two objectives. FINDINGS Changes in SMS intervention strategy. At the beginning of the Activity, SMS planned to facilitate media outlet and media association networking and consultations to identify needed legal and regulatory 6 Objective 2 indicators measured changes in media partners’ financial sustainability in terms of revenue increases, the rate of expansion of media partners’ online audience, media partners’ improved management practices or use of technologies, the percentage of for-profit media outlets that increased sales or donations, and the proportion of media partners that developed new business strategies. 7 The monthly grant data was provided by the SMS team on September 15, 2021. As such the data used in this report is based on data collected prior to September 15, 2021. Numbers reported in the Activity’s final report may differ from those in this evaluation report as the Activity’s period of performance was extended to February 2022. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 10 reforms, develop policy papers, and conduct related reform-oriented advocacy activities. Following changes in the operating environment and USAID’s decision to fund another activity, implemented by OSCE, that focused on media regulatory reform, SMS reoriented activities in Year 2 to focus primarily on Objective 2, strengthening the financial sustainability of independent media. It focused Objective 1 interventions on facilitating partnerships between and among media system actors to improve the business enabling environment. To do so, it facilitated access to e-payment platforms, promoted ethical standards in marketing, and encouraged use of common media audience measurement standards. These changes notwithstanding, SMS kept its original theory of change: If relationships within the media system are strengthened and the capacities of system actors are improved, the Serbian media system will be able to develop a more favorable enabling environment and adapt to the realities of the digital market, enabling diverse, objective, and professional news and information to survive and thrive in Serbia’s media market. According to SMS reporting and feedback from beneficiaries, the revised strategy had significantly less emphasis on strengthening relationships within the media system8 as a means of advocating for legal and regulatory reform. Still, SMS maintained its emphasis on capacity building for media system actors, focusing on building media outlets’ capacity to identify and implement new business models. SMS interventions were mainly directed towards supporting individual media partners to: ● Improve existing and develop new revenue streams ● Improve marketing and sales strategies ● Increase audience size and engagement In addition, SMS supported media support organizations—media associations, civil society organizations (CSOs), and private sector entities—to: ● Build the capacity of media outlets to implement new and strengthen existing business models ● Promote media market self-regulation ● Promote business innovation and support for media start-ups ● Implement joint efforts to improve the media market The SMS Chief of Party (COP) presented the logical relationship between, on the one hand, SMS interventions to strengthen and diversify media outlets’ revenue streams and, on the other, the Activity goal to help the independent media sector in Serbia survive and thrive, in a presentation written for an Activity-supported public advocacy campaign: And at the global level, every independent media must find new sources of income and be diversified. Part of that finance must come from the audience. To do that, the media must understand the market, their offer, use data and create business plans. This is the primary goal of the SMS project as well. We help independent media [in Serbia] improve their prospects for survival and growth.9 8 The SMS Activity design recognizes that the media system includes both media outlets and media support organizations, as well as other stakeholders including government, private sector, academia, CSOs, advertisers, start-ups, and research and development institutions. 9 Extracted from https://www.podrzimedije.rs/vesti/intervju-evana-trejsa-za-nedeljnik on September 3, 2021. 11 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV Exhibit 1 presents visually the SMS intervention strategy described above. Exhibit 1 SMS intervention strategy Overview of SMS assistance to media outlets and media support organizations. As of July 2021, SMS had awarded approximately $1,050,00010 in grants and provided technical assistance and mentoring to 44 diverse media outlets and 10 media support organizations. Media partners were geographically dispersed, located both inside and outside Serbia’s main cities. They used different mediums to reach their audience (print, TV, radio, web portals, podcasts) and featured content with diverse geographic scope (local, regional, national) and Exhibit 2 Distribution of SMS partners by media type 10 This figure is based on an internal document that SMS provided to the evaluation team. According to an SMS report, based on the SMS monthly grant tracker, approximately $856,000 in grants had been disbursed as of July 2021. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 12 thematic focus (news, investigative journalism, business and economy, sports). Media support organizations included national and regional professional associations and CSOs. Exhibit 2 provides information on the distribution of SMS grantees based on the type of media or organization. SMS selected media outlets and media support organization partners based on criteria outlined in its USAID-approved grants manual. According to the manual, a variety of organization types (CSOs, community-based organizations, private enterprises, professional associations, and universities) were eligible for grant assistance if the objectives of the grant proposal aligned with the Activity objectives and if the grantee accepted SMS technical assistance and mentoring. As a result, SMS had significant scope to award grants to organizations outside Serbia’s traditional media. SMS partners encompassed traditional media (print, radio, TV) as well as “new” and nontraditional media (web portals, podcasts). They also included a variety of media support organizations including professional associations of journalists and CSOs. Exhibit 3 Distribution of grants by size Most SMS partner organizations (41 out of 54 partners) received a grant from SMS, either directly or indirectly as a partner of one of SMS’s sustainability hub11 grant recipients. Grant sizes varied. SMS documentation12 lists three very large anchor grants of $75,000 to $100,000 to established independent media outlets: Danas, 021, and Juzne Vesti. As shown in Exhibit 3, SMS awarded 10 large grants of $20,000 to $50,000, 11 medium-sized grants of $10,000 to $20,000, and 17 small grants of less than $10,000. Detail on purposes and uses of grants is provided in the following subsections on SMS achievements by objective. Assessment of Project Performance under Objective 1 Under Objective 1, “support the development of a more enabling legal, regulatory, and market environment for media in Serbia,” SMS conducted research, facilitated consultations among media outlets and media support organizations on business enabling environment issues, and implemented targeted actions to improve the business environment for independent media. SMS conducted market-oriented research to identify market trends and areas where SMS support was needed. To identify business environment issues affecting independent media, SMS produced media market research and formed and facilitated working groups of media outlets and media support organizations. It then followed up with several targeted interventions to facilitate access to and use of e-payment systems, improve advertising market metrics, and strengthen media self-regulation, especially in advertising. According to SMS and media partner feedback, the market research provided useful information about the size and characteristics of the media market as well as information about audience behavior and demand. In Year 1, SMS also conducted a Political Economy Analysis and Social Network Analysis to 11 SMS gave grants to three partners—the Independent Journalists’ Association of Vojvodina, Impact Hub, and Podcast.rs—to provide support to local media and podcast producers. 12 Information reported here is based on the SMS monthly grant tracker. Reported grant sizes differed slightly depending on the document consulted. 13 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV identify challenges affecting independent media and to map key players within the media system. This analysis was mainly used internally, as it was in line with SMS’s original strategy to identify needed legal and regulatory reforms. Box 1 outlines SMS research products. Box 1 SMS Research Products The Political Economy Analysis assessed the media environment and media market; categorized key problems related to media business in Serbia; identified stakeholders, both institutional and noninstitutional, and their incentives, capacities, and power to deal with these problems; and outlined the socio-political forces affecting the media sector in Serbia. SMS developed a Legal and Regulatory Reform Roadmap based on the findings and inputs from key media stakeholders and legal media experts. The Social Network Analysis provided a comprehensive understanding of the relationships among media support organizations and CSOs working in the media reform space and identified existing collaborations, as well as gaps in collaboration that the SMS project could address strategically. The analysis was based on qualitative and quantitative data collected from 21 organizations: 10 media support organizations and 11 CSOs. The Strategic Media Market Trend Analysis provided information on the size and scope of the media market in Serbia, disaggregated by the main revenue streams. It identified the market’s main characteristics and trends and indicated the potential for future development. The analysis was based on primary data sources, as well as publicly available data from business registers and databases. The Demand-Side Study: Comprehensive Study on the Public’s Media Needs and Behaviors in Serbia was developed to help local, regional, and national media outlets strengthen their editorial and business decisions, both in their day-to-day operations and in their long-term strategic approaches. SMS-supported thematic working groups on shared business environment challenges produced mixed results in terms of coordinated advocacy efforts or changes in the business environment. However, the Activity identified needs it could address through assistance to individual media and business partnerships. Starting in Year 2, SMS facilitated thematic working groups, which it called “collaborative nodes,” around specific challenges related to the business environment. Three groups were formed to discuss (1) regulation of e-payments as a basis for business models in digital environments; (2) standardization of a measurement system for the media industry, with a special focus on local media, to produce reliable metrics that are valuable and relevant for advertisers; and (3) compliance challenges surrounding regulatory and legal provisions that are critical for media operations, especially regulations related to new user privacy standards. SMS followed up on initiatives identified in the working groups, albeit with somewhat limited results. ● Although media associations planned to conduct joint advocacy on business enabling environment issues, prospective partners’ initial enthusiasm fizzled with the onset of COVID-19, which changed their priorities.13 ● SMS relayed media partners’ concerns related to e-commerce regulations and support services provided by banks and payment processors to other USAID Activities dealing with broader business climate issues in Serbia. In addition, under Objective 2, SMS provided technical assistance to individual media outlets and media associations to establish e-payment gateways. ● The Activity granted $18,000 to the Association of Serbian Media (ASMEDI) to promote the standardization of online media audience measurement metrics in Serbia. Promotion was done by exploring the feasibility of developing a Joint Industry Committee (JIC) model in Serbia and by initiating cross-sectoral cooperation among media support organizations, media buying and research agencies, publishers, and advertisers. Among its grant-supported activities, ASMEDI 13 During COVID-19, the Government of Serbia imposed new controls on media. These restrictions were the subject of protest by media organizations. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 14 organized a national conference on media measurement. According to SMS management, efforts to develop the joint industry model and to reform and harmonize media measurement practices at the industry level were ultimately unsuccessful because of opposition from media operators with vested interests in the current system. However, under Objective 2, at the media outlet level, SMS supported media partners to improve their use of available tools to measure audience size and behavior and suggested ways they could use these measurements to market their ad space more effectively. ● SMS supported an aggregated advertising sales model to help overcome the unfavorable advertising market for small, local independent media outlets. Pooling small and medium independent media outlets’ advertising space enabled a “one-stop shop” for potential advertisers. A major setback occurred when SMS had to drop one of its three anchor grantees.14 SMS reported that it made its first sale in September 2021. This model is discussed in more detail under EQ2. ● In partnership with the SHARE Foundation, SMS provided technical support to media outlets on how to implement personal data protection regulations.15 SMS produced a journalists’ guide and an informational video. With the Association of Online Media and SHARE, SMS organized a workshop on personal data protection regulations that attracted 83 participants from 56 media outlets. SMS support was instrumental in the creation of the National Association for Ethical Standards in Advertising (NAESA), but the organization has yet to become fully operational. According to Activity documentation and KIIs, NAESA is an independent nongovernmental nonprofit association mandated to define ethical standards in advertising and marketing communications and to enable enforcement. With SMS support, NAESA was founded in 2019. In 2020, it became a member of the European Alliance for Standardization in Advertising (EASA).16 According to SMS program managers, one of the Activity’s objectives for NAESA was to foster a more favorable advertising market for independent media by distinguishing outlets that uphold international standards in advertising. In addition, NAESA was involved in SMS activities to harmonize media audience measurement. Key informants said that NAESA was recognized by OSCE and the Serbian government as an important SMS initiative. One government representative said, “There is a need to dedicate time to make this initiative operational and to have financial resources for their work. We need this type of self-regulatory body that would play an important role in the sector of audiovisual policy.” Many media partners were also supportive of the initiative. For example, 16 out of 25 survey respondents agreed (n = 11) or strongly agreed (n = 5) that NAESA is a useful initiative to promote self-regulation in media advertising in Serbia. However, in surveys and FGDs, media operators said that they perceived little movement from NAESA since its creation. According to KIIs, COVID-19 restrictions and illness within the executive team delayed planned activities, but NAESA expects to accelerate actions soon. 14 SMS dropped Danas, Serbia’s largest independent daily newspaper, from the model when it was bought by United Group and became ineligible for SMS assistance. 15 In November 2018, Serbia adopted a new law on personal data protection that is modelled after the EU General Data Protection Regulation. 16 EASA is a standard-setting body that brings together 34 advertising self-regulatory organizations and 16 organizations representing the advertising industry (advertisers, agencies, and media outlets). According to its website, EASA helps to make sure that ads are legal, decent, honest, truthful, prepared with a sense of social responsibility, and created with due respect for the rules of fair competition. 15 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV SMS supported a public advocacy/awareness-raising campaign with larger-than-expected reach. The campaign contributed to desired changes in audience attitudes and practices. In 2020, SMS implemented an advocacy and awareness raising campaign under the banner “Independent media depend on you” (“Nezavisni mediji zavise od tebe” in Serbian).17 Its goal was to educate and encourage Serbian citizens to support their preferred independent media outlets through donations and subscriptions; the campaign also aimed to change negative attitudes towards online payments. According to SMS’s Year 4 Q1 progress report, more than 50 national, regional, and local media outlets and media support organizations published information about the campaign. The campaign reached an estimated six million people via TV, websites, social media, Google display network ads, display banners, and other means. The campaign thus exceeded its expected reach by 42 percent. Research commissioned by SMS before and after the campaign shows it met some of its objectives. In a 2020 survey of Serbian citizens, 38 percent of respondents considered independent media to be an important source of information, and 18 per cent were ready to support independent media outlets financially—seven percentage points more than in 2019.18 The same research showed that citizens would be willing to set aside an average of 866 dinars per month ($8.50) for media support and that their preferred modalities were donations (43 percent) and subscriptions (24 percent). The research also highlighted increased public openness to using online payment systems. SMS managers attributed this shift primarily to changes induced by the COVID-19 pandemic. The research reported that 43 percent of respondents chose online payment as the preferred model of financial support, compared to 18 percent in 2019. SMS largely met its Objective 1 AMELP targets; however, the indicators tracked by the AMELP were not very useful to measure performance. SMS reporting shows that the Activity met or exceeded its output targets for Objective 1 in Year 3 (it had no outcome targets). In Year 4, it narrowly missed its target on the number of media support organizations assisted. According to SMS staff reports and the evaluation team’s analysis, the output indicators, which referred in a generic way to numbers of media organizations involved in advocacy and capacity-building activities, were not very useful to measure SMS achievements. For example, there were no indicators that measured changes in the number of advocacy or capacity-building activities pertaining to the business enabling environment issues and capacity areas on which SMS decided to focus. In addition, there were no outcome indicators to measure the extent of actual changes related to these issues and areas. Weaknesses in its Objective 1 AMELP indicators notwithstanding, SMS collected some useful data to measure outcomes. For example, the pre- and post-public advocacy campaign research yielded useful data to measure campaign outcomes. For an analysis of SMS performance against Objective 1 baseline and targets, refer to Annex G. Assessment of Project Performance under Objective 2 To “support the development of financial sustainability, innovation, and partnerships in the Serbian media sector”, SMS provided technical assistance, mentoring, and grants to media outlet and media support organization partners. According to SMS and media partner accounts, SMS tailored its interventions to each partner’s needs and opportunities, using a mix of intervention strategies. Exhibit 4 provides a snapshot of the diverse approaches and business models that SMS supported to increase its media partners’ revenues and expand their audience base. Most of the strategies involved innovation, most notably, e-payment gateways, crowdfunding, podcasts, and new partnerships with other outlets or 17 The campaign’s website is https://www.podrzimedije.rs/vesti/intervju-evana-trejsa-za-nedeljnik 18 In 2019, 79 percent of citizens answered that they would not financially support the work of independent media outlets. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 16 media service providers, particularly organizations specialized in crowdfunding campaigns and media analytics as well as SMS sustainability hub partners. Exhibit 4 SMS strategies to support media partners’ financial sustainability Create new or improve existing revenue streams Promoting audience subscriptions and donations ✔ E-payment gateways ✔ Crowdfunding campaigns Introducing new products ✔ Newsletters ✔ Podcasts ✔ Content in ethnic minority languages Increasing advertising revenue ✔ Strengthened sales strategies ✔ Aggregated advertising model ✔ Optimized banner and other ad spaces Increasing revenue from complementary businesses ✔ Restructured management teams ✔ Development of new services and products Expand media partners’ audiences Improving content delivery ✔ Better media web portals ✔ New mobile applications or optimization of web content for mobile Increasing audience size and loyalty ✔ Effective use of social media to drive users to media websites ✔ Effective use of standard web analytics ✔ Use of Content Insights/Smart Octo, a specialized media analytics tool, to facilitate media outlets’ understanding of audience needs and habits and of advertising sales strategies After receiving SMS assistance to implement new business models, most media partners generated new revenue, although the reported value varied widely. SMS monitoring data show that 27 grantees (one media association and 26 out of 44 media partners) generated new revenue through SMS assistance, with the average increase equal to $40,000. New revenue values were unevenly distributed by media partners, as can be seen in Exhibit 5. Exhibit 5 Value of new revenue by media partner 17 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV For most media partners, SMS assistance resulted in increases in annual revenue and more diversified revenue streams. Available data on the absolute value of reported new revenue does not provide sufficient information to assess the size of the gain relative to grantees’ annual revenues or the effect on the diversification of revenue streams. According to AMELP indicator reporting, the 15 SMS partners that were structured as business firms (versus nonprofit organizations) increased their annual sales or donations from baseline by 47 percent in Year 3 and 54 percent in Year 4. To complement this information and fill information gaps on changes for nonprofit grantees, the grantee survey asked about the effect of SMS support on media partners’ revenue. Out of 25 respondents, 12 reported an increase in annual revenue of 10 percent or less, six reported an increase between 11 and 50 percent, and one reported a large increase of more than 50 percent, as shown in Exhibit 6. In addition, eight partners reported they had diversified their revenues, three reported becoming less dependent on international donors, and three reported no change. Exhibit 6 Effects of SMS assistance on grantee revenue reported by survey respondents The evaluation team had limited information to assess the extent to which partners diversified their revenue or became less donor dependent. Anecdotally, one FGD participant reported modest changes in their outlet’s reliance on donor-funded project financing. In 2020, 65 percent of this outlet’s income came from business activities and 35 percent from donor-funded projects. By comparison, in 2019, a much larger proportion of annual revenue—57 percent—came from donor-funded projects. Exhibit 7 ROI by grant size Taken together, SMS partners generated revenue equal to roughly 150 percent of the value of the grants disbursed, with very large and small grants providing the greatest return on investment (ROI). Using SMS grant monitoring data, the evaluation team examined how much grantees earned to date per grant dollar as a measure of ROI.19 On average, looking only at grantees that reported new revenue generated with SMS support, every grant dollar generated about $1.50 in revenue. Many of the new revenue streams reported by grantees, while varying in value from month to month, are recurring, so it is reasonable to expect that ROI will continue to grow over time. Looking at ROI by grant size, as in Exhibit 7, the evaluation team found ROI was highest, at 204 percent, for anchor grants awarded to larger, older media outlets; the large mix of interventions for these grantees included support for organizational restructuring. Next were small grantees that received less than $10,000. Their average ROI was 144 percent. Many small grants were innovation-oriented, focused on funding, for example, e-portals, podcast development, or crowdfunding campaigns. ROI analysis suggests, and SMS program managers and media partners affirmed, that some relatively simple, low-cost interventions 19 ROI was not an AMELP indicator, but SMS began looking at income generated relative to grant value in Years 3 and 4 when it noticed that some relatively small grants contributed to relatively substantial revenue. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 18 were effective at generating new revenues. However, more substantial, long-term, and complex interventions were critical to transforming the business models of larger traditional media outlets. SMS was effective in developing innovative business models and improving existing income generation strategies. The evaluation survey asked grantees to identify the three income-generating strategies supported by SMS that they perceived to have been the most effective in helping create sustainable revenue for their organization.20 Among the 25 respondents, the top choices were alternative revenue streams (n = 14), donation/crowdfunding model (n = 13), e-payment gateway (n = 12), advertising (n = 9), and subscription/membership model (n = 8). These choices reflect a mix of existing and innovative revenue streams. Many media partners piloted more than one income generation strategy with their grant. SMS assistance helped several media partners enhance their revenues from complementary business activities. The survey defined “alternative revenue streams” to include revenue from services and from complementary businesses. In the minds of survey respondents, services may have included project-based revenue streams traditionally funded by international donors. As will be discussed under EQ4, donors have been moving away from this funding modality. Media partners reported successful outcomes of SMS efforts to strengthen their complementary business activities: ● 021, a regional media outlet that uses income from its café to finance its media activities, increased café monthly revenues by 50 percent within a few months of receiving SMS assistance. Over the 20 months from November 2019 to June 2021—including some months with slow or no activity due to COVID-19 restrictions—021 generated nearly $63,000 from its café business. ● Nova Ekonomija, an online business magazine, used SMS support to launch a business intelligence service that enabled the outlet to land new contracts and to establish a partnership with a UK risk analysis firm. ● Juzne Vesti, a digital media outlet operating in Serbia’s third-largest city, Nis, financed a portion of its media activities using revenue from Simplicity, an IT/online business solutions company of which the media outlet is a subsidiary. SMS helped Simplicity improve the IT business so that, between August 2020 and June 2021, it generated $167,000. The crowdfunding/donation and memberships/subscription models, both of which rely on online payment platforms, were among SMS’s most innovative and successful strategies. The crowdfunding/donation, membership/subscription, and e-payment gateway models all rely on voluntary contributions from media users. Media fundraising campaigns were operated either through Catalyst’s Serbia21 crowdfunding platform (https://www.donacije.rs/) or by installing a donation page with an e-payment gateway directly on the media partner’s website. According to key informants across all stakeholder groups, these models were innovative, not only because they relied on e-commerce technology that was still relatively underdeveloped in Serbia, but also because few media outlets had ever tried raising funds from their user community. For example, one key informant reported, “Looking to your own constituents or your own audience for financial support is not native to the thinking of civil society [in Serbia] that has been 95 percent funded from the outside.” The SMS intervention resulted in a paradigm shift, remarked by several FGD participants. One media partner said, “Thanks to the SMS 20 The choices were e-payment gateway, subscription or membership model, donation/crowdfunding model, advertising, Google AdSense optimization, Patreon membership model, alternative revenue streams (revenue from services, side businesses), and none of the above. 21 To implement the crowdfunding strategy, SMS worked with Catalyst, a regional NGO specializing in helping CSOs in the Balkans to mobilize philanthropic contributions to support their activities. 19 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV program, we realized that we have a community.…This was an opportunity to find out who our readers and supporters are, to talk to them and to build mutual trust.” Crowdfunding/donation model Exhibit 8 SMS media partners' revenue from donations . Data on revenue generated through individual donations or crowdfunding was available for 10 SMS media partners in the SMS monthly grantee tracker data, illustrated in Exhibit 8. Donation revenue generated by these 10 organizations equaled an average of $8,683 or $548 per month. Given the novelty of the approach, the baseline for most partners was zero. According to data from five SMS media partners, the average number of donations was 682 and the average size of donation was $28.22 FGD participants said that the most successful fundraising campaigns were implemented by media outlets with a loyal audience base, including both mature outlets like Danas and emerging outlets like the Agelast podcast. Also successful were outlets that were perceived to be under attack for covering government corruption or organized crime, including investigative journalism outlets KRIK, Pescanik, and Voice. Donation data shows that audience contributions were an irregular source of income, but they were not inconsequential in terms of the total value of revenues generated. For example: ● Pescanik raised $50,000 from January 2020 to October 2020 through an online crowdfunding campaign, achieving a nearly 300 percent ROI on its SMS grant. The campaign revenues reportedly covered nearly half of Pescanik’s annual operational costs. ● With a grant of $3,500, KRIK generated about $6,000 in donations during a four-month campaign. KRIK created a new crowdfunding staff position, which it continued to fund after SMS support ended. ● TV Forum in Prijepolje, the first female-led TV station in the Balkans, generated $5,400 from an online crowdfunding campaign to help replace the studio’s broadcasting and production equipment. Membership/subscription model. The membership/subscription model relied on audience or member contributions, mainly paid online using an e-payment gateway, in exchange for content or services. This revenue was more often recurring than donations were, because consumers committed to pay at regular intervals for the content or services. According to SMS monthly grant tracker data, seven media 22 Data on the number of donations was not available for all media organizations that reported donations. This figure is calculated by taking the total value of donations divided by the total number of donations, using only data from partners that reported both. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 20 partners implemented the model, generating, on average, $1,068 in monthly revenue.23 To generate membership/subscription revenue, SMS helped some partners develop new products and enabled others to collect payment for existing services. Both modes met with mixed success. ● Danas, a national daily newspaper, generated recurring revenue through its Readers’ Club, a new model that gave members access to exclusive content and other benefits, such as discounts on advertisers’ products. Between January 2020 and February 2021, Danas generated over $70,000 from club members, with over 500 members on average. ● Fonet, a Serbian wire service, developed The Balance Sheet, a newsletter focused on the economy, and made it available for subscription. According to available data, this newsletter did not attract subscribers in large numbers. ● The Journalist Association of Serbia created an e-payment gateway on its website to collect dues from members. This strategy was reported as largely unsuccessful; few members paid their dues this way, and the processing fees were relatively high. Exhibit 9 Changes in SMS anchor grantee advertising revenues SMS helped media partners increase advertising revenues by restructuring sales teams and testing new marketing strategies. In addition to supporting new income generation strategies, SMS also helped media partners to improve existing advertising revenues. Grantee monthly tracker data show that 10 grantees reported revenue from ads. SMS’s three anchor grant partners and a few other grant recipients received assistance to restructure their ad sales strategies and thereby realized new revenue. Exhibit 9 shows that the changes in the three anchor grantees’ advertising revenues were unevenly distributed over time and among the three media partners. SMS also reported that some grantees increased advertising revenues by optimizing banners and other ad space and increasing audience numbers. In addition to conventional online advertising, SMS grantees experimented with a Groupon model24 with Ozon. This tactic was unsuccessful, largely because COVID-19 shut down many businesses typically interested in this type of marketing. SMS also piloted a sponsored content approach with Boom 93, which reported earning just over $7,000 between January and August 2021. SMS support for new income generation activities laid a foundation for reducing donor dependence and increasing financial sustainability, but most media partners are still in the early stages of changing how they finance their businesses. SMS media partners varied in their perceptions of the extent to which SMS positively contributed to their financial sustainability. Nearly all media respondents agreed that their organizations needed to develop their business strategies. One media representative affirmed, “The best result of the SMS program was that everybody had to start 23 Media that reported data on revenue from subscriptions were Danas, Talas, the Journalist Association of Serbia, NUNS, Liceulice, Loznicke, and Nova Ekonomija. 24 This model was based on selling ads to local businesses including coupons giving special deals or cost reductions to coupon holders. 21 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV seriously thinking about business development.” Nevertheless, most SMS media partners perceived that, although revenue from SMS-supported business models represented progress towards sustainability, it would take time for these new revenue streams to scale. A survey respondent wrote, “My impression is that all projects supported by SMS are fragile, and I am afraid that our market ‘seeks’ longer-term support towards self-sustainability.” A few respondents emphasized that SMS revenue streams were complementary to, but not yet a sufficient substitute for, international donor funds. Thirteen grantee survey respondents ranked lack of access to donor funding in the top four challenges affecting media outlets’ ability to survive financially; this ranking suggests that respondents continue to rely significantly on donors. SMS met or surpassed all its Objective 2 outcome and output targets in Years 3 and 4. 25 According to SMS reporting, the Activity exceeded its targets for the number of media outlets assisted and the percentage of those outlets that increased their revenue, expanded their online audience, and developed new business models to monetize content. For the 15 partners registered as for-profit companies, SMS reporting shows an increase in annual donations or sales because of SMS assistance that was more than three times greater than predicted. AMELP indicators provide limited information on the extent to which these changes in individual media organizations contributed to improving their financial viability or reducing donor dependence. For example, the indicators do not track whether the increases in revenue, audience size, consumer donations, and sales were small, medium, or large relative to total operating costs and annual revenue or the extent to which the proportion of revenues from business activities versus international donor funds changed. For an analysis of SMS performance against Objective 2 baseline and targets, refer to Annex G. CONCLUSIONS SMS interventions enhanced most media partners’ income generation strategies, which resulted in the adoption of innovative practices and business models and in new revenue streams. After receiving SMS assistance to implement new business models, most media partners successfully implemented one or more new revenue-generating strategies. These produced new revenue to support partners’ activities, albeit with varying absolute and relative values. Most media partners reported that SMS assistance resulted in increases of 10 percent or less in annual revenue and in more diversified revenue streams. Existing data is not sufficient to measure the extent of these changes relative to media partners’ financial situations prior to receiving SMS assistance. The 15 grantees structured as firms reported substantial annual increases in sales and consumer contributions, suggesting that these firms were able to capitalize on SMS assistance to raise revenues to a greater extent than the nonprofits could. SMS grants also had a positive multiplier effect, with the value of new revenue exceeding the value of grants disbursed by 50 percent. Moreover, returns will likely increase over time, as some revenue streams are recurring. Even in the case of one-off donations, media partners developed capacity to implement new fundraising campaigns and therefore are likely to see increased revenue from such activities in the future. SMS was successful in helping media partners to generate new revenue using innovative business models and improvements to existing strategies. Crowdfunding/donation and memberships/subscription models, both of which rely mainly on online payment platforms, were innovative approaches for Serbia. These strategies proved to be successful when implemented by partners of varying sizes and capacity. Although relatively fewer media partners tried to generate revenue from complementary business activities, this strategy also resulted in revenue increases and is an approach that may be replicated with other outlets. Several grant partners were also successful in increasing revenue generated through advertising. However, many media partners, especially local 25 SMS redeveloped its indicators in Year 2 and began tracking them in Year 3. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 22 media, reported that they faced significant challenges in increasing their advertising revenues because of the difficult advertising market conditions including the small size of the advertising market and large number of media outlets vying for advertising income, unfair distribution of government grants for public interest media projects,26 and bias in allocation of advertising contracts by government and pro￾government businesses in favor of state affiliated media. SMS found effective ways to improve independent media business strategies within Serbia’s current legal, regulatory, and market constraints. Through consultations, research, and direct support to media, SMS identified areas for practical improvements in the business environment. It piloted ways forward by facilitating access to e-commerce and donation management systems, proposing alternative advertising market measurements, and promoting ethical standards in advertising and personal data protection. SMS’s public awareness activity highlighted the value of supporting independent media and encouraged members of the public to make financial contributions to their media outlet of choice. This innovative effort demonstrated an area of untapped potential in Serbia’s media market. SMS effectively adapted its implementation strategy to comply with USAID’s post-award decision to focus SMS interventions primarily on reinforcing the financial sustainability of independent media outlets. However, the AMELP indicators were not sufficiently updated to reflect changes and measure results. SMS was effective in redirecting resources for planned interventions, whose purpose was to promote general legal and regulatory reforms, to focus on efforts to improve the business enabling environment for media and to pilot innovative business strategies with independent media partners. For the purposes of evaluation, to support SMS decision making, and to inform future activity design, it would have been useful to revise the SMS theory of change and associated objectives to reflect more explicitly how results were intended to be achieved and to articulate the key SMS assumptions and risk factors. Although the AMELP indicators were revised, they did not adequately measure how and why diverse interventions affected media partners’ financial sustainability. SMS’s practice of collecting data monthly on changes in grantees’ revenue and audience reach was very useful. Nevertheless, other data was needed to establish financial and organizational benchmarks to enable fuller and more holistic assessment of changes in media partners’ financial sustainability. EVALUATION QUESTION 2 To what extent did SMS identify and address challenges and opportunities in the media sector in Serbia, in particular digitization, approaches to content monetization, innovative business models, and new ways to increase financial sustainability of independent media? To answer EQ2, the evaluation team examined what various media experts, independent media managers, USAID, and SMS Activity staff perceived as opportunities and challenges in the media market that helped and/or hindered SMS progress, particularly related to digitization, content monetization, and business development in general. This section assesses the strategies SMS implemented to seize these opportunities and overcome challenges, focusing on the extent to which these strategies were effective. 26 Funding by local, provincial, and state governments is available for media projects in the public interest. Serbian law requires that funding be allocated through a transparent selection process. 23 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV FINDINGS Digitization Independent media in Serbia have been slow to adapt to the media revolution, despite available technology and increasing demand for online content. SMS staff, media consultants, and FGD participants reported that former state-owned and corporate media in Serbia have the resources and access to expertise necessary to update their digital platforms. In contrast, at the inception of SMS, most independent media did not have sufficient access to financial resources or specialized technical and business expertise; they were five to ten years behind in their digitization strategies. Nevertheless, media experts said that digitization offered opportunities for newcomers to enter Serbia’s media market, allowing them to capitalize on the relatively low barriers to entry in the digital media era. Additionally, the online media audience in Serbia is gradually growing, particularly among the younger population. While television remains the most used media source, a quarter of the Serbian population uses online media, including news portals, online news magazines, and blogs, to stay informed.27 SMS digitization support, a core component of most SMS assistance packages regardless of the type of media, helped partners improve their online presence and online audience outreach. SMS helped independent media outlets in Serbia to overcome their digitization challenges and capitalize on opportunities to reach both existing and new online audiences. SMS grants and technical support helped partners with website improvements, mobile applications, podcast development, and social media marketing strategies. SMS reported that relatively basic changes, such as optimizing websites for mobile devices, enhancing online advertising space, and paying more attention to social media communities, contributed to substantial improvements in page views and increased basic advertising revenues from Google AdSense. Several SMS media partners expressed appreciation for the Activity’s digitization support. One FGD participant reported, “SMS support helped to strengthen our [digital] infrastructure. Very few of us are capable of modernizing our internal infrastructure [with our own resources] to make our production more competitive.” SMS support to the podcast community in Serbia was a particularly innovative and successful intervention capitalizing on media digitization opportunities. According to stakeholder feedback, SMS took advantage of this emerging medium to build a “whole new ecosystem” for Serbian podcasters. Before SMS, podcasters were few and, because they were spread out across different platforms, hard to find. One of SMS’s grantees reported that, after the grant, “Many of the local independent media houses as well as hobby podcasters finally had a base and someone they could reach out to get help and production tips.” In its final grant report, Podcast.rs owner noted that user traffic on the website had multiplied sevenfold over one quarter, with up to 19,000 users on the platform. Box 2 provides details on some of SMS’s reported digitization success stories. Box 2 Media Partner Success Stories Podcast.rs is an NGO specialized in development of the Serbian podcasting scene, with activities extending into the Western Balkan region. In partnership with Impact Hub, a start-up incubator with activities in Serbia, SMS provided technical capacity building to podcast producers; developed the first podcast aggregator, Podcast.rs; and supported an audience outreach campaign. Today Podcast.rs hosts 239 podcasts and covers Serbia and the region. SMS identified the potential for developing a podcast platform by organizing a “hackathon,” an approach commonly used to promote entrepreneur-driven innovative start-ups, especially in IT. Through its partnership with Impact Hub, SMS successfully adapted the approach and organized a competition for new media models that resulted in 27 IREX Media Sustainability Index, 2019. https://www.irex.org/sites/default/files/pdf/media-sustainability-index europe-eurasia-2019-serbia.pdf ­ USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 24 the development of Podcast.rs. Innovative approaches such as the hackathon leveraged new and creative ideas and facilitated the arrival of fresh, independent voices to the country’s independence media scene. With SMS support to rebuild its web portal, Danas became one of the top 15 most-read websites in Serbia, averaging more than 20 million monthly page views in 2020. In addition, it launched one of the most-listened-to news podcasts in the region. SMS helped FONET, a news service, to rebuild its digital platform and strengthen its digital sales activities, expanding its client base from traditional wire service media users to individual users and other clients. Digital sales generated over $70,000 between January 2020 and February 2021. Agelast was one of the first podcasts to receive an SMS grant. During the grant period, the podcast recorded a 3,606 percent increase in listeners (from about 300,000 to 14.1 million YouTube views) and generated $52,000 in advertising revenue and donations. Further infrastructure investments are needed, as well as mentoring and technical capacity building on content management, data analytics, and online audience engagement, to address media partners’ enduring challenges. Despite progress made on digitization, according to media managers and experts, sustained progress on digitization is constrained by independent media partners’ limited access to financial resources and technical expertise. Although USAID and other donors provide support in this area, SMS stakeholders and media partners reported that further support is needed. Content Monetization and Innovative Business Models Independent media outlets in Serbia face both the same content monetization challenges that have forced traditional media worldwide to reevaluate their business models for digital media services and constraints specific to the Serbian market. SMS staff and media experts reported that traditional independent media outlets in Serbia need to replace old content monetization models, many of which have become obsolete with digitization. At the same time, all Serbian independent media outlets have struggled to find effective ways to compete with social media platforms,28 search engines, international online media sites, and international and national telecom and content providers, all of which have amassed large online audiences by understanding and appealing to changing audiences, new kinds of content, and consumer needs and behaviors.29 With their large online media audiences, these platforms attract the lion’s share of Serbia’s growing online advertising market.30 Background documents, including research produced by press freedom watchdog groups and commissioned by Western donors, highlighted challenges specific to Serbia’s media market. These include the small size of the advertising market and large number of media outlets vying for advertising 28 According to a 2020 survey conducted by the Pew Research Center, about half of US adults (53 percent) say they get their news from social media “often” or “sometimes.” See https://www.pewresearch.org/journalism/2021/01/12/news-use-across-social-media-platforms-in-2020/. 29 According to media experts, most of Serbia’s digital advertising market is reserved for two kinds of players: Big Tech (e.g., Facebook and Google) and large corporate publishers that have huge traffic and wide user bases along with long-lasting partnerships with media buying agencies. 30 Within the entire digital advertising market, the biggest growth in 2020 was recorded by mobile advertising (45.9 percent) and advertising on social networks (33 percent), according to Gemius Explorer statistics from June 2021. 25 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV income,31 unfair distribution of government grants for public interest media projects,32 and bias in allocation of advertising contracts by government and pro-government businesses in favor of state affiliated media. For years the state has been the biggest advertiser in the country, through its ministries, public companies, and state agencies, buying power that it has reportedly used to exert strong pressure on advertising agencies to favor state-affiliated media.33 In addition, KIIs noted that the COVID-19 pandemic worsened the economic situation of Serbian media outlets, which were hit by a significant reduction in advertising revenues. Despite these obstacles, some positive trends support new content monetization strategies. Serbia’s digital advertising market expanded more than sixfold in the last decade; in 2020, it increased by 19.5 percent, reaching €56.22 million (about $65.2 million) in value.34 COVID-19 brought changes in consumers’ habits, including increased openness to e-commerce and use of online payment systems, the foundation for many online content monetization strategies. In 2020, compared to 2019, the number of e-commerce transactions increased by 104 percent, and the overall value of transactions increased by 81 percent, according to the National Bank of Serbia.35 Although media experts highlighted that consumer spending power in Serbia is lower than in Western Europe, SMS market research showed that a growing, although still small, proportion of media users are willing to pay for content.36 SMS helped many media partners generate revenue using online payment systems, although the process was more difficult and time-consuming than anticipated. As highlighted in EQ1 findings, SMS provided media partners with financial and technical assistance to implement donation/crowdfunding and membership/subscription models. These SMS-supported business models require e-payment gateways. SMS and media partner staff reported e-payment challenges due to technical issues and consumer habits. Furthermore, the country’s conservative banking system prohibits 31 There are 2,636 media outlets registered with the Agency for Business Registers, according to the 2019 Media Sustainability Index produced by IREX (https://www.irex.org/sites/default/files/pdf/media-sustainability-index￾europe-eurasia-2019-serbia.pdf). This number is difficult to sustain in a country of 7 million people; it represents one media outlet per 2,700 citizens. According to the SMS Strategic Media Market Analysis, the total value of Serbia’s media market is $478 million. The bulk of revenue (43 percent) comes from advertising ($205.9 million), followed by income generated through other sources and services provided by media outlets, which combined are equal to 26 percent or $124.3 million. The funding for two public broadcasters represents a significant portion of the media market, accounting for $99.1 million (or 21 percent), while state funding accounts for $28.5 million (or 6 percent). Media organizations with the highest revenues operate in the capital and other big cities. 32 Funding by local, provincial, and state governments is available for media projects in the public interest. Serbian law requires that funding be allocated through a transparent selection process. 33 In September 2021, ten media companies, including public service broadcaster Radio-Television of Serbia, all television stations with a national frequency, the country’s largest news agency, Tanjug, and the highest-circulating pro-government tabloid newspapers, are now marked on Twitter social network with the tag “Serbian state￾affiliated media”. State-affiliated media is defined as outlets where the state exercises control over editorial content through financial resources, direct or indirect political pressures, and/or control over production and distribution. https://www.slavkocuruvijafondacija.rs/en/twitter-labels-10-media-outlets-in-serbia-as-being-under-state￾control/The 2021 European Commission Serbia Report notes that political and economic influence over the media continues to be a source of concern. It notes needed reforms, foreseen in Serbia’s media strategy to address fairness in financing media content and advertising from state resources. https://ec.europa.eu/commission/presscorner/detail/en/ip_21_5275. 34 https://serbia-business.eu/the-digital-market-of-serbia-has-increased-more-than-six-times-in-one-decade/. 35 https://ecommserbia.org/2021/03/03/astronomski-rast-elektronske-trgovine-u-srbiji/ 36 The SMS IPSOS research showed that citizens would be willing to set aside an average of 866 dinars per month ($8.50) for media support and that their preferred modalities were donations (43 percent) and subscriptions (24 percent). USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 26 transactions in foreign currency and is resistant to serving new clients with low transaction volumes, such as small media organizations. The Activity overcame many of these challenges by providing technical support to media organizations and by working with banks and payment processing service providers to overcome their resistance to offering services to small media outlets. Although SMS was successful in setting up e-payment gateways with at least eight partners, the process was reported to be more difficult and time-consuming than anticipated. One SMS media partner recounted, “In the US, Canada, or Western Europe, it would take three days max to implement the payment solution and then start earning money. It took us nine months, because of compliance issues and because the whole thing is new to the banks, the web developers, and the payment processors.” SMS media partners reinforced their audience outreach strategies with SMS support for enhanced use of social media and data-driven decision making. SMS staff and several media managers indicated that, prior to SMS interventions, many media outlets spent little time trying to understand their audience needs, tailoring their content to demand, or building a community of users. Many FGD participants reported that SMS assistance helped them use social media more effectively to drive users to their websites; they also understood their audience’s needs better. In many, but not all, cases, these improvements resulted in large increases in the media partner’s audience. According to SMS AMELP indicator reporting, in Year 3, 94 percent of participants in SMS technical assistance activities expanded their online audience; in Year 4, it was 100 percent.37 According to SMS monthly grant data on page views, 22 media partners reported that on average their audience increased month-to-month due to SMS assistance. Among these partners, the average monthly rate of increase in page views was 13 percent. Five partners reported negative trends in the average monthly change in number of page views during their grant period. The average monthly change, including both positive and negative trends in reported page views, was a 9 percent increase. Exhibit 10 shows the average monthly change in page views reported by SMS media partners.38 Exhibit 10 Average monthly change in page views reported by SMS media partners 37 The indicator defined online audience expansion in terms of the number of users and page views. 38 The increase is measured in page views. SMS grantees reported on other audience outreach indicators including the numbers of users, social media followers, and podcast listeners. 27 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV SMS helped media partners use media data analytics to better understand their audiences and to adapt their production and content management approaches accordingly. These steps led to growth in audience size and loyalty. According to SMS staff, media experts, and some media partners, few SMS media partners had previously used analytical tools—either basic metrics from Google Analytics or more advanced media analytics—to systematically track and understand audience interests, behaviors, and preferences. Through a partnership with Smart Octo, formerly known as Content Insights, SMS provided access to a media metrics tool and mentoring to track audience interaction with online media content. For example, the tool shows how long users spend reading a piece of content, as well as other metrics. The partnership with Smart Octo generated $180,000 in cost share, according to SMS reports. Media partners said that SMS assistance contributed to changes in the way they used data and resulted in improvements in their audience outreach strategies. To increase small media partners’ access to advertising revenues, SMS implemented an aggregated advertising sales (AAS) model, but it is too soon to tell whether this model will be successful in improving local media advertising revenues. Building on its work with Smart Octo, SMS developed an AAS model to boost media partners’ access to the digital advertising market. This model attempted to overcome constraints in the advertising market that put smaller, typically local, independent media outlets at a disadvantage, as volume-based advertising metrics favor media with large numbers of page impressions and users. The initiative involves seven media partners from across Serbia39 that “have agreed to work together to offer tailored native advertising content to brands that seek to penetrate local markets.” The AAS approach uses a common technology platform to pool network member page views and achieve wider reach. In addition, the model capitalizes on Smart Octo audience engagement metrics to measure depth of audience loyalty and trust, a metric on which small media outlets with close connections to their audiences can compete with larger outlets. According to SMS accounts, the AAS model offers potential advertisers access to an audience of approximately 11 million viewers, which is competitive with a medium-size web portal like Radio Television of Serbia but is still far smaller than the audiences of Serbia’s biggest online media outlets.40 With agreement from the network of media partners, in September 2021, SMS successfully negotiated a deal with Motorola, the first of its kind, to pilot a tailored native advertising campaign for the company worth $800. CONCLUSIONS SMS successfully tested innovative approaches to address challenges and seize opportunities related to digitization, content monetization, and business development. These approaches helped many partner media outlets to adapt to global trends and improve their revenue and audience outreach. The Activity helped media partners update their digital platforms, making them more competitive in Serbia’s media market. Notably, the Activity’s support for Podcast.rs enabled the expansion of this relatively new medium in Serbia and facilitated the arrival of fresh, independent voices to the county’s independent media scene. SMS assistance to enable its partners to strengthen their audience outreach and use data to better understand the needs of their audience inspired many media partners to reevaluate, reinforce, and capitalize on their loyal user community—an essential step forward for both mobilizing audience contributions and attracting advertising revenues. 39 021 Novi Sad, Južne vesti Niš, Ozon Press Čačak, Glas Šumadije Kragujevac, Glas Zaječara, Boom93 Požarevac, Bujanovačke. 40 Based on SMS reports, the industry standard digital auditor Gemius shows that Blic has more than a third of total domestic internet users (968,000), while Kurir has over 13 percent of total monthly traffic on a national level (187,412,279 page views). USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 28 The SMS AAS model attempted to overcome constraints created by volume-based advertising metrics that put smaller, typically local, independent media outlets at a disadvantage. This model was a creative way to tackle constraints related to audience size. Access to Serbia’s digital advertising revenues was a significant challenge for most independent media outlets, given their relatively small audience and high levels of competition, together with reported political interference favoring state-affiliated media in the Serbian advertising market. Highlighting the value of audience loyalty using Smart Octo metrics was innovative for Serbia and potentially useful to attract advertisers. Although its recent first sale is promising, the AAS model still needs to be proven. Moreover, although SMS was successful in boosting its media partners’ competitiveness by enabling them to upgrade their digital platforms, further investments to improve partners’ digital platforms are needed as well. To improve sustainability, independent media partners also need mentoring and technical capacity building on content management, use of data analytics, and online audience engagement. EVALUATION QUESTION 3 How well did SMS adapt to the changing operational environment, including incorporating adaptive management and continuous learning in planning and delivering its assistance? EQ1 and EQ2 analyses assessed, among other things, the Activity’s response to national and industry challenges and opportunities facing independent media in Serbia. To answer EQ3, the evaluation team examined SMS staff perceptions of the practical operational challenges they experienced in delivering the program, as well as media partners’ perspectives on the relevance and effectiveness of SMS assistance. The evaluation team then explored to what extent and in what ways SMS effectively incorporated adaptive management and used lessons learned to improve its assistance over the life of the Activity. FINDINGS Operational Challenges In addition to the challenges related to digitization, content monetization, and business development, evaluation data shows SMS faced a variety of operational challenges that affected its results. SMS media partners’ limited capacity to absorb assistance was a fundamental implementation challenge. Notably, SMS staff, media experts, and implementing partners emphasized the need for adaptive management in order to support media partners that had very different needs and internal capacities to absorb SMS assistance. SMS staff named the following issues: ● Partners’ lack of buy-in for the SMS financial sustainability goal. SMS staff reported that many media managers did not believe that media activities should be financed using business models and were skeptical about the feasibility of developing new revenue streams in Serbia given the difficult political and economic context. Because of this skepticism, many potential media partners were not initially interested in or not fully committed to the support SMS provided. ● Partners’ management capacity, especially business development capacity. Managers of small, local, and niche media outlets, many of which operate with two or three full-time staff, were especially vocal about this challenge. For example, one participant said that, even if SMS were to “bring in the Director General of the BBC to offer advice,” most small media outlets would not be able to absorb the assistance because they are too busy with day-to-day activities. SMS staff also noted that the lack of business expertise among media partners made it challenging to collaborate and implement new business ideas. One staff member indicated, “Most of the 29 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV leaders of the media that we work with are journalists, editors, or editors-in-chief. They're not business folks.” ● Diversity of media and media partners’ needs. The Activity served a large variety of media partners who worked in different mediums, were dispersed geographically, and had very different levels of human resource capacity. With this diversity came a wide variety of needs. The Activity had to tailor its assistance and develop and monitor a large variety of assistance mechanisms—all of which was a challenge for a relatively small management team. ● Matching media partners’ needs and expectations with available expertise. SMS reported challenges finding consultants and service providers with the right mix of skills, media sector business expertise, and experience in Serbia. Media Partner Feedback on SMS Assistance Overall, media partner feedback on program assistance was positive. One FGD participant reported, “The people running the SMS program were very supportive. [They were] always ready to listen and adjust their approach according to our needs.” Survey responses to four questions about grantee satisfaction with SMS grants, technical assistance, events, and consultation meetings were largely positive. On average, 19 survey respondents out of 25 indicated they were satisfied or extremely satisfied with SMS assistance, with little divergence by type of assistance. Only 2.5 respondents, on average, indicated they were dissatisfied. Improvements suggested in KIIs, FGDs, and survey responses converged around the scope, size, and duration of assistance. Many grantees expressed strong opinions that assistance for business development needed to be complemented with other kinds of support that SMS did not provide, most notably support for content production and operational expenses. Several media partners commented that they were currently able to monetize their content through consumer donation and subscription models because they had previously received support from international donors to develop high-quality content for their users. Except for support for the development of new media products (newsletters, content in minority languages, and podcasts), SMS did not provide grants for content production. These same media managers noted that international donor support for content production was dwindling and that income from other sources was not sufficient to replace it. One media partner said, “Our message to every donor from now on is when they work on [business] sustainability, they also need to have a core support or content production component. Because if you're in the media business, you need to have content for your readers.” In addition, local and niche media outlet managers were adamant about the need for operational support to pay staff salaries. Media and media experts highlighted that the financial crisis brought on by COVID￾19 increased financial strain on many independent media outlets. SMS staff and media partners reported that SMS supported media staff salaries in a few cases, but on a small scale, mainly by paying consultant fees for pilot initiatives. Only a few grants, primarily anchor grants, included significant assistance for restructuring media operations, including short-term support to finance new staff positions. Several SMS media partners said that SMS assistance packages (grants, technical assistance, and mentoring) were too short relative to the expected results and available human resources. Most SMS grants had relatively short implementation periods—six months to a year. In addition, according to USAID and SMS staff, to incentivize results, the Activity increasingly used a pay-for-performance (P4P) approach, which conditioned grant disbursements on the achievement of key benchmarks such as new revenue generation or audience growth. Whether because of the P4P modality or grant timelines, many grantees reported feeling challenged to produce quick results from new revenue-generating strategies. However, a few said that they were given extra time and additional resources when they needed them. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 30 A few grantees indicated the grant sizes were too modest. For example, one FGD participant reported, “Regarding grant support, six months is very short to develop any business idea. Also, for local media, financial support could not surpass half our annual revenue, which was a limitation.” Another media partner reported that the SMS grant, while useful, was “not enough funding to go beyond the basic level.” Based on grant data, SMS provided follow-on grants to some media partners to consolidate initial successes and to address identified needs. Examples of Adaptive Management SMS adjusted its grant modalities during the life of the Activity so that its assistance adapted more closely to partners’ needs. To meet the media partners’ diverse needs in light of their varying capacity to absorb assistance, SMS modified its grant mechanisms and developed assistance packages tailored to partners’ needs. According to one SMS staff member, “We definitely tried almost every shape and size of a mechanism that we could.” Starting in FY 2018–19, SMS used an annual program statement as its main grantmaking mechanism. Potential grantees now had some leeway in terms of the types of assistance they could request as well as more flexibility regarding when and how they could submit grant requests. Nevertheless, the scope of grants remained focused on business development, in line with IREX’s contract with USAID. Because not all media outlets had the capacity to develop detailed proposals, SMS reported that it asked for concept notes to begin the grant process. After validating concept notes, SMS helped partners develop more detailed action plans. In addition, SMS also developed rapid-response mechanisms that allowed it to react relatively quickly to provide needed technical assistance to partners, with or without a grant award. SMS scaled its assistance to its partners’ capacity, providing longer and larger assistance packages to the few relatively developed media outlets in Serbia through anchor grants. Later, it also offered larger grants to “small anchor” partners that had the potential to absorb business development assistance. SMS considered partner feedback when selecting service providers and consultants. Overall, most FGD participants highlighted their appreciation for SMS personnel, consultants, and service providers. Nevertheless, SMS staff and media partners reported several cases of dissatisfaction with consultants and other service providers. In response, SMS terminated agreements or did not use those consultants or service providers again. For example, a SMS staff member reported that the Activity discontinued its contract with an early implementing partner after receiving negative feedback from media partners. According to SMS, the implementing partner, a technology company, “was unable to make the case to media outlets on the importance of technology” because it “did not understand the context and approach of media outlets and could not speak their language.” SMS staff and media partners also reported that the Activity changed the experts assigned to some grantees based on grantee feedback; in some cases, SMS allowed grantees to identify and choose their own consultants. SMS used short, experimental pilots to test innovative business ideas. SMS and media partners reported that SMS promoted experimentation in the form of short, limited-scale pilot projects. Not all approaches were successful.41 Various SMS, USAID, and media partner respondents highlighted that early, relatively quick wins generated success stories that helped overcome partner skepticism and facilitated buy-in for new business models, which were then replicated with other partners. 41 For example, SMS supported Vreme, a weekly magazine, to develop a podcast dedicated to youth. The podcast was discontinued when funding ended because it was not effective in reaching the target audience. SMS also supported BIRN to create an online learning platform to serve as a knowledge base for independent media; SMS reported that this effort was unsuccessful in attracting users. 31 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV CONCLUSIONS Adaptive management was evident in how SMS designed and delivered its technical assistance packages, including in the evolution of its assistance mechanisms to be more flexible and responsive to media partners’ needs. SMS successfully enabled partners to adopt new business models and practices. Media partners expressed high levels of satisfaction with SMS services despite contextual challenges, including the difficult political environment and COVID-19. These results are a positive testament to the SMS team’s management and the relevance of Activity assistance to partners’ needs. Moreover, the Activity’s willingness to start small, try new approaches, accept failure, and scale successful business models was critical to its success in promoting innovation. Evaluation findings and conclusions for EQ1 and EQ2 also highlighted examples of SMS adaptive management. For example, in its answer to EQ1, the evaluation team concluded that SMS responded effectively to USAID’s post-award directive to repurpose resources and retool strategies to focus on improving the business enabling environment for media and helping media partners to develop their financial sustainability. EQ1 and EQ2 findings and conclusions highlighted the Activity’s largely successful efforts to pilot innovative solutions in response to opportunities and challenges associated with digitization and content monetization; these too are examples of agile management. SMS assistance modalities did not sufficiently address some media partners’ needs as they transitioned to greater financial sustainability. Rather than using old assistance models, which provided subsidies for media partners’ general operating expenses such as rent, equipment and payroll, SMS focused on business development so that in the long run, media partners could cover these costs without external support. This approach was designed to reduce the dependence of independent media outlets on government and international donors funding and to increase their financial resilience. Nevertheless, media partners argued convincingly that future USAID activities should consider providing at least some short-term support for operational costs (mainly salaries) to partners that commit to developing their media in line with business principles. Providing at least some short-term support to help cover operational costs would help alleviate the resource burden that many media face and allow them to invest resources into testing and scaling innovative business models. Most importantly, the time and human resources required to test and scale new business models, such as crowdfunding campaigns, while continuing to produce high-quality content and maintain a loyal user base is a significant barrier to many media partners, particularly under-resourced investigative journalism and local media outlets. EVALUATION QUESTION 4 To what extent has SMS influenced changing approaches —both in the donor community and among beneficiaries —to media development in Serbia? To answer EQ4, the evaluation team examined to what extent and how SMS assistance influenced media partners’ attitudes, knowledge, and practices. In addition, the team examined the extent to which SMS influenced other donors’ practices by looking at trends in donor assistance and the reported reasons behind these trends. Finally, this subsection examines SMS efforts to coordinate and share information with other donors and USAID Activities. FINDINGS SMS Influence on Media Partners As reported in EQ3 findings, one of the obstacles facing SMS staff was limited awareness among independent media of the need to generate revenue from their activities using business models. In USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 32 addition, SMS staff and media experts said many media partners were initially highly skeptical that revenue generation strategies could be successful in the prevailing political and market context. Many were particularly doubtful of the feasibility of mobilizing audience donations or subscriptions, given the population's low purchasing power and the proliferation of “free” content on social media and other online media sites. Early successes of SMS partners with new business models and advice from SMS’s experienced consultants helped to change media partners’ perceptions regarding the validity of new business models over the life of the Activity. According to both SMS and its media partners, the early successes of SMS partners and advice from SMS’s experienced consultant media mentors were influential in changing media partners’ mindsets and moving them to try new business strategies. In the words of one SMS staff member, “Once the money came in [from implementing membership or online contribution approaches], then those challenges that I highlighted about mindset and culture, while they were still there, became less constraining because we had evidence there was money and a new audience to be picked up.” According to SMS AMELP indicator reporting, nearly 100 percent of participants in SMS activities developed new business strategies. This shift was corroborated by the evaluation survey, which showed that 18 out of 25 survey respondents agreed or strongly agreed with the statement, “To survive and progress, independent media outlets need to be operated as a business.” Media partners reported that SMS media experts, many of whom were experienced media professionals with international experience, were influential in changing partners’ practices because these consultants provided practical advice based on their experiences. Most SMS media partners gained capacity to strengthen their business operations and developed new business partnerships and networks with SMS assistance. Eighteen out of 25 survey respondents agreed or strongly agreed with the statement, “Through SMS assistance, I have gained skills and capacity to strengthen my media outlet’s business operations.” In FGDs, media partners reported they had acquired new competencies in key areas such as data analytics, fundraising campaigns, and audience engagement. One FGD participant reported, “Community management, crowdfunding, and other skills were developed within this project, which wouldn't have been otherwise, at least not in time.” Media partners highlighted that customized mentoring from SMS consultants was more effective in helping them gain new skills than group trainings were. SMS and its implementing partners emphasized that some media partners needed additional assistance to address emerging needs to strengthen their business and financial management capabilities. For example, SMS staff noted that some media outlets’ financial accounting systems needed to be strengthened to support new business models. Media partners noted that they needed additional training and support on marketing strategies and use of data analytics. 18 out of 25 survey respondents agreed or strongly agreed with the statement, “Through SMS assistance, I have gained skills and capacity to strengthen my media outlet’s business operations.” One media partner affirmed during FGDs, “Community management, crowdfunding, and other skills were developed within this project, which wouldn’t have been otherwise, at least not in time.” In addition to gaining new skills, many media outlet and media association managers benefited from SMS-facilitated business partnerships and networks. Eighteen out of 25 evaluation survey respondents agreed or strongly agreed with the statement, “SMS helped my media outlet or association develop new business partnerships or networks.” FGD participants reported that 33 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV SMS-sponsored events, including its summer school and town hall meetings,42 were useful for networking. Media partners also highlighted positive collaboration with private media support firms, including Smart Octo and Catalyst, as well as business partnerships such as the AAS model. For example, one FGD participant reported that cooperation with Smart Octo “was the most important partnership. I can’t think of any other donor support that we received that was as effective.” One SMS approach that was largely successful to create new business and support networks among its media partners was sustainability hub grants. Successful examples include the partnerships with Impact Hub and Podcast.rs, which, as highlighted under EQ2, contributed to creating an active network of Serbian podcasters. In addition, SMS provided a grant to the Independent Journalists’ Association of Vojvodina to provide support to local media in the region. Several local media outlets reported that this connection was, in the words of one respondent, “a breakthrough cooperation.” Trends in Donor Assistance for Media Freedom in Serbia and in Donor Coordination Activities International donor support for media activities has decreased overall since the fall of the Milosevic regime. In recent years, direct support for media moved away from large subsidies for content production to more diverse forms of assistance. SMS’s Strategic Media Market Trends Analysis provided useful analysis on trends in donor support. This analysis and media partners’ reports confirm that foreign donors are fewer in number and less active in the media sector than in the years during and immediately after the Milosevic regime. The EU is the biggest donor in Serbia; since 2002, it has invested €42 million ($47.2 million) in the media sector through direct grants to media outlets and activities to update the legal framework and to restructure Serbia’s public broadcasting service. Smaller media support activities are funded by foreign governments including the US, UK, Sweden, Switzerland, and Norway, as well as by a few foundations: Open Society Foundation, National Endowment for Democracy, and Olaf Palme Centre, among others. Foundation funding largely focused on empowering media associations and media outlets to promote and strengthen free, critical, and analytical journalism as well as to produce TV programs and digital content, particularly investigative journalism. SMS’s media market analysis showed that USAID is currently the second-largest donor in the media sector. USAID/Serbia funding for SMS followed a previous IREX-implemented activity, the Serbia Media Assistance Program, which ended in 2012. This program had three objectives: (1) strengthening the legal and regulatory environment for media, (2) increasing the managerial and financial capacity of independent media outlets, and (3) enhancing the quality and availability of information. During the SMS implementation period, USAID/Serbia’s Support to Media Reforms Activity, implemented by OSCE, was ongoing; it mainly focused on the creation of a sustainable regulatory environment for media development. SMS also overlapped with a $10 million USAID regional activity, the Balkans Media Assistance Program (BMAP), which is being implemented by FHI 360 and Internews in five countries in the Balkan region, including Serbia. Like SMS, this regional Activity aims to reinforce the financial sustainability of media outlets through support for digitization, audience outreach, and networking. SMS’s strong focus on developing new business models diverged from past donor programs, but more recent programs have taken a similar approach. One SMS staff member said, “SMS is different from a lot of other donor support, which was mostly premised on giving survival grants…. The SMS project was premised on not doing that and trying to get into the organizational and 42 SMS summer school was a three-day workshop that brought together many SMS partners to discuss new business ideas. SMS reported that it organized town hall meetings to share the results of its research activities, in particular the Demand-side Study. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 34 business issues.” SMS and USAID reported that at least two donor-supported programs outside USAID, one ongoing and the other still in its design stages, had drawn extensively on SMS approaches and lessons learned.43 The ongoing program is the British Council’s Media for All, which launched in September 2019. In partnership with the Thompson Foundation and the Balkan Investigative Reporting Network (BIRN), this program provides tailored business development and other support to independent media outlets.44 The EU recently made changes in its assistance programs for media outlets in Serbia based on its assessment of media needs post–COVID-19. For many years, the EU has supported independent media with long-term grants for investigative reporting and creative content production in the areas of rule of law, environmental protection, cultural diversity, media and digital literacy, good governance, human rights and minority protection, freedom of expression, and regional reconciliation. These were mainly project grants directly supporting the production of journalistic content and associated operating costs such as salaries, rent, equipment, and other overhead costs needed to achieve the media’s core mission to inform the public. In September 2020, the EU allocated €2.4 million to be disbursed over 36 months to help media outlets to cope with the consequences of the COVID-19 pandemic, including losses in advertising revenue and media restrictions. This new program, managed by the European Endowment for Democracy, offers support in the form of institutional grants for a wide range of media partner needs. According to an EU representative, the change in media support modalities enabled the agency to more effectively reach local media outlets, which generally lacked sufficient capacity to qualify for the project-oriented grants that dominated past EU support. The new approach also helped the EU to avoid perceptions of political bias that arose because the content whose production its grants supported was sometimes critical of the Serbian government. Still in the development stage, the EU is preparing a regional media support program, potentially to be implemented by German Agency for International Cooperation , which will include a component on the economic sustainability of independent media. USAID and SMS led coordination efforts among donors and other USAID Activities with shared objectives. SMS and USAID staff reported that SMS staff actively led or participated in coordination activities involving other donors working in the media sector. For example, in late 2018, SMS organized a meeting to relaunch coordination and knowledge sharing among donors and organizations involved in supporting media activities in Serbia. Following the 2018 meeting, SMS staff reported, the group held periodic coordination meetings, which SMS used to solicit feedback and data from donors on their work with key media partners. According to one SMS staff member, the information SMS garnered in this way “explains the apparent shift in donor approaches with regard to media, and also facilitated broader openness to SMS initiatives among media partners, as they could no longer easily pitch and repitch the same concepts to multiple donors.” To coordinate among USAID activities, SMS staff and the managers of the USAID-supported OCSE Activity reported that they regularly shared information. An SMS staff member indicated that, based on information SMS shared with OSCE, strategies to address business-related concerns raised by SMS partners were integrated into Serbia’s New Media Strategy Action Plan. OSCE staff also reported that SMS media partner consultations, especially those organized during the first two years of SMS, were extremely useful for OSCE’s efforts to identify independent media outlets’ concerns and raise them with the Serbian government. SMS also coordinated closely with the regional BMAP Activity manager for 43 Respondents did not name the donor for the program in the design stage because of an ongoing procurement process. 44 The program also has gender and content support components to “improve gender equality in the workplace and enable [media] to produce more quality, relevant, gender sensitive content.” See https://kosovo.britishcouncil.org/en/programmes/society/media-for-all. 35 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV Serbia. SMS and BMAP collaborated on several projects, dividing responsibilities for different elements. In addition, SMS and BMAP each sat on the other’s grant selection committee so that they were well informed about the other’s activities. CONCLUSIONS SMS was successful in changing its partners’ negative perceptions of the feasibility of generating new revenue from innovative business models and in building media managers’ capacity to run their organizations as businesses. To varying degrees, partners embraced approaches such as online crowdfunding campaigns, subscription/membership models, and the use of data analytics to understand audience needs. SMS’s media outlet mentoring approach, which capitalized on the experience of media professionals who understood the Serbian media market, was effective in enabling media partners to acquire and practice new skills—more effective, according to respondents, than SMS group trainings were. Moreover, the Activity fostered productive new business networks and partnerships among independent media, media associations, and media service providers; these networks also contributed to changing practices. Several international donors changed their types and modes of media assistance in recent years. Some donors used approaches like that of SMS, aiming to improve media partners’ financial sustainability. The evaluation team did not find sufficient evidence to attribute observed changes to SMS rather than to other contextual and institutional factors. The EU, the largest donor for media support activities in Serbia, has been moving away from providing large-project style grants but continues to offer funding for a variety of purposes based on partners' needs, as do other smaller donors. SMS exerted influence on media assistance programs through information sharing and coordination with other USAID Activities as well as through its efforts to encourage donor coordination on activities to promote media freedom. These efforts by SMS and USAID effectively promoted complementarity and avoided duplication of effort, particularly within USAID’s media portfolio. EVALUATION QUESTION 5 Based on the experience and knowledge from this Activity, what approaches are most likely to result in gender equity improvements in the area(s) of focus? To answer EQ5, the evaluation team examined the findings of available gender analyses and assessed the extent to which SMS design, implementation, and monitoring and evaluation incorporated effective GESI approaches. FINDINGS State of Gender Equity and Social Inclusion in Serbia’s Media Sector Little research on gender in the media sector in Serbia had been conducted before the launch of the SMS Activity. USAID completed a Gender Analysis of its activities in 2020 to inform Serbia’s Country Development Cooperation Strategy (CDCS) 2020–2025.45 In addition, OSCE, which received USAID funding for its media freedom activities, produced a report in 2021 on the gender structure of managerial and chief editorial positions in news media outlets in Serbia.46 45 https://pdf.usaid.gov/pdf_docs/PA00WNPZ.pdf 46 https://www.osce.org/mission-to-serbia/491860 USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 36 Women are found in large numbers at all levels of media in Serbia except the highest levels. Women are underrepresented at the highest managerial levels in journalism, such as boards of directors and Chief Executive Officers, according to the recent OSCE gender analysis. That analysis also showed that more women than men work in journalism and editorial roles, with their representation significantly higher than the European average (60 percent compared to the European average of 40 percent).47 The USAID Gender Analysis Report found that women are often in leadership positions in small local media outlets. FGD participants highlighted high levels of female representation in journalism. One participant reported, “I think that journalism in Serbia is a female job, not because of the number of women who are in journalism, but because of their visible efforts to do professional journalism in difficult circumstances.” One media expert reported that women not only are dominating journalism in terms of numbers but also are very successful: “Both the higher and lower ranks of journalism are quite open for women. Equality is acknowledged. The awards last year were all given to women journalists.” Many of the challenges highlighted in available research on GESI in the media sector raise issues that are not explicitly linked to SMS’s financial sustainability objectives or do not provide sufficient information to form such a link. For example, the USAID Gender Analysis found that female journalists are exposed to higher risks of in-person and cyber violence. The 2020 findings of the Global Media Monitoring Project National Report for Serbia48 reported that women made up 20 percent of persons heard, read about, or seen in newspapers, television, radio, and digital news in Serbia, two percentage points less than when similar research was conducted 15 years ago and eight percentage points less than the European average. The report also shows that women are significantly underrepresented in news stories; they are mainly seen in stories on gender-related topics, whose share is only 1 percent of 605 analyzed stories. The USAID Gender Analysis found that women-led media outlets need support to reinforce their financial sustainability, but it did not highlight any specific business challenges faced by women-run media that were distinct from independent media in general—possibly because no such analysis was done. A media expert interviewed by the evaluation team opined, “I think women-led small media outlets face the same business challenges as other small outlets.” However, the same key informant noted that, because females still carry a higher burden of family care than men, they may be disproportionately affected by difficult working conditions for Serbian journalists, including low pay, lack of childcare, and long hours. Available data on social inclusion issues in the media is likewise limited. In 2020, OSCE published an analysis on media outlets that produce content in national minority languages.49 Although the report notes challenges in identifying minority-language media due to incomplete or inaccurate official data, it says that media privatization carried out in 2015 and 2016 significantly reduced the level of information provided in Serbia’s official minority languages.50 The report findings point to potential for a further decline in minority language content due to regulatory changes and increasing production costs. Moreover, the report finds that many minority language media, like other media outlets in Serbia, are subject to political pressure that negatively affects the quality and independence of their production. 47 https://www.osce.org/sr/mission-to-serbia/491866 48 https://whomakesthenews.org/wp-content/uploads/2021/07/Serbia-Report-GMMP.pdf 49 https://www.osce.org/files/f/documents/1/e/473673.pdf 50 There are 15 official minority languages in Serbia: Albanian, Bosnian, Bulgarian, Bunjevac, Croatian, Czech, German, Hungarian, Macedonian, Romani, Romanian, Pannonian Rusyn, Slovak, Ukrainian, and Vlach. 37 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV SMS Compliance with USAID Directives on Conducting Gender Analysis Despite USAID mandates, the SMS Activity did not conduct a gender analysis during its design phase and did not have any specific objectives that addressed GESI. All USAID Activities are mandated to conduct gender analysis and use GESI-sensitive approaches. USAID’s Gender Equality and Female Empowerment Policy51 and the Women’s Entrepreneurship and Economic Empowerment Act of 201852 established that all programs and activities should be designed using gender analysis as a tool for examining potential differences between women and men in the target sector, including not only their different needs, constraints, and opportunities but also the potential impact of the Activity on their lives. The evaluation document review showed that the SMS team did not conduct gender analysis or include gender-sensitive indicators in the AMELP. SMS did not include gender analysis in most of its research. The evaluation team’s review found that the documents and analyses produced by SMS lack insights or information from a GESI perspective. The notable exception is the SMS Comprehensive Study of the Public’s Media Needs and Behaviors in Serbia (the demand-side study), which included GESI-related insights, sex- and age-disaggregated data, and GESI-responsive recommendations. The Social Network Analysis contains no gender analysis and incorporates no GESI-related measurements. The Political Economy Analysis, although it contains some data and information disaggregated by sex, does not apply the findings from that disaggregation in its recommendations. In addition, no GESI-related performance indicators were developed for the SMS AMELP. GESI Approaches Implemented by SMS The lack of explicit GESI objectives and data in Activity documents notwithstanding, SMS integrated GESI-sensitive approaches in its activities. SMS staff reported that the implementation team received input from IREX gender specialists and discussed GESI strategies internally. SMS aimed to achieve GESI goals when selecting partners and showcased the talents of women leaders in Serbia’s independent media scene by selecting female media experts and featuring female media managers as presenters in its meetings and events. According to one SMS staff member, “We made a conscious decision with our [annual program statement], to make sure that we tried to find organizations that had females in leadership roles…and we were able to do that.” Nevertheless, the Activity did not have any explicit GESI criteria or targets in its grant manual. Despite the lack of explicit GESI targets and gender analysis, SMS achieved gender parity. It also included a few minority-language media outlets among its media partner organizations. In SMS-reported data, 50 percent of media outlets and media support organizations that received SMS support were women-owned or women-led.53 In addition, SMS reported that its partners included several media outlets that provided minority-language content or that served socially 51 https://www.usaid.gov/sites/default/files/documents/1865/GenderEqualityPolicy_0.pdf 52 https://www.congress.gov/115/plaws/publ428/PLAW-115publ428.pdf 53 Because SMS did not report gender-disaggregated data, the evaluation team requested this information during the data collection phase. The following is the list of women-owned or -led media: Juzne Vesti (editor-in-chief), Nova Ekonomija, Loznicke Novosti (founder and editor), TV Forum, BETA, Istinomer, Vugl (editor-in-chief), CINS (editor-in-chief), Pescanik, Glas Zajecara, Pistaljka, Maglocistac, Storyteller, Agro Smart, Fabrika Kreativnosti, City Smart (founder and editor), Infocentrala, Dizajn Prica, Future Generations, SoccHer (author and editor), Autonimija (editor-in-chief), Glas Sumadije, Association of Media, BIRN, Local Press, NAESA, Impact Hub (co￾founder). The civil society media outlet Liceulice serves people suffering from homelessness. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 38 excluded groups.54 Moreover, although they could not point to any specific GESI policy, most SMS partners perceived the Activity to have been inclusive. For example, 19 of 24 evaluation survey respondents either agreed (n = 13) or strongly agreed (n = 6) that SMS activities were effective in building the capacities of diverse media outlets, including local and regional, women-led, and ethnic minority outlets. One KII participant remarked, “SMS must have been doing something right, because at any event or gathering of their grantees where I was present, there was always at least an equal number of male and female participants.” The following are examples of SMS activities to support women-led and minority-language media outlets: ● TV Forum, a civil society media outlet established by the Women’s Forum of Prijepolje, is the only female-led TV station in the region. It is dedicated to promoting gender equality in Serbian society. SMS supported TV Forum’s crowdfunding campaign, which enabled the outlet to purchase new equipment. ● SMS supported Bujanovačke, a local news web portal in Southern Serbia, to develop news content in the Albanian language. Since May 2020, the number of articles published in Albanian has consistently grown, leading to a 191 percent increase in page views between May and July 2021. In one year, Bujanovačke doubled the number of advertisers from five to 11 by strengthening cooperation with Albanian businesses. According to SMS staff, their advocacy led Bujanovačke to hire women and members of minority groups in the new positions supported by SMS. ● The street paper Liceulice is a civil society media outlet intended to educate citizens on the perspectives of marginalized and underprivileged groups in Serbia and to promote civil society values. When COVID-19 restrictions prevented Liceulice from selling its print magazine, SMS supported Liceulice to install an e-payment gateway and an online store. From August 2020 to June 2021, this gateway generated nearly $5,000 in contributions. Many SMS partners integrated GESI-sensitive approaches in their activities. During FGDs, many SMS media partners reported that gender equity and tolerance were core organizational principles. A few indicated that they regularly produced content promoting feminist values and the rights of ethnic minority groups. In addition, NAESA, the organization supported by SMS to contribute to advertising standards in Serbia, reported that it plans to promote GESI values, such as avoidance of gender stereotypes, in its future activities to educate media organizations and advertising agencies. Despite these examples of GESI, a few SMS staff, media experts, and media partners emphasized that future Activities like SMS could adopt more systematic GESI approaches. For example, one FGD participant proposed, “In the future, it would be good to have dedicated activities to support the role of women in leadership positions in the media.” CONCLUSIONS SMS efforts to support media partners that respect or promote Gender Equity and Social Inclusion (GESI) values, including its support for the creation of the National Association for Ethical Standards in Advertising (NAESA), was an effective approach to address discriminatory cultural norms and promote greater gender equity. SMS built the capacity of media with gender-sensitive policies and practices, including some which reported regularly producing content promoting feminism and diversity. By building the capacity of these media partners, SMS boosted their financial sustainability and contributed to the continuity of their efforts and subsequently the sustained promotion of positive gender norms. Similarly, NAESA’s commitment to promoting ethical 54 Minority-language media partners included Storyteller (Slovak), Autonomíja (Hungarian), and Bujanovacke (Albanian). 39 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV advertising, specifically plans to educate its members on how to ensure content is sensitive and does not perpetuate gender and ethnic minority stereotypes, has the potential to diminish prejudice and promote greater gender equity and social inclusion over time. Identifying and working with female media managers and experts to lead project activities and capacity building efforts helped elevate female voices and promote positive gender norms. As noted in the Organization for Security and Co-Operation in Europe’s (OSCE) gender analysis, despite making up a large percentage of the media workforce, women are underrepresented at the highest managerial levels in journalism. SMS purposefully selected female media managers and experts to lead workshops and mentor media partners. This approach helped promote positive female role models in the media sector and potentially opened new avenues for women-in-leadership. Targeting local and niche media outlets was an effective approach for reaching women-led and minority-language media. As noted in the USAID Gender Analysis Report, women leaders are more commonly found in local and niche media in Serbia, and minority-language media outlets are often located in communities with mixed ethnic group populations. Despite a lack of a GESI analysis and explicit targets and indicators, SMS’ was effective in reaching women-led and minority-language media by targeting local and niche media outlets during partner selection and ensuring that SMS technical assistance was tailored to the needs of those media. Specifically, tailoring its grant and technical assistance mechanisms to meet the diverse needs of different sizes and types of media organizations located across Serbia, contributed to its success in reaching women-led and minority-language media. While generating significant knowledge and learning on financial sustainability for independent media in Serbia, SMS did not identify specific business constraints, if any, facing women-led and minority media outlets or define approaches to overcome these challenges. SMS was largely noncompliant with USAID directives to conduct a GESI-sensitive needs assessments and measure results using GESI-sensitive indicators. Because of these gaps, SMS did not specifically test approaches or generate significant learning about approaches to improve gender equity in media business development and financial sustainability. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 40 RECOMMENDATIONS Based on the findings and conclusions, the evaluation team makes the following recommendations for future USAID activities to support independent media in Serbia. Build on SMS achievements in supporting the collective efforts of media outlets and media support organizations to improve the media business climate. ● Monitor the experience of media partners and media buyers or advertising agencies with the AAS model. Evaluate whether and how the model generates new advertising revenues for AAS media partners. If feasible, help to improve the model. If it proves successful, help scale and/or replicate the AAS model to include other media partners or new partnerships among other media organizations. ● Continue to foster wider acceptance of ethical standards in advertising, including standards related to gender equity and tolerance. Future support may include promoting partnerships between independent media outlets and multinational companies doing business in Serbia to pilot ethical advertising campaigns showcasing the firms’ commitment to responsible business values like tolerance, green development, and democratic values. ● Support advocacy and coalition building among media partners and others in Serbia’s business community to facilitate better regulation for e-commerce, increased access to innovative payment systems (fintech), and improvements related to other media financing issues. To promote transformative change, consider offering grants and technical assistance for financial management, marketing, and organizational development of media partners that meet minimum business readiness requirements. ● Even relatively large independent media outlets lack human resources with expertise in sales, marketing, and financial management. Provide short-term grant and technical assistance to enable these partners to hire new personnel as they restructure and to train and mentor staff in these areas. Anchor grant assistance packages, for example, could help meet this need. ● For niche and local media partners that are too small to absorb this kind of assistance, test collective solutions to enable them to implement innovative business ideas and access needed technical expertise. Solutions might include sharing part-time staff in areas like information technology (IT), sales and marketing, or business development—building on the SMS sustainability hub approach. ● To reduce dependence on international donors, continue to condition grant or investment disbursements on meeting outcome benchmarks including increased revenue, diversified revenue, and increased audience reach. Ensure that prospective implementers conduct a GESI analysis during start-up and to include explicit GESI-responsive strategies in their implementation plan. ● Use explicit gender and social inclusion criteria when selecting partners, for example prioritize media led by women and ethnic minorities and/or that have organizational values that promote gender equality and non-discrimination. ● Conduct research to identify specific financial sustainability needs and opportunities facing women-led and ethnic minority media. ● Include GESI-sensitive indicators and disaggregate data by gender. ● Promote the capacity and leadership of women media managers. Further sensitize all media managers to principles of gender-sensitive management and journalism. 41 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV Develop a monitoring and evaluation framework that measures business outcomes aligned to the Activity intervention strategy. ● Continue to follow SMS’s practice of tracking changes in revenue and audience size, as well as return on investment, to monitor media partners’ progress towards financial sustainability and to measure the efficiency of USAID support. In addition, establish and monitor indicators that measure changes in annual revenue, profitability (among for-profit media outlets), and the proportion of income generated through business activities and audience donations versus international donor contributions. ● Periodically assess changes in organizational capacity. Relevant organizational capacity indicators include changes in the numbers and capacity of media outlet personnel with marketing, sales, data analysis, and financial management skills and, for larger media partners, the extent to which business functions have been separated from editorial functions. ● Reduce reporting frequency and provide technical support for data collection, when feasible, to help media partners collect relevant financial and organizational capacity data. Many partners— particularly small, local, and niche media outlets—face human resource capacity constraints. Data collection and reporting support would improve the Activity’s access to quality data. This can be done through annual or quarterly surveys rather than monthly data collection. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 42 ANNEX A. EVALUATION STATEMENT OF WORK C.1.1. BACKGROUND The purpose of the Strengthening Media Systems activity is to enable Serbian media system actors to improve the legislative, regulatory and market environment for media, so that Serbian media outlets can provide Serbian citizens with the information to enable them to effectively engage in informed citizenship and hold their government accountable. The Activity must achieve the purpose through two interrelated objectives: 1. Support the development of a more enabling legal, regulatory and market environment for media in Serbia; and 2. Support the development of financial sustainability, innovation, and partnerships in the Serbian media sector. Detailed background information on the SOW for this activity can be found in Attachment # of this RFTOP (“SMS SOW”). C.1.2. PURPOSE OF THE EVALUATION USAID/Serbia requires a full, evidence-based and independent final performance evaluation of the USAID/Serbia Strengthening Media Systems (SMS) Activity, and a report that provides a qualitative as well as quantitative evaluation of the progress made towards achieving expected results per the contract. Specifically, the purpose of the evaluation is to: ● Examine the effectiveness of implemented project activities; ● Determine whether the project has achieved the planned results; ● Identify successes and gaps in performance against targets; and ● Provide information on whether the project has unintended consequences. The Contractor must not simply provide an accounting of performance against targets but provide independent analysis on why targets were realized, met, or significantly surpassed. The Contractor must also provide an analysis of the reasons behind gaps between expected and actual performance, including an identification of the likely source(s) of these gaps (design, project management, changes in operating environment, change in context, success factors, if contractor’s approach has led to intended results and goals, etc.). This evaluation will start o/in early August 2021. C.1.3. EVALUATION QUESTIONS, METHODOLOGY AND DATA SOURCES Evaluation Questions: Evaluation questions that Contractor must address are as follows: 1. 1. Has the activity demonstrated results that are consistent with its theory of change and to what degree it has achieved its stated objectives? 2. To what extent did SMS identify and address challenges and opportunities in the media sector in Serbia, in particular digitization, approaches to content monetization, innovative business models and new ways to increase financial suitability of independent media? 3. How well did the SMS adapt to the changing operational environment, including incorporating adaptive management and continuous learning in planning and delivering its assistance? 43 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV 4. To what extent has SMS influenced changing approaches - both in the donor community and among beneficiaries - to media development in Serbia? 5. Based on the experience and knowledge from this activity what approaches are most likely to result in gender equity improvements in the area(s) of focus? Evaluation Methodology: In developing an appropriate methodology, the contractor must consider potential limitations and incorporate best-practice mitigation measures. The evaluation must be supported by relevant data and information gathered from meetings with beneficiaries and all the relevant stakeholders. The evaluation team should clearly demonstrate links between the data collected and the subsequent findings, conclusions, and recommendations drawn. Where applicable, responses to the questions above and resulting data must be dis-aggregated by sex and gender. Available Data Sources: The contractor will have access to all performance data, M&E database, surveys, and other deliverables, that have been collected and reported by the implementing partner as part of their Performance Monitoring and Evaluation plan for SMS. The contractor must also review the following documents in preparation for the SMS evaluation: ● Relevant project reports, assessments, annual and life of project work plans. ● Full data set in their formats available and collected by the implementing partner. ● Annual performance reports of SMS partners and sub-partners. ● USAID/Serbia 2020-2025 CDCS. ● USAID/Serbia Media Sector Assessment. ● Rapid Assessment of USAID/Media Portfolio. ● Serbia’s Strategy of Public Information System Development and other relevant policy and regulatory documents. In addition, the evaluation team must consult with key informants and stakeholders, including: ● Meetings with USAID/Serbia. ● Meetings with current/former SMS implementing partners in Serbia. ● Meeting with relevant stakeholders. ● Meeting with direct project beneficiaries. ● Meetings with other donors, including the EU Delegation and those donors engaged in the media sector. C.3 GENERAL CONTRACTOR REQUIREMENTS – STATEMENT OF WORK The Contractor must conduct a methodologically sound evaluation which must be fully compliant with USAID’s Evaluation Policy and Automated Directives System (https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf), and must utilize tested and scientific evaluation methodologies. C.4 TEAM COMPOSITION The Contractor(s) must utilize a team with the experience and skills necessary to complete this task as specified in Section F.4. C.5 SCHEDULE AND LOGISTICS USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 44 The schedule (with precise due dates for deliverables) will be defined in the work plan before the start of the field work for each evaluation. Prior to commencing the field work, all team members will review background program documents to gain a firm understanding of the situation in Serbia and the activity/ies. The Contractor(s) will be responsible for logistical arrangements for conducting the evaluation, including scheduling meetings, making copies of documents and drafts, making travel arrangements, etc. Field visit logistics will be organized and managed by the Contractor(s). This includes travel within country and in-country transportation. USAID/Serbia will provide the team(s) with input and guidance in setting up a schedule of interviews and site visits, but the responsibility for the schedule for the evaluation resides with the Evaluation Teams. 45 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ANNEX B. EVALUATION MATRIX EVALUATION MATRIX EVALUATION QUESTION EVALUATION SUB -QUESTIONS DATA SOURCES 1. Has the activity demonstrated results that are consistent with its theory of change, and to what degree has it achieved its stated objectives? 1.1 How have relationships between and among actors within the media system changed over the course of the Activity? ● Document review ● Secondary data review ● Net-map55 ● KIIs ● FGDs 1.2 To what extent has the Activity supported policy reform efforts related to improving the media business environment? ● Document review ● Secondary data review ● KIIs ● FGDs 1.3 What innovative approaches to business development have been most successful and why? ● Document review ● Secondary data review ● KIIs ● FGDs 1.4 What types of activities have been most effective in pushing forward policy reform efforts related to improving the media business environment (i.e., advocacy, network development, building monitoring and evaluation (M&E) capacity of media actors, enhancing research and policy development capacity of media actors)? ● Document review ● Secondary data review ● KIIs ● FGDs 55 Given the Activity’s theory of change, the evaluation team originally proposed a net-map as part of the data collection and analysis methods. However, due to the Activity’s shift in strategy in Year 2, the evaluation team, in consultation with USAID, determined that the net-map was no longer an appropriate analysis approach for answering the EQs. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 46 2. To what extent did SMS identify, and address challenges and opportunities in the media sector in Serbia, in particular digitization, approaches to content monetization, innovative business models, and new ways to increase financial sustainability of independent media? 2.1 What financial models or approaches to monetization show the most promise for scalability and sustainability? ● Document review ● Secondary data review ● KIIs ● FGDs 2.2 To what extent and in which ways has the Activity worked with the private sector to promote financial sustainability of media actors? ● Document review ● Secondary data review ● KIIs ● FGDs 3. How well did SMS adapt to the changing operational environment, including incorporating adaptive management and continuous learning in planning and delivering its assistance? 3.1 What mechanisms and approaches did SMS utilize to continuously monitor Activity implementation and contextual factors? And how did SMS incorporate these mechanisms into program management? ● Document review ● KIIs ● FGDs 3.2 How did SMS help media system actors adapt to challenges and opportunities that arose during implementation? ● Document review ● KIIs ● FGDs 4. To what extent has SMS influenced changing approaches – both in the donor community and among beneficiaries – to media development in Serbia? 4.1 How has SMS influenced or facilitated other donor funded initiatives or partners to apply lessons learned, best practices, or innovative approaches to media development within the Serbian context? ● Document review ● Secondary data review ● Net-map ● KIIs ● FGDs 4.2 What SMS interventions or approaches have been most effective in facilitating changes in the media system as it relates to partnerships, financial models, audience outreach, and the use of data analytics? ● Document review ● Secondary data review ● Net-map ● KIIs ● FGDs 47 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV 5. Based on the experience and knowledge from this Activity, what approaches are most likely to result in gender equity improvements in the area(s) of focus? 5.1 What linkages or relationships within the media system promoted or hindered increased gender equity and participation in the media sector? How and why were these linkages or relationships successful? ● Document review ● Secondary data review ● Net-map ● KIIs ● FGDs USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 48 ANNEX C. DATA COLLECTION INSTRUMENTS CONSENT SCRIPT: KII Thank you for taking the time to talk to me. My name is [INSERT NAME]. I am from IMPAQ International, a U.S.-based international policy research, technical assistance, and evaluation company. IMPAQ is conducting a final evaluation of the USAID’s Strengthening Media Systems Activity or SMS. SMS aims to increase media independence in Serbia by helping media system actors to improve and diversify their business strategies, improve their audience outreach, and capitalize on media digitization trends as well as by improving the overall business enabling environment for Serbian media outlets. The Activity, which is being implemented by the International Research & Exchanges Board (IREX), was launched in October 2017, and is scheduled to end in February 2022. The purpose of the evaluation is to identify the extent to which SMS achieved its main objectives, identify key achievements, its strategies that have worked well, as well as any challenges and lessons learned. I would like to ask you questions related to your experience with the Activity. This interview will last approximately one hour (1h). With your permission, I would like to audio record this session for report writing and analysis purposes only. The recordings will be destroyed once we complete our analysis. Is this okay with you? Yes No In the evaluation report, your name will not be mentioned except in an annex in a list of people consulted and this list will only be shared with the donor (USAID) The evaluation team will make every effort to protect the anonymity and confidentiality of our discussion and hope you will feel comfortable providing honest feedback on your experiences and points of view. Participation is voluntary; also, you may decline to respond to our questions or end the interview at any time. Do you have any questions? Can we start now? 49 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV CONSENT SCRIPT: FOCUS GROUP DISCUSSION Thank you for taking the time to talk to participate in this focus group discussion. My name is [INSERT NAME] and I will be facilitating our discussion today. With me are my colleagues [INSERT NAME] and [INSERT NAME] who may also participate by asking questions and taking notes. We are from IMPAQ International, a U.S.-based international policy research, technical assistance, and evaluation company. IMPAQ is conducting a final evaluation of the USAID’s Strengthening Media Systems Activity or SMS. SMS aims to increase media independence in Serbia by helping media system actors to improve and diversify their business strategies, improve their audience outreach, and capitalize on media digitization trends as well as by improving the overall business enabling environment for Serbian media outlets. The Activity, which is being implemented by the International Research & Exchanges Board (IREX), was launched in October 2017, and is scheduled to end in February 2022. The purpose of the evaluation is to identify the extent to which SMS achieved its main objectives, identify key achievements, its strategies that have worked well, as well as any challenges and lessons learned. Through this focus group discussion, we hope to learn about your experiences with SMS, including which types of SMS support were effective or ineffective, the challenges you experienced (if any), as well as the outcomes of your collaboration with the Activity. This discussion will last approximately two hours (2h). With your permission, we would like to audio record the session for report writing and analysis purposes only. The recordings will be destroyed once we complete our analysis. Is this okay with you? Yes No In the evaluation report, your names will not be mentioned except in an annex in a list of people consulted and this list will only be shared with the donor (USAID) The evaluation team will make every effort to protect the anonymity and confidentiality of our discussion and hope you will feel comfortable providing honest feedback on your experiences and points of view. Participation is voluntary; also, you may decline to respond to our questions or end the interview at any time. Do you have any questions? Can we start now? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 50 FOCUS GROUP DISCUSSION GUIDELINES Date: Interviewer Name: Primary Notetaker Name: Location of participants: Name Title Organization Gender Informed Consent: The FGD facilitator will explain the purpose of the meeting and administer the informed consent protocol (which includes information about SMS, the purpose of evaluation, and explains that their participation is voluntary, and that information shared will be kept confidential. The facilitator will also request that participants in the group respect the confidentiality of their co-participants by not discussing what was discussed with others outside the group). Introductions: We will ask each participant to introduce herself/himself and briefly how his/her institution collaborated with SMS. FGD Questions: 1. To what extent and how has the SMS Activity been effective in helping independent media outlets in Serbia to identify and implement new business ideas and approaches (including digitization) and income generation models? What SMS-supported business ideas and income generation models have been most successful to date and why? (Probe for crowdfunding, subscription, and membership models, new income generating services, development of new content delivery approaches) Follow-up questions: ⋅ What are some success stories? What factors contributed to success? ⋅ What are examples of less successful initiatives? What factors limited results? 51 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV 2. To what extent and how has the SMS Activity helped independent media outlets in Serbia to understand its target audiences’ needs and behaviors and increase and/or improve audience outreach and engagement? What SMS-supported approaches have been most useful and effective to date and why? In what ways, if any, could SMS assistance have been more effective? (Probe for usefulness of SMS research [e.g., the demand-side study] and support on improving data analytics, training on the use of social media, support for alternative content delivery mechanisms)? Follow-up questions: ⋅ What are examples of success stories? What factors contributed to these successes? ⋅ What are examples of less successful initiatives? What factors limited results? ⋅ To what extent and how did SMS foster audience outreach and engagement activities to underserved groups (women, minority groups, youth, people living in rural areas, LGBTQI and/or integrate gender concerns into their programming?) 3. In what ways, if any, has the SMS Activity helped foster business or advocacy-orientated partnerships and collaboration among independent media outlets and/or with media support organizations (either private sector service providers or NGOs) to improve the business environment for media outlets? (Probe for collaboration with service providers like ICT Hub, SmartOcto, e payment solution providers, as well as collaboration within project supported working groups such as the Startup Committee, Thematic Nodes) ⋅ Have any new partnerships /collaborations been established by or with women-owned or led media organizations? ⋅ What are examples of successful partnerships/collaborations? What were the success factors? ⋅ What are examples of less successful initiatives? What factors limited results? 4. What have been the strengths and weaknesses of SMS support mechanisms, such as grants, events, technical assistance, collaboration networks/nodes? (probe for adequacy of funding amounts, access to funding for different types of investments, quality of technical assistance, timeliness, and efficiency in delivery) ⋅ In what ways, if any, did SMS identify and capitalize on lessons learned to improve its support to media partners during the implementation period? (probe for if/how SMS tested, assessed, and documented the results of different approaches? Created opportunities for partners to provide feedback?)? Reminder: The following are examples of SMS support mechanisms (1) Larger grant technical and financial assistance packages such as Media Accelerator, Media Digitizer, (2) Small grant facilities such as Mobile Support to Media, Direct Support to Media, (3) SMS Events: Media Jam, Summer School (4) Convening mechanisms: E payment, Media Measurement, Regulations, and Podcast Nodes, Media Talks 5. To what extent and how was SMS effective in identifying and helping media outlets which are managed by and/or serve diverse and marginalized populations? What approaches or strategies (if any) were effective ensuring SMS was supportive of: ⋅ Women-owned or run media outlets? ⋅ Outlets run by or primarily serving youth? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 52 ⋅ Outlets run by or serving ethnic minority groups? ⋅ Outlets run by or serving other marginalized populations? How might SMS have been more effective in identifying and supporting media outlets managed by and/or serving diverse and/or marginalized audiences? 6. To what extent and how was SMS effective in extending support to media outlets with varying degrees of maturity (start-ups as well as more mature media organizations), of different sizes, and based both inside and outside Serbia’s major cities? How might SMS have been more effective in identifying and supporting start-ups and media outlets serving local populations outside the big cities? 7. To what extent and how has SMS helped its media partners understand and uphold ethical standards and/or adapt to existing or anticipated regulations affecting the media business practices? (Probe for usefulness of SMS support to partners to implement EU data privacy regs, creation of National Association for Ethical Standards in Advertising [NAESA], other advocacy activities) 8. In what ways, if any, has SMS support for media outlets’ new business ideas (including digitalization) and business models contributed to strengthening independent voices within Serbia’s media landscape? ⋅ What are examples of success stories? What factors contributed to these successes? ⋅ What were the most important limitations of SMS’s “business” orientation in promoting/enabling independent voices to grow and flourish in Serbia? 9. In what ways, if any, has SMS Activity helped promote greater gender equity in the media sector, specifically as it relates to: o Representation of women in leadership roles in media organizations? o Addressing the risk of violence (in-person or cyber) against female journalists? o Addressing the media’s depiction or reinforcement of gender stereotypes? In what ways, if any, do you think the SMS Activity could have addressed these issues more? 10. In what ways, if at all, has SMS influenced the approaches of other donors in their support for media freedom in Serbia? 11. What recommendations do you have for USAID that reflects on if/how to sustain and build on the accomplishments of the SMS activity? 53 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV USG KIIS Date: Interviewer Name: Primary Notetaker Name: Respondent Name: Respondent title: Respondent Organization: Sex of respondent: 1. Please introduce yourself and provide a brief overview of your role overseeing the SMS Activity. How long have you held that role? 2. What were the most significant overarching challenges or opportunities that SMS faced reaching its objectives over the life of the Activity? How well did SMS adapt its strategies and management to capitalize on opportunities, mitigate risks, and/or overcome challenges? ⋅ What approaches or tools were helpful in enabling SMS to adapt to and capitalize on contextual changes? (probe for role of research activities, monitoring activities, consultations, other) ⋅ What factors, if any, hindered SMS ability to adapt? 3. To what extent and how has the SMS Activity been effective in helping independent media outlets in Serbia to become stronger and more financially viable media businesses? What SMS-supported business ideas, income generation models or other strategic changes have been most successful to date and why? (Probe for crowdfunding, subscription, and membership models, new income generating services, development of new content delivery approaches) Follow-up questions: ⋅ What are some success stories? What factors contributed to success? ⋅ What are examples of less successful initiatives or persistent gaps? What factors limited results? 4. In what ways, if any, has the SMS Activity helped foster business and advocacy-oriented partnerships and collaboration among independent media outlets and/or with media support organizations (either private sector service providers, business associations, or NGOs) to improve the business environment for media outlets? (Probe for collaboration with service providers like ICT Hub, SmartOcto, e payment solution providers, as well as collaboration within project supported working groups such as the Startup Committee, Thematic Nodes) ⋅ What are examples of successful business and/or advocacy-oriented partnerships/collaborations? What were the success factors? ⋅ What are examples of less successful initiatives? What factors limited results? 5. What have been the strengths and weaknesses of SMS’ various support mechanisms, such as grants, events, technical assistance, collaboration networks/nodes? (probe for adequacy of funding amounts, access to funding for different types of investments, quality of technical assistance, timeliness, and efficiency in delivery) What might SMS have done differently, if anything, to be more effective delivering support to its partners? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 54 Reminder: The following are examples of SMS support mechanisms (1) Larger grant technical and financial assistance packages such as Media Accelerator, Media Digitizer, (2) Small grant facilities such as Mobile Support to Media, Direct Support to Media, (3) SMS Events: Media Jam, Summer School (4) Convening mechanisms: E payment, Media Measurement, Regulations, and Podcast Nodes, Media Talks 6. To what extent and how was SMS effective in identifying and helping media outlets which are managed by and/or serve diverse and marginalized populations? What approaches or strategies (if any) were effective ensuring SMS was supportive of: ⋅ Outlets run by women and/or which focus on women’s concerns? ⋅ Outlets run by or primarily serving youth? ⋅ Outlets run by or serving ethnic minority groups? ⋅ Outlets run by or serving other marginalized populations? How might SMS have been more effective (if at all) in identifying and supporting media outlets managed by and/or serving diverse and/or marginalized audiences? 7. To what extent and how was the SMS Activity effective in extending its support to media outlets with varying degrees of maturity (start-ups as well as more mature media organizations), of different sizes, and based both inside and outside Serbia’s main cities? How, if at all, might SMS have been more effective reaching diverse partners? 8. To what extent and how has the SMS Activity helped its media partners understand and uphold ethical standards and/or adapt to existing or anticipated regulations affecting the media business practices? (Probe for usefulness of SMS support to partners to implement EU data privacy regs, creation of National Association for Ethical Standards in Advertising [NAESA], other advocacy activities) 9. In what ways, if any, has SMS support for media outlets’ new business ideas (including digitalization) and their implementation of new business models contributed to strengthening independent voices within Serbia’s media landscape? ⋅ What are examples of success stories? What factors contributed to these successes? ⋅ What were the most important limitations (if any) of SMS’s “business” orientation in promoting/enabling independent voices to grow and flourish in Serbia? 10. In what ways, if any, has SMS Activity helped promote greater gender equity in the media sector, specifically as it relates to: o Representation of women in leadership roles in media organizations? o Addressing the risk of violence (in-person or cyber) against female journalists? o Addressing the media’s depiction or reinforcement of gender stereotypes? In what ways, if any, do you think the SMS Activity could have addressed these issues more? 11. In what ways, if at all, has SMS influenced the approaches of other donors in their support for media freedom in Serbia? ⋅ What are examples of ways that SMS has been influential? ⋅ What factors contributed to and/or limited SMS influence? 55 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV 12. Is there anything more you would like to add? Is there anything that we did not ask about that is important for us to understand in the context of this evaluation? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 56 SMS STAFF KII Date: Interviewer Name: Primary Notetaker Name: Respondent Name Respondent title: Respondent Organization: Sex of respondent 1. Please introduce yourself and provide a brief overview of your role within the SMS Activity (including how long you have held that role). 2. What were the most significant overarching challenges or opportunities that SMS faced reaching its objectives over the life of the Activity? How did SMS adapt its media outlet capacity building strategies to capitalize on these opportunities, mitigate risks, and/or overcome challenges? ⋅ What approaches or tools were helpful in enabling SMS to adapt to and capitalize on contextual changes? (probe for role of research activities, monitoring activities, consultations, other) ⋅ What factors, if any, hindered SMS ability to adapt? 3. To what extent and how has the SMS Activity been effective in helping independent media outlets in Serbia to identify and implement new business ideas and income generation models? What SMS￾supported business ideas and income generation models have been most successful to date and why? (Probe for crowdfunding, subscription, and membership models, new income generating services, development of new content delivery approaches) Follow-up questions: ⋅ What are some success stories? What factors contributed to success? ⋅ What are examples of less successful initiatives or persistent gaps? What factors limited results? 4. To what extent and how has the SMS Activity helped independent media outlets in Serbia to increase and/or improve audience outreach and engagement? What SMS-supported approaches have been most useful and effective to date and why? (Probe for usefulness of SMS research [e.g., the demand-side study] and support on improving data analytics, training on the use of social media for marketing, support for alternative content delivery mechanisms)? Follow-up questions: ⋅ What are examples of success stories? What factors contributed to these successes? ⋅ What are examples of less successful initiatives? What factors limited results? 5. In what ways, if any, has the SMS Activity helped foster business or advocacy-oriented partnerships and collaboration among independent media outlets and/or with media support organizations (either private sector service providers, business associations, or NGOs) to improve the business environment for media outlets? (Probe for collaboration with service providers like ICT Hub, SmartOcto, e payment solution providers, as well as collaboration within project supported working groups such as the Startup Committee, Thematic Nodes) ⋅ What are examples of successful business and/or advocacy-oriented partnerships/collaborations? What were the success factors? 57 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ⋅ What are examples of less successful initiatives? What factors limited results? 6. What have been the strengths and weaknesses of SMS’ various support mechanisms, such as grants, events, technical assistance, collaboration networks/nodes? (probe for adequacy of funding amounts, access to funding for different types of investments, quality of technical assistance, timeliness, and efficiency in delivery) Reminder: The following are examples of SMS support mechanisms (1) Larger grant technical and financial assistance packages such as Media Accelerator, Media Digitizer, (2) Small grant facilities such as Mobile Support to Media, Direct Support to Media, (3) SMS Events: Media Jam, Summer School (4) Convening mechanisms: E payment, Media Measurement, Regulations, and Podcast Nodes, Media Talks 7. To what extent and how was SMS effective in identifying and helping media outlets which are managed by and/or serve diverse and marginalized populations? What approaches or strategies (if any) were effective ensuring SMS was supportive of: ⋅ Outlets run by women and/or which focus on women’s concerns? ⋅ Outlets run by or primarily serving youth? ⋅ Outlets run by or serving ethnic minority groups? ⋅ Outlets run by or serving other marginalized populations? How might SMS have been more effective in identifying and supporting media outlets managed by and/or serving diverse and/or marginalized audiences? 8. To what extent and how was SMS effective in extending its support to media outlets with varying degrees of maturity (start-ups as well as more mature media organizations), of different sizes, and based outside Serbia’s main cities? How, if at all, might SMS have been more effective? 9. To what extent and how has SMS helped its media partners understand and uphold ethical standards and/or adapt to existing or anticipated regulations affecting the media business practices? (Probe for usefulness of SMS support to partners to implement EU data privacy regs, creation of National Association for Ethical Standards in Advertising [NAESA], other advocacy activities) 10. In what ways, if any, has SMS support for media outlets’ new business ideas (including digitalization) and their implementation of new business models contributed to strengthening independent voices within Serbia’s media landscape? ⋅ What are examples of success stories? What factors contributed to these successes? ⋅ What were the most important limitations of SMS’s “business” orientation in promoting/enabling independent voices to grow and flourish in Serbia? 11. In what ways, if any, has SMS Activity helped promote greater gender equity in the media sector, specifically as it relates to: o Representation of women in leadership roles in media organizations? o Addressing the risk of violence (in-person or cyber) against female journalists? o Addressing the media’s depiction or reinforcement of gender stereotypes? In what ways, if any, do you think the SMS Activity could have addressed these issues more? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 58 12. In what ways, if at all, has SMS influenced the approaches of other donors in their support for media freedom in Serbia? ⋅ What are examples of ways that SMS has been influential? ⋅ What factors contributed to and/or limited SMS influence? 13. What recommendations do you have for USAID that reflects on if/how to sustain and build on the accomplishments of the SMS activity? 14. Is there anything more you would like to add? Is there anything that we did not ask about that is important for us to understand in the context of this evaluation? 59 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ⋅ MEDIA OUTLETS AND MEDIA SUPPORT ORGANIZATIONS KII Date: Interviewer Name: Primary Notetaker Name: Respondent Name Respondent title: Respondent Organization: Sex of respondent 1. Please introduce yourself and provide us with a brief overview of your organization as well as the ways you and your organization have collaborated with SMS over the last four years. 2. To what extent and how has the SMS Activity been effective in helping your organization or the organizations you serve to identify and implement new business ideas and income generation models? ⋅ What SMS-supported business ideas and income generation models have been most successful to date and why? (Probe for crowdfunding, subscription, and membership models, new income generating services, development of new content delivery approaches). What factors contributed to success? ⋅ What are examples of less successful initiatives? What factors limited results? 3. (This question is for media support organizations only) To what extent and how has the SMS Activity been effective in helping your organization to identify and implement services and intervention strategies to strengthen your support for independent media organizations? ⋅ What SMS-supported services or intervention strategies have been most successful to date and why? ⋅ What are examples of less successful initiatives? What factors limited results? 4. (This question is for media outlets only) To what extent and how has the SMS Activity helped your organization to improve audience outreach and engagement? ⋅ What SMS-supported approaches have been most useful and effective to date? What factors contributed to these successes? (Probe for usefulness of SMS research [e.g., the demand-side study] and support on improving data analytics, training on the use of social media, support for alternative content delivery mechanisms)? ⋅ What are examples of less successful initiatives? What factors limited results? ⋅ To what extent and how did SMS foster audience outreach and engagement activities to underserved groups (women, minority groups, youth, people living in rural areas, LGBTQI and/or integrate gender concerns into their programming? 5. In what ways, if any, has the SMS Activity helped foster partnerships and collaboration among independent media outlets and/or with media support organizations (either private sector service providers or NGOs) to improve the business environment for media outlets? (Probe for collaboration with service providers like ICT Hub, SmartOcto, e payment solution providers, as well as collaboration within project supported working groups such as the Startup Committee, Thematic Nodes) ⋅ What are examples of successful partnerships/collaborations? What were the success factors? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 60 ⋅ What are examples of less successful initiatives? What factors limited results? ⋅ Have any new partnerships /collaborations been established by or with women-owned or led media organizations? 6. What have been the strengths and weaknesses of SMS support mechanisms, such as grants, events, technical assistance, collaboration networks/nodes? (probe for adequacy of funding amounts, access to funding for different types of investments, quality of technical assistance, timeliness, and efficiency in delivery) ⋅ In what ways, if any, did SMS identify and capitalize on lessons learned to improve its support to media partners during the implementation period? (probe for if/how SMS tested, assessed, and documented the results of different approaches? Created opportunities for partners to provide feedback?)? Reminder: The following are examples of SMS support mechanisms (1) Larger grant technical and financial assistance packages such as Media Accelerator, Media Digitizer, (2) Small grant facilities such as Mobile Support to Media, Direct Support to Media, (3) SMS Events: Media Jam, Summer School (4) Convening mechanisms: E payment, Media Measurement, Regulations, and Podcast Nodes, Media Talks 7. To what extent and how has SMS helped your organization or the media organizations you serve to understand and uphold ethical business practices and/or adapt to existing or anticipated regulations affecting media outlets’ business practices? (Probe for usefulness of SMS support to partners to implement EU data privacy regs, creation of National Association for Ethical Standards in Advertising [NAESA]) 8. In what ways, if any, has SMS enabled your organization or the media organizations that you support to advocate for laws and regulations to create a more supportive business environment for independent media outlets? 9. In what ways, if any, has SMS support for your organization’s (or the organizations you serve) new business ideas and business models contributed to strengthening independent voices within Serbia’s media landscape? What factors have either supported or limited SMS’ effectiveness in this regard? 10. In what ways, if any, has SMS Activity helped promote greater gender equity in the media sector, specifically as it relates to: o Representation of women in leadership roles in media organizations? o Addressing the risk of violence (in-person or cyber) against female journalists? o Addressing the media’s depiction or reinforcement of gender stereotypes? In what ways, if any, do you think the SMS Activity could have addressed these issues more? 11. In what ways, if at all, has SMS influenced the approaches of other donors in their support for media freedom in Serbia? 61 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV 12. What recommendations do you have for USAID that reflects on if/how to sustain and build on the accomplishments of the SMS activity? 13. Is there anything more you would like to add? Is there anything that we did not ask about that is important for us to understand in the context of this evaluation? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 62 GOVERNMENT OF SERBIA Date: Interviewer Name: Primary Notetaker Name: Respondent Name Respondent title: Respondent Organization: Sex of respondent 1. How would you assess the media framework in Serbia in the context of the recently adopted Media Strategy? 2. Could you provide an overview of the activities that arose from the Action Plan, in particular drafting of the amendments to the Law on Public Information and Media, project co-financing of media outlets? 3. What have been the most significant obstacles and opportunities affecting the financial viability of independent media outlets in Serbia over the last four years? 4. In what ways, if at all, has USAID’s SMS activity contributed positively to helping individual independent media outlets to seize opportunities and overcome obstacles affecting their financial viability since 2017? What are some examples? 5. To what extent, if at all, has the SMS activity or its partners highlighted legal, regulatory, or other sector wide issues that should be addressed to improve the business environment for independent media outlets? What are some examples? 6. In what ways, if any, has the Media Strategy included actions to improve the business environment for independent media outlets in Serbia? What are examples? In what areas has progress been made? Which are the most significant gaps? 7. In what ways, if any, has SMS Activity helped promote greater gender equity in the media sector, specifically as it relates to: o Representation of women in leadership roles in media organizations? o Addressing the risk of violence (in-person or cyber) against female journalists? o Addressing the media’s depiction or reinforcement of gender stereotypes? In what ways, if any, do you think the SMS Activity could have addressed these issues more? 8. What recommendations do you have for USAID has it reflects on if/how to build on the accomplishments of the SMS activity? 63 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV OTHER USAID IMPLEMENTING PARTNERS, OTHER DONORS, ACADEMIA, MEDIA EXPERTS KII Date: Interviewer Name: Primary Notetaker Name: Respondent Name Respondent title: Respondent Organization: Sex of respondent 1. Please introduce yourself and provide a brief overview of your organization, its activities promoting media freedom, and the ways you have collaborated with the SMS Activity. 2. How would you assess the media framework in Serbia in the context of the recently adopted Media Strategy? 3. Could you provide an overview of the activities that arose from the Action Plan, in particular drafting of the amendments to the Law on Public Information and Media, project co-financing of media outlets? 4. What have been the most significant obstacles and opportunities affecting the financial viability of independent media outlets in Serbia over the last four years? 5. In what ways, if at all, has USAID’s SMS activity contributed positively to helping individual independent media outlets to seize opportunities and overcome obstacles affecting their financial viability since 2017? What are some examples? 6. In what ways, if at all has the SMS Activity contributed positively to improving the broader business environment for independent media in Serbia? What are some examples? 7. To what extent, if at all, has the SMS activity or its partners been effective in highlighting legal, regulatory, or other sector-wide issues that should be addressed to improve the business environment for independent media outlets? What are some examples? 8. In what ways, if any, has SMS support to improve the financial viability of individual independent media outlets and/or to improve the business environment for independent media strengthened independent voices within the Serbia media landscape? What factors have either supported or limited SMS’ effectiveness in this regard? 9. In what ways, if any, has SMS Activity helped promote greater gender equity in the media sector, specifically as it relates to: o Representation of women in leadership roles in media organizations? o Addressing the risk of violence (in-person or cyber) against female journalists? o Addressing the media’s depiction or reinforcement of gender stereotypes? In what ways, if any, do you think the SMS Activity could have addressed these issues more? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 64 10. In what ways, if at all, has SMS influenced the approaches of other donors (or for donors, your approaches) in their support for media freedom in Serbia? 11. What recommendations do you have for USAID that reflects on if/how to sustain and build on the accomplishments of the SMS activity? 12. Is there anything more you would like to add? Is there anything that we did not ask about that is important for us to understand in the context of this evaluation? 65 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV SMS PARTNER/GRANTEE SURVEY IMPAQ is conducting a final evaluation of the USAID’s Strengthening Media Systems Activity or SMS. SMS aims to increase media independence in Serbia by helping media system actors to improve and diversify their business strategies, improve their audience outreach, and capitalize on media digitization trends as well as by improving the overall business enabling environment for Serbian media outlets. The Activity, which is being implemented by the International Research & Exchanges Board (IREX), was launched in October 2017, and is scheduled to end in February 2022. The purpose of the evaluation is to identify the extent to which SMS achieved its main objectives, identify key achievements, its strategies that have worked well, as well as any challenges and lessons learned. As part of the evaluation, IMPAQ is conducting a short survey with SMS partners and grantees to understand how the Activity has supported your organization in the areas of financial sustainability, audience outreach and engagement, and partnership and collaboration with other media sector actors. The survey will help IMPAQ to better understand the effectiveness of SMS interventions. The survey should take approximately 20-30 minutes to complete. We ask that you complete the survey as honestly and accurately as possible. Your individual responses will remain confidential. Data will be aggregated and presented as such in the final evaluation report. In the evaluation report, your name will not be mentioned except in an annex in a list of people consulted and this list will only be shared with the donor (USAID). Your participation is voluntary. If you have any questions regarding the survey or its use, please contact [NAME] at [EMAIL ADDRESS AND/OR PHONE NUMBER]. Demographic Information 1. Gender: ● Male ● Female ● Prefer not to disclose 2. Organization type: ● NGO ● Media Association ● Media Support Organization ● National Media Outlet ● Regional or local media outlet ● Other (please specify) 3. Type of assistance received from SMS (check all that apply): ● Grant ● Technical Assistance ● Mentorship ● Other (please specify) Survey 1. To survive and progress independent media outlets need to be operated as a business. ● Strongly Agree ● Agree USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 66 ● Disagree ● Strongly Disagree 1a. Comment (please explain) 2. Through SMS assistance, I have gained skills and capacity to strengthen my media outlet’s business operations. ● Strongly Agree ● Agree ● Disagree ● Strongly Disagree 2a. Comment (please explain) 3. How did SMS support affect your media outlets’ revenue (Select all that apply)? ● No change ● Sources of revenue diversified ● Less dependent on donor funds ● Small increase in annual revenue (10% or less) ● Moderate increase in annual revenue (11%-50%) ● Large increase in annual revenue (61% or more) 4. Which of the following income generating strategies, if any, did SMS help you to apply? (Select all that apply) ● E-payment gateway ● Subscription or membership model ● Donation/crowdfunding model ● Advertising ● Google AdSense optimization ● Patreon membership model ● Alternative revenue streams (revenue from services, side businesses) ● Other (please specify) ● None of the above 5. Please choose the three income generating strategies that you feel were most effective in helping create sustainable revenue for your organization. ● E-payment gateway ● Subscription or membership model ● Donation/crowdfunding model ● Advertising ● Google AdSense optimization ● Patreon membership model ● Alternative revenue streams (revenue from services, side businesses) ● Other (please specify) ● None of the above 67 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV 5a. Comment (please explain) 6. Which of the following types of support did you receive to improve your content delivery? ● Improved website design ● Website optimized for mobile technology ● Developed podcast ● Developed newsletter ● Other (please specify) ● None of the above 7. Which of the following types of support did you find most useful to improving your content delivery? (select one) ● Improved website design ● Website optimized for mobile technology ● Developed podcast ● Developed newsletter ● Other (please specify) ● None of the above 8. SMS helped my media outlet or association develop new business partnerships or networks. ● Strongly Agree ● Agree ● Disagree ● Strongly Disagree ● Not applicable 9. What are examples of new business partnerships or knowledge sharing networks that SMS helped you to form. In what ways if any were these partnerships or knowledge sharing networks beneficial for your organization? 10. SMS activities were effective in building the capacity of diverse media outlets, including local and regional, women-led, and ethnic minority media entities. ● Strongly Agree ● Agree ● Disagree ● Strongly Disagree ● Not applicable 11. What are some examples of how SMS activities built the capacity of diverse media outlets, including local and regional, women-led, and ethnic minority media entities? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 68 12. Please provide examples of ways, if any, that you are using data to better understand your audience. 13. Which of the following approaches, if any, did SMS help you to implement to improve your audience outreach? (Select all that apply) ● Improved use of social media (i.e., Facebook, Twitter, Instagram, etc.) ● Content insights/Smart octo metrics ● Other research (please specify) ● Other data analysis tools (please specify) ● None of the above 14. Which of the following approaches did you find most useful for improving your audience outreach? (select one) ● Improved use of social media (i.e., Facebook, Twitter, Instagram, etc.) ● Content insights/Smart octo metrics ● Other research (please specify) ● Other data analysis tools (please specify) ● None of the above 15. What are the most important challenges affecting media outlets’ ability to survive financially? Please rank the following: ● Unfair advertising market ● Facebook/google quasi monopoly ● Lack of access to state funding ● Lack of access to donor funding ● Lack of access to investment funds to improve media platforms ● Lack of internal expertise in marketing, business management ● Limited willingness of public to fund independent media ● Poor enabling environment for e-commerce in Serbia ● Other (please specify) 16. The National Association for Ethical Standards in Advertising (NAESA) is a useful initiative to promote self-regulation in media advertising in Serbia. ● Strongly Agree ● Agree ● Disagree ● Strongly Disagree ● Not applicable 17. What is necessary for the NAESA to be more useful? Please rate your level of satisfaction with the following: 18. SMS grants 69 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ● Extremely satisfied ● Satisfied ● Unsatisfied ● Extremely unsatisfied ● Not applicable 18a. What could have been done better? 19. Technical assistance ● Extremely satisfied ● Satisfied ● Unsatisfied ● Extremely unsatisfied ● Not applicable 19a. What could have been done better? 20. Events (i.e., Media Jam, Summer School, etc.) ● Extremely satisfied ● Satisfied ● Unsatisfied ● Extremely unsatisfied ● Not applicable 20a. What could have been done better? 21. Convening mechanisms/Consultations (i.e., nodes, Media talks, etc.) ● Extremely satisfied ● Satisfied ● Unsatisfied ● Extremely unsatisfied ● Not applicable 21a. What could have been done better? 22. I believe the support provided by SMS will result in sustainable improvements in my media organization’s revenue ● Strongly Agree ● Agree ● Disagree ● Strongly Disagree ● Not applicable 23. What if anything could be done to improve the likelihood of sustainability? 24. What are the three most important external threats to independent media in Serbia? USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 70 ● Threats against journalists ● Limits in the application of laws on access to information ● Journalists’ poor pay and working conditions ● Political-oriented persecution of independent media outlets ● Unfair allocation of state resources to pro-government media ● Unfair media market for advertising ● Other (please specify) ● None of the above 25. To what extent do you think that by helping independent media reinforce their financial sustainability, SMS is helping independent media voices thrive? ● To a great extent ● Somewhat ● Very little ● Not at all 71 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ANNEX D. ACTIVITY BACKGROUND SMS ACTIVITY BACKGROUND Based on international press freedom watch dog and research organization reports, media freedoms have declined in Serbia over the last decade. Despite having a constitution that guarantees freedom of the press, Serbia ranks 93rd out of 180 countries on the 2021 World Press Freedom Index compiled by Reporters without Borders, and 89th out of 161 according to the 2019/2020 Global Expression Report. Pressure from political and economic interests, threats to independent journalists, and out-of-date and unsustainable business models are among the challenges reported to be hindering the ability of media outlets in Serbia to produce unbiased and timely information for the Serbian public. In line with Serbia’s ambitions of joining the European Union by 2025 and following a prolonged consultative process marred by disagreements with civil society participants, the Serbia Government adopted the Strategy for the Development of Public Information System for the period 2020 – 2025 in January 2020. Known as the Media Strategy, the document outlines planned actions to reform laws and regulations and improve the enabling environment for independent media in the Republic. SMS ACTIVITY GOAL AND OBJECTIVES In 2017, USAID/Serbia awarded the $6.5 million SMS Activity to IREX, Inc. The goal of the SMS Activity is to improve the legislative, regulatory, and economic environment for Serbia’s media, so that Serbian citizens have access to independent, reliable, and timely information that they can use to hold their government accountable. Toward this goal, the SMS Activity aims to achieve two interrelated objectives: 1. Support the development of a more enabling legal, regulatory and market environment for media in Serbia; and 2. Support the development of financial sustainability, innovation, and partnerships in the Serbian media sector. During the first year of implementation, the SMS Activity, in discussion with USAID, agreed to limit the focus of activities under objective one to supporting the development of the business enabling environment for media outlets in Serbia56. Starting in year two, USAID/Serbia mandated the OSCE, which also receives USAID funding, to promote broader legal and regulatory reforms as part of its activities to support the Media Strategy development process57. SMS retains its mandate under Objective 2 to support independent media outlets to develop innovative business approaches, including the design and implementation of new digital tools, and to promote business networking and cooperation. Additionally, SMS scope includes providing support to media outlets for the implementation of specific policies and regulations from the Media Strategy’s Action Plan. The SMS Activity began in September 2017 with an anticipated end date of September 2021. Due to COVID-19 related delays, USAID/Serbia extended the implementation period until February 2022. SMS THEORY OF CHANGE The SMS theory of change states: 56 IREX SMS Year One Workplan. 2017. IREX. 57 IREX SMS Year Two Workplan. 2018. IREX. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 72 If relationships within the media system are strengthened and the capacities of system actors are improved, the Serbian media system will be able to develop a more favorable enabling environment and adapt to the realities of the digital market, enabling diverse, objective, and professional news and information to survive and thrive in Serbia’s media market. 73 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ANNEX E. DETAILED METHODOLOGY IMPAQ applied a mixed methods final performance evaluation design, integrating quantitative and qualitative data collection and analysis methods, including secondary data analysis of MEL and external data, KIIs, FGDs, and an online survey with current and former SMS grantees, to answer the EQs and provide USAID/Serbia with an assessment of the Activity’s performance against their targets, why targets were or were not met, and reasons for gaps in performance. Using this approach, IMPAQ was able to help USAID: (1) examine the effectiveness of implemented activities, (2) determine whether the Activity achieved the planned results, (3) identify successes and gaps in performance against targets, and (4) provide information on whether the Activity has unintended consequences. Furthermore, IMPAQ was able to provide USAID/Serbia with key insights into the strategies, lessons learned, and promising practices to support a more enabling market environment for media and strengthen the capacity of media system actors to provide information effectively and sustainably to citizens. DATA COLLECTION TOOLS AND INSTRUMENTS To answer the EQs, the evaluation team used a mixed-method approach, integrating qualitative and quantitative approaches to data collection, analysis, and interpretation. GENDER EQUALITY AND SOCIAL INCLUSION (GESI) CONSIDERATION Given IMPAQ’s commitment to integrating gender equality and social inclusion (GESI) considerations across all its work, our data collection instruments critically examined SMS’s approach through a GESI lens. Reflecting GESI best practices, our approach integrated the key concerns of women, youth, LGBTQI, and other marginalized groups – not only by assessing the root drivers that inhibit their access to, or ability to benefit, from SMS activities; but also, through careful attention to how SMS interventions protected against unintended consequences and incorporate mechanisms that protect these vulnerable groups within the Activity’s scope of work. In addition to addressing the EQs, we assessed the extent to which SMS implementation reflected gender-aware programming, meaningfully engaging women, youth, and other marginalized groups not only in SMS activities but also in engaging in partnerships and collaborations with other media sector actors – thus promoting the long-term sustainability of a more equitable Serbia. Across all EQs we disaggregated data by sex and analyzed findings to understand the constraints and opportunities affecting both men, women, and beneficiaries from other marginalized groups. Additionally, we conducted all activities within a GESI-sensitive approach, such as scheduling KIIs or FGDs that are sensitive to different groups’ availability and ensuring representation among respondents. The following data collection tools and instruments were used during the evaluation: DESK REVIEW: The evaluation team conducted a thorough desk review of background information and Activity documentation to gain an in-depth understanding of SMS activities and to assess the availability and quality of secondary data for answering the EQs. As part of this process, the evaluation team mapped the main documents and key partners and stakeholders to ensure that all the crucial elements were considered and analyzed. The following activities were conducted as part of the desk review: ● Review of the SMS scope of work, annual workplans, and MEL plan, with focus on the intervention logic and the results framework, including indicators and benchmarks (baselines, targets, and sources of verification); ● Analysis of SMS quarterly progress and monitoring reports; USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 74 ● Review of other reports that the SMS team produced and provided, including SNA, Strategic Media Market Trends Analysis, Demand-Side Study, Political Economy Analysis (PEA), Gender Analysis, etc. ● Analysis of products (reports and analysis) from other organizations and development partners in the area of media development, including other USAID-funded media development programs in the region, and identify possible best practices or lessons learned that could apply to future USAID/Serbia media programming or policy; ● Review of national policies and strategies around media development in Serbia with a specific focus on the new Media Strategy and initiatives to improve media measurement, simplify online payments for media outlets, establish self-regulatory bodies within the media industry, and continue professional development for journalist and media professionals; and ● Analysis of international practices and the international obligations of Serbia, especially considering the EU accession process for the country. The desk review continued throughout the evaluation, with information and evidence organized by EQ to allow for structured data processing and analysis. The desk review informed the final EQs methodology, sampling, and data collection instruments. Desk review evidence was also used to triangulate findings from the other data analysis approaches. SECONDARY DATA ANALYSIS OF MEL AND EXTERNAL DATA: IMPAQ reviewed the SMS Activity MEL data as a starting point for assessing Activity progress towards achieving the stated objectives and targets, including the extent to which gender related variables were tracked, where applicable. IMPAQ assessed MEL data availability and relevance to determine which data could be used to answer the EQs58. Using independent data sources where appropriate and available, for example by obtaining media analytics data (e.g., data on audience engagement, revenue, etc.) from beneficiary media actors, the evaluation team conducted an independent analysis of which targets were met or significantly surpassed and identified reasons behind any performance shortfalls. Reasons for such shortfalls were further explored through qualitative data collection. Where possible, IMPAQ reviewed ad-hoc and qualitative monitoring reports conducted by the SMS Activity to help inform if and how the Activity adjusted to the changing environment, including identified challenges and opportunities. ONLINE SURVEY: In 2018, the SMS Activity conducted a social network analysis (SNA) of the media system in Serbia. The SNA focused on uncovering patterns of interaction between the media stakeholders targeted by the project. The specific objective of the 2018 SNA was to gain a comprehensive understanding of the relationships among media support organizations and civil society organizations (CSOs) working in the media reform space and to identify strong existing collaborations, as well as gaps in collaboration that the SMS Activity could address more strategically. 59 Given the Activity’s theory of change, the evaluation team originally proposed to conduct a net-map as part of the data analysis methods. A net-map, a tool used for SNA, is a visual representation of the key players and their relationships within a system. The evaluation team intended to construct two net-maps, (1) an initial net-map of the media system as it was at the beginning of the Activity in 2018 and (2) a current net-map of the media system at the time of the evaluation. To do this, the evaluation team was going to administer an online SNA survey, based on the SNA questionnaire used in 2018, to collect perspectives from current and former SMS media partners. However, due to the Activity’s shift in strategy in Year 2 58 Activity AMELP indicators were changed significantly in Year 2, as such, the evaluation team will mainly analyze available data based on the updated indicators. 59 SMS Social Network Analysis Report 75 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV to focus more on financial sustainability (objective two) than network and systems strengthening (objective one), the evaluation team, in consultation with USAID, determined that the net-map was no longer an appropriate analysis approach for answering the EQs. Instead of conducting an online SNA survey, the evaluation team developed and administered a short online survey to current and former SMS grantees. The purpose of the survey was to better understand how the Activity supported media organizations in the areas of financial sustainability, audience outreach and engagement, and partnership and collaboration with other media sector actors. Complemented with qualitative data collected through KIIs and FGDs, the survey was intended to help the evaluation team better understand the effectiveness of SMS interventions. KIIS: IMPAQ developed KII guides (refer to Annex C) tailored to each of the key stakeholder groups. The KIIs were used to delve deeper into understanding how changes occurred over the life of the Activity. The KIIs also assessed: (1) progress towards achieving the objectives; (2) the extent to which the activities addressed the challenges within Serbia’s media sector; (3) how well the Activity adapted to the changing context, including the impact of COVID-19; and (4) how well the Activity influenced the use of new approaches to strengthening media sector sustainability. Finally, the KIIs elucidated promising approaches to address gender and other social barriers within the media sector. KIIs followed a semi-structured approach to ensure comparability while allowing for follow-up and probing on topics of interest. Protocols were adapted as relevant to each stakeholder. FGDS: IMPAQ developed tailored FGD guides (refer to Annex C). FGDs were conducted with SMS partners/grantees to better understand and validate changes, what factors facilitated or hindered change, and whether the changes are sustainable, with regards to improving the business enabling environment To assess whether the Activity addressed challenges and leveraged opportunities to promote sector financial sustainability, IMPAQ inquired about approaches or strategies SMS used to support beneficiaries with digitization, content monetization, and innovative business models. IMPAQ also explored if and how the Activity adapted to the changing context, including whether the Activity helped beneficiaries adapt to the COVID-19 context. During all FGDs, IMPAQ facilitated a discussion around specific barriers women and other marginalized groups face in the media sector, to what extent the Activity helped address those barriers, and what approaches, if any, show promise for improving gender equality within the media sector. FGDs followed a semi-structured approach to ensure comparability while allowing for follow up and probing on topics of interest. Protocols were adapted as relevant to each stakeholder group. SAMPLING PLAN Based on a review of the SMS Activity documents and in consultation with USAID and SMS staff during the Design Workshop and Kick-Off Meeting, the evaluation team determined an appropriate sample size and sampling strategy to identify respondents for the online survey, KIIs, and FGDs. IMPAQ employed a purposive sampling approach to collect targeted information from specific stakeholders, to provide insights and perspectives that were used to identify useful findings and develop actionable recommendations. IMPAQ uses purposive sampling approaches to ensure that only necessary and relevant data is collected and to maximize field data collection efforts. In addition, the evaluation team employed a snowball sampling approach to enrich the KII sampling frame. ONLINE SURVEY: IMPAQ distributed the online survey to 52 current and former SMS grantees or partners in the media sector. This list was validated with the SMS Activity to ensure that all relevant SMS partners and grantees were included. Twenty-five individuals (13 female, 10 male, and 2 nondisclosed) completed the survey, for a response rate of 48 percent. Survey respondents included representatives from seven non-governmental organizations, five media associations, three national media outlets, nine regional or local media outlets, and one other. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 76 KEY INFORMANT INTERVIEWS: IMPAQ planned to conduct KIIs with a sample of 28 key stakeholders and informants, however based on information collected throughout the evaluation, IMPAQ expanded this sample to include a total of 31 KIIs, three more than originally planned. KIIs included relevant representatives from the US Government and USAID implementing partners, the Government of Serbia, SMS Activity staff, and representatives from other donors and actors engaged in the media sector (including but not limited to media associations and NGOs, other international donors funding or implementing media programming in Serbia, and academia). Included in the KII sample were several SMS Activity partners and grantees that received unique or specific support from the Activity that other partners or grantees did not receive. For example, one of the key informants, 021, received “Anchor Grants” from the SMS Activity. “Anchor Grants” are 12-month grants of substantial size and scope, designated for the most advanced SMS partners. These “Anchor Grants” serve as staged investments with concrete milestones and targets rather than traditional media support grants60. IMPAQ conducted KIIs with these stakeholders rather than including them in the FGDs, discussed in the following section, as they have unique perspectives and interactions with the SMS Activity that IMPAQ felt were important to capture as part of the evaluation. Exhibit 12 provides a list of actual respondents. Given COVID-19 restrictions, all KIIs were conducted remotely. 60 IREX SMS Year Four Workplan. 2020. IREX. 77 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV EXHIBIT 12. KII SAMPLE STAKEHOLDER TYPE RESPONDENTS NUMBER OF STAKEHOLDERS INTERVIEWED RATIONALE USG and USAID Implementing Partners ● USAID/Serbia SMS Contracting Officer’s Representative (COR) ● USAID Balkan’s Media Assistance Program ● USAID/Serbia Contracting Officer’s Representative (COR) for the OSCE Activity/Alternate COR for the SMS Activity 3 Key USAID/Serbia stakeholders involved in the design, implementation, and oversight of the SMS Activity provided insights into the effectiveness of implementation, how well the activity identified and addressed changing conditions and circumstances within the media sector, how the Activity influenced other donor initiatives, and whether the Activity addressed gender considerations. Representatives from the USAID Balkan’s Media Assistance Program provided critical insights into how the Activity coordinated with and promoted best practices and lessons learned within the broader media landscape. Government of Serbia ● Ministry of Culture and Information 1 Representatives from the Government of Serbia provided the latest updates on implementation of the Media Strategy in order to help identify if and how it could contribute to financial sustainability of the media, as well the Chamber of Commerce to analyze country business potentials and investments in the media sector. SMS Staff ● COP ● Deputy Chief of Party ● Digital Media Lead ● Project Officer ● MEL Manager/Adviser , 5 SMS Activity staff provided insight into progress towards achieving the Activity targets and objectives, how the activity identified and addressed changing conditions and circumstances within the media sector, how the Activity influenced other donor initiatives, and how the Activity addressed gender considerations. NGOs, Media, Startups, Media Service Providers ● Catalyst ● National Association for Ethical Standards in Advertising ● 021 ● Juzne vesti ● Smart Octo (former Content Insights) ● Podcast.rs ● Impact Hub 7 The list includes SMS partners who were critical in delivering trainings, served as flagship Activity partners (National Association for Ethical Standards in Advertising) and/or anchor leaders of business network hubs for smaller outlets in their respective regions and Startups. These stakeholders provided insights into what methods and approaches were most effective in promoting innovative and sustainable business models. USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 78 Market Research ● IPSOS ● CESID 4 Research partners involved in the SMS Activity identified if and how the Activity identified and responded to the needs on the ground, and which approaches, or methods show the most promise for promoting sustainability within the media sector. Media Experts ● Economist/Media Expert ● Business Media Expert ● Media Marketing Expert ● Lawyer/Expert in Media Law 4 Experts engaged in delivering legal and market analysis identified if and how the Activity responded to the needs on the ground, and which approaches, or methods show the most promise for promoting sustainability within the media sector. Other donors ● European Union Delegation in Serbia, ● Organization for Security and Cooperation in Europe (OSCE) ● British Council 4 Donor coordination group identified if and how the Activity facilitated or promoted the use of new or innovative approaches or best practices in media sector development in order to promote sustainability. Academia ● University of Belgrade, Faculty of Organisational Science ● University of Belgrade, Faculty of Political Science, Media Department ● University of Belgrade, Faculty of Economics 3 Relevant academic informants and key experts in the media space provided insights into how the media landscape has changed over the life of the Activity and what approaches and methods show the most promise for promoting sustainability. 31 (11 female and 20 male) 79 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV FOCUS GROUP DISCUSSIONS: IMPAQ planned to conduct six FGDs based on the different types of media system actors the SMS Activity engaged, however, to accommodate schedules and respondents preferred language, IMPAQ conducted seven FGDs with 26 participants. FGD participants included representatives from media associations, traditional media outlets, investigative reporting, local/regional media, and new media portals/podcasts. When determining the composition of FGDs, IMPAQ grouped SMS grantees and partners by the different project activities they engaged in or support services they received as part of the SMS Activity. Organizing the FGDs in this manner, and separating FGDs by media type, allowed IMPAQ to delve deeper into if and how the SMS Activity provided relevant and targeted support to each actor. IMPAQ sought to ensure gender parity within the FGDs to the extent possible. Given COVID-19 restrictions, FGDs were conducted remotely and with four to six participants per group to allow for in-depth discussion in a remote format. To avoid domination by the more influential and eloquent participants, we limited the discussion time for each FGD question and ensured all participants had a chance to answer the questions. EXHIBIT 13. FGD SAMPLE STAKEHOLDER TYPE RESPONDENTS Media Associations ● Independent Journalists’ Association of Vojvodina (IJAV) ● Journalists’ Association of Serbia (UNS) ● Media Association (ASMEDI) ● Local Press ● Association of Online Media Traditional Media ● BETA ● FONET ● Vreme ● Danas ● Nova Ekonomija Investigative Reporting ● CINS ● KRIK ● Peščanik ● Pistaljka ● Istinomer Local/Regional Media ● Bujanovacke Portal ● Loznicke Novosti ● Zajecar Online ● Boom 93 ● Pressek Kragujevac ● Ozon Media Cacak ● Glas Šumadije, ● TV Forum Prijepolje ● Liceulice Portals/Podcasts ● Libek, Talas ● Maglocistac, part of NDNV Media Hub ● Storyteller ● Autonomija TOTAL 28 participants (15 female and 13 male) USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 80 DATA ANALYSIS IMPAQ took an iterative approach to data analysis, incorporating and analyzing different data sources as the evaluation progressed, ensuring each phase of data collection captured the most relevant and appropriate information to answer the EQs. DESK REVIEW AND SECONDARY DATA ANALYSIS: Data analysis began during desk review as the team identified and documented key progress made towards reaching the Activity’s objectives and targets. A review of MEL data revealed which indicators the Activity has made progress on, as well as those where the Activity may have fallen behind. IMPAQ also reviewed monthly grantee data on revenues and audience outreach which contributed to the team’s evaluation of progress against the objectives and indicators. A review of annual workplans, reports, and activity briefs, helped identify if and how the Activity adapted to the changing context on the ground, including changes related to COVID-19. ONLINE SURVEY ANALYSIS: The data collected through the online survey of current and former SMS grantees and partners was used to provide additional insights into grantee and partners perceptions around the effectiveness of SMS interventions, including activities and support that they found most useful to promoting the financial sustainability of their organization. This data was triangulated with MEL data as well as KII and FGD data to provide a comprehensive understanding. KII AND FGD ANALYSIS: The qualitative data collected through the KIIs and FGDs was used to delve deeper into understanding how changes occurred over the life of the Activity. With the informed consent of the interviewees, KIIs and FGDs were audio-recorded for accuracy. For each KII and FGD, the team took notes using a semi-structured protocol to capture key inputs and noteworthy statements. Following transcription, the team conducted content analysis to summarize findings and analyze them for common themes based on the EQs. The team’s preliminary analysis was then incorporated into a validation session to draw out final findings and conclusions with the evaluation’s COR and alternative COR. Following the validation session, IMPAQ completed its data analysis and held an Out-Briefing with USAID/Serbia. The Out-Briefing session provided USAID an opportunity to provide input to help the evaluation team refine and prioritize its recommendations, ensuring they are useful for informing future programming and policy. 81 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ANNEX F. ANALYSIS OF KEY PERFORMANCE INDICATORS Indicator Base line Targe t Yr. 361 Yr. 4 Percentage change in the development of a more enabling legal, regulatory, and market environment for media in Serbia 2.75 5% 0 1.81 Percentage of media system actors assisted through SMS that apply gained knowledge and skills 0 30% 100% 85% Number of USG-assisted media-sector civil society organizations (MSOs) and/or institutions that serve to strengthen the independence media or journalists (DR.5.3-2) 0 7 8 5 Number of civil society organizations receiving USG assistance engaged in advocacy interventions (DR.4.2-2) 0 5 5 8 Percentage of SMS supported media outlets that show improved financial sustainability62 0 25% 100% 98% Number of media system actors that receive assistance on content format, promoting media partnerships, developing management and financial strategies 0 25 35 55 cum Number of non-state news outlets assisted by USG (DR.5.3-1) 0 15 31 44 cum Percentage of participants in SMS technical assistance activities that expanded their online audience 0 33% 94% 100% Number of private sector firms that have improved management practices or technologies as a result of USG assistance (EG.5.2-2) 0 15 15 17 cum Percentage increase of sales or donations of firms receiving USG-funded assistance (EG.5-1b) 0 15% 47% TBD63 Percentage of participants in SMS activities (such as Digitizer and MSM) that develop new business strategies and models to monetize content 0 33% 57% 98% 61 Given the change in scope in Year 2, reporting on these indicators did not start until Year 3. 62 Tracked percentage of media outlets that showed increased income earned through the revenue streams supported/developed through SMS. 63 In year three, SMS reported an average increase in sales of 15 media outlets registered as firms is 47.04%. Calculated as: (Y3-Y2)/Y2*100. Highest increase – Nova ekonomija (270.47%), lowest – Loznicke Novosti (­ 32.98%) USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 82 ANNEX G. LIST OF DOCUMENTS REVIEWED MONITORING AND EVALUATION ● SMS Monitoring, Evaluation, and Learning (MEL) Plan ● SMS Year 3 AMELP Tracker ● Media business development Index - Project Year 2-3 ● Success Study - Danas Case Study SMS QUARTERLY PROGRESS REPORTS ● FY 2018-Q1 (Oct-Dec 2017) ● FY 2018-Q2 (Jan-March 2018) ● FY 2018 Q3 (April-June 2018) ● FY2018 Q4 (July-Sep 2018) ● FY2019 Q1 (Oct-Dec 2018) ● FY2019 Q2 (Jan-Mar 2019) ● FY2019 Q3 (April-Jun 2019) ● FY2019 Q4 (Jul - Sept 2019) ● FY2020 Q1 (Oct - Dec 2019) ● FY2020 Q2 (Jan-Mar 2020) ● FY2020 Q3 (Apr-June 2020) ● FY2020 Q4 (July-Sep 2020) ● FY2021 Q1 (Oct-Dec 2020) ● FY2021 Q2 (Jan-Mar 2021) RESEARCH PRODUCTS ● SMS Social Network Analysis Report ● SMS Political Economic Analysis Report ● SMS Demand Side Report Serbia ● USAID Serbia Gender Analysis Report 83 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV ● Strategic Media Market Trends Analysis ● Rapid Assessment of Media Portfolio WORKPLANS ● SMS Year One Workplan ● SMS Year Two Workplan ● SMS Year Three Workplan ● SMS Year Four Workplan GRANTEE REPORTING ● Grantees SMS Monthly Tracker ● BETA Narrative Reports ● Nova Ekonomija Narrative Reports ● BIRN Narrative reports ● Istinomer Narrative Reports ● NDNV Narrative Reports ● Vreme Narrative Reports ● Asmedi Narrative Reports ● Radio 021 Narrative Reports ● NAESA 01 Narrative Reports ● Bujanovacke Narrative Reports ● FoNet Narrative Reports ● Danas Readers Club Narrative Reports ● Loznicke Narrative Reports ● PESCANIK Narrative Reports ● KRIK Narrative Reports ● Pressek Narrative Reports ● Danas Daily Narrative Reports USAID.GOV FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY | 84 ● NUNS Narrative Reports ● UNS Narrative Reports ● IMPACT Hub Reports OTHER PROJECT DOCUMENTS ● NAESA Activity Description and Progress (March 2019) ● SMS USAID Presentation (January 2019) ● Brief on SMS ● List of SMS partners ● SMS Quick Facts ● SMS Grants Manual ● SMS E-payment Progress Report 85 | FINALE PERFORMANCE EVALUATION OF SMS ACTIVITY USAID.GOV