Performance Evaluation of the Alternative Resources in Media (ARM) Project Mid-term Evaluation Report January 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Arnold R. Isaacs and Suren Deheryan through Social Impact, Inc. Cover Photo: Master class for cameramen in convergent media environment: a well-known Ukrainian cameraman Mikhayil Lebedev explaining the structure of the camera. Photograph courtesy of ARM Project, April 2012. PERFORMANCE EVALUATION OF THE ALTERNATIVE RESOURCES IN MEDIA (ARM) PROJECT Mid-term Evaluation Report USAID/ARMENIA January 25, 2013 Contract # AID-RAN-I-00-09-00019 Task Order # AID-111-TO-12-00002 Evaluation of Five USAID/Armenia Activities DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENTS The evaluation team is grateful to the ARM chief of party and the heads of the three consortium partners for their cooperation in providing information and candidly responding to our questions. Staff members of Internews Armenia, Eurasia Partnership Foundation, and the Yerevan Press Club were similarly open and helpful, as were the partners and beneficiaries whom we interviewed for this report. We thank them all for their assistance. 4 CONTENTS Acronyms............................................................................................................................................................................. 6 Executive Summary ........................................................................................................................................................... 7 Evaluation Purpose & Evaluation Questions.............................................................................................................. 12 Project Background......................................................................................................................................................... 13 Evaluation Methods & Limitations............................................................................................................................... 16 Findings, Conclusions & Recommendations.............................................................................................................. 18 Key Question #1......................................................................................................................................................... 18 Findings...................................................................................................................................................................... 18 Conclusions.............................................................................................................................................................. 19 Key Question #2......................................................................................................................................................... 20 Findings...................................................................................................................................................................... 20 Conclusions.............................................................................................................................................................. 22 Key Question #3......................................................................................................................................................... 24 Findings...................................................................................................................................................................... 24 Conclusions.............................................................................................................................................................. 25 Key Question #4......................................................................................................................................................... 26 Findings...................................................................................................................................................................... 26 Conclusions.............................................................................................................................................................. 27 Key Question #5......................................................................................................................................................... 28 Findings...................................................................................................................................................................... 28 Conclusions.............................................................................................................................................................. 28 Gender...................................................................................................................................................................... 29 Recommendations...................................................................................................................................................... 30 Annexes ............................................................................................................................................................................. 32 Annex I: Evaluation Statement of Work................................................................................................................ 33 Annex II: Evaluation Methods and Limitations..................................................................................................... 42 Annex III: Sources of Information........................................................................................................................... 44 Annex IV: Data Collection Instruments................................................................................................................ 49 Annex V: Disclosure of Any Conflicts of Interest............................................................................................... 51 5 ACRONYMS ARM Alternative Resources in Media FY Fiscal Year IMS Internews Media Support IREX International Research & Exchanges Board KQ Key Evaluation Question MAG Media Advocacy Group MEO Media Ethics Observatory NGO Non-governmental organization SOW Statement of Work USAID U.S. Agency for International Development 6 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The main purpose of this mid-term performance evaluation is to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Specifically, findings from the mid-term evaluation are designed to inform future work plans of the relevant projects, as well as designs of future similar activities. The evaluation findings are intended to be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. Relevant recommendations from this evaluation may be used by project stakeholders in developing future work plans. The evaluation addresses a set of five Key Questions (KQs) focused on identifying (a) what changes should be made in the approaches and directions of the current program, and (b) what recommendations can be made for the design of future media programs. The KQs, as stated in the Evaluation Statement of Work (SOW) are: (1) How relevant has the ARM project intervention been to the media field challenges in Armenia? (2) To what extent has the project contributed to creating opportunities for (a) a diversity of alternative sources of information and opinion, (b) increased access to alternative sources of information and opinion, and (c) advocacy for improved enabling environment for media independence? (3) What has the project done or mechanisms has it created to ensure that the project products will be sustained? (4) What have been the added value and/or drawbacks of the project management structure (consortium of three local organizations with an unaffiliated Chief of Party)? (5) How successful has the project been in leveraging resources of (a) the skills and organizational strengths of the consortium members, and (b) private and other donor partnerships? PROJECT BACKGROUND The Alternative Resources in Media (ARM) is a $4 million, four-year program now entering the second half of its period of implementation. The program has three objectives: (1) "supporting production and dissemination of alternative content through multiple high and low tech platforms"; (2) "building consumer demand for news and information by promoting media literacy, critical analysis, and use of latest web and mobile-based media applications; and (3) "encouraging public advocacy on media-related issues. The broader goal, as stated by USAID, is to “assist citizens to transform themselves from passive users to active information producers in an otherwise controlled media environment.” The ARM project’s inception and implementation has coincided with two recent nationwide trends. One is widespread distrust of news in traditional media, particularly television, which is commonly viewed as controlled by state authorities and powerful business interests linked to them. The other trend is an explosion of “new media” and a rapid increase in Internet use. The change is dramatically highlighted by 7 research indicating that the number of Armenians who receive news from Internet sources may now be equal to the audience for TV news shows. The first trend increases demand for alternative information, while the second increases the means to transmit it and people's ability to receive and use it. Thus, both phenomena make ARM a timely and relevant endeavor. ARM's management is a four-legged stool: three Armenian partners (Internews Armenia, Eurasia Partnership Foundation, and the Yerevan Press Club) and a Chief of Party not affiliated with any of the partners. Although the leadership operates as a consortium and commonly uses that term to describe itself, in fact it is not formally a consortium. The recipient of the USAID grant is the international U.S.- based Internews organization, which also hired the Chief of Party, and the Armenian partners are technically sub-grantees. In practice, however, they have functioned from the start as members of a consortium – a word that will be used in this evaluation with the understanding that it refers to a de facto arrangement and not a formal one. To avoid confusion, the name Internews-U.S. will be used for the international Internews (though not its formal name) and Internews Armenia for the Armenian organization (also referred to in ARM documents as Internews Media Support, or IMS). ARM's activities are profuse and diverse. Significant projects include: • Establishing regional information centers (known as Infotuns) that offer computer literacy training and online interaction opportunities to residents in five regional towns. • Multiple training programs for professional journalists, "citizen contributors," bloggers, social media users, etc. • Grants to individuals and organizations for a broad range of projects aimed at widening the audience for alternative information and giving citizens a platform to make their voices heard on community and policy issues. Among these are grants assisting two traditional news organizations navigate the transition to multimedia coverage, and one in support of the "i￾Ditord" ("i-Observer") project that helped journalists, election observers and voters use alternative media resources to monitor and report abuses or irregularities in the May 2012 parliamentary election. • Drafting a proposed media literacy curriculum for use in Armenian schools. This is one of numerous activities aimed at raising awareness and increasing public knowledge and understanding of alternative media sources. • Numerous activities to monitor media and encourage public awareness and discussion of media issues, advocate greater media freedom, and promote self-regulation of disputes in place of legal action. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation team employed a comprehensive qualitative evaluation design consisting of thorough program document review, key informant interviews with program beneficiaries and stakeholders, and direct observation of ARM project sites. The findings of this evaluation are based primarily on interviews – nearly 50 in total – with program organizers and managers, beneficiaries, and expert observers; review of work plans, progress reports, and other relevant documents; and visits to selected sites (including two outside Yerevan) where ARM or its grant recipients are conducting or have carried out a variety of project-related activities. Interview subjects were selected in an attempt to capture as wide a variety of involvement in differing media activities and different perspectives on ARM and the Armenian media 8 environment as possible. Key informants included the Chief of Party and the heads of the three organizations primarily responsible for designing and implementing the program (Internews Armenia, Eurasia Partnership Foundation, Yerevan Press Club). Also interviewed were staff members of those organizations who are the coordinators for specific program areas -- e.g. grants, regional information centers, media monitoring, proposed media literacy curriculum for schools, advocacy for legal reform and voluntary self-regulation, etc. -- and selected recipients of grants, both those awarded to individuals and those given to existing media or civil society organizations. To ascertain the subjects and range of alternative information the program has helped create, the content of several websites supported by ARM grants was also reviewed by the evaluation team. Given the short amount of time allotted for this evaluation, data gleaned from interviewees is subject to some degree of selection bias. To mitigate this bias, as previously stated, the evaluation team made every attempt to collect data from subjects representative of the media landscape as a whole. This was done in consultation with USAID staff and through thorough review of project reports and other documents. ARM's goal, broadly stated, is changing the consciousness of the Armenian public. By definition, this is not subject to exact measurement. Nor is it possible to isolate the effects of a particular program from those of broader changes in the media world and the public life of Armenian society. Thus the available quantitative data, such as numbers of people receiving training, or the count of visitors to ARM￾supported online media sites, cannot constitute a comprehensive measurement of ARM's mid-term results. While it is possible to identify particular news stories initiated by ARM-supported projects that have led to citizen action challenging government decisions, such examples only serve to illustrate, rather than quantify, a much broader trend in the public life of the country. A further limitation is that ARM's activities are too numerous and varied for an evaluation to independently examine more than a limited number of activities in the time available. The evaluation team’s findings and conclusions are thus necessarily based on examples, rather than the full range of activities -- e.g., visits to two of the five regional information centers and interviews with eight of 33 grant recipients. SUMMARY OF FINDINGS AND CONCLUSIONS FINDINGS ARM has conducted multiple training programs for professional journalists, "citizen contributors," bloggers, social media users, etc., while awarding grants to individuals and organizations for a wide range of projects aimed at widening the audience for alternative information and giving citizens a platform to make their voices heard on community and policy issues. ARM also assists traditional news organizations navigate the transition to multimedia coverage, and has developed a proposed media literacy curriculum for use in Armenian schools. Most of ARM’s projects are integrated in existing structures that were well established and active before the ARM program began and are expected to continue after ARM ends. In some cases, the sustaining organization will be one of the three consortium members. In others, it will be one of the many groups ARM has partnered with to carry out different parts of the program. Regional information centers (known as Infotuns, or "info-houses") have offered computer literacy training and online interaction opportunities to residents of all ages in five regional towns. Through the Infotuns, in ARM's first two years of operation, 2,800 citizens received training in new media tools, basic computer skills, and citizen reporting. In addition, ARM organized a variety of community meetings and discussions on media issues and self-regulation. 9 Leaders of partner organizations have known each other for many years, have cooperated in many ways in the past, share common values and goals, and have strong mutual respect and trust. They report that they have been able to resolve different views without damaging their relations, and have carried out their parts of the project without competing for resources. An additional significant benefit of the consortium structure is that joint efforts and financial and organizational support from ARM help strengthen capacities of the three consortium partners, making their ongoing activities more effective. An illustrative example is the Media Ethics Observatory (MEO), a project originally undertaken by the Yerevan Press Club. In the media advocacy area, ARM-supported activities are conducted in cooperation with numerous other groups, such as the 20 members of the Media Advocacy Group (MAG) that was formed at ARM's initiative and coordinated by the Yerevan Press Club. This coalition-building approach has helped bring together industry representatives, legal specialists, media professionals, and advocates of media reform. CONCLUSIONS • ARM's efforts have been and continue to be highly relevant to the rapidly changing media environment in Armenia (Question 1); • ARM has made significant contributions in developing alternative media sources, promoting the creation of alternative content, and expanding citizens' access to new information from new and diverse outlets, and has contributed to concrete advances on issues relating to media freedom, such as promoting self-regulation and in the ongoing effort to reform the broadcasting law (Question 2); • ARM has achieved success in designing projects that will be sustainable after the ARM grant expires. A major reason for this is that many of its activities are carried out together with established and ongoing organizations, including the three consortium partners and a variety of outside groups (Question 3); • ARM's leadership structure has been effective, with benefits far outweighing disadvantages. The partners have worked collegially, sharing common goals and values. The consortium arrangement allows the program to take advantage of the long experience and complementary strengths of the three partner organizations, while in turn further strengthening those organizations' capacities. We also conclude that the structure contributes greatly to the sustainability of its many projects (Question 4); • ARM has had considerable success in leveraging skills and resources both from its consortium member organizations and from many other groups it has collaborated with to implement its projects (Question 5). SUMMARY OF RECOMMENDATIONS In the remaining project period, it is generally recommended that the ARM project look for ways to be more selective in the activities it supports, giving more focus and resources to those that are most relevant and effective. Specifically, it is recommended that ARM: • Establish additional Infotuns (regional information centers); 10 • Take initiatives to support the use of new media tools to promote democratic processes and principles in the February 2013 presidential election. In cooperation with national and international election monitoring groups, ARM should use its experience and knowledge of new media in projects to help inform citizens about issues, candidates, interest groups, and proper election process, and to assist journalists, monitors, and voters to observe and to report and correct irregularities in campaigning, balloting, and vote counting; • Be even more creative and proactive in efforts to promote media literacy and educate the Armenian public about the new media environment and how to understand and use the new knowledge that it makes available. • Make additional grants to assist traditional media making the transition to multimedia news coverage. • Initiate exploratory discussions on the changing media market with media managers (editorial and business), advertisers and ad agencies, rating companies, and other relevant stakeholders. These discussions should seek to collect data and ideas on how media will have to adapt to the new environment, and develop concepts for future projects to assist in that process.. • Our final recommendation looks toward the longer-term future of media assistance in Armenia. The media landscape is changing too profoundly and too fast for us to predict what it will look like after the ARM program concludes. But we are certain that the evolving media environment will continue to pose existing challenges and new ones (for example, Armenia's scheduled shift to digital TV, which may endanger some independent broadcast voices in the period immediately following the end of the current ARM project). Thus there will be a continuing need for development of information resources and for citizen education on new information platforms and on ways to participate in public discussion. As we report in this evaluation, USAID's concept of this project was prophetically apt, and ARM's implementation has been capable and imaginative. Looking ahead, we strongly urge USAID to think about continuing and new challenges and about ways to continue the valuable work that ARM has started. 11 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The main purpose of this mid-term performance evaluation is to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Specifically, findings from the mid-term evaluation are designed to inform future work plans of the relevant projects, as well as designs of future similar activities adding to the Armenian media landscape. The evaluation findings are intended to be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. Relevant recommendations from this evaluation may be used by project stakeholders in developing future work plans to support a rapidly growing media environment in Armenia. The evaluation addresses a set of five Key Questions (KQs) focused on identifying (a) what changes should be made in the approaches and directions of the current program, and (b) what recommendations can be made for the design of future media programs. EVALUATION QUESTIONS The KQs, as stated in the Evaluation Statement of Work (SOW) are: (1) How relevant has the ARM project intervention been to the media field challenges in Armenia? (2) To what extent has the project contributed to creating opportunities for (a) a diversity of alternative sources of information and opinion, (b) increased access to alternative sources of information and opinion, and (c) advocacy for improved enabling environment for media independence? (3) What has the project done or mechanisms has it created to ensure that the project products will be sustained? (4) What have been the added value and/or drawbacks of the project management structure (consortium of three local organizations with an unaffiliated Chief of Party)? (5) How successful has the project been in leveraging resources of (a) the skills and organizational strengths of the consortium members, and (b) private and other donor partnerships? 12 PROJECT BACKGROUND Alternative Resources in Media (ARM) is a multi-faceted, $4 million four-year project designed to (1) generate a freer, more diverse flow of information to the Armenian public, (2) foster reporting on traditional and new platforms that will give more citizens a voice in discussions of national, regional, and local issues, and (3) defend and promote greater freedom of expression for both old and new forms of media, while also promoting public understanding of media and improved professional and ethical standards in traditional and "citizen" journalism. The broader underlying goal, as expressed in USAID's description of the project, is to “assist citizens to transform themselves from passive users to active information producers in an otherwise controlled media environment.” The vision is of a society where citizens can use new information technologies to acquire knowledge, make their needs and their opinions known, and come together with like-minded individuals and organizations to work on issues of common concern. ARM's work in the past two years has coincided with two strong trends in the Armenian environment. One trend is a pervasive distrust of news from traditional media, particularly television, which has been Armenia's dominant source of news, but is widely seen as controlled by state authorities and business interests linked to them. In 2011 and 2012, IREX's Media Sustainability Index1 gave Armenia faintly positive ratings on freedom of expression, largely because of legal reforms that decriminalized libel. The Freedom House Nations in Transit report for 2012 gave Armenia the second-lowest score on its 7-point scale for media independence, and Reporters Without Borders listed Armenia 77th among 142 countries ranked in its World Press Freedom Index for 2011-2012. The Freedom House report, quoting the Yerevan-based Committee to Protect Freedom of Expression, noted that "violence against journalists decreased in 2011 with three incidents reported in the first nine months of 2011, compared to eight incidents reported during the same period in 2010. However, several violent attacks in 2011 raised concerns about the safety of journalists. For the third time in five years the car of Lori TV’s editor-in-chief, Narine Avetisyan, was set on fire. Lori TV is known for its independent, critical reports on local government and business, and Avetisyan believes the attacks on her property are intended to intimidate her out of investigative journalism." Earlier, in its report on developments in 2010, the U.S.-based Committee to Protect Journalists gave this description of the broadcast law and the media climate: 1 See IREX Armenia Media Sustainability Index 2012; http://www.irex.org/system/files/u105/EE_MSI_2012_Armenia.pdf; Freedom House, Armenia Freedom of the Press 2012 and Nations in Transit 2012, Ratings and Democracy Score Summary; http://www.freedomhouse.org/report/freedom-press/2012/armenia; http://www.freedomhouse.org/sites/default/files/2012%20%20NIT%20Tables.pdf; Reporters Without Borders, 2011-2012 World Press Freedom Index; http://en.rsf.org/press-freedom-index-2011-2012,1043.html; Committee to Protect Journalists, Attacks on the Press 2010: Armenia, http://www.cpj.org/2011/02/attacks-on-the-press-2010-armenia.php; Committee to Protect Freedom of Expression, Quarterly Reports and 2011 Annual Report; http://khosq.am/en/reports/; http://khosq.am/en/reports/annual-report-2011-2/; See also Open Society Foundation-Armenia and Journalists for the Future, Monitoring of Armenian Online Media Coverage of May 6. 2012 Elections to RA National Assembly; Civilitas Foundation, Armenia 2011, Without Illusions. 13 "New legislation granted regulators broad new powers to award and revoke licenses, while putting severe limits on the number of provincial broadcast licenses. Self-censorship remained widespread in the media, as lawlessness curbed the activities of journalists, human rights defenders, and opposition leaders." The second trend is a sudden increase in the development and adoption of new information technology. This increase not only makes alternative news available through multiple channels to a rapidly expanding audience; it also offers unprecedented opportunities for citizens to contribute information, express their views, and demand greater accountability from officials at all levels of government. By September 2012, reports from two rating services (Circle.AM and AGB Nielsen Media Research) suggested that the Armenian audience for online news may have approached or possibly even passed the audience for television news. Using Circle.AM's daily reports on the number of visitors to online news sites, one analyst has estimated that the top 10 online sites draw as many viewers as the top 10 TV news program. The 10 most visited sites had between 20,000 and 100,000 visitors. (Two sites supported by ARM, Hetq.com and Lragir.com, ranked 17th and 22nd on Circle.AM's list, incidentally.) Social media were also growing at a dizzying pace. In only one six-month period, the number of Facebook users more than doubled, reaching 350,000. Thus ARM's first two years have seen not only a sharply increased demand for the kind of information it seeks to promote, but also an expansion in the means of meeting that demand -- a technological revolution making it possible to communicate that new information to a much wider range of citizens than could have been predicted even a few years ago. ARM's activities are profuse and diverse. Significant projects include: • Establishing regional information centers (known as Infotuns) that offer computer literacy training and online interaction opportunities to residents in five regional towns. • Multiple training programs for professional journalists, "citizen contributors," bloggers, social media users, etc. • Grants to individuals and organizations for a broad range of projects aimed at widening the audience for alternative information and giving citizens a platform to make their voices heard on community and policy issues. Among these are grants assisting two traditional news organizations navigate the transition to multimedia coverage, and one in support of the "i￾Ditord" ("i-Observer") project that helped journalists, election observers and voters use alternative media resources to monitor and report abuses or irregularities in the May 2012 parliamentary election. • Drafting a proposed media literacy curriculum for use in Armenian schools. This is one of numerous activities aimed at raising awareness and increasing public knowledge and understanding of alternative media sources. • Numerous activities to monitor media and encourage public awareness and discussion of media issues, advocate greater media freedom, and promote self-regulation of disputes in place of legal action. ARM's management is a four-legged stool: three Armenian partners (Internews Armenia, Eurasia Partnership Foundation, and the Yerevan Press Club) and a Chief of Party not affiliated with any of the partners. Although the leadership operates as a consortium and commonly uses that term to describe itself, in fact it is not formally a consortium. The recipient of the USAID grant is the international U.S.- based Internews organization, which also hired the Chief of Party, and the Armenian partners are 14 technically sub-grantees. In practice, however, they have functioned from the start as members of a consortium – a word that will be used in this evaluation with the understanding that it refers to a de facto arrangement and not a formal one. To avoid confusion, the name Internews-U.S. will be used for the international Internews (though not its formal name) and Internews Armenia for the Armenian organization (also referred to in ARM documents as Internews Media Support, or IMS). 15 EVALUATION METHODS & LIMITATIONS The evaluation team employed a comprehensive qualitative evaluation design consisting of thorough program document review, key informant interviews with program beneficiaries and stakeholders, and direct observation of ARM project sites. The findings of this evaluation are based primarily on interviews – nearly 50 in total – with program organizers and managers, beneficiaries, and expert observers; review of work plans, progress reports, and other relevant documents; and visits to selected sites (including two outside Yerevan) where ARM or its grant recipients are conducting or have carried out a variety of project-related activities. Interview subjects were selected in an attempt to capture as wide a variety of involvement in differing media activities and different perspectives on ARM and the Armenian media environment as possible. Key informants included the Chief of Party and the heads of the three organizations primarily responsible for designing and implementing the program (Internews Armenia, Eurasia Partnership Foundation, Yerevan Press Club). Also interviewed were staff members of those organizations who are the coordinators for specific program areas -- e.g. grants, regional information centers, media monitoring, proposed media literacy curriculum for schools, advocacy for legal reform and voluntary self-regulation, etc. -- and selected recipients of grants, both those awarded to individuals and those given to existing media or civil society organizations. To ascertain the subjects and range of alternative information the program has helped create, the content of several websites supported by ARM grants was also reviewed by the evaluation team, including Media.am, MyNews.am, Lragir.am, the Ecolur website, and the Circle.AM site, which records data on visitors to online news sites.. In addition, mainly with respect to KQs 1 and 2, a variety of journalists and journalism educators, media law specialists, NGO activists, and other informed observers who are knowledgeable about the wider Armenian media landscape were interviewed. A list of 42 interview subjects is in Annex IV of this report; there were several others who asked not to be named. Of all interview subjects, in addition to the four consortium leaders, 10 are staff members of the consortium organizations administering various ARM programs; 9 are recipients of ARM grants, and 5 are Infotun partners or staff members. Also consulted was the USAID project management specialist who works with ARM. Representatives of two organizations that collaborate with ARM advocacy and freedom-of-expression projects and four teachers attending an ARM media literacy training session were additionally contacted. The remaining seven subjects are journalists, journalism educators, or persons involved in civil society organizations. In response to KQ 1, we also considered various published reports from international media monitoring programs such as the Media Sustainability Index (IREX), the World Press Freedom Index (Reporters Without Borders), and Nations in Transit (Freedom House). Given the short amount of time allotted for this evaluation, data gleaned from interviewees is subject to some degree of selection bias. To mitigate this bias, as previously stated, the evaluation team made every attempt to collect data from subjects representative of the media landscape as a whole. This was done in consultation with USAID staff and through thorough review of project reports and other documents. ARM's goal, broadly stated, is changing the consciousness of the Armenian public. By definition, this is not subject to exact measurement. Nor is it possible to isolate the effects of a particular program from 16 those of broader changes in the media world and the public life of Armenian society. Thus the available quantitative data, such as numbers of people receiving training, or the count of visitors to ARM￾supported online media sites, cannot constitute a comprehensive measurement of ARM's mid-term results. While it is possible to identify particular news stories initiated by ARM-supported projects that have led to citizen action challenging government decisions, such examples only serve to illustrate, rather than quantify, a much broader trend in the public life of the country. A further limitation is that ARM's activities are too numerous and varied for an evaluation to independently examine more than a limited number of activities in the time available. The evaluation team’s findings and conclusions are thus necessarily based on examples, rather than the full range of activities -- e.g., visits to two of the five regional information centers and interviews with eight of 33 grant recipients. 17 FINDINGS, CONCLUSIONS & RECOMMENDATIONS KEY QUESTION #1 How relevant has the ARM project intervention been to the media field challenges in Armenia? FINDINGS An appraisal of ARM's relevance must take into consideration the state of Armenian media and the challenges to creating more independent media that will better serve citizens and the democratic process. As noted above in the Project Background section, Armenia's public and private television channels, which have traditionally been the overwhelmingly dominant source of news, were reported by respondents to be generally seen as providing government-approved reports instead of independent journalism. (Journalists on Armenian television "are not journalists," one respected reporter told us, but "part of a team" serving political interests and their allies in the business community.) It was reported that print media are freer, but have few readers and little influence. The state of trust in media at the outset of the ARM program was documented in the "Armenia Media Landscape" survey conducted for ARM by the Caucasus Research Resource Centers in early 2011. Just five percent of survey respondents declared that they "fully trust" Armenian news media, while 43 percent said they trusted "somewhat" in the news they receive. On the other side, 30 percent "somewhat distrusted" media, and 18 percent said they "don't trust at all." The split was thus exactly even, with 48 percent trusting media at least partially and the identical percentage distrusting partially or entirely. Meanwhile, use of computers and the Internet has spread dramatically from middle-class urban youth to a much wider cross-section of the Armenian population. When teachers attending an ARM media literacy training session in the city of Vanadzor were asked how many of their students had computers at home, estimates ranged from 30 to 50 percent -- even in schools with sizable numbers of students from disadvantaged backgrounds. When asked how many students' homes might have had computers as recently as two years ago, the teachers guessed it would have been fewer than 10 percent. ARM's activities -- described in more detail in our response to KQ 2 -- intersect with these trends in many ways and many places and across a very wide spectrum of Armenian society. Manana Aslamazyan, chief of party, says ARM was developed "at the right time in the right place." Its regional information centers (known as Infotuns, or "info-houses") offer computer literacy training and online interaction opportunities to residents of all ages in five regional towns. ARM has conducted multiple training programs for professional journalists, "citizen contributors," bloggers, social media users, etc., while awarding grants to individuals and organizations for a wide range of projects aimed at widening the audience for alternative information and giving citizens a platform to make their voices heard on community and policy issues. The ARM project also assists traditional news organizations navigate the 18 transition to multimedia coverage, and has developed a proposed media literacy curriculum for use in Armenian schools. In another significant activity, ARM supported the project known as "i-Ditord" ("i-Observer"), using alternative media resources to help journalists, election observers and voters monitor and report abuses or irregularities in the May 2012 parliamentary election. In cooperation with two other civil society organizations, Counterpart International-Armenia and Open Society Foundation-Armenia, ARM assisted in training some 500 monitors and citizens on election procedures and how to report questionable incidents (with photos or videos, if available) via mobile phones or other devices to the i￾Ditord.org website. By ARM's post-election compilation, nearly 900 verified reports were received during the pre-election period and on election day. Among regular viewers of the complaints were election monitoring bodies and the Armenian police. The site is expected to be activated for the coming presidential election as well. Across the entire span of its projects, ARM estimates its activities have potentially reached over 200,000 people, about seven percent of the total Armenian population. Preliminary calculations for the program's first two years estimate that its awareness campaigns -- including nearly 100 TV and radio broadcasts -- had a potential audience of up to 235,000 Armenians. The number who gained improved access to alternative information sources is estimated at 205,000, including visitors to ARM or its grantees' websites, people who had contact with the regional Infotuns, participants in training sessions, and viewers of "E-club," a series of weekly online TV broadcasts that presents discussions of online technologies and news and features on Armenia's blogosphere. Altogether, ARM assistance has reached approximately 80 non-state news outlets across the country. An estimated 62 active blogs are operated by ARM trainees. Snapshots of two ARM-supported websites and what readers could find on them during a typical week give a sense of the variety of material generated by ARM projects and its relevance to audience members' lives. MyNews.am was created for citizen reporters and also for professional journalists who are working in traditional media but cooperate with citizen contributors. Through various special projects and competitions, MyNews seeks to help passive readers become active content creators. Contributors can submit text, photos, video or multimedia products. Postings are not grouped by subject or limited to particular issues or themes; they can be on any topic someone wanted to write about. Media.am is a platform for news about media and media-related issues. Most contributors are professional journalists, but outside commentators also contribute. CONCLUSIONS The sharp increase in computer and Internet use means that ARM-supported online material is available and thus potentially relevant not only to more people but also to an audience that is more representative of the population. As the preceding summary shows, ARM's activities in expanding the reach and use of alternative information resources have been numerous, creative and wide-ranging. Those activities have helped train and organize a growing community of reporters and activists using new media technology to meet a wide variety of needs, including a more transparent and legitimate election process. The projects reviewed, findings on the media landscape, and the information and judgments received from informants within and outside ARM structures all indicate that ARM's efforts have been and continue to be highly relevant to the rapidly changing media environment in Armenia. 19 KEY QUESTION #2 To what extent has the project contributed to creating opportunities for (a) a diversity of alternative sources of information and opinion, (b) increased access to alternative sources of information and opinion, and (c) advocacy for improved enabling environment for media independence? FINDINGS As noted above, the activities undertaken by this program are numerous and diverse. Findings are presented below in two sections: the first focused on topics (a) and (b) of this question (developing alternative sources and increasing access and use), and the second focused on topic (c) (activities that center on monitoring, advocating for media freedom, and improving ethical and professional standards). FINDINGS (1): Activities aimed at developing more diverse alternative information sources and greater access to them. Activities in this area include: • Establishment of Infotuns ("information houses) in five regional towns, housed in and operated in cooperation with local NGOs. The Infotuns' purposes are (1) to widen the audience for alternative news sources, (2) increase public demand for information and independent reporting on issues of local and national concern, and (3) help citizens use new media to contribute to public discussion by creating and disseminating their own reports on pertinent issues and events. Through the Infotuns, in ARM's first two years of operation, 2,800 citizens received training in new media tools, basic computer skills, and citizen reporting. • Multiple training programs for professional journalists, citizen contributors, bloggers, social media users, etc. Training covers both technical skills and journalistic skills and principles. Of 1,280 people who have participated in these training sessions, ARM estimates about half are media professionals and half are citizen contributors. • Grants to individuals and organizations for a wide range of projects aimed at increasing public knowledge of new media, expanding the audience for alternative information, and giving citizens a platform to make their voices heard on community and policy issues. In the two-year period under review, ARM awarded 33 grants through a competitive application process (five of these were continuations of previous grants). Another three grants were for recipients who were invited to apply. Two of the invited grants went to established news outlets, the daily newspaper Aravot and the online investigative newspaper Hetq.am, and are intended to help them navigate the transition from text-only to multimedia news coverage. The third supported the i-Ditord election monitoring project, described under Question 1 above, which developed tools and trained journalists, election observers, and citizens to monitor the May 2012 parliamentary election. A feature of several of these grant projects (examples include lragir.am; the Taghinfo.am "hyperlocal" news site, directed by Gayane Mirzoyan; and MyNews.am, directed by Seda Muradyan) is collaboration between citizen contributors and trained professional journalists, who oversee contributions and seek to promote professional standards of accuracy, fairness, and verification of facts. 20 • A variety of public events, including televised discussions and debates, town meetings, exhibitions, and technology festivals, aimed at raising awareness of new information sources and bringing users together to share experiences and ideas. As alternative sources become stronger and reach a wider audience, government authorities and traditional media have less ability to control or manipulate the news. When people are informed about events and situations from alternative media, traditional media are far less able to shut those subjects out of their own coverage. As ARM grantee Seda Muradyan told us, "it is much more difficult for them to filter information" when that information is already widely known. Anna Israelyan, the online editor for Aravot, pointed out that even while Armenia's TV channels remain subject to direct and indirect political pressure, the new media environment makes them report more openly because "if they don't cover stuff that's on the Internet, people will not watch them." Informants generally reported that Armenian TV coverage of this year’s parliamentary election was far more balanced and gave much fairer coverage to opposition candidates and views than in the 2008 election. The change in the media environment was not the only reason; many believe the improved coverage reflected the authorities' decision to satisfy OSCE standards for a fair, democratic election. Another multiplier is that content generated or disseminated by ARM-supported projects is widely picked up by other online and traditional media outlets. In a typical month this year, for example, of 165 reports posted on Seda Muradyan's MyNews.am site, 52 were republished by other media. Similarly, Inga Zarafyan of the environmental NGO Ecolur (see below under Question 3) and the editors of Lragir.am, both ARM grant recipients, reported that citizen contributions posted on their sites have generated significant coverage in other media, thus letting contributors' voices be heard by a larger audience. FINDINGS (2): Activities centered on monitoring, advocating for media freedom, and improving ethical and professional standards. The background of ARM's advocacy programs includes a 2010 change in Armenian law that decriminalized libel and defamation. Originally, this was hailed as a positive step toward greater media freedom. But the change led to a sudden spike in civil lawsuits and a series of judicial decisions imposing disastrously punitive damage awards -- a trend that reformers charged made the media climate more repressive, not less. In cooperation with the national ombudsman's office, The Media Advocacy Group (MAG), an ARM initiative, played an instrumental role in obtaining rulings from the Court of Cassation, Armenia's highest appeals court, and an interpretive ruling from the Constitutional Court that sharply limited damage awards. The constitutional ruling did not declare that fines for defamation are unconstitutional, but found that media outlets cannot be held liable for “statements of fact” and “value judgments.” The court suggested non-financial compensation, such as an apology, as an alternative to monetary damages.2 The decisions -- which set binding standards for trial courts -- effectively ended the practice of excessive damage awards, and gave a major boost to the movement toward self-regulation through the Media Ethics Observatory, as reported by respondents. After participating in a working group established by the ombudsman's office to draft a reform proposal, two of ARM's consortium members (Internews Armenia and the Yerevan Press Club) and the Committee to Protect Freedom of Expression concluded the working group draft was inadequate. 2 See Freedom House, "Armenia: Freedom of the Press 2012," at http://www.freedomhouse.org/report/freedom￾press/freedom-press-2012. 21 Working on their own, the two groups drafted an alternative proposal and presented it to the Armenian parliament. Their draft has not been voted on, but is now the only one before the parliament and thus will be the basis for action when legislators take up the issue. Specific advocacy and monitoring activities include: • Creation of the Media Advocacy Group (MAG), with a membership drawn from a number of organizations active in media and civil society spheres along with Armenian government organizations such as the National Commission on TV and Radio. MAG has given a unified voice to media members and advocates on issues such as excessive damage awards in defamation and libel cases and reform of the national broadcast laws. • Support for dealing with defamation complaints through a self-regulatory council, the Media Ethics Observatory (MEO), rather than the legal system. The MEO, modeled on the Press Councils that operate in many countries in Europe and elsewhere, was initiated by the Yerevan Press Club in 2007, before the ARM project began. Its activities have significantly increased during the period of ARM support, largely as a result of the successful movement to limit punitive damage awards handed down by Armenian courts. ARM's support has helped the MEO become much more visible and active. Together with the Council of Europe, ARM funded a series of 10 television programs focusing on media ethics and self-regulation issues, helping publicize MEO as an alternative to legal action. In addition, ARM organized a variety of community meetings and discussions on media issues and self-regulation. ARM's involvement helped spark a significant increase in its caseload, from four or five cases a year to 12 or 15 in 2012, according to Irene Aloyan, a media lawyer who coordinates dispute resolution activities for the Press Club. (If the TV discussions were to continue for a full season, Aloyan added, she believes the caseload would double again.) Meanwhile, the number of news organizations that have accepted MEO's Code of Conduct and agreed to be bound by its decisions has risen to 48. That pattern has occurred similarly with other activities of all three consortium partners. • Creating a "media literacy" course for submission to the Ministry of Education as a proposed addition to the national secondary school curriculum. In addition to drafting the curriculum, ARM has presented it to teachers in training sessions in Yerevan, Vanadzor and Goris. • Ongoing review and analysis in various forums monitoring Armenian journalism and its role in informing and educating the public. These monitoring projects drew interest far beyond the media world. As an example, ARM reports the Media.AM website, which posts reports on media-related events and issues, drew 121,000 visitors in its first 1-1/2 years of operation -- many of them, it is probable, from outside the media professional world that is the site's intended audience. CONCLUSIONS On parts (a) and (b) of this question, a review of ARM's activities leads to the conclusion that they have cumulatively made a significant contribution to developing alternative media sources, promoting the creation of alternative content, and expanding citizens' access to new information from new and diverse outlets. It can be assumed that sources and access would have expanded without ARM, given the rapid and revolutionary changes in media technology. But these findings indicate that ARM has helped shape the effects of those changes and has actively assisted a large number of people to become more informed and active through new media in the public life of their communities and the country. As noted 22 in the findings, the impact of these contributions is multiplied because the content generated in alternative media makes coverage in conventional media more open and more responsive to public needs. Also of note is that in its training programs and designing grant projects in which trained journalists monitor and guide amateur contributors, ARM has also demonstrated an awareness of the need to promote professional standards and values among citizen reporters, so their work will command public trust and make citizens' voices more effective in public discussion. ARM has recognized the need for citizen reporters to be trained and develop their understanding of professional skills and values. On part (c), relating to media monitoring and advocacy, ARM and its collaborators have contributed to concrete advances in the areas of self-regulation and efforts to reform the broadcasting law. The impact of ARM's broader efforts to monitor media and promote public knowledge and discussion of media issues is impossible to measure with any precision, but the audience for its programs indicates that there is keen public demand for the material and knowledge ARM is providing on these subjects. ARM￾supported media advocates are currently in the process of finalizing their draft; thus, it is premature to determine success in this area. As well as making alternative information available, ARM has recognized, it is just as important to help citizens become informed consumers of that information, so they can use it to exercise their rights as active citizens instead of being manipulated or controlled by others. 23 KEY QUESTION #3 What has the project done or mechanisms has it created to ensure that the project products will be sustained? FINDINGS Most of ARM’s projects are integrated in existing structures that were well established and active before the ARM program began and are expected to continue after ARM ends. In some cases, the sustaining organization will be one of the three consortium members. In others, it will be one of the many groups ARM has partnered with to carry out different parts of the program. For example, the five regional information centers, called Infotuns, are all housed in and operated under the auspices of local NGOs that are long established in their cities and regions. As a result, instead of being established as new, unattached facilities identified only with a time-limited aid project, the Infotuns are operated as part of an ongoing enterprise. The Infotuns thus start out in touch with a population that is already connected to the host NGOs, while Infotun activities -- teaching computer skills, promoting awareness of new media and citizen reporting, etc. -- add an information component to services those NGOs are already providing, as well as meeting ARM's goals of a more informed, media-literate society. Similarly, many other ARM activities are carried out in cooperation with organizations that can be expected to remain in operation after ARM is terminated. From many possible examples, there are two cases that best illustrate the link between ARM's partnerships with other organizations and the potential sustainability of its projects: The Infotun for the Gegharkunik region is one of two Infotuns visited by the evaluation team. The Infotun is located in the office of the Martuni Women's Community Council, which has been in operation for 12 years, is led by a well-known and well-connected long-time local activist, and has a network of four branch offices and long-standing relationships with a wide range of Armenian and international organizations. The Infotun occupies a spacious room in the council's newly renovated headquarters, with new desks and chairs, five new computers and a printer/fax/copying machine. (The equipment was donated by the Orange Foundation.) The ARM-trained staff not only conducts classes for local residents in the center, but also fans out across the region to bring training to people in their villages and even in the high pastures where farmers spend the summers with their cows and sheep. A popular training topic is the use of Skype, which helps people stay in touch with family members working in Russia and other countries. NGO and Infotun staff members have also created a network of 12 citizen contributors in different towns, who have publicized issues such as the need for a new electricity substation in one district, opposition to a proposed increase in bus fares, and protests against a planned hydroelectric plant that opponents feared would disrupt local farming. Another example showing the same pattern is ARM's support for Ecolur, an NGO that focuses on environmental issues, particularly the situation in 40-plus "hot spots", or zones that have suffered particularly severe ecological damage. Ecolur, which has been active since 2006, is regarded as one of Armenia's most active environmentalist groups. ARM's grant supported multimedia training for activists in eight localities, teaching them how to publicize local issues through blogs, social media, and other new information channels. Material originated by those contributors generated extensive news coverage in traditional media and helped mobilize public action on a variety of environmental threats from mining, hydropower, and other projects. In this case, ARM's grant to Ecolur has terminated, but it continues to produce and distribute information and promote public demands for more transparent procedures and 24 environmentally responsible decisions by government regulatory agencies. Thus, the goals of ARM's support will continue to be met. Both of these partner organizations – the Martuni Women’s Council and Ecolur – have other donors. Those donors will be familiar with their activities, including activities supported by ARM, and therefore if new funding is required to sustain ARM-initiated projects, those partners will be in a much stronger position to get the needed support because of their existing donor relationships. The same is true of many of ARM's other collaborators. A third project that illustrates how sustainability is built into the design of ARM's projects is the Media Ethics Observatory (MEO), described under KQ 2 above. In this case, the project is implemented by a consortium member, the Yerevan Press Club, and as with the two examples presented above, is planned to continue after the ARM program ends. CONCLUSIONS Based on the aforementioned findings, ARM has a clear strategy in place for preparing projects that are realistically sustainable after the ARM grant expires. Cooperation with ongoing organizations is not the only criterion for assessing potential sustainability, but it is logical to conclude that a project implemented within a continuing structure has a better prospect of being sustainable than a stand-alone project whose structure will disappear or will have to be reinvented when ARM terminates. Among ARM's many projects, some are clearly temporary activities that were never intended to be permanent, and it is not possible to make any overall assessment of sustainability while program activities are continuing. However, ARM has designed many important projects in ways that maximize the chance of sustainability, and there are strong reasons to believe that a great deal of the work ARM has initiated will continue to produce valuable results in the future. For example, with the Martuni Women’s Council, some adjustments may be necessary, i.e. staffing, after the ARM program ends; however, because the Women’s Council will remain, as will the computers and the presence of the training recipients, the Infotun will continue to serve the purposes for which it was established. ARM support has already ended for Ecolur, yet the sustainability of the project is already in evidence. 25 KEY QUESTION #4 What have been the added value and/or drawbacks of the project management structure (consortium of three local organizations with an unaffiliated Chief of Party)? FINDINGS While ARM's leadership operates as a consortium and commonly uses that term to describe itself, the leadership structure is not formally a consortium. As explained by USAID's project manager for ARM in Yerevan, the recipient of the grant is the international U.S.-based organization Internews (referred to in this document as Internews-U.S.), rather than the three Armenian partners (Internews Armenia, Eurasia Partnership Foundation, and the Yerevan Press Club). In practice, however, the Armenian partners have functioned more like members of a consortium than like sub-grantees (e.g., they were the principal designers of the project proposal, although the proposal was formally submitted by Internews-U.S.). Manana Aslamazyan, the Chief of Party, and the heads of the three consortium members all reported that the advantages of ARM's management structure have far outweighed the disadvantages. A principal reason is that each of the three partner organizations came to the program with years of experience and highly knowledgeable staffs in different but complementary areas: Eurasia Partnership in grant administration, Internews Armenia in training and media production, and the Yerevan Press Club in activism and advocacy on ethical and legal issues. Additionally, they were unanimous in stating that the personalities involved were also a key factor. Leaders of the partner organizations have known each other for many years, have cooperated in many ways in the past, share common values and goals, and have strong mutual respect and trust. For all those reasons, they report that they have been able to resolve different views without damaging their relations, and have carried out their parts of the project without competing for resources. One partner's view of the arrangement is expressed in an e-mail to the evaluators from Boris Navasardyan, president of the Yerevan Press Club: “In the case of the three local partners the management structure was very efficient, since they share the values, approaches, they do not differ much in terms of capacity and influence, they also have long history of fruitful cooperation, thus, it is very common for them to discuss things and come up with a collective decision." Navasardyan added that having Internews Network as the external lead organization has been helpful in establishing an "efficient division of tasks and fair cooperation" among the Armenian partners. All three, he explained, "through their 15 years and more operational history have proved each to be quite strong in many components of media development – advocacy, training, production, research, etc., and each might claim that is able to implement almost all components of the ARM project itself. Thus, Internews Network and its representative as CoP helps to identify the stronger sides of the partners and finds best implementers (or their combination) for each project component or sub-component, ensuring synergy effect of various efforts." The three organization heads and all others asked gave Manana Aslamazyan the highest possible marks for her leadership – dedicated, charismatic, with an extraordinary depth of experience in all realms of media assistance, and able to draw on a very wide network of people she has worked with over her many years in the field. The decentralization of the program, along with its wide range of activities, occluded a larger understanding of consortium structure for some informants. However, informants did not provide any significant criticism about their own relations with ARM's managers or general management style (the 26 only adverse comment came from one grant recipient, whose project has been approved but not yet implemented, felt that ARM was being unduly slow in releasing the designated funds). An additional significant benefit of the consortium structure reported by respondents was that joint efforts and financial and organizational support from ARM help strengthen capacities of the three consortium partners, making their ongoing activities more effective. An illustrative example is the Media Ethics Observatory (MEO), a project originally undertaken by the Yerevan Press Club. The MEO is the centerpiece of the effort to handle defamation and privacy cases and other disputes over media practices through a voluntary self-regulation process instead of through the legal system. ARM's management was also found to have been capable of changing course when circumstances proved beneficial. An example was the decision not to proceed with a proposed new printing plant for regional newspapers, as was envisaged in the original program description but reconsidered in light of a continued decline in newspaper readership. Another example was a turn to give less attention to blogs and more to social media, which have increasingly become a more important platform for discussion of public issues. CONCLUSIONS ARM's collegial decision-making process at times takes more effort and is slower than it would be if there were a single organization involved. However, that effect is far outweighed by the many advantages of the consortium structure. Those benefits include: the experience and expertise each partner has in complementary areas of activity; the strengthened capacity all three will gain from ARM's resources and organizational support; and the greatly improved sustainability that comes from working with organizations that were strong and well-established before ARM and will remain strong and well￾established after ARM concludes. Those considerations, plus the experience and attitudes of the personalities involved, demonstrate that ARM's management structure has significantly helped make its many activities more creative and effective. ARM’s management has effectively shown its ability to remain flexible and adapt to changing needs, circumstances, and ideas. 27 KEY QUESTION #5 How successful has the project been in leveraging resources of (a) the skills and organizational strengths of the consortium members, and (b) private and other donor partnerships? FINDINGS Within the consortium, a good example of leveraging resources was found to be the Media Ethics Observatory, described above, which draws on the Yerevan Press Club's depth of experience in promoting free expression and ethical and professional standards in Armenian media. Another example is management of ARM's grant program by a different consortium partner, Eurasia Partnership Foundation. EPF administers ARM grants with a background of having conducted a grant program in Armenia for many years, supporting a wide range of civil society and NGO activities. The expertise it has developed in that area was found to have greatly benefitted ARM's operations. Similarly, Internews Armenia’s expertise in training, one of its principal activities for much of the last two decades, was cited as a highly important asset for ARM's programs. As to leveraging resources from organizations outside the consortium, ARM-supported activities are conducted in cooperation with numerous other groups, such as the 20 members of the Media Advocacy Group (MAG) that was formed at ARM's initiative and coordinated by the Yerevan Press Club in the media advocacy area, for instance. The members represent journalistic associations, traditional and new media outlets, civil society organizations, and Armenian government agencies. Similarly, ARM participates with other media advocates in the Information Dispute Council, which advises complainants on self-regulation possibilities, and in the movement to revise the broadcasting law. This coalition￾building approach brings together industry representatives, legal specialists, media professionals, and advocates of media reform. Similar leverage of partners' skills and resources was seen to occur in other ARM programs beside those discussed above. The i-Ditord election monitoring project, supported by ARM jointly with the Armenian affiliates of Counterpart International and the Open Society Foundation, is one example. The linkage of the five Infotuns to existing NGOs with proven viability, also described earlier, is another. With respect to finding additional donor support, Chief of Party Manana Aslamazyan reported that ARM so far has obtained approximately 250,000 USD from non-U.S. government international sources, putting it on track to meet USAID's requirement of 400,000 USD in co-funding over the four-year life of the program. ARM has also received modest financial support from Armenian donors, such as the Orange Foundation's donation to the Martuni Infotun and funding from other Armenian telecom firms for a BarCamp, where participants exchange ideas and get expert guidance on topics such as blogging, social media, web and application design, etc. CONCLUSIONS Both ARM's structure and its project design approach were found to have effectively made use of the skills and strengths of collaborating organizations – the consortium partners themselves, and a wide range of other groups that have cooperated with various ARM projects. Their combined knowledge and resources strengthen the broader effort to expand media freedom, raise professional standards, and improve public knowledge. These findings indicate that ARM has had considerable success in leveraging skills and resources both from its consortium member organizations and from many other groups it has collaborated with to implement its projects. 28 ARM has been creative and imaginative in finding ways to contribute to revolutionary changes in the Armenian information landscape. Meanwhile, through its monitoring and advocacy activities, it has also found ways to support traditional principles and values that need to be preserved in the new media environment -- recognizing that if alternative media do not observe standards of honesty, independence, and respect for fact, they risk losing public trust and becoming discredited in the same way that traditional Armenian media have. ARM's organizational structure and its approach to designing its profusion of activities have also maximized the chance that the projects it supports will be able to remain active and effective after ARM support ends. Additionally, in collaborating with many other organizations and individuals to carry out those projects, ARM has both benefited from its partners' contacts, experience and expertise, and in turn, has given those partners significant new ideas and experience and improved capacity for future efforts. GENDER ARM's senior leadership reflects an equal division in gender, with two females (Chief of Party and the director of Internews Armenia) and two males (president of the Yerevan Press Club and country director of Eurasia Partnership Foundation). At the staff level, of the six people we met who manage specific ARM programs for one or another of the consortium partners, four were women and two were men. At the two Infotun centers we visited (Martuni and Vanadzor), one Infotun coordinator was male and one female; the heads of both host NGOs were women. As to beneficiaries, a large majority of participants in ARM's training programs have been women. In the first year, according to the Year One summary of results, women made up 75 percent of those receiving training from Internews Armenia and 70 percent of those trained under the Infotun program. A similar gender breakdown for training programs in the second year was unavailable to the evaluation team during their fieldwork; however, those proportions are not expected to have changed significantly. In ARM's grant programs, individual awards went to roughly equal numbers of men and women. While there is no way to precisely identify recipients of organizational grants by gender, it can be reasonably assumed that ARM's grants in that sector benefited a significant proportion of women, since women are generally well represented in the NGO sector. Finally, it is speculated that ARM may help reduce the gender imbalance among younger journalists, where women vastly outnumber men. (An example is that of approximately 150 students in the journalism faculty at the Yerevan State Linguistic University, only four are men.) Computer technology, on the other hand, is traditionally of greater interest to men, so it may be possible that the new-media technology taught in ARM's training could draw more men into the world of information producers. In at least one reporting period, the gender balance in ARM training courses on blogging was much more even, with men still in the minority, but by a much narrower margin. 29 RECOMMENDATIONS ARM is a whole made up of many, many parts. The second half of the program should look for ways to be more selective, giving more focus and resources to activities that are most relevant and effective. Those considered high-priority category include: • Infotuns, which informants unanimously described as exceptionally successful in extending the reach of new media to a wide range of citizens in Armenia's regions. Without suggesting a specific target, expansion of this program should be a priority. To the extent that resources permit and effective partnerships can be established, the launch of new centers in other regions beyond the five presently served is highly recommended. • Using new media tools to promote democratic processes and principles in the February 2013 presidential election. Alternative information sources can play a huge role in informing citizens about issues, candidates, interest groups, and proper election process. They can also greatly assist journalists, monitors, and voters to observe and to report and correct irregularities in campaigning, balloting, and vote counting. ARM is strongly urged to look for all possible ways to cooperate with other election-monitoring organizations and use its knowledge and experience in new media and the platforms it has created in projects relating to voter education and ensuring freedom and fairness in the election process. • Media literacy. This is a crucial element – unless people learn to be knowledgeable and critical consumers of information, promoting alternative information sources will not reach the broader goals of empowering citizens and strengthening democracy. ARM is urged to find ways to be even more creative and proactive in educating the Armenian public about the new media environment and how to understand and use the new knowledge that it makes available. In connection with the proposed media literacy curriculum for schools, one specific idea to explore would be a system of modest incentives for teachers trying out the new course in the experimental phase. A possible model reported by respondents comes from Doctors Without Borders, which paid a small stipend to teachers who participated in a pilot program teaching that organization's proposed course on healthy lifestyle and living. In that connection, it was also reported that some of the teachers in the training session attended by the evaluation team expressed a wish for more technical material to be covered in the training. This would be relatively easy to implement, and it is recommended that ARM adjust its training agenda accordingly. It is also recommended that the training-of-trainers component in this particular program be strengthened, so teachers receiving ARM's training on the media literacy curriculum can then pass on their knowledge to other colleagues. • Assistance to traditional media for the transition to multimedia news coverage. ARM has given grants to help two news organizations develop multimedia capability. Other news organizations could benefit from this support, and ARM is urged to consider making additional grants in this area. • Related to the previous recommendation, reshaping for the new media age is necessary not only in newsrooms but in the industry as a whole, which must adapt to an entirely new and still largely undefined market model. Therefore, it is recommended that ARM begin a discussion with media managers (editorial and business), advertisers and ad agencies, rating companies, and other relevant stakeholders, to begin collecting data and ideas on how that process may unfold 30 and what news organizations need to do to be sustainable in the changed media environment. A better grasp of the present situation and existing and future challenges will lay the groundwork for designing future programs to assist the transition to new and more viable market structures that can preserve the principles and practices of professional journalism in the new media world. • With respect to "citizen reporting," ARM is strongly urged to continue and strengthen efforts to team citizen contributors with trained professional journalists, so that the accuracy and fairness of citizen reporting will reach the best possible standard and earn and keep the trust of its audience. • In giving higher priority to the areas listed above, ARM should consider whether resources for an array of awareness-raising activities ought to be kept at the same level as in the first two years. The need for promoting awareness may be less now than when ARM began; therefore, ARM managers should reexamine that category of projects and determine whether some resources can be reallocated to other activities. • For the remaining term of ARM's program, ARM should consider offering more extended in￾depth training, particularly when the focus is on reporting skills rather than technology. The program should also look for ways to extend training to a wider range of participants, including media managers and professionals in related fields. Teaching journalistic principles and skills should be recognized as an element in all activities promoting content by “citizen contributors”. The more citizen reporters and their work can be trusted to meet standards of accuracy, balance, independence, and ethical conduct, the more effectively they will help meet the goal of creating a more educated and empowered public. • Looking ahead, ARM should develop strategies and educate its constituents and its audience in advance of Armenia's scheduled shift to digital TV, which is planned to take place in the period immediately following the end of the current ARM program. There is concern in the media advocacy community that some independent broadcast voices may be endangered in that transition.3 In view of that concern, ARM should put appropriate resources into monitoring the preparations for digitalization, promoting an open process that will protect free expression in broadcast media, and informing and mobilizing citizens about the process and the importance of the issues it raises. • Looking ahead, USAID/Armenia is urged strategize ways of approaching new challenges to continue the valuable work that ARM has started. The evolving media environment will continue to pose existing challenges and new ones -- for example, the coming shift to digital TV. Thus, there will be a continuing need for development of information resources and for citizen education on new information platforms and on ways to participate in public discussion, as well as a continuing need to protect media freedoms and open discussion of public concerns from a diversity of voices and viewpoints. 3 See Committee to Protect Freedom of Expression task group report, "Transition Process to Digital, TV and Radio Broadcasting in Armenia," Dec. 2010. 31 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK ANNEX II: EVALUATION METHODS AND LIMITATIONS ANNEX III: SOURCES OF INFORMATION ANNEX IV: DATA COLLECTION INSTRUMENTS ANNEX V: SUMMARY NOTES FROM INTERVIEWS ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST 32 ANNEX I: EVALUATION STATEMENT OF WORK Statement of Work Evaluation of USAID/Armenia Health Systems Strengthening Project, Civil Society and Local Government Support Program, Alternative Resources in Media Project, Assistance to the Energy Sector to Support Energy Security and Regional Integration Program, Pension and Labor Market Reform Project Summary: USAID/Armenia requires performance evaluations of the following activities: Health Systems Strengthening Project (HS-STAR), Civil Society and Local Government Support Program (CSLGSP), Alternative Resources in Media Project (ARM), Assistance to the Energy Sector to Support Energy Security and Regional Integration Program (ESRI), and Pension and Labor Market Reform Project (PALM). The purposes of this Task Order are to evaluate the success of these projects in their relevant technical areas and to assess the overall effectiveness of the projects in achieving set programmatic goals and USAID/Armenia’s strategic objectives. Three of the five planned evaluations are designed as mid￾term performance evaluations (HS-STAR, CSLGSP, and ARM), while the remaining two are designed as end-of-project performance evaluations (ESRI and PALM). The purposes of the mid-term evaluations are to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Findings from the mid-term evaluations must inform future work plans of the relevant projects, as well as designs of future similar activities. The purpose of the end-of-project evaluations is to assess the effectiveness of resources spent and to inform design and development of future strategies and projects. The evaluation findings must be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. The respective project AORs/CORs will develop plans for incorporation of relevant recommendations from the evaluations in their future work plans. Contractor Responsibilities and Projects: The evaluations should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, guided by the evaluation questions formulated for each individual activity. Each evaluation question must be answered empirically, relying on factual evidence, and must be addressed distinctly in the final reports. HS-STAR Jan 2011-Jan 2014, $9.6m This is a three-year activity in the second year of its implementation. The project aims to address key constraints in health financing, leadership and governance, human resources, and information systems that impede access to and delivery of quality health services. The project relies on an approach that simultaneously aims to strengthen the health system while improving the quality of care and increasing population knowledge in priority service areas, including maternal and child health, reproductive health and family planning, tuberculosis, non-communicable diseases and emergency medicine. HS-STAR aims to significantly enhance local capacity to design, implement and monitor reform interventions to foster sustainability. To this end, HS-STAR provides technical assistance to the Armenian government through program components of health financing and governance, quality improvement (including MCH, TB, RH/FP), and the adoption of proactive health care seeking behaviors through civil society engagement. 33 CSLGSP Sep 2010-Sep 2014, $15.5m This is a four-year activity in the second year of its implementation. The project aims to increase the level of informed and organized civic activism at the local and national levels, along with more participatory, decentralized, accountable governance that leads to a more democratic society. The project will reach this objective through three programmatic components: (1) local government and civil society collaboration, (2) fostering civic participation, advocacy and activism, and (3) facilitating decentralization and local fiscal autonomy. The project expands opportunities for the public to organize and advocate on behalf of their needs and concerns; it also increases local government accountability and capacity to be responsive to citizen interests. The program takes advantage of opportunities for reform at the local level by creating models of democratic governance through a highly integrated community development approach to local self-governance, fueled by increased civic participation from Civil Society Organizations (CSOs) and businesses. ARM July 2010-June 2014, $4m. ARM is a four-year activity in its second year of implementation. The project is aimed at (1) supporting the production and dissemination of alternative content though multiple high- and low-tech platforms; (2) building consumer demand for news and information at local and regional levels by promoting media literacy, critical analysis, and use of the latest and most useful web and mobile-based media applications and (3) encouraging timely, concerted public advocacy by NGOs, think tanks and citizens on specific key media-related issues. Alternative Resources in Media supports the diversification of media content, utilizing both high- and low-tech means to produce and distribute news and information. It assists citizens to transform themselves from passive users to active information producers in an otherwise controlled media environment. ESRI Sep 2009-Sep 2012, $5.9 million The project provides expert services to the Government of Armenia to ensure (1) that the new Nuclear Unit meets western safety and environmental standards; (2) that Armenia develops the human and institutional capacity to plan and successfully implement the project; and (3) that the contribution of the project to Armenia’s economic growth and development is maximized. The project’s objectives also include improving the enabling environment for the development of renewable energy in Armenia, and supporting Armenia’s integration into regional energy systems and markets. PALM Oct 2009-Sep 2012, $5.4 million PALM is USAID’s primary activity aimed to assist individuals, households, and communities to better manage social risks or needs, provide support to the vulnerable employed and jobless individuals (particularly the low-skilled, women, and the disabled), and give greater attention to countering the effects of the global economic crisis. The program is focused on (1) building institutional and human resource capacity to implement a multi-pillar pension system that will provide an adequate income for its elderly population in an environment that is fiscally sound and sustainable, with reformed current pay-as-you-go (PAYG) scheme and the newly created private retirement savings system; and (2) countering the effects of the economic downturn, through catalyzing new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps, and through improving the country’s workforce competitiveness. Evaluation Questions: 34 The evaluation questions for mid-term evaluations are designed to help the projects adjust and/or modify future implementation and work plans, as well as designs of future similar projects. The questions for end-of project evaluations will help assess the effectiveness of the resources spent and will help inform the design of future projects and strategies. The contractor must review and summarize the data collected from document review and from field work to answer the following evaluation questions: HS-STAR • To what extent is the project on track in achieving its expected results of a) establishing transparent and accountable health financing and governance; b) institutionalizing a system of continuous improvement of the quality of provided services; c) building the capacity of the national TB program; and d)enabling civil society to exercise their health rights and responsibilities? • Are the project implementation approaches relevant to the current state of health reforms in Armenia? • How did the implementer perform in terms of project management? o Were the project leadership and the management structure appropriate for its implementation? • What internal and external factors slow down organization of procurement and training processes? • Is there evidence that project interventions will be sustainable beyond the project lifetime? CSLGSP • How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia? • To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms? • To what extent has the program been successful in promoting decentralization? • How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? o Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)? • Do local governments/community councils, and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? ARM • How relevant has the ARM project intervention been to the media field challenges in Armenia? • To what extent has the project contributed to creating opportunities for (a) a diversity of alternative sources of information and opinion, (b) increased access to alternative sources of information and opinion, and (c) advocacy for improved enabling environment for media independence? 35 • What has the project done or mechanisms has it created to ensure that the project products will be sustained? • What have been the added value and/or drawbacks of the project management structure (consortium of three local organizations with an unaffiliated Chief of Party)? • How successful has the project been in leveraging resources of (a) the skills and organizational strengths of the consortium members, and (b) private and other donor partnerships? ESRI • Is there evidence that the independent expert services to Government of Armenia for new Nuclear Power Plant contributed to procedures and activities implemented in accordance with the western safety and environmental standards in preparation of documentation for the New Nuclear Power Plant? • Is there evidence that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? • Is there evidence that the regional integration task contributed to an increase of regional cooperation with Georgia? • What do stakeholders groups see as the results of the project? • How relevant was the intervention in terms of Armenia’s priorities in energy sector? PALM • What do stakeholder groups - the Ministry of Labor and Social Affairs, the Central Bank of Armenia, the Ministry of Finance of Armenia, State Revenue Service, State Employment Service Agency, “Pension System Awareness Center” foundation, “Youth Orientation Center” NGO – see as the achievements of the program? • To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? • To what extent has the support given to the GOAM to build the foundation of Multi Pillar Pension System contributed to the preparedness of the System to be implemented? • To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency to match labor market demand and to develop/implement active labor market policies? • How adequate was the project PMP in reporting on project progress? USAID’ S Role in the Evaluation The USAID Mission in Armenia will: • provide relevant programmatic and budgetary information to the Contractor; • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff 36 and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor must: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Identify data collection methodology to provide the best possible evidence to answer the evaluation questions, also considering feasibility issues. 3. Identify informants and stakeholders, samples and/or other relevant data sources. 4. Prepare a field work plan. 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports addressing comments by USAID and submit final reports to USAID/Armenia for acceptance. 8. Address implementing partner comments within one week as necessary after USAID/Armenia shares the final reports with implementing partners, and if partners choose to submit “Statements of Differences”. The proposed methodology should address the need for data collection from qualitative and quantitative sources, and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. All evaluation questions need to be answered empirically; therefore the data collection methods should be tailored to ensure that adequate evidence is collected to answer each of the questions in a definitive manner. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. Data should come from facts, rather than be based on anecdotal evidence. Conclusions should be based on findings received from multiple sources, and strengths and limitations of the methodology should be explicitly communicated. All people-level data should be disaggregated by sex to allow analysis of findings by sex. Baseline data for all projects is available from their monitoring data. A sample of indicators used for monitoring of each of the projects is provided in the Annex. Some of the baseline data sources include surveys, official statistics, automated information systems, and project records. In the case of CSLGSP baseline data was collected not only from participating 43 communities, but also from 15 control communities. Although not randomly assigned to treatment and control groups, communities nonetheless are comparable, and can therefore serve as comparisons for the achievements in citizen engagement in decision making and in community development projects. Deliverables The Contractor’s deliverables must include: 1. Written methodology plans (evaluation designs and work plans). The evaluation designs must include detailed evaluation design matrices (including the key questions, the methods and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final designs require COR approval. The work plans must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation teams. The methodologies will be shared with relevant stakeholders, including the implementing partners for their comments before finalizing. 37 2. Verbal debriefing The evaluation teams must meet with USAID/Armenia upon arrival. The teams must also provide oral briefings of findings and recommendations to the USAID/Armenia senior management and relevant technical teams prior to departure. 3. Draft Evaluation Reports Prepare draft evaluation reports which must analyze data and summarize key findings, conclusions and recommendations. The Findings section must list all facts and evidence received from desk research and from field work in relation to each evaluation question. The Conclusions section must analyze the facts specified in the Findings section, and must discuss what worked, what did not work and why. The Recommendations section must address issues of what can be improved for future programming, based on Findings and the analysis provided in the Conclusions section. More specifically the Recommendations section must address the following: For HS-STAR – a) what can be changed for future implementation of the project, b) what recommendations can be made to ensure sustainability of project results; for CSLGSP - what can be improved for the remainder of the project given the resources available; for ARM a) what changes should be made in the approaches and directions of the current program, and b) what recommendations can be made for the design of future media programs; for ESRI - what recommendations can be made for future energy sector programming; for PALM - what recommendations can be made for design of future pension reform projects. The Evaluation Reports must at a minimum contain: 1) a 3-5 page Executive Summary summarizing key points (purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations); 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action-oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work; all evaluation tools; all sources of information properly identified and listed; any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers and/or members of the evaluation team; disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Draft reports must be submitted to USAID/Armenia within three weeks after completing the fieldwork for each evaluation. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. Final Reports The Contractor must submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The reports shall follow USAID branding procedures. The reports must include an executive summary and not exceed 30 pages (excluding appendices). 38 The final reports must meet the following quality standards (Please see the USAID Evaluation Policy): a) The reports must represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports must address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings must assess outcomes and impact on males and females; f) Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. Evaluation Data The Contractor must submit Evaluation data to USAID/Armenia along with the final reports for record keeping and future use by the Mission. The data must be in an easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Schedule: The anticipated duration of all evaluations is not to exceed nine months. Staffing and Level of Effort: It is expected that the Contractor will provide five different teams of consultants to carry out the evaluation of the five activities described above. Consultants with diverse or cross-cutting skills may serve on more than one evaluation team. The mix of skills necessary for each evaluation team is specified below. Each evaluation team should consist of two experts, although other team compositions might also be considered. Inclusion of qualified local Armenian experts in the evaluation teams is highly encouraged. All evaluation team members must be required to provide a written disclosure of conflict of interest. Required Qualifications of the Personal For all evaluation teams: • Excellent English writing skills • Experience with evaluation designs, quantitative and qualitative methods and data analysis • Substantial experience with program evaluations. Prior experience with USAID or other donor project evaluations is a plus. For HS-STAR: • Substantial experience in health systems management, public health, international health or a related field • Substantial experience with program management and/or organizational development • Experience working with health projects, health systems project evaluations, or with program evaluations in the region is a plus 39 For ARM: • Substantial experience in communications, media, journalism or related field • Experience working with media projects, media project evaluations, or with program evaluations in the region is a plus For CSLGSP: • Substantial experience in public administration, international relations or related field with emphasis on civil society and local government • Experience working with civil society and/or local government projects, experience with civil society and local government project evaluations, or with program evaluations in the region is a plus For PALM: • Substantial experience in public administration, international relations or related field with emphasis on social services and pension reform • Experience working with social services or pension reform projects, experience with social services project evaluations, or with program evaluations in the region is a plus For ESRI: • Substantial experience in engineering with emphasis on energy systems, nuclear energy or renewable energy • Experience working with energy system projects, experience with energy project evaluations, or with program evaluations in the region is a plus Instructions on Preparation of Branding Implementation Plan and Marking Plan As part of the proposal submission, the Contractor will develop a Branding Implementation Plan (BIP) and a Marking Plan in accordance with the policies found at Automated Directive System (ADS) Chapter 320, revised on May 5, 2009, or any successor branding policy, and with the “USAID Graphics Standard Manual” that is available at www.usaid.gov/branding. Among other provisions, ADS 320 states that: 1. Contractors and subcontractors' corporate identities or logos must not be used on USAID￾funded program materials. 2. Marking is not required on contractor vehicles, offices, office supplies or other commodities used solely for administration of the USAID-funded program. 3. Marking is not permitted on any communications that are strictly administrative, rather than programmatic, in nature. USAID identity is also prohibited on contractor and recipient communications related to award administration, such as hiring/firing of staff or renting office space and/or equipment. The Contractor shall also develop a Marking Plan for public communications, commodities, program materials, deliverables, and other items that visibly bear or will be marked with the USAID identity. The marking plan may include requests for exceptions to marking requirements, to be approved by the Contracting Officer. Contract deliverables to be marked with the USAID identity must follow design guidance for color, type, and layout in the Graphic Standards Manual (available at www.usaid.gov/branding) or any successor branding policy. 40 With respect to this Task Order, the Contractor should develop a BIP and Marking Plan bearing in mind the following branding strategy: 1. Program Name: Evaluation of Five USAID/Armenia Projects. 2. Positioning: This task order is funded through the USAID/Armenia Mission. Materials and communications must be positioned as from the American People, using the USAID Identity. 3. Outreach to Beneficiaries and Host-Country Citizens: No special outreach efforts to beneficiaries and host-country citizens are planned under this Task Order. 4. Level of Visibility: The findings of the final evaluation report will be used by USAID in its implementation and further planning its activities. The report will be submitted to USAID’s Development Experience Clearinghouse for wider access. 5. Other Organizations to be Acknowledged: No other organizations are required to be acknowledged. 6. Specific branding issues: The only branding issue expected to arise as a result of implementing this Task Order is the proper use of graphics standards for all reports and other printed or electronically distributed information. Environmental Considerations The Evaluation Services IQC has an Initial Environmental Examination (IEE) approved on April 8, 2010. The IEE has determined that the activities described under the Evaluation Services IQCs qualify for a categorical exclusion per 22 CFR 216. Any activities found to be outside the scope of the approved Regulation 216 environmental documentation shall be halted until an amendment to the documentation is submitted and written approval is received from USAID. 41 ANNEX II: EVALUATION METHODS AND LIMITATIONS The evaluation team employed a comprehensive qualitative evaluation design consisting of thorough program document review, key informant interviews with program beneficiaries and stakeholders, and direct observation of ARM project sites. The findings of this evaluation are based primarily on interviews – nearly 50 in total – with program organizers and managers, beneficiaries, and expert observers; review of work plans, progress reports, and other relevant documents; and visits to selected sites (including two outside Yerevan) where ARM or its grant recipients are conducting or have carried out a variety of project-related activities. Interview subjects were selected in an attempt to capture as wide a variety of involvement in differing media activities and different perspectives on ARM and the Armenian media environment as possible. Key informants included the Chief of Party and the heads of the three organizations primarily responsible for designing and implementing the program (Internews Armenia, Eurasia Partnership Foundation, Yerevan Press Club). Also interviewed were staff members of those organizations who are the coordinators for specific program areas -- e.g. grants, regional information centers, media monitoring, proposed media literacy curriculum for schools, advocacy for legal reform and voluntary self-regulation, etc. -- and selected recipients of grants, both those awarded to individuals and those given to existing media or civil society organizations. To ascertain the subjects and range of alternative information the program has helped create, the content of several websites supported by ARM grants was also reviewed by the evaluation team, including Media.am, MyNews.am, Lragir.am, the Ecolur website, and the Circle.AM site, which records data on visitors to online news sites.. In addition, mainly with respect to KQs 1 and 2, a variety of journalists and journalism educators, media law specialists, NGO activists, and other informed observers who are knowledgeable about the wider Armenian media landscape were interviewed. A list of 42 interview subjects is in Annex IV of this report; there were several others who asked not to be named. Of all interview subjects, in addition to the four consortium leaders, 10 are staff members of the consortium organizations administering various ARM programs; 9 are recipients of ARM grants, and 5 are Infotun partners or staff members. Also consulted was the USAID project management specialist who works with ARM. Representatives of two organizations that collaborate with ARM advocacy and freedom-of-expression projects and four teachers attending an ARM media literacy training session were additionally contacted. The remaining seven subjects are journalists, journalism educators, or persons involved in civil society organizations. In response to KQ 1, we also considered various published reports from international media monitoring programs such as the Media Sustainability Index (IREX), the World Press Freedom Index (Reporters Without Borders), and Nations in Transit (Freedom House). Given the short amount of time allotted for this evaluation, data gleaned from interviewees is subject to some degree of selection bias. To mitigate this bias, as previously stated, the evaluation team made every attempt to collect data from subjects representative of the media landscape as a whole. This was done in consultation with USAID staff and through thorough review of project reports and other documents. ARM's goal, broadly stated, is changing the consciousness of the Armenian public. By definition, this is not subject to exact measurement. Nor is it possible to isolate the effects of a particular program from those of broader changes in the media world and the public life of Armenian society. Thus the available quantitative data, such as numbers of people receiving training, or the count of visitors to ARM￾supported online media sites, cannot constitute a comprehensive measurement of ARM's mid-term results. While it is possible to identify particular news stories initiated by ARM-supported projects that 42 have led to citizen action challenging government decisions, such examples only serve to illustrate, rather than quantify, a much broader trend in the public life of the country. A further limitation is that ARM's activities are too numerous and varied for an evaluation to independently examine more than a limited number of activities in the time available. The evaluation team’s findings and conclusions are thus necessarily based on examples, rather than the full range of activities -- e.g., visits to two of the five regional information centers and interviews with eight of 33 grant recipients. 43 ANNEX III: SOURCES OF INFORMATION Interviews ARM leadership Manana Aslamazyan, chief of party Boris Navasardian, president of Yerevan Press Club Gevorg Ter-Gabrielyan country director, Eurasia Partnership Foundation Nouneh Sarkissian, managing director, Internews Armenia ARM consortium staff Arevhat Grigoryan, Yerevan Press Club, editor of weekly media review Irene Aloyan, Yerevan Press Club, lawyer & coordinator of Media Ethics Observatory, YPC advocacy activities Marina Mkhitaryan, Eurasia Partnership Foundation, program manager for Infotuns, co-manager for grants Alice Ter-Ghevondian, Eurasia Partnership Foundation, program manager for grants Artur Papayan, Internews Armenia trainer, new media expert Gegham Vardanyan, internews Armenia, trainer Lusine Grigoryan , Internews Armenia, coordinator for media literacy curriculum Ashot Hambartsumyan, Internews Armenia, ARM program manager Moses Hakobyan, Internews Armenia, lawyer Vahram Martirosyan, Internews Armenia, project manager for media musem ARM grantees Anna Muradyan Inga Zafarian, president of Ecolur Seda Muradyan, founder and president, Public Journalism Club Lia Mkhitaryan Arpine Grigoryan Anna Israelyan, online editor of Aravot Haykazn Ghahriyan, editor-in-chief of lragir.com Naira Hayrumyan, deputy editor of lragir.com Gayane Mirzoyan Infotun partners and project staff Anahit Gevorkyan, head of Martuni Women’s Community Council Vruyr Malkhasyan, Martuni Infotun blog coordinator Deborah Hall, Peace Corps volunteer at Martuni Women's Council Arpine Hakobyan, coordinator of NGO Center, Vanadzor Nora Minasyan, project coordinator, NGO Center, Vanadzor Representatives of organizations collaborating with ARM projects 44 Ashot Melikyan, chairman, Committee to Protect Freedom of Expression Ara Ghazaryan, media lawyer, member of Information Dispute Council USAID/Armenia Anahit Khachatryan, project management specialist Teachers attending media literacy training in Vanadzor Rita Mkhitiryan Rusana Hakhverdyan Nouneh Hovhannisyan Manush Azumanyan Journalists, journalism educators, civil society experts Tatul Hakobyan, author and journalist, Civilitas Tatyana Hovhannisyan, international journalism faculty, Yerevan Linguistic University Sasun Khachatryan, journalist Lilit Arakelyan, journalist, medialab.am Nvard Hovhannisyan, journalist Vladimir Osipov, Institute of Philosophy and Law, Armenian National Academy of Sciences, specialist on gender issues Armine Haladjyan, officer-in-charge, UN Information office in Yerevan Site visits Martuni: visited Martuni Women's Community Council, toured Infotun facility, spoke with NGO and Infotun coordinators Vanadzor: visited NGO Center, Vanadzor, observed part of training session for teachers on proposed media literacy curriculum, toured Infotun facility, spoke with NGO and program coordinators and several of the teachers Yerevan: visited offices of ARM grant recipients Ecolur, Lragir.am, Public Journalism Club, Aravot daily; attended ARM "Social Art Action" event in English Park, spoke with participants and several ARM grantees ARM documents Reviewed workplans, quarterly and year-end progress reports to USAID, newsletters and announcements ARM-supported Websites Reviewed contents of Media.am, MyNews.am, Lragir.am 45 Published reports on Armenian media from media and human rights monitoring groups Reporters Without Borders, 2011-2012 World Press Freedom Index, http://en.rsf.org/press-freedom-index-2011-2012,1043.html Freedom House, Armenia Freedom of the Press 2012 and Nations in Transit 2012, Ratings and Democracy Score Summary http://www.freedomhouse.org/report/freedom-press/2012/armenia http://www.freedomhouse.org/sites/default/files/2012%20%20NIT%20Tables.pdf IREX Armenia Media Sustainability Index 2012 http://www.irex.org/system/files/u105/EE_MSI_2012_Armenia.pdf Committee to Protect Journalists, Attacks on the Press 2010: Armenia http://www.cpj.org/2011/02/attacks-on-the-press-2010-armenia.php Committee to Protect Freedom of Expression, Quarterly Reports and 2011 Annual Report http://khosq.am/en/reports/ http://khosq.am/en/reports/annual-report-2011-2/ Open Society Foundation-Armenia and Journalists for the Future, Monitoring of Armenian Online Media Coverage of May 6. 2012 Elections to RA National Assembly Civilitas Foundation, Armenia 2011, Without Illusions Interview log 10/15 Manana Aslamazyan, chief of party, Boris Navasardian, president of Yerevan Press Club Overview of ARM program, activities, leadership structure 10/15 Tatul Hakobyan, author and journalist, Civilitas State of Armenian media 10/15 Arevhat Grigoryan, Yerevan Press Club, editor of weekly media review Overview of media monitoring activities 10/15 Irene Aloyan, Yerevan Press Club, lawyer & coordinator of Media Ethics Observatory, YPC advocacy activities Reviewed legal and advocacy activities 10/16 Manana Aslamazyan, Gevorg Ter-Gabrielyan country director, Eurasia Partnership Foundation, Nouneh Sarkissian, managing director, Internews Armenia Overview of ARM program, activities, leadership structure 10/16 Tatyana Hovhannisyan, international journalism faculty, Yerevan Linguistic University perspectives on Armenian media, professional standards, state of journalism education, public attitudes toward new media and traditional media 46 10/16 Inga Zafarian, president of Ecolur, ARM grantee Reviewed Ecolur activities supported by ARM grant using alternative media to report on environmental issues & followup activities 10/17 Marina Mkhitaryan, Eurasia Partnership Foundation, program manager for Infotuns, co-manager for grants, Reviewed Infotun program, training and social media activities, partnerships, details of programs at Martuni and Vanadzor infotuns 10/17 Alice Ter-Ghevondian, Eurasia Partnership Foundation, program manager for grants reviewed grant competition, selection, specific projects e.g. "crowd-sourced" reporting of election violations 10/17 Ashot Melikyan, chairman, Committee to Protect Freedom of Expression overview of press freedom issues, CPFE cooperation with Yerevan Press Club in media monitoring, advocacy activities; training on legislative & ethics issues; "self-regulation" of defamation cases; proposed reform of broadcast law 10/17 Artur Papayan, new media trainer reviewed I-DTORD election monitoring project, training approach, positive experience with Infotuns, landscape of traditional and new Armenian media, problem of maintaining professional standards in new media 10/17 Anna Muradyan, ARM grantee Reviewed her activity in creating online platform "info village," work with secondary school students reporting on events & issues in their schools in various regions 10/17 Seda Muradyan, ARM grantee, founder Public Journalism Club Reviewed her project preparing for new media, mobile phone apps for monitoring/coverage of presidential election, work on MyNews site, material republished in other online media 10/18 Ara Ghazaryan, media lawyer, member of Information Dispute Council Reviewed structure and work of IDC, history of defamation cases & current issues, ongoing work to reform broadcasting law 10/18 Nouneh Sarkissian, director Internews Armenia reviewed management structure issues; training programs for professionals and "citizen contributors," Media.am website, media literacy projects 10/18 Gegham Vardanyan, internews Armenia, trainer reviewed their monitoring program, paid contributors Write media analysis pieces, 2-3-4/month, monitor entertainment as well as news content 10/19 visited Martuni Infotun. presentations/interviews: Anahit Gevorkyan, head of Martuni Women’s Community Council; Vruyr Malkhazyan, Martuni Infotun blog coordinator; Deborah Hall, Peace Corps volunteer at Martuni Women's Council; Tigran Baghishjanyan, one of their local contributors reviewed Infotun program, history of Women's Community Council & cooperation with ARM, use of contributors from 12 villages in region, program to teach Skype to help people communicate with family members in Russia, viewed refurbished computer room w equipment from Orange telecom co. 47 10/19 Anna Israelyan, online editor of Aravot Reviewed activities to be supported by ARM grant supporting transition to "convergent newsroom" 10/20 at Social Art Action event in English Park, Yerevan, interviewed ARM grantees Lia Mkhitaryan, founder of eiwa.am, and Arpine Grigoryan of Pan-Armenian network Reviewed grant activities to raise awareness, promote use of alternative media by artists and others to provide content on social, other public issues 10/21 visit to Vanadzor Infotun. observed training for teachers conducted by Tigran Hakobyan, media mgr & consultant, Mesrop Harutunyan, expert on media literacy handbook; interviewed Lusine Grigoryan (Internews, coordinator for media literacy); Nora Minasyan, project coordinator, NGO Center, Vanadzor; Arpine Grigoryan, infotun coordinator; Vagham Martirosyan, Internews staff reviewed teacher training & other infotun activities including training on internet use, Skype, "citizen contributors" in villages, "civil society day" observance & coop with other NGOs interviewed 4 teachers, Rita Mkhitiryan, Rusana Hakhverdyan, Nouneh Hovhannisyan, Manush Azumanyan; asked about no. of students in their schools who have computers at home, student knowledge & use of social media 10/22 further interview w Manana Aslamazyan, additional details on training programs, no. of professionals and citizen contributors trained, visits to ARM supported sites, media literacy programs, focus on remaining period of program; Vagham Martirosyan, on background & need for media literacy program 10/22 also interviewed Ashot Hambartsumyan, Internews Armenia, ARM program manager; reviewed media literacy, outreach to editors & media managers; short interview w Moses Hakobyan, Internews lawyer, reiterated and confirmed account of broadcast law reform effort 10/23 Haykazn Ghahriyan, editor-in-chief, and Naira Hayrumyan, deputy editor of lragir.com Reviewed examples of local stories reported by citizens w ARM support 10/25 Gayane Mirzoyan, ARM grantee reviewed her activities collecting and editing "hyperlocal" citizen contributions, training contributors in cooperation w UNDP Yerevan 10/25 group interview w/ Yerevan journalists, graduates of Caucasus School of Journalism in Tbilisi. Group included Sasun Khachatryan, Lilit Arakelyan, Nvard Hovhannisyan, one other who doesn't want name used Discussed state of media, censorship & selfcensorship, lack of public trust esp in TV channels 10/26 Armine Haladjyan, officer-in-charge, UN Information office in Yerevan; Vladimir Osipov, Institute of Philosophy and Law, Armenian National Academy of Sciences, specialist on gender issues Perspective on media landscape, increasing use of social media & other new platforms, loss of trust in TV broadcasters & other traditional media, weakness of print media. 48 ANNEX IV: DATA COLLECTION INSTRUMENTS Following are sample interview questions for respondents including responsible program managers for ARM and the three consortium organizations; grantees and participants in ARM-supported projects; and outside observers including journalists, journalism educators, civil society activists, etc. Questions for ARM director and consortium representatives: -- What specific examples can you give where the program has developed alternative platforms for disseminating information and expression of citizens' questions, needs, opinions, and demands for accountability? -- How have institutions and officials responded? Have these projects led to any change in reporting by traditional news media? -- Has your experience led you to identify which platforms and skills best serve to improve public knowledge and empower citizens? Blogging? Social media such as Facebook? Telecommunication hubs? Support for local production of video/audio materials, documentaries, etc.? -- Which platforms and skills will you concentrate on in the remaining two years of the program? -- What steps have you taken to ensure that the projects you have supported will be sustained? -- How do you assess the project management structure? -- How effectively have you made use of the skills and organizational strengths of the consortium members and other donor partners? Questions for grantees and program participants: -- Summarize your project's goals, your planned course of action, and what was actually carried out. -- What community, population group. policy area or institution did you hope to influence? What changes were you able to observe? Did you see any change in reporting by traditional media on issues or subjects relevant to your project? -- What new technology or communication systems were developed or used in your project? Who used them? Did experience show that some platforms or skills were more effective than others? If so and if the project is continuing, which technologies will you concentrate on for the remaining activities? -- Will this project continue after the ARM program ends? What steps are you taking to make it sustainable? -- What was your relationship with ARM and its constituent organizations? Did you need training, guidance, administrative support, and were those provided? Questions for journalists, journalism educators, NGO and civil society activists, and other observers of media and civic life: 49 -- How do you assess Armenian media on the following issues: -- Professional standards (accuracy, independence, news judgment) -- Making public officials and other power holders accountable to the public -- Giving a voice to those without power or influence, and reporting on their needs and concerns -- What is your experience and perspective with new media and alternative information sources and their role in Armenian public life? What has been the effect of those new platforms on independent reporting, citizen advocacy, and accountability of public officials? -- If you are familiar with ARM or any of the projects it supports, how relevant have they been to the challenges facing media and civil society? -- As far as you are aware, how has ARM contributed to creating opportunities for alternative sources of information and opinion, increased access to those alternative sources, and an improved environment for media independence? 50 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Arnold R. Isaacs Title Evaluation Specialist Organization Social Impact, Inc. Evaluation Position? Team Leader Evaluation Award Number (contract or other instrument) Contract No: AID-RAN-I-00-09-00016 Task Order No: AID-111-TO-12-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Alternative Resources in Media (ARM) I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 24 January 2013 51 Name Suren Deheryan Title Evaluation Specialist Organization Social Impact, Inc. Evaluation Position? Local Evaluator Evaluation Award Number (contract or other instrument) Contract No: AID-RAN-I-00-09-00016 Task Order No: AID-111-TO-12-00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Alternative Resources in Media (ARM) I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 24 January 2013 52 USAID ARMENIA 1 AMERICAN AVENUE YEREVAN 0082, ARMENIA TELEPHONE: +37410 46 47 00 FAX: +37410 46 47 28 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 53