March 2012 This publication was produced for review by the United States Agency for International Development. EVALUATION Support For Armenia-Turkey Rapprochement (SATR) Project: Mid-Term Evaluation Report USAID AID|ARMENIA SATR EVALUATION REPORT Submitted to USAID on March 29, 2012 This publication was produced for review by the United States Agency for International Development. SUPPORT FOR ARMENIA-TURKEY RAPPROCHEMENT (SATR) PROJECT: MID-TERM EVALUATION REPORT MARCH 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016-00, Task Order Number AID-111-TO-11-00002 Evaluation of ABA-ROLI, CEPPS, SATR AND SSIP Projects DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Authors: Lawrence Robertson and Stepan Grigoryan SATR Evaluation Report International Business & Technical Consultants, Inc. Acknowledgements The team would like to thank Mariam Gevorgyan and Anahit Khachatryan at USAID/Armenia and the leadership and staff of the Armenian and Turkish organizations that are in the consortium or partners in the project for their openness and help with the evaluation. SATR Evaluation Report International Business & Technical Consultants, Inc. TABLE OF CONTENTS Executive Summary ......................................................................................................................... i I. Introduction ................................................................................................................................. 1 II. Scope and Methodology ............................................................................................................. 5 III. Findings..................................................................................................................................... 7 A. Major Achievements and Lessons Learned from Implementation ........................................ 7 B. Lessons Learned from Implementation of Project Design ................................................... 12 C. Challenges in Project Management ...................................................................................... 14 D. Overall Lessons Learned ...................................................................................................... 16 IV. Responses to the Evaluation Questions Presented in the Scope of Work .............................. 19 A. General USAID/Armenia Evaluation Questions .................................................................. 19 B. SATR-specific Evaluation Questions ................................................................................... 22 V. Conclusions .............................................................................................................................. 26 VI. Recommendations................................................................................................................... 29 Annex I: Evaluation Scope of Work ............................................................................................. 33 Annex II: SATR Work Plan .......................................................................................................... 41 Annex III: Bibliography ................................................................................................................ 51 Annex IV: Conflict of Interest Statements.................................................................................... 54 Annex V: Statement of Difference ............................................................................................... 56 SATR Evaluation Report International Business & Technical Consultants, Inc. LIST OF ACRONYMS AATTC Association of Armenian and Turkish Travel Companies ATCP Armenia-Turkey Cinema Platform AIRCA Association of International Road Carriers of Armenia B2B business-to-business BSEC Black Sea Economic Cooperation BTD Bridging the Divide CAS Country Assistance Strategy CRRC Caucasus Research Resource Center DRL Bureau of Democracy, Human Rights and Labor, U.S. Department of State EPF Eurasia Partnership Foundation GPoT Global Political Trends Center ICHD International Center for Human Development MFA Ministry of Foreign Affairs MOU Memorandum of Understanding n.d. no date NK Nagorno-Karabakh NGO Non-governmental Organization P2P people-to-people SATR Support to Armenia-Turkey Rapprochement RFA Request for Applications TABDC Turkish-Armenian Business Development Council TEPAV Economic Policy Research Foundation of Turkey TESEV Turkish Economic and Social Studies Foundation THM Town Hall Meeting TOG Toplum Gönüllüleri TURKONFED Turkish Enterprise and Business Confederation TURSAB Association of Turkish Travel Agencies TV Television UBCCE Union of Black Sea and Caspian Confederation of Enterprises UMBA Union of Manufacturers and Businessmen of Armenia U.S. United States SATR Evaluation Report International Business & Technical Consultants, Inc. USG United States Government USAID United States Agency for International Development YB Youth Bank YPC Yerevan Press Club SATR Evaluation Report International Business & Technical Consultants, Inc. Page i EXECUTIVE SUMMARY Troubled relations between Armenia and Turkey have had a negative influence on Armenia since independence. Armenia and Turkey are neighbors that share hundreds of years of history of living together - as well as the last century of tensions. The border has been closed by Turkey for most of the 20 years of Armenian independence and the two countries have not established diplomatic relations. However, from a 10 year perspective there have been dramatic changes in Armenia-Turkey rapprochement - from almost no contact to significant civil society and business ties. Improved Armenia-Turkey relations are important to the United States Government in its efforts to support peace and stability in the Caucasus and the development of a strong democratic Armenia with economic and other engagement with her neighbors. But conditions are not propitious for a breakthrough in the status quo because there are few opportunities for a significant change in state-to-state relations. USAID thus needs a medium- to long-term perspective to support future rapprochement, as well as to provide support to keep the ideas and vision of Armenia-Turkey rapprochement alive and to build on growing concrete ties in civil society and businesses. Introduction The main objective of USAID Armenia’s Support to Armenia-Turkey Rapprochement (SATR) project is to promote improved Armenia-Turkey relations. The purpose of SATR is to: engage civil society in reconciliation processes; establish and develop business partnerships and regional professional networks; and facilitate government-to-government dialogue. The initial SATR project objectives were to: 1. Develop new business partnerships and regional professional networks; 2. Engage civil society in alliance-building to further contribute to Armenia-Turkey normalization; 3. Build a favorable environment for dialogue and facilitate interaction among politically active groups and policy advocates; and 4. Support multi-lateral efforts toward rapprochement with research. The project began October 1, 2010, and is planned to continue until September 30, 2012. USAID/Armenia awarded the project after a competitive procurement to the Eurasia Partnership Foundation (EPF) as the lead member of an Armenian consortium that linked the Yerevan Press Club (YPC), International Center for Human Development (ICHD), and Union of Manufacturers and Businesspeople of Armenia (UMBA) – each with a set of Turkish organizations as their partners (subcontractors). In the third quarter of FY 2011, Objective 3, state-to-state component, was revised. Scope and Methodology USAID/Armenia requested a mid-term evaluation of the SATR project to examine progress towards achieving programmatic goals, the appropriateness of project design, and the impact of strategic adjustments made during the first year of implementation. The evaluation aims to contribute to understanding issues in Armenia-Turkey rapprochement and to strengthen project implementation. SATR Evaluation Report International Business & Technical Consultants, Inc. Page ii The evaluation was carried out by a team of two IBTCI consultants – one international specialist in peace building in development and one Armenian foreign policy specialist. The team worked together during 14 days of field-work in Armenia in December 2011. The evaluation covers the first year of work of the project, as well as on-going work currently in process. Findings Although Armenia-Turkish rapprochement has not progressed as anticipated at the time that USAID designed the request for applications (RFA) or as expected by the Consortium in their application, SATR has been able to conduct a wide range of activities towards its objectives. However, the project is troubled in all strands by issues of information, collective action problems, and a variety of asymmetries between the Armenian and Turkish sides. The Consortium’s design was overly ambitious, and the way it was proposed without substantial discussion with Turkish partners meant that there were many revisions needed once awarded by USAID. The design also sets up a variety of challenges for project management – within the Armenian consortium, between consortium members and their Turkish subcontractors, and between Armenian consortium members and other non-governmental organizations in the country. The impact of the SATR project to-date appears to be limited due to the following: the design led to a slow start of implementation; there were issues with the quality of project activities; the program focused on activities rather than objectives and project goals; the program was changed to reduce the emphasis on reaching key actors in government under the project; and mechanisms for the project to reach large numbers of people, especially in Turkey, were absent. Conclusions SATR is a wide ranging project, with many strands and numerous activities rather than a tight, focused, strategic project concentrating on Armenia-Turkey rapprochement. The project has not used lessons from worldwide people-to-people peace building programming, nor from past Armenia-Turkey activities, to develop high quality activities and push for substantial movement towards rapprochement, especially in an environment where state-to-state relations have been frozen. Although the project does function, the impact of its activities has been modest. Few people know about the project or its activities, which are not sustainable at present due to the problematic design of the consortium and problems with implementation. Problems of Armenia￾Turkish relations remain important to the development of a peaceful, prosperous Caucasus that is in the U.S. national interest. Improved management can produce better results from SATR even as the project heads towards the mid-point of its second year of implementation. Increased efforts and stronger results are especially important in Turkey. Recommendations Modest changes are possible within SATR in the nine months left under this cooperative agreement to reach more people through better information management. SATR partners and contractors can also reach out more informally to their own government counterparts to facilitate policy change. Both of these aspects can increase project results and the sustainability of these results. SATR Evaluation Report International Business & Technical Consultants, Inc. Page iii SATR needs to strategically refocus to have a greater and longer lasting impact. USAID and the Consortium should rapidly decide on a nine-month strategy to maximize project impact through:  A push for policy influence with state officials by re-emphasizing the need to reach state officials in both countries;  An outreach strategy and implementation to push SATR products to broader target groups;  Greater visibility for SATR by increasing the visibility of the project among the public;  A strategic focus on improved prospects for rapprochement by increasing support by reaching beyond project partners;  A platform for information sharing that will be actively managed beyond the grant for sustainability;1  Targeting ways to make project outcomes more sustainable; and  Fewer but more influential activities within the strategic focus by ending activities that do not fit the approaches above. Additional project support for Armenia-Turkey relations is needed after the conclusion of the SATR project. Part of the long-term development challenge that must be addressed to support development in Armenia, and peace and security in the Caucasus, is the century of disagreement and the absence of almost all contact between Armenia and Turkey over this period. Building support for rapprochement and change is a long-term process for both societies and governments, and not something that can be concluded in the time-frame of a two-year project. While it is recommended that USAID consider developing and supporting successor Armenia-Turkey rapprochement projects, it must be mentioned that issues in addressing Armenia-Azerbaijan and the Nagorno-Karabakh conflict are more difficult and pressing from the standpoint of Armenia. The Mission should consider developing targeted programming in these areas, which should be kept separate from Armenian-Turkey relations. 1 The project had begun to develop a dedicated web site at the time of the evaluation. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 1 I. INTRODUCTION Armenia and Turkey are burdened by a long history of disagreement. The two countries have made several attempts over the 20 years since Armenia declared its independence from the USSR to set aside their differences and establish ties. Thus far, these efforts have not resulted in diplomatic relations and their mutual border has been closed by Turkey for all but the first few years of this period. The neighboring countries face issues over the history of Armenian communities in Turkey and the processes that led to the death or expulsion of almost all ethnic Armenians from Turkey during the First World War. The support of Armenia for the independence of Karabakh from Azerbaijan, with whom Turkey has established strong relations, is an additional issue that hampers relations. However, Turkey’s attempts to link improvement in Armenia-Turkey rapprochement with progress in the resolution of the Nagorno-Karabakh conflict and Armenia rejecting this link, the Turkey-Armenia normalization process has largely been left to public diplomacy initiatives in civil society and business. Recent Context and the Development of the Support for Armenia-Turkey Rapprochement Project The United States (U.S.) government has longstanding interests in peace, stability, and the development of regional ties in the Caucasus. The March 2009 United States Government (USG) Country Assistance Strategy (CAS) for Armenia has improved regional relations as the top priority. Priority Goal 1 is “Developing Armenia's ability to be a constructive and peaceful regional neighbor.” The CAS notes how Armenia's longstanding conflict with Azerbaijan over Nagorno-Karabakh has both harmed economic development and heightened the risk of armed conflict. The analysis concludes that “settlement of the Nagorno-Karabakh dispute and achieving consensus with Turkey on addressing shared history are critical to Armenia's full realization of democratic governance, sovereignty, independence and broad-based economic prosperity.” At this time, the quiet diplomacy between Armenia and Turkey mediated by the Swiss and other governments – including the U.S. – was making substantial progress that led to the development of protocols on opening the border and establishing diplomatic relations. On October 10, 2009, the Turkish and Armenian foreign ministers signed two protocols in Zurich that were intended to develop and normalize relations between the two countries. The protocols deliberately did not include language on the following two most controversial problems in bilateral relations: the 1915 killing of Ottoman Armenians and the Nagorno-Karabakh conflict. Initially, there was great optimism about the historic agreement, but political bickering between the two countries resulted in a suspension of the protocols just six months later. The United States Agency for International Development (USAID) Mission in Armenia funded Armenia-Turkey civil society contacts earlier in 2009 via a one-year unsolicited proposal from Eurasia Partnership Foundation (EPF) for the “Days 2 and 3” project. EPF began work in this area in 2006 with an “Armenia-Turkey Cross-Border Dialogue and Cooperation Program” that provided small grants to many organizations that remain their key partners. With the opening in state-to-state engagement that became apparent with the “football diplomacy,”2 USAID/Armenia developed the Request for Applications (RFA) for the larger Support to Armenia-Turkey 2 The opening first became publically visible when the Turkish President accepted an invitation from his Armenian counterpart to attend the Turkey-Armenia World Cup qualifying match in Yerevan. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 2 Rapprochement (SATR) project as the state-to-state engagement of the countries became public and controversial in both countries in late 2009 and early 2010. SATR was competed and then awarded on the last day of FY 2010 to EPF as the head of a consortium. The consortium partners then moved to reach formal agreements with Turkish partners and to adjust the program in light of changing realities in Armenia-Turkish rapprochement and the experience, knowledge, and ideas of their Turkish partners. This took substantial time. The first major event to launch the project, a meeting that connected the Armenian and Turkish partners plus other stakeholders in Yerevan, did not occur until November 23, 2010. A follow-up Armenian-Turkish workshop in Istanbul was held in December 2010. SATR began with a request for proposals to support innovative Armenian-Turkish pilot projects; the start-up of the business survey in Armenia and Turkey; the elaboration of media research methodology; the development of the project’s Facebook page; and three town hall meetings (THM) on Armenia-Turkey relations in Armenia. Context of the Evaluation USAID/Armenia contracted with IBTCI to do a mid-term process evaluation of the first year of the two-year SATR project. The activity is being implemented in a changed environment of Armenia-Turkey relations. The Mission asked that the evaluation examine the appropriateness of the design of the project as well as any issues in implementation. The evaluation should contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach and in the future planning of this project and any potential successor activity. The Armenia-Turkey rapprochement process at the state level remained frozen through the fall of 2010 and all of 2011. To enter into force, the two protocols had to be ratified by the parliaments of both countries; neither country has brought the protocols to the floor of parliament for a vote. Although the government of Turkey repeatedly stated that “quiet diplomacy” has continued between the Turkish and Armenian authorities, the Armenian government has denied these contacts and maintained the position that Armenia has no intention of continuing the rapprochement process at the state level unless Turkey ratifies the protocols. Contacts between civil society and businesses in Armenia and Turkey have continued in spite of the absence of state-to-state contacts. The optimism from early in the “football diplomacy” period that Armenia-Turkey ties were ripe for resolution has substantially faded. Trends are not promising for change from either Turkey or Armenia. As one perceptive Turkish commentator has recently summarized: Given that the pursuit of normal economic relations with Armenia comes with the price tag of deteriorating relations with Azerbaijan, and the fact that Armenia’s economic potential pales in comparison to Azerbaijan’s, the prospective gain of opening the border with Armenia today is not enough incentive for Turkey to follow through with the already controversial course of normalizing relations. … Strategically, economically, and in consideration of public opinion, it is not in the interests of the Turkish government to open its border with Armenia.”3 Neither these interests, nor the perception of them by the Turkish authorities, are likely to change in the next few years. The current government of Turkey (GOT) is stable, as seen through three 3Diba Nigar Göksel. “Turkey’s Caucasus Policies in the Framework of Ankara’s New Foreign Policy.” Caucasus Analytical Digest No. 30: Emerging Regional Powers: Turkey and Iran in the South Caucasus. (Zurich: Swiss Federal Institute of Technology, 28 October 2011), p. 5-6. http://www.isn.ethz.ch/isn/Digital￾Library/Publications/Detail/?ots591=0c54e3b3-1e9c-be1e-2c24-a6a8c7060233&lng=en&id=133750 SATR Evaluation Report International Business & Technical Consultants, Inc. Page 3 electoral victories over the last nine years, each with a larger number of votes than before. The Erdogan administration has a strong mandate for its domestic and foreign policies from the Turkish people, and has substantially limited the roles of the military and national security administration in foreign and security decision-making processes that they had long-dominated (whether Turkey was under military or civilian rule). In Armenia as well, there are few incentives to change the foreign policy status-quo in relations towards Turkey. Extensive interviews in Armenia with foreign policy analysts and political parties for the evaluation found almost no support among these circles in Armenia for new Armenian government initiatives to break the current stalemate in Armenia-Turkey relations.4 Even strong supporters of rapprochement felt the protocols already embody concessions to Turkey such as "giving the Turks what they want on the historical sub-commission on the genocide and the recognition of the existing border.”5 Project Approach and Development Hypothesis The goal of the SATR project to promote improved Armenia-Turkey relations has been implemented through activities under three objectives: 1) engaging civil society in the reconciliation processes; 2) establishing and developing business partnerships and regional professional networks; and 3) facilitating government-to-government dialogue. The SATR project does not have an explicitly formulated development hypothesis. However, the management and staff of the program, as well as the program design, come out of a robust peace building tradition, which has many different strands and alternatives. Instead of a formally stated development hypothesis, for this program, SATR partners “share the common vision of the project as an effort by leading civil society organizations in Armenia and Turkey to support the process of normalization at the level of state, business and civil society.”6 The implicit SATR development hypothesis is that building a variety of general ties between societies on both sides of the border creates conditions to eventually change state policies that will bridge the divide between Armenia and Turkey.7 The EPF application notes that: “The longer-term impact of the Project will be gauged by the follow-on activities of the four organizations and their Armenian, Turkish, and international partners. It will also be reflected in attitude changes at the societal level. Regardless of how State-to-State relations progress, the Support to Armenia-Turkey Rapprochement Project will prepare grounds for rapprochement to reach a point of no return.”8 The SATR project proposed to “act as a catalyst for diplomacy, creating and supporting conditions for both governments to have a variety of options to develop relations.” The project outlined the ways the consortium would be a catalyst: the “formation of cross-border networks will contribute to confidence and decrease stereotypes among the two societies, accelerate the 4 Interviews with academic, civil society, and political party leaders in Yerevan. 5 Interview with Armenian academic expert. 6 Eurasia Partnership Fund. “Support to Armenia-Turkey Rapprochement” Cooperative Agreement No. AID-111-A￾10-00004: Quarterly Progress Report April – June 2011.” (Yerevan: EPF), p. 10. 7 Interview with project management. 8 Partnership Fund. “Support to Armenia-Turkey Rapprochement” Cooperative Agreement No. AID-111-A-10- 00004: Quarterly Progress Report April – June 2011.” (Yerevan: EPF), p. 7. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 4 normalization of inter-state relations and, in the long term, help to address more controversial issues.”9 SATR project activities focus on four priority areas: business partnerships, civil society, and inter-governmental dialogue, and research informing and supporting all three areas. The main goal is to promote improved Armenia-Turkey relations by engaging civil society in reconciliation processes; establishing and developing business partnerships and regional professional networks; and facilitating government-to-government dialogue. The four initial project objectives were to: 1. Develop new business partnerships and regional professional networks; 2. Engage civil society in alliance-building to further contribute to Armenia￾Turkey normalization; 3. Build a favorable environment for dialogue and facilitate interaction among politically active groups and policy advocates; and 4. Support multi-lateral efforts toward rapprochement with research. The project began October1, 2010, and is planned to continue until September 30, 2012. To implement the SATR project, the EPF and three Armenian civil society organizations developed a consortium with a division of labor around these objectives. The consortium partners include the Yerevan Press Club (YPC), International Center for Human Development (ICHD), and Union of Manufacturers and Businesspeople of Armenia (UMBA). The business objective is led by UMBA and contributes to regional economic cooperation between Armenia and Turkey by building opportunities for profit and non-profit cooperation between Armenians and Turks in the private sector through activities such as expos, conferences, round-tables, and the promotion of public-private partnerships. The leading organizations in the civil society strand are YPC and EPF and conduct a host of activities in the media, with youth, and through culture. The lead organization in the state-to-state strand is ICHD, which planned four cycles of targeted assessments starting at the end of year one based on priorities and opportunities identified through a series of six, off-the-record discussions in each country; each cycle was expected to lead to a brief paper in Armenian, Turkish and English on opportunities for state-level cooperation targeting government stakeholders. The fourth objective consisted of research activities by EPF, ICHD, UMBA, and YPC to contribute to rapprochement in the strands where they focus their activities. After receiving the grant from USAID, the four Armenian consortium organizations then reached agreements with partners in Turkey as sub-contractors. The SATR project now has seven main Turkish partners: Turkish Armenian Business Development Center (TABDC), GAYA Research Institute, Turkish Economic and Social Studies Foundation (TESEV), Anadolu Kultur, Global Political Trends Center (GPoT), Izmir University of Economics, Faculty of Communication, and Toplum Gönüllüleri. The project was restructured during the third quarter of the first year of implementation. In response to the USG’s advice, the project revised the state-to-state component (Objective 3). The re-conceptualization reallocated resources and priorities from the state-to-state strand to strengthen the business and civil society strands of the project. 9 Ibid., p. 10. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 5 The purpose of the mid-term evaluation is to examine whether the SATR project is on track towards achieving programmatic goals, the appropriateness of the project design, and the impact of strategic adjustments made during the first year. II. SCOPE AND METHODOLOGY As a mid-term project evaluation, the evaluation scope of work (SOW) (see Annex A) focuses on project implementation and pending issues for USAID, the implementing partner EPF, its partners in Armenia and Turkey, project beneficiaries, and project stakeholders. The evaluation aims to contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach to dealing with the issues regarding Armenia-Turkey rapprochement and support any further adjustments as necessary in the approach and planning of SATR to strengthen project implementation. Evaluation Methodology The evaluation is based on the first year of work of the project, plus some of the work in progress in the first quarter of year two while the team was conducting the evaluation in December 2011. The evaluation team used as resources the four Quarterly Progress Reports, and extensive annexes, that aggregate the results of the consortium and its partners and grantees over FY 2011. USAID and the Consortium also provided the work plan for the first year and draft work plan for year two, the original RFA, and the grant award (which includes the original application of the consortium). The evaluation was carried out by a team of two IBTCI consultants – one international specialist in peace building and development and one Armenian foreign policy specialist. The team worked together during 14 days of field-work in Armenia in December 2011. The team conducted interviews with 70 people, prioritizing USAID staff and management, the management and staff of the implementing organizations, project beneficiaries, and other stakeholders. To encourage candor and open discussion, all interviewees were assured at the start of the interview that nothing they said would be attributed to them directly or discussed in such a way that could be clearly attributed to them or their organizations in the evaluation report. The international consultant benefitted greatly from observing some of an externally-facilitated workshop held in Yerevan with project staff and partners that reviewed the first-year of the project and discussed plans to move forward in the second year of SATR. Most interviews were set up interdependent of project staff by soliciting lists of project beneficiaries and stakeholders from consortium partners. The international consultant selected some participants from the larger lists, using sampling criteria to ensure a diversity of respondents (e.g. male and female, larger and smaller firms, in the capitals and from the regions). Other interviews were set up using another independent track - having the national consultant identify, contact, and arrange interviews with stakeholders that are outside of SATR process. These interviews provided information about other activities in Armenia-Turkey rapprochement and views of SATR’s work from stakeholders that do not work with SATR. Based on the SOW, the evaluation team formulated 17 specific questions from the longer list of SATR-specific and general questions provided by USAID. The team used these questions in SATR Evaluation Report International Business & Technical Consultants, Inc. Page 6 interviews with project management and staff, beneficiaries, and stakeholders in face-to-face interviews in Armenia and through telephone interviews with Turkish interviewees who were not in Armenia. 1. Are SATR approaches feasible given current Armenia-Turkey relations? 2. What aspects of the project work or do not work in the current political situation in Armenia? 3. What aspects of the project work or do not work in the current political situation in Turkey? 4. How flexible is SATR in adjusting activities to the changing political context? 5. How has SATR built on the previous “Days 2 and 3”project? 6. What concrete examples and evidence demonstrates mid-term impact? Why? 7. How sustainable are project results likely to be? 8. How do key external stakeholders perceive the project’s role and impact in Armenia and Turkey? 9. What recommendations can improve impact? 10. What are the key obstacles that impede project objectives? 11. What recommendations can be made to minimize their effects? 12. Are there alternative approaches and activities to better reach project goals? 13. How appropriate is decentralized implementation (four local partners with Turkish counterpart organizations) to project objectives? 14. How responsive is the implementing partner (as a consortium of local organizations) to the donor and project stakeholders? 15. Did the agreement provide clear and achievable results against which progress and impact could be measured? 16. How much of an impact has USAID involvement had on project successes? 17. What recommendations can improve consortium management and operations? Limitations of the Evaluation Methodology 1. Key staff of the main consortium partner strongly felt that the national consultant held biases that precluded his objective evaluation of EPF activities. To conduct the evaluation and manage this issue, the team developed a work plan that divided key responsibilities for USAID review, discussion, and approval (Annex B). In implementing the planned work plan, the anticipated division of labor between the international and the national consultant proved impractical. As planned, the international consultant, in his role as team leader, did all of the interviews and reviews of EPF activities, but instead of the national consultant taking the lead interviewing the other partners, both team members worked together to interview and review the activities of other consortium members and their Turkish partners as well. 2. The relatively short 14-day timeframe for field work created evaluation limitations. The team could attend only a few project activities and could not interview all key partners or top government stakeholders in Armenia and Turkey. The evaluation did not have sufficient time to travel to Turkey so the evaluation was conducted only in Armenia, limiting face-to-face interviews to Turkish participants that came to Armenia, which was fortunately a substantial number since an external facilitator was holding a review of the project’s first year of work in Yerevan. Other interviews with Turkish partners, SATR Evaluation Report International Business & Technical Consultants, Inc. Page 7 beneficiaries, and stakeholders were initiated and carried out over the telephone, which made it more difficult to develop rapport and trust and to encourage openness. The findings and analysis that leads to the conclusions and recommendations are based on the products and reporting of the project as well as other written analyses on Armenia-Turkey rapprochement, the workshop, and these extensive interviews. The analysis draws conclusions from the work to date of the SATR project relative to best practices in people-to-people work in peace building and Track II diplomacy, as well as inferences about what may have been possible based on conditions in Armenia and Turkey at present.10 As other 2012 work has shown, the activities undertaken, overall strategy, and challenges the project faces are familiar ones in the 20 year history of Armenia-Turkey civil society contacts.11 The team has evaluated the work of the SATR project to date and the accomplishments of the project in light of expert judgment, the extant literature (including that of the SATR project), and these key informant interviews. III. FINDINGS The finding section aggregates and organizes empirical facts collected by the team during the evaluation. The findings are organized by objective and activities of the SATR project, project design, project management, and key results of the work to date. A. Major Achievements and Lessons Learned from Implementation The findings in this section, organized by objective, review the work to date of the SATR project. Findings for the final two objectives are combined because the conceptual thinking was similar and adjustments were made because USAID advised EPF not to engage in activities involving or targeting state officials. As a result, the objectives pertaining to the business-to￾business and civil society were strengthened. Business Partnerships and Regional Professional Networks The goal identified by the partners in this area is to generate and establish more business partnerships because they are seen as contributing to normalization. UMBA’s conceptual thinking underlying this part of the project has three stages: 1) conduct business dialog; 2) build cooperation and confidence; and 3) promote reconciliation. UMBA views the process currently in stage two, building cooperation. UMBA and its primary Turkish partner, TABDC, use two main techniques to achieve these ends: joint business meetings and research to reveal obstacles to direct trade and relations. Meetings 10 Track II diplomacy is the practice of interested, organized citizens from different states meeting to discuss issues that divide them internationally. It is distinguished from Track I which is carried out by diplomats. These processes can edge closer to official ones, by involving ex-diplomats or state officials acting in their individual capacities - which is referred to as Track 1.5. 11 Esra Çuhadar and Burcu Gültekin Punsmann, “Reflecting on the Two Decades of Bridging the Divide: Taking Stock of Turkish-Armenian Civil Society Activities” (Ankara: TEPAV, January 2012). http://www.tepav.org.tr/upload/files/1326455092- 5.Reflecting_on_the_Two_Decades_of_Bridging_the_Divide___Taking_Stock_of_Turkish_Armenian_Civil_Societ y_Activities.pdf SATR Evaluation Report International Business & Technical Consultants, Inc. Page 8 spread information and set up direct business-to-business (B2B) discussions and deals. Once the meetings have occurred, the partners can work to lobby and solve obstacles or figure out ways to get around them. To date, after the first four quarters of SATR, UMBA has conducted nine round-tables in Armenia and Turkey plus two larger conferences, each with associated and formal and/or informal sets of B2B meetings. UMBA has commissioned one study on business perceptions in both countries and conducted smaller research projects on business registration and start-up in Turkey and Armenia, textiles, transport, and freight forwarding. SATR has an explicit focus on women-led businesses that helps ensure that women are participants in other activities as well connecting women in business. Business-to-business ties are hampered by numerous factors. SATR activities touch on these difficulties, which include the following: Information problems - Implementers, beneficiaries, and stakeholders asserted that the lack of information about potential partners and ways of doing business together continues to hamper both sides. There is not an easily recognizable venue to serve as a clearing house for information either internationally, in Armenia, in Turkey, or regionally within parts of either country. As a result, particular products produced by SATR get limited visibility in business circles in part from this general information problem. For example, the UMBA-commissioned report “Armenian-Turkish Business Relations in the Eyes of Business Opinion Leaders” is not readily available or widely known, nor is the UMBA Business Startup Guide. Some initiatives are in progress to address information problems in particular business areas, such as the work of the Narekavank Tour NGO under a sub-grant from SATR, to cooperate with the TURSAB Van Chapter to share information that promotes tourism. But the problem is a general one that affects all business-to-business ties regardless of sector, which needs a more general publically available and widely used way to develop and share information in both countries in both national languages. Collective action problems - UMBA forums and other business networking events, such as the Gyumri-Kars Second Economic Forum, are able to bring business leaders together to explore potential opportunities. While this work has some multilateral and shared interests among businesses, particular firms are often focused on their own rather than general issues. Specifically, business leaders are not neutral mediators focused on common interests, rather they concentrate on their particular business interests. For example, one UMBA leader noted that meetings are useful “for our private business.”12 There are modest achievements beyond bilateral deals, such as the Travel Association efforts to stimulate regional tourism, but this regional-level association is not a strong platform to lobby for national-level changes like opening the border. Other business lobbying is in progress including work with the Van Chamber of Commerce to push for the opening of a Yerevan-Van flight, UMBA partnering with TEPAV to lobby for the end of the trade embargo. But these businesses and regional associations have modest influence in both countries, especially relative to some of the big businesses that benefit from limited competition and a quasi-monopoly in Armenia today – or the many international Turkish conglomerates that play prominent roles in Turkey. These efforts may lead to some progress, such as a work-around of customs procedures to allow the current Armenia-Turkey trade through 12 Interview at UMBA. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 9 Georgia to continue, but are unlikely to break through the larger barriers to broader economic engagement that limit both Armenia and Turkey. Asymmetries - Turkish businesses have operated in competitive domestic and international markets for many years, while Armenian firms have been limited internationally and domestically. The small Armenian market is of limited interest to Turkish firms, although companies in Eastern Turkey appear to be more interested. The two sides also have different “legal cultures” from their different pasts: Turkish firms tend to take regulations and laws as the given framework for work, while Armenian companies look for ways to get around these formal rules. The absence of international legal protection for businesses leaves agreements between companies on the two sides regulated by ethics rather than law; this is not the way Turkish firms think about contracts, although it is in consonance with Armenian business culture.13 Ad-hoc approach to Governments - Implementers in this strand do not have a consistent approach to interacting with governments on either side. This leads to scrambling, such as getting visas for Turkish green passport holders coming to Armenia for events, and a problem￾by-problem approach to addressing issues. This approach leaves trade vulnerable to shocks, such as the forthcoming unification of Georgian and Turkish customs posts which will make the current dual bookkeeping system of trade impossible.14 Governments on both sides have important leverage with large firms, as do local authorities with smaller companies. For example, Turkish authorities in Van are perceived to have actively discouraged companies from the region from participating in the Gyumri-Kars Economic Forum, which lowered the attendance and reduced the sectors covered dramatically from those anticipated beforehand. Engage civil society in alliance building to further contribute to Armenia -Turkey normalization The goal identified by the partners in this area was that “extensive and intensive engagement of civil society in the reconciliation processes will contribute to mutual understanding, assist decision-makers in resolving technical obstacles to regional integration, and will enhance the validity and legitimacy of policy actions aimed at improving relations.”15 The application noted that the “strategy aims to link a large array of civil society initiatives to strengthen their impact, maintain civil society’s momentum and accelerate progress, address existing and new challenges, and fill the gaps where more, or more effective, work needs to be done.”16 The conceptual thinking in YPC and EPF behind this part of the initiative is that the media is a powerful tool to reach the public and selected audiences and that cultural programs are widely seen as less political and provide ways for each side to learn about the other. Working with young people was added as a new opportunity to expand on areas from the “Days 2 and 3” project. The primary Turkish partners for YPC are GPoT and Izmir University and the primary counterparts for EPF are Toplum Gönüllüleri (TOG) for youth banks and Anadolu Kultur for cultural 13 Interviews with implementing partners in B2B activities, an Armenian small businessperson, and a Turkish partner businessperson, 14 UMBA interviewees explained that now Turkish trucks leave Turkey with an invoice that says the goods are being exported to Georgia which is stamped by Turkish customs at the border. Then a different invoice is stamped by Georgian customs that says the goods are going to Armenia. UMBA and its partners are working on this issue and recognize that they have two to three-months to work out a resolution – or a new work-around - of this system to enable trade to continue, which is now $210 million in Turkish imports to Armenia a year. 15 USAID, “Cooperative Agreement No. AID-111-A-10-00004,” (Yerevan: USAID, September 30, 2010), p. 14. 16 Ibid. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 10 programming. Activities in this strand through YPC include media research, media monitoring, a column exchange, media round tables, and journalism awards. Other civil society and cultural activities in this strand are through EPF: the Facebook “4th wave: Armenia-Turkey page;” television talk shows; youth bank activities and exchanges; Pecha Kucha ties; and a variety of cultural activities/exchanges. To date, through the first year of SATR, completed activities under this strand have been two meetings, the production and broadcast of four talk shows in Armenia, the development and training of three youth banks in Turkey, a youth bank visit to Armenia, a media and policy discussion in each country, a visit of Armenian journalists to Istanbul to cover the Turkish Parliamentary elections, the development and completion of a documentary on April 24 genocide commemoration in each country – shown in Armenia, a film workshop in Yerevan, a publicized award for journalism in each country, a report on media research covering both countries, and the start of an exchange of opinion columns in newspapers from both countries. It does not appear that women have been explicitly considered in this objective. SATR has also supported the production of an additional film in each country through the Armenia-Turkey Cinema Platform (ATCP) for planned viewing in 2012. Civil society strand activities are hindered by factors similar to those in business, including: Asymmetries - SATR finds that it is a challenge to get substantial public exposure for the project in Turkey. The design of parts of the media component of the project did not fit the realities of the Turkish media. It took time for the Turkish partners to educate the Armenian consortium that some activities, such as the planned talk show, were not feasible in Turkey. Some SATR events have attracted extensive media coverage in Turkey, though this has occurred mostly in Armenia. Moreover, while the television talk shows based on SATR activities are broadcast and available online in Armenia often host project participants from Turkey, the total audience watching these programs has not been large. Information problems - As with the business strand, there is no simple way for the public or interested stakeholders to find information about the project; and the push outward of information from project implementers and grantees is also limited. The Facebook page could provide a mechanism for communication among invested stakeholders because, as a closed group, it provides ways to communicate among close stakeholders of the project. However, thought there is some facilitation and management by YPC, the page does not make the material more accessible beyond simply reporting and reposting it; translation is seldom provided to spread understanding across language barriers and little discussion is facilitated through the page. In general, Turkish-speakers cannot make sense of Armenian or Russian content – and Armenian speakers cannot make sense of Turkish content (and some on both sides may not understand the English content). Little information is available in translation – and even less in both Armenian and Turkish. The absence of an open platform to serve as a clearing house for information from the media and civil society strand is notable. With that said, at the time of the evaluation EPF had begun to develop an open web-based platform to make project information more widely available. Build a favorable environment for dialogue and facilitate interaction among state actors; and Support government and non-government efforts toward rapprochement with research SATR Evaluation Report International Business & Technical Consultants, Inc. Page 11 The goal identified by the partners in this area, which correspond to two objectives of the project, is to conduct a type of Track II diplomacy and move this understanding towards government – towards Tack 1.5. ICHD’s conceptual thinking about this part of the initiative is that support from civil society is needed to help state-to-state relations. The primary Turkish partners for ICHD are TESEV and GAYA. Activities in this strand include expert discussions, town hall meetings in both countries on aspects of country relations, and research papers on Armenia￾Turkey relations in the EU context. To date, throughout the first year of SATR, project partners have conducted expert discussions and videoconferences on energy sector and regional cooperation as well as on rapprochement in both Armenia and Turkey (two meetings in each country and videoconferences largely between the partners), written and disseminated a policy brief in Armenian and English, and conducted regular town hall meetings and large-scale consultations using this methodology in both countries (five plus a large-scale meeting in each country). An explicit focus on women has not been noted in these strands. In response to the USG’s advice, the project revised the state-to-state component (Objective 3). This change weakened the potential impact of the entire project. Establishing diplomatic relations and opening the international border are both inherently governmental activities. Making progress in these areas requires state action; for the project to have an influence, reaching out to state decision makers and processes is the key to establishing diplomatic relations and opening the border. Interviewees noted on both sides that if the GOA and GOT wanted to take action on either of these aspects of international relations, they could do so without civil society. But their unwillingness to take these actions is precisely why civil society engagement and advocacy is important – to keep the issues alive, build understanding of the costs and benefits of these prospective changes, and increase understanding of the constituencies and interests involved for state actors. Activities in this strand have reached out to government counterparts and key decision makers with some effort on both sides. For example, outside of this project, TEPAV facilitated meetings in Ankara in September 2011 for consortium members with GOT officials, including the Foreign Minister. ICHD facilitates similar meetings on the sidelines of some project activities in Armenia. Only the sub-goal of working to “build the capacity of state offices and employees in Armenia and Turkey to constructively interact with their counterparts across the border” was rejected by both governments.17 Although encouraging government-to-government contacts directly has been rejected by the GOA, the project has ways to indirectly have these effects. ICHD argued that they could still facilitate civil society discussion on political topics, between citizens and their governments, and informally connect experts from one country with government officials and parliamentarians in the other in off-the-record discussions. Project events and opportunities through other projects provided ways for substantial influence. ICHD interviewees felt that USAID and EPF direction had weakened the impact of their work. ICHD has now adjusted and argued for future plans for training and networking in Armenia and Turkey. 17 Ibid., p. 18-19. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 12 Turkish partners from TESEV agreed that the work in Turkey had become more difficult when official rapprochement had stalled, but they also argued this official hesitancy made additional civil society work more important. While the GOT may officially state that they do not need Track II civil society engagement, activities of organizations in this area can raise awareness among officials and in politics. Awareness is needed since many in government and parliament for example “do not even know that the border is closed.”18 As with the business and civil society strands, concerns arise in this area over the following: Asymmetries - Turkey and Armenia are quite different in foreign policy and in civil society-state relations. In Armenia, issues of rapprochement remain relatively high on the foreign policy agenda (as well as in domestic politics). But in Turkey, especially with world and regional events since the signing of the protocols, Armenia has fallen far down the list of priorities. Information –The development of a centralized web-based clearing house is now underway and will allow for dissemination of information to government officials or other stakeholders, which was not possible before. Some project products are written in the original language and English and this limits their reach. Further, consortium partners have stopped or minimized the push of information produced by the project to the authorities. An example is in the THMs, where promoting them with the GOA and GOT as planned in the application was not done. Instead the next step was a direct youth push through “advocacy summits” as was also planned in the application. Without this active marketing to officials, these activities are another youth initiative in civil society with a different theory of change than this strand – the theory that eventually the next generation will be the ones to change inter-state politics. B. Lessons Learned from Implementation of Project Design The RFA written by USAID outlined an approach focusing on key areas in rapprochement. The coalition’s application and design in response appears to have been ambitious and, with the changes in Armenia-Turkey relations at the state level stalled, overambitious. This was a problem of both ambition and timing. In addition, not designing the RFA with the close collaboration of Turkish partners led to feasibility issues with activities planned for Turkey. The design also appears to lack a strategic analysis of rapprochement as a problem, which then leaves activities less than strategic in their targeting. Overly Ambitious The consortium’s plans outlined in their successful application in response to USAID were ambitious. The optimism about change in Armenia-Turkey relations came in early 2009 as news of the process and the roadmap to the protocols became public. Then over the spring and summer it became clear that rapprochement was finding a difficult reception in both counties leading to contentious debate in both countries, especially in domestic politics. This points to the importance of a broader understanding in both Armenia and Turkey of each other as well as to the relevance of engaging as neighbors. The Armenian consortium was put together in April 2010 as the optimistic period was closing. With the drawing back from improving relations in 18 Interview with ICHD. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 13 late 2009 and early 2010 by the two states, the plans drawn up by the Armenian partners proved overly ambitious and were overcome by events. Although the Consortium response to the USAID RFA was written while problems were becoming apparent in the prospects for the protocols, the consortium’s application appears to have reflected the optimism of the earlier period. Then the actions of political leaders in both Armenia and Turkey suggested moving forward was a much tougher proposition, as did the reception the agreement received in both societies. Although both the GOT and GOA signed the protocols October 10, the prospects for ratification at this point were troubled. By a few months into the project, many stakeholders were arguing that the protocols were essentially “dead.” Imbalances in the SATR project Asymmetries in the country context are not adequately addressed by the project. Armenia￾Turkey relations are more complicated issues in Turkish politics and more difficult to address in Turkey relative to Armenia. Although there is an apparent effort to balance the project with equivalent activities on each side of the border, these activities are much less likely to have substantial results in Turkey than Armenia. This is the case for the civil society, business, media, and state-to-state (or influence government) strands of the projects. The project matches a small state with a regional powerhouse. Turkey is much larger in terms of population and has a more diverse society. Almost 79 million people live in Turkey, while Armenia has fewer than three million people; Armenia is almost entirely ethnically Armenian, while Turkey has many ethnic minority populations, including large groups like Kurds that have been bringing this diversity into public life in recent years. Turkey has had a competitive market economy for many years with tremendous growth and engagement around the world. Armenia has had a difficult time in the transition to a market system. The Armenian GDP of some $10 billion is 1.3 percent of the Turkish GDP which is approaching $750 billion. The media business in Turkey is huge and highly competitive, with the media not independent of the main political forces in the country. Armenia is of little interest in the Turkish news. In Armenia, some of the main media sources are widely seen as controlled by the government and the reach of independent media is limited, but the Armenian media readily covers Turkey issues. The political competition between democratic political parties in Turkey was (and is) more open than the Armenian system. This difference makes the political contest over changes in foreign and domestic politics, such as relations with Armenia, more heated and the results of debate and voting uncertain. In Armenia, by contrast, while the protocols were met with hostility by some political parties and segments of society, even after their withdrawal from the governing coalition over this issue, there was no doubt that the governing Republican Party would be able to ratify the protocols through a parliamentary vote. The views of the constituents of members of parliament (MP) are thought to be much more relevant to ruling party MPs in Turkey than in Armenia. Another way there are imbalances in the project is that a project on bilateral rapprochement is designed and managed by only one side – the Armenian one. As a USAID/Armenia project, awarded to EPF, which leads a consortium of Armenian NGOs, it is clear that the project is not an equal partnership. While the project is on bilateral relations, SATR is seen as Armenia-driven with only contracts between Armenian consortium members and Turkish sub-contractors. Years SATR Evaluation Report International Business & Technical Consultants, Inc. Page 14 of work on bilateral issues by with Turkish partners by these Armenian organizations does not resolve this inequality. The project is not bilateral. This Armenian drive has several consequences. First, the objectives and activities were all conceived of by the Armenian side, and thus do not necessarily fit Turkey. Second, the structure sets up problems in management and achieving results, as will be discussed below. The third way interviewees noted imbalances was within Armenian civil society, where some civil society leaders that are not involved in the project view the SATR coalition as having a “near monopoly” on Armenia-Turkey rapprochement activities.19 During the time of the previous USAID “Days Two and Three” project, EPF, UMBA, YPC, and ICHD agreed to form a consortium to work on Turkey-Armenia issues. This formidable group of CSOs was seen as overwhelming by other smaller NGOs in Armenia. Other organizations resented what they saw as a large grant where the resources and opportunities were mainly kept within the consortium, leaving only limited funding for other organizations through re-granting. Some CSO leaders argued that they were largely shut out of opportunities to work on rapprochement through the ways the funding was allocated to a large consortium. Some argued that USAID should have instead competed multiple small awards, perhaps through an Annual Program Statement, to numerous organizations rather than compete one large award to gain the benefits of diverse approaches and multiple implementers.20 While this approach would have the potential benefit of stimulating additional entrants into rapprochement and might encourage alternative approaches, it is far from clear that this approach would better target key people, reach more people, and be more sustainable than a larger consortium-based approach that has this potential. C. Challenges in Project Management The consortium faces notable challenges in its internal relations and in ties with its Turkish partners. Issues within the Armenian Consortium In general, Armenian civil society is contentious and internally divided, comprising NGOs that are competitors. The Consortium links four different organizations with different histories, organizational attributes, goals, and views. Tensions between the organizations and their leaderships were clear in meetings and interviews. Relations worsened when the original division of labor in the project broke down approximately six months into the project with the redesign of the state-to-state component. Managing the grant with USAID and having the staff-work done by one partner creates additional tensions from communications with and reporting to the funder. Relations with Turkish Partners The Consortium is made up of Armenian organizations that contacted their regular Turkish partners with the funds in hand after winning the RFA, rather than involving them at the beginning of the RFA process. First, the lack of consultation meant that the activities under this structure had to be revised almost immediately, as work with Turkish partners demonstrated that many Consortium plans from the application for Turkey were not feasible. Second, this structure left the Turkish “partners” as not real partners, but subcontractors. This has not inspired the 19 Separate interviews with several different NGO leaders in Yerevan. 20 Interviews with CSO leaders in Yerevan. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 15 strongest commitment or drive from Turkish civil society.21 Track II negotiations are often structured to be managed by a neutral convening power. The consortium is not neutral but Armenian. While the management by one side of the disputes does not seem to have led to worsening relations between organizations that already have substantial experience working together, the system has also led to misunderstandings and smaller tensions rather than deepening cooperation and promoting sustainability. Third, the consortium has developed few new partners. As the consortium in Armenia was formed in April 2010 in anticipation of bidding on the forthcoming project, the Armenian partners all enlisted their regular counterparts in Turkey that they planned to work with. The SATR consortium only brought in one new partner, but even then EPF noted that they had already been trying to develop a relationship with TOG, a large Turkish youth volunteer network with thousands of members, and that TOG had contacted EPF in spring 2009 to partner with EPF Armenia to develop Youth Banks in Turkey. In the wake of discussions with USAID and changes in the project, the consortium has brought in some new Turkish partners that bring organizations from Eastern Turkey that are especially interested in Armenian relations into SATR. This brings welcome regional diversity and some of the actors that are the most interested in better relations into the partnership. It does not build enough support for rapprochement with organizations that focus on mainstream Turkish society and politics. The consortium and contracting relationships are thus making only a little headway in expanding the circle of NGOs active in Armenia-Turkey rapprochement in either country. Personalities Civil society organizations around the world are often driven by their leadership. The importance of individual leadership is notable across post-Soviet NGOs, and is so in Armenia and within the consortium as well. The ways the individual leaders develop the organizations that they lead has enduring effects. In the Consortium, there are very different views on relations with government partners and key questions in Armenia-Turkey relations that make collaboration and communication difficult. These disagreements are not only about what to do but how to do activities and the emphasis in the project. In other ways the heads of these Armenian CSOs are rivals competing for resources for NGO projects. Relations with Other Armenian Organizations Other civil society organizations complain about the Consortium and its partners as overly strong in Armenia, and as being barriers to competition in the sector. SATR is a large grant in the context of Armenian civil society. Open-door grant applicants and recipients noted the small size of these awards relative to the work they do and the total resources of the project. USG/USAID Relations with Consortium Partners Issues were raised by both USG/USAID staff and consortium partners about their relationship in interviews. Some project partners do not find that communications with the donor by the project have adequately represented their perspectives. Some USG stakeholders did not have much 21 The subcontracting structure and Armenian drive and design behind the project has led to dissatisfaction from both sides about bilateral civil society working relationships in SATR. In interviews, the Armenian partners were often critical of the limited responsiveness and initiative of their Turkish partners, while Turkish organizations were critical of the limited understanding of Turkey of their Armenian counterparts, the lack of consultation or discussion in their work together, and inequalities in the design and execution of the project. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 16 information on project activities. While some project activities drew high level USG interest, such as the Ambassador and Mission Director, others did not. Little Attention to Gender On the whole, SATR has not focused on gender issues in working on rapprochement. The key staff members of most organizations are male and it is not clear that the project has considered whether there are systemic gender issues that impede rapprochement or could be used to build Armenia-Turkey relations in a systematic way. On the other hand, there are areas where gender has been considered explicitly, such as in the business strand, where business women were linked to good effect. D. Overall Lessons Learned The impact of the SATR project to date appears to be limited. There are numerous reasons for these limits. Broad Challenges of Armenia-Turkish Rapprochement Making progress in Armenia-Turkish rapprochement is challenging in the current context. No stakeholders and experts on Armenia-Turkish relations interviewed from either side suggested unilateral initiatives that their governments could take to initiate change in rapprochement. Universally, interviewees felt that the next official steps in rapprochement were up to the other country’s leadership or the international community, who could push the other party into compromise. Stakeholders all favored civil society contacts across the border to reduce tension and mistrust and build productive ties between Armenia and Turkey. But many stakeholders noted that the key issues were inherently governmental and demanded government action; for example, only the two governments can choose to establish diplomatic relations and open the border. These conditions and sentiments are precisely why civil society efforts are needed – to help both societies understand issues and each other as well as explore alternatives to develop rapprochement, and to convey this knowledge and these experiences to their broader societies and own governments. Appears to be a Slow Start Awarded on the last day of FY 2010, the “the launch of the project” was not until November 23, 2010 in a Yerevan meeting with Armenian and Turkish partners.22 Despite having a pre-made consortium of Armenian partners and roster of planned activities in their application, the project was invisible for the weeks between award and launch. The launch meeting focused inward on the work and plans of the project, although there were limited guests and press attention. The first work on the project in Turkey was a follow-up Armenian-Turkish workshop in Istanbul in December 2010 that was also inward-looking with limited media attention. Quality Concerns 22 Eurasia Partnership Fund, “Support to Armenia-Turkey Rapprochement Cooperative Agreement No. AID-111-A￾10-00004: Quarterly Progress Report October – December 2010” (Yerevan: EPF), p. 2. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 17 Many stakeholders and beneficiaries expressed concerns about the quality of project results.23 The project has not developed formal monitoring and evaluation tools to work with its beneficiaries and stakeholders. Although several methods were outlined in the application, they have not been implemented by the Consortium or its partners. A strong commitment to monitoring and persistent use of participant feedback surveys, an example given in the application, would provide clearer evidence for quality from particular activities through the beneficiaries themselves.24 Including and Incorporating other Funding and Activities into Project-Specific Activities SATR blends activities on a regular basis, but it is not clear that SATR adds value to other activities. For example, UMBA notes that all kinds of meetings are held with businesses and that information is forwarded to TABDC on Armenian firms for “matchmaking.” Yet, it is not clear that SATR aids in developing these types of connections between the long-time partner organizations UMBA and TABDC; nothing in the structure of the project helps deepen partner contacts, spread information or connect companies in an open, transparent way that benefit other companies in Armenia or Turkey or other partnerships between companies across the border. Project partners that have other grants to work on Armenia-Turkey relations are sometimes wrapped into the project and outside partners from other projects are brought into SATR activities and events. While the SATR project has a cost share from the main consortium partners which encourages them to pull in other activities into the project, the ways consortium partners blend activities and funding makes it difficult to distinguish what is funded by SATR and what share of activities is SATR relative to other projects. Combining funding streams makes it difficult to identify the role of the project in joint activities. Difficult to Aggregate Results of Project Activities to Project Objectives or Goal The project has many activities, but the cumulative effects of these discrete parts of the project are difficult to find. Activities themselves appear to be treated as results of the program rather than as parts of a longer-process to develop Armenia-Turkish rapprochement. Sustainability concerns were also raised about activities and whether they would persist beyond the life of the project. Many Activities Focus within the Consortium on Member Organizations and Partners Many of the activities of SATR have focused on consortium members and a few partners; some activities simply focus on the development, review, and finalization of project products. These meetings are often expanded modestly to include additional stakeholders. There has not been much outreach to stakeholders and others in Armenia or Turkey. The results of SATR activities are used by consortium members, but have not been widely promoted in either country. There is not much reach to leaders, broad target groups, or the general population and few direct beneficiaries participate in SATR activities. Difficult to See the Development Hypothesis or Strategy of Change in Action 23 Interviews in Yerevan with civil society stakeholders. 24. USAID, “Cooperative Agreement No. AID-111-A-10-00004” (Yerevan: USAID, September 30, 2010), p. 24. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 18 SATR does not sharply focus and target its activities. The project deals with rapprochement from many different directions and approaches that leave SATR unfocused. It is not clear how the project activities and objectives are designed to come together to change the minds of influential actors in Turkey or Armenia or to shape larger social attitudes and promote change in rapprochement on both sides. Limited Impact To date, many activities have largely stopped at the time of activity completion, such as holding a meeting or producing a report – rather than extending the activity to push the results of the meeting into the media, to influential political forces or government officials, or to spread knowledge to additional businesses or social groups. Concerns about limited impact were brought up in interviews with stakeholders and partners, including from consortium partners themselves. For example, while the project did a workshop in Armenia in April to serve as a clearing house for projects in Armenia-Turkey relations, concrete information sharing remained limited to the participants and their close partners rather than receiving wide circulation through a web site.25 Through EPF, the project produces a brief quarterly overview drawn from reporting to USAID, but this document provides only the outlines of main activities and does not include how to follow-up to find out more. The publication of this abbreviated information lags behind the quarterly report in timing as well. Little High-level Influence Apparent SATR does not seem to have been thinking about stakeholders in a well defined way. A classic view in negotiation work is to define stakeholders as any actor with the potential to block progress. This view notes the important roles of opponents and potential spoilers. The project seems not to have reached out to the many potential spoiler stakeholders on both sides with information or analysis. SATR also has not focused on how to quietly pull more officials from both countries into the reach of project products and ideas. Project Implementation Perspectives Rather than a Focus on Track II to Change Project reporting and management tools focus on the delivery and execution of particular tasks and activities on a contract-like basis. There does not appear to be a focus on the project objectives and broad goals of rapprochement; the long-term view has been minimized even under a two-year grant. While the agreement between USAID and EPF provides clear and achievable results against which progress and impact could be measured, the way that these are based on activities rather than the larger objectives and overall goals has detracted attention from these higher-order ends. Only Some Ties are Sustainable On the whole, the project has not focused on sustainability. While the Armenian organizations in the consortium are likely to remain in close contact with their Turkish partners, the activities within the civil society strand are often unlikely to sustain bilateral engagement. For example, while the Turkish youth banks developed under the project may continue, they have no reason to 25 See Annex EPF-4 of Eurasia Partnership Fund, “Support to Armenia-Turkey Rapprochement” Cooperative Agreement No. AID-111-A-10-00004: Quarterly Progress Report April – June 2011” (Yerevan: EPF). SATR Evaluation Report International Business & Technical Consultants, Inc. Page 19 focus on Armenia-Turkey rapprochement or expectation that they will have funding to continue bilateral exchanges. Business-to-business contacts are more likely to be sustained based on the common incentives to work together in their particular markets such as tourism. IV. RESPONSES TO THE EVALUATION QUESTIONS PRESENTED IN THE SCOPE OF WORK A. General USAID/Armenia Evaluation Questions 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? SATR has been able to conduct a wide range of discrete activities towards its objectives. The project is or is likely to reach its expected results when these expectations were framed as numerical targets in the proposal and when project activities have been directed towards these concrete goals (e.g. “at least five mutually beneficial business projects implemented jointly by Turkish and Armenian businesses”).26 Where the original agreement set out broad expected results (e.g. “reduced stereotyping and increased appreciation of opportunities to engage in business on both sides of the border), SATR appears to have made less progress in the first year of implementation.27 SATR processes realistically should not be expected to have much of an impact on these broader trends with the Armenian-centered structure and limited resources of the project, the difficult and limited history of cooperation by business and civil society across the border, and the large size and regional interests Turkish businesses, media, and society. SATR also has not developed monitoring and evaluation tools to be able to clearly substantiate progress towards these broader types of results - but can point to specific concrete examples where project activities are implemented that affect more limited numbers of people in these areas (such as interest in business engagement across borders). 2. How sustainable are project results likely to be? (Are the processes, innovations, institutions, partnerships, linkages introduced sustainable?) The sustainability of SATR project results varies. Few project results in civil society are likely to be sustainable. The links between Armenian and Turkish NGOs depend on program funding; some will continue if NGOs can win additional projects to support their work together. Many of the business-to-business partnerships that share mutual interests and have well-founded plans to profit from their relationship together will persist as long as their relationship remains profitable. Research and advocacy products can be made more influential and sustainable with additional efforts to reach more people and influential people, plus a permanent dedicated web-platform to champion Armenia-Turkey relations and make these products accessible and promote their use. The development of a web site is now underway through SATR. 3. What were the main achievements of the program? 26 USAID, “Cooperative Agreement No. AID-111-A-10-00004” (Yerevan: USAID, September 30, 2010), p. 14. 27 Ibid.. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 20 The main achievements of SATR are keeping a wide range of efforts to improve Armenia￾Turkey relations moving forward in an environment where neither state’s authorities have encouraged progress. SATR has had important achievements in its work to continue to build interest among companies for business relationships across the border and in providing support to concrete partnerships. SATR’s civil society and media strand have helped keep project-based relationships between Armenian and Turkish NGOs going and helped to keep information about Armenia-Turkey rapprochement in the media, which is especially difficult in Turkey. SATR has been able to provide information to government counterparts through civil society organizations in each society and contribute to direct links on an informal basis between government stakeholders and influential project participants through some activities in each capital city. 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? With a history of discord and nearly 100 years of barriers that have limited almost all interaction between Armenia and Turkey, project-based support for business engagement, civil society ties, media contacts, and engaging governments is important as part of the longer-term process of Armenia-Turkey rapprochement. The substantial progress that has been made in both countries over the last decade has not built the momentum to strive for rapprochement in ways that are strong enough to overcome the many barriers to Armenia￾Turkey relations. Project support is relevant to maintain the civil society and media engagement across borders and interest within each society about the other, as well as to stimulate business contacts and relationships, encourage fair media coverage of the other country and bilateral relations, and inform (and keep some lobbying pressure on) governments. The SATR application left the theory of change and development hypothesis implicit, which has contributed to the lack of attention to a unifying theory in implementation and hampered a focus on the broader results sought through the program. 5. How did the implementer perform in terms of project management and how effective was the project leadership? The EPF, its consortium partners, and their Turkish subcontractors face many challenges in an environment in both countries that is not supportive of the main project goals. The structure of the project with a direct grant to EPF to be managed by the Consortium that combines EPF and three other strong Armenian organizations with different histories, organizational cultures, goals, and personalities making project management difficult. These challenges are compounded by the need to work in Turkey through numerous Turkish subcontractors. Neither EPF nor the board at the head of the consortium has been able to get the consortium in Armenia to manage their overall effort or many of the activities toward project objectives. EPF and consortium partners have been able to execute discrete activities under SATR, although these have sometimes been smaller, less influential among key people, and less known in the wider population than required for progress towards the challenging objectives of the project. The leadership of the project has not been able to overcome the constraints on the project by its Armenian-centered structure or the challenges of striving for substantial results in working towards the broad objectives of SATR. These problems impede the consortium in small ways such as adequately consulting and reflecting SATR Evaluation Report International Business & Technical Consultants, Inc. Page 21 partner viewpoints in quarterly reports to USAID and in larger policy differences such as reshaping the state-to-state component of SATR. 6. What lessons learned can be provided for future USAID programming in this area? Because reconciliation processes take substantial time and effort, USAID should keep project management of its implementing partners focused on the broad objectives of rapprochement and progress towards these objectives rather than focusing largely on discrete activities in many diverse areas. USAID can use best practices in reconciliation programs from around the world and from past work in Armenia-Turkey relations to try to develop more equal structures for bilateral activities, frameworks for influencing larger numbers of people and more key people, and ways to address the asymmetries in rapprochement in both countries in a successor project. Any subsequent project in this area should develop more sophisticated monitoring and evaluation tools, including strong baselines, to measure changes in attitudes and practices within both countries and focus on attribution to be able to make plausible links between project activities and any behavioral or attitudinal changes. 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? With only nine months remaining in the SATR project, the consortium should rethink some of its activities, focus on the broad objectives of the project and expected results, and redouble its efforts to influence larger numbers of people and more key people to build and sustain constituencies for rapprochement in both countries. The consortium, its partners, and subcontractors should informally connect project activities and products to government stakeholders using national NGOs, businesses, and the media to reach their own states. The project has begun development of an open web site dedicated to Armenia-Turkey rapprochement to serve as a clearing house for information from the project and other activities towards these goals. The project should continue its development with the understanding that the web site needs to be actively managed and made sustainable to have a long-term impact. The site should include all project-generated materials which should be translated into both Armenian and Turkish in ways that make clear that these products are identical in both languages. 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? (Did the agreement provide clear and achievable results against which progress and impact could be measured?) The agreement between EPF and USAID set out the four objectives, outlined the wide variety of activities in the four strands, and in general terms described anticipated outcomes (e.g. “Independent and unbiased media coverage of Armenia-Turkey relations is expanded” as point 3 of component 2). The plan proposed to monitor activities in-line with goals and general results towards these goals that are appropriate in advancing rapprochement in a fluid situation under a cooperative agreement. However, the illustrative Performance Monitoring and Evaluation Plan in the agreement then often proposed to measure processes rather than outcomes in ways that distract from focusing on achieving the goals of the project. Other proposed measures focused on broad goals outside of the project’s ability to influence (e.g. “changes in the level of mutual stereotyping in print media and TV”). The agreement also left SATR Evaluation Report International Business & Technical Consultants, Inc. Page 22 sustainability as “ensured through the key partners’ continuing work on Armenia-Turkey issues after the completion of the project” which did not focus on the need to build sustainability through means other than successor grants. 9. How much of an impact has USAID involvement had on project successes? (Analyze attribution of project successes to USAID involvement.) USAID engagement is important to demonstrate international support for Armenia-Turkey rapprochement with limited support for ties from the GOA and GOT. USAID and U.S. government support helps provide cover in Armenia from potential government opposition to project activities. In response to the USG’s advice, the project revised the state-to-state component (Objective 3). This change reduced the effort under the project to influence key state leaders that is needed to support Armenia-Turkey rapprochement. It is precisely because inter-state relations are so troubled that the project exists, and opening the border and establishing diplomatic relations are fundamentally state functions and tasks that require a focus on state actors for success. 10. Are there alternative approaches and activities to better reach project goals? (Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming.) SATR has an encompassing approach and wide range of activities that sensibly simultaneously works on civil society, business, and government relations. One alternative approach would be to target the work more directly to government officials; if welcomed by the GOA or GOT, this approach might encourage state-to-state relations in a more direct way than the research-driven methods now employed by the project to influence governments. Goals, objectives, and activities can move to emphasize this approach in implementation – and back off the government focus on a case by case basis if not productive in each individual case. USAID can consider a different approach in any future rapprochement project. As an alternative to a large consortium, the Mission can consider issuing an Annual Program Statement to fund a number of smaller awards for other ideas from a variety of organizations to develop Armenia-Turkey relations. If pursued, this approach should be combined with one that ensures that there is a central champion organization for this work and a central web site to serve as a permanent clearing-house for information on Armenia-Turkey engagement. B. SATR-specific Evaluation Questions 1. Are the project implementation approaches relevant and feasible in the current state of affairs in the Armenia-Turkey relations? SATR approaches are feasible and necessary given the current state of Armenia-Turkey relations. Approaches that support the development of mutually beneficial business-to￾business ties can be and are successful even without an open border or diplomatic relations. Civil society engagement can and does build connections between groups that share interests in Armenia and Turkey, and broadens understanding of the other country, society, and processes of civil society engagement among the wider public, opinion makers, and media figures in both countries. SATR can and does support government and non-government SATR Evaluation Report International Business & Technical Consultants, Inc. Page 23 efforts towards rapprochement with research and help to build a favorable environment for dialogue and facilitate interaction among state actors. This latter objective was put on hold based on GOA objections but key SATR partners argue persuasively that they can subtlety work directly with their own government counterparts to support Armenia-Turkey rapprochement. The current frozen condition of state-to-state relations however reduces the impact of SATR approaches in all areas. The two-year project should not be expected to successfully change Armenia-Turkey relations in a fundamental way given the longstanding nature of the bilateral issues that divide the countries, but should focus and work towards the longer-term process of building neighborly relations - including at the official state-to-state level. SATR approaches are feasible given the current state of Armenia-Turkey relations in that they can be implemented. SATR approaches are necessary given the current state of Armenia-Turkey relations; SATR support is needed because progress towards rapprochement remains limited and the constituencies for rapprochement would be weakened by the absence of support. However SATR needs to be more strategic and effective in project implementation to have more substantial effects. 2. Which specific aspects of the project are or are not working in the given political situation? [This question should be explored in the Armenian and Turkish contexts separately, because some approaches/activities may be still feasible in Armenia but not in Turkey and vice versa.] Armenian Context SATR can and should continue to work to develop business ties, civil society relations, and to influence and reach state actors directly and through civil society and the media in Armenia. While GOA officials may continue to not participate in public SATR functions and there are not opportunities for joint activities with Armenian and Turkish officials, the project can still target policy influence through informal, off-the-record activities and support both government and non-government efforts to improve relations with research. Turkish Context The SATR proposal outlined activities that the consortium’s Turkish partners argued and demonstrated would not work under current conditions in the country. SATR thus adjusted and sought and gained USAID concurrence for changes to the work plan for Turkey to meet these realities. Now as in Armenia, all of the aspects of the project work in the current political situation in Turkey. Implementation in Turkey however is more difficult than in Armenia in almost all areas since the issues of Armenia are of less importance in politics and business and of less interest to the public. This is clear for example in the continuing challenge of getting Turkish participation in the column exchange that continues to be difficult – but possible – to resolve. 3. How flexible is SATR in adjusting activities to the changing political context? SATR has adjusted its activities to fit the changing political context. Two adjustments have been prominent. The consortium changed several activities planned for Turkey through SATR Evaluation Report International Business & Technical Consultants, Inc. Page 24 discussion with their Turkish partners after the award of the grant. Partner experience was, for example, that TV talk shows on Armenia-Turkey relations were not feasible in the highly competitive Turkish media. SATR thus adapted to support media dialogues instead. In response to the USG’s advice, the project revised the state-to-state component (Objective 3), This modification of SATR has reduced the impact of the project by impeding efforts to reach key people in the GOA. 4. How has SATR built on the previous “Days 2 and 3”project? EPF worked with many of the partners in SATR in both countries during this 11-month grant on Armenia-Turkey relations with USAID. The strands of business-to-business ties, media contacts, and research and policy dialogues in SATR were also part of the structure of the prior project. EPF interviewees asserted that working with young people was added to SATR as a new opportunity to expand support for rapprochement based on the experience of the “Days 2 and 3” project. During the project in April 2010, and reported on as one of the “other program activities” in the Final Report of the “Days 2 and 3” project, the consortium partners signed a Memorandum of Cooperation and wrote their successful SATR proposal (which was attached to the final report of the “Days 2 and 3” project).28 5. What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given? SATR has developed a significant number of products and processes to support Armenia￾Turkey rapprochement. However the project has not developed adequate techniques to promote and use these products and processes with a significant number of people in Armenia or Turkey or with key people in both countries in order to have much of an impact on public attitudes or either state to date. SATR has also not developed tools to demonstrate and measure impact in these areas. SATR can identify a number of business and civil society partnerships that have been formed and/or strengthened through project processes – but again has not developed systems or tools to demonstrate the broader effects of these partnerships on rapprochement. 6. What is the external perception of the project’s role and impact both in Armenia and Turkey, according to the key stakeholders not involved in the project? Many external stakeholders interviewed were not knowledgeable about the SATR project or saw the impact of the project to date as modest. SATR does not stand out from the other projects in Armenia-Turkey relations, some of which are conducted by the same organizations and project partners with other USG support. 7. What recommendations can be provided to improve impact? What are the key obstacles and what recommendations can be made to minimize their effect? SATR needs to focus its efforts on reaching more people and reaching key people in both Armenia and Turkey in the nine months remaining under the grant. SATR should push out the knowledge and products produced to date through a dedicated public web site, translating 28 Eurasia Partnership Fund, “’Days Two and Three’ in Armenia-Turkey Rapprochement: Cooperative Agreement No. AID-111-A-10-00001, Final Progress Report, January – November 2010” (Yerevan: EPF, n.d.) p. 8-9. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 25 these products into both Turkish and Armenian when only available in one of these languages. EPF has begun to develop this public web site. SATR also should use its connected partner organizations to promote this knowledge and these products with government and political party leaders in both countries and continue to work hard to increase the visibility of Armenia-Turkey activities in the mass media, especially in Turkey. Issues of information, collective action problems, and a variety of asymmetries between the Armenian and Turkish sides are key obstacles that impede reaching project objectives. The way SATR was designed by the consortium and its management also has impeded progress – leading to a slow start of implementation, issues with the quality of project activities, a focus on activities rather than objectives and project goals, and the absence of mechanisms for the project to reach large numbers of people – especially in Turkey. The revision in the third quarter to reduce the emphasis on reaching key actors in government under the project based on GOA reluctance to engage officially (conveyed through diplomatic channels) in joint or unilateral activities on Armenia-Turkey rapprochement has also impeded efforts to improve relations. Stepped-up management, moving to address the obstacles above, and implementing the recommendations above can produce better results from SATR even as the project heads towards the mid-point of its second year of implementation. SATR can return to processes that informally inform and encourage GOA and GOT participation in activities and aggressively market project products and results with both governments. Increased efforts and stronger results are especially important in Turkey where Armenia is a significantly smaller foreign policy and domestic issue. 8. How appropriate is decentralized implementation (four local partners with Turkish counterpart organizations) to project objectives? The consortium approach is potentially appropriate but as implemented, the consortium has had problems with internal communication and links between the partners that have had negative effects on relations between the organizations and on implementation. Management needs a stronger focus to keep the focus of partners and Turkish subcontractors on the larger goals of the project rather than simply implementing their activities. There are a variety of challenges for project management within the Armenian consortium, between consortium members and their Turkish subcontractors, and between Armenian consortium members and other non-governmental organizations in the country. Consortium relations have issues due to the inter-institutional rivalries between Armenian organizations and ideological and personal differences between their leaderships. Structuring Turkish participation through contracts has meant that many of the Turkish organizations appear to feel less invested in the project than in their other work where they have more authority and control. And the contentious politics between civil society organizations in Armenia has left some NGOs that are not in the consortium feeling frozen out of potential work in Armenia-Turkey relations through SATR, which they believe is limited to the consortium and its partners. The most serious and structural issue is the need to make Turkish participants full partners in project activities with equal roles in decision-making and management; this cannot be done within the current SATR award and structure. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 26 9. How effective is the implementing partner as a consortium of local organizations: responsiveness to the donor, responsiveness to project stakeholders, information flow. What recommendations can be made to improve consortium management and operations? EPF and the members of the consortium appear to have reasonably close communications with USAID and have been responsive to USG concerns. For example, in response to the USG’s advice, the project revised the state-to-state component (Objective 3), and EPF has expanded the project partners in the second year, which also appears to have been at least in part driven by USAID. Communication and adjustment processes can take time since there are many layers to project management from USAID, EPF, the consortium partners, their Turkish subcontractors, and other partners (including “open door” grant competition winners). Consortium management and operations need to be amplified for program impact. Limited measures are possible in the nine months remaining under the grant but it is important to reach more people, reach key people, and promote sustainability in project activities and have more significant influence on Armenia-Turkey rapprochement. The consortium should:  Push for policy influence with state officials by reemphasizing the need to reach state officials in both countries;  Develop an outreach strategy and implementation to push SATR products to broader target groups;  Seek greater visibility for SATR, by increasing the visibility of the project among the public;  Craft strategic focus on improved prospects for rapprochement by increasing support for rapprochement by reaching beyond project partners;  Create an actively-managed, enduring web-based platform for information sharing, which EFP is currently working on, and have all project materials translated into both Armenian and Turkish;  Target other ways to make project outcomes more sustainable; and  Focus on fewer but more influential activities by ending activities that do not fit the approaches above. V. CONCLUSIONS The findings above and our analysis of them in light of best practices in People-to-People (P2P) peace building and Track II diplomacy projects lead to conclusions that identify a number of key issues with the SATR project. A great deal has been learned over the years from worldwide experience in P2P activities in peace building. This experience has usefully been summarized by SATR Evaluation Report International Business & Technical Consultants, Inc. Page 27 USAID.29 In addition, the analysis is informed by reflections on earlier Track II work from Armenia-Turkish relations compiled by ICHD.30 Project Has Become Activity-Driven Rather than Objective-Driven Peace building projects should be driven by their analysis of the problem. In this case, the SATR project should be driven by a strategic analysis of the key problems inhibiting rapprochement between the two states and societies. What to do about these problems should be based on what the main problems are and how to support change. However, the application did not make a clear analysis of why rapprochement has been a problem. As they implement, the project does not appear to have a clear shared understanding of the key problems that the consortium partners seek to address or a strategy to have an impact on large numbers of people or key people in either Armenia or Turkey. Modest Impact to Date Project activities appear to be modest in scale and impact at present. Now the project has low visibility even among key stakeholders in Armenia-Turkey rapprochement. Stakeholders that have not engaged with the project often do not know of the SATR project or its activities. Some stakeholders know the partners in the consortium and can associate activities with these partners, but they cannot perceive or grasp the project as a whole. Little is available in translation in the relevant languages. To reach larger numbers of people and key people in both countries, SATR products need to be translated into both national languages and made widely available in both Armenia and Turkey.31 More to Come SATR, like many projects, will have a burst of activities towards the end of the grant period, and project activities should come to fruition in the months ahead. Some areas are back loaded, including influencing government (Objective 3, now phrased as to “build a favorable environment for dialogue and facilitate interaction among state actors” and Objective 4, to “support government and non-government efforts toward rapprochement with research). It is particularly important to focus on reaching key state officials and opinion makers in society with knowledge about the results of rapprochement to date and perspectives on the benefits of greater engagement in order to build support for policy change in the medium- to long-term. Problems in Armenia-Turkey Relations Remain Critical As the USG has long-pointed out, Armenia has limited economic and political prospects without relations with its neighbors. Yet influential Armenian political and social forces note that Armenia has survived for 20 years of independence without these positive relations. Despite its 29 Office of Conflict Management and Mitigation (CMM), “People-to-People Peacebuilding: A Program Guide” (Washington: USAID, January 2011), http://www.usaid.gov/our_work/cross￾cutting_programs/conflict/publications/docs/CMMP2PGuidelines2010-0203.pdf. 30 ICHD, “Track 2 Diplomacy, Armenian-Turkish Track 2 projects: Assessment of Best Practices” (Yerevan: ICHD, 2006). http://www.ichd.org/?laid=1&com=module&module=menu&id=8&page=2&dateDay=&dateMonth=&dateYear=. 31 When products in the Armenian or Turkish are translated, they are often translated into English, which limits its reach in both countries. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 28 importance, there is still limited momentum and support for change in relations with their neighbors within Armenia. The issues of rapprochement with Armenia are not prominent ones in Turkish foreign policy or domestic politics. Project Support Remains Important for the Growth of Contacts between Armenia and Turkey Precisely because the challenges in building better relations are so great, continued USG support remains important in supporting Armenia-Turkey rapprochement. While some business-to￾business ties may be sustainable, the connections between the two societies remain at an early stage. As these issues of rapprochement have declined as governmental and societal priorities, project-based support to keep attention on the benefits and prospects for engagement between Armenia and Turkey is critical. The SATR Project Can Do Better and Do More More management is needed from USAID, EPF, the Armenian consortium partners, and their Turkish contractors to increase project impact through higher quality activities, a more strategic design to have activities contribute directly to project objectives and goals, and much greater visibility. SATR needs to reach both more people and key people in both Armenia and Turkey in civil society, business, and government. The current structure and strands should continue but refocus and aim for more substantial results through better management. Armenian and Turkish Partners Need to Engage their Own States USAID and the Embassy should revisit the decision to reduce the emphasis on government actors. The main problems that need to be addressed and the processes to do so vis-à-vis Armenia-Turkey rapprochement are first and foremost government ones. Greater attention is needed to encourage GOA and GOT consideration of rapprochement. The USG and others drove the negotiating process towards the protocols and should keep the drive going; without this push, the GOA and GOT will continue to pay little attention to each other which not only stalls state￾to-state rapprochement but also has deleterious effects on civil society and business ties. Armenian Consortium Management Needs to Become More Effective Any management system has drawbacks and assets. SATR project implementers have chosen to form a coalition, but they need to make this structure work more effectively. EPF as the secretariat of the Consortium needs to be more inspiring and demanding of its partners to press together for SATR impact. Consortium Partners Need to Inspire their Turkish Partners to Greater Efforts Consortium partners need animate their Turkish counterparts. Substantially stronger efforts are needed to have a notable impact in Turkey, but these efforts are unlikely because Turkish participants tend to view their roles as subcontractors only. This limited ownership of the project and limited commitment to project goals reduces the effectiveness of project activities. All implementing organizations, whether as management or subcontractors, need to learn to encourage and support their partners and learn to listen to and trust them in their areas of expertise. This trust can be combined with high expectations for project results that can be used SATR Evaluation Report International Business & Technical Consultants, Inc. Page 29 to inspire, challenge, and test partners - both Armenian-Armenian and Armenian-Turkish partners. With nine months left in the SATR project, there is time for modest adjustments in each of the three remaining quarters. However the structure of the Consortium and need for negotiations within it with consortium partners and Turkish counterparts and with USAID will make this process unwieldy. But the need for more results, more substantial results, and more sustainable results is critical and requires more action to achieve project goals. VI. RECOMMENDATIONS The recommendations suggested are based on the findings and conclusions stated above and are organized along the lines of the debrief provided by the evaluation team to Mission management, staff, and Embassy personnel on December 19, 2011. First, concrete actionable recommendations are outlined that focus on the current SATR project and actions that may be taken in the nine months remaining under the grant. Second, broader recommendations are presented for potential successor projects of the Mission in reconciliation or rapprochement. For the Current SATR Project The findings and conclusions above note numerous challenges that the project has faced and suggest ways the consortium and its Turkish partner organizations can make changes to increase project impact. The recommendations below make these suggestions explicit while taking into account the limited time that remains under the two-year project and the limits to USAID managerial control when working under a cooperative agreement. These latter two factors suggest that expectations for change may need to be modest. At this stage in the project, the recommendations do not suggest fundamental changes to address asymmetries or change the structure of the consortium. Some suggestions in these areas are for any future project. Target Policy Influence SATR needs to clarify project goals and invigorate its effort to influence in policy circles in both countries on issues in rapprochement. While the project has appropriately dropped plans to work directly with government officials from both sides together, influencing the GOA and GOT remains not only feasible but necessary in order to have an impact on key issues in rapprochement. Activities under project objectives can be informally brought to the appropriate government stakeholders as a central way to build governmental interest and understanding of the stakes in rapprochement. Reach Out The project needs to emphasize reaching outside of its core participants. Too many activities are limited almost entirely to project partners. Instead, project activities should bring together partners plus stakeholders and beneficiaries of all sorts to increase impact. Project products need to be used with key stakeholders to support rapprochement. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 30 Increase Visibility SATR needs to be much more visible through a focus beyond consortium partners and their Turkish counterparts. All project activities need to focus on how to get public attention as well as reach influential people in both countries. Project products need to be widely distributed to reach more people in Armenia and Turkey. SATR reports need to be translated into both Armenian and Turkish and made widely available in an easily accessible form to reach more people in both countries. EPF is now developing a public platform to serve as a clearinghouse for project￾created materials as part of the Year 2 Work Plan. Invigorating the column exchange to get editorials into the hands of Turkish readers is important, even if the papers that run these columns are not the largest circulation dailies. No interviewees suggested feasible ways to get SATR talk shows on mainstream Turkish television channels; their commercial interests in high ratings and the lack of interest within Turkish society means that since the audience is limited, so is the willingness of channels to air this content. Focus on Goals, Problems to Overcome, and Objectives not Activities The project needs to focus on the broad goal of improving prospects for rapprochement. To do so, consortium partners should review proposed activities in the remaining period of the grant and justify to USAID why these activities are expected to benefit rapprochement today and how these components will be aggregated to create larger impacts. An additional process that can encourage this broad focus is changing SATR project reporting to tell the bigger story of rapprochement rather than providing a blow-by-blow account of individual activities under project objectives as has been done to date. This focus should also facilitate broadening outreach goals. The project should give a greater role to Turkish partners since the problems of rapprochement are more difficult in Turkey and the country is more difficult to reach. While the structure of the project, remaining funding, and time limit what can be done, SATR needs a larger emphasis on reaching people in Turkey through its partners. Make Outcomes More Sustainable SATR needs to explicitly consider how to ensure that project impact can be extended to last beyond the two-years of SATR. One way to increase coherence, outreach, visibility, and sustainability is through a single, centralized web space that can champion Armenia-Turkey rapprochement. EPF has begun work in this area. This site needs to remain active after the life of the project through a commitment by site management to actively monitor/solicit and spread information on Armenia-Turkey relations. Other activities may have ways that the consortium partners could consider to lengthen interest and engagement. Consider Dropping Less-targeted Activities Some SATR activities appear less valuable than others in terms of their potential influence on Armenia-Turkish rapprochement. USAID and consortium partners should consider not implementing activities that do not fit the refocused project. Activities that do not either influence key people in rapprochement or broaden understanding by reaching more people could be dropped, and the resources redirected to help the refocusing effort. Illustrative activities that might be appropriate to drop include support for public opinion surveys on relations and the youth bank development in Turkey. Simply knowing public opinion does not work to change SATR Evaluation Report International Business & Technical Consultants, Inc. Page 31 views, and public opinions are relatively well known based on Caucasus Research Resource Center (CCRC) polls from 2010. The development of a new form for civic engagement of young people through youth banks in Turkey has only tenuous connections to rapprochement through meetings with Armenian counterparts. Consortium partners should review their proposed activities with a focus on broad impact on rapprochement, reaching key people, impacting more people, and sustainability. As part of the approval of the second year work plan, USAID should consider having consortium partners report on each of their proposed activities using these criteria and articulate how each contributes to the project goal before approving individual activities and the overall work plan. SATR needs to strategically refocus to have a larger, more targeted, broader, and longer-lasting impact. USAID and the Consortium should rapidly decide on a nine-month strategy to maximize project impact by reemphasizing the need to reach state officials in both countries, increasing support for rapprochement by reaching beyond project partners, increasing the visibility of the project among the public, targeting ways to make project outcomes more sustainable and ending activities that do not fit these tactics. Six points for this refocus should be: 1. A push for policy influence with state officials 2. An outreach strategy and implementation to push SATR products to broader target groups 3. Greater visibility for SATR 4. A strategic focus on improved prospects for rapprochement 5. A platform for information sharing that will be actively managed beyond the grant for sustainability 6. Fewer but more influential activities within the strategic focus For Any Future Projects A century of disagreement and the absence of contacts between Armenia and Turkey is the context of the long-term development challenge that must be addressed to support development in Armenia, and to support peace and security in the Caucasus region. Building support for rapprochement and change is a long-term process for both societies and both governments. It is not something that could be concluded after a two-year project. USAID should thus consider developing and supporting successor Armenia-Turkey rapprochement projects. Any subsequent project should be designed in light of the experience of the SATR project and the six points suggested in the previous section for refocusing the project in the last nine months of implementation. A successor project should also employ a greater diversity of implementers/partners, either through a number of direct grants under a mechanism like an Annual Program Statement or by working under the leadership of a neutral organization – one not from Armenia or Turkey – as an impartial broker that can smooth over relations and avoid or address any issues with asymmetry between Armenian and Turkish partners. Given the huge asymmetries in the challenges of rapprochement and the scale of the two countries, a successor project should be unbalanced rather than symmetrical. A successor project should have a substantially smaller scale in Armenia and a much larger one in Turkey to have an impact on the key barriers to rapprochement. Because the barriers to change are stronger in Turkey and because the large size of the country makes reaching the pubic and decision makers more difficult, the project should focus more effort (and funds) on supporting change in Turkey to SATR Evaluation Report International Business & Technical Consultants, Inc. Page 32 make progress on Armenia-Turkish rapprochement. Since the direct impediments to opening the border and establishing diplomatic relations are political and demand action by governments, any successor project needs to target governments and aim for a higher level influence to get ideas and discussion about policy changes to the political and policy levels in Armenia and Turkey. Interviews with experts from both Armenia and Turkey found little expectation for the opening of the border, start of diplomatic relations, or resumption of negotiations on these issues between the two governments in the near future, given the preoccupation of both governments with domestic issues, with Parliamentary and Presidential elections due in Armenia in 2012 and 2013, the crowded Turkish foreign policy agenda where relations with Armenia remain a low priority issue. There was much more optimism about the longer term trajectory for Armenia-Turkey relations based on the dramatic changes in Turkish society and the polity over the past few years as the country has become more open and tolerant about some discussion of the difficult historical issues of Turkey’s minority communities, including Armenians. A number of scholarly and cultural initiatives, such as the reconstruction of the Akhtamar Church on the island of Van and the unprecedented public commemoration of April 24th testify to this remarkable transformation. Nagorno-Karabakh and Armenia-Azerbaijan relations are bigger and more difficult issues than Armenia-Turkey rapprochement for Armenia and in contrast to Armenia-Turkey relations, current trends are worrisome rather than suggestive of progress. While Turkey-Armenia rapprochement has come a long way in a decade, the challenges of bilateral relations with Azerbaijan and the Nagorno-Karabakh peace process have become more difficult and the risks of war have increased. The government of Azerbaijan and many elements of Azeri society have hardened their perspectives on Karabakh and relations with Armenia and Armenians. These challenges pose greater problems than Armenia-Turkey relations for the development of a democratic, prosperous Armenia at peace with all its neighbors. USAID should consider support Armenia-Azerbaijan Track II and society contacts in an effort to slow the negative trends and build some understanding and contacts between the societies of each country. Any program in Armenia-Azerbaijan relations should be completely distinct from Armenia-Turkey relations. Grant support is also needed to facilitate expert research on the costs of conflict and potential benefits of compromise on Nagorno-Karabakh and resolving issues in Armenia-Azerbaijan relations and to disseminate this work to policymakers and the public in Armenia and Azerbaijan. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 33 ANNEX I: EVALUATION SCOPE OF WORK Evaluation of USAID/Armenia Armenia-Turkey Rapprochement (SATR), Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA￾ROLI), and the Small Scale Infrastructure Program (SSIP) Summary: USAID/Armenia requires evaluations of the following activities: Armenia-Turkey Rapprochement (SATR), IFES and NDI electoral and political process Associate Awards under the Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA-RO LI), and the Small Scale Infrastructure Program (SSIP). The purposes of this Task Order are to evaluate the success of these projects in their relevant areas and assess effectiveness of these in achieving set programmatic goals and the USAID/Armenia’s strategic objectives. Two of the five planned evaluations are designed as midterm evaluations (Armenia￾Turkey Rapprochement, and ABA-ROLI), while the other three are designed as end-of-project evaluations (IFES, NDI and SSIP). In the case of mid-term evaluations the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives. They will feed information into the future work plans. In the case of end-of-project evaluations, the findings will be used to inform design and development of future projects. Therefore, the evaluations will identify “lessons learned”; assess strengths and weaknesses of strategies and activities performed under these projects; and provide recommendations to USAID for project planning purposes for the next three to five years. The Contractor will seek to capture effective approaches; analyze the utility of performance monitoring efforts; consider respective outcomes and results; and assess the influence of internal and external changes on the achievement of results. Contractor Responsibilities and Projects: The evaluation should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, including an “effectiveness and efficiency assessment” that looks at how successful the programs have been in achieving their set targets, and how effectively USG resources have been used. Additionally, USAID/Armenia would like to measure the sustainability of the project results on respective beneficiaries where applicable and possible. The Contractor shall review each project’s implementation methodology and to the degree possible, verify the results achieved, the relevance of the project in addressing USAID priorities, and to what extent USAID can be accountable for achieving those objectives. Final recommendations to USAID will help improve program outcomes, weigh sustainability factors, and address program relevance as well as cost efficiency and effectiveness. The evaluations will serve to guide how similar projects, approaches and/or work plans can be improved. The evaluations will also validate (or not) the feasibility of the initial designs of the projects and of their respective development hypotheses. SATR SATR Evaluation Report International Business & Technical Consultants, Inc. Page 34 This two-year activity is in its first year of implementation. The main objective is to promote improved Armenia-Turkey relations by engaging civil society in the reconciliation processes; establishing and developing business partnerships and regional professional networks; and facilitating government-to government dialogue. The activity is being implemented in a rapidly changing environment of Armenia-Turkey relations. Therefore the appropriateness of the design of the project needs to be explored in addition to the actual project implementation issues. Thus, this evaluation should contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach and in the future planning of this project. The evaluation should aim at obtaining feedback from stakeholders and project partners both from Armenia and Turkey. CEPPS These are two separate Associate Awards under CEPPS III, implemented by IFES and NDI, both of which are in their third year of implementation and are expected to end in FY2012. Both activities are primarily aimed at improving political processes. The IFES activity supports the strengthening the administration of electoral processes and assists the electoral administration to meet international standards for free and fair elections through its support to the Central Election Commission and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR). The NDI activity aims to improve the ability of citizens to effectively participate in political processes as members of political parties and helps political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and in 2013 respectively. The findings and recommendations from this evaluation will be reviewed for results achieved, and effectiveness of program approaches. Best practices and lessons learned will be identified which will contribute to the Mission’s decisions on future election-related activities. ABA-ROLI This is in its third year of implementation and will end in FY2012. The main objectives are to support curriculum reform in the Judicial School; cooperate with the Judicial Department of the Republic of Armenia for enhancing the judicial reforms; and provide greater access to justice through Law School legal clinics with a particular focus on protection of human rights. The purpose of evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the project to inform future USAID/Armenia decision￾making with regard to similar undertakings. SSIP This is a 27-month project ending in FY2012. The goal of the project is to mitigate the consequences of Global Economic Crisis through the creation of temporary employment opportunities in vulnerable rural communities by means of implementation of small scale infrastructure projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls and drinking water supply systems. This project deals with multiple communities and partners in jointly carrying out construction and work with target communities. Executing water projects has been challenging due to different factors, including multiple construction partners, delayed contribution from other donors, weather conditions, etc. However, civil construction has been progressing timely and with very good quality. Project has been modified twice to increase the number of projects (from SATR Evaluation Report International Business & Technical Consultants, Inc. Page 35 48 to 58), labor days and expand the geographic coverage. The final evaluation will look at few aspects of the program: how the targets have been met vis-à-vis original targets and benchmarks; how the partnership between all parties (IFAD, local NGO Shen, local construction companies, village Mayors, Local Supervisors, etc.) worked; what are mechanisms that would help communities sustain project gains after its completion; did the project leverage contributions or matching funds from other donors or stakeholders as anticipated originally. Evaluation Questions: The Contractor shall review and summarize the implementation and results achieved by all five activities to answer the following evaluation questions and additional questions that may be developed by the Contractor after reviewing the provided materials. For ALL projects: 11. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? 12. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? 13. What were the main achievements of the program? 14. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? 15. How did the implementer perform in terms of project management and how effective was the project leadership? 16. What lessons learned‟ can be provided for future USAID programming in this area? 17. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? 18. Did the agreement provide clear and achievable results against which progress and impact could be measured? 19. Analyze attribution of project successes to USAID involvement. 20. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Project-specific questions/Tasks: For SATR:  Are the project implementation approaches relevant and feasible in the current state of affairs in the Armenia-Turkey relations?  Which specific aspects of the project are or are not working in the given political situation? [This question should be explored in the Armenian and Turkish contexts separately, because some approaches/activities may be still feasible in Armenia but not in Turkey and vice versa.]  How flexible is the project in terms of adjusting activities to the changing political context?  How has this activity been able to build on the advances made under the previous “Days 2 and 3”project? SATR Evaluation Report International Business & Technical Consultants, Inc. Page 36  What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given?  What is the external perception of the project’s role and impact both in Armenia and Turkey, according to the key stakeholders not involved in the project?  What recommendations can be provided to improve impact? What are the key obstacles and what recommendations can be made to minimize their effect?  How appropriate is the decentralized project implementation approach (four local partners with their Turkish counterpart organizations) to the project objectives?  How effective is the implementing partner as a consortium of local organizations: responsiveness to the donor, responsiveness to project stakeholders, information flow. What recommendations can be made to improve consortium management and operations? For CEPPS:  How can the projects be better designed in the future to measure impact, given the political situation faced in Armenia? NDI  Have NDI youth and women’s Leadership Training Academies, the NDI Women’s Candidate Schools, and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party memberships and activities, political engagements, political public gatherings, etc.) among youth and women?  Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political and policy activities.  To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations?  Have NDI/Yerevan Press Club public debates had value for participating political parties?  What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? IFES  Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code?  Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? SATR Evaluation Report International Business & Technical Consultants, Inc. Page 37  Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to electoral administration reforms?  Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? For ABA-ROLI:  What are the most notable accomplishments of the project in the ROL area since the inception of the current agreement in 2009?  What are the factors hindering judicial independence in Armenia and what has the project done to strengthen the judiciary?  Has the project been able to increase the capacity of the Armenian judiciary to play a role in making the government more accountable?  What did the project do to ensure proper implementation and enforcement of new or existing laws?  What are the activities aimed to establish mechanisms for oversight of court proceedings o How do these mechanisms affect judicial performance? o Are the existing oversight mechanisms sufficient to activity build on the existing efforts to promote accountability and transparency in the justice sector? o If yes, please name, if not, please mention gaps.  Is there capacity or interest within the legal professionals to promote reform in ROL? If yes, what does the project do to support that interest?  How does the project support the country’s only Bar Association in helping to play a balancing role within the justice sector?  What has the project accomplished in regard to defending human rights through legal protection?  How did the project affect legal education in law schools?  Where is the most viable stakeholder support for rule of law reform likely to be found?  What targeted activities could be proposed to address Armenia specific deficiencies in the justice sector?  How successfully does the project coordinate with the international community to promote human rights?  What recommendations can be made for a more effective, integrated project design? For SSIP:  What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)?  What are social, economic and other impacts of the project on target communities and beneficiaries.  How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability? SATR Evaluation Report International Business & Technical Consultants, Inc. Page 38  How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why?  How did the partnerships impact the project (timeliness, quality, and cost-benefit)?  In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met?  How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.?  Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented?  What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies?  If a similar activity is considered in the future, what should be changed in the design and technical approach? USAID’S Role in the Evaluation The USAID Mission in Armenia will: • organize a small USAID advisory group to support the Contractor in the implementation of this scope of work; • provide relevant programmatic and budgetary information to the Contractor (some relevant portions of contracts and assistance agreements are attached); • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor will: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Develop additional research and evaluation questions as needed based on the development hypothesis and on the above-mentioned evaluation questions; identify informants and stakeholders, samples and/or other relevant data sources. 3. Develop data collection tools based on the best possible methodology in accordance with the evaluation questions and feasibility considerations and provide to USAID prior to commencing field work. 4. Prepare a field work plan. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 39 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports as requested by USAID and submit final reports to USAID/Armenia for acceptance. The proposed methodology should address the need for data collection from qualitative and quantitative sources; and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. To the extent possible, data should come from facts, rather than be based on anecdotal evidence, and conclusions should be based on findings received from multiple sources. Clear, standardized data collection methodology should be described in detail to ensure reliability and consistency of the evaluation findings. Deliverables The Contractor’s deliverables shall include: 1. A written methodology plan (research design and operational work plan). 2. Provide a verbal debriefing at the end of the field work to Mission management and technical teams. 3. Prepare draft evaluation reports (electronic and hard copy) which will analyze data and summarize key findings, conclusions and recommendations. The Evaluation Report shall at a minimum contain 1) an Executive Summary; 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work, all evaluation tools, all sources of information. Submit these to USAID/Armenia within three weeks after completing the fieldwork. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. The Contractor shall submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The final reports will meet the following quality standards: a) The reports will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports shall address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings will assess outcomes and impact on males and females; f)Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of people’s opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) SATR Evaluation Report International Business & Technical Consultants, Inc. Page 40 Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. The Contractor will submit Evaluation data to USAID/Armenia along with the final reports for warehousing and future use by the Mission. The data will be in easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Evaluations Timeline 1. Evaluations should be initiated around September-October 2011. The preliminary findings of the evaluations should be submitted to USAID/Armenia immediately after the completion of the field work. 2. The first drafts of the five evaluation reports shall be submitted to USAID/Armenia no later than three weeks following the completion of the field work. (For CEPPS USAID expects to have two separate reports – one on the IFES activity and one on the NDI activity). 3. The final Evaluation Reports shall be submitted no later than two weeks after final comments on the draft evaluation reports are submitted by USAID/Armenia. 4. Once finalized, the contractor is responsible for ensuring that the final approved reports are also submitted to USAID’s Development Experience Clearinghouse within three months of the completion of the reports. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 41 ANNEX II: SATR WORK PLAN INTRODUCTION Armenia-Turkey relations have been troubled for the entire period of Armenian independence, and the border has been closed since 1993. Diplomatic activity by both the Government of Armenia (GOA) and the Government of Turkey (GOT) in 2007 led to the drafting of protocols on restoring relations and opening the border in 2008, which were not ratified by either country. PROJECT PURPOSE USAID/Armenia developed a request for proposals to carry out a program to support the opening in Armenia-Turkey relations. The USAID Mission selected the Eurasia Partnership Foundation (EPF) to implement SATR. The EPF proposal built a consortium with the Union of Manufacturers and Businessmen of Armenia (UMBA), International Center for Human Development (ICHD), and Yerevan Press Club (YPC) to implement the project. The four partner organizations have been at the forefront of Armenia-Turkey rapprochement and have played a major role in civil society, business, media, youth, cultural dialogue, and track two and people’s diplomacy. Each of the organizations has partnerships with counterpart organizations in Turkey. The main objective of USAID Armenia’s Support to Armenia-Turkey Rapprochement (SATR) project is to promote improved Armenia-Turkey relations. The purpose of SATR was to engage civil society in reconciliation processes; establish and develop business partnerships and regional professional networks; and facilitate government-to-government dialogue. The initial project objectives were to: 1. Develop new business partnerships and regional professional networks; 2. Engage civil society in alliance-building to further contribute to Armenia-Turkey normalization; 3. Build a favorable environment for dialogue and facilitate interaction among politically active groups and policy advocates; 4. Support multi-lateral efforts toward rapprochement with research. The project began 1 October 2010, and is planned to continue until 31 September 2012. The Armenia-Turkey rapprochement process at the government level has remained frozen rather than thawed as hoped at the time the project was designed. However, public diplomacy initiatives between the two countries have continued or intensified. The evaluation of SATR will examine the ways USAID, EPF, and its partners and grantees have worked in this difficult political climate and adjusted to advance project objectives and the overall goal of rapprochement. EVALUATION PURPOSE Support for Armenia-Turkey Rapprochement is into the second year of implementation of the two-year activity. SATR is being implemented in a rapidly changing environment of Armenia￾Turkey relations. Given the context described above, the scope of work (SOW) for the evaluation includes exploring the appropriateness of the project design and the strategic adjustment made SATR Evaluation Report International Business & Technical Consultants, Inc. Page 42 during the first year. As with any mid-term project evaluation, the SOW also asks the evaluation to focus on project implementation and any issues identified by USAID, the implementing partner EPF, its partners in Armenia and Turkey, project beneficiaries, and project stakeholders. The evaluation will contribute to the Mission’s understanding of whether or not further adjustments are necessary in the approach and future planning of SATR. The evaluation will gather feedback from stakeholders and project partners both from Armenia and Turkey to help Mission management of the project. The evaluation will provide answers to 17 questions (See Annex I), which have been adapted from the questions in the SOW for SATR. During the fieldwork, the evaluation team will evaluate role/impact of SATR on women and men. TEAM MEMBERS The SATR evaluation will be carried out by a team of two - an international expert on conflict mitigation and reconciliation and a local expert knowledgeable on Armenian-Turkish relations and the current state of affairs. Lawrence Robertson has a doctoral degree in political science and has worked on conflict transformation as an independent consultant on conflict, stabilization, and democracy in international development since 2006, as Senior Conflict Specialist in the Office of Conflict Management and Mitigation in the Bureau for Democracy, Conflict, and Humanitarian Assistance in USAID/Washington 2004-2006, and as a Senior Social Science Analyst in the Program Office of the Bureau for Europe and Eurasia in USAID/Washington 1998-2003. He led a team to Armenia to conduct the first conflict assessment for USAID in 2002 and helped facilitate the development of the Mission strategy in 2003. Stepan Grigoryan is Chair of the Board of the Analytical Center on Globalization and Regional Cooperation in Yerevan. A former Member of Parliament, Dr. Grigoryan has also served as Extraordinary and Plenipotentiary Minister of the Republic of Armenia in the Russia Federation. He is the author of numerous works on Armenian foreign policy. METHODOLOGY To evaluate the SATR project, the IBTCI team will review project documents, interview project management and staff, and interview partners, beneficiaries and stakeholders of the project’s activities. Robertson will take lead responsibility with the Eurasia Partnership Foundation (EPF), which is responsible for the overall management of the project including oversight of sub-grants to the three Armenian partner organizations: Union of Manufacturers and Businessmen of Armenia (UMBA), International Center for Human Development (ICHD), and Yerevan Press Club (YPC). EPF manages the Cooperative Agreement and monitors the work/grants of the other three partners, reports to USAID, and carries out some activities (most activities under the three components are delegated to the consortium partners or other grantees). EPF has an Armenia￾Turkey Civil Society Project Manager specifically responsible for the management of EPF’s Media and Civil Society work under the project as well as the open-door grants. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 43 Robertson, assisted by Grigoryan, will also work directly with the consortium members - the Union of Manufacturers and Businessmen of Armenia (UMBA), International Center for Human Development (ICHD), and Yerevan Press Club (YPC). He will examine their work in implementing their SATR grants and work with their Turkish partner organizations. The focus will be on how they carried out their work plans, their relations with other Armenian organizations and grantees, and their ties with Turkish organizations. The evaluation will also examine the perspective of a sample of participants and individual beneficiaries of these activities as the three consortium partners carried out the three components of the program Document Review: The team will review all project documents, including quarterly reports, work plans, and modifications of the program. Assessment of the project M&E system: The team will examine, monitor, and evaluate the use of the PMP, annual work plans and the proposed results therein by the consortium. Units of analysis: The evaluation will examine the performance of organizations and individuals towards project objectives, and the impacts of the project on both individuals and organizations. Interviews: The team’s primary methodology will be face-to-face semi-structured interviews with key informants, including:  Eurasia Partnership Foundation (EPF)  Union of Manufacturers and Businessmen of Armenia (UMBA)  International Center for Human Development (ICHD)  Yerevan Press Club (YPC)  Golden Apricot NGO  Successful and unsuccessful applicants for support Open Door Armenia-Turkey initiatives Annex II contains three sets of illustrative open-ended questions that will be used during the key informant interviews with the Implementing Partners and Grantees; Stakeholders; and Beneficiaries. The methodology for conducting the follow-up on interviewee responses is explained and illustrated in Annex III. Likely interviewees include:  Gevorg Ter-Gabrielyan, EPF  Vazgen Karapetyan, EPF  Varya Meruzhanyan, EPF  Arthur Ghazaryan, the Business Strand Manager with UMBA  Kamo Mailyan, Program Coordinator, UMBA SATR Evaluation Report International Business & Technical Consultants, Inc. Page 44  Tevan Poghosyan, ICHD  Ashot Khurshudyan, research, young leaders ICHD  Vahan Asatryan, research, training ICHD  Boris Navasardian, YPC  Mikayel Zolyan, YPC  Alice Ter-Ghevondyan, EPF  Heghine Manasyan, CCRC The team will conduct semi-structured telephone interviews with key informants in Turkey, including representatives from the following:  International Transporters’ Association of Turkey  Association of Turkish Travel Agencies  Economic Policy Research Foundation of Turkey  Toplum Gönüllüleri  Global Political Trends Center  TABDC  Anadolu Kultur Likely interviewees include:  Murat Çağatay, GAYA  Ozgun Ozcer, Taraf  Hakki Ozdal, Radikal  Dr. Mensur Akgun  Silvia Tiryaki  Susae Elanchenny  Narod Maraslyan  Sylvia Tiryaki, GPoT  Can Yirik, GPoT  Noyan Soyak, TABDC SATR Evaluation Report International Business & Technical Consultants, Inc. Page 45  Aybars Görgülü, TESEV  Sabiha Senyücel Gündoğar, TESEV Geographic coverage: The team will focus on implementing partners, beneficiaries, and stakeholders in Yerevan, reaching partners, stakeholders, and beneficiaries in Turkey via Skype, e-mail, and telephone. The team may visit Gumryi or other regional partners, beneficiaries, and stakeholders in Armenia. DELIVERABLES AND WORKPLAN Deliverables, with due dates, are: 1. A written methodology plan (research design and operational work plan) – Monday, December 5, 2011. 2. Verbal debriefing at the end of the field work to Mission management and technical teams – Monday, December 19, 2011. 3. Draft evaluation report (within three weeks of completing the fieldwork, electronic) – Monday, January 9, 2012 The draft report will contain 1) an Executive Summary; 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action oriented recommendations, organized around the evaluation questions defined for the project above; and 5) Annexes, including the Scope of Work, all evaluation tools, interviews conducted and all sources of information. The report will address all evaluation questions in the scope of work by objectively evaluating what worked in the project, what did not, and why. Evaluation findings will assess outcomes and the impact on males and females. Any limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology. Evaluation findings will be presented as analyzed facts, evidence and data and not based on anecdotes. The opinions of interviewees will, however, inform the evaluation process. All sources of information will be properly identified and listed in an annex. Recommendations will be supported by specific findings and be action-oriented, practical and specific, with defined responsibility for these actions. USAID/Armenia expects to comment on the draft report within three weeks of submission. 4. Final evaluation report (within two weeks after receipt of USAID's comments on the draft) - on or about Monday, February 20, 2012. The following table provides a detailed work plan with deliverables and timing. The fieldwork will be conducted in Armenia December 5 – 20 (14 work days). SATR Evaluation Report International Business & Technical Consultants, Inc. Page 46 Table 1: Field Work Plan for SATR Evaluation Team Week Day/date Planned activities Deliverable Team Leader Team Member Week 1 Mon, Dec 05 Initial Meeting with USAID COTR to discuss this work plan, initial meeting with EPF Detailed Evaluation Design and proposed Work Plan Robertson Grigoryan Tues, Dec 06 Initial Meetings with consortium partners Robertson Grigoryan Wed, Dec 07 Fieldwork Robertson Grigoryan Thurs, Dec 08 Fieldwork Robertson Grigoryan Fri, Dec 09 Fieldwork Robertson Grigoryan Sat, Dec 10 Fieldwork Robertson Grigoryan Week 2 Mon, Dec 12 fieldwork Robertson Grigoryan Tues, Dec 13 fieldwork Robertson Grigoryan Wed, Dec 14 fieldwork Robertson Grigoryan Thurs, Dec 15 fieldwork Robertson Grigoryan Fri, Dec 16 reach findings, conclusions, recommendations Robertson Grigoryan Sat, Dec 17 prepare debrief Robertson Grigoryan Week 3 Mon, Dec 19 prepare and present debrief 2. Verbal debrief to Mission management and technical teams Robertson Grigoryan Tues, Dec 20 Team Leader departure Robertson SATR Evaluation Report International Business & Technical Consultants, Inc. Page 47 KEY INFORMANT INTERVIEWS The final report will include Annex II – a list of all people or groups interviewed, the organization(s) that they represent, their contact information, and the date of the interview. The team will ask different open-ended questions, and follow up on interviewee responses, to gather information for the findings. The following three lists of questions are illustrative of the kinds of questions we will ask (when appropriate). Illustrative Questions for SATR Implementing Partners and Grantees 1. Have changes in Armenia-Turkey relations affected your work under SATR? How? 2. Are there aspects of your work under SATR that work better or less well due to the current political situation in Armenia? 3. Are there aspects of your work under SATR that work better or less well due to the current political situation in Turkey? 4. Have you been able to adjust your work under SATR to the changing political context? How? 5. Has SATR built on the previous EPF “Days 2 and 3” project? How? 6. What concrete examples from your work demonstrate the impact of SATR to date? What are the mid-term impacts of your work? 7. How has your work addressed sustainability? How sustainable do you think your project results are likely to be? 8. How do you think key external stakeholders perceive SATR’s role and impact in Armenia and Turkey? 9. Do you have any recommendations that you think would improve the impact of SATR? 10. What do you see as key obstacles that impede the achievement of SATR objectives? 11. What recommendations would you make to minimize the effects of these obstacles on achieving SATR objectives? 12. Do you think there are alternative approaches and activities that might better reach SATR’s goals? What might these approaches and activities look like? 13. Do you find that the decentralized implementation is beneficial to project objectives? How or how not? 14. How do you respond to requests from USAID and project stakeholders? 15. Do you have clear and achievable results outlined in your agreement with EPF (USAID) against which progress and impact could be measured? What are these expected results? SATR Evaluation Report International Business & Technical Consultants, Inc. Page 48 16. How has USAID been directly involved in your work under SATR? What affect to you think this involvement has had on project successes? 17. Do you have any recommendations to improve consortium management and operations? Illustrative Questions for SATR Stakeholders 1. What is your relationship with the SATR project? 2. How do you think current Armenia-Turkey relations affect the SATR project? 3. Are there aspects of SATR that do not work in the current political situation in Armenia? 4. Are there aspects of SATR that do not work in the current political situation in Turkey? 5. Do you think SATR has been flexible in adjusting activities to the changing political context? How? 6. Do you know of EPF’s previous “Days 2 and 3”project? If so, how has SATR built on this project? 7. Do you see concrete examples and evidence that demonstrates the mid-term impact of SATR? 8. How sustainable are project results likely to be? 9. Do you have any recommendations to improve the impact of SATR? 10. What do you see as key obstacles that impede reaching SATR objectives? 11. Do you have any recommendations to minimize the effects of these obstacles? 12. Do you think that there are alternative approaches and activities to better reach SATR’s goals? 13. Do you find that the implementing consortium is responsive to you and your colleagues? 14. Do you see a direct role of USAID in the SATR project? If so, what has USAID’s management role been? 15. Do you have any recommendations to improve consortium management and operations? Illustrative Questions for SATR Beneficiaries 1. What is (was) your relationship with the SATR project? 2. Has current Armenia-Turkey relations affected your work with SATR? If so, how? 3. Did SATR try to adjusting its activity with you to the changing political context? SATR Evaluation Report International Business & Technical Consultants, Inc. Page 49 4. Do you know of EPF’s previous “Days 2 and 3”project? If so, has SATR built on this previous project? 5. What do you think the impact of your work with SATR has been to date? Why do you think your work has had this impact? 6. How sustainable do you think your work with SATR has been? 7. How do you think key external stakeholders in Armenia and Turkey perceive your work with SATR? 8. Do you have any recommendations to improve the impact of your work with SATR? 9. What do you see as key obstacles that impede your work with SATR? 10. Do you have any recommendations to minimize their effects? 11. Are there alternative approaches and activities that you think would work better reach the goals of your work with SATR? 12. Who has been your partner in your work with SATR? 13. Do you think this partner been responsive to the donor and project stakeholders? How? 14. Have you noted the involvement of USAID staff in the SATR project? If so, how? 15. Do you have any recommendations to improve your partner’s management and operations under SATR? FOLLOW-UP TECHNIQUES IN INTERVIEWS One of the challenges in evaluating the SATR project is the number and diversity of activities of the four partners in the consortium, their Turkish partners, and other grantees. These activities are also part of the overall context in Armenia and Turkey where there are other projects that work on Armenia-Turkey relations, including ones that are implemented by these same partners and grantees. Consortium members, their partners, grantees and beneficiaries all work across￾issues and across strands of the project. This reality makes it not feasible to design separate questions for the different categories of implementers, beneficiaries, and stakeholders. These multiple contacts and roles suggest an interview strategy that begins with open, relatively general questions to key informants and an active, back-and-forth follow-up style to move from these general opportunities for informants to say whatever is on their minds to target specific aspects of the environment around the SATR project, activities of the project, and outcomes of project work. This strategy works for implementers, beneficiaries, and stakeholders. For example, starting with “What is (was) your relationship with the SATR project?” as a question to a beneficiary, quickly leads to several tracks of discussion about information flow, what they did with the project, how they felt the activity went, the roles of other organizations, SATR Evaluation Report International Business & Technical Consultants, Inc. Page 50 and its impact on their organization and potentially Armenia-Turkey relations. Follow up questions include things like: How did you find out about _____________ (the activity)? What did you do with _____________ (the activity)? Who else was there? How did you engage with them? How was _____________ (the activity) conducted? What it professionally done? How were any sensitive issues handled? What was the impact of _____________ (the activity) on your organization? Do you think this impact will be a lasting one? Do you think _____________ (the activity) had an impact on Armenia-Turkish relations? How? The approach also avoids potentially biasing the evaluation. Deciding on detailed questions before the evaluation rather than developing them through the interview process telegraphs to implementers, beneficiaries and stakeholders what aspects of the program to talk about. This would bias the evaluation since these may or may not be what the key informant has actually worked on within the program or the main areas they believe are important to discuss with the team. Instead we want the interviews to start with how they understand and see the project. Thus, for the SATR evaluation, starting with general questions and then focusing in and following up on their initial thoughts promises to be a superior strategy to deliver valid and reliable answers about what project implementers have focused on as well as what the project has meant to beneficiaries and stakeholders. This avoids the potential problem that detailed questions to guide interviews might create – distortions in the experience and perspectives of implementers, beneficiaries and stakeholders that they convey to the team through the direction provided by the guides. Starting with general questions organized by broad target groups above (see ANNEX II) with implementers, beneficiaries, and stakeholders has worked well to provide us with their SATR objectives (as they understand them) and elicit a dialog about specific interests, project objectives, activities, and achievements. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 51 ANNEX III: BIBLIOGRAPHY Anderson, Mary B. and Lara Olson. April 2003. Confronting War: Critical Lessons For Peace Practitioners. Cambridge: The Collaborative for Development Action, Inc.. http://www.cdainc.com/cdawww/pdf/book/confrontingwar_Pdf1.pdf Busnessin Andarag LTD and Turkish-Armenian Business Development Council (TABDC). 2011. “Armenian-Turkish Business Relations through the Eyes of Business Opinion Leaders: Study Report.” UMBA: Yerevan. Church, C. and Rogers, M. 2006. Designing for Results: Integrating Monitoring and Evaluation in Conflict Transformation Programs. Washington: Search for Common Ground. http://www.sfcg.org/programmes/ilr/ilt_manualpage.html Çuhadar, Esra and Burcu Gültekin Punsmann. January 2012. “Reflecting on the Two Decades of Bridging the Divide: Taking Stock of Turkish-Armenian Civil Society Activities.” Ankara: TEPAV. http://www.tepav.org.tr/upload/files/1326455092- 5.Reflecting_on_the_Two_Decades_of_Bridging_the_Divide___Taking_Stock_of_Turkish_Ar menian_Civil_Society_Activities.pdf de Waal, Thomas. October 2010. “Armenia and Turkey: Bridging the Gap.” Policy Paper No. 87. Washington: Carnegie Endowment for International Peace. http://carnegieendowment.org/files/armenia_turkey.pdf Eurasia Partnership Fund. n.d. “Support to Armenia-Turkey Rapprochement Cooperative Agreement No. AID-111-A-10-00004: Quarterly Progress Report July – September 2011.” Yerevan: EPF. Eurasia Partnership Fund. September 30, 2011. “Support to Armenia-Turkey Rapprochement Cooperative Agreement No. AID-111-A-10-00004: Work Plan, Year Two (October 1, 2011 – September 30, 2012.” Yerevan: EPF. Eurasia Partnership Fund. n.d. “Support to Armenia-Turkey Rapprochement Cooperative Agreement No. AID-111-A-10-00004: Quarterly Progress Report April – June 2011.” Yerevan: EPF. Eurasia Partnership Fund. n.d. “Support to Armenia-Turkey Rapprochement Cooperative Agreement No. AID-111-A-10-00004: Quarterly Progress Report January – March 2011.” Yerevan: EPF. Eurasia Partnership Fund. n.d. “Support to Armenia-Turkey Rapprochement Cooperative Agreement No. AID-111-A-10-00004: Quarterly Progress Report October – December 2010.” Yerevan: EPF. Eurasia Partnership Fund. n.d. “’Days Two and Three’ in Armenia-Turkey Rapprochement: Cooperative Agreement No. AID-111-A-10-00001, Final Progress Report, January – November 2010.” Yerevan: EPF. European Stability Initiative (ESI). January 2011. “Turkey – Armenia Manual.” Berlin: ESI. http://www.esiweb.org/pdf/esi_picture_story_-_turkey_armenia_manual.pdf SATR Evaluation Report International Business & Technical Consultants, Inc. Page 52 ESI. August 2009. Armenia-Turkey: The Great Debate.” Berlin: ESI. http://www.esiweb.org/pdf/esi_picture_story_-_turkish_armenian_relations_-_august_2009.pdf Göksel, Diba Nigar. 28 October 2011. “Turkey’s Caucasus Policies in the Framework of Ankara’s New Foreign Policy.” Caucasus Analytical Digest No. 30: Emerging Regional Powers: Turkey and Iran in the South Caucasus. Zurich: Swiss Federal Institute of Technology. http://www.isn.ethz.ch/isn/Digital-Library/Publications/Detail/?ots591=0c54e3b3-1e9c-be1e￾2c24-a6a8c7060233&lng=en&id=133750 Göksel, Diba Nigar. September 26, 2011. “Can Turkey Inspire? Part II Turkish Civil Society: From Black Sheep to Synergy in the Black Sea.” Washington: German Marshall Fund of the United States. http://www.scribd.com/doc/66533076/Can-Turkey-Inspire-Part-II-Turkish-Civil￾Society-From-Black-Sheep-to-Synergy-in-the-Black-Sea Göksel, Diba Nigar. May 2010. “The Rubik’s Cube of Turkey-Armenia Relations.” UNISCI Discussion Paper No 23. Madrid: Research Unit on International Security and Cooperation (UNISCI). http://independent.academia.edu/NigarGoksel/Papers/438835/THE_RUBIKS_CUBE_OF_TUR KEY_ARMENIA_RELATIONS Görgülü, Aybars, Iskandaryan, Alexander, and Mınasyan, Sergey. May 2010. “Turkey-Armenia Dialogue Series: Assessing the Rapprochement Process.” Working Paper. Istanbul and Yervan: TESEV and Caucasus Institute. http://www.caucasusinstitute.org/wp￾content/uploads/2011/03/CI-TESEV-2010.pdf Görgülü, Aybars. November 2008. “Turkey-Armenia Relations: A Vicious Circle.” Istanbul: Turkish Economic and Social Studies Foundation (TESEV). http://www.tesev.org.tr/UD_OBJS/PDF/DPT/ERM/Turkey￾Armenia%20Relations%20A%20Vicious%20Circle.pdf International Crisis Group (ICG). 14 April 2009. “Turkey and Armenia: Opening Minds Opening Borders.” Europe Report No. 199. Brussels: ICG. http://www.crisisgroup.org/~/media/Files/europe/199_turkey_and_armenia___opening_minds_o pening_borders_2.pdf International Center for Human Development (ICHD). 2006. “Track 2 Diplomacy, Armenian￾Turkish Track 2 projects: Assessment of Best Practices.” Yerevan: ICHD. http://www.ichd.org/?laid=1&com=module&module=menu&id=8&page=2&dateDay=&dateMo nth=&dateYear= Iskandaryan, A., ed. June 2011. “Исследовательские записки Института Кавказа №. 4.” Yerevan: Caucasus Institute. http://www.caucasusinstitute.org/wp￾content/uploads/2011/03/Policy_brief_041.pdf Iskandaryan, Alexander, and Sergey Minasyan. January 2010. “Pragmatic Policies vs. Historical Constraints: Analyzing Armenia-Turkey Relations.” Yerevan: Caucasus Institute. http://www.caucasusinstitute.org/wp-content/uploads/2011/02/CI-RP_1_en.pdf SATR Evaluation Report International Business & Technical Consultants, Inc. Page 53 Iskandaryan, A.. 2008. Caucasus Neighborhood: Turkey and the South Caucasus. Yerevan: Caucasus Institute. http://www.caucasusinstitute.org/wp￾content/uploads/2011/03/Caucasus_Eng3.pdf Mikaelian, Hrant. April 2010. “Armenia and Armenians, Turkey and Turks in Armenian Media.” Yerevan: Caucasus Institute. http://www.caucasusinstitute.org/wp￾content/uploads/2011/03/armtur_en.pdf Office of Conflict Management and Mitigation (CMM). January 2011. “People-to-People Peacebuilding: A Program Guide.” Washington: USAID. http://www.usaid.gov/our_work/cross￾cutting_programs/conflict/publications/docs/CMMP2PGuidelines2010-0203.pdf Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee. 2008. Guidance on Evaluating Conflict Prevention and Peacebuilding Activities. Paris: OECD. http://www.adb.org/Documents/Papers/DAC-Guidance/Approach￾DACGuidance.pdf. Phillips, David L. 2005. Unsilencing the past: track two diplomacy and Turkish-Armenian reconciliation. New York: Berghahn Books Reimann, Cordula. 2007. “Assessing the State-of-the-Art in Conflict Transformation - Monitoring, Evaluation and Learning for Fragile States And Peacebuilding Programs: Practical Tools For Improving Program Performance And Results.” Washington: USAID. http://www.socialimpact.com/resource-center/downloads/fragilestates.pdf Ross, Marc H. May 2001. “Action Evaluation in the Theory and Practice of Conflict Resolution.” Peace and Conflict Studies, Vol. 8, No. 1. http://www.gmu.edu/academic/pcs/Ross81PCS.htm TESEV. 30 June 2011. “Policy Brief on Turkey-Armenia Relations I.” Istanbul: TESEV. http://www.tesev.org.tr/UD_OBJS/PDF/DPT/ERM/Policy%20Brief_Off%20the%20Record%20 Meeting.pdf TESEV. 30 June 2011. “Policy Brief on Turkey-Armenia Relations II.” Istanbul: TESEV. http://www.tesev.org.tr/UD_OBJS/PDF/DPT/ERM/Policy%20Brief_Off%20the%20Record%20 Meeting_II.pdf Turan. İlter. August 10, 2011. “Exit the Commanders.” Washington: German Marshall Fund of the United States. http://www.scribd.com/doc/63491955/Exit-the-Commanders USAID. September 30, 2010. “Cooperative Agreement No. AID-111-A-10-00004.” Yerevan: USAID. Yerevan Press Club. 2009. “Armenian-Turkish Team Reporting Project.” Yerevan: YPC. http://www.ypc.am/upload/Arm-Turk_book2009_eng.pdf Zaman, Amberin. April 28, 2011. “The Turkish Media: At Long Last Turkey Becomes a World Leader!” Washington: German Marshall Fund of the United States. http://www.scribd.com/doc/54133053/The-Turkish-Media-At-Long-Last-Turkey-Becomes-a￾World-Leader SATR Evaluation Report International Business & Technical Consultants, Inc. Page 54 ANNEX IV: CONFLICT OF INTEREST STATEMENTS SATR Evaluation Report International Business & Technical Consultants, Inc. Page 55 SATR Evaluation Report International Business & Technical Consultants, Inc. Page 56 ANNEX V: STATEMENT OF DIFFERENCE Statement of Difference Response of EPF to SATR Mid-Term Evaluation Report Introduction EPF provides this response to the mid-term evaluation report of the Support to Armenia Turkey Rapprochement (SATR) project, produced by International Business & Technical Consultants, Inc. (IBTCI) on behalf of the United States Agency for International Development (USAID). As outlined below, EPF disputes the accuracy of some of the information included in the evaluation report and also questions some of the conclusions drawn by the evaluators. EPF recognizes that there are a number of relevant recommendations made to USAID and EPF for both this project and future Turkey-Armenia rapprochement work; however, the tone and structure of the report, in addition to the pre-existing conflict of interest with the local expert serving as a member of the evaluation team, undermine the credibility and utility of the report for the implementers. EPF is concerned by the fact that the Executive Summary does not accurately reflect the findings of the report, but rather presents a more pessimistic assessment of the project. On page 20, the evaluators included a more or less fair assessment of the achievements of the SATR project, which we highlight here: “The main achievements of SATR are keeping a wide range of efforts to improve Armenia￾Turkey relations moving forward in an environment where neither state’s authorities have encouraged progress. SATR has had important achievements in its work to continue to build interest among companies for business relationships across the border and in providing support to concrete partnerships. SATR’s civil society and media strand have helped keep project-based relationships between Armenian and Turkish NGOs going and helped to keep information about Armenia-Turkey rapprochement in the media, which is especially difficult in Turkey. SATR has been able to provide information to government counterparts through civil society organizations in each society and contribute to direct links on an informal basis between government stakeholders and influential project participants through some activities in each capital city.” SATR Evaluation Report International Business & Technical Consultants, Inc. Page 57 In what follows, EPF outlines discrepancies in the evaluators’ facts, argumentation, and conclusions. Impact The mid-term evaluation report repeats in several places that the SATR project had set overly ambitious goals (pp. ii, 12, 13, 18 etc). While this may be correct for the state-to-state strand, the objectives in the civil society and the business strands are likely to be reached, and the report itself, in other places (p. ii), agrees with this. The evaluators also suggest that when designing a project of such scale, it is recommended to have a theory of change and a risk analysis, which are based on more than one version of possible developments. EPF and its partners based their calculus on three different scenarios: the best case, the medium case, and the worst case. The reality proved to be a mixture of the medium and worst case and the project goals and activities have been modified accordingly and under the advisement of USAID. The report asserts that it will be difficult to measure the impact of the SATR project. Unfortunately, the evaluators make reference only to the illustrative Performance Monitoring and Evaluation Plan included in EPF’s original proposal (p. 22), while overlooking the revised SATR Performance Management Plan (approved by USAID in early December 2011). The evaluators also suggest that USAID consider cutting the parallel (baseline) public opinion research (p. 31). For funding reasons, this activity has already been removed. Nonetheless, EPF remains convinced that this type of evidence-based research would provide useful information for assessing the impact of the SATR project, and regrets that this possibility is not included in the current agreement. The mid-term evaluation report likewise claims that the qualitative impact envisioned by EPF will be difficult to measure (i.e. “reduced stereotyping and increased appreciation of opportunities to engage in business on both sides of the border”). While in June 2011 it was impossible to include a business expo component in the business conference, by November 2011, after several months of lobbying and careful preparation, an additional large-scale business conference did include a business expo. Similarly, after the demise of the ‘football diplomacy,’ the mainstream Turkish TV channels would hardly air positive reports on Armenia or the prospects of Armenia-Turkey relations; however, thanks to consistent lobbying by TABDC, TGRT Haber did air a TV program featuring one of the key Turkish participants of the large￾scale Armenia-Turkey business conference. The program length was 40 minutes and it was entirely devoted to the prospects of Armenia-Turkey business cooperation. These are just some of the examples of reduced stereotyping as well as of appreciation of opportunities, which the evaluators did not provide in response to the question on p. 24 (“What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given?”). SATR Evaluation Report International Business & Technical Consultants, Inc. Page 58 While underlining the importance of knowing the theory of Track 2 Diplomacy (T2D), the evaluators do not connect this theory with the on-going debate about the difficulty of measuring the impact of confidence-building processes. While leaving the more encompassing assessments of the impact to a later stage, EPF has presented indicators to USAID, which the evaluators failed to include in their report. Despite the political difficulties, there were important media and social media outcomes: CNN Turk covered Armenia in a series of four video reports and a representative of the office of the President of Armenia published the President’s condolences to Turkey after the Van earthquake on the SATR Facebook page. Some opportunities capitalized upon by SATR are mentioned in the report, i.e. the meeting with Turkish FM brokered by TEPAV, where an important exchange took place related to state-to-state messages. Yet the evaluators failed to see that these were cases leading to possible future impact. Some of these cases were conveyed to the evaluators, but none of them were reflected in the report noting their significance. EPF would gladly present additional recommendations for measuring impact in conflict transformation, peace-building, and confidence-building projects to USAID at a later stage, upon request. While in some parts of the report (pp. ii, 14, 20, 25, etc.) the evaluators mention the need to expand the partnership base and the number of beneficiaries (as a means to strengthen the impact), they recommend to cut down the Youth Bank activity (p. 31). Evidently, the evaluators did not delve deeply into what Youth Bank is or how it works. As part of the SATR project, the Youth Bank methodology has been successfully adopted by the Turkish partner Toplum Gonulluleri, after which Toplum Gonulluleri received additional funding from the Mott Foundation for further expansion of Youth Bank in Turkey. In addition, the Belgian government supports EPF to further promote cross-border collaboration between Youth Bank committees from Armenia and Turkey, and also other countries, in a specific youth peace network. If the evaluators had spoken to Toplum Gonulluleri, they would learn more about the value of the Youth Bank activity in confidence building between the youth constituencies of the two countries. Theory of Change The mid-term evaluation report suggests that the SATR project does not have an elaborated theory of change (p. 20). This is incorrect: EPF described its theory of change in the SATR application, which was approved by USAID. The theory of change is: To support the Armenia-Turkey rapprochement by leading civil society, business, and informal state-level dialogue to a point where Armenia and Turkey will be able to establish relations and open the border with ease; to act as a catalyst for diplomacy, creating and supporting conditions for both governments to have a variety of options to develop relations; the formation of cross-border networks will contribute to confidence and decrease stereotypes among the two societies, accelerate the normalization of inter-state relations and, in the long term, help to address more controversial issues. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 59 EPF and partners aimed at sustaining contacts between key business and civil society stakeholders and increasing them, so that even if relations at the state level deteriorate, these business and civil society stakeholders will develop additional momentum for peace at the grass￾roots level. We are ready to capitalize on the momentum if the situation at the state level becomes conducive to dialogue again. In fact, the theory of change that EPF originally proposed proved realistic, and EPF is working towards the accomplishment of this particular theory of change. In places, the evaluators conflate rapprochement with reconciliation; they also do not compare the major difference between SATR’s theory of change and methodology with those of other projects (see below for an explanation about the need for an international mediator). In several places, the report acknowledges the professionalism of the implementers – EPF, the Armenian consortium, and Turkish partners. However, the evaluators mention several times that the implementers did not adequately employ the theoretical and practical literature on T2D. While the report provides some references to such literature, it does not clarify what exactly from the methods of T2D is not being used or taken into account by the SATR project. In fact, the mid-term evaluation report does not convey a message of a thorough knowledge of the very rules and methods of T2D by the evaluators themselves. Instead, the report repeats that the project needs to include more people and key people. EPF and most experts of T2D methodology acknowledge that the concept of ‘key people’ must include a much broader group of key people and not only political decision-makers, especially when government parties refuse to engage. As included in the various project documents, EPF and the Consortium focus on key people in civil society, media, and business, in addition to including political actors before the suspension of the state-to-state strand, and still influencing them via indirect means afterwards (see above e.g. the TEPAV meeting). At some point, the report gives an impression that EPF was responsible alongside with USAID for suspending the state-to-state strand. However, this revision to the work plan was a USG decision. At the same time, EPF and the Consortium partners, in agreement with USG, did keep the goal of influencing governments on both sides. Given the above change, this has been often done outside the project’s direct activities, such as in the case of the meeting brokered by TEPAV. Context While the report gives an overview of the local contexts several times, it fails to see the connection between the project design, implementation practices, and the local context. For instance, the report requires feedback questionnaires, which, in many cases, are simply inappropriate. While feedback questionnaires were mentioned in the proposal as a general tool that EPF utilizes for monitoring alongside with other means, for many SATR events they would have been dysfunctional and thus have not been used. In these instances, USAID received SATR Evaluation Report International Business & Technical Consultants, Inc. Page 60 explanations (see on this also below). The report fails to mention that the visibility of the project had to be somewhat muted because of its sensitivity. Also in Turkey, promoting USAID has in the past inspired a negative reaction from the beneficiaries. On p. 12, the evaluators write that Turkish officials state “that they do not need Track II civil society engagement.” Even if such statements are documented, this reflects a fundamental misunderstanding of the context by the evaluators: it is primarily the Armenian government which is uncertain about the worth of civil society contacts. Its position is that there is no progress unless the Protocols are ratified or the border opened, and that informal and civil society contacts may be used as a device that will allow the Turkish government to claim that there is progress in relations between the two states. As mentioned in the mid-term evaluation report, the evaluators did not visit Turkey as part of the field work. Unfortunately, the evaluators’ analysis of the context in Turkey repeats some of the statements by diplomats and analysts and betrays a limited knowledge of Turkish politics. The report contradicts itself, on the one hand claiming that the Turkish incumbent government is strong and has won several elections, and on the other hand, fully buying in into the argument that the current government would be unsuccessful in the parliament if they put the Protocols for ratification. Also, the report praises SATR’s expansion into the eastern regions of Turkey, and mentions that some organizations there are committed but perhaps too weak to successfully advocate with the central government. However, the report fails to mention the risk in expanding into the eastern regions of Turkey without careful preparation, given that any misstep in this respect may be misperceived by some in Turkey as mixing Armenia-Turkey issues with the issue of minorities in Turkey. Management Structure The evaluators emphasize several times that the Armenia-based management structure of the SATR project is deficient. EPF would like to underline that this structure was determined by many factors: a) It is USAID/Armenia that released the request for applications. b) EPF has successfully employed a similar management structure in previous projects. c) There are already several parallel US-funded projects, which have been either advertised from Washington DC and/or from the US Embassy in Ankara, and are led by US-based organizations. d) This structure is logically connected with the theory of change and the context. The theory of change that the project utilizes does not include an external mediator (e.g. one side, another side, and the broker). EPF’s theory of change targets support to larger scale civil society movements to keep the momentum of rapprochement going. In this case, implementers, participants, and beneficiaries work together to sustain the momentum and expand the impact of Armenia-Turkey rapprochement activities. Such a model is SATR Evaluation Report International Business & Technical Consultants, Inc. Page 61 particularly relevant for a project between sides which have so many asymmetries (a fact which the report underlines several times), and at the same time are not isolated from each other and are in constant interaction in different ways. Rather than have an external mediator dictate to the two sides, local actors are empowered to initiate and undertake rapprochement activities themselves, which increases local buy-in and promotes sustainability of outcomes. e) The structure reflects the local context in both Armenia and Turkey, which is itself incongruous. There was less enthusiasm among the Turkish partners to engage in rapprochement activities, since Turkey-Armenia relations are not high on the government agenda. At the same time, there is a tendency in Turkey to interpret Armenia-Turkey rapprochement as an internal matter related to a domestic national minority and not to the state of Armenia. On the one hand, the report claims that the SATR project is asymmetrical because it is managed by an Armenian consortium. The report further claims that the engagement of the Turkish side is smaller than that of the Armenian side. On the other hand, the report recommends that follow-on activities be “unbalanced rather than symmetrical” (p. 32) with a larger focus on Turkey and substantially smaller on Armenia. This change in focus, however, does not reflect the local context, where resistance to opening the border between Armenia and Turkey is higher among the population in Armenia than in Turkey.32 In addition, as a practical matter, USAID has a mission in Armenia, but no office in Turkey. The report neglects to recognize the major success of the consortium approach: the SATR project has proceeded steadily despite all the impediments. In fact, the SATR Consortium, with all its different leader-organizations, is able to address rapprochement issues while covering the entire spectrum of existing approaches, from conservative to liberal in both contexts. Similarly, the variety of partners in Turkey allows the project to engage organizations that have the strongest capacity to deliver a message to key decision-makers. This increases the legitimacy of the project activities in the current tense political context. The report failed to properly weigh the deficiencies in the consortium setup against the implications of having a less representative implementation base and risking a disagreement with the extremely sensitive governments. At the same time, the fact that the project enjoys less enthusiasm in Turkey and among some circles in Armenia is correct and well known. EPF and the Consortium members are aware of that reality, and they are working to address this issue, in particular via engaging more partners during the project. The mid-term evaluation report claims that many of the SATR activities focus on the implementing partners. A more careful study of the attachments to the quarterly progress reports would have revealed that the average ratio of activity participants to the SATR implementers is at least eight to one. This documentation was provided to the evaluators in 32 See PPT presentation on http://www.epfound.am/english/thought-leadership/policy-research/data-on-public￾perceptions-regarding-armenia-turkey-relations-2010.html SATR Evaluation Report International Business & Technical Consultants, Inc. Page 62 order to offset some of the perceptual biases that they would encounter while interviewing beneficiaries. In fact, hundreds of participants have already taken part in the project activities. On page 18, the evaluation reads: “The project seems not to have reached out to the many potential spoiler stakeholders on both sides with information or analysis.” While it is unclear what information or analysis is meant by the evaluators, one example of SATR’s engagement of potential spoilers is the collaboration with Yerkir Media TV on production of the TV talk shows. Yerkir Media TV is closely linked to the Armenian Revolutionary Federation Dashnaktsutyun party, which is highly critical of the Protocols. However, the party leaders are engaged in many activities of the SATR project, including production of the first-ever film by a Turkish director on April 24. This is one of several similar examples of engaging spoilers. The report then reads: “SATR also has not focused on how to quietly pull more officials from both countries into the reach of project products and ideas.” While this has not been the only focus of the project, particularly after suspending the state-to-state strand, officials at the highest levels are aware of the project and its logic and outputs on both sides. President Serj Sargsyan mentioned recently in his speech in Brussels the fact that there are many more civil society members in Turkey today who actively present a different view of Armenians than it has been a few years ago. He connected this fact with the work of civil society and, though he does not mention the SATR project specifically, EPF believes that such indirect references indicate the government’s knowledge of the project activities (SATR is currently the largest rapprochement project). In terms of engaging more officials, as encouraged in the report, the SATR project no longer focuses on engaging large numbers of officials now that the state-to-state strand has been suspended. That said, several ministries and other major departments in both countries are aware of some of the project developments. The project has also been communicated in some detail to numerous international organizations, including the European Commission, EU Delegations to Armenia and Turkey, European Parliament, the Council of Europe (whose Parliamentary Assembly’s former Chair, Mevlut Cavusoglu, was introduced to the EPF Armenia-Turkey activities at the onset). Several Yerevan and Ankara-based embassies have been briefed on EPF’s Armenia-Turkey rapprochement strategies and SATR. This type of outreach is an important element of the project’s communication strategy, aimed at influencing the state actors indirectly, which is not mentioned in the report. The report is critical of EPF’s and the partners’ efforts to integrate the SATR activities with other relevant activities and projects. It is quite contradictory: on the one hand, the report states several times the need to expand the number of partners and beneficiaries, and build sustainability apart from future USG funding. On the other hand, the report expresses a critical view of many cases where SATR cooperated with other entities and activities that were locally-driven and/or funded from different sources. In fact, the SATR project cooperates with as many outside projects as possible. Many of these other actors approached EPF and the Consortium members for different types of support, which was provided and is being provided whenever possible. SATR greatly benefited from this synergy with other activities, such as the Ani Dialogue, run by the Boell SATR Evaluation Report International Business & Technical Consultants, Inc. Page 63 Foundation and Caucasus Institute, or the strategic meetings organized by David Phillips with support from Swedish Government Armenian and Turkish partners’ large-scale, in-person strategic meetings take place at least twice annually, often co-funded from other sources. SATR’s outreach to governments in some cases happened via outside projects, not necessarily foreign-government funded, some of which are mentioned in the report and in this statement, such as the TEPAV meeting. All of the cases of cooperation are cleared with USAID in advance. Visibility The report claims that the overall picture of the project (a) is not clear to many stakeholders and (b) is not known to the general public. Regarding the stakeholders, EPF is pursuing two major priorities in terms of visibility: the visibility of Armenia-Turkey rapprochement issues as a primary objective and the visibility of SATR as a secondary one. EPF has focused more on making sure that the Armenia-Turkey issues rather than the SATR project are on the agenda in both countries. Experts in T2D would know that, as different from usual development projects, projects on confidence-building across conflict divides should have a sophisticated and cautious communication strategy, combining transparency with a certain amount of confidentiality. The USG and USAID have been supportive of this approach and have provided guidance to EPF and its partners. Regarding reaching out to the general public, the report mentions in passing the TV talk shows, but it dismissively mentions that their audience is limited. Having an audience of tens of thousands of viewers on TV in Armenia is quite a significant number both from the perspective of the controversial issue that is being addressed, as well as given the size of the population. The voluntary feedback that this audience provides is crucial for understanding public attitudes and how they change, given that there is no baseline research apart from a few questions that EPF/CRRC included in the Caucasus Barometer in 2010 (which was funded from outside sources). In Turkey, the project’s on-line TV audience reaches up to thirty thousand viewers, if not more. Indeed, it has been difficult to access mainstream TV in Turkey, however, EPF did brief the evaluators on the CNN Turk coverage arranged by the project with outside funding, and other cases of large-scale coverage (such as the demonstration of jointly produced Armenian￾Turkish films at the Istanbul Film Festival and at the Golden Apricot Film Festival in 2011), which is not mentioned in the report. In 2011, SATR supported joint participation of Turkish and Armenian film professionals and cinema journalists in three film festivals in Armenia and in one film festival in Turkey, which were attended by hundreds of people and also enjoyed extensive media coverage. None of them are mentioned in the report. The report additionally criticizes the implementers for not having a specific website dedicated to the SATR project. However, it also mentions several times (on pp. iii, 20, 21, 25, etc.) that the web-site is currently being developed by EPF. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 64 The public versions of quarterly reports that the evaluators criticize have been an add-on activity that EPF voluntarily took upon itself to develop. They were not part of the original work plan approved by USAID. They cannot be delivered until the quarterly report is submitted to USAID and until USAID clears the public version. However, after they are published, they are circulated widely via the EPF database and reach several thousands of subscribers. They are currently available in English, Armenian, and Turkish. Factual Inaccuracies The report is factually incorrect in a number of instances. For example, the report states that contracts were negotiated with the Turkish partners after the project was approved. In fact, the general lines of the project’s strategy and activities were discussed and agreed upon with the key Turkish partners during the project preparation period if not prior. Service contracts were indeed signed after the commencement of the SATR project, which is a general practice in the industry. The report contains unfounded generalizations which become factual inaccuracies. In the report, the evaluators repeatedly claim that the entire project had to be renegotiated at the inception phase. In reality, only the media strand had to be partly revised in the beginning. Other changes took place in the process of the SATR implementation as needed, based on mutual agreement between all the stakeholders. Some of these changes are mentioned correctly in the report. Contradictorily, the evaluators are both critical of these changes at the same time that they encourage SATR to be flexible. The report claims several times that the SATR activities started late. In fact, the activities did not start late: the project was approved by USAID on September 30, 2010, and EPF and the Consortium members started implementation on October 1, 2010. The first month was devoted mainly to administrative issues and finalizing operational strategies with USAID and among the Consortium members. By the fifth week, the Consortium had already started implementing activities. The official launch event, mentioned on page 17 of the mid-term evaluation report, took place approximately seven weeks after the project start and was designed to bring all Armenian and Turkish partners together to appreciate the scope and scale of the launched project and establish links between each other. The report states that the Consortium members did not undertake activities to make sure that the launch event was covered in the media. Given the downward slope in the state-to-state relations, EPF and USAID were not certain to what extent broad coverage of the event in the media would benefit the project. As a result, USAID advised EPF to keep the event low key and not issue a press release. Despite this, the event was covered in a number of Armenian newspapers and on-line news outlets. In some cases, the paper mentions a complaint or concern voiced by a stakeholder or beneficiary as fact. EPF and others are committed to the quality of the project events, whereas the subheading “Quality Concerns” on p. 17 presents concerns about the quality as proof of lack of quality of many activities. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 65 Monitoring and Evaluation The evaluators suggest that project activities lack appropriate follow up, failing to mention at least one case of numerous instances of EPF and partners following up. As one example, the success of the Pecha-Kucha Night event in Yerevan inspired the implementers to undertake the same activity in Istanbul, even though it was not originally included in the annual work plan. There are many other examples of a variety of follow up activities flexibly undertaken after the first instance proved a success with potential impact. In the second year of its activities, SATR supports again the Armenian-Turkish Cinema Platform, brings journalists from Turkey to Armenia on April 24, encourages Turkish partners and stakeholders to build a communication network after a similar event proved a success in Armenia, etc. All these sub-projects were not included in the initial plan, but the implementers and USAID decided to undertake them given their success the previous year. The entire project is based on the idea of capitalizing on successful undertakings and focusing on them to increase their impact. It is true that the project does not require the implementers to collect feedback after each activity. This is due to the fact that many joint events include high level policy specialists and seasoned professionals, and it is unrealistic to expect them to eagerly fill in questionnaires. Feedback is collected and stored through other means, including an internal mid-term evaluation, audience opinions after media exposure, program staff visits, and interviews with project participants. Active participants are also constantly updating the SATR’s Facebook page. In addition, feedback has been gathered through less conventional means, such as the statements left by the Turkish Youth Bank participants in the visitor’s book at the Genocide Museum in Yerevan. The visit itself was not planned, but was a reaction to the interest of the Turkish youth to visit it. In July 2011, an internal self-assessment and a reflection were completed by the consortium. The work was reassessed while preparing the second year annual plan. Bridging the Divide, a US NGO, is tasked by EPF with doing another evaluation. Additional comments on the business strand from UMBA and EPF The report states on page 8: “There are modest achievements beyond bilateral deals, such as the Travel Association efforts to stimulate regional tourism, but this regional-level association is not a strong platform to lobby for national-level changes like opening the border.” It would not be realistic to expect for the Association to lobby for the opening of the border from the very beginning. However, after they succeed with the development of regional tourism and a significant increase of tourists in the region, which will visibly contribute to the economy of both countries, they will become strong enough to start lobbying. On page 9 the report reads: “Ad-hoc approach to Governments - Implementers in this strand do not have a consistent approach to interacting with governments on either side. This leads to scrambling, such as getting visas for Turkish green passport holders coming to Armenia for events, and a problem-by-problem approach to addressing issues.” Both UMBA and TABDC SATR Evaluation Report International Business & Technical Consultants, Inc. Page 66 have permanent access to both governments. UMBA has been able to quickly resolve issues related to acquiring visas for businesspeople from Turkey, if they had special (green) passport. Special passports holders still encounter impediments to free travel from Turkey to Yerevan in some cases. On page 13, the report says: “The project matches a small state with a regional powerhouse. Turkey is much larger in terms of population and has a more diverse society.” It then continues on: “Turkey has had a competitive market economy for many years with tremendous growth and engagement around the world. Armenia has had a difficult time in the transition to a market system. The Armenian GDP of some $10 billion is 1.3 percent of the Turkish GDP which is approaching $750 billion.” Economic asymmetries would be an obstacle, if the purpose of the project was the integration of both economies. However, the purpose of the SATR business strand is to foster increased cooperation, and economic asymmetries do not undermine the ability for the project to develop this cooperation. On the same page, the report states: “For example, UMBA notes that all kinds of meetings are held with businesses and that information is forwarded to TABDC on Armenian firms for ‘matchmaking.’ Yet, it is not clear that SATR aids in developing these types of connections between the long-time partner organizations UMBA and TABDC; nothing in the structure of the project helps deepen partner contacts, spread information or connect companies in an open, transparent way that benefit other companies in Armenia or Turkey or other partnerships between companies across the border.” It seems that there are two critiques included here: that the project does not foster closer relations between UMBA and TABDC and also does not provide opportunities to other businesses to connect across the conflict divide. The first is incorrect since both organizations cooperate now for more than ten years. As for the second, companies are sometimes reluctant to transparently share information. However, UMBA is currently preparing a web-site to present possible business opportunities to interested businesses from Armenia and Turkey, while Narekavank Tour LLC via an open-door grant is establishing an Information and Consulting Center in Yerevan to serve as a “one-stop shop” for Armenian and Turkish entrepreneurs interested in cooperation with one another. Conclusions The report, in general, is written in an unhelpfully negative style. EPF believes it would be more logical, professional, and constructive if the evaluators were first to gauge the achievements before moving to criticisms. For instance, the section titled “Little Attention to Gender” (p. 16) begins by mentioning a lack of focus on gender and ends with mentioning the focused work on gender that is included in the project. If it were titled neutrally, e.g. “Attention to Gender,” it would dispel the impression that there was a conscious desire to challenge the project in an unconstructive manner. Another sub-heading titled “Only Some Ties are Sustainable” on p. 19 could have been phrased as “Some Ties are Sustainable,” which would then not give an impression of a prejudice. Similarly, the paragraph entitled “Limited Impact” is written so as to SATR Evaluation Report International Business & Technical Consultants, Inc. Page 67 diminish every mention of the SATR impact that the paragraph in fact contains. In many cases, when enumerating activities accomplished in different strands, the report immediately criticizes, if not wholeheartedly dismisses, the philosophy behind the particular activity and the strand. While the report mentions several times that it is a mid-term evaluation, many statements are formulated in a style that would qualify this paper as a final evaluation. The structure of the report contributes to this impression: criticism precedes acknowledgement of any achievements or progress toward impact. This gives the impression that the evaluators expected all components to be in place and no obstacles to be encountered during project implementation. Only in the second half of the report and in later sections do the evaluators recognize that things have actually been accomplished, while many other things are in progress, etc. Of course, impacts will become visible and/or easier to gauge toward the end of the project, if not some time after the completion of their active stage, and not during a mid-term evaluation. The evaluators could have acquired a better understanding of the SATR project if they had approached the key implementing staff with additional and more focused questions. In the methodology section on p.5, they justify their free style of interview procedure. However, free style interviewing does not often give sufficient information on concrete questions. Some of the project key staff was interviewed in such a way as to assume that this was just an introductory interview, and a more structured one would follow, which proved not to be the case. Some others have been omitted. For instance, the head of TABDC was not interviewed. The evaluators do not provide a list of those interviewed. Despite the overwhelmingly negative tone of the report, the evaluators suggest that SATR not change the general direction of the project. Rather than major strategic changes to the SATR project, the evaluators introduce a number of lessons learned that should be incorporated into any future intervention. EPF is thankful for these suggestions. However, at this point, EPF and partners are discussing how the next level needs should be conceptualized for a follow up strategy. This includes bolder work on the issue of 2015, engagement of the Diasporas, possibly a religious dialogue, etc. If not the main evaluator (Lawrence Robertson), at least his local colleague (Stepan Grigoryan) could have recognized the importance of these quite obvious issues. Instead, however, they underline, several times, the importance of having a strategy to address the Nagorno-Karabakh conflict, which is indeed important but further away from issues that need to be addressed in Armenia-Turkey relations more immediately. EPF regrets that there was a pre-existing conflict of interest with the local expert selected to perform the evaluation. The mid-term evaluation report clearly states (p. 6) that EPF expressed reservations about the local expert and that the team leader attempted to ensure that this did not have an impact on the evaluation. Unfortunately, it is impossible to judge how this conflict of interest influenced the evaluation’s findings, which diminishes the credibility of the report for all interested parties. SATR Evaluation Report International Business & Technical Consultants, Inc. Page 68 EPF recommends that the reader read the evaluation in its entirety, since it is only later in the report that the assessment and recommendations become more constructive. For instance, on page 23 it reads: “SATR approaches are feasible and necessary given the current state of Armenia-Turkey relations.” The report’s credibility would be strengthened, if this sentence were incorporated into the Executive Summary and this understanding included in the initial sections of the report. Overall, the final recommendations include some interesting and useful ideas, and EPF and the Consortium look forward to making appropriate adjustments to improve the effectiveness of the SATR project.