EVALUATION OF THE COMPETITIVENESS OF THE ARMENIA PRIVATE SECTOR (CAPS) PROJECT USAID/ARMENIA SEPTEMBER 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Donovan Rudisuhle of Social Impact, Inc. and Anna Sedrakyan of Management Systems International. EVALUATION OF THE COMPETITIVENESS OF THE ARMENIA PRIVATE SECTOR (CAPS) PROJECT USAID/ARMENIA DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document was submitted by Social Impact, Inc., with Management Systems International, to the United States Agency for International Development under: USAID Contract No. RAN-I-00-09-00019, Task Order No. AID-111-TO-10-00002 Evaluation Services Indefinite Quantity Contract Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com Acronyms CAPS Competitiveness of the Armenia Private Sector EIF Enterprise Incubator Foundation GMP Good Manufacturing Practices GOAM Government of Armenia MOE Ministry of Economy MPI Union of Medicine Producers and Importers NSS National Statistical Service TAIEX Technical Assistance and Information Exchange USAID United States Agency for International Development YSMU Yerevan State Medical University Table of Contents Acronyms ........................................................................................................................................ 3 Executive Summary ........................................................................................................................ 1 I. Description of the Project ....................................................................................................... 1 II. Purpose of the Evaluation ....................................................................................................... 1 III. Evaluation Methodology ...................................................................................................... 2 IV. Overall Findings and Conclusions ....................................................................................... 3 A. Sectoral Findings, Conclusions and Recommendations .................................................. 5 1. Information Technology Cluster .................................................................................. 5 Conclusions: ............................................................................................................................ 6 Recommendations: .................................................................................................................. 7 2. Pharmaceutical Cluster ................................................................................................. 7 Conclusions: ............................................................................................................................ 9 Recommendations: .................................................................................................................. 9 3. Tourism Cluster .......................................................................................................... 10 Conclusions: .......................................................................................................................... 11 Recommendations: ................................................................................................................ 12 V. Summary of Recommendations ............................................................................................ 12 VI. Lessons Learned................................................................................................................. 13 ANNEXES .................................................................................................................................... 14 1 Executive Summary The CAPS project was successful in meeting its stated objectives. Interviewees in all three of the industry clusters voiced favorable opinions regarding their interactions with the project. There was virtually universal agreement among interviewees and focus group participants that their respective industry cluster had received valuable benefits from CAPS’ activities and that the project was a positive factor in fostering economic growth and employment opportunities in Armenia. The principal vehicles used by CAPS were training, funding of international experts, sponsorship of conferences and trade association events, assistance with regulatory and legislative reform, identification of foreign markets and potential business partners, development of promotional literature and materials and creation of industry cross-linkages. The vast majority of persons contacted stated that they would like to see a continuation CAPS’ activities in a future project based on the same successful model. Many of them offered valuable suggestions to improve the effectiveness of any future activities under consideration by USAID. The evaluation team found that there was balanced gender participation in all aspects of the activities conducted throughout the three clusters. I. Description of the Project The CAPS project was launched in September 2005 under a $14.1 million grant from USAID for the purpose of assisting Armenia in its efforts to develop its key industries with the specific objective of increasing employment, promoting exports and growing local businesses by promoting productivity, competitiveness and policy reform. The mechanism used by CAPS is a cluster approach that incorporates all of the major stakeholders and participants within a specific industry. This includes private companies, individual entrepreneurs, government agencies and regulatory authorities, educational institutions, trade associations and civil society organizations. Information technology, pharmaceuticals and tourism are the three clusters that were identified as targets of CAPS financial and technical assistance. CAPS’ methodology is a demand-driven approach that stresses job creation, cluster-based development, local ownership of strategies, action and results and market-oriented themes. The principal vehicles used by CAPS to carry out its activities are educational seminars, facilitation of international technical assistance, sponsorship of trade fairs, identification of foreign markets and partners, assistance with legislative reform efforts and preparation of marketing and promotional literature. The project strives to demonstrate results and achieve sustainability by: • Building sustainable institutional capacity • Transferring the organization of initiatives to counterparts • Building public awareness on key competitiveness issues • Achieving clear policy reform results The CAPS project is scheduled to terminate in February 2011. II. Purpose of the Evaluation The primary purpose of this evaluation was to provide USAID/Armenia with an objective external economic impact assessment of this important program that will soon be concluding and to apply the experiences and lessons learned to the next generation of economic growth 2 initiatives. The evaluation endeavored to measure and analyze the impact of CAPS project activities through an “effectiveness assessment” lens. That is, how well did the project activities catalyze economic growth and contribute to productivity gains and workforce development in the three selected sectors? III. Evaluation Methodology The objective of the evaluation was to provide USAID with defensible conclusions and actionable recommendations for the purpose of helping USAID and its Armenian partners establish priorities and plan their activities for any future interventions in these designated sectors. To accomplish this, the evaluation team met with a variety of stakeholders for each of the designated clusters. With the assistance of CAPS officials, the consultants met with cluster coordinators, trade associations, private companies, entrepreneurs, government agencies, educators, students, civil society organizations and other beneficiaries of the project. In addition, focus groups were held with groups of stakeholders for the purpose of obtaining their opinions of project effectiveness and to provide a forum for them to voice their recommendations regarding ways to improve the project in any future extension or redesign by USAID. The consultants conducted their interviews in Yerevan, Gyumri and Vayk and focus groups were also held at all three of these locations. In accordance with its standard qualitative information collection technique, the team convened three focus groups of business representatives and concerned individuals in Yerevan, one in Gyumri and one in Vayk. These groups were composed of individuals such as business owners, startup entrepreneurs, trainees, university professors and other direct beneficiaries of CAPS assistance. These groups averaged about 6 to 10 individuals who were gathered together at each location for a series of discussions lasting 1 to 2 hours where they were requested to voice their experiences regarding working and collaborating with CAPS. In addition, they were asked about particular sector needs and the degree of success they achieved using CAPS’ various approaches to project identification and implementation In addition to the above, concerned individuals and local government representatives at project sites were interviewed using an open semi-structured questionnaire format. The topics and the wording of the questions were very flexible to allow for the diverse nature of the individuals that met with the team. In summary, within the very short time available for this impact assessment, the team conducted dialogues with as varied a group of participants as is possible. The study also referred to existing documentation to ensure that the team obtained the best possible understanding of all aspects of the interventions. Assisted by the CAPS project staff, the team referred to various analytical reports prepared by third parties as secondary data to assess the catalytic effect of the project and the corresponding productivity increase. In particular, the team used following reports: • Armenian Information Technology Sector, Software and Services. 2009 Industry Report • , Ministry of Economy of the RA, USAID/CAPS, EIF Armenian Pharmaceutical Industry. Investment Handbook 2008, USAID/CAPS, MPI Union, Armenian Development Agency 3 • Armenian Tourism Industry. Investment Handbook 2008 • , USAID/CAPS, Armenian Tourism Development Agency, Armenian Development Agency Jermuk Comprehensive Development Plan, 2008 • , Ministry of Economy, USAID/CAPS Information Technology Growth Model, 2007 • , EIF, Economy and Values Research Center Tourism Strategy For Armenia, 2007-2030 • National , Ministry of Economy, USAID/CAPS, Economy and Value Research Center, Armenia 2020 Competitiveness Report of Armenia, 2009 • A number of other published sources provided by various government and analytic entities. , Economy and Values Research Center" IV. Overall Findings and Conclusions Overall, the CAPS-assisted clusters demonstrated varying degrees of growth over the period from 2005 to 2010. The IT Cluster (excluding telecom) saw a 17% increase from 2008 to 2009 and there was a corresponding 3% increase in the Tourism Cluster, this in spite of the global economic crisis. (the increase counts the overall turnover) for the same period. The pharmaceutical sector registered a 3% decline overall in 2009, although this figure includes data on pharmacies. Pharmaceutical production, the focus of CAPS interventions in the sector, experienced an average growth of 7% last year. The increase was primarily a consequence of increased domestic sales, with the export of local pharmaceutical products increasing by only 4.7%, according to the National Statistical Service (NSS). However, the pharmaceutical industry has not yet seen any CAPS-related growth, as CAPS has only intervened recently, and its efforts have been focused on obtaining GMP certification for Armenian laboratories and manufacturers. When this goal is achieved, Armenian firms should be in a position to such to actually increase their export sales by having unfettered access to the EU market. According to CAPS semi-annual and annual reports to USAID, in 2009 3,500 new jobs were created, just short of the target of 3,800. This figure includes jobs created directly or indirectly as a result of CAPS activity. Sales and revenue targets were surpassed and overall productivity increased by 8%, exceeding the target of 6%. Armenian firms participating in the CAPS clusters have made significant progress in adopting international best practices, with 15 of them adopting them, while only 4 were envisioned in the target. The project also supported the adoption of six pro-business reforms which benefited the CAPS cluster, surpassing the target of four. The structures and processes established in the IT, Tourism and Pharmaceutical Clusters have resulted in improved cooperation and communication among stakeholders. Interviewees from both the private sector and the project itself stated that they had good cooperation with those governmental agencies with which they routinely interacted. However, numerous stakeholders mentioned that there was an occasional failure of the government agencies to collaborate effectively with each other. 4 The Head of the Drug Agency expressed frustration in dealing with the Ministry of Health, and so did the CAPS project staff. In particular, the project staff mentioned that there was absolutely no one in the Ministry of Health responsible for the development and follow-up of GMP regulations. It was only after the Ministry of Economy took over the responsibility for developing these regulations and presenting them to the Government for approval that the Ministry of Health finally got involved in the process. The lack of capacity and poor motivation at the lower levels of the ministries was cited as the principal factor that contributed to time-consuming and burdensome experiences in dealing with government organizations. There is no reliable way of establishing a specific percentage of growth attributable to CAPS’ intervention, but stakeholders universally stated that the favorable outcomes they experienced would not have been possible in the absence of CAPS’ support. In terms of sustainability and cost effectiveness, it can generally be concluded from the interviews that project activities were relevant to the current industry needs, and beneficiaries were optimistic about their long-term sustainability. However, it is not possible to make a statement regarding cost-effectiveness because the contractor's records do not disaggregate expenditures in a way that they can be easily matched to project activities. The information is available, but a time-consuming effort would be required in order to analyze and organize the numbers in a way that they could be associated with specific project activities. Strengths of the CAPS Project – All Three Sectors: • CAPS strengthened the foundation for IT sector development and created linkages with the Tourism and Pharmaceutical Clusters. • CAPS promoted start-up support, capacity building and workforce development activities and encouraged co-funding and in-kind assistance from the private sector. • New business associations were created and existing associations were strengthened through CAPS’ intervention. • Targeted trainings and education were made available to a wide variety of business stakeholders. • CAPS assistance helped to improve the marketing, branding and packaging capabilities of the firms that were assisted. • CAPS supported improvements in policy and regulatory environment. • International certification programs were introduced in all three industries. Weaknesses of the CAPS Project: • CAPS’ scope of work was too broad and too many small activities were dispersed across the target sectors, reducing the focus of efforts. These activities were not connected to each other, and in some instances even contradicted each another. • This excessively broad scope led to uncertainty in planning the project activities at the inception, thus impeding the effective startup of the project. When the activities began to reach a point where they were actually having a demonstrable positive effect, the project was beginning to wind down its five and a half year life. 5 • The future sustainability of the newly created business associations that received CAPS funding is uncertain, particularly those in the tourism industry. This is attributable in part by a failure to attract a majority of the potential members, who may not perceive a benefit in membership, or who are unwilling to pay the membership fees. Key challenges ahead: • One recurring comment that was voiced to the evaluators was that a continuing effort is required to change the mindset of managers, professionals and the public in general in terms of accepting new ways of thinking. In particular, they stressed the importance of educating and persuading managers at all levels that they can improve the efficiency of their businesses and increase their revenues by adopting modern IT practices. • Along the same lines, stakeholders in the pharmaceutical sector said that it is necessary to educate doctors, pharmacists and patients about the fact that locally produced drugs are of equal quality and effectiveness as imported products. • Corruption was also mentioned frequently as an impediment to business efficiency and economic growth. A. Sectoral Findings, Conclusions and Recommendations 1. Information Technology Cluster Findings: Information technology has become one of the driving sectors of Armenia’s economy, contributing to innovation and productivity growth in the country and generating substantial export sales. According to the Enterprise Incubator Foundation (EIF) survey 2009 on Armenian Information Technology Sector Software and Services, the IT sector has registered an average annual growth in revenues of 24.2% from 2006 to 2009. In absolute figures, IT sector revenues reached $129.9 million in 2009, an increase from $84.2 million in 2006 and $111.3 million in 2008. According to the 2009 State of IT Industry Report, prepared jointly by the EIF and CAPS based on the figures provided by the National Statistical Service, the IT sector accounts for 1.3% of GDP in Armenia, which is comparable with that of other leadings economies. By comparison, the IT industry share in India’s GDP is 1.4% and in Germany’s it is 1.3%. From 2003 to 2009, the IT industry’s contribution to total exports rose from 3.6% to 5.6%, and domestic sales and revenues increased from $42 million in 2008 to $59 million in 2009. A wide range of CAPS activities contributed to this growth, both on the export and domestic sales side. There were a number of country and industry promotion initiatives, such as supporting companies to participate in global IT fairs and conferences which assisted Armenian firms to find new partners and sign sales and services contracts with them. Issues of industry confidentiality precluded obtaining quantitative data, but numerous instances of these partnerships and agreements were said to have occurred as a consequence of CAPS’ involvement. The promotion of IT solutions in other sectors, particularly in tourism, contributed to increased domestic sales of IT companies, as local firms began to adopt new technologies for the first time. 6 Numerous industry-wide events were organized locally with broad international participation, and this created awareness gave local Armenian firms more visibility. In particular, it is worth mentioning such major events as the ArmTech Congress, DigiTec Expos, Meetings Without Ties and the annual Armenian IT Competitiveness Conference. These activities created a favorable environment that enabled members of the IT Cluster to collaborate effectively and helped create a positive business climate for IT, as well as improve the competitiveness of Armenia’s IT sector in international and domestic markets. In addition, these industry events also had the effect of creating awareness of other local industries on IT use for their needs, and furthermore stimulated IT industry representatives to raise their concerns regarding legal and regulatory matters to the Government. CAPS also contributed to improvements in the regulatory and legal environment. With CAPS’ assistance in 2008, the Government adopted a 10-year development strategy and vision for the IT sector. In 2009, CAPS developed an E-Government Road Map which it is using to outline its activities and execute its IT strategy. This provided the basis for including funding in the 2010 budget for the Digitec Expo, Meetings Without Ties and other events. CAPS’ firm-level assistance was directed towards international IT standards localization and the establishment of internationally recognized certification programs. As a result, participating firms were successful in meeting the expectations of foreign partners and customers in terms of quality, price and service. The project focused its firm-level assistance program in critical operational areas, such as sales and marketing, motivation, innovation and knowledge management, among others. CAPS also worked towards the localization of international IT standards to improve their understanding among local firms. This, combined with training and international certification programs enhanced the capability of industry members to cooperate in joint ventures, outsourcing opportunities and project management. In 2009, the total workforce in the IT sector reached around 5,200 specialists. However, the availability of up-to-date, practical IT vocational and university programs is still inadequate in Armenia. Therefore CAPS’ role in this area was to bridge the gap between the skills acquired in higher education and the needs of employers. This was accomplished through internship programs, upgrading of university curricula, localization of IT international standards and the promotion of IT specializations within society and business. The training initiatives at vocational schools and universities that were promoted by CAPS facilitated the upgrading of IT education and increased the number of trained individuals with the skills and experience demanded by the marketplace. During the focus group that was conducted in Gyumri, the consultants were told that 47 out of the 58 participants in the CAPS-funded training programs found immediate employment. Conclusions: As a result of numerous meetings with project beneficiaries and industry representatives, and a review of project documentation, the evaluation team has arrived at the following conclusions with regard to CAPS’ support to IT Cluster: • Workforce development activities, such as short and long-term training programs and vocational education programs were of considerable value to the industry. In addition, interviewees consistently stated that internship programs allowed recent graduates to find jobs more easily. 7 • CAPS’ market development initiatives, particularly industry promotion activities, helped IT companies to increase both domestic and international sales. • Cluster development support resulted in a more cohesive IT industry that is able to collaborate effectively and take ownership of initiatives so as to create a more favorable business environment. For example, in its role as an industry mirror, the Union of Information Technology Enterprises of Armenia paved the way for the industry to lobby the government and promote its interests in terms of the legal and regulatory environment. • Firm-level assistance led to improved capacity of local companies to effectively compete in the international marketplace. However, if continued, a more comprehensive approach to this component should be considered. This might include elements such as seed financing, training and certification, capacity development and marketing assistance. Recommendations: • A comprehensive firm-level assistance package should be provided that includes financing, capacity and workforce development, certification support, company promotion, all with the special emphasis on IT start-up companies. • Consideration should be given to increasing access to financing by creating a development credit authority and/or a venture fund type structure. • Design and implement activities to strengthen managerial and entrepreneurial skills. This could include things such as specialized training programs and seminars and lectures conducted by recognized business executives. There could also be support for universities and larger GDA projects, such as opening of a new entrepreneurship or business school with the joint participation of an internationally recognized university. • Support initiatives designed to foster development of e-society, such as supporting those activities established under the “Armenia E-Society Development Concept, 2010-2012” which was developed by the Ministry of Economy and approved by the Government of Armenia (GOAM). This program has the objective of creating a wide broadband network for the purpose of making e-services accessible throughout the country. • Continue to promote the IT sector as a vehicle for encouraging local firms to increase their efficiency by embracing modern IT strategies in their everyday conduct of business. • Explore the possibility of offering assistance to the government to help implement their 10-year IT strategy. 2. Pharmaceutical Cluster The Pharmaceutical Cluster was added to the CAPS scope in 2007, and initial activities in this area only began in 2008, which was very late in the game. These efforts focused mainly on the development of the cluster strategic action plan, which was designed to identify key initiatives and build consensus among the industry stakeholders. Based on the results of this assessment exercise, CAPS’ assistance focused on two key initiatives: 1) broad adoption of Good Manufacturing Practices (GMP) by all participating 8 producers, and 2) improved market research and targeted promotion of Armenia’s pharmaceutical industry, emphasizing the industry’s core strengths and the expected benefits of broad adoption of GMP. To accomplish this, the CAPS project supported a number of activities designed to help move the industry towards the acceptance of international operational standards, develop the capacity of firms to target new markets and bring key educational institutions and business associations into active cluster roles so as to strengthen collaboration and improve the competitiveness of the industry. CAPS helped the Union of Medicine Producers and Importers (MPI) to establish the GXP Center of Excellence and conduct Armenia’s first GMP training. The Center will be providing GMP training, certification and assessment for the purpose of building GXP competency within the industry. Furthermore, CAPS technical assistance strengthened the Center’s facilities and training capacity through improvement of their library and procurement of equipment used to conduct GMP training for all member and non-member companies in the sector. The services offered by the Center of Excellence are fee-based and it is expected that the Center will be sustainable upon the phase-out of CAPS support. In 2009, CAPS organized a number of training courses with the GXP Center of Excellence (CoE) on GMP, Good Distribution Practices (GDP), pharmaceutical marketing, and GMP workplace assessments. A total of 113 representatives of pharmaceutical companies, the Ministry of Health, the National Institute of Health, the Scientific Center of Drug and Medical Technology Expertise (SCDMTE) and various universities participated in the training courses. The Government is planning to adopt a new GMP regulation by the end of 2010. Therefore, CAPS provided valuable and timely assistance to these public entities in terms of developing the GMP regulatory framework. This was accomplished by working with state bodies on drafting and adopting the GMP regulations, supporting the Drug Agency to finalize the inspectorate manual, assisting with the publication of an Armenian “Orange Book” that establishes GMP requirements and supporting the Drug Agency’s participation in training programs and study visits under the EU-funded Technical Assistance and Information Exchange (TAIEX) program. CAPS firm-level support to the 10 cluster companies has helped them move closer to GMP compliance, or at least acknowledge the importance of GMP compliance. The project has helped to strengthen general management skills within cluster companies through the development of an investment plan for one company, and support for customer relationship management for two other firms. The short period of time available following the technical assessment exercise and the lack of preparation at the other seven firms precluded in-depth assistance on the part of CAPS. CAPS’ Market Development Component activities focused on improving the domestic perception of Armenia’s pharmaceutical manufacturing industry and its products through a local promotional campaign organized with CAPS support and co-funding from the industry. Due to funding restrictions, the campaign had very limited effect on the local market. Under the Workforce Development Component of the Pharmaceutical Cluster, collaboration between Yerevan State Medical University (YSMU) and the University of Southern California (USC) was established to enhance production-related curricula at YSMU and strengthen linkages with the private sector companies. The project also helped YSMU to access support from the 9 European Union’s TAIEX program to review a broader range of curricula. Internships implemented through the framework of the project facilitated the strengthening of linkages between universities and the labor market. CAPS continues its efforts with existing initiatives in the cluster by supporting the GXP Center of Excellence, building the capacity of the Union of Medicine Producers and Importers (MPI), developing a University Partnership with YSMU and others so as to help ensure their continuation after the project. Conclusions: • CAPS’ assistance resulted in providing a demonstration effect to other clusters, such as IT and pharmaceuticals. However, cluster coordination still requires significant improvements in order for them to become more cohesive and effective. • Lack of proper government regulations and a coordinating ministry led to uncertainly in the regulatory and legal environment, especially with regard to GMP regulations, whose adoption is necessary in order for local companies to export. • Because of their late inception, CAPS activities in Pharmaceutical Cluster were limited both in terms of timing and funding. An example of this was the local PR campaign that was undertaken with a very limited amount of funding and as a consequence, had an almost undetectable effect, as evidenced in local sales figures provided by the National Statistical Service. • Expensive advertising campaigns and bribery practiced by foreign firms and their importer partners was cited as an obstacle for Armenian firms to increase their shares in local market. Still, it is necessary to conduct campaigns to educate the public and professionals alike that locally produced products are of equal quality and less expensive. • The vocational education and training activities were appreciated by the industry and were successfully implemented on a cost-sharing basis and this served to strengthen linkages between the private sector and educational institutions. • The industry needs further institutional and technical support to complete the GMP certification process. Recommendations: • USAID should continue to support the GMP certification process until it is finalized. This involves the adoption of the respective decree by the government and the subsequent completion of the certification of all of the individual companies. • USAID should consider providing an international expert to participate in the first round of producer certifications by accompanying the local staff during their assessments. • A campaign should be conducted to educate doctors, pharmacists and the general public about the fact that national products produced by Armenian companies are safe and equally effective as more costly imported products. • Consideration should also be given to pursuing ISO 17025 laboratory testing and calibration certification, especially if the government labs might be used in 10 counterterrorism efforts by providing quick and accurate analysis of suspected narcotic substances. • An anti-corruption component should be included to address the pervasive problem of bribes or other illegitimate incentives paid by pharmaceutical manufacturers to doctors and pharmacists for the purpose of promoting their brands. 3. Tourism Cluster In 2007, the CAPS project helped the Ministry of Economy (MOE) to conduct a survey of tourist arrivals. Since then, the MOE and the NSS have continued collecting this information. Their data reveals that despite a 4% decline in global tourism numbers in 2009, Armenia saw an increase in its number of visitors. In the past year, the Tourism Cluster has made progress developing regional cooperation with neighboring countries, and continued targeted promotion directed at Italy and France as desirable geographical markets. It also addressed the religious travel segment, upgraded key tourist attractions and strengthened the quality of tourism education by collaborating with internationally-accredited foreign universities, increasing the skills of lecturers and introducing international curricula. Overall, the National Statistical Service reported that 86,569 tourists visited Armenia during the first three months of 2009, which represents a 2.8 % increase over the same period last year. One of the evaluation focus group participants, Mr. Mekhak Apresyan, the Head of the Tourism and Regional Economic Development Department at the Ministry of Economy, was recently quoted in the news media as stating: “Armenia’s annual growth in tourist arrivals has been 23-25% over the past seven years”...“A 9.4 % increase was recorded in the number of the tourists in 2008 (558,443) compared to 2007 (510,287).” Legal and institutional support provided by CAPS included assistance to develop and adopt a Tourism Master Strategy, draft and submit a new Tourism Law to the Ministry of Economy (MOE) and provide legal recommendations to the MOE and private sector organizations. CAPS attempted to approach the General Department of Civil Aviation about performing an Air Transport Competitiveness Assessment. However, this seems to be too sensitive area to get involved at this time. CAPS also assisted the National Statistical Service (NSS) to obtain more reliable visitor information statistics and identifying the problem behind the inaccuracy of existing data. Upon the request of the Minister of Economy, CAPS provided assistance for the creation of a Jermuk Tourism Development Strategy that was approved in January 2010. To increase the workforce capacity in this cluster in accordance with its mandate, CAPS worked with universities specialized in tourism education. CAPS facilitated two university academic partnerships, a long-term partnership between Virginia Tech University and the Armenian Greek College and a curricula-based partnership between George Washington University and the American University of Armenia. Programs were implemented in food safety and destination management. CAPS also organized an internship program for students from tourism-related universities, a tourism summer school and follow-up support to lecturers who participated in the Community Connections Tourism Educators Exchange Program. The numerous stakeholders interviewed agreed that the targeted promotion of Armenia as a tourist destination in the Italian and French travel markets advocated by CAPS was highly valued by industry representatives, both private and public. Artak Ghazaryan, the CAPS Director, was quoted in an American Chamber of Commerce in Armenia publication as stating that CAPS' 11 targeted promotions to the Italian market during 2007 and 2008 resulted in a 35% increase in Italian tourist arrivals to Armenia and a doubling of visa applications. Among other activities, CAPS has continued to support World Federation of Tourist Guide Associations (WFTGA) training and development of the capacity of the Armenian Guides Guild to improve customer service. The visitor experience at many of Armenia’s attractions was further improved last year through support for the Armenian Monuments Awareness Program (AMAP), which placed high-quality, USAID-branded panels, directional signs and placards at more than 40 locations throughout the country. During their field trips, the consultants personally witnessed foreign and domestic tourists spending time reading the informational placards that gave the historical and cultural background of the site in four languages. The Tourism Cluster continued to gain strength over the past year with more evidence of collaboration, networking, information sharing, partnership development and joint actions among members of the cluster, including government and academia. Examples of successful collaboration include the regional tourism workshop in Istanbul, the annual Tourism Competitiveness Conference and the CTS Travel Fair. Conclusions: • The project has a long list of activities in the Tourism Cluster. However, they are less efficient than comparable activities in two other clusters. The degraded infrastructure, the closed aviation sector and the lack of modern, affordable hotels are certainly contributing factors, but these areas were not within the scope of the CAPS project. Addressing difficult structural issues such as these is a long-term process that will require proactive initiatives on the part of both business and government. • Country promotion activities targeted at Italy and France had a significant effect of raising the level of awareness in those markets and increasing the number of tourists. There are no precise figures available as of this writing, but there is delayed data being gathered by the NSS which will likely support the anecdotal accounts heard by the consultants during their interviews. However, it is impossible to determine whether these tourists were informed by the CAPS campaign, a friend or a relative or some other source. • Some areas crucial for tourism development, such as aviation, are still closed to collaboration and effective dialogue. • CAPS supported the establishment of associations in the Tourism Cluster and provided funding for them. However, their sustainability is uncertain due to the fact they are too weak at the moment and too dependent on donor-funding. In addition, they are fragmented along occupational lines and do not represent the whole industry, and consequently do not have a clear strategy for expansion and development. • Natural synergies with IT companies led to increased productivity of tourism companies. An example of this sort of mutually beneficial business collaboration would be the adoption of appropriate software designed, installed and serviced by an Armenian IT 12 company that would allow a tour operator to sell a tour packaged online and provide booking and reservation services at the same time. • Quality service in this industry is lacking, despite the number of activities designed to improve it. The usual quality measures applicable to any service, such as timely and accurate responses and polite and comprehensive satisfaction of a customer’s request, are frequently absent. • The tourism infrastructure is in a state of deterioration and there is a lack of middle-tier hotels to serve tourists who are on a limited budget and cannot afford luxury accommodations. Recommendations: • If support to Tourism Cluster is to be maintained in the future, targeted promotions of Armenia as a tourist destination should be continued. • USAID should provide detailed firm-level assistance to develop, package and sell new and attractive tourism products on international markets. • Assistance in increasing the quality and quantity of proposed service spectrum would also help develop this sector. V. Summary of Recommendations Future projects should build on the successes of the CAPS project, both in terms of productivity increases and cluster strengthening, both of which have been major success stories to date. Specifically, the recommendations are: Narrow the scope To achieve more tangible outcomes, USAID should consider narrowing down the scope of their new program so as to focus on several select activities than can demonstrate results. Having too many smaller dispersed activities over several clusters make management and planning much more complicated, and the outcomes become diluted. Small scale activities should be dropped and the program should concentrate on large sub￾projects within a particular industry, such as GMP certification for pharmaceuticals. Another component could be an incubation/start-up support effort which would encompass an integrated approach including a seed financing facility, training and capacity development, marketing and promotion on international markets. Another different approach could entail a scenario whereby the new project would only cover the IT sector and focus its efforts on the cross-sectoral use of technology. IT services are generally underutilized in Armenia, and promoting local IT companies on the domestic market would help support not only them, but also their non-IT partners who sell their products to Armenian companies from other industries e.g. tourism, pharmaceuticals, chemistry, mining, etc. In any of these scenarios, collaboration with the industry associations and the GOAM is going to be necessary to ensure state support and understanding of the donor’s activity. 13 Use a combination of industries to create natural synergies Having more than one sector included in the development program can lead to the creation of natural synergies between the companies operating in these areas. The CAPS experiences demonstrated that the use of IT solutions has significantly increased in the Tourism Cluster since they had the opportunity to communicate with each other and understand the benefits of mutual cooperation. Include new promising sub-sectors in the new program, such as engineering services (industrial automation, precision electronics, etc) To maximize the impact of local economic development, productivity increases and job creation, USAID must clearly identify potential obstacles and develop realistic opportunities that foster competitive economic growth in Armenia. Further capacity building should be provided to targeted businesses, business service providers and associations, all of which have the potential to encourage local investment and leverage other donor and public funds. VI. Lessons Learned After looking closely at the documents provided and having interviewed numerous beneficiaries of the project, the evaluation team concludes that the processes, innovations, institutions, partnerships and linkages that were introduced are fundamentally sustainable in that CAPS activities were instrumental in building capacity in the country and stimulating growth in the industries that were targeted, thus validating this particular form of development model. However, long-term sustainability will be influenced by global economic factors and the ability to secure financing for some of the more costly elements of the various programs, such as hardware and software replacement, international conferences and seminars, study tours and foreign experts and speakers. In terms of cooperation, CAPS was effective in collaborating with the government, private companies and universities. There are other programs targeting the IT sector in Armenia sponsored by corporations such as Microsoft, Hewlett-Packard and D-Link. These would appear to be equally appreciated by the beneficiaries and serve to foster healthy competition in the sector. Gender equity in all project activities was evident to the evaluation team, which found that in most instances there was a balance of gender participation in project activities. It was the evaluators’ impression that the widespread gender equality that was observed at all of the project sites was a consequence of sociological characteristics prevalent in Armenian society. That is, there was equal demand among men and women for the services and benefits provided by USAID's projects. 14 ANNEXES 15 Annex 1: Documents Reviewed CAPS Reports 111-C-00-05-00059-00 - Quarterly Report Competitive Armenian.pdf CAPS Quarterly Report - 1st Quarter 2010 - FINAL.docx CAPS Quarterly Report Oct Dec 2009.docx CAPS Quarterly Report Jul Sep 2009.doc CAPS Quarterly Report_April-June 2009.doc CAPS_Quarterly_Report_Jan-Apr 2009.doc CAPS_Quarterly_Report_Oct-Dec 2008.doc CAPS_Quarterly_Report_Oct-Dec 2008_A CAPS_Quarterly_Report_July-Sep 2008.doc CAPS Quarterly Report_Apr-June 2008.doc CAPS Quarterly Report_Jan-Mar 2008.doc CAPS Quarterly Report Oct-Dec 2007.doc CAPS Quarterly Report Jul-Sep 2007.doc CAPS Quarterly Report_Apr-Jun 2007.doc CAPS Quarterly Report Jan-Mar 2007.doc CAPS_Quarterly Report_2nd Qtr 2006.doc CAPS_Quarterly_Report_3rd Qtr 2006.doc CAPS_Quarterly_Report_4th_Qtr 2006.doc CAPS M&E Annual Reporting Memo Nov 2006.doc 111-C-00-05-00059-00 - CAPS Quarterly Report December 2006.pdf CAPS Work Plans CAPS 2009 Workplan & Resource Est_19June.xls CAPS 2010-11 Work Plan Master File_Final.xlsx CAPS 2010-11 Work Plan Master File_Final_REVISED.xlsx CAPS 2010-11 Work Plan Master File_Final_USAID COmments.xlsx CAPS SOW 081004.doc CAPS Work Plan (Final).DOC CAPS Work Plan _6 MONTH_Sep 2008-Feb 2009.doc CAPS Work Plan _FY2007.doc CAPS Work Plan Institut Sppt FY09-10.doc CAPS Work Plan Narrative FY09-10.doc CAPS Work Plan Policy Actions FY09-10.doc CAPS Work Plan Status_Jan2010.doc CAPS Work Plan_FY2008_10 Oct.doc CAPS Work Plan_FY2008_Status.doc CAPS Workplan Tables FY09-10.xls CAPS Workplan Tables FY09-10_w-telecom_1Apr 2009.xls CAPS_SOW_081004.doc CAPS Operational Plan Indicators.xls Final_Annual_workplan_for_CAPS Appendix A.XLS Final_Annual_workplan_for_CAPS.DOC Indicative Work Plan Task Schedule_Final.xls Indicative Work Plan Task Schedule_Final_REVISED.xls NATHAN-#231539-v1-CAPS_2010-11Workplan_Final.DOC NATHAN-#231539-v1-CAPS_2010-11Workplan_Final_REVISED.DOC Notes CAPS Operational Plan Indicators.xls FY2008 Work Plan Progress March-June 2008.doc 16 FY2008 Work Plan Progress Jan -March 2008.doc FY2007 Work Plan_Sept 2007 Progress.doc FY2007 Work Plan _June 2007 Progress.doc FY2007 Work Plan _March 2007 Progress.doc CAPS FY2007 Work Plan_Dec 2006 Progress.doc Progress Against Workplan September 2006.xls CAPS PAR Indicators FY2006.xls WorkPlanUpdate.xls CAPS Activity Summaries Activity Summary Cumulative through June2007.xls Activity Summary Cumulative through March2007.xls Activity Summary Cumulative through Sep2007.xls Activity Tracking for Qtr 3.xls CAPS Activity Summary April -June 2008.xls CAPS Activity Summary April -June 2009.xls CAPS Activity Summary Apr -June2007.xls CAPS Activity Summary Cumulative Calendar Year 2006.xls CAPS Activity Summary Cummulative.xls CAPS Activity Summary Jan -2010.xls CAPS Activity Summary Jan -Mar 2009.xls CAPS Activity Summary Jan -Mar2007A.xls CAPS Activity Summary July -Sep 2009 Final 2.xls CAPS Activity Summary July -Sept 2008.xls CAPS Activity Summary July -September 2006.xls CAPS Activity Summary July -September.xls CAPS Activity Summary Oct -Dec 2006.xls CAPS Activity Summary Oct -Dec 2008.xls CAPS Activity Summary Oct -Dec 2009.xls CAPS Activity Summary Year1.xls CAPS Activity Summary Cumulative through June 2008.xls CAPS Activity Summary Cumulative through March 2008.xls CAPS Activity Summary January -March 2008.xls CAPS M&E Association M&E September 2006.xls CAPS M&E Plan Indicator Table Proposed_Sept2006.XLS CAPS M&E Plan_Sept2006.doc CAPS M&EFullYearReporting_FY2006.XLS CAPS ME Data forUSAID_11Oct.doc CAPS ME Data forUSAID_2008.doc CAPS M&E_AnnualReport09Plan10_Final.doc CAPS M&E_AnnualReport09Plan10_Final_REVISED.doc CAPS M&E ResultsFY07&PlanFY08.XLS CAPS M E Annual Report 2007 & Plan 2008.doc First_M_E_plan_for_CAPS.doc First_M_E_plan_for_CAPS_Table_B -1.XLS M&E Results 2006 Mem.doc M&E Associations Qtr June 2006.xls M&EReportingYear1.XLS CAPS M&E Plan Add Year2 Targets.XLS Association M&E September 2006.xls 17 CAPS M&E Plan Indicator Table Proposed_Sept2006.XLS CAPS M&E Plan_Sept2006.doc CAPS M&EFullYearReporting_FY2006.XLS CAPS ME Data forUSAID_11Oct.doc CAPS ME Data forUSAID_2008.doc CAPS M&E_AnnualReport09Plan10_Final.doc CAPS M&E_AnnualReport09Plan10_Final_REVISED.doc CAPS M&E ResultsFY07&PlanFY08.XLS CAPS M&E AnnualReport07&Plan08.doc First_M_E_plan_for_CAPS.doc First_M_E_plan_for_CAPS_Table_B-1.XLS M&E Results 2006 Mem.doc M&E Associations Qtr June 2006.xls M&E ReportingYear1.XLS CAPS M&E Plan Add Year2 Targets.XLS Other Documents Pharmaceutical Sector Development Council, Concept Paper 2009 The Impact of Targeted Promotion of Armenia as a Tourism Destination to the Italian Travel Trade & Public Transparency International Global Corruption Barometer 2009 U.S. Ambassador’s Speech at the American University of Armenian 2010 Global Integrity Scorecard Armenia 2007 Establishment of Advanced Educational and Technology Resources in Armenia - USAID 2010 18 Annex 2: List of Interviews General Interviews – June 14 to June 24, 2010 Name Title Organization Jatinder Cheema, Ph.D Mission Director USAID Timothy Alexander Director, Program Office USAID Simon Sargsyan, Ph.D Project Management Specialist, Economic Growth Office USAID Diana Avetyan Economic Growth Office, Private-Sector Specialist USAID Dr. Marina Vardamyan Office of Economic Growth, Energy, Water and Environmental Officer USAID Mariam Gevorgyan Monitoring and Evaluation Specialist USAID Artak Ghazaryan Director CAPS Armen Abrahamyan IT Cluster Coordinator CAPS Lala Margaryants Pharmaceutical Cluster Coordinator CAPS Gera Voskanyan Tourism Cluster Coordinator CAPS Armen Shahbazyan Business Associations Specialist CAPS Timothy Moore Associate, Enterprise and Industry Development Nathan Associates, Inc. Sophia Muradyan Senior Analyst Enterprise Incubator Foundation (EIF) Norayr Vardanyan Project Manager Sun Incubator Project Hrayr Ter-Nikoghosyan Project Manager Sun Solution Lab Vladimir Yeghiazaryan Head of Applied Math in Informatics Department Slavonic University (Russian￾Armenian University) Eduard Philiposyan Sun Campus Ambassador Russian Armenian University Karen Vardanyan Executive Director Union of Information Technology Enterprises Davit Sandukhchyan Chief Legal Officer Beeline Andrew Hovhannisyan Deputy General Manager Synopsis Gurgen Paronyan Executive Director President Gyumri IT Center 3-D Modeling Union Davit Kocharyan Executive Director developWay CJSC Davit Grigoryan Director Flexible Applications CJSC Arman Atoyan Founder, Creative Director X-TECH Sofi Baroyan Director Vericel Service Emil Gabrielyan M.D. Director President Drug Agency Pharmacological Society of Armenia Gevorg Yaghjyan, M.D., PhD Vice-Rector Yerevan State Medical University 19 Gevorg Safaryan Director LimeTech Frans Stobbelaar Pharmaceutical Expert Pharin International Robert Harutyunyan, PhD Director-General Armenian Development Agency Mekhak Apresyan Head of Tourism Department Ministry of Economy Pegor Papazian Chief Executive Officer Competitiveness Foundation of Armenia Varoozhan Harikian Dean of Extension Programs American University of Armenia Hayk Chobanyan Director Ministry of Labor and Social Issues, "Nork" Information￾Analytical Center Alex Sardar Country Team Representative Counterpart Karine Avetisyan Head of Actuarial Mathematics Department Yerevan State University, Sun Training Laboratory Syuzanna Azoyan Marketing Director Competitive Foundation of Armenia, Armenian Tourism Development Agency (former) Noubar Tatarian President Armenian Hotel Association Amalia Stepanyan Executive Director Armenian Hotel Association Yeghishe Tanashyan President, Armenia Chapter, Managing Partner American Society of Travel Agents Five Stars Travel Pharmaceutical Focus Group Participants – June 18, 2010 Name Title Organization Vardan Mkrtchyan Marketing and Sales Manager Liqvor CJSC Sona Khachatryan Quality Assurance Manager Esculap Ltd. Shahe Kassis Director Medical Horizon Gevorg Yaghjyan, M.D., PhD Vice-Rector Yerevan State Medical University Samuel Zakarian Director Medicine Producers and Importers Union of Armenia Azam Ghazaryan Director GXP Center of Excellence Tourism Focus Group Participants – June 18, 2010 Name Title Organization Naira Sukiasyan President AGG Luisa Khalatyan AGG Varoozhan Harikian Dean of Extension Programs American University of Armenia Shushan Khachatryan Fairyland Travel Agency Amalia Stepanyan Executive Director Armenian Hotel Association Yeghishe Tanashyan President, Armenia Chapter, Managing Partner American Society of Travel Agents Five Stars Travel 20 Mekhak Apresyan Head of Tourism Department Ministry of Economy Lusine Martirosyan Sima Tours Hovhannes Morgovyan Armenian travel Bureau IT Focus Group Participants – June 18, 2010 Name Title Organization Hayk Chobanyan Director Ministry of Labor and Social Issues, "Nork" Information￾Analytical Center Arman Atoyan Founder, Creative Director X-TECH Gevorg Safaryan Director LimeTech Davit Kocharyan Executive Director developWay CJSC Grigor Barseghyan Country Manager Microsoft Sofi Baroyan Director Vericel Service Davit Grigoryan Director Flexible Applications CJSC Tatevik Sakradyan Marketing Specialist Flexible Applications CJSC