Rritje Albania / Competitive Enterprise Development Mid-term Project Evaluation of Chemonics Contract # 182-C-00-09-00002-00 Nov 28-Dec 20, 2011 Final Report Submitted Jan 20, 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by independent consultant Todd Kirkbride under REQ-182-12-000003 and E&E Senior Competitiveness Advisor Neal Nathanson. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CED – Mid-Term Project Evaluation Dec 2011 2 CONTENTS I. ACKNOWLEDGMENTS ...................................................................................................... 3 II. ACRONYMS AND ABBREVIATIONS ............................................................................. 4 III. EXECUTIVE SUMMARY ................................................................................................. 5 IV. INTRODUCTION ............................................................................................................... 9 - Background ......................................................................................................................... 9 - Project Goals & Objectives ............................................................................................... 10 - Purpose of Evaluation ....................................................................................................... 11 - Assessment Methodology.................................................................................................. 11 V. FINDINGS .......................................................................................................................... 13 Overall Project Design, Implementation Strategy and Operations ...................................... 13 - Appropriate Resource Allocation to Accomplish Objectives ................................... 13 - Appropriate Implementation Strategy to Accomplish Objectives ............................ 14 - Achieving Long-term Sustainability ......................................................................... 15 Project Impact ...................................................................................................................... 15 - Analysis of Project Indicators ................................................................................... 15 - Engaging with Start-ups ............................................................................................ 18 - Engaging with Existing Enterprises .......................................................................... 19 - Impact on Productivity, Diversification, and Job Creation ....................................... 19 VI. CONCLUSIONS AND LEASONS LEARNED ............................................................... 20 - Maximizing Impact for SME Competitiveness ................................................................. 20 - Value-added Practices Provided by CED .......................................................................... 21 - Collaboration with Other Donor Activities ....................................................................... 21 - Innovative Development Solutions ................................................................................... 22 VII. PRACTICAL RECOMMENDATIONS .......................................................................... 23 - Proposed Project Modifications ........................................................................................ 24 - Advantages of Shifting Project Focus ............................................................................... 25 - Revisions to Expected Results/Indicators ......................................................................... 25 - Revisions to Project Components ..................................................................................... 27 - Addressing Overarching Business Constraints ................................................................. 28 - Considerations to Engage Agribusiness Sector ................................................................. 29 VIII. ANNEXES ...................................................................................................................... 31 Annex I Scope of Work ................................................................................................. 31 Annex II CED Evaluation Workplan .............................................................................. 37 Annex III Stakeholder Interview Questions ..................................................................... 38 Annex IV List of Persons/Organizations Interviewed ...................................................... 39 Annex V Mission Debrief Topline Recommendations ................................................... 41 Annex VI List of Documents Reviewed ........................................................................... 42 CED ALBANIA – Mid-Term Project Evaluation Dec 2011 3 I. ACKNOWLEDGEMENTS The Review Team would like to express its sincere appreciation to a number of persons who have facilitated this assessment. First and foremost, we would like to thank Alma Kospiri, the project COTR, for arranging the assessment and for her guidance and assistance in making this review a reality. Along with other key USAID staff, she prepared the scope of work for the assessment, selected and arranged for many of the key stakeholder interviews and accompanied the team to many of their meetings throughout Albania. As with any assessment, an overwhelming amount of logistics coordination is required and this report would not be possible without the tireless support from Luljeta Qorlaze (also as primary translator), Ketrina Janaqi at the Mission and Tara Lala-Pritchard with the CED Project. We would also like to thank other members of the USAID/Albania Mission for their assistance and advice on the assessment. These include the overall support from Mission Director Joe Williams, General Development Officer Marc Ellingstad and M&E Specialist Alken Myftiu. We also wish to thank the very professional USAID driver for our field trips, Arben Afezolli who succeeded in reaching all our client visits successfully and who brought us back safely to our lodging each night. Other staff members in the Executive and Controllers Offices also provided important and timely assistance. We also appreciate the attention paid by the entire Mission management team to the evaluators‟ debriefing on Thursday December 15th . The assessment benefited greatly from the briefings and meetings with the CED project staff in order to better understand the strengths and constraints of the project. The Chief of Party (COP), Victor Luboyeski, and the M&E Specialist, Tara Lala-Pritchard were extremely helpful in arranging for meetings with the various parties involved in the project and provided important insights on project implementation both at the beginning of the assessment and after the team‟s field trips. Finally we would also like to thank the representatives of the various private sector companies and partner organizations that have worked with CED to achieve project objectives. The list is too long to mention here but these organizations and their comments are reflected throughout this report. They provided essential information that allowed us to better understand the challenges of their industries, the relationships between them and the CED staff and potential for achieving greater impact in the future. CED – Mid-Term Project Evaluation Dec 2011 4 II. ACRONYMS AND ABBREVIATIONS AADF Albanian-American Development Foundation AAQM Authentic Albania Quality Mark Program ACIT Albanian Center for International Trade AFS Albanian Factoring Services AIDA Albanian Investment Development Agency AITA Albanian IT Association AmCham American Chamber of Commerce APS Annual Program Statement ATA Albanian Tourism Association ATACT Association of Traditional and Artistic Crafts and Trades ATOU Albanian Tour Operators Union B2B Business to Business CDCS Country Development Cooperation Strategy CED Competitive Enterprise Development CFA Chamber of Fasson Albania CGP Client Growth Plan CMMI Capability Maturity Model Integration CPAR Contract Performance Assessment Report EBRD European Bank for Reconstruction and Development EG Economic Growth ETI Education and Training Institutions EU European Union GoA Government of Albania GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GFSWG Garment and Footwear Stakeholders Working Group GSTC Global Sustainable Tourism Council ICT-TRC ICT Training and Resource Center KPI Key Performance Indicator KRA Key Results Area METE Ministry of Economy, Trade, and Energy MTCYS Ministry of Tourism, Culture, Youth, and Sports MOU Memorandum of Understanding NAIS National Agency for Information Society NCGFM National Chamber of Garment and Footwear Manufacturers NTA National Tourism Agency PBMP Performance-Based Monitoring Plan PEP Productivity Enhancement Program PIR Project Intermediate Result PPD Public-Private Dialogue PPP Public-Private Partnership PUT Polytechnic University of Tirana RCI USAID‟s Regional Competitiveness Initiative SNV Stichtig Nederlandse Vrijwilligers (Netherlands Development Organization) SPI Software Process Improvement STTA Short-Term Technical Assistance TAM-BAS Turn Around Management – Business Advisory Services ToT Training-of-trainers VAT Value added Tax VET Vocational Education Training WEF World Economic Forum CED – Mid-Term Project Evaluation Dec 2011 5 III. EXECUTIVE SUMMARY On December 2011 USAID/Albania requested a mid-term evaluation for the Competitive Enterprise Development (CED) 1 project implemented by Chemonics International with the purpose of answering four key questions: 1. How successful has been the overall project design, implementation strategy, and operational approach? 2. Specifically, what has been the project impact (successes) so far? 3. In the context of lessons learned, what are the major challenges faced by CED? 4. What future strategic decisions are to be considered and what are the alternatives for any possible project improvements, particularly in light of the new Mission strategy (CDCS)? The main headline from answering these four questions is that USAID/Albania should continue to implement the CED project with some modifications. Even though CED‟s original assumptions, goals and objectives (with a primary focus on firm-level assistance) now appear to be in conflict with Mission‟s new Country Development Cooperation Strategy, evaluators believe that by capitalizing on the successes and momentum achieved thus far at the enterprise level, assistance can be successfully elevated to the industry level. As stated in CDCS‟s Development Objective #2, “emphasis is on establishing the conditions for growth, rather than on providing assistance directly to specific enterprises. Interventions at the firm and farm level will be phased out as emphasis shifts to working with key business and farming associations in identifying and addressing with the GOA constraints to doing business.” Project Background When the five-year Albania Competitive Enterprise Development (CED) contract was awarded in March of 2009, the main focus was to effect positive change in competitiveness at the enterprise level as measured through two key indicators: increased sales and jobs. At the time of this mid-term evaluation, CED was meeting almost all indicators set for the project and making notable progress in achieving its overall goals. However as this report details, project indicators appear to be a bit misleading in measuring the true nature of the impact of CED which is making most its progress in the second category of its three main components: (1) Strengthening trade and investment capacity; (2) Increasing enterprise productivity; (3) Improving workforce development. Upon project implementation, CED recommended to work with the five largest, non￾agricultural sectors2 with the greatest potential for sustained growth, which are Garment, Footwear, Tourism, ICT and Recycling. Once the sectors were identified, the project brings a mix of technical assistance, training, targeted grants and linkages to financing to help remove business constraints. 1 Also known as Rritje Albania, which means „growth‟. 2 Recycling has since been eliminated as a key sector. CED – Mid-Term Project Evaluation Dec 2011 6 Evaluation Methodology Evaluators Todd Kirkbride and Neal Nathanson conducted an extensive review of project reports and other technical documents produced by CED and USAID/Albania Mission before arriving in country. In order to obtain a reasonable picture of the strengths and weaknesses of project implementation, the team leader and COTR then selected a representative sample of almost 40 donors, government institutions, associations and chambers, NGO‟s, universities and private sector clients to interview in 4 key cities throughout Albania. Key findings and recommendations were presented to the Mission Director and key staff on December 16th 2011 and detailed here in this mid-term evaluation report. Key Findings Most every business and partner organization interviewed stated that CED has implemented this competitiveness project as it was originally designed in an effective and professional manner generating a fair amount of goodwill for both the project and USAID. While it may not be possible now to demonstrate high levels of revenue and job growth from activities to date, progress has been made in setting the stage at the firm-level to eventually realize this goal (however some of this may extend beyond the life of this project). From the projects‟ primary components, most progress was noted at enhancing firms‟ productivity even though an extensive amount of activities have been implemented in all three. And an argument could be made that workforce development is simply the flip side of Productivity Enhancement. When workers are trained and assisted to become more productive, these actions are both workforce development and productivity enhancement. In the case of garment/footwear businesses, the Key Performance Indicator strategy appears to be greatly assisting with the reduction of defects and the tracking and improving of assembly line operations. In both the tourism and ICT sectors, this is being realized primarily through the establishment and promotion of standards (Authentic Albania Quality Mark for tourism and SPI-IT Mark for technology). The evaluation team was explicitly asked to analyze project impact in the context of indicators which revealed that while some of these indicators are reasonably helpful at gauging quantitative and qualitative aspects of the project, overall they do not convey a complete or accurate picture of project activity and benefits. In some instances the numbers present a more favorable view of activity, while in other cases the numbers actually understate or diminish the benefits. There is a significant amount of overlap in reported activities, and while the project does attempt to avoid double-counting, there is a significant amount of arbitrariness in the way data is presented. In addition there may be some question around the original targets set since the project is reporting that it is achieving (in some cases) as much as 400% of the contract‟s targets. Challenges and Lessons Learned As with any development project, CED has come across its fair share of challenges but appears to have taken steps to learn from their mistakes and successfully implement improvements. Not to mention that throughout the life of the project, Europe (and the rest of the world) has been experiencing an economic recession that has caused many of the weaker firms to close but at the same time opened up new opportunities for growth not previously experienced. According to most the businesses and associations interviewed, the project has fortunately experienced a full turn-around from a very uncharacteristically slow start missing most of its CED – Mid-Term Project Evaluation Dec 2011 7 first year targets. Also, great efforts were made trying to merge the two leading garment and footwear associations to no avail which lead to the recent creation a public-private working group of key industry stakeholders. Lastly, little measureable progress seems to have been made in helping firms actually receive new forms of capital (also not an explicit deliverable for the project). CED has demonstrated productive collaboration in their four key sectors with other donor activities including SNV, GIZ, AmCham, TAM-BAS, AADF, and USAID‟s AAC project. The project was given very high remarks from each of these donors who described CED‟s level of assistance as proactive, technically competent and delivered professionally. Although CED has engaged in several public private partnerships and dialogs, the evaluation team still believes there are more possibilities to be explored in this area in order to maximize leverage from other interested partners working in the same sectors. The signing of the ICT Training Resource Center MOU was a large milestone in the process of launching a partnership that has been many years in the making. And the PPD working groups established around making revisions to the tourism law and establishing an open forum platform around the garment and footwear stakeholders are good examples at the project‟s attempt to leverage the influence, talent and resources of numerous organizations involved in these issues. Key Recommendations As stated early, the evaluators recommend continuing to implement the CED project through the life of the contract but consider making the following overall modifications: 1. CED should shift its focus away from primarily providing one-on-one firm level assistance towards more mesa level support. However the evaluators believe that some degree of firm-level assistance should be maintained but only to key companies within each sector who could provide the greatest multiplier effect from this continued technical assistance. In order to most effectively make this transition, CED should evaluate key associations (most they are already working with) and NGO‟s best positioned in each sector to support and expand upon the momentum gained with firms. 2. Provide assistance at the macro level only in areas previously engaged including:  Garment/footwear: increase practical deliverables within new working group; enhance coordination with Ministry of Economy and Trade around policy and industry priorities; develop „Made in Albania‟ campaign promoting industrial production.  Tourism: re-engage with tourism working group to pass tourism law; support NTA to develop and implement tourism promotion action plan and address constraints between central and municipal implementation; elevate regional promotion efforts through partnerships at both public and private levels.  VAT: examine key issues raised by stakeholders to balance the application and collection of various taxes that disadvantage businesses in projects‟ sectors and determine whether worthwhile to pursue. 3. Modify Workforce Development component to no longer double count activities delivered in other components, expand and ensure sustainability of the internship program and explore new options to reduce the large skills gaps between the workforce and current business needs. 4. Continue to deliver/sponsor technical workshops and trainings across key sectors but modify implementation (see full report). CED – Mid-Term Project Evaluation Dec 2011 8 5. Do not add agribusiness as a new sector for CED to support. 6. Explore ways within the CED contract to streamline their ability to evaluate and support new public private partnership opportunities. Specific to garment/footwear: 1. Reduce firm level assistance to only companies with ability to shift away from traditional „fasson‟ to full-cycle AND to be willing to expand sales to new markets. 2. Investigate potential for sourcing raw materials thru joint venture opportunity for warehousing with foreign companies (intermediaries vs. production). Specific to tourism: 1. Expand support of AAQM by expediting website development, implement sound marketing strategy to promote standards, integrate more stakeholder feedback in providing more flexibility and growth within grades and sub-categories. 2. Strengthen ATA to ensure ability to implement and expand AAQM program. 3. Support organizations best suited to open new tourism markets including Meetings, Incentives, Conferences, Expositions (MICE) and Cruise ships. Specific to ICT: 1. Assist AITA in transitioning to include „C‟ Communications in sector. 2. Ensure successful launch and operations of ICT Resource Center. 3. Expanding marketing and support of certifications (more TOT?). 4. Increase B2B type activities linking IT to other sectors. New opportunity areas: 1. Increase access to capital by developing more specialized forms of financing tailored to each sector and ensure companies have ability to access. 2. Assist with the implementation of Intellectual Property Rights by working at the mesa/macro level in a PPP model. 3. Coordinate technical assistance activities more in line with EU Accession agenda. 4. Potentially support AmCham with introducing Corporate Social Responsibility to businesses. CED – Mid-Term Project Evaluation Dec 2011 9 IV. INTRODUCTION Background The development of competitive private enterprises, especially SMEs, is critical for Albania to increase gainful employment and household incomes, and to reduce poverty and unemployment. Furthermore, as the efficiency and productivity of the agricultural sector improves, the need to create more non-agricultural jobs to absorb a continuing inflow of migrants from rural to urban areas becomes increasingly urgent. Thus, a key development challenge in Albania has been to improve the competitiveness and productivity of existing enterprises so that they can expand and/or diversify their businesses in domestic and regional markets, thereby increasing revenues and profits while creating more jobs. At the time of the design, it was considered focusing assistance on existing enterprises has not been considered sufficient in itself; but it was also necessary to spur competitive start-up enterprises to meet the changing demands of open markets and Albanian consumers whose incomes have risen in recent years. At the time of the design of the Albania Competitive Enterprise Development (CED) project in 2009, the Government of Albania (GoA) and donors‟ focus on the development of competitive SMEs – both existing and start-up – was considered an appropriate strategy to promote sustained, broad-based economic growth. “To increase competitiveness and promote growth, Albania must come up with innovative approaches (and less replication) and/or production based on low labor costs. Enterprises need to: (1) respond to end market demands through improved products and processes; (2) continuously invest in human and capital resource development; (3) develop more efficient organizational structures within firms and value chains; and (4) improve production to marketing linkages within enterprises and address cross-cutting constraints that affect related businesses.”3 In order to help Albanian SMEs achieve these objectives, on March 18, 2009, USAID/Albania awarded Chemonics International the Albania Competitive Enterprise Development (CED) contract, a five-year activity with a value of $9,774,154. Now two and a half years later, the Mission has commissioned this mid-term evaluation to assess the effectiveness of project implementation and impact achieved, and also to provide recommendations on how CED‟s future activities should be modified to better align with the Mission‟s new Country Development Coordination Strategy (CDCS) released in November of 2011, which now states: “Emphasis is on establishing the conditions for growth, rather than on providing assistance directly to specific enterprises. Interventions at the firm and farm level will be phased out as emphasis shifts to working with key business and farming associations in identifying and addressing with the GOA constraints to doing business. This change is predicated on a belief that targeting precise areas of the country or specific enterprises to reduce poverty and regional disparities in income would not be the most effective use of limited USG resources.”4 CED‟s original assumptions, goals and objectives with a primary focus on firm level assistance now appear to be in conflict with Mission‟s new CDCS. Therefore this report will address some options for bringing CED and the CDCS into closer alignment. 3 CED Evaluation Scope of Work 4 USAID Albania Country Development Cooperation Strategy 2011-2015 CED – Mid-Term Project Evaluation Dec 2011 10 Project Goals and Objectives The goal of CED is to enable Albanian private enterprises in selected regions to expand their businesses and improve competitiveness, thus contributing to the country‟s continued, broad￾based economic growth. Its main objectives are to support the sustained growth of non￾agricultural enterprises, increase household income and create profitable employment by developing and strengthening the competitiveness of existing enterprises and start-ups in targeted regions of Albania. These objectives will be achieved through three component areas of intervention: (1) Strengthening trade and investment capacity; (2) Increasing enterprise productivity; (3) Improving workforce development.5 CED works with entrepreneurs, investors and other stakeholders involved in selected sectors (Garment, Footwear, Tourism and ICT) and firms within those sectors to identify constraints to competitiveness. Once identified, the project brings a mix of technical assistance, training, targeted grants and linkages to financing to help remove these constraints. The main focus of the project is to effect positive change in competitiveness at the enterprise level as measured through two key indicators: increased sales and jobs. Over the life of the project, CED looks to contribute to at least $98 million USD in increased sales at assisted businesses. The project contract directs CED to focus on both start-up firms and downstream/upstream extension by existing enterprises that will emerge to take advantage of opening markets and increase domestic incomes over the life of the project. It has actively searched out start-ups and companies with high growth potential to provide the demand-driven support needed to generate sustainable new income and jobs for Albanians. However, it has been relatively unsuccessful at finding start-up firms with the growth potential that exists with more established firms. The project contract also directed CED to focus a substantial share of its resources at the firm level, and only secondarily at constraints that are hindering the development of more competitive enterprises at the meso and macro levels of the targeted sectors. However, even before the adoption of the CDCS, which mandates a shift toward higher-level interventions, the project already was beginning to identify opportunities for industry-level impacts through meso and macro levels of the tourism, garment, footwear and ICT sectors6 . According to CED Annual Report, requests for project support are measured against their contribution to the achievement of the following objectives for CED‟s target industries:  Promote Albania and its enterprises as sources of secure, dependable, quality￾conscious and competitively priced products and services. 5 The project contract uses terminology from USAID‟s “F” process of 2008-2009. Under that terminology, the “Program Area” is economic growth, with three “Program Elements”. The CED Project reports, and the common understanding in the mission refers to three “Components” which are exactly the same as the three “Program Elements”. This evaluation uses the terminology of Components instead of Program Elements, and the program “goals and objectives” are the same as the “Program Area” of Economic Growth. 6 As indicated in CED Year 3 Work plan CED – Mid-Term Project Evaluation Dec 2011 11  Support and strengthen key meso-level actors, including associations, chambers of commerce, educational institutions and business services providers (BSPs).  Support and strengthen public-private-partnerships (PPP) and public-private-dialogue (PPD) forums by working with the private sector and the Government of Albania to help create a well-functioning enabling environment for the development and expansion of competitive enterprises.  Improve the competitiveness of Albanian enterprises through scalable, tailored solutions delivered at the firm level, including movement into higher value-added products and services. Purpose of Project Evaluation The purpose of this external mid-term evaluation is to assess: (1) the project‟s overall performance, (2) the appropriateness and effectiveness of the interventions; (3) the convergence (or otherwise) of the project‟s objectives with the October 2011 approved Country Development Cooperation Strategy (CDCS) for USAID/Albania; (4) current or potential synergy with other USAID- and other donor-funded projects, in order to leverage additional resources; (5) the potential for long-term broad-based impact and sustainability; and (6) lessons learned and practical recommendations for increasing impact and assuring congruence with the Mission‟s strategy. Although all aspects regarding CED project objectives, operations and results are of interest to this evaluation, particular emphasis should be given to identifying lessons learned, and formulating practical recommendations. Evaluation Methodology The evaluation was conducted during December 2011 by Todd Kirkbride, Independent Consultant, and Neal Nathanson, USAID Senior Competitiveness Advisor with the E&E Bureau. The evaluation consisted of a review of project documents (scope of work, work￾plans, quarterly and annual reports, PBMP plans, Contract (and modifications), Mission CDCS Strategy), and other technical papers prepared by the project, followed by field visits and interviews with Chemonics staff in Tirana, and 37 beneficiaries and partners7 throughout Albania. Upon arrival in Tirana, the evaluation team leader spent the first couple days orienting with key USAID and CED staff. This included an overview briefing by the Mission Director to describe the overall project, key challenges that he feels the project faces and the need to not only evaluate the past performance of the implementer but to make key recommendations on how the Mission can best enhance the project to better align with the recently released CDCS. Subsequent meetings with COTR and M&E Officer helped answer questions raised during document review, clarify and edit SOW, confirm target list of key organizations and businesses to interview during evaluation (including a representative sample of sectors, geographic locations and organizations with both strong/weak interaction with CED) and begin making scheduling arrangements for partner visits. Additionally, Team leader and COTR met with CED COP and M&E Specialist to discuss many of the questions raised from reviewing key project documents as well as reviewing schedule for interviews of clients and associations. 7 Out of 530 total companies assisted of which only 74 are CGP clients and approx. 30 other participating partners. CED – Mid-Term Project Evaluation Dec 2011 12 In order to obtain a reasonable picture of the strengths and weaknesses of project implementation, the team leader and COTR selected a representative sample of almost 40 donors, government institutions, associations and chambers, NGO‟s, universities and private sector clients to interview in key commercial cities of Tirana, Shkodra, Durres, Fier and Berat8 (Annex III). Interview questions were prepared from the evaluation SOW but adapted to each organization‟s level of engagement with the project and general knowledge about specific sectors (Annex II). During all of these interviews, partners provided frank and open discussions of project implementation issues and suggestions on how to maximize project impact during the second half of the contract. The team purposely did not invite the CED staff to accompany them on these visits in order to have unbiased conversations with the partners and beneficiaries of the activities. Throughout the beneficiary interviews, the team conferred with both Mission and CED staff regularly to provide additional information and clarification on issues raised in meetings with partners. A final meeting was also held with CED COP to ground-truth a number of the conclusions and suggestions the evaluation team had arrived at after conducting its interviews. The evaluation team presented its findings, conclusions and recommendations at a debrief with the Mission Director on Thursday December 16th (Annex V). Final draft report was submitted to the Mission for comments prior to team leader‟s departure on Monday, December 19th 2011. 8 List initially prepared by CED staff CED – Mid-Term Project Evaluation Dec 2011 13 V. FINDINGS Overall Project Design, Implementation Strategy and Operations The first questions in the evaluation SOW asks whether CED project implementation has validated the hypothesis that the development of competitive private enterprises in the non￾agriculture sector, especially SMEs, is in fact critical for Albania to increase gainful employment and household income and reduce poverty and unemployment. Although CED has achieved some measurable impact in these macro economic growth indicators, and likely will achieve more in the future, it is still too early to answer this question definitively and affirmatively. The project had a slow start-up, so in reality there has been barely more than one year of effective operations. With the exception of only one private sector tourism operator, the other sector clients and partner organizations stated that CED has provided beneficial and relevant firm-level assistance in an effective and professional manner. While it may not be possible now to demonstrate high levels of revenue and job growth from activities to date, progress has been made in setting the stage at the firm-level to eventually realize this goal as demonstrated by a Finish textile company recent trip to meet/source with Albanian garment producers (however some of this may extend beyond the life of this project). These successful firm-level interventions have also established a base of knowledge, trust and support that allows for the elevation of USAID‟s assistance to the next level of working more directly at the mesa/macro levels. Appropriate Resource Allocation to Accomplish Objectives The project‟s main objective to support the sustained expansion of non-agricultural enterprises (and start-ups), increase household income and create gainful employment is very ambitious considering the projects relatively small size (less than 10 million) and limited time window of five years. Since resolving its initial start-up problems, beneficial activities have grown rapidly, and by the end of CED‟s second year, most targets were being met and exceeded, and interviews confirmed that progress being made at the firm level likely will lead to increased sales and better jobs created.9 Observations, particularly client interviews, affirm that the technical expertise offered by CED is of high quality and appropriate to the needs of the firms. Recipients of the assistance appreciate the practical, non-theoretical assistance. These firms describe very specific and tangible benefits that result from the assistance – particularly in areas of productivity, efficiency, and quality improvements. Through our review of industry reports (Annex IV) and stakeholder interviews, it appears that CED has selected the four key industries with the greatest overall impact and growth potential to help expand Albania‟s economy outside the agriculture sector. With regard to the cross-cutting theme of gender integration, two of the largest industries (garment and footwear) overwhelmingly employ women. However, what is more significant is that there is not a gender separation, with women in the lower-level jobs and men at the top. Women are fully integrated into all levels of firms in all sectors, as well as in the industry associations and other support institutions. Additional assistance targeting women appears throughout many of the toolkits designed by the project along with specific criteria in their grant application process favoring assistance to women. 9 There are some issues regarding the suitability of indicators and targets that will be addressed later. Nevertheless, interviews confirm that the quality of work is high and greatly appreciated by clients. CED – Mid-Term Project Evaluation Dec 2011 14 Finally, current staff size, skillsets and professionalism now appear appropriately aligned to most efficiently implement the project. This is a vast improvement from USAID‟s initial CPAR (1 year into project), which indicated that the project was not achieving many of its targets and that the project personnel were unable to provide the required technical assistance. Additionally, while there are some legitimate concerns about the reoccurring high costs of bringing in international consultants (STTA) to provide technical assistance, company interviews confirm that these international experts have added considerable value to Albanian firms. Madison Avenue‟s (garment producer) General Manager Buyar Dervishi remark‟s summarize the impressions of all other companies visited as he stated that this level of international expertise does not exist in-country (mostly referring to garment and footwear) and these experts‟ perspective is critical to advancing Albanian business strategies, competencies and knowledge of other markets. Nevertheless, a key part of the second half of the project will be the transfer of this knowledge and capability to Albanian institutions and service providers. Appropriate Implementation Strategy to Accomplish Objectives CED was designed to primarily enhance the economic growth of Albania by providing direct firm level assistance. This strategy was to be realized through three primary component interventions: trade & investment (essentially outreach to new markets), enhanced productivity, and workforce development. However, it was noted in half dozen businesses that they have realized domestic market expansion for sourcing and sales of their products but the overall low-priced market is not ideal for long-term growth. So far the evaluation team has confirmed numerous activities in all three areas across all four sectors but has noted more progress and results being made at enhancing firms‟ productivity than the others.10 In the case of garment/footwear businesses, the Key Performance Indicator strategy appears to be greatly assisting with the reduction of defects and the tracking and improving of assembly line operations. In both the tourism and ICT sectors, this is being realized primarily through the establishment and promotion of standards (AAQM for tourism and SPI-IT Mark for technology). Efforts are underway as well to expand the market outreach of firms, particularly to help them find potential buyers outside of the traditional Italian market. However, these efforts are not likely to show a real pay-off for some time. This market outreach recently led to an exploratory visit by a group of buyers in Finland looking for new production capabilities. They were quite favorably impressed with the quality of production capabilities in Albania, but as of yet have not placed firm orders. European buyers who have been sourcing production in Tunisia, Egypt, and Libya and looking for more stable alternatives are considering Albania as well, so additional market outreach efforts both by individual firms and the industry overall would be timely and productive. Nevertheless, the time lag between these initial outreach efforts and the actual realization of revenue growth could be one or two years, so these efforts are not likely to show up in actual indicator performance right away. Up till now, assistance directly to the companies has been critical to not only learn the exact constraints that each of these industries face but also to gain their trust and generate tangible (and measureable) business growth that can now be better capitalized on at higher levels. 10In many respects, Workforce Development is simply the flip-side of Productivity Enhancement. When workers are trained and assisted to become more productive, these actions are both workforce development and productivity enhancement. However, because these actions have the dual nature, at times they are counted twice in the indicators. The productivity enhancement of workers should continue, but it is not necessary to score the indicators twice. CED – Mid-Term Project Evaluation Dec 2011 15 Interviews across each of the four key sectors and types of organizations (businesses, associations, chambers, NGO‟s, public agencies, donors, and national government)11 confirm that the current strategy should be modified to include a higher level of mesa/macro assistance combined with a mix of more targeted firm level assistance to key companies. Achieving Long-term Sustainability With a continued focus exclusively on firm level assistance, could CED produce sizeable increases in sales while creating new and better jobs within these five years? Although CED is not likely to show such sizable increases, that does not necessarily disprove the core hypothesis. Five years is actually a short time to bring about fundamental economic transformation, and the loss of the first year‟s activity makes this outcome even less likely. There are (and will continue to be) numerous individual success stories and a steady increase in indicator results, but not on a grand scale and not exclusively attributable to CED‟s assistance. However, most clients and partners agreed that this is enough time to set many of these „change agent‟ businesses on the right path for incremental growth and therefore create a ripple effect among their suppliers, subcontractors and competitors that will continuously improve the entire industry. Additionally, with two and a half years remaining and a shift towards more capacity building and assistance at the mesa level, industry associations, chambers, foundations and local NGOs expressed their belief that this would be enough time to elevate their own resources, advocacy and technical assistance to member companies to ensure long-term sustainability well beyond the life of any donor assisted project. According to AITA (IT Association) Executive Director Lindita Komani, transitioning the activities provided by CED through their direct technical assistance and workshops will empower them to amplify these valuable skills to all their members and provide continuous improvement upon these as their industry grows. Project Impact The Evaluation Scope of Work asks for a review of project results, but also to look behind the reported numbers to ascertain the economic impact of CED on its clients and the quality of new jobs. The evaluators have examined the various impact measures, indicators, targets, and reported outcomes. The project has already undergone one modification to refine and simply the reporting requirements, reducing the original reporting framework down from 33 to 26 separate indicators. While some of these indicators are reasonably helpful at gauging quantitative and qualitative aspects of the project, overall they do not convey a complete or accurate picture of project activity and benefits. In some instances the numbers present a more favorable view of activity, while in other cases the numbers actually understate or diminish the benefits. There is a significant amount of overlap in reported activities, and while the project does attempt to avoid double-counting, there is a significant amount of arbitrariness in the way data is presented. In addition there may be some question around the original targets set since the project is reporting that it is achieving (in some cases) as much as 400% of the contract‟s targets. Therefore, this section first analyzes the indicators overall – focusing on both data reliability and suitability of the indicators as measures of project impact and then concludes by addressing specific (sub) topics raised in the SOW. 11 See Annex IV for specific list CED – Mid-Term Project Evaluation Dec 2011 16 Analysis of Project Indicators The CED contract initially established 4 “Program Area” level indicators and 29 “Element” level indicators. The contract was later revised down to 26 in total, grouped according to these categories:  Impact Level Indicators (4) – Macro-level indicators that attempt to measure the primary goals and objectives of the project - changes in revenues and jobs for assisted enterprises  Intermediate Results Indicators (10) – Primarily output indicators quantifying number of specific activities, events, etc., and degree of participation by firms and individuals  Key Result Area Indicators (12) – A mix of output and outcomes indicators that quantify specific program activities of CED12 The data is not reported separately by sector. The Mission may want to consider having the project provided supplemental information by sector to offer a more complete picture of project results. From the client interviews and sector overviews, it is clear that one sector may contribute overwhelmingly to a particular indicator, while a different sector may contribute overwhelmingly to a different indicator (or not at all) – but there is no way to know this from the data. For example, the “international certifications attained” indicators are made up almost entirely by hotels completing the Quality Mark program (six IT companies are in the process of obtaining IT-Mark/CMMI certification). None of the apparel or footwear companies are participating in any type of international certification programs (there are no such programs offered in Albania). On the other hand, the sheer size of garment/footwear manufacturing relative to other sectors means that practically all growth in revenues and employment come from those sectors. Just 1% growth of employment would be almost equal to 100% of the IT sector employment. The “Impact Level” indicators are the ones associated with the project‟s overall goals and objectives, and those are the ones that matter the most to USAID. These measure growth in revenues and jobs from the firms receiving assistance from CED. Specifically, these are: 1. Number of assisted enterprises that have increased sales of products and services 2. Percent change in total sales of assisted enterprises in targeted municipalities 3. Value of sales of Albanian products and services by assisted enterprises 4. Number of jobs created in assisted enterprises However, there are some very basic data problems with these indicators:  Indicator #2 makes reference to sales in targeted municipalities. The reference to municipalities is not at all relevant to the work of the project, is not part of any of the identification or selection criteria, and should be dropped (it appears to be a carry￾over from earlier EG portfolio considerations).  The project attempts to report data based on full-year program activity, even though firms enter (and occasionally leave) at various times during the program year. To avoid data distortions, the project only reports sales and employees for client firms 12 The distinction between “Intermediate Results Indicators” and “Key Result Area Indicators” is not clear, and in one case there is an exact duplication of the reported result. CED – Mid-Term Project Evaluation Dec 2011 17 only after one full year of program participation. As a result, the revenue and job data reported in the FY 2011 annual report is based only on submissions by 37 companies. In FY 2012 the project expects to report the data on 124 companies. Consequently, the value of sales and jobs reported in 2012 will be substantially above the 2011 levels related simply from the increase in the numbers of clients, not based on any actual project activity. The project will restate its baseline, and will recalculate the FY 2012 change based on the total sales and jobs that were present in the 124 companies at the end of 2011.  The FY 2012 Annual Report will accurately report changes in revenues and jobs for the 124 client companies during that year. However, the data will give no indication whether the changes have anything to do with actual program activity. The project makes no claims about whether changes in revenues and jobs are in any way related to program activity – it simply reports out the data at two different points in time. Because of this, there is no way to judge whether the assistance provided by the project has any beneficial impact on these key impact-level indicators. As the project moves away from firm level assistance, this indicator becomes harder to track. However, our recommendation is not to abandon firm-level altogether, but to continue with a core, catalytic group of companies. The mission seemed to accept this as well. I think having one stable sample of companies that can be tracked for the life-of￾project will actually provide some interesting insights about the growth of those companies - and to the extent that they can be seen as a proxy for the industry, gives insight into the growth overall. In reality, this is not a random sample. The firms that continue to receive assistance are generally the most progressive, growth oriented companies, so the sample is skewed that way. Still, I favor allowing CED to continue with firm-level assistance, alongside other industry-level assistance, for this group of 30 - 40 companies, and that CED would continue tracking revenue and job growth with that sample for the remainder of the project. This does not solve the issue of attribution/cause-and-effect. But at least there will be a consistent 3 - 4 year track of growth within this specific sub-set of the industry, and that would be useful information to have. The interviews with client firms do generally express favorable opinions about the quality and benefits of the assistance being provided from the project. However:  A majority of clients interviewed in the garment and footwear companies give the highest marks to the “productivity” improvements and are less certain about the direct contributions to revenue and employment growth. This is not entirely surprising. Productivity gains are more immediate, while actual sales growth will lag by a year or more. Most of the firms do believe they will eventually gain sales from CED‟s efforts but with data being collected and reported in the current manner, these gains cannot be captured and/or isolated. Having the revenue and job-growth data on the companies referenced above allows for an easy calculation of productivity improvements. The definition of productivity is output per unit of labor. A slightly simpler, perhaps not quite as precise measure of productivity would be revenue per employee. And limiting this calculation to the firms that receive assistance does allow for a reasonable assertion that some degree of the positive change is attributable to project assistance. Again, maybe not scientifically precise, but a reasonable indicator.  There is no direct correlation between revenue growth and job growth. In fact, since most of the assistance so far has been targeted toward productivity improvements, this makes it possible for firms to increase output and revenues without adding new jobs. CED – Mid-Term Project Evaluation Dec 2011 18 Almost certainly these productivity improvements have helped the companies maintain competitiveness in the face of difficult Italian markets – probably contributing to at least the stabilization of revenues if not outright growth.  As the project shifts away from firm-level, some of the other indicators become less relevant. Most of the Group 2 indicators (2.1-2.4) will become static over time. If the project stays with a fixed number of companies receiving firm-level assistance, the number never changes. On the other hand, to the extent that the focus of the project shifts so that the project works with intermediary institutions and then those institutions work with firms, these still could be relevant indicators. If the tourism association continues provide training that certifies quality-mark hotels, the number of firms assisted will continue to grow. The role of the project shifts from direct assistance to a strengthening of TA providers. The indicators are still relevant - just a different matter regarding data collection and reporting. There are 10 “Project Intermediate Results Indicators” and 12 “Key Result Area Indicators”. It is not always clear what distinguishes one group from the other. These indicators are grouped into the three components (elements) of the project. In the “Trade and Investment Capacity Strengthened” component, the Intermediate Results” indicators do attempt to get at a beneficial outcome – quantifying the number of enterprises that are making use of the skills that have been imparted during various trainings. The “Key Result Area” indicators within this component are simply counting the number of training events and participants (businesses and individuals) that attended these events and it is simply assumed that the attendees actually make use of the information. In only one instance: “1.2 Number of assisted enterprises that obtain certification with international quality control, environmental and other process voluntary standards or regulations” is there an independent test of the application of the knowledge. For this, there is a third-party certifier that verifies that the knowledge is being applied. In the case of the other two components, there is no clear factor that distinguishes “Project Intermediate Results Indicators” from “Key Result Area Indicators.” Within the “Private Sector Productivity” component, 7 of the 8 indicators simply count numbers of enterprises or individuals participating in various training events, or the numbers of events themselves. One of the indicators is stated slightly differently: “2.3 Number of enterprises able to reduce production costs as a result of USG assistance”. However, the project simply assumes that every enterprise participating in trainings about production cost improvements are, in fact, able to reduce production costs. There is no verification of actual result of cost-savings. There is a good deal of double counting within the Productivity component. Typically a training workshop touches multiple indicators. For example, a single training about using technology to improve management and reduce production costs could be scored under 7 of the 8 indicators. Within the third component (Workforce Development), the project conducts various job￾readiness workshops – helping people prepare either for entering the workforce, or to improve employability. The most significant indicator in this group should be “3.1 Number of people gaining employment or better employment as a result of participation in USG￾supported workforce development programs.” However, the methodology used by the project simply assumes that any person receiving any kind of training (a total of 740 individuals in 2011) or technical assistance from the project by definition has improved employment prospects. There is no verification of whether a person actually did gain or CED – Mid-Term Project Evaluation Dec 2011 19 improve employment (make more money) – only that their employment would be “better” because of added knowledge. The project contract establishes baselines and targets that seem arbitrary, and in many cases rather low. The implementer reports its actual achievements as a percentage of its target. Out of its 26 indicators, the implementer reports achievement of results of more than 200% of the target in 11 of the indicators, and between 100% and 200% in another 13 indicators. The implementer reports that its results fell short of target on only 2 of its indicators. In 7 of the 26 instances, the implementer reports achievements of greater than 300% of the target. But it is not clear from these results whether the apparent success came from exceptionally strong implementation, or simply low targets that were easy to beat. The greatest overage occurs with indicator 3.1.2 “Number of persons participating in USG￾supported pre-employment vocational and technical training”. The 2011 target was 150 participants, and the reported result is 740 – a 493% result over the target. Indicator 3.3 is stated as “Number of persons participating in USG-supported workforce development programs” with a 2011 target of 300. The reported result is also 740, which in this case represents an achievement of 247% of target. It appears that the same result is being reported for two different indicators, but against a different target. Consequently the same situation reported two different ways produces in one case a 493% overage and in a different case a 247% overage. What was the relationship between “participation” vs. “completion” and the ability to obtain new or better employment? If, as Indicator 3.1 suggests, 293 people obtained jobs after completing the workforce program, this represents almost 75% success rate – and certainly would point to something for future continuation and replication. But unfortunately, as noted previously, there is no identifiable relationship between participation in a project-supported activity and improved job prospects, and moreover, the project simply assumes that any trained individual has automatically enhanced the job prospects. A similar problem can be seen with indicator 2.4 “Number of enterprises receiving USG assistance to access bank loans or private equity”. The FY 2011 target was 15, and the reported result was 49, giving a result 327% of the target. However, the interviews suggest that few if any businesses are actually able to obtain bank financing, so the fact that companies received training on that topic is of little consequence. Engaging with Start-ups The project has engaged with only two start-ups, and the growth in revenues and jobs by these two companies is not substantial. A further emphasis on start-ups would not be recommended particularly in light of the CDCS-mandated shift away from firm-level assistance. It should be noted that the ICT Resource Center quite likely will provide some incubation capabilities for new IT companies. Engaging with Existing Enterprises While the evaluation did not conduct a one-by-one verification of project participation, according to an audit of the project database, it does appear that the annual report indicating 74 enterprises engaged does match to verifiable source documents. The significant, intensive firm-level assistance is conducted within the framework of Client Growth Plans (CGP), which are agreements signed between the project and the individual clients. These CGP‟s spell out the nature of the intended assistance, and require the clients to report quarterly to the CED – Mid-Term Project Evaluation Dec 2011 20 project. The project maintains databases of the client reports, registers of participants in the various project trainings, and other similar back-up documents on project participation. The project can, and will, accurately report changes in revenues and employment within the client firms. However, as noted previously, there is no clear way to relate these changes to any project activities. That‟s not to say that the project activities are without benefit. As also noted previously, client interviews clearly cite the value of assistance, most notably in productivity and quality improvements. Some of the assistance certainly will translate into sales and employment gains in the future. However, the current indicator set cannot capture the exact cause-and-effect of assistance and sales gains. Impact on Productivity, Diversification, and Job Creation Client interviews clearly point to improvements in productivity as a direct result of project activity, although there is no specific indicator that accurately captures this change. They also noted that CED assistance could help them both diversify target markets and increase number of employees but this data has yet to be captured. If the project maintains a data-collection relationship with a sufficient number of the CGP firms (currently 124) for the life of project, it is likely that some of this kind of result data can be extracted from the reports. Having 3 or 4 years of these reports will allow for a clear track of the changes in revenues and jobs within the client firms. Comparing those growth paths to the specific types of assistance that was provided (market outreach, productivity, etc.) will at least allow for some cross-relationships to be tested. However, as with every kind of measure of this nature, full cause-and-effect attribution will be impossible to prove or assert. CED – Mid-Term Project Evaluation Dec 2011 21 VI. Challenges and Lessons Learned As with any development project, CED has come across its fair share of challenges but appears to have taken steps to learn from their mistakes and successfully implement improvements that have helped create the very good reputation of the project and USAID. We will highlight several of the key challenges encountered by the project below but discuss future implications of these (if there are any) in the Recommendations section. As noted earlier, the project began with a very uncharacteristically slow start missing most of its first year targets. However, a full turn-around can be credited to the leadership of Chief of Party Victor Luboyeski and his skilled staff that has since gained the trust and respect of most the businesses and associations the evaluators interviewed. Additionally, as highlighted in CED‟s second annual report, great efforts were made trying to merge the two garment and footwear associations to no avail. Although a working group has been created instead, it has yet to be seen whether this new entity will be effective despite the extensive amount of time and resources invested in this merger. Lastly, little measureable progress seems to have been made in helping firms actually receive new forms of capital (also not an explicit deliverable for the project). A clear majority of all firms interviewed by the team reinvest their own money into building up their business. A final note should be made that throughout the life of this project both Europe and global capital markets have been experiencing a recession that has clearly challenged the potential for increased exports and market expansion forcing many smaller, less stable businesses to close in Albania. But at the same time each of the sectors targeted have also experienced growth due to supply disruptions in other parts of the world and a steady currency offering good value to both travel and investing in Albania. Maximizing Impact for SME Competitiveness Although CED appears to have a very comprehensive mixture of activities supporting increased investment/trade, enhanced productivity and improved workforce in each of the four sectors, certain interventions were noted repetitively throughout the interviews. In the garment and footwear industry, the introduction of Key Performance Indicators delivered through international STTA clearly has made the most immediate impact on the operations of the companies we visited. And although reducing production costs, minimalizing defects and waste, increasing efficiency and outputs are not immediately realized at the sales and jobs indictors, clearly these improvements are critical to remain competitive in an industry with razor thin margins and minimal competitive advantages. According to OMEGA (women owned garment manufacturer in Tirana), the one-on-one direct technical assistance combined with their participation of 4 workshops has helped them reorganize their production lines more efficiently and introduce an incentive program encouraging increasing quality and quantity of their garments. The evaluators also commend CED‟s effort to transfer the expertise provided by this STTA to local faculty at PUT to be able to scale this type of assistance at a much more cost effective and accessible level. In the tourism sector, the creation of the Authentic Albanian Quality Mark clearly has the greatest potential to deliver long-term, sustainable impact for the industry as a whole. From ATA to TAM-BAS and GIZ to all the hotels interviewed, each organization unanimously agreed that the introduction of an accredited standards initiative is critical to not only elevate the quality of the services and tourism products but to raise awareness and promote Albania as a destination to both local and foreign tourists. Even though only 20 hotels have yet CED – Mid-Term Project Evaluation Dec 2011 22 received an AAQM rating, approx. 60 are in the cue and through the support of ATA and the launch of the official website, the evaluation team believes that this initiative has the greatest chance to capitalize and accelerate the growth in this critical sector. Within the ICT sector, the support of SPI-IT Mark as well as the efforts made to establish the ICT Training and Resource Center both received strong marks from participating companies and associations. DM Consulting in Tirana mentioned that through this certification not only has his businesses improved their service delivery to clients, but they have received three new international contracts as a result of this accreditation. Not to mention they are also able to transfer this level of expertise to other IT companies increasing the competitiveness of the industry as a whole. Similarly, the American Albanian Development Foundation has agreed to enter into a PPP agreement with CED/USAID to operate the ICT Center as a way to reduce the gap between ICT training and the skills demanded by business while serving immediately as „legacy‟ organization for the investments and efforts already made by CED in this sector. Value-added Practices Provided by CED CED‟s approach to engaging with businesses in each sector appears to follow a fairly strategic and straightforward process of collecting the necessary information to then propose the most effective assistance the staff can offer. Many of the associations confirmed that CED had originally contacted them to get their member lists and advice on the different types of companies they might be able to engage with. The Client Growth Profiles created by CED then appear to have served as an effective tool to capture the necessary and important information on a businesses‟ operations that not only gathers data for reporting purposes but more importantly helps the company and CED jointly identify the right interventions to increase sales and jobs. Bella Confex undergarment producer in Shkodra credited this introduction/orientation process with them being able to receive the right technical assistance in the right order at the right time. As CED has continued to add new CGP companies to their portfolio, the evaluators believe that the business constraints identified through this process have dictated the type of skills and staff the project has in turn hired to work in these areas (and the type of STTA that needs to be brought in to assist further). While CED‟s value-add of how they delivered their technical assistance to businesses in the garment, footwear and ICT sectors seems effective, it was not as warmly received by some associations and several tour operators. Both Albanian Experience and Albanian Holidays felt that the project staff needed to be more proactive in learning from their many years experience in the industry before determining the best assistance to provide. Additionally they suggested the project be more flexible in adapting other models or lessons learned to fit both the timing and the situation that their industry is facing in Albania. Regarding the associations, this was not too much of a surprise to the evaluation team since the project was designed to primarily offer firm-level assistance and only secondarily to engage at the mesa levels. This said there still appeared to be some variance of opinions among some associations that greatly complemented all the assistance they have received (AITA, ATA) to others that indicated they would like much deeper support (NCGFM, CFA). Collaboration with Other Donor Activities Specifically, the evaluation team met with SNV, GIZ, AmCham, TAM-BAS, AADF, and USAID‟s AAC project to get a representative sample of the current levels of coordination in the four sectors CED has targeted. Generally, the project was given very high remarks from each of these donors who described CED‟s level of assistance as technically competent and delivered professionally. Floreta Lul-Faber, Executive Director of AmCham described CED‟s CED – Mid-Term Project Evaluation Dec 2011 23 approach to be very proactive, easy to work with and successful in identifying key local organizations to partner with in order to ensure sustainability of the initiatives the project has invested in. AADF President Ian Schneider said the ICT Training Resource Center never would have come together without the leadership of CED and the creative and inclusive approach they used to establish a PPP. Since GIZ works in each of the sectors CED has targeted, they felt that the project staff works together very well at the donor coordination level and that their projects activities have been designed well to complement the work GIZ has and continues to do in these areas. TAM-BAS described the successful integration of using their consultants to deliver some of the technical assistance needed by CED companies as well as coordinating on sponsoring series of roundtables on tourism that they hope to increase in the future. Innovative Development Solutions Although CED has engaged in several PPP‟s and PPD‟s, the evaluation team still believes there are more possibilities to be explored in this area in order to maximize leverage from other interested partners working in the same sectors. All of CED‟s activities in this area are still relatively new so it is impossible to measure impact or sustainability yet from these initiatives. The signing of the ICT Training Resource Center MOU was a large milestone in the process of launching a partnership that has been many years in the making. Hopefully much of the delay from USAID in supporting this partnership has been addressed so that CED can continue to move ahead in the building renovation and operationalization of this much needed training center for the ICT industry (and its support of all other industries). The PPD working groups established around making revisions to the tourism law and establishing an open forum platform around the garment and footwear stakeholders are good examples at the project‟s attempt to leverage the influence, talent and resources of numerous organizations involved in these issues. Unfortunately, these types of multi-party volunteer networks often get stalled in more theoretical discussions and don‟t have proper guidance to translate words into actions. Even though efforts to revise the tourism law got quite close to being approved, with a change in the government and certain stakeholders not feeling their interests have been accommodated, CED should make sure to re-engage to see this across the finish line. Similarly, concrete action plans and accountability need to be incorporated into GFSWG in order for all participants to see any benefit both in the short and long-term. In the future, the evaluation team sees several possibilities to explore future partnerships that would accelerate the impact of CED. Two such areas to be discussed in further detail in the Recommendations section include PPP‟s in establishing VET-type schools or enhanced curriculums in existing schools to reduce the workforce skills gap primarily in the garment and footwear sector. In the ICT sector numerous interviews revealed the need for securing Intellectual Property Rights that have been done successful by USAID through partnerships in numerous other countries. Lastly there remain other possibilities to expand the DCA program as well as look at integration of social capital markets as possible incentives to encourage further investment in Albania. The only caution to engaging in more innovative partnership-type solutions to address the projects‟ development objectives is that these activities take resources and time and with really only 2 years left on the project, starting such initiatives from scratch may be risky. Leveraging the expertise of organizations such as AADF which have virtually built their enterprise model around partnerships would be good partners to explore new possibilities in areas of mutual interest. CED – Mid-Term Project Evaluation Dec 2011 24 VII. Practical Recommendations The main headline for this evaluation is that USAID/Albania should continue to implement the CED project. Evaluators believe that the following recommendations shifting some of the activities of the project are still in compliance with the original contract Section C.B (3) Market-led Private Sector-Owned Approach; Section C.B (6) Enabling Competitiveness; and Section C.B (7) Building Local Capacity and Ownership for Sustainability. But in order for the evaluation team to recommend future improvements to CED, we must first re-frame the context within which these suggestions are being offered. In November 2011 USAID/Albania issued its new five-year Country Development Cooperation Strategy. Certain aspects of CED are now out of compliance with the new CDCS. Specifically, page 29 of the CDCS states: “The current EG portfolio will focus at the reform and regulatory level and less on interventions at the farm/firm level. This will require altering or ending some current activities; for example, the Mission will address the agriculture sector through the prism of agro-business enabling reforms and regulations. And all aspects of the Mission’s Economic Growth program will aim to achieve transformational reform and work to remove systemic constraints to private sector-led growth.” EG/Competitiveness projects carried out by USAID Missions around the world tend to work at one or more of three possible levels within the economy, and one factor that distinguishes projects one from the other is often the degree to which they concentrate on the different levels. Firm-Level Assistance – Economic growth and job creation take place within firms. Particularly in countries like Albania where people were isolated from world markets for 50 years, basic business knowledge is often quite deficient and needs to be brought up to world￾standards for firms and industries to compete globally. This can be done by supplying international experts to work with businesses to raise the knowledge level. Industry/Mesa-level Assistance – Mesa-level assistance can take place in three different forms: 1) Trainings and other interventions that engage numbers of businesses at the same time so as to achieve economies of scale; 2) Development/transfer of capacities to local entities that can then deliver assistance more cost-effectively than international experts; 3) Assistance that broadly benefits an industry or segment of the business community but does not directly engage individual businesses. Business Environment/Macro Improvements – The CDCS notes that Albania ranks poorly in World Bank and World Economic Forum indices. Improvements to the overall business environment (e.g. – tax policy, regulatory reform, education, infrastructure, trade agreements, etc.) will benefit all businesses (although not always every business equally). Although there are numerous specific examples of beneficial firm-level assistance provided by CED, conditions necessitate a shift in emphasis away from those activities. Nevertheless, as will be detailed here, and consistent with the description of Industry-level assistance CED – Mid-Term Project Evaluation Dec 2011 25 above, there are numerous ways to continue the beneficial activities of CED, by strengthening and working through intermediaries and partners in Albania. Proposed Project Modifications As referenced throughout this report and consistent with the new CDCS, the evaluation team confirms through its almost 40 stakeholder interviews that CED should shift its focus away from primarily providing one-on-one firm level assistance towards more mesa level support. This would include halting any specific outreach and assistance efforts to start-up companies as well as reducing the target number of individual companies CED is delivering firm-level assistance to. However the evaluators believe that some degree of firm-level assistance should be maintained but only to key companies within each sector who could provide the greatest multiplier effect from this continued technical assistance (factoring in their supply chain, sub￾contractors, willingness and potential to enter new markets, etc.). In order to most effectively make this transition, CED should evaluate key associations (most they are already working with) and NGO‟s best positioned in each sector to support and expand upon the momentum gained with firms but now be able to address the business constraints at broader reaching level. CED should only engage at the macro level in areas they have already engaged or feel that tangible results could be measured within the life of the project (and in partnership with others for sustainability). Specifically in tourism:  CED could re-engage with the working group established to propose a new Tourism law and to ensure that all the outstanding stakeholder input is taken into consideration and that the law is enacted.  CED could engage with the National Tourism Agency (and other public/private stakeholders) to develop a realistic action plan to implement the National Tourism Strategy. This would most likely need to include an overall market study to understand the key constraints and opportunities. Other specific activities could include leveraging assets from the private sector to help promote and “brand” Albania to foreign and regional markets, and establishing the necessary certification and licensing requirement to elevate overall industry standards.  Working with ATA and NTA, CED could help link national-level tourist activities with programs/support done at the municipal level (potentially working with Destination Management Organizations).  Regional support could accelerate tourism development by linking CED to more RCI activities and to sponsor a regional public agency convening with NTA to share and coordinate strategies (and resources) to benefit the entire Balkans. In the garment and footwear sectors:  CED could work with key associations/Ministry and potentially AIDA to develop a „Made in Albania‟ campaign educating buyers on the advantages of manufacturing garment and footwear in Albania.  Initiate better coordination with METE with quarterly update meetings and joint participation in steering committees. CED – Mid-Term Project Evaluation Dec 2011 26 And across all sectors:  There are inconsistencies and imbalances in the application and collection of various taxes (VAT primarily) that disadvantage the businesses, e.g., Garment/footwear producers are being charged VAT on raw material that is then exported for sale without the ability to recoup this taxes; for tourism, Albania‟s hotel tax is higher than in neighboring countries, creating an unfair disadvantage against Macedonia; and IT sales of computers and other IT equipment compete against most products being sold on the black market for no taxes. Advantages of Shifting Project Focus It almost goes without saying the advantage of shifting resources and activities according to this sections‟ recommendations is to better align CED with the new CDCS and to then to realign the project to maximize its impact within its remaining two and a half years. As far as the evaluators are concerned, it is no longer an option to continue with „business as usual‟ but time for CED to capitalize on its assistance provided thus far and to take it to the next level. With regard to the grants program, the guidance manual and evaluation criteria appear sound and in line with the project‟s original goals. Although only 7 grants have been approved YTD, the companies we interviewed who have received funding have directly linked this assistance to an increase in number of jobs, enhanced productivity and eventually sales. The use of the grant also seems to have encouraged a more fundamental shift in how these businesses intend to grow their operations in the future. However, we recommend that the target number of 50 grants in the contract be reduced, and greater flexibility offered for both size and eligibility. With current resource constraints, the administrative costs of identifying and processing 43 additional grants are too great. As noted early, the evaluators support the project‟s request to the Mission to no longer target recycling as one of its primary sectors of engagement. It is up to the project to determine if the companies already assisted in this industry meet the criteria they will establish on whether these businesses represent key „multiplier‟ firms they choose to continue to support. As a much lower priority area, the Mission and project may want to consider working with AmCham to develop the private sectors‟ understanding and investment in Corporate Social Responsibility (CSR). This has already surfaced as an interest from the overall business community with the Chamber and the project could potentially link this to its other direct assistance provided to associations. Revisions to Expected Results/Indicators Both the changes in the project priorities and some of the identified deficiencies in the current indicator set suggest a restructuring of the indicators and targets. However, to maintain consistency with the current contract and overall project goals and objectives, a radical restructuring of the indicator set is not recommended or warranted. Almost all indicators have some deficiencies – it is almost impossible to determine a perfect cause-and-effect relationship and results typically require long time frames to be realized. However, a group of indicators taken together can tell a reasonably good story about the effectiveness of a project. CED – Mid-Term Project Evaluation Dec 2011 27 Impact Level Indicators As noted previously, the most important indicators are the Impact-level indicators that attempt to quantify increases in revenue and employment for firms receiving direct assistance. The current set is deficient in two respects: 1) there is no attempt to relate the assistance provided to the outcomes that occur; and 2) changes in revenues and jobs, if they result at all from the assistance, will only happen with some significant time lags. These indicators will become even more problematic in the future because as the project shifts away from firm-level assistance, fewer firms will be engaged directly by the project. Nevertheless, it is recommended that these impact level indicators remain, but with some modifications:  The project will remain engaged directly with a smaller subset number of firms, and the changes in revenues and jobs for these firms should be reported.  The project will shift some resources from firm-level assistance to assistance provided by intermediaries that the project works to strengthen. The project should track the changes in revenues and jobs in those firms, although separated so that it is clear whether the firms are being assisted directly or through intermediaries.  The project‟s narrative reports should attempt to provide greater insights into the specific nature of assistance provided, and to what extent it might be possible to demonstrate cause-and-effect. The Current Intermediate Results and Key Area Results Indicators As noted previously, most of these indicators simply count the numbers of trainings offered, number of participating companies, and numbers of participating individuals. This quantification is useful at least as a gauge of the level of project activity. However, more effort should be made – at least in the project narratives if not in the indicators themselves – to show the qualitative outcomes of the work.  In the Workforce Development component, more effort should be put into determining just how effective the various trainings are in helping individuals obtain new employment or better employment. This can be done by surveying a sample of participants six to twelve months after participating in a job-readiness program.  Within the Productivity component, it should be possible to quantify specific productivity gains for those companies participating in the KPI/Productivity Enhancement initiatives. Possible New/Modified Indicators New indicators will be needed to track new or substantially modified activities:  As the project shifts to higher-level activities, specifically the strengthening of intermediaries, associations, and other local service providers, indicators should be added to measure factors such as clients served by these intermediaries, the effectiveness of the assistance to those clients, and trends toward sustainability of those intermediaries.  Workforce Development will be strengthened/expanded using internship programs between the PUT and garment/footwear firms (and any other training institutes assisted). Indicators both quantifying the numbers of interns engaged and the impact of those internships on both the individuals and the firms should be added. CED – Mid-Term Project Evaluation Dec 2011 28 Specifically, the project should look at the wage differentials received by these interns upon securing permanent employment, and the secondary effects these interns have on the productivity and expansion prospects for the recipient firms.  One of the two double-counted indicators in Workforce Development could be dropped (either 3.3 or 3.1.2).  The project contemplates making a large grant from its LIF to the Innovation/Technology center. A suitable indicator for tracking success of that grant should be added.  There are currently no indicators tracking the impacts of grants. The project needs to be reporting the numbers and impacts of the grants that are awarded.  It is possible that the project will engage in a limited number of policy reform efforts (e.g. – modifications to VAT rules, international trade regulations regarding the targeted industries, the Tourism Law). An indicator that quantifies engagements between high-level government officials and the project‟s beneficiaries (associations, etc.) would be appropriate. Revisions to Project Components Workforce Development Some of the Workforce Development activities of the project are simply the flip side of productivity enhancement. If workers are trained and become more productive, that is addressing both productivity enhancement and workforce development. It is redundant to count these actions twice. However, a particularly promising area for Workforce Development would be expansion of the higher-level technical trainings for the garment and footwear industries carried out within the Polytechnic University (PUT), and linked to specific internships within companies. The early results suggest that these initiatives could be greatly enhanced and as a result both offer substantially higher-level employment for students and much-needed labor skills within the industry. Although not fully tested, similar arrangements may be possible for both the tourism and IT industries. University programs exist in both of these industry areas, and opportunities for similar kinds of linkages should be explored. ICT Training and Resource Center The project has agreed to use a portion of its Local Incentive Funds (LIF) to co-fund the development of a technology center for the ICT industry in Albania. Because of certain regulations regarding the use of project grant-funds, CED may have to maintain something of an arms-length relationship from the management of this center. If that is the case, other intermediaries working with the project (IT Association, local training firms, etc) should be assisted to engage with the center. This center could become an important training and incubation site for IT professionals, and could help feed personnel to the local IT firms. One concern expressed by the IT Association is that this center should not become a competitor to existing training organizations and other services providers. The Project should do what it can to assure that this does not happen. CED – Mid-Term Project Evaluation Dec 2011 29 Standards and Certifications The project has had some early successes promoting quality standards and certifications in both the Tourism and IT industry. The Authentic Albania Quality Mark (AAQM) for hotels and guesthouses has been awarded to 20 hotels, with the goal of expanding to 100. Up to now, the project has paid for most of the costs associated with training and certifying the hotels. The “ownership” and management of the AAQM needs to be fully transitioned to the Albania Tourism Association. Ideally, members will pay fees that will sustain the ATA and allow for its maintenance and expansion. As the numbers hit a critical mass, external identification and marketing efforts could expand its visibility and effectiveness. In the IT sectors, two certification initiatives have been carried out in partnership/co-funding with the E&E Bureau‟s Regional Competitiveness Initiative (RCI). These are IT-Mark certification for software firms and Project Management Professional (PMP) certifications. These appear to be successful and well received. These should be continued, and where possible, shift more of the training to local entities. Strengthening Local Associations As noted previously, both the Albania Tourism Association (ATA) and the Albania IT Association (AITA) have been active partners with CED – but both are still small and struggling to achieve sustainability. CED can help these associations develop new programs that would enhance their ability to secure more fee-paying members, or deliver fee￾generating services. In the case of tourism, two suggestions were raised by local tour operators: 1) to promote Tirana more for meetings, conventions, and expos; and 2) investigate the possibilities for cruise ships stopping at Albanian ports. In the case of IT, the suggestion has been to reach out to communications companies (telecoms, etc) to add the “C” in ICT. There are two associations working with the garment/‟fasson‟ industry. Last year an unsuccessful effort to merge the two consumed a great deal of time and resources. The project should maintain whatever positive relationships it can with both, but it does not appear that merging the two is worth the expenditure of additional resources. Addressing Overarching Business Constraints It is not recommended that the project engage broadly in trying to improve the overall business environment. However, there are a few initiatives that are more industry-specific that would be worthwhile exploring. Investigate Opportunities for Sourcing Raw Materials for Garment and Footwear Industry Some of the garment and footwear companies are interested in full-cycle production, which requires purchasing of raw materials (fabric, leather, rubber, etc). Although at one time Albania had a cotton textile industry, restoring it is probably not feasible. However, there may be possibilities for joint ventures with Turkish or Egyptian producers to establish warehouses and supply depots that would make it easier, quicker, and cheaper for local firms to source this material. CED – Mid-Term Project Evaluation Dec 2011 30 Along with this investigation, it will be necessary to look at the various trade rules about sources of materials and domestic content so as to avoid conflicts with buyers in the European Union. Access to Capital It is almost impossible for SME‟s to get financing on reasonable terms. The project did conduct some workshops for businesses on the topic, but the constraints are deeper than just knowledge. The project should look deeper into the core problems to see if there are any possible solutions. Using USAID‟s Development Credit Authority (DCA) guarantees may offer some possibilities. Intellectual Property Rights (IPR) A high percentage of software sold and used in Albania is unlicensed. The larger multi￾national software companies (Microsoft, etc) are interested in strengthening IPR. The project may consider partnering with some of these firms to work with the government on stronger regulations and enforcement. European Union Accession Ultimately Albania should be able to enter the European Union. Already visa regulations have been liberalized making it easier for Albanians to travel back and forth into European countries. This makes it possible for workers to go legally, and eases the flow of remittance funds back from Europe. Remittances can be an important source of capital. Diaspora returnees can contribute a great deal of business knowledge and market connections. The project should investigate possibilities for expanding in these areas. Considerations to Engage Agribusiness Sector There is concern within the Mission that when the Albania Agricultural Competitiveness (AAC) project ends in 2012, USAID/Albania will have no activities within this very important sector (agriculture and food-processing represent 50% of the GDP and livelihoods of the population). While recognizing the importance of agriculture to both Albania and USAID, there is no easy way to add agriculture and/or food-processing to the workload of CED. Here are some thoughts and suggestions on this subject:  Agricultural processing would be a closer fit to the current activities of CED, as compared to agricultural production. Consequently, any possible additions to the CED scope of work should be limited only to processing.  According to AAC, the associations and other business support organizations linked to the agribusiness sector (that CED would want to be targeting according to the CDCS strategy) are very unorganized providing little value to firms and the industry overall.  CED would need to earn „credibility‟ with firms or associations in this industry which would shorten an already shrinking window of just over two years to achieve results. Not to mention adding additional time to modify the contract (including additional resources) and adding new capabilities.  CED is willing to consider adding another sector, but has stated that they would need additional specialized personnel (and almost certainly would want additional funding). CED – Mid-Term Project Evaluation Dec 2011 31  Executive Director of ACIT (Competitiveness NG) concurred with evaluators‟ assessment of challenges to engage this industry at this level with this timeline.  There may be some possibility for a high-impact partnership with a multi-national investor that could generate some favorable returns, but nothing specific along these lines has been identified or contemplated. CED ALBANIA – Mid-Term Project Evaluation Dec 2011 32