MID-TERM EVALUATION OF THE KOSOVO PRIVATE ENTERPRISE PROGRAM (KPEP) Mid-Term Evaluation of KPEP ii Prepared for the United States Agency for International Development, USAID Contract Number RAN- I - 00- 09-00015, Task Order Number AID - 167-TO-00003, Mid-Term Evaluation of the Kosovo Private Enterprise Program (KPEP) Implemented by: Development & Training Services, Inc. (dTS) 4600 North Fairfax Drive, Suite 304 Arlington, VA 22203 Phone: +1 703-465-9388 Fax: +1 703-465-9344 www.onlinedts.com Mid-Term Evaluation of KPEP iii MID-TERM EVALUATION OF THE KOSOVO PRIVATE ENTERPRISE PROGRAM (KPEP) September 10, 2011 DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Mid-Term Evaluation of KPEP iv ACRONYMS1 ADI Association Development Index BEEP Business Enabling Environment Project () BSPs business services providers CBK Central Bank of Kosovo COP Chief of Party DCOP Deputy Chief of Party DG Democracy & Governance ERO Energy Regulatory Office EU European Union FTE full-time-equivalent FY Fiscal Year GDP Gross Domestic Product GoK Government of Kosovo HCN host country national ICT Information & Communications Technology IMCI International Management Consultants Institute IMF International Monetary Fund KBS Kosovo Business Support KCBS Kosovo Cluster and Business Services KEK Kosovo Energy Corporation KPEP Kosovo Private Enterprise Program KPST Kosovo Pension Savings Trust KTA Kosovo Trust Agency LOP Life of Project LTTA Long Term Technical Assistance MEM Ministry of Energy and Mines MFE Ministry of Finance and Economy NGO Non-governmental Organization NOA USAID New Opportunities in Agriculture PAK Privatization Agency of Kosovo SPS Sanitary and Phytosanitary Standards STTA Short Term Technical Assistance TAK Tax Administration of Kosovo TO Task Order UN United Nations USAID United States Agency for International Development USG United States Government WTO World Trade Organization 1 For the purpose of this report, all projects will be referred to by acronyms Mid-Term Evaluation of KPEP v CONTENTS EXECUTIVE SUMMARY ........................................................................................................ 1 I. INTRODUCTION ................................................................................................................ 5 II. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE ................................... 6 Problem Statement ........................................................................................................................................ 6 USAID’s Intervention in Response to the Problem Statement ............................................................. 6 III. PURPOSE OF THE EVALUATION ................................................................................ 8 IV. RESEARCH DESIGN AND EVALUATION .................................................................... 9 V. FINDINGS ......................................................................................................................... 10 Management and implementation of the program. ................................................................................ 10 Outcome and results findings about the project’s achievements and consequences ........................ 11 Component 1. Increased Competitiveness of Key Sectors ................................................................... 12 Component 2. Demand-driven development of business support services ...................................... 18 Component 3: Business Enabling Environment .................................................................................... 19 Component 4: Workforce Development ................................................................................................. 21 VI. CONCLUSIONS AND LESSONS LEARNED ............................................................... 22 Project Effectiveness ................................................................................................................................... 22 Project Efficiency ......................................................................................................................................... 22 Project Impact .............................................................................................................................................. 23 Project Sustainability ................................................................................................................................... 23 Project Relevance ......................................................................................................................................... 23 Responses to Subordinate Questions ....................................................................................................... 24 VII. RECOMMENDATIONS ................................................................................................ 28 VIII. APPENDICES ................................................................................................................ 30 Appendix A. Evaluation Scope of Work ................................................................................................. 31 Appendix B. KPEP Project Monitoring Methodology ......................................................................... 38 Appendix C. Persons Interviewed ............................................................................................................. 41 Appendix D. Quantitative Project Indicators, with results through June 30, 2011 .......................... 43 Appendix E. Results of Survey of Grant Recipients .............................................................................. 56 Appendix F. References .............................................................................................................................. 57 FIGURES AND TABLES Table 1. KPEP Cumulative expenditures by component ..................................................................... 11 Table 2. KPEP Results* through Q3 FY 2011 ....................................................................................... 16 Table 3. Ranking of KPEP Sub-sectors ................................................................................................... 17 Mid-Term Evaluation of KPEP 1 EXECUTIVE SUMMARY The USAID Kosovo Private Enterprise Program (KPEP) is being implemented satisfactorily, and should exceed or achieve the quantitative goals for sales and employment established at the outset of the project. The project goal for sales has already been achieved with 15 months remaining in the project. Export sales (a goal adopted only later during implementation) have also exceeded planned targets. Employment growth has been slower, but project management expects that this goal will also be achieved by the end of the project. KPEP was a follow-on to two previous USAID projects, KBS and KCBS, which both worked in the same area, and with some of the same clients. Despite this, the project had a slow start. The initial COP was replaced after a few months, followed by an interim COP for 18 months. The current COP has been in place since August 2010. Initially, the project had four areas of activity, or components:  Private sector development of supported sectors,  Demand-driven development of business support services, including promotion of associations in supported sectors,  Improvements in the business-enabling environment, and  A cross-cutting focus on workforce development, aimed at employment creation. Project funding was $17.3 million, of which $3.76 million was allocated for a strategic activities fund that provided grants and sub-contracts. (Another $2.3 million was provided by USAID for the project to promote employment in Serbian areas in Northern Kosovo, but that activity is outside the scope of this evaluation.) Initially, the project relied heavily on LTTA expatriate advisors in each of six areas, supported by Kosovar professionals. Over time, these expatriates have been replaced by Kosovar sector leaders, with only two expatriates (the COP and deputy) in place by the end of the second year of operation. The Kosovar staff in each area is judged by the evaluation team to be of very high quality and professionalism. Over time, the focus of the project has shifted mainly into the first component, private-sector development, supported by activities in the other three components. This appears to be an appropriate response to conditions on the ground in Kosovo. Progress under each component is as follows: COMPONENT 1: PRIVATE SECTOR DEVELOPMENT OF SUPPORTED SECTORS The Task Order for the project required that the implementer work in two areas: road construction and construction materials; and agriculture. The implementer was required to examine opportunities in these areas, along with suggestions for seven sub-sectors where opportunities were seen as potentially promising. The project worked initially with the construction sector, but later concluded that the size of projects, along with the issues relating to a single buyer (the Government of Kosovo) meant that any project influence would be marginal. The project implementers, in conjunction with USAID, ultimately have chosen to focus on 9 areas within these two broad guidelines:  Livestock products, particularly dairy and poultry  Fruits and vegetables  Non-wood forest products Mid-Term Evaluation of KPEP 2  Forest products  Wood processing  Construction materials  Information and communications technology  Tourism, and  Recycling Results from the project activities have varied with the opportunities in each sector.  In the dairy sector, impressive results have been achieved in increased production, higher quality, and exports.  Poultry, a recent addition, is also providing significant results.  In the fruits and vegetables sector, peppers are the outstanding success, with rapidly increasing production and exports.  Non-wood forest products, targeted in particular on EU markets, have had modest positive results, and have generated substantial employment.  The emphasis in forest products has been largely on certification of sustainable harvesting from protected forests and prevention of illegal logging. Both are important to the upcoming EU requirements for wood product imports.  Wood products, notably furniture, have gained traction in regional markets, and, modestly, into the EU (but they remain uncertified at present).  Construction materials production, notably bricks, plastics and metal products, has been successfully expanded into the domestic market, replacing imports.  Information and communications technology included promotion of call centers and business process outsourcing services targeted on the EU. Progress has been made, but the sector has yet to gain real dynamism. Nevertheless, the activity did not begin until year 2, and the current approach has been underway for less than a year, so judgment on its effectiveness should be deferred.  Tourism has made some progress from an extremely low base. The employment benefits from promoting this sector are only likely to emerge over the medium term, once policy and institutional constraints are addressed.  Recycling has made very modest progress. COMPONENT 2: BUSINESS SUPPORT SERVICES This component was meant to help build a number of private-sector firms (Business Services Providers) capable of delivering specialized services to private sector enterprises in Kosovo. In addition, it sought to strengthen business associations as vehicles for providing services to members in a sector, as well as the ability to lobby government to address problems facing the sector. In each case, the goal was to promote self￾sustaining enterprises. Mid-Term Evaluation of KPEP 3 Progress toward the goal of strengthening the business services sector has been very modest. In part, pursuing self-sustaining BSPs appears to have been premature because of the small size of most Kosovar businesses and their lack of willingness to pay for assistance from outside firms. The large number of donor￾financed sources of free or subsidized technical assistance surely exacerbated the challenge. More progress was made in a number of cases in pushing associations in the direction of greater service to their members and greater self sufficiency. Overall, there was a significant increase in the capabilities of assisted associations. Nevertheless, most of these associations are still far from achieving self-sufficiency so the project results should be regarded as modest. COMPONENT 3: BUSINESS ENABLING ENVIRONMENT This component was intended to support the businesses in Component 1 by improving the overall business environment in Kosovo. Various problems, including frequent changes in the national government, limited its effectiveness during the early period of project implementation. Moreover, the gap between policy issues relevant to the sectors in which KPEP was operating and national government policy was perhaps too wide to make for coherence with the rest of the project. USAID decided to implement a separate project, BEEP, to address these national issues, leaving KPEP to focus on narrower matters relevant to the project. KPEP’s notable success under this component was the elimination of the use of reference prices for the import of potatoes and milk into Albania. The Albanian customs shift to transaction pricing should allow Kosovo exports to develop more broadly. COMPONENT 4: WORKFORCE DEVELOPMENT This area included a wide variety of activities intended to develop the skills of Kosovo’s large population of unemployed people. Training, job fairs, internships, and promotion of certifications were among the tools used. This is the activity furthest at present from reaching the project’s targets. As noted above, project management expects that the goal will be achieved by end of project. To some extent, this component works at cross-purposes with Component 1, particularly in the short run. Major gains in productivity (i.e., output per worker) have occurred in many of the assisted firms. This means that employment in assisted firms has grown much more slowly than sales. This is an inherent part of increasing the competitiveness of such firms. Employment will grow faster to the extent that firms are able to continue to grow because of their increased capacity to meet both domestic and export demand. RECOMMENDATIONS We recommend that the project be extended two additional years, with some modifications. We offer three reasons for this recommendation. 1. We consider the KPEP project team at present to be excellent, poised to deliver increasingly valuable benefits during the remaining 14 months of the project, and we expect that project extension would provide even larger benefits. 2. A full and open competition for a successor project, in our opinion, would risk serious loss of momentum in the areas where the project is currently most successful. Uncertainties about future employment would be likely to lead to gradual erosion of the capabilities of project staff, as concern about continued employment would be likely to lead to loss of key staff. 3. The extension would align project completion with the USAID strategy for Kosovo, which covers 2010- 2014. Kosovo is still a very young country, and the environment for business is still evolving rapidly. Mid-Term Evaluation of KPEP 4 Designing a new project in the context of a new USAID strategy for 2014 would allow for better targeting of the most appropriate approaches and activities for the environment at that time. The modifications that we would propose for extending the project are as follows: 1. No grant component would be included in the extended project. 2. Instead, the project would hire a specialist in linking producers with commercial financing, in order to promote the faster development of ongoing relationships between clients and the financial sector. 3. The project would graduate the most successful enterprises, and reach out to a new generation of small and micro firms through technical support and training, provided increasingly through BSPs or business associations. 4. The number of expatriate staff would shrink from two to one by the end of the first year of the extension. 5. The project would shift much of its work in the fruit and vegetables sector to the USAID NOA project, leaving KPEP with a narrower focus. 6. Our current judgment is that a narrower focus would include livestock (dairy, poultry and possibly other targets of opportunity), non-wood forest products, certified wood processing and tourism. There would be a phase-out of construction materials, recycling and forestry, and possibly ICT. Nevertheless, much may happen during the last year of the project, particularly in the relatively new sector of ICT and some of the newer building materials products KPEP is beginning to support. Therefore, the sector choices should be revisited in light of developments during the next few months of implementation. Mid-Term Evaluation of KPEP 5 I. INTRODUCTION This mid-term evaluation of the Kosovo Private Enterprise Program (KPEP) was carried out by dTS online in July and August, 2011, during the final quarter of its third year of operation. The project, with a value of $17.3 million, was initiated in September, 2008, and is to be completed by September 2012. Mid-Term Evaluation of KPEP 6 II. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE PROBLEM STATEMENT Kosovo is a young country, having only come into existence as an independent nation in 1998 at the close of a war with Serbia that featured massive repression and genocide of Kosovars, followed by U.S. military intervention to liberate the nation. Even today, its existence is questioned by some other nations, greatly complicating trade relations and travel. The country’s transition to independence has proceeded on a forward path despite setbacks. Starting from virtually zero, Kosovo today is an established state, with a functioning government and a society that is growing stronger and more self-confident. The process of handover of Kosovo’s nascent institutions from UN control to the new Government of Kosovo (GoK) has been a notable success and has highlighted the young state’s growing maturity and capacity. Government institutions at the central level are establishing a track record of successful operations. The country has adopted a democratic Constitution, established a Constitutional Court, and has held consistently represented the people’s will through elections. There have been important foreign policy and economic successes as well, including the historic border demarcation agreement between Kosovo and Macedonia, recognition of statehood from 81countries to date, and the admission of Kosovo to the World Bank and International Monetary Fund (IMF). There are also significant hurdles in the international community that are still impeding Kosovo’s economic development, including lack of membership in the World Trade Organization (WTO). Kosovo is widely acknowledged as having one of the most open economies in the region. Economic growth rates steadily increased over the past ten years to 5.4 percent in 2008 before falling to about 3 percent growth in 2009 during a period of global financial crisis, when most regional economies experienced contractions, before recovering modestly to about 4 percent in 2010. In 2002 Kosovo adopted the euro as its official currency. Inflation has been relatively low and the country’s nascent economic institutions have maintained fiscal stability. Yet serious challenges remain. Unemployment is very high, particularly among Kosovo’s burgeoning youth population. Three years after independence, the regional environment continues to be difficult with Serbia still following a policy of obstruction toward its southern neighbor. Border conflicts with Serbia in August, 2010 have demonstrated the fragility of that border. Kosovo needs urgently to reduce unemployment. Imports are twice the level of exports, though the latter have been growing rapidly. Remittances from Kosovars working in Western Europe have so far helped fill the gap, along with large amounts of foreign aid inflows, but both are likely to decline over time. The country needs to offer more and better employment opportunities for its citizens in order to stem the outflow of its working population. USAID’S INTERVENTION IN RESPONSE TO THE PROBLEM STATEMENT USAID has played a central role in establishing economic institutions and systems in Kosovo. The basic institutions, laws, and regulations are in place, as is a sound financial system. Through U.S. technical assistance and partnership with other donors, much progress has been made in building the capacity of Mid-Term Evaluation of KPEP 7 central GoK institutions, such as the Ministry of Finance (MF) and the Ministry of Economic Development (MED), the Central Bank of Kosovo (CBK), the Tax Administration of Kosovo (TAK), the Privatization Agency of Kosovo, the Privatization Agency of Kosovo (PAK), the Kosovo Pension Savings Trust (KPST), Kosovo Energy Corporation (KEK), Energy Regulatory Office (ERO), and others. Several institutions, such as the CBK, the KPST, and the privatization agency, now function at a level of proficiency that does not require ongoing donor support. The country has continued to maintain fiscal and macroeconomic stability during a period of global financial crisis. Nonetheless, Kosovo is far from reaching its economic potential. Economic growth in 2009-2010 has averaged less than 4%. Kosovo has weathered the global crisis primarily because it remains isolated from the world economy and is largely dependent on remittances and foreign aid, with each contributing approximately 10% to GDP. But this is not a model for sustainable growth and, moreover, it is not sufficient to create jobs for the roughly 40% unemployed and the 30,000 new entrants to the labor market each year. Lack of reliable energy, lack of a conducive business enabling environment, poor infrastructure, and an unskilled workforce are among the major constraints facing Kosovo’s economy. Kosovo has a highly import￾dependent economy, with exports comprising less than 10% of GDP in 2008 – the lowest proportion of exports to GDP in any transition economy in Europe and Eurasia. Kosovo’s productivity in the agriculture sector declined severely as a result of the conflict with Serbia, during which processing plants and equipment were systematically destroyed, so that the country has shifted from a net agricultural producer to a net agricultural consumer. Reviving and building the agricultural potential of Kosovo with an export-led strategy is a. major development challenge. Kosovo needs to develop long-term, sustainable economic opportunities that can only be created by the private sector. There are four critical elements of private sector development in Kosovo – the enterprises that make up the private sector, the people that are needed to run the enterprises, the capital that is required to finance business operations, and the environment in which they all operate. These elements are the analytical focus of this evaluation. Mid-Term Evaluation of KPEP 8 III. PURPOSE OF THE EVALUATION This is a mid-term evaluation of a project scheduled to terminate on September 30, 2012. The terms of reference for the project required that it be assessed in five dimensions: effectiveness, efficiency, impact, sustainability, and relevance. In addition, USAID requested the evaluation team’s judgments on 11 subordinate questions. The team was also requested by the Mission to make recommendations on any “mid￾course corrections” for the remainder of the project. The work plan for the final year of the project was prepared before the evaluation team completed its assignment. In view of this, the team was requested to offer its judgments on appropriate follow-on activities following the end of the project, though this was not contemplated in the scope of work. Mid-Term Evaluation of KPEP 9 IV. RESEARCH DESIGN AND EVALUATION Methodology Members of the two-person evaluation team, assisted by an interpreter, were in Kosovo between July 15 and August 6, 2011. During that time, they reviewed KPEP quarterly and annual reports, as well as numerous other documents prepared under the project. They also interviewed the entire professional staff of KPEP, including the chief of party and his deputy. In addition to the interviews with KPEP staff, team members visited and interviewed 25 clients of the project, in person, usually on site, and 44 grant clients by telephone. Some interviewees were suggested by KPEP staff, but others were chosen at random. For the telephone interviews, the team attempted to interview all 150 grant recipients from KPEP, but was able to reach only 44. The team also interviewed USAID/Kosovo staff, and the implementers of several other USAID projects relevant to KPEP, as well as other donors operating in the same general area. They also interviewed other stakeholders in Kosovo (bankers, business associations, GoK representatives) to learn their views of the KPEP project. A list of all interviewees is presented in Annex B. A summary of the results of the survey of grant recipients is contained in Annex E. As noted above, the evaluation took place in late July and early August, 2011. In Kosovo, this is a period where many possible interlocutors were away on vacation. This selectivity of the persons interviewed could have led to some unintended bias. Similarly, the fact that fewer that one-third of KPEP grant recipients could be reached by phone during early August could have produced different results than interviews with a larger sample. In addition, no detailed comparison of the sales and exports reported by KPEP with individual reports by participating firms. This could have led to overstatements of results from the project. In general, however, we were impressed by the seriousness of KPEPs efforts to report sales and exports correctly. As to the other sources of potential bias, we consider the weaknesses in the evaluation methodology to be relatively minor. We would not expect significantly different results from a post-vacation￾period survey. Mid-Term Evaluation of KPEP 10 V. FINDINGS MANAGEMENT AND IMPLEMENTATION OF THE PROGRAM. QUALITY OF TOP MANAGEMENT OF THE PROGRAM We judge that the current top management of the project, the COP and deputy, are extremely well-qualified to lead the project. Both have substantial experience elsewhere, and both are extremely engaged in overseeing the project, and in motivating excellence in performance by the project’s local staff. . Interviews with other donors, other USAID projects, and other relevant institutions have been unanimous in praising the efforts of the project’s top management to cooperate and coordinate with others in achieving project goals, and more broadly, to promote the development of the Kosovar economy. In the judgment of the evaluation team, the praise for the current team by others far exceeds the norm for such projects. That said, it seems clear from our interviews that the original COP, though skilled in his primary area of expertise, lacked the skills needed for overall project management. He was replaced after six months by a second COP, who appears to have put the project back on track. After his departure for family reasons after 18 months, the current leadership team was brought in to replace him. Our assessment is that the leadership of the project went from unsatisfactory, to good, and then to outstanding. Since the arrival in mid-2010 of the current top management, coordination with other actors was regarded by interviewees as substantially improved. One knowledgeable observer stated that the original six expatriate advisors fielded earlier in the project were less willing to give responsibility to Kosovar project staff, limiting their effectiveness. QUALITY OF PROJECT STAFF The project staff is unusually well-qualified and committed, indeed often passionate, about their work. From client interviews, it is evident that project staff visits clients frequently, has gained their trust, and is regarded by clients as very knowledgeable and helpful. FOCUS ON PROJECT GOALS The current project management has focused project activities clearly on project goals. Their emphasis on exports has increased the project’s attention to quality and productivity. The team argues, correctly in our view, that pursuing export opportunities exposes client firms to quality and price requirements that are important drivers of competitiveness. The ability to meet export standards places firms in a stronger position to compete with imported products in Kosovo’s very open economy. Overall, we judge project staff to be very focused on achieving project goals. MONITORING AND EVALUATION OF THE PROGRAM The project uses an elaborate system for monitoring and evaluating the program. The approach, adapted from the earlier KCBS project, takes a conservative stance on attribution of firm-level results to project activities. Quarterly reports from the project provide clear reporting that allows USAID to understand clearly how the project is proceeding toward its goals. (The methodology for the monitoring and evaluation system is described in more detail in Annex C.) Mid-Term Evaluation of KPEP 11 OUTCOME AND RESULTS FINDINGS ABOUT THE PROJECT’S ACHIEVEMENTS AND CONSEQUENCES The KPEP project’s strategy is to emphasize a value-chain approach to promoting private sector development in key sectors. This emphasis is evident in Table 1 below, which presents KPEP’s relative expenditures on each of the four original components of the project through the end of Quarter 3 of FY 2011. Table 1. KPEP Cumulative expenditures by component As shown in the table, Component 1 accounted for two-thirds of total project expenditures through the end of the third quarter of FY2011, with the other three components accounting for the residual one-third. But this does not fully reflect KPEP’s strategic focus on a value-chain approach, because in fact the activities within Components 2 through 4 also have been geared toward the provision of support to KPEP’s key sectors. An example would be the support KPEP has provided to workforce training of heavy equipment operators in the road construction sector. Moreover, concurrent with the project’s re-organization and installation of the third Chief-of-Party, Components 2 and 3, dealing with development of business services providers and reform of the business enabling environment, were effectively de-emphasized, an indication that the original project objectives for these components might have been too ambitious and not well enough integrated with the project’s overall purpose. For example, KPEP frequently provided grants to core clients within its key sectors designed to achieve results targets within Component 1, relating to increased sales (including exports), investment and employment in those sectors. But the grants from KPEP, coupled with a prevalence of grant-making by entities funded by other donors (such as the European Commission and the World Bank) may have worked against the development of business services providers in Kosovo that would be self-sustaining based on fees they charged to the same client base. By the third year of KPEP activities, general USAID support for business enabling environment reform in Kosovo had been re-allocated to another project, the Business Enabling Environment Project (BEEP), with the result that KPEP’s policy and institutional reform activities were re-focused exclusively on impediments encountered within its key sectors. An example would be the imposition by Albanian customs authorities of reference pricing on imported potatoes and milk products from Kosovo, a non-tariff barrier imposed in contravention of Albania’s WTO’s commitments that nonetheless required high-level policy dialogue led by KPEP – on behalf of some of its core clients -- before being reversed. Cross‐ Direct Sectoral* TOTAL Component 1: Increased competitiveness of key sectors   44.5% 22.2% 66.7% Component 2:  Development of business support services 5.1% 2.5% 7.6% Component 3: Improved business enabling environment 7.2% 3.6% 10.8% Component 4: Workforce development 9.8% 4.9% 14.7% Source:  Derived from figures supplied by KPEP (cumula tive through Q3 FY2011)    * Alloca ted by component according to di rect s ha re of cumula tive expendi tures. Mid-Term Evaluation of KPEP 12 COMPONENT 1. INCREASED COMPETITIVENESS OF KEY SECTORS The project has had substantial impact on the economy of Kosovo, fostering innovations by a set of dynamic enterprises in key sectors with increased capacity to compete in domestic, regional and European markets. These innovations have increased sales, employment and exports of the assisted firms and, with adoption of similar innovations by their competitors, should engender broader impact within the sectors in which they operate. INITIAL CONDITIONS IN PRIVATE SECTOR At the inception of the KPEP project in 2008, USAID reported that 98% of Kosovar businesses had fewer than nine employees. The domestic economy was mostly service, retail trade and small-scale oriented, while the largest export was scrap metal. Private enterprise growth centered on construction, agricultural production and processing for the domestic market, and retail trade. The private sector was not sufficiently diversified and was reliant on a few low-productivity activities. The key constraints faced by private enterprises included lack of product standards, unreliable electricity supply, insufficient transport corridors, high cost of financing, weak courts and contract enforcement, and the overall weakness of economic demand. PROJECT ACTIVITIES TO ADDRESS PROBLEMS KPEP’s activities to address these problems have targeted support to key enterprises within selected sectors and sub-sectors with the objective of increasing value-chain competitiveness, with resulting increases in domestic sales, exports, employment and investment. The sectors selected for intervention were an amalgam of those, such as the road construction sector, that were pre-selected by USAID and named in the Request for Proposals to which contract bidders responded, and those that were selected by the contractor based on analysis of the potential to increase value-chain competitiveness. The modalities of support to the firms selected for KPEP support within Component 1 have ranged from intensive technical assistance and training to matching grants for key investments or marketing efforts. The following presents the sectors and sub-sectors selected for KPEP support and describes the experience in each with activities illustrative of those that KPEP pursued. Construction  Road construction. This sub-sector was selected for intervention in part because of the very large road construction contracts that were being funded by the Government of Kosovo. In practice, the opportunities to provide high-payoff technical and matching grant support to key players in this sector were limited. Nonetheless, KPEP has proved innovative in varying the application of the tools at its disposal to meet the needs of this sector – emphasizing workforce training of heavy equipment operators, for example, rather than grants to the large road construction firms that bid on government contracts.  Construction materials. Demand in this sub-sector has been driven by the general construction boom that is occurring in Kosovo presently. Technical support and a matching grant have encouraged a brick￾making plant, for example, to invest in a higher quality production line that makes it competitive with imports from Serbia. Although everyone in Kosovo is seemingly in the market for bricks nowadays, there Mid-Term Evaluation of KPEP 13 is unlikely to be much potential for export of construction materials, in part because of transport costs, limiting the scope for entry by other Kosovar firms into this subsector.2 Agriculture  Dairy. This sector has been very amenable to quality improvements, driven in part by a Ministry of Agriculture decision to implement a quality control incentives program (with support from KPEP), and partly to investments and technical innovations implemented by some of KPEP’s key clients, who are purchasers of fresh milk products. One KPEP client – an ice cream producer – believes that quality improvements based on the use of fresh milk will increase its sales within Kosovo, at the expense of imported products. KPEP also identified a window for exporting fresh milk products to Albania that looks promising, and played a key role in relieving a non-tariff barrier imposed by low-level Albanian customs officials in contravention of their WTO commitments.  Poultry meat processing. As with the absence of quality products made from fresh milk in Kosovo, the absence of fresh poultry products on the Kosovar market was an opportunity that KPEP moved to fill by providing technical support and a matching grant to encourage a new product entry into the market. Although this intervention has occurred too recently to assess the results, the prospect looks very promising, and eventually may have export potential within the immediate region of Albania, Macedonia and Montenegro.  Fruits & vegetables. KPEP activities in support of this sector have been particularly effective, through technical assistance, training and matching grants provided, for example, in support of investments in post-harvest handling and cold storage of fresh produce. Although the principal markets so far have been domestic, effectively substituting for imports, fruits and vegetables are very competitive products for Kosovo, and the potential is strong to increase exports of horticultural products, both fresh and processed. Noting this potential, USAID has already fielded the New Opportunities in Agriculture project, which targets assistance to horticulture.  Non-wood forest products. The potential market for export to the EU of dried and processed wild fruits and herbs is large, and KPEP’s clients, including one large entrant into the production of processed fruits, is gearing up to meet that demand, while developing natural and organic products for the higher￾end Kosovo market. KPEPs’ clients drawtheir raw material from thousands of collectors of wild forest products from Kosovo as well as neighboring Albania, Macedonia and Montenegro. One issue that arises has to do with ensuring that the collectors do not reduce future yields of these uncultivated products; currently there is little coordination or regulation of the collection season. Forestry & wood processing  Wood processing. The project has provided some support to the producers of furniture from Kosovo forests, but if the target market is in the EU, Kosovar furniture firms are likely to run into the problem that they lack access to lumber that is certified as sustainably harvested, whereas the EU will begin requiring certified wood in all its imports in 2014. KPEP has provided technical, training and grant support to clients who are utilizing forestry by-products to produce pellets that can be used to heat buildings more efficiently, but the market so far appears to be in EU countries like Italy. The evaluation team also visited one furniture producer who relied completely on imported inputs, assembling ready-to- 2 In recent months the project reports making progress with pipe and pvc manufacturers as well as paints, all of which, given lower transport costs, could well be competitive in export markets; these could be the project’s focus during the next fifteen months of operations. Mid-Term Evaluation of KPEP 14 install kitchen and bedroom furniture, but this producer utilized no domestic wood products in that assembly line.  Forestry. The issue of ensuring that Kosovo’s forests are maintained and harvested in a sustainable manner is unlikely to be resolved anytime soon because of the inability of the Government of Kosovo to protect its forests from illegal logging. Partly this results from lack of funds to hire guards to protect the forests, and partly from the fact that a large proportion of the population still relies on firewood as a principal heating source during the winter, making enforcement problematic. ICT KPEP’s support to this sector only began in the second year following project initiation, and had no antecedent in prior USAID projects. KPEP has provided technical and grant support to Business Process Outsourcers that are targeting EU firms looking for call centers and IT services providers. The key ingredient is to establish a relationship with the buyer that is based on assurances of consistently high-quality product provided responsively and in the language of the buyer. In one enterprise visited by the evaluation team, KPEP had provided technical support but the promised grant support from a third entity – the Community Development Fund, financed by the World Bank – had been delayed for four months. The result of this delay had been to imperil the basis of the client’s relationship with its buyer, raising the question of whether this kind of product is amenable to support from donor-financed institutions, which can be slow and unresponsive. Tourism. KPEP’s activities in the tourism sector have included some technical and grant support, for example to development of wine tours that feature overnights at bed-and-breakfast inns. But the sector has lacked the dynamism that might be provided by development of a site-specific tourism strategy centered on developing Kosovo’s extensive natural and cultural heritage as a tourism draw. It also suffers from an enabling environment little conducive to development of the sector; for example, the principal tourism site from the pre-war period, a ski resort, is located in a Serbian enclave that has resisted privatization. Recycling. KPEP’s activities in the recycling sector have run up against an extremely negative enabling environment in which municipalities are not allowed to put garbage collection services up for bid, instead reserving them for a state-owned company that is not required to recycle. This leaves the recycling companies in the position of having to persuade households and businesses to give them their garbage to be sorted and recycled, without compensation, even though they must still pay fees to the state-owned company for collection services. EFFECTIVENESS OF KPEP INTERVENTIONS A general strategy for implementation of an effective value-chain approach to achieve quick and replicable results is as follows:  Targeting assistance to value chains that are relatively free of policy and institutional barriers, and in which market entry is fairly open, and in which there are many buyers and the potential market is large.  Identifying and working with key elements of the selected value chain to enhance quality and productivity.  Providing intensive technical support and grant / loan financing to selected businesses in identified sectors to enhance their value added and to demonstrate their potential to increase sales and profits through specific innovations. Mid-Term Evaluation of KPEP 15  Providing extensive technical and training support and access to affordable financing to competitors of the selected businesses to allow them to adopt the specific innovations demonstrated to be successful and thereby expand sales and employment within the sectors in which they operate. KPEP’s value-chain support activities in the sectors it is working in have tended to emphasize the first three elements of this sequence; the challenge for the remainder of the project will be to ensure that the competitors of KPEP’s primary clients are enabled rapidly to adopt the innovations their clients have used to increase value added, so that sales and employment in the target sectors can be increased more broadly. Accordingly, KPEP’s successes with sector leaders will need to be followed up with support to the extension of their successes more broadly within the sectors. While some direct technical and training support to firms might continue to be provided by KPEP, the training of trainers in key institutions, like business associations or services providers, would be a more sustainable means to encourage broader adoption of the innovations that have proved successful. Facilitating access to affordable financing should also be a KPEP priority. It should also be recognized that, while there are opportunities to gain sales in substitution for imports, nonetheless, especially in a country as small as Kosovo, with limited internal markets, a viable competitiveness approach must be driven by an export-led growth strategy. Although KPEP’s clients have been able to harvest “low-hanging fruit” by concentrating first and foremost on import substitution, as sectoral production expands to take advantage of Kosovo’s competitive advantage, firms will increasingly need to look to export markets for further expansion. Finally, as Kosovar firms move into their natural export markets, policy and institutional constraints may become increasingly important, again motivating the need to strengthen business or trade associations to represent their concerns within the political system. OUTCOMES FROM INTERVENTIONS As reported in its Quarterly Report for Q3 FY 2011, KPEP projected that it would be capable of meeting its life-of-project target results fully in the areas of sales and export increases by the end of year three of project operations, and was on track to meet all its major targets, including in employment creation, by the end of year four. Mid-Term Evaluation of KPEP 16 Table 2. KPEP Results* through Q3 FY 2011 As shown in Table 2, KPEP reported that increases in sales in the sectors it supported had totaled more than $61 million through the end of Q3 FY2011, of which increased exports comprised more than $28 million. Investment increased by $21 million and nearly 3,000 new, full-time-equivalent (FTE) jobs were created. EVALUATION TEAM ASSESSMENT OF OUTCOMES KPEP’s results are impressive, especially given the slow start it suffered in the first two years of operation; and it looks very likely that the project will achieve the results targets set out for it at project inception in the areas of sales, exports, investment and employment creation. Nonetheless, celebration of these results should be tempered by two observations. First, the method that was agreed at the outset of the project for calculating increases in key indicators does not reflect any attribution of KPEP’s direct contributions to those increases in the various sectors and subsectors. After year one of operations, KPEP staff started to calculate such attributions for each subsector, on a quarterly basis. For the most recent quarter, the percentage attributions to employment creation, for example, ranged from just over 10% in the road construction sector to nearly 63% in the recycling sector. The average attribution to the project of employment generated in all sectors during the quarter was less than 37%. Sector/subsector Construction 20,695 10,776 11,759 598 Construction materials 2,443 126 5,363 199 Road construction 18,252 10,650 6,396 399 Agriculture 21,972 13,360 5,432 1,253 Dairy 12,414 997 1,651 323 Fruits & vegetables 10,650 5,182 2,511 425 Non‐wood forest products ‐1,302 7,181 1,270 367 Poultry meat processing 210 0 0 138 Forestry & wood processing 13,594 3,454 3,618 450 Wood processing 13,281 3,454 3,294 425 Forestry 313 0 324 25 ICT 2,202 37 240 161 Business services providers 1,292 0 0 110 Tourism 972 638 236 108 Recycling 80 69 3 5 Other 448 135 15 239 TOTAL 61,255 28,469 21,303 2,924 Source:  KPEP Qua rte rly Report, Q3 FY2011, p. 115 *As compa red wi th baseline figures for each sector/sub‐sector US$  (000,s) Cumulative sales increases Cumulative export increases Cumulative investment increases Cumulative employment increases (FTE) Mid-Term Evaluation of KPEP 17 The second observation relates to the impacts not calculated within the indicator definition agreed with USAID. Even if one utilizes the results figures reported in Table 2, above, which are not deflated to reflect directly attributable results, based on project expenditure figures provided by KPEP the average cost per FTE job created was $4,995 through the end of Q3 FY 2011. But this surely underestimates the potential impact of the project in Kosovo, because its impact on the broader sectors in which KPEP is operating is not calculated or projected. In the case of employment generation, for example, the indicator calculates direct employment generated by the project itself, but it neglects indirect employment, generated via purchases from the relevant supply chain; induced employment, generated via spending by persons employed directly or indirectly by the project; and catalytic employment, generated by improved productivity and performance of the sector due to the innovations made with KPEP’s support. This points once again to the need for KPEP to begin re-focusing its efforts in the final 15 months of the project toward generating these types of impacts in the target sectors, through extension of the innovations introduced throughout those sectors. Prior to doing so, it is recommended that the project re-focus its energies on the sectors where it is likely to have the greatest impact. Table 3 below, presents an ordinal ranking of the cost effectiveness of expenditures in each of the subsectors KPEP has worked in, in terms of attributable jobs created as well as other variables relating to the replicability and scalability of the results achieved in each sector. The ranking tends to indicate that there is a top tier of sectors to focus on, consisting of non-wood forest products, livestock (e.g., dairy and poultry) production, fruits and vegetables and wood processing, whereas those successes that have been achieved in ICT, tourism, construction, recycling and forestry may not prove as easy to replicate and scale up to sector-wide increases in sales and employment. Table 3. Ranking of KPEP Sub-sectors Non‐wood forest products  Livestock**  Fruits & vegetables  Wood processing   ICT   Tourism  Construction materials    Recycling  Forestry  Road construction *Reverse order ranki ng.  Sources:  Expendi ture es tima tes by sub‐sector provi ded by KPEP;  cumula tive empl oyment res ul ts based on KPEP a ttributi ons of res ponsi bili ty for job crea ti on by sub‐sector. **Lives tock includes dai ry and poul try production. Many buyers Scope to increase value‐ KPEP added Sub‐sectors Ranking by Cost per Attributable Job* Export‐ oriented Employment‐ intensive Ease of entry Conducive enabling environment Mid-Term Evaluation of KPEP 18 Finally, Appendix E presents information derived directly from a survey, carried out in Albanian, of KPEP clients who have received matching grants from the project. The objective of the survey was to obtain some responses directly from KPEP clients relating to results sustainability and replicability. A total of 44 grant recipients were contacted and interviewed successfully. Of these, some 37 had received just one grant, and seven had received more than one from KPEP and its predecessor, KCBS. Interestingly, only 24 of the 44 grant recipients said they had received accompanying technical support or training from the project. About twice as many grant recipients used the funds received to purchase equipment as for other purposes (participation in trade fairs, meeting quality standards, etc.) In response to the key question relating to replicability of the innovations introduced, 10 of the 44 respondents said that other firms in the sectors they competed in had already adopted similar innovations since the grant was received; while 14 said this had not occurred, and 20 did not know. Interventions that result in rapid adoption by competitors are much more likely to be replicated across the sectors of interest, and in a project still under implementation an adoption rate of one-quarter may be considered quite robust. Relating to the sustainability of the innovations introduced, the key question was whether, if the client had to do it over, and the grant(s) were not available, having seen the results of the innovations that they facilitated, whether they would have funded the investments in question themselves. To this, three of four recipients responded yes, an indication that there may be scope for developing a market for business support services in Kosovo once the innovations introduced by KPEP have proven themselves to be effective in terms of increasing the sales and profits of sector leaders. COMPONENT 2. DEMAND-DRIVEN DEVELOPMENT OF BUSINESS SUPPORT SERVICES INITIAL CONDITIONS IN SECTOR At the outset of the project, the market for business support services was nascent. Private providers of business services, aside from ICT, were few and mostly dependent on donor funding. Business associations were weak. USAID has used an association development index, first used at the outset of project in 2008 (reported in Hajdari, 2009) to measure the status of business associations in Kosovo using eight variables linked to sustainability of the association. At that time, the median score for the sixteen business associations reviewed was only 1.59 on a scale of 0-5, indicating extremely limited capacity to be financially self-sufficient, to provide services to members, or to represent the membership to the government PROJECT ACTIVITIES TO ADDRESS PROBLEMS This component was meant to help build a number of private-sector firms capable of delivering specialized services to private sector enterprises in Kosovo. In addition, it sought to strengthen business associations as vehicles for services to members in a sector, as well as the ability to lobby government to address problems facing the sector. In each case, the goal was to promote self-sustaining enterprises or institutions. The project undertook a variety of activities, including:  Assistance to individual business services providers (BSPs) through training and “twinning” with international firms for six BSPs.  A training program for women-owned BSPs, for 14 firms.  Training programs for business association leadership.  Promoting the creation of a Management consulting association that meets IMCI standards. Mid-Term Evaluation of KPEP 19  Implementation of an Association Development Index (ADI) to measure over time the progress of individual business associations toward self-sufficiency and high-quality services to members, with annual updating. EFFECTIVENESS OF KPEP INTERVENTIONS Progress toward the goal of strengthening the business services sector has been modest. In part, pursuing this objective appears to have been premature because of the small size of most Kosovar businesses and their lack of willingness to seek assistance from outside firms. The large number of donor-financed sources of free or subsidized technical assistance surely exacerbated the problem. Progress in strengthening business associations has been more effective. Several associations (notably, in banking and accounting) have further strengthened their capacity to serve their members. Several other associations, in ICT, wood processing, and microfinance, substantially increased their ratings on the index. More broadly, the project has disseminated a set of concepts about how business associations should manage their finances, their services to members, their strategic planning, and related issues that seem to be percolating through the association community. OUTCOMES FROM INTERVENTIONS Some BSPs have successfully marketed themselves, but the market for these services is still very young and thin. The relatively small-sized firms predominant in Kosovo are typically unwilling to pay outsiders to provide advice. The latest (draft) version of the ADI shows that most business associations in Kosovo have improved their performance during the KPEP period. On the scale of 1-5, with 5 being the highest, the ADI increased from a mean of 1.75 in January 2009 to 2.70 in August 2011. Associations in Kosovo are weak, but much less so than in 2009. EVALUATION TEAM ASSESSMENT OF OUTCOMES Significant progress was made in a few cases in pushing associations in the direction of greater service to their members and greater self-sufficiency. Despite the progress, the achievement should be regarded as modest for BSPs, and satisfactory for business associations. Clearly, strengthening BSPs and associations is still a work in progress, with much to be done in the future. Nonetheless, the impetus to form and support effective trade associations is likely to increase as growing Kosovar firms confront, first, the challenge of meeting buyers’ requirements regarding standards, quality and timely delivery according to contract (especially those of foreign buyers); and second, to ensure that their interests are adequately represented politically as they confront policy and institutional barriers to increasing exports. COMPONENT 3: BUSINESS ENABLING ENVIRONMENT INITIAL CONDITIONS IN SECTOR Despite an optimistic assessment of the business environment in Kosovo for USAID (Chemonics International 2005) several years before the project began, the first World Bank Doing Business report for Kosovo for 2010 (actually drawing on reported information from 2008 and 2009), ranked Kosovo very poorly, placing it 118th out of 180+ countries. The most serious problems identified in the World Bank report were starting a business, getting construction permits, trading across borders, protecting investors, and enforcing contracts. In all five areas, Kosovo had a ranking poorer than 100. Mid-Term Evaluation of KPEP 20 PROJECT ACTIVITIES TO ADDRESS PROBLEMS The KPEP project did not seek to address overall BEE issues, but rather to focus on specific problems relating to the businesses being promoted under component 1 of the project. This concentration on Component 1 activities was made concrete when USAID began another project, the Business Enabling Environment Project (BEEP), intended specifically to address the World Bank’s Doing Business indicators, working with the national government on a variety of legal and regulatory changes to increase Kosovo’s performance in this area. Even within this mandate, various problems, including frequent changes in the national government, limited the effectiveness of KPEP’s activities under this component during the early period of project implementation. The principal activities carried out under Component 1 have included:  Review and comment on proposed laws relating to food, construction, tourism, forestry, and other areas  Trade promotion in KPEP sectors, and  Promotion of foreign investment in Kosovo through a variety of initiatives EFFECTIVENESS OF KPEP INTERVENTIONS The notable success in this area was the elimination of the use of reference prices for the import of potatoes and milk into Albania. The Albanian customs shift to transaction pricing allowed these two export areas to develop – and should allow exports generally to Albania to follow suit. The specific activity involved KPEP seeking support from the GoK to seek redress from the Albanian government for their arbitrary action. A senior Ministry of Trade official stated that KPEP’s intervention in support of the milk industry was crucial to the resolution of the problem. The pleas of milk producers by themselves, he argued, would have fallen on deaf ears. OUTCOMES FROM INTERVENTIONS Although no assessment is possible about KPEP’s influence on the various proposed laws for which it provided comments, KPEPs efforts in trade were a clear victory for the relevant sectors (milk and potatoes) but also for the entire Kosovo export economy, as Albanian customs officials ended their use of reference prices for Kosovo’s exports. In promotion of foreign investment, the jury is still out. Clearly, the low rating of Kosovo in protecting investors, as well as other areas, is a deterrent to the inflow of foreign investment. The project has experimented with a variety of activities, some of which are likely to yield results in the medium term. EVALUATION TEAM ASSESSMENT OF OUTCOMES Given the environment and the level of effort in this area, we regard the outcome as satisfactory. To improve on this performance during the final 15 months of the project, KPEP may look to reinforcing key trade associations by helping them introduce industry (rather than government) standards to improve members’ ability to meet the requirements of foreign buyers; and second, to develop the capacity to advocate on behalf of their members for key policy or institutional reforms to relieve barriers to exports. Mid-Term Evaluation of KPEP 21 COMPONENT 4: WORKFORCE DEVELOPMENT INITIAL CONDITIONS IN SECTOR Unemployment is a serious problem in Kosovo, and the large number of new entrants to the labor force annually increases the challenge. PROJECT ACTIVITIES TO ADDRESS PROBLEMS The principal activities carried out under this component include:  Training,  Certifications,  Internships,  Job fairs, and  Entrepreneurship training The activities in this area have included a wide variety of activities intended to develop the skills of Kosovo’s large population of unemployed people. Training, job fairs, internships, and promotion of certifications were among the tools used. This is the activity furthest at present from reaching the project target. As noted above, project management expects that the goal will be achieved by end of project. To some extent, this component works at cross-purposes with Component 1. Major gains in productivity (i.e., output per worker) have occurred in many of the assisted firms. This means that employment in assisted firms has grown much more slowly than sales. This is an inherent part of the increased competitiveness of such firms. Employment will grow faster to the extent that such firms are able to continue to grow because of their increased capacity to meet domestic and export demand. EFFECTIVENESS OF KPEP INTERVENTIONS This activity has taken an experimental approach, trying various activities, and identifying those that have yielded the largest benefits. Tailored internships in the dairy and wood product sectors appear to have been quite effective. OUTCOMES FROM INTERVENTIONS While the value of the training, certifications, and internships is not susceptible to easy evaluation, our judgment is that there has been positive synergy between the Component 1 activities and the workforce development work. Each has reinforced the other, most notably in the dairy sector, where interns were able to both transfer knowledge to dairy producers and serve to monitor the quality of milk being produced. A job fair in the ICT sector identified serious weaknesses in the education of students in this area, leading KPEP to promote closer relationships between ICT firms and academic institutions. EVALUATION TEAM ASSESSMENT OF OUTCOMES There is still much to be done to match the skills of Kosovars, particularly new job entrants, to the demands of the marketplace. The project has made important strides in illuminating the issues in this area. On one side, the project demonstrated that many young Kosovars lack the skills needed by business firms. On the other, the project has helped business firms to better articulate their skills needs, and to communicate it to both job entrants and educational institutions. On the whole, this has been a successful effort. Mid-Term Evaluation of KPEP 22 VI. CONCLUSIONS AND LESSONS LEARNED In accordance with the terms of reference for the evaluation, the sections below respond to the five principal evaluation questions and the eleven subsidiary questions. PROJECT EFFECTIVENESS Overall, the project has been highly effective. The project staff is highly professional. In most cases, they have excellent connections with the businesses and associations they are supporting. Staff members are strongly committed to the success of the project, and are focused effectively on achieving the stated goals of the project. The project has had its problems. It appears to have gotten off to a slow start, despite inheriting a client base and major elements of the approach from the previous project, KCBS. The project has been re-organized, with overall emphasis being given to a value-chain approach to promoting private sector development. The original project components dealing with reform of the business enabling environment and development of business services providers were effectively de-emphasized, an indication that the original project objectives might have been too ambitious and not well enough integrated with the project’s overall purpose. By year three, however, KPEP has fully hit its stride, and is promising even more impressive results in the final year of the project. Overall, as shown in Table 2 above, the project led to cumulative sales increases of more than $3 for each dollar expended, more than $1.50 in additional exports, and $1.20 in increased investment. The assisted firms performed much more positively than the overall Kosovo economy, suggesting that the project has been quite effective in raising the country’s productivity in the relevant sectors. The project’s focus on increased sales, exports, investment, and productivity continued to be highly relevant to Kosovo’s economic future. PROJECT EFFICIENCY Overall, the project appears to have provided good value for the dollars invested. But in the first two years of the project, it appears that despite inheriting many experienced and senior host-country national staff members from the KCBS project, there was a disproportionate utilization of expatriate personnel, both short￾term and long-term, with the result that in the final two years of the project almost no budget remains for expatriate STTA. The large number of expatriate LTTA advisors derived from the original design of the project by USAID, which envisioned substantial expatriate LTTA support at project outset, but the disproportionate front-loading of expatriate STTA support can only be explained by project decision-making following contract award. The extensive use of expatriate LTTA appears to have been less efficient in achieving project goals than greater use of Kosovar staff combined with more use of focused STTA. By year three, following a change in management, long term expatriate staff members had been cut from six to two, and the STTA assignments were almost exclusively being provided by host-country nationals. There also appears to have been something of a delay in placing grants in the first two years of the project, and in otherwise leveraging project resources. A major effort was made to extend grants in year 3; this change in strategy seems to have been very effective in turning the project around. Mid-Term Evaluation of KPEP 23 As mentioned under the previous point, there are substantial quantitative results from the project. Standard metrics for such indicators are scarce, but these appear quite satisfactory. PROJECT IMPACT The project has had substantial impact on the economy of Kosovo, fostering innovations by a set of dynamic enterprises with increased capacity to compete in domestic, regional and European markets. These innovations have increased sales, employment and exports of the assisted firms and, with adoption of similar innovations by their competitors, they will engender broader impact within the sectors in which they operate. The performance of these firms and sectors is much better than that of business in general in Kosovo. The result is that, by the end of year three KPEP is projected to be capable of meeting its life-of-project target results fully in the areas of sales and export increases; and is reportedly on track to meet all its major targets, including in employment creation, by the end of year four. The use of the value-chain approach has been an important ingredient of this success. The impact of the project on business associations – an important asset for competitiveness in the long run – is more mixed. Apparently, considerable learning and acquisition of trust among businesses in a sector is an important feature of strong associations. This seems likely to emerge only gradually. KPEP cannot be a substitute for cooperation among members of the business community with shared interests. PROJECT SUSTAINABILITY A number of sector leaders that received KPEP assistance are clearly launched onto a sustainable growth path in the agricultural, dairy, and construction materials sectors, and no longer seem candidates for further assistance. Additionally, anecdotal evidence suggests that competitors in the targeted sectors have invested in innovations similar to those invested in by KPEP’s primary clients with project support, an indication that these innovations are being adopted sector-wide; and, with support to these sectors through training and improved access to finance, it is probable that the innovations will lead to sustainable improvements in sector competitiveness. There is also evidence that at least some of the workforce training KPEP has supported is institutionally sustainable – the prime example being that of heavy equipment operators’ certification through a local center. On the other hand, there are a number of areas where sustainability is unlikely to be achieved by the end of the project. The KPEP scope-of-work targets the development of commercially viable business services while addressing issues of affordability and sustainability. Whether this is achievable within the context of the support KPEP has provided to associations is problematic. PROJECT RELEVANCE The project has been highly relevant to Kosovo’s economic circumstances. There are three factors to support this judgment: 1. Businesses in the country are still in an early stage of development of their understanding of how to successfully compete in a global market economy. The project has addressed this successfully in a number of individual cases, most with potential catalytic impact on the rest of the business community in the assisted sectors. 2. Unemployment is a critical problem in Kosovo, so the efforts of the project to create additional productive jobs in the country are of considerable importance. Mid-Term Evaluation of KPEP 24 3. The emphasis on exports, in light of the country’s huge trade imbalance and small size, is highly relevant. Not only does the effort to increase exports support the previous two points, but it also creates capabilities for greater import substitution. This is because, in the open trade regime, the lessons on quality and cost learned from exporting are extremely valuable in competing with imported products. RESPONSES TO SUBORDINATE QUESTIONS 1. Should there be any programmatic shifts? Several programmatic shifts relating to KPEP have already occurred. These have included:  An organizational shift from component management to sector management, and integration of all the component elements – technical assistance, business services providers strengthening, reform of the business enabling environment and workforce development – within the central focus on a sectoral value￾chain approach, supported by grants, training and technical assistance.  A staffing shift from expatriate component leaders to host-country national sector leaders, managed by a COP and DCOP, with reduced reliance on expatriate LTTA and STTA and more reliance on host￾country nationals.  A new USAID business enabling environment project put into place capable of pursuing policy and institutional reforms that are not specific to the value-chain approach, and to reinforce the project in pursuing those that are. The Mission has also put into place a new agriculture project that will extend KPEP’s work in the horticulture sector with an overlap of several months.  An emphasis on making sure that grant support to selected value chains gets out the door in year three of KPEP, in order to facilitate leveraging of that support in year four. No further programmatic shifts are thought to be necessary at this time. 2. Is there effective synergy / coordination with other EG and DG activities? The indications so far from interviews held with other USAID economic growth and democracy and governance activities are that cooperation and exchange of information with other USAID projects are excellent at present, and that it has improved substantially during the past year or so. With a renewed emphasis on export-led growth, the DG office’s emphasis on economic diplomacy through training of Kosovar embassy staff in key foreign countries could prove to be a real asset to achieving KPEP’s export goals. 3. Is KPEP working in the sectors with the greatest potential for growth? The results produced in livestock and horticulture, in particular, and in value-added agricultural production more generally, confirm that the potential for growth in these sectors is substantial. The Mission’s decision to extend KPEP’s work in horticulture by fielding a new project devoted principally to these commodities validates the emphasis on this sector. Similarly, while the results are still in the process of being developed, KPEP’s work in the dairy and poultry sectors seems to be on track. KPEP’s emphasis on non-wood forest products and certified wood processing also shows promise. Other sectors, such as road construction, also have been confirmed as strong growth sectors, but the question here has revolved around how to work with economic agents in these sectors to produce the greatest impact, given the tools that KPEP has at hand. KPEP has proved innovative in varying the application of these tools to meet the needs of these sectors– emphasizing workforce training of heavy equipment operators, for Mid-Term Evaluation of KPEP 25 example, rather than grants to the large road construction firms that bid on government contracts. This strategy seems to have been effective. Alternatives to the value-chain approach may be the key to working successfully with other sectors, e.g., policy and institutional reforms to reduce heavy weekend backups at border immigration posts, in order to facilitate cross-border commercial transport and weekend tourism; or integrated community development to facilitate forest protection, waste recycling and utilization of Kosovo’s enviable natural and cultural heritage as a tourism draw. Similarly, a concerted effort to link potential leaders in the ICT sector with reliable commercial financing might be a better tack to take than directly supporting them with matching grants. It is worth noting that while KPEP was successful eventually in rolling back a non-tariff barrier to the importation of fresh milk and potatoes by Albania, more generally the success of value-chain support to producers planning to export their products depends heavily on open access to foreign markets, and in particular regional markets. Without that open access, there would be very definite limitations on the potential for growth in these sectors. 4. How is the matching grant support for associations working? What about sub-contracting arrangements with business service providers? As regards subcontracting with business services providers, the evaluation team heard no comments to the effect that these were anything but effective. It is true, however, that matching grants to productive enterprises were reportedly delayed earlier this year because of changed USAID/Washington procedures relating to environmental due diligence. 5. How effective are the procedures and controls for managing grant funds in providing an objective review of candidates, allocating funds to the most effective projects that contribute to the results framework and in preventing fraud, waste and abuse? The evaluation team heard no evidence and has no reason to suspect that the grants program was subject in any case to waste, fraud or abuse. The procedures that have been put into place seem to be quite effective in providing objective review of candidate applications and allocating funds to the projects with the greatest payoff potential. That said, there is a case to be made for capping USAID grant contributions from all sources to any single client. The KPEP project seems to have continued grant support to a number of clients that also received support from the previous KCBS project, and to have provided more than one grant to several of the same clients during the LOP. It should be remembered that grants to a single client are useful in creating a development model that others in the sector must compete with, but it is important not to provide so much support that the competition finds itself at a competitive disadvantage. One innovation that might be considered, especially in the final 15 months of the project, would be for KPEP to hire an expert host-country national focused on the cross-cutting purpose of ensuring that value￾chain participants are able to gain access to (debt) financing from commercial sources. Over the longer term, Kosovar businesses need to establish relationships with commercial banks, the only source of sustainable financing for working capital and expansion as donor funding gradually fades. 6. Are the numbers, quality and relative cost of long-term and short-term technical staff generally appropriate? The organizational shift from expatriate component leaders to host-country national sector leaders (see answer to overall evaluation question #2, above) has effectively reorganized the relative cost of expatriate support by reducing the number of LTTA expatriate staff in the field in proportion to the number of HCN staff. The project also is now relying much less on expatriate STTA assignments than it did in the first two Mid-Term Evaluation of KPEP 26 years of the project. The current structure is definitely more appropriate to the KPEP project’s emphasis on a value-chain approach – implying that the development payoff from utilization of expatriate resources, both short and long-term, may have been disproportionately low. 7. Is there potential for greater KPEP collaboration with other USAID projects? As mentioned in the answer to subordinate question #3, given KPEP’s extensive knowledge of the opportunities and constraints in specific sectors, there should be substantial opportunities to effect synergies with BEEP, the USAID project devoted to business enabling environment reform. An example would be BEEP’s support to Kosovo’s bid for observership status at the WTO, which might be accompanied by substantial training and technical support relating to Sanitary and Phytosanitary Standards (SPS) prevalent among WTO members for the importation of fresh horticultural products. KPEP should also be quite proactive in ensuring that the administrative and technical handover of value chain work to NOA, the new USAID project focused on horticulture, is accomplished seamlessly and effectively. And, as mentioned in response to subordinate question #2, the DG office’s Economic Diplomacy project looks like a natural fit for KPEP collaboration in respect to implementation of an export-led growth strategy. The Young Entrepreneur Project appears to build on earlier KPEP work, so is also closely coordinated. 8. What has been learned from other USAID competitiveness/enterprise development projects in other countries? Are any of these lessons applicable to Kosovo? Among the lessons learned from USAID support to value-chain competitiveness in other countries are the following sequential steps:  Targeting assistance to value chains that are relatively free of policy and institutional barriers, in which market entry is fairly open, and there are many buyers and the potential market is large.  Identifying and working with key elements of the selected value chain to enhance quality and productivity.  Providing intensive technical support and grant / loan financing to selected businesses in identified sectors to enhance their value added and to demonstrate the potential to increase sales and profits through specific innovations.  Providing extensive training support and access to affordable financing to competitors of the selected agribusinesses to allow them to adopt the specific innovations demonstrated to be successful and thereby compete with selected businesses that have received intensive support. KPEP’s value-chain support activities in the sectors it is working in have tended so far to emphasize the first three elements of this sequence; the challenge for year four will be to ensure that the competitors of KPEP’s primary clients are enabled rapidly to adopt the specific innovations to increase value added so that they can increase sales and employment in the sectors more broadly. It should be recognized that, while there are opportunities to gain sales in substitution for imports, especially in a country as small as Kosovo, a viable competitiveness approach must be driven mainly by an export-led growth strategy. 9. Has Booz Allen effectively and efficiently managed KPEP to date? Are there recommendations to improve the management structure? There have been several changes in management structure to date, including changes in senior staff as well as organizational structure (see answer to subordinate question #1). These changes reportedly came about in response to a slow start in achieving project results. Now that the project has finally achieved a management and organizational structure that is more effective in producing the desired results, no further changes in the management structure are recommended by the evaluation team at this time. Mid-Term Evaluation of KPEP 27 10. What are the recommendations of the evaluation team regarding KPEP operations during the final year of operation? The evaluation team recommends that KPEP operations continue to consolidate gains the project has made in the sectors it is involved in, with the following objectives:  Working in collaboration with USAID’s new horticulture value-chain project, leverage the grants KPEP has made, the bulk of which occurred in year three, to produce the greatest possible impact among beneficiaries in the selected value chains / sectors.  In collaboration with USAID’s new business enabling environment project, identify and remove barriers to improvement of quality and expansion of sales, especially exports, in KPEP’s selected sectors.  In collaboration with banks, other donors, and other activities supported by the USG (e.g., Crimson Capital), work to ensure that competitors in selected sectors have access to financing to enable them to adopt the innovations that have made possible increased sales and profitability among KPEP’s primary client base.  In collaboration with trade associations, NGOs and other business services providers, work to ensure that training is available to enable competitors in selected sectors to have access to technologies that will allow them to adopt the innovations that have made possible increased sales and profitability among KPEP’s primary client base, and to continue innovating to increase sales and profitability in the future. 11. What are the recommendations of the evaluation team regarding future USAID work with the private sector? The evaluation team believes that the activities that have been pioneered by the KPEP team to date have the potential to greatly increase sales and profitability in several of the sectors it has been involved in, as long as the innovations that have been pioneered are adopted broadly across the sectors in which KPEP’s primary clients are involved. Future work will need to consolidate and extend those gains, recognizing that in many respects what has been achieved has been to harvest “low-hanging fruit,” meaning that some of the knottier issues, e.g., expanding exports and providing access to commercial financing, have yet to be addressed. Accordingly, if a KPEP project extension were to be considered, the evaluation team believes that it should adhere to the following guidance:  Consolidate and deepen the gains made in selected sectors other than horticulture (which is covered by the new NOA project)  End grant support to the clients in those sectors, and move them toward opportunities for commercial financing  Pursue export-led growth opportunities, especially in Kosovo’s natural regional market – comprised of Albania, Macedonia and Montenegro  Encourage relevant trade associations to establish direct relationships with their counterparts in Albania, Macedonia and Montenegro, and encourage the development of a regional forum to promote economic integration Mid-Term Evaluation of KPEP 28 VII. RECOMMENDATIONS We recommend that the project be extended two additional years, with some modifications. We offer three reasons for this recommendation. 1. We consider the KPEP project team at present to be excellent, and poised to deliver increasingly valuable benefits during the remainder of the life-of-project; and we expect that a project extension would provide even larger benefits. We strongly believe that it would be a mistake not to continue to support a project that is achieving excellent progress, with the promise of even better results if extended. 2. A full and open competition for a successor project, in our opinion, would risk serious loss of momentum in the areas where the project is currently most successful. Uncertainties about future employment would be likely to lead to gradual erosion of the capabilities of project staff, as concern about continued employment would contribute to the loss of key staff. And the choice of a new implementer, with a new COP, is one subject to significant uncertainties. where . Moreover, the timing of a new award itself, given the vagaries of USAID’s contracting processes could itself be an obstacle to achieving USAID’s goals in Kosovo. Further, the transitional costs involved in a rebid of the contract, even if the current firm wins the competition, are substantial. The process of reducing staff, disposing of property, and other close-out processes can be expected to be well-advanced before a new award is made; and the potential is substantial for lengthy delays in the award and start-up of a new project. 3. The extension would align project completion with the USAID strategy for Kosovo, which covers 2010- 2014. Kosovo is still a very young country, and the environment for business is still evolving rapidly. Designing a new project in the context of a new USAID strategy for 2014 would allow for better targeting of the most appropriate approaches and activities for that environment. We propose modifications for extending the project as follows: 1. No grant component would be included in the extended project. Kosovar enterprises need to understand that businesses must operate on commercial principles. 2. Instead, the project would hire a specialist in linking producers with commercial financing, in order to promote the faster development of ordinary commercial relationships between clients and the financial sector. This would be an important step toward the maturation of the Kosovar business community. 3. The project would graduate the most successful enterprises, and reach out to a new generation of small and micro firms. Many of the assisted firms are surely able to grow on their own, and need to be pushed out the door. On the other hand, nascent firms, encouraged by the improving business environment, need start-up help in the form of technical assistance, training and improved access to finance. 4. The number of expatriate staff would shrink from two to one. Though the large number of expatriate staff was specified by USAID at the outset, we judge that the capacity of Kosovar business firms has moved beyond the need for intensive LTTA support. Targeted STTA by expatriates will continue to be important, but lowering the profile of the project by reducing the expatriate staff would be a further demonstration of the increasing capabilities of Kosovars. 5. The project would shift much of its work in the fruit and vegetables sector to the USAID NOA project, leaving KPEP with a narrower focus. USAID has not provided a clear delineation between the two projects. This needs to be done. Our recommendation is to shift most work in the fruit and vegetables Mid-Term Evaluation of KPEP 29 area to NOA, but this is an issue that needs serious discussion between USAID and the staffs of the two projects. 6. Our current judgment is that the project’s narrower focus would include livestock (dairy, poultry and possibly other targets of opportunity), non-wood forest products, certified wood processing and tourism, phasing out of ICT, construction materials, recycling and forestry. Still, these are mid-term judgments, which could be overturned by developments during the next year or so. USAID needs to continue to follow the original philosophy of its work in the business development area: look for low-hanging fruit, and pursue opportunities where they seem likely to produce sales, exports and jobs in the near term. Mid-Term Evaluation of KPEP 30 VIII. APPENDICES A. Evaluation Scope of Work B. KPEP Project Monitoring Methodology C. Persons Interviewed D. Quantitative Project Indicators, with results through June 30, 2011 E. Results of Interviews with Grant Recipients F. References Mid-Term Evaluation of KPEP 31 APPENDIX A. EVALUATION SCOPE OF WORK Mid-Term Evaluation of the Kosovo Private Enterprise Program (KPEP) Statement of Work BACKGROUND GENERAL As USAID launched the new Kosovo Private Enterprise Program (KPEP), it intended to build on the successes of Kosovo Cluster and Business Support (KCBS) while moving to a higher plateau of enterprise and industry competitiveness. Conditions in Kosovo were more stable and normalized than they were several years before, and at the same time, markets locally, regionally, and globally were more competitive than ever. While enterprises may have been able to continue growing through the opportunities available in the domestic market and with the influx of new infrastructure funding, particularly for road construction, Kosovar industries needed to use existing growth opportunities as a spring-board to enter new markets, develop new products and processes, advance labor skills and productivity, and improve the business environment in anticipation of greater regional integration and eventual entry into the European Union. While overall progress in developing a market economy has been considerable, the private sector is not yet dynamic or competitive. Private investment in Kosovo remains low and unemployment is unacceptably high (estimated at 44% in 2004). Increased levels of private investment, and particularly productive investment, from both foreign and domestic sources are vital to create rapid and sustained economic growth. There is a need to help Kosovo develop competitive industries producing goods that are marketable domestically as well as regionally. Roughly 40% of Kosovo’s exports (e.g., base metals, raw hides and skins, prepared foodstuffs) flow to neighboring countries such as Macedonia. However, imports dwarf exports resulting in a massive trade imbalance.3 However, building competitive industries will face many challenges, such as the lack of industry standards in local agricultural and manufacturing enterprises; weak professional and trade associations; the lack of essential business and technical skills and little capacity for organizational and operational management; unfavorable business environment and poor knowledge of current market information, industry trends, buyer and supplier networks. To have a sustainable economy, Kosovo must be able to compete domestically and regionally in the export market. To be competitive, private enterprises must have the capacity to meet local and regional demand for goods and services at the standards required by the local population and regional buyers. The agriculture sector in particular receives targeted assistance as it supports a significant portion of the population and generates roughly 12-15% of Kosovo’s GDP. Under the USAID/Kosovo Strategic Plan (2010-2014), USAID is attempting to tackle these challenges to private sector-led development in Kosovo. The USAID/Kosovo Strategy “envision Kosovo as an effective state, with a viable economy and an inclusive democracy on the path to European integration.” The economic program, Assistance Objective 2: Increasing Private Sector Led Economic Growth, is supported by three Intermediate Results (IR): IR 1: Private Sector Growth and Investment is Increased IR 2: Kosovo Has a Reliable Energy Supply 3 Kosovo Customs estimates exports of appr. 300 million euros and imports of appr. 2 billion euros in 2010. Mid-Term Evaluation of KPEP 32 IR 3: Economic Institutions Ensure Fiscal Sustainability KOSOVO PRIVATE ENTERPRISE PROGRAM The goal of the Kosovo Private Enterprise Program is to stimulate the competitiveness of Kosovo’s private sector. The program focuses on four sub-objectives: 1) private sector development in targeted sectors, 2) demand-driven development of business-support services, 3) an improved business enabling environment, and 4) workforce development. The program is designed to improve the competitiveness of selected sectors for which there is growth potential. Agriculture, construction/construction materials, ICT and wood processing/forestry are the main sectors receiving assistance, Within targeted sectors, the program promotes the improved quality and efficiency of locally produced goods and services, with the expectation of expanding employment for all of Kosovo’s citizens including youth and minorities, reducing Kosovo’s massive trade imbalance, and furthering Kosovo’s integration into the Balkans and beyond. Core component 1: Private sector development of targeted sectors: The core component will be to develop targeted sectors with clear end-market opportunities. Agriculture and construction were pre-identified as two areas for technical assistance. A post-award sector selection (based on a set of selection criteria such as market potential, cost factors and skills availability) determined other possible sub-sectors for involvement, such as ICT, wood processing /forestry as well as domestic tourism and recycling on a smaller scale. Areas for focus include such areas as new product development, quality and food safety standards, post-harvest handling (cold storage, grading/sorting/packing), and marketing. Component 2: Demand-driven development of business support services: This sub-component addresses the need to develop a local cadre of professional and specialized consultants to improve business or private sector competitiveness, particularly those with internationally-recognized certifications and qualifications in the targeted sectors. Areas include facilitating business service markets, particularly specialized consulting advice in such areas as total quality assurance and food safety standards such as Hazard Analysis Critical Control Points (HACCP); strengthening private sector providers of business services; and strengthening business associations to generate fee-based services and advocate for members. Component 3: Improved business enabling environment: This sub-component addresses the specific barriers that constrain the targeted sectors. Trainings and other types of focused short-term technical assistance for improved enterprise growth and competitiveness is provided to relevant stakeholders. Areas for involvement include standards, certifications, and testing (such as food safety, forest certification, product quality standards, and construction standards); licensing and registration; implementation of construction standards in government procurement, particularly new road construction that is envisioned post-status; energy efficiency; and advocacy skills. Component 4: Workforce development: The workforce development activities generally align with the targeted sectors for competitiveness. Areas of focus include construction skills training, ICT skills, internships and apprenticeships, technical training in food safety for laboratory technicians and institutional trainings; Strategic Activities Fund (SAF): The program has a Strategic Activities Fund (SAF) for grants and sub￾contracts in the amount of $3,760,000. A total of $2 million of this amount is set aside to support micro￾enterprises. This mechanism provides cost-shared funding and financing in the form of grants and sub￾contracts to businesses, associations, chambers of commerce, or NGOs in order to advance competitiveness within the program components. Safeguards and selection criteria include the following: 1) clear and transparent procedures, 2) catalytic impact, 3) cost-sharing, 4) avoid the crowding out of the private sector, and 5) a clear exit strategy. Cross-cutting themes are also incorporated into every area of assistance of the Mid-Term Evaluation of KPEP 33 program to include minorities, gender, youth, information and communication technologies, and energy efficiency. EVALUATION SCOPE OBJECTIVE The purpose of this mid-term evaluation is to provide USAID/Kosovo with an objective external assessment of the performance of the Kosovo Private Enterprise Program to date. The KPEP evaluation team is tasked with assessing the effectiveness, efficiency, impact, sustainability, and relevance of KPEP. The results of this evaluation will provide feedback to USAID/Kosovo and KPEP for any necessary mid-course corrections over the life of the project. This evaluation will also guide future planning for USAID/Kosovo. METHODOLOGY The contractor is asked to use to the maximum extent possible the best available social science methods and tools that will deliver an unbiased, relevant, and transparent evaluation. This should include all available and relevant quantitative instruments, as well as qualitative instruments. At a minimum, the evaluation team will carry out on-sight research and data collection (via interviews, focus groups, and/or other methods) in Kosovo for approximately 20 person-days per evaluation team member. The team will collect information from key informants and stakeholders (e.g., client enterprises, associations, government counterparts), KPEP project staff, relevant USAID staff, and other donors. The contractor should identify a list of key research questions and include them in the evaluation design. The evaluation will also draw on project documents and reports. An additional ten person-days are authorized for pre- and post-trip preparations (e.g., planning, reviewing documents, report writing and editing). Note that the contractor may suggest an alternative approach if appropriate. SCOPE OF WORK The contractor will provide a two person team to conduct a mid-term evaluation of the USAID Kosovo Private Enterprise Program, implemented by a consortia led by Booz Allen Hamilton. The team will develop and adopt an approach that elicits and analyzes information, and provides key findings, conclusions, and recommendations on the issues below. Priority Questions: 1. EFFECTIVENESS: Are expected results occurring? Are the stated goal, purpose and outputs as defined in the project description still relevant? Are the performance indicators and means of measurement that KPEP has adopted sufficient to adequately measure results? 2. EFFICIENCY: Are the results being obtained by KPEP being produced at an acceptable cost compared with alternative approaches to accomplishing the same objectives? 3. IMPACT: What has been the quantitative and qualitative impact of USAID-funded activities in the sectors where KPEP is working? How has KPEP made a difference in the development of the private sector? Has KPEP assistance made a difference in those enterprises’ ability to compete? What effect did the approach adopted by KPEP have on the impact achieved? Is the adopted value chain approach working in Kosovo? 4. SUSTAINABILITY: As currently implemented, are KPEP’s activities likely to have a sustainable development impact after USAID funding has stopped? Will the organizations supported under KPEP, Mid-Term Evaluation of KPEP 34 whether farmers, enterprises, and processors or business support organizations (e.g., skills training providers, associations, business service providers) have the capacity to provide future support to the private sector when USAID funding has stopped? 5. RELEVANCE: As the flagship project in the private sector, how relevant is the KPEP project toward achieving USAID Assistance Objective 2 “Increasing Private Sector-Led Economic Growth”? Have outcomes for the private sector in Kosovo changed as a result of USAID’s investment in KPEP? Subordinate Questions:  Should there be any programmatic shifts?  Is there effective synergy / coordination with other EG and DG activities?  Is KPEP working in the sectors with the greatest potential for growth?  How is the matching grant support for associations working? What about sub-contracting arrangements with business service providers?  How effective are the procedures and controls for managing grant funds in providing an objective review of candidates, allocating funds to the most effective projects that contribute to the results framework, and in preventing fraud, waste, and abuse?  Are the numbers, quality, and relative cost of long-term and short-term technical staff generally appropriate?  Is there potential for greater KPEP collaboration with other USAID projects?  What has been learned from other USAID competitiveness/enterprise development projects in other countries? Are any of these lessons applicable to Kosovo?  Has Booz Allen Hamilton effectively and efficiently managed KPEP to date? Are there recommendations to improve the management structure?  What are the recommendations of the team regarding KPEP operations during the final year of operation?  What are the recommendations of the team regarding future USAID work with the private sector? CONTRACTOR TASKS AND DELIVERABLES TASK ONE: LITERATURE REVIEW AND EVALUATION METHODOLOGY PREPARATION Prior to beginning the interview process, the contractor shall prepare for the evaluation by reviewing key documents on competitiveness and enterprise development;4 background material on Kosovo’s economy; and applicable USAID design and project documentation.5 4 See for example at www.dec.org the USAID-funded report “Promoting Competitiveness in Practice: An Assessment of Cluster-Based Approaches” by the Mitchell Group, Nov. 2004, “Enterprise Growth Initiatives: Strategic Directions and Options” by Snodgrass and Winkler, DAI, February 2004, “ICT as a catalyst to enterprise competitiveness by J.E.Austin, BGI, February 2010 5 See for example the USAID/Kosovo Strategic Plan (2010-2014) and KPEP quarterly reports, annual reports, work plans, and reports from short-term technical assistance. Mid-Term Evaluation of KPEP 35 TASK TWO: CONDUCTING FIELDWORK The contractor shall meet with the USAID/Kosovo Program Office and Economic Growth Office to review objectives of this evaluation. The contractor shall collect data from a broad range of stakeholders familiar with the KPEP, following the approved evaluation design. These stakeholders may include private sector representatives (e.g., enterprises, associations, Chambers of Commerce), Kosovo government officials, all current long-term technical advisors (LTTA), short-term technical advisors (STTA) currently in town, and other donors. C. TASK THREE: REPORT PREPARATION AND BRIEFING The contractor shall provide an oral briefing of its findings to the USAID/Kosovo senior management, the Program Office, and the Economic Growth Office prior to departure. The evaluation team will present a draft report within ten business days of returning to the U.S. The final report will be due within 15 days following receipt of comments from USAID. See deliverables below for more detail. DELIVERABLES 1. Oral Briefings. The evaluation team will meet with USAID/Kosovo upon arrival in Pristina. The team will also provide an oral briefing of its findings and recommendations to the USAID/Kosovo senior management and economic growth office prior to departure. 2. Draft Report. The evaluation team will present a draft report in English of its findings and recommendations to the USAID/Kosovo Economic Growth Office within ten business days from the time of return to the United States. 3. Final Report. The Final Report will be provided to the USAID/Kosovo Economic Growth Office in electronic form within 15 days following receipt of comments from USAID. The report shall include an executive summary and not exceed 30 pages (excluding appendices). The report shall follow USAID branding procedures. The contractor shall also submit an electronic copy of the report to the USAID Development Experience Clearinghouse at www.dec.org. An acceptable report will meet the following requirements:  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  The evaluation report should address all evaluation questions included in Section II C of the scope of work.  The evaluation report should include the scope of work as an Annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline shall be agreed upon in writing by the COTR.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report.  Evaluation findings will assess outcomes and impacts using gender disaggregated data.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Mid-Term Evaluation of KPEP 36  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions.  Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an Annex, including a list of all individuals interviewed.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. TEAM COMPOSITION 1. Evaluation Team Leader (senior): The team leader shall have at a minimum, a Master’s degree in economics or a related discipline, and 10-15 years of experience in economics, policy reform, or private sector development. Experience with project evaluation/assessment is required. The individual shall have excellent written and oral communication skills, as well as exceptional organizational and analytical abilities. Knowledge of USAID programs is required. Background in the Europe and Eurasia region is a plus. 2. Private Sector Specialist (senior): The private sector specialist shall have a minimum of 10-15 years of experience in the private sector and/or private sector development programs. Knowledge of competitiveness methodologies is desirable as are analytical skills in enterprise development, business support services, and association development. Knowledge of USAID programs is preferred. Background in the Europe and Eurasia region is a plus. LOGISTICS PERIOD OF PERFORMANCE AND DUTY STATION The evaluation will take place in mid-2011 in Pristina, Kosovo and surrounding areas (three weeks). Up to two weeks total pre- and post-trip will be authorized in the U.S. (or home location). A six-day work week is authorized. Total level of effort (LOE) will be for two expatriate advisors for 30 work-days each. ADMINISTRATIVE AND LOGISTICAL SUPPORT The BAH Kosovo Private Enterprise project team will provide administrative and scheduling assistance. The Contractor will be responsible to provide transport and translation services. USAID can assist in identifying potential service providers. USAID will provide administrative assistance in scheduling initial interviews with stakeholders. EVALUATION CRITERIA Firms submitting a proposal to provide services under the Evaluation Services IQC task order for the Kosovo Private Enterprise Program evaluation should present an evaluation design proposal with identified methodology and a list of key research questions. The information provided should be under 15 pages (not including resumes). Prospective contractors should list staff proposed for this task order and include their resumes and statements of availability. The selection criteria/assessment factors are as follows: Mid-Term Evaluation of KPEP 37 Personnel (50 percent) – The education, experience, and expertise of key staff. The staff should have experience in economics/private sector/enterprise development and in evaluation. See experience required under section V. “Team Composition”. Overall Evaluation Design (30 percent) – Quality of contractor’s overall evaluation design. Prospective contractors shall describe their overall approach to the tasks defined in this scope of work, including alternative suggestions to specifications in this scope of work if so justified. Past Performance and Corporate Capabilities (20 percent) – Prior experience and track records of contractor in evaluation, especially of private sector competitiveness programs. Mid-Term Evaluation of KPEP 38 APPENDIX B. KPEP PROJECT MONITORING METHODOLOGY This Appendix presents the evaluation team’s assessment of KPEP’s monitoring methodology. Cumulative project results through June 30, 2011 are presented in Appendix D. KPEP’s project monitoring methodology is well-developed and responsive to the requirements of its contract with USAID. Nonetheless, modifications may be considered more fully to capture the extent to which the broader objectives of the project are being achieved. Several suggestions in this regard are offered below. Perhaps the potentially most important, though difficult to quantify, would be the impact of the investments by KPEP firms on their competitors. Non-assisted firms in some areas have adopted productivity-increasing technologies first pioneered by KPEP firms. KPEP’s methodology relies on several levels of indicators. The following summarizes the ‘context-level’ and ‘results-level’ indicators that the project monitors and presents the evaluation team’s comments on the efficacy and applicability of the indicators at each level. (‘Activity-level’ indicators are more of a management tool than a results monitoring tool, and hence are not assessed here.) 1. “Context-level.” These are objectives-level results over which the project has only limited influence, but which nonetheless provide perspective about the context in which the project is operating. At this level, KPEP collects and reports several results at an aggregate level, including: a Exports as a percentage of imports. USAID’s expectation for this indicator was that it would move from around 10% (2008 baseline) to 13% in 2012. It appears that the economy will exceed this expectation, because the actual proportion in 2011 was 14.6%, based on reports available for the customs office. The measure itself is not very useful conceptually, however, because the objective should be to increase the levels of exports versus imports, and not the proportion of exports to imports, which could be achieved by restricting imports rather than by export-led growth. b Foreign investment. USAID’s expectation was that this indicator would move from Euro 357 million (2008 baseline) to Euro 500 million in 2012. The actual number for 2011 was Euro 354 million, indicating that this expectation is unlikely to be met. But the real indicator of interest should be investment, both domestic and foreign, because Kosovo is still in an early stage of developing its foreign markets and normally increases in foreign investment may be expected to follow increases in foreign trade. Tracking both foreign and domestic investment would also more closely align this indicator with indicators tracking project results in respect to increasing client investments. c Value of sales. USAID’s expectation here was that sales (i.e., ‘turnover’) in all sectors would increase from Euro 4.4 billion (2008 baseline) to Euro 6.7 billion by 2012, an increase of Euro 2.3 billion. But reported aggregate sales fell by more than one-quarter in 2010 and by 2011 were falling far short of expectations. The main problem with this indicator is that it depends on official sources, when small and micro enterprises have an interest in minimizing their reporting of turnover to official sources, to avoid VAT tax collections. Disaggregating the figures by sector would only compound this reporting error. Unfortunately, this means that there is no aggregate-level data series that may be utilized to assess the extent to which increases in sales by KPEP’s clients are being emulated across the sectors in which they are operating. d Number of people employed. USAID’s expectation was that employment in all sectors would increase from 481,000 in 2008 to 534,000 in 2012. With the actual result on this indicator reaching 518,000 in 2011, the target appeared difficult to achieve within KPEP’s life-of-project. But a USAID report in Mid-Term Evaluation of KPEP 39 2009 concluded that total employment in Kosovo was only 367,0006 which calls into question the baseline figure that was utilized. 2. “Results-level.” What the KPEP’s Performance Based Management System manual (dated 31 March 2009) terms “results-level” results are in fact outcomes attributable to the project and its technical assistance, training and grant support of key clients. These include, the following indicators, by component: Component 1: Cumulative increases in sales, exports, investment and employment in target sectors, over baseline values. KPEP tracks these variables quarterly for all its clients, establishing new baseline levels each time a new client is added to the list. According to these “results-level” indicators, as reported in the body of the evaluation report, KPEP is well on its way to achieving its targets by 2012 in all but employment creation, in the sectors in which it is active (see Table D.1 and accompanying charts in Appendix D for an explication of the results achieved). But different sectors have experienced different levels of support from KPEP, and accordingly the extent to which the outcomes of increased sales, exports, investment and employment by KPEP’s clients in these sectors are directly attributable to project inputs varies. Table D.2 (in Appendix D) presents KPEP’s percentage attributions in each subsector, which as reported in the body of the evaluation report, range from just over 10% in the road construction sector to nearly 63% in the recycling sector. The average attribution to the project of employment generated in all sectors during the quarter was around 37%. Also as reported in the body of the main evaluation report, these results are somewhat misleading because they are exclusively focused on the direct impacts of KPEP’s assistance to its clients, without consideration of the potential impact of the project in Kosovo more broadly, through encouraging economic agents in the value chains KPEP is supporting to adopt the innovations that have successfully increased sales and employment by its primary clients. In the case of employment generation, for example, the evaluation report states that the “results-level” indicator calculates direct employment generated by the project itself, but it neglects indirect employment, generated via purchases from the relevant supply chain; induced employment, generated via spending by persons employed directly or indirectly by the project; and catalytic employment, generated by improved productivity and performance of the sector due to the innovations made with KPEP’s support. KPEP has endeavored to address this issue by utilizing sector-level multipliers to gauge the broader impact of the project within the Kosovar economy, but the source and justification for the multiplier assumptions is not provided. If anything, the multipliers would appear to underestimate the potential impact these innovations may have in the sectors in which the project is engaged. As discussed in the evaluation report, the key challenge facing KPEP in the final 15 months of the project will be to consolidate and extend the gains that have been achieved through direct support to sector leaders by enabling the broader adoption of value-chain innovations broadly across the sectors in which KPEP’s primary clients are involved. The evaluation team’s assessment is that there is a considerable gap between the output-level results which are the main indicators currently being utilized to measure Component 1’s impact, and ‘context-level” results, 6 USAID, ‘A Modern Workforce Development System is Key to Kosovo’s Growth,’ May 2009, p. 9. Mid-Term Evaluation of KPEP 40 which are at too aggregate a level to really measure the project’s more extended impact. To address this gap, KPEP may consider conducting surveys of its clients, and of a random sample of economic agents within the sectors of interest, to ascertain whether such adoption is occurring. (See Appendix E for an example of where the evaluation team conducted such a survey of KPEP’s grant recipients.) Component 2: Increases in the number of associations and companies providing business services, and of their sales and employment, as well as on an index of sustainable institutional development. The KPEP project is performing well beyond expectations on all the “results-level” indicators it is tracking for this component, and indeed had delivered more than triple the number of jobs that were targeted among BSPs by Q3 FY 2011. The main issues to be addressed had to do with the self-sustainability of the institutions that were being supported, as indicated by the lagging performance of the index designed to measure such performance. As regards trade association development, as mentioned in the body of the evaluation report, to a certain extent the project works at cross-purposes by providing clients with technical assistance and training, as well as grants, while at the same time encouraging BSPs to provide similar services for a fee. A similar observation may be made as regards KPEP’s support for trade associations that are supposed to become self￾sustaining entities, while a number of them have been recipients of both grant and sub-contract support from the project. Component 3: Progress on the BizCLIR Business Environment Index. Compilation of this index for Kosovo, while it may have been a useful tool in the past, may not be as valid an exercise currently, for three reasons. First, the indicators measure performance at a level that is considerably beyond the monitoring of project outcomes; so, for example, KPEP’s recent success in eliminating a non-tariff barrier to exportation to Albania was not captured. Second, KPEP has largely shelved its business enabling environment reform activities, other than those in direct support of its activities within Component 1. Third, USAID’s new Business Enabling Environment Project (BEEP), which has largely supplanted KPEP’s Component 3 activities in Kosovo, utilizes a different index, the World Bank’s ‘Doing Business’ ranking, to target and monitor the results of its activities. Accordingly, the evaluation team believes that the BizCLIR Index might be shelved altogether as an exercise falling within KPEP’s monitoring and evaluation responsibility, in order better to reserve resources for other activities more directly related to KPEP’s revised mandate. Component 4: Number of training certifications and number of program participants that obtain employment. The KPEP project appears to be performing well above the levels targeted by USAID, and in fact in 2011 the target was exceeded by more than double as regards training certifications, and by more than fourfold in respect to the number of program participants obtaining employment. The issue here, once again, is the extent to which the workforce development programs are having a broader impact on unemployment in the sectors in which KPEP is operating, something that the current alignment of results indicators does not capture, a gap that, as with Component 1, may be addressed through follow-up surveys. Mid-Term Evaluation of KPEP 41 APPENDIX C. PERSONS INTERVIEWED USAID KOSOVO Craig Buck, Mission Director Greg Olson, Economic Growth Office Ardian Spahiu, Program and Project Office KPEP Mark Wood, COP Maury Wray Bridges, Deputy COP David Cowles, former COP Artan Avidiu, Workforce Development and BSS Specialist Rhedon Begoli, Competitiveness and ICT Specialist Musli Berrisha, Agricultural Sector Specialist Fisanik Dragusha, Tourism Specialist Sadete Kastrati, Special Activities Fund Manager Burim Meqa, Wood Processing Specialist Arben Musliu, Agribusiness Specialist Nazmi Pllana, Performance-Based Management Specialist Bujar Prestreshi, Construction Specialist Samir Riza, Horticulture Specialist Valbona RRaci, Tourism Specialist Diella Rugova, Workforce Development Specialist Hysen Shabanaj, Forestry Specialist KPEP CLIENTS [Note: The individuals listed below were interviewed in person, usually on-site. In addition to these interviews, the evaluation team did telephone interviews with 44 recipients of KPEP grants. The results of those short interviews are shown in Appendix E.] Lulzim Aliu, Owner, Magic Ice Dairy Products Gani Basha, Manager, Hit Flores Forest Products Imer Berisha, Managing Director, Bylmeti Dairy Products Mustaf Borovcki, Managing Director, Brikos Brick Company Vjollca Cavolli, Manager, Kosova Association of ICT Mid-Term Evaluation of KPEP 42 Musa Cena, Director, Rahovec Promotion Office Gazmend Daka, President, Daka Winery Naim Gacaferri, Gacaferri Wood Products Naser Hoxha, General Manager, Deco Home Furniture Baka Hoti, Manager, Kosovo Alternative Tourism Association Fitim HOti, Director, Hosa Fresh Vegetables Fatos Islami, Manager, ETC Supermarkets Shkelzen Kuqi, Manager, Denisi Logging Krusha e Madhe, President, Perdrini Farmers Association Erton Namoni, Manager, ASK Foods Zef Pjetri, Owner, Konsoni Poultry Gezim Pula, Director, Mea Recycling Muhamed Rexhepi, Director, Kosovo Business Support Centre Mentor Sahiti, Director, Adaptavit IT Agim Shahini, Manager, Kosovo Business Alliance Sheqer Ukaj, General Manager, Helnor Furniture Other Relevant Professionals Albina Berisha, Microenterprise Officer, Community Development Fund Heini Conard, Swiss Cooperation Michael Gold, Managing Director, Crimson Capital Valdrin Lluka, Advisor to the Minister of Trade and Industry Luljeta Rizvanolli, Senior Implementation Officer, Community Development Fund Fatmir Selimi, Deputy COP, USAID New Opportunities in Agriculture Project Terry Slywka, COP, USAID Business Enterprise Environment Project Uran Ismaili, Advisor to the Minister of Agriculture, Food, and Rural Development Arian Zeka, American Chamber of Commerce Mid-Term Evaluation of KPEP 43 APPENDIX D. QUANTITATIVE PROJECT INDICATORS, WITH RESULTS THROUGH JUNE 30, 2011 The following tables and charts present KPEP’s report on its cumulative progress against performance indicators from project inception (2008 baseline) through the third quarter of fiscal year 2011. Table D.1 presents cumulative results as reported by KPEP for its context, results-level and activity-level results, as against annual targets established with USAID. This is followed by a series of charts illustrating the cumulative performance of selected results indicators against targets. Finally, Table D.2 presents KPEP’s analysis, by subsector, of the extent to which project outcomes at the results level can be attributed directly to project activities. Attribution figures by subsector were provided by KPEP; where there are several subsectors, the evaluation team has weighted the attribution percentages by levels (of sales, exports, investment and FTE jobs) in each subsector to arrive at a sectoral attribution percentage. The evaluation team’s assessment of these indicators and results is presented in Appendix B. Mid-Term Evaluation of KPEP 44 SO# IR# SO Name Unit measure Base-line Year Base-line Value 2009 Target 2010 Actual 2011 Target 2011 Actual 2012 Target 2012 Actual 1.3 Indicators 1 (AR) Exports as % of imports, by sector Percent 2008 10.20% 11.50% 12.80% 12.00% 14.57% 13.00% 0.00% 2 Foreign direct investment, by sector Thousand Euros 2008 357,400 451,000 354,300 475,000 354,300 500,000 0 3 Value of sales, by sector Thousand Euros 2008 4,418,000 5,400,000 3,294,933 6,000,000 2,473,112 6,700,000 0 4 Number of people employed, by sector Number 2008 481,000 512,000 517,594 523,000 517,955 534,000 0 1.3.1 Indicators %increase in sales over current baseline Percent 2008 0 16.40% 33.60% 32.80% 41.80% 41.00% 0.00% 2 Increase in investments in target sectors Euros 2008 0 8,000,000 15,256,256 16,000,000 21,304,066 20,000,000 0 3 (AR) Increase in jobs created in target sectors Number 2008 1,801 2,000 1,590 4,000 2,813 5,000 0 4 Number of enterprises assisted in target sectors Number 2008 36 85 66 110116140 0 5 Number of new start-ups in target sectors Number 2008 0 12 663 16 663 20 0 Table D.1: KPMP Performance Monitoring Plan from Q3 FY 2011 Report‐Annual Goals and Results 61 47 7 448 9.40% 25.70% 3,200,000 9,570,686 800 293 20,000,000 41,003,603 40,000,000 59,965,977 50,000,000 0 Increased Competitiveness of Key Sectors 1 Increase in sales in target sectors over the baseline Euros 2008 132,316,729 8,000,000 13,177,698 430,000 291,500 4,900,000 4,552,795 488,000 496,708 2009 Actual 2010 Target Accelerated Growth of Private Sector 11.00% 0.00% Mid-Term Evaluation of KPEP 45 SO# IR# SO Name Unit measure Base-line Year Base-line Value 2009 Target 2010 Actual 2011 Target 2011 Actual 2012 Target 2012 Actual 6 Number of entrepreneurs receiving services supported by USAID Number 2008 0 500 539 TBD 668 TBD 0 1.3.2 Indicators 1 Number of enterprises and associations providing business support services Number 2008 11 30 15 35 16 40 0 2 Increase of sales among BSPs Euros 2008 1,573,702 110,000 752,302 150,000 1,292,060 200,000 0 3 Increase in jobs created among USAID supported BSPs Number 2008 95 18 74 31 110 36 0 4 Progress on the Association Development Index Median score 2008 1.59 1.7 1.65 1.9 1.65 2 0 5 Number of consultants trained/certifie d to provide business support service Number 2008 0150 19 240 19 300 0 1.3.3 Indicators 1 Progress on the Business Environme nt Index (BizCLIR) Table D.1: KPMP Performance Monitoring Plan from Q3 FY 2011 Report‐Annual Goals and Results 70 19 Improved Business Operating Conditions 40,000 236,566 823 1.6 1.65 Improved and Demand Driven Business Support Services 20 15 0 496 2009 Actual 2010 Target Mid-Term Evaluation of KPEP 46 SO# IR# SO Name Unit measure Base-line Year Base-line Value 2009 Target 2010 Actual 2011 Target 2011 Actual 2012 Target 2012 Actual 1.1 Contracts Percent 2008 18% 25% 34% 40% 34% 50% 0% 1.1.1 Supporting Institutions (B.3) Percent 2008 10% 15% 21% 24% 21% 30% 0% 1.1.2 Social Dynamics (B.4) Percent 2008 26% 35% 41% 55% 41% 69% 0% 1.2 Internation al Trade Percent 2008 34% 41% 67% 59% 67% 70% 0% 1.2.1 Legal Framework (H.1) Percent 2008 41% 45% 82% 63% 82% 74% 0% 1.2.2 Implementin g Institutions (H.2) Percent 2008 27% 40% 45% 56% 45% 65% 0% 1.2.3 Supporting Institutions (H.3) Percent 2008 35% 40% 73% 56% 73% 65% 0% 1.2.4 Social Dynamics (H.4) Percent 2008 34% 40% 61% 62% 61% 76% 0% 1.3 Foreign direct investment , by sector Percent 2008 40% 48% 58% 55% 58% 60% 0% 1.3.1 Legal Framework (K.1) Percent 2008 43% 45% 74% 52% 74% 57% 0% 1.3.2 Implementin g Institutions (K.2) Percent 2008 36% 45% 53% 52% 53% 57% 0% 1.3.3 Supporting Institutions (K.3) Percent 2008 54% 60% 76% 70% 76% 76% 0% 1.3.4 Social Dynamics (K.4) Percent 2008 28% 40% 43% 47% 43% 50% 0% 2 Progress on the policy index Percent 2008 0 70% 92% 80% 93% 85% 0 Table D.1: KPMP Performance Monitoring Plan from Q3 FY 2011 Report‐Annual Goals and Results 0% 43% 72% 75% 0% 49% 0% 49% 0% 65% 0% 47% 0% 50% 0% 52% 0% 49% 0% 53% 0% 47% 0% 32% 0% 19% 0% 44% 2009 Actual 2010 Target Mid-Term Evaluation of KPEP 47 SO# IR# SO Name Unit measure Base-line Year Base-line Value 2009 Target 2010 Actual 2011 Target 2011 Actual 2012 Target 2012 Actual 1.3.4 Indicators 1 The number of individuals trained through USAID sponsored training Number 2008 0 700 4,183 2,500 6,290 3,000 0 2 Number of training certifications Number 2008 0 80 252 275 663 325 0 3 Number of program participants that obtain employment Number 2008 0 TBD 135 100 439 150 0 4 Number of individuals provided with services Number 2008 0 800 9,078 3,500 9,449 4,000 0 5 Number of internship programs Number 2008 0 3 13 819100 Indicators 1 Number of grants/sub￾contracts Number 2008 0 20 66 70 167 90 0 2 Value of grants and sub-contracts Euros 2008 0 400,000 1,335,357 2,370,000 3,055,624 2,625,000 0 26 45 447,837 1,750,000 Table D.1: KPMP Performance Monitoring Plan from Q3 FY 2011 Report‐Annual Goals and Results 7,326 1,600 36 Strategic Activities Fund 1,620 1,400 20 160 220 Improved Capacity of the Workforce to Engage in Employment Opportunities 2009 Actual 2010 Target Mid-Term Evaluation of KPEP 48 Mid-Term Evaluation of KPEP 49 Mid-Term Evaluation of KPEP 50 Mid-Term Evaluation of KPEP 51 Mid-Term Evaluation of KPEP 52 Mid-Term Evaluation of KPEP 53 Mid-Term Evaluation of KPEP 54 Mid-Term Evaluation of KPEP 55 Cumulative Cumulative KPEP Cumulative KPEP Cumulative KPEP Employment KPEP Sector/subsector sales attrib.* exports attrib.* investment attrib.* (FTE) attrib.* Construction 20,695 9.8% 10,776 0.1% 11,759 14.4% 598 14.3% Construction materials 2,443 29.2% 126 9.1% 5,363 22.9% 199 22.2% Road construction 18,252 7.2% 10,650 0.0% 6,396 7.2% 399 10.3% Agriculture 21,972 44.9% 13,360 39.8% 5,432 39.8% 1,253 40.7% Dairy 12,414 46.7% 997 46.7% 1,651 43.1% 323 47.9% Fruits & vegetables 10,650 41.7% 5,182 41.7% 2,511 38.4% 425 39.4% Non‐wood forest products ‐1,302 37.4% 7,181 37.4% 1,270 38.4% 367 35.5% Poultry meat processing 210 50.0% 0 50.0% 0 0.0% 138 41.7% Forestry & wood processing 13,594 38.0% 3,454 41.3% 3,618 32.5% 450 32.3% Wood processing 13,281 38.6% 3,454 41.3% 3,294 35.3% 425 33.3% Forestry 313 12.2% 0 0.0% 324 4.3% 25 14.7% ICT 2,202 51.9% 37 20.8% 240 47.8% 161 54.7% Business services providers 1,292 0.0% 0 0.0% 0 0.0% 110 100.0% Tourism 972 60.1% 638 54.0% 236 61.5% 108 61.3% Recycling 80 72.3% 69 41.7% 3 25.0% 5 62.7% Other** 448 8.3% 135 25.0% 15 0.0% 239 25.0% TOTAL 61,255 31.5% 28,469 25.2% 21,303 24.8% 2,924 36.5% Source: KPEP Qua rterly Report, Q3 FY2011, p. 115, and KPEP es tima tes. Note: sectoral a ttributions were calcula ted by KPEP qua rterly and averaged for the purpose of thi s evalua tion; al though BSP res ul ts a re included in total s, a ttributions of BSPs a re not calcula ted by KPEP, and so were not included in total s except for employment, where number assumed to be 100%. * Where applicable, sector averages a re weighted averages of subsector averages ** As reported for Q3 FY2011 Euros (000,s) Table D.2: KPEP Results through Q3 FY2011 Mid-Term Evaluation of KPEP 56 APPENDIX E. RESULTS OF SURVEY OF GRANT RECIPIENTS USAID KPEP Survey Evaluation Results of 44 Respondents # Survey question and optional answers No. of responses Comments 1 Have you received one or more grants from KPEP (and its predecessor project, KCBS)? No 0 Yes, one grant 37 Yes, more than one grant 7 2 Did you also receive training or technical support from the KPEP project? No 20 Yes 24 If, yes was it useful Yes 24 No 0 3 The purpose of the grant(s) was: The were 4 cases in which the grant(s) was/were used for equipment and for other purposes also. New equipment or technology? 33 Other purposes (e.g., trade fairs, participation, quality standards, etc)? 15 4 Following your receipt of the grant(s), have other firms in your sector adopted similar innovations (e.g., purchase of similar equipment, or efforts to meet quality standards) to that facilitated by the grant(s)? Yes 10 No 14 Not aware 20 5 How effective have the grant(s) been in increasing your sales and profits? There were 12 cases which have not finished the implementation of the project covered by the grant, hence no answers could be given on questions 5 and 6. Very effective 31 Not effective 1 One case of lack of own funding to finish the project In process 12 6 If you had to do it over, and the grant(s) were not available, now that you have seen the results of the innovations that they facilitated, would you have funded these investments yourself? Yes 31 No 0 Innovations still in process 13 Mid-Term Evaluation of KPEP 57 APPENDIX F. REFERENCES J.E. Austin Associates, “ITC as a Catalyst to Enterprise Competitiveness,” 2010. Chemonics International for USAID/Kosovo, 2005. Doing Business in Kosovo, 2005. International Monetary Fund, “Republic of Kosovo: 2011 Article IV Consultation and the Initiation of a Staff-Monitored Program, August 2011. Hajdari, 2009. “Baseline Report: Association Development Index as applied to selected Business Asociations in Kosovo, STTA Report for KPEP, January 2009. The Mitchell Group, “Promoting Competitiveness in Practice: An assessment of Cluster-Based Approaches,” Report for USAID, 2003. KPEP, annual reports, work plans, quarterly reports through July 2011, and numerous other documents and special studies. Snodgrass, Donald, and James Winkler, Enterprise Growth Initiatives: Strategic Directions and Options, Report for USAID/EGAT, 2004. World Bank, Doing Business in 2011.