MID-TERM PERFORMANCE EVALUATION FOR PRIVATE SECTOR DEVELOPMENT IN SOUTH AND SOUTHWEST SERBIA PROJECT (PSD) FINAL EVALUATION REPORT April 2016 This publication was produced for review by the United States Agency for International Development. It was prepared independently by SEGURA Consulting LLC, under the Midterm Evaluation for Private Sector Development in South and Southwest Serbia Project (PSD) under the Monitoring & Evaluation IDIQ. 1 MID-TERM PERFORMANCE EVALUATION FOR PRIVATE SECTOR DEVELOPMENT IN SOUTH AND SOUTHWEST SERBIA PROJECT (PSD) FINAL EVALUATION REPORT Prepared under Task Order AID-169-TO-16-00001, as part of the EVAL-ME IDIQ. Submitted to: USAID Serbia April 2016 Prepared by: Ronald Ivey, Team Leader/ Private Sector Analyst Brankica Obućina, Regional Economic Development Analyst Submitted by: SEGURA Consulting LLC 6110 Executive Boulevard, Suite 512 Rockville, MD 20852 T: (301) 469-4724 www.seguraconsulting.net DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 2 CONTENTS LIST OF TABLES AND FIGURES .................................................................................................................................................. 4 LIST OF ACRONYMS ................................................................................................................................................................................. 4 ACKNOWLEDGEMENTS ...................................................................................................................................................................... 5 EXECUTIVE SUMMARY .......................................................................................................................................................................... 6 I. PURPOSE OF THE EVALUATION .......................................................................................................................... 11 II. PROJECT BACKGROUND AND RATIONALE ........................................................................................... 12 III. EVALUATION METHODOLOGY ............................................................................................................................. 14 IV. ANSWERS TO THE EVALUATION QUESTIONS .................................................................................. 16 Specific Questions Posed by USAID......................................................................................................................................................16 I. To what extent has PSD achieved the expected overarching Mission-level results identified in the Activity Monitoring and Evaluation Plan?...........................................................................................................16 II. To what extent has the project resulted in improved competitiveness of SMEEs?...........................25 III. To what extent has the project contributed to increased linkages between small businesses and regional/national governmental support organizations?...............................................................................30 IV. How effective was PSD in creating the linkages among different PSD partners (including sub￾recipients) and across different mechanisms?................................................................................................................31 V. What have been the strengths and weaknesses of USAID’s approach to G2G management? . 34 V. RECOMMENDATIONS FOR the REMAINDER OF the PSD PROJECT .............................. 37 I. Drawing on PSD lessons learned, how should USAID best focus or target its limited assistance for the strengthening of the private sector businesses in targeted regions in the future?.........................................................................................................................................................................................................37 II. How can USAID Serbia strengthen the linkages between different PSD partners during phase II?.....................................................................................................................................................................................................40 ANNEXES ............................................................................................................................................................................................................ 41 Annex 1. Questionnaire Formats .................................................................................................................................................. 41 Annex I1. List of Conducted Interviews .................................................................................................................................. 62 Annex III. PMP Results and Targets ........................................................................................................................................... 69 Annex IV. Statement of Work ......................................................................................................................................................... 73 3 LIST OF TABLES AND FIGURES Table 1. Overarching Indicators of the PSD Project...............................................................................................16 Table 2. Conclusions on the Overarching Indicators.............................................................................................24 Table 3. Indicators on Strengthening SMEEs.............................................................................................................25 Table 4. Conclusions on Indicators on Strengthening SMEEs...............................................................................29 Table 5. Indicators on Strengthening Associations/Clusters.................................................................................32 Table 6. Conclusions on Indicators on Strengthening Associations/Clusters...................................................34 Figure 1. Breakdown of interviews by sectors.........................................................................................................15 Figure 2. Breakdown of interviews by type of clients.............................................................................................15 LIST OF ACRONYMS CDCS Country Development Cooperation Strategy DAS Development Agency of Serbia G2G Government to Government MoU NARD Memorandum of Understanding National Agency for Regional Development OBR Output Based Reimbursement PMP Activity Performance Monitoring Plan PSD Private Sector Development in South and Southwest Serbia Project RDA Regional Development Agency SIEPA Serbia Investment and Export Promotion Agency SOW Statement of Work SWOT USAID Strengths Weaknesses Opportunities and Threats (analysis) U.S. Agency for International Development 4 ACKNOWLEDGEMENTS The Evaluation Team wishes to thank the staff of USAID/Serbia and the Private Sector Development in South and Southwest Serbia (PSD) Project for providing necessary background documentation, being open and responsive to questions, arranging meetings with implementing partner representatives and beneficiaries, and providing comments on the draft Mid-Term Evaluation report. The Evaluation Team further extends its thanks to the representatives of national, regional, and local governments; industry associations; founders and employees of small and medium enterprises; and other beneficiaries who gave so generously of their time to participate as members of group or individual interviews. Their insights proved invaluable in shaping this report. 5 EXECUTIVE SUMMARY The following report is based on the mid-term performance evaluation of the first phase of the Private Sector Development in South and Southwest Serbia (PSD) Project. The evaluation was conducted by SEGURA Consulting LLC in January and February 2016, pursuant to a Task Order award under the EVAL-ME IDIQ. The Statement of Work for the evaluation was developed by USAID/Serbia and addressed by a team of two SEGURA consultants – a U.S. based team leader (Ronald Ivey) and a local economic development analyst (Brankica Obucina). During the evaluation, the team conducted more than 40 individual and group interviews with project staff and program beneficiaries in all 12 target municipalities in South and Southwest Serbia. COUNTRY/PROJECT OVERVIEW Although Serbia is well on the path of transitioning to a market economy, it has struggled to achieve balanced economic and regional development in many parts of the country. The issues are perhaps most acute in South and Southwest Serbia. The South Serbia and Sandžak regions are highly important for the country’s stability, as they have ethnically diverse populations struggling to overcome a recent history of regional ethnic conflicts. The economic underdevelopment of these two regions has led to high outmigration to either Central Serbia or abroad, as well as significant rural poverty in more remote locations. USAID/Serbia aimed to assist the private sector development in these two regions through programs preceding the PSD Project, namely through the USAID Preparedness, Planning and Economic Security Program (PPES) and the USAID Agribusiness Project. The PPES activities focused in particular on supporting further development of businesses in the textile/fashion and agribusiness sectors in Sandžak region and South Serbia. Although the USAID Agribusiness Project activities were initially implemented throughout the country, towards the end of that project specific technical assistance was designed for South Serbia to mitigate the factors limiting development of agribusinesses in underdeveloped municipalities. The PSD Project started on June 21, 2013 -- implemented by the Serbian National Agency for Regional Development (NARD) and its four counterpart Regional Development Agencies (RDAs) -- as one of the first Government-to-Government (G2G) USAID projects in Europe. The PSD Project was designed to support the development of the private sector in two of the poorest regions of Serbia, while serving as a learning laboratory for transitioning to a new development paradigm as envisioned by USAID Forward/Local Solutions. The PSD Project was designed as a four-year, $6.5 million USAID initiative to: a) promote small and medium enterprise and entrepreneurship development; and b) increase sales, promote exports and create new jobs in agribusiness, light industry and fashion (textiles and shoes) in 12 municipalities in the struggling economic regions of South and Southwest Serbia. Specifically, PSD works in the municipalities of Vranje, Leskovac, Preševo, Bujanovac, Medvedja, Raška, Novi Pazar, Sjenica, Tutin, Priboj, Prijepolje and Nova Varoš. The PSD Project was implemented by the National Agency for Regional Development (NARD), until its dissolution in late December 2015, through its network of accredited Regional Development Agencies (RDAs), namely: • Regional Development Agency – Leskovac (Center for Development of Jablanica and Pčinja Districts) 6 • Regional Development Agency – Kraljevo (Regional Agency for Spatial and Economic Development of Raška and Morava Districts) • Regional Development Agency – Sandžak (SEDA) • Regional Development Agency – Zlatibor USAID set four project-level objectives for the PSD Project: 1) strengthening small-to-medium sized enterprises and entrepreneurs; 2) strengthening business associations/clusters; 3) encouraging women and youth entrepreneurship; and 4) enhancing business development services of NARD and RDAs. To accomplish these objectives, the PSD Project employed a multi-pronged approach to private sector development • working with businesses (SMEEs, associations/clusters and start-ups) in three high-potential sectors: agribusiness, fashion/textile and light manufacturing (Component 1, implemented by NARD); • working with the Government of Serbia’s private sector development entities to open markets and create linkages (Component 2, implemented by NARD); and supplementary assistance to NARD and the RDAs to enhance SME competitiveness (Component 3, implemented by USAID/Serbia). A major change in PSD Project implementation occurred in December 2015 -- just prior to this Evaluation -- as the Government of Serbia decided to consolidate NARD and the Serbia Investment and Export Promotion Agency (SIEPA) into a newly-formed Development Agency of Serbia (DAS). Although much of the PSD Project core staff will likely be absorbed into the new agency, that process was not yet formally completed by the time of this evaluation. OVERVIEW OF EVALUATION METHODOLOGY Prior to the field evaluation work, the Evaluation Team gathered relevant information and data about the PSD Project by familiarizing themselves with the PMP; Year 1 PSD Project Work Plan; Year 1 annual performance report of NARD and the RDAs; USAID Serbia’s Amended Strategy and 2013-2017 CDCS; and the Economic Growth Assessment for Southwest and South Serbia. The Evaluation Team also reviewed the G2G Agreement establishing the implementation framework for the PSD Project. The online PSD Project database provided insight into more detailed PSD Project documentation, including Client Impact Survey Results, evaluation criteria forms for selection of clients, analysis and assessment reports on the assistance needs of clients, as well as technical documents on different assistance programs for all categories of clients (SMEEs, associations/clusters, and youth and women start-ups). Review of the PSD Project documentation enabled the Evaluation Team to prepare three different types of questionnaires (available in Annex 1) that were used for field interviews with PSD clients and implementing agencies. After the four implementing RDAs provided lists of PSD Project clients from their respective regions, the Evaluation Team made a random selection of 43 potential interviewees across all three sectors and types of beneficiaries. The Evaluation Team convened in Serbia for three weeks of in-country work, starting with a debriefing with USAID Mission staff and project management. After interviewing project staff, the Evaluation Team embarked on a grueling two-week trip (aided by an experienced translator and adept driver) in the Serbian winter to reach all target municipalities in South and Southwest Serbia. Along the way, the Team interviewed representatives from 25 SMEEs, four associations/clusters, and five start-ups helmed by women or youth. All interviews, with the exception of those with the NARD and RDAs, were conducted primarily in Serbian with English translation. 7 Once the field surveys were complete, the Evaluation Team prepared an out-briefing presentation for USAID/Serbia Mission management and the PSD technical staff. Upon return from the field, the Evaluation Team submitted a draft evaluation report for USAID comment. This final report is the product of those observations from USAID and further refinement by the Evaluation Team. SUMMARY OF FINDINGS AND CONCLUSIONS The first evaluation question asked the Evaluation Team to determine to what extent PSD achieved the expected Overarching Mission-level results set forth in the Activity Monitoring and Evaluation Plan. Those five indicators included the following: • the value of annual revenues of USAID client companies in targeted regions • the number of jobs created in targeted regions • the number of business services increased in the targeted regions • the number of beneficiaries with increased competitive skills • the number of firms receiving capacity building assistance with export The Evaluation Team found that the PSD Project did achieve or exceed all five Mission-level indicators. Notably, the annual revenue increase of nearly 8% within an economically challenging business environment is impressive. The job creation indicators – though surpassed -- were set at a very low level and should be increased for the next project phase. The number of business services as an indicator was not particularly meaningful, as it was set low and spoke only to whether services were offered by NARD and RDAs and not to what extent they had been accessed by beneficiaries. In exceeding the target number of beneficiaries with increased competitive skills, respondent opinions were most favorable regarding training and technical assistance opportunities. The provision of mentoring/consulting was noted as a potential area for improvement and expansion in the next project phase. While export training was positively received as a means of capacity building assistance, trade fair support was singled out as an area that can be revamped or better calibrated for even more positive development in the future. The second evaluation question asked to what extent the PSD project resulted in improved competitiveness of SMEEs. Applicable performance indicators included the following: • the number of SMEEs receiving USAID assistance • the number of USAID client SMEEs entering new markets • the number of USAID client SMEEs exhibiting at international trade fairs • the value of contracts signed by USAID client SMEEs exhibiting at international trade fairs • the number of USAID client SMEEs introducing international standards or certifications • the number of USAID client SMEEs adopting new technologies or management practices PSD Project success on this group of indicators was generally more of a mixed bag. The number of SMEEs receiving assistance (183) exceeded the PMP target; however, data provided by NARD to the Evaluation Team suggests that fewer companies were able to sell in new markets than hoped. Survey results indicated that only 23.5% of the respondents who attended trade fairs entered new markets as a result, and fewer SMEEs participated in trade fairs than targeted. However, the value of sales contracts at international trade fairs ($355,000) significantly exceeded the $250,000 trade fair sales target. While this indicator was technically met, it should be increased in the next project phase. The introduction of 20 international standards (usually quality and environmental certifications) solidly exceeded the target of 12 standards. The adoption of 50 new technologies or management practices under the Technical Assistance component was one of the more popular and cost-effective aspects of the PSD program. 8 The third evaluation question asked to what extent the PSD Project contributed to increased linkages between small businesses and regional/national governmental support organizations. This was one of the clear strengths of the PSD Project and should be a continued focus in the next phase. The level of appreciation for, willingness to cooperate with, rely on, and trust the RDAs was supremely high among respondents. Respondents particularly appreciated the on-going communication of opportunities for their businesses. Another strong point of emphasis was the extent to which SMEEs felt that PSD calls for grant applications (versus similar grant competitions sponsored by the government) really emphasized participation by small businesses rather than favoring and ultimately rewarding only larger, more well-established businesses. It seemed clear that even though not all PSD interventions were equally successful, there was a consensus from respondents that PSD was working hard for their benefit. This is a perfect example of the value of the FORWARD Initiative and unique strength to the G2G aspect of PSD. As mentoring and consulting were viewed as overly general, RDAs should work to better tailor their advice to the particular requirements of individual enterprises. The fourth evaluation question asked the Evaluation Team to gauge how effective the PSD Project was in creating linkages among different PSD partners (including sub-recipients) and across different mechanisms. This question had six indicators as follows: • the number of associations receiving USAID assistance • the value of joint revenues of USAID client associations/clusters • the number of new members joining USAID association/clusters • the value of contracts signed by USAID client associations/clusters at international trade fairs • the number of USAID client associations/clusters exhibiting at international trade fairs • the number of trained managers of USAID client associations/clusters Relative to the stated indicators, this is one potential area for improvement in the next phase of the PSD Project. It should be noted first that PSD staff did work with double the target number of associations and clusters, and that those associations/clusters receiving assistance showed an 11.5% annual growth rate. However, very few members joined USAID associations/clusters and associations/clusters were not very successful in exhibiting at trade fairs or earning contracts from their participation at such fairs. Additionally, while the number of managers trained exceeded the Year 1 target, it did not exceed the combined target for Years 1 and 2. Given the unique government-to-government (G2G) nature of the PSD Project, the important fifth evaluation question was for the Evaluation Team to assess the strengths and weaknesses of USAID’s approach to G2G management. In general, the Evaluation Team felt that the G2G approach was a real benefit for PSD. This unique model operated with a high-level of efficiency, transparency, and professionalism. The project is a strong marriage between USAID contractual requirements and operating principles, NARD’s project methodology, and the RDA as highly competent local networks. This cost-effective model should be replicated for use in other developing countries. While political realities can affect a G2G program in ways that they might not affect private contractors, the benefits of a novel approach that thoroughly advances USAID’s FORWARD Initiative cannot be overlooked. SUMMARY OF SELECTED RECOMMENDATIONS While several aspects of the first phase of PSD project implementation have been successful, the Evaluation Team was able to provide some recommendations for future implementation. These are discussed in more detail in section V below, but a selection is summarized here: 9 • Adjust the employment creation and trade fair sales indicators upwards to challenge project implementers and re-calibrate them to better reflect the needs of SMEEs in the targeted regions. Consider adding performance indicators related to the number of exports generated and availability of bank financing. • Develop a future call for technical assistance that adjusts SMEE selection criteria to give higher emphasis to those enterprises involved in strategic growth activities or value-chains as lead firms. These criteria may result in the selection of fewer enterprises, but should hopefully result in more focused, targeted assistance to those enterprises best poised to substantially expand sales, increase hiring, and export their products more widely. • Consider adding highly-qualified financial specialists to the PSD team to assist individual firms that are looking to obtain bank financing in order to accelerate their growth • Maintain a focus on trade fair participation, but consider a multi-stage approach that aligns SMEEs or clusters/associations with the trade fairs that are most appropriate to their lines of productions, quality of products, and language capabilities. Ensure that interested enterprises first get familiar with the demands of trade shows before they bring samples, attend B2B events, or seek out contract partners. • Consider bringing in regional or international trade show experts to provide courses to RDA staff to best prepare them to engage with businesses. Similarly, move forward with the ITC export and market research trainings which were planned as part of the practical module but not yet implemented. • Continue to provide SMEE trainings as a means of raising the business acumen in the region and maintaining the strong relationship of trust between businesses and RDAs, but consider calibrating trainings more carefully to the individual needs of the participants. For example, permit business owners who have been in business for a long time or have undergraduate/graduate business training to skip more basic trainings. • To the extent that funds allow, continue utilizing “access to technology” grants for SMEEs and machinery/equipment grants for start-ups. 100% grant funding to youth and women start-ups is great, but consider working with a greater number of female-owned startups. • To ensure start-up sustainability, expand the mentoring and consulting relationships past just the 50 and 5 hour-requirements, respectively, that are currently in place. • Consider a value chain approach to strengthening associations and clusters. One method would be a call for applications that places particular companies as “lead firms” that will map/implement development, and provide technical support to their suppliers to raise product quality. This could be most useful with dairies, honey producers, fruit growers, and grain producers. Another approach is for association/clusters to themselves take on the role of a “lead firm” that provides quality control and provide joint marketing and branding. • Continue to rely heavily on RDAs as trusted providers of critical ground-level assistance to Serbian enterprises, especially in light of the recent and as-of-yet not fully finalized transition from NARD to DAS. 10 I. PURPOSE OF THE EVALUATION This mid-term performance evaluation report was prepared by SEGURA Consulting LLC on the first phase of the Private Sector Development in South and Southwest Serbia Project implemented by the Republic of Serbia’s National Agency for Regional Development (NARD). The Evaluation’s objective was to determine the extent to which the PSD Project, launched in June 2013, achieved intended development results related to strengthening the capacity of the private sector in targeted regions. This included a review of higher Mission-level results referred to in CDCS IR 2.2 (Economic Opportunities in Targeted Regions) and the extent to which the activities undertaken supported the achievement of those higher-level results. The mid-term performance evaluation was structured to answer the following specific questions with respect to PSD Project performance: Specific Questions Posed by USAID • To what extent has PSD achieved the expected overarching Mission-level results identified in the Activity Monitoring and Evaluation Plan? • To what extent has the project resulted in improved competitiveness of SMEEs? • To what extent has the project contributed to increased linkages between small businesses and regional/national governmental support organizations? • How effective was PSD in creating the linkages among different PSD partners (including sub￾recipients) and across different mechanisms? • What have been the strengths and weaknesses of USAID’s approach to G2G management? The Statement of Work also asked that two specific recommendations be made: • Drawing on PSD lessons learned, how should USAID best focus or target its limited assistance for the strengthening of the private sector businesses in targeted regions in the future? • How can USAID Serbia strengthen the linkages between different PSD partners during phase II? To answer these questions and make these recommendations, the Evaluation Team reviewed activities, documentation and results information from the first phase of the PSD Project. The Activity Performance and Monitoring Plan (PMP) final data were not initially available for review; however, NARD helpfully provided the Evaluation Team the compiled results for the project as per the latest Client Impact Survey (as of July 2015). The Evaluation Team was also granted access to the PSD Project database, which enabled review of all key PSD Project documents and reports. The report was designed for presentation to USAID/Serbia, including both Mission management and technical staff involved in management of the project. USAID/Washington and other E&E Bureau personnel constitute a secondary audience for this evaluation report. 11 II. PROJECT BACKGROUND AND RATIONALE Although Serbia is well on the path of transitioning to a market economy, it has struggled to achieve balanced economic and regional development in many parts of the country. The EC Progress Report from 2012 demonstrated that Serbia’s GDP level still lagged significantly behind the country’s performance at the end of 1980s, explicitly emphasizing the need for “further state commitment to economic development in South Serbia and Sandžak.” It should be noted that during the first phase of the PSD Project, Serbia was technically in recession. Although regional development strategies were developed at the state and local level, GDP growth was at -1.8% (2014) and 1.2% (2015)1, while the rate of unemployment exceeded 20%. The issues are perhaps most acute in South Serbia, where unemployment reached almost 70%. The South Serbia and Sandžak regions are highly important for the country’s stability, as they have ethnically diverse populations struggling to overcome a recent history of regional ethnic conflicts. The Office for Sustainable Development of Underdeveloped Areas classified 40 Serbian municipalities as economically “devastated” (meaning where their development was measured at less than half the national average), with the greatest concentration of these municipalities in the South Serbia and Sandžak regions. The economic underdevelopment of these two regions has led to high outmigration to either Central Serbia or abroad, as well as significant rural poverty in more remote locations. USAID/Serbia aimed to assist the private sector development in these two regions through programs preceding the PSD Project, namely through the USAID Preparedness, Planning and Economic Security Program (PPES) and the USAID Agribusiness Project. The PPES activities focused in particular on supporting further development of businesses in the textile/fashion and agribusiness sectors in Sandžak region and South Serbia. Although the USAID Agribusiness Project activities were initially implemented throughout the country, towards the end of that project specific technical assistance was designed for South Serbia to mitigate the factors limiting development of agribusinesses in underdeveloped municipalities. With the institutionalization of the USAID Forward approach, the PSD Project was developed in the context of a major re-alignment of the private sector in the poorest regions of Serbia. The PSD Project design was to a large extent unique for Serbia -- creating synergy and fully utilizing existing knowledge, practices, procedures and staff of Serbia’s Regional Development Agencies engaged in the private sector development and USAID’s goals, policies and procedures. It also constituted one of the first Government-to-Government USAID programs in Europe. Project implementation focused on delivering the 67 milestones defined in the PSD Project’s Statement of Work. The PSD Project was designed as a four-year, $6.5 million USAID initiative to: a) Promote small and medium enterprise and entrepreneurship development; and b) Increase sales, promote exports and create new jobs in agribusiness, light industry and fashion (textiles and shoes) in 12 municipalities in the struggling economic regions of South and Southwest Serbia. Specifically, PSD works in the municipalities of Vranje, Leskovac, Preševo, Bujanovac, Medvedja, Raška, Novi Pazar, Sjenica, Tutin, Priboj, Prijepolje and Nova Varoš. The PSD Project was implemented by the National Agency for Regional Development (NARD), until its dissolution in late December 2015, through its network of accredited Regional Development Agencies (RDAs), namely: 11 Official GDP rates based on National Bank of Serbia statistics. 12 • Regional Development Agency – Leskovac (Center for Development of Jablanica and Pčinja Districts) • Regional Development Agency – Kraljevo (Regional Agency for Spatial and Economic Development of Raška and Morava Districts) • Regional Development Agency – Sandžak (SEDA) • Regional Development Agency – Zlatibor USAID set four project-level objectives for the PSD Project: 1) strengthening small-to-medium sized enterprises and entrepreneurs; 2) strengthening business associations/clusters; 3) encouraging women and youth entrepreneurship; and 4) enhancing business development services of NARD and RDAs. To accomplish these objectives, the PSD Project employed a multi-pronged approach to private sector development • working with businesses (SMEEs, associations/clusters and start-ups) in three high-potential sectors: agribusiness, fashion/textile and light manufacturing (Component 1, implemented by NARD); • working with the Government of Serbia’s private sector development entities to open markets and create linkages (Component 2, implemented by NARD); and • supplementary assistance to NARD and the RDAs to enhance SME competitiveness (Component 3, implemented by USAID/Serbia). Prior to the launch of the PSD Project, USAID contracted Nathan Associates, Inc. to implement the Economic Growth Assessment for Southwest and South Serbia. The Assessment recommended three high-potential sectors: agribusiness (in particular dairy, meat processing, and vegetables), fashion (textiles, apparel and shoes) and light manufacturing (wooden and aluminum joinery, doors, windows, bathroom fixtures, scaffolding). Additionally, the Assessment recommended utilizing (to the extent possible) the sector/value chain approach for activities, focusing on the non-agricultural sector for quick returns on investment and the agricultural sector for long-term results, and prioritizing the choice of partners with the greatest potential to affect sales and growth. Based on documents reviewed by Evaluation Team, we note that the PSD Project performed value chain analyses of the private sector to select high-potential sectors (agribusiness and fashion/textile and light manufacturing industry sectors) and determine priority support activities. These three sector/value chain analyses were developed based on the implemented interviews and workshops with key "stakeholders" in selected intervention municipalities. The RDAs identified potential private sector project clients and inquired as to their needs, while the final selection of SMEE clients was conducted by NARD and independent consultants using pre-determined criteria to ensure impartiality. The PSD Project clients (SMEEs, associations/clusters, youth and women start-ups) were offered a mixed package of assistance, which encompassed some combination of: 1) business training courses, 2) mentoring and consulting by RDAs; 3) technical assistance with cost-sharing grants to enhance their businesses operations (50%-50% for SMEEs, and 80%-20% for associations/clusters); 4) trade fair attendance and export training; and 5) 100% grants for start-ups to purchase machinery. The Evaluation Team specifically assessed the effect of these efforts by directly interviewing project clients who had received the combined assistance, and evaluated whether and to what extent the technical assistance and grants achieved tangible results. It should be noted that the PSD implementation approach was slightly changed from the initial G2G agreement, as the implementation of the Component 2 was originally envisioned as a role for the Serbia Investment and Export Promotion Agency (SIEPA). This Component was ultimately implemented by 13 NARD/RDAs. Additionally, the Coordination Body for the Municipalities of Preševo, Bujanovac and Medveđa, originally incorporated to assist with the PSD Project implementation, ultimately had a more minor role in the overall project. The major change in PSD Project implementation occurred in December 2015 -- just prior to this Evaluation -- as the Government of Serbia decided to consolidate the NARD and SIEPA into a newly￾formed Development Agency of Serbia (DAS). Although much of the PSD Project core staff will likely be absorbed into the new agency, that process was not yet formally completed by the time of this evaluation. This transition has created some uncertainty as to project implementation, as roughly a dozen project milestones remain and some already-initiated activities (awards to grantees, disbursements of equipment and funds to SMEEs, etc.) have not yet been completed. The Evaluation Team was made aware that NARD operations were being transferred to the DAS and strived to capture inputs from implementing partners NARD/RDAs regarding future requirements for the PSD Project. Despite the newness and significance of this transition, the Evaluation Team felt that NARD/RDA input was quite objective and thoughtful, allowing for solid recommendations for the DAS (or any other implementing partner) selected to implement the latter phase of the PSD Project. III. EVALUATION METHODOLOGY The evaluation was conducted by SEGURA Consulting, featuring a two-person evaluation team. Ronald Ivey served as Team Leader and Private Sector Analyst, along with Brankica Obucina as the Serbian Regional Economic Development Specialist. They were capably supported by a translator (Branislava Jurasin) and driver. In line with the requirements of the SOW, the Evaluation Team convened in Serbia on January 27, 2015, for three weeks of in-country work. A Work Plan for the evaluation mission was submitted to USAID/Serbia prior to arrival of the Team Leader in Belgrade. As indicated in the Work Plan, the Evaluation Team undertook a comprehensive review of available PSD Project-related documentation before conducting interviews with key stakeholders, including: PSD Project staff from NARD and all four implementing RDAs, randomly selected sector SMEEs, associations/clusters, and youth and women start-up businesses. Prior to the field evaluation work, the Evaluation Team gathered relevant information and data about the PSD Project by familiarizing themselves with the PMP; Year 1 PSD Project Work Plan; Year 1 annual performance report of NARD and the RDAs; USAID Serbia’s Amended Strategy and 2013-2017 CDCS; and the Economic Growth Assessment for Southwest and South Serbia. The Evaluation Team also reviewed the G2G Agreement establishing the implementation framework for the PSD Project. The online PSD Project database provided insight into more detailed PSD Project documentation, including Client Impact Survey Results, evaluation criteria forms for selection of clients, analysis and assessment reports on the assistance needs of clients, as well as technical documents on different assistance programs for all categories of clients (SMEEs, associations/clusters, and youth and women start-ups). Review of the PSD Project documentation enabled the Evaluation Team to prepare three different types of questionnaires (available in Annex 1) that were used for field interviews with PSD clients and implementing agencies, namely: • Questionnaire for NARD and RDAs • Questionnaire for associations and clusters • Questionnaire for SMEEs (including youth and women start-ups) The four implementing RDAs provided lists of PSD Project clients from their respective regions, from which the Evaluation team made a random selection of 43 potential interviewees. This random selection process provided a balance of clients across all three sectors and types of beneficiaries supported by the 14 PSD Project. The largest category for interviews was SMEEs (25 SMEEs interviewed out of 121 in Year 1 and additional 62 in Year 2) as they have been the primary focus group of the PSD Project. Out of eight PSD Project client associations/clusters, the Evaluation Team met with four associations to assess the PSD Project assistance and gain deeper knowledge about sector development context and needs. Additionally, the Evaluation Team met with five out of a total of 36 youth and women start-ups. The Evaluation Team also met with NARD and all four implementing RDAs to capture specific insights into their assistance within the regional context of the PSD Project. The RDAs from Leskovac and RDA Sandžak (SEDA) facilitated interviews with PSD Project clients on their premises, due to the remote locations of some selected clients. The Evaluation Team met with PSD project clients from all 12 municipalities (on average three private sector clients per municipality). One participating textile sector SMEE was ultimately unavailable for an interview, as was the representative of the Ministry of Economy. A list of interviewed clients, along with specifics on their type of business, length of time in business and PSD Project assistance provided is available in Annex 2. Figures 1 and 2 below show the breakdown of interviews conducted by sectors and types of clients. Figure 1. Breakdown of interviews by sectors Figure 2. Breakdown of interviews by type of clients Prior to initiating fieldwork and meeting PSD Project participants, the Evaluation Team conducted an USAID briefing explaining this methodology. The questionnaires for the interviews were designed to elicit information and data concerning the implementation effectiveness of the PSD Project. All interviews, with the exception of those with the NARD and RDAs, were conducted primarily in Serbian with English translation. This enabled both Evaluators to ask follow-up questions and capture interviewees’ remarks. The SMEE questionnaire was revised after an initial test to ensure that it could be administered in about 1.5 hours. The most notable challenge was some resistance to providing quantitative information (sales data, sales forecasts, the exact number of employees at the enterprise, etc.). Otherwise, interviewees spoke quite freely about their businesses, any challenges faced, and what they had learned and achieved as PSD project participants. The interview information was collated by the Evaluation Team on a rolling basis and then finalized upon the conclusion of fieldwork. Once the field surveys were complete, the Evaluation Team prepared a debriefing presentation for USAID/Serbia Mission management and the PSD technical staff. Quantitative cross-referencing was done with the Year 1 and Year 2 indicators, which were delivered to the Evaluation Team just prior to the USAID debriefing meeting. 15 In preparing the USAID debriefing and the final report, the Evaluation Team referred to the overarching and specific questions mentioned in the SOW. Answers to these questions are recorded in Section IV below. Additional questions used in field interviews are also addressed within the body of the report, as a means of drawing specific attention to some key findings and recommendations. IV. ANSWERS TO THE EVALUATION QUESTIONS As noted above, one key purpose of the mid-term evaluation was to answer the Evaluation Questions posed in the Statement of Work. Thus, this section is organized around the responses to these questions. SPECIFIC QUESTIONS POSED BY USAID I. TO WHAT EXTENT HAS PSD ACHIEVED THE EXPECTED OVERARCHING MISSION￾LEVEL RESULTS IDENTIFIED IN THE ACTIVITY MONITORING AND EVALUATION PLAN? We reviewed and considered the results presented in the Activity Monitoring and Evaluation Plan for the PSD Project. Table 1 summarizes the baseline, targets and actual results provided by NARD to the Evaluation Team. The Evaluation Team’s analysis and the respondents’ comments follow below. Table 1. Overarching Indicators of the PSD Project Overarching Indicators Baseline Y1 Result (Jan￾Oct) Y1 Target Year 2 Result (Nov￾July) Y2 Target I-1 Value of annual revenues of USAID client companies in targeted regions $141 M $124 M $145 M $126.79 M $152 M I-2 Number of jobs created in targeted regions 5,570 59 50 361 50 I-3 Number of business services in targeted regions 8 2 2 2 2 I-4 Number of beneficiaries with increased competitive skills n/a 318 324 165 144 I-5 Number of firms receiving capacity building assistance n/a 44 20 n/a 10 Value of Annual Revenues. The actual revenues per USAID client companies can be calculated by dividing the Revenues by the number of SMEEs in the program by their total revenues. At the start of the program the 122 companies had a reported baseline of $141 million in sales, or an average of $1,155,737 in sales revenue per company. The Year 1 target was an increase of total sales to $145 million, an average of $32,787 per firm equivalent to 2.8% growth. The PSD Project results were only provided for nine months of Year 1, with an actual result of $124 million for nine months in Year 1 which the Evaluation Team estimated (via linear extrapolation) at $165 million for Year 1 or $1,355,191 per company. This would represent an estimated annual increase of 17.25%. Thus, we can conclude that Year 1 revenue targets were exceeded by the PSD Project. The Year 2 target was $152 million, with the estimated result (again linearly extrapolating for the entire Year 2, based on the 9 month survey results) 16 being $169,053,333, again surpassing the Year 2 target, and demonstrating growth of 2.4% compared to the Year 1 actual result. Thus, PSD beneficiaries’ revenues grew at approximately 8% over the first phase of the project. The Evaluation Team considers the data from the Client Impact Surveys reliable because, as per information provided by NARD, company revenue data were cross-referenced with official data from the national Agency for Business Register (ABR). Further, the revenue growth estimate matched sales growth estimates of around 10-15% for 2016, as reported by those SME respondents who actually responded to the question about their revenues. The Evaluation Team feels that this revenue growth is impressive in light of the economic context in the region, and that USAID/Serbia exceeded this Overarching Indicator. A few notes to this conclusion should be mentioned. First, some of the interviewed companies have been in business over 20 years, so it was difficult to determine whether to properly attribute revenue growth to PSD implementation efforts. Second, it proved difficult at times to cross-reference sales statistics with survey responses, as some respondents were reluctant to provide either current or historical quantitative sales data. It is possible then that revenue growth is actually understated as a result of underreporting by SMEEs (either due to taxation concerns or the strong presence of the “gray economy” in the country.) Number of Jobs Created. The baseline for the 122 SMEEs was 5527 jobs or an average number of jobs per company of 45. This is approximately in line with the average number of employees per firm interviewed (see Table 11 in Annex II). The targets set by NARD for Years 1 and 2 totaled 100 jobs. The Evaluation Team concluded that the PSD Project exceeded these targets by creating 113 full-time jobs. In addition, the PSD Project also supported the creation of 307 part-time or seasonal jobs. The results recorded by the PSD Project are in line with the results from the survey conducted by the Evaluation Team. Out of 25 interviewed SMEs, 12 SMEs did not report employing new workers based on the assistance provided by the PSD, while 13 interviewed SMEs reported adding a total of 20 full-time workers and 52 part-time & seasonal workers. Additionally, five interviewed youth and women businesses reported employing a total of 8 full-time and 4 seasonal workers in their newly established businesses. The Evaluation Team notes that the high employment tax of around 62% and seasonality of certain businesses operations are two key factors reported by interviewees as favoring part-time job creation over full-time job creation. While the PSD program set low initial job creation targets (perhaps understandably given regional conditions), USAID/Serbia achieved this Overarching Indicator. Based on the sample of interviewed SMEEs, the Evaluation Team observed the highest job creation in the agribusiness sector, while the overall PSD Project data shows a slightly higher job creation in the light manufacturing sector. Notable successes include the following: • The PSD project helped one dairy operation with packaging and labels, which had positive effects upon their operation. The company added four full-time workers as demand and sales volume increased. • The PSD Project helped one agribusiness juice company exhibit at the Food & Life trade fair in Munich. In order to deliver on new post-fair sales contracts with Swiss and German companies, this company employed ten additional seasonal workers. • A confectionary company developed an Iraqi market for its products based on assistance provided by the PSD Project in developing new recipes and adjusting production process according to the requirements of the Halal standard. In order to fulfill new contract and production process requirements, this company employed 16 new part-time workers through the Youth Employment Association. • An agribusiness specializing in production of mushrooms and forest fruit products in South Serbia contracted 20 new seasonal workers based on the PSD Project assistance. Specifically, PSD helped the company in introducing IT technologies to their operation by developing new 17 software for tracking stocks in their cold-store, thus optimizing management of stocks and delivery of products. PSD assistance also led to the development of a new company website. Number of Business Services Increased. USAID/Serbia and NARD set a target of just two additional business services to be developed and executed by the NARD/RDAs. According to the information provided to the Evaluation Team by NARD and the RDAs, these new business services were the Export Training and International Trade Fair Advisory services that were undertaken by NARD as a new approach to further internationalization and opening of new foreign markets for Serbian businesses. While this indicator was technically met with two services provided, the Evaluation Team found that it was not particularly meaningful as it focused only on the development of new business services within NARD/RDAs and not specifically across the private sector market for business services of South and Southwest Serbia. Further analysis of these two added business services will be addressed in later sections. Number of Beneficiaries with Increased Competitive Skills. The total combined target for the two years was 468 individuals with increased competitive skills. During this period, the actual number of persons with “increased competitive skills” was 483 individuals. Thus, this target was met. As indicated in the PMP, the increased competitive skills for businesses are observed jointly through a range of assistance provided to PSD clients, including: training extended to SMEEs, start-ups and associations; cost-share technical assistance for SMEEs, as well assistance to export and trade fair participation; mentoring and consulting; grants for youth and women start-ups; and cost-share technical assistance for associations. Below, the Evaluation Team provides a detailed overview of assistance provided to all three categories of the PSD Project clients (SMEEs, youth and women start-ups, and associations/clusters). SMEE TRAINING The initial training for SMEEs was a relatively comprehensive, 6-module, 72-hour course provided to 121 SMEEs (200+ individuals) in Year 1. The basic training course was previously developed by NARD/RDAs and delivered by certified trainers from the RDAs. It included sessions on topics like financial management, business plan preparation, quality standards, exporting, and product marketing. SMEEs were also able to attend a 3-module advanced training in 1) Management & Adaption to Company Growth Cycle; 2) Branding; and 3) Leadership Transition. Advanced Training modules were delivered by external consultants. The Needs Analysis of Selected SMEEs was developed by the Easy4Business consulting company, in part by analyzing the training needs of SMEEs and identifying a list of training modules. It listed specific training modules that interviewed SMEEs identified as lacking and highly needed (e.g. production specific trainings, especially in textiles; market research training on how to find new, foreign buyers, etc.). Training courses were not mandatory, but it was generally “implied” that SMEE owners, managers or employees should participate. One RDA specifically noted that an enterprise could not participate in a technical assistance call until someone in the enterprise had completed all the modules of the course. More than 60% of respondents indicated that the training provided to SMEEs was useful. Around a third (11 out of 30 SMEEs) indicated that these courses were somewhat elemental, especially for those who had already been in business for a while or for those who had studied business at the university level. Some found that the course was useful as a “refresher” despite their familiarity with the topics. One of four RDAs noted that they had received feedback that the courses were overly “theoretical.” While a handful of respondents reported making changes to their enterprise operations due to information gleaned from the trainings, many respondents indicated that they had made no such changes. A sample of comments from SMEE owners and managers regarding SMEE trainings follows: • One agribusiness owner said that the training was inappropriate for Serbia and lacked a realistic understanding of the business environment there. The course did contribute to a better 18 understanding of her financial position. According to her, Serbia needs to “do business more” and not educate so much. • Another agribusiness owner who thought that the training was overly basic nevertheless pointed to new cost control measures that were discussed in training. As a result, that business had re￾organized to better control costs, including cutting fuel costs of their trucks. • A woman-owned light manufacturer noted that the RDA trained her in how to create a business plan, which she deemed highly useful and opened her up to possibilities and issues that she had not contemplated before. It also led her to the introduction of time sheets, an employee satisfaction survey and better organization of new plant premises and tools. • One respondent noted the training module on transfer of ownership prompted him to involve his son in the business and give him increasing responsibilities. • Another agribusiness received training in export processes that proved useful, especially customs procedures, which they put into practice with confectionary exports to Iraq. They also received RDA orientation on sales to Czech Republic. This business suggested that RDA might develop separate training courses for each sector. These comments reflect that some participants did find technical value in the RDA-delivered SMEE training. Others mentioned that RDA trainers offered “up-to-date” information on best business practices. We noted in the Year 1 Annual Report that the RDAs flexibly scheduled training courses towards the end of the working day so that they were more convenient for the enterprise owners and managers to attend. Even for those respondents who did not make changes as a result of the trainings, these offerings still facilitated a “relationship of trust” with the RDAs. As one respondent put it: “We knew that we could go back to the RDAs to get further assistance after getting to know them during the training modules.” SMEE TECHNICAL ASSISTANCE Respondent SMEEs indicated that they were most interested in the cost sharing technical assistance provided by the PSD Project. Three rounds of public calls for technical assistance for SMEEs were issued. Technical assistance was normally defined and provided in the PSD Project for such support as: accounting and production software; website development and website sales; standards certification, new product and packaging development, etc. PSD SMEEs were eligible to apply for cost share technical assistance (50% “co-share grants”). The selection criteria applied for the technical assistance calls encompassed: • Importance of the proposed activity for the further development of the applicant • Business productivity • Business economy and efficiency • Number of the applicant’s employees • Growth indicators achieved • Experience and available resources of the service provider Each selected SMEE signed the contract for financial assistance with NARD. In Year 1, 38 SMEEs received cost-share technical assistance grants valued in total RSD 5.9 million. During the second call 45 SMEEs from Year 1 utilized additional technical assistances valued RSD 7.3 million. In the Year 2, 35 19 SMEEs received cost-share technical assistance grants amounting to RSD 7.5 million. After delivering the service, the service provider was paid in full by the SMEES, who was subsequently reimbursed the PSD share after completion of the process and presentation of the proof of payment. The TA application process employed by NARD/RDAs was deemed by a large majority of the respondents as clearer and less confusing than those of other donors. NARD/RDAs were also able to process the applications faster and deliver commitments for providing funds to SMEEs faster (usually in about one or two months) than other donor grant programs. A majority of interviewed participants reported having the opportunity to discuss with the RDAs any issues related to filling out of their applications. Additionally, some respondents reported familiarity with the forms used for the TA application process from similar Government of Serbia calls implemented over the past years. These respondents felt that under the PSD Project, businesses from underdeveloped regions had greater opportunity for receiving assistance compared to previous nationwide Government calls in which more experienced and larger firms from the Belgrade region usually scored more points and thus received the TA. This gave PSD a clear advantage over other donors in this regard. Other donors were reported to have quite rigid guidelines to which it took longer to respond, invariably involving the submission of more paperwork. Additionally, businesses appreciated the possibility to submit PSD applications in Serbian. Also, overall PSD’s four-year time frame held a clear advantage over donors having shorter￾term projects, sometimes with only a one or two year duration. RDAs appreciated USAID’s “content” over “form” approach. They also noted USAID’s transparency built in to every step, largely protected them from political influence. SMEE respondents frequently remarked that the process was transparent and the NARD/RDAs implemented it in a nonpolitical way. Several also noted USAID’s further comparative advantage over other donor’s in introducing and managing economic and private sector development projects. Further, the SMEEs had to provide quotations from at least three suppliers of services (e.g. certification bodies or software companies, depending on the type of proposed technical assistance). When the SMEEs had difficulties obtaining price quotations, the RDA’s staff often facilitated this process. In many cases, SMEE matching grants under technical assistance were too small (relative to businesses’ overall financial situation2) to make a very substantial difference. Both NARD and the RDAs themselves reported that Technical Assistance grants were too small to have the impact that they had hoped to achieve. Nevertheless, qualitative findings from the interviews reveal grant assistance from PSD was impactful for some enterprises, as follows: • Novateks Ltd is an excellent example of how a small investment in the company website and institution of a website sales program stimulated the enterprise’s trajectory way beyond what might be expected, given the size of the grant. • Other respondents pointed to the packaging and brand and logo assistance as transformative for their operations and establishing their market image. • One textile manufacturer with 54 full-time employees was able to purchase specialized accounting software that helped monitoring production line costs and generate more profits. Given the challenging state of the Serbian economy (and in particular the scarcity of term lending by commercial banks), cost-share grants for potential useful “building blocks” are still attractive and useful 2 If the average sales of a PSD SMEE are roughly $1.1M (taken from baseline sales figures divided by the initial number of SMEE participants), then the average estimated grant size of $1,552 (based on reported cost-share grants to 38 SMEEs in Year 1) is a fairly small fraction of overall revenue. 20 to SMEEs, even if they do not quite capture the originally anticipated impact. Additionally, many of the business steps taken by enterprises with grants funding (obtaining quality standards, improved marketing and logos, etc.) may well enhance future sales in ways that are not immediately measurable. One benefit of the four-year span for the PSD project is that there are still multiple program years remaining to trace and measure positive effects. SMEE MENTORING AND CONSULTING The MOUs concluded with the NARD included the stipulation that RDA staff would provide 50 hours of mentoring to the enterprise and 5 hours of consulting. Mentoring was based upon a SWOT analysis of the enterprise, with RDA staff helping the owner/manager focus upon resolving weaknesses. RDAs were reported to be in touch with participants’ progress, which continued the communication and trust-building between the RDAs and the PSD clients – and the RDAs have a number of well-qualified, regionally-based personnel available to do this. Despite, the quality of the staff and value of the concept, mentoring provided by RDA staff seemed to have only a marginal impact. While the SMEE owners or managers responded that the RDAs “visited their businesses from time to time” and that was positive, the RDAs genuinely felt that they were mentoring the businesses. SMEEs viewed through mentoring visits by RDAs as positive with respect to selection of proper assistance under the calls for technical assistance co-sharing funds. For example, Kop-promet (one of the submitted success stories) reported learning about tailoring production software through the mentoring process of the RDA. Few SMEEs, however, could point to a change made in their businesses operations as a result of these mentoring visits. Furthermore, no respondent could meaningfully distinguish between the types of advice they received on “mentoring” visits vs. “consulting” visits. A few respondent comments may illuminate some issues with the mentoring and consulting aspect of SMEE assistance: • A textile manufacturer said: “There was no mentoring under this program. RDA visited very often, but didn’t go to the production facility. They really did not provide technical advice.” • Another textile manufacturer had an opposite view: “RDA suggested how to go about organizing their production process and they used those concepts to organize themselves better.” • An agribusiness said that “RDA understood their situation as far as they could; an advisor cannot deal with the private sector unless you have experience with the private sector.” One RDA (of four) agreed that its staff needed more on-the-job training on how to offer specialized consulting services, as they had learned only the fundamentals of business mentoring through a previous Japanese aid program. The export-related assistance provided to SMEEs by the PSD Project will be discussed under the indicator “Number of Firms Receiving Capacity Building to Export” below. YOUTH AND WOMEN START-UPS Youth and women entrepreneurs responded to an RDA call for applications for training, by submitting a business concept. Each business plan was measured by a set of comprehensive, standard criteria that included items like whether applicants had clearly outlined goals, or the extent to which prospective grantees showed an understanding of financial aspects of the grant program. The Start-up entrepreneurs were then given a training course, “How to Start Your Business,” which included instruction and support on starting the business, business plan development, financial planning and dealing with banks (similar to 3 modules of the SMEE basic training). In total 296 youth and women trainees took the 3- module Business Start-up training. Out of this number, 116 youth and women in Year 1 and 75 in Year 2 developed their business plans and submitted grant applications. From among those who completed 21 their individual business plans, often with the additional assistance of the RDAs, a smaller number of entrepreneurs were selected to receive a grant (36 in Year 1 and 24 in Year 2). The selected grantees were eligible to take part in the remaining 3 modules of the Basic SMEE training (quality standards, marketing and sales, and innovation management). By writing their own business plan the Youth and Women Start-ups reportedly “owned” their plan. Some RDAs noted that other donor projects did not directly provide such support and thus resulted in entrepreneurs having to hire outside consultants to develop the business plans, which was expensive for them and created an immediate obstacle to initiating their enterprises on a solid basis. The Youth and Women Start-ups selected in the PSD Project became eligible for 100% grants to procure machinery or equipment valued at up to $12,000, as identified in the business plan. The Youth and Women Start-up had to acquire quotations from machinery or equipment suppliers. Prior to final approval the Start-ups had to undergo an environmental assessment of their production processes. Actual grants provided reportedly averaged $7,500 for 36 Startups in Year 1. The twenty-four Year 2 Youth and Women Start-ups were still going through individual environmental reviews at the time of this Evaluation and the funds were not disbursed yet. The contracts for grants-based financial assistance for Youth and Women Start-ups were concluded with local RDAs who also led the public procurement process for the machinery and equipment. RDA staff often facilitated obtaining quotations from suppliers to ensure the purchase of the lowest cost, appropriate machinery or equipment. This extra effort from RDAs was very positively viewed by Start￾ups. One woman respondent who spoke only Albanian reported that the equipment suppliers were not taking her request for price quotations seriously and were not responsive. RDA staff stepped in, assuring the suppliers that this was a serious potential transaction, and obtained the quotations, which served as the basis for a grant amounting to approximately $4,500. The training in basic three modules provided to Youth and Women Start-ups prior to the grant award was viewed by the entrepreneurs as very positive and useful for initiation of their businesses (e.g. practical advice related to registration of businesses). Start-ups were also highly appreciative of the mentoring process by RDAs, reporting that RDA staff members were instrumental in proving various information and advice to them, and that this process should continue in the future. Business-specific advice to the Youth and Women Start-ups was also provided by the suppliers of machinery (e.g. instructions on operating the machinery; information on potential suppliers of raw materials, etc.). Additionally, the Youth and Women Start-ups particularly valued the grants for equipment which enabled them to start their business. One RDA felt that these grants created the greatest impact in the PSD Project. For many such Start-ups, PSD offers one of the only avenues for obtaining any financial support for their family business. There are no fully-functional micro-financial institutions in Serbia, let alone a micro-financial framework law to support their establishment, so other finance for Start-ups is difficult to come by. Many of the start-ups were developed as family-run businesses, either as parent-child or husband-wife. The evaluators noted a handful of instances where women-owned start-ups were not actually woman￾managed and may have represented attempts by husbands to take advantage of the 100% grant funding operated to woman-owned businesses. However, there were undoubtedly numerous legitimate female entrepreneurs who applied for the startup grants. This represents one potential area of PSD expansion in the second phase, as only roughly a quarter of PSD beneficiaries were female-owned businesses. Out of the five interviewed Youth and Women Start-ups only two had actually started their business operations. Some Start-ups were behind in the initiation of their operations due to the pending investments in facilities where they will place the equipment or due to the seasonal aspect of their business. All interviewed Start-ups were legally registered, often with more than one employee, and the 22 majority had an active status (i.e. had already started paying taxes) despite delays in initiation of operations. ASSOCIATIONS/CLUSTERS The PSD Project assisted eight associations/clusters, doubling the initial target of four associations, and also expanded this assistance beyond the 12 targeted municipalities. The RDAs were responsible for making an inventory of all of the associations and clusters in their respective municipalities, including those formed with the support of USAID and other donor-supported projects. Similar to the SMEEs, the PSD Project undertook an independent and outsourced Rapid Inventory & Gap Analysis of Existing Associations and Clusters, as well as Technical Needs Assessments of Associations and Clusters to identify the key priority areas for intervention. Needs were assessed in terms of i) human resource and staff development; ii) planning; and iii) execution. Ultimately, specific needs were articulated for each client association in areas of: i) administration training; ii) business communication training; iii) management training; iv) export procedures training; v) cluster support and business improvement; and vi) priority trade fair representation. All associations received substantial assistance (manager training; 80% cost-share technical assistance, including purchase of equipment for associations; and trade fair attendance). Initially, all association/cluster managers took part in a 3-module training course encompassing: 1) Cluster Management and the Role of Cluster Manager; 2) Business Skills in Marketing and Sales; 3) Development Documentation for Strategic Positioning. Association managers found the training modules useful for further association/cluster development; however, it was unclear if the training resulted in advancement of knowledge and skills of an association’s members. Associations/clusters were mostly interested in the cost-share technical assistance grants, offered after a public call for application. Most association and cluster managers were already familiar with the application process and found both the process and required documentation fairly straightforward. The biggest challenge for the associations/clusters was to agree on required common priority actions and thus apply for appropriate assistance. The scope of requested cost-share technical assistance was diversified, and, inter alia, encompassed joint market research, website and online sale development, development of promotional materials, redesign of packaging, development of procedures for joint services center of association, purchase of laboratory testing equipment, packing equipment, laptops, etc. Some of the technical assistance resulted in embedding new joint services within the association/clusters for their members and thus ensuring additional sources of revenues for association if further commercialized (e.g. laboratory equipment; online sales, packing machine, etc.). Collectively, associations reported adding only two new members in Year 1 as a result of the PSD Project assistance. Number of Firms Receiving Capacity Building to Export. The indicators show a Year 1 and 2 combined target of 30 individuals to receive capacity building for export. The actual number of businesses receiving capacity building to export in Year 1 only was 44, showing that PSD soundly met its target on this indicator. RDAs delivered the export training on “Mechanisms for Financing Exports” to 30 participants SMEEs. The export development training by those SMEEs not already exporting was found to be quite useful. This training was rated highly by SMEEs contemplating exports or having difficulties penetrating export markets. One advantage cited by RDAs and SMEEs was the interaction and relationship building among the enterprise principals to learn and strategize as well as building trust between the RDA and SMEEs. The development of this training module was outsourced and RDAs were positive about their ability to provide input for development of export training topics and curriculum. Given the fact that RDAs are most familiar with the needs of SMEEs in their regions, this practice should be applied any time services are outsourced. 23 Component 2 of the PSD Project supported export development by a) providing training on mechanisms for financing exports, including reference to export development strategies and techniques and b) offering support for attending trade fairs. Although PSD exceeded its target for this indicator, the Evaluation Team would comment that this indicator alone does not provide that much information about export success. The evaluation surveys suggested mixed results on export capacity. While export training was generally viewed positively by respondents as a good introduction, support to businesses and associations to attend trade fairs did not often result in increased sales. The trade fair attendance PSD support for SMEEs will be further discussed under the indicator “Number of USAID Client SMEEs Exhibiting at International Trade Fairs and the Value of Contracts.” Additionally, USAID provided a specialized training to NARD and RDAs staff in export market research (course “How to Find Markets”) through the International Trade Center (ITC). The goal of the training was to further enhance the capacities of RDAs in providing export related research and assistances to businesses. NARD and RDAs staff were trained in using different online tools and databases of ITC during the export market research, and reported that the training further enhanced their knowledge and skills with respect to exports. However, only one training course was delivered by ITC during the first phase of project implementation. (There had been plans, however, to deliver a more practical second training focusing on selected products from the three core assistance sectors.). One RDA strongly advocated that this training should continue. This echoes the conclusion of the Evaluation Team that RDAs have not yet been able to adequately pass on knowledge gained through the ITC courses to their PSD clients. Conclusion to the Question: To what extent has PSD achieved the expected Overarching Mission-level results identified in the Activity Monitoring and Evaluation Plan? Table 2. Conclusions on the Overarching Indicators Overarching Indicator Conclusion Value of annual revenues of USAID client companies in targeted regions The PSD Project exceeded the annual revenue targets, increasing the revenues of client companies by 8% annually over the first phase of the PSD Project compared to the baseline. Estimates of sales growth, when given by companies in interviews, were between 10 and 20%. Number of jobs created in targeted regions The PSD Project exceeded its targeted number of full-time jobs, achieving a full-time job creation total of 113. Additionally, the Project created 307 part￾time and seasonal jobs as well, positive in an environment of 50-70% unemployment in the southern and southwest regions. Number of business services increased in targeted regions The PSD project achieved this Mission-level result by undertaking two new business services: Export Education and Trade Fair Attendance. Given the fact that NARD/RDAs were previously not involved in supporting trade fair attendance, the Evaluation Team concludes that new business services were successfully integrated in their practice. The Evaluation Team recommends to consider extending this support beyond building business service capacity of NARD/RDA, and expand to private sector service providers, as per identified needs of PSD clients, e.g. vocational training in textile sector. Number of beneficiaries with increased competitive skills The PSD Project achieved this Mission-level result by providing a combined package of assistance, including training, mentoring, technical assistance and export support to 3 categories of PSD clients: SMEEs, youth and women start￾ups and associations to the satisfactory of all interviewed clients. The actual number of beneficiaries with increased competitive skills (483) exceeded the Year 1 and Year 2 combined targets of 468. Some adjustment needs to be made regarding the selection of trainees when the business has been in operation for considerable time or the potential trainee has recently graduated 24 from business or economics at the university level. Also, although the association managers participated in these training courses there was little effect. Some adjustment is required in the support provided to associations and clusters, as the Evaluation Team noted little sustainability of supported associations/clusters, e.g., each association should be supported in developing an annual development plan similar to the business plan exercise of the Start￾ups. This should be useful as a means to structure support to enable these associations to achieve a useful, self-sustaining role. This may prove that these efforts will not be productive as it may be impossible to demonstrate individual association’s financial sustainability. As one part of this, the planning exercise should include, whenever possible, the consideration of a value chain development and financing approach, implementing sales through a lead firm or one in which the association acts as the lead firm. Number of firms receiving capacity building assistance to export By building capacity for export of 44 firms, the PSD project exceeded the combined Years 1 and 2 targets of 30. The Evaluation Team concludes that the export training on “Mechanisms for Financing Exports” was particularly well structured and received by client SMEEs. The same format of the training module should be continued in the future. Additionally, the trade fair support could be further strengthened with staging of the trade fair attendance, as there were a number of businesses and associations attending International Trade Fairs with quite limited results. In addition to staging of the trade fair attendance, additional consideration should be given to selection of trade fairs as per company’s needs, company or sector focused market research and B2B scheduling prior to the fair to ensure high learning and ultimate results from the trade fair experience. The PSD Project has achieved all five of the Overarching Mission-level Indicators. An annual revenue increase of nearly 8% within an economically challenging business environment is impressive. The job creation indicators – though surpassed -- were set at a very low level and should be increased for the future phase. The number of business services as an indicator was not particularly meaningful. PSD exceeded the target number of beneficiaries with increased competitive skills; respondent opinions regarding training, technical assistance, and mentoring/consulting were mainly positive, but also evinced areas for improvement. While export training was positively received, trade fair support must be improved. II. TO WHAT EXTENT HAS THE PROJECT RESULTED IN IMPROVED COMPETITIVENESS OF SMEES? Again, we start by presenting USAID’s relevant performance indicators and the actual results. Table 3. Indicators on Strengthening SMEEs Performance Indicator Baseline Y1 Result (Jan-Oct) Y1 Target Year 2 Result (Nov￾July) Y2 Target 01-1 Number of SMEEs receiving USAID assistance n/a 122 120 121 180 01-2 Number of USAID client SMEEs entering new markets n/a 9 10 20 01-3 Number of USAID client SMEEs exhibiting at International Fairs n/a 9 10 19 20 01-4 Value of contracts signed by USAID client SMEEs exhibiting at n/a $91,956 $100,000 $263,250 $150,000 25 International Trade Fairs 01-5 Number of USAID client SMEEs introducing international standards n/a 9 6 11 6 01-6 Number of USAID client SMEEs adopting new technology or management practice n/a 15 6 35 6 Number of SMEEs Receiving USAID Assistance. Initially, around 340 SMEEs expressed interest in participating in the PSD Project. Of the initial expressions of interest, 122 SMEEs were chosen in Year 1(one of which dropped subsequently dropped out) and an additional 62 in Year 2 using two sets of selection criteria based on whether the enterprise used double-entry or single-entry accounting systems. The RDAs reached out personally or by email to enterprises they knew from previous projects or site visits. RDAs completed surveys with SMEEs via personal interviews and recorded on a Profile of Potential Project Beneficiary form. Each RDA selected an initial group of candidate SMEEs and forwarded the profiles to a Selection Committee led by NARD. A local consulting company (Easy4Business) was contracted using a public tender to reduce local conflicts of interest. Easy4Business prepared the Report on Interviewed SMEEs and Gap Analysis (under Milestone 4). The same company also prepared the Analysis of the Needs of Selected SMEEs (Milestone 18) identifying weak points of businesses with respect to competitiveness, staff-related problems, and, most importantly, needs of SMEEs for technical assistance3. Some of the identified needs were addressed by a cost-share technical assistance program, but some were not addressed and continue to be outstanding issues for these businesses (e.g. lack of skilled workforce, access to financing, etc.). The Committee reviewed the list of eligible participants using predetermined selection criteria like cost effectiveness or ratio of capital to long-term assets (for businesses with double-entry bookkeeping) or number of employees and export orientation ratio (for businesses with single-entry or lump-sum bookkeeping systems). The SMEEs from Year 1 were eligible to take part in the cost-share technical assistance program or export related in Year 2. Following their selection, participant SMEEs signed a memorandum of understanding with NARD to receive a combination of PSD support, including: training; co-share technical assistance; mentoring and consulting, and export related assistance. In total the PSD project signed the MoUs and provided assistance to 183 SMEEs cumulatively for two implementing years, thus surpassing the cumulative target of 180 SMEEs for Year 2. As indicated earlier, the SMEEs benefited from package of assistance provided by the PSD Project encompassing: training, mentoring and consulting, 3 round of calls for 50% cost-share technical assistance for SMEEs, as well assistance to export and trade fair participation. Number of USAID Client SMEEs Entering New Markets. NARD performance indicators noted that nine light manufacturing companies entered the EU Market via an International Trade Fair in Slovenia. Out of 25 interviewed SMEEs and 4 associations, 17 of them reported foreign trade fair attendance. Of these 17, only 4 reported actually delivering sales contracts to new buyers (including locally, in Central/Eastern Europe, and in Western Europe). This means that only 23.5% of interviewees had increased sales as a direct result of trade fair participation. Other respondents indicated that they took samples of their goods to International Trade Fairs and that they sold them to pay expenses for getting to and staying at the Trade Fairs, or that they merely established contact with potential buyers. An additional eight interviewees indicated that after the Trade Fairs they maintained contacts with 3 In the PSD Project, technical assistance was not defined in the conventional way, i.e. provision of technical assistance. Technical assistance under the PSD Project was defined as some useful input, such as website development, organizational or environmental certification, a computerized accounting system, etc., which was paid for with a cost-share grant from the RDA. 26 potential buyers, sent samples or even negotiated the terms of a potential contract, but did not ultimately finalize any sales. Number of USAID Client SMEEs Exhibiting at International Trade Fairs and the Value of Contracts. There were 28 individual SMEEs4 that attended 12 trade fairs5 in Years 1 and 2, selling a total of $355,206 in merchandise at these shows. Forty-four individual participants went to Trade Fairs in Celje; Munich and Dusseldorf in Year 1 and 36 individuals from SMEEs attended Trade Fairs in Dusseldorf, Celje, Munich, and Brno in Year 2. As noted in the section above, only 23.5% of the 17 respondents (14 SMEEs, 3 associations) visiting trade fairs could point to increased sales contracts as a result of trade fair participation. The respondents indicated that the overall impact of attending trade shows was positive in that they learned about quality standards, pricing and other factors. Some attendees expressed surprise (one interviewee even used the term “shock”) at the level of product quality encountered in the fairs. NARD (with assistance from SIEPA) selected the trade fairs to be attended and provided support in the form of payment for a stand at the show and on-site technical guidance. The participants paid their own transport, accommodations and sample shipment expenses. Some respondents indicated that they sold the samples that they had taken to the Trade Fair, which could account for some of the sales reflected as an indicator. The following are examples of the range of results from trade fairs: • An agribusiness company producing juice exhibited at the Food & Life Trade Fair in Munich. The trade fair attendance resulted in new sales contracts to Switzerland valued at EUR 150,000, and close to 40,000 EUR to a German buyer. To deliver on these new sales contracts, this company employed ten additional seasonal workers. • A woman-owned and managed protective glove manufacturer demonstrated the impact of a successful trade fair experience, landing a contract in Celje resulting in shipments of about 15,000 pairs monthly in the local market through a foreign-owned Serbian company over the last seven months. • The shoemakers association reported attending five trade fairs, three financed by a previous EU project and 2 times through PSD. They were instructed on doing “market research” and received assistance on brochures and business cards. Under the PSD Project, this association landed a highly-touted contract from an Italian buyer for 1,000 pairs of shoes; however, the contract fell through because the buyer would not pay for transporting the shoes. Thus, they were ultimately unable to report much positive effect. • Numerous respondents stated that trade fair communications could be improved further, as they sometimes lacked useful information about market research or sales opportunities 4 Some SMEEs attended certain trade fairs more frequently than others, resulting in a total of 86 SMEEs participants in international trade fairs. 5 Five trade fairs were attended in Year 1, namely: MOS – Trade Fair of Light Manufacturing Industry, Celje, Slovenia; Food & Life, Munich; GDS – Footwear and leather goods (accessories) fair, Dusseldorf; IHM – Trade fair for craft trades and small businesses, Munich; FormaTool Fair, Celje and 7 trade fairs in Year 2, namely: GDS – Footwear and leather goods (accessories) fair, Dusseldorf; MOS – Trade Fair of Light Manufacturing Industry, Celje, Slovenia; A+A – Trade Fair for Safety, Security and Health and Work, Dusseldorf; Food & Life, Munich; Heim & Handwerk, Munich; AGRA – Trade Fair of Agriculture and Agribusiness, Gornja Radonja, Slovenia; KABO – Footwear Trade Fair, Brno. 27 The implications are several: a) some enterprises noted that there might have been better shows to attend, i.e. shows which were more appropriate to their particular lines of production; and b) putting an inexperienced owner in a booth with their samples at a European trade fair might be a daunting proposition, especially if that person did not speak English or other major European languages. One RDA noted similar concerns that “an enterprise should have something to do with the selection of the trade fair to be attended.” Number of USAID Client SMEEs Introducing International Standards. There were some examples of enterprises achieving new levels of competitiveness, primarily through access to “building blocks” such as ISO or other export certifications. Among the survey respondents, eight companies were assisted with thirteen certifications. These included certification for ISO 9001, 14000 and 18000; CE Mark; and HACCP. These relatively small inputs helped businesses to fulfill mandatory requirements or quality standards imposed by their buyers, enabled them to take part in public procurement tenders, and increased SMEEs owners’ confidence in marketing to the EU market and at International Trade Fairs. Several businesses cited these certifications as crucial to the credibility of their companies. NARD/RDAs were able to support 20 such certifications in Year 1 and 2, exceeding their target of 12. Selected respondents’ comments were as follows: • A light manufacturer that attended a trade fair with PSD support cited the importance of the ISO 9001 re-certification as important to buyers in the international markets. Another light manufacturer was assisted with the CE Mark certification as well as ISO 9001 and ISO 14000 certification and reported importance of standards in fulfilling requirement of demanding buyers, and verifying quality of product with the CE mark certificate. Owners generally felt that the certification process would contribute to their competitiveness. • A dairy was assisted with the HACCP certification. This certification enabled the dairy to comply with local mandatory food production hygiene requirements. • A textile manufacturer was certified as following the standards of ISO 9001, but claimed not to obtain the certification for marketing but rather to improve their organization. Number of USAID client SMEEs adopting new technology or management practice. In addition to the certifications, SMEEs were supported by the PSD Project to adopt a range of new technologies and management practices. Technical assistance included software support; better packaging and labeling; website and website sales. NARD/RDAs had set a target of 12 such installations over Years 1 and 2 and reported 50 such new technology or management practice inputs. These must have proven easier to implement and more popular than expected. Twelve of the SMEEs interviewed had received cost-share support for the installation of 16 new technologies. All interviewed SMEEs reported positive experience in applying new technology in their operations. Some of these SMEEs made the following comments regarding these technology or management practice shifts: • A textile manufacturer already in business for 10 years, had a website installed, which they followed up by getting a local consultant to help them do on-line sales, focusing on Google ads and key words. They had a small amount of export sales until the website was activated. Once activated, buyers from Montenegro and Croatia began to call by telephone to order as well as ones in Serbia. They had achieved RDS 4 million in on-line sales and had to hire a new person to handle those sales. • One large textile manufacturer installed accounting software that enables them to handle the requirements of their tax returns to the central registry, whereas another large textile manufacturer received a cost-share grant for production monitoring software. • A large dairy and small agribusiness both received packing and labeling assistance, which helped position their products better in the local market. 28 • One 60-year old food manufacturer following up a market lead needed to shift its production so that it could be Halal-certified and developed specialized packaging. This shift enabled them to open up a market in Iraq for their product. The RDA supported in obtaining this information needed for this and provided the firm with Export Training. It is clear that the PSD Project exceeded its targets for Years 1 and 2 and in several cases achieved some relatively dramatic results. Conclusion to the Question: To what extent has the project resulted in improved competitiveness of SMEEs? Table 4. Conclusions on Indicators on Strengthening SMEEs Performance Indicator Conclusions Number of SMEEs receiving USAID assistance The PSD project signed MoUs and assisted 183 SMEEs during project’s first phase, thus surpassing the cumulative target of 180 SMEEs for Year 2. Several SMEEs selected were inappropriate as they were large enterprises (one interviewed had 424 full-time employees) or parts of a large consortium that should not have received PSD assistance. One interviewed SMEE operating in the agribusiness sector was predominantly foreign owned. Number of USAID client SMEEs entering new markets In the indicators provided by NARD, only 9 companies were reported to enter new markets, while the target for Years 1 and 2 was 30. Given the fact that Year 2 data for this indicator were not available, the Evaluation Team cannot provide a definite opinion with regard to PSD performance on this indicator. Based on the completed interviews, the Evaluation Team indicates that the PSD Project was partially successful, as only 23.5% of interviewed clients reported delivering sales to new buyers. Number of USAID client SMEEs exhibiting at International Fairs The PSD Project fell short in this category, reporting 28 SMEEs at International Trade Fairs just missing the target of 30. As per the information provided by the interviewed clients, the Evaluation Team concludes that the selection of trade fairs was only partially successful as several respondents indicated that some other more specialized fairs would fit better to their business operations. Some SMEEs felt that locally organized B2Bs might be more effective. Value of contracts signed by USAID client SMEEs exhibiting at International Trade Fairs The PSD project did exceed a modest target of $250,000 in sales by selling $355,206. While the NARD/RDAs exceeded the target, the Evaluation Team feels that this approach needs adjustment and further highly-skilled technical advisory services to really have success in International Trade Fairs. Additionally, the Evaluation Tem concludes that the value of indicator could be further strengthened by focusing to delivered sales instead to contracted sales. Number of USAID client SMEEs introducing international standards NARD/RDA exceeded the target of 12 international standards introduced with a two-year total of 20 such standards. The Evaluation Team concludes that the SMEEs interviewed introducing quality standard have a more solid foundation to further increase sales and enter export markets. Number of USAID client SMEEs adopting new technology or management practice NARD/RDAs by far exceeded the targets for helping SMEEs to adopt new technology or management practice by registering 50 new technologies or management practices while having a target of 12. The Evaluation Team verifies the positive impact, and in some cases, excellent results these technologies had, based on the conducted interviews. 29 The number of SMEEs receiving assistance (183) exceeded the PMP target. Two firms surveyed were inappropriately selected because they were too large or were affiliated with a large consortium. According to data provided by the NARD to the Evaluation Team, the PSD Project underperformed in assisting companies sell in new markets. The Evaluation survey obtained information that only 23.5% of the respondents who attended trade fairs entered new markets as a result, and fewer SMEEs participated in trade fairs than targeted. The value of sales contracts at international trade fairs at $355,000 exceeded the $250,000 trade fair sales target. The Evaluation Team concluded that while this indicator was technically met, it must be increased in the next phase (and recommendations have been provided below to that effect.) The introduction of 20 international standards, usually quality and environmental certifications, exceeded the target of 12 such standards. The Evaluation Team notes that these standards can provide companies with a solid foundation for export, especially to the EU. By adopting 50 new technologies or management practices the Technical Assistance component, this proved to be one of the most popular and cost-effective aspects of the PSD Project. The Evaluation Team notes the following further observation regarding the selection process of SMEEs. It became obvious during the interviews that this selection process, despite being done carefully and transparently, did not necessarily result in the selection of strategic, high growth SMEEs or value chain possibilities for PSD investment. Rather, the benefits were spread too widely across too large a number of SMEEs. This ultimately resulted in less impact per beneficiary. III. TO WHAT EXTENT HAS THE PROJECT CONTRIBUTED TO INCREASED LINKAGES BETWEEN SMALL BUSINESSES AND REGIONAL/NATIONAL GOVERNMENTAL SUPPORT ORGANIZATIONS? RDA communications, training and mentoring interactions have clearly resulted in the creation of a “relationship of trust” with the SMEEs, Startups and Associations. This was one of the best results of the PSD Project’s first phase. The RDAs performed exceedingly well in terms of providing frequent, ongoing support, including information on donor opportunities, trade fairs, training courses, market and regulatory information, bank lending promotions and other opportunities for enterprises and associations. The RDAs have some definite advantages over traditional USAID contractor projects in that they can maintain quick, cost-effective communication and visits to SMEEs, Youth/Gender Start-ups, and Associations/clusters in all 12 target regions. Respondents viewed the quality and frequency of these communications far more favorably than those that they had with the Chamber of Commerce, which most felt did a poor job of keeping businesses informed of opportunities. PSD Project beneficiaries consistently indicated that they appreciated the geographic closeness of RDAs and the extent to which RDA members are often in the field directly talking to enterprises. As noted above, RDA staff provided 50 hours of mentoring to the enterprise and 5 hours of consulting. Visits made by the RDAs to company premises contributed to strengthening linkages of PSD clients with regional and national government support organizations. However, mentoring and consulting activities did not always have direct impact upon company operations, in part because level of technical knowledge offered by RDA personnel was not always sufficient to help address the particular, unique challenges experienced by individual businesses. The Evaluation Team notes that 100% of interviewed SMEEs, youth and women start-ups and associations reported strengthened linkages with the local RDAs/NARD through the PSD Project. Out of 34 interviewees, 18 of them (53%) reported having no past contact or very limited contact with local RDAs/NARD before the PSD Project. One SMEE respondent even expressed that given their recent experience through the PSD Project, they were regretful that they had not established relations with the RDA earlier. 30 All respondents reported obtaining a range of useful information from the RDAs and respondents almost uniformly noted RDAs as an efficient, primary source of reliable information, especially as compared to other state institutions. Multiple respondents noted the link between NARD and RDAs and indicated that they felt stronger linkages to NARD in part because of their experience with the RDAs. Interviewees left the Evaluation Team with the clear impression that beneficiaries feel able to contact RDAs to ask for any information or assistance they might need. This all underscores the importance of maintaining the communication between RDAs and beneficiaries, especially during the transition to DAS. The Evaluation Team concludes that the PSD Project was particularly successful in developing linkages between businesses and local/national RDAs. The prime reasons for such excellent performance by RDAs in this regard was their local presence, accessibility, high professionalism of the staff, continuous communication and worthy assistance and information. The RDAs should continue maintaining these relations of trust with businesses in their regions to further enhance private sector and regional development. Conclusion to the Question: To what extent has the project contributed to increased linkages between small businesses and regional/national governmental support organizations? The Evaluation Team believes that this was undoubtedly one of the strongest positive elements of the PSD Project. The level of appreciation for, willingness to cooperate with, rely on, and trust the RDAs was supremely high among respondents. They especially appreciated the on-going communication of opportunities for the business. This is a perfect example of the value of the FORWARD Initiative and G2G aspect of PSD. While mentoring and consulting was not particular effective, RDAs should be able to better tailor their advice to individual enterprises. Recommendations along these lines are provided below. IV. HOW EFFECTIVE WAS PSD IN CREATING THE LINKAGES AMONG DIFFERENT PSD PARTNERS (INCLUDING SUB-RECIPIENTS) AND ACROSS DIFFERENT MECHANISMS? At least two opportunities were open for creating linkages among different PSD partners (including sub￾recipients) and across different mechanisms. One was for SMEE beneficiaries to work through Association/clusters to share information, market together under a common logo, and coordinate production for larger contracts and other activities. The second opportunity was to work through value chains, working with lead firms to drive marketing efforts and assisting suppliers with technical advice, product purchases, and financing assistance. Some SMEEs reported that they started cooperating and learning from other PSD clients engaged in similar business activities. Several businesses reported identifying new suppliers and buyers in local market through the PSD. It appears that the PSD Project focus in extending assistance to sub-recipients was through the assistance provided to associations/clusters, as value chains were not a prominent part. Initially, the RDAs were responsible for inventorying all associations and clusters in their respective municipalities, including those formed with the support of USAID and other donor-supported projects. Similar to the SMEEs, the PSD Project undertook an independent and outsourced Rapid Inventory & Gap Analysis of Existing Associations and Clusters, as well as Technical Needs Assessments of Associations and Clusters to identify the key priority areas for intervention. Needs were assessed in areas of i) human resource and staff development; ii) planning; and iii) execution. Ultimately, specific needs were articulated for each client association in areas of: i) administration training; ii) business communication training; iii) management training; iv) export procedures training; v) cluster support and business improvement; and vi) priority trade fair representation. 31 Table 5. Indicators on Strengthening Associations/Clusters Performance Indicator Baseline Y1 Result (Jan-Oct) Y1 Target Year 2 Result (Nov￾July) Y2 Target 02-1 Number of associations receiving USAID assistance n/a 6 4 8 4 02-2 Value of joint revenues of USAID client associations/clusters $33 million $26.6 million $35 million $38.97 million $40 million 02-3 Number of new members joining USAID associations/clusters 47 2 4 10 02-4 Value of contracts signed by USAID client associations/clusters at International Trade Fairs n/a n/a n/a $50 million 02-5 Number of USAID client associations/clusters exhibiting at International Trade Fairs n/a n/a n/a 6 02-6 Number of trained managers of USAID client associations/clusters n/a 6 4 4 Number of Associations/clusters Receiving USAID Assistance. The PSD Project did assist eight Associations/clusters, doubling the initial target of four associations, and also expanded assistance beyond the 12 targeted municipalities. Value of Joint Revenues of USAID Client Associations/Clusters. The recorded Association/cluster sales for Year 1 were $26.6 million, which can be extrapolated to be $35,466,467 for the full year, which means that NARD/RDAs exceeded the goal for Year 1. Year 2 sales for the nine￾month period can be extrapolated to $51,969,333. Over the two years, the Association/Cluster companies achieved a reported 11.5% annual sales growth. Number of New Members Joining USAID Client Associations/Clusters. Only 2 new members joined the eight Associations/Clusters over the two-year period, considerably below the expected target. The Evaluation Team notes that interviewed associations were still struggling with identifying common interests across their membership and ensuring sustainability. Two of the interviewed associations/clusters (one from the shoe sector and one from agribusiness) showed some positive planning with respect to developing joint products and joint marketing and sales. Value of Contracts and Number of Association/Clusters Exhibiting at Trade Fairs. NARD statistics showed no contracts for associations/clusters exhibiting at the Trade Fairs. The Evaluation Team notes that some Associations/Clusters reported exhibiting at Trade Fairs, but interview results matched the NARD statistics regarding no increased sales as a result of participation. Number of Association/Cluster Managers Trained. During Year 1 the RDAs trained six Association/Cluster managers; none were reported in Year 2. Initially, all association/cluster managers took part in a 3-module training course encompassing: 1) Cluster Management and the Role of Cluster Manager; 2) Business Skills in Marketing and Sales; 3) Development Documentation for Strategic Positioning. All interviewed association managers found the training modules useful for further association/cluster development. According to the Evaluation interviews, associations/clusters were mostly interested in 80% cost-share technical assistance grants initiated by a public call for application. Many association and cluster managers 32 were already familiar with the application process and found both the process and required documentation fairly straightforward. The biggest challenge for the associations/clusters was to agree on required common priority actions and thus apply for appropriate assistance. The scope of requested cost-share technical assistance was diversified, and, inter alia, encompassed joint market research, website and online sale development, development of promotional materials, redesign of packaging, development of procedures for joint services center of association, purchase of laboratory testing equipment, packing equipment, laptops, etc. Some of the technical assistance resulted in embedding new joint services within the association/clusters for their members and thus potentially supporting additional sources of revenues for the association if its operations could be further commercialized (e.g. laboratory equipment; online sales, packing machine, etc.). Collectively, associations reported adding only two new members in Year 1 as a result of the PSD Project assistance. The Evaluation Team noted the following responses: • The Cluster concept is highly challenged in this environment; despite significant assistance received by association managers, interviewees seemed to lack clear vision or progress. Associations were not close to financial viability, and they had not achieved export sales despite a number of association members attending trade fairs abroad. • Association managers reported that overall their members were struggling to achieve adequate quantities and quality in order to actively engage in export activities, and could further benefit from additional training on enhancing technological and production processes, joint market research, B2B meetings and study tours for their members. • Association members were struggling with lack of trust and engendering common goals/vision (This is the common challenge to a cluster-based concept.6) One RDA commented that the two clusters in its region “were not really functioning well. One cluster is sort of a rivalry among the producers. Another cluster went abroad to a trade fair with no sales achieved and overall little was being done through association and there has been no increase in membership.” • The Novi Pazar Shoemakers Association had great ideas about creating a common logo, joint marketing and common response for large orders. However, few associations had significant joint sales or joint product development (i.e. a true cluster approach). This Association reported attending five trade fairs, two of which were financed by PSD. They were instructed on how to do “market research” and received assistance on producing brochures and business cards. • The Agro-Pešter Cluster reported development of a joint product under the common brand for industrially produced traditional product “kajmak.” They strongly believe in the concept that the cluster should have joint services, equipment and develop prototypes for joint products. 6 The cluster concept was originally introduced by Michael Porter of the Harvard Business School as a means to improve competitiveness. This approach is challenging as cluster members rarely trust one another to the point of building a stronger joint business model. As a result, USAID has increasingly turned to application of “value chains” in private sector development projects. Associations can become parts of value chains if they function as the aggregator or lead firm, purchasing inputs or products from members and contracting for product sales. 33 Table 6. Conclusions on Indicators on Strengthening Associations/Clusters Performance Indicator Conclusion Number of associations receiving USAID assistance PSD Project has exceeded the target number of Associations/Clusters. Value of joint revenues of USAID client associations/clusters The value of joint revenues exceeded both the Year 1 and Year 2 targets with a reported average of 11.5% annual growth, an even higher rate than the SMEE annual revenue growth rate. It was not clear from interviews that this revenue growth was attributable directly to PSD interventions. Number of new members joining USAID associations/clusters The fact that only 2 new members joined these associations coincides with the survey findings that organizations are severely challenged and are not offering many benefits to their membership. The association/cluster concept requires adjustment in future phases of the PSD project. Value of contracts signed by USAID client associations/clusters at International Trade Fairs The lack of contracts achieved at International Trade Fairs confirms the survey findings that this is one area for improvement in future phases of PSD. Number of USAID client associations/clusters exhibiting at International Trade Fairs The lack of client associations/clusters exhibiting at International Trade Fairs is consistent with the survey findings and requires adjustment. Number of trained managers of USID client associations/clusters The number of managers trained exceeded the Year 1 target. It did not achieve the combined targets for Years 1 and 2. Conclusion to the Question: How effective was PSD in creating the linkages among different PSD partners (including sub-recipients) and across different mechanisms? The PSD worked with eight associations/clusters, more than the targeted 4. The associations/clusters with which the PSD Project was working achieved an 11.5% annual growth rate, which is at an even higher rate than the SMEE sales revenue increases registered above. However, as indicated there were challenges in this aspect of the program in terms of new members joining the targeted associations/clusters; number of contracts signed at international trade fairs; the number of clusters/associations exhibiting at international trade fairs. The number of managers trained exceeded the Year 1 target but not the combined targets for Years 1 and 2. This element of the PSD Project requires adjustment. We note that “creating linkages among different partners (and sub-recipients) should not be restricted to associations/clusters. We recommend that associations be given an opportunity to put forth applications for assistance based on value chains. V. WHAT HAVE BEEN THE STRENGTHS AND WEAKNESSES OF USAID’S APPROACH TO G2G MANAGEMENT? NARD served as USAID's lead partner and the umbrella organization of the PSD Project while the RDAs served as partner institutions implementing field activities. NARD existed as a coordinator of donor programs well before the design of the PSD Project and was established reportedly to facilitate EU accession. Based on a series of donor projects, most of them focused on private sector development, NARD and the RDAs developed their overall administrative processes and technical competence well before the PSD Project initiation. After the PSD G2G agreement was signed, NARD combined Serbian procedures with USAID requirements, evidently without making the administrative procedures overly complex. According to the director, 13 of NARD’s 57 staff were dedicated to PSD Project implementation and oversight. The G2G agreement was an Output Based Reimbursement Agreement (OBR Agreement), which was to track 34 PSD Project implementation against 70 pre-defined milestones (36 in Year 1 and 34 in Year 2), and provide fixed reimbursements based upon NARD’s presentation and USAID’s verification and approval of the implemented milestones. At the time of this Evaluation, 18 milestones were pending implementation, which underscores the necessity of a smooth transition between NARD and then newly combined DAS for continuing PSD success. Some PSD Project clients felt that NARD as central administrator did not really represent what was going on at the regional level. Many instead credited the RDAs and their 12+ years of experience with advancing private sector development. However, NARD played numerous implementation roles. For example, the selection criteria and the final selection of participants were approved by NARD. The training modules discussed below were standardized by NARD, so that the same business training content would be presented by the RDAs and outside trainers to enterprises and associations across the South and Southwestern Regions. Additionally, the PSD Project clients signed the Memorandum of Understanding (MoU) with NARD. NARD along with the directors of RDAs took part in the Evaluation Committee approving the grants made under the PSD Project. Contracts on financial assistance were signed with NARD under the cost-share technical assistance program for SMEEs, while contracts for equipment grants were signed with local RDAs. NARD identified the trade fairs that SMEEs and associations/clusters might attend. So, while NARD might have not been as directly visible to project participants, they did play many important roles. The RDAs definitely function, as noted above, as a hub for all kinds of information and opportunities (other donor programs, grant opportunities, government initiatives, etc.) for SMEEs, Start-up Enterprises and Associations/Clusters. Almost every respondent spoke positively about the value and frequency of this information, which resulted in many opportunities to take advantage of training courses, apply for grant funds of other donors, and possibilities for attending trade fairs. The activities of RDAs were viewed as more efficient in providing key information to PSD Project clients compared to any other source, e.g. chambers of commerce, other state or municipal institutions. Different modes of communications were utilized by RDAs including emails, telephone and direct communication with businesses. There was one case of a SMEE interviewed that reported a collaborative business relationship with another SMEE due to their meeting in an RDA-implemented training course. Strengths. PSD, the first G2G project developed by USAID/Serbia, has been a true opportunity for the NARD/RDAs to demonstrate that they are the foundations of private sector development in the country. The G2G approach offers the following strengths: 1. This program is more cost-effective than a standard USAID program (run by private contractors), better strengthens local linkages, and far better epitomizes the USAID Forward Initiative. 2. The G2G approach resulted in a relatively unique private sector development project with NARD having a central role in designing the processes and contours of project implementation, and RDAs doing most of the direct work with beneficiaries. Thus, NARD and RDAs were able to best play to their respective strengths. 3. The milestone approach developed to track the implementation process provided sufficient guidelines and kept the NARD/RDAs on the right implementing track, while simplifying progress monitoring by USAID. Additionally, operation of the online portal database enabled continuous insight into the PSD Project activities and reports, as well as systematic approach to documentation and data recording by all RDAs. 4. USAID/Serbia extended crucial and timely assistance to NARD, ensuring compliance with the USAID implementing requirements, in particular with respect to development of the Performance and Monitoring Plan (PMP), Grants Manual, Client Impact Survey and environmental assessment compliance for technical assistance and grant programs. 35 5. Private sector implementation projects by private contractors or from other donor agencies are often shorter-term (one or two years) and thus fail to offer as robust an opportunity for true development. Having four-years for NARD/RDAs to work with beneficiaries was seen as a big plus. RDAs appreciated that USAID’s approach with PSD really emphasized “content” over “form,” meaning that it permitted more than just superficial or one-dimensional efforts at development. 6. The three sectors chosen by USAID as focuses for PSD -- agribusiness, fashion (shoes and textiles) and light manufacturing were the correct ones for the targeted regions. Local implementers are often best in position to understand the contemporary economic realities of the regions in which they work and the beneficiaries that they serve. 7. The challenge grant and competitive bid processes for technical assistance and machinery developed by NARD have created a transparent, reliable way of operating, generating a lot of confidence in NARD/RDAs on the part of the enterprises. Because of their history of operating in the country and visibility, NARD and RDAs are able to engender local familiarity and trust far more readily than private implementers who may lack significant in-country experience. 8. Despite the fact that government entities are project implementers, USAID, NARD, and RDAs managed to successfully build transparency into every step, largely protected from political influence. SMEE respondents frequently remarked that processes were transparent and the NARD/RDAs implemented the project in a nonpolitical way. This effectively answers one possible concern about G2G implementation efforts. 9. The positioning of the RDAs as permanent, reliable local hubs for information and opportunities was viewed by all respondents as more effective as compared to other sources, e.g. chambers of commerce, other state or municipal institutions, etc. RDAs can connect to enterprises in a more meaningful, hyper-local way that cannot be replicated as easily by international implementers. Weaknesses. The weaknesses of the G2G approach can be summarized as followed: 1. Local project staff (particularly given their myriad responsibilities) may lack the deep knowledge and foundation to be able to train and advise businesses as effectively and flexibly as private contractors. This has tended to result in more “one-size-fits-all” training that does not afford great differentiation between the capacities of individual business owners. 2. The sudden dissolution of NARD and transition from a stand-alone agency to part of a joint agency (and any resulting turnover or uncertainty) evidence how a government implementer might be more affected by political realities than a private contractor implementer. While a substantial number of NARD/PSD employees will be invited to the new agency and there is great hope that project implementation will not be substantially affected, there is still a large amount of funding (~$2.1 million) left to expend. It is yet unclear whether (and if so, to what extent) the change to DAS will delay project implementation and whether any delay will affect program results or the strong relationships of trust that RDAs developed with their clients. 3. NARD/RDA staff seemed to have considerable abilities in business development, but there was a perceived shortfall in more advanced technical capabilities related to technical industrial processes and systems, market linkages through international trade fairs, vocational education, and banking and finance. We note an absence of banking and finance advisory capabilities in this program, despite a strong worldwide USAID focus on financial components to assist businesses and banks to find common ground in providing investment and working capital finance. This is one area where private implementers, with access to a wide network of international consultants, might offer an advantage. 36 Without access to finance, it may prove challenging for companies like the PSD beneficiaries to grow optimally.7 4. Having a government implementer sometimes results in logistical challenges that a private implementer would not. For example, there was a discrepancy in the US/Serbian fiscal years and local legislative changes with respect to submissions of official financial statements of businesses that occurred during the PSD Project implementation. Specifically, the deadline for submission of official financial statements was changed from February to June. For this reason, the final PMP data for the Year 2 (2015) will be available and verified after June 2016, far too late to be included in this evaluation. Conclusion to the Question: What are the strengths and weaknesses of USAID’s approach to G2G management? The Evaluation Team concluded that the model used was a unique government-run model that operated with a high level of efficiency, transparency and professionalism. USAID/Washington should note that this G2G model is generally performing quite well. Many developing countries would be fortunate to have a private sector development agency with the competency of NARD/RDAs. This is due in part to the blend of USAID contractual requirements and project methodology previously developed by NARD. The use of a network of highly competent RDAs is a strong way to advance the FORWARD Initiative that should be utilized in other developing countries. This G2G approach is more cost-effective than using international consultants. Despite the extent to which political realities may be more likely to affect a government implementer, there is confidence that the existing RDA networks and continuing NARD/PSD project staff will smooth out potential rough spots in the transition to the second phase of the PSD Project. Some recommendations are made below on the strategic targeting of growth firms, strategic value chain support, financial sector services and upgrading the RDA staff technical capabilities that can result in an even better performance than the first phase of the PSD Project. V. RECOMMENDATIONS FOR THE REMAINDER OF THE PSD PROJECT I. DRAWING ON PSD LESSONS LEARNED, HOW SHOULD USAID BEST FOCUS OR TARGET ITS LIMITED ASSISTANCE FOR STRENGTHENING THE PRIVATE SECTOR BUSINESSES IN TARGETED REGIONS IN THE FUTURE? As the reader of this Evaluation can appreciate, there are many aspects of the PSD Project that are positive and when taken in their entirety show a unique combination of support activities for the Serbian businesses in the targeted regions. However, it is obvious from the analysis above that not every component of the PSD was highly successful. We have indicated some potential areas for improvement in the next phase of project implementation. To summarize those PSD components that should undergo adjustment, we note the following: • PSD business training courses did not work for everyone and were sometimes cited as “too elemental” or basic. 7 USAID Serbia does have two projects touching upon access to finance: the Business Enabling Project and the Access2Finance Project with Opportunity Bank Serbia. No respondents mentioned these programs in interviews with the Evaluation Team, however. 37 • Mentoring and consulting services were not recognized by the businesses as an effective means to assist them. The RDA staff members and local consultants could use more training and experience with how to provide targeted, in-plant technical advice. • The association/cluster support activities did not result in joint, concerted action to achieve competitiveness. Linkages between businesses (value chains) were not achieved. • The attendance at international trade fairs did not result in many contracted and delivered sales or continued commercial relationships once the attendees returned from the fairs. What proved effective in the PSD Project? • Probably the most sought after and positive element of the PSD program was the “building block” technical assistance in the form of website development, on-line sales, certification, computerized accounting and production monitoring software installations. • RDA communications regarding a range of opportunities for the SMEEs (donor grants, trade fair information, bank loan promotions, B2Bs, regulatory changes, etc.) proved to be one of the strong pluses of the PSD Project. We again note that RDAs successfully developed an important relationship of trust and confidence with the PSD clients. • Business planning education for the start-ups, plus some modestly-sized grants for critically required equipment was a popular and positive element. This assistance resulted in establishment of new legally established businesses that should be further supported through extended mentoring and technical assistance programs. • Export training for companies not yet undertaking exports was particularly valuable to PSD clients and should be continued, with the addition of the recommendations on trade show market linkage execution included below. The Evaluation Team recommends that for the remainder of the project with the remaining funds that the following steps be taken: • Adjust the employment creation, trade fair sales indicators up to a point that they are challenging to the PSD implementer and address the needs of the SMEEs in the targeted regions. A bank financing indicator should be added to the performance indicators. • Develop a future call for technical assistance, for which the selection criteria should be adjusted to include the ability to identify and emphasize those enterprises involved in strategic growth activities or in value-chains as lead firms. These criteria will reduce the number of enterprises targeted, but would select those firms poised to expand sales, employment and exports within one year based upon recent verifiable performance and their business plans. Examples of this included the following: This recommendation also includes the concept that the RDAs will be able to focus greater amounts of cost-share grants within the remainder of the PSD to individual firms that are taking steps and achieving the fundamental indicators of sales, employment and exports. When this topic was discussed with one RDA, the director expounded on her vision for creating higher impact through the PSD (followed up by an email). Her concept was to focus on 20% of the enterprises to create 80% of the impact of the PSD. We concur with this approach and have seen it work in a transformative way in other USAID private sector development projects around the globe. To achieve this focus, the RDA staff must be prepared to follow up on the applications from companies with site visits. On these visits, RDA employees can better understand each step that a potential awardee would take towards greater levels of production and employment. PSD support might involve implementing in-plant vocational education strategies for expanding their workforce; adding critical equipment and technology as well as access to financing for 38 plant space; or using short-term technical advice by European and U.S. industrial experts on product refinement to overcome what seems to be a lag in competitiveness and quality of Serbian industrial production. These short-term technical advisors should be required to provide a short course to RDA staff to help them expand their technical skills and understanding of production processes of strategic and high-potential growth SMEEs. To achieve this kind of strategic company-level investment, we recommend that highly-qualified financial specialists be added to the PSD team who can help individual firms that are poised to grow to obtain bank financing to achieve this growth. The Evaluation Team, based on its interviews with trade fair participants supported by the PSD and reflecting upon the PSD results indicators, believes that a stronger initiative needs to be undertaken by the RDAs to raise the level of success for SMEEs desiring to export: • Overall, the PSD implementer/RDAs must carefully align SMEEs for participation in trade shows that are calibrated to their particular lines of production, quality levels and language capabilities. • RDAs should conduct company-focused market research and B2B scheduling prior to trade fairs so that participants are able to get the most out of their trade fair experiences. Trade fair participants must be armed with information about prospective buyers (where in the fair are they located? When is the optimal time to interface with them?) as well as competitors (where are they located? What can be learned from interacting with them?) Initial trade fair participation should be a learning opportunity, so participants should not be pressured to rent booths or bring samples. The goal should be to determine which buyers might be open to purchase and to better understand expectations of quality, price, volume, and delivery. • After the trade fair, RDAs should spend targeted time with each participant to determine what type of specific support is required for the next stage of this process. Technical production advisors could visit the plant to assess how to improve quality or solve operational issues. RDAs could also help SMEEs work through potential “what if” scenarios (e.g. “How would you respond if a potential trade partner proposes an arrangement that you could only meet by curtailing production on another line?”), to better position them for future potentialities. • For the second stage of this process, trade fair attendance should continue to be carefully targeted for individual SMEEs. Participants should be supported with cost-sharing of sample transport costs (especially where, as in the case of heavier items, transport costs are high). This round should feature B2B breakfasts or other meetings to integrate sellers with potential buyers. This stage would also feature delivery of samples for test marketing, quality assessments, laboratory testing or whatever else is required. • The third round of trade fair participation would occur after attendees had received mentoring from RDAs regarding likely contract targets and how to make promotional pitches, and were thoroughly able to answer questions regarding plant capacity, logistics, delivery experience, etc. This multi-stage approach means that To help develop this strategy, the Evaluation Team encourages USAID to bring in a regional or international trade show expert to provide learning courses in the South and Southwest regions, and to work with RDAs. This would be a prime opportunity for the ITC and market research training module which was planned for RDAs, but has not yet been implemented. The Evaluation Team recommendation on the training provided to the PSD clients is based on our overall view of this training as positive and the PSD Project should continue the courses for additional promising SMEEs as a means up raising the level of business acumen in these regions as well as continue to create the relation-building of the RDAs. We further recommend that any 39 contemplated SMEE trainees to identify potential participants who already have up-to-date business experience/education to have the option of bypassing the basic training modules. Upgrade the RDA staff training capabilities through seminars with visiting sectoral and international marketing experts. To the extent that funds allow, USAID should continue the “access to technology” grants for SMEEs and machinery grants for start-ups. We conclude that the assistance to youth and women start-ups is operating mostly as it should. The 100% grant levels are unique and the business planning training is welcomed by participants. To ensure sustainability of start-ups, mentoring process by RDAs could be extended beyond 50 hours if requested by participants. II. HOW CAN USAID SERBIA STRENGTHEN THE LINKAGES BETWEEN DIFFERENT PSD PARTNERS DURING PHASE II? The Nathan Associates report, which identified the three strategic sectors, also laid a strong emphasis on value chains. USAID has learned across the world that a focus on well-chosen value chains can shift the economic picture for a country in a relatively short time. Serbia is especially well-situated to begin to take advantage of value chains, exporting immediately into the Balkans region and shortly thereafter into the European Union. However, no substantial value chain work was evident in the PSD Project approach as assistance was instead provided to associations/clusters. USAID/Serbia should consider giving weight in a call for assistance that would help a company proposing to operate as the “lead firm” in a value chain to map, plan the development of and implement development in this way. This would enable PSD to affect positively an even greater number of beneficiaries and make production improvements and raise volumes over broader areas. This could affect the incomes of numerous low-income smallholders. As noted above, RDAs pointed out that supporting primary agricultural production in combination with agro-processing would probably yield more results in the agribusiness sector. Our interviews with dairies, honey producers, fruit growers and grain producers (e.g. buckwheat and spelt) indicated that a special call should be developed to encourage these operations to demonstrate how they would, as lead firms, provide technical support to their suppliers to raise product quality and volumes and become instrumental in access to finance on the part of smaller producers. The two interviewed dairies, for example, were purchasing milk from 1464 smallholders that really are not being assisted to expand production and improve quality. This call for assistance should also be extended to the associations to determine whether they can envision one of their members playing the lead firm role, purchasing the production from smaller producers. It could also be that the association/cluster takes on the role of lead firm providing the quality control and inspections, traceability mechanisms, certification of the final product. Similar to what the Novi Pazar Shoe Producers Association or Agro Pešter Cluster have envisioned for the future, the association could also provide the joint logo, common brand and joint marketing. Either the association or a strong member company should actually take control of the merchandise and be responsible for sales and delivery to clients. While the concept of the cluster has fallen to the wayside in international development, the value chain concept should be strengthened in the PSD project, particularly in those value chains that are aiming to export in the near future. If the associations/clusters cannot respond to a call for applications that involves value chains, the Evaluation Team recommends that assistance to associations be shifted to the SMEEs that could achieve strategic growth. Value chains, like the strategic businesses, discussed above know their businesses and understand the subsequent steps linkages that they need to take to grow their businesses, improve quality and expand their markets. 40 ANNEXES ANNEX 1. QUESTIONNAIRE FORMATS PSD PROJECT EVALUATION QUESTIONNAIRE GOVERNMENT REPRESENTATIVES (NARD & RDAs) Respondent(s) Name(s): _________________________________________________________ Organization and Location: ______________________________________________________ Position(s): ___________________________________________________________________ PSD PROJECT 1. How specifically has your organization contributed to implementation of the PSD? What resources that were not provided by USAID did your organization provide to achieve the targeted PSD results? _________________________________________________________________ ________________________________________________________________ Financial (what level of Serbian government financial resources have been provided through your organization to the PSD Project to date?) US$__________________ Personnel (how many full-time NARD and RDA staff have been involved with the PSD Project?) ______________ 41 Other ________________________________________________________________ 2. Based on the overarching results agreed for the PSD Project at its conception on June 21, 2013, what were the PSD Project/your region’s actual results to date? a. Total PSD/Regional sales increases of SMEEs $___________ b. Total PSD/Regional sales increases of W/Y enterprises$ _________ c. Total PSD/Regional sales increases of Associations $________ d. Number of total PSD/Regional jobs created supported _____ Full-time _____ Part-time _____ Seasonal ______ Were revenues and jobs created in your region or municipality consistent with targets set in the project’s performance monitoring plan? Y ___ we achieved or exceeded the revenue targets N ____ Y ___ we achieved or exceed the job creation targets N ___ e. Number of new PSD/Regional business services created ______ f. If yes, what new business services provided? _______________________________ __________________________________________________________ g. Number of PSD Project/Regional business skill training courses since June 2013 delivered ______ h. Number of PSD/Regional training course attendees _____ i. What was the number of PSD/Regional SMEEs receiving capacity building to export? _____ j. What number of persons were trained in exporting skills? _____ SMEES 42 3. In your opinion, were the priority PSD sectors (fashion – shoes and textiles; agribusinesses; light manufacturing) well selected? Y___ N___ Maybe ___ In your region, what is the number of SMEEs in the PSD Project in each sector? F ____ A ____ LM ____ Approximately what percentage of SMEEs active in each sector in your region participates in the PSD Project? F ____% A ____% LM ____% Are there other sectors that might have been selected that would have provided an even higher result for the south and southwestern regions? Y ___ N____ U ___ If yes, what other sector (s) might have been chosen? ______________________________ 4. To date, do you think that technical assistance activities provided through PSD have proven valuable for each sector, especially in terms of contributing to the improved competitiveness of SMEEs? Y___ N___ U ___ If N or U, what are some ways that these activities might be improved? _________________________________________________________________ _________________________________________________________________ 5. Did your organization evaluate the potential PSD participants and grantees? Y ___ N ___ 6. If Y, what criteria did you use to determine their potential for growth and impact? ________________________________________________________________ ________________________________________________________________ 7. To what extent do you feel that the beneficiaries selected could be considered among the strongest enterprises in sector, municipality, or regional economic growth? Y ___ N ___ U ___ Can you cite a particular example? ____________________________________________ 43 Can you cite another example? _______________________________________________ In your view, what would be the success story from assistance provided by PSD Project in your region? _____________________________________________________________ _____________________________________________________________ 8. Do you feel that project selection criteria were sufficiently transparent and straightforward to enable SMEEs to easily become project beneficiaries? (5 highly transparent and workable; 1 complicated and not really transparent) ____ How about associations/clusters? (5 highly transparent and workable; 1 complicated and not really transparent) ____ How about youth/women entrepreneurs? (5 highly transparent and workable; 1 complicated and not really transparent) ____ How might this process be improved further? ____________________________________ ________________________________________________________________ ________________________________________________________________ 9. Besides the initial group of SMEEs, have other SMEEs subsequently become a participant of the PSD Project in your region? Y ___ N ___ If yes, how many? _____ How about associations/clusters? Y ___ N ___ Other youth/women-owned start-ups? Y___ N ___ 10. Have SMEEs supported by PSD increased their sales into new domestic and international markets as a direct result of that support? Y ___ N ___ U ___ In your opinion, would SMEEs generally be able to increase their sales without PSD assistance? Y ___ N ___ U ___ 44 If N, what are some examples and what were the major factors for the SMEEs’ increased competitiveness? ______________________________________________________________ ______________________________________________________________ 11. Have SMEEs supported by PSD increased the overall employment in the companies as a direct result of that support? Y ___ N ___ U ___ If Y, what are some examples where that support was directly helpful in employment creation? ________________________________________________________________ _________________________________________________________________ 12. Have SMEEs supported by the PSD Project developed new competitive skills as a result of the PSD- sponsored training or workshops? Y ___ N ___ U ___ If Y, which training courses and resulting skills seem to give the best results? ________________________________________________________________ ________________________________________________________________ 13. Do SMEEs Have sufficient financial resources to maintain or further improve their employee training or support the competitive skills already gained through PSD training? Y ___ N ___ U ___ 14. How well do you feel PSD through your organization has provided capacity building or training assistance for specific sectors? (5-1; 5 excellent and 1 not very good) In which sectors, has the impact been the highest? _______________________________________ YOUTH/WOMEN ENTREPRENEURS 15. How have PSD efforts directly benefited existing youth and female entrepreneurs? 45 Financial assistance ___ If yes, what has been the average size of the grant given? _____ If yes, what were the grants used for? _________________________________________ Business planning ____ How many business plans were supported? ____ Business skills training ____ How many training attendees since the beginning of the PSD Project? _____ Other assistance __________________________________________________________ 16. How many youth and female entrepreneurs supported by PSD would you say become self-sustainable? _____ 17. For youth or female entrepreneurs that are not yet self-sustainable, do you anticipate that they will be by the end of PSD funding in 2017? How many? _________ 18. If not, why not? What have been the primary obstacles? ________________________________________________________________ _________________________________________________________________ 19. Do these entrepreneurs have access to outside financial resources, in addition to the project support, to implement their business plans? Y ___ N ___ U ___ In your opinion, how many of these small entrepreneurs could qualify for a microfinance loan? ____ In your opinion, how many could qualify for a commercial loan? ____ Can you cite an example of a youth/female entrepreneur who has obtained outside financing? _______________________________________________________________ 20. What is a success story of youth or women entrepreneurship in your regions? __________________________________________________________________ 46 ASSOCIATIONS/CLUSTERS 21. Which PSD-supported associations/clusters operate in your region? ___________________ __________________________________________________________________ 22. To what extent have sector associations and clusters increased their membership with the PSD Project support? _____ new members. 23. In your opinion, are the associations/clusters in your region financially and operationally sustainable? Y ___ N ___ U ___ If not, why not? ________________________________________________________________ ________________________________________________________________ 24. For associations/clusters that are not yet sustainable, do you anticipate that they will be by the end of PSD funding in 2017? Y ___ N ___ U ___ What challenges do the associations/clusters in your region generally face? ________________________________________________________________ 25. In your opinion, to what extent has the PSD project resulted in increased linkages between small businesses and associations/clusters (5 to 1; 5 = strong linkages formed; 1 = not too much evidence of linkages) ____ Can you provide some examples? _____________________________________________ 26. In your opinion, to what extent has the PSD project resulted in increased linkages between small businesses and other government or donor support projects (5 to 1; 5 = strong linkages formed; 1 = not too much evidence of linkages) ____ Can you provide some examples? _____________________________________________ 27. Do you believe that the PSD Project been successful in increasing number of available business services through your organization? Y ___ N ____ 47 If Y, which types of new business services are being provided? ________________________________________________________________ 28. Besides your organization, are similar business services being provided by other government support organizations or other donor programs? Y ___ N ___ If yes, please cite the other government programs ________________________________________________________________ _________________________________________________________________ 29. Were any private service providers organized as a result of the PSD Project’s support? Y ___ N___ If Y, please name them. __________________________________________ ________________________________________________________________ 30. Do the SMEEs pay for business services? Y ___ N ___ U ___ Do you feel that SMEEs are willing to pay for such services Y ___ N ___ U ___ EXPORT DEVELOPMENT 31. Do you think that PSD project participants can meet demands of export markets in terms of: sufficient quantity, adequate quality/standards, and price competitiveness? Y ___ N ___ U ___ 32. What are the key obstacles for Serbian enterprises in export markets? __________________ ________________________________________________________________ 33. Has PSD Project taken any technical assistance actions to overcome these key obstacles? Y ___ N ___ If Y, what are they? _______________________________________________________ 48 34. For which exports do you feel PSD through your organization has provided the best capacity building or training assistance? _______________________________________________ 35. How have PSD Project participants been able to meet export market standards as a result of this capacity building assistance? ______________________________________________ ________________________________________________________________ 36. How effective has PSD Project been in creating the linkages among different PSD businesses? (5 -1; 5 a lot, 1 not really) ____ Have these linkages or cooperation led to more exports? Y ___ N ___ U ___ 37. If you feel these linkages have been effective, how specifically did your organization ensure synergy and cooperation among different partners and Project participants? ________________________________________________________________ 38. What else might be done to encourage these kinds of relationships/synergy? ________________________________________________________________ PSD GENERAL OBSERVATIONS 39. If you know, what have been the strengths and weaknesses of USAID’s approach to G2G management for supporting the Serbian Government to be effective in programs like the PSD Project? _____________________________________________________________ ________________________________________________________________ 40. Throughout the PSD Project to date was your organization able to efficiently monitor individual enterprises activities and to identify and address issues and concerns? Y ___ N ___ U ____ 41. In your opinion, were the PSD Project indicators and targets realistically defined? Y ___ N ___ U ___ 49 Were they too high or too low? (5 too high and unrealistic; 1 low and easy to achieve) _____ 42. Would you say that the PSD Project(s)’s implementing approach still effective in achieving the proposed results? Y ___ N ___ U ___ If N or U, what changes should be made? ______ ________________________________________________________________ 43. What is your opinion about current restructuring of NARD into the Development Agency of Serbia, and will it affect the PSD Project implementations? ________________________________________________________________ ________________________________________________________________ 44. What further actions could be undertaken to ensure effectiveness on the part of the PSD Project? ________________________________________________________________ ________________________________________________________________ 45. Do you have any other comments that have occurred to you about the PSD Project? ________________________________________________________________ 50 PSD PROJECT EVALUATION QUESTIONNAIRE SMEE AND YOUTH AND WOMEN START-UPS Initiate the interview with the statement that all these responses will be confidential in that there will be no specific attribution to any enterprise, association/cluster or other organization. The respondent is encouraged to speak frankly and fully about their experience with the PSD Project. The purpose of the interview is to help improve the PSD Project over its next several years of operations. PSD RELATIONSHIP AND ENTERPRISE CHARACTERISTICS 1. What is the size of your business in terms of number of employees? _________ 2. In which sector do you operate: F ____ A ____ LM ____ Other ____ What is your core activity? _____________________________________________ 3. Are you a youth or female entrepreneur/owner? Youth ____ Female ____ 4. In what year did you start your business? 5. Which RDA has provided you with PSD assistance? ___________________________ 6. Did your enterprise receive a grant from the PSD Project? Y ____ N ____ If yes, what was its value? ______________ How long did the procedure for approval and reimbursement take and what was your recollection of how the process went? _____ months 7. Could you tell us why you think that your enterprise received the grant? __________________________________________________________________ __________________________________________________________________ 8. What specific kinds of assistance has the PSD Project provided you? 51 Training ___ Access to new technology Grants _____ Trade Fair Participation _____ 9. Has your enterprise increased its sales into new domestic markets as a direct result of the PSD support? Y___ N___ Please elaborate. ENTERPRISE COMPETITIVENESS 10. If your enterprise has experienced increased revenues from new or expanded markets, has that led to new and sustainable job creation within your enterprise? Y___N___ 11. Do you have any problems in using these new technologies? Y____ N_____ Elaborate. TRAINING 12. Please describe the length and content of the relevant training course(s) you or your employees have received through PSD Project. ________________________________________________________________ ________________________________________________________________ 13. Have these new skills from the training courses assisted in the growth of your enterprise? Y ___N___ U___ If yes, please explain your answer. _____________________________________________ 14. Has the PSD training changed management practices at your company? Y___ N____ Elaborate. FINANCIAL VIABILITY 15. Do you feel that your enterprise has enough financial resources, in addition to the project grant support, to implement your business plan for 2016? Y___ N___ U___ 16. Have you ever gotten a commercial loan from a bank? Y ___ N ___ 52 17. If you cannot access such loans, what is your main source of your operating capital? Personal ___ Family ___ Buyer financing ___ Informal lender ___ Other ___ EXPORT SUPPORT 18. Have you or any person related to your enterprise received specific capacity building assistance for exports through PSD Project? Y___ N___ U___ 19. Has your enterprise increased its sales into new domestic or export markets as a direct result of the PSD support? Y ____ N ____ Indicate domestic or export ______________ 20. With PSD support, did your enterprise participate in an International Trade Show(s)? Y___ N ___ Did your enterprise achieve sales or contracts at this International Trade Show? Y ___ N ___ If Y, please provide locations and values. ______________________________________ 21. Do you think that your enterprise can meet demands of export markets in terms of: sufficient quantity, adequate quality/standards, and price competitiveness? Y ___ N ___ U ___ What are the key obstacles? __________________________________ 22. Has PSD provided any technical assistance actions to help overcome these key obstacles? Y ___ N ___ If Y, what are they? _____________________________________________ ________________________________________________________________ OTHER TECHNICAL SUPPORT 23. To what extent do you feel that the PSD project(s) resulted in increased linkages between your enterprise and other regional/national governmental support 53 organizations? Please give specific examples __________________________________________________________ 24. What kind of information do you receive from the RDA that’s important for your business? Elaborate ________________________________________________________________ 25. Since 2013, have business/advisory services been made available to your enterprise from sources other than the PSD Project? Y ___ N___ If Y, what were they? Other government offices ___ Sector associations ____ Chambers of commerce ____ Other programs ____ To what extent have you utilized these business services? (5 extensively; 1 minimally) ____ 26. Are there any private service providers independently providing such business service support? Y___N___U___ If Y, what kind of services are available? ______________________________________ If yes, do they charge fees that are affordable by your enterprise? Y___ N___ U____ 27. Overall, did the RDA mentoring the progress of your business so that they could provide assistance when it was needed? Y ___ N ___ U ___ 28. Did you participate in the PSD Client Impact Survey? Was the survey able to adequately capture your experience and results generated by working with PSD? GENERAL QUESTIONS 29. What other type of assistance would you like to receive for your enterprise from PSD project in upcoming year? _________________________________________________________ 30. What are your business goals in terms of annual sales growth percentage in 2016? ______% 54 31. What is your enterprise’s potential for growth in sales/revenues over the next 5 years? _______________ 32. What type of assistance is the most needed for your sector that PSD project could provide? _________________________________________________________________ 33. Is there any other comment that you would like to make about the PSD Project? _________________________________________________________________ _________________________________________________________________ Respondent Name: Organization/Company: Position: Specific PSD Initiative(s) Worked On or Familiar with: 55 PSD PROJECT EVALUATION QUESTIONNAIRE ASSOCIATIONS/CLUSTERS Respondent Name: Organization and Location: Position: 1. Has your Association/Cluster received financial assistance in the form of a grant from the PSD Project since June 2013? Y ___ N ____ If Y, how much financial assistance has your organization received? ___________________ 2. Can you provide your recollection of how the proposal or grant approval process went? _________________________________________________________________ _________________________________________________________________ 3. How many enterprise members does your Association/Cluster have? _________ How is your Association financed? _________________________________________________________________ __ What portion of your financing comes from membership fees? ______% Do you consider your Association/Cluster financially sustainable? Y ___ N ___ U ___ If N or U, will it be sustainable by the end of the PSD Project? Y ___ N ___ U ____ 4. What are total annual revenues of your Association/Cluster? _________________ 5. Does your Association/Cluster actually become involved with the business transaction as an agent, integrator of products, shipments etc. Y ___ N ___ If Y, do you earn a commission for those transactions? Y ___ N ___ If Y, how much income is this for your Association? ________________________ 6. Since your Association/Cluster began working with the PSD Project, how many new members have joined? _____________ 56 If you have new members, what is the gender of these new members? W ___ M ____ How many of the new members have been youth? ______ 7. Has this Association/Cluster received technical advisory services through the PSD Project? Y ___ N ___ If yes, what were the specific services that you received? Market strategies and advice ____ Management and organizational advice _____ Financial management advice ____ Mentoring ____ Other ____ MEMBERSHIP PERFORMANCE 8. Do you feel that your members are generally more competitive now as a result of its working and receiving training from the PSD Project? Y___ N___ U___ Please explain your answer ________________________________________________________________ 9. Has your association experienced increased revenues because of PSD’s assistance in opening up new or expanded markets? Y ___ N ___ U ____ 10. Have increased sales of your members led to new and sustainable job creation within their enterprises? Y___N___ 11. If Y, what kinds of jobs and how many have been created? Full-time _____ Part-time _____ Seasonal _____? 12. Have your members adopted new technology(ies) as a result of PSD assistance? Y ___ N __ If Y, please give examples ________________________________________________________________ ________________________________________________________________ 57 13. Do your members sufficient resources (financial, technical, personnel, etc.) to fully implement and continue using these new technologies? Y___ N___ U___ 14. How effective has PSD been in creating linkages among your members’ enterprises? (5 -1; 5 a lot, 1 not really) ______ 15. If you feel it has been effective, how specifically did your association’s role in PSD support synergy and collaboration among different partners and Program participants? Collaborated in marketing ____ Collaborated in aggregate their goods for sale ____ Collaborated in production ____ Other ____ 16. What else might be done to encourage this kind of collaboration and synergy? _________________________________________________________________ _________________________________________________________________ TRAINING 17. How well do you feel that PSD through your organization has provided capacity building or training assistance your sector? (5-1; 5 excellent and 1 not very good) _______ 18. Please describe the length and content of the relevant training course(s) you or your members have received through PSD Project. _________________________________________________________________ _________________________________________________________________ 19. Have these new skills gained from PSD training courses assisted in the growth of your members’ enterprises? Y ___N___ U___ If yes, please explain your answer. _____________________________________________ 20. Have your members adopted new management practice(s) as a result of PSD training courses? Y ___ N ____ U ___ 58 21. If Y, what are some examples? _________________ ______________________________ 22. Do the members have sufficient resources (financial, technical, personnel, etc.) to fully implement and maintain these management practices? Y___ N___ U___ 23. Have your members adopted new technology(ies) as a result of PSD training courses? Y ___ N ____ U ___ 24. If Y, what are some examples? ________________________________________________ EXPORT ASSISTANCE 25. Have your members received specific training to assist them to expand or achieve export sales? Y___ N___ U ____ Can you elaborate? ________________________________________________________ 26. Has your Association/Cluster exhibited at an International Trade Fair with the assistance of the PSD Project? Y ___ N ____ If yes, could you please provide details ________________________________________________________________ ________________________________________________________________ 27. How many of your member’s enterprises have attended an International Trade Fair as a result of the PSD Project financing and other support? ____________________________ 28. What number of contracts was signed by the Association/Cluster or its members at an International Trade Fair where attendance was facilitated by the PSD Project? __________ Value? Were the contracts fulfilled? Y___ N ____ If N, why not? ________________________________________________________________ ________________________________________________________________ 59 29. What kind of information do you regularly provide your members? Market information and opportunities ____ Technical advice ____ Other ____ How do you do this? Email ___ Regular mail ___ Meetings ____ Other ___ 30. Have you provided them any information that has come to you from the PSD Project? Y ___ N ___ U ___ If Y, please give an example _________________________________ GENERAL QUESTIONS 31. Would you say that the PSD Project(s)’s implementing approach for assisting associations is effective in achieving the proposed results? Y ___ N ___ U ____ If N, what changes should be made to the Project? ________________________________ _________________________________________________________________ _______ 32. When your association/cluster became a participant in the PSD Project, were specific indicators and targets (sales, employment, etc.) set? Y___ N___ U____ If Y, in your opinion were these indicators realistic? Y___ N ___ U ___ 33. Based on your knowledge of the PSD Project, how would you rate the overall impact that PSD has achieved in your region? (5 very high impact-1 very low impact) _____ 34. Do you feel that the associations selected could be considered “drivers” of sector, municipality, or regional economic growth? Y ___ N ___ U ___ Please elaborate? __________________________________________________________ 35. Based on your experience, will your association/cluster continue to participate in the PSD Project as long as possible? Y___ N___ 36. What advice would you give to another association regarding PSD’s role and assistance to maximize the results achieved by their participation? 60 _________________________________________________________________ _________________________________________________________________ 37. What other type of high priority assistance for your sector would you like to receive from PSD project in upcoming year? ________________________________________________________________ 38. What are your association’s goals in terms of annual sales growth percentage in 2016? ___% 39. What is your association’s potential sales/revenues growth over the next 5 years? ________% 40. What type of assistance is the most needed for your sector that PSD project could provide? ________________________________________________________________ ________________________________________________________________ 41. Do you have any other comment regarding the effectiveness or lack of effectiveness of the PSD Project? _______________________________________________________________ ________________________________________________________________ 61 ANNEX I1. LIST OF CONDUCTED INTERVIEWS Date Name of the organization Interviewed persons Municipality Type of organization Sector Core activities Type of assistance provided Year of establis hment Number of employees Female business 27.1.2016 National Agency for Regional Development – NARD Ana Žegarac; Katarina Aksentijevic Belgrade RDA RDA 13 27.1.2016 USAID Belgrade USAID USAID 28.1.2016 Regional Development Agency – Leskovac (Center for Development of Jablanica and Pčinja Districts) Biljana Stanković Jelena Pavlović Dragana Belenzada Dobrila Sudimac Mratinković Zoran Stojiljković Leskovac RDA RDA 28.1.2016 Regional Fruit Cluster of South Serbia (Regionalni voćarski klaster Južne Srbije) Milan Živković Leskovac Association Agribus iness Fruit and vegetable production, 15 members • Training • Mentoring&Consulting • TA for web site development • TA for fruit lab • TA for e-commerce • Three trade fairs 2013 1 Full-time No 28.1.2016 Jablanica Fruit Dragan Kulić Medvedja SMEE Agribus iness Purchase and cold storage of forest fruits • Training • Mentoring&Consulting • TA for purchase of cold storage equipment 2001 8 Full-time 30-40 Seasonal No 28.1.2016 SZTR Tomica Tomica Stamenković Medvedja SMEE Light manufa cturing Wood processing • Training • Mentoring&Consulting • Trade fair in Munich 2007 3 Full-time 20 Seasonal No 62 28.1.2016 Strela ltd Aleksandra Prokopović Leskovac SMEE Agribus iness Purchase and processing of forest fruits, fruits and vegetables • Training • Mentoring&Consulting • TA for production software • TA for web site development • Two trade fairs (Food & Life, Munich) 1991 86 Full-time 100 Seasonal No 28.1.2016 Tomaco Line Ltd Marina Kostić Leskovac SMEE Agribus iness Production of candies and waffles • Training • Mentoring&Consulting • TA for development production elaborate to adjust recipes for foreign markets • Trade fair (Food & Life, Munich) 2008 48 Full-time 70 Seasonal No, Majority foreign ownership 29.1.2016 White Stars Ltd Miroslav Ristić Bujanovac SMEE Agribus iness Production of honey • Training • Mentoring&Consulting • TA for development of product visual identity 1990 4 Full-time 2 Seasonal No 29.1.2016 Tomyplast Ltd Ružica Zdravković Bujanovac SMEE Light manufa cturing Dairy - production of cheese • Training • Mentoring&Consulting • TA for development of new labels 2012 13 Full-time 3 Part-time No 29.1.2016 Irena Stamenkovic Irena Stamenković Bujanovac Women/Youth Agribus iness Production and processing of mushrooms and pine honey • Training • Mentoring&Consulting • Grant for cooling system and production machinery 2015 2 Full-time 4 Part-time Yes 29.1.2016 Fluidi Bujar Mustafa Preševo SMEE Agribus iness Production of juices • Training • Mentoring&Consulting • Two trade fairs (Food & Life, Munich, Germany) 1994 65 Full -time 15-20 Seasonal No 29.1.2016 Demiri Envera Nehmie Demiri Envera Nehmie Preševo Women/Youth Light manufa cturing Production of adhesive tapes • Training • Mentoring&Consulting • Grant for equipment for cutting of tapes 2015 2 Full-time Yes 29.1.2016 ZENITI OD Medat Zulfiu Preševo SMEE Light manufa cturing Building materials (steel nets) • Training • Mentoring&Consulting 2002 15 Full-time 3-5 Seasonal No 30.1.2016 Klameks/SIK Ltd Branislav Vranje SMEE Light manufa Sales of pneumatic tools and production • Training • Mentoring & Consulting 1993 14 Full-time No 63 Stanković cturing of staples for furniture • TA for recertification of ISO standard • Trade fair (Munich, Germany) 30.1.2016 Ukras Ltd Svetozar Milovanović Vranje SMEE Textile Production of protective gloves • Training • Mentoring&Consulting • TA for CE mark certification • Trade fair (Celje, Slovenia) • Ongoing TA for certification of ISO 9001 and ISO 14000 1993 4 Full-time 3 Seasonal Yes 30.1.2016 Cerak Dragan Đorđević Vranje SMEE Light manufa cturing Production of PVC and Al window frames, doors and fences • Training • Mentoring&Consulting • TA for certification of ISO 14000 and ISO 18000 • Four trade fairs (2xCelje, Slovenia, 2xMunich, Germany) 1989 8 Full-time 8 Seasonal No 30.1.2016 Hanibal Ltd Milica Dimitrijević Vranje SMEE Light manufa cturing Production of inner springs for mattresses, pocket springs for mattresses, mattresses and furniture • Training • Mentoring&Consulting • TA for web site development (not implemented) • Trade fair (Celje, Slovenia) 1989 6 Full-time Yes 1.2.2016 Regional Development Agency – Kraljevo (Regional Agency for Spatial and Economic Development of Raška and Morava Districts) Radojka Savić Jelena Jevtić Kraljevo RDA RDA 1.2.2016 Ljin Ltd Ivanka Pendić - Filipović Raška SMEE Agribus iness Dairy - Processing of milk and production of cheese • Training • Mentoring&Consulting • TA for HACCP certification • TA for ISO 9001 certification 1990, 1994 dairy 18 Full-time Female manager 1.2.2016 SZR Čaki Termo tim Danka Paunović - Terzić, Zoran Raška SMEE/Women Light manufa Processing of metals, production of heating boilers and • Training • Mentoring&Consulting 2014 2 Full-time Yes 64 Terzić cturing production of lamps • Grant for welding equipment and compressor • Trade fair (Celje, Slovenia) 1.2.2016 Regional Development Agency – Sandžak (SEDA) Samir Kačapor Džemil Huseinović Almir Šaćirović Novi Pazar RDA RDA 1.2.2016 Pešter Agro Cluster (Pešter Agro klaster) Ertan Bogućanin Novi Pazar Association Agribus iness Production of milk and dairy products, 40 members (5 dairies) • Training • Mentoring&Consulting • Trade fair (Slovenia) • TA for examination of water in Tutin • TA for design of joint￾brand products (redesign of web site, redesign of logo, redesign of packaging) • Purchase of packing machine • TA for development of joint services of cluster (development of rules and procedures) 2012 1 - manger No 2.2.2016 “NP Shoes” Association of Footwear Manufacturers (“NP Shoes“ asocijacija obućara) Senad Abdović Novi Pazar Association Light manufa cturing Association of shoe makers, 9 members + municipality • Training • Mentoring&Consulting • TA for German market research • Trade fair attendance (2 trade fairs in Dusseldorf) • TA for development of promotional materials (brochures and catalogues) • Purchase of equipment for association (laptops) 2013 1 manager No 2.2.2016 Kop promet Ltd Emir Koca Novi Pazar SMEE Textile Production of clothes (COP brand) • Training • Mentoring&Consulting • TA for web site development and online 1997 40 Full-time No 65 sales • TA for development of software for tailoring 2.2.2016 Amra Čuljević (Protex Has) Amra Čuljević Tutin Women/Youth /SMEE Textile/ Light Manufa cturing Production of textile covers (for furniture, sofas, floors, etc.) • Training • Mentoring & Consulting • Grant for machine to produce covers 2015 1 Yes 2.2.2016 Elan MMS Ltd Sarik Fakić Tutin SMEE Light manufa cturing Production of furniture • Training • Mentoring&Consulting • TA for new production and accounting software 1993 80 Full-time No 2.2.2016 Beni -komerc Ltd Ninioslav Marić Sjenica SMEE Agribus iness Dairy - Processing of milk, production of cheese, kajmak, peppers in sour cream • Training • Mentoring&Consulting • TA for application of software for accounting and monitoring purchase of milk 1991 10-12 No 2.2.2016 Sanatex Ltd Senad Prepoljac Sjenica SMEE Textile Production of female lingerie • Training • Mentoring&Consulting • TA for accounting software 1991 424 Yes 2.2.2016 The Carpet of Sandzak (Sandžački ćilim) Indira Kuburović Sjenica Association Textile Production of carpets, 25 members • Training • Mentoring&Consulting • TA for: Belgrade trade fair attendance; Development of promotional materials and catalogues; Purchase of materials for production of samples • Purchase of association equipment (printing machine, loom, laptop) • Ongoing TA for web site development, promotional material, new materials for samples for trade fairs in Belgrade and Vrnjacka Banja 2006 1 No 3.2.2016 Aleksandar Varaklic Aleksandar Varaklić Prijepolje Women/Youth Agribus iness Drying of fruits, vegetables and • Training • Mentoring&Consulting • Grant for tunnel fruit 2015 1 No 66 medicinal herbs drying machine 3.2.2016 Idi -Ninčić Ltd Igor Ninčić Prijepolje SMEE Light manufa cturing Production and trade of spare parts for motor vehicles • Training • Mentoring&Consulting • TA for certification of ISO 9001 • Two trade fair attendances (Celje, Slovenia and Munich, Germany) 1992 12 Full-time 5-6 Part - time No 3.2.2016 Gold Star Plus Ltd Aida Hasanagić Prijepolje SMEE Textile Production and retail sale of clothes • Training • Mentoring&Consulting • TA for certification of ISO 9001 • Trade fair attendance (Dusseldorf, Germany) 1993 54 Full-time No 3.2.2016 Dekor Ltd Aleksandar Zaković Priboj SMEE Light manufa cturing Production of furniture and paints and varnishes • Training • Mentoring&Consulting • TA for certification of ISO 9001 and ISO 14000 (not implemented) 1991 19 Full-time No 3.2.2016 Boka R Borko Radović Priboj SMEE Textile Production of hunting and working uniforms • Training • Mentoring&Consulting • TA for development of web site • Two trade fair attendances (Celje, Slovenia and Dusseldorf, Germany) 2006 4 Full-time 3 Seasonal Yes 3.2.2016 Poliester grupa Ltd Slaviša Janjušević Priboj SMEE Light manufa cturing Production of antihail rockets and pipes from composite materials • Training • Mentoring&Consulting • TA for development of new web site • TA for development of new product (not implemented) 1962 (2008 existing owners hip structur e) 44 No 4.2.2016 Agro- Zlatar ent Jekoslav Purić Nova Varoš SMEE Agribus iness Production of buckwheat and spelt flour • Training • Mentoring&Consulting • TA for development of new packaging • Trade fair attendance (Munich, Germany) • Ongoing TA for 2003 2 Full-time 4-5 Seasonal No 67 development of web site 4.2.2016 Novatex Ltd Nadežda Kolašinac Nova Varoš SMEE Textile Production of working and protective uniforms • Training • Mentoring & Consulting • TA for web site development and e-shop (online sales) • TA for development of new products (design and modeling) 2006 3 Full-time 8 Part time No 4.2.2016 MB Branko Mandić Nova Varoš SMEE Agribus iness Production of waffles and waffle-based edible products • Training • Mentoring&Consulting • TA for certification of HACCP • TA for design and redesign of packaging • Trade fair attendance (Munich, Germany) 2002 10 Full-time 20 Seasonal No 4.2.2016 Regional Development Agency – Zlatibor Slavko Lukić; Nataša Knežević Danijela Panić Biljana Radović Užice RDA RDA 10.2.2016 USAID Đorđe Boljanović Belgrade USAID USAID 10.2.2016 USAID Aleksandra Zorić Kržić Belgrade USAID USAID 11.2.2016 USAID briefing Miodrag Bogdanovic Belgrade USAID USAID 68 ANNEX III. PMP RESULTS AND TARGETS USAID/Serbia Mission-Level Indicators - Overarching Indicators Performance Indicator Methodology/ Data Source Baseline Y1 Result * (Jan-Oct 2014) Y1 Target Questionnaire 2 Result *(Nov 2014 – July 2015) Y2 Target I-1: Value of annual revenues of USAID client companies in targeted regions Client impact survey and cross reference check with APR $141M $124M $145 $126.79 M $152M I-2: Number of jobs created in targeted regions Client impact survey and cross reference check with APR 5,570 59 50 361 50 I-3: Number of business services increased in targeted regions Data input into NARD's Database 8 2 2 2 2 I-4: Number of beneficiaries with increased competitive skills Data input into NARD's Database n/a 318 324 165 144 I-5: Number of firms receiving capacity building assistance to export Data input into NARD's Database n/a 44 20 10 Objective 1: Strengthening SMEE's Performance Indicator Methodology/ Data Source Baseline Y1 Result * (Jan-Oct 2014) Y1 Target Questionnaire 2 Result *(Nov 2014 – July 2015) Y2 Target 69 O1-1: Number of SMEE's receiving USAID assistance Data input into NARD's Database n/a 122 120 121 180 O1-2: Number of USAID client SMEE's entering new markets Data input into NARD's Database n/a 9 10 20 O1-3: Number of USAID client SMEE's exhibiting at International Trade Fairs Data input into NARD's Database n/a 9 10 19 20 O1-4: Value of contracts signed by USAID client SMEE's exhibiting at International Trade Fairs Data input into NARD's Database n/a $91,956 $100,000 $263,250 $150,000 O1-5: Number of USAID client SMEE's introducing international standards Data input into NARD's Database n/a 9 6 11 6 O1-6: Number of USAID client SMEE's adopting new technology and/or management practice Data input into NARD's Database n/a 15 6 35 6 Objective 2: Strengthening Associations/Clusters Performance Indicator Methodology/ Data Source Baseline Y1 Result * (Jan-Oct 2014) Y1 Target Questionnaire 2 Result *(Nov 2014 – July 2015) Y2 Target O2-1: Number of associations/clusters receiving USAID assistance Data input into NARD's Database n/a 6 4 8 4 O2-2: Value of joint revenues of USAID client associations/clusters Client impact survey and cross reference check with NARD's $33M $26.6M $35M $0.00 M $40M 70 Database O2-3: Number of new members joining USAID client associations/clusters Client impact survey and cross reference check with NARD's Database 47 2 4 10 O2-4: Value of contracts signed by USAID client associations/clusters exhibiting at International Trade Fairs Client impact survey n/a n/a n/a $50K O2-5: Number of USAID client associations/clusters exhibiting at International Trade Fairs Data input into NARD's Database n/a n/a n/a 6 O2-6: Number of trained managers of USAID client associations/clusters Data input into NARD's Database n/a 6 4 4 Objective 3: Encouraged Women and Youth Entrepreneurship Performance Indicator Methodology/ Data Source Baseline Y1 Result * (Jan-Oct 2014) Y1 Target Questionnaire 2 Result *(Nov 2014 – July 2015) Y2 Target O3-1: Number of women and youth start-ups initiated thanks to USAID assistance Data input into NARD's Database n/a 0 10 32 40 O3-2: Number of women and youth receiving grant funds Data input into NARD's Database n/a 0 32 36 33 71 O3-3: Number of women and youth attending business training Data input into NARD's Database n/a 190 200 80 Objective 4: Enhanced Business Development Services of NARD and RDA's Performance Indicator Methodology/ Data Source Baseline Y1 Result * (Jan-Oct 2014) Y1 Target Questionnaire 2 Result *(Nov 2014 – July 2015) Y2 Target O4-1: Increased number of services by NARD and RDA's in targeted region Data input into NARD's Database 8 2 2 2 2 O4-2: Client satisfaction rating by USAID clients of NARD and RDA's business services Client Satisfaction Survey n/a 4.76 n/a 4.82,00 TBD O4-3: Increased number of NARD and RDA's clients in targeted region Data input into NARD's Database 52 290 280 200 140 72 ANNEX IV. STATEMENT OF WORK USAID/Serbia is planning a mid-term performance evaluation to take place at the completion of the first phase of its 4-year $6.5 million Support to Private Sector Development in South and Southwest Serbia Project (PSD). PSD was designed to support the development of the private sector in two of the poorest regions of Serbia, while serving as a learning laboratory for transitioning to a new development paradigm as envisioned in USAID Forward/Local Solutions. PSD was launched in June 2013, as cooperation with a host country government implementing Government to Government (G2G) agreement. Through the Private Sector Development project (PSD), USAID/Serbia aims to expand and deepen the gains made under past and current initiatives funded by USAID/Serbia (i.e. USAID Agribusiness project and USAID Economic Security project (ES)). In order to increase the probability of having a sustainable impact that will endure beyond USAID’s current assistance, USAID/Serbia has identified National Agency for Regional Development (NARD), a host government agency that has the absorptive capacity to adapt and build upon the gains accomplished to date. The purpose of this evaluation is two-fold. First, it will assess the extent to which the project (including all of its awards) achieved or is on track to achieving its intended development results related to strengthening the capacity of the private sector in targeted regions. While this requires an understanding of the performance of individual implementing mechanisms, the evaluation should put greater emphasis on the higher-level results of these activities referring to the CDCS’s IR 2.2 Economic Opportunities in Targeted Regions, and how their inter-relationship supported or hindered the achievement of those higher-level results. In order to monitor and evaluate performance of the PSD project, NARD developed Performance Monitoring Plan (PMP), which includes causal chain that links project activities to USAID/Serbia’s Development Objective: “Conditions for Broad-Based Inclusive Economic Growth Improved.” The causal chain clearly shows the relationships between the PSD project’s activities and targeted results. In this causal chain, the following four project objectives are set at the PSD project level: Objective 1: Strengthening small-to-medium size enterprises and entrepreneurs (SMEE’s) Objective 2: Strengthening business Associations/Clusters Objective 3: Encouraged Women and Youth Entrepreneurship Objective 4: Enhanced Business Development Services of NARD and RDA’s The project activities and assistance under each objective are designed to produce, in the first instance, the outputs, such as the numbers of SMEE’s and associations/clusters receiving assistance, number of trained managers of the client associations/clusters, number of women and youth receiving grant funds, number of client firms entering new markets, etc. These outputs, in turn, will lead to outcomes such as value of annual revenues of USAID client companies in the targeted region, new jobs, increased business services, etc. Both outputs and outcomes have largely been defined by USAID/Serbia to lead towards the following results: I. PURPOSE 73 IR 2.2: Economic Opportunities Increased in Targeted Regions Sub-IR 2.2.1: Economic and Business Development Services Enhanced in Selected Regions Sub-IR 2.2.2: Workforce/ Entrepreneurship Skills Improved Second, the evaluation will assess the implementation approach utilized by PSD and how effective it was in facilitating the transition to direct implementation by local partners. In particular, the evaluation will assess the capacity development role of the G2G implementing partner and the extent to which it contributed to the capability of USAID’s local partners. The results of the evaluation will be used by USAID/Serbia and shared with partners, from the GoS and other stakeholders/donors for managerial and design purposes. The results will also be of interest to a broader audience of USAID/Washington stakeholders as they continue to roll out reforms and learn from the global experience in implementing Local Solutions. Identifying Information: Project Title: Private Sector Development Project Number(s): N/A Project Performance Period: 4-year Project Funding: $6.5 million Implementing Organizations: National Agency for Regional Development (NARD) and its network of accredited Regional Development Agencies (RDAs) present in two of the poorest regions in Serbia. Contract Officer‘s Technical Representative (COR): Djordje Boljanovic Background and Context Private Sector Development (PSD) project was developed in the context of both a major re-alignment of private sector in the poorest regions of Serbia and a major paradigm shift in how USAID does business, namely the implementation of USAID Forward/Local Solution. As existing firm-level support projects neared their end, USAID/Serbia’s Economic Growth Office analyzed the current needs for support for private sector development. The findings drove the development of the Private Sector Development Project. More detail is included in the approved Concept Paper attached as Annex E, along with a map of the targeted regions in Annex A. The purpose of the project is to expand market linkages among businesses in the targeted regions and to connect them to the broader Serbian and regional markets through the improved capacity of public and private business services to provide support. Serbia’s Role in the Region and Current Economic Performance Improving the economic performance of Serbia is critical both from the country’s perspective, and from that of the region. Serbia is also pivotal to the regional integration of the Balkans. The 2011 EC Progress Report noted Serbia’s strategic location as important, geo-politically and economically, as II. Project Overview 74 Serbia plays a stabilizing role in the Western Balkans. However, the continuing disenfranchisement in the South and regional economic disparities continue to prevent the country from reaching its full economic potential and indeed still have the potential to destabilize the country. The current GDP level of Serbia remains below its equivalent 1989 level, and in fact lags the furthest behind of all Balkans countries in this generally accepted litmus test of economic progress. The average wage in May 2012 was under $500 nationwide and the overall unemployment rate hovers around 25 percent. Salaries in the economically depressed southern regions of the country are even lower than the national average, with the unemployment rate running as high as 70 percent in southern Serbia, leading to substantial outmigration and requests for asylum in the EU. With this uneven economic performance, Serbia will find it extremely difficult to continue on a steady path towards EU accession. If the economic conditions continue to deteriorate in the targeted regions, the resulting tensions could slow Serbia’s EU accession process and even reverse the gains the country has made since its emergence from the wars of the 1990s. It is little surprise that the 2012 EC Progress Report explicitly noted the need for “further state commitment to economic development in South Serbia and Sandzak.” Underdevelopment of South Serbia and Sandžak As the above statistics show, the development of the regions of South Serbia and Sandzak lags behind the rest of the country and is hindered by numerous socio-economic and political factors and trends. For instance, out of a total of 160 municipalities, Serbia’s Office for Sustainable Development of Underdeveloped Areas has classified 46 as “underdeveloped,” of which 40 have been further classified as “devastated” (i.e., their development levels are less than half the national average). While there are municipalities throughout Serbia that lag behind, the greatest concentration is by far in the targeted regions. Half of the devastated municipalities are in South Serbia alone, and a majority of the remaining underdeveloped municipalities are in Sandzak. Rural Poverty The areas of South Serbia and Sandzak are also predominantly rural, posing an additional challenge to their development. Although rural poverty was on the decline in Serbia in 2008, rural areas, including those in South Serbia and Sandzak, have suffered disproportionately from the economic crisis. As a result, the percentage of the population falling below the consumption-based absolute poverty line in rural areas increased from 7.5% to 9.6% in 2009, which is twice as high as in urban areas, 4.9%. An added challenge to the development of rural areas is that the human resources needed to improve this situation are also scarcer, given the extensive out-migration to urban areas and abroad. Vulnerability to Ethnic Tensions Serbia’s increasing regional disparities are particularly troubling in the targeted regions given the recent history of inter- and intra-ethnic conflicts that have marked both South Serbia and Sandzak. The economic marginalization, geographic isolation and poor infrastructure of these regions are compounded by political tensions as well as a pervasive perception that the regions are ignored by the central Government of Serbia. All of these are serious obstacles which dis-incentivize the private sector investment that is necessary to drive economic growth and create jobs. Context: USAID and NARD, based on their respective experience, technical expertise and best practices in implementing development activities, launched project to strengthen private sector firms in the 75 underdeveloped, isolated and vulnerable targeted districts in South and Southwest Serbia. By helping to create linkages between businesses in the targeted region and new and expanded markets, the project increases opportunities for sales and exports and create jobs. As the project is implemented by NARD, businesses in the targeted regions also benefit from linkages with their respective Regional Development Agencies, which have a mandate to support private sector development in their geographic regions. And finally, through NARD the project increases the linkages between businesses in the isolated targeted regions and these national level support entities. Given their importance to the sustainability of rural development, particular attention is focused to the involvement of youth and women in the selected municipalities and disaggregation of the data collected during evaluation. Targeted regions: 12 Municipalities in South and South-west Serbia PSD aims to strengthen private sector firms in the underdeveloped, isolated and vulnerable targeted districts in South and South-west Serbia, with the overall goal: "Businesses in targeted regions strengthened through broader linkages." PSD consists of the following three components: Component 1: Support Private-Sector Development in Target Municipalities by Working with Businesses in High-Potential Sectors: Assist businesses to better understand and serve local, regional and international markets; develop new products and improve the quality of existing and traditional products; lower production costs; broaden networks of suppliers; and increase operational scale and profitability. Component 2: Strengthen Government of Serbia Private-Sector-Development-Support Entities to Open New Markets and Create Linkages between Businesses: Support improved flow of information and opportunities between the target regions and governmental institutions responsible for supporting private-sector development to increase exports and broaden market opportunities. Component 3: Supplementary support to enhance the competitiveness of the SME sector. Overall Objective: Through Component 3, USAID/Serbia provides additional technical support to NARD and the RDAs in implementing Components 1 and 2. This support includes technical assistance in developing annual work-plans, small grants manual, environmental compliance documents, annual reports, PMP, etc. Components 1 and 2 are managed by NARD, while Component 3 is implemented by other partners and managed by USAID/Serbia. NARD is USAID's lead partner, and the umbrella organization of 76 the project. The RDAs are partner institutions with the field work role. During inception and implementation, PSD project considered activities funded by other USAID funded activities, such as Serbia Local Development Project (SLDP), Junior Achievement Serbia (JAS), other donors, etc. Other NARD and RDAs partners ( Office of sustainable development for underdeveloped areas, Coordination Body for the Municipalities of Presevo, Bujanovac and Medvedja, etc) also contributed and add value to the Project. USAID Forward/Local Solutions: In order to increase the impact, cost-effectiveness and sustainability of the new PSD project, USAID/Serbia fully embraced the USAID Forward and Implementation and Procurement Reform (IPR) efforts. The project’s activities are implemented by the National Agency for Regional Development (NARD) through its network of existing Regional Development Agencies (RDAs). In this way, the project contributes to USAID/Serbia’s goal of increasing the use of host country systems and local organizations in order to maximize resources and build local capacity. By working directly through and with these host-country entities, the project supports private sector development, and also: Facilitates greater access to the host country’s regional and national private sector support resources for firms in the targeted regions; Builds the capacity of Government of Serbia (GoS) entities to provide valuable support to the private sector; and, Develops their ability to implement donor programs and comply with rigorous requirements for deliverables and reporting. Project Goal and Objectives Project Purpose: Expand market linkages among businesses in the targeted regions, connect them to the broader Serbian and regional export markets through improved business services. Broader business linkages will be created and the capacity of firms and institutions will be increased on three levels: Market (Value Chains): Increased linkages between businesses in these target districts and markets in the rest of Serbia and beyond; National (within Serbia): Increased national linkages and support for businesses through the Government of Serbia’s national business support entities; Regional (Balkans): Increased regional linkages and support for businesses through existing Serbian entities that are engaged in private sector development. The overall goal of the project is: “Businesses in targeted regions strengthened through broader linkages.” This goal will be achieved through three project components, two of them led by NARD and component 3 managed by USAID Serbia. Component 1: Support to private sector development through NARD and the Regional Development Agencies (RDAs) (Novi Pazar, Uzice, Kraljevo, and Leskovac, in conjunction with the Small Business Development Agency of Vranje) Overall Objective: NARD and the RDAs to identify key sectors that have the potential for increased income and employment generation, and then work with the stakeholders in these sectors to strengthen their businesses. The activities will help the businesses to: better understand and serve local/regional markets; improve the quality of traditional products, as well as develop new ones; 77 lower the cost of production; and, broaden the network of suppliers, increasing the profitability and size of operations. USAID will ensure that lessons learned under the previous USAID projects are shared with NARD and the RDAs, including the methodology used, names and contact information of beneficiaries, list of consultants, and cluster action plans. Component 2: Strengthening of GoS national PSD-support entities to open new markets and create linkages between businesses (NARD in collaboration with SIEPA) Overall Objective(s): To ensure better two-way flow of information and to create opportunities between the targeted regions and GoS institutions dedicated to supporting private sector development in Serbia, especially the National Agency for Regional Development and the network of RDAs. In order to expand opportunities for businesses and regions, NARD will look for options to cooperate with the Serbian Investment and Export Promotion Agency (SIEPA) to engage client companies of PSD project to participate at international trade fairs and B2B events organized through cooperation between NARD and SIEPA. Component 3: Supplementary support to enhance the competitiveness of the SME sector Overall Objective: Through Component 3, USAID/Serbia’s EG Office will provide additional support to NARD and the RDAs, in implementing Components 1 and 2. This support could include both capacity building and “surge capacity” to supplement the activities of NARD and RDAs. According to Activity Monitoring and Evaluation Plan (AMEP), the PSD project’s indicators are divided into the following two major categories: USAID/Serbia Mission-level Indicators – Overarching Indicators, and Project-level Indicators Each category includes relevant indicators; the mission-level “higher” indicators are set and pre-defined by USAID in accordance with the project’s scope, while the Project-level indicators are set by NARD and linked to each objective. The overarching and project-level indicators together provide quantitative and qualitative results of the PSD’s project assistance to SMEE’s and associations/clusters in the targeted regions. The approved AM&EP and the log-frame are presented as an Attachment IV. Alignment with Country Development Cooperation Strategy (CDCS) for Serbia Private Sector Development Project supports the above goals via USAID’s Development Objective 2, "Conditions for Broad-based Inclusive Economic Growth Improved." It specifically accomplishes this by supporting Intermediate Result IR 2.2 “Economic Opportunities Increased in Targeted Regions” the CDCS Result Framework. Finally, program is precisely defined under Sub IR 2.2.1 “Economic and Business Development Services Enhanced in Selected Regions” and Sub IR 2.2.2 “Workforce/Entrepreneurship Skills Improved”. Objective III. EVALUATION RATIONALE AND STATEMENT OF WORK 78 USAID/Serbia aims to conduct a full, evidence-based and independent mid-term performance evaluation at the completion of the Phase I of the Private Sector Development project. The primary audience for this evaluation will be USAID/Serbia and its PSD implementing partners. A secondary audience is USAID/Washington, who will use the results of this evaluation to inform its global efforts to identify lessons learned in implementing Local Solutions. Evaluation Questions As mentioned previously, the purpose of the evaluation is to assess both the results achieved and the effectiveness in USAID’s use of direct partnerships with local implementing partners. Evaluation questions include the following: To what extent has PSD achieved the expected overarching Mission-level results1 identified in the Activity Monitoring and Evaluation Plan? To what extent has the project resulted in improved competitiveness of SMEEs? To what extent has the project contributed to increased linkages between small businesses and regional/national governmental support organizations? How effective was PSD in creating the linkages among different PSD partners (including sub￾recipients) and across different mechanisms? What have been the strengths and weaknesses of USAID’s approach to G2G management? Request for Recommendations Drawing on PSD lessons learned, how should USAID best focus or target its limited assistance for the strengthening of the private sector businesses in targeted regions in the future? How can USAID Serbia strengthen the linkages between different PSD partners during phase II? In addressing these questions, the evaluation should pay particular attention to activities and results outside of Belgrade. The focus of evaluation should be on the targeted regions covering all 12 cities and municipalities in South and South-west Serbia. Evaluation Methodology It is anticipated that a mix of evaluation methodological approaches will be required to meet the requirements outlined in the Statement of Work section above. The evaluation team will first complete a desk study that will be used to establish an understanding of PSD’s activities and environment before arrival in Serbia. Suggested data sources include: (a) secondary data/background documents, (b) project plans, outputs, and reports, (c) key informant interviews ( with small businesses, business associations, RDAs) ,focus group discussions and (e) survey(s) of project stakeholders and beneficiaries. An emphasis will be placed on the collection of reliable empirical data and/or objectively verifiable evidence, as opposed to anecdotal evidence. Where surveys or interviews are used, appropriate sampling and questioning techniques will be utilized to ensure representative results; where references are made to data generated by PSD implementers and their partners, it will be complemented by references to independent data sources and any significant data differences must be explained. The evaluation will be conducted in Belgrade, as well as smaller towns and cities in the targeted regions of Serbia (South-east and South-west Serbia). The Evaluation Team Leader will provide data collection instruments for any planned interviews or surveys to USAID prior to field work. USAID reserves the right to approve /suggest additional questions. The evaluation should be backed up by relevant data and information gathered from 79 meetings with program partners and all the relevant stakeholders. The evaluation team should clearly demonstrate links between the data collected and the subsequent findings, conclusions, and recommendations drawn. Illustrative methodological approaches for each evaluation question and potential limitations are presented in Table One below. In developing an appropriate methodology, the evaluation contractor should consider these and other potential limitations and incorporate acceptable mitigation measures. In addition to these approaches, the Mission is looking for new, creative suggestions regarding this evaluation, and it is anticipated that the implementer will provide a more detailed explanation of the proposed methodology for carrying out the work. Table One: Overview of Illustrative Evaluation Methods and Limitations Evaluation Question Data Collection Methods Data Sources Data analysis method Limitations To what extent has PSD achieved, the expected overarching Mission-level results2 identified in the Activity Monitoring and Evaluation Plan? Key informant interviews; document review Project M&E plans and relevant project reports; Qualitative methods Key informant bias To what extent has the project contributed to the improved competitiveness of SMEEs? Key informant interviews; document review, mini-survey Relevant Project reports;relevant stakeholders, surveys Qualitative and statistical methods Key informant bias To what extent has the project resulted in increased linkages Between small Businesses and regional/national governmental support organizations? Key informant interviews; document review, mini-survey Relevant project reports;relevant stakeholders; surveys Qualitative and statistical methods Key informant bias How effective was PSD in creating the linkages among different PSD partners (including sub￾recipients) and across different mechanisms? Key informant interviews USAID/Serbia staff, PSD direct partners Qualitative methods Key informant bias 80 What have been the strengths and Weaknesses of USAID’s approach to G2G management for supporting the transition to local partners? Key informant interviews USAID/Serbia staff, PSD direct partners, GoS representatives Qualitative methods Key informant bias Available Data Sources: The evaluation contractor will have access to all performance data that has been collected and reported by implementing partners as part of their Performance Monitoring and Evaluation plan for Private Sector Development Project. The evaluation team should also review the following documents in preparation for the PSD evaluation: Relevant project reports, assessments, annual and life of project work plans. Annual performance reports of PSD partners and sub-partners. USAID/Serbia and Montenegro’s 2011-2015 Amended Strategy and 2013-2017 CDCS. In addition, the evaluation team is encouraged to consult with the following key informants and stakeholders: Meetings with USAID/Serbia’s Mission Director, Economic Growth Office Director, Program Officer, project COR and Alternate COR. Meetings with current/former PSD implementing partners in Serbia. Meeting with appropriate officials of the Government of Serbia’s National Agency for Regional Development, Regional Development Agencies and other relevant Government of Serbia officials. Meetings with other donors, including the EU Delegation and those donors engaged in private sector development. Meetings with small businesses and their associations in targeted regions of Serbia as beneficiaries of the project. Evaluation Stages: Background Review and Familiarization: Become familiar with PSD’s work, which includes reviewing: the contract agreements and all modifications; list of milestones; implementing partner’s annual reports; work plans and performance monitoring plans. In-Country Research and Evaluation: Field Work: Upon completion of a background review, the evaluation team will commence work in Serbia. The evaluation team will also meet with other donors, implementers and key government partners. The LOE allows for time in Belgrade to write a draft report. The evaluation team will provide out-brief to the Mission on key findings. The draft report will be submitted to the PSD COR and EG Office Director for review. Follow-up and Final Report: The evaluation team will receive comments on the draft report from USAID. The team will then incorporate these comments and provide a draft final draft report. USAID will provide final comments to the evaluation team upon which the team will then submit a completed final report to USAID. In addition to the final report, the evaluation team should prepare 81 and submit three communication products (success stories, impact blog, etc.) based on evidence that emerged during the evaluation. Up to 10 work days have been allocated for this final report writing time period, after returning from Serbia. The final report will be submitted to the PSD COR, EG Office Director, and evaluation COR. The evaluation team is required to submit all records from the evaluation (e.g. focus groups transcripts) and all quantitative data in an organized and machine-readable fashion and fully documented for use by those not familiar with the project or evaluation. The evaluation report and summaries must be submitted to the Development Experience Clearinghouse (DEC) within three months of completion by the evaluation COR at http://dec.usaid.gov. Agency criteria to ensure quality in the evaluation final report are presented in Attachment I. The contractor shall use all of these criteria in the preparation of the final report. The following deliverables will be required as a part of the PSD evaluation. In-brief meeting: The evaluation team will conduct an in-brief meeting with USAID Serbia Mission’s management in order to acquaint it with the work plan details (draft work plan to be presented at the meeting). Work plan: a detailed work plan to be submitted before arrival in country to be approved by USAID Serbia in which the evaluation’s methodology and activities will be clearly stated. Adjustments to this work plan will be made based upon comments and/or issues by USAID Serbia staff at the in-brief. The work plan should at a minimum include the following: The methodology for carrying out the evaluation; A draft schedule of targeted meetings and list of potential interviewees; Data collection and analysis plan (including interview questions and protocols); and Identification of potential major constraints. Out-briefing: The team will conduct an out-briefing to USAID covering its findings and recommendations at the conclusion of its field work in Serbia. The out-briefing will include a rough draft of the evaluation report. Draft Evaluation Report: The evaluation team will deliver an outline evaluation report prior to departing Serbia, highlighting the key findings and recommendations. Revised Draft Evaluation Report: The team will deliver a completed report including a four- to- six page executive summary and a full report of up to 30 pages. Additional material may be included as annexes. Final Evaluation Report: The evaluation team will incorporate feedback and provide its final report, with page length specifications as noted for the draft final evaluation report. Communication Products: The evaluation team will prepare at least three communication products that are supported by evidence gathered as part of the evaluation. This includes, but is not limited to, success stories and impact blog. IV. DELIVERABLES OF THE EVALUATION 82 The evaluation team shall consist of four members: two technical specialists, an interpreter / administrative assistant, and a driver. While it would be especially valuable that the each of the technical specialists have extensive and documented experience in conducting performance evaluations, it is a requirement that at least one specialist/team leader has this experience. Specific qualifications for each team member include the following: Evaluation Team Member/Private Sector Analyst: This analyst will serve as the team leader and will ultimately be responsible for the management of the team, the coordination of team activities, and the submission of the final report. The Private Sector analyst will serve as the expert in economic growth and will ultimately be responsible for evaluating the performance of the Project’s activities related to business enabling environment and competitiveness. Extensive and documented experience in program performance evaluation and knowledge related to business development, supply chains and economic growth in the Balkans and/or Eastern Europe is mandatory. The proposed individual shall meet the minimum level of academic and the work experience qualifications for the senior level expert. Evaluation Team Member/Regional Economic Development Analyst: This team member is expected to have excellent knowledge about economic growth-related policy reform challenges in countries in transition, along with familiarity with transitional economies. Documented experience in conducting performance evaluations is especially valuable. The proposed individual shall meet the minimum level of academic and the work experience qualifications for the Mid-level expert. Interpreter/Administrative Assistant: This team member will provide logistical, administrative, and clerical and translation support to the team throughout the evaluation. Special attention should be paid to this team member’s ability to translate technical language as it relates to the economics and business terminology. Driver. Implementation of this award will require two core staff members – one of these two core members will serve as the team leader – and an interpreter/administrative assistant. The team leader will be ultimately responsible for the management of the team, the coordination of team activities and the submission of the final evaluation; the other core team member will support the team leader in team coordination and management. In addition to the submission of detailed curriculum vitae for each of the proposed core personnel, the proposal shall include a summary of each proposed individual's experience specific to the following areas: Evaluation; Business development and market analysis; USAID experience; and Relevant regional experience. The evaluation team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest (see Attachment IV). VI. SUPPORT FROM USAID AND IMPLEMENTING PARTNERS V. TEAM COMPOSITION 83 To support the team’s initial literature review, USAID/Serbia will provide electronic copies to the contractor of all documents to be reviewed (listed above) one week prior to departure. USAID/Serbia will facilitate introductions with key informants where necessary and provide a list of recommended key informants and contact information. The evaluation team should not expect logistical support from USAID or its implementing partners. END OF SECTION C 84